An automated trigger fired every two minutes for 8 hours. 243 agent runs. 31M tokens. $63 wasted. The problem: nobody noticed until morning.
This is a case study in production safety when AI agents are on the loop — and a cautionary tale about human approval gates.
The technical breakdown:
An Ash/Oban cron trigger polling GitHub every 2 minutes — seemed fine on review
The check was level-triggered (state-based) instead of event-triggered
GitHub review feedback is immutable history, not a dismissable flag
So the agent kept trying to fix the same feedback, over and over
No cost counter, no alert, no visibility — just a morning invoice
Why the existing guards failed:
Freshness check existed but lived on an unreachable code path
Attempt counter got reset to 0 on every pass (accidentally disarmed)
Branch order made recovery impossible once one reviewer approved
The real lesson: when you put an expensive actuator (LLM tokens) on a loop, “harmless redundancy” stops being a category. Every pass has to earn itself.
Full postmortem:
Built with Elixir/Phoenix/Ash, and the hardest part wasn’t the code — it was the code review.
$63 / 31M tokens is about $2.03 per million blended across those 243 runs, so each pass averaged roughly 127k tokens — exactly the kind of loop where a level-triggered poll quietly becomes a meter. Two cheap guards that would have caught this before morning: a hard spend or token ceiling on the actuator itself (kill the Oban job when the cumulative prompt+completion for that trigger crosses a few dollars), and an alert on “same review thread / same feedback hash seen N times,” because immutable GitHub review history will never clear a level check. Seat plans hide this until the invoice; an API key at least makes every retry visible as line items, but only if something is watching the running total rather than a single-request freshness flag that can be skipped. The attempt counter resetting to 0 is the scarier bug than the missing alert — once the safety latch disarms itself, redundancy stops being free.
Exactly. What saved me that time is that it was subscription not API usage. So this time it just hit a session usage limit and that’s it But with API usage it could be costly.
