A $63 Mistake in Production: How an AI Agent's Loop Escaped Without Guards

$63 / 31M tokens is about $2.03 per million blended across those 243 runs, so each pass averaged roughly 127k tokens — exactly the kind of loop where a level-triggered poll quietly becomes a meter. Two cheap guards that would have caught this before morning: a hard spend or token ceiling on the actuator itself (kill the Oban job when the cumulative prompt+completion for that trigger crosses a few dollars), and an alert on “same review thread / same feedback hash seen N times,” because immutable GitHub review history will never clear a level check. Seat plans hide this until the invoice; an API key at least makes every retry visible as line items, but only if something is watching the running total rather than a single-request freshness flag that can be skipped. The attempt counter resetting to 0 is the scarier bug than the missing alert — once the safety latch disarms itself, redundancy stops being free.

1 Like