Stop Terminal Retry Storms After AI API 429s
A 429 in your terminal is a scheduling signal
Your coding agent launches several requests, a background indexing job starts, and suddenly every terminal prints 429. Adding a sleep inside each worker usually moves the spike rather than fixing it. The real problem is coordinating demand across requests, processes, and retry layers.
I treat a 429 as feedback to the scheduler, not proof that a request will succeed after one second. First inspect the response body and headers: temporary request throttling, token limits, concurrency limits, and exhausted quota can require different actions. Do not blindly retry a quota or billing failure.
Give exponential backoff a deadline and full jitter
For retry index n starting at zero, calculate a local ceiling of min(cap, base × 2^n), then sample a random delay uniformly between zero and that ceiling. A reasonable starting configuration is a 500-millisecond base, a 20-second cap, and at most four retries. These are tuning inputs, not universal provider limits.
Full jitter matters because deterministic exponential delays synchronize workers. Twenty processes sleeping for exactly two seconds become twenty processes retrying together. Randomizing only the initial delay is insufficient; draw a fresh delay on every retry.
- Set an overall deadline that includes requests and sleeps.
- Stop when the remaining deadline cannot accommodate another useful attempt.
- Make both sleeping and network operations respond to cancellation.
- Record exhausted retries separately from requests that eventually succeed.
For an interactive editor action, a 15-second deadline might be more appropriate than a generous background-job policy. A capped delay does not cap total execution time, especially when individual requests also have long timeouts.
Respect Retry-After without creating another retry wave
When present, parse Retry-After as either delay seconds or an HTTP date. Clamp past dates to zero and handle malformed values explicitly. Treat the valid server delay as a minimum: wait max(server delay, local jitter delay), then optionally add a small positive jitter to spread clients receiving the same reset time.
If that minimum exceeds your remaining deadline, stop rather than silently retrying earlier. Use a monotonic clock for elapsed-time budgets; interpreting an HTTP date still requires wall-clock time. Do not assume every upstream or relay exposes identical reset headers, and do not invent missing limits.
Put retry ownership in one layer
Before adding a wrapper, inspect your SDK's retry configuration. Application retries multiplied by SDK retries can turn one editor action into many attempts. Disable one layer or explicitly account for the combined budget. Also check whether your agent runtime retries independently.
With Fluxo, one key and one base URL, https://api.fluxorelay.com, cover multiple model providers. It works with Claude Code, Codex, Gemini CLI, and any OpenAI SDK. That simplifies configuration, but it does not make every upstream rate-limit policy identical. Keep retry decisions based on actual responses rather than assumptions about the shared endpoint.
Coordinate token budgets across terminal workers
A per-process semaphore cannot protect a key shared by several terminals. Use a shared limiter when needed, with separate accounting for request rate, estimated token demand, and in-flight concurrency. After repeated 429s, reduce concurrency; restore it gradually after a stable success window.
For example, queue background requests to Claude Sonnet 5.5 behind interactive work. If you deliberately route eligible work to GPT-6.1 Sol, treat that as a policy decision with quality and cost checks, not an automatic assumption that another model has spare capacity.
Fluxo bills per real token, including cache reads and writes, with no monthly fee. Track usage alongside retries, but do not assume every failed attempt is billed or free. Referral shares accrue when invited users pay; keep that separate from operational request-cost accounting.
Protect streams and measure retry amplification
A rejected request before streaming begins differs from a stream interrupted after partial output. Retrying the latter can duplicate text or repeat tool-driven side effects. Never assume generation requests are idempotent; deduplicate application actions and use idempotency mechanisms only where explicitly supported.
Log attempt number, delay source, status, model, elapsed time, and any available request identifier without logging secrets. Watch attempts per successful action and deadline exhaustion, not just final error rate. To try this policy in your own terminal workflow, you can create a Fluxo account.