Stop Queue Retries from Doubling Your AI API Bill
Stop Queue Retries from Doubling Your AI API Bill
The expensive bug usually looks boring: a worker sends an AI request, waits, then loses its connection. The queue retries. Both requests finish upstream, but your application records only the second answer. You bought two generations and shipped one.
At-least-once delivery is normal queue behavior. The money-saving move is separating delivery attempts from the logical operation they represent. A retry should recover existing work whenever possible, not automatically purchase another generation.
Give each billable operation a durable identity
Create an operation record before enqueueing anything. Store an operation ID, tenant ID, request fingerprint, status, attempt count, and eventual result reference. Put only the operation ID in the queue message. A unique database constraint should prevent two records for the same business action.
For example, summarizing document revision 42 with Claude Sonnet 5 is one operation. Reprocessing revision 43 is another. Include the model, prompt version, generation settings, and input revision in the fingerprint. Do not deduplicate solely by prompt text: two customers can legitimately submit identical content.
- Use a stable operation ID across every queue redelivery.
- Reject reused operation IDs when their request fingerprints differ.
- Keep completed results longer than your maximum queue redelivery window.
- Make intentional regeneration explicit by creating a new operation.
Claim work atomically before opening the connection
Workers should claim an operation with an atomic database update or transaction. If its status is complete, return the stored result and acknowledge the message. If another worker holds a valid lease, defer delivery rather than starting another request.
A lease needs an owner, expiration, and heartbeat. Use a fencing token so an expired worker cannot overwrite a newer worker's result. Set the lease around realistic request durations, not your happiest local benchmark.
This prevents concurrent local execution, not every duplicate upstream charge. A worker can lose its lease while its request continues remotely. Fencing protects your database; it cannot cancel computation already accepted elsewhere.
Treat read timeouts as unknown outcomes
A timeout means you lack an answer, not necessarily that execution failed. Record ambiguous attempts separately. If the endpoint offers documented idempotency or request-status retrieval, persist the relevant key or identifier before sending and reuse it within the documented scope and retention period.
Do not assume an arbitrary Idempotency-Key header makes a generation endpoint safe. A relay, provider, and endpoint may have different guarantees. Without upstream deduplication or reconciliation, a crash between remote completion and local persistence leaves an unavoidable duplicate-risk window.
- Retry clearly transient failures with exponential backoff and jitter.
- Respect Retry-After when present, alongside documented endpoint retry guidance.
- Stop automatic retries for authentication failures and invalid requests.
- Route ambiguous expensive attempts to reconciliation before purchasing another generation.
Commit results before acknowledging queue messages
Persist the response and completion status in one transaction, then acknowledge the job. If acknowledgment fails, redelivery finds the completed record and exits cheaply. For downstream notifications, write an outbox event in that same transaction; otherwise, a recovery path may send duplicate emails or trigger duplicate workflows.
Also inspect SDK retries. Three queue attempts combined with three total SDK attempts can produce nine requests. Choose one layer to own the retry budget, and cap attempts, elapsed time, output tokens, and estimated spend per operation.
Measure duplicate spend across your relay
Fluxo gives this setup one key and one base URL, https://api.fluxorelay.com, for Claude, GPT, Gemini, DeepSeek, Kimi, GLM, Qwen, Grok, and more. It works with Claude Code, Codex, Gemini CLI, and any OpenAI SDK. Keep your operation ledger application-owned when switching models or clients.
Fluxo bills per real token, including cache reads and writes, with no monthly fee. Cache hits can reduce some costs, but they do not make duplicate generations free. Track usage per attempt and operation, including ambiguous attempts awaiting reconciliation. A cheaper fallback such as DeepSeek V4 Flash still deserves a fresh budget check.
If this fits your workflow, create a Fluxo account and test the crash-after-response case before production. Users also earn a referral share when people they invite pay; track that separately from engineering savings. The win here is fewer accidental purchases, not merely cheaper retries.