A Terminal Queue That Keeps Batch API Costs Predictable

A Terminal Queue That Keeps Batch API Costs Predictable

The expensive part of a batch script often starts with an innocent change: replacing a sequential loop with a thousand concurrent requests. Throughput jumps, then timeouts appear, retries multiply, and you lose track of which results actually landed. My default now is a durable queue, a small worker pool, and a spending ceiling checked before each dispatch.

Concurrency does not change the price of a token. It changes how quickly you commit money and how much unfinished work exists when something breaks. The savings come from preventing duplicate calls, stopping bad batches early, and choosing sensible output limits.

Measure 100 Jobs Before Launching 10,000

Start with a representative sample, including the longest inputs. Record input tokens, output tokens, latency, failures, and whether each result passes your checks. For classification or extraction, try a candidate such as DeepSeek V4 Flash, then compare difficult cases with Claude Haiku 4.5. Choose using cost per accepted result, rather than token price alone.

In Fluxo, usage is billed per real token, including cache reads and writes, with no monthly fee. Estimate the batch using the applicable rates for each token category. Shared prompts may benefit from caching where supported, but cache writes and reads still belong in the estimate. Avoid treating repeated prompt text as automatically free.

• Set output limits from observed successful responses, leaving reasonable headroom.
• Reject malformed inputs locally before they consume any paid tokens.
• Reserve part of the budget for failures and difficult records.

Use Four Workers and a Bounded Waiting Queue

For a first run, use four workers and keep perhaps 100 jobs in memory. Store the remaining work in SQLite or another durable store. This bounds memory use and gives the process a restart point. Four is a starting experiment, not a Fluxo account limit; your actual allowance depends on the service and model.

Track both request throughput and token throughput. Four short classification calls behave differently from four document summaries. If token limits bind first, use a token-aware dispatch budget alongside the worker count. Increase concurrency gradually while watching accepted results per minute, tail latency, and throttling.

A waiting job should contain an identifier, input reference, configuration version, attempt count, and status. Mark success only after saving the result durably. On restart, reconcile interrupted jobs before sending them again.

Make Retries Spend from a Separate Budget

Retry transient failures with exponential backoff and jitter, honoring Retry-After when supplied. Do not immediately retry authentication errors or invalid requests. Put delayed retries back into the queue so one failing record cannot occupy a worker indefinitely.

A timeout is ambiguous: the upstream request may have completed even though your client never received the response. Do not assume a retry is free or deduplicated. Use idempotency only where explicitly supported, and keep a local result ledger to prevent resending work you already saved.

• Cap each job at three total attempts before manual review.
• Pause dispatch when throttling or transient failures exceed your threshold.
• Stop new requests when estimated committed spending reaches the ceiling.

Include in-flight requests in that ceiling. If you check only completed charges, active workers can push the final total beyond your target. Reserve conservatively before dispatch, then reconcile against reported usage.

Keep Provider Switching Behind the Same Queue

Fluxo gives this worker setup one key and one base URL, https://api.fluxorelay.com, for Claude, GPT, Gemini, DeepSeek, Kimi, GLM, Qwen, Grok, and more. It works with Claude Code, Codex, Gemini CLI, and any OpenAI SDK. Keep queue state outside those clients so changing a model or restarting your terminal does not erase progress.

If you want to try this workflow, create a Fluxo account and run your sample batch first. Users also earn a referral share when people they invite pay; track that separately from workload costs so your measurements stay useful.

Before the overnight run, interrupt the sample deliberately and verify that it resumes correctly. A queue earns its keep when your laptop sleeps, a request stalls, or a result fails validation—and you can continue without paying to repeat the whole batch.