A Terminal Playbook for Cheaper AI Load Tests
A Terminal Playbook for Cheaper AI Load Tests
The expensive way to test an AI endpoint is to point a load generator at production credentials, crank up concurrency, and watch tokens disappear. You learn that something breaks, but not necessarily where. My cheaper approach separates application capacity from model behavior, then spends real tokens only on questions a stub cannot answer.
Write the budget before writing the load script
Start with a small test manifest: target model, prompt fixtures, output limit, arrival rate, maximum in-flight requests, duration, and dollar ceiling. Keep it beside the load script in your repository. For a first pass, I might allocate $10 and stop admitting requests at $8, leaving room for requests already running. Those are test budgets, not model price estimates.
Estimate cost from input tokens, generated tokens, and applicable cache-read and cache-write rates. Use current pricing, not a remembered blended rate. An output limit bounds generation, but it does not bound input costs or make retries free. Track estimated spend during the run and reconcile it against actual usage afterward.
Load-test your plumbing without buying model output
First, replace the upstream model call with a local stub behind the same application interface. Make it simulate slow first tokens, uneven streaming chunks, rate limits, and dropped connections. This exercises authentication, queues, connection pools, streaming proxies, and cancellation without paying for generation.
- Verify that disconnected clients cancel upstream work instead of abandoning it.
- Keep queues bounded so overload produces controlled failures, not memory exhaustion.
- Disable automatic retries initially so failures remain visible and measurable.
- Confirm that secrets and full user prompts never enter load-test logs.
Push the stub harder than your expected launch traffic. If your application falls over here, buying more real tokens will not explain much. Fix the plumbing first.
Spend real tokens on a representative prompt mix
Next, replay sanitized fixtures through the actual endpoint. Include short questions, long context, cache-friendly shared prefixes, and unique requests. Preserve the rough production mix instead of repeating one tiny prompt. Otherwise, you can accidentally benchmark an unusually favorable cache path.
Fluxo gives me one key and one base URL, https://api.fluxorelay.com, across Claude, GPT, Gemini, DeepSeek, Kimi, GLM, Qwen, Grok and more. It works with Claude Code, Codex, Gemini CLI, and any OpenAI SDK. For this test, I would compare DeepSeek V4 Flash and Claude Sonnet 5.5 using their documented model identifiers, changing one variable at a time.
Fluxo bills per real token, including cache reads and writes, with no monthly fee. That makes cache behavior part of the budget worksheet, not just a latency optimization. Measure cold-prefix and reused-prefix runs separately, and verify reported cache usage rather than assuming repeated text guarantees a hit.
Ramp arrivals while watching streaming latency
Run a smoke test first, then short stages at one, two, four, and eight concurrent requests. After that, test your expected arrival rate with a maximum in-flight limit. Concurrency-only tests can hide overload because slow responses automatically reduce how quickly new requests arrive.
- Record time to first token separately from total response duration.
- Report p50 and p95 latency alongside completed requests per second.
- Count timeouts, rate limits, cancellations, retries, and incomplete streams separately.
- Attach fixture identifiers and token usage to every request record.
Keep generation settings and output limits consistent between runs. Restore retries only after establishing a baseline, with a strict attempt limit and backoff that honors server guidance. A retry storm can turn one capacity problem into a much larger bill.
Make launch readiness a repeatable budget check
Define pass criteria before seeing the graphs: acceptable p95 first-token latency, error rate, queue depth, and cost per completed task. Stop immediately when the spend threshold or safety limits trigger. Save the manifest, results, and application revision so the next comparison means something.
If you want to run this playbook through one relay, you can create a Fluxo account. Keep referral income separate from performance economics: users earn a referral share when people they invite pay, but launch decisions should stand on measured request costs. The useful deliverable is not a heroic traffic spike. It is a repeatable command, a bounded bill, and evidence that your endpoint survives the traffic you actually expect.