Skip to content

devlog/token-math/cache-hits-in-the-terminal-structuring-prompts-for-reuse-1104.md

Token mathEN

Cache Hits in the Terminal: Structuring Prompts for Reuse

Prefix drift between editor saves and terminal reruns

Cache reuse in an editor loop rarely fails because of the model or the network. It fails because the first few hundred tokens of the request keep changing. Providers keep a cached block for a short window and read it back when the next request starts with the same tokens. In Claude Code or Codex that means one regenerated header, one reordered section, or one insert at the top of the context resets the chain and charges you full input price for text you already paid to write. The cost shows up as a falling hit rate that looks like a provider problem.

The mistake is treating the prompt like a document you can edit in any order. A cache-friendly prompt is a fixed pipeline: stable material first, variable material last. Inserting a line at the top changes every token after it, while appending a question at the bottom leaves the prefix untouched and lets the model read what it already stored. That habit usually saves more than switching models or rewriting your whole template.

  • Keep the system prompt, tool definitions, and long reference documents at the very top of every request so they stay inside the reusable prefix.
  • Append questions, diffs, and new user turns at the end of the conversation instead of inserting them anywhere earlier in the message list.
  • Resist reordering messages between runs because the cache matches the token sequence and not the logical content of each message.
  • Freeze the exact header block your agent writes and version it deliberately instead of letting a template regenerate it on every invocation.

Once the prefix is stable, the rest of the loop gets cheaper on its own. A long session where you only append turns will keep reading the same block until it expires, and the per-turn cost drops to the tokens you actually changed.

Volatile values placed above stable instructions

Nothing destroys a prefix faster than a timestamp near the top. Session ids, random seeds, git commit hashes regenerated per build, test counts printed by a previous command, and a clock string all sit above your instructions and change on every call. A single differing byte invalidates everything after it, so a two-token timestamp can cost you a cached context of tens of thousands of tokens.

Move every changing value into a single block at the end of the request. If you must include environment details, collect them once per session and reuse the same rendered block instead of rebuilding it per call. Deterministic ordering matters here too: if your serializer emits map keys in a random order, the JSON text differs even when the data is identical.

  • Place timestamps, run ids, and machine names in the last user message so they never touch the cached prefix.
  • Render environment details once per session and paste the same rendered block into every follow-up call.
  • Sort object keys and normalize whitespace in serialized tool arguments because byte-level differences break prefix matching.
  • Strip terminal noise such as progress bars, escape sequences, and build counters before pasting command output into a prompt.

This is also why prompt templates should separate constants from slots. Constants live in the fixed prefix, slots are filled at the end, and a debug mode prints both halves so you can see exactly where the boundary sits.

Tool schemas and system text rewritten on every call

Agent frameworks love to rebuild their tool list per turn. A tool gets added, removed, or renamed; the schema is regenerated with slightly different whitespace; the system text mentions the current date. All of that sits above the conversation and breaks reuse. If you switch between Claude Code and an OpenAI SDK script in the same session, the two shapes will not share anything anyway, so treat them as separate pipelines with their own stable prefixes.

Version your system prompt and tool schemas explicitly. When you change them, accept the cold cache and measure the new baseline instead of mutating small pieces every run. Keep tool names and descriptions long lived, and put experimental tools in a second block that you can append rather than splice into the middle.

  • Declare your full tool set once and keep the same order and formatting for the entire session or job.
  • Bump a version string inside the stable prefix when schemas change instead of editing descriptions in place on every run.
  • Avoid injecting the current date or locale into system text because those values change across calls and invalidate the prefix.

Small models in a tight loop benefit most from this discipline. A cheap fast model with a warm cache beats a bigger model that reprocesses the whole context on every turn.

Chunk boundaries and the minimum reusable block

Cached blocks usually have a minimum size and a short lifetime, so a tiny prompt may never become reusable and a long pause between turns may expire the block anyway. Batching related calls inside one window is worth more than spreading them across minutes. When you grow context, append to the end; trimming from the front rewrites the prefix and triggers a fresh write.

Another frequent mistake is duplicating the same document in two messages to keep it near the question. Duplication adds tokens and adds a second write. Reference it once, early, and let the later turns refer to it by name. If you do need to restate a key fact, put the restatement at the end where it costs only incremental tokens.

  • Batch related requests into a single working window so the cached block stays alive between calls.
  • Append new context at the end and never trim or reorder earlier messages during an active session.
  • Store long documents once near the top and reference them later instead of repeating them in every turn.

Cache reads, cache writes, and the hit rate you actually log

You cannot improve what you do not measure. Most SDK responses expose three token buckets: plain input tokens, cache creation tokens, and cache read tokens. Add them up per session and divide the read bucket by the total input to get a real hit rate. A healthy long session in a terminal loop often sits well above eighty percent after the first few calls, while a fresh prompt every turn stays near zero no matter which model you pick.

Log the buckets next to the model name and the session id so you can spot regressions. A sudden drop usually means someone changed the system text, inserted a timestamp, or reordered tool definitions, and the numbers will point you at the call where the prefix moved. This is where an API relay helps, because one key and one base URL, https://api.fluxorelay.com, give you the same usage shape across Claude, GPT, Gemini, DeepSeek, Kimi, GLM, Qwen, Grok and more while you compare cache behavior between models.

Fluxo is billed per real token including cache reads and writes, with no monthly fee, so the accounting lines up with the usage you see in your logs whether you drive it from Claude Code, Codex, Gemini CLI or any OpenAI SDK. Referrals earn a share when invited developers pay, so the people you bring along keep the economics simple.

  • Record input tokens, cache creation tokens, and cache read tokens for every call so the hit rate stays visible across sessions.
  • Compare the same workflow on two shelf models, for example Claude Haiku 4.5 and DeepSeek V4 Flash, before blaming your template for a drop.
  • Use one key and one base URL across your tools so the usage report stays comparable no matter which terminal front end you use.

If you want to try these patterns against real traffic, create a Fluxo account and run the same session twice to see the second run read instead of write. If anything in your numbers looks off, leave a note on the contact page after signing in.

Next in the buffer