Prompt Caching: The 90% Discount You're Missing

Last updated: October 2026

There is a 90% discount sitting in your API bill, and most developers are either not using it or using it in a way that doesn't work. Prompt caching — paying a tenth of the input price for tokens the provider has already seen — is the highest-leverage cost optimization available on modern LLM APIs. Here's how it actually works and how to get the savings.

How Prompt Caching Actually Works

The idea is simple. When you send a request, the provider processes your input tokens through the model — that's the expensive computation you're paying for. But if your next request starts with the exact same tokens — the same system prompt, the same repo context, the same few-shot examples — the provider doesn't need to recompute all of that. It kept the intermediate results cached. So it charges you a fraction of the price for those repeated tokens.

The key phrase is stable prefix. Caching works from the start of your prompt forward: the provider matches your prompt's beginning against what it has cached, and everything that matches gets the discounted rate. The moment the tokens diverge, matching stops and the rest is billed at full price.

This has one big architectural implication: put stable content first, variable content last. System instructions, repo context, few-shot examples, reference documents — all of that goes at the top, byte-identical every time. The thing that changes per request goes at the end.

The Math: What 90% Off Actually Means

On GPT-5.6 Sol, the numbers are published and stark:

  • Fresh input: $4.00 per 1M tokens
  • Cached input reads: $0.40 per 1M tokens

Run it through the reference workload — 1,100 tasks/month, each with 12K fresh input, 48K cached input, and 6K output tokens. The 48K cached tokens — four times the volume of fresh tokens — cost less than half as much. Without caching, those 48K tokens at full price would cost $0.1920 per task instead of $0.0192. Across 1,100 tasks a month, caching saves roughly $190/month on this workload alone.

One cost people miss: cache writes. The first time a prompt prefix is cached, the provider charges a premium — on GPT-5.6 Sol, cache writes cost 1.25x the input rate. Every reuse after that is nearly free.

Structuring Prompts for Cache Hits

1. Fixed order, every time. The stable prefix must be byte-identical across calls. Build your prompts programmatically from fixed templates.

2. Nothing dynamic in the prefix. Timestamps, request IDs, user names — if any of it sits inside the cached region, every request has a different prefix and nothing ever hits the cache. Move all dynamic content after the stable block.

3. Longest stable content first. Order the stable blocks by how rarely they change: system prompt first, then reference documents, then conversation summaries, then the live request.

4. Don't reshuffle tool definitions. In agentic loops, the tool schemas are often a large, stable block — perfect cache material. Pin them.

5. Verify, don't assume. After restructuring, check the cached-token ratio in your usage dashboard. You're aiming for 80%+ of input tokens served from cache on repetitive workloads.

Realistic Hit Rates

  • Coding agents with stable repo context: 70–90% of input tokens cached is realistic.
  • Chatbots with fixed system prompts: 50–80%.
  • One-shot or highly variable tasks: 0–30%. Don't force it; caching is for repetition.

Which Workloads Benefit Most

Biggest winners: agentic coding loops, RAG systems with fixed document sets, chatbots with long system prompts, eval harnesses. Modest benefit: mixed workloads with partial repetition. Skip it: truly one-off inference where the input is never repeated.

Debugging a Cache That Won't Hit

Capture two consecutive prompts, diff them, and look at exactly which tokens differ. Common offenders: timestamps injected by templates, session IDs in instructions, tool definitions regenerated per call with non-deterministic ordering, conversation history prepended before the system prompt instead of appended after it.

The Bottom Line

Prompt caching is the closest thing to free money in LLM APIs: a ~90% discount on your largest token category, available for the cost of ordering your prompts correctly. Fix the order, check the dashboard, and watch the input line of your invoice collapse.

Popular posts from this blog

Argon vs Wonder: Which AI Should Developers Build On?

Image
Last updated: October 2026 Two flagship models, two very different price tags — and one of the prices is about to double. If you're choosing a foundation model to build on right now, the decision isn't just about capability. It's about timing. Here's the honest comparison: what each model costs, what the benchmarks actually say, and which one to pick depending on what you're optimizing for. The Pricing: Read the Dates Gemini 4 Argon is on introductory pricing: $2.00 per 1M input tokens / $10.00 per 1M output tokens . That intro rate runs through November 30, 2026 . From December 1 , the price doubles to $4.00 / $20.00 . Muse Wonder is priced at $4.00 / $20.00 per 1M tokens — no intro period, no scheduled change. So today, Argon costs half what Wonder does. On December 1, they cost exactly the same. That convergence matters more than it looks. Every cost comparison you run today has a 6-week expiry date. If you're doing the math — and you should be — do...
Read more

GPT-5.6 API Pricing: The November 21 Deadline

Image
Last updated: October 2026 If you're running GPT-5.6 through the API right now, you're on a promotional price. And that promotion has an expiry date: November 21, 2026 . OpenAI's pricing page states the current rate is "available at least through November 21, 2026." Read that carefully. "At least through" — not "until." The promo could extend. It could also end exactly on that date. OpenAI has published no pricing for what comes after. This post lays out exactly what the pricing is today, what it costs on a realistic workload, and how to budget for the unknown that follows November 21. The Current Pricing, Exactly GPT-5.6 is available in several variants. The one most developers are using — the flagship reasoning model, referred to here as GPT-5.6 Sol — is currently priced at: Input: $4.00 per 1M tokens Cached input: $0.40 per 1M tokens Output: $20.00 per 1M tokens That is promotional pricing. Before the promo, announced August 21, ...
Read more