How I Reduced AI API Costs by 80% Using Prompt Caching

The graph that ruined my morning was in the API usage dashboard: a cost curve climbing noticeably faster than my user count. The support agent was doing its job well — that was the problem. Every conversation, every message, was expensive in a way I hadn't budgeted for.

When I dug into why, the answer was almost stupid: I was paying to send the same twenty thousand tokens over and over. And over. AI prompt caching fixed it — my input costs dropped by roughly 80% — but only after I understood what actually makes caching work, and found the one line in my prompt that had been silently sabotaging it.

The problem: you keep paying for the same tokens

Look at what an LLM app sends per request. Mine was:

• System instructions and policies: ~3,000 tokens — identical every request
• Product documentation context: ~15,000 tokens — identical every request
• Tool definitions: ~2,000 tokens — identical every request
• The actual user message and history: ~500–2,000 tokens — the only new part

Over 90% of every request was a photocopy of the previous request. And the API billed full price for the photocopies. That's the entire problem prompt caching exists to solve: providers let you reuse the processed version of a repeated prompt prefix instead of paying to reprocess it.

How caching works on the two big providers

Anthropic (Claude) uses explicit cache breakpoints. You mark where the stable part of your prompt ends with cache_control, and that prefix gets cached (default lifetime around five minutes, refreshed on every hit). Cached reads cost roughly 90% less than normal input tokens; writing the cache the first time costs a ~25% premium. You're placing a bet: this prefix will be reused within minutes. For an active support agent, that bet wins almost every time.

const response = await anthropic.messages.create({
  model: 'claude-sonnet-4-5',
  max_tokens: 1024,
  system: [
    {
      type: 'text',
      text: BASE_INSTRUCTIONS + POLICIES + PRODUCT_DOCS, // the big static block
      cache_control: { type: 'ephemeral' },
    },
  ],
  messages: conversationMessages, // the small dynamic part
});

OpenAI caches automatically — no markup needed — for prompts past a minimum size, matching on the prompt prefix, with cached input tokens billed at roughly half price. Zero code changes to benefit... if your prompt is structured so the repeated content is a byte-identical prefix.

That "if" is where I was failing on both providers at once.

The audit: finding my cache busters

Caching matches prefixes exactly. One changed character early in the prompt and everything after it is a cache miss. I printed two consecutive requests and diffed them. Three saboteurs, in ascending order of embarrassment:

Third place: user-specific facts injected near the top of the system prompt. Everything below them — all fifteen thousand tokens of docs — became uncacheable, different for every user.

Second place: tool definitions assembled from an object whose key order wasn't stable. Same tools, occasionally different serialization. Invisible to humans, fatal to prefix matching.

First place, with trophy: Current time: ${new Date().toISOString()} on line two of my system prompt. A timestamp. Changing every second. Above twenty thousand static tokens. I had built a machine for generating cache misses and paid for the privilege for weeks.

The restructure: static first, dynamic last

The fix is one principle applied ruthlessly — order your prompt by how often things change:

1. Base instructions          (changes: never)        ← cached
2. Policies + product docs    (changes: weekly)       ← cached
3. Tool definitions           (changes: rarely)       ← cached
   ---- cache boundary ----
4. User facts, current time   (changes: per user/req)
5. Conversation messages      (changes: every turn)

The timestamp moved to the bottom, next to the user facts. Tool serialization got a stable sort. On Anthropic, the cache_control marker sits right at that boundary; on OpenAI, the identical prefix now matches automatically. Same information, same behavior, different order — that's the whole trick.

The real math

My support agent's shape: ~20,000 static tokens per request, a few thousand requests a day, conversations bursty during working hours (which keeps the cache warm — gaps longer than the cache lifetime mean paying the write premium again).

After the restructure, the usage logs showed cache hit rates around 90% during active hours. Input cost per request dropped to roughly a fifth of before. Blended over the month, input spend fell ~80%. Two honest caveats so this doesn't oversell: output tokens are never cached — you pay full price for every generated word, so total-bill savings depend on your input/output ratio (mine is very input-heavy because of the docs). And low-traffic apps with cold caches will see far less; caching rewards exactly the apps that are getting expensive.

Verify with the API's own accounting — the usage object reports cached tokens (cache_read_input_tokens on Anthropic, cached_tokens on OpenAI). If those numbers aren't large, your prefix isn't matching, and something in your prompt is fidgeting.

The bonus nobody mentions

Cached prompts aren't just cheaper — they're processed faster. My time-to-first-token improved noticeably, because the model skips re-reading twenty thousand tokens it has already digested. Users noticed the snappier replies before I told anyone about the bill.

One morning-ruining graph, one diff, one reordering. If your app sends big stable context — docs, schemas, long instructions — go diff two consecutive requests today. There's a decent chance a timestamp is quietly eating your margin, like mine was.