Prompt Caching: How to Cut LLM Cost and Latency

If you're building anything on top of an LLM API and you haven't turned on prompt caching, there's a decent chance you're paying five to ten times more than you need to — and shipping slower responses on top of it. It's one of the rare optimizations that's close to free money. Here's how it works and how to actually get the win.

The waste it eliminates

Most real AI applications send the model the same long chunk of text over and over. A detailed system prompt. A set of instructions and examples. A big document the user is asking questions about. That preamble might be 10,000 tokens, and it's identical on every single request — only the last little bit (the user's new question) changes.

Normally, the model reprocesses that entire preamble from scratch every time. Ten thousand tokens of identical text, recomputed on every call, billed on every call. That's the waste.

What caching does

Prompt caching is a provider-side feature that lets the API reuse the processed form of a prompt's prefix across requests. The first time it sees your long preamble, it does the work and stores the result. On the next request that starts with the same prefix, it loads the stored result instead of recomputing it — the model skips re-reading the part it has already read.

You get two benefits at once:

• Cost: cache reads are dramatically cheaper than fresh input. On Anthropic, cached tokens are billed at roughly 10% of the normal input rate; other providers offer discounts in the 25–50% range. A workload that costs $5.00 uncached can drop to well under a dollar.
• Latency: when most of your prompt is cached, time-to-first-token drops sharply — commonly 40–80% faster on cache-heavy prompts, because the model isn't grinding through thousands of repeated tokens before it starts answering.

The one rule that makes it work: order matters

Here's the thing that trips people up. Caching works on the prefix — the beginning of your prompt — and it only hits if the cached portion is byte-for-byte identical to what was cached before. The moment the text diverges, the cache stops applying from that point on.

The practical consequence: put your stable content first and your variable content last. Structure every prompt as [ stable stuff ] + [ changing stuff ]:

• Stable, cacheable prefix: system prompt, instructions, few-shot examples, the big reference document.
• Variable suffix: the user's specific question, the current input, anything that changes per request.

If you put the user's question before the reference document, you've defeated caching — every request now has a different prefix and nothing gets reused. This reordering is often the single change that unlocks the entire savings.

How to turn it on

The mechanics differ by provider, and the split is worth knowing:

• Explicit (Anthropic): you mark what to cache with cache_control breakpoints on the parts of the prompt you want stored, and choose a time-to-live (commonly 5 minutes, with a longer 1-hour option). You're in control of what gets cached.
• Automatic (OpenAI): caching kicks in on its own for prompts above a length threshold (around 1,024 tokens), with no explicit markers and no guaranteed TTL.
• Other providers (Gemini, DeepSeek, and most major hosted APIs) support caching too, with their own pricing models — Gemini, for instance, bills for cache storage by time rather than per cached token.

One cost nuance to know: on explicit-cache providers, the first write to the cache can cost slightly more than a normal request (around 1.25x on Anthropic). You pay a small premium once to store, then save big on every read. So caching pays off when a prefix is reused enough times to earn back that write — which, for a busy system prompt or a document in an active chat session, is almost immediately.

Where it pays off most

• Chatbots and assistants with a long system prompt sent on every turn.
• Document Q&A, where many questions hit the same large document — cache the document, vary the question.
• Agents, whose tool definitions and instructions are large, static, and re-sent on every step of the loop (see How to Build Your First AI Agent).
• Few-shot-heavy prompts, where a big block of examples rides along on every call.

The takeaway

Prompt caching is the highest-return, lowest-effort optimization in applied AI right now. The recipe is short: identify the big chunk of your prompt that never changes, move it to the front, and turn caching on. You keep the exact same quality — this isn't a tradeoff — while cutting cost by up to ~90% and latency by up to ~80% on the requests that matter most. Few optimizations are that one-sided.

For the broader craft, see The Complete Guide to Prompt Engineering in 2026; and if a cached preamble is feeding a data pipeline, pair it with reliable structured output. Prompts to build on are in the prompt library.