How does prompt caching work, and what breaks the cache?

Prompt caching lets an AI provider reuse the processed form of an identical prompt prefix, which reduces response delay and the cost of repeated input. The cache only hits when everything up to the cache point matches exactly. One changed tool definition or a timestamp near the top can silently turn every request into a miss.

What is cached

The cache covers a prefix, not the whole prompt and not the output. Anthropic's documentation describes the order as tools, then system, then messages, up to the marked block. OpenAI describes the same idea: the cache stores the processed state of a reusable prefix such as instructions, tool definitions and earlier conversation. Both say reuse requires the entire rendered prefix to match.

Caching does not change the model's answer. The Anthropic page states the response is identical to an uncached one, and that output tokens are unaffected.

How you turn it on

There is also a minimum length. Shorter prefixes cannot be cached, and Anthropic notes no error is returned when that happens, so a too-short prompt just never hits. The minimum differs by model, so look it up for yours.

How long it lasts

Anthropic documents a default lifetime of five minutes that is refreshed each time the cached content is used, with a longer one hour option at additional cost. OpenAI documents retention settings that differ by model generation, from short in-memory retention to longer windows on some models. Treat lifetime as a setting to verify, not a constant. If traffic is bursty with long gaps, you may pay to write the cache repeatedly without enough reads.

What breaks it

The Anthropic page lists these invalidations, and the pattern is that a change flows downward from the level where it happens:

1. Changing tool definitions invalidates everything after them.
2. Changing the system prompt, including toggling some built-in features, invalidates system and message caches.
3. Changing tool choice, adding or removing images, or changing the thinking or effort settings invalidates message blocks.
4. Putting a marker on content that changes on every request, such as a timestamp, means no hits at all.

OpenAI adds a routing note: cache entries live on specific machines, and heavy volume can overflow to machines without your entry, so hit rates can fall under load.

How to check it is working

Look at the usage fields in each response. Anthropic reports cache creation tokens, cache read tokens, and the uncached remainder, and says total input is the sum of the three. If both cache fields are zero, nothing was cached, often because the prompt was under the minimum. OpenAI reports cached tokens in usage details, and dividing cached by total input gives a hit rate.

A practical layout

Put stable content first: tool definitions, system instructions, reference documents, examples. Put volatile content last: the user question, current date, per-request IDs. OpenAI's latency guide gives the same advice, placing dynamic content later to benefit from caching.

Caching also helps limits. On most models Anthropic does not count cache reads toward the input token limit, as covered in cache-aware input token limits. For batch jobs with a shared prefix, see how batch APIs work.

Worked example: a support assistant

These token counts are illustrative. A support assistant sends the same 6,000 token block on every call: instructions, a product guide and tool definitions. Each user question adds about 150 tokens.

Without caching, every call processes about 6,150 input tokens. With a working cache, the 6,000 token prefix is read from the cache and only the 150 new tokens plus the output are processed fresh. The first call writes the cache, and later calls within the cache lifetime read it.

Now introduce a bug. A developer adds the current time to the top of the system prompt, "Current time is 14:03:22". Every call now has a different prefix, so the cache never hits, and the team pays the write cost again and again without any reads. The usage fields show cache reads stuck at zero. Moving the time to the end of the prompt, after the stable block, restores hits.

A debugging routine

1. Log the three usage fields for each call: cache creation, cache read and uncached input.
2. Compute a hit rate over a day. A low rate on a repeated prefix means something is changing it.
3. Diff two consecutive prompts byte for byte. Look for timestamps, request IDs, random ordering of tools or examples, and user names.
4. Check that tool definitions are identical and in the same order every call.
5. Confirm the prefix is over the model's minimum length.
6. Check call spacing against the cache lifetime. If calls are minutes apart and the lifetime is five minutes, you may be missing by a little.

When not to cache

A prompt that is unique every time, such as a one-off document summary, gains nothing from caching. A cache write may even cost more than plain input, since the Anthropic page says writes cost more than standard input tokens. Cache only content you will reuse within the lifetime, and confirm with a week of usage data.

Pairing with other levers

Caching combines with batch jobs, where the page suggests a longer cache lifetime for shared context. It interacts with limits, since cache reads do not count toward input token limits on most models. And it depends on stable tool lists, as covered in MCP tool definitions and tokens.

Testing a cache change before shipping

Before you change a system prompt or tool list in production, replay a sample of recent requests against the new version in a test workspace and compare cache read rates with the old one. A change that looks harmless, such as renaming a tool, can reset the cache for every user at once and show up as a bill jump the next morning. A short replay test catches that.

Glossary

A prefix is the start of a prompt that stays identical across calls. A cache write stores the processed prefix, and a cache read reuses it. A breakpoint marks where the cacheable portion ends. Lifetime is how long an entry stays valid without use. A miss means the provider reprocessed content that could have been reused. Time to first token is the delay before output begins. Share these terms so reviews of caching problems start from a common vocabulary.

Try it yourself

Click through it at your own pace, or open it full screen. It is about filtering out work that does not matter. The parallel is loose but useful: caching avoids repeating work that has not changed, and the demo shows how removing irrelevant items leaves a smaller, clearer list.

Quick check

Sources

• Prompt caching, Claude API documentation
• Prompt caching, OpenAI API documentation
• Latency optimization, OpenAI API documentation

Last reviewed: 19 October 2026.