A familiar setup: the same block rides along with every request. It might be a system instruction several pages long, a database schema or a slice of documentation. The model reads it from scratch on each call, and each read lands on your bill.
When prompt caching pays off
Three conditions have to hold at once:
- The repeated block is large. Squeezing a couple of paragraphs is not worth the effort.
- It is identical from request to request, down to the last character.
- Requests come often. A cache lives for a limited time, and a long pause between calls means it will not help.
Typical fits: an agent with a long rule set, a chatbot over big documentation, batch processing of similar documents under one shared instruction.
What breaks the cache
Change a single character in the repeated part and the cache misses. A classic mistake is to inject today's date or a request id into the instruction. The cache breaks on every call, and it is easy to miss.
Keep the constant and the variable parts apart: constant first, variable last.
Order inside the request
The cache is read from the start of the request, so the stable block goes first and the changing one last. Put the variable part at the front and everything after it falls out of the cache as well.
Estimating the savings in money
Take the size of the repeated block, multiply by requests per month and by the input rate. That is what you currently spend on re-sending the same thing.
Then compare it with the cached-read rate, which is markedly lower than the normal input rate.
The site has a calculator for trying this on your own volumes.
Cache writes cost more
The first write is more expensive than a normal read. If you hit the cache only once or twice, you lose money. It starts paying off on the third or fourth hit.
Hence the rule: do not cache what you use once.
The short version
Prompt caching pays on a large, stable start of the request with frequent access. Put the stable content first and the variable content last, and make sure dates and ids do not leak into the constant part.