Skip to main content
When the same system prompt, tool definitions or reference documents are sent with every request, prompt caching lets the provider bill those repeated tokens at a fraction of the normal input price. nOps measures how much of each owner’s input is read from the cache, flags owners whose cache hit rate is below 25%, and estimates the saving from raising it to 35%.

Why it matters

Agents and chat applications resend the same context on every call. Without caching, all of it is paid at the full input price every time.

How nOps finds it

nOps flags an owner when cache reads are less than 25% of all input tokens, and estimates the saving from moving uncached input into cache reads until 35% of input is cached. On the Anthropic API (per API key) the saving is the moved tokens times the difference between your uncached and cache-read rates, less the cost of one cache write for every five reads. On the OpenAI API (per project) it is your billed uncached input cost times the share that would move, times the cached-price discount (your own measured ratio, or 90% when it cannot be measured) less a 5% allowance. Both use the last 30 days and need a saving of at least $25 a month. On AWS Bedrock (per person or workload, last 32 days, owners with at least $10 of spend) nOps uses Bedrock’s price multipliers (cache reads at 0.1 times input, writes at 1.25 times), counts 75% of the moved input cost as saved, and needs at least $5 a month. Claude Enterprise users are not included, because they cannot set caching themselves.

Before you start

Access to the application code that calls the model, on a model and platform that supports prompt caching.

How to implement

  1. Find the part of each request that repeats: the system prompt, tool definitions, examples and reference documents.
  2. Put that static content at the start of the request and keep it byte-for-byte identical between calls. Put anything that changes (user input, timestamps, IDs) after it.
  3. Anthropic API: add a cache_control breakpoint at the end of the repeated prefix (system prompt, tools, long context).
  4. AWS Bedrock: add a cache point after the repeated prefix in the Converse or InvokeModel request.
  5. OpenAI API: caching is automatic on a stable prefix. Set prompt_cache_key so similar requests share a cache.
  6. After rollout, check the cache hit rate and the cost of cache writes.

Risks and considerations

  • Writing to the cache costs more than normal input. A prompt that is rarely reused within the cache lifetime can cost more with caching.
  • Caches expire after a few minutes without use, so low-traffic workloads benefit less.
  • Any change to the cached prefix, even one character, misses the cache.
  • To roll back, remove the cache breakpoints.

FAQs

nOps flags a cache hit rate below 25% and estimates the saving at 35%. Many workloads can go higher, so the real saving can be larger.