Prefix Caching Fundamentals
Learning Headroom
Why Repeated Prefixes Are Wasted Work
**The problem**
Every agent turn re-sends system instructions, tool definitions, and prior conversation — most of it byte-for-byte identical to the last turn.
**Builds on what you already know**
- 1.1 Headroom and Context-Window Economics: tokens cost money and count against the context window — reprocessing the same tokens repeatedly wastes both.
- 1.4 Repeated Wakes and Prompt Layers: an agent 'wakes up' again and again, and each wake re-sends layered prompt content (system layer, history layer, new turn).
**Where this leads**
This idea — that a stable, repeated prefix can be reused instead of recomputed — is the whole foundation for 2.2, Byte-Identical Prefix Requirements, where you'll learn exactly what 'identical' has to mean for the cache to actually fire.
What a Prefix Cache Actually Stores
**Cacheable prefix**
The leading portion of a prompt — from the very first token — that stays byte-identical across turns. Usually: system prompt + tool definitions + early conversation history.
**Cached attention state**
When the model processes tokens, it computes internal attention values (keys and values) for them. The provider can save this computed state instead of throwing it away.
```mermaid
flowchart LR
subgraph Turn1["Turn 1 prompt"]
A["System prompt"] --> B["Tool defs"] --> C["History so far"] --> D["New user message"]
end
D -."compute and save state".-> S[("Cached attention state for prefix A+B+C")]
subgraph Turn2["Turn 2 prompt"]
A2["Same system prompt"] --> B2["Same tool defs"] --> C2["Same history plus Turn 1 reply"] --> D2["New user message"]
end
S -."reused, not recomputed".-> C2
```
**Why it works**
Attention math is deterministic: identical input tokens in the identical order always produce the identical internal state. So if the prefix truly hasn't changed, recomputing it is guaranteed to reproduce the same numbers — which means it's safe to skip and reuse them.
Cache Hit vs. Cache Miss
**Two outcomes when you send a prompt**
- Cache hit — the sent prefix matches a previously cached prefix exactly. Provider reuses the saved attention state. Only the new suffix tokens get freshly processed. Fast, cheap.
- Cache miss — the sent prefix does NOT match (even one token off, or reordered, or expired). Provider recomputes the entire prefix from scratch. Slow, full price.
**Turn-by-turn trace**
- Turn 1: prefix has never been sent before → nothing to match → MISS. Provider processes all 2,000 prefix tokens plus the 50-token user message: \(2000 + 50 = 2050\) tokens computed. Provider caches the 2,000-token prefix state.
- Turn 2: same 2,000-token prefix is sent again, unchanged, plus turn 1's reply (about 40 tokens) and a new 50-token message appended → prefix matches the cached one → HIT on those 2,000 tokens. Provider only computes the newly appended tokens: \(40 + 50 = 90\) tokens.
- Turn 3: someone edits the system prompt by one word → prefix no longer matches the cached version → MISS again. All 2,000+ prefix tokens are recomputed from scratch, and a new cache entry is stored.
**The rule**
A hit needs the ENTIRE prefix to match, not 'mostly' match. One changed, inserted, or reordered token anywhere in the prefix invalidates the match for everything after it.
Identifying What's Actually Cacheable
**Spot the cacheable prefix**
1. System prompt describing the agent's role — identical every turn ✅ cacheable
2. Full tool/function definitions — identical every turn ✅ cacheable
3. Earlier conversation turns 1 through N−1 — identical every turn, as long as nothing before them was edited ✅ cacheable
4. The current user's newest message and the model's response to it — different every turn ❌ not cacheable yet (becomes cacheable NEXT turn once it's fixed history)
5. A live timestamp or token-usage counter injected into the system message — changes every turn ❌ breaks the cache for everything after it, even though it looks tiny
**Common trap**
A single small dynamic field placed early in the prompt (like a timestamp) does more damage than a whole paragraph of dynamic content placed at the end — because it sits inside the stable prefix and breaks the match for everything downstream.
```tikz
\begin{tikzpicture}[scale=1]
\def\w{9}
\def\h{0.9}
% stable prefix block
\draw[fill=blue!12] (0,0) rectangle (6.5,\h) node[midway] {};
\node at (3.25,\h/2) {\small stable prefix: system + tools + history};
% timestamp sliver near the start
\draw[fill=red!35] (1.4,0) rectangle (1.9,\h);
\node[below] at (1.65,-0.15) {\footnotesize timestamp};
% new suffix block
\draw[fill=green!20] (6.5,0) rectangle (\w,\h);
\node at (7.75,\h/2) {\small new};
% broken match arrow
\draw[->,thick,red] (1.9,-0.6) -- (6.5,-0.6);
\node[below,red] at (4.2,-0.6) {\footnotesize everything after this point is a MISS};
\end{tikzpicture}
```
Why Caching Pays Off — The Economics
**Rough cost per turn**
\[ \text{cost}_{\text{turn}} = \underbrace{c_{\text{hit}} \cdot |\text{prefix}|}_{\text{cached tokens}} + \underbrace{c_{\text{full}} \cdot |\text{new tokens}|}_{\text{fresh tokens}} \]
Here \(c_{\text{full}}\) is the price per token for full processing, and \(c_{\text{hit}}\) is the (much lower) price per token for reading from cache. As \(c_{\text{hit}}\) drops toward zero, or as the prefix grows relative to the new tokens, the savings from hitting the cache grow larger.
**Numbers: 2,000-token prefix, 90 new tokens, hit vs. miss**
Assume \(c_{\text{full}} = 1.00\) unit per 1,000 tokens and \(c_{\text{hit}} = 0.10\) units per 1,000 tokens.
MISS cost (everything processed fresh):
\[ \frac{2000 + 90}{1000} \times 1.00 = \frac{2090}{1000} \times 1.00 = 2.09 \text{ units} \]
HIT cost (prefix from cache, new tokens at full price):
\[ \frac{2000}{1000} \times 0.10 = 0.20 \text{ units (cached part)} \]
\[ \frac{90}{1000} \times 1.00 = 0.09 \text{ units (new part)} \]
\[ \text{HIT total} = 0.20 + 0.09 = 0.29 \text{ units} \]
Savings from the hit:
\[ 2.09 - 0.29 = 1.80 \text{ units saved} \]
\[ \frac{1.80}{2.09} \approx 0.86 \rightarrow \text{about 86% cheaper this turn} \]
**Ties back to 1.1**
This is exactly the headroom math you learned in 1.1, applied at the per-turn level: fewer tokens genuinely reprocessed means lower cost AND less pressure on the context window's effective budget.
Putting It Together: Hit, Miss, and What Comes Next
```mermaid
flowchart TD
Start(["New turn arrives"]) --> Check{"Does the sent prefix byte-match a cached prefix?"}
Check -- "Yes" --> Hit["CACHE HIT: reuse saved attention state, only process new suffix tokens"]
Check -- "No: even 1 token differs, order changed, or cache expired" --> Miss["CACHE MISS: recompute entire prefix, store new cache entry"]
Hit --> Cheap["Lower cost, faster turn"]
Miss --> Costly["Full cost, slower turn"]
```
**The core chain of reasoning**
- A repeated prefix means identical tokens in identical order across turns.
- Identical tokens always produce identical attention state (deterministic computation).
- So the provider can legally SAVE and REUSE that state instead of recomputing it — that's a cache hit.
- Any change anywhere in the prefix breaks the exact match — that's a cache miss, full recompute.
**Where this leads next**
Node 2.2, Byte-Identical Prefix Requirements, zooms into exactly what counts as 'identical' — whitespace, field ordering, dynamic values — so you can design prompts that keep hitting the cache instead of silently falling back to misses.
بازگشت به دوره