Byte-Identical Prefix Requirements
Learning Headroom
Why 'Almost the Same' Prefix Isn't Good Enough
Recap — Prefix Caching (2.1)
- Provider caches the prompt prefix once, reuses it on later turns
- Cache hit = skip re-processing those tokens = lower latency + lower cost
The catch:
- Cache hit requires the prefix to be **byte-identical** to what's cached
- Not "similar" — not "same meaning" — **same bytes, in the same order**
```mermaid
flowchart LR
A["New request prefix"] --> B{"Byte-identical to cached prefix?"}
B -->|Yes| C["Cache HIT: fast + cheap"]
B -->|No, even 1 byte differs| D["Cache MISS: full reprocessing"]
```
Builds on ← 2.1 Prefix Caching Fundamentals: this is the exact-match rule behind the cache hit you learned there.
Where this leads → 2.3 Stable-Prefix Layout will show you how to *design* prefixes so this check keeps passing.
Defining Byte-Identical
Byte-identity requirement
Let prefix \(P_t\) be the serialized bytes sent as the stable portion of the prompt on turn \(t\).
Cache hit on turn \(t+1\) requires:
\[ P_{t+1} = P_t \quad \text{(byte-for-byte)} \]
Equivalently: \( \text{hash}(P_{t+1}) = \text{hash}(P_t) \)
- Not \(P_{t+1} \approx P_t\) — approximate similarity does **not** count
- Not "same tokens after re-tokenizing" — tokenization happens *after* the cache check
- A single added/removed/reordered byte anywhere in \(P\) invalidates the whole prefix from that byte onward
**Why this shape?** Providers cache the *raw serialized text*, keyed by a hash (or byte compare) of it. Hashing is cheap and exact — there's no cheap way to hash "approximately," so the check is necessarily exact. Flip one bit of input and a hash function's output changes completely (this is called the avalanche effect) — so a hash can never tell you "close, but not quite." It only ever tells you "same" or "different."
Worked Example: Spot the Byte Drift
Turn 1 prefix (serialized, simplified):
```
{"system":"You are an agent.","tools":["search","exec"],"session_id":"abc123"}
```
Turn 2 prefix sent by the caller:
```
{"system":"You are an agent.","tools":["exec","search"],"session_id":"abc123"}
```
Step-by-step byte comparison:
1. `{"system":"You are an agent.",` — identical, bytes match
2. `"tools":["search","exec"]` vs `"tools":["exec","search"]` — **tool order swapped**
3. First differing byte: right after `"tools":["` — turn 1 has `s`, turn 2 has `e`
4. Everything from that byte onward counts as different, even though `session_id` afterward is identical text
Result: **Cache MISS** — the reordering of two list elements, with zero change in meaning, is enough to invalidate the entire prefix from that point forward.
```tikz
\begin{tikzpicture}[scale=1]
\node[anchor=west] at (0,1.4) {\footnotesize turn 1: \ldots "tools":["s..."};
\node[anchor=west] at (0,0.6) {\footnotesize turn 2: \ldots "tools":["e..."};
\draw[-] (0,1.1) -- (4.3,1.1);
\draw[-] (0,0.3) -- (4.3,0.3);
\draw[red,thick] (3.55,-0.1) -- (3.55,1.7);
\node[red,above] at (3.55,1.7) {\footnotesize first mismatch};
\node[below] at (1.7,-0.3) {\footnotesize identical bytes};
\node[below] at (4.0,-0.55) {\footnotesize diverges here};
\draw[->,gray] (2.5,-0.15) -- (1.7,-0.05);
\end{tikzpicture}
```
Volatile Fields: The Usual Suspects
Volatile prefix field detection — common culprits
| Field type | Example | Why it drifts |
|---|---|---|
| Timestamps | `"sent_at":"14:32:07"` | Changes every turn by definition |
| Random/session IDs regenerated per call | `"req_id":"9f3a..."` | New UUID each request |
| Unordered collections | tool lists from a set/dict | Iteration order not guaranteed stable |
| Live counters | `"turn":7` inside prefix zone | Increments every turn |
| Environment-dependent values | working directory, hostname | Differs across machines/runs |
| Floating-point serialization | `0.1` vs `0.10000000001` | Different float→string formatting paths |
**Rule of thumb:** if a field's value is a function of *time*, *randomness*, or *unordered data*, it does not belong in the stable prefix.
Builds on ← 1.2 Agent Message Anatomy: these are exactly the layer boundaries between "stable" and "live" content that message anatomy first drew.
Serialization Stability: Same Data, Different Bytes
Message serialization stability
The object you build and the bytes you send are not the same thing — serialization sits between them.
```mermaid
flowchart LR
Obj["Message object in memory"] -->|serialize| Bytes["Prefix bytes P_t"]
Bytes -->|sent to provider| Cache["Provider cache keyed by hash"]
```
Same logical object, unstable serializer, different bytes:
- Dict key order not fixed → `{"a":1,"b":2}` vs `{"b":2,"a":1}`
- Whitespace/pretty-printing toggled → `{"a":1}` vs `{"a": 1}`
- Numeric formatting differs → `7` vs `7.0`
- Optional-field omission vs `null` → `{"x":null}` vs `{}`
**Requirement:** serialization must be a deterministic function of the message content — same content in \(\Rightarrow\) same bytes out, every single time, on every machine, in every process.
This is *necessary but not sufficient*: stable serialization only helps if the underlying content is also unchanged (see previous slide).
Reasoning Backward: Cache Invalidation Checklist
Prefix cache invalidation reasoning
Symptom: cache hit rate suddenly drops on turn \(t\). Diagnose by asking, in order:
1. **Content change** — did any field value in the prefix actually change (timestamp, counter, regenerated ID)?
2. **Order change** — did a list or dict get rebuilt from an unordered source (set, hash map)?
3. **Formatting change** — did the serializer's output format change (whitespace, number formatting, key omission)?
4. **Boundary drift** — did volatile content leak from the live zone *into* the stable prefix zone?
5. **Environment change** — did something machine/process-specific (paths, hostnames) get embedded?
Three changes that WILL invalidate an otherwise-identical prefix:
- Reordering the tool list (or any list built from a set)
- Adding/removing whitespace or changing key order in the serialized JSON
- Letting a live timestamp or turn counter sit inside the stable-prefix bytes
```mermaid
flowchart TD
Miss["Cache hit rate drops"] --> Q1{"Content differs?"}
Q1 -->|yes| Fix1["Remove volatile field from prefix"]
Q1 -->|no| Q2{"Order differs?"}
Q2 -->|yes| Fix2["Sort/fix ordering before serializing"]
Q2 -->|no| Q3{"Formatting differs?"}
Q3 -->|yes| Fix3["Lock serializer config"]
Q3 -->|no| Q4["Check environment and cache TTL"]
```
Where this leads → 2.4 CacheAligner watches for exactly this kind of drift automatically and flags it before it costs you cache hits.
Back to course