Cache-Safe Multi-Turn Forwarding
Learning Headroom
Why Multi-Turn Loops Break the Cache
**Recall (2.3):** stable prefix first, volatile live zone last — the layout the provider hashes and caches.
**Recall (2.4):** CacheAligner *detects* when that prefix drifted (hit rate drops) but cannot fix it — it's observability only.
**New problem:** a real agent loop sends turn 1, turn 2, turn 3... Each `/v1/compress` call is *stateless* — it sees one isolated request, with no memory of what the provider already cached.
**The trap:**
- Turn 1: you compress + send messages. Provider caches the bytes it received.
- Turn 2: if you resend the *pristine originals* instead of what you actually sent, the provider sees a **different prefix** → cache miss → full re-read from scratch.
On Anthropic, a cache read is ~90% cheaper than a fresh read — so a silent cache-bust can cost *more* than compression saved.
**Where this leads →** 3.4 Cache Mode vs Token Mode Tradeoffs depends on getting this loop right.
Forwarded State vs Pristine Originals
**Pristine originals:** the raw messages your agent generated, before Headroom touched them.
**Forwarded message state:** the *exact bytes you actually sent* to the provider last turn — after compression, formatting, everything.
\[
\text{pristine} \xrightarrow{\text{compress}} \text{forwarded}, \quad \text{forwarded} \neq \text{pristine}
\]
Compression changes bytes. The provider cached the **forwarded** version — not the pristine one. So the forwarded output *is* the new ground truth for what to send next turn.
```mermaid
flowchart LR
A["Pristine original message"] -->|compress turn 1| B["Forwarded bytes"]
B -->|provider caches these bytes| C[("Provider prefix cache")]
B -->|must resend THIS not A| D["Turn 2 request"]
A -.->|reintroducing this equals cache miss| X(("Cache busted"))
```
**Rule 2 (of 2):** send back your own previous *output* (the forwarded version), never the pristine original, as the leading messages next turn.
Pinning the Prefix: frozen_message_count
**Rule 1 (of 2):** pass `config.frozen_message_count` — the number of leading messages the provider has *already cached*.
**What it does:** tells the compressor "don't touch or re-derive these N leading messages — return them exactly as given."
\[
\texttt{frozen\_message\_count} = k \;\Rightarrow\; \text{messages } [1..k] \text{ returned byte-identical to input}
\]
**Why this shape:** the compressor has no memory between calls (stateless). `frozen_message_count` is how *you*, the caller, tell it "this many messages are the cached prefix — freeze them," restoring the state a stateful proxy would track automatically.
**The catch — it pins whatever you HAND it:**
- Hand it your **forwarded** output → pins the exact cached bytes → cache hit. ✅
- Hand it **pristine originals** → pins pristine bytes → you get pristine back → still a cache miss. ❌
`frozen_message_count` alone does not save you — it only guarantees round-trip fidelity for whatever you pass in.
Worked Example: A 3-Turn Agent Loop
**Setup:** agent has a system prompt + 4 tool-result messages. Turn 1 compresses tool result #2 (a long log) down.
**Turn 1:**
- Input: pristine `[sys, tool1, tool2 (2000 tok), tool3, tool4]`
- `frozen_message_count = 0` (nothing cached yet)
- Output (forwarded): `[sys, tool1, tool2' (300 tok), tool3, tool4]` ← provider caches these exact 5 messages
**Turn 2 — CORRECT pipeline:**
- New turn adds `tool5`. Caller resends **forwarded** turn-1 output + tool5:
`[sys, tool1, tool2' (300 tok), tool3, tool4, tool5]`
- Sets `frozen_message_count = 5` (the 5 the provider already cached)
- Compressor returns messages 1–5 byte-identical (frozen), processes only `tool5`.
- Provider hashes the prefix (messages 1–5): matches the cached prefix exactly → **cache hit**. Only `tool5` (the new message) is read fresh.
**Turn 2 — NAIVE pipeline (bug):**
- Caller resends **pristine** `[sys, tool1, tool2 (2000 tok), tool3, tool4, tool5]`
- Sets `frozen_message_count = 5` anyway
- Compressor freezes these 5 exactly as given — but they're pristine, so message 3 is still the **2000-token** `tool2`, not the 300-token `tool2'`
- Provider hashes this prefix: it does **not** match the cached prefix (byte 3 differs) → cache miss
- Result: **full re-read of all ~2000+ tokens in the prefix**, silently, every single turn — even though `frozen_message_count` was set "correctly."
Edge Case: Compression Isn't Uniform Over Time
**Why not just recompute compression fresh each turn?** Because compression strength depends on *position in the conversation*, not just content.
- A tool result near the end of the conversation sits inside a "recent-read protection window" → compressed lightly (or not at all).
- As the conversation grows, that same message drifts further from the end → falls **outside** the protection window → gets compressed **harder** next time it's reprocessed.
```mermaid
flowchart TD
T1["Turn 1: tool_result is near the end - light compression"] --> T5["Turn 5: same tool_result now far from end"]
T5 --> T5b["Recompressed HARDER - different bytes than turn 1's cached version"]
T5b --> Miss["Prefix no longer matches provider cache"]
```
**Consequence (prior_output_reuse):** you cannot regenerate the cached prefix by recompressing the original — you must **store and resend your actual prior forwarded output**, not recompute it.
**Cold-prefix exception:** only when the session goes idle past the provider's cache TTL is the cache already dead — that's the *one* safe moment to recompact the whole prefix from scratch (covered later; off by default).
Putting the Two Rules Together
**Cache-safe multi-turn forwarding procedure:**
1. After each turn, **store** the forwarded output you sent (not the pristine input).
2. On the next turn, build the request from **stored forwarded output + new pristine messages only**.
3. Set `config.frozen_message_count = N`, where N = number of leading messages the provider already cached.
4. Never substitute pristine originals back in for messages `1..N` — that reintroduces bytes the provider never cached.
```mermaid
flowchart LR
S["Stored: prior forwarded output"] --> M["Merge with new pristine messages"]
M --> F["Set frozen_message_count = N"]
F --> C["compress call"]
C --> R["Response: msgs 1..N unchanged, N+1.. compressed"]
R --> S2["Store this as new forwarded output"]
S2 -.next turn.-> S
```
**Pass condition check:** does your procedure (a) preserve previously forwarded messages, (b) pin the cache via `frozen_message_count`, and (c) avoid reintroducing pristine originals? All three, every turn.
**Where this leads →** 3.4 Cache Mode vs Token Mode Tradeoffs weighs *when* this cache-preserving path is worth it versus optimizing for raw token count instead.
بازگشت به دوره