Cache Mode and Token Mode Tradeoffs
Learning Headroom
Two Ways to Save Tokens
**The core tension**
You can freeze the prompt to protect the provider's prefix cache, OR rewrite the prompt aggressively to shrink token count. Doing both at once is not possible on the same turn.
**Builds on ← 2.5 and 3.3**
- 2.5 (Cache-Safe Multi-Turn Forwarding): forwarding prior turns byte-identical is what keeps a prefix cache hit alive.
- 3.3 (Live-Zone-Only Compression): compressing only the newest turn, leaving old turns untouched, is exactly what protects that stable prefix.
- 3.4 asks: when do you WANT that protection, and when do you trade it away for raw compression?
Cache Mode: Freeze the Past, Compress the Delta
**Cache mode behavior**
\[
\text{prompt}_t = \underbrace{P_{0..t-1}}_{\text{byte-identical, frozen}} \;\Vert\; \underbrace{C(\Delta_t)}_{\text{compressed newest turn}}
\]
**Why this shape?**
The provider's cache only hits if the bytes it already has match the new prefix exactly. One changed character anywhere in turns 0 through t-1 invalidates the cache for the whole prefix. So cache mode never edits old turns — it only ever appends compressed content at the end.
**What you get**
- Savings show up as cheaper cached-input billing, not as a big 'tokens compressed' number.
- The compression tile can read near zero on short prompts — that's expected, not a bug.
- This is the effective default in Headroom's 'coding' profile, because long coding sessions repeat the same huge prefix turn after turn.
Token Mode: Maximize Compression, Risk the Cache
**Token mode behavior**
\[
\text{prompt}_t = C\big(P_{0..t-1} \Vert \Delta_t\big)
\]
**Why this shape?**
To hit an aggressive savings target (say ~70% or ~90%), you can't leave old turns untouched forever — they're most of the prompt's bulk. So token mode rewrites prior turns too. That maximizes visible compression but almost certainly changes bytes in the prefix, which breaks the provider's cache match.
**What you get**
- Big 'tokens compressed' numbers on the dashboard — this restores the old 0.27.0-style compression figures.
- Provider prefix-cache hit rate drops, because prior turns are no longer byte-identical across calls.
- Selected via `--mode token`, or savings profiles like `balanced` (~70% target) or `agent-90` (~90% target).
The Tradeoff, Side by Side
**Cache mode vs token mode**
| Dimension | Cache mode | Token mode |
|---|---|---|
| Prior turns | frozen, byte-identical | rewritten as needed |
| Compression scope | newest turn delta only | entire prompt |
| Dashboard 'tokens saved' | can look near-zero | large, visible number |
| Where savings actually land | cheap cached-input billing | fewer raw tokens billed |
| Provider prefix-cache hit rate | preserved | reduced or lost |
| Default in | 'coding' savings profile (default) | opt-in via `--mode token` |
```mermaid
flowchart LR
subgraph CacheMode["Cache Mode"]
P1["Turns 0..t-1 frozen"] --> J1[Join]
D1["Delta t compressed"] --> J1
J1 --> O1["Prefix cache HIT: cheap cached tokens"]
end
subgraph TokenMode["Token Mode"]
P2["Turns 0..t-1"] --> C2["Compress whole prompt"]
D2["Delta t"] --> C2
C2 --> O2["Prefix cache MISS: fewer raw tokens"]
end
```
Worked Example: Long Coding Session
**Workload: a coding agent, 40 turns, same large system prompt + file context every call**
**Setup:** System prompt + repo context = 18,000 tokens, identical every turn. Each new turn (user message + tool result) adds about 400 tokens.
**Step 1 — Prior-turn token count by turn 40:**
\[
39 \times 400 = 15{,}600 \text{ tokens of prior deltas}
\]
\[
18{,}000 + 15{,}600 = 33{,}600 \text{ tokens total prior history}
\]
**Step 2 — If we used token mode on turn 40:**
The compressor rewrites the full prompt (33,600 prior tokens + this turn's 400 tokens = 34,000 tokens) targeting a ~70% savings, i.e. keeping ~30%:
\[
34{,}000 \times 0.30 = 10{,}200 \text{ tokens}
\]
But every one of those 18,000 prefix tokens got rewritten at least once → provider cache never matches → all 10,200 tokens billed at FULL input price.
**Step 3 — If we use cache mode on turn 40:**
Prior 33,600 tokens forwarded byte-identical → provider recognizes the match → billed at CACHED input rate (say ~10% of list price).
Only the newest delta (~400 tokens) is compressed, say down to ~150 tokens, and billed at full price.
**Step 4 — Compare effective cost (illustrative units: full price = 1.0/token, cached = 0.1/token):**
\[
\text{Token mode cost} = 10{,}200 \times 1.0 = 10{,}200 \text{ units}
\]
\[
\text{Cache mode cost} = (33{,}600 \times 0.1) + (150 \times 1.0) = 3{,}360 + 150 = 3{,}510 \text{ units}
\]
**Conclusion:** Even though token mode shows a bigger 'tokens compressed' number, cache mode wins on actual billed cost here — about 3x cheaper — because the frozen prefix is huge and repeats every turn.
Worked Example: Short One-Off Task
**Workload: single-shot Q&A, no repeated calls, one giant pasted document**
**Setup:** User pastes a 25,000-token document and asks one question. There is no turn 2, no repeated call — the conversation ends after this response.
**Step 1 — Cache mode here:**
Cache mode freezes 'prior turns' and compresses only the newest delta — but there ARE no prior turns yet, and there's no future turn 2 to benefit from a cache hit either. The prefix cache saves you money only on the NEXT call reusing this prefix. Since this is one-shot, that benefit is worth zero, so the whole 25,000 tokens is billed at full price:
\[
25{,}000 \times 1.0 = 25{,}000 \text{ units}
\]
**Step 2 — Token mode here:**
Compress the whole 25,000-token document toward the ~90% `agent-90` target, i.e. keep ~10%:
\[
25{,}000 \times 0.10 = 2{,}500 \text{ tokens}
\]
Billed once, at full price, but on a much smaller number:
\[
2{,}500 \times 1.0 = 2{,}500 \text{ units}
\]
**Step 3 — Compare:**
Cache mode cost (this call, no future reuse) ≈ 25,000 units.
Token mode cost ≈ 2,500 units.
**Conclusion:** With no repeated calls, there's no prefix to protect — cache mode's whole advantage disappears. Token mode wins decisively, by 10x.
**Mode-selection rule**
Ask: will this prefix be reused across MANY future calls? Many reuses + stable prefix → cache mode. One-shot or highly variable prompts → token mode.
Edge Cases and What's Next
**Common misconception**
A near-zero 'tokens compressed' number does NOT mean compression is broken. In cache mode it often means savings moved to the prefix-cache-impact panel. Check that panel before concluding anything is wrong.
**Mixed / edge workloads**
- Long session that occasionally pastes a huge one-off document: consider cache mode as the session default, since most turns reuse the prefix — the rare huge paste is one delta, not a reason to abandon the whole strategy.
- Session where earlier turns constantly change (e.g. a live-edited scratchpad resent each call): the prefix was never stable anyway, so cache mode buys nothing — token mode is the honest choice.
- You can switch modes per profile/run (`--mode token` vs default cache) — it's a per-workload decision, not a permanent global setting.
**Where this leads**
- 6.4 Observability Metrics and Tuning Decisions: you'll read the prefix-cache-impact and compression-vs-cache dashboard panels to CONFIRM which mode is actually winning for real traffic.
- 7.2 End-to-End Headroom Tuning Playbook: mode selection becomes one deliberate step in a full tuning workflow.
Back to course