End-to-End Headroom Tuning Playbook
Learning Headroom
From Pieces to a Playbook
## The problem: you know the parts, not the plan
So far you've learned each lever separately:
- Cache mode vs token mode (3.4)
- SmartCrusher retention tuning (4.5)
- CCR routing metadata (5.5)
- Observability metrics (6.4)
- Repeated-wake digests (7.1)
**But a real agent system needs all of them tuned together, in the right order.**
```mermaid
flowchart LR
A["Baseline metrics"] --> B["Workload characterization"]
B --> C["Mode selection"]
C --> D["Compressor parameters"]
D --> E["Tool profile selection"]
E --> F["Cache drift check"]
F --> G["Iterate and re-measure"]
G -.-> A
```
**Where this leads →** this node IS the capstone: the pass condition asks you to produce one end-to-end plan covering all of it.
Step 1 & 2: Baseline Collection and Workload Characterization
## Step 1 — Baseline collection
Before changing anything, capture, per representative session:
- \(T_{before}\): tokens sent to the model (pre-compression)
- \(T_{after}\): tokens actually billed/sent (post-compression)
- Savings: \(\text{savings\%} = \dfrac{T_{before}-T_{after}}{T_{before}}\times 100\)
- Cache hit rate (from CacheAligner, 6.4)
- Task accuracy / eval score
- p50 / p95 latency
*Why it works:* you can't judge a tuning change without a "before" number for every axis it could affect — token cost, cache health, AND accuracy. Tuning on token savings alone hides accuracy regressions.
### Worked example — computing the baseline
A representative session sends **12,000 tokens** before compression and **7,200 tokens** after compression.
\[
\text{savings\%} = \frac{T_{before}-T_{after}}{T_{before}}\times 100 = \frac{12000-7200}{12000}\times 100
\]
Step by step:
1. \(T_{before}-T_{after} = 12000 - 7200 = 4800\)
2. \(\dfrac{4800}{12000} = 0.4\)
3. \(0.4 \times 100 = 40\)
So this session shows **40% token savings**. On its own that looks great — but suppose cache hit rate on the same session is only **52%** (down from a healthy ~85% elsewhere) and eval score is **3 points below** the pre-compression baseline. The 40% number alone would have hidden both problems.
## Step 2 — Workload characterization
Classify the traffic that produced the baseline:
- **Turn shape**: short bursty chats vs long multi-turn sessions
- **Repeated-wake pattern**: does the agent re-invoke often with a growing history? (builds on 7.1)
- **Tool mix**: which tools return large payloads (search results, file dumps) vs small ones
- **Content type**: code-heavy, log-heavy, natural language, structured JSON
Step 3: Mode Selection and Compressor Parameters
## Step 3a — Mode selection (builds on ← 3.4)
| Signal from baseline | Choose |
|---|---|
| High cache hit rate matters, stable prefix reused often | **Cache mode** — protect the hot zone, compress only live zone |
| Prefix rarely reused (one-shot calls, no repeat wakes) | **Token mode** — optimize raw token count directly |
```tikz
\begin{tikzpicture}[scale=1]
\draw[fill=gray!20] (0,0) rectangle (3,1);
\node at (1.5,0.5) {$\text{hot zone (stable prefix)}$};
\draw[fill=blue!15] (3,0) rectangle (5,1);
\node at (4,0.5) {$\text{live zone}$};
\draw[->,thick] (1.5,-0.6) -- (1.5,-0.05);
\node at (1.5,-1) {$\text{cache mode: kept byte-identical}$};
\draw[->,thick,red] (4,-0.6) -- (4,-0.05);
\node at (4,-1) {$\text{compressed each turn}$};
\end{tikzpicture}
```
```mermaid
flowchart TD
Q{"Baseline: cache hit rate high
and prefix reused across turns?"}
Q -- yes --> CM["Cache mode:
keep hot zone byte-identical,
compress live zone only"]
Q -- no --> TM["Token mode:
optimize total tokens,
no hot-zone constraint"]
```
## Step 3b — Compressor parameter selection (builds on ← 4.5)
```python
config = HeadroomConfig(
default_mode=HeadroomMode.OPTIMIZE,
smart_crusher_target_ratio=0.3, # keep ~30% of eligible content
)
```
- Start conservative (\(target\_ratio \approx 0.4\)) if accuracy baseline is thin
- Lower toward \(0.2\)–\(0.3\) only after eval score holds steady across a tuning cycle
- **Why it works:** ratio directly trades recall of older content for token savings — dropping too fast outruns your ability to detect an accuracy regression
### Worked example — applying the decision rule
Baseline from Step 1: cache hit rate 85%, and the workload characterization shows the same session prefix is replayed on every one of 6 repeated wakes.
- Cache hit rate high? Yes (85%). Prefix reused across turns? Yes (6 repeated wakes).
- Rule says → **cache mode**.
- Since the accuracy baseline only has 20 eval examples (thin), start with \(target\_ratio = 0.4\), not 0.3, and only lower it once a tuning cycle confirms eval score is stable.
Step 4: Tool Profile Selection
## Per-tool overrides
```python
response = client.chat.completions.create(
model="gpt-4o",
messages=[...],
headroom_mode="optimize",
headroom_keep_turns=5,
headroom_tool_profiles={
"important_tool": {"skip_compression": True},
"search_results": {"target_ratio": 0.5},
"debug_logs": {"target_ratio": 0.15},
},
)
```
**Decision rule, from workload characterization (Step 2):**
- Tool output the agent must quote verbatim later (e.g. exact file diff, legal clause) → `skip_compression: True`
- Tool output that's bulky but summarizable (search snippets, logs) → lower `target_ratio`
- `headroom_keep_turns` — how many recent turns stay fully uncompressed regardless of tool
*Why it works:* a global target ratio treats every tool the same, but a 3-line calculator result and a 3,000-line log dump are not the same compression problem — tool profiles let the plan match each tool's actual information density.
### Worked example — deriving profiles from Step 2's tool mix
Step 2's tool-mix characterization for this workload found three tools in play:
1. `diff_tool` — returns an exact file diff the agent later applies verbatim → must stay byte-exact → `skip_compression: True`
2. `search_results` — returns 20 snippets, only the gist matters → summarizable → `target_ratio: 0.5` (keep about half)
3. `debug_logs` — returns thousands of log lines, mostly noise → heavily summarizable → `target_ratio: 0.15` (keep about 15%)
Each ratio traces directly back to a fact recorded in Step 2 — none of it is guessed.
Step 5: Cache Drift Remediation and CCR Policy
## Cache drift: the silent failure mode
**Cache drift** = the stable prefix stops being byte-identical across turns, so cache hit rate collapses even though token savings look fine.
```mermaid
flowchart LR
A["Change a setting"] --> B["Redeploy"]
B --> C{"CacheAligner:
hit rate vs baseline"}
C -- drop greater than threshold --> D["DRIFT DETECTED"]
C -- stable --> E["Change is cache-safe"]
D --> F["Remediate: revert layout change
or move it out of hot zone"]
```
**Common causes:** a prompt-layer edit landed in the stable prefix, a timestamp or per-request ID got baked into the hot zone, tool schema changed between calls.
### Worked example — catching drift with numbers
Baseline cache hit rate (Step 1): **85%**. Rollback threshold (set in advance, Step 6): drop of more than 10 percentage points.
After deploying a change that adds a live "last-updated" timestamp inside the system prompt:
1. New measured cache hit rate: **58%**
2. Drop = \(85\% - 58\% = 27\) percentage points
3. \(27 > 10\) → **DRIFT DETECTED**, even though token savings for the same deploy actually improved (\(T_{after}\) went down)
4. Remediation: move the timestamp out of the stable prefix into the live zone (or drop it), redeploy, re-measure — hit rate returns to ~84%, confirming the fix
## CCR policy (builds on ← 5.5)
- Set routing metadata so instruction-bearing / reversible content goes through CCR, not SmartCrusher
- Confirm CCR **lifetime** setting matches session length — a retrieval past its lifetime fails closed, not silently
*Why it matters together:* drift remediation protects the **hot zone's byte-identity**; CCR policy protects **which content is even eligible** for reversible vs lossy compression. Both must be right or the plan optimizes the wrong thing.
Step 6: The Iterative Loop and Rollback Thresholds
## The iterative tuning loop
```mermaid
flowchart TD
A["Baseline captured"] --> B["Apply ONE change"]
B --> C["Run representative traffic"]
C --> D["Compare to baseline:
tokens, cache hit, accuracy, latency"]
D -- all within thresholds --> E["Keep change,
new baseline"]
D -- any threshold breached --> F["ROLLBACK"]
F --> A
E --> B
```
## Example rollback thresholds (set BEFORE tuning, not after)
| Signal | Rollback trigger |
|---|---|
| Accuracy / eval score | drops \(> 2\) points vs baseline |
| Cache hit rate | drops \(> 10\%\) absolute |
| p95 latency | increases \(> 20\%\) |
| Token savings | negative (config made it worse) |
**Rule:** change ONE variable per loop (mode, ratio, tool profile, or CCR routing) — never several at once — so a threshold breach tells you *which* change caused it.
### Worked example — one loop iteration end to end
Baseline (from Step 1): eval score 91, cache hit 85%, p95 latency 800 ms, savings 40%.
Loop iteration: lower `smart_crusher_target_ratio` from 0.4 to 0.3 (ONE variable), redeploy, re-measure:
1. Eval score: 89 → drop of \(91-89=2\) points → threshold is "drops > 2", and 2 is not greater than 2 → **within threshold**
2. Cache hit: 84% → drop of \(85-84=1\) point → well within the 10-point threshold
3. p95 latency: 820 ms → increase of \(\frac{820-800}{800}\times100 = 2.5\%\) → within the 20% threshold
4. Token savings: 46% → improved, not negative → within threshold
5. All four signals pass → **keep the change**, and 0.3 / 89 / 84% / 820ms / 46% becomes the new baseline for the next loop
**Complete plan =** baseline → workload profile → mode → SmartCrusher ratio → tool profiles → CCR policy → drift check → thresholds → loop.
Back to course