Simulation-Driven Configuration Comparison
Learning Headroom
From Tuning Knobs to Proving They Work
**Builds on ← 6.1 (scopes) and 6.2 (budget/output buffer)**
You've set config knobs: eligibility gates, retention strategy, context budget, output buffer.
**The question those lessons couldn't answer:** which combination is actually best for THIS traffic?
**The answer:** run simulation mode — replay real messages through the compression pipeline, no LLM call, no cost.
```mermaid
flowchart LR
A["Your messages"] --> B["headroom.simulate"]
B --> C["SimulationResult"]
C --> D["tokens_before / tokens_after"]
C --> E["transforms_applied"]
C --> F["waste_signals"]
```
**Where this leads →** 6.4 turns these same numbers into ongoing observability metrics and live tuning decisions.
Reading a SimulationResult
**SimulationResult (Python)**
```
tokens_before int tokens before compression
tokens_after int tokens after compression
tokens_saved int tokens removed
savings_percent float % of tokens saved
transforms_applied list which compressors fired
waste_signals obj breakdown of waste sources
```
**Core identity — why it works:**
\[ \text{tokens\_saved} = \text{tokens\_before} - \text{tokens\_after} \]
\[ \text{savings\_percent} = \frac{\text{tokens\_saved}}{\text{tokens\_before}} \times 100 \]
Subtraction gives the raw win; dividing by the *starting* size turns it into a comparable rate — a 500-token save means very different things on a 2,000-token vs a 50,000-token prompt.
Worked Example: Comparing Three Configs
**Setup:** Same 10 messages (agent log with JSON tool results + code), simulated under 3 configs.
| Config | tokens_before | tokens_after | tokens_saved | savings_percent | transforms_applied |
|---|---|---|---|---|---|
| A: cache mode, default gates | 20,000 | 17,000 | 3,000 | 15.0% | [SmartCrusher] |
| B: token mode, balanced profile | 20,000 | 6,200 | 13,800 | 69.0% | [SmartCrusher, Kompress] |
| C: token mode, agent-90 profile | 20,000 | 2,100 | 17,900 | 89.5% | [SmartCrusher, CodeCompressor, Kompress] |
**Step-by-step for Config A:**
1. tokens_saved = 20,000 − 17,000 = 3,000
2. savings_percent = 3,000 / 20,000 = 0.15
3. 0.15 × 100 = 15.0%
**Step-by-step for Config B:**
1. tokens_saved = 20,000 − 6,200 = 13,800
2. savings_percent = 13,800 / 20,000 = 0.69
3. 0.69 × 100 = 69.0%
**Step-by-step for Config C:**
1. tokens_saved = 20,000 − 2,100 = 17,900
2. savings_percent = 17,900 / 20,000 = 0.895
3. 0.895 × 100 = 89.5%
**Requirement:** save ≥ 60% tokens AND keep CodeCompressor active (preserve code structure).
→ A saves only 15% — fails the savings requirement outright.
→ B saves 69% but transforms_applied lacks CodeCompressor — fails the preservation requirement.
→ C saves 89.5% and includes CodeCompressor — meets both.
**C is the only config meeting both requirements.**
Waste Signals: Diagnosing WHY, Not Just How Much
**WasteSignals breakdown (tokens attributed to each cause):**
```
json_bloat_tokens formatting waste in JSON (indentation, repeated keys)
html_noise_tokens leftover HTML tags/markup
whitespace_tokens excess blank space/newlines
dynamic_date_tokens timestamps that churn every turn
repetition_tokens repeated content across messages
```
**Same example, Config A's 3,000 waste_signals breakdown:**
| Signal | Tokens |
|---|---|
| json_bloat_tokens | 1,800 |
| whitespace_tokens | 700 |
| repetition_tokens | 500 |
| html_noise_tokens | 0 |
| dynamic_date_tokens | 0 |
Check: 1,800 + 700 = 2,500; 2,500 + 500 = 3,000; 3,000 + 0 + 0 = 3,000 = tokens_saved. ✓
**Why it matters:** if json_bloat_tokens dominates, the fix is tightening SmartCrusher's JSON retention (recall ← 4.5 Tuning SmartCrusher Safely) — not raising the overall budget.
Designing a Fair Configuration Experiment
**Experiment design checklist:**
1. Fix the input — same message set across all configs (else comparison is meaningless)
2. Vary ONE setting per config where possible (mode, profile, gate threshold)
3. Run ≥ 3 configs — enough to see a trend, not just a coin flip
4. Record tokens_before/after, savings_percent, transforms_applied, waste_signals for each
5. Score against explicit requirements (e.g. "≥60% saved AND preserve code structure") — not just "highest %"
**Common trap:** picking the config with the single highest savings_percent while ignoring that it dropped a required transform (recall Config B above) or broke a preservation rule.
**Reminder ← 6.2:** context budget and output buffer settings are exactly the kind of "one setting" you vary between runs.
**Where this leads →** 6.4 takes this same experiment habit — fix, vary, record, score — and applies it continuously in production using live observability metrics.
بازگشت به دوره