Observability Metrics and Tuning Decisions
Learning Headroom
From Simulations to Live Signals
**The gap simulation leaves open**
Simulation-driven comparison (6.3) tells you which config wins on recorded traffic. It cannot tell you if that config is still winning next week, on real traffic, under real load.
**Why metrics matter now**
- Builds on ← 6.3 Simulation-Driven Configuration Comparison: replay compared configs offline; live metrics check the chosen config in production.
- Builds on ← 2.4 CacheAligner as an Observability Detector: that detector emits the cache hit/miss signals we now read as ongoing metrics.
- Builds on ← 3.4 Cache Mode and Token Mode Tradeoffs: metrics are how you tell, after the fact, whether that tradeoff is paying off.
- Where this leads → 7.2 End-to-End Headroom Tuning Playbook: this lesson's diagnostic method becomes step one of the full tuning workflow.
**Today's target skill**
Given a snapshot of compression, cache, and latency metrics, decide: change the threshold, change retention, change the mode, or change prompt assembly — or leave it alone.
The Metric Families You're Reading
**Compression & savings metrics**
- \(\text{tokens\_saved} = \text{headroom\_tokens\_saved\_total}\) — raw tokens removed from prompts this session, across message compression AND tool-schema deferral.
- \(\text{compression\_ratio} = \dfrac{\text{tokens after}}{\text{tokens before}}\), tracked as a histogram — read the median with \(\text{histogram\_quantile}(0.5, \ldots)\).
- transform_usage — counts of WHICH compressor fired (SmartCrusher, CCR, specialized routers) per request.
**Cache & latency metrics**
- \(\text{cache\_hit\_rate} = \dfrac{\text{hits}}{\text{hits}+\text{misses}}\) — fraction of requests that reused a cached prefix instead of reprocessing it.
- \(\text{latency\_p99} = \text{histogram\_quantile}(0.99, \text{headroom\_latency\_seconds\_bucket})\) — worst-case request time, tail not average.
- overhead_ms — time Headroom itself spends compressing, separate from model latency.
**Why these four together, not one alone**
A single metric lies by omission. High tokens saved with low cache hit rate can mean you're compressing so aggressively you're breaking cache-safe prefixes. You need all four families read as one picture.
Worked Example 1 — Healthy vs. Degrading Snapshot
**Two snapshots, same service, one week apart**
Given (Prometheus counters at two points in time):
Week 1 — hits=8400, misses=600, tokens_before_total=1,200,000, tokens_after_total=480,000, tokens_saved_total=720,000, latency p99=1.8s
Week 2 — hits=6000, misses=3000, tokens_before_total=1,200,000, tokens_after_total=360,000, tokens_saved_total=840,000, latency p99=3.1s
**Step 1 — cache hit rate**
\[
\text{Week 1: } \frac{8400}{8400+600} = \frac{8400}{9000} = 0.933 \rightarrow 93.3\%
\]
\[
\text{Week 2: } \frac{6000}{6000+3000} = \frac{6000}{9000} = 0.667 \rightarrow 66.7\%
\]
**Step 2 — compression ratio (after/before)**
\[
\text{Week 1: } \frac{480{,}000}{1{,}200{,}000} = 0.40 \qquad \text{Week 2: } \frac{360{,}000}{1{,}200{,}000} = 0.30
\]
**Step 3 — tokens saved changed**
\[
840{,}000 - 720{,}000 = 120{,}000 \qquad \frac{120{,}000}{720{,}000} = 0.167 \rightarrow +16.7\%
\]
**Step 4 — latency changed**
\[
3.1 - 1.8 = 1.3\text{s} \qquad \frac{1.3}{1.8} = 0.72 \rightarrow +72\%
\]
**Diagnosis**
Tokens saved went UP and compression ratio went DOWN (more aggressive) — looks good in isolation. But cache hit rate dropped 26.6 points and p99 latency jumped 72%. The extra compression is very likely rewriting the stable prefix, breaking byte-identical cache reuse (2.4, 2.2), forcing full reprocessing on 1 in 3 requests instead of 1 in 15.
The Diagnostic Decision Map
```mermaid
flowchart TD
A["Read snapshot"] --> B{"Cache hit rate
dropped sharply?"}
B -- yes --> C{"Compression ratio
also dropped a lot?"}
C -- yes --> D["Threshold too aggressive
Raise compression threshold"]
C -- no --> E["Prefix layout drifting
Check prompt assembly / stable-prefix order"]
B -- no --> F{"Compression ratio
stuck near 1.0?"}
F -- yes --> G{"Transform usage
mostly none?"}
G -- yes --> H["Threshold too conservative
Lower compression threshold"]
G -- no --> I["Retention too generous
Tighten SmartCrusher retention"]
F -- no --> J{"Latency p99 high
but cache hit rate fine?"}
J -- yes --> K["Wrong mode for workload
Revisit cache vs token mode"]
J -- no --> L["System healthy
leave configuration alone"]
```
**Symptom to lever, at a glance**
- Cache hit rate falls + ratio falls together → threshold too aggressive, rewriting the stable prefix
- Cache hit rate falls + ratio stable → prompt-assembly drift, something moved into or reordered the stable prefix
- Ratio near 1.0 + transform usage mostly none → threshold too conservative, nothing is being compressed
- Ratio near 1.0 + transform usage active but small savings → retention too generous, SmartCrusher keeps too much
- Latency high + cache hit rate fine → mode mismatch, cache mode overhead not worth it for this traffic shape
Worked Example 2 — Applying the Decision Map
**A different service, different symptom**
Given (single snapshot):
\[
\text{cache\_hits}=9200,\ \text{cache\_misses}=800 \;\Rightarrow\; \text{hit rate} = \frac{9200}{10000} = 0.92 \;(92\%,\ \text{healthy})
\]
\[
\text{tokens\_before}=900{,}000,\ \text{tokens\_after}=873{,}000 \;\Rightarrow\; \text{ratio} = \frac{873{,}000}{900{,}000} = 0.97 \;(\text{barely compressing})
\]
transform_usage: SmartCrusher fired on 640 of 10,000 requests (6.4%), CCR fired on 0
tokens_saved_total this window = 27,000 (small)
latency p99 = 1.1s (fine)
**Walk the map**
- Step 1: cache hit rate dropped sharply? No — 92% is healthy.
- Step 2: compression ratio stuck near 1.0? Yes — 0.97.
- Step 3: transform usage mostly 'none'? Yes — only 6.4% of requests triggered any compressor.
**Diagnosis**
Cache is healthy, latency is fine, but almost nothing is being compressed. Ratio near 1.0 with transforms rarely firing means the compression threshold is set too conservatively — content that should qualify for compression is being left alone. Fix: lower the threshold so more eligible content is routed to SmartCrusher/CCR, then re-check compression ratio and tokens saved on the next window — NOT a retention, mode, or prompt-assembly change.
Edge Cases and What Metrics Can't Tell You
**Cases that need extra care**
- Low traffic windows: a 66% hit rate on 9 requests is noise, not signal — wait for enough volume before diagnosing, or widen the time window.
- Cold start: right after a deploy or restart, cache hit rate is naturally near 0% until the prefix warms up — don't diagnose a threshold problem from a cold cache.
- overhead_ms rising alongside tokens_saved: Headroom's own compression work is now the bottleneck, not the model — this points at mode or retention complexity, not threshold.
- Metrics show WHAT changed, never WHY on their own — pair a metrics diagnosis with the simulation replay from 6.3 before shipping a config change to production.
**Tie back to the goal**
Diagnosing from metrics (this lesson) plus comparing candidate configs offline (6.3) plus scoping where a change applies (6.1) are the three legs of the full playbook in 7.2.
بازگشت به دوره