Tuning SmartCrusher Safely
Learning Headroom
Why Tuning SmartCrusher Is the Real Job
**Recap the pipeline:**
- 4.3: dedup + change points find *what's structurally distinct*
- 4.4: relevance scoring finds *what matters to the query*
- SmartCrusher scores every item across 5 dimensions, then keeps a shortlist
**Builds on ←** Deduplication/Change Points (4.3) and Relevance Scoring (4.4): those give SmartCrusher its *signals*. This lesson gives you the *knobs*.
**The problem tuning solves:**
A fixed default config (`max_items_after_crush=15`, `first_fraction=0.3`, ...) is a guess. Real tool outputs vary wildly:
- 1,000-row API log → needs heavy compression
- 40-row error report → barely needs touching
**Your job as a tuner:** hit a *token target* while never losing errors, recency, or query-relevant records.
**Where this leads →** 6.3 (Simulation-Driven Configuration Comparison) and 7.2 (End-to-End Headroom Tuning Playbook) both assume you can already do this by hand.
The Five Knobs, One at a Time
**1. `max_items_after_crush`** (int, default 15)
Hard cap on surviving items. *Why it works:* directly controls output size — the single biggest lever on token count.
**2. `variance_threshold`** (float, default 2.0)
Anomaly = value more than this many standard deviations from the mean.
\[ |x_i - \mu| > t \cdot \sigma \]
*Why it works:* smaller \(t\) → more items flagged as anomalies → more kept. Larger \(t\) → only extreme outliers survive.
**3. `first_fraction` / `last_fraction`** (default 0.3 / 0.15)
Fraction of the array *always* kept from the start and end, regardless of score.
*Why it works:* guarantees pagination context and recency — the newest events survive even if they score low statistically.
**4. `preserve_change_points`** (bool)
When true, items marked as transitions (from 4.3's change-point detection) are exempt from being dropped.
*Why it works:* a transition (e.g. status flips from `ok` to `failed`) carries information no single point around it does — losing it breaks the story of the data.
**5. Tool-specific profile**
Different tools (logs vs. search results vs. DB rows) need different combinations of the above — one global config rarely fits all.
Picture It: What Survives the Crush
```tikz
\begin{tikzpicture}[scale=0.62, every node/.style={font=\small}]
\foreach \i in {0,...,19} {
\pgfmathsetmacro{\x}{\i*0.9}
\ifnum\i<4
\fill[blue!55] (\x,0) rectangle (\x+0.8,0.8);
\else\ifnum\i>15
\fill[blue!55] (\x,0) rectangle (\x+0.8,0.8);
\else\ifnum\i=7
\fill[red!70] (\x,0) rectangle (\x+0.8,0.8);
\else\ifnum\i=11
\fill[orange!80] (\x,0) rectangle (\x+0.8,0.8);
\else\ifnum\i=13
\fill[violet!70] (\x,0) rectangle (\x+0.8,0.8);
\else
\fill[gray!30] (\x,0) rectangle (\x+0.8,0.8);
\fi\fi\fi\fi\fi
}
\node at (1.6,-0.6) {\footnotesize first\_fraction};
\node at (15.5,-0.6) {\footnotesize last\_fraction};
\node[red!70!black] at (6.3,1.3) {error};
\node[orange!80!black] at (9.9,1.3) {anomaly};
\node[violet!70!black] at (11.7,1.3) {change pt};
\node[gray] at (5,1.9) {gray = dropped (score too low)};
\end{tikzpicture}
```
**The key insight**
Every knob answers one question: which gray boxes should turn a color? Tighten a threshold and fewer boxes are protected — the array shrinks. Loosen it and more survive.
Worked Example: Hitting a Token Target on a Real Payload
**Given:** a monitoring tool returns 400 JSON records (health checks), current output = 9,800 tokens. Target: **≤ 2,500 tokens**, must preserve: all `status=error` records, the last 20 minutes of checks, and any records matching the query "database timeout".
**Step 1 — Baseline default config:**
`max_items_after_crush=15, first_fraction=0.3, last_fraction=0.15, variance_threshold=2.0`
Simulate: `plan.tokens_saved = 6,200`.
Tokens after \(= 9800 - 6200 = 3600\). Still over 2,500 — defaults aren't enough.
**Step 2 — Diagnose why:**
\(0.3 \times 400 = 120\) items are kept from the start alone by `first_fraction=0.3` — far more than needed, since "last 20 minutes" is a *recency* requirement, not a *first-chunk* requirement.
**Step 3 — Tune the fraction knobs:**
- New `first_fraction = 0.02` → \(0.02 \times 400 = 8\) items, just enough for opening context.
- New `last_fraction = 0.05` → \(0.05 \times 400 = 20\) items ≈ last 20 minutes at 1 check/min.
**Step 4 — Tighten the cap:**
`max_items_after_crush`: \(15 \to 12\).
**Step 5 — Re-simulate:**
`plan.tokens_saved = 7,450`.
Tokens after \(= 9800 - 7450 = 2350\).
\(2350 \le 2500\) ✓ target met.
**Step 6 — Verify preservation (non-negotiable):**
- Errors: kept (100% rule, untouched by any knob).
- Query match "database timeout": kept (relevance scoring, 4.4, ranks high scorers above the cap regardless of `max_items_after_crush`).
- Recency: kept via `last_fraction=0.05` covering the last 20 items ≈ last 20 minutes.
Edge Cases and the Misconception That Breaks Configs
**The misconception to kill:** "Lower `max_items_after_crush` always saves the most tokens."
**Reality:** if `first_fraction + last_fraction` alone already guarantees more items than `max_items_after_crush` allows, the cap can't be honored cleanly — the algorithm must reconcile them (fractions guarantee a *floor*, the cap sets a *ceiling*). Always check:
\[ \lceil \text{first\_fraction} \times n \rceil + \lceil \text{last\_fraction} \times n \rceil \le \text{max\_items\_after\_crush} \]
**Check it against Slide 4's numbers:** \( \lceil 0.02 \times 400 \rceil + \lceil 0.05 \times 400 \rceil = 8 + 20 = 28 \). If you'd set `max_items_after_crush = 10` here, \(10 < 28\) — a direct conflict, since the fractions alone promise more items than the cap allows.
**Small-array skip:** if item count \( < \text{min\_items\_to\_analyze} \) (default 5) or tokens \( < \text{min\_tokens\_to\_crush} \) (default 200), SmartCrusher does **nothing** — tuning a config for a 3-item array is pointless, it never fires.
**Tool-specific profiles (the 5th micro-skill):**
| Tool type | Suggested profile |
|---|---|
| Paginated search results | high `first_fraction`, low `last_fraction` |
| Time-series / logs | low `first_fraction`, high `last_fraction`, `preserve_change_points=True` |
| Error/status reports | small `max_items_after_crush` is safe — errors are preserved regardless |
**Where this leads →** 6.3 lets you *simulate several profiles side by side* before shipping one; 7.2 shows how to assign a profile per tool across a whole agent.
بازگشت به دوره