SmartCrusher Retention Strategy
Learning Headroom
From Eligibility to Retention
**Recap — Builds on ← 4.1 SmartCrusher Eligibility Gates**
- Gates decide: *should this array be crushed at all?*
- `min_items_to_analyze` (default 5)
- `min_tokens_to_crush` (default 200)
**Today's question:** once an array passes the gates, *which items survive the crush?*
**The scenario:** a tool returns 1,000 JSON log records. Most are identical success entries. A few are errors. One is a weird outlier.
**Goal:** shrink 1,000 items down to ~50 — without losing anything the LLM actually needs to answer a question like *"find errors in the last 24 hours."*
**Where this leads →** the exact scoring math behind "relevant" and "anomalous" items is node 4.4 (Relevance Scoring for Retention); deduplication logic is node 4.3.
The Five Retention Signals
SmartCrusher scores every item across **five dimensions** before deciding what to drop:
1. **First / Last items** — pagination & recency context
2. **Error items** — 100% preserved, never dropped
3. **Anomalies** — statistical outliers ( > 2 std devs from the mean )
4. **Relevant items** — match the user's query (BM25 / embeddings)
5. **Change points** — sudden shifts in running values
```mermaid
flowchart TD
A["1000-item JSON array"] --> B{"Score each item"}
B --> C["First / Last"]
B --> D["Error items"]
B --> E["Anomalies"]
B --> F["Relevant matches"]
B --> G["Change points"]
C --> H["Retained set ~ 50 items"]
D --> H
E --> H
F --> H
G --> H
B -."ordinary, non-scoring items".-> I["Dropped"]
```
**Key idea:** these are *safety guarantees* — error items, numeric/string anomalies, and change points are kept **even if they exceed the K budget**.
Worked Example: 1,000 Log Records
**Input:** array of 1,000 API log records, `id` 1–1000, mostly `{"status": "success", "latency_ms": 40}`
**Step-by-step retention:**
| Rule | What's in the array | Kept? |
|---|---|---|
| First items | `id=1, id=2, id=3` | ✅ kept (first 3) |
| Last items | `id=999, id=1000` | ✅ kept (last 2) |
| Error item | `id=998: {"status": "error", "msg": "timeout"}` | ✅ kept — error keyword |
| Anomaly | `id=512: {"latency_ms": 4300}` (mean ≈ 40ms, std ≈ 8ms) | ✅ kept — see z-score below |
| Change point | `id=701: status flips success→error→success` | ✅ kept — running-value shift |
| Ordinary items | `id=4 ... id=997` (except above), all near-identical successes | ❌ dropped — no signal triggered |
**Computing the anomaly z-score for id=512:**
\[
z = \frac{x - \mu}{\sigma} = \frac{4300 - 40}{8} = \frac{4260}{8} \approx 532.5
\]
**Result:** ~50 items kept via representative sampling of the ordinary bulk + all flagged items, from 1,000 originally.
**Why the anomaly formula works:** \( z = \dfrac{x - \mu}{\sigma} \) measures *how many standard deviations away* a value sits. A huge \(z\) (here, over 500!) means this point could not plausibly come from the normal pattern — it's signal, not noise, so it's flagged regardless of budget.
Relevance Scoring & Representative Sampling
**Relevance scoring** — matches items to the user's actual question
- Query: *"find errors in the last 24 hours"*
- Method: BM25 keyword match or embedding similarity
- Items with high relevance score get priority even without being errors/anomalies
**Representative sampling** — the K budget isn't fixed, it's adaptive:
\[
K_{\text{items}} = 0.30\, K_{\text{start}} + 0.15\, K_{\text{end}} + 0.55\, K_{\text{scored}}
\]
- 30% of budget → array start
- 15% of budget → array end
- 55% of budget → importance-scored items (errors, anomalies, relevance, change points)
**Why this split works:** start/end give *structural* orientation cheaply (few items needed); the *bulk* of the budget goes where the actual information density is — the scored, non-obvious items — because that's what answers real questions.
**Sizing method:** Kneedle algorithm finds the point on the bigram-coverage curve where extra items stop adding new information — this sets K, not a hardcoded number.
```mermaid
flowchart LR
A["Coverage curve: items vs new info"] --> B{"Kneedle finds the knee"}
B --> C["K = elbow point"]
C --> D["30% start / 15% end / 55% scored"]
```
**Worked mini-example:** suppose Kneedle sets \(K = 50\) for our 1,000-item log array.
\[
K_{\text{start}} = 0.30 \times 50 = 15, \quad K_{\text{end}} = 0.15 \times 50 = 7.5 \approx 8, \quad K_{\text{scored}} = 0.55 \times 50 = 27.5 \approx 28
\]
So about 15 items come from the start, 8 from the end, and the remaining ~28 slots go to whichever errors, anomalies, relevant matches, and change points scored highest — this is where `id=998`, `id=512`, and `id=701` from the previous slide earn their seats.
Edge Cases & What SmartCrusher Never Sacrifices
**Safety guarantees — always kept, budget or not:**
- Error items (`error`, `exception`, `failed`, `critical`) — across **all** array types
- Numeric anomalies ( > `variance_threshold` std devs, default 2.0 )
- String-length anomalies ( > 2 std devs from mean length )
- Change points
**Tuning knobs that shape retention (from `SmartCrusherConfig`):**
| Parameter | Default | Effect |
|---|---|---|
| `max_items_after_crush` | 15 | Upper bound on retained items |
| `variance_threshold` | 2.0 | Lower = more items flagged as anomalies |
**Edge case:** if error/anomaly items alone exceed `max_items_after_crush`, SmartCrusher keeps them **all anyway** — safety guarantees override the cap.
**Edge case:** an array that's mostly errors has no "ordinary bulk" to sample from — nearly everything gets kept, and compression is naturally small. That's correct behavior, not a bug.
**Where this leads →** node 4.3 covers how near-duplicate items get merged via SimHash fingerprinting before scoring even happens, and 4.4 unpacks the BM25/embedding relevance math in full. Node 4.5 (Tuning SmartCrusher Safely) will have you adjust these exact parameters against real traffic.
Back to course