SmartCrusher Eligibility Gates
Learning Headroom
Why SmartCrusher Needs Gates
Recall from 3.2: ContentRouter detects JSON arrays and sends them to SmartCrusher.
But detection ≠ compression. SmartCrusher checks **eligibility gates** first.
**Why gates exist:**
- Compression itself costs tokens (scoring, restructuring, summaries)
- A tiny array has nothing worth crushing
- Crushing small payloads can *increase* net tokens — the opposite of the goal
**Builds on ← 3.2 / 1.3:** gates use the routing decision from Content Detection and the token math from Token Measurement.
**Where this leads →** 4.2 (Retention Strategy) only runs on payloads that already passed these gates.
The Two Gate Conditions
SmartCrusher applies **two independent gates**. Both must pass for compression to run.
**Gate 1 — Item count**
\[ \text{item\_count} \geq \text{min\_items\_to\_analyze} \]
default: \( \text{min\_items\_to\_analyze} = 5 \)
**Gate 2 — Token count**
\[ \text{token\_count} \geq \text{min\_tokens\_to\_crush} \]
default: \( \text{min\_tokens\_to\_crush} = 200 \)
**Builds on ← 1.3:** token_count here is the same token measurement you already learned to compute for any payload.
If **either** gate fails → payload **passes through untouched**.
```mermaid
flowchart TD
A["JSON array detected"] --> B{"item_count >= min_items_to_analyze?"}
B -- No --> P["Pass through unchanged"]
B -- Yes --> C{"token_count >= min_tokens_to_crush?"}
C -- No --> P
C -- Yes --> D["Eligible: SmartCrusher runs scoring"]
```
Worked Example: Three Payloads Through the Gates
Given: `min_items_to_analyze = 5`, `min_tokens_to_crush = 200`
**Payload A — small log list**
- 3 items, ~80 tokens total
- Gate 1: \(3 \geq 5\)? **No** → fails immediately
- Result: **PASS THROUGH** (gate 2 never even checked)
**Payload B — medium result set**
- 12 items, ~140 tokens total
- Gate 1: \(12 \geq 5\)? **Yes**
- Gate 2: \(140 \geq 200\)? **No**
- Result: **PASS THROUGH**
**Payload C — large search results**
- 40 items, ~1{,}600 tokens total
- Gate 1: \(40 \geq 5\)? **Yes**
- Gate 2: \(1600 \geq 200\)? **Yes**
- Result: **ELIGIBLE → SmartCrusher scores and compresses**
Structure Still Matters: JSON Array Eligibility
Gates only apply to payloads that are actually a **JSON array of items**.
**Eligible shape:**
```json
[{"id":1,"...":"..."}, {"id":2,"...":"..."}, "...", {"id":40,"...":"..."}]
```
→ item_count = length of array, checked against gates
**Not array-shaped → gates don't apply at all:**
```json
{"status":"ok", "data": {"nested":"object"}}
```
→ Single object, no list of items to score → **passes through**, regardless of token size
**Rule:** structure gate comes *before* the counting gates.
```mermaid
flowchart TD
A["Tool output"] --> S{"Is it a JSON array of items?"}
S -- No --> P["Pass through, count gates skipped"]
S -- Yes --> B{"item_count >= 5?"}
B -- No --> P
B -- Yes --> C{"token_count >= 200?"}
C -- No --> P
C -- Yes --> D["Eligible for compression"]
```
Overhead Reasoning: Why the Thresholds Exist
**Why not crush everything, even 1 item?**
Compression overhead includes:
- Statistical analysis (mean, variance, outlier detection)
- Relevance scoring (BM25/embeddings) per item
- Rebuilding a smaller JSON structure
Each of these costs *processing* and can add *tokens* (e.g., a summary note explaining what was dropped).
**Net savings condition (informal):**
\[ \text{tokens\_saved\_by\_dropping\_items} > \text{tokens\_spent\_on\_overhead} \]
Small arrays fail this: few items to drop, so little to save, but overhead is roughly fixed.
**Worked check — why 200 tokens is a sane floor:**
1. Original payload: 150 tokens
2. Crushed payload: 100 tokens
3. Raw tokens saved: \(150 - 100 = 50\)
4. Overhead of running the crush pipeline (scoring + rebuilt structure + drop-note): estimated 40–60 tokens
5. Net saving: \(50 - 50 = 0\) (using the midpoint of the overhead estimate, 50)
6. Net saving ≈ 0, sometimes negative → **not worth running the pipeline**, hence the gate at 200 tokens
Putting It All Together: Five Payloads
Decision chain, in order:
1. Is it a JSON array of items?
2. \( \text{item\_count} \geq 5 \)?
3. \( \text{token\_count} \geq 200 \)?
All three yes → **COMPRESS**. Any no → **PASS THROUGH**.
| Payload | Shape | Items | Tokens | Verdict |
|---|---|---|---|---|
| 1. User profile object | single object | — | 300 | Pass through (not an array) |
| 2. Error log | array | 4 | 220 | Pass through (item gate fails) |
| 3. Weather readings | array | 8 | 90 | Pass through (token gate fails) |
| 4. API search results | array | 60 | 3,200 | **Compress** |
| 5. Config list | array | 6 | 210 | **Compress** (barely clears both) |
**Where this leads →** Payloads 4 and 5 now move to 4.2, where SmartCrusher decides *which* items survive.
Back to course