Specialized Compressors and Preservation Rules
Learning Headroom
From One Big Hammer to a Toolbox
**The problem:** a generic compressor can't tell an error line from a passed test, or a changed line from unchanged context. It either keeps too much or throws away what matters.
**Builds on ← 3.2 Content Detection and Content Routing**
- You already know Headroom detects what TYPE of content a message holds (JSON, code, or something else) and routes it accordingly.
- This node covers the 'something else' branch: once content is routed as log, search output, diff, table, or prose, it needs a compressor that understands THAT shape.
- Each specialized compressor has one job: know the structure of its input, keep the load-bearing parts, drop the noise.
**Where this leads →** 5.2 (Instruction-Bearing Content Routing) asks *should this content be compressed at all?* — but only after you trust what today's compressors preserve.
```mermaid
flowchart LR
A["Raw content"] --> B["Content detection (3.2)"]
B --> C["Log"]
B --> D["Search output"]
B --> E["Diff"]
B --> F["Table"]
B --> G["Prose"]
C --> H["LogCompressor"]
D --> I["SearchCompressor"]
E --> J["DiffCompressor"]
F --> K["Table rules"]
G --> L["TextCrusher / KompressCompressor"]
```
The Five Specialized Compressors
| Compressor | Input Type | What It Preserves | Typical Savings |
|---|---|---|---|
| `LogCompressor` | build/test logs | errors, warnings, stack traces, summaries | 85-95% |
| `SearchCompressor` | grep/ripgrep output | relevant matches, file diversity | 80-95% |
| `DiffCompressor` | unified diffs | changed lines, context | 60-80% |
| `TextCrusher` | general prose | relevant sentences, anchors | 30-60% |
| `KompressCompressor` | general text fallback | learned token scoring via ONNX | 30-50% |
Notice the savings gradient: structured, repetitive input (logs, search) compresses harder than dense or free-form input (diffs, prose).
LogCompressor: keep the needle, drop the haystack
**Input:** 500-test pytest run, one failure
```
===== test session starts =====
collected 500 items
tests/test_foo.py::test_1 PASSED
... (498 more PASSED lines) ...
tests/test_bar.py::test_fail FAILED
AssertionError: expected 5, got 3
===== 1 failed, 499 passed =====
```
**Line-by-line decision:**
- `===== test session starts =====` → KEPT (section header)
- `collected 500 items` → KEPT (summary)
- 499x `... PASSED` → DROPPED (verbose success, zero new info)
- `test_fail FAILED` → KEPT (matches FAILED)
- `AssertionError: expected 5, got 3` → KEPT (this is the debug-relevant detail)
- `===== 1 failed, 499 passed =====` → KEPT (summary)
**Compression math:**
- Total input lines: \( 2 + 500 + 1 \text{(AssertionError)} + 1 \text{(summary)} = 504 \) lines, of which 499 are PASSED lines and 5 are kept.
- Kept lines: 5 (header, collected, FAILED line, AssertionError, summary)
- Dropped lines: \( 504 - 5 = 499 \)
- Compression ratio: \( \dfrac{499}{504} \approx 0.99 \), so **~99% of lines removed**
**Result:** ~504 lines → 5 lines, compression_ratio ≈ 99%
DiffCompressor: keep the change, keep the frame around it
**Input:** a one-line fix inside `process()`
```
diff --git a/src/main.py b/src/main.py
--- a/src/main.py
+++ b/src/main.py
@@ -42,7 +42,7 @@
def process(items):
- return [x for x in items]
+ return [x.strip() for x in items if x]
```
**What's kept and why:**
- File header (`a/src/main.py`, `b/src/main.py`) → KEPT: which file changed?
- `@@ -42,7 +42,7 @@` hunk marker → KEPT: which line numbers?
- `def process(items):` context line → KEPT: which function is this inside?
- The `-` and `+` lines → KEPT: this is the actual change
**Without the anchor line**, the LLM would only see:
```
- return [x for x in items]
+ return [x.strip() for x in items if x]
```
Two floating fragments with no idea which function or file they belong to — that's why the context line is not optional.
Why only 60-80% savings: a diff is already dense — little redundant filler to strip.
```tikz
\begin{tikzpicture}[scale=1]
\node[anchor=west,font=\ttfamily\small] at (0,2.4) {@@ -42,7 +42,7 @@};
\node[anchor=west,font=\ttfamily\small] at (0,1.8) {def process(items):};
\node[anchor=west,font=\ttfamily\small,red] at (0,1.2) {- return [x for x in items]};
\node[anchor=west,font=\ttfamily\small,green!50!black] at (0,0.6) {+ return [x.strip() for x in items if x]};
\draw[dashed,gray] (-0.3,2.6) rectangle (9.3,1.5);
\node[gray,font=\small] at (4.5,2.9) {kept: hunk marker + context anchor};
\draw[dashed,blue] (-0.3,1.4) rectangle (9.3,0.3);
\node[blue,font=\small] at (4.5,-0.1) {kept: the actual change (+/-)};
\end{tikzpicture}
```
Search Output and Tables: diversity and grid preserved
**SearchCompressor example:** `grep -r "TODO"` returns 40 matches: 30 in `utils.py`, 5 in `main.py`, 5 spread across 5 other files.
- Naive truncation (keep first 20 lines, in file order) → all 20 come from `utils.py` alone, since it's listed first and has 30 matches — the LLM never learns TODOs exist anywhere else
- SearchCompressor instead samples ACROSS files → e.g. keeps a handful from `utils.py`, all 5 from `main.py`, and all 5 from the other files, so the LLM sees TODOs exist in 7 files, not 1
- Keeps file:line address + matching line per hit, drops near-duplicate hits within the same file
**structure_preservation, generalized:**
Every compressor keeps the parts of the structure that carry meaning:
- headers, hunk markers, error lines, distinct files, table headers/columns
**Tables specifically:** header row + column alignment always kept; rows pruned/summarized, but never in a way that shifts a value into the wrong column.
**Why this matters — a wrong-column example:** if a table of \( \text{(server, CPU %, memory %)} \) gets a row dropped carelessly and the remaining values re-aligned incorrectly, a memory reading could end up read as a CPU reading. That's not compression, that's corrupted data — which is why column alignment is treated as sacred, never as a place to save space.
TextCrusher and the Fallback: when structure runs out
**Input (prose):** "The deployment failed because the database connection pool was exhausted. This has happened three times this week, always during peak traffic between 2pm and 4pm. The team suspects a connection leak in the new caching layer introduced last Tuesday."
**TextCrusher's picks:**
- KEPT: "The deployment failed because the database connection pool was exhausted." (anchor — states the core fact)
- KEPT: "The team suspects a connection leak in the new caching layer introduced last Tuesday." (actionable hypothesis)
- SHORTENED, not dropped: "...happened three times this week, always during peak traffic 2pm-4pm" → "recurred 3x this week, peak hours" (frequency detail still relevant, just compressed)
**Word-count check:** original passage ≈ 50 words; compressed version ("Deployment failed: database connection pool exhausted. Recurred 3x this week, peak hours. Suspected cause: connection leak in new caching layer, added last Tuesday.") ≈ 22 words. That's \( \dfrac{50-22}{50} = 0.56 \), about **56% savings** — right in the expected 30-60% band.
Why only 30-60% savings: prose has little pure repetition; almost every sentence adds something.
**KompressCompressor (fallback):** when content isn't log/diff/search/table/recognizable prose, an ONNX model scores each token's importance and keeps the highest scorers. Learned, not hand-designed.
The Unifying Rule and What's Next
**One question, five answers:** "What does an LLM actually need from THIS content type to keep working?"
- Logs → what broke (errors/warnings/traces/summaries)
- Search output → where, across the whole codebase (diverse matches)
- Diffs → what changed, in what context (hunks + anchors)
- Tables → which value belongs to which column (headers + alignment)
- Prose → which sentences carry the actual claims (anchors, not filler)
**Where this leads →**
- 5.2 Instruction-Bearing Content Routing: decides WHETHER content should be compressed at all before these rules even apply
- 5.3 CCR Reversible Compression Architecture: makes compression reversible, so dropped detail can still be retrieved later
Today's preservation rules are the foundation both stand on.
Back to course