Deduplication, Similarity, and Change Points
Learning Headroom
From Retention Strategy to Retention Detail
**Builds on ← 4.2 SmartCrusher Retention Strategy**
You already know SmartCrusher keeps the *start*, the *end*, and a scored sample of the *middle* of a big array (via `first_fraction`, `last_fraction`, `max_items_after_crush`).
But before it can even decide *what to score*, it has to answer three prior questions:
1. Which items are exact duplicates? → **dedup_identical_items**
2. Which items are *basically* duplicates? → **similarity_threshold**, **uniqueness_threshold**
3. Which items mark a real turning point that must never be silently dropped? → **preserve_change_points**
**Where this leads →** 4.5 (Tuning SmartCrusher Safely) is exactly about turning these knobs without breaking things — today you learn what each knob *does* so tomorrow's tuning makes sense.
Identical-Item Deduplication
**dedup_identical_items** (bool, default `True`)
Removes items that are **byte-for-byte identical** — every field matches exactly.
```mermaid
flowchart LR
A["Item A: status=ok, msg=Success"] --- B["Item B: status=ok, msg=Success"]
B -.identical.-> C["Kept: 1 representative + count=2"]
D["Item C: status=ok, msg=Success, ts differs"] -->|not identical if ts counts| E["Kept separately"]
```
**Worked example** — 1,000 search results:
- 997 items: `{status: "ok", message: "Success"}`, all sharing the same timestamp field too → truly identical on every field
- 2 items: same status and message, but a *different* timestamp field → **not identical** (timestamp is part of the record, so these 2 stay separate from the 997 and from each other unless their timestamps also match)
- 1 item: `{status: "error", message: "Connection timeout"}`
Step by step with dedup ON:
1. Group items where **every field** matches exactly.
2. The 997 fully-matching items form one group → collapse to 1 representative + `count: 997`.
3. The 2 items with differing timestamps don't match the group *or* each other → each stays as its own item (2 items).
4. The 1 error item matches nothing → stays as its own item (1 item).
**Result:** 1,000 items → 1 + 2 + 1 = **4 items** before any further scoring even happens.
Near-Duplicate Grouping: similarity_threshold
**similarity_threshold** (float, default `0.8`)
Items with a similarity score **above this threshold** get merged into one group (with a count), even if not byte-identical.
\[ \text{similarity}(a,b) \in [0,1], \quad \text{merge if } \text{similarity}(a,b) \ge \text{similarity\_threshold} \]
- Higher threshold (→1.0): only near-exact matches merge — **less** grouping, more items kept
- Lower threshold (→0.0): loosely related items merge — **more** grouping, fewer items kept
**Worked example** — log messages:
| Item | Text | Similarity to A |
|---|---|---|
| A | "Connection timeout on host db-1" | — |
| B | "Connection timeout on host db-2" | 0.92 |
| C | "Connection timeout after 30s" | 0.75 |
| D | "Disk write failed" | 0.10 |
With `similarity_threshold = 0.8`:
1. Compare B to A: \(0.92 \ge 0.8\) → merge B into A's group. Group = {A, B}, count = 2.
2. Compare C to A: \(0.75 < 0.8\) → does not merge. C stays separate.
3. Compare D to A: \(0.10 < 0.8\) → does not merge. D stays separate.
4. Result: 3 groups — {A, B} (count 2), {C}, {D}.
Now drop the threshold to `0.7`:
1. Compare C to A again: \(0.75 \ge 0.7\) → now merges too.
2. Result: 2 groups — {A, B, C} (count 3), {D}.
3. You've lost the distinction that C's timeout had a different cause (a 30-second limit) rather than a specific host failing.
Uniqueness Threshold: the Other Side of the Coin
**uniqueness_threshold** (float, default `0.1`)
During scoring, an item needs a **uniqueness score above this bar** to be preserved as its own distinct entry rather than folded away as unremarkable.
\[ \text{keep distinctly if } \text{uniqueness}(item) \ge \text{uniqueness\_threshold} \]
- Low threshold (0.1, default): even mildly distinct items are kept separately → more variety preserved
- High threshold (e.g. 0.5): only strongly unique items survive as their own entry → aggressive shrinkage
**Why uniqueness is scored this way:** an item's uniqueness score roughly tracks *how rarely that kind of item appears* in the array — items that repeat constantly contribute little new information each time, so their score is low; items that appear once or almost never carry more information, so their score is high.
**Worked example** — 200 status items, mostly `"ok"`, a few `"degraded"`, one `"critical"`:
- `"ok"` items: appear constantly (about 180 of 200) → low uniqueness score ≈ 0.05
- `"degraded"` items: appear occasionally (about 19 of 200) → medium uniqueness score ≈ 0.3
- `"critical"` item: appears once (1 of 200) → high uniqueness score ≈ 0.9
Step by step with `uniqueness_threshold = 0.1`:
1. Check `"ok"`: \(0.05 < 0.1\) → falls below the bar → summarized/dropped from individual listing.
2. Check `"degraded"`: \(0.3 \ge 0.1\) → clears the bar → kept distinctly.
3. Check `"critical"`: \(0.9 \ge 0.1\) → clears the bar → kept distinctly.
Now raise the threshold to `uniqueness_threshold = 0.4`:
1. Check `"degraded"` again: \(0.3 < 0.4\) → now also falls below the bar → folded away.
2. Check `"critical"` again: \(0.9 \ge 0.4\) → still clears it.
3. Result: only the one `"critical"` item survives as distinct.
Change-Point Preservation
**preserve_change_points** (bool, default `True`)
Guarantees items marking a **significant transition** in the sequence survive compression — regardless of dedup, similarity, or uniqueness scoring.
```tikz
\begin{tikzpicture}[scale=1.0]
\draw[->] (0,0) -- (9,0) node[right] {$\text{items}$};
\draw[->] (0,0) -- (0,3) node[above] {$\text{value}$};
\foreach \x in {0.5,1,1.5,2,2.5,3,3.5,4}
\fill[blue] (\x,1) circle (2pt);
\fill[red] (4.5,2.4) circle (3pt);
\node[above] at (4.5,2.6) {$\text{change point}$};
\foreach \x in {5,5.5,6,6.5,7,7.5,8,8.5}
\fill[blue] (\x,2.4) circle (2pt);
\draw[dashed,gray] (4.5,0) -- (4.5,2.4);
\node[below] at (4.5,0) {\small item 9};
\end{tikzpicture}
```
**Worked example** — CPU usage log, 100 readings:
- Items 1–8: hover around 20% (near-identical → dedup/similarity would merge most of them)
- Item 9: jumps to 85% — a real change point
- Items 10–100: hover around 88%
Step by step:
1. Items 1–8 are all close in value → similarity_threshold merges most of them into one group around "~20%".
2. Items 10–100 are all close in value → similarity_threshold merges most of them into one group around "~88%".
3. Item 9, scored alone, might get a low uniqueness/similarity score since it's just 1 item among 100 — a naive scorer could drop it.
4. With `preserve_change_points = True`: the pipeline detects the jump from ~20% to 85% and force-keeps item 9, even though it's buried in a merged region — it's the evidence *something happened here*.
5. With `preserve_change_points = False`: item 9 can be scored low and dropped like any other item, silently erasing the moment CPU usage spiked.
Putting the Four Knobs Together
**Pipeline order (conceptually):**
```mermaid
flowchart TD
A["Raw array"] --> B{"dedup_identical_items?"}
B -->|True| C["Collapse exact duplicates"]
B -->|False| D["Keep all items separate"]
C --> E["Apply similarity_threshold"]
D --> E
E --> F["Merge near-duplicate groups"]
F --> G["Apply uniqueness_threshold"]
G --> H["Keep only distinct-enough items individually"]
H --> I{"preserve_change_points?"}
I -->|True| J["Force-keep transition items"]
I -->|False| K["Transitions scored like anything else"]
J --> L["Final retained set: first_fraction + last_fraction + scored middle"]
K --> L
```
**Predicting the effect of a change:**
| Setting change | Effect on retained array |
|---|---|
| `dedup_identical_items: False` | Exact duplicates reappear individually → array grows |
| `similarity_threshold` ↓ | More items merge together → array shrinks, fewer distinctions |
| `uniqueness_threshold` ↑ | Fewer items pass the bar → array shrinks, more summarizing |
| `preserve_change_points: False` | Transitions may get scored/dropped like anything else → risk of losing the moment things changed |
**Where this leads →** 4.5 (Tuning SmartCrusher Safely) teaches you how to pick values for these together — building on the 4.2 retention strategy shape (`first_fraction` / `last_fraction` / `max_items_after_crush`) — without breaking downstream reasoning.
Back to course