Relevance Scoring for Retention

Learning Headroom

From Retention to Relevance: Which Items Survive?

**The gap retention strategy leaves open** Builds on ← 4.2 SmartCrusher Retention Strategy: dedup removes copies, first/last-fraction keeps structural anchors, change points keep turning points. None of that asks: is this item actually about what the agent is asking right now? **Where this leads →** 4.5 Tuning SmartCrusher Safely will have you dial `relevanceThreshold` and `hybridAlpha` in `RelevanceScorerConfig`. Today you learn what those numbers actually control. **The new ingredient: the query** - Dedup, anchors, change points score items using ONLY the item itself - Relevance scoring adds a second input: the current query or task context - Same conversation history, different query → different items get kept - This is called **query-conditioned scoring**: relevance is not a fixed property of an item, it's a relationship between item and query

BM25: Scoring by Word Overlap

**BM25 scoring concept** BM25 scores an item by how well its exact words match the query's exact words, weighted so rare words count more than common words, and long items don't win just by being long. **BM25 score for a query against one item** \[ \text{score}(q, d) = \sum_{t \in q} \text{IDF}(t) \cdot \frac{f(t,d)\,(k_1+1)}{f(t,d) + k_1\left(1 - b + b\frac{|d|}{\text{avgdl}}\right)} \] **Why it's shaped this way** - $\text{IDF}(t)$: rare terms (like "ECONNRESET") get a high weight, common terms (like "the") get near zero — matching a rare word is strong evidence, matching a common one isn't - $f(t,d)$ in the numerator: more occurrences of the term = higher score, but with diminishing returns (it saturates) - The $|d|/\text{avgdl}$ term: penalizes long items so they don't score high just by containing every word somewhere - $k_1$ controls how fast term-frequency saturates; $b$ controls how much length is penalized (0 = no penalty, 1 = full penalty)

BM25 Worked Example

**Query: "database timeout error"** - Corpus: 10 items total. "database" appears in 4 of them, "timeout" appears in 2, "error" appears in 8. - $\text{IDF}(t) \approx \log(N/df(t))$: $\text{IDF}(\text{database}) = \log(10/4) = \log(2.5) \approx 0.92$ - $\text{IDF}(\text{timeout}) = \log(10/2) = \log(5) \approx 1.61$ - $\text{IDF}(\text{error}) = \log(10/8) = \log(1.25) \approx 0.22$ - Item A: "Connection database timeout after retry" → contains database + timeout → score $= 0.92 + 1.61 = 2.53$ - Item B: "Unhandled error in render function" → contains error only → score $= 0.22$ - Result: Item A (2.53) ranks far above Item B (0.22) — it shares the RARE terms, not just the common one

Embedding Scoring: Meaning Over Wording

**Embedding scoring concept** An embedding model maps text into a vector in high-dimensional space. Relevance = how close the query's vector and the item's vector point in that space — measured by cosine similarity. **Cosine similarity between query vector q and item vector d** \[ \text{sim}(q, d) = \frac{q \cdot d}{\lVert q \rVert \, \lVert d \rVert} \] ```tikz \begin{tikzpicture}[scale=1.4] \draw[->] (-0.3,0) -- (3.5,0); \draw[->] (0,-0.3) -- (0,3.2); \draw[->, thick, blue] (0,0) -- (2.6,2.2) node[above right] {$q$: "db timeout"}; \draw[->, thick, darkgray] (0,0) -- (2.9,1.5) node[right] {$d_1$: "connection dropped"}; \draw[->, thick, red] (0,0) -- (0.4,2.8) node[above] {$d_2$: "recipe for soup"}; \draw[dashed] (0.9,0.55) arc (25:39:1.0); \node at (1.35,0.9) {\small $\theta_1$ small}; \end{tikzpicture} ```

Embedding Worked Example

**Toy vectors (2 dimensions, for hand-calculation)** - Query vector: $q = (3, 4)$ - Item vector: $d = (6, 8)$ — same direction as q, just scaled up - Dot product: $q \cdot d = (3)(6) + (4)(8) = 18 + 32 = 50$ - Norm of q: $\lVert q \rVert = \sqrt{3^2 + 4^2} = \sqrt{9+16} = \sqrt{25} = 5$ - Norm of d: $\lVert d \rVert = \sqrt{6^2 + 8^2} = \sqrt{36+64} = \sqrt{100} = 10$ - Cosine similarity $= \dfrac{50}{5 \times 10} = \dfrac{50}{50} = 1.0$ - Interpretation: similarity of 1.0 = same direction = same meaning, even though d is a longer/scaled vector

Hybrid Scoring: Blending Both Signals

**Hybrid scoring concept** \[ \text{score}_{\text{hybrid}} = \alpha \cdot \text{score}_{\text{bm25}} + (1-\alpha) \cdot \text{score}_{\text{embedding}} \] **Choosing a scorer for stated conditions** - Query has exact identifiers (error codes, function names, IDs) → BM25 wins: exact string match is what matters, embeddings can blur distinct codes together - Query is a paraphrase / natural-language question, wording varies across items → Embedding wins: catches meaning without needing shared vocabulary - Mixed traffic: some queries are exact lookups, others are natural language → Hybrid with moderate alpha (e.g. 0.5) or `adaptiveAlpha`, so neither signal is thrown away - Very small, cheap deployment, latency-sensitive → BM25 alone: no embedding model call needed, much cheaper and faster **Relevance ranking interpretation** Whichever scorer runs, every item ends up with ONE relevance number. Sort items by that number, keep items above `relevanceThreshold` (or the top N), drop the rest. A higher score always means "more likely to survive retention."

Putting It Together: Choosing the Right Scorer

**Scenario: debugging agent, tool-output history** - Data: 40 tool-call results (stack traces, log lines, file diffs) — many reuse exact error codes and function names - Query: "why did ECONNRESET happen in the payment service" - Choice: BM25 (or hybrid with high alpha) — ECONNRESET and payment service are exact, rare identifiers; embedding could conflate this with unrelated connection errors - Expected retention: items containing the literal string ECONNRESET or payment service rank highest and survive; generic log noise sits near the bottom and gets crushed **Scenario: research assistant, mixed documents** - Data: 40 knowledge-base snippets, varied phrasing, no shared vocabulary guaranteed - Query: "what slows down our checkout flow" (paraphrase of underlying issue, no exact keyword match expected) - Choice: Embedding — the concept "slows down checkout" may appear as "latency in payment step" or "timeout during purchase", zero word overlap but same meaning - Expected retention: semantically related snippets survive even with different wording; purely keyword-matching BM25 would have missed most of them **Where this leads →** 4.5 Tuning SmartCrusher Safely: you'll set `relevanceThreshold`, `bm25K1`/`bm25B`, and `hybridAlpha` for a real config, and reason about how each change shifts what survives retention.

Back to course