AI Search
Failure Mode Analysis

1. Benchmark Overview — what we measured
2. Scorecard — where we're good vs. bad
3. Six Failure Modes — taxonomy of why retrieval fails

4. Evidence from Benchmarks — mapping data to failure modes
5. Fix / Mitigate / Redirect — product strategy per failure
6. What We Could Do — options on the table

Problem: A chunk has 10 sentences. The answer is in one sentence. The embedding represents the average — the fact contributes ~10%.

Result: A chunk that's entirely about the topic ranks higher than the one with the answer.

Worse with: Larger chunks, dense config refs, mixed-content pages

Problem: Embedding models are trained on similarity, not Q&A alignment. "Discusses OAuth" and "States Service X uses OAuth" look identical.

Result: Top-K is full of explanatory content, the one-liner answer is pushed out.

Worse with: Large corpora with deep topic coverage, well-written docs

Problem: Facts live in separate chunks with no relationships. Multi-hop questions fail.

Math: If single-hop recall = 80%, then 2-hop = 64%, 3-hop = 51%

Especially hard for structured data: Join keys (owner_team → team_id → rotation) are opaque identifiers with no semantic meaning

Example: "Who is on-call for the payments service?"

services.json → svc-payments → team-billing
teams.json → team-billing → billing-primary
oncall.csv → billing-primary → carol@example.com

3 hops across 3 files. Search finds "payments" but never reaches the on-call CSV.

Problem: 10-token query vs. 300-token passage. Config blocks, JSON, CSV embed poorly compared to prose.

The format gap:

Our benchmark evidence:

• code-eval: 0.33 R@10
• json-eval: 0.49 R@10
• FinQA (tabular): 0.86 R@10

Structured content consistently retrieves worse than prose.

Prose negation Partial

• Negation in text: "regions that do NOT support X" — modern embeddings have weak but real negation sensitivity
• Negation-aware training helps (Jina, DEO) but signal is still overwhelmed when many topically similar chunks compete
• Absence: "What features are missing?" — weak signal for things that aren't there

Structured ops Impossible

• Comparison: "Products under 50 euros" — 29.99 and 189.00 embed identically
• Filtering: category = 'electronics' AND price < 50
• Aggregation: "How many active users on enterprise?"
• Sorting: "Three most recent deployments"

Helps +28pp

• CourtListener (Qwen3): +28pp (citations need exact match)
• emanual (Gemma): +13pp
• NewsQA (Gemini2): +8pp

Qwen3 and Gemini2 hybrid is safe almost everywhere. Most gains come from fuzzy match (next slide).

BGE-M3 catastrophic -26pp

• MS MARCO (BGE-M3): -26pp
• PubMedQA (Gemma): -19pp
• Qwen3: worst -4pp (emanual)
• Gemini2: worst -0.5pp (safe everywhere)

Pattern: weaker embeddings can't absorb BM25 noise. Model-dependent, not dataset-dependent.

Wins (Qwen3, best pipeline)

• TechQA: +10.7pp
• FinQA: +5.4pp
• QASPER: +5.0pp
• emanual: +4.5pp
• beir-scifact: +2.4pp

Pattern: diverse vocabulary, technical terms, academic content

Losses (Qwen3, best pipeline)

• PubMedQA: -32.6pp
• Legal CR: -12.3pp
• CourtListener: -10.6pp
• MS MARCO: -3.0pp (was -49pp before pipeline fix)

Pattern: citation-heavy, short lexically similar docs where fuzzy BM25 overwhelms the vector signal

Where we've been investing

• Chunking pipeline v2 — table parsing, improved boundary detection, overlap fixes. Drove the v6→v8 lift (+20pp HotpotQA)
• Reranking — BGE-reranker-base shipped. Addresses FM2 (aboutness)
• Hybrid search tuning — rank-based scoring, chunk correctness validation
• Benchmark infrastructure — 18 datasets, 4 models, automated eval pipeline

Where we're headed

• Header injection — simplified contextual retrieval (FM1, FM5)
• Content categories — detect content type, adapt chunking per file (FM1, FM5)
• Code processing — format-aware chunking for source code (FM5)
• Chunk diversity — exploring similar chunk detection (FM3)