The previous article ended on a warning: an embedding keeps the gist of a sentence and blurs the details. This article is about what that does to a real RAG system, and the three standard fixes. Each fix is introduced only after we've watched retrieval fail without it.
Some words first:
- Retrieval: given a question (the query), return the few pieces of text most likely to contain the answer.
- Corpus: all the text you can search. Document: one item in it.
- Dense search: search with embeddings (dense because every one of the 384 numbers is used). Sparse / keyword search: search by matching words.
- Top-k: the k best results. The LLM only ever sees these.
WHY THIS MATTERS
An LLM can only answer from what retrieval hands it. If the right passage isn't in the top-k, no prompt engineering will save the answer, and the model will usually still answer, confidently and wrongly. Most "the bot hallucinated" bugs in RAG are retrieval bugs.
Part I · Before search: chunking
Chapter 1: Chunking decides what can be found
The problem
Policies and manuals are long. You can't embed a 40-page PDF as one vector (it would blur into "a document about orders"), and you can't paste all of it into every prompt. So you split documents into chunks, embed each chunk, and retrieve chunks.
MENTAL MODEL
Cutting a book into index cards. Retrieval can only return whole cards. If a card is too big, its vector is a blur of many topics. If a card is too small, or the cut falls mid-thought, the answer is split across two cards and neither one makes sense alone. Overlap (repeating the last few words of one card at the start of the next) is tape across the cut.
Details, built on the model
policy = ("Refunds. Damaged items are refunded within 7 days. "
"Late deliveries get a voucher. "
"Exceptions. Perishable goods cannot be returned, "
"but damaged perishables are refunded in full.")
def chunk(text, size, overlap=0):
words = text.split()
step = size - overlap
return [" ".join(words[i:i + size])
for i in range(0, len(words), step)
if words[i:i + size]]
for c in chunk(policy, 8):
print("|", c)
print()
for c in chunk(policy, 8, overlap=3):
print("|", c)
| Refunds. Damaged items are refunded within 7 days. | Late deliveries get a voucher. Exceptions. Perishable goods | cannot be returned, but damaged perishables are refunded | in full. | Refunds. Damaged items are refunded within 7 days. | within 7 days. Late deliveries get a voucher. | get a voucher. Exceptions. Perishable goods cannot be | goods cannot be returned, but damaged perishables are | damaged perishables are refunded in full. | full.
Look at where the cuts fall without overlap. "Perishable goods" ends one card and "cannot be returned" starts the next, so neither card states the rule on its own: the first just mentions perishable goods, the second says something "cannot be returned" without saying what. With an overlap of 3 words, the card "goods cannot be returned, but damaged perishables are" keeps the rule and its exception together. Overlap doesn't guarantee a perfect card; it gives every cut a second chance.
In practice you split on structure first (headings, paragraphs), then fall back to a size limit of a few hundred tokens with 10–20% overlap, and store the heading with each chunk so a card knows which section it came from.
MODEL RECAP
Recap: retrieval returns whole index cards, so where you cut decides what can ever be found.
Part II · Two kinds of search
Chapter 2: Dense search misses exact terms
The problem
Let's set up a small corpus and watch embeddings fail.
MENTAL MODEL
Dense search reads for the topic; it doesn't read the serial number. As we saw, the vector for "Order ORD-48213 is late" sits right next to the one for "Order ORD-99107 is late". When a query hinges on an exact token, such as an ID, an error code or a product SKU, dense search returns the right kind of document, not the right document.
Details, built on the model
import numpy as np
from sentence_transformers import SentenceTransformer
docs = [
"ORD-48213: courier marked delivered, customer says not received",
"ORD-99107: delivery delayed by weather, new ETA Friday",
"ORD-30551: refund approved for damaged blender",
"Policy: damaged items are refunded within 7 days",
"Policy: missing parcels are investigated with the courier",
"Error E-1043 means the payment gateway timed out",
"ORD-99170: courier marked delivered, customer confirmed",
"Error E-1034 means the card was declined by the bank",
]
emb = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
D = emb.encode(docs, normalize_embeddings=True)
def dense(q, k=3):
s = D @ emb.encode([q], normalize_embeddings=True)[0]
return [(int(i), round(float(s[i]), 2))
for i in np.argsort(-s)[:k]]
for q in ["was ORD-99107 delivered?", "E-1043"]:
print(q, "->", dense(q))
was ORD-99107 delivered? -> [(6, 0.76), (1, 0.66), (0, 0.63)] E-1043 -> [(5, 0.45), (7, 0.41), (6, 0.32)]
Each result is (document number, cosine score). For the first query, dense search puts doc 6, ORD-99170, first: a different order whose ID shares most of its characters and whose text is about delivery. The right order, doc 1, is second. For the error code, the right doc 5 wins, but only by 0.45 to 0.41 over E-1034, the wrong error. The vectors saw "an order ID about delivery" and "an error code"; they didn't read the digits. We need a search that treats ORD-99107 as a word that must match exactly.
MODEL RECAP
Recap: dense search finds the right kind of document, not the one with the exact serial number.
Chapter 3: BM25, built by hand
The problem
BM25 ("Best Matching 25") is the classic keyword ranking formula behind Elasticsearch and Lucene. It's decades old and still hard to beat on exact terms.
MENTAL MODEL
Rare words shout, repeated words get hoarse, long documents get discounted. A document scores points for each query word it contains. A rare word (like ORD-99107, in one document) is worth far more than a common one (like "the"); this weight is IDF, inverse document frequency. Saying a word again earns less each time (saturation, controlled by k1). Long documents are discounted a little, so they don't win just by containing more words (controlled by b).
Details, built on the model
The per-word score is IDF × tf·(k1+1) / (tf + k1·(1 − b + b·len/avg_len)), where tf is how many times the word appears in this document. Each piece is one part of the model:
import math, re
from collections import Counter
def tokens(text):
return re.findall(r"[a-z0-9]+(?:-[a-z0-9]+)*", text.lower())
class BM25:
def __init__(self, docs, k1=1.5, b=0.75):
self.toks = [tokens(d) for d in docs]
self.k1, self.b = k1, b
self.avg = sum(map(len, self.toks)) / len(self.toks)
df = Counter(w for t in self.toks for w in set(t))
n = len(docs)
self.idf = {w: math.log((n - f + 0.5) / (f + 0.5) + 1)
for w, f in df.items()} # rare -> big
def score(self, q):
out = []
for t in self.toks:
tf, s = Counter(t), 0.0
for w in tokens(q):
if w not in tf:
continue
norm = 1 - self.b + self.b * len(t) / self.avg
s += self.idf[w] * tf[w] * (self.k1 + 1) / (
tf[w] + self.k1 * norm)
out.append(s)
return np.array(out)
bm = BM25(docs)
print("idf ord-99107:", round(bm.idf["ord-99107"], 2),
"| idf damaged:", round(bm.idf["damaged"], 2))
def sparse(q, k=3):
s = bm.score(q)
return [(int(i), round(float(s[i]), 2))
for i in np.argsort(-s)[:k]]
for q in ["was ORD-99107 delivered?", "E-1043"]:
print(q, "->", sparse(q))
idf ord-99107: 1.79 | idf damaged: 1.28 was ORD-99107 delivered? -> [(1, 1.77), (7, 1.58), (6, 1.43)] E-1043 -> [(5, 1.77), (0, 0.0), (2, 0.0)]
Each result is (document number, BM25 score). The ID appears in only one document, so it has the highest IDF in the query and doc 1 now ranks first. Docs 7 and 6 still score something because they share the ordinary words "was" and "delivered", which is why the margin is modest here; in a real corpus of thousands of documents, common words get much smaller IDF and the exact ID dominates more. The tokenizer keeps ord-99107 as one token on purpose; split it on the hyphen and "ord" would match every order.
The model also predicts BM25's weakness: it has no idea that "parcel" and "delivery" are related.
print("my parcel never came ->", sparse("my parcel never came"))
print("my parcel never came ->", dense("my parcel never came"))
my parcel never came -> [(0, 0.0), (1, 0.0), (2, 0.0)] my parcel never came -> [(4, 0.58), (0, 0.52), (6, 0.45)]
BM25 scores every document 0: no document contains "parcel", "never" or "came" (doc 4 says "parcels", a different token). Dense search finds the missing-parcel policy without effort.
MODEL RECAP
Recap: BM25 scores shared words, rare words most, with diminishing returns and a length discount.
Chapter 4: Hybrid search with reciprocal rank fusion
The problem
Dense is good at meaning, BM25 at exact words. Run both and combine? Their scores live on different scales (cosine near 0.5, BM25 near 3), so adding them is meaningless.
MENTAL MODEL
Merge the two ranked lists by position, not by score. Reciprocal rank fusion (RRF) ignores the scores and gives each document 1 / (60 + rank) from each list it appears in, then adds those up. Being near the top of either list helps; being near the top of both helps most. The 60 is a damping constant that keeps the first place from dominating.
Details, built on the model
def rrf(rankings, k=60):
fused = Counter()
for ranking in rankings:
for rank, doc in enumerate(ranking, start=1):
fused[doc] += 1 / (k + rank)
return [d for d, _ in fused.most_common()]
def hybrid(q, n=3):
by_bm25 = list(np.argsort(-bm.score(q)))
qv = emb.encode([q], normalize_embeddings=True)[0]
by_dense = list(np.argsort(-(D @ qv)))
return [int(i) for i in rrf([by_bm25, by_dense])[:n]]
for q in ["was ORD-99107 delivered?", "my parcel never came"]:
print(q, "->", hybrid(q))
was ORD-99107 delivered? -> [1, 6, 0] my parcel never came -> [0, 4, 1]
Both queries now get a sensible top result: the ID query is anchored by BM25's exact match, and the paraphrase query is carried by dense search, which BM25 couldn't help at all. Neither list alone was right for both.
MODEL RECAP
Recap: RRF merges ranked lists by position, so each search contributes where it is strong.
Part III · Reading carefully
Chapter 5: Rerank the shortlist
The problem
Both searches compare a query vector or word list with each document independently. Neither actually reads the query and the document together, which is why negation and subtle relevance slip through.
MENTAL MODEL
A fast librarian, then a careful one. Retrieval is the fast librarian: scans everything, grabs maybe 20 plausible cards in milliseconds. A reranker (a cross-encoder: a model that takes the query and one document together as input and outputs a relevance score) is the careful librarian who reads each of those 20 against the question. Too slow for the whole corpus, perfect for a shortlist.
Details, built on the model
from sentence_transformers import CrossEncoder
rerank = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
q = "was ORD-99107 delivered?"
shortlist = [i for i, _ in dense(q, k=4)] # dense alone
scores = rerank.predict([(q, docs[i]) for i in shortlist])
order = [shortlist[j] for j in np.argsort(-scores)]
print("dense order: ", shortlist)
print("reranked order:", order)
print("top:", docs[order[0]])
dense order: [6, 1, 0, 2] reranked order: [1, 6, 0, 2] top: ORD-99107: delivery delayed by weather, new ETA Friday
Here we deliberately reranked the dense-only shortlist, which had the wrong order on top. The cross-encoder read the query and each document together, noticed the exact ID, and moved doc 1 to first place.
The cost model is simple: a cross-encoder runs one model call per (query, document) pair, so reranking 20 candidates costs 20 calls, while reranking a million documents is out of the question. That's why it's always the last stage: chunk → hybrid retrieve ~20 → rerank → give the top 3–5 to the LLM.
WHY THIS MATTERS
When you evaluate a RAG system, measure this pipeline stage by stage: is the answer chunk in the hybrid top-20 (recall), and is it in the reranked top-3 (precision)? The fix depends on which stage lost it. The next article builds the harness for exactly that kind of measurement.
MODEL RECAP
Recap: retrieve fast and wide, then let a reranker read query and document together.
All the mental models, in one place
MODEL RECAP
- Index cards. Retrieval returns whole chunks; size and overlap decide whether an answer survives the cut.
- Dense reads the topic, not the serial number. Embeddings find the right kind of document, not exact IDs.
- Rare words shout. BM25 rewards shared words weighted by rarity, with saturation and a length discount; it misses synonyms.
- Merge by position. RRF adds
1/(60+rank)from each list, combining keyword and dense without comparing scores. - A fast librarian, then a careful one. A cross-encoder reranks a short list by reading query and document together.