Joe Abou Nakoul Docs

hybrid-retrieval-and-reranking

Properties1
tagsrag, retrieval, bm25, reranking, evaluation

8 min read

Hybrid retrieval and reranking

Run a keyword search and a vector search over the same corpus, merge them by rank, then re-score the survivors with a model that reads the query and the document together.

The problem

Your RAG answer is wrong and you cannot tell whose fault it is — two different failures look identical from the outside. A user asks “containers keep getting killed when they use too much RAM” and keyword search returns nothing useful, because the document says OOMKilled and memory limit and shares not one token with the question. Another user pastes ERR_CONN_4471 and vector search returns three plausible pages about connection pools, none of them the page that defines that code — the embedder split the string into subword fragments and buried it among generic pool-ish text. One system fails on paraphrase, the other on literals. A bigger generator fixes neither.

How it works

Two retrievers, two properties, and they are complementary.

BM25 scores exact term overlap, weighting each matched term by inverse document frequency and saturating repeated occurrences (k1=1.2) and long documents (b=0.75). A rare token like an error code gets an enormous IDF, so one occurrence dominates the ranking. A paraphrase scores zero — not “low”, zero.

A bi-encoder embeds query and document independently and ranks by cosine similarity. It generalises across wording — and that same smoothing is what loses the identifier.

You cannot just add the two scores: BM25 is unbounded and shifts scale with query length and corpus statistics, cosine sits in [-1, 1], and any normalisation you pick is tuned to today’s corpus. Reciprocal Rank Fusion throws the scores away and fuses ranks:

RRFscore(d) = Σ over systems  1 / (k + rank(d))     k = 60

The k is the trick. At k=60, rank 1 contributes 1/61 and rank 5 contributes 1/65 — nearly the same. That deliberately flattens the head of each list, so a single retriever that is confidently wrong cannot own the fused top. Cormack et al. reported RRF beating both Condorcet Fuse and CombMNZ on TREC collections (MAP .3686 vs .3586 for the best individual system on Robust), untrained and untuned.

Fusion buys recall — the right document is now somewhere in the top 50 — but orders it badly, because neither input ranking ever compared query terms against document terms directly. A cross-encoder does exactly that: it concatenates [query, document] into one sequence and runs full attention across the pair, so “RAM” in the query can attend to “memory limit” in the document. The output is a single relevance score. Nothing is precomputable — one forward pass per candidate — which is why it runs on the top-k and never on the corpus. ms-marco-MiniLM-L6-v2 scores 74.30 NDCG@10 on TREC DL 19 at roughly 1800 docs/sec.

So: retrievers for recall, fusion to combine their different blind spots, cross-encoder for precision at the top.

Code

First run downloads ~130 MB of weights. CPU is fine; no GPU.

# pip install sentence-transformers   -- CPU only, no GPU. First run pulls ~130 MB of weights.
import math, re
from collections import Counter
from sentence_transformers import SentenceTransformer, CrossEncoder
 
DOCS = [
 "Pods are terminated with OOMKilled when the container exceeds its memory limit; raise limits.memory.",
 "Horizontal pod autoscaling scales replicas on CPU utilisation, not on resident set size.",
 "Error ERR_CONN_4471 is returned when the connection pool is exhausted; raise pool_max_size.",
 "The connection pool defaults to 16 sockets per worker and blocks callers once every socket is out.",
 "Worker processes are recycled after 500 requests to bound heap growth in long-lived jobs.",
 "Retries use exponential backoff with jitter; the default ceiling is 30 seconds.",
 "A memory leak in the PDF parser was fixed in 3.2.1; earlier versions grow unbounded on uploads.",
 "The ingest queue drops messages older than 24 hours to keep the backlog bounded.",
 "Set GOMEMLIMIT so the garbage collector runs before the kernel's OOM killer intervenes.",
 "Blue/green deploys keep the previous ReplicaSet warm for a five minute rollback window.",
]
STOP = set("a an the is are was were be to of in on for and or it its with when as by from not".split())
tok = lambda s: [t for t in re.findall(r"[a-z0-9_]+", s.lower()) if t not in STOP]
 
def bm25(query, docs, k1=1.2, b=0.75):                  # the lexical half, in full
    D = [tok(d) for d in docs]; N = len(D); avgdl = sum(map(len, D)) / N
    df = Counter(t for d in D for t in set(d)); scores = []
    for d in D:
        tf, dl, s = Counter(d), len(d), 0.0
        for t in tok(query):
            if t not in tf: continue
            idf = math.log((N - df[t] + 0.5) / (df[t] + 0.5) + 1)   # rare literal token -> huge idf
            s += idf * tf[t] * (k1 + 1) / (tf[t] + k1 * (1 - b + b * dl / avgdl))
        scores.append(s)
    return scores
 
def rrf(*rankings, k=60):                               # the fusion, in full: ranks, never scores
    fused = Counter()
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            fused[doc_id] += 1 / (k + rank)
    return [doc_id for doc_id, _ in fused.most_common()]
 
bi = SentenceTransformer("all-MiniLM-L6-v2")
ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
E = bi.encode(DOCS, normalize_embeddings=True)
 
for q in ["containers keep getting killed when they use too much RAM", "ERR_CONN_4471"]:
    s = bm25(q, DOCS)
    lex = sorted(range(len(DOCS)), key=lambda i: -s[i])[:5]
    vec = (E @ bi.encode(q, normalize_embeddings=True)).argsort()[::-1][:5].tolist()
    cand = rrf(lex, vec)[:5]                            # recall: union of two different failure modes
    top = [cand[i] for i in (-ce.predict([(q, DOCS[c]) for c in cand])).argsort()]  # precision
    print(f"\nq: {q}\n  bm25   {lex}\n  vector {vec}\n  rrf    {cand}\n  rerank {top}\n  -> {DOCS[top[0]]}")

The pure-Python half is worth running on its own first. For the RAM query, BM25 scores docs 0 and 8 — the two correct ones — at exactly 0.0, and puts retry backoff at rank 1. For ERR_CONN_4471 it is the only retriever that can possibly be right.

Where it fits

This lives in the retrieval layer of any RAG service over a corpus that mixes prose with identifiers: runbooks, ticket histories, API docs, contracts with clause numbers, catalogues with SKUs. Postgres gives you both halves in one place (tsvector + pgvector); Elasticsearch, Qdrant, Weaviate and Vespa expose hybrid queries natively, most with RRF built in — so the code above is what to understand, not necessarily what to ship.

The real payoff is diagnostic. Instrument the rank of the known-correct chunk. Not in the top 100 → retrieval problem, fix chunking or add the missing retriever. In the top 100 but not the top 5 → ranking problem, add or upgrade the reranker. In the top 5 and the answer is still wrong → generation problem, and no amount of retrieval work will help.

When not to

Reranking 50 chunks costs tens of milliseconds on GPU but hundreds on a busy CPU box, all of it on the critical path before the first token. Under a few thousand chunks, one retriever plus a larger top-k fed to a long-context model usually wins — cheaper, simpler, fewer knobs. And on a corpus of pure natural language with no identifiers, BM25 may add nothing but a second index to keep in sync.

Check yourself

1. Your reranker adds 300 ms and lifts recall@5 by 2 points. What do you measure before keeping it?

Answer Not recall@5 — end-to-end answer correctness on a fixed eval set, split by whether the reranker changed the top-5 at all. Two points of recall matter only if those queries were failing before. And check p95, not the mean: reranking cost scales with candidate count, and the tail is where the budget breaks.

2. Why does RRF use a constant k=60 rather than fusing normalised scores?

Answer Because the scores are not comparable and their distributions move per query and per corpus, so any normalisation is a hidden tuning parameter. Ranks are scale-free. `k=60` additionally flattens the head of each list so a single confident outlier retriever cannot dominate the fused ordering.

3. Your fused top-50 always contains the right chunk, but the reranker puts it at rank 12. Where is the bug most likely to be?

Answer In the chunk, not the model. Cross-encoders are trained on passage-length text; a chunk that buries the relevant sentence in 2000 tokens of surrounding context dilutes the pair score. Shrink the chunk, or rerank on a smaller window and return the larger parent.

Go deeper

Graph View