Skip to content

Hybrid Retrieval and Reranking for RAG: BM25, Vectors, RRF and Cross-Encoders in Production

I run the retrieval tier of an enterprise RAG system that serves internal docs, legal references, and a parts catalog for our engineering teams. For a long time my pipeline was embarrassingly simple: embed everything, do a cosine similarity search, take the top 5, stuff them into the prompt. It worked well enough for "what is our policy on remote access" style questions. Then a support engineer asked the system for the replacement interval of a bearing and got a hallucinated answer about a completely different component. That is the story of how I stopped trusting vectors alone.

The query was short and exact: SKF-6204-2RSH. Our catalog had a spec sheet for that exact bearing. The vector search returned documents about "deep groove ball bearings" and "greased seals" — semantically adjacent, literally useless. The answer cited the wrong part number, and an engineer ordered the wrong inventory. The root cause is a fundamental property of dense embeddings: they compress meaning into a vector and, in doing so, flatten exact tokens. "SKF-6204-2RSH" as a string is a precise identifier, but as a vector it is just another point near other bearing-related text. Cosine similarity has no notion of "this token must appear verbatim."

The same blind spot hits error codes (E_CANNOT_FIND_DEVICE), legal citations (F.R.C.P. 12(b)(6)), and model numbers. In every case the user knows the exact token, and the vector search politely returns the neighborhood instead of the address.

The three-stage pipeline I run now

I rebuilt retrieval as three explicit stages, each with a measurable cost and benefit.

Stage 1 — Candidate generation. Two independent retrievers each pull a candidate list from the same corpus. The BM25 (lexical) retriever handles exact tokens: it scores how many query terms appear in a document and how rare those terms are. The dense retriever handles paraphrase and semantics. I run both in parallel so the wall-clock cost is roughly the slower of the two, not the sum. I keep topN around 50 per retriever — large enough to give the next stage room, small enough to stay fast.

Stage 2 — Fusion (RRF). The two lists are merged. I do not add scores. I use Reciprocal Rank Fusion.

Stage 3 — Reranking. The fused top 30 go through a cross-encoder, which scores each query-document pair jointly. The cross-encoder is slow but accurate, and it sees the actual question alongside each candidate rather than comparing two independent vectors.

def hybrid_search(query: str, k: int = 5) -> list[Document]:
    bm25_hits = bm25_retriever.search(query, top_n=50)
    dense_hits = dense_retriever.search(query, top_n=50)
    fused = reciprocal_rank_fusion([bm25_hits, dense_hits], k=60)
    reranked = cross_encoder.rerank(query, fused[:30])  # only top 30
    return reranked[:k]

Why RRF instead of adding the scores

The first version I hacked together literally summed the two scores. It failed, and the reason is not a tuning problem, it is a scale problem. BM25 is an unbounded term-frequency score; for a long document with repeated query terms it can reach into the hundreds. Cosine similarity is bounded in [0, 1]. If I add them, the dense score is a rounding error next to BM25, and the dense retriever effectively stops mattering. If I normalize them, I have to pick a normalization strategy that is itself a tuning knob, and any choice I make on my corpus breaks on the next one.

RRF avoids the whole problem because it uses only rank position:

RRF(d) = Σ_r 1 / (k + rank_r(d))

where rank_r(d) is the rank of document d in retriever r's list, and k is a smoothing constant. The default k=60 comes from the original paper and is what I use. The value dampens the influence of deep ranks: moving from rank 1 to 2 changes the score by roughly 1/(k+1) - 1/(k+2), which is small but meaningful, while a document ranked 50th contributes almost nothing. A document that ranks well in both retrievers wins; a document that ranks first in one but nowhere in the other survives but does not dominate.

def reciprocal_rank_fusion(retriever_lists: list[list[Document]], k: int = 60) -> list[Document]:
    scores: dict[str, float] = {}
    for hits in retriever_lists:
        for rank, doc in enumerate(hits, start=1):
            scores[doc.id] = scores.get(doc.id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

The beauty of RRF is that it is score-free. I can swap embedding models, change BM25 weighting, or add a third retriever, and the fusion logic never has to change.

What the numbers said

I measured on a golden set of 100 real queries with labeled answers, before and after each change:

Pipeline recall@5 MRR p95 latency
Dense only 0.61 0.58 40 ms
Hybrid (BM25 + dense + RRF) 0.89 0.71 120 ms
Hybrid + cross-encoder rerank 0.92 0.84 310 ms

The recall jump from 0.61 to 0.89 is the BM25 contribution: it caught the exact-token queries that dense search was missing. The reranker cost 190 ms extra but bought a big precision gain — the answer at rank 1 was now the right one far more often. I keep the rerank pool at 30, not 50; going bigger barely moved MRR and added latency. That 310 ms is why I only rerank when the pipeline is in the interactive path, and I cap concurrent rerank requests.

Local deployment on our GPU box

We run everything in-house on a single A100, and the cost math was not even close. Cloud embedding and rerank APIs priced out at thousands per month at our volume. Local costs one GPU we already owned and a power bill.

For embeddings and BM25 I use BGE-M3, which conveniently outputs dense vectors, sparse (lexical) weights, and a multi-vector representation from one model — so the dense and lexical retrievers can share the same encoder. For reranking I use BGE-reranker-v2-m3. Both run through vLLM for the encoder side and Ollama for quick experiments.

# Serve the embedding/reranker models via vLLM
vllm serve BAAI/bge-m3 --dtype float16 --max-model-len 2048 --port 8001
vllm serve BAAI/bge-reranker-v2-m3 --task rerank --dtype float16 --port 8002

from elasticsearch import Elasticsearch
from sentence_transformers import CrossEncoder

es = Elasticsearch("http://localhost:9200")
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")

def dense_search(query: str, top_n: int = 50):
    vec = embed(query)  # BGE-M3 dense vector
    return es.search(index="docs", knn={"field": "vector", "query_vector": vec, "k": top_n})

def bm25_search(query: str, top_n: int = 50):
    return es.search(index="docs", query={"match": {"content": query}}, size=top_n)

BM25 lives in Elasticsearch's match query, which is battle-tested and needs no custom code. The dense path uses the knn query with a vector we get from the same BGE-M3 model. One model, two retrieval modes, zero API bills.

Five incidents from my runbook

Case 1 — Part number not retrievable (the vector blind spot). This is the opening incident. Fixed by adding BM25 as a co-retriever. Pure vector still would have failed.

Case 2 — RRF k mis-tuned, fusion goes silent. I once dropped k to 5 while experimenting. The fusion became noisy: a document ranked 2nd in both retrievers now beat the consensus rank-1, and precision fell. Reverting to k=60 restored order. Lesson: tune k on the golden set, not by feel.

Case 3 — Candidate pool too small for the reranker. I fed only the top 8 to the cross-encoder to save latency. MRR dropped because the correct document was at rank 12 and never reached the reranker. The reranker can only promote what it is shown. Enlarging the pool to 30 recovered the gain.

Case 4 — Bad chunking breaks context. A multi-part failure investigation was split so that the "root cause" paragraph and the "recommended action" paragraph landed in different chunks. No retriever could reassemble them. Chunking is a retrieval decision, not a formatting decision. I moved to separator-aware chunking with token overlap.

Case 5 — Cost blowup from reranking everything. A teammate wired the reranker onto the full 50-document pool for every query, and GPU saturation pushed p95 to over a second at peak. Cap the rerank pool, and cache rerank results for identical query pairs.

The tuning order that saved me from thrashing

The order matters because each stage's output feeds the next. If I tune the reranker before the chunker, I am tuning noise.

  1. Build a golden set. ~100 real queries with correct document IDs. Without it I am guessing.
  2. Sweep chunking. Change chunk size and overlap, measure recall@5. This dominates everything downstream.
  3. Tune topN per retriever. Start at 50; the goal is to not lose the right answer before fusion.
  4. Tune RRF k. Sweep 30/60/90/120 on the golden set.
  5. Tune the rerank pool. 20 vs 30 vs 50. Watch MRR against latency.
  6. Add guardrails. Min-score thresholds, a fallback to pure BM25 for exact-token queries, and a hard latency budget with a fast-path that skips reranking.
def guarded_search(query: str) -> list[Document]:
    if is_exact_token_query(query):        # regex for part numbers, codes, citations
        return bm25_search(query, top_n=5) # skip dense entirely
    return hybrid_search(query)

FAQ

Why can't I just add BM25 and cosine similarity scores directly? The two scales are incomparable: BM25 is an unbounded term-frequency score while cosine similarity is bounded in [0,1]. Summing them means one term silently dominates or vanishes. RRF sidesteps this by using only rank position.

What should the RRF k parameter be and why 60? k dampens the influence of low-ranked candidates. k=60 is the commonly cited default and is stable for candidate pools around 50-100. If k is too small, the fusion becomes noisy; too large and ranking differences shrink to nothing.

When is a cross-encoder reranker actually worth the latency? When precision matters more than speed and your candidates are a mixed bag. In our data it lifted MRR from 0.71 to 0.84 but p95 latency jumped from ~120ms to ~310ms. Rerank only the top 20-30, not the full pool.

How do I tune a hybrid pipeline without burning GPU hours? Build a golden set of ~100 real queries first, sweep chunking, then topN per retriever, then RRF k, then the rerank pool. Change one variable at a time and keep the best on the golden set before moving on.

Why did my part number query still fail after I enabled hybrid search? Probably chunking, not retrieval. If a part number spans a chunk boundary, no retriever can see it. Check your chunker for separator-aware boundaries and token-level overlap before blaming BM25.