Reranker (Cross-Encoder)

A reranker is a cross-encoder model that scores query/document pairs jointly (concatenated as input), producing a relevance score per pair. Used as a second pass after retrieval: retrieve top-100 by cheap method, rerank to top-10 with the cross-encoder.

Cross-encoders dramatically outperform bi-encoder dense retrieval on relevance — typically +5 to +15 NDCG@10 — because they let the query and document attend to each other. The cost is compute: scoring 100 pairs is 100× slower than the original retrieval.

Common open rerankers: BGE-reranker-v2-m3, Cohere Rerank (API), Jina-Reranker-v2. For local RAG, the m3 family is the standard pick.

After hybrid retrieval returns its top 50 candidate chunks for a RAG query, an operator runs BGE-reranker-v2-m3 as a second pass to cut that down to the top 5 actually sent to the LLM's context window. The reranker correctly demotes a chunk that merely shares keywords with the query but discusses an unrelated topic, and promotes a chunk that phrases the answer differently but is clearly the right match — something BM25 and even dense retrieval alone both got wrong in the initial pass. Because cross-encoders score each query-document pair jointly, 50 pairs means 50 forward passes; on a local GPU with a small reranker model this typically adds well under 100ms, which is cheap insurance against feeding the LLM irrelevant context and triggering a hallucinated answer.