BM25 From Scratch, and Why It Still Beats Embeddings Sometimes

BM25 treats a query and a document as bags of words: order is ignored, only which terms occur and how often. A document's score is a sum over the query's terms. Each term found in the document contributes three things multiplied together:

• IDF: how rare the term is across the whole collection. A term in every document tells you nothing; a term in one document tells you a lot.
• Term frequency, saturated: how often the term appears in this document, with diminishing returns controlled by k1.
• Length normalization: how long this document is compared with the average, controlled by b.

Written out, with f the term's count in the document, dl the document length and avgdl the average length:

score = Σ IDF(term) × f × (k1 + 1) / (f + k1 × (1 − b + b × dl / avgdl))

A query term that does not occur in the document contributes zero, so a document with none of the query's terms scores zero and is never returned. That is BM25's precision and its main weakness at once.

This is the implementation used for every number in this guide:

1import math, re2from collections import Counter3 4def tokenize(text):5    return re.findall(r"\w+", text.lower())6 7class BM25:8    def __init__(self, docs, k1=1.2, b=0.75):9        self.k1, self.b = k1, b10        self.docs = [Counter(tokenize(d)) for d in docs]11        self.lengths = [sum(tf.values()) for tf in self.docs]12        self.avgdl = sum(self.lengths) / len(docs)13        self.df = Counter(term for tf in self.docs for term in tf)14        self.n = len(docs)15 16    def idf(self, term):17        df = self.df.get(term, 0)18        return math.log(1 + (self.n - df + 0.5) / (df + 0.5))19 20    def score(self, query, i):21        tf, dl, s = self.docs[i], self.lengths[i], 0.022        for term in tokenize(query):23            f = tf.get(term, 0)24            if f:25                norm = self.k1 * (1 - self.b + self.b * dl / self.avgdl)26                s += self.idf(term) * f * (self.k1 + 1) / (f + norm)27        return s28 29    def search(self, query, k=3):30        scores = [(self.score(query, i), i) for i in range(self.n)]31        return [i for s, i in sorted(scores, reverse=True)[:k] if s > 0]

__init__ computes everything that depends on the collection: each document's term counts and length, the average length, and each term's document frequency (how many documents contain it). score applies the formula for one document, and search scores all of them and returns the top k with a positive score.

Use it like this:

1docs = ["Error E1042: the payment gateway timed out ...", "Order SKU-88231 ships in a red box ...", ...]2bm = BM25(docs)3print(bm.search("SKU-88231"))   # indexes of the best documents

The tokenizer is deliberately minimal: lowercase, then split on anything that is not a letter, digit or underscore. SKU-88231 becomes sku and 88231. Real search engines use analyzers that also stem words (refunds to refund), remove stop words and handle accents, and those choices often matter more than the scoring function.

Robertson and Zaragoza derive BM25's IDF from the Robertson/Spärck Jones relevance weight with no relevance information, which gives:

IDF = log((N − n + 0.5) / (n + 0.5))

where N is the number of documents and n is the number containing the term. In the ten-document test corpus used later, e1042 appears in one document and the appears in seven:

• e1042: log(9.5 / 1.5) = 1.846
• the: log(3.5 / 7.5) = −0.762

The second value is negative. Any term in more than half the documents gets a negative weight, so matching it lowers a document's score. That is defensible in theory, but surprising in practice. Lucene adds 1 inside the logarithm, log(1 + (N − n + 0.5) / (n + 0.5)), which keeps every weight positive. With that version, used in the code above, e1042 scores 1.992 and the scores 0.383. The ratio is what matters: the error code carries about five times the weight of the.

This is why BM25 is good at identifiers. A product code, an error number or a person's surname usually appears in very few documents, so it gets one of the highest IDF values in the collection, and a document containing it rises to the top.

A document that mentions "refund" ten times is probably more about refunds than one that mentions it once, but not ten times more. BM25's term-frequency factor f × (k1 + 1) / (f + k1) (at average length) rises quickly and then levels off towards a ceiling of k1 + 1. With k1 = 1.2:

• 1 occurrence: 1.000
• 2 occurrences: 1.375
• 3 occurrences: 1.571
• 5 occurrences: 1.774
• 10 occurrences: 1.964, against a ceiling of 2.2

A smaller k1 saturates sooner. At k1 = 0.5, ten occurrences score only 1.429, so presence matters and repetition hardly does. A larger k1 keeps rewarding repetition: at k1 = 2.0, ten occurrences score 2.5.

Saturation is the main difference from classic TF-IDF, where the term-frequency part grows without limit (as f or 1 + log f). It also limits keyword stuffing: past a few occurrences, repeating a word barely moves the score. The (k1 + 1) in the numerator does not change the ranking, since it multiplies every term equally. Robertson and Zaragoza note it is there so that a single occurrence gets the same weight as in the plain relevance-weight formula, which is why one occurrence scores exactly 1.0 above.

Long documents contain more words, so they match more query terms by chance. Robertson and Zaragoza describe two reasons a document might be long. Verbosity: the author uses more words for the same content, which suggests dividing counts by length. Scope: the document covers more topics, which suggests not penalizing length at all. Real collections mix both, so BM25 applies a partial normalization controlled by b between 0 and 1.

The factor 1 − b + b × dl / avgdl scales k1. For two occurrences of a term (k1 = 1.2):

• Document twice the average length, b = 0: no length penalty, factor 1.375.
• Same document, b = 0.75: factor 1.073.
• Same document, b = 1: full normalization, factor 1.000.
• Document half the average length, b = 0.75: factor 1.600.

So with the usual default, two hits in a short document are worth about 50% more than two hits in a long one. Robertson and Zaragoza report that values of b between 0.5 and 0.8 work reasonably well in many collections, and Lucene defaults to 0.75. They also note that the best values depend on the type of documents and queries, so the defaults are a starting point, not a result.

To see the difference concretely, the code above and an embedding model were run on ten short support articles. The articles include three error codes that differ by one digit (E1042, E1024, E1402) and two product codes that differ by two swapped digits (SKU-88213, SKU-88231). Five queries were run against them; the results are in the table.

The product-code query shows the failure mode. The embedding model ranked the wrong SKU first, with cosine similarities of 0.700 and 0.695: the two strings are almost identical as text, so their vectors are almost identical too. BM25 sees 88231 as a separate token that occurs in exactly one document, gives it a high IDF, and ranks the correct document first by a factor of more than two. The error-code query went the right way for the embeddings, but by a margin small enough that a slightly different corpus could flip it.

The paraphrase query shows the reverse. "How long until I get my money back" shares no words with "Refunds are sent to the original payment method within five business days", so BM25 returns nothing. The embedding model matches the meaning and ranks the refunds article first. The sign-in query shows a smaller BM25 weakness: the second result, "Invoices can be downloaded...", matched only on the word can.

This is a demonstration of the mechanism, not a benchmark. Ten documents and five queries say nothing about average accuracy, and a larger or newer embedding model may separate the SKUs. The point is which kind of query each method is structurally good or bad at.

The BEIR benchmark (Thakur et al., 2021) tested ten retrieval systems zero-shot on 18 datasets, from biomedical search to argument retrieval to fact checking. Its main finding was that BM25 held up across domains better than most of the neural systems tested. In nDCG@10 averaged against BM25:

• The dense retrievers did worse on average: TAS-B −2.8%, ANCE −7.4%, DPR −47.7%.
• docT5query, which expands documents with generated queries and then uses BM25, did +1.6%.
• BM25 followed by a cross-encoder reranker did best, at +11%, beating BM25 on 16 of 18 datasets.

The mechanism the authors point to is domain shift. Dense models trained on MS MARCO beat BM25 by 7 to 18 points on MS MARCO itself, then fell behind on datasets far from it, such as BioASQ and Touché-2020. BM25 has no training data to drift away from. It applies the same statistics to any text, so it degrades less when the domain changes.

The boundary is time. BEIR's dense models are from 2020 and 2021, and embedding models trained on far larger and more varied data have since raised scores on BEIR's retrieval tasks, so do not read these figures as a current ranking. What has not changed is the structure of the result: a lexical method has no domain to drift from, and a strong reranker on top of a cheap first stage remains a reliable design. For measuring retrieval on your own data, see RAG Retrieval Metrics Explained.

Since BM25 and embeddings fail on different queries, the common fix is to run both and merge the results. Their scores cannot be added directly: BM25 scores are unbounded and depend on the collection, while cosine similarities sit in a narrow band. Reciprocal rank fusion (RRF) avoids the problem by using only ranks. Each document gets 1 / (k + rank) from each list it appears in, and the sums are sorted:

1def rrf(rankings, k=60, top=3):2    s = {}3    for r in rankings:4        for rank, d in enumerate(r, start=1):5            s[d] = s.get(d, 0) + 1 / (k + rank)6    return sorted(s, key=s.get, reverse=True)[:top]

Cormack, Clarke and Büttcher fixed k = 60 in a pilot study and found it near-optimal; Elasticsearch's RRF uses 60 as its default rank_constant. In the test above, RRF put the right document first for all five queries, but look at how. The paraphrase query worked because BM25 returned nothing, so the dense list stood alone. For SKU-88231 the two SKUs tied exactly: each was ranked first by one method and second by the other, scoring 1/61 + 1/62. The correct one came first only because the BM25 list was passed in first.

That tie shows RRF's limit. It ignores how confident each retriever was, so BM25's clear 3.61-to-1.49 margin counts the same as dense's 0.700-to-0.695 near-tie. When one signal must win, such as an exact ID match, add a rule or a weight rather than relying on fusion, and put a reranker after fusion for the final order. For the reranking step, see Rerankers vs Embeddings.

Do not deploy the 31-line class. Use an engine that keeps an inverted index, updates statistics incrementally and has proper analyzers:

• Lucene-based engines (Elasticsearch, OpenSearch, Solr) use BM25 as their default similarity, with Lucene's k1 = 1.2 and b = 0.75.
• Python libraries that implement BM25 work for prototypes and small collections held in memory.
• PostgreSQL's built-in full-text search is not BM25. Its documentation states that ts_rank and ts_rank_cd "do not use any global information", so there is no IDF. Rare and common terms are weighted by frequency within the document only.

Whichever engine you choose, most of the quality comes from the analyzer: tokenization, lowercasing, stemming, synonyms and how identifiers are split. An ID like SKU-88231 should stay searchable both whole and in parts, and an analyzer that strips digits or splits on hyphens differently from your users' typing will lose the main advantage BM25 has.

For a RAG system, a reasonable default is hybrid retrieval (BM25 plus embeddings, fused with RRF) followed by a reranker. Evaluate it against BM25 alone and embeddings alone on real queries from your logs.

Work through these with the code in this guide:

1. Index the ten-document corpus (three error codes, two SKUs, password, lockout, refunds, dark mode and invoices articles) and run the five queries from the table. 2. Print idf("e1042") and idf("the"), then change the IDF to the classic form without 1 + and find which terms go negative. 3. Plot the term-frequency factor for k1 = 0.5, 1.2 and 2.0 for counts 1 to 10 and check it against the saturation diagram. 4. Add a document that repeats "payment" twenty times. With b = 0.75 and then b = 0, does it outrank the E1042 article for "payment timed out"? 5. Embed the corpus with any embedding model and rerun the queries. Note where it ranks a near-identical ID first. 6. Fuse both rankings with the rrf function, print the fused scores for the SKU query, and add a rule that boosts documents containing the exact query string.

Pass criterion: your BM25 scores match the guide to two decimal places, you can explain the step 4 result from the formula, and you can explain why the two SKUs tie under RRF and how to break the tie in favour of the exact match.