Ranking search results with BM25

So far in this series we have crawled pages, broken them into tokens, and built an index that can tell us which documents contain a term. That leaves the question every search engine actually lives or dies by: in what order do we show them?

A query for "puppet ubuntu" might match four hundred documents. The user will look at five. Getting those five right is the ranking problem, and for keyword search the answer has been more or less settled since the 1990s: BM25.

BM25 is the default scoring function in Lucene, Elasticsearch, OpenSearch, and Solr. If you have ever run a full-text query against any of them, you have used it. It is worth understanding what it is doing, because the two knobs it exposes are the difference between a search box people trust and one they abandon.

Why TF-IDF isn't enough

The obvious first attempt at ranking is TF-IDF: multiply how often the term appears in the document (term frequency) by how rare the term is across the whole collection (inverse document frequency). Rare terms count for more, and documents that mention the term more often score higher.

It works, right up until it doesn't. Two problems break it.

Term frequency grows without limit. In plain TF-IDF, a document that mentions "puppet" forty times scores forty times higher than one that mentions it once. But that is not how relevance works. The jump from zero mentions to one mention is enormous — it tells you the document is about the topic at all. The jump from thirty to forty tells you almost nothing. Relevance saturates, and linear TF ignores that completely.

Long documents cheat. A 5,000-word page will naturally contain more instances of any term than a 300-word page, without being any more relevant. Under plain TF-IDF, length is a free ranking boost. This is exactly the loophole that produced two decades of bloated, keyword-stuffed SEO pages.

BM25 fixes both. It is still fundamentally TF-IDF — rare terms matter more, repeated terms matter more — but it bends the term frequency curve so it flattens out, and it penalises documents that are longer than average.

The formula

For a query Q consisting of terms q₁ … qₙ, the score of document D is the sum of a per-term score:

                         f(qᵢ, D) · (k₁ + 1)
score(D, Q) = Σ IDF(qᵢ) · ─────────────────────────────────
                    f(qᵢ, D) + k₁ · (1 − b + b · |D| / avgdl)

Every symbol in that expression is something you already have in your index:

And the IDF component:

              N − n(qᵢ) + 0.5
IDF(qᵢ) = ln( ─────────────── + 1 )
                n(qᵢ) + 0.5

where N is the number of documents in the collection and n(qᵢ) is the number of documents containing the term. The trailing + 1 is the Lucene variant, and it matters: without it, any term appearing in more than half the collection gets a negative IDF and actively subtracts from the score. With it, common terms simply contribute very little.

Read structurally, the whole thing says: score = rarity × saturating frequency, discounted by length. That is all.

What k₁ and b actually do

These are the only two parameters you can tune, and both have an intuitive reading.

k₁ controls how fast term frequency saturates. Hold everything else constant and watch what the frequency factor does as a term repeats, with k₁ = 1.2:

The ceiling is k₁ + 1 = 2.2, and you can never reach it. The first occurrence buys you 45% of the total available benefit. The hundredth buys you essentially nothing. Keyword stuffing stops paying almost immediately — which is precisely the intent.

At k₁ = 0 the whole factor collapses to 1 for any non-zero frequency, making the model purely binary: does the term appear, yes or no. Raise k₁ and the curve straightens toward classic linear TF. Values between 1.2 and 2.0 are the usual range, and 1.2 is a sensible default for most collections.

b controls how hard long documents are punished. At b = 0 length is ignored entirely. At b = 1 the score is fully normalised by the ratio of document length to average length. The 0.75 default sits deliberately closer to full normalisation.

When to move it: if your collection is homogeneous — product descriptions, log lines, forum posts that are all roughly the same size — length carries no signal and you can lower b toward 0.4. If you index a mix of one-paragraph notes and 8,000-word guides, keep it at 0.75 or push higher. And if long documents in your corpus are genuinely more comprehensive rather than just padded, lowering b stops you burying your best content.

A worked example

Let us score three documents against the query "puppet ubuntu" by hand.

Collection statistics: 1,000 documents, average length 500 tokens. "puppet" appears in 50 documents, "ubuntu" in 200. Using k₁ = 1.2 and b = 0.75.

First the IDF values, which are fixed per term regardless of which document we are scoring:

IDF(puppet) = ln(950.5 / 50.5 + 1)  = ln(19.82) = 2.99
IDF(ubuntu) = ln(800.5 / 200.5 + 1) = ln(4.99)  = 1.61

"puppet" is worth roughly 1.9× as much as "ubuntu" per match, purely because it is rarer. Now the three candidates:

The length normaliser k₁ · (1 − b + b · |D|/avgdl) comes out to 0.84 for A, 3.00 for B, and 1.20 for C. Plugging through:

Doc A — puppet: 2.99 × (8 × 2.2) / (8 + 0.84) = 5.95, ubuntu: 1.61 × (3 × 2.2) / (3 + 0.84) = 2.76. Total: 8.71

Doc B — puppet: 2.99 × (20 × 2.2) / (20 + 3.00) = 5.71, ubuntu: 1.61 × (5 × 2.2) / (5 + 3.00) = 2.21. Total: 7.92

Doc C — puppet: 2.99 × (3 × 2.2) / (3 + 1.20) = 4.69, ubuntu: 1.61 × (3 × 2.2) / (3 + 1.20) = 2.53. Total: 7.22

Doc A wins. Note what just happened: Doc B mentions "puppet" two and a half times as often as Doc A and still loses, because it is five times longer. Meanwhile Doc C, at exactly average length, is beaten by A because A's higher density more than compensates. Density beats raw count, which is the behaviour we wanted and could not get from TF-IDF.

Implementation

The whole thing is about fifteen lines. Given a document as a term-frequency map and a collection-wide document-frequency map:

function bm25(
    array $query,
    array $doc,
    array $df,
    int $N,
    float $avgdl,
    float $k1 = 1.2,
    float $b = 0.75
): float {
    $len   = array_sum($doc);
    $score = 0.0;

    foreach ($query as $term) {
        $tf = $doc[$term] ?? 0;
        if ($tf === 0) {
            continue;
        }

        $n    = $df[$term] ?? 0;
        $idf  = log(($N - $n + 0.5) / ($n + 0.5) + 1);
        $norm = $k1 * (1 - $b + $b * $len / $avgdl);

        $score += $idf * ($tf * ($k1 + 1)) / ($tf + $norm);
    }

    return $score;
}

Two things worth noticing. Terms not present in the document contribute exactly nothing, so you only ever iterate over the posting lists you actually retrieved — this is why BM25 is cheap enough to run over millions of candidates. And IDF depends only on collection statistics, so precompute it once per term at index time rather than per query.

Where BM25 falls short

It is a bag-of-words model, and every one of its weaknesses follows from that.

It cannot handle synonyms or paraphrase. A document about "configuration management automation" scores zero for a query about "puppet" if the word never appears. This is the vocabulary mismatch problem, and no amount of parameter tuning touches it.

It ignores word order and position entirely. "puppet ubuntu" and "ubuntu puppet" produce identical scores, and a document where the two terms sit in the same sentence scores no better than one where they are 4,000 words apart. This is exactly the gap that proximity scoring exists to close, and it is the standard first upgrade on top of a BM25 baseline.

It knows nothing about the document itself — not freshness, not authority, not whether anyone has ever clicked it. BM25 is a textual relevance score, one signal among several. In production it is almost always blended with the kind of factors covered in the on-page SEO article, plus link and behavioural signals.

The modern answer to the first two is hybrid retrieval: run BM25 and a vector similarity search in parallel, then merge the two ranked lists with something like reciprocal rank fusion. The embedding side catches semantic matches that BM25 misses entirely; BM25 catches the exact-match cases — product codes, error strings, proper nouns — where embeddings are famously unreliable.

Which is the real argument for learning BM25 properly in 2026, even now that vector search is everywhere. It has not been replaced. It is still half of what a good retrieval system does, it costs almost nothing to compute, and it is the baseline every fancier approach has to beat before it earns its place.