Cross-encoder reranking: what it costs, and whether you need it

Reranking is usually introduced as the next thing you add after hybrid search. Retrieve broadly, then rescore the top candidates with a model that reads query and document together. Relevance improves, everyone is pleased.

The part that gets skipped is that you have just put a transformer forward pass into your request path — once per candidate, on every search. So let us start with the bill.

The measured cost

These numbers are from a benchmark I ran on a single vCPU with eager PyTorch, no ONNX, no quantisation. Sequence length 256, batch size 1, median of five runs. Reranker architectures at the three common sizes:

Reranking fifty documents with the smallest commonly used reranker takes four and a third seconds on one core. With a base-sized model it is twenty-six seconds. For a search box, both are simply broken.

That is a deliberately unflattering baseline — one core and an unoptimised runtime — so here is the same workload with the usual optimisations applied. Only the first row is measured; the rest are scaling estimates and should be treated as such:

The shape of that table is the whole engineering problem. Cross-encoder reranking is viable, but only with real optimisation work and probably a GPU — and the moment a GPU is involved, where it physically sits becomes a question with legal consequences.

Getting under 100 ms

Four seconds is not a latency problem, it is a broken feature. A search box needs the whole response inside about 100 ms to feel instant, and reranking is only one item on that bill — retrieval, network and rendering all want a share.

From the measured baseline that is a 43× speedup. It sounds impossible and is not. Three levers do the work, and here they are measured individually on the same single vCPU:

Sequence length: 2.1× for halving it. Not the 4× you would predict from attention being quadratic — at these small hidden sizes the feed-forward layers dominate the arithmetic, and those are linear in sequence length. Worth knowing, because it means truncation helps less than the textbook suggests while still being the cheapest lever you have, and the one that costs the least accuracy.

Depth: exactly linear. Six layers to two is precisely 3×. No surprises, and it makes distillation easy to budget for.

Width: 2.0× for 384 to 256 hidden. Close to the quadratic (384/256)² = 2.25 the matmul shapes predict.

Stacking them

Compose the levers and the arithmetic works out:

k=50, L6/384, seq 256, 1 core      4,334 ms   baseline
  truncate to seq 128              2,064 ms   x2.1
  rerank k=25 instead of k=50      1,032 ms   x2.0
  distil to 2 layers                 348 ms   x3.0
  narrow to 256 hidden               170 ms   x2.0
  ONNX Runtime + int8 (est. ~3x)      57 ms   under budget

Everything above the last line is measured. Only the ONNX step is an estimate — and you may not even need it: k=10 with a 2-layer, 256-hidden model at sequence length 128 is 68 ms on a single core, unoptimised, straight out of eager PyTorch.

Which gives the finding that matters most here. The GPU is what you buy when you refuse to compromise on model size, sequence length, or k. Compromise on all three and one CPU core is enough. That changes the EU hosting question from an infrastructure project into a container you already know how to run — no accelerators to procure, no GPU quota, no region availability to check.

The trade is real and should not be glossed. A 2-layer, 256-hidden model reranking ten truncated candidates is meaningfully less accurate than a base-sized model over fifty full documents. It is also roughly 250 times cheaper. Whether the accuracy gap matters for your corpus is precisely what a judged set answers — and if the tiny model still beats your unreranked baseline on nDCG, it has cost you nothing to find out.

Why it costs this much

The expense is structural, not an implementation detail, and it comes down to what can be computed in advance.

A bi-encoder — what generates the embeddings in your hybrid setup — encodes each document into a vector independently, offline, at index time. At search time you encode one query and do an approximate nearest-neighbour lookup. The document side of the work was done days ago.

A cross-encoder concatenates query and document into a single input and runs them through the model together, with attention flowing across both. That is why it is more accurate: it can resolve that "bank" in the query and "bank" in the document refer to different things, because it sees them in each other's context.

It is also why nothing can be precomputed. The input does not exist until the query arrives. Fifty candidates means fifty forward passes, and there is no cache to warm because the pairs are new every time.

Hence the standard architecture: cheap retrieval to narrow millions of documents to tens, then expensive reranking on those tens. The cross-encoder never sees the corpus, only the shortlist. Retrieval optimises for recall; reranking optimises for precision.

The benchmark, in full

Every number above came from the script below. Two things about the method, stated plainly because they affect how much you should trust the figures.

The models are randomly initialised. There are no trained weights here. A forward pass costs the same regardless of what the weights contain — cost is determined by architecture, sequence length and batch size, which are all fixed by the model shape. So the timings are representative of a real reranker of the same dimensions. What this cannot tell you is anything about quality; for that you need real weights and a judged set.

The environment is deliberately unflattering. One vCPU, eager PyTorch, no ONNX, no quantisation, no batching across candidates. That is a floor, not a typical deployment, and it is chosen so the optimisation levers have somewhere to work from.

#!/usr/bin/env python3
"""
Cross-encoder reranking latency benchmark.

Latency is determined by architecture, sequence length and batch size --
not by the values of the weights -- so randomly initialised models of the
correct shape give representative timings.

    pip install torch
    ./bench.py
"""

import json
import time

import torch
import torch.nn as nn

torch.manual_seed(0)


class Encoder(nn.Module):
    """Standard pre-LN transformer encoder with a classification head."""

    def __init__(self, layers, hidden, heads, vocab=30522, max_len=512):
        super().__init__()
        self.tok = nn.Embedding(vocab, hidden)
        self.pos = nn.Embedding(max_len, hidden)
        layer = nn.TransformerEncoderLayer(
            d_model=hidden,
            nhead=heads,
            dim_feedforward=hidden * 4,
            batch_first=True,
            norm_first=True,
            activation="gelu",
        )
        self.enc = nn.TransformerEncoder(layer, num_layers=layers)
        self.head = nn.Linear(hidden, 1)

    def forward(self, ids):
        pos = torch.arange(ids.size(1), device=ids.device).unsqueeze(0)
        x = self.tok(ids) + self.pos(pos)
        x = self.enc(x)
        return self.head(x[:, 0])          # score from the [CLS] position


def bench(model, batch, seqlen, runs=5, warmup=1):
    """Median wall-clock ms for one forward pass."""
    ids = torch.randint(0, 30000, (batch, seqlen))
    with torch.inference_mode():
        for _ in range(warmup):
            model(ids)
        times = []
        for _ in range(runs):
            t0 = time.perf_counter()
            model(ids)
            times.append((time.perf_counter() - t0) * 1000)
    times.sort()
    return times[len(times) // 2]


# (label, architecture, sequence length)
CONFIGS = [
    ("L6/384  seq256", dict(layers=6,  hidden=384, heads=12), 256),
    ("L6/384  seq128", dict(layers=6,  hidden=384, heads=12), 128),
    ("L6/384  seq64",  dict(layers=6,  hidden=384, heads=12),  64),
    ("L4/384  seq128", dict(layers=4,  hidden=384, heads=12), 128),
    ("L2/384  seq128", dict(layers=2,  hidden=384, heads=12), 128),
    ("L2/256  seq128", dict(layers=2,  hidden=256, heads=8),  128),
    ("L12/384 seq256", dict(layers=12, hidden=384, heads=12), 256),
    ("L12/768 seq256", dict(layers=12, hidden=768, heads=12), 256),
]


def main():
    print(f"threads={torch.get_num_threads()}  "
          f"cuda={torch.cuda.is_available()}\n")
    print(f"{'config':<18}{'params':>9}{'ms/pair':>9}{'k=10':>9}"
          f"{'k=25':>9}{'k=50':>9}")

    results = []
    for label, cfg, seqlen in CONFIGS:
        model = Encoder(**cfg).eval()
        nparams = sum(p.numel() for p in model.parameters()) / 1e6
        ms = bench(model, batch=1, seqlen=seqlen)
        print(f"{label:<18}{nparams:>8.0f}M{ms:>9.1f}"
              f"{ms * 10:>8.0f}ms{ms * 25:>8.0f}ms{ms * 50:>8.0f}ms")
        results.append(dict(
            config=label,
            params_m=round(nparams, 1),
            seqlen=seqlen,
            ms_per_pair=round(ms, 1),
            k10_ms=round(ms * 10),
            k25_ms=round(ms * 25),
            k50_ms=round(ms * 50),
        ))

    json.dump(results, open("bench.json", "w"), indent=1)
    print("\nwrote bench.json")


if __name__ == "__main__":
    main()

Run it on the hardware you actually intend to deploy on. The absolute numbers will differ — probably by a lot, since one vCPU is a pessimistic floor — but the ratios between configurations hold, and those ratios are what you budget with.

If you want a number for a specific reranker rather than an architecture class, install sentence-transformers, load the real model, and time model.predict() over a list of query-document pairs. Same measurement, real weights, and you can score quality against a judged set in the same script.

Try the cheap things first

This is the section most reranking articles skip, and it is where most of the value is for a site of ordinary size.

Raise k on your existing retrieval. If you fuse the top 20 from each leg, try 100. Costs almost nothing, and if your problem is that good documents are not reaching the fusion stage at all, this fixes it outright. No reranker can promote a document that retrieval never returned.

Tune your fusion. Weight the lexical and vector legs against a judged set. Free, and it is the experiment your evaluation harness exists for.

Fix the analysers. Language-appropriate stemming, compound splitting for Danish, a synonym list for the terms your users actually use. Cheaper than inference hardware and it improves every query, not just the top fifty results of the ones you rerank.

Late interaction (ColBERT-style) is the genuine middle ground. It precomputes per-token vectors for documents and does a cheaper interaction at query time — better than a bi-encoder, far cheaper than a cross-encoder. The cost is index size, since you store many vectors per document rather than one.

Only after those should a cross-encoder be on the table. And the honest position for most sites is that it will not be needed: if you have ten thousand documents and a well-tuned hybrid retriever, the ceiling you are hitting is more likely to be your content than your ranking.

The EU hosting question

Now the part specific to running this from Denmark or anywhere else in the EU, because reranking has a materially different privacy profile from embedding.

Embedding your corpus is a batch job. It happens at index time, it can run anywhere including a machine under your desk, and if you use an external provider you send them your documents once. Unpleasant, but bounded and schedulable.

Reranking sends the user's query. Every search, in real time, paired with document text. On a public site that is search terms; on an intranet, a customer portal, a case management system, or anything municipal, those queries are personal data of a fairly sensitive kind. People type things into internal search boxes — names, case numbers, diagnoses, complaints — that they would never put in a public form.

So a reranker on an API endpoint outside the EU means every internal query crosses a border, continuously, as a matter of normal operation. That is not a footnote in a DPIA; it is the main event.

Options, in increasing order of comfort

Self-hosted on your own hardware is the strongest position and, for reranking specifically, more achievable than it sounds. Rerankers are small — the whole point of MiniLM-L6 is that 23M parameters is enough. This is not a model needing a rack of H100s; a single mid-range GPU handles a substantial query load, and for lower volumes an optimised CPU deployment is viable. No data leaves your infrastructure, no processor agreement, no transfer analysis.

A European cloud provider — OVHcloud, Scaleway, Hetzner and others offer GPU instances in EU regions. You still run the model; they supply the metal. The analysis reduces to a normal processor relationship with an EU-established company, which is the well-trodden path.

A European inference API is the least operational work: managed endpoints from EU providers, where you send the pair and get a score back. This is a genuine processing relationship, so read the terms rather than assuming EU incorporation settles it. The questions worth asking are whether inputs are retained or logged, whether anything is used for training, and where support and administrative access physically sit.

A US-headquartered provider with an EU region is where it gets genuinely contested. Data may rest in Frankfurt while the parent company remains subject to US jurisdiction. The EU-US Data Privacy Framework exists precisely to address this and is under ongoing legal challenge — its two predecessors were both struck down. Whether that is acceptable for your data is a question for your DPO and your risk appetite, not for a blog post. But it is a question, and "the region is in Europe" does not close it.

What this buys you technically

Worth noting that the sovereignty-friendly option is also the technically better one here, which is not always the case.

Reranking is latency-critical in a way batch embedding is not — it sits in the request path, and a network round trip to an external API adds tens of milliseconds before the model does any work. A reranker on the same network as your search cluster, or on the same machine, eliminates that entirely. You also get to batch all fifty pairs into one forward pass rather than paying per-request overhead fifty times.

So the arrangement that keeps queries inside your jurisdiction is also the one that gives you the lowest latency and the most control over throughput. That alignment is worth pointing out because it means you are not trading performance for compliance.

If you do build it

Rerank 20 to 50 candidates, not 200. Latency is linear in k, and the gain from reranking falls off quickly past the first few dozen. If a document at rank 150 is the right answer, your retrieval is the problem.

Truncate aggressively. Attention cost scales quadratically with sequence length. Reranking a title plus the first 200 tokens is far cheaper than a full article and often nearly as accurate, because the relevant passage is usually near the top. If it is not, you want passage-level reranking, which is a different design.

Cache by query. Head queries repeat constantly. As shown in the query log analysis, a handful of queries can account for a third of all searching — and those are exactly the ones you never need to rerank twice. Cache on the query plus the retrieved document IDs, and invalidate on reindex.

Export to ONNX and quantise. The single largest practical speedup available, typically several times faster than eager PyTorch, with quality loss that is usually negligible at int8. Measure it on your judged set rather than assuming.

Have a fallback. If the reranker is slow or down, return the retrieval ordering. Degraded ranking is a much better outcome than a search box that times out, and this should be a timeout, not an exception handler.

Then prove it worked

Reranking has a real, measurable cost — hardware, latency, an extra service to operate. That cost has to be justified by a real, measurable gain, and this is precisely what the evaluation harness is for.

Measure nDCG@10 with and without the reranker on the same judged query set. Look at the per-query distribution rather than the mean: rerankers characteristically help discursive natural-language queries a great deal and can actively hurt exact-identifier lookups, where the lexical retriever already had it right and the model second-guesses it into fourth place.

If the improvement is a couple of points of nDCG on a corpus of ten thousand documents, you have bought a GPU and a p99 latency problem for something your users will not notice. If it is a large gain on the natural-language queries that make up most of your distinct traffic, it is worth every bit of the infrastructure.

You cannot tell which from reading about it, including from reading this. The only way to know is to measure it on your own corpus — which is the same conclusion as every other article in this series, and remains the only honest one.