Here is how relevance tuning usually goes. Someone complains that searching for the company name returns a press release above the About page. You raise the title boost. You try the query. It is fixed. You ship it.
What you do not know — cannot know, from that process — is what happened to the other four thousand queries people run against your search box. Maybe nothing. Maybe you just broke a hundred of them in a way nobody will report, because people who do not find what they want do not file bug reports. They leave.
This is the gap between tuning search and engineering it, and it is entirely a measurement problem. Every previous article in this series has been about building retrieval machinery. This one is about the instrument you need before any of that machinery can be honestly said to work.
Offline first
There are two families of evaluation. Online evaluation measures real users — A/B tests, click-through, abandonment. Offline evaluation measures against a fixed set of queries with human-assigned relevance judgements, and needs no users at all.
Start offline, for three reasons: it is reproducible, so the same change measured twice gives the same answer; it is fast, so you can evaluate twenty configurations in an afternoon; and it works before you have traffic, which matters if you are building the thing rather than optimising an existing one.
Online evaluation is the ground truth eventually. But it is slow, noisy, and confounded in ways covered further down.
Building a judged set
The instrument is a list of queries, each with documents marked for relevance. Everything else follows from its quality, so it is worth some care.
Where the queries come from
Your search logs, not your imagination. The queries you would invent are the ones you already think about, which is precisely the population where your search already works.
Sample across the frequency distribution rather than taking the top 50. Head queries — the handful everyone runs — dominate traffic and are usually already fine. The torso and tail is where search fails, and collectively the tail is often most of your volume. A reasonable split is roughly a third head, a third torso, a third tail.
Then add a deliberate handful of adversarial cases: an exact product code, a misspelling, a two-word phrase whose words are individually common, a natural-language question. These are your regression tests.
How many
Fifty is the floor for a number that means anything. A hundred is comfortable. Two hundred is good and starts to be a real labelling commitment.
Below fifty, per-query variance swamps you — search quality varies enormously between queries, and with twenty queries a single hard one moving from bad to good shifts your average enough to look like a systemic improvement.
Grading
Binary — relevant or not — is faster to produce and enough for precision, recall and MAP. Graded, usually 0 to 3, distinguishes "perfect answer" from "mentions the topic" and is required for nDCG. Graded is better; binary is what you will actually finish. Start binary, upgrade later if the metric stops discriminating.
You cannot judge every document against every query — that is millions of judgements. The standard trick is pooling: run every configuration you want to compare, take the top k results from each, judge the union of those, and treat everything unjudged as irrelevant.
Pooling has a known bias worth understanding. A genuinely new approach that surfaces good documents no pooled system ever retrieved gets penalised, because those documents were never judged and therefore count as irrelevant. If you later add a fundamentally different retriever — a vector leg, say — re-pool and judge the new documents rather than reusing the old judgements.
Who judges
Ideally not one person alone. Have two people judge an overlapping subset and check how often they disagree. If they disagree on a third of documents, your judgements encode one person's opinion rather than relevance, and any metric difference smaller than that disagreement rate is noise.
Write down your criteria before judging, and keep them short. "Would a user asking this be satisfied by this document" beats anything more elaborate.
The metrics
Four worth knowing, each answering a different question.
Precision@k — of the top k results, what fraction were relevant. Directly reflects what a user sees on the first page. Ignores ordering within k, and says nothing about what you missed.
Recall@k — of all relevant documents, what fraction appeared in the top k. Requires knowing the full relevant set, which pooling only approximates.
The relationship between these two is the single most useful thing on this page, because which one matters depends on which stage you are measuring. For a search box showing ten results, precision@10 is the user experience. But for the retrieval stage of a hybrid system, or anything feeding a reranker, recall@100 is what matters — you cannot rerank a document you failed to retrieve. Optimise early stages for recall and late stages for precision. Measuring the first stage on precision will lead you to tighten retrieval exactly when you should be widening it.
MRR — the reciprocal of the rank of the first relevant result, averaged over queries. If the right answer is at position 3, that query scores 1/3. Excellent for navigational or known-item search where exactly one document is correct. Useless when many documents are legitimately relevant, since it ignores everything after the first hit.
nDCG — the one to use if you use one. It accounts for both ordering and degree of relevance, and it is comparable across queries. Worth understanding properly rather than treating as a number a tool emits.
nDCG, worked through
Discounted Cumulative Gain sums the relevance of each result, discounted by how far down the list it sits:
k 2^rel_i − 1
DCG@k = Σ ─────────────
i=1 log₂(i + 1)
Two ideas, both worth stating plainly.
Gain. The 2^rel − 1 form makes relevance superlinear: on a 0–3 scale the grades are worth 0, 1, 3 and 7. A perfect result is more than twice as valuable as a merely good one, which matches how people actually experience search.
Discount. Dividing by log₂(i + 1) means position 1 counts fully, position 2 at about 63%, position 3 at 50%, position 10 at 29%. Diminishing, but gently — it does not assume nobody looks past result three.
Take a query where the top five results are graded 3, 1, 2, 0, 1:
7/1.000 + 1/1.585 + 3/2.000 + 0/2.322 + 1/2.585
7.000 + 0.631 + 1.500 + 0.000 + 0.387 = 9.518
That number alone is meaningless — it depends on how many relevant documents happen to exist for this query. So normalise by the best score achievable with the same documents, which is the ideal ordering 3, 2, 1, 1, 0:
7/1.000 + 3/1.585 + 1/2.000 + 1/2.322 + 0/2.585
7.000 + 1.893 + 0.500 + 0.431 + 0.000 = 9.823
nDCG@5 = 9.518 / 9.823 = 0.97
Now reverse that ranking to 0, 1, 1, 2, 3 — the same five documents, badly ordered:
0/1.000 + 1/1.585 + 1/2.000 + 3/2.322 + 7/2.585
0.000 + 0.631 + 0.500 + 1.292 + 2.708 = 5.131
nDCG@5 = 5.131 / 9.823 = 0.52
Identical retrieval, 0.97 versus 0.52. That gap is what nDCG measures and precision@5 — identical at 0.8 for both — cannot see at all.
Because it is normalised to 0–1 per query, a query with twenty relevant documents and one with two are directly comparable, which is what makes averaging across a judged set meaningful.
Do not report only the mean
This is where most relevance work goes wrong, and it needs no statistics background to avoid.
You measure your current configuration at nDCG@10 of 0.62 and your new one at 0.64. Is that better? Possibly. It is also entirely consistent with a change that improved thirty queries slightly and destroyed ten badly.
So look at the per-query distribution, not the average. How many queries improved, how many regressed, and by how much. A change with twenty wins and two catastrophic losses is usually a bad change even with a higher mean, because those two losses are real users hitting a wall — and if one of them is your company name, the mean will not save you from that meeting.
If you want a significance test, use a paired one — paired t-test or Wilcoxon signed-rank — run over the same queries in both configurations. Pairing matters enormously here: query difficulty varies far more than the difference between two ranking configurations, and pairing removes that variance from the comparison. An unpaired test on search metrics will almost never find anything.
Break results out by query class as well — navigational, informational, exact-identifier. The aggregate hides exactly the trade you most need to see. Adding a vector leg to your retrieval typically improves natural-language queries and degrades identifier lookups; a single average shows you a modest gain and conceals both.
Judgements from clicks, and why they lie
Hand-labelling does not scale, and eventually you want judgements derived from behaviour. OpenSearch's User Behaviour Insights captures query and click events and derives implicit judgements from them.
This works, with one large caveat: click data is position-biased. Users click the first result substantially more often than the second, largely because it is first. So if you rank a document at position 1 and it gets clicked, you have learned very little — you may simply have measured your own ranking decision reflected back.
Naively training on click data therefore entrenches whatever your current ranking does. The corrections are either a click model that estimates and divides out position bias, or interleaving — merging two rankings into a single list so both compete at every position, which controls for position bias by construction and is dramatically more sensitive than A/B testing whole result pages.
Two behavioural signals are more honest than click-through and cost nothing to collect: query reformulation (they searched again immediately, so the first attempt failed) and abandonment (they searched and clicked nothing). Both are unambiguous failures in a way a click never is unambiguously a success.
Tooling
You can do all of this in a spreadsheet and a script, and for fifty queries that is genuinely fine.
If you are on OpenSearch, the Search Relevance Workbench builds it in. Its model maps onto everything above: a query set is your query list, a judgment list is your relevance labels, a search configuration is one version of your query, and an experiment runs them together. It reports nDCG, MAP, precision and coverage, both aggregated and per query, with dashboards from version 3.2.
Two things worth knowing before you rely on it. Recall and MRR are not currently in the standard output — there is an open feature request — so if you are measuring a retrieval stage rather than a final ranking, you may still need to compute recall yourself. And there is a dedicated hybrid search optimisation experiment type that sweeps parameter combinations for hybrid queries, which is the honest way to choose between RRF and score normalisation rather than taking my word for it.
The minimum viable version
If the above sounds like a project, it does not have to be. This is an afternoon:
- Pull 50 queries from your logs — roughly a third each from head, torso and tail.
- Run them against your current search. Judge the top 10 of each, binary, relevant or not.
- Compute precision@10 and, if you can approximate the relevant set, recall@50.
- Write the numbers down with the date and the configuration.
- Rerun before and after every relevance change. Look at which individual queries moved.
That is not a rigorous evaluation. It will still catch the change that improves your test query and quietly wrecks fifteen others, which is the failure mode that matters and the one nobody catches by hand.
The wider point: everything in this series — BM25 parameters, field boosts, analyser chains, fusion strategy — is a knob. Turning knobs without an instrument is not tuning. It is guessing with extra steps, and the reason so much relevance work feels like superstition is that it usually is.