How to Measure Context Quality
Most teams measure only the final answer, which tells you that something is wrong but not where, so they end up retuning the prompt at random while the real fault sits upstream in retrieval or memory. Measuring context quality separately fixes this by putting a meter on the window itself, before the answer, so a bad answer can be traced to a bad window and a bad window to the stage that produced it. These steps build that measurement, from capturing the window through tracking the metrics over time, and they connect the context pipeline to the broader practice of LLM evaluation.
Step 1: Log the assembled window
Everything starts with capturing the exact context sent to the model on each request, broken down by source so you can see what the instructions, retrieved knowledge, memory, history, and tool results each contributed. Without this log there is nothing to measure, the window is ephemeral and gone the moment the model responds, so the first and non-negotiable step is to persist it. Record it alongside the request, the answer, and any quality signal you have, so the window and its outcome stay linked. This is the same observability that a good pipeline builds in from the start, and every metric below is computed from this log.
Step 2: Measure context recall
Context recall asks the most important question: was the information needed to answer this request actually present in the window? A wrong answer from a window that never contained the needed fact is a retrieval or memory failure, not a model failure, and recall is the metric that catches it. Measure it by taking a set of requests with known correct answers, identifying what information each answer required, and checking whether that information was in the assembled window. Low recall means the upstream selection is missing what matters, which is the highest-priority problem to fix because no downstream tuning can recover an answer whose evidence was never in the window.
Step 3: Measure relevance density
Where recall asks whether the needed content was present, relevance density asks how much of the window was actually relevant, the fraction of tokens that bore on the request versus padding and noise. High recall with low density means the window contained the answer but buried it under irrelevant content, which invites distraction and wastes budget. Estimate density by judging, for a sample of windows, what share of the included content was pertinent, which can be scored by a model-based grader for scale. Tracking density alongside recall separates two distinct problems, missing information versus diluted information, that have different fixes, better retrieval for the first and stricter filtering for the second.
Step 4: Run failure-mode checks
Beyond the aggregate scores, check the assembled window for the specific failure modes, because each one is detectable in the log and each has a distinct fix. Scan for poisoning by flagging low-confidence or stale items that entered as fact, for confusion by detecting contradictory facts present together, and for clash by finding retrieved content that conflicts with the instructions. These checks turn the general failure modes into concrete detectors you can run over the logged windows, so you learn not just that quality is low but which failure is driving it. Counting how often each mode occurs tells you where to invest, whether in reliability gating, memory reconciliation, or retrieval filtering.
Step 5: Correlate context with answer quality
The payoff step is linking the context metrics to answer correctness, so you can see which context problem actually drives failures for your application. With the window logged and scored, and the answers graded for correctness, you can ask whether wrong answers cluster with low recall, low density, or a particular failure mode. This correlation is what points you at the stage to fix: if failures track low recall, invest in retrieval, if they track low density, invest in reranking and filtering, if they track a specific failure mode, fix its source. Without the correlation you know your context metrics and your answer quality separately, with it you know which context problem is costing you answers.
Step 6: Track the metrics over time
Finally, compute these metrics on a fixed evaluation set and track them as the system changes, so a regression is caught the moment it appears rather than discovered by users. A context pipeline drifts: a retrieval index grows, a prompt gets edited, a memory store fills, and any of these can quietly lower recall or density. A standing test set of representative requests, scored on every change, turns these silent regressions into a visible number that moves, which is the same discipline that LLM evaluation applies to model outputs generally. Tracking over time is what keeps the gains from context engineering from eroding as the system evolves.
Measure context quality by logging the window, then scoring context recall for whether the needed information was present, relevance density for how much of the window mattered, and failure-mode counts for specific defects. Correlate these with answer quality to find the stage to fix, and track them on a fixed set to catch regressions.
Why Context Metrics Beat Answer-Only Metrics
Grading only the final answer is necessary but not sufficient, because it collapses many distinct problems into one verdict. An answer can be wrong because the model reasoned poorly over a good window, or because the window was missing the fact, or because the fact was present but buried, or because two facts contradicted, and answer-only grading cannot tell these apart. Context metrics decompose the verdict into its causes, attributing the failure to the stage responsible, which is what makes the difference between knowing your system is at seventy percent and knowing that ten of the thirty missing points come from retrieval recall and the rest from distraction. That decomposition is the entire value of measuring context, it converts a single unactionable score into a set of actionable ones.
Where Memory Shows Up in the Metrics
A memory layer affects these metrics directly, and measuring context quality is how you see whether your memory is helping or hurting. Good memory raises context recall by supplying facts that pure retrieval would miss, the things a user told you in a past session that bear on the current request, so a jump in recall when memory is added is the signal that it is doing its job. Poor memory, by contrast, lowers relevance density by flooding the window with loosely related recollections, or causes confusion by returning stale facts, both of which show up in the metrics as a density drop or a rise in the confusion count. Because Adaptive Recall ranks stored facts by relevance and confidence and returns only the pertinent ones, it is built to raise recall without paying in density, and the context metrics are exactly where you verify that trade is landing in your favor. The persistence side of this is covered across the AI memory pillar.
Building a Context Evaluation Set
Every metric above depends on a fixed set of requests to measure against, so building a good evaluation set is the practical foundation that makes context measurement possible. The set should be a collection of representative requests, each paired with its correct answer and, importantly, with the specific information the answer requires, because that required-information annotation is what lets you compute context recall automatically rather than by hand. Draw the requests from real traffic where you can, so the set reflects the questions users actually ask, and include the hard cases deliberately: the requests that need a fact from a past session, the ones where the answer hinges on a single buried document, the ones where stale and current facts both exist. A set that only contains easy requests will report high scores while the system fails on exactly the cases that matter.
Keep the set fixed once you have it, so that a change in the metrics reflects a change in the system rather than a change in what you measured, and grow it deliberately by adding new cases as you discover failure patterns in production. When a user hits a wrong answer that traces to a context problem, distill it into a new evaluation case with its required information annotated, so the set accumulates exactly the situations your system has struggled with. Over time this turns the evaluation set into an institutional memory of your context failures, and running it on every change is what catches a regression the moment it appears. This is the same discipline the LLM evaluation pillar applies to model outputs, focused here on the window rather than the answer, and it is what lets the metrics in this guide function as a standing measurement rather than a one-time audit.