How to Detect LLM Quality Regressions
LLM systems regress for reasons that have no equivalent in conventional software. A prompt edit intended to fix one case quietly breaks three others. A model-provider version update changes behavior overnight with no code change on your side. A growing retrieval index starts returning different documents. User inputs drift toward patterns the system was never tuned for. The question is never whether a system will regress, only whether you will detect it on the change that caused it or weeks later through complaints.
The first line of defense is running your evaluation dataset in continuous integration on every change to a prompt, model, retrieval component, or any code that touches the LLM path. The evaluation reports per-dimension scores against the previous baseline, and a significant drop blocks the change or flags it for review. This catches the self-inflicted regressions, the prompt edit that helped one case and hurt others, before they ever reach production. A change that cannot pass the evaluation does not ship.
A regression is a drop relative to a baseline, so you need a recorded baseline score per metric and per segment, and a threshold that defines how large a drop counts. The threshold must account for the inherent variance of LLM evaluation: a two-point move might be noise, while a ten-point move is real. Set thresholds from observed variance rather than guessing, and set them per segment, because a drop concentrated in one important query category can be real even when the aggregate barely moves.
Because LLM outputs vary across runs, a single evaluation run can show a score change that is pure noise. Run multiple samples per example, especially at non-zero temperature, and compare the distribution of scores rather than a single point. Report a mean with a confidence interval, and only call a regression when the change exceeds what the variance can explain. Skipping this step produces false alarms that erode trust in the evaluation and real regressions that hide inside the noise band.
Two regression sources have no corresponding code change and so evade change-gated evaluation. Model drift happens when the provider updates the model behind an endpoint; pin model versions where the provider allows it, and run a scheduled evaluation against production traffic so a silent provider change shows up as a score drop even with no deploy. Data drift happens when the input distribution shifts or the retrieval corpus changes; monitor the distribution of incoming query types and the composition of retrieval results so you notice when the world the system operates in has moved.
Offline evaluation cannot cover the long tail of real inputs, so production monitoring is the second line of defense. Run online evaluators on sampled traffic, compute quality metrics over rolling windows, and compare each window against the trailing baseline. Alert when a metric drops beyond its threshold. This is where the model-provider updates and data drifts that slipped past offline gating get caught, because they show up as a degradation in the live quality metric even when nothing in your codebase changed.
Detection is only useful if it leads to a fix, and a fix requires knowing the cause. When a regression fires, correlate it with the deploy history (did a change ship at that time), the model version (did the provider update), and the failing traces (what specifically is now wrong). The failing traces also become new dataset examples, so the regression becomes a permanent test. A regression you detected but could not attribute is a half-solved problem; the trace and deploy correlation is what closes it.
Detect regressions in two layers: offline evaluation gates the changes you make, production monitoring catches the drift you did not, from model-provider updates to shifting inputs. Sample multiple runs to separate real drops from noise, and tie every regression to its cause through traces and deploy history.
Improvements That Are Secretly Regressions
Not every regression looks like a falling number; some hide inside an apparent improvement. A change that raises average accuracy by improving common queries while quietly degrading a rare but important category is a net regression for the users in that category, and it is invisible unless you segment. A change that improves a quality metric the judge measures while worsening a quality the judge does not capture, such as tone or concision, trades a visible gain for an invisible loss. A prompt that boosts faithfulness by making the model refuse more often improves the faithfulness metric while hurting helpfulness, because a refusal is trivially faithful. These cases are why a single headline metric is dangerous: optimizing it can drag down the dimensions it does not cover.
The defense is to evaluate the full set of dimensions together and to segment every one of them, so that a gain in one metric or one segment cannot mask a loss in another. When a change improves the headline number, the question to ask is what got worse, and the evaluation should be structured to answer it: per-dimension scores, per-segment breakdowns, and operational metrics all reported side by side. A change ships only when it holds or improves across the board, or when the tradeoff it makes is explicit and accepted. Treating every improvement with the same skepticism as a suspected regression is what prevents the slow erosion where each individually reasonable change quietly degrades a dimension nobody was watching.
Regressions in Memory and Retrieval
Systems with persistent memory have a regression source unique to them: the memory store itself evolves. Consolidation can merge away a useful distinction, decay can remove information that was still needed, and confidence can inflate without supporting evidence. These degrade retrieval quality with no code or prompt change at all, so they only surface through monitoring the retrieval metrics over time. Adaptive Recall's status tool reports retrieval quality and confidence distribution, which lets you put memory-layer regression detection on the same footing as the rest of the system, comparing current retrieval precision and confidence calibration against a baseline and alerting when the store starts drifting in the wrong direction. This connects to the broader discipline of an observability layer for AI learning.