LLM Evaluation and Observability

Why Evaluation and Observability Matter

An LLM application is non-deterministic, sensitive to small prompt changes, and dependent on data that drifts. The same prompt that worked yesterday can produce a worse answer today because the model provider silently updated a model version, because a retrieval index grew and started returning different documents, or because users started asking questions the system was never tested on. Traditional software has deterministic outputs you can assert against. LLM software has a distribution of plausible outputs, and the only way to know whether that distribution is good is to measure it deliberately.

The cost of skipping measurement is not hypothetical. Teams that ship LLM features without evaluation routinely discover problems through customer complaints rather than dashboards: a support bot that started hallucinating refund policies, a coding assistant whose accuracy dropped after a model upgrade, a RAG system that returns confident answers from outdated documents. By the time a human notices, the system has been degraded for days or weeks. Evaluation and observability replace this reactive cycle with a proactive one, where you catch regressions on the change that caused them and you measure quality continuously instead of waiting for the next complaint.

The two disciplines are complementary and most production teams need both. Evaluation is what you run before shipping and on every pull request: you take a fixed set of inputs, generate outputs, and score them against expectations. Observability is what runs in production: every request emits a trace of the prompts, retrieved context, tool calls, token counts, latencies, and final outputs, and metrics aggregate those traces so you can see trends and set alerts. A regression caught in evaluation never reaches a user. A regression that slips through is caught by observability. The teams that ship reliable AI treat both as non-negotiable infrastructure, the same way web teams treat tests and application performance monitoring.

There is also an economic argument that often gets the project funded when the quality argument does not. LLM calls cost money per token, and a system without measurement tends to overspend in ways nobody notices: a prompt that grew over time and now sends twice the necessary context, a model tier more expensive than the task requires, a retrieval step that pulls ten chunks when three would do. Observability surfaces cost per request and per conversation as a first-class metric, and evaluation lets you prove that a cheaper model or a leaner prompt holds quality before you switch. The same infrastructure that protects quality routinely pays for itself by exposing waste, which is why cost and quality belong on the same dashboard rather than in separate conversations.

What LLM Evaluation Measures

LLM evaluation is the process of scoring model outputs against a definition of quality for your task. The definition of quality is the hard part, because quality is multidimensional. A good answer is correct, but it is also relevant to what was asked, grounded in the provided context rather than invented, appropriately concise, free of policy violations, and delivered within acceptable latency and cost. A single accuracy number hides most of this. Mature evaluation tracks several dimensions at once and reports them separately so you can see which one regressed.

There are four broad approaches to evaluation, and production systems usually combine them. Reference-based evaluation compares the output against a known correct answer using exact match, F1, or semantic similarity, and it works well for tasks with a defined ground truth like classification or extraction. Reference-free evaluation scores the output without a gold answer, typically using a model to judge qualities like helpfulness or faithfulness, and it is the only practical option for open-ended generation where there is no single right answer. Human evaluation puts trained reviewers in the loop and remains the gold standard for nuanced judgments, but it is slow and expensive. Behavioral and assertion-based evaluation checks concrete properties such as whether the output is valid JSON, contains a required citation, or avoids a banned phrase, and it is cheap, deterministic, and catches a surprising fraction of real bugs.

The output of an evaluation is not just a number, it is a decision. You evaluate to decide whether a change is safe to ship, which of two prompts is better, whether a cheaper model is good enough, or whether the system has regressed. That decision frame is what keeps evaluation honest. A metric that does not change a decision is a vanity metric. Before adding any metric, the question to answer is what action you would take if it moved, and if there is no action, the metric is not worth the cost of computing it.

A common mistake is to treat evaluation as a one-time gate before launch rather than a continuous practice. The systems that stay reliable run evaluation on every change, the same way a mature engineering team runs its test suite on every commit. This changes the character of development: prompt engineering stops being a matter of trying something and eyeballing a few outputs, and becomes a measured activity where every edit produces a score you can compare against the previous one. The first time a team wires evaluation into continuous integration, they almost always discover that a change they were confident about actually made the system worse on some slice of inputs, which is precisely the kind of silent regression that ships to users when evaluation is skipped.

What LLM Observability Captures

LLM observability is the practice of instrumenting an AI application so that every request produces a structured record of what happened, and aggregating those records into metrics, traces, and alerts. The core unit is the trace: a single user request expands into a tree of operations, including the system prompt, the retrieved documents, each LLM call with its inputs and outputs and token usage, each tool or function call, and the final response. Capturing the full tree is what makes debugging possible, because the failure is rarely in the final output alone. The answer was wrong because retrieval returned the wrong chunk, or because a tool call failed silently and the model improvised, or because the prompt template dropped a variable.

Observability differs from plain logging the same way it does in conventional systems. Logging records events; observability answers questions you did not anticipate when you wrote the code. A good observability layer lets you ask, after the fact, which prompts produced the most negative feedback this week, what the p95 latency of the retrieval step is, how token cost per conversation changed after the last deploy, and which specific traces triggered a content-policy filter. The OpenTelemetry project has converged on semantic conventions for generative AI spans, and most LLM observability platforms either build on OpenTelemetry or interoperate with it, so the instrumentation you add is increasingly portable across tools.

In production, observability is also where online evaluation lives. You cannot run a full reference-based evaluation on live traffic because you do not have gold answers for real user questions, but you can run cheap reference-free evaluators continuously: a faithfulness check on RAG answers, a toxicity classifier on outputs, a JSON-validity assertion on structured responses. These run on a sample of live traffic, attach scores to the traces, and feed the same dashboards and alerts as your latency and cost metrics. This is the bridge between evaluation and observability, online evaluators turn observability traces into continuous quality signals.

Observability also has practical concerns that distinguish it from a simple log dump. Prompts and outputs frequently contain personal or confidential data, so a real observability layer has to decide what to capture in full, what to redact or hash, and what to drop entirely, in line with privacy and compliance obligations. High-traffic systems cannot afford to store every trace, so sampling becomes necessary, and the smart pattern is to keep every failed or low-scoring trace while sampling the successful ones, so the interesting cases are never lost to cost control. These decisions are easy to defer and expensive to retrofit, which is why they belong in the design of the observability layer from the start rather than being bolted on after the first incident exposes a gap.

The Metrics That Actually Matter

LLM metrics fall into a few families, and choosing the right ones depends entirely on the task. Quality metrics measure whether the output is good: correctness or accuracy against ground truth, faithfulness or groundedness for whether the answer is supported by the provided context, answer relevance for whether it addresses the question, and completeness for whether it covers what was asked. Retrieval metrics apply to any RAG or memory system: context precision (what fraction of retrieved chunks were relevant), context recall (what fraction of the relevant chunks were retrieved), and mean reciprocal rank (how high the first relevant result appears). Operational metrics cover the practical constraints: latency at p50 and p95, cost per request and per conversation, token usage, and error and timeout rates. Safety metrics cover toxicity, PII leakage, jailbreak success rate, and policy-violation rate.

The most important discipline with metrics is segmentation. An aggregate accuracy of 0.85 can hide the fact that accuracy on one important query category dropped to 0.6 while another rose to 0.95. Always segment metrics by query type, user segment, and time window, because local degradation is invisible in global averages and it is local degradation that generates the complaints. The second discipline is correlating quality metrics with operational ones. A change that improves accuracy by two points but doubles latency and cost is usually not worth shipping, and you can only see that tradeoff when the metrics sit side by side.

The third discipline is accounting for variance. LLM outputs are non-deterministic, so a single evaluation run produces a noisy estimate, and a two-point move between runs may be nothing more than sampling noise. Serious evaluation runs multiple samples per input and reports a mean with a confidence interval rather than a single score, so that you can distinguish a real change from run-to-run jitter. Without this, teams chase phantom regressions and miss real ones that hide inside the noise band. The size of a meaningful change should be calibrated against the observed variance of the metric, not guessed, which is the difference between an evaluation that informs decisions and one that generates false alarms.

Evaluation Methods

The method that has reshaped LLM evaluation in the last few years is LLM-as-a-judge: using a strong model to score the outputs of the system under test. It scales to open-ended generation where no reference answer exists, it correlates reasonably well with human judgment when the rubric is clear, and it is dramatically cheaper and faster than human review. It is also fallible. Judges have known biases, including a preference for longer answers, a preference for the first option in a pairwise comparison, and a tendency to favor outputs that resemble their own style. The countermeasures are concrete: use a clear rubric with explicit criteria, ask for a structured verdict with reasoning, randomize position in pairwise comparisons, and periodically validate the judge against a human-labeled set to confirm it still agrees with people.

Human evaluation remains essential for the judgments that models cannot reliably make: subtle tone, domain-specific correctness, and whether an answer is genuinely helpful rather than merely plausible. The practical approach is a pyramid. Cheap deterministic assertions run on every output and catch format and policy bugs. LLM-as-a-judge evaluators run on every evaluation pass and on a sample of production traffic, catching most quality regressions. Human review runs on a small sample and on cases where the automated evaluators disagree or report low confidence, providing the ground truth that keeps the automated layers calibrated. This pyramid is how teams get most of the rigor of human evaluation at a fraction of the cost.

Evaluating RAG and Agents

Retrieval-augmented generation needs evaluation at two layers, because a RAG answer can fail in two distinct ways. The retrieval layer can fail to surface the right context, in which case no amount of generation quality can save the answer, and this is measured with context precision and context recall against a labeled set of relevant documents. The generation layer can fail even with perfect context, by hallucinating beyond what the context supports or by missing relevant information that was retrieved, and this is measured with faithfulness and answer relevance. Evaluating the two layers separately is what lets you diagnose a bad answer: a low context-recall score points at the retriever and embeddings, a low faithfulness score with good retrieval points at the prompt and the generation model.

Agent evaluation is harder still, because an agent produces a trajectory rather than a single output. The agent decides which tools to call, in what order, with what arguments, and when to stop, and a good final answer reached through a wasteful or incorrect path is still a problem in production. Agent evaluation therefore scores both the outcome (did the task get completed correctly) and the trajectory (did the agent select the right tools, pass valid arguments, recover from errors, and avoid unnecessary steps). This trajectory-level evaluation is where observability and evaluation merge most tightly, because the trace is the trajectory, and the same captured tree of tool calls that you debug with is the artifact you score.

The same layered thinking applies to every multi-step LLM system, not just agents. A pipeline that classifies a query, retrieves context, generates an answer, and then post-processes it can fail at any stage, and an end-to-end score cannot tell you which one. The general principle is to instrument and score each stage independently, so that a quality drop localizes to the component responsible rather than leaving you to bisect the whole pipeline by hand. This is the single most important structural decision in evaluating any non-trivial LLM application: evaluate the parts, not just the whole, because the whole only tells you that something is wrong, while the parts tell you what to fix.

From Offline Eval to Production Monitoring

Offline evaluation runs against a fixed dataset, gives you a repeatable score, and gates changes before they ship. Online evaluation runs against live traffic, cannot use gold answers, and catches the problems that only appear with real inputs. Neither replaces the other. The offline set is curated and stable so that a score change means a real change in the system, but it is always incomplete because real users are more creative than any test set. Online evaluation covers the long tail of real inputs but is noisier and limited to reference-free metrics. The mature pattern is to run offline evaluation in continuous integration on every change, run online evaluators continuously on sampled production traffic, and feed the failures discovered online back into the offline dataset so the regression that slipped through once is caught automatically next time.

Regression detection is the payoff of this pipeline. Because model providers update models, because data drifts, and because prompt and code changes have non-obvious effects, the question is not whether the system will regress but when, and whether you will notice. A regression-detection setup compares current metrics against a rolling baseline, alerts when a quality metric drops beyond normal variance, and ties the alert back to the change that caused it through the trace and deploy history. This is the same discipline as performance monitoring in conventional systems, applied to quality instead of latency.

Two regression sources deserve special attention because they evade change-gated evaluation entirely. The first is the silent model update: a provider revises the model behind an endpoint, and behavior shifts overnight with no code change on your side, so the only way to catch it is a scheduled evaluation that runs against production traffic and flags the drop. The second is data drift, where the distribution of incoming questions or the contents of a retrieval index move away from what the system was tuned for, degrading quality gradually with nothing in your codebase to blame. Both are invisible to a test suite that only runs when you change something, which is why continuous production monitoring is not optional for a system that must stay reliable over months rather than days.

Evaluating Memory and Retrieval Quality

Any system that adds persistent memory to an LLM inherits the full evaluation problem at the retrieval layer, plus a new dimension: the quality of what was remembered. A memory system can fail by retrieving the wrong memory, by retrieving a stale or contradicted memory with high confidence, or by failing to retrieve a relevant memory that it holds. These map directly onto context precision, faithfulness, and context recall, which means the same RAG evaluation toolkit applies to memory retrieval. The added dimension is confidence calibration: a memory system that assigns high confidence to information that later proves wrong is worse than one that is appropriately uncertain, because downstream components trust the confidence score.

Adaptive Recall exposes the signals that make this measurable. The status tool reports retrieval quality and confidence distribution, and because each memory carries a confidence score that rises with independent corroboration and falls under contradiction, you can evaluate whether confidence actually predicts correctness over time. Treating a memory layer as a measurable component rather than a black box is the difference between a system that quietly degrades and one whose retrieval quality you can put on a dashboard next to your other LLM metrics. The guides below cover how to build these evaluations and connect them to production observability.

Core Concepts

Foundations

Implementation Guides

Building Evaluations

Production Monitoring