How to A/B Test LLM Prompts in Production
Offline evaluation tells you whether a prompt change is better on a curated dataset. A/B testing tells you whether it is better in the real world, where the inputs are messier, the context is richer, and the definition of "better" includes user behavior, not just automated scores. A prompt that scores higher on faithfulness in offline eval may still produce longer, more formal answers that users skip, resulting in lower engagement. A prompt that scores slightly lower on a relevance metric may include a conversational tone that keeps users interacting longer. Only live experimentation resolves these tradeoffs, because the test set cannot capture everything that matters.
Step 1: Define the Hypothesis and Primary Metric
Every A/B test starts with a hypothesis: "Changing the system prompt to include explicit formatting instructions will increase the rate at which users copy the generated output." The hypothesis has two parts: the change and the expected effect. Without both, you are exploring rather than testing, and exploration produces interesting data but not actionable decisions.
Choose a single primary metric that captures the effect you care about. This metric should be as close to real user value as possible. Thumbs-up rate, task completion rate, response copy rate, follow-up question rate (lower is better if the first answer should be sufficient), and session engagement duration are all reasonable choices depending on your application. Avoid using LLM evaluation scores as the primary metric for an A/B test, because the whole point of the test is to validate whether the offline score improvement translates to real user impact. LLM scores belong as secondary metrics for diagnosis, not as the decision metric.
Also define guardrail metrics that must not regress: latency, cost per request, safety classifier pass rate, and error rate. A prompt change that improves engagement but doubles latency or triples cost is not a winner, and guardrail metrics make this visible without requiring a separate analysis.
Step 2: Set Up Traffic Splitting
Assign users to the control (current prompt) or variant (new prompt) group, and keep each user in the same group for the duration of the test. User-level assignment means a user who gets the variant on their first request continues to get it on all subsequent requests, which prevents confusing experiences where the system's behavior changes mid-session. If your application does not have user identity, assign at the session or device level.
Start with a small percentage in the variant group, typically 5% to 20%, depending on your traffic volume and risk tolerance. High-risk changes, such as a major prompt rewrite or a model swap, deserve a smaller initial allocation with close monitoring before ramping up. Low-risk changes, like formatting adjustments or minor wording tweaks, can start at higher percentages. Most experimentation platforms (LaunchDarkly, Statsig, Eppo, GrowthBook, Braintrust) support percentage-based rollouts with user-sticky assignment.
For LLM-specific experiments, you can also split at a finer grain than user level. Some teams split at the request level (each request is independently assigned) when the interactions are stateless and there is no consistency concern. This reduces the sample size needed to detect a difference, but it sacrifices within-session consistency, so it works better for one-shot tools like code generators than for multi-turn chatbots.
Step 3: Instrument Both Variants
Every request in both the control and variant groups needs to log the prompt version identifier, the full prompt sent to the model, the model name and configuration (temperature, max tokens), the user input, the model output, all quality metrics (LLM-as-a-judge scores, assertion results), operational metrics (latency, token count, cost), and any user feedback signals (thumbs up/down, copy events, follow-up actions). These logs are what make the test analyzable after it ends.
Use your observability layer to capture this data rather than building a separate logging system. The trace should include the variant assignment as a tag or attribute, so you can filter and segment production traces by experiment arm. LangSmith, Braintrust, and Phoenix all support tagging traces with experiment metadata. If you built your own tracing, add a variant_id field to every trace span and make sure it propagates through multi-step pipelines.
Critically, instrument the control group with the same thoroughness as the variant. The value of an A/B test comes from the comparison, and a comparison is only valid if both arms are measured identically. A common mistake is to add detailed logging to the variant while relying on existing, sparser logging for the control, which introduces measurement asymmetry that biases the analysis.
Step 4: Run Until Statistical Significance
The test needs to run long enough to collect a sample size that distinguishes a real difference from random noise. The required sample size depends on the baseline rate of your primary metric, the minimum detectable effect (the smallest improvement worth caring about), and the significance level and power you want (the convention is 95% confidence and 80% power). For most LLM applications, this means thousands to tens of thousands of requests per arm, which at moderate traffic volumes means running the test for days to weeks.
Do not check the results daily and stop the moment you see a significant p-value. This practice, called peeking, inflates the false positive rate because you are running multiple comparisons. Either pre-commit to a fixed sample size and analyze only after reaching it, or use a sequential testing method (like the always-valid p-values in Statsig or the Bayesian approach in Eppo) that is designed for continuous monitoring. Sequential methods let you stop early if the effect is large, while maintaining the statistical guarantees that prevent false conclusions.
LLM outputs add a complication that traditional A/B testing does not face: non-determinism. The same prompt on the same input can produce different outputs across runs. This means within-variant variance is higher than in deterministic systems, which increases the sample size needed to detect a given effect. Reducing variance by lowering temperature helps with this, but most production systems want some temperature for output diversity, so the practical solution is simply accepting larger sample sizes and longer test durations.
Step 5: Analyze and Ship or Revert
When the test reaches the required sample size, compare the primary metric between control and variant. If the variant wins on the primary metric and holds on all guardrail metrics, promote it to 100% traffic. If it loses, revert to the control. If the primary metric is flat but a secondary metric improved meaningfully, decide based on whether the secondary improvement justifies the operational cost of the change.
Always segment the results before declaring a winner. A prompt that improves average performance can still regress on an important subset. Check performance by query category, user segment, input length, and language. A 5% overall improvement that comes from a 20% gain on easy queries and a 10% loss on hard queries may not be the right tradeoff, depending on how much your hard queries matter. Segmented analysis also reveals opportunities: if the variant is dramatically better on one category, you might ship it only for that category and keep the control for the rest.
After shipping a winner, add the winning prompt version and its metrics to your evaluation dataset as the new baseline. The next A/B test will compare against this version, creating a ratchet where each test either confirms the current prompt is best or replaces it with something better. Over months, this ratchet produces prompt quality that no amount of one-time optimization can match, because it is driven by real user behavior rather than offline estimates.
A/B testing bridges the gap between offline evaluation and real user impact. Define a primary metric tied to user value, split traffic with sticky user assignment, instrument both arms identically, run to statistical significance without peeking, and segment results before shipping. The ratchet of continuous testing produces compounding prompt quality over time.