How to Add LLM Evaluation to Your CI/CD Pipeline
The concept is simple: treat evaluation like tests. The implementation has challenges that traditional tests do not, specifically non-determinism, API costs, and latency. A unit test runs in milliseconds and always returns the same result. An LLM evaluation run calls an external model API for every example and every judge metric, takes minutes, costs money, and can produce slightly different results on each run. These differences require practical accommodations, but they do not change the fundamental architecture: a CI step that runs evaluation and gates the merge based on the results.
Step 1: Set Up the Evaluation Dataset and Metrics
Before writing any CI configuration, you need two things: a versioned dataset and a defined set of metrics. The dataset should contain 100 to 500 examples that represent the core use cases of your application, including happy-path inputs, edge cases, and examples of past failures that you never want to regress on. Each example should have an input (what the user sends), optionally a reference answer or expected properties, and any context the system needs (for RAG applications, the documents to retrieve against).
Version the dataset alongside your code, either in the same repository or in a dedicated dataset repository that the CI pipeline pulls from. Versioning matters because the evaluation score is meaningful only relative to the dataset it was computed on. If the dataset changes between runs, a score change could reflect the dataset change rather than a code change, which breaks the comparison. Tools like DVC (Data Version Control) handle dataset versioning well, but a simple directory of JSON files in the repository also works for datasets under 500 examples.
Choose 3 to 5 metrics that capture the quality dimensions you care about most. For a RAG application, faithfulness, answer relevance, and context precision are a good starting set. For a coding assistant, correctness (pass@1 against test cases) and code quality are essential. For a chatbot, response relevance, helpfulness, and safety. Each metric needs a clear implementation, either from a framework like DeepEval or Ragas, or a custom LLM-as-a-judge prompt. Avoid adding metrics speculatively. Every metric in the CI check adds cost and latency to the pipeline, and a metric that never changes a decision is pure overhead.
Step 2: Write the Evaluation Script
The evaluation script is a standalone program that loads the dataset, runs each example through your application, scores the outputs with your metrics, and produces a structured report. The report should include the overall score per metric, the score per metric compared to the baseline (the last evaluation run on the main branch), per-example results for any examples that regressed, and the total cost and duration of the run.
If you use DeepEval, the script is a pytest test file where each test case is a dataset example. DeepEval's test runner handles metric computation, baseline comparison, and reporting natively. If you use Ragas, write a Python script that calls the evaluate() function on the dataset and compares the results to a saved baseline. If you use a custom setup, structure the script as: load dataset, run examples, compute metrics, load baseline, compute deltas, output report.
Handle non-determinism by running each example multiple times (3 to 5 runs is practical) and averaging the results. This reduces the variance of the score and makes it possible to distinguish real regressions from run-to-run noise. The cost of multiple runs is the primary tradeoff: 3 runs on a 200-example dataset with 4 metrics roughly triples the API cost compared to a single run. For teams with tight budgets, running once but using a wider threshold for declaring a regression (allowing for noise) is the practical compromise.
Store the baseline results alongside the dataset. After every successful merge to main, the CI pipeline should update the baseline with the results of the current evaluation, so the next PR compares against the latest known-good scores. This rolling baseline means the team is always comparing against the most recent shipped version rather than an arbitrary historical snapshot.
Step 3: Add the Eval Step to CI
In GitHub Actions, the evaluation runs as a workflow step triggered on pull requests that touch relevant files (prompt templates, model configuration, retrieval logic, or evaluation datasets). Gate the check so it only runs when the change could affect LLM output quality, not on documentation or infrastructure changes, to avoid unnecessary cost and latency.
The CI environment needs access to the LLM API keys for both the application model and the judge model. Store these as encrypted secrets in your CI platform (GitHub Actions secrets, GitLab CI variables, or your platform's equivalent). Use a separate API key with spend limits dedicated to evaluation, so a misconfigured evaluation loop cannot run up an unlimited bill. Set a per-run budget cap in the evaluation script as an additional safety net.
Post the evaluation results as a comment on the pull request, showing the metric scores with deltas from baseline, any regressed examples with their inputs and outputs, and the overall pass/fail verdict. This makes the evaluation results visible to reviewers without requiring them to open a separate dashboard. GitHub Actions supports posting PR comments via the GitHub API, and most CI platforms have similar capabilities. For teams using LangSmith or Braintrust, the evaluation run URL can be included in the comment so reviewers can drill into individual examples.
Set the CI check as required for merge on branches that contain prompt or model changes, so a regressing change cannot be merged without an explicit override. This is the enforcement mechanism that turns evaluation from optional to mandatory.
Step 4: Define Pass and Fail Criteria
The pass/fail criteria determine whether the CI check blocks the merge. Setting the thresholds correctly is critical because thresholds that are too tight produce false failures that annoy the team and erode trust in the check, while thresholds that are too loose let real regressions through.
The practical approach is to define two kinds of thresholds. Absolute thresholds set a floor below which the metric should never fall regardless of the baseline: faithfulness above 0.70, safety pass rate at 1.0, error rate below 0.02. These catch catastrophic regressions. Relative thresholds compare the current run to the baseline and flag regressions beyond a tolerance: faithfulness must not drop more than 0.05 from baseline, relevance must not drop more than 0.03. The relative thresholds account for the fact that different prompts have different baseline scores, and a drop of 0.05 from 0.95 is different in character from a drop of 0.05 from 0.75.
Account for variance by making the relative threshold wider than the observed run-to-run variance. If your evaluation scores vary by plus or minus 0.02 between identical runs (due to LLM non-determinism), set the regression threshold at 0.04 or 0.05 to avoid triggering on noise. Measure the variance by running the evaluation five times with no code changes and computing the standard deviation. This calibration step takes one afternoon and prevents months of false-alarm frustration.
When a check fails, the developer should be able to see exactly which examples regressed, what the model output was, and what the expected behavior is, so they can diagnose whether the regression is real or a false alarm. Include an escape hatch for false alarms: a review process where the developer explains why the regression is acceptable and a senior team member approves the override. This prevents the check from becoming a barrier without accountability.
Step 5: Maintain and Evolve the Dataset
The evaluation dataset is a living artifact that needs ongoing maintenance. Add new examples when production monitoring surfaces failures that the dataset did not cover. Remove or update examples when the product changes and old examples no longer represent valid use cases. Periodically review the dataset for staleness, bias, and completeness, ideally on a quarterly cadence.
The most valuable source of new examples is production failures. When the production monitoring system catches a low-quality response, or when a user reports a bad answer, add the input to the evaluation dataset with the correct expected behavior. This creates a feedback loop where every production incident makes the evaluation suite stronger, ensuring that the same failure cannot recur without being caught in CI. Over months, this feedback loop produces an evaluation dataset that is deeply representative of real user behavior and real failure modes.
Watch for dataset growth becoming a cost problem. A dataset that grows from 200 to 2,000 examples over a year increases CI costs by 10x. One approach is to maintain a core set of 200 to 300 critical examples that run on every PR, plus an extended set of 1,000+ examples that runs nightly or on release branches. The core set catches most regressions quickly, and the extended set provides deeper coverage on a less frequent schedule. This tiered approach keeps CI fast while maintaining thorough coverage where it matters.
Wire LLM evaluation into CI the same way you wire tests: a required check on every PR that touches prompts or model configuration, with a versioned dataset, defined metrics, calibrated thresholds that account for LLM variance, and a feedback loop from production failures back into the dataset. This turns prompt engineering from guesswork into a measured, iterative process where regressions are caught before they ship.