How to Evaluate a Fine-Tuned LLM Before Production

Build a Held-Out Test Set

The test set must be created before training begins and must never be used during any part of the training process, including hyperparameter tuning. Reserve 10% to 20% of your curated dataset as the test set, using random sampling to ensure the distribution matches the training data. If your data contains distinct categories or difficulty levels, use stratified sampling to maintain the same proportions in both splits.

The test set should include three types of examples. First, representative examples that match the typical distribution of production inputs. These tell you how the model performs on the bread-and-butter cases. Second, edge cases that test the boundaries of what the model should handle: unusual inputs, rare categories, long inputs, very short inputs, and inputs that require the model to refuse or acknowledge uncertainty. Third, adversarial examples designed to expose failure modes: inputs that are similar to training examples but require different outputs, inputs that test for common biases, and inputs that probe whether the model memorized training examples rather than learning generalizable patterns.

A test set of 100 to 500 examples is typically sufficient for a meaningful evaluation. Fewer than 50 examples produces results with high variance that make it hard to distinguish real improvement from noise. More than 500 is rarely necessary unless your task has a very large number of distinct categories and you need statistical significance per category.

Measure Task-Specific Quality

Run the fine-tuned model on every example in the test set and compute metrics appropriate to your task. The right metrics depend on what you are measuring:

Classification tasks: Accuracy, precision, recall, and F1 score. If classes are imbalanced, weighted or macro-averaged F1 is more informative than raw accuracy. Report a confusion matrix to identify which categories the model confuses.

Generation tasks: BLEU and ROUGE compare the model's output to reference outputs at the n-gram level, which works for tasks where the correct output is relatively constrained (summarization, translation). For open-ended generation where multiple outputs are valid, these metrics are less useful and human evaluation becomes primary.

Structured output tasks: Format compliance rate (percentage of outputs that parse as valid JSON, XML, or whatever structure you require), field accuracy (percentage of individual fields that are correct), and schema adherence (whether all required fields are present).

Code generation: Pass@1 (percentage of generated code that passes the test suite on the first attempt), and if your tests have varying difficulty, pass rate broken down by difficulty level.

Compare every metric to two baselines: the base model with the same prompt (no fine-tuning) and, if applicable, any previous version of the fine-tuned model. The improvement should be statistically significant, not just a few percentage points that could be noise. For small test sets, use bootstrap confidence intervals to assess whether the observed difference is real.

Check General Capability Retention

Fine-tuning can degrade the model's general capabilities if the training overwrites important pretrained knowledge, a phenomenon called catastrophic forgetting. To detect this, run the fine-tuned model on a standard general-purpose benchmark and compare the scores to the base model.

The most commonly used benchmarks are MMLU (Massive Multitask Language Understanding, a multiple-choice test spanning 57 academic subjects), HumanEval (a code generation benchmark with 164 programming problems), and HellaSwag (a commonsense reasoning benchmark). For instruction-tuned models, MT-Bench (a multi-turn conversation benchmark) tests conversational quality. You do not need to run every benchmark; pick 1 to 2 that cover capabilities important for your application.

A drop of 1 to 2 percentage points on general benchmarks is normal and acceptable for most applications. A drop of 5 or more percentage points suggests excessive forgetting and indicates a problem with your training process. The most common causes are: training for too many epochs, using a learning rate that is too high, using full fine-tuning when LoRA would suffice, or training on data that is too narrow and repetitive. Addressing the cause and retraining is better than deploying a model with degraded general capability.

Run Human Evaluation

Automated metrics measure specific, quantifiable properties of model outputs but miss the subjective qualities that determine whether users will trust and use the system. Human evaluation catches what metrics miss: tone, helpfulness, readability, confidence calibration, and overall usefulness.

The standard protocol is a blind side-by-side comparison. Present evaluators with a prompt and two responses, one from the fine-tuned model and one from the base model, without indicating which is which. Ask the evaluator to select the better response or indicate a tie. Run this comparison on at least 50 prompts, ideally 100 or more, with 2 or more evaluators per prompt. Compute the win rate (percentage of prompts where the fine-tuned model was preferred) and inter-annotator agreement (Cohen's kappa or similar).

For domain-specific applications, the evaluators should be domain experts, not general annotators. A legal fine-tuned model needs lawyers to evaluate it, a medical model needs clinicians, and a code generation model needs experienced developers. General annotators cannot reliably assess domain-specific correctness and will miss errors that matter in production.

If the fine-tuned model does not win at least 55% of side-by-side comparisons (with the remainder being ties or base model wins), the fine-tuning did not add sufficient value to justify the added complexity. Go back to data quality and training configuration before trying again. The LLM evaluation pillar covers evaluation methodology in broader depth.

Test Edge Cases and Safety

Production inputs are messier than test sets. Before deployment, probe the model with inputs designed to expose failure modes. Include: out-of-distribution inputs (topics the model was not trained on, to verify it handles them gracefully rather than hallucinating domain-specific answers), adversarial prompts (prompt injection attempts, requests for harmful content, attempts to make the model contradict its training), malformed inputs (truncated text, wrong language, garbage characters), and boundary conditions (very long inputs near the context limit, empty inputs, single-word inputs).

The model should fail gracefully on these inputs: acknowledging uncertainty rather than hallucinating, refusing harmful requests, and producing sensible output even on unusual inputs. If it produces confident nonsense on out-of-distribution inputs, that is a sign of overfitting to the training distribution. If it is easily jailbroken or manipulated, the DPO alignment step may need to be applied or strengthened.

Document the results of edge case testing alongside the standard evaluation metrics. This documentation serves as the model card that accompanies the deployed model and informs the team about known limitations and failure modes.