Model Distillation vs Fine-Tuning: What Is the Difference

How Distillation Works

Classical knowledge distillation, introduced by Hinton et al. in 2015, trains a student model to match the teacher model's output probability distribution rather than just the final answer. When a teacher model classifies an input as "cat" with 90% confidence and "dog" with 8% confidence, the standard fine-tuning label would be just "cat." Distillation preserves the full probability distribution, teaching the student that "dog" is a plausible second choice. This "soft label" information captures the teacher's understanding of relationships between classes, which gives the student a richer learning signal than hard labels alone.

For language models, distillation extends this idea to next-token prediction. The teacher model processes an input and produces a probability distribution over all possible next tokens. The student model is trained to match this distribution, not just to predict the single most likely token. By matching the full distribution across thousands of training examples, the student absorbs the teacher's nuanced understanding of language, including its uncertainty about ambiguous inputs and its relative preferences among multiple valid continuations.

In practice, LLM distillation in 2026 most commonly takes a simpler form: the teacher model generates high-quality outputs for a large set of inputs, and the student model is fine-tuned on those teacher-generated outputs using standard supervised fine-tuning. This is sometimes called "black-box distillation" because it does not require access to the teacher's internal probabilities, only its generated text. This approach is what powers most practical distillation workflows because access to the teacher's logits requires running the teacher model yourself, which is feasible for open models but impossible for proprietary APIs like GPT-4 or Claude.

The Alpaca, Vicuna, and Orca models were all created through this black-box distillation process: GPT-4 or GPT-3.5 generated training data, and smaller models were fine-tuned on that data. The distinction between "distillation" and "fine-tuning on synthetic data" blurs when the synthetic data comes from a larger model, and many practitioners use the terms interchangeably in this context.

How Fine-Tuning Differs

Standard fine-tuning uses human-labeled data rather than teacher-generated data. The training signal comes from ground-truth annotations: a human expert determined that this input maps to this output, and the model learns to replicate those mappings. The model does not learn from another model's behavior; it learns from human judgments about what the correct output is.

This distinction matters for quality in two ways. First, human-labeled data can be more accurate than teacher-generated data for tasks where the teacher model itself makes errors. If GPT-4 achieves 85% accuracy on your task, distilling from GPT-4 gives the student an accuracy ceiling of 85%. Fine-tuning on human-labeled data has no such ceiling; the model can potentially achieve 99%+ accuracy if the labels are perfect and the task is within the model's capacity.

Second, human-labeled data captures domain expertise that no language model possesses. For specialized tasks like medical diagnosis, legal analysis, or engineering assessment, the correct answer depends on expert knowledge that the teacher model may not have. Fine-tuning on expert labels transfers that expert knowledge directly, while distillation can only transfer what the teacher model already knows.

The trade-off is scale and cost. Human labeling is expensive ($5 to $150 per hour depending on expertise level) and slow (20 to 100 examples per hour). Teacher-generated data costs $0.01 to $0.05 per example and can be produced at thousands of examples per hour. For most teams, the practical path is a combination: use distillation (synthetic data from a teacher model) for the bulk of the training data, and supplement with human-labeled data for critical edge cases and quality anchoring.

When to Use Distillation

Distillation is the right approach in four specific situations.

You are building a smaller, faster version of a capability that a large model already has. If GPT-4 or Claude already performs your task well but is too expensive or too slow for production, distillation transfers that capability to a 7B or 13B model that costs 20 to 50 times less per request. This is the most common distillation use case and the reason it has become a standard production technique. The small model fine-tuning guide covers the economics in detail.

You lack labeled training data but have access to a strong teacher model. If you cannot afford or find human annotators for your task, the teacher model acts as a substitute annotator. The quality is lower than human labels on average, but the volume compensates: 5,000 teacher-generated examples often outperform 500 human-labeled examples for training purposes. The synthetic training data guide covers the generation and filtering pipeline.

You want to capture the teacher's reasoning process, not just its answers. By generating "chain-of-thought" outputs from the teacher (asking it to show its reasoning before giving the answer), you can distill the reasoning process itself into the student model. This produces a student that not only gives the right answer but arrives at it through a reasoning process similar to the teacher's. The Orca papers demonstrated that this approach, called "explanation tuning," produces stronger students than distilling from answers alone.

You need to transfer general capability rather than task-specific behavior. If your goal is a model that is broadly good at a domain (general-purpose medical reasoning, general-purpose legal analysis) rather than one specific task, distillation from a strong teacher across a diverse set of domain inputs produces a more broadly capable student than fine-tuning on narrow task-specific data.

When to Use Standard Fine-Tuning

Standard fine-tuning on human-labeled data is the right approach in three situations.

Your task requires accuracy that exceeds what any teacher model can provide. If the best available teacher model achieves 85% accuracy on your task and you need 95%+, distillation from that teacher caps your performance at 85%. Human-labeled data, assuming the labels are correct, provides an accuracy target that is independent of any model's limitations. For safety-critical tasks (medical, financial, legal), where errors have real consequences, human-labeled data is often a requirement rather than an option.

Your task involves domain expertise that language models lack. If the correct output depends on specialized knowledge, proprietary information, or expert judgment that no public language model has been trained on, teacher-generated outputs will be wrong or superficial. A radiologist labeling medical images, a patent attorney classifying prior art, or a structural engineer assessing building specifications brings knowledge that cannot be distilled from a general-purpose LLM.

You have enough labeled data and the budget to produce it. If you already have 5,000+ labeled examples from production logs, customer interactions, or historical decisions, there is no reason to generate synthetic data from a teacher model. Real data from your actual use case is always more representative than synthetic data, and the model trained on it will generalize better to production inputs.

Combining Both Approaches

The strongest results come from combining distillation and fine-tuning in a staged pipeline. The specific workflow depends on your data availability and quality requirements.

Stage 1: Distillation for volume. Generate 5,000 to 50,000 training examples from a strong teacher model. Filter for quality using the pipeline described in the synthetic data guide. Fine-tune the student model on this data. This gives the student a strong baseline capability across the full range of inputs.

Stage 2: Fine-tuning for precision. Take 500 to 2,000 human-labeled examples, focusing on edge cases, ambiguous inputs, and high-stakes scenarios where accuracy matters most. Fine-tune the stage-1 model on this human-labeled data, either by continuing training or by doing a second LoRA fine-tuning pass. This shifts the model from the teacher's general capability toward the specific quality standard your application requires.

Stage 3: Preference alignment (optional). Use DPO or RLHF with human preference data to polish the model's output quality on subjective dimensions like helpfulness, tone, and detail level. This stage is optional but can meaningfully improve user satisfaction.

This three-stage pipeline is more effective than either approach alone because distillation provides the broad coverage that would be too expensive to get from human labeling alone, while the human-labeled fine-tuning provides the precision and domain expertise that the teacher model cannot.

White-Box vs Black-Box Distillation

Black-box distillation uses only the teacher's generated text as training data. It works with any teacher model, including proprietary APIs, because you only need the outputs, not the internal model state. This is the practical default for most teams because the strongest teacher models (GPT-4, Claude) are available only through APIs. The quality is good but not optimal because the student only learns from the final output, not from the teacher's internal reasoning process.

White-box distillation uses the teacher's internal probability distributions (logits) as the training target. The student learns not just what the teacher outputs but how confident it is about each possible token, which provides a richer learning signal. White-box distillation requires running the teacher model yourself to extract logits, which means it only works with open-weight models (Llama, Mistral, Qwen). The quality improvement over black-box distillation is typically 2 to 5% on benchmarks, which matters for competitive ML research but may not justify the infrastructure complexity for production applications.

A middle ground is chain-of-thought distillation, which is black-box (uses only generated text) but captures more of the teacher's reasoning process by asking the teacher to show its work. This provides some of the richness of white-box distillation without requiring access to the teacher's logits, making it the practical recommendation for most teams.