How to Prepare a Training Dataset for LLM Fine-Tuning
The temptation with training data is to prioritize volume, collecting as many examples as possible on the assumption that more data means a better model. This is wrong for fine-tuning. Unlike pretraining, where the model benefits from seeing diverse language patterns across billions of tokens, fine-tuning needs to teach a specific behavior, and every example that deviates from the target behavior teaches the model something you do not want. The discipline is to be ruthless about quality and to treat data curation as the most important step in the fine-tuning pipeline.
Define the Task and Quality Bar
Before collecting a single example, write down exactly what the model should do and what a correct output looks like. This is your task specification, and it serves the same purpose as a product requirements document: it removes ambiguity and gives everyone involved in data creation a shared standard. The specification should include the expected input format (what the model will receive), the expected output format (structure, length, tone), the types of inputs it must handle (the distribution of topics, difficulty levels, and edge cases), and explicit examples of correct outputs for at least 10 representative inputs.
The quality bar is equally important. Define a rubric that every training example must pass. A practical rubric has 4 to 6 criteria, each binary: the output is factually correct (yes/no), the output follows the required format (yes/no), the output is complete (yes/no), the output uses appropriate domain terminology (yes/no). Any example that fails any criterion gets fixed or discarded. Do not train on examples you would not want the model to produce in production, because it will learn to produce exactly those outputs.
Collect Raw Examples
The best training data comes from real production interactions where a human provided the correct response, because these examples represent the actual distribution of inputs the model will see. If you have logs of customer support conversations, code review feedback, document analyses, or any other task where experts produced correct outputs, these are your first data source. Extract input-output pairs and filter for cases where the expert's response meets your quality bar.
If production data is not available or not sufficient, you have two other sources. The first is expert creation: hire domain experts to write correct outputs for a representative set of inputs. This is expensive but produces the highest-quality data. The cost for a human-written dataset of 2,000 examples typically ranges from $2,000 to $10,000 depending on the domain complexity and the expertise required. The second is synthetic generation: use a stronger model (GPT-4, Claude) to generate candidate outputs, then have humans review and filter them. Synthetic data is cheaper to produce but must be audited carefully, as the generating model's biases and errors will transfer to your fine-tuned model if not caught.
How many examples do you need? For a narrow task with clear patterns (classification, format conversion, structured extraction), 200 to 500 high-quality examples are often sufficient. For broader tasks with diverse inputs (open-ended Q&A, creative writing, complex reasoning), 1,000 to 5,000 examples produce noticeably better results. Going beyond 10,000 examples shows diminishing returns for most tasks unless the input distribution is very broad. The right number is the one where adding more examples stops improving validation performance.
Clean and Deduplicate
Raw data almost always contains problems that will degrade model quality if not addressed. The cleaning process should check for and fix: duplicate or near-duplicate examples (which cause the model to overweight those patterns), formatting inconsistencies (which teach the model to produce inconsistent output), factual errors (which the model will learn to reproduce confidently), contradictory examples (where similar inputs have conflicting outputs, which teaches the model to be uncertain), and examples that fall outside the scope of your task specification.
For deduplication, exact duplicates are easy to find with hashing, but near-duplicates require similarity comparison. Two examples with the same input but slightly different wording in the output are near-duplicates and should be resolved to a single canonical version. Two examples that are semantically identical but phrased differently in both input and output should be kept if they represent valid variation, or merged if they are just noise. A practical approach is to embed all examples with a sentence embedding model and flag any pair with cosine similarity above 0.95 for manual review.
After cleaning, re-verify every remaining example against your quality rubric. This second pass catches problems that the initial collection missed and ensures the dataset actually meets the standard you defined. If more than 10% of examples fail the rubric after cleaning, the collection process needs to be improved before proceeding.
Format for Training
The training framework expects data in a specific format, and getting this wrong produces silent failures where the model trains on garbage and appears to learn nothing. For instruction tuning (the most common form of fine-tuning in 2026), the standard format is a list of chat messages with role and content fields, matching the model's chat template. Most models expect messages with roles "system" (optional), "user", and "assistant", where the assistant's message is the target output the model learns to generate.
The tokenizer's apply_chat_template method converts your structured messages into the model's native token format, adding special tokens (beginning-of-turn, end-of-turn, role markers) that vary by model family. Do not hardcode these tokens manually, as getting them wrong causes the model to learn incorrect turn boundaries and produces broken outputs at inference time. Let the tokenizer handle formatting and verify by decoding a few examples back to text to confirm they look correct.
For supervised fine-tuning on non-conversational tasks (classification, extraction, format conversion), the simplest format is a text column with the full input-output sequence separated by the model's special tokens. The training framework needs to know which portion of the sequence is the target (the assistant's response) so it only computes loss on that portion. In TRL's SFTTrainer, this is handled automatically when you provide data in chat format.
Store your formatted dataset as JSONL (one JSON object per line) or as a Hugging Face Dataset. Both are standard inputs for the training frameworks. Include a unique identifier for each example so you can trace training results back to specific data points during evaluation.
Split and Validate
Reserve 10% to 20% of your data for validation before any training begins. The validation set must be created by random sampling from the full dataset, not by taking the last N examples (which may have a different distribution if data was collected over time). The validation set should never be used during training. It serves exclusively to detect overfitting and measure whether the model is generalizing beyond the training examples.
Verify that the split preserves the distribution of your data. If your dataset contains examples from 5 different categories, the validation set should contain roughly the same proportion of each category as the training set. If you have rare edge cases that are important to test, consider a stratified split that ensures at least a few instances of each rare case appear in the validation set.
Before training, run the base model (without fine-tuning) on your validation set and record its performance. This baseline tells you what the model can already do with just prompting, and any fine-tuning that does not beat this baseline is not worth deploying. If the base model with good prompting already achieves 95% of your target quality, the marginal gain from fine-tuning may not justify the effort, and you should reconsider whether fine-tuning is the right approach for your task.
Data quality determines model quality. Define a clear task specification and quality rubric before collecting data. Aim for 500 to 5,000 high-quality examples depending on task complexity. Clean ruthlessly for duplicates, inconsistencies, and errors. Format using the model's chat template, not hardcoded tokens. Split before training and establish a baseline so you can measure whether fine-tuning actually helped.