Key Hyperparameters for LLM Fine-Tuning

Learning Rate

The learning rate is the single most important hyperparameter. It controls the size of each weight update during training: too high and the model overshoots the optimal weights, destabilizing its pretrained capabilities; too low and the model barely changes, wasting compute without learning the task.

For LoRA fine-tuning, the recommended starting learning rate is 1e-4 to 2e-4. This is higher than the 1e-5 to 5e-5 typical of full fine-tuning because the LoRA adapter parameters are initialized to produce zero output (the B matrix starts at zero), so they need larger updates to begin contributing. The 2e-4 default in most LoRA tutorials works for the majority of tasks, and it is the right place to start unless you have a specific reason to change it.

For full fine-tuning, the recommended starting learning rate is 1e-5 to 5e-5. The weights start at their pretrained values rather than at zero, so large updates risk destroying the pretrained representations. The exact value depends on model size: larger models typically need lower learning rates because each weight update has a larger cumulative effect. A 7B model can tolerate 5e-5, while a 70B model is safer at 1e-5 to 2e-5.

For continued pre-training, the learning rate should be 1e-5 to 5e-5, similar to full fine-tuning, because you are modifying pretrained weights and need to avoid catastrophic forgetting. The lower end (1e-5) is safer for domains that are far from the pretraining distribution, where the weight changes need to be gradual.

Diagnosing learning rate problems: If the training loss spikes or oscillates wildly, the learning rate is too high. If the training loss decreases very slowly and has not plateaued after one full epoch, the learning rate may be too low. If the training loss decreases smoothly and validation loss follows it down, the learning rate is in the right range. The most reliable approach is to try 3 values (1e-4, 2e-4, 5e-5) on a small subset of your data and pick the one with the best validation loss curve.

Learning Rate Schedule

The learning rate does not stay constant during training. A schedule adjusts it over the course of training, typically starting low (warmup), reaching a peak, and then decreasing (decay). The schedule matters because the optimal learning rate changes as training progresses: early in training, the model needs to make large adjustments to start learning the task; late in training, it needs to make small adjustments to fine-tune the details without destabilizing what it has already learned.

Warmup ramps the learning rate from near-zero to the peak value over the first few percent of training steps. For LoRA fine-tuning, warmup over 3 to 10% of total steps is standard. Warmup prevents the model from being destabilized by large gradient updates at the very start of training, when the gradients may be noisy because the model has not yet adjusted to the training data distribution. For most fine-tuning jobs, a warmup of 100 steps or 5% of total steps (whichever is larger) works well.

Cosine decay is the most common schedule after warmup. The learning rate follows a cosine curve from the peak value down to near-zero by the end of training. This produces a smooth transition from exploration (large updates early) to refinement (small updates late). The alternative is linear decay, which decreases the learning rate at a constant rate. Cosine decay is slightly better in practice because it keeps the learning rate higher for longer in the middle of training, giving the model more time to learn complex patterns before the rate drops.

Constant schedule (no decay after warmup) is simpler and works reasonably well for short training runs (1 to 2 epochs). For longer runs (3+ epochs), decay helps the model converge to a better final state. If you are not sure which schedule to use, cosine decay with 5% warmup is the safe default.

Batch Size and Gradient Accumulation

The batch size determines how many training examples the model sees before each weight update. Larger batches produce more stable gradient estimates (the average over more examples is less noisy), while smaller batches provide more frequent updates and can be more effective for learning with limited data.

For LLM fine-tuning, the effective batch size (batch size per GPU multiplied by gradient accumulation steps multiplied by number of GPUs) should typically be 16 to 64. An effective batch size of 32 is a common starting point. Larger batch sizes (128+) can cause the model to converge to sharp minima that generalize poorly, while very small batch sizes (1 to 4) produce noisy gradients that make training unstable.

Because LLM fine-tuning often runs on a single GPU with limited memory, the per-GPU batch size may be only 1 to 4 (especially for 7B+ models on consumer GPUs). Gradient accumulation compensates by accumulating gradients over multiple forward passes before performing a weight update. If your per-GPU batch size is 2 and you set gradient accumulation steps to 16, the effective batch size is 2 x 16 = 32, which gives the same training dynamics as a true batch size of 32 while fitting in GPU memory.

The practical approach: set the per-GPU batch size to the largest value that fits in your GPU memory without running out of VRAM (use trial and error, starting from 4 and reducing if you get OOM errors). Then set gradient accumulation steps to reach an effective batch size of 32. If your per-GPU batch size is 2, set gradient accumulation to 16. If it is 4, set gradient accumulation to 8.

Number of Epochs

An epoch is one complete pass through the training data. The number of epochs determines how many times the model sees each training example, and getting this right is the primary control for overfitting.

For most fine-tuning tasks, 2 to 4 epochs is optimal. With 3,000 to 5,000 training examples, 3 epochs typically produces the best validation metrics. The model sees each example 3 times, which is enough to learn the patterns without memorizing specific examples. Training beyond 4 epochs almost always causes overfitting, where the model achieves excellent performance on training data but degrades on new inputs because it memorized the training examples rather than learning the underlying patterns.

The signs of overfitting are visible in the training curves: training loss continues to decrease while validation loss stops decreasing or starts increasing. The divergence point between training and validation loss is approximately when you should stop. This is called early stopping, and most training frameworks support it automatically by evaluating on a validation set every N steps and saving the checkpoint with the best validation performance.

For small datasets (fewer than 500 examples), overfitting is a bigger risk and you should use 2 epochs as the starting point, possibly increasing to 3 if validation metrics are still improving. For large datasets (10,000+ examples), 1 to 2 epochs is often sufficient because each example only needs to be seen once or twice when the dataset is diverse enough to cover the task's patterns.

A common mistake is training for 10, 20, or 50 epochs because the training loss keeps decreasing. The training loss will always decrease with more training because the model is memorizing the data. This is not evidence of learning. Validation loss is the only reliable signal, and if it is increasing while training loss decreases, you have already passed the optimal stopping point.

LoRA-Specific Parameters

When using LoRA, three additional hyperparameters control the adapter's capacity and influence.

Rank (r): Determines the number of trainable parameters per adapter layer. Rank 16 is the recommended default for most tasks. Rank 8 works for simple tasks like format changes or classification. Rank 32 to 64 may be needed for complex tasks that require the model to learn significant new patterns. Going above 128 is rarely beneficial. Higher rank means more parameters, more memory, and longer training, with diminishing returns above the point where the adapter has enough capacity for your task.

Alpha (scaling factor): Controls how much the LoRA adapters influence the model's output. The effective influence is scaled by alpha/rank. The standard recommendation is to set alpha equal to rank (alpha = 16 when rank = 16) for a neutral scaling. Setting alpha to double the rank (alpha = 32 when rank = 16) increases the adapter's influence, which can speed learning but risks instability. Start with alpha = rank and adjust only if the model is learning too slowly (increase alpha) or producing unstable outputs (decrease alpha).

Target modules: Which layers of the model receive LoRA adapters. The minimum effective set is the attention query and value projections (q_proj, v_proj). Adding the key projection (k_proj) and output projection (o_proj) improves results for most tasks. Including the feed-forward network layers (gate_proj, up_proj, down_proj) trains more parameters but captures adaptations in the MLP layers. For most tasks, targeting all linear layers gives the best results without significantly increasing training time, and this is the recommended default in 2026.

LoRA dropout: Applies dropout to the LoRA adapter outputs during training as a regularization technique. A dropout of 0.05 to 0.1 can help prevent overfitting on small datasets. For datasets with 3,000+ examples, dropout of 0 is fine because the data itself provides enough regularization. LoRA dropout is a secondary knob that rarely makes or breaks a training run.

Weight Decay

Weight decay is a regularization technique that slightly penalizes large weight values during training, preventing the model from relying too heavily on any single weight and encouraging more distributed representations. For LLM fine-tuning, a weight decay of 0.01 to 0.1 is standard, with 0.01 being the most common default.

Weight decay matters more for full fine-tuning than for LoRA, because full fine-tuning modifies all weights and has more capacity to overfit. For LoRA fine-tuning, the adapter parameters are already a small fraction of the model, which acts as a natural regularizer, so weight decay has less impact. If you are using LoRA and unsure about weight decay, 0.0 (no decay) or 0.01 both work fine for most tasks.

Max Sequence Length

The max sequence length determines the longest input-output pair the model will process during training. Longer sequences use more GPU memory and take more time per training step. Setting this correctly saves significant compute without sacrificing quality.

Analyze your training data to find the 95th percentile of sequence lengths (the length that covers 95% of your examples). Set max sequence length to that value plus a small buffer (10 to 20%). Examples longer than the max length will be truncated, which may cause the model to learn from incomplete examples. If truncation affects more than 5% of your data, increase the max length or consider splitting long examples.

For most fine-tuning tasks with short outputs (classification, short-form generation, extraction), max sequence length of 512 to 1024 tokens is sufficient. For long-form generation (article writing, code generation, document summarization), you may need 2048 to 4096 tokens. Every doubling of sequence length roughly doubles the memory required for that batch, so there is a real trade-off between handling long sequences and fitting larger batches in memory.

Quick Reference: Default Starting Values

These values are safe defaults for a QLoRA fine-tuning run on a 7B to 8B model with 3,000 to 5,000 training examples:

Learning rate: 2e-4. Schedule: cosine decay with 5% warmup. Effective batch size: 32 (using gradient accumulation as needed). Epochs: 3. LoRA rank: 16. LoRA alpha: 16. LoRA target: all linear layers. LoRA dropout: 0.05. Weight decay: 0.01. Max sequence length: 1024 (adjust to your data). Optimizer: AdamW (the default in Transformers). BF16 precision: yes, if your GPU supports it (all NVIDIA Ampere and newer GPUs do).

Start with these values, run a training job, evaluate the results, and adjust. In most cases, the only parameters you will need to change from these defaults are the number of epochs (if overfitting) and the LoRA rank (if the task requires more or less capacity). The learning rate and batch size defaults work for the vast majority of tasks.