Full Fine-Tuning: When and How to Train All Weights

What Full Fine-Tuning Actually Changes

In full fine-tuning, the optimizer computes gradients for all of the model's parameters and updates them at every training step. For a 7B model, that means 7 billion parameters receive gradient updates, each one shifting slightly to better fit the training data. For a 70B model, 70 billion parameters are updated. Compare this to LoRA, which typically trains 10 to 50 million parameters (less than 1% of a 7B model) while keeping the rest frozen.

This difference in the number of updated parameters has three practical consequences. First, full fine-tuning has more capacity to learn complex adaptations because it can modify any weight in any layer, including the embedding layer, early attention layers, and feed-forward networks that LoRA typically does not touch. Second, it requires proportionally more compute and memory: you need enough GPU VRAM to hold the model weights, the optimizer states (which are 2x the model size for AdamW), the gradients (1x the model size), and the activations during the forward pass. For a 7B model in BF16 precision, this adds up to roughly 56 GB of VRAM for full fine-tuning versus about 14 GB for QLoRA. Third, because all weights are modified, there is no way to "undo" the fine-tuning by removing an adapter. The original model is overwritten, and you must keep a separate copy of the base model if you want to revert.

The result is a model where the fine-tuning changes are deeply integrated into every layer rather than applied as a surface-level perturbation. For tasks that require the model to fundamentally change how it processes information at the lowest levels, such as learning a new language, understanding a completely novel document format, or acquiring a radically different reasoning style, this deep integration is what makes full fine-tuning superior. For tasks that only require behavioral shifts at the output level, such as changing the response format, adjusting tone, or learning a classification mapping, LoRA's surface-level adaptation is sufficient and more practical.

When Full Fine-Tuning Is Justified

Full fine-tuning is justified in a small number of specific situations. If your use case does not match one of these, LoRA or QLoRA is almost certainly the better choice.

You have a large, high-quality dataset (10,000+ examples). Full fine-tuning's larger capacity is wasted on small datasets because the model simply memorizes the examples rather than learning generalizable patterns. With 10,000 or more diverse, high-quality examples, the additional capacity lets the model learn deeper patterns that LoRA's limited parameters cannot capture. The crossover point where full fine-tuning consistently outperforms LoRA is approximately 10,000 to 20,000 examples for a 7B model, though this varies by task complexity.

You have verified that LoRA underperforms on your task. The correct process is to try LoRA first (it is cheaper and faster), measure its performance on your evaluation set, and only move to full fine-tuning if LoRA's quality is measurably insufficient. "Measurably" means you have a specific metric (accuracy, F1, BLEU, human preference rating) that LoRA fails to meet, not a vague sense that the outputs could be better. Many teams assume full fine-tuning will be better without testing, which wastes compute on a marginal improvement that does not justify the cost.

The task requires changes in the model's early layers. LoRA adapters are applied to the attention and feed-forward layers but do not modify the token embedding layer or the early processing layers that determine how the model represents basic concepts. Tasks that require the model to understand a fundamentally new type of input, such as a novel programming language, a highly specialized notation system, or a domain with completely different structural patterns from natural language, may need early-layer modifications that only full fine-tuning provides.

You are building a foundation model variant for broad domain use. If your goal is not a single task but a broadly capable domain-specific model (a medical foundation model, a legal foundation model, a financial foundation model), full fine-tuning on a large, diverse domain dataset produces a model with deeper domain integration than LoRA adapters can achieve. This is the continued pre-training use case, which almost always uses full-weight training.

Hardware Requirements

Full fine-tuning demands significantly more GPU memory than LoRA because every parameter needs optimizer states (two additional copies of each weight for AdamW's momentum and variance tracking), gradient storage, and activation memory for backpropagation.

For a 7B model in BF16 precision: the model weights occupy about 14 GB, the optimizer states occupy about 28 GB, the gradients occupy about 14 GB, and activations add another 4 to 8 GB depending on batch size and sequence length. Total: approximately 60 to 64 GB of VRAM. This requires a single A100 80GB or H100 80GB GPU. It does not fit on consumer GPUs (RTX 4090 has 24 GB) without using CPU offloading, which reduces training speed by 5 to 10x.

For a 13B model: approximately 110 to 120 GB of VRAM. This requires two A100 80GB GPUs with tensor parallelism or a single H100 with NVLink-connected H100 clusters.

For a 70B model: approximately 600 to 700 GB of VRAM. This requires 8 to 10 A100 80GB GPUs or 4 to 8 H100 80GB GPUs with tensor parallelism. The cloud cost for this hardware runs $20 to $50 per hour, which means a 10-hour training run costs $200 to $500.

DeepSpeed ZeRO is the standard technique for reducing per-GPU memory requirements by partitioning the model, optimizer states, and gradients across multiple GPUs. ZeRO Stage 3 partitions all three, enabling training of models that would not fit on any single GPU. With ZeRO Stage 3, a 7B model can be fully fine-tuned on two 24GB GPUs (like two RTX 4090s), and a 70B model on 8 A100 40GB GPUs. DeepSpeed is integrated into the Hugging Face Trainer and requires minimal configuration: a JSON file specifying the ZeRO stage, optimizer settings, and gradient accumulation parameters.

FSDP (Fully Sharded Data Parallel) is PyTorch's native alternative to DeepSpeed ZeRO, providing similar memory optimizations. FSDP is increasingly popular because it is built into PyTorch core and does not require a third-party library. For new projects starting in 2026, FSDP is a strong default choice if your training pipeline is based on PyTorch directly rather than Hugging Face Trainer (which supports both DeepSpeed and FSDP).

Training Configuration

Full fine-tuning uses lower learning rates than LoRA because the weight updates are applied directly to the pretrained parameters rather than to randomly initialized adapters.

Learning rate: 1e-5 to 5e-5. The 2e-5 starting point works for most tasks. Larger models (70B+) are more sensitive and should use 1e-5. This is 10x lower than the 2e-4 typical of LoRA, because each update directly modifies the pretrained weights and large updates risk destroying valuable pretrained representations.

Epochs: 1 to 3. Full fine-tuning overfits faster than LoRA because all parameters can adapt to the training data, so fewer epochs are needed. For datasets under 10,000 examples, 1 to 2 epochs. For datasets over 10,000, up to 3 epochs with careful validation monitoring.

Batch size: The largest effective batch size your hardware supports after gradient accumulation, typically 32 to 128. Full fine-tuning benefits from larger batches more than LoRA does because the gradient signal needs to be stable when updating all 7B+ parameters simultaneously. Noisy gradients from small batches can push weights in conflicting directions, which is more damaging when all weights are being updated than when only 1% of weights are.

Weight decay: 0.01 to 0.1. Weight decay is more important for full fine-tuning than for LoRA because the larger parameter space provides more opportunity for overfitting. A weight decay of 0.01 is the standard starting point.

Warmup: 5 to 10% of total steps. Slightly longer warmup than LoRA helps stabilize training because the full-weight updates are more sensitive to initial gradient noise.

Managing Catastrophic Forgetting

Catastrophic forgetting, where the fine-tuned model loses general capabilities it had before training, is the primary risk of full fine-tuning. LoRA mitigates this by keeping the original weights frozen, but full fine-tuning directly overwrites those weights, so the model can genuinely forget how to do things it used to do well.

The most effective mitigation is data mixing: include a percentage of general-purpose training data alongside your task-specific data. If you fine-tune on 10,000 task-specific examples, add 2,000 to 5,000 examples of general instruction-following, general knowledge Q&A, or other broad capabilities you want to preserve. This teaches the model the new task while reminding it of its general capabilities. The mix ratio depends on how much general capability degradation is acceptable: a 20% general data mix preserves most general capability, while a 5% mix allows more aggressive task specialization at the cost of some general degradation.

The second mitigation is low learning rates and few epochs. Every training step overwrites pretrained weights, so fewer steps and smaller updates mean less forgetting. This is why the recommended learning rate for full fine-tuning (1e-5 to 5e-5) is much lower than for LoRA (1e-4 to 2e-4), and the recommended epochs (1 to 3) are fewer.

The third mitigation is evaluation on general benchmarks. Run the fine-tuned model on a standard general-purpose benchmark (MMLU, ARC, HellaSwag) before and after training. If scores drop by more than 5%, the forgetting is significant and you should increase the general data mix, decrease the learning rate, or reduce the number of epochs. The evaluation guide covers the full testing pipeline.

Full Fine-Tuning vs LoRA: The Practical Decision

In nearly all practical situations, LoRA is the better choice. It trains in a fraction of the time, requires a fraction of the hardware, preserves general capabilities by design, produces swappable adapters for multi-task serving, and achieves quality within 1 to 3% of full fine-tuning on most benchmarks. The cases where full fine-tuning is genuinely better are narrow: very large datasets, tasks requiring deep model changes, and foundation model variants.

If you are reading this guide because you think full fine-tuning might give you better results, the recommended process is: (1) Train a LoRA adapter first. (2) Evaluate it rigorously. (3) If it meets your quality requirements, ship it. (4) Only if LoRA measurably fails your quality bar should you invest in full fine-tuning. This process saves significant time and money because most teams that try both end up shipping the LoRA version, and the teams that do need full fine-tuning benefit from having LoRA results as a baseline for comparison.

The one exception is continued pre-training, which should always use full-weight training because its goal is to fundamentally expand the model's knowledge, not to teach it a surface-level task adaptation. If you are doing continued pre-training followed by instruction fine-tuning, use full-weight training for the CPT stage and LoRA for the instruction fine-tuning stage, combining the strengths of both approaches.