How to Fine-Tune Any LLM on a Single GPU with QLoRA
What QLoRA Changes
Standard LoRA freezes the base model in FP16 or BF16 precision (16 bits per parameter) and trains small adapter matrices on top. This already cuts trainable parameters to less than 1%, but the frozen model still takes its full memory footprint to store. For a 7B model, that is about 14GB in FP16, and for a 70B model, about 140GB, which exceeds the VRAM of any single consumer GPU.
QLoRA, introduced by Dettmers et al. in 2023, solves this by quantizing the frozen base model to 4-bit precision using a technique called NF4 (4-bit NormalFloat), which is specifically designed for normally distributed neural network weights. This cuts the model's memory footprint by roughly 75%: a 7B model drops from 14GB to about 3.5GB, and a 70B model drops from 140GB to about 35GB. The LoRA adapter parameters are still trained in full BF16 precision, so the quality of the learned adaptation is preserved.
An additional innovation is double quantization, where the quantization constants themselves (the scaling factors used to dequantize the 4-bit weights during computation) are quantized to 8-bit, saving another 0.5 to 1GB of memory. Together, NF4 quantization and double quantization bring a 70B model's memory footprint down to about 33GB, which fits on a single 48GB A6000 or with careful memory management on a 40GB A100.
The remarkable finding from the QLoRA paper is that this aggressive quantization of the base model does not significantly degrade the quality of the fine-tuned result. On the Guanaco benchmark, QLoRA fine-tuning of a 65B model reached 99.3% of the performance of full 16-bit fine-tuning while using a fraction of the GPU memory. This is because the quantization errors in the frozen weights are consistent, and the LoRA adapters learn to compensate for them during training.
Hardware Requirements
The minimum hardware for QLoRA depends on the model size you want to fine-tune. Here are the practical requirements based on real-world usage in 2026:
7B to 8B models (Llama 3.1 8B, Mistral 7B, Qwen2.5 7B): A GPU with 16GB VRAM is the minimum, and 24GB is comfortable. An RTX 4090 (24GB), RTX 3090 (24GB), or A5000 (24GB) handles these easily. Cloud cost: $0.40 to $1.50 per hour on RunPod, Vast.ai, or Lambda Labs. Training 3,000 examples for 3 epochs takes 30 to 90 minutes.
13B to 14B models (Llama 3 13B, Qwen2.5 14B): Requires 24GB VRAM. An RTX 4090 or A5000 works with a batch size of 1 and gradient accumulation. Training takes 1 to 3 hours for a typical dataset.
70B models (Llama 3.1 70B, Qwen2.5 72B): Requires 40 to 48GB VRAM for comfortable training. An A6000 (48GB) or A100 (40/80GB) is the practical choice. On a 24GB card, it is technically possible with aggressive memory optimization (gradient checkpointing, batch size 1, offloading to CPU RAM) but painfully slow. Cloud cost: $2 to $6 per hour. Training takes 6 to 12 hours.
Beyond GPU VRAM, you need 32GB or more of system RAM (CPU memory) for model loading and data preprocessing, and at least 100GB of fast SSD storage for the model weights, dataset, and checkpoints.
Set Up the Environment
Install the required Python packages. The core stack is: transformers (model loading and tokenization), peft (LoRA adapter management), bitsandbytes (4-bit quantization), trl (training with SFTTrainer), datasets (data loading and formatting), and accelerate (GPU management). All are installable via pip and should be at their latest versions, as QLoRA support has improved significantly through 2025 and 2026. Pin your package versions in a requirements file to ensure reproducibility.
Verify your GPU is visible to PyTorch by running a quick check that prints the device name and available VRAM. If you are on a cloud instance, confirm the GPU matches what you rented before starting a training run. Also verify that bitsandbytes detects CUDA correctly, since 4-bit loading depends on GPU kernel support that can silently fall back to CPU if the installation is wrong.
Load the Model in 4-Bit
Configure BitsAndBytesConfig with load_in_4bit set to True, bnb_4bit_quant_type set to "nf4", bnb_4bit_compute_dtype set to torch.bfloat16 (or torch.float16 if your GPU does not support BF16), and bnb_4bit_use_double_quant set to True. Pass this config to AutoModelForCausalLM.from_pretrained along with the model name and device_map="auto".
The model will download (if not cached) and load in quantized form. A 7B model should show about 4 to 5 GB of VRAM usage after loading. If it shows 14GB or more, the quantization config was not applied correctly. Load the tokenizer separately with AutoTokenizer and set the padding token if the model does not have one (pad_token = eos_token is the common workaround for Llama-style models).
Configure LoRA Adapters
Create a LoraConfig with r (rank) set to 16 or 32, lora_alpha set to equal the rank, lora_dropout set to 0.05, and target_modules set to "all-linear" (or the specific module names for your architecture: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj for Llama-style models). Set task_type to "CAUSAL_LM".
Apply the LoRA config to the model using get_peft_model(), which wraps the base model and inserts the trainable adapters. Print the model's trainable parameter count: it should show less than 1% of the total parameters as trainable. A rank-16 config on Llama 3.1 8B targeting all linear layers trains about 42 million parameters out of 8 billion total, or 0.5%.
Prepare and Format Training Data
Format your data as conversations in the model's expected chat template. For most models this means a list of messages with role ("system", "user", "assistant") and content fields. The tokenizer's apply_chat_template method converts these into the model's native format with special tokens. Each example should have the complete conversation including the assistant's response, which is the target the model learns to generate.
Split your dataset before training: 90% for training, 10% for validation. The validation set must never be used during training, as it serves to detect overfitting. Remove duplicate or near-duplicate examples, verify that every example meets your quality standard, and check that the distribution of topics and difficulty levels in your training set is representative of what the model will see in production. The guide on preparing training data covers this step in full detail.
Train with SFTTrainer
Configure the TrainingArguments with: output_dir for checkpoints, num_train_epochs set to 3, per_device_train_batch_size set to 2 or 4 (reduce to 1 if you run out of memory), gradient_accumulation_steps set to reach an effective batch size of 8 to 16, learning_rate set to 1e-4 or 2e-4, warmup_ratio set to 0.03 to 0.1, lr_scheduler_type set to "cosine", bf16 set to True, logging_steps set to 10, eval_strategy set to "steps", eval_steps set to 50 to 100, and save_strategy set to "steps". Enable gradient_checkpointing to trade compute for memory, which is essential for fitting larger models on smaller GPUs.
Create the SFTTrainer with the model, training arguments, train and eval datasets, tokenizer, and max_seq_length (typically 2048 or 4096 depending on your data). Call trainer.train() and monitor the training and validation loss. Training loss should decrease steadily. Validation loss should decrease and then flatten, and if it starts increasing while training loss continues to decrease, the model is overfitting and you should stop training or reduce the number of epochs.
Merge and Deploy
After training, save the adapter with trainer.save_model() or model.save_pretrained(). This saves only the LoRA adapter weights (typically 20 to 100 MB), not the base model. To deploy, reload the base model in full precision (FP16 or BF16), load the adapter with PeftModel.from_pretrained(), and call model.merge_and_unload() to fuse the adapter weights into the base model. The result is a standard model file that runs at the same speed as the original with no adapter overhead.
For local deployment with tools like llama.cpp, vLLM, or Ollama, convert the merged model to the target format (GGUF for llama.cpp, the native format for vLLM). Test the merged model on your held-out validation set and on a diverse set of new inputs to verify it behaves as expected. Compare its outputs to the base model on both your task-specific inputs and a few general-purpose prompts to confirm it has not lost general capability. The full evaluation process is covered in how to evaluate a fine-tuned model.
QLoRA makes fine-tuning accessible on consumer hardware by quantizing the frozen base model to 4-bit while training LoRA adapters in full precision. A 7B model fine-tunes in under an hour on a $1,600 GPU. The merged result runs at full speed with no adapter overhead, and the quality matches full-precision LoRA within a fraction of a percent.