How to Merge and Deploy LoRA Adapters

Why Merging Matters

During training, LoRA keeps the base model frozen and trains small adapter matrices that modify the model's behavior. At inference time, both the base model and the adapter must be loaded, and the adapter computation adds a small amount of overhead to every forward pass. For a single adapter, this overhead is negligible, typically 1 to 3% latency increase. But there are good reasons to merge the adapter into the base model rather than serving them separately.

The first reason is compatibility. Many inference engines, especially optimized ones like llama.cpp, Ollama, and quantized model formats, expect a single model file rather than a base-plus-adapter combination. If you want to convert your fine-tuned model to GGUF format for local inference, or to AWQ/GPTQ format for GPU serving, you need a merged model as the starting point. The adapter format is specific to the Hugging Face PEFT ecosystem and is not directly compatible with most production inference stacks.

The second reason is simplicity. A merged model is a single artifact that you deploy, version, and manage. There is no risk of version mismatch between the base model and adapter, no need to ensure the correct adapter is loaded with the correct base, and no additional code path in your serving infrastructure. For production systems where reliability matters, reducing the number of moving parts is worth the small amount of disk space the merged model requires.

The third reason is that merging is irreversible in a useful way: it commits you to the adapter's changes, which forces proper evaluation before deployment. If you deploy an unmerged adapter, it is tempting to swap adapters in production without proper testing because it is easy to do. If you merge and deploy a single model, the deployment process is the same as deploying any model, with the same quality gates and rollback procedures.

The Basic Merge Process

The merge operation is mathematically simple. For each layer that has a LoRA adapter, the merged weight is: W_merged = W_base + (alpha / rank) * B * A, where W_base is the frozen base weight, A and B are the LoRA adapter matrices, alpha is the scaling factor, and rank is the LoRA rank. The PEFT library handles this with a single method call, but understanding the math matters when troubleshooting quality issues after merging.

The standard workflow uses the Hugging Face Transformers and PEFT libraries. You load the base model, load the adapter on top of it, call the merge method, and save the result. The entire process takes 5 to 30 minutes depending on model size, because it requires loading the full model into memory, performing the matrix additions, and saving the merged weights. For a 7B model, you need about 14 GB of RAM (or VRAM) to hold the model in float16 precision during the merge. For a 70B model, you need about 140 GB, which typically means using CPU RAM rather than GPU VRAM.

The key function is model.merge_and_unload() in the PEFT library. The "unload" part removes the adapter hooks from the model after merging, so the resulting model object behaves exactly like a standard Transformers model with no PEFT dependency. You then save it using the standard model.save_pretrained() method, which writes the merged weights in safetensors format. The tokenizer should be saved alongside the model using tokenizer.save_pretrained() to the same directory, since some fine-tuning processes modify the tokenizer by adding special tokens.

Handling Multiple Adapters

Some workflows produce multiple LoRA adapters that need to be combined into a single model. This happens when you train separate adapters for different capabilities (one for formatting, one for domain knowledge, one for safety) and want to deploy a model that has all three. There are several strategies for combining adapters, each with different trade-offs.

Sequential merging applies one adapter at a time: merge adapter A into the base to get model_A, then merge adapter B into model_A to get model_AB. This is the simplest approach and works well when the adapters were trained for complementary rather than overlapping tasks. The order of merging can matter because each merge changes the base weights that the next adapter is applied to, so if you observe quality differences based on merge order, try both and keep the better result.

Weighted merging applies multiple adapters simultaneously with different weights. Instead of merging each adapter at full strength, you scale each adapter's contribution: W_merged = W_base + w1 * adapter_A + w2 * adapter_B, where w1 and w2 are weights between 0 and 1. This gives you a dial to control how much influence each adapter has. If adapter A handles formatting and adapter B handles domain knowledge, you might use w1=0.8 and w2=1.0 to slightly reduce the formatting adapter's influence while keeping full domain knowledge. The mergekit library supports this workflow with configuration files that specify merge recipes.

TIES merging (Trim, Elect Sign, and Merge) is a more sophisticated approach that resolves conflicts between adapters by trimming parameters that changed least during training, resolving sign conflicts when two adapters push a weight in opposite directions, and averaging the remaining parameters. TIES produces cleaner merges when adapters were trained on overlapping tasks where conflicts are likely. The mergekit library implements TIES along with several other merge strategies.

DARE merging (Drop And REscale) randomly drops a fraction of each adapter's weight updates before merging, then rescales the remaining updates to compensate. This acts as a regularizer that prevents the merged model from being overly influenced by any single adapter, and it has been shown to produce better multi-adapter merges than simple averaging in many cases.

Quantization After Merging

The merged model is in the same precision as the base model, typically float16 or bfloat16, which means a 7B model is about 14 GB and a 70B model is about 140 GB. For production deployment, especially on consumer hardware or cost-constrained cloud instances, you almost always want to quantize the merged model to a smaller precision.

The most common quantization target is GGUF format for llama.cpp and Ollama deployment. GGUF supports quantization from Q2_K (about 2.5 bits per weight) to Q8_0 (8 bits per weight), with Q4_K_M (about 4.5 bits per weight) being the sweet spot for most use cases: it reduces model size by about 75% with minimal quality loss. A 7B model at Q4_K_M is about 4 GB, small enough to run on any modern laptop. The conversion is done with the llama.cpp convert and quantize tools, which take the merged safetensors model as input.

AWQ (Activation-aware Weight Quantization) is the preferred quantization method for GPU serving with vLLM or TGI. AWQ quantizes to 4-bit precision while preserving the weights that have the most impact on model quality, determined by analyzing activation patterns on a calibration dataset. The result is a model that is about 75% smaller and significantly faster on GPU inference, with quality very close to the full-precision model. The autoawq library handles the quantization process and produces models compatible with vLLM and the Hugging Face inference stack.

GPTQ is an older quantization method that also produces 4-bit models but uses a different algorithm (approximate second-order optimization) to choose which precision to assign each weight. GPTQ models are widely supported but AWQ has largely superseded it for new deployments due to better quality at the same compression ratio. If you see a GPTQ model in the wild, it works fine, but for new quantization, AWQ is the better choice in most cases.

Deployment Options

Once you have a merged (and optionally quantized) model, the deployment path depends on your serving requirements.

vLLM is the dominant choice for high-throughput GPU serving in 2026. It supports continuous batching, PagedAttention for efficient memory use, and automatic tensor parallelism across multiple GPUs. A merged model in safetensors or AWQ format loads directly into vLLM with a single command. vLLM serves an OpenAI-compatible API, so your application code does not need to change when you swap between models. For a 7B model on a single A100, vLLM achieves 500 to 2,000 tokens per second depending on batch size and sequence length.

Ollama and llama.cpp are the choices for local or edge deployment. Convert the merged model to GGUF, load it into Ollama with a one-line Modelfile, and serve it locally. This is the best path for applications that need to run on-premise, on user devices, or without cloud infrastructure. A Q4 quantized 7B model runs at 30 to 60 tokens per second on an M2 MacBook Pro, which is fast enough for interactive use.

Hugging Face TGI (Text Generation Inference) is another production-grade serving option with Docker-based deployment, built-in monitoring, and support for multiple quantization formats. It is particularly well-integrated with the Hugging Face ecosystem, so if your workflow already uses Hugging Face for training, TGI provides a seamless deployment path.

API-based deployment through providers like Together AI, Fireworks, or Anyscale lets you upload your merged model and serve it through their infrastructure without managing GPU instances. This is the easiest path for teams that want fine-tuned model serving without DevOps overhead. Costs vary but typically run $0.10 to $0.30 per million tokens for 7B models, significantly cheaper than frontier model API pricing.

Verifying the Merge

After merging, always verify that the merged model produces the same outputs as the unmerged base-plus-adapter combination. The merge should be mathematically lossless (the outputs should be identical within floating-point precision), but bugs in the merge code, version mismatches between libraries, or incorrect adapter loading can produce a merged model that behaves differently from what you tested during development.

The verification process is straightforward: prepare 50 to 100 test inputs, run them through both the unmerged model (base + adapter) and the merged model, and compare the outputs. For greedy decoding (temperature = 0), the outputs should be identical or differ only in the last few tokens due to floating-point rounding. For sampled decoding, set a fixed random seed and compare. If you see meaningful differences, the merge went wrong and you should debug before deploying.

After quantization, the outputs will differ from the full-precision model because quantization is a lossy operation. The question is whether the differences matter for your task. Run your evaluation suite on the quantized model and compare to the full-precision scores. For Q4_K_M quantization of a 7B model, you should see less than 1% degradation on most benchmarks. If you see more than 2 to 3% degradation, try a higher quantization level (Q5_K_M or Q6_K) or use AWQ instead of GGUF quantization.