LLM Quantization Explained: GGUF, GPTQ, and AWQ Compared

What Quantization Does

Neural network weights are normally stored as 16-bit floating point numbers (BF16 or FP16). Each parameter occupies 2 bytes. A 70 billion parameter model needs 140GB of memory at full precision, far more than any single GPU holds. Quantization converts these 16-bit values into lower-precision representations: 8-bit integers (INT8), 4-bit integers (INT4), or other compact formats.

The conversion is lossy, meaning some information is permanently discarded. The engineering challenge is minimizing the quality impact while maximizing the compression ratio. Modern quantization methods are remarkably effective at this. A well-quantized 4-bit model retains approximately 95% of the full-precision model's benchmark performance while using 75% less memory.

The practical benefits are immediate. A 70B model that requires two A100 80GB GPUs at full precision fits on a single A100 at 4-bit quantization. A 7B model that needs a 16GB GPU at full precision runs on an 8GB consumer GPU when quantized. This is what makes open source LLMs accessible on consumer hardware.

GGUF: The Standard for Local Inference

GGUF (GPT-Generated Unified Format) is the native format for llama.cpp and Ollama. It was designed specifically for efficient inference on diverse hardware, including CPU, GPU, and hybrid CPU+GPU configurations.

GGUF's key advantage is flexibility. When a model does not fit entirely in GPU VRAM, GGUF automatically splits it across GPU and system RAM. The GPU handles as many layers as it can fit, and the CPU handles the rest. This is slower than full GPU inference but dramatically faster than pure CPU inference, and it lets you run models that would otherwise be impossible on your hardware.

GGUF offers multiple quantization levels, each trading quality against size:

Q8_0 (8-bit): Virtually indistinguishable from full precision on most benchmarks. File size is roughly 50% of BF16. Use when you have ample VRAM and want maximum quality.

Q6_K (6-bit): Near-BF16 quality with about 37% of the original file size. Excellent for applications where quality is paramount but you need some memory savings.

Q5_K_M (5-bit medium): Very close to full quality at roughly 33% of BF16 size. A strong choice when you want quality assurance without Q8's memory demands.

Q4_K_M (4-bit medium): The community consensus sweet spot. Roughly 25% of BF16 size with approximately 95% quality retention. This is what Ollama uses by default, and it is the right choice for most developers. Unless you have a specific reason to use a different level, start here.

Q4_K_S (4-bit small): Slightly more aggressive than Q4_K_M, saving about 5% more memory at the cost of marginally lower quality. Useful when Q4_K_M barely does not fit and you need to squeeze out a few hundred megabytes.

Q3_K (3-bit): Noticeable quality degradation begins here. Factual accuracy drops, creative writing becomes less coherent, and complex reasoning suffers. Only use when hardware constraints are severe.

Q2_K (2-bit): Significant quality loss. The model becomes unreliable for anything beyond simple conversational responses. The memory savings over Q3 are small in absolute terms, making this level hard to justify.

The general rule: never go below Q3 unless you are experimenting. The quality degradation below 3-bit accelerates sharply while the memory savings in absolute gigabytes become marginal.

GPTQ: GPU-Only Quantization

GPTQ (GPT Quantization) is a post-training quantization method that uses a small calibration dataset to minimize the quantization error for each weight tensor. Unlike GGUF's round-to-nearest approach, GPTQ optimizes the quantization of each weight while considering the impact on other weights in the same layer, producing a tighter approximation of the original model's behavior.

GPTQ is GPU-only. It does not support CPU inference or CPU+GPU hybrid splitting. This limits its use to machines with sufficient GPU VRAM to hold the entire quantized model. The benefit is faster inference than GGUF on GPUs, because the kernels are optimized specifically for GPU execution without the overhead of managing CPU fallback.

GPTQ models are typically quantized at 4-bit or 8-bit. The quality at 4-bit GPTQ is generally comparable to GGUF Q4_K_M, though some benchmarks show GPTQ performing slightly better on tasks requiring factual precision, likely due to the calibration-based optimization. The difference is small enough that it rarely drives the format choice.

vLLM supports GPTQ natively through the Marlin and Machete GPU kernels, which are heavily optimized for GPTQ's specific data layout. If you are serving models with vLLM in production, GPTQ is a solid choice that integrates cleanly with the serving stack.

AWQ: Activation-Aware Weight Quantization

AWQ takes a different approach to minimizing quantization error. Instead of treating all weights equally, AWQ identifies the most important weights by analyzing activation patterns, meaning which weights contribute most to the model's outputs. It then protects these critical weights with higher precision while quantizing less important weights more aggressively.

The result is quality that matches or slightly exceeds GPTQ at the same bit width, with faster inference on modern NVIDIA hardware. AWQ models are also GPU-only, like GPTQ, and integrate well with vLLM.

AWQ's practical advantage over GPTQ is inference speed. The data layout is more friendly to modern GPU architectures, resulting in higher tokens-per-second rates. On Ampere (A100, RTX 3090) and later GPUs, AWQ is typically 10% to 20% faster than GPTQ at equivalent quantization levels.

For new deployments on GPU-only infrastructure, AWQ is generally the better choice over GPTQ. The quality is equivalent, the speed is better, and the ecosystem support in vLLM and other serving frameworks is equally mature.

FP8: The Production Standard on Modern Hardware

FP8 (8-bit floating point) is the newest and increasingly dominant quantization format for production deployments on NVIDIA Hopper (H100, H200) and later GPUs. Unlike integer quantization (INT4, INT8), FP8 preserves the floating-point representation, just with fewer bits for the mantissa and exponent.

The quality impact of FP8 quantization is virtually undetectable on standard benchmarks. Most evaluations show less than 0.5% quality degradation compared to BF16. This makes FP8 a "free" optimization on compatible hardware: you cut memory usage in half with no meaningful quality trade-off.

FP8 requires hardware support. The Hopper architecture added native FP8 tensor core operations, making it fast. On older GPUs (Ampere, Ada Lovelace), FP8 is either unsupported or emulated without the hardware acceleration that makes it worthwhile. If you are running on H100s or newer, FP8 should be your default. If you are on A100s or consumer GPUs, use Q4/Q8 GGUF or AWQ instead.

vLLM supports FP8 natively, and over 20% of vLLM deployments in 2026 reportedly use FP8 on Hopper hardware. The combination of FP8 quantization with vLLM's PagedAttention produces the highest throughput per dollar currently achievable for LLM serving.

How to Choose a Quantization Format

The decision tree is straightforward:

Running locally with Ollama or llama.cpp? Use GGUF. Ollama handles this automatically. If downloading from Hugging Face for llama.cpp, look for Q4_K_M files as the default choice.

Serving in production with vLLM on Hopper GPUs (H100/H200)? Use FP8. The quality loss is negligible and the throughput benefit is substantial.

Serving in production with vLLM on Ampere or Ada GPUs (A100/RTX 4090)? Use AWQ at 4-bit. Faster than GPTQ on these architectures with equivalent quality.

Need CPU+GPU hybrid inference (model too large for VRAM)? GGUF is your only option. Neither GPTQ, AWQ, nor FP8 support CPU offloading.

Targeting mobile or embedded devices? GGUF at Q4_K_M or Q4_K_S for the smallest models (Phi-4, Gemma 3 2B). Some mobile frameworks support their own quantization formats (Core ML on Apple devices, TensorFlow Lite on Android).

The Bigger Model Principle

One of the most important insights in practical LLM deployment: a larger model at lower precision almost always outperforms a smaller model at higher precision, even when the quantized file sizes are similar.

A 70B model quantized to Q4 (approximately 40GB) significantly outperforms a 7B model at full BF16 precision (approximately 14GB) on every benchmark. The 70B model's deeper understanding of language, reasoning patterns, and world knowledge is largely preserved through quantization, while the 7B model never had that understanding to begin with.

This means your hardware strategy should prioritize fitting the largest possible model, not the highest precision. If you have 24GB of VRAM, run a 30B model at Q4 rather than a 14B model at Q8. If you have 80GB, run a 70B at Q4 rather than a 30B at FP16. The quality difference is substantial and consistent across use cases.

The exception is edge deployment on extremely constrained devices (phones, embedded systems), where the smallest models at the lowest viable quantization levels are the only option. In these cases, model architecture matters more than size: Phi-4 at 3.8B with Q4 outperforms older 7B models at the same quantization because its training data was more carefully curated.

Measuring Quality Loss

Published benchmarks show average quality across standard tasks. Your application may be more or less sensitive to quantization than these averages suggest. Before deploying a quantized model in production, run your own evaluations on a representative sample of your actual workload.

Pay particular attention to tasks involving precise factual recall (quantization can introduce subtle errors in less common facts), mathematical computation (quantization affects numerical precision), and long-form generation (quality degradation accumulates over longer outputs). Tasks involving classification, sentiment analysis, and short-form question answering are typically more robust to quantization.

A practical testing approach: run 100 representative prompts through both the full-precision and quantized versions of your chosen model. Have the full-precision outputs serve as ground truth and measure how often the quantized model produces meaningfully different answers. For most applications at Q4_K_M, the disagreement rate is under 5%.