What GPU Do You Need to Run LLMs Locally?

The Memory Formula

LLM memory requirements come from two components: the model weights and the KV cache. The weights are the model itself, stored permanently in memory during inference. The KV cache stores attention state for the current context and grows with the length of the conversation.

Weight memory is straightforward to calculate. Multiply the number of parameters by the bytes per parameter at your chosen quantization level:

Full precision (BF16/FP16): 2 bytes per parameter. A 7B model needs 14GB. A 70B model needs 140GB.

8-bit (Q8_0/INT8): 1 byte per parameter. A 7B model needs 7GB. A 70B model needs 70GB.

4-bit (Q4_K_M/AWQ/GPTQ): 0.5 bytes per parameter. A 7B model needs 3.5GB. A 70B model needs 35GB.

3-bit (Q3_K): 0.375 bytes per parameter. A 7B model needs 2.6GB. A 70B model needs 26GB.

KV cache memory depends on context length, model architecture, and the number of concurrent requests. A rough estimate for a single request: 1GB of KV cache per 32K tokens of context for a 7B model, 4GB per 32K tokens for a 70B model. With multiple concurrent requests, multiply by the number of active requests.

Runtime overhead adds 1 to 2GB for the inference framework itself (CUDA context, framework buffers, temporary compute memory).

Total VRAM needed = weight memory + KV cache + runtime overhead. Add 10% safety margin to avoid out-of-memory errors during peak usage.

VRAM Requirements by Model Size

2B to 3B parameters (Gemma 3 2B, Phi-4 3.8B):

At Q4: 1.5 to 2.3GB weight memory. Total with KV cache: 3 to 4GB. Runs on: any GPU with 4GB+ VRAM, Apple Silicon with 8GB+ memory, even some phones. These models fit everywhere and generate tokens extremely fast, often exceeding 100 tokens per second on modern GPUs.

7B to 8B parameters (Qwen 3 8B, Llama 4 8B, Mistral 7B):

At Q4: 3.5 to 4.9GB weight memory. Total with KV cache: 5 to 8GB. Runs on: RTX 4060 (8GB), RTX 3060 12GB, RTX 4070 (12GB), Apple Silicon with 16GB+. This is the sweet spot for consumer hardware. Generation speed is typically 40 to 80 tokens per second on a modern GPU.

13B to 14B parameters (Qwen 3 14B):

At Q4: 7 to 8.5GB weight memory. Total with KV cache: 9 to 12GB. Runs on: RTX 4070 (12GB), RTX 4080 (16GB), Apple Silicon with 16GB+. A meaningful quality step up from 8B models, particularly for reasoning and instruction following.

30B to 34B parameters (Qwen 3 30B-A3B):

At Q4: 15 to 20GB weight memory. Total with KV cache: 18 to 24GB. Runs on: RTX 4090 (24GB), RTX 5090 (32GB), A6000 (48GB), Apple Silicon with 32GB+. Mixture-of-experts variants like Qwen 3 30B-A3B activate only 3B parameters per token, giving near-30B quality at 8B-like speed.

65B to 72B parameters (Qwen 3 72B, Llama 4 70B):

At Q4: 33 to 43GB weight memory. Total with KV cache: 38 to 50GB. Runs on: A100 80GB (single GPU), A6000 48GB (tight), two RTX 4090s (48GB combined with tensor parallelism), Apple Silicon with 64GB+. This is the largest model size that fits on practical single-machine setups. Quality approaches proprietary model levels.

100B+ parameters (Qwen 3 235B, Llama 3 405B):

At Q4: 50 to 200GB+ weight memory. Requires: multiple A100 80GB GPUs (2 to 8 depending on model), H100 clusters, or specialized infrastructure. These models require significant investment and are typically only justified for organizations with specific quality requirements that smaller models cannot meet.

GPU Recommendations by Budget

Under $300 (used market): RTX 3060 12GB. Handles 7B to 8B models at Q4 with room for decent context length. The 12GB variant (not the 8GB) is essential. Available used for under $200 in many markets. This is the minimum viable GPU for meaningful local LLM inference.

$400 to $600: RTX 4060 Ti 16GB or RTX 4070 12GB. The 16GB variant handles 13B to 14B models at Q4. Better performance per watt than older cards. Good for developers who want daily local LLM usage without noise or heat concerns.

$800 to $1,200: RTX 4080 16GB or RTX 4070 Ti Super 16GB. Comfortable with 14B models and can handle 30B models with aggressive quantization. Fast enough for interactive use with larger models.

$1,500 to $2,000: RTX 4090 24GB or RTX 5090 32GB. The consumer sweet spot for serious local LLM work. Handles 30B models at Q4 easily and can run 70B models at Q3 or with CPU layer offloading. The RTX 5090 at 32GB is the best consumer GPU for LLMs in 2026.

$4,000 to $6,000: NVIDIA A6000 48GB (workstation). Handles 70B models at Q4 in a single GPU. ECC memory for reliability. Quieter and designed for sustained workloads. Preferred for small team server deployments.

$10,000+: A100 80GB or H100 80GB (data center). The production standard. Handles any single model at Q4. H100's FP8 support provides additional efficiency. Available for rent from cloud providers at $0.80 to $3.50 per hour.

Apple Silicon Recommendations

Apple Silicon Macs use unified memory shared between CPU and GPU, which means all system RAM is available for model weights. The Metal GPU provides reasonable inference speed, and Ollama 0.19+ uses the MLX backend for optimal performance.

M2/M3/M4 with 16GB: Handles 7B to 8B models at Q4. Usable for development and personal AI assistant use. Token generation around 15 to 25 tokens per second for 8B models.

M2/M3/M4 Pro with 32GB: Handles 14B models comfortably, 30B models with Q4 quantization. Good for developers who want a capable local model as part of their daily workflow. 12 to 18 tokens per second for 14B models.

M2/M3/M4 Max with 64GB: Handles 30B to 34B models easily, 70B models with Q3 or Q4 (tight). The high memory bandwidth of Max chips provides competitive token generation speeds despite using Metal rather than CUDA.

M2/M3/M4 Max with 128GB: Handles 70B models at Q4 comfortably with room for large context windows. 15 to 20 tokens per second for 70B models. This is the best laptop setup for running large models without an external GPU or cloud rental.

The trade-off with Apple Silicon: memory bandwidth is lower than dedicated NVIDIA GPUs, so token generation is slower. But the ability to use all system RAM for models, combined with the silent, cool operation and long battery life, makes Macs the preferred platform for developers who want large models on a portable machine.

Cloud GPU Rental

For models too large for your local hardware, or for production serving where uptime matters, cloud GPU rental provides on-demand access to powerful hardware without capital expenditure.

Vast.ai offers marketplace pricing where GPU owners rent their machines, typically 30% to 60% cheaper than major cloud providers. A100 80GB instances start around $0.80 per hour. RTX 4090 instances start around $0.25 per hour. The trade-off is less reliability than major providers and no SLA guarantees.

Major cloud providers (AWS, GCP, Azure) offer A100 and H100 instances with enterprise SLAs, typically at $1.50 to $3.50 per hour for an A100. These are preferred for production workloads where downtime costs more than the higher hourly rate.

For intermittent use (development, experimentation, batch processing), cloud rental is almost always cheaper than owning hardware. For sustained 24/7 production use, purchasing GPUs breaks even with rental costs in 6 to 12 months for consumer GPUs and 12 to 24 months for data center GPUs.

Multi-GPU Setups

When a model does not fit on a single GPU, you have two options: CPU layer offloading (GGUF only, slower but works with any hardware) or tensor parallelism (splits the model across multiple GPUs, requires NVLink or PCIe for inter-GPU communication).

Tensor parallelism with vLLM or llama.cpp distributes model layers across GPUs. Two RTX 4090s (48GB combined) handle a 70B Q4 model. Four RTX 4090s (96GB combined) handle a 70B model at full FP16. The GPUs must be in the same machine, and NVLink connectivity provides significantly better performance than PCIe for inter-GPU communication.

For multi-GPU consumer setups, ensure your motherboard has sufficient PCIe lanes and physical space for multiple GPUs. Cooling becomes critical with two or more GPUs generating 300W+ each in a desktop case. Water cooling or open-air setups are common for dual-GPU LLM workstations.