How to Serve LLMs with vLLM for Production Inference
Ollama works well for development and single-user scenarios, but production workloads serving multiple concurrent users need fundamentally different infrastructure. The core problem is GPU utilization: autoregressive text generation is memory-bound, not compute-bound. Each token depends on the previous one, and a naive server wastes most GPU compute cycles waiting. vLLM solves this by batching multiple requests together so the GPU processes tokens from different users simultaneously, dramatically increasing throughput.
Install vLLM
vLLM requires Python 3.9+ and NVIDIA CUDA GPUs with compute capability 7.0 or higher (V100, A100, A6000, RTX 3090/4090, H100). AMD GPU support exists but is experimental. Apple Silicon is not supported.
Install with pip: pip install vllm. For the latest features, install from source: pip install git+https://github.com/vllm-project/vllm.git. The installation pulls CUDA dependencies automatically if your system has NVIDIA drivers installed.
Verify the installation by importing vllm in Python: python -c "import vllm; print(vllm.__version__)". Check GPU detection: python -c "import torch; print(torch.cuda.device_count())" should show the number of available GPUs.
For containerized deployments, use the official Docker image: docker run --gpus all -p 8000:8000 vllm/vllm-openai --model Qwen/Qwen3-72B-Instruct.
Launch the Server
Start the OpenAI-compatible API server:
vllm serve Qwen/Qwen3-72B-Instruct --port 8000
This downloads the model from Hugging Face (first run only), loads it onto available GPUs, and starts the HTTP server. The server accepts requests at http://localhost:8000/v1/chat/completions using the same format as the OpenAI API.
Specify a quantized model for lower memory usage:
vllm serve TheBloke/Qwen3-72B-Instruct-AWQ --quantization awq --port 8000
For FP8 quantization on Hopper GPUs:
vllm serve Qwen/Qwen3-72B-Instruct --quantization fp8 --port 8000
The server logs show model loading progress, GPU memory allocation, and the URL where the API is available. Initial startup takes 30 seconds to several minutes depending on model size and download speed.
Configure for Production
Tensor parallelism splits the model across multiple GPUs. For a 70B model on two A100 40GB GPUs: --tensor-parallel-size 2. Each GPU holds half the model. This is required when a model does not fit on a single GPU at your chosen quantization level.
Max model length sets the maximum context window: --max-model-len 32768. Larger values use more GPU memory for KV cache. Set this to the maximum context your application actually needs, not the model's maximum capability, to leave more memory for concurrent request batching.
GPU memory utilization controls what fraction of GPU memory vLLM reserves: --gpu-memory-utilization 0.90. The default 0.90 leaves 10% as buffer. Increase to 0.95 for maximum capacity on dedicated inference machines, or decrease if other processes share the GPU.
Max concurrent requests controls how many requests vLLM processes simultaneously. vLLM handles this dynamically by default, but you can set an upper bound with --max-num-seqs 256. Higher values increase throughput but also increase memory pressure and per-request latency.
Enforce eager mode with --enforce-eager disables CUDA graph optimizations. This uses more memory but avoids rare CUDA graph-related issues. Use for debugging, remove for production performance.
Add Load Balancing
For serving beyond what a single GPU or multi-GPU server handles, run multiple vLLM instances behind a load balancer. Each instance runs on its own GPU or GPU set and handles requests independently.
Use NGINX, HAProxy, or a cloud load balancer (ALB, Cloud Load Balancing) to distribute requests. Round-robin distribution works adequately for uniform request sizes. For workloads with highly variable prompt lengths, least-connections or request-queue-depth-based routing performs better.
Autoscaling based on GPU utilization or request queue depth handles traffic spikes. Monitor the vLLM Prometheus metrics endpoint (/metrics) for queue depth, GPU utilization, and tokens-per-second. Scale up when queue depth exceeds a threshold that would cause unacceptable latency.
Adding a response cache (Redis or a dedicated semantic cache) in front of your vLLM instances reduces GPU load by 30% to 60% for applications with repetitive queries. Exact match caching catches identical requests. Semantic caching catches paraphrased variations.
Monitor and Optimize
vLLM exposes Prometheus-compatible metrics at the /metrics endpoint. Key metrics to monitor:
vllm:num_requests_running: currently processing requests. Consistently at maximum indicates you need more capacity.
vllm:num_requests_waiting: queued requests. Growing queue means throughput is insufficient. Scale up or reduce max_model_len to free memory for more concurrent requests.
vllm:avg_generation_throughput_toks_per_s: output tokens per second across all requests. This is the primary throughput metric.
vllm:gpu_cache_usage_perc: KV cache utilization. Above 95% means the PagedAttention memory manager is running tight. Reduce max_model_len or max_num_seqs.
Common optimization techniques: enable FP8 quantization on Hopper GPUs for a near-free 2x memory reduction. Use AWQ or GPTQ for older GPUs. Reduce max_model_len to the actual maximum your application uses. Enable chunked prefill (--enable-chunked-prefill) for workloads with long prompts to improve time-to-first-token latency.
PagedAttention Explained
PagedAttention is vLLM's core innovation and the reason it achieves dramatically higher throughput than naive serving. During text generation, the model maintains a key-value (KV) cache that stores intermediate attention computations for all previous tokens. This cache grows with context length and must be kept in GPU memory for fast access.
Traditional serving frameworks pre-allocate a fixed block of GPU memory for each request's KV cache based on the maximum possible sequence length. Most of this memory goes unused because actual sequences are shorter than the maximum. When serving multiple concurrent requests, this wasted memory limits how many requests can run simultaneously.
PagedAttention borrows virtual memory concepts from operating systems. Instead of pre-allocating contiguous memory blocks, it allocates small pages on demand and maps them virtually. Memory is only consumed as tokens are generated. Pages from completed requests are immediately recycled. This eliminates memory waste and allows vLLM to serve 2x to 4x more concurrent requests than frameworks using traditional memory management.
vLLM vs Ollama vs SGLang
Use Ollama for development, prototyping, single-user scenarios, and when you need CPU+GPU hybrid inference. Ollama prioritizes simplicity and broad hardware support.
Use vLLM for production serving with multiple concurrent users, when you need maximum throughput per GPU dollar, and when running on NVIDIA GPUs. vLLM prioritizes throughput and efficiency.
Use SGLang for agent-style workloads that require structured generation (JSON output, function calling, constrained decoding). SGLang optimizes the generation process for schema-constrained outputs, reducing latency for agentic AI applications where every model call must produce valid structured data.
These tools are not mutually exclusive. A common architecture uses Ollama for local development, vLLM for production serving, and SGLang for agent-specific endpoints, all running the same underlying model weights.
Integration with Memory Systems
vLLM servers are stateless. Each request is independent, with no built-in conversation history or user context. For applications that need continuity across interactions, your application layer must handle memory retrieval and inject relevant context into each prompt before sending it to vLLM.
This architecture actually simplifies scaling. Because the serving layer has no state, you can add and remove vLLM instances freely without worrying about session affinity. The memory architecture runs as a separate service that all vLLM instances can access.
For RAG applications, the retrieval and generation stages can run on the same GPU or different GPUs depending on your performance requirements. Co-locating the embedding model and the generation model on the same machine eliminates network latency between retrieval and generation.
vLLM is the production standard for serving open source LLMs to multiple concurrent users. Its PagedAttention mechanism delivers 16x throughput over single-request serving. Install it, point it at a model, and scale horizontally with a load balancer. Use FP8 on Hopper GPUs or AWQ on older hardware for the best throughput per dollar.