How to Self-Host a Private LLM for Your Organization
The motivations for self-hosting fall into three categories. Regulatory compliance mandates data residency for healthcare (HIPAA), finance (SOC 2), legal (attorney-client privilege), and government (FedRAMP) organizations. Cost optimization makes self-hosting cheaper than API calls above roughly 5 million tokens per day. Control requirements demand pinned model versions, zero downtime dependency on external services, and the ability to customize model behavior through fine-tuning.
Define Requirements
Before selecting hardware or models, answer four questions that determine every subsequent decision:
What are the use cases? Internal chatbot for employees, customer-facing support, code generation, document processing, or a combination? Each use case has different quality thresholds, latency requirements, and model size preferences.
How many concurrent users? A team of 10 developers using an internal coding assistant has fundamentally different infrastructure needs than a customer support bot serving 1,000 simultaneous conversations. Plan for peak load, not average load.
What latency is acceptable? Interactive chat needs sub-2-second time-to-first-token. Batch document processing can tolerate minutes. Latency requirements determine whether you need the fastest GPUs or can use cost-effective alternatives.
What compliance frameworks apply? HIPAA requires specific access controls, audit logging, and data encryption. GDPR requires data residency in specific regions and the ability to delete user data on request. SOC 2 requires documented security controls and regular audits. Identify these requirements early because they constrain hardware and deployment choices.
Select Hardware
GPU selection is the most impactful hardware decision. The calculation starts with model memory requirements (see our GPU requirements guide) and adds memory for the KV cache that grows with context length and concurrent users.
For small teams (under 20 users): A single NVIDIA A6000 (48GB) or RTX 5090 (32GB) runs a 70B model at Q4 quantization with room for 4 to 8 concurrent requests. Total hardware cost: $4,000 to $10,000. Monthly power and cooling: roughly $50 to $100.
For medium deployments (20 to 200 users): Two A100 80GB GPUs with tensor parallelism handle a full-precision 70B model or a quantized model with extensive KV cache for concurrent requests. Total hardware cost: $20,000 to $40,000 if purchased, or $1,200 to $3,000 per month if rented.
For large deployments (200+ users): Multiple A100 or H100 GPU servers behind a load balancer, with autoscaling based on demand. Cloud GPU rental through services like Vast.ai provides flexibility without capital commitment. Budget $5,000 to $20,000 per month depending on scale and model size.
Non-GPU considerations: fast NVMe storage for model loading (models can be 40GB+ and you want cold-start times under 60 seconds), ECC RAM for reliability, and redundant power for on-premises deployments.
Choose a Model
Model selection for self-hosting prioritizes three factors beyond raw quality: licensing, efficiency, and community support.
Licensing for commercial use: MIT-licensed models (GLM-5.2, DeepSeek V4 Pro, Phi-4) have zero restrictions. Apache 2.0 models (Qwen 3) are similarly unrestricted. Llama 4's community license works for most businesses but includes a revenue threshold. Have your legal team review the specific license before deploying.
Efficiency for your hardware: Mixture-of-experts models (Qwen 3 235B, Mistral Large 2) activate fewer parameters per token, offering better quality-per-FLOP but requiring the full parameter set in memory. Dense models (Llama 4, GLM-5.2) have simpler memory requirements but use all parameters on every token.
Community support: Models with large communities have more documentation, more fine-tuned variants, and faster bug fixes. Llama and Qwen have the largest communities. Practical tip: choose a model that has a well-tested GGUF or AWQ quantized version on Hugging Face rather than quantizing yourself.
Deploy the Serving Stack
The minimal production stack consists of three components: the inference server (vLLM or Ollama), a reverse proxy (NGINX or Caddy), and monitoring (Prometheus + Grafana).
Inference server: Use vLLM for multi-user production serving. Use Ollama for single-user or small-team deployments where simplicity matters more than throughput. Both expose OpenAI-compatible APIs, so your application code works identically with either.
Reverse proxy: Neither vLLM nor Ollama includes authentication. Place NGINX or Caddy in front to handle TLS termination (HTTPS), API key authentication, rate limiting, and request logging. A basic NGINX configuration validates an API key header, proxies to the inference server, and logs requests for audit purposes.
Monitoring: vLLM exports Prometheus metrics natively. Ollama requires a custom metrics exporter. Track GPU utilization, request queue depth, tokens per second, error rates, and P99 latency. Set alerts for GPU memory exhaustion, sustained queue growth, and error rate spikes.
Deploy using Docker Compose for single-server setups or Kubernetes for multi-server deployments. Container orchestration simplifies model updates (rolling deployment with zero downtime), autoscaling, and failure recovery.
Secure the Deployment
Network isolation: The inference server should not be directly accessible from the internet. Place it on a private network segment behind the reverse proxy. For on-premises deployments, use VLAN segmentation. For cloud deployments, use private subnets with no public IP.
Authentication: Implement API key validation or OAuth2 token verification at the reverse proxy layer. Issue unique API keys per application or team to enable per-consumer monitoring and revocation.
Audit logging: Log every request with timestamp, authenticated identity, input token count, and output token count. For compliance-sensitive deployments, log full prompts and responses to a secure, tamper-evident log store. Retention policies should match your compliance requirements.
Data encryption: Enable TLS for all network communication. For HIPAA and similar requirements, encrypt model weights and cached data at rest using full-disk encryption or file-system-level encryption.
Access controls: Restrict who can load or change models, who can access the monitoring dashboard, and who can view audit logs. Role-based access control at the reverse proxy and monitoring layers prevents unauthorized model swaps or log tampering.
For organizations subject to GDPR or similar privacy regulations, ensure that the memory system storing user conversation histories supports individual data deletion requests. The model weights themselves do not contain user data, but any memory, caching, or logging layer does.
Ongoing Operations
Model updates: Unlike API services that update models automatically, self-hosted models stay at the version you deployed. This is both an advantage (no surprise behavior changes) and a responsibility (you must proactively evaluate and deploy newer models). Establish a quarterly model evaluation cycle: benchmark new releases against your current model on your specific use case data, and upgrade when improvements justify the effort.
Capacity planning: Track request volume trends and GPU utilization over time. Plan for 30% headroom above peak usage to handle traffic spikes without queuing. Add capacity before utilization consistently exceeds 70%, as GPU memory pressure increases non-linearly with concurrent requests.
Failover: For applications that cannot tolerate downtime, run at least two inference servers with the load balancer automatically routing around failures. Health check endpoints on each server detect GPU errors, model loading failures, and out-of-memory conditions.
Cost monitoring: Track compute costs, power consumption, and staff time spent on operations. Compare regularly against the equivalent API cost to validate that self-hosting remains economical. The breakeven analysis shifts as API prices change and newer, more efficient models become available. See our cost optimization guide for frameworks to evaluate this trade-off.
Self-hosting a private LLM requires upfront infrastructure work but gives you complete data privacy, predictable costs at scale, and full control over model behavior. Start with vLLM behind NGINX, add monitoring from day one, and plan capacity with 30% headroom. The total cost of ownership drops below API pricing at roughly 5 million tokens per day.