How to Run LLMs Locally on Your Own Hardware
Local inference gives you zero-latency networking (no round-trip to a cloud API), complete data privacy (prompts never leave your machine), and predictable costs (no per-token billing). The setup takes less than 10 minutes for a basic configuration. More advanced setups with custom models, GPU tuning, and application integration take an afternoon.
Step 1: Check Your Hardware
The first constraint is memory. LLMs need to fit their weights in either GPU VRAM (for GPU inference) or system RAM (for CPU inference). GPU inference is 5x to 20x faster than CPU, so you want to use your GPU whenever possible.
For a 7B to 8B parameter model at Q4 quantization (the most common starting point), you need approximately 4.5GB of memory. An 8GB GPU (RTX 3060, RTX 4060) handles this comfortably. For a 13B to 14B model, you need about 8GB, requiring a 12GB GPU (RTX 4070) or better. For 70B models, you need roughly 40GB, which means an A100, A6000, or two consumer GPUs.
On Apple Silicon Macs, the unified memory architecture means all system RAM is accessible to the GPU. An M2 Pro with 32GB can run a 30B model, and an M4 Max with 128GB can comfortably run quantized 70B models at 15 to 20 tokens per second.
Check your GPU memory on Linux with nvidia-smi, on Mac with "About This Mac" under Memory, and on Windows through Task Manager's Performance tab under GPU.
If your local hardware is insufficient, cloud GPU rental is the alternative. Services like Vast.ai offer A100 80GB instances starting around $0.80 per hour, giving you access to high-end hardware without a capital expenditure.
Step 2: Install Ollama
Ollama is the simplest path from zero to running a local model. It handles model downloading, GGUF quantization, GPU detection, and API serving in a single binary.
On macOS, download from ollama.com and drag to Applications, or run brew install ollama. On Linux, run the install script: curl -fsSL https://ollama.com/install.sh | sh. On Windows, download the installer from ollama.com. After installation, Ollama runs as a background service. Verify the installation by opening a terminal and running ollama --version.
Ollama automatically detects NVIDIA GPUs on Linux and Windows (requires CUDA drivers), Apple Silicon GPUs on macOS, and falls back to CPU inference if no compatible GPU is found. On Apple Silicon Macs running Ollama 0.19 or later, it automatically uses the MLX backend for optimal Metal GPU performance.
Step 3: Pull a Model
Models are downloaded with the ollama pull command. Choose a model based on your hardware and use case:
For 8GB VRAM or less: ollama pull qwen3:8b (4.9GB download, strong general purpose). Alternative: ollama pull phi4:3.8b (2.3GB, surprisingly capable for its size).
For 12GB to 24GB VRAM: ollama pull qwen3:14b (8.9GB, noticeably better quality than 8B). Alternative: ollama pull llama4:8b (Meta's latest compact model).
For 48GB+ VRAM or Apple Silicon with 64GB+: ollama pull qwen3:72b (43GB, near frontier quality). Alternative: ollama pull deepseek-r1:70b (for reasoning-heavy workloads).
The pull command downloads the model to ~/.ollama/models/ on your machine. Models are downloaded once and reused across sessions. The download size is the quantized weight file size. Ollama defaults to Q4_K_M quantization, which provides the best balance of quality and size.
Step 4: Run Inference
Start an interactive chat session with ollama run qwen3:8b (or whichever model you pulled). Type your prompt and press Enter. The model generates a response locally, with no network calls. Type /bye to exit.
For programmatic access, Ollama exposes an HTTP API at http://localhost:11434. The API is compatible with the OpenAI chat completions format, so existing code using the OpenAI SDK works with a URL change:
curl http://localhost:11434/v1/chat/completions -d '{"model":"qwen3:8b","messages":[{"role":"user","content":"Explain vector search in two sentences."}]}'
In Python, use the OpenAI SDK pointed at your local server:
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
The first request after pulling a model takes a few seconds longer as the model loads into GPU memory. Subsequent requests are fast because the model stays loaded.
Step 5: Configure for Your Use Case
Default settings work for casual experimentation, but production applications benefit from tuning several parameters.
Context size: The default context window in Ollama varies by model (typically 2048 to 8192 tokens). Increase it with ollama run qwen3:8b --ctx-size 32768 for longer conversations or document processing. Larger context windows use more memory, so balance context size against your available VRAM.
System prompts: Set a system prompt to control the model's behavior. In the API, include a message with role "system" as the first message. For interactive sessions, create a Modelfile with a SYSTEM instruction.
Temperature: Lower values (0.1 to 0.3) produce more deterministic, focused outputs. Higher values (0.7 to 1.0) increase creativity and variation. For factual question answering and code generation, use lower temperatures. For creative writing and brainstorming, use higher values.
GPU layers: If a model does not fit entirely in VRAM, Ollama automatically offloads some layers to CPU. You can control this with environment variables. More GPU layers means faster inference but higher VRAM usage.
Step 6: Connect Your Application
With the Ollama server running, any application that can make HTTP requests can use the local model. Common integration patterns:
Python applications: Use the OpenAI Python SDK with base_url="http://localhost:11434/v1". LangChain and LlamaIndex both have Ollama integrations. For RAG pipelines, point the LLM component at your local Ollama instance while keeping everything else (embedding, vector search, retrieval) unchanged.
Web applications: Call the Ollama API from your backend server. Never expose the Ollama port directly to the internet, as it has no authentication. Use a reverse proxy with authentication if external access is needed.
Memory integration: Pair the local model with an external memory layer to maintain conversation state across sessions. The model itself is stateless, so each request needs to include relevant context from your memory system in the prompt.
Agent frameworks: Tools like LangGraph and CrewAI work with local Ollama models the same way they work with cloud APIs. The agent logic, tool calling, and orchestration remain identical. Only the LLM endpoint URL changes.
Using llama.cpp Directly
For more control than Ollama provides, use llama.cpp directly. This gives you access to every configuration parameter, custom sampling strategies, and fine-grained GPU layer allocation.
Clone the repository from GitHub and build it with your GPU backend (CUDA for NVIDIA, Metal for Apple Silicon, Vulkan for AMD). Download a GGUF model file from Hugging Face. Run inference with the llama-cli binary, specifying the model path, context size, GPU layers, and other parameters.
The main reasons to use llama.cpp over Ollama are: running models not in Ollama's library, fine-grained control over sampling parameters, embedding generation with the same model, and running inference as part of a larger C/C++ application. For most developers, Ollama provides a sufficient interface and handles the complexity of llama.cpp configuration internally.
Common Issues and Solutions
Out of memory errors: Reduce context size, use a more aggressively quantized model (Q3_K instead of Q4_K_M), or switch to a smaller model. On NVIDIA GPUs, check that other applications are not consuming VRAM with nvidia-smi.
Slow generation speed: Verify that inference is using your GPU, not falling back to CPU. On NVIDIA, check GPU utilization with nvidia-smi during inference. If utilization is near zero, reinstall CUDA drivers. On Mac, ensure you are running a recent Ollama version that supports MLX acceleration.
Poor quality responses: Try a larger model if your hardware allows it. The quality jump from 8B to 14B is substantial. Also verify you are using at least Q4_K_M quantization, as lower quantization levels (Q2, Q3) degrade quality noticeably.
Install Ollama, pull a model sized for your GPU, and start generating responses locally in under 10 minutes. Use the OpenAI-compatible API at localhost:11434 to integrate with any application. Start with Qwen 3 8B for limited hardware or Qwen 3 72B for high-end GPUs.