How to Set Up and Use Ollama for Local LLM Inference
Ollama has become the default entry point for local LLM development, with over 172,000 GitHub stars and broad community adoption. Its appeal is simplicity: one install, one command to download a model, one command to start chatting. The OpenAI-compatible API means existing code that calls GPT-4 or Claude can switch to a local model by changing a single URL.
Install Ollama
macOS: Download from ollama.com and drag the application to your Applications folder. Alternatively, install via Homebrew: brew install ollama. Ollama runs as a menu bar application on macOS and automatically starts the background server.
Linux: Run the official install script: curl -fsSL https://ollama.com/install.sh | sh. This installs Ollama and registers it as a systemd service. The service starts automatically and persists across reboots. For NVIDIA GPUs, ensure CUDA drivers are installed first (version 12.0+ recommended).
Windows: Download the installer from ollama.com and run it. Ollama installs as a Windows service. NVIDIA GPU support requires up-to-date GeForce or Studio drivers.
Docker: For containerized deployments, use the official Docker image: docker run -d --gpus all -p 11434:11434 ollama/ollama. The --gpus all flag passes NVIDIA GPUs to the container.
Verify the installation: ollama --version should return the version number. Test the server: curl http://localhost:11434/ should return "Ollama is running".
Pull and Manage Models
Download models from Ollama's model library with ollama pull. The library hosts models in various sizes, pre-quantized to Q4_K_M by default:
ollama pull qwen3:8b downloads Qwen 3 at 8 billion parameters (approximately 4.9GB). ollama pull llama4:70b downloads Llama 4 at 70 billion parameters (approximately 43GB). ollama pull phi4 downloads Phi-4 at 3.8 billion parameters (approximately 2.3GB).
Manage your local model library with these commands:
ollama list shows all downloaded models with their sizes and modification dates. ollama show qwen3:8b displays model details including architecture, parameter count, quantization level, and context length. ollama rm qwen3:8b deletes a model to free disk space. ollama cp qwen3:8b my-custom-model copies a model as a starting point for customization.
Models are stored in ~/.ollama/models/ on macOS and Linux, and C:\Users\USERNAME\.ollama\models\ on Windows. The directory can grow large if you download many models, so monitor your available disk space.
Create Custom Modelfiles
A Modelfile is Ollama's equivalent of a Dockerfile. It defines a custom model configuration: which base model to use, what system prompt to set, and what parameters to override. Create a file named Modelfile (no extension) with the following structure:
FROM qwen3:8b
SYSTEM "You are a technical documentation assistant. Respond with precise, factual answers. Always cite specific version numbers and configuration values."
PARAMETER temperature 0.2
PARAMETER num_ctx 16384
Build the custom model: ollama create my-docs-assistant -f Modelfile. Run it: ollama run my-docs-assistant. The custom model appears in ollama list alongside downloaded models.
Useful Modelfile parameters include: temperature (0.0 to 2.0, controls randomness), num_ctx (context window size in tokens), top_p (nucleus sampling threshold), top_k (limits token selection pool), repeat_penalty (reduces repetition, 1.0 is no penalty), and num_predict (maximum tokens to generate, -1 for unlimited).
For importing custom GGUF models not in the Ollama library, create a Modelfile with FROM /path/to/your/model.gguf and build it. This lets you use any GGUF model from Hugging Face with Ollama's interface.
Use the API
Ollama runs an HTTP server on port 11434 that accepts requests in two formats: its native API and an OpenAI-compatible API.
The OpenAI-compatible endpoint at /v1/chat/completions is the recommended integration point. Any code or library that works with the OpenAI API works with Ollama by changing the base URL:
Python with the OpenAI SDK:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
response = client.chat.completions.create(model="qwen3:8b", messages=[{"role": "user", "content": "Explain embeddings"}])
The API key value does not matter (Ollama has no authentication), but the SDK requires the parameter to be present. Set it to any string.
The native API at /api/chat and /api/generate provides additional Ollama-specific features like streaming with context management, embedding generation via /api/embed, and model management endpoints. Use the native API when you need Ollama-specific features. Use the OpenAI-compatible API when you want portability between local and cloud models.
Both endpoints support streaming responses. For the OpenAI-compatible endpoint, set "stream": true in the request body. The server sends tokens as server-sent events, which most HTTP client libraries handle natively.
Configure Advanced Settings
Ollama's behavior is controlled through environment variables. Set these in your shell profile, systemd service file, or Docker compose configuration.
OLLAMA_NUM_PARALLEL controls how many requests the server handles concurrently. The default is 1, meaning requests queue. Set it higher for applications serving multiple users, but each concurrent request multiplies memory usage. A value of 4 with a 7B model needs roughly 4x the VRAM.
OLLAMA_MAX_LOADED_MODELS controls how many models stay loaded in memory simultaneously. The default is 1, meaning switching models requires unloading the current one and loading the new one (a few seconds of latency). Increase this if your application uses multiple models and you have sufficient VRAM.
OLLAMA_HOST changes the bind address. The default 127.0.0.1:11434 only accepts local connections. Setting it to 0.0.0.0:11434 allows connections from other machines on the network. Never do this without a reverse proxy and authentication in front, as the Ollama API has no built-in access control.
OLLAMA_MODELS changes the model storage directory. Useful when your home directory is on a small SSD but you have a larger drive available for model files.
CUDA_VISIBLE_DEVICES controls which GPUs Ollama uses on multi-GPU systems. Set to "0" for the first GPU only, "0,1" for the first two, etc.
Ollama with Agent Frameworks
Ollama integrates with all major agent and RAG frameworks through its OpenAI-compatible API. In LangChain, replace ChatOpenAI with ChatOllama or use ChatOpenAI(base_url="http://localhost:11434/v1"). In LlamaIndex, set the LLM to use the Ollama provider. In any framework that accepts an OpenAI-compatible endpoint, pointing at localhost:11434 works without additional configuration.
For agentic AI applications that require function calling, Ollama supports tool use with models that have been trained for it (Qwen 3, Llama 4, and Mistral all support function calling through Ollama). The tool definitions follow the same JSON schema format used by the OpenAI API.
When building applications that need persistent memory across sessions, Ollama itself provides no state management. Each API call is independent. Your application needs to implement memory retrieval and inject relevant context into each prompt. This is the same pattern used with cloud APIs, the only difference is the endpoint URL.
Performance Tuning
Token generation speed depends on model size, quantization level, hardware, and context length. Typical throughput on common hardware configurations:
A 7B model at Q4 quantization generates approximately 40 to 80 tokens per second on an RTX 4090, 30 to 50 on an M4 Pro, and 5 to 10 on CPU only. A 70B model at Q4 generates approximately 15 to 25 tokens per second on an A100 80GB, 12 to 18 on an M4 Max with 128GB, and is impractical on CPU.
If performance is below expectations, check these common bottlenecks: insufficient GPU layer offloading (verify with nvidia-smi that GPU utilization is high during generation), context window set larger than necessary (larger contexts slow generation), and thermal throttling on laptops (extended inference sessions can trigger thermal limits on consumer hardware).
For applications requiring higher throughput or concurrent user support beyond what Ollama handles, consider vLLM, which provides continuous batching and PagedAttention for production-grade serving.
Ollama provides the fastest path from zero to local LLM inference. Install it, pull a model, and start generating. Use Modelfiles for custom configurations, the OpenAI-compatible API for application integration, and environment variables for advanced tuning. It handles the complexity of llama.cpp so you can focus on building your application.