How to Find and Fix LLM Latency Bottlenecks
LLM latency is fundamentally different from traditional API latency, and misunderstanding the difference leads teams to waste effort on the wrong optimizations. In a traditional web application, latency is dominated by database queries, network calls, and compute, all of which can be improved by scaling infrastructure. In an LLM application, the dominant latency component is usually the model's generation time, which is proportional to the output token count and controlled by the model provider's infrastructure, not yours. Adding more servers to your application does not make the model generate faster. This means LLM latency optimization is primarily about reducing how much work you ask the model to do and structuring your pipeline so that the remaining work happens as efficiently as possible.
Step 1: Instrument Per-Span Timing Across the Pipeline
Before you can fix a latency problem, you need to see where the time goes. Create a separate observability span for each stage of your pipeline and record the start time, end time, and derived duration on each. A typical LLM pipeline has these stages, each with its own latency characteristics:
Query processing (1 to 50 milliseconds): parsing the user input, rewriting the query for retrieval, classifying the intent. This is rarely the bottleneck but should still be instrumented to confirm.
Embedding (10 to 100 milliseconds): converting the query text into a vector for retrieval. Latency depends on the embedding model and provider. OpenAI's text-embedding-3-small typically returns in 20 to 40 milliseconds for short queries. Local embedding models can be faster (5 to 15 milliseconds) but require GPU infrastructure.
Retrieval (20 to 500 milliseconds): searching the vector database for relevant documents. Latency depends on the database, index size, number of results requested, and whether filters are applied. Pinecone and managed databases typically return in 30 to 80 milliseconds for indexes under 10 million vectors. Self-hosted databases can be faster or slower depending on configuration. Multiple retrieval calls (searching different collections or doing hybrid search) multiply this time.
Re-ranking (50 to 300 milliseconds): if your pipeline includes a cross-encoder re-ranker, it adds significant latency because it processes each candidate document against the query. Cohere Rerank typically takes 100 to 200 milliseconds for 10 to 25 candidates.
Prompt assembly (1 to 10 milliseconds): constructing the final prompt from the template, retrieved context, conversation history, and system instructions. Rarely a bottleneck.
Generation (200 milliseconds to 30 seconds): the LLM call itself. This is subdivided into time-to-first-token (TTFT), which is the model's pre-fill time processing the input prompt, and inter-token latency, which is the time between successive output tokens. TTFT scales with input length: a 2,000-token prompt might have a 200-millisecond TTFT, while a 50,000-token prompt might have a 2-second TTFT. Total generation time scales with output length at the model's tokens-per-second rate (typically 50 to 150 tokens per second for cloud-hosted models).
Tool execution (variable): if the model makes tool calls, each tool's execution time adds to the total. External API calls can add hundreds of milliseconds or seconds. Database queries are typically faster. Multiple tool calls in sequence multiply the delay.
Post-processing (1 to 50 milliseconds): parsing the model output, applying formatting, running safety checks. Rarely the bottleneck unless the safety check calls another model.
Step 2: Identify the Bottleneck Component
With per-span timing in place, build a stacked latency chart that shows the average and p95 time distribution across pipeline stages. This chart immediately reveals the bottleneck. In most LLM applications, the distribution falls into one of three patterns.
Generation-dominated (most common): the generation span consumes 50 to 80 percent of total request time. This is the default pattern for applications with moderate prompt lengths and no complex retrieval. Optimization targets are prompt length reduction, prompt caching, model speed, and streaming.
Retrieval-dominated: the retrieval span (including embedding and re-ranking) consumes more than 30 percent of total request time. This happens when the pipeline makes multiple retrieval calls, when the vector database is slow or distant, or when a re-ranker adds significant overhead. Optimization targets are retrieval architecture, database performance, and caching.
Tool-dominated: tool-call spans consume significant time because external APIs are slow or unreliable. This is common in agentic applications where the model calls multiple tools sequentially. Optimization targets are tool response time, parallelizing independent tool calls, and caching tool results.
Also look at the latency distribution, not just the average. If the p95 latency is three to five times the p50, there is a long tail caused by specific conditions: certain query types that trigger more retrieval, certain model outputs that are much longer than typical, or intermittent slow responses from external services. The long tail often reveals the most impactful optimization opportunities because it affects the worst user experience.
Step 3: Optimize the Generation Step
If generation is the bottleneck, four techniques consistently reduce latency.
Reduce prompt length. Every token in the prompt adds to the model's pre-fill time (TTFT). The most common source of prompt bloat is retrieved context: injecting ten document chunks when three would suffice, or including full documents when summaries would work. Reducing retrieved context from 10 chunks to 5 typically cuts input tokens by 30 to 50 percent, which reduces TTFT proportionally. Also audit your system prompt for instructions that accumulated over time without pruning. Many production system prompts contain redundant or contradictory instructions that add tokens without improving output quality.
Enable prompt caching. Anthropic, OpenAI, and Google all offer prompt caching where the provider caches the pre-processed representation of the prompt prefix (system instructions, few-shot examples, tool definitions) and reuses it on subsequent requests. This can reduce TTFT by 50 to 85 percent for the cached portion. The requirement is that the cached portion must be identical across requests, so structure your prompt with the static content first and the variable content (user query, retrieved context) last. Monitor the cache hit rate to confirm the caching is working; a low hit rate means the "static" prefix is varying between requests.
Use a faster model for eligible tasks. Smaller models generate tokens faster and have lower TTFT. GPT-4o mini is roughly two to three times faster than GPT-4o for the same output. Claude Haiku 3.5 is significantly faster than Claude Sonnet 4. If your evaluation shows that the smaller model handles a specific task adequately, switching models is the highest-impact latency reduction available. Many production systems route different task types to different models: simple classification and extraction to a fast model, complex reasoning and generation to a capable model.
Set appropriate max_tokens. If your application expects short responses (a classification label, a brief answer, a JSON object), set max_tokens to a value slightly above the expected output length. This prevents the model from generating unnecessarily long responses, which directly reduces generation time. Without a max_tokens limit, a model can sometimes produce verbose outputs that take seconds to generate when a concise answer would have taken a fraction of a second.
Step 4: Optimize the Retrieval Step
If retrieval is the bottleneck, four techniques apply.
Reduce round trips. If your pipeline makes multiple sequential retrieval calls (search collection A, then search collection B, then search collection C), each call adds its latency in series. Consolidate into a single collection where possible, or parallelize independent searches so they run concurrently. Switching three sequential 80-millisecond searches to three parallel searches reduces retrieval time from 240 milliseconds to 80 milliseconds.
Tune the approximate nearest neighbor index. Vector databases use ANN algorithms (HNSW, IVF, ScaNN) that trade recall for speed. If retrieval latency is high, you may have the index configured for maximum recall at the cost of speed. Reducing the HNSW ef_search parameter (from 200 to 100, for example) or the IVF nprobe parameter can cut retrieval latency significantly with a modest recall reduction that may not affect end-to-end quality. Run your evaluation suite after the change to verify.
Batch embedding calls. If you embed multiple queries per request (for hybrid search or multi-collection search), batch them into a single embedding API call rather than making separate calls. The embedding API processes a batch in roughly the same time as a single item, so batching N queries cuts embedding latency from N * single_latency to approximately single_latency.
Cache frequent queries. In many applications, a significant fraction of queries are repeated or very similar. Caching retrieval results for exact query matches eliminates the embedding and search latency entirely for cache hits. Even a small cache (10,000 entries with a 1-hour TTL) can provide a 20 to 40 percent cache hit rate in applications with repetitive query patterns, reducing average retrieval latency proportionally.
Step 5: Optimize Pipeline Structure
Pipeline-level optimizations reduce latency by changing how operations are orchestrated rather than making individual operations faster.
Parallelize independent operations. If your pipeline embeds the query and classifies the intent before retrieval, and these two operations are independent, run them concurrently rather than sequentially. In Python, use asyncio.gather() or concurrent.futures. The total time becomes the maximum of the two operations rather than the sum. Identify all pairs of operations in your pipeline that do not have a data dependency and run them in parallel.
Eliminate unnecessary sequential steps. Some pipelines include steps that were added during prototyping and never removed: a summarization step that compresses retrieved context (adding a second model call), a validation step that checks the output with another model call, or a translation step for content that is already in the target language. Each unnecessary step adds its full latency to the pipeline. Audit your pipeline for steps that can be removed, combined, or made conditional.
Implement streaming. Streaming does not reduce total generation time, but it dramatically reduces perceived latency by showing the first tokens to the user while the model is still generating. A response that takes 3 seconds to generate fully can show the first tokens in 200 milliseconds with streaming, which feels nearly instant. For conversational applications, streaming is the single most impactful latency improvement from the user's perspective. All major model providers support streaming, and most LLM frameworks provide streaming utilities.
Consider speculative execution for multi-path pipelines. If your application decides between different processing paths based on the user's query (for example, answering from a knowledge base versus performing a calculation), and the classification step takes significant time, you can speculatively start both paths in parallel and cancel the one that is not needed once classification completes. This trades compute cost for latency, which is appropriate when the classification step is a meaningful fraction of total time and the speculative execution cost is low.
Step 6: Monitor Improvements and Prevent Regression
After applying optimizations, set latency baselines and alerts so the improvements stick and new regressions are caught immediately.
Record the p50, p95, and p99 latency for each pipeline stage as your new baseline. Set alerts that trigger when any stage's p95 latency exceeds 150 percent of the baseline for a 15-minute window. This catches both infrastructure issues (a slow database, a provider outage) and code changes that inadvertently add latency (a new retrieval step, an expanded prompt, an unoptimized tool call).
Track latency trends over time on a dashboard with daily or weekly granularity. Latency tends to creep upward as features are added and prompts grow, and a trending view catches the slow creep that individual alerts miss. Review the trend monthly and investigate any sustained upward movement.
When deploying changes that might affect latency (prompt modifications, model switches, retrieval configuration changes), compare the before and after latency distributions explicitly rather than relying on gut feeling. Your observability platform should support this comparison either through trace filtering by deploy version or through A/B analysis tools. A change that improves average latency but increases p99 latency is not a clear win and deserves scrutiny.
LLM latency optimization starts with measurement. Instrument per-span timing, build the stacked breakdown, and the bottleneck will be obvious. Most pipelines are generation-bound, where prompt caching, model routing, and prompt shortening deliver the largest gains. Streaming reduces perceived latency independently of actual generation time and should be enabled in every conversational application.