How to Optimize Prompts for Cost and Speed

Understanding the Cost Structure

LLM API costs are calculated per token, with separate rates for input tokens (what you send) and output tokens (what the model generates). Output tokens are typically 3x to 5x more expensive than input tokens across all major providers. For OpenAI's GPT-4o, input tokens cost $2.50 per million and output tokens cost $10 per million. For Anthropic's Claude Sonnet, input tokens cost $3 per million and output tokens cost $15 per million. These rates mean that both prompt length and response length directly affect your costs.

Latency is primarily driven by output tokens. The time to process input tokens (the "prefill" phase) is fast because it is parallelized. The time to generate output tokens (the "decode" phase) is sequential, with each token generated one at a time. A response that generates 500 tokens takes roughly 5x longer than one that generates 100 tokens. This means that the most effective latency optimization is reducing the number of output tokens.

The cost and latency interaction creates a clear optimization hierarchy. Reducing output tokens gives you both cost savings and latency improvement. Reducing input tokens gives you cost savings but only modest latency improvement. Using a cheaper or faster model gives you cost savings and often latency improvement. Caching gives you cost savings and latency improvement for repeated requests.

Reducing Input Tokens

Audit your prompt for redundancy. Many production prompts grow over time as developers add instructions to fix specific issues. This accretion often produces redundant or contradictory instructions. Review your system prompt periodically and remove instructions that duplicate each other, that no longer apply, or that address edge cases you no longer encounter.

Compress verbose instructions. "You must always ensure that you provide a response that includes specific details and real-world examples to support your analysis" (25 tokens) can be "Include specific details and real examples" (7 tokens). The meaning is identical. The model follows both equally well. This kind of compression across a 1,000-token system prompt can save 200 to 400 tokens per call.

Use shorter, clearer examples. Few-shot examples are often longer than they need to be. If your examples include long input texts, trim them to the minimum that demonstrates the pattern. The model learns the input-to-output mapping from the structure, not from the full content of the example input. A 3-sentence example that demonstrates the classification pattern is as effective as a 3-paragraph example and uses one-third the tokens.

Remove unnecessary context. If your prompt includes retrieved documents or reference material, include only the sections relevant to the current query, not the entire document. This is a context engineering concern that directly affects cost: every irrelevant sentence in the context window costs tokens without improving output quality. Techniques like semantic chunking and targeted retrieval minimize the irrelevant context you include.

Reducing Output Tokens

Set max tokens appropriately. If your task produces a 200-token response, set max tokens to 300 (with buffer) rather than leaving it at the default 4,096. Some pricing models charge for the max tokens allocation, and even when they do not, a tight max tokens limit prevents the model from generating unnecessarily long responses.

Instruct the model to be concise. "Be concise. Limit your response to 2-3 sentences" or "Respond in under 100 words" directly reduces output tokens. Without length constraints, models default to verbose responses because their training data rewards completeness and thoroughness. Explicit brevity instructions override this default.

Use structured output to control response size. A JSON response with defined fields produces a predictable number of tokens. A free-form text response can vary by 5x or more. Structured output gives you both consistency and cost predictability.

Suppress chain-of-thought when you do not need it. If you are using chain-of-thought prompting, the reasoning tokens add to your output cost. For tasks where CoT does not improve accuracy (simple classification, factual lookups, format conversion), removing the "think step by step" instruction eliminates the reasoning tokens and reduces both cost and latency.

Prompt Caching

Prompt caching allows you to avoid re-processing the static parts of your prompt on every request. If your system prompt, few-shot examples, and instruction set total 2,000 tokens and are identical across requests, caching lets you process them once and reuse the processed representation for subsequent requests.

Anthropic's prompt caching reduces the cost of cached input tokens by 90% (you pay 10% of the normal rate for cached tokens). OpenAI offers similar caching through their Batch API and automatic prefix caching. The savings are dramatic for applications with long system prompts and high request volumes. A system prompt of 3,000 tokens at $3/million tokens costs $0.009 per thousand requests without caching and $0.0009 per thousand requests with caching. The prompt caching guide covers the implementation details for each provider.

To maximize cache hit rates, structure your prompt so that the static content (system instructions, examples) comes first and the dynamic content (user input) comes last. Caching works on prefix matches: the longer the identical prefix across requests, the more tokens get cached. If you interleave static and dynamic content, the cache breaks at the first dynamic token.

Model Routing

The most impactful cost optimization is using the right model for each task. A simple classification task does not need a $15/million-output-token frontier model when a $0.60/million-output-token model handles it with the same accuracy. Model routing assigns each request to the cheapest model that meets the quality requirements for that task type.

The simplest routing strategy is task-based: all classification requests go to the small model, all analysis requests go to the medium model, all complex reasoning requests go to the frontier model. This requires categorizing your tasks in advance but produces consistent, predictable savings.

A more sophisticated approach uses a classifier (often a small model itself) to assess the complexity of each input and route it accordingly. Simple inputs go to the cheap model, complex inputs go to the expensive model. This dynamic routing captures more savings than task-based routing because even within a single task type, some inputs are harder than others. The routing by complexity guide and the broader AI cost optimization pillar cover these patterns in detail.

Batch Processing

If your workload includes requests that do not need real-time responses (processing uploads, generating reports, analyzing datasets), batch processing offers significant cost savings. OpenAI's Batch API offers 50% cost reduction for batch requests with a 24-hour turnaround. Anthropic offers similar batch pricing. The trade-off is latency: batch requests are processed asynchronously and may take hours to complete.

Batch processing works particularly well for: overnight report generation, bulk data extraction, content moderation at scale, embedding generation, and any workload where you are processing many items and do not need the results immediately. Queue the requests, submit them as a batch, and process the results when they arrive.

The Optimization Hierarchy

When optimizing prompts for cost and speed, apply techniques in this order for maximum impact:

1. Use the right model. Moving from a frontier model to an adequate smaller model for a task provides the largest savings, often 80 to 95% cost reduction per call.

2. Reduce output tokens. Conciseness instructions, tight max tokens, and structured output reduce both cost and latency.

3. Enable prompt caching. 90% reduction in cached input token costs for applications with stable system prompts.

4. Reduce input tokens. Compress instructions, trim examples, remove irrelevant context.

5. Use batch processing. 50% cost reduction for non-real-time workloads.

The cumulative effect of applying all five can reduce costs by 90 to 98% compared to naively using a frontier model with verbose prompts for every request. For an application processing 100,000 requests per day, this is the difference between a $3,000 daily API bill and a $100 daily API bill.