How to Track and Reduce LLM API Costs in Production

Step 1: Capture Token Counts on Every LLM Call

Every model API response includes the token usage in its response metadata. OpenAI returns it in the usage field (prompt_tokens, completion_tokens, total_tokens). Anthropic returns it in the usage field (input_tokens, output_tokens). Google returns it in usageMetadata. AWS Bedrock returns it in the metrics field. Your instrumentation code should extract these values and record them as attributes on the generation span.

Record four token values for each generation call: input tokens (the total tokens in the prompt, including system message, user message, and any injected context), output tokens (the tokens generated in the response), cached tokens (input tokens that were served from the provider's prompt cache, which are priced at a discount), and total tokens (input plus output). Also record the model identifier (the specific model version, not just the model family, because pricing varies by version) and the finish reason (which tells you whether the output was truncated by the max_tokens limit).

For applications that make multiple LLM calls per request (a classification call followed by a generation call, or an agent that makes several calls in a loop), each call should be its own span with its own token counts. The total request cost is the sum of all generation spans in the trace. This per-call granularity is essential because it reveals which calls in the pipeline are the most expensive and which are candidates for optimization.

If you use Helicone's proxy, token capture is automatic because the proxy parses every API response. If you use LangSmith, Langfuse, or Arize Phoenix, their auto-instrumentation libraries handle token extraction for supported providers. For custom instrumentation, add the token extraction after every model call in your application code.

Step 2: Compute Per-Request Cost from Model Pricing

Token counts become actionable when converted to dollar costs. Each model has a specific price per million input tokens and per million output tokens, and these prices differ between models, between providers, and between standard and cached tokens. As of mid-2026, representative pricing includes: GPT-4o at $2.50 per million input tokens and $10.00 per million output tokens, GPT-4o mini at $0.15 per million input and $0.60 per million output, Claude Sonnet 4 at $3.00 per million input and $15.00 per million output, Claude Haiku 3.5 at $0.80 per million input and $4.00 per million output. Cached input tokens typically cost 50 to 90 percent less than standard input tokens.

Your cost computation logic should maintain a pricing table mapping model identifiers to per-token prices, and update this table when providers change pricing (which happens several times per year). The formula is straightforward: cost = (input_tokens * input_price / 1,000,000) + (output_tokens * output_price / 1,000,000). For cached tokens, replace the standard input price with the cached price. Most observability platforms include built-in pricing tables that they update automatically, so you only need to manage custom pricing if you use models that the platform does not recognize.

Attach the computed dollar cost as an attribute on each generation span. This makes cost visible in every trace, so when you investigate a specific request, you can immediately see how much it cost and which call was most expensive. The per-request cost also feeds into the aggregation step, where it becomes the basis for dashboards and alerts.

Step 3: Aggregate Cost by Business Dimensions

Per-request cost data becomes strategically useful when aggregated by the dimensions that map to business decisions. The aggregations that consistently reveal optimization opportunities are cost by model, cost by feature, cost by user segment, and cost by prompt template.

Cost by model shows how much you spend on each model across all features. This is the fastest way to identify cases where an expensive model is being used for a task that a cheaper model could handle. If 40 percent of your spend goes to GPT-4o and half of those calls are classification tasks that GPT-4o mini handles equally well, the savings opportunity is obvious and quantifiable from the dashboard.

Cost by feature shows how much each product feature costs in LLM API spend. This maps directly to product economics: if your document Q&A feature costs $0.08 per query and your chat feature costs $0.02 per message, you know the document Q&A feature needs cost optimization more urgently. It also enables cost attribution to product teams, which creates accountability and motivation to optimize.

Cost by user segment reveals whether specific user cohorts are disproportionately expensive. Enterprise users with long conversation histories, power users who trigger complex agent workflows, or users in specific regions with different query patterns can drive costs that are invisible in aggregate metrics. This data informs pricing decisions, usage limits, and targeted optimizations.

Cost by prompt template shows how different prompt versions affect spending. When you iterate on a prompt (adding instructions, changing the format, including more examples), the token count changes, and the cost changes with it. Tracking cost by prompt version lets you see the cost impact of prompt changes alongside their quality impact from evaluation, so you can make informed trade-offs between quality and cost.

Step 4: Set Up Cost Anomaly Detection

Cost anomalies in LLM applications are usually engineering bugs, not billing problems. A prompt template change that accidentally doubled the context length, a retrieval pipeline that started returning all documents instead of the top five, a retry loop that makes the same API call repeatedly, or a new feature that went to production without cost testing can all cause sudden cost spikes. Anomaly detection catches these before they accumulate into a large bill.

Set two types of cost alerts. The first is a per-request cost alert that triggers when any individual request costs more than a threshold (for example, $1.00 per request). This catches runaway requests caused by bugs like infinite tool-call loops, extremely long context injection, or accidental use of the wrong model. The second is a daily cost rate alert that triggers when the trailing 24-hour cost exceeds 150 percent of the 7-day daily average. This catches systemic changes that increase cost across all requests, like a prompt template deployment or a model tier change.

Both alerts should include context from the trace: which model was used, how many tokens were consumed, which feature or endpoint was involved, and a link to the full trace for investigation. Without this context, a cost alert is just a number that requires manual investigation. With context, the alert often points directly at the cause.

Also monitor the ratio of cached tokens to total input tokens if you use prompt caching. A drop in cache hit rate can indicate that the system prompt or few-shot examples changed, breaking the cache prefix. Since cached tokens cost 50 to 90 percent less than standard tokens, a cache break can double input costs overnight. Alert when the cache hit rate drops more than 20 percentage points from its baseline.

Step 5: Identify and Validate Savings Opportunities

With cost data flowing, the optimization process becomes systematic: identify the highest-cost component, form a hypothesis about how to reduce it, validate the hypothesis with evaluation to ensure quality holds, deploy the change, and confirm the savings through monitoring.

The most common optimization opportunities, in order of typical impact, are model downgrading, prompt shortening, retrieval tuning, and prompt caching.

Model downgrading means switching from a larger model to a smaller one for tasks where the smaller model performs adequately. The cost difference between model tiers is typically 5x to 20x (GPT-4o to GPT-4o mini is roughly 17x cheaper per token), which makes this the single largest savings lever. The process is: identify calls that use an expensive model, run your evaluation suite against the same inputs using the cheaper model, compare quality scores, and if the quality difference is within your tolerance, switch. Cost monitoring confirms the savings after deployment.

Prompt shortening means reducing the token count of your prompts without losing instruction quality. Common techniques include removing redundant instructions, replacing verbose examples with concise ones, moving static context to the system prompt for caching, and compressing retrieved context by summarizing or truncating documents. Track input tokens per request before and after the change to verify the reduction, and run evaluation to verify quality is maintained.

Retrieval tuning means adjusting how many documents are retrieved and injected into the context. If your RAG pipeline retrieves ten chunks but the model only uses three to generate the answer, you are paying for seven chunks of unused context. Experiment with reducing top-k from ten to five or three, evaluate whether answer quality changes, and monitor cost per request after the change. In many cases, retrieval of three to five highly relevant chunks produces the same answer quality as ten chunks at 30 to 50 percent lower input cost.

Prompt caching means structuring your prompts so that the static prefix (system instructions, few-shot examples, tool definitions) is identical across requests, allowing the provider to cache and reuse the pre-processed prefix. Anthropic, OpenAI, and Google all offer prompt caching with discounts of 50 to 90 percent on cached tokens. The implementation requires careful prompt template design to maximize the shared prefix, and cost monitoring to confirm the cache is actually being hit.