The Tool Selection Problem in AI Agents

Why Selection Gets Harder with Scale

Three factors compound as the tool set grows. First, token cost: each tool definition adds 100 to 500 tokens to the model's input. With 50 tools, that is 5,000 to 25,000 tokens consumed by tool definitions alone, leaving less room for conversation history, system instructions, and retrieved context. At current pricing, this overhead adds measurable cost to every API call.

Second, semantic overlap: as you add more tools, the probability increases that two or more tools serve partially overlapping purposes. A search_products tool and a find_product_by_name tool might both be valid for the query "find the Widget Pro." The model must reason about subtle differences between tools to choose correctly, and its accuracy decreases as the number of overlapping options increases.

Third, attention dilution: language models allocate attention across their entire input. When tool definitions take up a large fraction of the input, the model has less effective attention for the user's message, the conversation history, and the system instructions. This can cause the model to miss nuances in the user's request that would have led to better tool selection if the input were less crowded.

The Token Budget Problem

Every tool definition consumes tokens that could be used for other context. A model with a 128K context window might seem like it has unlimited room, but in practice, prompt caching, context assembly, and response quality all benefit from keeping the input lean. An agent with 50 tools averaging 300 tokens per definition spends 15,000 tokens on tool definitions, roughly the equivalent of 10 to 15 pages of conversation history that no longer fits.

The cost impact is direct: more input tokens means higher API costs per call. For an agent handling thousands of conversations daily, the difference between sending 50 tools and sending 8 tools per call translates to meaningful savings. But the accuracy impact is more important than the cost impact, because selection errors cause retries, user frustration, and failed workflows that cost far more than the token difference.

The Ambiguity Problem

Tool descriptions are the model's primary signal for selection decisions, and writing unambiguous descriptions for similar tools is genuinely difficult. Consider an enterprise agent with these tools: get_customer, search_customers, find_customer_by_email, lookup_customer_account. Each has a slightly different purpose (exact ID lookup, criteria search, email lookup, account details), but their descriptions inevitably overlap because they all deal with customer data.

The model resolves ambiguity by reasoning about the descriptions and the user's message, but this reasoning is probabilistic, not deterministic. With well-written descriptions, the model makes the right choice 95% of the time. With vague descriptions, accuracy drops to 70% or lower. And in multi-tool workflows where the wrong choice at step 1 cascades into failures at steps 2 and 3, even a 5% error rate at each step compounds into a 14% workflow failure rate over three steps.

Strategies for Scaling Tool Selection

Schema Consolidation

The first strategy is reducing the number of tools. If several tools serve the same entity type (get, search, update, delete for customers), consider whether some can be merged. A single customer tool with an action parameter trades some schema clarity for a smaller tool set. Alternatively, consolidate tools that are always used together into composite tools: if every order lookup is followed by a status check, a single get_order_with_status tool eliminates a chain link and reduces the tool count.

Dynamic Tool Selection

Instead of passing all tools to every call, select the most relevant tools for each query. Keyword matching, embedding similarity, and intent classification can each narrow a 50-tool set to the 5 to 10 most relevant options. The model sees fewer tools, reducing token cost and ambiguity while improving selection accuracy. See the tool router guide for implementation details.

Hierarchical Tool Organization

Organize tools into categories and route queries to the appropriate category before the model sees individual tools. A first-pass classifier determines that a query is about orders, and the model receives only order-related tools. This two-stage approach scales to hundreds of tools because the first stage is a lightweight classification problem and the second stage always sees a manageable tool set.

Memory-Informed Selection

For returning users, past tool usage patterns provide strong signals about which tools are likely to be needed. If a user consistently asks questions that require the order lookup and shipping status tools, memory can boost those tools in the selection ranking. Adaptive Recall supports this pattern through cognitive scoring: tool usage memories with high recency and frequency scores surface first when the system queries for relevant tool history, enabling memory-informed pre-selection that improves with every interaction.

Measuring Selection Quality

Track tool selection accuracy by logging every tool call and whether it succeeded or was immediately followed by a different tool call (indicating the first selection was wrong). Compute accuracy as the percentage of tool calls that were correct on the first attempt. Also track the number of tool calls per conversation as a proxy for efficiency: an increasing average suggests the model is making more selection errors or using tools unnecessarily.

Break down selection accuracy by tool to identify which tools are confused most often. If search_products and get_product_details are frequently swapped, their descriptions need better disambiguation. If a rarely used tool is never called when it should be, its description may not clearly communicate when it is the right choice.

Build a confusion matrix that tracks which tools are selected when other tools were correct. This matrix reveals the specific pairs of tools that the model conflates, which is far more actionable than an aggregate accuracy number. If tool A and tool B are confused 30% of the time, you know exactly which two descriptions need sharper differentiation. If tool C is never selected (all queries that should go to C end up at D), tool C's description may not cover the use cases that trigger it.

Monitor these metrics continuously, not just at launch. Selection accuracy changes as user behavior evolves, as new tools are added, and as the underlying model is updated. A schema set that achieved 97% accuracy on one model version may drop to 92% when the provider releases a new model, because different model versions interpret descriptions differently. Regular monitoring catches these regressions before they impact user experience.