# Your AI Agent Picks the Wrong Tool. Here's How to Measure It and Fix It.

Your agent has a tool that's worked for months. You add a new one, test it, ship it. A week later the **old** tool starts getting skipped on requests it used to handle, and nothing in your logs says so.

That's not a model bug. It's a tool selection bug, and it comes from tools that look alike, not from how many tools you have.

**TL;DR**

*   The model picks a tool by matching the request against your tool names and descriptions, fresh on every turn.
    
*   Count drives the **token bill**. Overlap drives **wrong tool calls**. Different causes, different fixes.
    
*   Wrong tool calls rarely error. The wrong tool returns real data and the model answers confidently.
    
*   Tool search (deferred loading) cuts cost a lot, but look-alike tools still come back together in the short list.
    
*   Measure collisions with **routing probes**, fix them in a specific order, and re-run the probes on every tool change.
    

![](https://cdn.hashnode.com/uploads/covers/6a44b5d24b41ab0145e5cf63/c2b4acf7-9a3b-43e2-81f6-531e47e9eb73.png align="center")

## How an LLM chooses a tool

When you use function calling or MCP, the model never executes anything itself. Every request ships the full list of tool definitions (name, description, JSON schema), and the model emits a call to the one that best fits. If that's new to you, the [tool calling and MCP explainer](https://simplyexplained.hashnode.dev/how-ai-learned-to-actually-do-things-tool-calling-and-mcp-explained) covers the loop.

The useful mental model is a multiple-choice exam:

*   the user's request is the question
    
*   every tool is an answer option
    
*   the model answers from your descriptions alone, every turn
    

Exams don't get hard because of option count. They get hard because of **distractors**: wrong answers that look right. A tool list of 12 unrelated tools is easy. A list of 3 where two overlap is hard.

So adding a tool is not a local change. It adds a new option to every question your agent already answers. That's why the regression shows up in the old feature: the new tool got tested, the old one didn't, and only its competition changed.

## Why wrong tool calls are silent

Suppose an agent has `search_orders` and you add `search_tickets` (past support conversations). The request "Where is my refund for order 4471?" now has two plausible homes. If the model picks `search_tickets`, it gets a real ticket that mentions a refund, and writes a fluent answer about the wrong refund.

HTTP 200. Normal latency. No exception. In multi-step agents, the next step builds on that result, which is how [small errors compound into agent failures](https://simplyexplained.hashnode.dev/why-ai-agents-fail).

This matches what model providers report. Anthropic's [advanced tool use post](https://www.anthropic.com/engineering/advanced-tool-use) names wrong tool selection and incorrect parameters as the most common tool failures, especially with similar names like `notification-send-user` vs `notification-send-channel`.

## The two costs of a tool list

|  | Token bill | Confusion bill |
| --- | --- | --- |
| Driven by | Number and size of definitions | Overlap between definitions |
| Symptom | Slow, expensive requests; cache misses | Wrong tool calls that look successful |
| Main fixes | Scope, defer, trim | Merge, draw boundaries, namespace |

On the token side, Anthropic's example of five MCP servers comes to 58 tools and about 55K tokens of definitions before the conversation starts; they report seeing 134K internally before optimization. Tool definitions also sit at the front of the cache prefix: the [prompt caching docs](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) say modifying tool definitions invalidates the entire cache (same mechanism as in [why trimming your prompt can raise your AI bill](https://simplyexplained.hashnode.dev/trimming-prompt-raises-ai-bill)).

OpenAI's [function calling guide](https://developers.openai.com/docs/guides/function-calling) gives a soft suggestion of fewer than 20 functions available at the start of a turn. That's a sensible ceiling for cost. It says nothing about which 20.

## Measure it: routing probes and a confusion table

### Step 1: find suspect pairs

List tool pairs that share a verb or noun (`search_*`, `get_*`, `send_*`, `*_user`), plus any catch-all like `run_sql` or `http_request`. Common sources of look-alikes:

*   two MCP servers exposing the same verb (`search_issues` on GitHub and Jira)
    
*   a REST API mirrored one-to-one (`get_user`, `find_user`, `list_users`)
    
*   a `_v2` next to the original
    
*   descriptions written for insiders ("Searches records")
    

### Step 2: write probes

Three per suspect tool: plain, paraphrased, and **border** (deliberately close to the look-alike).

```yaml
# probes/routing.yaml
- tool: search_orders
  probes:
    - "Where is order 4471?"
    - "Has the thing I bought last week shipped yet?"
    - "I was promised a refund on order 4471. Where is it?"   # border
- tool: search_tickets
  probes:
    - "What did support tell me last time?"
    - "I spoke to someone about this already, what did they say?"
    - "I complained about a refund last month. Any update?"   # border
```

### Step 3: run them several times and tabulate

Tool choice varies between runs, so sample each probe more than once. `pick_tool` is your own thin wrapper: one model call with the real tool list, returning the chosen tool name (or `None`).

```python
from collections import Counter, defaultdict
import yaml

def confusion_table(probe_file, pick_tool, runs=5):
    suites = yaml.safe_load(open(probe_file))
    table = defaultdict(Counter)          # expected -> Counter(chosen)
    misses = []
    for suite in suites:
        expected = suite["tool"]
        for request in suite["probes"]:
            for _ in range(runs):
                chosen = pick_tool(request)
                table[expected][chosen] += 1
                if chosen != expected:
                    misses.append((expected, chosen, request))
    return table, misses

def print_table(table):
    for expected, picks in table.items():
        total = sum(picks.values())
        hit = picks[expected] / total
        others = ", ".join(f"{t} x{n}" for t, n in picks.items() if t != expected)
        print(f"{expected:16} {hit:5.0%}  {others}")
```

Anything below 100% is a collision with a number attached. Border probes tell you where the line is blurry.

![](https://cdn.hashnode.com/uploads/covers/6a44b5d24b41ab0145e5cf63/f0c8e7db-1354-4eeb-97a4-cba4b1f44432.png align="center")

## Fix it, in this order

1.  **Merge tools with the same intent.** Anthropic's [guide to writing tools for agents](https://www.anthropic.com/engineering/writing-tools-for-agents) gives the example of one `schedule_event` tool instead of separate list and create calls. Watch for parameter sprawl: a merged tool with a dozen optional args moves the guessing into the arguments.
    
2.  **Draw the boundary in both descriptions.** Say what the tool is for and what it's not for, naming the neighbour. One-sided boundaries leave half the border unmarked.
    
3.  **Rename and namespace.** `jira_search`, `asana_search`. Batch renames, since each change to tool definitions resets the prompt cache.
    
4.  **Scope tools to the task.** Billing mode doesn't need the GitHub tools. Cuts both bills, but choosing the mode is one more decision, so keep modes coarse.
    
5.  **Defer with tool search** once the catalog is large (Anthropic suggests it beyond roughly 10K tokens of definitions or 10+ tools).
    

Here's step 2 applied to the example:

```json
[
  {
    "name": "search_orders",
    "description": "Look up orders and their current state: items, payment, shipping and refund status. Use for any question about where an order, delivery or refund is right now. Not for past conversations with support (use search_tickets)."
  },
  {
    "name": "search_tickets",
    "description": "Look up past conversations with support agents. Use when the customer refers to something they were told, a complaint they filed, or a promise support made. Not for the current status of an order or refund (use search_orders)."
  }
]
```

Longer descriptions cost tokens on every call, so spend the words on the border. The same principle as in [designing schemas the model can answer honestly](https://simplyexplained.hashnode.dev/structured-outputs-still-hallucinate-designing-llm-schemas-that-can-say-i-don-t-know): the model can only be as precise as what you hand it.

![](https://cdn.hashnode.com/uploads/covers/6a44b5d24b41ab0145e5cf63/bd9d5f32-c9c6-4bf3-98e4-474b06abde15.png align="center")

## Does tool search fix tool confusion?

Partly. Tool search gives the model a search tool and loads matching definitions on demand. In Anthropic's example it cut definition tokens by about 85%, and on their internal MCP evals selection accuracy rose from 49% to 74% (Opus 4) and from 79.5% to 88.1% (Opus 4.5). Removing noise helps.

But retrieval ranks tools by how well their descriptions match the request, and look-alikes match the same requests. They come back together in the short list. A 2025 preprint on retrieval-based tool selection (RAG-MCP) likewise reported failures appearing as descriptions in a large pool began to overlap. Clean the boundaries first, then defer.

## Make it a regression test

The probes are only useful if they run when tools change: added, renamed, description edited, or a remote MCP server updated its definitions without any deploy on your side (the [config-change blind spot](https://simplyexplained.hashnode.dev/nobody-deployed-anything-production-broke-anyway)).

```python
# test_tool_routing.py
from routing import confusion_table, pick_tool   # the helpers above

THRESHOLD = 0.8   # pick a bar for your domain; border probes are the ones to watch

def test_no_tool_collisions():
    table, misses = confusion_table("probes/routing.yaml", pick_tool, runs=5)
    for expected, picks in table.items():
        hit = picks[expected] / sum(picks.values())
        assert hit >= THRESHOLD, f"{expected}: {hit:.0%} {dict(picks)}"
```

These tests call a model, so they cost money and aren't fully deterministic. Run them on tool-definition changes and on a schedule, not on every commit.

## Key Takeaways

*   Tool selection is multiple choice from your descriptions. Every new tool is a new option for every old request.
    
*   Count drives cost; overlap drives wrong tool calls. Fix them separately.
    
*   Wrong tool calls return real data, so they don't show up as errors.
    
*   Tool search cuts cost substantially but look-alikes still surface together.
    
*   Write border probes, tabulate a confusion table, and re-run it on every tool change.
    

## FAQ

### Why does my AI agent call the wrong tool?

Usually because two tools' names or descriptions plausibly match the same request. The model chooses from your descriptions alone, so overlapping descriptions have no single correct answer. Draw the boundary in both descriptions and test with border probes.

### How many tools is too many for an AI agent?

There's no fixed number. OpenAI softly suggests fewer than 20 functions at the start of a turn, and Anthropic recommends tool search beyond about 10 tools or 10K tokens of definitions. Both are mostly about cost; correctness depends on overlap.

### Does adding MCP servers make an agent worse?

It can. Each server adds definitions to every request and may add tools that overlap with ones you have, such as two `search_issues` tools from different servers. Namespace them and probe the overlapping pairs.

### Does tool search fix wrong tool selection?

It helps and it reduces token cost, but it doesn't remove overlap. Retrieval returns look-alike tools together, so the model still has to choose between them. Fix boundaries before deferring.

### How do I test AI agent tool selection?

Write routing probes (plain, paraphrased, border) for each tool, run each several times, and build a confusion table of expected vs chosen tool. Re-run it whenever tool definitions change.

### Do tool definitions affect prompt caching?

Yes. On Anthropic's API, tools sit at the start of the cache prefix, and modifying tool definitions invalidates the entire cache. Batch tool changes rather than editing them one at a time.

## Related reading

*   [How AI Learned to Actually Do Things: Tool Calling and MCP Explained](https://simplyexplained.hashnode.dev/how-ai-learned-to-actually-do-things-tool-calling-and-mcp-explained)
    
*   [Prompt Caching for AI Agents: Stop Breaking Your Own Cache](https://simplyexplained.hashnode.dev/trimming-prompt-raises-ai-bill)
    
*   [Why AI Agents Fail in the Real World](https://simplyexplained.hashnode.dev/why-ai-agents-fail)
    
*   [Structured Outputs Still Hallucinate: Designing LLM Schemas That Can Say "I Don't Know"](https://simplyexplained.hashnode.dev/structured-outputs-still-hallucinate-designing-llm-schemas-that-can-say-i-don-t-know)
    
*   New here? [Start Here: A Plain-English Map to Understanding AI](https://simplyexplained.hashnode.dev/start-here-understand-ai)
    

## The bottom line

Don't ask how many tools your agent can handle. Ask which two a new hire would mix up, then prove they don't with probes.

> Your agent doesn't need fewer tools. It needs fewer wrong answers that look right.

Get one of these in your inbox each week: [subscribe to the Simply Explained newsletter](https://simplyexplainedbyaj.substack.com/subscribe).

*Adam Jaber is a software engineer who writes Simply Explained: complex topics, made simple. No jargon, no hype.*