Tool Selection for AI Agents: Tool Search, Deferred Loading, Poisoning

An AI agent chooses a tool by reading every tool name, description and JSON Schema in its context and matching them to the request. Past a few dozen tools, accuracy drops and definitions eat the context window. Tool search and deferred loading fix both, but tool descriptions stay untrusted model input that can be poisoned.

This post goes one level deeper than our earlier pieces on MCP tools, resources and prompts, context engineering for AI agents and AI agent evaluation. Attack walk-throughs live in our MCP tool poisoning and line jumping posts. Here we stay on selection: how a model picks among many tools, what it costs, how the Model Context Protocol (MCP) and the providers handle large catalogs, and where the security boundary sits. Anthropic and OpenAI facts come from their own documentation and MCP facts from modelcontextprotocol.io, linked under Sources.

Key takeaways

• A model sees tools as text: a name, a description and an input schema. Tool selection is reading comprehension over that text, so the description is the single biggest lever you control, and also an input an attacker can try to control.
• Anthropic's tool search documentation says Claude's ability to pick the right tool degrades once you exceed 30 to 50 available tools. OpenAI's function calling guide suggests fewer than 20 functions available at the start of a turn, as a soft target.
• Tool definitions are billed as input tokens. Anthropic's worked example puts five common MCP servers at 58 tools and roughly 55,000 tokens before any work starts.
• Tool search with deferred loading keeps most definitions out of context until the model searches for them. Anthropic's tool search tool returns up to 5 matching tools per search by default and supports up to 10,000 deferred tools per request. OpenAI's tool_search works with deferred functions, namespaces and MCP servers on gpt-5.4 and later models.
• MCP gives catalogs structure: paginated tools/list, list change notifications, a 128 character naming rule, collision guidance for clients that aggregate servers, and annotations such as readOnlyHint and destructiveHint. The specification says annotations are hints and must be treated as untrusted unless the server is trusted.
• Selection is not authorisation. Tool search decides what the model sees; allowlists, per tool approval, scoped credentials and checks in the downstream system decide what it may do.
• Pin the tool definitions you reviewed by hash, and treat any change, including one announced by a list change notification, as a new review.

How does a model choose a tool?

Every major tool calling API works the same way at the level that matters for selection. You send a list of tools with the request. Each tool has a name, a natural language description and a JSON Schema for its input. The model reads them as part of its context, decides whether a tool is needed, picks one (or several), and emits a structured call that your code or the provider then runs.

Anthropic's tool use overview describes it plainly: Claude determines when to call a tool based on the user's request and the tool's description, then returns a structured call that your application runs (client tools) or that Anthropic runs (server tools such as web search, web fetch, code execution and tool search). With the default tool_choice of auto, Claude decides on each turn whether to call a tool or answer directly, calling one when the request maps to a tool's described capability and the answer is not already in context. You can force a call with tool_choice set to any or to a named tool, or suppress tools with none.

OpenAI's function calling guide says the same thing from the other side: under the hood, functions are injected into the system message in a syntax the model has been trained on, so function definitions count against the context limit and are billed as input tokens. Its tool_choice options are auto, required, a forced function, and an allowed_tools mode that restricts calls to a subset of the tools you passed.

Three consequences follow:

1. The model only knows what the text says. Similar names with thin descriptions give it nothing to separate them. Anthropic's advanced tool use article names wrong tool selection and incorrect parameters as the most common failures, especially with names such as notification-send-user and notification-send-channel.
2. Every tool competes with every other tool. Each new tool is a candidate the model must rule out on every turn where it does not apply.
3. The description is an instruction channel. The model treats tool text as guidance. That makes good descriptions work, and it is what a poisoned description abuses.

What the model actually reads

Here is what a single tool looks like in each format. The fields line up closely.

Note the naming mismatch in the first row. MCP allows a dot in tool names (the specification gives admin.tools.list as a valid example), while Claude's API pattern does not. Any client that bridges MCP tools into Claude has to rewrite names, and Claude Code's documentation shows how it does that for plugin servers: the callable name becomes mcp__plugin_<plugin-name>_<server-name>__<tool-name>, with any character outside letters, digits, underscore and hyphen replaced by an underscore. The rewritten name is what permission rules and hooks match on, which matters later in this post.

Why does selection accuracy fall as catalogs grow?

Three independent lines of evidence say the same thing: more tools in context means worse selection, and narrowing the candidate set before the model chooses brings accuracy back.

Provider documentation. Anthropic's tool search page states that Claude's ability to pick the right tool degrades once you exceed 30 to 50 available tools, and recommends tool search when you have 10 or more tools, definitions over 10,000 tokens, or several MCP servers that add up to 200 or more tools. OpenAI's guide says to keep the number of initially available functions small for higher accuracy, to evaluate with different numbers of functions, and to aim for fewer than 20 at the start of a turn, calling that a soft suggestion.

Provider measurements. Anthropic's advanced tool use article (24 November 2025) reports internal MCP evaluations with large tool libraries where turning on the tool search tool moved Claude Opus 4 from 49% to 74% and Claude Opus 4.5 from 79.5% to 88.1%. These are Anthropic's own internal numbers on its own evaluation, so read them as a direction and a size of effect, not as a benchmark you can reproduce.

Research. The idea that retrieval should sit in front of tool choice predates MCP:

• Gorilla (Patil and colleagues, arXiv, 24 May 2023) found models of the time tended to hallucinate API usage, and showed that pairing a fine-tuned model with a document retriever let it adapt to changed documentation at test time.
• ToolLLM (Qin and colleagues, arXiv, first posted 31 July 2023) gathered 16,464 real REST APIs across 49 categories and used a neural API retriever to recommend relevant APIs per instruction.
• RAG-MCP (Gan and Sun, arXiv, 6 May 2025) retrieves the most relevant MCP servers from an external index before the model sees anything, and reports cutting prompt tokens by over 50% and raising tool selection accuracy from 13.62% to 43.13% on its benchmark tasks.

Benchmarks. The Berkeley Function Calling Leaderboard, from the same group as Gorilla, evaluates how accurately models call functions. Version 4 adds holistic agentic evaluation to earlier single and multi-turn tests, and the page showed a last update of 12 April 2026 when we checked. Look there for model comparisons; we do not rank models here.

Why does more choice hurt? Big catalogs contain near duplicates (search, search_issues, find_ticket), the few words that separate two tools get diluted across tens of thousands of tokens, arguments get filled from a neighbouring schema, and every definition token is one not available for the task. On parameters, Anthropic's article reports that tool use examples raised accuracy on complex parameter handling from 72% to 90% in its internal testing.

What do tool definitions cost in context?

Tool definitions are not free metadata. Anthropic's pricing section for tool use lists three sources of extra input tokens: the tools parameter itself (names, descriptions and schemas), tool_use blocks, and tool_result blocks. On top of that the API adds a tool use system prompt whenever tools are present. For Claude Opus 5.5 that is 286 tokens with tool_choice set to auto or none; older models in the same table range from 264 to 804 tokens depending on model and tool_choice. OpenAI's guide says function definitions are billed as input tokens and count against the context limit.

Worked example 1: Anthropic's five server setup

Anthropic's advanced tool use article gives a concrete catalog:

That is an average of about 950 tokens per tool, but the spread is wide: about 740 per GitHub tool and about 1,900 per Slack tool. The same article says adding a Jira server alone adds about 17,000 tokens, and that Anthropic has seen tool definitions reach 134,000 tokens before optimisation.

The article then compares two setups for a library of 50 or more MCP tools. Loading everything up front cost about 72,000 tokens of definitions and about 77,000 tokens of context before any work began. With tool search, only the search tool loaded up front (about 500 tokens) and the 3 to 5 tools actually needed were discovered on demand (about 3,000 tokens), for about 8,700 tokens in total, which Anthropic describes as an 85% reduction.

Worked example 2: your own catalog

Run the same arithmetic on your own numbers before you decide anything. Suppose an internal platform agent connects to six MCP servers with 300 tools between them, and you measure an average of 400 tokens per definition.

• All tools in context: 300 × 400 = 120,000 tokens of definitions on every request, plus the tool use system prompt. On a 200,000 token context window that leaves 80,000 tokens for instructions, files and history, and each of the 300 tools competes on every turn.
• Tool search: the search tool plus 4 frequently used tools loaded up front (4 × 400 = 1,600 tokens, plus the search tool's own definition), then about 5 discovered tools per task (5 × 400 = 2,000 tokens). That is under 5,000 tokens in context for a typical task, and the model chooses among 9 candidates instead of 300.
• Tool search is not metered separately, but loaded definitions count as input tokens and the search adds latency, which Anthropic says pays off when savings and accuracy gains outweigh it.

Two caveats. You still send every definition in the tools array on every request, because the API needs them to run the search; deferral controls what enters context, not what you transmit. And prompt caching matters: Anthropic says deferred tools stay out of the cached prefix and discovered tools are appended inline, and OpenAI says its tool search preserves the cache by injecting discovered tools at the end of the context.

How do tool search and deferred loading work?

The core idea is the same everywhere: keep a searchable index of tool definitions outside the model's context, give the model one small tool that searches that index, and load only what the search returns.

Claude's tool search tool

Anthropic's documentation describes two server-side variants:

• Regex (tool_search_tool_regex_20251119): Claude writes Python re.search() patterns, matched case-insensitively, up to 200 characters.
• BM25 (tool_search_tool_bm25_20251119): Claude writes natural language queries, up to 500 characters.

The flow, as documented:

1. Put a tool search tool in your tools list.
2. Send every tool definition, and set defer_loading: true on the ones that should not load up front. At least one tool, normally the search tool itself, must stay non-deferred. Marking every tool deferred returns a 400 error.
3. Claude's context initially holds only the search tool and any non-deferred tools.
4. When Claude needs something else, it calls the search tool.
5. The API runs the search over tool names, descriptions, argument names and argument descriptions, and returns matches as tool_reference blocks, up to 5 by default. Claude can set a limit from 1 to 10,000.
6. The API expands those references into full definitions before Claude sees them.
7. Claude calls the discovered tool, and your code handles the tool_use block as usual.

Other documented details that matter in production:

• The maximum is 10,000 deferred tools per request.
• Anthropic recommends keeping your 3 to 5 most frequently used tools non-deferred so Claude can call them without searching.
• Discovered tools stay usable in later turns without searching again.
• For tools that come through Anthropic's MCP connector (a beta feature that connects to remote MCP servers from the Messages API), you set defer_loading once in the mcp_toolset entry's default_config, or per tool in its configs, instead of on individual definitions. The same connector lets you allowlist or denylist individual tools.
• You can run your own search, for example with embeddings, by returning tool_reference blocks from a custom tool. Every referenced tool must still be defined in the top-level tools array.

The tool search tool is supported on Claude Haiku 4.5, Sonnet 4.5, Opus 4.5 and later models in Anthropic's compatibility table. Claude Opus 4.1 and earlier do not support it.

OpenAI's tool search

OpenAI's tool search guide describes tool_search in the Responses API, supported on gpt-5.4 and later models. You add tool_search to the tools array and mark deferred functions with defer_loading: true; for MCP servers you set defer_loading: true on the MCP server tool definition. Two modes are documented:

• Hosted tool search: OpenAI searches the deferred tools you declared in the request and returns the loaded subset in the same response. The guide recommends this when you know the candidate tools when you create the request.
• Client-executed tool search: the model emits a tool_search_call, your application does the lookup, and you return a tool_search_output.

One point is easy to miss. For an individual deferred function, the model still sees its name and description up front, so tool search mostly defers the parameter schema. For a namespace or MCP server, the model sees only that group's name and description until search loads the functions. OpenAI recommends namespaces with clear, high-level descriptions and fewer than 10 functions each. allowed_tools still applies to whatever is callable in the turn.

Claude Code

Claude Code's MCP documentation says tool search is on by default, only tool names and server instructions load at session start, and there is no fixed per server tool cap. Server instructions tell Claude when to search for a server's tools.

It also documents description limits: by default, server instructions and up-front tool descriptions are cut at 4,096 characters, and descriptions of tools loaded through search at 16,384. Search matches only within the shorter limit. A server can request up-front loading for a tool with "anthropic/alwaysLoad" in its _meta, and the person configuring the server can override it.

Retrieval over descriptions and hierarchical routing

You can build the same pattern yourself in three shapes:

• Lexical search over names and descriptions, which is what the regex and BM25 variants do. Cheap, predictable and easy to debug, but it misses synonyms unless descriptions contain the words users actually say. Anthropic's troubleshooting advice is to add common keywords to descriptions.
• Semantic retrieval, embedding each definition and the request, then taking the nearest neighbours. This is the approach of RAG-MCP and the retrievers in Gorilla and ToolLLM, and Anthropic's docs point to an embeddings recipe for a custom search tool.
• Hierarchical routing, where a first step chooses a group (a server, a namespace, or a specialist subagent) and a second step chooses the tool inside it. OpenAI's namespaces are a two-level hierarchy in one request. A router agent that hands off to specialist agents, each with a small fixed toolset, is the same idea spread across several model calls.

All three move a decision out of the model's context into an index you control, which helps accuracy and cost and makes the index, and the text it ranks on, part of your attack surface.

What does MCP provide for large catalogs?

MCP is where most large catalogs come from, so its rules shape everything above. The current specification revision is dated 2026-07-28. Here is what it gives a client that has to manage many tools.

Discovery with tools/list. A server that offers tools declares the tools capability, optionally with listChanged. Clients call tools/list to get definitions. The 2026-07-28 revision adds three rules aimed squarely at catalogs:

• The tool set may change over time and may vary by the authorisation presented on the request (for example, returning only the tools the caller's scopes permit), but must not vary per connection or as a side effect of other requests.
• Servers should return tools in a deterministic order, which the specification says lets clients cache the list reliably and improves prompt cache hit rates when tools sit in model context.
• List results carry ttlMs and cacheScope fields, a freshness hint and a scope (public or private) for caching the list.

Pagination. List operations use opaque cursor-based pagination. The server decides the page size, returns a nextCursor when more results exist, and clients must not assume a fixed page size. A client that reads only the first page silently sees a partial catalog.

Change notifications. When the tool list changes, a server that declared listChanged should send notifications/tools/list_changed. In the 2026-07-28 revision, change notifications flow over a subscriptions/listen stream that clients opt into per notification type (toolsListChanged for tools), replacing the older GET endpoint. Claude Code's documentation says that when such a notification arrives in an interactive session it fetches the updated list from that server without a reconnect.

Names and collisions. Tool names should be 1 to 128 characters, case-sensitive, using only ASCII letters, digits, underscore, hyphen and dot, and unique within a server. The specification is explicit that uniqueness is scoped to one server: clients or proxies that aggregate several servers may hit collisions (two servers each exposing search) and should disambiguate, for example by prefixing tool names with a server identifier. It also warns that the server name from serverInfo is not guaranteed unique and should not be relied on for disambiguation.

Annotations. The ToolAnnotations schema defines four hints:

The schema says all of these are hints, not guaranteed to describe behaviour faithfully, and that clients should never make tool use decisions based on annotations received from untrusted servers. The tools page says clients must consider annotations untrusted unless they come from trusted servers. Note the defaults: an unannotated tool is assumed to be destructive and open world, which is the safe reading.

Security considerations. The tools page says a human should always be able to deny tool invocations. Servers must validate inputs, enforce access controls, rate limit and sanitise outputs. Clients should confirm sensitive operations, show inputs before calling, validate results, set timeouts and log tool usage.

How should tools be named and described?

Good naming and description work makes selection more accurate and makes review easier. Both providers publish concrete guidance.

Namespace by service and resource. Anthropic's tool definition guide recommends prefixing tool names with the service, such as github_list_prs and slack_send_message, and calls this especially important with tool search, because one search then matches a whole group. Anthropic's "Writing effective tools for agents" (11 September 2025) adds resource-level prefixes such as asana_projects_search and asana_users_search, and reports that choosing prefix versus suffix namespacing had non-trivial effects on its evaluations, varying by model. OpenAI's namespaces do the same job structurally.

Resolve collisions before the model sees them. If you aggregate servers, prefix every tool with a server identifier you control, not the server's self-reported name, as the MCP specification advises. Two tools named search from two servers should reach the model as tickets_search and wiki_search, never as two identical names.

Write descriptions for a newcomer. Anthropic's guidance calls detailed descriptions by far the most important factor in tool performance and asks for at least 3 to 4 sentences per tool: what it does, when to use it and when not to, what each parameter means, and caveats such as what it does not return. OpenAI's version is the "intern test": could an intern use the function correctly with nothing but what you gave the model?

Use examples and tight schemas. Anthropic's input_examples field takes schema-validated example inputs for format-sensitive parameters; OpenAI notes examples can hurt reasoning models. OpenAI recommends enums so the schema cannot express invalid states, and Anthropic recommends consolidating related operations, such as one pull request tool with an action parameter.

Keep descriptions about the tool. A description should say what the tool does and when to use it. It should not tell the model how to treat other tools, what to do with the user's data in general, or to keep anything from the user. That rule is what makes the next section's risks detectable.

Here is a short contrast for a security platform tool:

{
  "name": "vulns_list_for_project",
  "description": "Lists known vulnerabilities affecting one project in the current workspace. Use it when the user asks which CVEs affect a named project or wants findings filtered by severity. It returns CVE id, severity, affected package and version, and fixed version when known. It does not start a scan or change any data; use the project's scan status to see how fresh results are.",
  "input_schema": {
    "type": "object",
    "properties": {
      "project_id": {"type": "string", "description": "Project id from the project list tool"},
      "severity": {"type": "string", "enum": ["critical", "high", "medium", "low"]}
    },
    "required": ["project_id"]
  }
}

A weak version named list and described as "Gets vulns." would collide with every other list and give search nothing to match.

What are the security risks?

Tool definitions are text written by whoever runs the server, and the model reads that text much as it reads your instructions. We describe the risk classes at concept level only; for depth see our tool poisoning threat model for Claude.

Poisoned tool descriptions

A poisoned description carries instructions aimed at the model instead of describing the tool: read data the task does not need, put it in an argument, call a different tool, or stay quiet about it. Users often see only a title while the model reads the full text. OWASP's Top 10 for LLM Applications 2025 classes this as indirect prompt injection under LLM01: instructions arrive from an external source, here a server's metadata, and they need not be visible or readable to a human to affect the model.

The defensive signals are recognisable, and Step 6 below checks for them: references to other tools, files or credentials unrelated to the tool's purpose, requests for secrecy, invisible characters, and descriptions far longer than the job needs.

Definitions that change after approval

You approve a server's tools; later it changes a description or adds a tool. MCP allows this and announces it with list_changed, and clients that refresh automatically send the new text to the model. Nothing in the protocol ties it to your earlier approval, so the step by step below pins approved definitions by hash. Our post on MCP server capability drift detection covers the monitoring side.

The 2026-07-28 rule that tools/list may vary by the caller's authorisation adds a wrinkle: the definitions a reviewer saw with an admin token may differ from what a production token receives. Snapshot with the same credentials the agent will use.

Tool shadowing across servers

When servers share a context, one server's text can try to steer how the model uses another server's tools; its own tools never need to be called. Unprefixed name collisions make it worse, since the model may pick the wrong server's send_email. Prefix by a server identifier you control and keep untrusted servers out of high-trust sessions.

Search and routing manipulation

Tool search changes the attack, not the principle. With Claude's server-side search, a deferred tool's definition does not enter the model's context until a search returns it, so a poisoned description on a tool that is never retrieved never reaches the model. But retrieval ranks on description text, so a description stuffed with popular keywords can push its tool into results for unrelated requests. With OpenAI's design, an individual deferred function's name and description are visible from the start, so deferral alone does not keep its text out of context. Claude Code's longer 16,384 character limit for searched tools also means more room for text in tools that load on demand. Our analysis: treat the search index as a supply chain input and review what goes into it, rather than assuming deferral is a filter.

Annotations that lie

readOnlyHint: true is a claim by the server. A client that auto-approves anything marked read-only has handed its approval policy to the server author. Use annotations from trusted servers for UI defaults, never for authorisation.

Mapping to the OWASP Top 10 for LLM Applications 2025

LLM06 is the closest fit for selection design. OWASP's mitigations read like a tool catalog policy: minimise extensions, minimise their functionality, avoid open-ended ones such as "run a shell command", minimise their permissions, execute in the user's context with minimum scope, require user approval for high-impact actions, and enforce complete mediation in downstream systems rather than relying on the model to decide what is allowed.

How do permission layers help?

Selection and permission are separate layers. Tool search decides which definitions the model sees; permissions decide which calls run. Finding a tool through search is not authorisation to use it.

Allowlists at the client. Anthropic's MCP connector lets you enable all tools from a server, allowlist specific ones or denylist unwanted ones. OpenAI's allowed_tools restricts calls to a subset without changing the tools list, which the guide notes also helps prompt caching. Claude Code permission rules, a skill's allowed-tools, a subagent's tools field and hook matchers all take the full tool name, so you can name single MCP tools instead of whole servers. Our Claude Agent SDK explainer covers those mechanics in depth.

Per tool approval. MCP asks for a human able to deny invocations. Reserve approval for calls that change something, so people do not learn to click through it; see our post on out of band confirmation for irreversible tool calls.

Read only first. Start every new server with only its read tools enabled. Add write tools one at a time when a task needs them, with approval on each. This is OWASP's "minimise extension functionality" applied to MCP, and it also shrinks the catalog the model selects from.

Scoped credentials and complete mediation. MCP's security best practices warn that a stolen broad-scope token enables lateral access. Give each server the narrowest credential and enforce authorisation downstream, so a wrong selection fails closed (scoped credentials per MCP server).

Audit. Log which tools the model was shown, searched for and called, with arguments; without the first two you cannot tell a selection failure from an injection (tool call audit).

Comparison: four ways to give an agent many tools

Most production agents blend them: a few always-loaded tools, search over the rest, and a stricter agent for anything that writes.

Step by step: test selection, add tool search, pin definitions

This walkthrough uses Python and the Anthropic SDK because tool search is documented for it; the same structure works with OpenAI's Responses API. It assumes you can export each server's tools/list result to JSON with your MCP client or the MCP Inspector (check the Inspector's documentation for the CLI flags of your version, since version 2 changed them).

Step 1: snapshot the catalog with the credentials the agent uses

Export one file per server, using the same token the agent will run with, since the 2026-07-28 specification allows the list to vary by authorisation. Follow nextCursor until it is absent, or you will snapshot only the first page.

catalog/
  tickets.tools.json   # {"tools": [...]} merged across all pages
  wiki.tools.json
  repo.tools.json

Step 2: convert to API tools with collision-proof names

# build_tools.py
import json, pathlib, re

def api_name(server: str, tool: str) -> str:
    # Prefix with a server id you control; Claude names allow only [a-zA-Z0-9_-], max 128.
    return re.sub(r"[^a-zA-Z0-9_-]", "_", f"{server}_{tool}")[:128]

tools, seen = [], {}
for path in sorted(pathlib.Path("catalog").glob("*.tools.json")):
    server = path.name.split(".")[0]
    for t in json.loads(path.read_text())["tools"]:
        name = api_name(server, t["name"])
        if name in seen:
            raise SystemExit(f"name collision: {name} from {server} and {seen[name]}")
        seen[name] = server
        tools.append({
            "name": name,
            "description": t.get("description", ""),
            "input_schema": t["inputSchema"],
        })

pathlib.Path("tools.json").write_text(json.dumps(tools, indent=2))
print(f"{len(tools)} tools written")

The collision check fails loudly instead of letting two servers' tools merge into one name.

Step 3: write a selection test suite

Each case has a prompt, the tool you expect, and optionally tools that must not be called. Include cases where the right answer is no tool at all, and cases that sit between near duplicate tools.

{"id": "t01", "prompt": "Which open tickets mention the login outage?", "expect": "tickets_search", "forbid": ["tickets_close"]}
{"id": "t02", "prompt": "Find the runbook page for rotating the database password", "expect": "wiki_search"}
{"id": "t03", "prompt": "What does HTTP status 418 mean?", "expect": null}
{"id": "t04", "prompt": "List the last five merged pull requests on the api repo", "expect": "repo_list_pull_requests"}

Anthropic's tool writing guidance suggests many realistic tasks, each with a verifiable outcome. Keep a held-out set you do not tune descriptions against.

Step 4: measure baseline accuracy with every tool loaded

# eval_selection.py
import json, sys, anthropic

client = anthropic.Anthropic()
MODEL = "claude-opus-5-5"          # use the model you deploy
RUNS = 3                           # selection is not deterministic; repeat each case

def chosen_tool(resp):
    for block in resp.content:
        if block.type == "tool_use":
            return block.name
    return None

def evaluate(tools, cases):
    correct = forbidden_hits = total = 0
    input_tokens = 0
    for case in cases:
        for _ in range(RUNS):
            resp = client.messages.create(
                model=MODEL, max_tokens=1024, tools=tools,
                messages=[{"role": "user", "content": case["prompt"]}],
            )
            got = chosen_tool(resp)
            total += 1
            correct += got == case["expect"]
            forbidden_hits += got in case.get("forbid", [])
            input_tokens += resp.usage.input_tokens
    return {"accuracy": correct / total, "forbidden_calls": forbidden_hits,
            "avg_input_tokens": input_tokens / total}

if __name__ == "__main__":
    tools = json.load(open(sys.argv[1]))
    cases = [json.loads(line) for line in open("cases.jsonl") if line.strip()]
    print(json.dumps(evaluate(tools, cases), indent=2))

Run python eval_selection.py tools.json and record the three numbers. This scores only the first call; for multi-step tasks, score the trajectory as in our agent evaluation post.

Step 5: add tool search and measure again

# with_search.py
import json

tools = json.load(open("tools.json"))
HOT = {"tickets_search", "wiki_search", "repo_list_pull_requests"}   # 3 to 5 most used

deferred = [t if t["name"] in HOT else {**t, "defer_loading": True} for t in tools]
search_tools = [{"type": "tool_search_tool_bm25_20251119", "name": "tool_search_tool_bm25"}] + deferred
json.dump(search_tools, open("tools_search.json", "w"), indent=2)

Then run python eval_selection.py tools_search.json; chosen_tool skips the server_tool_use search block. Also log retrieval recall: did the expected tool appear in the tool_search_tool_result block's tool_references? Low recall means descriptions lack the words users say. Expect input tokens to fall; accuracy should hold or rise.

Step 6: review descriptions before they enter the index

Run a simple, defensive lint over every description before approving a server. It does not prove a description is safe; it surfaces text a human should read.

# review_descriptions.py
import json, re, sys, unicodedata

tools = json.load(open(sys.argv[1]))
names = {t["name"] for t in tools}
MAX_LEN = 1500

for t in tools:
    d = t.get("description", "")
    flags = []
    if len(d) > MAX_LEN:
        flags.append(f"long description ({len(d)} chars)")
    if any(unicodedata.category(c) == "Cf" for c in d):
        flags.append("invisible format characters")
    others = [n for n in names if n != t["name"] and n in d]
    if others:
        flags.append(f"mentions other tools: {others}")
    if re.search(r"\b(do not|don't|never) (tell|mention|show|inform)\b", d, re.I):
        flags.append("asks for secrecy")
    if re.search(r"\b(ssh|\.env|credential|api[_ ]?key|token|password)\b", d, re.I):
        flags.append("mentions secrets; check it fits the tool's purpose")
    if flags:
        print(f"{t['name']}: " + "; ".join(flags))

Any flag means a person reads the full text, parameter descriptions included, before approval.

Step 7: pin approved definitions by hash

# pin_tools.py  usage: python pin_tools.py lock|check
import hashlib, json, pathlib, sys

FIELDS = ("name", "title", "description", "inputSchema", "outputSchema", "annotations")

def digest(tool):
    canon = json.dumps({k: tool.get(k) for k in FIELDS}, sort_keys=True,
                       separators=(",", ":"), ensure_ascii=False)
    return hashlib.sha256(canon.encode("utf-8")).hexdigest()

def current():
    out = {}
    for path in sorted(pathlib.Path("catalog").glob("*.tools.json")):
        server = path.name.split(".")[0]
        out[server] = {t["name"]: digest(t) for t in json.loads(path.read_text())["tools"]}
    return out

lock = pathlib.Path("tools.lock.json")
if sys.argv[1] == "lock":
    lock.write_text(json.dumps(current(), indent=2, sort_keys=True))
    sys.exit(0)

pinned, now, problems = json.loads(lock.read_text()), current(), []
for server in sorted(set(pinned) | set(now)):
    a, b = pinned.get(server, {}), now.get(server, {})
    problems += [f"NEW      {server}/{n}" for n in sorted(set(b) - set(a))]
    problems += [f"REMOVED  {server}/{n}" for n in sorted(set(a) - set(b))]
    problems += [f"CHANGED  {server}/{n}" for n in sorted(set(a) & set(b)) if a[n] != b[n]]
print("\n".join(problems) or "all tool definitions match the lock")
sys.exit(1 if problems else 0)

Commit tools.lock.json with the agent's code so definition changes get reviewed like code. Add any other fields your client passes to the model.

Step 8: check on a schedule and on every change notification

Run the check in CI and on a timer, and fail closed:

# .github/workflows/tool-pins.yml
name: tool-definition-pins
on:
  schedule:
    - cron: "17 */6 * * *"
  pull_request:
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Export current tool lists
        run: ./scripts/export_tools.sh        # your client or Inspector export, same credentials as prod
      - name: Compare with approved pins
        run: python pin_tools.py check

In the agent's client, do the same at runtime: on notifications/tools/list_changed, refetch, hash, hide new or changed tools until approved, and send the diff to the owning team.

Common mistakes

• Near duplicate tool names. search, find and query from three servers in one context. Prefix by server and resource, and merge tools that do the same job.
• Vague descriptions. One-line descriptions give the model and the search index nothing to work with. Write 3 to 4 sentences: what, when, when not, what it returns.
• Loading every server for every task. Connecting all servers globally because it is convenient. Load per task or per agent, and defer the rest.
• Trusting descriptions from unreviewed servers. A description is input from a third party. Review it like code, and pin it.
• Treating annotations as permissions. readOnlyHint is the server's claim, not your policy.
• Reading only the first page. Ignoring nextCursor gives you a partial catalog and a partial review.
• Confusing selection with authorisation. Tool search narrows what the model sees; it does not decide what the model may do.

FAQ

How does an AI agent decide which tool to call?

It reads the names, descriptions and input schemas of the tools in its context and matches them to the request. With tool_choice set to auto, the model decides each turn whether a tool is needed and which one. You can force a call or restrict the set with tool_choice options.

How many tools can an AI agent handle?

There is no hard limit in context, but accuracy falls as the list grows. Anthropic's docs say Claude's selection degrades past 30 to 50 tools, and OpenAI suggests fewer than 20 at the start of a turn as a soft target. With tool search, Anthropic supports up to 10,000 deferred tools per request.

Does tool search reduce token costs?

It reduces the tokens in context. Anthropic's example went from about 77,000 to about 8,700 tokens. You still send every definition with each request, and the tools that search loads count as input tokens.

What is MCP tool poisoning?

It is hiding instructions for the model inside a tool's description or parameter descriptions. OWASP classes it as indirect prompt injection. Review and pin descriptions, limit which servers share a context, and require approval for calls that change things.

What is a tool rug pull in MCP?

It is the informal name for a server changing a tool's definition after you approved it. MCP allows tool lists to change and announces changes with notifications/tools/list_changed. Pinning approved definitions by hash and blocking changed tools until review handles it.

Can I trust MCP readOnlyHint and destructiveHint?

Only from servers you trust. The MCP schema says annotations are hints that may not describe behaviour faithfully, and that clients should never make tool use decisions based on annotations from untrusted servers.

How do I avoid tool name collisions across MCP servers?

Prefix every tool with a server identifier you control. The MCP specification says names need only be unique within one server, that aggregating clients should disambiguate, and that the server's self-reported name is not guaranteed to be unique.

Is tool search a security control?

No. It manages selection and context, and it ranks tools by description text. It does not replace allowlists, approvals, scoped credentials or downstream authorisation.

Where Safeguard fits

Safeguard does not select tools for your agent, run tool search or review MCP tool descriptions. Its fit is in two places.

First, Safeguard runs an MCP server of its own, and it follows the read only first pattern described above: a new workspace starts with a small read-only set of tools, and admins enable more per tool. You can add Safeguard from Claude's connector directory and sign in with OAuth, and the first demo below shows choosing which Safeguard tools Claude may use, the same least privilege idea as an allowlist.

Second, the agent itself is software with a supply chain. The MCP server packages you launch, the SDKs, their transitive dependencies and the base image are all third party components that receive vulnerability advisories. The Safeguard CLI can scan the agent's repository and built image (safeguard scan --dir . and safeguard scan --image agent-image), match components against vulnerability databases, rank findings with reachability analysis, and evaluate policies and gates in the platform so a build carrying a vulnerable version fails before it ships. The platform shows which projects contain a given component and version, which answers "which of our agents run this MCP server release?" Governing live agent tool traffic is not something we describe here as generally available; MCP Guardrail, which generates guardrail proxy URLs for MCP servers, is generally available.

An agent's tool catalog is assembled from packages: MCP servers installed from public registries, SDKs and the open source libraries beneath them, all pinned at versions that will one day carry advisories. The fit here is indirect, since tool selection itself is a prompt and configuration problem, so the steps below apply to the dependencies and images you build and ship around the agent.

Fix it without waiting for upstream

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation categories include dependencies, licenses, security posture and findings, and each one is set separately. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds.

3. Gold Registry: start from components that are already fixed

Safeguard Gold rebuilds patched versions of open-source packages and images from source, signs each build, attaches an SBOM, SLSA provenance and VEX, and serves them from registry.safeguard.sh to your organization. Every Gold artifact is held continuously to zero exploitable critical or high CVEs and a completed license check, with signed build provenance (signed with Safeguard's key and verifiable with cosign, SLSA Build Level 3 as the design target) and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. Ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

None of these demos shows tool search. They show Safeguard's own agent surfaces handling which tools an agent may use and who approves changes. Download Safeguard and see the CLI docs to try them.

This demo connects Claude to the Safeguard MCP server with OAuth and lets you choose which tools Claude can use. In practice, the simpler route is to add Safeguard from Claude's connector directory and sign in. Picking individual tools here is the allowlist step from this post, applied to a real server.

Click through it at your own pace, or open it full screen.

This is a real Safeguard CLI session. Asked to fix a lodash CVE, the assistant reads the finding and the advisory, asks before any command that changes something, edits package.json to lodash 4.17.21, runs npm install and npm test with approval, prints a summary and pushes nothing. It then shows /permissions, the same read only first and approve-on-change idea this post recommends for MCP tools.

Click through it at your own pace, or open it full screen.

This demo sets up the Safeguard runner on your own machine, approves it with its device code, then scans from the terminal. For an agent project, that is how you would scan the repository that holds your MCP server configuration and its dependencies without sending the code anywhere else first.

Click through it at your own pace, or open it full screen.

Quick check

Sources

• Anthropic: Tool search tool
• Anthropic: Tool use with Claude, overview and pricing
• Anthropic: Define tools
• Anthropic: MCP connector
• Anthropic: Claude Code, connect Claude Code to tools via MCP
• Anthropic: Introducing advanced tool use on the Claude Developer Platform (24 November 2025)
• Anthropic: Writing effective tools for agents (11 September 2025)
• Anthropic: Effective context engineering for AI agents
• OpenAI: Function calling guide
• OpenAI: Tool search guide
• Model Context Protocol specification 2026-07-28: Tools
• Model Context Protocol specification 2026-07-28: Schema reference (ToolAnnotations)
• Model Context Protocol specification 2026-07-28: Pagination
• Model Context Protocol specification 2026-07-28: Key changes
• Model Context Protocol: Security best practices
• Model Context Protocol Inspector README
• OWASP Top 10 for LLM Applications 2025
• OWASP LLM01:2025 Prompt Injection
• OWASP LLM03:2025 Supply Chain
• OWASP LLM06:2025 Excessive Agency
• Patil et al., Gorilla: Large Language Model Connected with Massive APIs (arXiv:2305.15334)
• Qin et al., ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (arXiv:2307.16789)
• Gan and Sun, RAG-MCP: Mitigating Prompt Bloat in LLM Tool Selection via Retrieval-Augmented Generation (arXiv:2505.03275)
• Berkeley Function Calling Leaderboard (BFCL) V4

Last reviewed: 25 October 2026. Tool search, deferred loading and the MCP specification change often; check the linked Anthropic, OpenAI and modelcontextprotocol.io pages before you build.