Repository navigation
Find, hold back and load tools on a large tool surface - #3
Merged
Merged
Conversation
added 19 commits
September 27, 2026 15:19
…rn search hits best first Tool retrieval indexed name, description and hint as one bag of lower-cased words and re-tokenised the whole catalogue on every query. A Russian query against English descriptions matched nothing unless a tool carried a Russian hint, identifiers such as BrowserOpen never matched "browser open", and ToolRegistry.search re-sorted its hits by name, so the model could not tell which one fit best. A tool is now indexed as five weighted fields: name, search_hint, the first sentence of the description, the rest of it, and the parameter names and descriptions. Text is split at identifier boundaries with the joined form kept, case-folded with ё spelled е, stripped of English and Russian stopwords including conversational fillers, and stemmed with pure-Python implementations of the Snowball Russian and original Porter algorithms. Queries are scored with BM25F, and a Russian query is expanded through a bundled Russian-to-English lexicon of generic developer vocabulary, which is what lets it reach tools described only in English. The field weights, k1, b and the expansion weight are LoopConstants fields; the loop passes them to the registry as RetrievalSettings. The analysed catalogue is built once per catalogue version and cached on the registry instance, with one bounded set of scoring tables per settings value, so a query costs about a third of a millisecond at 700 tools instead of 15 to 20. The normalized fallback remains the second stage when nothing scores, now for the per-turn clip as well as for search, and no longer counts one-letter words of a description as matches. search returns tools in rank order; the advertised surface stays in name order. A host may supply its own ranker through IToolRetriever, fused with the lexical ranking by reciprocal rank fusion, and its own lexicon or none. The old BM25 helpers are removed.
…n the clip off by default The clip kept the pinned, forced and always-load tools and gave retrieval whatever was left of tool_retrieval_top_k. A floor as large as the default of twelve left nothing, so the clip retrieved no tool at all, whatever the user asked. top_k now counts only the tools retrieval adds, and a surface whose unpinned tools already fit is not clipped. The in-memory registry clips the same way. Clipping per user message re-ranks the surface on every message, which changes the tool list and the provider's cached prefix with it, and a Russian message against English descriptions leaves the right tool out about half the time. It stays available as an opt-in: tool_retrieval_top_k now defaults to 0, which the loop passes as no clip.
A model that misremembers a tool name is usually one case change or one word away, Mcp_github_x for Mcp_Github_x, and an error that only repeats the wrong name leaves it guessing again. The unknown-tool error now ends with up to three registered names close to the one asked for, compared case-insensitively, drawn only from the names the run's visibility policy admits so that a blocked tool is never revealed by being near.
A model can emit a runaway batch: the same search more than a thousand times in one message has been seen, and every call of it would run. max_tool_calls_per_turn (default 64, 0 for no limit) bounds one message. The calls past it are routed to the serial path and each answered there with an error saying it was not run, so every call keeps its result in the order the model asked and the next request stays valid. A refused call is not charged to the circuit breaker or the error streaks, because the tool never ran, and the snapshot is written once after the last refusal rather than once per call.
…hem with ToolSearch Advertising every tool works for as long as the definitions fit. Past a token budget, or past the fixed number of tools a provider accepts, a request fails outright, and connecting one large MCP server is enough to get there. A host can now sort its catalogue into groups: a tool's optional tool_group class attribute, or a name prefix given to IToolRegistry.declare_group, with dynamic=True for membership the host does not write, such as an MCP server's proxies. Membership never reaches the wire or the surface digest. With tool_deferral_mode "auto", once per run and again only when the registry's catalogue changes, every dynamic group is held back, and the other groups go largest first while the surface is over tool_definitions_ratio of the window, a budget that was computed and never enforced, or over max_advertised_tools with room left for the tools the run may load. The forced floor, pinned and always-load tools and ungrouped tools are never held back, and nothing is held back unless a discovery tool is registered. Held-back tools stay admitted by the policy. The groups are named in one byte-stable system-prompt block after the skill catalogue, with exact tool names, or an exact prefix and a count for a large prefix group, and a sentence saying that a dedicated tool beats a workaround with a general one. Below the limits the surface and the prompt are unchanged. protocore.tools.ToolSearchTool searches the admitted catalogue under the live policy: a free-text query lists matches best first, one line each with the parameters and the first sentence of the description, and loads the first few; select: loads exact names and answers an unknown one with the nearest admitted names. It is advertised only while something is held back. The tool reports what it loaded in its result metadata and the loop does the loading, believing only a tool in the discovers_tools role and only names the policy and a child's declared tool set admit. Loaded tools are appended to the end of the surface in discovery order, so the name-sorted base and every earlier load keep their bytes and a prefix cache re-reads only the tail. The context manager's pin LRU is replaced by discovery order with each tool's last use; pinned_tool_max_count unloads the least recently used only after compaction, at a turn boundary and when a run starts, and a provider tool limit leaves the least recently used off one request. A call of a registered tool the request did not advertise still runs, and now loads it. The snapshot moves to version 7: discovered tools with their last use, and the held-back groups, which a resumed run replays rather than remeasures; the upcaster lifts version 6. QueryEngineConfig.discovered_tools seeds a new run of a session. tool_surface_advertised carries the discovered tools in order, the held-back groups and the reasons, and a discovered source per tool; tool_discovered and tool_unadvertised_call are new events. An engine built without a host run state now carries its own constants on run_state.rc, where core tools read their tunables.
docs/tools.md and its Russian mirror describe how groups are declared and when they are held back, the catalogue, ToolSearch, why loaded tools are appended in discovery order and when they are unloaded, what happens to a call of a tool that was not advertised, the per-message call cap, why clipping per user message is discouraged, and why schemas delivered as text behind a generic call tool were not built. The constants pages describe the deferral fields as a set, the architecture pages list the new module and events, the READMEs replace the claim about a clipped surface with what the surface now does, and the changelog records the changes.
ToolSearch shows the first sentence of a description as the tool's line, and the cut stopped at any full stop followed by a space. A description reading "Transition an issue to a new status (e.g. Done)" was shown as "... to a new status (e.g.", which says nothing about the tool. The cut now skips the full stop of an abbreviation (e.g., i.e., vs., cf., т.е., напр. and a few more), and a full stop inside an open bracket unless the bracket never closes. "etc." and "т.д." end the sentence only when a capital letter starts the next one, since they close a sentence as often as not. The same sentence is the index's summary field, so the ranking reads the corrected cut too.
…as its own argument
ToolSearch reported every tool it matched as loaded, including one that was
on the request's tool list all along. A model asked to use a skill searched
for it, read "Loaded, and callable from your next step: WebSearch", took the
skill for loaded and never opened it. The loop now stamps the names the
calling request advertised on the tool context (ADVERTISED_TOOLS_METADATA_KEY,
set after the operator's envelope is merged so the envelope cannot claim a
tool was listed), and the result says "Already in your tool list" for those
and "Loaded" only for the rest. The description adds that the tool loads
tools only: skills are not tools.
A model that passed {"select": ...} used to get a validation error for the
extra key and spend a turn retrying with "select:" inside the query. select
is now an argument of its own, a list or a comma-separated string, and names
win over a description sent beside them. A call that asks for nothing is
refused.
The one-line form of a tool moves to tool_retrieval.tool_line, so other
answers can show a tool the way the search results do.
A held-back tool called by its exact name runs, and is loaded for the next request, but the call itself was written without the schema. When the guess at the arguments was wrong, the model learned only that it failed, and the next attempt was often a second guess. The error now ends with the tool's line in the form ToolSearch lists tools in, "It takes: Name(param1*, param2) — first sentence", taken from the advertised names the loop stamps on the context. The line is added to a dispatch validation error, to an exception that is a TypeError, ValueError (a pydantic ValidationError among them) or KeyError, and to an error result the tool counts as a failure, because a tool that checks its own arguments reports a bad one as a result, not an exception. A tool that was on the list never gets it. The text a parallel batch's replay rebuilds a failure from carries the line too, or the transcript-order rebuild would drop it.
A group's declaration outlived its tools: when the operator switched off or removed an MCP server, the host unregistered the proxies but had no way to drop the group, so it stayed declared until the process restarted and would have claimed the prefix of a server later added under the same name. IToolRegistry.undeclare_group(name) removes it, idempotently, in the core registry and the in-memory one. Like declaring, it leaves the catalogue generation alone. Groups change no search result, and the loop keys its deferral decision on the declared groups themselves, so the next request plans again without a rebuilt index.
… and deployments
An MCP server describes its tools in the English of its domain, and the
operator asks in Russian loanwords and slang that the lexicon did not know:
"покажи открытые пулреквесты" found no pull request tool at all, "заархивируй
тикет поддержки" ranked the ticket archive fourth behind the GitHub issue
tools, and "назначь созвон" or "какие встречи" matched nothing.
The lexicon gains generic entries for that vocabulary: pull and merge
requests, tickets, issues and tasks ("ишью", "таска"), meetings and calls,
open and show, deploy and release ("выкати"), merge ("смержи"), review and
approve, message, send and email, archive, alerts and firing, bugs, crashes,
vulnerabilities, dependencies, and the names of common trackers and chat
services. A new test runs Russian requests against a small catalogue of
server tools in English, with no search hints and a host tool that also
talks about pull requests, and requires the right tool among the first
three, since that is what ToolSearch loads.
"написать" and "покажи" stay out of the send and list entries: they appear in
too many requests that are about neither.
The decision was keyed on the registry's tool names and groups. A host that switches a server's tools on mid-run does it by replacing the visibility policy, and when those tools were already registered for another session it registers nothing, so the key did not move: the decision still said nothing was held back, and every tool of the server went onto the surface as it stood, however many there were. Switching one off left it named in the catalogue. The key now carries the host's policy as well. The pins the engine adds for the tools the run loaded are not part of it, so loading a tool does not reopen the decision.
A decision a snapshot carried was replayed as the whole answer. A snapshot taken before a large MCP server connected named no groups, so the resumed run held nothing back and sent the server's every tool, over a provider's limit on the number of tools; a snapshot from a smaller window kept its groups but was never held to the limits again. The restored groups are now a floor: they stay held back while they have tools, a dynamic group they do not name is still held back, and the token budget and the tool limit are enforced on top. The same floor keeps a run's own decision when the catalogue or the policy changes under it. Made from scratch instead, it flipped whenever anything in the process moved: another session switching a server on grows this session's refusals, a tool switched off shrinks the surface, and each flip rewrote the catalogue at the head of the cached prompt or let a group that had been held back onto the surface mid-run. A server switched on in this run still joins the catalogue, because it is a newly admitted dynamic group. Without a discovery tool nothing was held back at all, so a registry with no admitted search sent every tool to a provider that refuses more than its limit. Groups are now held back when the count is over, dynamic ones first, and the catalogue says to call a tool by its exact name, which serves it and loads it. The token budget is not enforced without a search: a group held back for it could only come back by a blind call.
…mitted blind calls with a tool's line The dispatcher put the visibility policy on the tool context with setdefault under the bare key "tool_visibility_policy", so a value already in the bag, a stale one or a host key of the same spelling, won over the policy the gate was about to enforce, and ToolSearch trusted it. The key is now "protocore.tool_visibility_policy" and is assigned on every dispatch; a child run's declared tool set is stamped beside it under "protocore.tool_allowlist", so the search neither lists nor reports as loaded a tool outside that set, which the gate would refuse. ToolSearch without a readable policy admits nothing instead of everything. The line a blind call is answered with when its arguments fail was attached to any tool the request had not advertised. The argument checks run before the gate, so a blocked tool called with bad arguments was given its parameters and told it was "loaded now", which a refused tool never is. The line is now given only for a tool the policy and the declared set admit, and the near names offered for an unknown tool honour the declared set too.
A search loads its best three matches whether or not the model wanted them, and a host carrying the run's loaded tools into the session's next run carried every unused near-miss with them, one definition on every later request. The context manager now marks a discovered tool called when it is dispatched, and called_discovered_tool_names() returns those in discovery order: the list worth carrying. A held-back tool called by its exact name is loaded and called at once. A tool a host seeds the run with counts as called, since an earlier run called it. Snapshot rows carry the mark; a row without it is taken as called.
…or its tools A group now declares a load mode. A lazy group is held back whenever the run has a discovery tool, even when everything fits, so a family the run rarely needs costs one line of the catalogue instead of its definitions on every request; without a discovery tool it is advertised as before. An eager group is never held back for size and gives way only to the provider's limit on the number of tools, which would otherwise refuse the request. A run can override the declarations through QueryEngineConfig.tool_group_loads. ToolSearch takes a group name, alone or as group: entries in select, and loads every admitted tool of it; the loop keeps such a group as one entry under the loaded-tool cap so eviction never leaves half of it. A group may carry instructions. They are given once per run where its tools first come in front of the model: in the catalogue for tools on the surface or loaded when the run starts, and again after a compaction; in the search result that loads them; or, for a blind call of such a tool, instead of running it, with the tool loaded so the next call runs. The groups whose rules were given travel in the snapshot, and each load is announced as tool_group_loaded.
…un held whole QueryEngineConfig.loaded_tool_groups loads each named group as a whole-group search would: only the tools the run may call, one entry under the cap on loaded tools, its rules in the first catalogue, and a tool_group_loaded event with via "seed". The groups are loaded at the first decision rather than at construction, so a group declared after the engine was built is still found, and a host that asks for the decision before the first request sees them. Seeding a large group's tools by name made each one an entry, and two groups together pushed the rest of the seed out. ContextManager.loaded_tool_group_names() lists the groups a run holds whole, so a host can carry them into the session's next run; a seeded group's tools do not count as called, which lets an unused group go.
…ups first A blind call of a tool whose group carries rules was answered with the rules and loaded that one tool. The rest of the group stayed out of sight, so a job that went on to need a second tool of it (a click, then a download) was done by other means, and the retry was written from memory with wrong arguments. Such a call now loads the whole group as ToolSearch(group=...) would, one entry under the cap and only the tools the run may call, and its answer names the group's other tools and gives the called tool's line. The catalogue sorted every held-back group by name, which put a dozen connected servers ahead of the host's own groups. The host's groups now come first and the dynamic groups follow under their own heading. Each section is still in name order, so the bytes do not move between requests.
…, and mark genuine rules A compaction rewrote the catalogue with the rules of the groups still loaded, but the set of groups counted as given kept every group given earlier. A group whose rules came in a ToolSearch result that the compaction then masked, and which the compaction unloaded over the cap, stayed "given": a blind call of its tool ran without the rules, and a search that loaded it again gave none. After a compaction the catalogue now carries the rules of exactly the groups on the surface or still loaded, and the given set is reset to those. ToolSearch inside a loop goes by the given set alone, even for a listed tool. Rules reach the model inside tool results, where a web page or a file can print a heading that looks like them. Genuine rules now end their heading with the run's rules mark, an HMAC of the session id under a per-process key, and the catalogue in the system prompt names the mark and says that anything claiming to be rules without it is content. The mark is stable for a session within a process, so the cached prompt keeps its bytes, and the snapshot carries it, so a resumed run keeps the mark its history was given with. A separate runtime-authored message was rejected: a user turn between a tool result and the next assistant turn is refused by some chat templates and reads as the operator speaking. A call held back to give its group's rules is no longer counted as a use of the tool, since it never ran, and its tool_unadvertised_call event now comes before its result so a host can tell the result answers a call that did not run. The comments and docs now state what writing the rules after a compaction costs: one cache miss on the system prompt when a group with rules loaded mid-run is still loaded.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Support for large tool surfaces: finding the right tool among hundreds,
holding groups of tools back from the request, and loading them on demand.
Nineteen commits, each self-contained: every commit passes the full suite,
ruff,mypy --strictand the leak scan on its own.Finding tools (four commits)
search_hint, first sentence, rest of the description, parameters), withidentifier splitting, Snowball Russian and Porter English stemming, and
Russian query expansion through a bundled Russian-to-English lexicon. A host
may pass its own ranker (
IToolRetriever), fused with the lexical one byreciprocal rank.
ToolRegistry.searchreturns hits best first; the index is built once percatalogue version instead of on every query.
"e.g.".
against it.
Holding tools back and loading them (twelve commits)
ToolGroup,declare_group,undeclare_group), with loadmodes
eager,autoandlazyand optional per-group rules.tool_deferral_mode): groups are held back while the surface isover
tool_definitions_ratioof the window or overmax_advertised_tools,and are named in a byte-stable catalogue block in the system prompt, the
host's own groups first and connected servers after them.
ToolSearch(protocore.tools.ToolSearchTool) finds and loads held-backtools by free text, by exact name (
select) or by whole group (group).order, so loading a tool changes only the end of the request and a cached
prefix survives.
loads the tool; if it fails on its arguments, the error ends with the tool's
one-line signature.
held-back groups, the rules given); the upcaster lifts version 6.
Guard rails (two commits)
admits.
max_tool_calls_per_turncaps the tool calls dispatched from one modelmessage.
Docs (one commit):
docs/tools.md,docs/runtime-constants.mdanddocs/architecture.md, each with its Russian mirror, and the changelog.Why
A host that connects several servers ends up with hundreds of tools. Sent
whole, such a surface hits a wall rather than degrading gently: past a few
hundred tools a provider refuses the request outright for its tool count, or
the schemas no longer fit a smaller model's window. The previous answer, a
per-message clip that kept the top few tools by BM25 over English
descriptions, missed the needed tool on most Russian requests and was where
agents started inventing tool names.
The old retriever also re-tokenised every tool on every query (15–20 ms at
700 tools), re-sorted its hits by name so the best one was hidden, and counted
the article "a" in a description as a match for any query token containing
the letter.
Evidence (aggregate)
Retrieval. A labelled set of 656 English and Russian queries, written both
as a user would and as a model would, against a catalogue of 713 tools (a
host's own tools plus real server tools as distractors), and a held-out set of
80 queries written by hand and not tuned on.
search_hintThe last row uses search hints written to the convention in
docs/tools.md. Without the lexicon, Russian user queries fall back to 0.18.Agent benchmark. Around 45–56 tasks, a fifth to a half of them in
Russian, graded on the tools called and their key arguments, run against
several models over thousands of episodes, with 100 to 600 distractor tools.
once it does not. Keeping the host's tools loaded and holding connected
servers back stayed within noise of sending everything at 100 distractors
and stayed flat up to 600, a size at which sending everything was either
refused for its tool count or overflowed the window; input per episode grew
little.
default.
episodes. The failures that remain are a model not searching and improvising
with a general tool (a shell command instead of the dedicated one). Models
mostly load by exact name from the catalogue, so the catalogue's wording and
order matter more than ranking.
one.
lazy) halves the input perepisode and is as good as sending everything on stronger models, but costs a
weaker model success on those tasks; the load mode is therefore per group and
per run, and the default for a group is
auto.What changes for a consumer
This is a pre-release. Most of the new behaviour is inert until a host opts
in, but four defaults change and one engine is replaced, which does not follow
the "new behaviour defaults off" convention. They are listed here with the way
back.
come best first rather than by name. The lexicon is on;
tool_retrieval_lexicon_weight = 0orToolRegistry(lexicon=None)turnsquery expansion off.
retrieve_tools,bm25_score,compute_idf,compute_avgdl,reduce_query,build_candidateandToolRetrievalCandidateare removed.tool_retrieval_top_kdefaults to 0: the per-message clip is off (itwas 12). Set it above zero to clip again; it now counts only the tools
retrieval adds, not the pinned ones.
tool_deferral_modedefaults to"auto", which does nothing unless ahost declares a tool group, sets
max_advertised_tools, or registersToolSearch; without those the tool list and system prompt are byte forbyte what they were. Set it to
"off"to rule deferral out entirely.max_tool_calls_per_turndefaults to 64: calls past it in one modelmessage are answered with an error and not run.
0removes the cap.tool_definitions_ratiodefaults to 0.25 (it was 0.05) and is nowenforced by deferral; before, it was derived into a budget nothing read. It
only matters once groups are declared.
sorted into it. The context manager's
pin_toolandpinned_tool_namesarereplaced by
discover_tool,note_tool_used,discovered_tool_namesandevict_discovered_tools.tool_surface_advertisedcarriesdiscovered_tool_namesin place oftoolsearch_pinned_tool_names, and the sourcediscoveredin place oftoolsearch_pin.IToolRegistrygainsdeclare_group,undeclare_groupandtool_groups,and
searchandcompute_effective_surfacetake an optionalretrieval.A third-party registry must add them.
Checklist
uv run pytest .passes: 3908 passed, 139 skipped, coverage 93.30%. Every commit also passes on itsown.
uv run ruff check .is cleanuv run mypy --strictis clean (353 source files, no path argument)uv run bandit -r protocore -q -c pyproject.tomlis cleantest_tool_retrieval.py,test_tool_retrieval_quality.py,test_tool_retrieval_devops_ru.py,test_tool_deferral.py(unit andscenario),
test_tool_group_loads.py,test_tool_search.py,test_tool_call_cap.py,test_tool_dispatch.py,test_snapshot_upcasters.pyRuntimeConstantsfields with a description. Thechanged defaults are listed above.
docs/tools.md,docs/runtime-constants.md,docs/architecture.md, each in both languages, the READMEs and thechangelog