Repository navigation
fix(github-mcp): rank the requested tool first in pi's tool_search - #16
Merged
Merged
Conversation
pi's tool_search ranks deferred tools with BM25 and adds each MCP server's description and instructions to every one of its tools' search text. The old texts listed the nouns the tools are named after (issues, pull requests, runs), so those terms stopped telling tools apart and an exact-name query such as issue_view ranked 5th. The server descriptions in pi/index.ts and the instructions in config-read.json and config-write.json now share no word with any tool name. The vendored SDK does not send the instructions yet, but a later version would otherwise undo the fix. host-pi.md now tells the model to call a loaded tool directly and otherwise search by exact name with limit: 1, searching again when a different tool loads, because tool_search skips loaded tools and the default limit loads up to eight tools per search. search, repo_file, and with the write server pr_comments, pr_reviews and pr_comment still rank 2nd to 5th, since their own descriptions share terms with another tool. tool-search-ranking.test.ts runs pi's own ranker over both tool catalogs, with and without the instructions, and requires every tool to rank first except those five, each with a worst allowed rank. A second test fails when a server text shares a word with a tool name. pi_e2e.bats checks that the real tool_search with limit: 1 loads exactly pr_view and, in a new case, issue_view. The two pi unit test files now share one fake pi in fake-pi.ts. The shared tool directive now lists release_list and the root README lists releases. Both were missing. Closes #15 Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #15.
pi's
tool_searchranks deferred tools with BM25. Each tool's search text includes its server'sdescriptionandinstructions, so every word in those texts appears in every tool of that server. The old texts listed the nouns the tools are named after (issues, pull requests, runs, …). Those words stopped telling tools apart, andissue_viewranked 5th for the queryissue_view.Changes
pi/index.ts):Remote GitHub data through the gh CLI, not local git.andChanges to remote GitHub data through the gh CLI, not local git. Empty unless enable_write_server is true.They share no word with any tool name. "Not local git" keeps some scope in pi'smcp_serverssummary without naming tool categories, because every category word is also a tool name.config-read.json,config-write.json): rewritten under the same rule. The vendored SDK (v5.2.0) does not sendinstructionsyet; once a later SDK does, the old texts would have putissue_viewback at 5th. The read instructions no longer namerepo_tree/repo_file; the tool directive names every tool for every host.hooks/prompts/host-pi.md): while the tools are deferred, call a tool that is already loaded directly; otherwise search by exact name withlimit: 1, and search again if a different tool loads.tool_searchonly ranks tools that are not loaded yet, so the default limit of 8 loaded up to eight tools per lookup.mcp-tool-directives.md): listsrelease_list, which was missing for every host. The root README's category list now includes releases.Remaining misses
Five names share terms with another tool through their own catalog descriptions, so no server text can fix them:
search(up to 3rd),repo_file(3rd), and with the write serverpr_comments(5th),pr_reviews(3rd) andpr_comment(3rd). Underlimit: 1each takes one more search per place it ranks below first.limit: 1is still the cheaper choice: all other tools rank first, so most lookups load one tool. The real fix is in pi's ranker (exact-name matches first, or server text kept out of per-tool documents).