Skip to content

pickTool: name the chosen tool in the error and match tool names leniently - #64

Open
walcz-de wants to merge 1 commit into
mudler:mainfrom
walcz-de:fix/picktool-lenient-match-and-error
Open

walcz-de wants to merge 1 commit into
mudler:mainfrom
walcz-de:fix/picktool-lenient-match-and-error

Conversation

@walcz-de

@walcz-de walcz-de commented Jul 7, 2026

Copy link
Copy Markdown

Two small, related robustness fixes in tools.go for local-model tool selection. We hit these repeatedly in production (agent pool via LocalAI) — 4 hard turn-failures in one evening across two different Qwen-class local models.

1. The error hides the chosen name (pickTool)

When the model's chosen tool can't be resolved, pickTool returned a bare chosen tool not found. The name was only xlog.Debug-logged, so operators running at info level saw repeated failures with zero diagnosability. Now: chosen tool %q not found.

2. Tools.Find matched exact-only

Local models routinely emit namespaced / case / separator variants — functions.coach_regeln, Coach_Regeln, tools/coach_regeln — each of which missed and became a hard failure. Find now keeps exact match as the fast path and adds a lenient fallback that only triggers on a miss: it strips a leading namespace segment (functions./tools//ns::), case-folds, and treats - ≡ _. When an exact match exists it is always returned, so existing resolutions are unaffected (covered by a test).

Tests

Table-driven tests for normalizeToolName, lenient Find, and the exact-wins-over-lenient guarantee. go test, go vet, gofmt clean.

Related in spirit to #60 / #62: helper/selection steps benefit from diagnosability and tolerance of model quirks. Happy to adjust (e.g. gate the lenient match behind an option) if preferred.

🤖 Generated with Claude Code

@mudler-agent

Copy link
Copy Markdown
Contributor

Thanks for this. Lenient matching helps local models, but as written it lets the name that approval sees differ from the tool that runs, which matters for embedders that gate tool calls by name.

The approved name is not the executed tool. pickTool stores the model's spelling in ToolChoice.Name (tools.go ~L837 on main: Name: toolName), and the tool-call callback receives that ToolChoice (~L1693). Execution then resolves it again with tools.Find(tc.Name) (~L1810, and ~L1862/L1911 on the other paths), which this PR makes lenient. So a model that calls Bash or functions.bash gets the bash tool executed, while the callback sees Bash.

In nib (the main embedder), a PreToolUse hook configured for bash then does not fire for that call. In auto-approve mode, hooks are the only gate. Name-based approval rules also stop matching; those fall back to prompting, which is safe but noisy.

Ambiguous matches. normalizeToolName keeps only the segment after the last ., / or :, so github/list_issues and gitlab/list_issues both become list_issues, and Find returns whichever comes first.

Suggested fix:

  1. When the lenient pass matches, rewrite ToolChoice.Name to the resolved tool's real name before any callback runs (in pickTool and in the native tool-call path), so what gets approved is what runs.
  2. Only accept a lenient match when exactly one tool matches. On two or more, treat it as not found, and name the candidates in the error.
  3. A test that the callback receives the canonical name for a lenient match, and one for the ambiguous case.

The error-message change (chosen tool %q not found) is good as is.

…ently

Local models routinely call a tool by a variant of its name ("Bash",
"functions.bash", "tools/bash", "coach-regeln"). Tools.Find now falls
back to a normalized match when there is no exact one, and a pickTool
miss names the tool the model chose.

The lenient match is made safe for embedders that gate tool calls by
name (review on mudler#64):

- Before any tool-call callback runs, every selected tool's name is
  rewritten to the real name of the tool it resolves to, in pickTool
  and for native tool calls, so what gets approved is what gets
  executed. A model that calls "Bash" no longer runs "bash" past an
  approval hook configured for "bash".
- A lenient match only counts when exactly one tool matches. When two
  or more tools normalize to the same name ("github/list_issues" and
  "gitlab/list_issues"), the name is not found and the error names the
  candidates.

Tests: the callback receives the canonical name for lenient matches
(three spellings, end to end through ExecuteTools); an ambiguous name
resolves to nothing, the error names both candidates, and end to end
neither tool runs. Without the canonicalization the callback test sees
the model's spelling; without the uniqueness rule the ambiguous tests
pick and run the first tool.

Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
Assisted-by: Claude:claude-opus-5-5
@walcz-de
walcz-de force-pushed the fix/picktool-lenient-match-and-error branch from 201beec to b2f239c Compare September 27, 2026 21:25
@walcz-de

Copy link
Copy Markdown
Author

Thanks for the careful review — both points were real. Updated and rebased on current main (single commit b2f239c):

  1. Approved name = executed tool. Before any tool-call callback runs, every selected tool's name is rewritten to the real name of the tool it resolves to (Tools.canonicalizeToolChoice). That happens in pickTool (both the single and the multiple selection) and once more in ExecuteTools right after the selection, which also covers native tool calls that carry the model's spelling. So a model that calls Bash gets bash both approved and executed, and a PreToolUse hook for bash fires.
  2. Only unique lenient matches. Find accepts a lenient match only when exactly one tool matches. github/list_issues + gitlab/list_issues asked for as list_issues → not found, and the pickTool error names both candidates (chosen tool "list_issues" not found: ambiguous, matches github/list_issues, gitlab/list_issues). Exact names still resolve.
  3. Tests (tools_lenient_test.go): the callback receives the canonical name for Echo, functions.echo, tools/echo end to end through ExecuteTools, and the tool runs once; ambiguous Find returns nil and the error names both candidates; end to end an ambiguous call runs neither tool. Counter-checked: without the canonicalization the callback sees the model's spelling, without the uniqueness rule the first tool is picked and run.

The non-e2e suite passes; "Sub-Agent Spawning — Background agent spawning" fails occasionally, but it does so on untouched main as well (1 in 4 runs on each here), so it's pre-existing timing flakiness rather than this change. The [e2e] specs need a live LocalAI and weren't run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants