Description
With vulnerability findings seeded (real CVEs in vulnerabilities + device_vulnerabilities), the org-scoped AI chat never chooses get_vulnerability_report / get_device_vulnerabilities, even when the prompt uses the tools' own vocabulary:
- "Do any of our devices have known vulnerable software? Give me the highest-risk issues."
- "Check our vulnerability findings: which open CVEs do we have across the fleet?"
- "Give me a vulnerability report: our open CVE findings, counts by severity..."
All three routed to get_security_posture, manage_patches, manage_alerts, get_s1_threats, get_huntress_incidents (and once get_cis_compliance, see #2603) — so the answers described patch KBs and posture scores while the actual CVE findings (including known-exploited entries) were never surfaced.
Likely a tool-selection problem at ~100 registered tools: the vuln tools' descriptions don't win against the louder security tools.
Proposed Fix
- Tune
get_vulnerability_report/get_device_vulnerabilities descriptions to claim the CVE/vulnerability-findings vocabulary explicitly (and cross-reference them from get_security_posture's description).
- Consider a system-prompt hint mapping common intents to canonical tools, or evaluating tool-choice quality as part of the flagged-chat review loop.
Affected Files
apps/api/src/services/aiToolsVulnerability.ts (descriptions)
apps/api/src/services/aiAgent.ts (system prompt)
Description
With vulnerability findings seeded (real CVEs in
vulnerabilities+device_vulnerabilities), the org-scoped AI chat never choosesget_vulnerability_report/get_device_vulnerabilities, even when the prompt uses the tools' own vocabulary:All three routed to
get_security_posture,manage_patches,manage_alerts,get_s1_threats,get_huntress_incidents(and onceget_cis_compliance, see #2603) — so the answers described patch KBs and posture scores while the actual CVE findings (including known-exploited entries) were never surfaced.Likely a tool-selection problem at ~100 registered tools: the vuln tools' descriptions don't win against the louder security tools.
Proposed Fix
get_vulnerability_report/get_device_vulnerabilitiesdescriptions to claim the CVE/vulnerability-findings vocabulary explicitly (and cross-reference them fromget_security_posture's description).Affected Files
apps/api/src/services/aiToolsVulnerability.ts(descriptions)apps/api/src/services/aiAgent.ts(system prompt)