Skip to content

[AI] Vulnerability tools are never selected — model routes CVE questions to posture/patch tools instead #2605

Description

@ToddHebebrand

Description

With vulnerability findings seeded (real CVEs in vulnerabilities + device_vulnerabilities), the org-scoped AI chat never chooses get_vulnerability_report / get_device_vulnerabilities, even when the prompt uses the tools' own vocabulary:

  • "Do any of our devices have known vulnerable software? Give me the highest-risk issues."
  • "Check our vulnerability findings: which open CVEs do we have across the fleet?"
  • "Give me a vulnerability report: our open CVE findings, counts by severity..."

All three routed to get_security_posture, manage_patches, manage_alerts, get_s1_threats, get_huntress_incidents (and once get_cis_compliance, see #2603) — so the answers described patch KBs and posture scores while the actual CVE findings (including known-exploited entries) were never surfaced.

Likely a tool-selection problem at ~100 registered tools: the vuln tools' descriptions don't win against the louder security tools.

Proposed Fix

  • Tune get_vulnerability_report/get_device_vulnerabilities descriptions to claim the CVE/vulnerability-findings vocabulary explicitly (and cross-reference them from get_security_posture's description).
  • Consider a system-prompt hint mapping common intents to canonical tools, or evaluating tool-choice quality as part of the flagged-chat review loop.

Affected Files

  • apps/api/src/services/aiToolsVulnerability.ts (descriptions)
  • apps/api/src/services/aiAgent.ts (system prompt)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions