Skip to content

docs(audit): reconcile LLM-tool descriptions with [Description] attributes in code#20

Merged
SchwartzKamel merged 1 commit into
mainfrom
docs/audit-2026-05-llm-tool-coherence
May 3, 2026
Merged

docs(audit): reconcile LLM-tool descriptions with [Description] attributes in code#20
SchwartzKamel merged 1 commit into
mainfrom
docs/audit-2026-05-llm-tool-coherence

Conversation

@SchwartzKamel

Copy link
Copy Markdown
Owner

Pass 3 of 2026-05 docs audit — LLM-tool doc-vs-code coherence

Cross-checked every [Description]-decorated AIFunction registered by the LLM tool catalog against doc claims in docs/MODEL_BEHAVIOR.md, docs/LEARNING_LOOP.md, docs/FIGHTS.md, AGENTS.md, and .github/copilot-instructions.md.

Tool inventory verified (14 LLM-visible tools)

Recon (LlmToolCatalog over ReconToolbox):
nmap_scan, http_probe, tls_probe, dns_probe, smb_probe, ftp_probe, ssh_probe, snmp_probe, ldap_probe, rpc_probe, kerberos_probe, dns_zone_transfer, http_content_discovery, tls_cipher_enum (conditionally registered via AddIf).

Exploit (LlmExploitTools.BuildAiFunctions):
exploit_plan, execute_cred_spray, run_post_ex_linux, run_post_ex_windows, pivot_and_scan, extract_flags_from_dir, run_multi_stage, generate_macro, deliver_phish_smb, deliver_phish_webdav, start_phish_stager, exec_shell, get_cms_chain_template.

Notebook (LlmNotebookTool):
take_note — params category, body, tags[], target_host.

Drift fixed

Tool name Code source Doc claim (before) Drift fixed
execute_cred_spray src/Drederick/Agent/LlmExploitTools.cs:238 docs/MODEL_BEHAVIOR.md:328 listed it as password_spray Replaced placeholder row with full 13-tool list + permission-gate summary; pointed at [Description] as the canonical wording source.

Note: The strings password-spray in docs/DB_SCHEMA.md:166, docs/FIGHTS.md:137, and docs/MODEL_BEHAVIOR.md:97 refer to the underlying IExploitTool.Name (the kebab-case tool registered with ExploitToolbox), not the LLM-visible AIFunction name. Those are correct and were not touched.

take_note cross-check

LlmNotebookTool.TakeNoteAsync signature:

  • category (string), body (string), tags? (string[]), target_host? (string)

Doc claims in docs/MODEL_BEHAVIOR.md:329 ((category, body, tags[], optional target_host)), docs/LEARNING_LOOP.md:205-247, and docs/FIGHTS.md:187-208 all match the [Description] strings on the method and parameters. No drift.

Code-side flags

None. Every [Description] attribute on LlmExploitTools.cs and LlmNotebookTool.cs is consistent with its method/parameter signature and behavior. No stale parameter descriptions found.

Files changed

  • docs/MODEL_BEHAVIOR.md — one row in the LLM-visible tools table.

No code touched. No other docs altered.

@SchwartzKamel SchwartzKamel merged commit 45ff059 into main May 3, 2026
1 check passed
@SchwartzKamel SchwartzKamel deleted the docs/audit-2026-05-llm-tool-coherence branch May 3, 2026 01:23
…butes in code

Pass 3 of the 2026-05 docs audit: cross-checked the LLM-visible
AIFunction inventory in src/Drederick/Agent/LlmExploitTools.cs and
LlmNotebookTool.cs against doc claims.

Drift fixed:
- docs/MODEL_BEHAVIOR.md L328: example tool list named 'password_spray',
  but the actual AIFunction registered by LlmExploitTools.BuildAiFunctions
  is 'execute_cred_spray' (LlmExploitTools.cs:238). The 'password-spray'
  string elsewhere (DB_SCHEMA.md, FIGHTS.md, MODEL_BEHAVIOR.md L97)
  refers to the underlying IExploitTool name and is correct as-is.

Replaced the placeholder row with the full LLM-visible tool list
(13 wrappers) plus the per-flag permission gate summary, and pointed
at the [Description] attribute as the canonical wording source.

No code changes. No other doc-vs-code drift found in the LLM-tool
surface (take_note params match LlmNotebookTool.TakeNoteAsync exactly:
category, body, tags[], target_host).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant