Repository navigation
llm(claude-config-validator): package validate-ai and validate-ai-local as skills - #288
SaintPatrck wants to merge 6 commits into
Conversation
9951293 to
5c5258e
Compare
Claude Code validationResult: Pass Re-ran against the same merge base as the previous round ( Not covered: Findings
Already raised on the pull request — not re-posted:
Dropped after documentation check:
Fixed since the previous roundBoth findings from the 15:30 round are confirmed fixed at
Checked and clean
|
🤖 Bitwarden Claude Code ReviewOverall Assessment: APPROVE This PR moves Not covered: Skill review did not run. This review path cannot launch |
33beb0b to
562a75a
Compare
562a75a to
9bb1beb
Compare
| --- | ||
| name: validate-ai | ||
| argument-hint: "[PR#] | [PR URL] | (blank for the checked-out PR)" | ||
| allowed-tools: Read, Edit(//tmp/validation-summary.md), Grep, Glob, Task, Skill, WebFetch(domain:code.claude.com), Bash(gh pr view:*), Bash(gh pr diff:*), Bash(git rev-parse:*), Bash(printenv GITHUB_ACTIONS), Bash(ls:*) |
There was a problem hiding this comment.
allowed-tools grant out of the only pass in this plugin's own pipeline that audits tool grants.
Details and fix
This is a different claim from the settled model-invocability threads above — it is about which validation bucket the file lands in, not about who may invoke it. The grant on line 4 is unchanged by this PR; what changed is whether anything reviews it.
At the old path, commands/validate-ai/validate-ai.md matched the Command-files bucket (skills/reviewing-claude-config/reference/validate-ai-scope.md:45) and routed to reviewing-command-definitions (skills/reviewing-claude-config/SKILL.md:92), which owns the grant check — "Where the command declares allowed-tools, check the grant against what the body actually runs."
At the new path it matches the Skill-files bucket, which the router marks "Not reviewed here" (skills/reviewing-claude-config/SKILL.md:95), and step 4b of this very file scopes skill-reviewer to "skill frontmatter, description and trigger quality, word count and writing style, progressive disclosure, and referenced files that do not exist" — no permission scoping. Line 149 then says "4c deliberately does not review SKILL.md files."
So the grant on line 4 — five Bash(...) prefixes plus Edit(//tmp/validation-summary.md) on a fixed path in a world-writable directory — and the matching grant in skills/validate-ai-local/SKILL.md:4 are now covered by no pass in the pipeline these files define. Nothing is exposed today; a future widening of either line would pass this pipeline unexamined. The broader gap (no skill's allowed-tools is grant-audited) predates this PR — the move is what pulls these two files into it.
Fix — narrow the carve-out at lines 149–150 so it excludes only the duplicate-prose concern, not the grant pass:
4c deliberately does not review `SKILL.md` prose — a second rule set over the same file
produces duplicate findings a reader cannot tell from independent confirmation. A changed
`SKILL.md` that declares `allowed-tools` is still routed to `reviewing-command-definitions`
for the tool-grant pass, which 4b does not cover.Reference: https://code.claude.com/docs/en/skills (allowed-tools grants tools for the invoking turn)
validate-ai
There was a problem hiding this comment.
This gap predates the move. No skill's allowed-tools gets grant-audited today, and the suggested carve-out wouldn't close it anyway, since the reviewing-claude-config router still marks SKILL.md as not reviewed there.
Grant-auditing skills belongs in its own change that covers every skill, not just these two.
| @@ -16,7 +17,7 @@ reach the diff as inline comments. Follow it exactly. | |||
|
|
|||
| For a review of your own uncommitted work, use `/validate-ai-local` instead. | |||
There was a problem hiding this comment.
🎨 SUGGESTED: Neither moved file points at its sibling README.md, which the move turned into a skill-directory support file.
Details and fix
Raised independently by the skill reviews of both relocated files. Under commands/validate-ai/, the sibling README.md was plainly human documentation sitting next to a command, and the router excludes a command's sibling README from review by name. Under skills/validate-ai/ it is a bundled file inside a skill directory — the layout progressive disclosure is built around — and nothing in SKILL.md routes a reader to it.
Both READMEs hold material the running skill cannot see and would benefit from: the plugin-dev install requirement, the reasoning behind each allowed-tools grant, and troubleshooting entries for the exact failure modes each SKILL.md tells it to handle (skills/validate-ai/README.md:120-182, skills/validate-ai-local/README.md:63-179).
Fix — add one pointer to each. After this line in skills/validate-ai/SKILL.md:
Requirements, the reasoning behind each pre-approved tool, and troubleshooting are in
`${CLAUDE_PLUGIN_ROOT}/skills/validate-ai/README.md`.and the equivalent line in skills/validate-ai-local/SKILL.md, pointing at its own README.
Reference: https://code.claude.com/docs/en/skills (progressive disclosure through bundled files)
validate-ai
There was a problem hiding this comment.
READMEs moved to plugin docs/ folder and renamed to reflect the process they describe.
…al as skills Skills are the marketplace's standard packaging for user-invocable workflows. The slash names stay the same, so the validate-ai action and anyone running them by hand are unaffected.
9bb1beb to
22f7936
Compare
| IMPORTANT / SUGGESTED / OPTIONAL classification. The skill owns the single write of the | ||
| report document, the labels it gives each severity, and any inline comments. |
There was a problem hiding this comment.
Details and fix
Before this edit the sentence read "The command owns the single write of the report document, the labels it gives each severity, and any inline comments." The noun did the disambiguating work: this file is a skill, its caller was a command, so "the command owns ... any inline comments" could only mean the caller.
Now both actors are skills, and the sentence reads "The skill owns ... any inline comments" seven lines after the file's own prohibition at line 170:
This skill produces findings. It does not deliver them anywhere, so never post a comment, even where a comment-posting tool happens to be available: callers that post run the findings through their own classification and validation first, and posting directly would bypass that.
Throughout the rest of this file, bare "this skill" / "the skill" refers to reviewing-claude-config itself — line 170, line 15 ("no step here depends on it"), line 268 ("those skills, which hold grants this skill does not"). So the nearest-referent reading of line 177 is that reviewing-claude-config owns the report write and the inline comments, which is the precise behavior line 170 exists to prevent. The stated consequence of getting it wrong is in line 170 itself: findings reach the pull request without the caller's classification and validation pass.
This is the one edit in this file where the rename removed information rather than just updating a label. The other five all sit next to an explicit validate-ai qualifier ("the two validate-ai skills", "both validate-ai skills"), which keeps them unambiguous.
Fix — restore the disambiguator by naming which skill:
| IMPORTANT / SUGGESTED / OPTIONAL classification. The skill owns the single write of the | |
| report document, the labels it gives each severity, and any inline comments. | |
| IMPORTANT / SUGGESTED / OPTIONAL classification. The calling `validate-ai` skill owns the | |
| single write of the report document, the labels it gives each severity, and any inline | |
| comments. |
validate-ai
There was a problem hiding this comment.
Addressed by removing the unnecessary reasoning.
| For a review of your own uncommitted work, use `/validate-ai-local` instead. | ||
|
|
||
| **This command produces a report. It does not fix anything.** The report is the deliverable | ||
| Requirements and the reasoning behind each pre-approved tool are in `${CLAUDE_PLUGIN_ROOT}/skills/validate-ai/README.md`. |
There was a problem hiding this comment.
🎨 SUGGESTED: The two added README pointers are the only unwrapped lines in either file, and nothing in the toolchain will rewrap them.
Details and fix
This line is ~121 characters; the prose around it wraps at ~88, consistently, for the whole file. The same applies to the matching line added in skills/validate-ai-local/SKILL.md:17.
Worth fixing by hand rather than leaving to the formatter: there is no Prettier config in the repo, so prettier --check . runs with the default proseWrap: "preserve" and will not reflow prose. pnpm run lint passes either way, which means this does not get corrected on its own — it just makes these two lines render as outliers in every future diff of a file whose wrap is otherwise uniform.
Fix — wrap to match the surrounding prose:
| Requirements and the reasoning behind each pre-approved tool are in `${CLAUDE_PLUGIN_ROOT}/skills/validate-ai/README.md`. | |
| Requirements and the reasoning behind each pre-approved tool are in | |
| `${CLAUDE_PLUGIN_ROOT}/skills/validate-ai/README.md`. |
and the equivalent break in skills/validate-ai-local/SKILL.md:17.
validate-ai
There was a problem hiding this comment.
Addressed by removing the unnecessary reasoning from the Skill.
| name: validate-ai-local | ||
| argument-hint: "[base-ref] (defaults to the repository default branch)" | ||
| allowed-tools: Read, Edit(~/.claude/plugins/data/claude-config-validator*/ai-validation/*), Grep, Glob, Task, Skill, WebFetch(domain:code.claude.com), Bash(git diff:*), Bash(git fetch origin:*), Bash(git rev-parse:*), Bash(git symbolic-ref:*), Bash(git ls-files:*), Bash(date:*), Bash(ls:*) | ||
| description: Validate the Claude Code material you changed locally and write a timestamped report to the plugin's data directory |
There was a problem hiding this comment.
🎨 SUGGESTED: Both relocated descriptions are still written as /help labels, but under skills/ the description is what the model matches on.
Details and fix
Independent of the two settled threads above — this is not about whether the model should invoke these, but about the description doing its job given that it may. It applies the same either way, and more so if autonomous loading is intended.
The docs make description load-bearing specifically for the skills/ layout:
Your existing
.claude/commands/files keep working. Skills add optional features: a directory for supporting files, frontmatter to control whether you or Claude invokes them, and the ability for Claude to load them automatically when relevant.
Both strings came across from commands/ unchanged, where they only had to render in a list. This file's version spends roughly half its length on the output location and never says when to use it — and CHANGELOG.md:124 records that it was tuned for exactly that older job ("names the report location, which is the string /help shows").
It also lost the asymmetry race against its sibling: skills/validate-ai/SKILL.md:5 enumerates "(plugins, skills, agents, commands, hooks, CLAUDE.md, .claude/)" while this one does not, so "check my skills and agents before I push" has more lexical overlap with the pull request skill than with the local one.
This is the plugin's own published rule, in skills/reviewing-claude-config/reference/claude-code-requirements.md:30:
description: MUST include activation triggers (when to use the skill)
and reviewing-claude-config/SKILL.md:3 already follows it ("Use when reviewing changes to CLAUDE.md, agents, prompts, commands, hooks, or settings").
Fix — add the enumeration and a use-when clause here:
description: Validate the Claude Code material changed in the local checkout (plugins, skills, agents, commands, hooks, CLAUDE.md, .claude/), covering branch commits plus uncommitted work. Use before pushing, or when asked to validate, check, or review Claude config locally.skills/validate-ai/SKILL.md:5 has a milder version of the same gap — it enumerates the material but still never says when to use it; appending "Use when asked to validate or review a pull request's Claude configuration, by PR number or URL." would close it.
Reference: https://code.claude.com/docs/en/skills
validate-ai
There was a problem hiding this comment.
Speculative. Have not observed the described failure, so this will stay as-is. If triggering misses are observed we will address at that time.
🎟️ Tracking
Stacked on #276, from a review suggestion there to keep the repo consistent.
📔 Objective
Skills are how the rest of the marketplace packages user-invocable workflows, so we move
/validate-aiand/validate-ai-localfromcommands/toskills/to match. The slash names are unchanged, so the validate-ai action's/claude-config-validator:validate-aiprompt and anyone running them by hand keep working.🤖 AI-assisted review
A Standard review (code-review-local) ran on the branch and approved. Its one finding, stale "command" wording in the intentionally duplicated notes, is fixed on the branch. It could not run
plugin-dev:skill-reviewer, so the two movedSKILL.mdfiles have not had a skill-quality pass.