[Scanner]: skills-inspector — offline static analyzer for agent extensions

[atarico]

Scanner name

skills-inspector

Upstream URL

https://github.com/atarico/skills-inspector

Request type

New built-in scanner adapter

CLI/API shape

skills-inspector --json

Prints a single JSON document to stdout. Supports skill and plugin targets
(it walks up from a SKILL.md to the enclosing .claude-plugin/plugin.json,
marketplace.json or opencode.json when one exists, since auditing a SKILL.md
without its manifest produces a false clean).

Note on gating: the exit code is always 0 by design — the tool reports and
has no veto. Gate policy would need to be JSON-path based rather than
blockOnExitCode.

Credentials and setup

None. No API keys, no network access, no LLM calls, no telemetry.
Python 3.10+, standard library only, zero runtime dependencies.

Install: pip install "git+https://github.com/atarico/skills-inspector.git@"

Expected artifact

A JSON report. Each finding carries rule id, severity, confidence,
disclosure status (undeclared / euphemistic / declared), capability,
file+line, sanitized evidence, and impact.

The report also carries mandatory "not_analyzed" and "coverage_limits"
sections — a report that omits what it could not inspect implies a
completeness the tool does not have.

I'm currently adding a published JSON Schema and a schema_version key
before proposing anything concrete.

Example proof

I've been building this for about two months. It's a detection engine
rather than a harness: 111 implemented rules across 14 families, plus
single-file taint tracking (source → sink across variables, redirects and
env exports), a reachability graph that separates active from conditional
and dormant files, structural parsing of the agent control plane
(settings.json, .mcp.json, opencode.json, hooks, subagents, permissions),
and disclosure-gap scoring against the unit's own description.

Corpus: 143 fixtures (56 benign / 82 malicious / 3 documented known-misses),
with exact per-fixture expected finding sets pinned in fixtures/EXPECTED.json.
Those numbers are self-measured and I'd rather not lean on them — I'm working
toward running it against SkillTrustBench and clawhub-security-signals so
there's an external number to compare against clawscan-static on the same
corpus.

Rules catalogue: https://github.com/atarico/skills-inspector/blob/main/RULES.md

I'm not proposing a PR yet. Before that I want to land packaging, the
published schema, and those benchmark numbers. Two questions that would
shape the work:

1. Is a pinned git SHA install acceptable, following the SkillSpector
   precedent in docker/clawscan-runtime/Dockerfile, or is PyPI publication
   expected for a new adapter?
2. Should the runtime-image pin land in the same PR as the adapter, or
   separately? CONTRIBUTING.md asks for one topic per PR, but the adapter
   isn't usable in the default Docker sandbox without the image entry.

Separately: docs/scanners.md and CONTRIBUTING.md both link to
docs/scanners.md#adding-a-built-in-scanner-adapter, but that heading
doesn't exist in the tree. Happy to send a docs-only PR writing that
section if it'd be useful.

[clawsweeper]

Codex review: this still needs some work. Reviewed September 23, 2026, 3:54 PM ET / 19:54 UTC.

Summary
Keep open for a maintainer adoption decision. The requested built-in scanner is absent from current main and v0.2.0. The author has reported packaging, a versioned schema, and public-corpus results, but acknowledges that a shared-corpus comparison is still missing. A merged PR resolved the separate documentation request.

Reproducibility: not applicable. this requests a new built-in integration. Current source and registry tests establish that it is absent, while custom command scanners are supported.

Root-cause cluster
Relationship: canonical
Canonical: #53
Summary: The merged documentation PR addressed a secondary request in this issue; the built-in scanner adoption question remains here.

Members:

• partial_overlap: docs: repair scanner contribution guidance #57 - It merged the missing scanner-contribution guidance and explicitly left adoption undecided.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Maintainer decision needed

Question	Recommendation
Should ClawScan maintain skills-inspector as a built-in scanner, or keep it as a user-defined scanner after evaluation?	Evaluate before bundling: Keep the built-in request open while a pinned custom profile produces shared-corpus and sandbox-coverage evidence.

Why: The execution and JSON-gate mechanisms exist, but accepting another bundled dependency and its output and coverage contract requires a maintainer choice informed by comparable results.

Next step
Built-in adoption needs sponsorship and comparable results; the existing product-decision and no-new-fix-PR signals rule out automated implementation.

Review details

Best possible solution:

Evaluate a pinned custom profile and sandbox image on a shared labeled corpus, then add a registry adapter, fixture tests, and a pinned runtime dependency if maintainers accept the results and upkeep.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this requests a new built-in integration. Current source and registry tests establish that it is absent, while custom command scanners are supported.

Is this the best way to solve the issue?

Unclear for built-in adoption: a custom profile is the narrowest supported evaluation path, but comparable detection and false-positive evidence is still needed to justify bundling.

AGENTS.md: found and applied where relevant.

Remaining risk / open question:

• The reported public-corpus results are self-measured and do not yet show detection and false-positive differences against clawscan-static on the same labeled cases.
• Before adoption, an in-sandbox skill and plugin scan needs to show how enclosing-manifest discovery reports coverage when the mounted target hides parent directories.

Codex review notes: model internal, reasoning high; reviewed against 6d4573ee6187.

Label changes

Label changes:

• add impact:security: The proposed scanner assesses risky capabilities, disclosure gaps, and taint paths in agent extensions.

Label justifications:

• P3: This is an optional scanner-adoption proposal with an existing custom-profile evaluation path and no reported ClawScan regression.
• impact:security: The proposed scanner assesses risky capabilities, disclosure gaps, and taint paths in agent extensions.

Evidence reviewed

What I checked:

• Built-in adapter absent: The current default registry lists the built-in adapters, and its registry test expects nine IDs; neither includes skills-inspector. A current-tree name search also returned no matches outside generated dist files. (internal/runner/scanner_registry.go:201, 6d4573ee6187)
• Runtime dependency absent: The runtime image installs a Git-SHA-pinned SkillSpector dependency but has no skills-inspector entry. (docker/clawscan-runtime/Dockerfile:22, 6d4573ee6187)
• Latest release check: The v0.2.0 registry blob is 1cce62a3de37ed35d28c66deae608e747eb1e37d, identical to the inspected current-main registry; the release does not contain the requested adapter. (internal/runner/scanner_registry.go:201, ef02eb65260b)
• Existing evaluation path: Profile resolution adds user-defined command scanners to the run registry and supports explicit skill and plugin target kinds. (internal/profiles/resolver.go:66, 6d4573ee6187)
• JSON gate coverage: A focused test shows a custom command scanner's JSON can trigger a declarative gate while its raw evidence remains unchanged; this supports the author's always-zero-exit-code design. (internal/profiles/resolver_test.go:1030, 6d4573ee6187)
• Adapter contribution guidance: The repaired guide says to evaluate through a custom profile first, discuss built-in adoption, register an accepted adapter, preserve raw JSON, and pin a runtime dependency when needed. (docs/scanners.md:26, 6d4573ee6187)

Likely related people:

• jesse-merhi: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
• steipete: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
• Patrick-Erichsen: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

[atarico]

Thanks for the review. Two of the three evidence items you named are now on main at 87df0a8. The third is not, and I would rather say so plainly than let "still being prepared" cover it.

Versioned output schema. schemas/report-v1.schema.json is published and every report carries schema_version. A gate validates live output against it on every run, and it also feeds 22 deliberately corrupted reports through the validator to prove the rejection path still rejects (e4d1f82).

Packaging. pyproject.toml with zero runtime dependencies, so a pinned pip install git+https://github.com/atarico/skills-inspector@<sha> behaves the same way your SkillSpector install already does. A guard checks that nothing is declared rather than merely that no lockfile is present (0c7f0b8).

Enclosing-manifest discovery inside a mounted target boundary. This is the part worth reading closely, because validating it found a real defect on my side. Reports now carry scope_search (how the upward walk ended, six values, three of them conclusive) and scope_levels (how many ancestors were actually examined). Under a target-only mount, reached_filesystem_root is reachable in two steps against a filesystem that is not the user's, so the report names how the search ended instead of publishing a bare boolean that reads as proof of absence (c626aba).

Testing exactly the case you raised turned up the defect: a plugin root scanned directly reported scope_search: "depth_limit" with scope_levels: 1, a budget of eight that had never run out. Fixed in 87df0a8. That case is now stopped_at_unit_marker and classified inconclusive, since a marketplace manifest can enclose a plugin and the walk stops before it ever looks.

Benchmark. Not started. I have no numbers to offer yet, so please read the adoption-readiness question as still open on that axis.

On your recommendation, evaluating through a custom-scanner profile before any bundling decision makes sense to me. I am happy to supply a profile definition and a pinned image reference for that, and to keep the built-in adapter question parked until the benchmark evidence exists.

[atarico]

The benchmark axis I left open is now closed enough to report a number, and
the number comes with the reasons it is weaker than it looks.

What was measured. Anthropic's claude-plugins-official marketplace
manifest, 305 entries. 52 carry no git sha — a relative path into the
marketplace repository rather than an external reference — so there is nothing
immutable to fetch and compare against tomorrow. They are excluded, and each
exclusion is recorded with its reason rather than quietly dropped from a
percentage. That leaves 253 entries pinned by exact sha and vendored into the
repository, which resolve to 248 distinct trees (more on that below).

248 discovered   248 scanned   clean 31 (12%)   crashes 0
median 4   mean 16.19   p90 38   max 253
36399 findings total, 4015 of them headline
85 rules produced a finding; 59 produced a headline finding

What "clean" means here, and what it does not. Clean is zero headline
findings — nothing the tool thinks a human needs to decide about before
installing. It is not a malice verdict. The 217 units that are not clean are
overwhelmingly not malicious; the common shape is a real capability that the
unit's own description never mentions. The top headline rules are
NET-001 (560), FSW-004 (335), BND-005 (301), HOK-003 (293),
PRV-001 (274). Read that as "how often a published extension does something
its description does not tell you about", not as "88% of the marketplace is
dangerous". I would rather hand you a number that is boring than one that is
quotable.

Reproducibility. The corpus is pinned by sha and committed, and the result
is frozen in bench/public-baseline.json, so a re-run is a comparison rather
than a fresh anecdote. Nothing is executed: the scanner reads, it never
imports, runs, or network-resolves the target.

A correction, because it bears on how much weight this number carries.
While preparing this I found that my own denominator was wrong. The marketplace
publishes some plugins under more than one name, so 253 listings are really 248
trees — one repository appeared three times. Scanning each name separately
counted the same tree, and the same findings, more than once: 911 findings,
2.4% of the total, were duplicates, and the per-rule census carried the same
inflation. Fixing it reduced the reported finding count, which is to say the
error had been making the ecosystem look worse than it is, and correcting it
made my own headline less impressive. The clean rate did not move (31 of 253 is
12.25%, 31 of 248 is 12.50%); none of the duplicated trees was clean.

Two gaps in this measurement that I would rather state than have found.

1. It is still self-measured. This is my scanner, my corpus choice, and my
   definition of "headline". It is not the external comparison against
   SkillTrustBench or clawscan-static on a shared corpus that I said would be
   the real evidence, and it does not substitute for one. The adoption-readiness
   question stays open on that axis; this only closes "has it ever been pointed
   at something that is not its own fixtures".
2. 33 link entries across 16 trees were dropped before extraction and never
   scanned — symlinks and hardlinks discarded at unpack time as a containment
   measure. Zero containment escapes were recorded. Those 33 entries are counted
   and published rather than rounded away, but they are 33 things the scanner
   did not see, and the clean rate above is computed without them.

Coverage limits, unchanged. 17 rule ids are specified in RULES.md and not
implemented in this build, each listed with a reason. One of them, NET-002, is
CRITICAL and genuinely missing rather than minor. A rule appearing in the
catalogue is not a guarantee that this build enforces it, and a clean report is
not a proof of safety — the report says so itself, in a mandatory section.

On process: your recommendation to evaluate through a custom-scanner profile
before any bundling decision still seems right to me, and this number does not
change that. I am happy to supply the profile definition and a pinned image
reference whenever it is useful, and to leave the built-in adapter question
parked until there is a shared-corpus comparison worth putting next to
clawscan-static.

[steipete]

Thanks for the detailed proposal, @atarico. We're going to pass on a built-in adapter for now. skills-inspector is still settling its rule precision (several false-positive fixes are in flight upstream), and with an always-zero exit code there's no stable gate signal yet; we'd have to invent JSON-path policy around a schema that is still moving.

Happy to revisit once the tool has tagged releases and a documented, stable JSON output schema. Feel free to reopen this then.