From c8beba90bc2033656d9c48eee33476e705cfbaba Mon Sep 17 00:00:00 2001 From: Chris Pezza <153260989+chrispezza@users.noreply.github.com> Date: Mon, 12 Oct 2026 14:51:32 -0400 Subject: [PATCH] fix(checks): make the manifest check graduatable and graduate it The manifest check reported a finding whenever the claude CLI was absent, so at block status it would have stopped every local session on a machine without the CLI. That kept it at warn in the first graduation. - Report a missing CLI as a note locally; frontmatter fields are still checked - Add --require-validator, passed by CI, which makes a missing CLI a finding so the official validator cannot silently stop running there - Graduate manifest warn -> block; append the log entry to ADR 0008 - Add unit tests for the three cases Closes #63 Co-Authored-By: Claude Opus 5.5 --- .github/workflows/validate.yml | 8 +++--- CHANGELOG.md | 1 + checks/README.md | 1 + checks/enforcement.config.json | 2 +- checks/run_checks.py | 16 +++++++++-- checks/tests/test_manifest_check.py | 39 +++++++++++++++++++++++++++ docs/adrs/0008-plugin-architecture.md | 15 +++++++++++ docs/adrs/README.md | 2 +- 8 files changed, 76 insertions(+), 8 deletions(-) create mode 100644 checks/tests/test_manifest_check.py diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml index 19a4a33..361fd8e 100644 --- a/.github/workflows/validate.yml +++ b/.github/workflows/validate.yml @@ -50,9 +50,9 @@ jobs: run: pip install -r plugin/scripts/requirements.txt # The manifest check (ADR 0008 TC-1) wraps the official validator, - # `claude plugin validate --strict`, and degrades to a warning when the - # CLI is absent. Install it so CI runs the real validator instead of - # reporting a standing warning. Pinned exactly (no lockfile covers a + # `claude plugin validate --strict`. Locally a missing CLI is only a + # note; here `--require-validator` makes it a finding, so the validator + # cannot silently stop running in CI. Pinned exactly (no lockfile covers a # global install); bump deliberately, like any other pinned tool. - uses: actions/setup-node@v7 with: @@ -70,7 +70,7 @@ jobs: env: # No update checks or telemetry from a CI-only validator run. CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC: '1' - run: python checks/run_checks.py --ci --json enforcement-report.json + run: python checks/run_checks.py --ci --require-validator --json enforcement-report.json # ADR 0018: push the suite's finding count to the ops dashboard, which # expects `audit.finding_count` from plugin_skill repos. Push-only so a diff --git a/CHANGELOG.md b/CHANGELOG.md index 28c8511..01aef34 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `skills-claude-ai/` — Claude.ai Agent Skill ports of product-ideation and product-flow (issue #12): bundled prompts with skill-relative paths, conversation-scoped artifact ledger with `product-dev-artifacts.md` export replacing the `.product-dev/` registry ### Changed +- `manifest` check: a missing `claude` CLI is now a note locally and a finding only under `--require-validator`, which CI passes. With that fixed the check graduates warn → block (issue #63); 13 of 18 checks now block - Plugin 0.5.1: tier escalation is now offered, not applied, in every workflow skill. Four rules that existed only in this repo's `CLAUDE.md` ship in `plugin/docs/registry-operations.md` § Workflow Rules (offer Tier 2 prompts and let the user pick; never auto-escalate to Tier 3; ask when a gate condition is unclear; entry points are exclusive). `product-ideation` previously inserted Tier 2 prompts on a signal; `product-flow` and `tech-spec` had no escalation procedure (issue #62) - `CLAUDE.md` is now the contributor guide (ADR 0019): about 650 words of pointers and change rules, down from about 2,000 that restated the skills, the tier model, the plugin tree, and the ADR list. Four runtime rules that never shipped in the plugin stay in a labelled section pending issue #62 - PreToolUse guard: an ADR is protected only once it exists in `HEAD`, so a draft can be revised before its first commit diff --git a/checks/README.md b/checks/README.md index 5d5e1e9..af306db 100644 --- a/checks/README.md +++ b/checks/README.md @@ -13,6 +13,7 @@ python checks/run_checks.py --json report.json # + machine-readable report python checks/run_checks.py --check word-limit # single check python checks/run_checks.py --list # checks, statuses, graduations python checks/run_checks.py --ci # exit 1 iff BLOCKERs (CI mode) +python checks/run_checks.py --ci --require-validator # as CI runs it: a missing claude CLI is a finding python checks/adr_index.py --write # regenerate the ADR index ``` diff --git a/checks/enforcement.config.json b/checks/enforcement.config.json index 60e921c..b488d73 100644 --- a/checks/enforcement.config.json +++ b/checks/enforcement.config.json @@ -9,7 +9,7 @@ { "id": "placeholder-resolvability", "adr": "0003", "tc": "0003:TC-3", "status": "block", "added": "2026-07-12", "graduated": "2026-10-12" }, { "id": "dependency-graph", "adr": "0003", "tc": "0003:TC-4", "status": "block", "added": "2026-07-12", "graduated": "2026-10-12" }, { "id": "word-limit", "adr": "0009", "tc": "0009:TC-5", "status": "warn", "added": "2026-07-12", "graduated": null }, - { "id": "manifest", "adr": "0008", "tc": "0008:TC-1,TC-3", "status": "warn", "added": "2026-07-12", "graduated": null }, + { "id": "manifest", "adr": "0008", "tc": "0008:TC-1,TC-3", "status": "block", "added": "2026-07-12", "graduated": "2026-10-12" }, { "id": "component-census", "adr": "0004, 0008, 0013", "tc": "0004:TC-1,TC-2,TC-3,TC-4; 0008:TC-2", "status": "block", "added": "2026-07-12", "graduated": "2026-10-12" }, { "id": "no-legacy", "adr": "0001, 0002, 0005, 0007", "tc": "0001:TC-2; 0002:TC-1; 0005:TC-1; 0007:TC-1,TC-2", "status": "block", "added": "2026-07-12", "graduated": "2026-10-12" }, { "id": "self-containment", "adr": "0011", "tc": "0011:TC-1,TC-2", "status": "warn", "added": "2026-07-12", "graduated": null }, diff --git a/checks/run_checks.py b/checks/run_checks.py index 6d81c77..fa9a548 100644 --- a/checks/run_checks.py +++ b/checks/run_checks.py @@ -258,8 +258,17 @@ def check_manifest(ctx): if r.returncode != 0: detail = (r.stdout + r.stderr).strip().splitlines() out.append(finding("0008", "TC-1", f"claude plugin validate --strict failed: {detail[-1] if detail else 'no output'}", "plugin/.claude-plugin/plugin.json")) + elif ctx.get("require_validator"): + # CI passes --require-validator: there the CLI is installed on purpose, + # so its absence means the official validator silently stopped running. + out.append(finding("0008", "TC-1", "claude CLI unavailable but --require-validator is set — the official validator did not run; restore the CLI install step", "plugin/.claude-plugin/plugin.json")) else: - out.append(finding("0008", "TC-1", "claude CLI unavailable — official validator skipped; run locally with `claude plugin validate --strict ./plugin`", "plugin/.claude-plugin/plugin.json")) + # Locally the CLI is optional. Its absence is an environment fact, not + # a defect in the plugin, so it is a note and never a finding. + ctx["notes"]["manifest"] = ( + "claude CLI unavailable — official validator skipped (frontmatter fields " + "still checked); run `claude plugin validate --strict ./plugin` where the CLI exists" + ) surfaces = ( [(f, SKILL_FIELDS, "skill") for f in sorted(PLUGIN.glob("skills/*/SKILL.md"))] + [(f, COMMAND_FIELDS, "command") for f in sorted(PLUGIN.glob("commands/*.md"))] @@ -498,6 +507,8 @@ def main(): ap.add_argument("--check", metavar="ID", help="run a single check by id") ap.add_argument("--ci", action="store_true", help="exit 1 iff any BLOCKER") ap.add_argument("--list", action="store_true", help="list checks and statuses") + ap.add_argument("--require-validator", action="store_true", + help="treat a missing claude CLI as a manifest finding (CI sets this)") args = ap.parse_args() config = load_config() @@ -514,7 +525,8 @@ def main(): print(f"config references unknown check(s): {sorted(unknown)}", file=sys.stderr) return 2 - ctx = {"prompts": load_prompts(set(config.get("exclude", []))), "notes": {}} + ctx = {"prompts": load_prompts(set(config.get("exclude", []))), "notes": {}, + "require_validator": args.require_validator} to_run = [args.check] if args.check else list(CHECK_FUNCS) if args.check and args.check not in CHECK_FUNCS: print(f"unknown check: {args.check}", file=sys.stderr) diff --git a/checks/tests/test_manifest_check.py b/checks/tests/test_manifest_check.py new file mode 100644 index 0000000..e8cf1f4 --- /dev/null +++ b/checks/tests/test_manifest_check.py @@ -0,0 +1,39 @@ +"""Tests for how the manifest check treats a missing claude CLI (ADR 0008 +TC-1, issue #63). Run: python -m unittest discover -s checks/tests +""" +import sys +import unittest +from pathlib import Path +from unittest import mock + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +import run_checks # noqa: E402 + + +class ManifestCliCase(unittest.TestCase): + def run_without_cli(self, require_validator): + ctx = {"notes": {}, "require_validator": require_validator} + with mock.patch.object(run_checks.shutil, "which", return_value=None): + return run_checks.check_manifest(ctx), ctx + + def test_missing_cli_is_a_note_locally_not_a_finding(self): + findings, ctx = self.run_without_cli(require_validator=False) + self.assertEqual([f for f in findings if "CLI" in f["message"]], []) + self.assertIn("validator skipped", ctx["notes"]["manifest"]) + + def test_missing_cli_is_a_finding_when_the_validator_is_required(self): + findings, ctx = self.run_without_cli(require_validator=True) + self.assertEqual(len([f for f in findings if "--require-validator" in f["message"]]), 1) + self.assertNotIn("manifest", ctx["notes"]) + + def test_validator_failure_is_a_finding(self): + failed = mock.Mock(returncode=1, stdout="✘ bad manifest\n", stderr="") + ctx = {"notes": {}, "require_validator": False} + with mock.patch.object(run_checks.shutil, "which", return_value="/usr/bin/claude"), \ + mock.patch.object(run_checks.subprocess, "run", return_value=failed): + findings = run_checks.check_manifest(ctx) + self.assertEqual(len([f for f in findings if "validate --strict failed" in f["message"]]), 1) + + +if __name__ == "__main__": + unittest.main() diff --git a/docs/adrs/0008-plugin-architecture.md b/docs/adrs/0008-plugin-architecture.md index 925df87..946ba85 100644 --- a/docs/adrs/0008-plugin-architecture.md +++ b/docs/adrs/0008-plugin-architecture.md @@ -137,3 +137,18 @@ consecutive clean days — zero findings in every CI run since 2026-07-27. `claude` CLI is absent, so at block it would stop every local session on a machine without the CLI. `checks/enforcement.config.json` is authoritative for current status. + +--- + +## Graduation log entry (2026-10-12, second): `manifest` + +`manifest` (TC-1, TC-3): **warn → block** (issue #63). The obstacle recorded +in the entry above is removed: a missing `claude` CLI is now reported as a +note, not a finding, so it cannot stop a local session. CI runs the suite +with `--require-validator`, which turns a missing CLI back into a finding +there, so the official validator cannot silently stop running. Basis +(ADR 0012 §5): 7+ consecutive clean days — zero findings from this check in +every CI run since the CLI was installed in CI on 2026-09-02. + +TC-1 now reads: `claude plugin validate --strict ./plugin` exits 0 wherever +the CLI is available, and it is always available in CI. diff --git a/docs/adrs/README.md b/docs/adrs/README.md index be8f762..33e1b90 100644 --- a/docs/adrs/README.md +++ b/docs/adrs/README.md @@ -17,7 +17,7 @@ listed under "Amended by" override its earlier text. | [0005](0005-prompt-enhancement-schema.md) | Prompt Enhancement Schema | Deferred | `no-legacy` (block) | — | — | | [0006](0006-progressive-disclosure-tiered-engagement.md) | Progressive Disclosure and Tiered Engagement | Accepted | `frontmatter-v2` (block) | 0019 | 3, latest 2026-10-12 | | [0007](0007-json-index-deprecation.md) | Deprecation of JSON Index Files and Stream Deck Document | Accepted | `no-legacy` (block) | — | — | -| [0008](0008-plugin-architecture.md) | Plugin Architecture | Accepted | `manifest` (warn), `component-census` (block) | 0011 | 1, latest 2026-10-12 | +| [0008](0008-plugin-architecture.md) | Plugin Architecture | Accepted | `manifest` (block), `component-census` (block) | 0011 | 1, latest 2026-10-12 | | [0009](0009-prompt-enhancement-pattern.md) | Prompt Enhancement Pattern v2 | Accepted | `frontmatter-v2` (block), `name-uniqueness` (block), `body-structure` (block), `placeholder-syntax` (block), `word-limit` (warn) | — | 1, latest 2026-07-25 | | [0010](0010-spec-package-compilation-target.md) | Spec Package as Compilation Target | Accepted | `spec-fixture` (block) | — | — | | [0011](0011-plugin-self-containment.md) | Plugin Self-Containment via Bundled Assets | Accepted | `self-containment` (warn) | — | 1, latest 2026-07-25 |