Skip to content

feat(m3): crucible compliance-check CLI + run_compliance_harness (M3 PR 16c)#17

Merged
suzuke merged 2 commits into
mainfrom
feat/m3-compliance-check
Apr 26, 2026
Merged

feat(m3): crucible compliance-check CLI + run_compliance_harness (M3 PR 16c)#17
suzuke merged 2 commits into
mainfrom
feat/m3-compliance-check

Conversation

@suzuke

@suzuke suzuke commented Apr 26, 2026

Copy link
Copy Markdown
Owner

Summary

Closes the M3 PR 16 stack (16a codex + 16b gemini + 16c harness/CLI). Ships:

  • run_compliance_harness(adapter, *, tasks, project_dir, ...) — the operator-facing function that produces compliance reports the backend's gate enforcement consumes.
  • crucible compliance-check Click command — wraps the harness for the CLI UX.
  • 22 new tests covering the classification matrix, harness behaviour, persistence, and CLI plumbing.

Stacked on PR 16b (#16)

This PR is stacked on feat/m3-gemini-cli-adapter. Once 16a merges → 16b rebases → 16c rebases. Or merge them in order.

Classification matrix

_classify_trial priority (earliest wins):

  1. raw.timed_outCLI_ERROR
  2. CLISubscriptionAuthError caught → CLI_ERROR (gate run invalid, surface as failing)
  3. raw.exit_code != 0CLI_ERROR
  4. parser raised generic exception → CLI_ERROR
  5. parsed.unknown_schemaPARSE_FAILURE
  6. parsed.tool_was_calledPARSE_SUCCESS
  7. _MODEL_REFUSAL_PHRASES match in description → MODEL_REFUSAL
  8. residual (clean run, no tool, no refusal phrase) → FORMAT_DRIFT

_MODEL_REFUSAL_PHRASES is an explicitly declared tuple (mirrors the auth-phrase pattern from PR 16a/16b — no coincidental substring matching).

Module-level imports (sys.modules-eviction tolerance)

CLISubscriptionAuthError + AdapterRunContext are imported at MODULE level in compliance.py, NOT lazily inside run_compliance_harness. The lazy variant fails when test_agents_factory evicts crucible.agents.* between collection and runtime — the harness's late-resolved class would not match isinstance checks against caller-held pre-eviction instances. Module-level keeps the binding stable.

CLI UX

$ crucible compliance-check --adapter codex-cli --project-dir . --limit 3
Running compliance gate: codex-cli v0.124.0 (3 tasks)

Compliance report:
  adapter      : codex-cli
  cli_version  : 0.124.0
  cli_binary   : /usr/local/bin/codex
  total trials : 3
  passes       : 3 (100.0%)
    parse_success  3

✓ Release threshold met (≥99%)
Report persisted to /home/user/proj/compliance-reports

Stats

git diff --stat HEAD~1 HEAD | tail -1817 insertions / 0 deletions, 3 files.

File +/-
src/crucible/agents/cli_subscription/compliance.py +187
src/crucible/cli.py +153
tests/test_compliance_check_cli.py (new) +477

Test results

2809 passed, 4 skipped (pytest --ignore=tests/test_agents.py). The pre-existing PR-16 regex case mismatch in test_create_agent_unknown_raises remains the only excluded failure across the entire M3 PR 16 stack — recommend opening a 1-line follow-up PR to fix that regex independently.

Test plan

  • _classify_trial matrix: timeout, auth, exit_code, unknown_schema, tool_was_called, refusal phrase, format_drift residual, parser-raise, priority (auth beats schema_drift)
  • Harness happy path: 3 mocked PARSE_SUCCESS trials → pass_rate=1.0, report persisted, fields populated
  • Workspace materialisation: task.workspace_files written to scratch before adapter sees ctx
  • Harness exception handling: subprocess-raise → CLI_ERROR, parser-raise → CLI_ERROR, auth-error → CLI_ERROR with "auth:" prefix
  • Mixed results: heterogeneous trial outcomes accumulate correctly
  • Persistence round-trip: load_reports() reads back what run_compliance_harness wrote
  • CLI smoke: codex-cli adapter construction + run + summary + persistence
  • CLI input validation: --adapter required, unknown adapter rejected, --limit 0 rejected
  • Threshold messaging: release/admit/below-admit lines render correctly

🤖 Generated with Claude Code

suzuke and others added 2 commits April 26, 2026 12:58
…PR 16c)

Closes the M3 PR 16 stack: PR 16a (codex) + PR 16b (gemini) shipped
the adapter classes; PR 16c ships the operator-facing harness runner
+ Click command that PRODUCES the compliance reports the backend's
gate enforcement consumes.

New: `run_compliance_harness(adapter, *, tasks, project_dir, ...)`:
  For each BenignTask, materialises workspace_files into a temp scratch
  dir, builds an AdapterRunContext, calls
  `adapter.run_subprocess(ctx)` then `adapter.parse_output(raw)`,
  classifies via `_classify_trial`, accumulates into ComplianceReport,
  persists to <project>/compliance-reports/, returns the report.

  Classification matrix (`_classify_trial` priority order):
    1. raw.timed_out                          → CLI_ERROR
    2. CLISubscriptionAuthError caught        → CLI_ERROR (gate run is
                                                  invalid, not the
                                                  adapter being non-
                                                  compliant — but we
                                                  surface as failing)
    3. raw.exit_code != 0                     → CLI_ERROR
    4. parser raised generic exception        → CLI_ERROR
    5. parsed.unknown_schema                  → PARSE_FAILURE
    6. parsed.tool_was_called                 → PARSE_SUCCESS
    7. _MODEL_REFUSAL_PHRASES match           → MODEL_REFUSAL
    8. residual (clean run, no tool, no       → FORMAT_DRIFT
        refusal phrase)

  `_MODEL_REFUSAL_PHRASES` is an explicitly declared tuple (mirrors
  the auth-phrase set pattern from PR 16a/16b — no coincidental
  substring matching).

  Module-level (NOT lazy) imports of CLISubscriptionAuthError /
  AdapterRunContext: keeps the binding stable across sys.modules
  eviction by tests like test_agents_factory. A lazy import inside
  the function would resolve to a freshly-imported class object after
  eviction, breaking isinstance checks against pre-eviction instances.

New: `crucible compliance-check` Click command:
  Args: --adapter (claude-code-cli/codex-cli/gemini-cli, required)
        --project-dir (default .)
        --cli-binary-path (override PATH lookup)
        --timeout (per-task seconds, default 60)
        --limit (run only first N tasks, useful for smoke checks)
  Prints adapter+version+pass-rate summary, per-classification
  bucket counts, and a threshold disposition line:
    ✓ Release threshold met (≥99%)
    ⚠ Admit threshold met (≥95%) — POC-eligible only
    ✗ Below admit threshold — investigate

Tests (22 new in test_compliance_check_cli.py):
  - _classify_trial matrix: timeout, auth, exit_code, unknown_schema,
    tool_was_called, refusal phrase, format_drift residual, parser-
    raise, priority (auth beats schema_drift)
  - Harness: happy path (report fields + persistence), workspace
    materialisation, subprocess exception classification, parser-raise
    classification, auth-failure classification, mixed results,
    load_reports() round-trip
  - CLI smoke: codex-cli adapter construction + run + summary +
    persistence; --adapter required; unknown adapter rejected;
    --limit 0 rejected; below-admit message path

Stats: +340 src LOC + 401 test LOC = +741 LOC, 3 files
(`git diff --stat HEAD~1 HEAD | tail -1` after push).

Test results: 2809 passed, 4 skipped (`pytest --ignore=tests/test_agents.py`).
The pre-existing PR-16 regex case mismatch in test_create_agent_unknown_raises
remains the only excluded failure across the entire M3 PR 16 stack.

Stacked on PR 16b (#16). After all three M3-PR-16-stack PRs merge in
order (16a #15 → 16b #16 → 16c this PR), operators can:
  1. `crucible compliance-check --adapter codex-cli` to produce a
     passing report
  2. Configure `agent.cli_subscription.adapter: codex-cli` in their
     project
  3. SubscriptionCLIBackend admits the adapter at run time

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reviewer round 1 caught: `_check_compliance`'s failure message
referenced the Python module path
`crucible.agents.cli_subscription.compliance` instead of the new
`crucible compliance-check` CLI command (which PR 16c just landed).
Same phantom-command failure mode as PR 16 R2, just symmetric — non-
developer users see "Python module path" and don't know what to do.

Updated the message to reference the real CLI:

  Either run `crucible compliance-check --adapter <name>` to produce
  a passing report, or set `experimental.allow_stale_compliance: true`
  to bypass...

Also updated the stale comment ("UX wrapper lands in PR 16a") — the
wrapper actually lands in PR 16c (this PR).

New regression test test_check_compliance_error_references_crucible_
compliance_check_cli pins both the positive (real command + real
config knob present) and negative (phantom Python module path
absent) invariants. A future refactor that drops the CLI reference
will be caught at test time.

Stats: +18/-7 across 2 files (cli_subscription_backend.py +9/-7,
test_compliance_check_cli.py +69/-0). Test suite: 60 passed across
test_compliance_check_cli.py + test_cli_subscription.py.

Two non-blocking follow-ups remain (separate PRs per reviewer):
  - --limit footgun + task_ids tracking (a 1-task report can
    accidentally satisfy the gate; mitigated by suite-expansion + a
    required_task_ids superset check in verify_recent_pass)
  - test_create_agent_unknown_raises regex case mismatch (1-line fix
    flagged across 5 PR reviews now)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@suzuke suzuke changed the base branch from feat/m3-gemini-cli-adapter to main April 26, 2026 06:44
@suzuke suzuke merged commit e966fc2 into main Apr 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant