feat(m3): crucible compliance-check CLI + run_compliance_harness (M3 PR 16c)#17
Merged
Conversation
…PR 16c)
Closes the M3 PR 16 stack: PR 16a (codex) + PR 16b (gemini) shipped
the adapter classes; PR 16c ships the operator-facing harness runner
+ Click command that PRODUCES the compliance reports the backend's
gate enforcement consumes.
New: `run_compliance_harness(adapter, *, tasks, project_dir, ...)`:
For each BenignTask, materialises workspace_files into a temp scratch
dir, builds an AdapterRunContext, calls
`adapter.run_subprocess(ctx)` then `adapter.parse_output(raw)`,
classifies via `_classify_trial`, accumulates into ComplianceReport,
persists to <project>/compliance-reports/, returns the report.
Classification matrix (`_classify_trial` priority order):
1. raw.timed_out → CLI_ERROR
2. CLISubscriptionAuthError caught → CLI_ERROR (gate run is
invalid, not the
adapter being non-
compliant — but we
surface as failing)
3. raw.exit_code != 0 → CLI_ERROR
4. parser raised generic exception → CLI_ERROR
5. parsed.unknown_schema → PARSE_FAILURE
6. parsed.tool_was_called → PARSE_SUCCESS
7. _MODEL_REFUSAL_PHRASES match → MODEL_REFUSAL
8. residual (clean run, no tool, no → FORMAT_DRIFT
refusal phrase)
`_MODEL_REFUSAL_PHRASES` is an explicitly declared tuple (mirrors
the auth-phrase set pattern from PR 16a/16b — no coincidental
substring matching).
Module-level (NOT lazy) imports of CLISubscriptionAuthError /
AdapterRunContext: keeps the binding stable across sys.modules
eviction by tests like test_agents_factory. A lazy import inside
the function would resolve to a freshly-imported class object after
eviction, breaking isinstance checks against pre-eviction instances.
New: `crucible compliance-check` Click command:
Args: --adapter (claude-code-cli/codex-cli/gemini-cli, required)
--project-dir (default .)
--cli-binary-path (override PATH lookup)
--timeout (per-task seconds, default 60)
--limit (run only first N tasks, useful for smoke checks)
Prints adapter+version+pass-rate summary, per-classification
bucket counts, and a threshold disposition line:
✓ Release threshold met (≥99%)
⚠ Admit threshold met (≥95%) — POC-eligible only
✗ Below admit threshold — investigate
Tests (22 new in test_compliance_check_cli.py):
- _classify_trial matrix: timeout, auth, exit_code, unknown_schema,
tool_was_called, refusal phrase, format_drift residual, parser-
raise, priority (auth beats schema_drift)
- Harness: happy path (report fields + persistence), workspace
materialisation, subprocess exception classification, parser-raise
classification, auth-failure classification, mixed results,
load_reports() round-trip
- CLI smoke: codex-cli adapter construction + run + summary +
persistence; --adapter required; unknown adapter rejected;
--limit 0 rejected; below-admit message path
Stats: +340 src LOC + 401 test LOC = +741 LOC, 3 files
(`git diff --stat HEAD~1 HEAD | tail -1` after push).
Test results: 2809 passed, 4 skipped (`pytest --ignore=tests/test_agents.py`).
The pre-existing PR-16 regex case mismatch in test_create_agent_unknown_raises
remains the only excluded failure across the entire M3 PR 16 stack.
Stacked on PR 16b (#16). After all three M3-PR-16-stack PRs merge in
order (16a #15 → 16b #16 → 16c this PR), operators can:
1. `crucible compliance-check --adapter codex-cli` to produce a
passing report
2. Configure `agent.cli_subscription.adapter: codex-cli` in their
project
3. SubscriptionCLIBackend admits the adapter at run time
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reviewer round 1 caught: `_check_compliance`'s failure message
referenced the Python module path
`crucible.agents.cli_subscription.compliance` instead of the new
`crucible compliance-check` CLI command (which PR 16c just landed).
Same phantom-command failure mode as PR 16 R2, just symmetric — non-
developer users see "Python module path" and don't know what to do.
Updated the message to reference the real CLI:
Either run `crucible compliance-check --adapter <name>` to produce
a passing report, or set `experimental.allow_stale_compliance: true`
to bypass...
Also updated the stale comment ("UX wrapper lands in PR 16a") — the
wrapper actually lands in PR 16c (this PR).
New regression test test_check_compliance_error_references_crucible_
compliance_check_cli pins both the positive (real command + real
config knob present) and negative (phantom Python module path
absent) invariants. A future refactor that drops the CLI reference
will be caught at test time.
Stats: +18/-7 across 2 files (cli_subscription_backend.py +9/-7,
test_compliance_check_cli.py +69/-0). Test suite: 60 passed across
test_compliance_check_cli.py + test_cli_subscription.py.
Two non-blocking follow-ups remain (separate PRs per reviewer):
- --limit footgun + task_ids tracking (a 1-task report can
accidentally satisfy the gate; mitigated by suite-expansion + a
required_task_ids superset check in verify_recent_pass)
- test_create_agent_unknown_raises regex case mismatch (1-line fix
flagged across 5 PR reviews now)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the M3 PR 16 stack (16a codex + 16b gemini + 16c harness/CLI). Ships:
run_compliance_harness(adapter, *, tasks, project_dir, ...)— the operator-facing function that produces compliance reports the backend's gate enforcement consumes.crucible compliance-checkClick command — wraps the harness for the CLI UX.Stacked on PR 16b (#16)
This PR is stacked on
feat/m3-gemini-cli-adapter. Once 16a merges → 16b rebases → 16c rebases. Or merge them in order.Classification matrix
_classify_trialpriority (earliest wins):raw.timed_out→CLI_ERRORCLISubscriptionAuthErrorcaught →CLI_ERROR(gate run invalid, surface as failing)raw.exit_code != 0→CLI_ERRORCLI_ERRORparsed.unknown_schema→PARSE_FAILUREparsed.tool_was_called→PARSE_SUCCESS_MODEL_REFUSAL_PHRASESmatch in description →MODEL_REFUSALFORMAT_DRIFT_MODEL_REFUSAL_PHRASESis an explicitly declared tuple (mirrors the auth-phrase pattern from PR 16a/16b — no coincidental substring matching).Module-level imports (sys.modules-eviction tolerance)
CLISubscriptionAuthError+AdapterRunContextare imported at MODULE level in compliance.py, NOT lazily insiderun_compliance_harness. The lazy variant fails whentest_agents_factoryevictscrucible.agents.*between collection and runtime — the harness's late-resolved class would not match isinstance checks against caller-held pre-eviction instances. Module-level keeps the binding stable.CLI UX
Stats
git diff --stat HEAD~1 HEAD | tail -1→ 817 insertions / 0 deletions, 3 files.Test results
2809 passed, 4 skipped (
pytest --ignore=tests/test_agents.py). The pre-existing PR-16 regex case mismatch intest_create_agent_unknown_raisesremains the only excluded failure across the entire M3 PR 16 stack — recommend opening a 1-line follow-up PR to fix that regex independently.Test plan
_classify_trialmatrix: timeout, auth, exit_code, unknown_schema, tool_was_called, refusal phrase, format_drift residual, parser-raise, priority (auth beats schema_drift)🤖 Generated with Claude Code