Context
Fathom's value proposition is explainability — result.reason and the rule trace matter alongside the decision. But the built-in fathom test harness only compares result.decision == expected_decision, so a rule pack that returns the right verdict for the wrong reason (or via the wrong rule) passes silently. This makes the YAML harness weaker than the Python-based rule-pack tests already in the repo (tests/rule_packs/test_ssvc.py already asserts on decision metadata directly via engine.evaluate()).
Note: this issue extends the YAML test harness to match what the Python tests already do. The shipped packs (owasp/nist/hipaa/cmmc) are currently tested in Python, not via YAML test files — there are no expected_decision YAML files in the repo today. Adding richer YAML assertions makes the YAML path a credible authoring-time alternative.
Task
Extend the YAML test-case schema and the test command so each case may optionally assert:
expected_reason — substring match against result.reason (use substring for robustness; document the semantics).
expected_rules — list of rule names that must be a subset of result.rule_trace.
All new fields are optional and additive; existing expected_decision-only suites must keep passing unchanged. On failure, report which field(s) mismatched.
Scope note (read before implementing): result.reason and result.rule_trace are directly available on the returned EvaluationResult (models.py:341-346), so expected_reason/expected_rules need no engine changes. The proposed expected_facts field was deliberately dropped: asserted_facts is NOT on EvaluationResult — it is computed inside Engine.evaluate() and handed only to the audit log (AuditRecord, models.py:365). Asserting on derived facts would require the CLI to additionally call the public engine.query() per template and diff. If desired, do it as a separate follow-up; keep this issue to reason + rules.
Where
src/fathom/cli.py:401-434 — the test command per-case loop; today line 404 reads only expected_decision and line 423 is the sole assertion (result.decision == expected). Add reads of expected_reason/expected_rules and extend the pass/fail logic to accumulate per-field mismatches into the failure message.
docs/reference/cli/test.md — currently only shows usage; add a "Test-case schema" section documenting name, facts, expected_decision, expected_reason, expected_rules. (The proposal pointed at docs/how-to/writing-rules.md, but that file is rule-authoring only and has no Testing section; the CLI reference is the natural home. Optionally cross-link from docs/how-to/cli.md.)
tests/test_cli.py:336-409 — extend alongside the existing test_passing_test/test_failing_test cases.
Acceptance criteria
Size: S
Context
Fathom's value proposition is explainability —
result.reasonand the rule trace matter alongside thedecision. But the built-infathom testharness only comparesresult.decision == expected_decision, so a rule pack that returns the right verdict for the wrong reason (or via the wrong rule) passes silently. This makes the YAML harness weaker than the Python-based rule-pack tests already in the repo (tests/rule_packs/test_ssvc.pyalready asserts on decision metadata directly viaengine.evaluate()).Note: this issue extends the YAML test harness to match what the Python tests already do. The shipped packs (owasp/nist/hipaa/cmmc) are currently tested in Python, not via YAML test files — there are no
expected_decisionYAML files in the repo today. Adding richer YAML assertions makes the YAML path a credible authoring-time alternative.Task
Extend the YAML test-case schema and the
testcommand so each case may optionally assert:expected_reason— substring match againstresult.reason(use substring for robustness; document the semantics).expected_rules— list of rule names that must be a subset ofresult.rule_trace.All new fields are optional and additive; existing
expected_decision-only suites must keep passing unchanged. On failure, report which field(s) mismatched.Scope note (read before implementing):
result.reasonandresult.rule_traceare directly available on the returnedEvaluationResult(models.py:341-346), soexpected_reason/expected_rulesneed no engine changes. The proposedexpected_factsfield was deliberately dropped:asserted_factsis NOT onEvaluationResult— it is computed insideEngine.evaluate()and handed only to the audit log (AuditRecord,models.py:365). Asserting on derived facts would require the CLI to additionally call the publicengine.query()per template and diff. If desired, do it as a separate follow-up; keep this issue to reason + rules.Where
src/fathom/cli.py:401-434— thetestcommand per-case loop; today line 404 reads onlyexpected_decisionand line 423 is the sole assertion (result.decision == expected). Add reads ofexpected_reason/expected_rulesand extend the pass/fail logic to accumulate per-field mismatches into the failure message.docs/reference/cli/test.md— currently only shows usage; add a "Test-case schema" section documentingname,facts,expected_decision,expected_reason,expected_rules. (The proposal pointed atdocs/how-to/writing-rules.md, but that file is rule-authoring only and has no Testing section; the CLI reference is the natural home. Optionally cross-link fromdocs/how-to/cli.md.)tests/test_cli.py:336-409— extend alongside the existingtest_passing_test/test_failing_testcases.Acceptance criteria
expected_decisionbut wrongexpected_reason(substring not present) fails.expected_rulesentry not inresult.rule_tracefails.docs/reference/cli/test.mddocuments the new fields.uv run ruff check src/ tests/,uv run mypy src/, anduv run pytestpass.Size: S