Skip to content

Assert reason and fired rules in YAML test cases, not just decision #139

Description

@se-jo-ma

Context

Fathom's value proposition is explainability — result.reason and the rule trace matter alongside the decision. But the built-in fathom test harness only compares result.decision == expected_decision, so a rule pack that returns the right verdict for the wrong reason (or via the wrong rule) passes silently. This makes the YAML harness weaker than the Python-based rule-pack tests already in the repo (tests/rule_packs/test_ssvc.py already asserts on decision metadata directly via engine.evaluate()).

Note: this issue extends the YAML test harness to match what the Python tests already do. The shipped packs (owasp/nist/hipaa/cmmc) are currently tested in Python, not via YAML test files — there are no expected_decision YAML files in the repo today. Adding richer YAML assertions makes the YAML path a credible authoring-time alternative.

Task

Extend the YAML test-case schema and the test command so each case may optionally assert:

  • expected_reason — substring match against result.reason (use substring for robustness; document the semantics).
  • expected_rules — list of rule names that must be a subset of result.rule_trace.

All new fields are optional and additive; existing expected_decision-only suites must keep passing unchanged. On failure, report which field(s) mismatched.

Scope note (read before implementing): result.reason and result.rule_trace are directly available on the returned EvaluationResult (models.py:341-346), so expected_reason/expected_rules need no engine changes. The proposed expected_facts field was deliberately dropped: asserted_facts is NOT on EvaluationResult — it is computed inside Engine.evaluate() and handed only to the audit log (AuditRecord, models.py:365). Asserting on derived facts would require the CLI to additionally call the public engine.query() per template and diff. If desired, do it as a separate follow-up; keep this issue to reason + rules.

Where

  • src/fathom/cli.py:401-434 — the test command per-case loop; today line 404 reads only expected_decision and line 423 is the sole assertion (result.decision == expected). Add reads of expected_reason/expected_rules and extend the pass/fail logic to accumulate per-field mismatches into the failure message.
  • docs/reference/cli/test.md — currently only shows usage; add a "Test-case schema" section documenting name, facts, expected_decision, expected_reason, expected_rules. (The proposal pointed at docs/how-to/writing-rules.md, but that file is rule-authoring only and has no Testing section; the CLI reference is the natural home. Optionally cross-link from docs/how-to/cli.md.)
  • tests/test_cli.py:336-409 — extend alongside the existing test_passing_test/test_failing_test cases.

Acceptance criteria

  • A case with correct expected_decision but wrong expected_reason (substring not present) fails.
  • A case asserting an expected_rules entry not in result.rule_trace fails.
  • A case with correct decision + correct reason + correct rules passes.
  • Existing decision-only suites pass unchanged.
  • Failure message names each mismatched field.
  • docs/reference/cli/test.md documents the new fields.
  • uv run ruff check src/ tests/, uv run mypy src/, and uv run pytest pass.

Size: S

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium priority: roadmap gap or DX/toolingcliCLI (validate, test, bench, info, repl)enhancementNew feature or requestsize/S<half-day: small focused change

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions