feat(rules): add rules test --json machine-readable result contract

[ronheichman]

What

Adds an optional --json mode to numbat rules test that emits a versioned NDJSON result stream on stdout — one event_result per evaluated fixture line plus one terminal summary object. The stream distinguishes the result classes a downstream consumer needs so it never has to parse human CLI text or infer a clean no-match from empty stdout:

1. findings — matches with rule id, version, severity, enforcement_eligible (mirroring the live-enforce gate including shell-safety), and via (engine vs sequence)
2. no-match / completed — findings: [] (or field omitted)
3. malformed input — error.kind ∈ {decode, validate, scan}
4. evaluator failure — per-rule CEL runtime error and/or sequence-tracker error; both may co-occur for one event and neither suppresses the other. error.kind is sequence when the tracker failed, otherwise evaluation.
5. coverage / evaluation health — shell_parse ∈ {ok, degraded, unusable}, sequence_tracker_active

The result-envelope schema versions independently of the wire schema (rules-test-result.v1 vs model.SchemaVersion="0.3.0"). Consumers must ignore unknown keys and treat unknown enum values as unrecognized rather than completed.

Exit codes

• 0: summary.status == "completed" with a passed or unchecked assertion, delivered as a well-formed NDJSON stream terminated by exactly one summary object.
• 1: summary.status == "partial" OR a failed assertion, delivered as a well-formed NDJSON stream terminated by exactly one summary object. This is a successfully delivered result, not a delivery failure.
• 2: one of two cases that share this exit code and are distinguished by inspecting stdout and stderr:
  • Handled JSON delivery failure. A stdout write for an event_result or the terminal summary returned an OS write error (ENOSPC, read-only descriptor); the process caught it, wrote a write rules-test-result: diagnostic to stderr, and short-circuited the remainder of the stream. Any partial stream on stdout is not terminated by a summary.
  • Usage/setup failure. Raised before any stream is produced — e.g. rules test --json without --fixture prints a usage diagnostic to stderr and exits 2 with empty stdout. This is existing CLI behavior, not new to --json.

Exit code alone is not sufficient to identify delivery failure. A usage/setup failure exits 2 before any stream, and abnormal process termination (SIGPIPE on a real broken pipe, SIGKILL, panic, host death) may end the process before the handled exit-2 path runs and before a terminal summary is written — the observed exit status in that case is signal-encoded (for example a Python subprocess returncode of -13 for SIGPIPE), not 2. This PR does not add a signal handler, and no envelope is emitted after an abnormal termination.

Consumer rule. Only a well-formed NDJSON stream terminated by exactly one summary object AND an exit status compatible with that summary proves the documented outcome. A missing terminal summary, a malformed or truncated final JSON line, or an abnormal/signal-encoded termination cannot prove a clean no-match and must be treated as an indeterminate delivery outcome — not silently reclassified as delivery failure, and not silently accepted as success.

Why

Guardian and other downstream consumers of numbat rules test today have to parse tab-separated stdout and infer a clean no-match from empty output; there is no way to distinguish a clean no-match from an unusable shell parse, from a per-rule CEL runtime error, from malformed input, from a scan-time truncation, or from a broken output channel. This blocks Guardian's ability to make an action-level enforcement decision from the direct evaluator without a parallel evaluator or fragile text parsing.

How

Extends the existing rules test seam. rule.Engine gains an additive EvalDetailed companion to Eval that returns per-rule errors and shell-analysis diagnostics without changing Eval's error text or the existing legacy tab-separated output. Every JSON write goes through a single emit closure that flips a deliveryFailed sentinel on any encoder error and short-circuits further stdout writes, so a stdout failure at either an event_result or the terminal summary propagates to a distinct exit code (2) rather than a silent success.

Legacy stdout is byte-identical when --json is absent — verified against all five shipped fixtures (secrets, sequence, custom, benign, exfil).

No new subsystem, no parallel evaluator, no wire-schema bump, no new dependencies.

Contract & docs

Authoritative machine-readable contract at docs/schema/rules-test-result.v1.md. CLI reference updated at docs/cli.md. The schema doc pins exact counter semantics (matches may exceed events_evaluated when several direct rules match one event, and each sequence finding adds to matches without adding to events_evaluated), delivery-failure semantics, and the exit-code table.

Tests

cmd/numbat/rules_test_json_test.go exercises the result classes end-to-end through runCLI plus adversarial cases:

• decode vs validate vs scan malformed-input kinds
• per-rule CEL error AND real sequence-tracker error co-occurring on the same event (deterministic tags[N]-out-of-bounds inputs), asserting both errors AND error.kind: "sequence" classification AND summary.status: "partial" AND exit 1
• count identities: two direct rules matching one event produces matches=2, events_evaluated=1; partial final event still counts in events_evaluated; sequence findings add to matches without adding to events_evaluated
• delivery failure at the first event_result write returns exit 2 with a stderr diagnostic
• delivery failure at the terminal summary write (after event_results succeeded) also returns exit 2
• compiled-CLI test spawning numbat rules test --json with stdout redirected to /dev/full — exits nonzero (Linux only; skipped when /dev/full unavailable)
• enforcement downgrade on shell-unsafe wrappers
• assertion-outcome matrix; partial run always unchecked
• backward-compatible byte-identical legacy stdout

Independent review

This PR was reviewed by two independent Perplexity Computer subagents (adversarial, evidence-only, verifying every claim against the exact head). A first-pass review found three concrete defects at head 5b3df36:

1. JSON output paths silently discarded enc.Encode errors — writing to a broken descriptor returned exit 0 with no stderr, while the legacy path returned exit 1 with a diagnostic.
2. The schema doc and a test asserted a false invariant matches <= events_evaluated.
3. The "sequence + CEL co-failure" test's inputs did not actually produce a sequence-tracker error, so the co-existence contract was not really exercised.

All three were reproduced against the compiled CLI with real OS file descriptors and synthetic read-only inputs, then fixed as one package on this branch. A second-pass independent review of the fix head confirmed PASS with no blocking findings, plus a small set of non-blocking polish items that were applied in a follow-up amendment; a third review confirmed the polish delta introduces no behavior change and all gates remain green.

No upstream GitHub reviewer has approved this PR yet.

Compatibility

• Additive: no existing behavior changes when --json is absent
• rule.Engine.Eval and its error text unchanged; existing callers unaffected
• Legacy tab-separated stdout byte-identical on all shipped fixtures
• No wire-schema bump (model.SchemaVersion stays 0.3.0)
• New envelope schema id rules-test-result.v1; future breaking changes will bump to v2

Gates

Local: go build ./..., go test ./... -count=1, go test -race ./cmd/numbat/... ./internal/rule/... -count=1, go vet ./..., gofmt -l . empty, go mod tidy clean, golangci-lint run 0 issues, go generate ./rules no drift. Compiled-CLI write-failure to /dev/full returns exit 2 as designed; legacy mode continues to return exit 1 on the same failure.

Not in this PR

• Guardian integration and demo. This PR is the upstream contract; Guardian will consume it in a separate follow-up after this lands and a reviewed dependency-pin + JSON-parser integration change ships on the Guardian side.
• Merge. Please review and hold merge — the caller has explicitly asked me not to merge this on their behalf.

Guidance

Please skim docs/schema/rules-test-result.v1.md first — that is the load-bearing artifact. The Go code is the smallest supported implementation that satisfies the contract.

---

Built with Codex