feat(validation): add cross-engine conformance inventory contract

[seonghobae]

Advances #1077.

Bounded product slice

Introduce a provider-neutral, source-free fast_mlsirm.cross_engine_conformance contract for the first machine-readable cross-engine conformance inventory slice.

• distinguish fixed-parameter equation, fitted-result, and neutral/adversarial evidence layers;
• preserve covered, partially_covered, no_independent_engine, not_comparable, and planned capability states;
• preserve passed, failed, indeterminate, not_executed, and not_applicable execution states without collapsing unavailable optional engines into success;
• bind comparison engines, parameter mappings, fixtures, environments, artifacts, package versions, and protected-main source commits to deterministic source-free manifests;
• reject caller-defined text/container subclasses at semantic-control boundaries before caller callbacks;
• keep external engines isolated from runtime/build/package dependencies and treat them as validation instruments rather than production numerical owners.

Review and replay-integrity remediation

• The original current-lineage review required covered / partially_covered capabilities to contain at least one actually executed evidence row; that finding is resolved in the branch and covered by dedicated regressions.
• Text normalization now uses NFC before content-addressed hashing so canonically equivalent Unicode does not produce divergent inventory identities.
• RED c0f6fa724b5629816cad19227360790ad20eb9b2 adds regressions showing that Python's object.__setattr__ can otherwise mutate an exact frozen/slots record after construction and let replay bypass semantic-control/container admission.
• GREEN 9b278f22ae223f83bc670e93292f9d609b08b02e revalidates exact engine, evidence, capability, and inventory records before manifest/fingerprint replay. Mutated enum controls are rejected before .value callbacks and rebound collection subclasses are rejected before iteration.
• Evidence commit f021a4fd918e36ff3697556d963e1dc521579965 records the replay-integrity boundary in the governed changelog without moving numerical ownership.
• f9ca1c2a786935db017377d262b85be859fbcd75 is a compatible successor that adds only docstrings to replay test-fixture builders.
• Current head 4f14a754a1e016edd9a81bbd4ad4bfa367f8f4cc is the direct successor to f9ca1c2...; it repairs the authoritative changelog fragment to the canonical # Title / ## Added structure. Production source is unchanged from the replay-integrity remediation lineage.

Ownership boundary

This slice is Python schema, validation, provenance, and serialization only. It adds no likelihood, estimator, fitting, scoring, alignment, discrepancy, uncertainty, Monte Carlo, or other psychometric/statistical arithmetic. Production numerical work remains Rust-owned. External R/Stan/commercial engines remain optional isolated validation dependencies, never package/runtime dependencies or sole correctness oracles.

Acceptance boundary

This PR does not close #1077. Fixed-parameter harness execution, fitted-result alignment, bias/MAE/RMSE and discrepancy computation, ADEMP/Monte Carlo evidence, full protected-main capability inventory generation, isolated external-engine workflows, accessible HTML, disagreement registers, and release qualification remain subsequent bounded slices.

Current exact head is 4f14a754a1e016edd9a81bbd4ad4bfa367f8f4cc against protected main@04d0bc21a2a20693bcf16108cd76d394fe844d23; the branch is 18 commits ahead / 0 behind with merge-base exactly protected main. The stacked #1085 base tip is this exact head. Keep Draft until exact-current-head CI/security/package/coverage/docstring/review evidence is terminal and clean; predecessor-head evidence does not transfer.

Summary by CodeRabbit

• New Features

  • Added a provider-neutral conformance inventory for comparing numerical results across engines.
  • Captures estimands, parameterization, identification scope, engine provenance, mappings, fixtures, and execution outcomes.
  • Provides deterministic manifests and inventory fingerprints for traceable validation.

• Documentation

  • Added guidance on conformance scope, evidence statuses, provenance requirements, limitations, and methodology references.

• Tests

  • Added comprehensive validation and regression coverage for manifests, fingerprints, status consistency, malformed inputs, duplicates, and record integrity.

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 3d061ffc-ee9e-4c59-88c4-0b7bb533c52f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds a provider-neutral Python conformance inventory. It validates immutable engine, evidence, capability, and inventory records, produces deterministic manifests and fingerprints, documents the contract, and adds extensive edge-case and record-sealing tests.

Changes

Cross-engine conformance inventory

Layer / File(s)	Summary
Contract foundation python/fast_mlsirm/cross_engine_conformance.py, docs/doctoring/cross_engine_conformance_inventory.md, docs/changelog.d/1077-cross-engine-conformance-inventory.md	Defines conformance layers and statuses, strict field validation, deterministic JSON and hashing helpers, public exports, and the inventory documentation.
Immutable conformance records python/fast_mlsirm/cross_engine_conformance.py	Adds immutable ComparisonEngine, ConformanceEvidence, ConformanceCapability, and ConformanceInventory records with provenance, coverage rules, manifests, sorting, and fingerprints.
Validation and sealing coverage tests/test_cross_engine_conformance.py, tests/test_cross_engine_conformance_edges.py, tests/test_cross_engine_conformance_record_sealing.py	Tests deterministic output, execution and coverage consistency, malformed inputs, duplicate identities, nested revalidation, fingerprint changes, and rejection of subclasses before field access.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: 🟡 Moderate · up to b2106

The contract can currently publish covered or partially covered capabilities without any executed evidence, allowing conformance manifests to overstate validated coverage; merge should wait until at least one executed result is required. Unicode fingerprint normalization and stricter test patterns are localized follow-ups.

Possibly related PRs

• ContextualWisdomLab/fast-mlsirm#976: Introduces numerical engines whose capabilities can be cataloged by this conformance inventory.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Linked Issues check	⚠️ Warning	The changes implement the inventory contract, but [#1077] also requires live capability coverage, numerical harnesses, and conformance evidence outputs.	Implement or separately link the remaining [#1077] requirements, including capability coverage, comparison execution, reproducibility evidence, and required reports.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Out of Scope Changes check	✅ Passed	The implementation, documentation, and tests directly support the cross-engine conformance inventory contract described in [#1077].
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely identifies the main change: adding a cross-engine conformance inventory contract for validation.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/cross-engine-conformance-inventory-1077

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

Exact-head review completed at b210652. The inventory remains provider-neutral and source-free; the Git provenance validator now accepts full SHA-1 and SHA-256 commit identifiers, and the APA 7 doctoring record is present. Focused contract, fail-closed edge, and record-sealing tests pass (38 passed); Ruff and diff checks pass. Full external-engine harness execution remains intentionally outside this bounded slice.

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.

[seonghobae]

Current-head RCA and repair: CodeRabbit identified that covered or partially_covered capabilities could contain only not_executed or not_applicable evidence. At 05b7b8f, both statuses now require at least one passed, failed, or indeterminate executed evidence row; unavailable/planned evidence remains explicit and cannot overstate validation. Added regression coverage for both statuses, updated APA doctoring/changelog guidance, and preserved the source-free/Rust-owned boundary. Focused contract, edge, and record-sealing tests: 38 passed; Ruff and diff checks pass. Re-review and regenerate all protected Checks for this exact head.

[seonghobae]

Review exact current head 05b7b8f. The prior actionable finding is fixed: covered and partially_covered now require executed evidence. Review the evidence-state invariant, nested record sealing, SHA-1/SHA-256 provenance boundary, Rust numerical ownership, and current protected Checks only. @opencode-agent review @cwl-noema-review review

[seonghobae]

Exact-head review request for aef1598:\n\nThe remote head advanced. Please run the established review agents on exact current HEAD ; the conformance edge suite and docstring evidence were re-run locally against this exact head.

[seonghobae]

Exact-head review request for aef1598: the remote head advanced; please run the established review agents on this exact HEAD. The conformance edge suite and changed-file docstring evidence were re-run locally against this exact head.

[seonghobae]

@opencode-agent review @cwl-noema-review review

Exact-current-head review request for 4f14a754a1e016edd9a81bbd4ad4bfa367f8f4cc against main@04d0bc21a2a20693bcf16108cd76d394fe844d23.

Revalidated the prior review findings against this exact head: NFC normalization is present before conformance text hashing; all three schema-version regex assertions match literal 1.0; covered/partially-covered capabilities require at least one executed evidence status. The focused suite tests/test_cross_engine_conformance.py tests/test_cross_engine_conformance_edges.py tests/test_cross_engine_conformance_record_sealing.py passes 38 tests, and all required hosted Checks are successful. No new source patch is warranted. Please bind any formal approval or finding to this SHA only; do not reuse predecessor coverage verdicts, self-approve, or merge.

[seonghobae]

Exact-head triage for inline finding 3822650386: the reported coverage asymmetry is already fixed in current head 4f14a75. ConformanceCapability now requires at least one PASSED, FAILED, or INDETERMINATE row for covered/partially_covered status and retains the empty-evidence error. The NFC normalization and literal schema-version regex fixes are also present. Focused proof: PYTHONPATH=.:python python -m pytest -q tests/test_cross_engine_conformance.py tests/test_cross_engine_conformance_edges.py tests/test_cross_engine_conformance_coverage_execution.py tests/test_cross_engine_conformance_record_sealing.py -> 40 passed. No source edit is needed; please publish a fresh exact-head formal review against this SHA rather than carrying the predecessor inline comment.

[seonghobae]

Current-head disposition

The downstream conformance-provenance PR #1085 was merged into this root branch at 3a85c075fde5443cedafbe0a65abb73391c72650; predecessor-head evidence is discarded. The current source already contains the fail-closed coverage invariant reported by CodeRabbit comment 3822650386: covered and partially_covered require at least one passed, failed, or indeterminate evidence row, while empty evidence retains the original error and planned/non-comparable states remain constrained.

Exact-head local verification at 3a85c075fde5443cedafbe0a65abb73391c72650:

• conformance, edge, coverage-execution, record-sealing, and mutation-replay suites: 47 passed
• Ruff: passed
• interrogate on cross_engine_conformance.py: 100%
• compileall: passed
• git diff --check: passed

No source edit is warranted. Fresh protected Checks and formal review must bind to this exact head; no stale approval or predecessor result is reused.

[devin-ai-integration]

Devin Review found 10 new potential issues.

[devin-ai-integration]

Devin Review found 1 new potential issue.

[devin-ai-integration]

Devin Review found 2 new potential issues.

[seonghobae]

Self-correction: reverted my own `051bc1b` in `e48eebf`.

That commit removed `escape(_CSP, quote=True)` on a false-positive premise (same one `fast-mlsirm#1230` was independently closed for): HTML character references in attribute source markup are decoded by the HTML parser before the attribute value is consumed, so `content="default-src 'none'"` yields the DOM attribute value `default-src 'none'` — the CSP pragma reads the decoded string, not the literal `'` bytes. See https://developer.mozilla.org/en-US/docs/Web/API/Element/getAttribute#decoded_character_references_in_attribute_values . Strix's original finding on this PR was a false positive; escaping the attribute value was correct standard practice and removing it reduces defense-in-depth without fixing anything. Restored the escape call and reverted the two test assertions that pinned the (unnecessary) unescaped serialization spelling; the underlying 8-test suite passes unchanged.

[coderabbitai]

Caution

Review failed

An error occurred during the review process. Please try again later.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}