feat(validation): complete conformance runtime provenance by seonghobae · Pull Request #1095 · ContextualWisdomLab/fast-mlsirm

seonghobae · 2026-08-20T21:18:01Z

Fixes #1094. Advances #1077. Stacks directly on Draft #1093.

RED → GREEN

RED d659b1292872bb64783816a939c6b873da2463af adds public regressions requiring runtime/environment/configuration/redistribution identities in run manifests, binding each to the inventory fingerprint, and rejecting rebound caller-defined text/enum values before callbacks.
GREEN 02af11bc0975a3508640926b860f6a184b51d1bc adds explicit environment-kind and redistribution-status enums plus operating-system, architecture, model-configuration SHA-256, and convergence-controls SHA-256 provenance fields. All are revalidated through the existing exact-record replay boundary before manifest projection.
Compatibility 00ea56b5c9fc7e39edb207ea00000270f0796558, 76a6b96c4d5ada4e2fcf4acc1f7b8a74b250923a, and 861253d6f070f85a7e0b9e57e43a28d5271b3405 update inherited conformance fixtures without changing their numerical semantics.
Release record b46410fe3697b47764dc626759968c01e2328875 adds the governed changelog fragment.

Contract

ConformanceRunProvenance now distinguishes a container image from an exact environment lock and records source-free runtime platform, model configuration, convergence controls, license classification, and redistribution status. Deterministic manifests and inventory_fingerprint cover every new identity. The existing executed-evidence/output-hash gate from #1092 remains unchanged.

Ownership boundary

Python remains validation/provenance/serialization only. No likelihood, fitting, scoring, alignment, discrepancy, bias/MAE/RMSE, uncertainty, Monte Carlo, or other psychometric/statistical arithmetic is introduced. External engines remain isolated validation instruments; production numerical ownership remains Rust-first.

Stack / acceptance

Base branch is the exact current #1093 head lineage. Keep Draft until #1093 and its predecessors are integrated and this exact head has terminal hosted CI/security/package/coverage plus independent current-head review evidence. Predecessor-head checks and reviews are historical only.

coderabbitai · 2026-08-20T21:18:10Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 2823d060-22d7-4089-ba88-24eb3768bf7a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

seonghobae · 2026-08-20T21:48:44Z

Current exact head c233b236f6dd6be48cc8ab7bd5c8f7fc4c9c247c has one reproducible inherited-suite failure: tests/test_cross_engine_conformance_mutation_replay.py::test_inventory_manifest_rejects_mutated_container_before_iteration_callback constructs executed evidence without the now-required run_provenance, so collection reaches ValueError: run_provenance is required for executed evidence. Update that fixture with matching executed run provenance before advancing #1096; this is a parent-stack compatibility fix, not a replay implementation change.

seonghobae · 2026-08-20T22:02:22Z

Parent compatibility PR #1099 is now open against this exact branch head c233b236f6dd6be48cc8ab7bd5c8f7fc4c9c247c. It supplies the required raw/normalized run provenance to the executed mutation fixture, making the inherited replay test reach its intended hostile-container assertion. #1097 remains a Draft child and will inherit the fix after #1099 merges.

seonghobae · 2026-08-21T01:48:22Z

Current-head review and fix record: runtime provenance adds immutable environment kind/digest, operating system, architecture, model and convergence-control digests, and redistribution status to source-free conformance manifests; to_manifest() revalidates the exact record before serialization. The parent branch initially had one mutation-fixture test still constructing executed evidence without the new required provenance, so its own suite failed. I carried the existing #1099 fixture fix into this parent branch to keep the parent independently green. Exact new HEAD 5200c0a6 passes all cross-engine conformance/provenance suites (71 passed), Ruff, production interrogate 100%, compileall, and git diff --check. Review this exact stacked HEAD only; merge normally after its base PR, without bypass.

devin-ai-integration

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

# Conflicts: # tests/test_cross_engine_conformance_mutation_replay.py

seonghobae · 2026-08-21T04:13:48Z

Exact-head stack repair at 235c96782b1c273f1d70b7394659baaab7e041d3: merged the concurrent base-branch fixture commits normally, resolved the single mutation-replay test conflict by preserving the richer runtime-provenance contract, and revalidated the parent stack. Evidence: 69 conformance/provenance tests passed; Ruff, production Interrogate 100%, compileall, and diff checks passed. Hosted Checks are running on this exact merge head; no bypass or self-approval was used.

) * test(conformance): require strict persisted manifest replay * feat(conformance): strictly replay persisted manifests * docs(changelog): record strict conformance manifest replay * test(conformance): cover nested manifest replay boundaries * test(conformance): require stable JSON resource failures * fix(conformance): stabilize bounded JSON replay failures * fix(conformance): bound parsed manifest nesting * test(conformance): provide provenance for executed mutation fixture

…1093) * test(conformance): require output provenance for executed evidence * fix(conformance): bind executed evidence to run outputs * test(conformance): align fixtures with executed provenance contract * docs(changelog): record executed conformance provenance gate * test(conformance): seal mutated output provenance before execution gate * test(conformance): provide provenance for executed mutation fixture * Revert "test(conformance): provide provenance for executed mutation fixture" This reverts commit b94cf2e. * test(conformance): provide legacy provenance fixture * feat(validation): complete conformance runtime provenance (#1095) * test(conformance): require runtime and redistribution provenance * feat(conformance): bind runtime and redistribution provenance * test(conformance): align inventory fixtures with runtime provenance * test(conformance): carry runtime identities in provenance replay fixture * test(conformance): carry runtime identities in execution fixture * docs(changelog): record runtime provenance contract * test(conformance): cover malformed runtime provenance controls * test(conformance): provide provenance for executed mutation fixture * feat(validation): strictly replay persisted conformance manifests (#1097) * test(conformance): require strict persisted manifest replay * feat(conformance): strictly replay persisted manifests * docs(changelog): record strict conformance manifest replay * test(conformance): cover nested manifest replay boundaries * test(conformance): require stable JSON resource failures * fix(conformance): stabilize bounded JSON replay failures * fix(conformance): bound parsed manifest nesting * test(conformance): provide provenance for executed mutation fixture

* feat(validation): add conformance run provenance * fix(changelog): restore fragment headings * test(validation): reproduce run provenance replay bypass * fix(validation): replay run provenance before serialization * docs(validation): record provenance replay integrity * fix(validation): bind executed conformance to run output provenance (#1093) * test(conformance): require output provenance for executed evidence * fix(conformance): bind executed evidence to run outputs * test(conformance): align fixtures with executed provenance contract * docs(changelog): record executed conformance provenance gate * test(conformance): seal mutated output provenance before execution gate * test(conformance): provide provenance for executed mutation fixture * Revert "test(conformance): provide provenance for executed mutation fixture" This reverts commit b94cf2e. * test(conformance): provide legacy provenance fixture * feat(validation): complete conformance runtime provenance (#1095) * test(conformance): require runtime and redistribution provenance * feat(conformance): bind runtime and redistribution provenance * test(conformance): align inventory fixtures with runtime provenance * test(conformance): carry runtime identities in provenance replay fixture * test(conformance): carry runtime identities in execution fixture * docs(changelog): record runtime provenance contract * test(conformance): cover malformed runtime provenance controls * test(conformance): provide provenance for executed mutation fixture * feat(validation): strictly replay persisted conformance manifests (#1097) * test(conformance): require strict persisted manifest replay * feat(conformance): strictly replay persisted manifests * docs(changelog): record strict conformance manifest replay * test(conformance): cover nested manifest replay boundaries * test(conformance): require stable JSON resource failures * fix(conformance): stabilize bounded JSON replay failures * fix(conformance): bound parsed manifest nesting * test(conformance): provide provenance for executed mutation fixture

devin-ai-integration

Devin Review found 4 new potential issues.

devin-ai-integration · 2026-08-21T10:08:37Z

cross_engine_conformance.py

+def _validate_manifest_nesting(value: object) -> None:
+    """Reject parsed JSON containers deeper than the replay contract allows."""
+    stack: list[tuple[object, int]] = [(value, 0)]
+    while stack:
+        current, depth = stack.pop()
+        if type(current) is dict:
+            children = dict.values(current)
+        elif type(current) is list:
+            children = current
+        else:
+            continue
+        if depth >= MAX_MANIFEST_NESTING:
+            raise ValueError("manifest JSON nesting is too deep")
+        stack.extend((child, depth + 1) for child in children)


📝 Info: Nesting guard is a secondary bound after parse

_validate_manifest_nesting runs after json.loads. Very deep input raises RecursionError during parse (caught at cross_engine_conformance.py); the iterative stack check covers the middle band that parsing accepts. Both paths map to the same stable error.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-08-21T10:08:38Z

+        if replayed.inventory_fingerprint != supplied_fingerprint:
+            raise ValueError("inventory_fingerprint does not match canonical manifest")
+        if replayed.to_manifest() != manifest:
+            raise ValueError("manifest must already be canonical")
+        return replayed


📝 Info: Replay requires byte-exact canonical manifests

from_manifest re-validates: text is NFC-normalized and stripped, and capabilities/evidence are re-sorted by id. Any persisted manifest not already byte-exact canonical is rejected as non-canonical rather than rehydrated. Callers must store exactly what to_manifest emits.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-08-21T10:08:39Z

cross_engine_conformance.py

        return {
+            "architecture": sealed.architecture,
+            "convergence_controls_sha256": sealed.convergence_controls_sha256,
+            "environment_kind": sealed.environment_kind.value,
            "environment_sha256": sealed.environment_sha256,
            "harness_commit": sealed.harness_commit,
            "license_classification": sealed.license_classification,
            "mapping_schema_version": sealed.mapping_schema_version,
            "mapping_sha256": sealed.mapping_sha256,
+            "model_configuration_sha256": sealed.model_configuration_sha256,
            "normalized_output_sha256": sealed.normalized_output_sha256,
+            "operating_system": sealed.operating_system,
            "raw_output_sha256": sealed.raw_output_sha256,
+            "redistribution_status": sealed.redistribution_status.value,
            "rng_algorithm": sealed.rng_algorithm,
            "rng_seeds": list(sealed.rng_seeds),
            "tolerance_rationale": sealed.tolerance_rationale,


📝 Info: New provenance fields consistently threaded through all revalidation sites

The six added fields (environment_kind, operating_system, architecture, model_configuration_sha256, convergence_controls_sha256, redistribution_status) are consistently applied across all five sites that must stay in sync: dataclass declaration (cross_engine_conformance.py), __post_init__ validation (:194-269), to_manifest projection (:274-292), _run_provenance revalidation (:299-317), and __all__ export (:723,728). Because to_manifest projects every new field, inventory_fingerprint correctly binds each new identity, matching the fingerprint-binding regression tests. No missing site was found.

(Refers to this code)

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-08-21T10:08:41Z

cross_engine_conformance.py

            "license_classification",
            _identifier(self.license_classification, "license_classification"),
        )
+        object.__setattr__(
+            self,
+            "redistribution_status",
+            _enum_value(
+                self.redistribution_status,
+                ConformanceRedistributionStatus,
+                "redistribution_status",
+            ),
+        )


📝 Info: Anti-callback validation ordering preserved for new enum/text fields

The 'reject rebound caller-defined text/enum before callbacks' invariant is preserved for the new fields. _enum_value (cross_engine_conformance.py) checks type(value) is enum_type then type(value) is not str before calling enum_type(value), so a str subclass is rejected before its __hash__ runs; _text (:59-68) checks type(value) is not str before .strip(). This is why test_runtime_enums_reject_string_subclasses_before_lookup_callbacks and test_mutated_runtime_text_revalidates_before_caller_callbacks hold for environment_kind, redistribution_status, and operating_system.

(Refers to this code)

Was this helpful? React with 👍 or 👎 to provide feedback.

) * test(validation): define cross-engine conformance inventory contract * feat(validation): add cross-engine conformance inventory * docs(changelog): record cross-engine conformance inventory * test(validation): cover conformance contract fail-closed edges * fix(validation): seal conformance package records * test(validation): seal conformance record subclasses * fix(validation): complete conformance commit provenance boundary * fix(validation): require executed conformance evidence * test(validation): reproduce unsupported conformance coverage claim * test(validation): pin execution-backed coverage invariant * test(validation): reproduce Unicode conformance identity drift * fix(validation): normalize conformance text before hashing * test(validation): make schema-version regex literal * test(conformance): expose post-init mutation replay gap * fix(conformance): revalidate records on manifest replay * docs(conformance): record replay integrity hardening * test(conformance): document replay fixtures * fix(changelog): restore authoritative fragment format * feat(validation): add conformance run provenance (#1085) * feat(validation): add conformance run provenance * fix(changelog): restore fragment headings * test(validation): reproduce run provenance replay bypass * fix(validation): replay run provenance before serialization * docs(validation): record provenance replay integrity * fix(validation): bind executed conformance to run output provenance (#1093) * test(conformance): require output provenance for executed evidence * fix(conformance): bind executed evidence to run outputs * test(conformance): align fixtures with executed provenance contract * docs(changelog): record executed conformance provenance gate * test(conformance): seal mutated output provenance before execution gate * test(conformance): provide provenance for executed mutation fixture * Revert "test(conformance): provide provenance for executed mutation fixture" This reverts commit b94cf2e. * test(conformance): provide legacy provenance fixture * feat(validation): complete conformance runtime provenance (#1095) * test(conformance): require runtime and redistribution provenance * feat(conformance): bind runtime and redistribution provenance * test(conformance): align inventory fixtures with runtime provenance * test(conformance): carry runtime identities in provenance replay fixture * test(conformance): carry runtime identities in execution fixture * docs(changelog): record runtime provenance contract * test(conformance): cover malformed runtime provenance controls * test(conformance): provide provenance for executed mutation fixture * feat(validation): strictly replay persisted conformance manifests (#1097) * test(conformance): require strict persisted manifest replay * feat(conformance): strictly replay persisted manifests * docs(changelog): record strict conformance manifest replay * test(conformance): cover nested manifest replay boundaries * test(conformance): require stable JSON resource failures * fix(conformance): stabilize bounded JSON replay failures * fix(conformance): bound parsed manifest nesting * test(conformance): provide provenance for executed mutation fixture * feat(validation): render accessible cross-engine conformance evidence (#1164) * test(validation): require accessible conformance evidence report * feat(validation): render accessible conformance evidence * docs(changelog): record conformance evidence report * test(validation): require downloadable long-form conformance rows * feat(validation): export long-form conformance evidence rows * docs(changelog): record long-form conformance export * fix(report): stop over-escaping the CSP meta-tag content in cross-engine report Strix flagged this exact pattern on this PR (CWE-693, CVSS 6.5): escape(_CSP, quote=True) converts the CSP's literal 'none' source expression to 'none', which browsers do not parse as valid CSP syntax, silently disabling the meta-delivered policy. _CSP is a fixed module constant with no user-controlled content, so interpolating it directly is safe. This mirrors the same fix already applied to every other report generator on main in #1230. Added a regression assertion that the CSP is embedded unescaped. Verified: pytest tests/test_cross_engine_conformance_report.py -- 8 passed. * Revert "fix(report): stop over-escaping the CSP meta-tag content in cross-engine report" This reverts commit 051bc1b.

seonghobae added 6 commits August 20, 2026 14:14

test(conformance): require runtime and redistribution provenance

d659b12

feat(conformance): bind runtime and redistribution provenance

02af11b

test(conformance): align inventory fixtures with runtime provenance

00ea56b

test(conformance): carry runtime identities in provenance replay fixture

76a6b96

test(conformance): carry runtime identities in execution fixture

861253d

docs(changelog): record runtime provenance contract

b46410f

test(conformance): cover malformed runtime provenance controls

c233b23

This was referenced Aug 20, 2026

feat(validation): strictly replay persisted conformance manifests #1096

Closed

feat(validation): strictly replay persisted conformance manifests #1097

Merged

seonghobae mentioned this pull request Aug 20, 2026

test(conformance): align mutation fixture with runtime provenance #1099

Closed

test(conformance): provide provenance for executed mutation fixture

5200c0a

seonghobae marked this pull request as ready for review August 21, 2026 01:48

seonghobae enabled auto-merge (squash) August 21, 2026 01:48

devin-ai-integration Bot reviewed Aug 21, 2026

View reviewed changes

Merge remote-tracking branch 'refs/remotes/origin/base-1095' into HEAD

235c967

# Conflicts: # tests/test_cross_engine_conformance_mutation_replay.py

seonghobae merged commit 5c5b4b1 into fix/conformance-execution-provenance-1092 Aug 21, 2026
3 of 4 checks passed

seonghobae deleted the feat/conformance-runtime-provenance-1094 branch August 21, 2026 10:05

devin-ai-integration Bot reviewed Aug 21, 2026

View reviewed changes

Navigation Menu

Repository navigation

feat(validation): complete conformance runtime provenance - #1095

feat(validation): complete conformance runtime provenance#1095
seonghobae merged 10 commits into
fix/conformance-execution-provenance-1092from
feat/conformance-runtime-provenance-1094

seonghobae commented Aug 20, 2026

Uh oh!

coderabbitai Bot commented Aug 20, 2026 •

edited

Loading

Review skipped

Uh oh!

seonghobae commented Aug 20, 2026

Uh oh!

seonghobae commented Aug 20, 2026

Uh oh!

seonghobae commented Aug 21, 2026

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

seonghobae commented Aug 21, 2026

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Aug 21, 2026

Uh oh!

devin-ai-integration Bot Aug 21, 2026

Uh oh!

devin-ai-integration Bot Aug 21, 2026

Uh oh!

devin-ai-integration Bot Aug 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

seonghobae commented Aug 20, 2026

RED → GREEN

Contract

Ownership boundary

Stack / acceptance

Uh oh!

coderabbitai Bot commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

seonghobae commented Aug 20, 2026

Uh oh!

seonghobae commented Aug 20, 2026

Uh oh!

seonghobae commented Aug 21, 2026

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

✅ Devin Review: No Issues Found

Uh oh!

seonghobae commented Aug 21, 2026

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderabbitai Bot commented Aug 20, 2026 •

edited

Loading