fix(validation): bind executed conformance to run output provenance - #1093
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Current-head review and fix record: executed conformance evidence now requires an exact run-level provenance record plus both raw and normalized output SHA-256 identities; nested provenance is revalidated before the execution gate, and no raw output or external engine is introduced. During review, the parent mutation fixture failed because it omitted the new required provenance. Added a current-contract-only fixture correction (no future enum fields). Exact new HEAD |
* test(conformance): require runtime and redistribution provenance * feat(conformance): bind runtime and redistribution provenance * test(conformance): align inventory fixtures with runtime provenance * test(conformance): carry runtime identities in provenance replay fixture * test(conformance): carry runtime identities in execution fixture * docs(changelog): record runtime provenance contract * test(conformance): cover malformed runtime provenance controls * test(conformance): provide provenance for executed mutation fixture * feat(validation): strictly replay persisted conformance manifests (#1097) * test(conformance): require strict persisted manifest replay * feat(conformance): strictly replay persisted manifests * docs(changelog): record strict conformance manifest replay * test(conformance): cover nested manifest replay boundaries * test(conformance): require stable JSON resource failures * fix(conformance): stabilize bounded JSON replay failures * fix(conformance): bound parsed manifest nesting * test(conformance): provide provenance for executed mutation fixture
b297a2d
into
feat/cross-engine-conformance-provenance-1077
* feat(validation): add conformance run provenance * fix(changelog): restore fragment headings * test(validation): reproduce run provenance replay bypass * fix(validation): replay run provenance before serialization * docs(validation): record provenance replay integrity * fix(validation): bind executed conformance to run output provenance (#1093) * test(conformance): require output provenance for executed evidence * fix(conformance): bind executed evidence to run outputs * test(conformance): align fixtures with executed provenance contract * docs(changelog): record executed conformance provenance gate * test(conformance): seal mutated output provenance before execution gate * test(conformance): provide provenance for executed mutation fixture * Revert "test(conformance): provide provenance for executed mutation fixture" This reverts commit b94cf2e. * test(conformance): provide legacy provenance fixture * feat(validation): complete conformance runtime provenance (#1095) * test(conformance): require runtime and redistribution provenance * feat(conformance): bind runtime and redistribution provenance * test(conformance): align inventory fixtures with runtime provenance * test(conformance): carry runtime identities in provenance replay fixture * test(conformance): carry runtime identities in execution fixture * docs(changelog): record runtime provenance contract * test(conformance): cover malformed runtime provenance controls * test(conformance): provide provenance for executed mutation fixture * feat(validation): strictly replay persisted conformance manifests (#1097) * test(conformance): require strict persisted manifest replay * feat(conformance): strictly replay persisted manifests * docs(changelog): record strict conformance manifest replay * test(conformance): cover nested manifest replay boundaries * test(conformance): require stable JSON resource failures * fix(conformance): stabilize bounded JSON replay failures * fix(conformance): bound parsed manifest nesting * test(conformance): provide provenance for executed mutation fixture
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
📝 Info: New executed-evidence gate ordering preserves prior validation errors
The new fail-closed gate at cross_engine_conformance.py runs after package_version, source_commit, _capability_values, and _schema_version validation. I verified this ordering does not regress the untouched edge tests in tests/test_cross_engine_conformance_edges.py: constructions expecting errors (duplicate capabilities, bad schema 2.0, container-type errors) all trip earlier validation before the executed gate, so they still raise the expected messages. The gate also correctly uses the revalidated local capabilities/provenance variables rather than raw attributes, so post-construction mutation cannot bypass it.
(Refers to this code)
Was this helpful? React with 👍 or 👎 to provide feedback.
| def _validate_manifest_nesting(value: object) -> None: | ||
| """Reject parsed JSON containers deeper than the replay contract allows.""" | ||
| stack: list[tuple[object, int]] = [(value, 0)] | ||
| while stack: | ||
| current, depth = stack.pop() | ||
| if type(current) is dict: | ||
| children = dict.values(current) | ||
| elif type(current) is list: | ||
| children = current | ||
| else: | ||
| continue | ||
| if depth >= MAX_MANIFEST_NESTING: | ||
| raise ValueError("manifest JSON nesting is too deep") | ||
| stack.extend((child, depth + 1) for child in children) |
There was a problem hiding this comment.
📝 Info: Deep JSON nesting has two independent guards
from_json guards stack exhaustion twice: it catches RecursionError from json.loads, and it runs the iterative depth check _validate_manifest_nesting (cross_engine_conformance.py). Both raise the same 'too deep' error, so nesting rejection is stable across interpreter recursion limits. from_manifest reads a fixed schema depth, so a directly-passed dict cannot trigger unbounded recursion.
Was this helpful? React with 👍 or 👎 to provide feedback.
) * test(validation): define cross-engine conformance inventory contract * feat(validation): add cross-engine conformance inventory * docs(changelog): record cross-engine conformance inventory * test(validation): cover conformance contract fail-closed edges * fix(validation): seal conformance package records * test(validation): seal conformance record subclasses * fix(validation): complete conformance commit provenance boundary * fix(validation): require executed conformance evidence * test(validation): reproduce unsupported conformance coverage claim * test(validation): pin execution-backed coverage invariant * test(validation): reproduce Unicode conformance identity drift * fix(validation): normalize conformance text before hashing * test(validation): make schema-version regex literal * test(conformance): expose post-init mutation replay gap * fix(conformance): revalidate records on manifest replay * docs(conformance): record replay integrity hardening * test(conformance): document replay fixtures * fix(changelog): restore authoritative fragment format * feat(validation): add conformance run provenance (#1085) * feat(validation): add conformance run provenance * fix(changelog): restore fragment headings * test(validation): reproduce run provenance replay bypass * fix(validation): replay run provenance before serialization * docs(validation): record provenance replay integrity * fix(validation): bind executed conformance to run output provenance (#1093) * test(conformance): require output provenance for executed evidence * fix(conformance): bind executed evidence to run outputs * test(conformance): align fixtures with executed provenance contract * docs(changelog): record executed conformance provenance gate * test(conformance): seal mutated output provenance before execution gate * test(conformance): provide provenance for executed mutation fixture * Revert "test(conformance): provide provenance for executed mutation fixture" This reverts commit b94cf2e. * test(conformance): provide legacy provenance fixture * feat(validation): complete conformance runtime provenance (#1095) * test(conformance): require runtime and redistribution provenance * feat(conformance): bind runtime and redistribution provenance * test(conformance): align inventory fixtures with runtime provenance * test(conformance): carry runtime identities in provenance replay fixture * test(conformance): carry runtime identities in execution fixture * docs(changelog): record runtime provenance contract * test(conformance): cover malformed runtime provenance controls * test(conformance): provide provenance for executed mutation fixture * feat(validation): strictly replay persisted conformance manifests (#1097) * test(conformance): require strict persisted manifest replay * feat(conformance): strictly replay persisted manifests * docs(changelog): record strict conformance manifest replay * test(conformance): cover nested manifest replay boundaries * test(conformance): require stable JSON resource failures * fix(conformance): stabilize bounded JSON replay failures * fix(conformance): bound parsed manifest nesting * test(conformance): provide provenance for executed mutation fixture * feat(validation): render accessible cross-engine conformance evidence (#1164) * test(validation): require accessible conformance evidence report * feat(validation): render accessible conformance evidence * docs(changelog): record conformance evidence report * test(validation): require downloadable long-form conformance rows * feat(validation): export long-form conformance evidence rows * docs(changelog): record long-form conformance export * fix(report): stop over-escaping the CSP meta-tag content in cross-engine report Strix flagged this exact pattern on this PR (CWE-693, CVSS 6.5): escape(_CSP, quote=True) converts the CSP's literal 'none' source expression to 'none', which browsers do not parse as valid CSP syntax, silently disabling the meta-delivered policy. _CSP is a fixed module constant with no user-controlled content, so interpolating it directly is safe. This mirrors the same fix already applied to every other report generator on main in #1230. Added a regression assertion that the CSP is embedded unescaped. Verified: pytest tests/test_cross_engine_conformance_report.py -- 8 passed. * Revert "fix(report): stop over-escaping the CSP meta-tag content in cross-engine report" This reverts commit 051bc1b.
1 participant
Fixes #1092. Advances #1077. Stacks directly on #1085.
RED → GREEN
3b24452acd916304f4ce83676391dbc3ed5b5655adds public regressions proving that executedpassed,failed, andindeterminateevidence was accepted without run provenance, and that executed inventories could omit raw or normalized output hashes.8de61b4a275775c8bc723dcc86855c8440b5ece3centralizes executed-status identity and makesConformanceInventoryfail closed unless an executed evidence set has a revalidatedConformanceRunProvenancewith both raw-output and normalized-output SHA-256 identities.ef04b7a034623cc64cfcc1144603e195e31a2e97updates existing executed fixtures to carry complete provenance while preserving optional output hashes for genuinely nonexecuted plans.5c5edf601778c72c5ce4c6b480c796a299a41461adds the governed changelog fragment.Integrity boundary
The consistency check runs only after capability and nested provenance records are reconstructed through the existing exact-record admission path. Post-construction mutation therefore cannot bypass the new executed-run gate before validation.
Ownership boundary
Python validation/provenance only. No likelihood, estimator, scoring, alignment, discrepancy, bias/MAE/RMSE, Monte Carlo, uncertainty, or other psychometric/statistical arithmetic changes. External engines remain isolated validation instruments and production numerical ownership remains Rust-first.
Stack / acceptance
Base is the exact current #1085 branch. Keep Draft until the predecessor stack is integrated and this exact head has terminal CI/security/package/coverage and independent review evidence. Predecessor-head checks/reviews are historical only.