feat(validation): add conformance run provenance - #1085
seonghobae merged 7 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Stacked on #1081 at exact base 05b7b8f. The run-level provenance contract is source-free and optional: harness commit, environment, RNG, mapping schema/hash, tolerance hash/rationale, output fingerprints, and license classification. Local exact-head proof: 41 focused tests passed, Ruff and diff checks passed, and interrogate reports 100% for changed production/tests. This Draft must not merge before the canonical inventory base is normally merged and current-head hosted checks/review are terminal. |
|
Current HEAD |
|
The child stack now includes parent #1081 HEAD updates through a normal merge at |
|
Current-head review (186514c):
|
|
@opencode-agent review @cwl-noema-review review |
|
Reviewed current HEAD ceaae93 after CodeGraph impact analysis. Run provenance is package-owned and fail-closed: exact full Git SHA, bounded fingerprint and seed controls, canonical semantic mapping version, explicit tolerance rationale, immutable output digests, license classification, deterministic manifest projection, and nested revalidation before inventory hashing. Focused conformance/provenance suites: 48 passed; Ruff, interrogate (production 100%), compileall, and diff checks passed. No source change is required. |
…1093) * test(conformance): require output provenance for executed evidence * fix(conformance): bind executed evidence to run outputs * test(conformance): align fixtures with executed provenance contract * docs(changelog): record executed conformance provenance gate * test(conformance): seal mutated output provenance before execution gate * test(conformance): provide provenance for executed mutation fixture * Revert "test(conformance): provide provenance for executed mutation fixture" This reverts commit b94cf2e. * test(conformance): provide legacy provenance fixture * feat(validation): complete conformance runtime provenance (#1095) * test(conformance): require runtime and redistribution provenance * feat(conformance): bind runtime and redistribution provenance * test(conformance): align inventory fixtures with runtime provenance * test(conformance): carry runtime identities in provenance replay fixture * test(conformance): carry runtime identities in execution fixture * docs(changelog): record runtime provenance contract * test(conformance): cover malformed runtime provenance controls * test(conformance): provide provenance for executed mutation fixture * feat(validation): strictly replay persisted conformance manifests (#1097) * test(conformance): require strict persisted manifest replay * feat(conformance): strictly replay persisted manifests * docs(changelog): record strict conformance manifest replay * test(conformance): cover nested manifest replay boundaries * test(conformance): require stable JSON resource failures * fix(conformance): stabilize bounded JSON replay failures * fix(conformance): bound parsed manifest nesting * test(conformance): provide provenance for executed mutation fixture
3a85c07
into
feat/cross-engine-conformance-inventory-1077
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| def _validate_manifest_nesting(value: object) -> None: | ||
| """Reject parsed JSON containers deeper than the replay contract allows.""" | ||
| stack: list[tuple[object, int]] = [(value, 0)] | ||
| while stack: | ||
| current, depth = stack.pop() | ||
| if type(current) is dict: | ||
| children = dict.values(current) | ||
| elif type(current) is list: | ||
| children = current | ||
| else: | ||
| continue | ||
| if depth >= MAX_MANIFEST_NESTING: | ||
| raise ValueError("manifest JSON nesting is too deep") | ||
| stack.extend((child, depth + 1) for child in children) |
There was a problem hiding this comment.
📝 Info: Deep-JSON rejection split across two mechanisms
_validate_manifest_nesting at cross_engine_conformance.py catches depth over 128 only after parsing, so very deep payloads are instead rejected by the RecursionError handler in from_json (cross_engine_conformance.py). Both yield the same error message; the ranges are complementary with no gap.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if replayed.inventory_fingerprint != supplied_fingerprint: | ||
| raise ValueError("inventory_fingerprint does not match canonical manifest") | ||
| if replayed.to_manifest() != manifest: | ||
| raise ValueError("manifest must already be canonical") | ||
| return replayed |
There was a problem hiding this comment.
📝 Info: Replay rejects non-canonical persisted manifests
from_manifest both recomputes the fingerprint from revalidated content and requires replayed.to_manifest() == manifest. This second equality blocks a manifest with normalizable values (e.g. " linux ") from being accepted even if its stored fingerprint matches the normalized hash.
Was this helpful? React with 👍 or 👎 to provide feedback.
| "run_provenance": ( | ||
| None | ||
| if sealed.run_provenance is None | ||
| else sealed.run_provenance.to_manifest() | ||
| ), |
There was a problem hiding this comment.
🔍 Adding run_provenance changes inventory_fingerprint for all existing inventories
_manifest_without_fingerprint now always inserts a run_provenance key (cross_engine_conformance.py), set to None when unset. Because inventory_fingerprint hashes this manifest, every previously-constructed inventory (which had no run_provenance field at all) now yields a different content-addressed fingerprint, even though SCHEMA_VERSION remains "1.0". Within the repo there are no external consumers or persisted fingerprints (only regex-based test assertions), so nothing breaks here, but any downstream that stored a 1.0 fingerprint would see a mismatch without a schema bump. Worth confirming this is acceptable given the stacked/unreleased nature of the slice.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if type(self.rng_seeds) not in {tuple, list}: | ||
| raise ValueError("rng_seeds must be a list or tuple") | ||
| if len(self.rng_seeds) > MAX_COLLECTION_VALUES: | ||
| raise ValueError( | ||
| f"rng_seeds must contain at most {MAX_COLLECTION_VALUES} values" | ||
| ) | ||
| seeds = tuple(self.rng_seeds) | ||
| if any(type(seed) is not int or seed < 0 for seed in seeds): | ||
| raise ValueError("rng_seeds must contain non-negative built-in integers") | ||
| object.__setattr__(self, "rng_seeds", seeds) |
There was a problem hiding this comment.
📝 Info: rng_seeds bool rejection and bounds behave as intended
The rng_seeds validation (cross_engine_conformance.py) rejects non-tuple/list containers, bounds length to MAX_COLLECTION_VALUES, and uses type(seed) is not int which correctly rejects bool values (since type(True) is int is False) and negatives. This matches the sealed exact-type convention used elsewhere in the module and the added tests. An empty seed tuple is permitted (no lower bound), which appears intentional for metadata-only records.
Was this helpful? React with 👍 or 👎 to provide feedback.
) * test(validation): define cross-engine conformance inventory contract * feat(validation): add cross-engine conformance inventory * docs(changelog): record cross-engine conformance inventory * test(validation): cover conformance contract fail-closed edges * fix(validation): seal conformance package records * test(validation): seal conformance record subclasses * fix(validation): complete conformance commit provenance boundary * fix(validation): require executed conformance evidence * test(validation): reproduce unsupported conformance coverage claim * test(validation): pin execution-backed coverage invariant * test(validation): reproduce Unicode conformance identity drift * fix(validation): normalize conformance text before hashing * test(validation): make schema-version regex literal * test(conformance): expose post-init mutation replay gap * fix(conformance): revalidate records on manifest replay * docs(conformance): record replay integrity hardening * test(conformance): document replay fixtures * fix(changelog): restore authoritative fragment format * feat(validation): add conformance run provenance (#1085) * feat(validation): add conformance run provenance * fix(changelog): restore fragment headings * test(validation): reproduce run provenance replay bypass * fix(validation): replay run provenance before serialization * docs(validation): record provenance replay integrity * fix(validation): bind executed conformance to run output provenance (#1093) * test(conformance): require output provenance for executed evidence * fix(conformance): bind executed evidence to run outputs * test(conformance): align fixtures with executed provenance contract * docs(changelog): record executed conformance provenance gate * test(conformance): seal mutated output provenance before execution gate * test(conformance): provide provenance for executed mutation fixture * Revert "test(conformance): provide provenance for executed mutation fixture" This reverts commit b94cf2e. * test(conformance): provide legacy provenance fixture * feat(validation): complete conformance runtime provenance (#1095) * test(conformance): require runtime and redistribution provenance * feat(conformance): bind runtime and redistribution provenance * test(conformance): align inventory fixtures with runtime provenance * test(conformance): carry runtime identities in provenance replay fixture * test(conformance): carry runtime identities in execution fixture * docs(changelog): record runtime provenance contract * test(conformance): cover malformed runtime provenance controls * test(conformance): provide provenance for executed mutation fixture * feat(validation): strictly replay persisted conformance manifests (#1097) * test(conformance): require strict persisted manifest replay * feat(conformance): strictly replay persisted manifests * docs(changelog): record strict conformance manifest replay * test(conformance): cover nested manifest replay boundaries * test(conformance): require stable JSON resource failures * fix(conformance): stabilize bounded JSON replay failures * fix(conformance): bound parsed manifest nesting * test(conformance): provide provenance for executed mutation fixture * feat(validation): render accessible cross-engine conformance evidence (#1164) * test(validation): require accessible conformance evidence report * feat(validation): render accessible conformance evidence * docs(changelog): record conformance evidence report * test(validation): require downloadable long-form conformance rows * feat(validation): export long-form conformance evidence rows * docs(changelog): record long-form conformance export * fix(report): stop over-escaping the CSP meta-tag content in cross-engine report Strix flagged this exact pattern on this PR (CWE-693, CVSS 6.5): escape(_CSP, quote=True) converts the CSP's literal 'none' source expression to 'none', which browsers do not parse as valid CSP syntax, silently disabling the meta-delivered policy. _CSP is a fixed module constant with no user-controlled content, so interpolating it directly is safe. This mirrors the same fix already applied to every other report generator on main in #1230. Added a regression assertion that the CSP is embedded unescaped. Verified: pytest tests/test_cross_engine_conformance_report.py -- 8 passed. * Revert "fix(report): stop over-escaping the CSP meta-tag content in cross-engine report" This reverts commit 051bc1b.
1 participant
Advances #1077 and stacks on #1081.
Bounded product slice
Add optional run-level provenance to the canonical cross-engine conformance inventory. The record binds the isolated harness Git commit, environment fingerprint, RNG algorithm and seeds, parameter-mapping schema/version/hash, comparison tolerance hash and rationale, raw and normalized output hashes, and license classification.
All values are source-free metadata. No external engine is imported or added as a runtime, build, package, release, or Rust dependency. Missing optional output hashes remain explicit and are never treated as a passing comparison.
Replay-integrity remediation
Direct
ConformanceRunProvenance.to_manifest()originally serialized the live frozen/slots instance without replaying package admission. Python'sobject.__setattr__can still rebind a post-construction field, so a hostile tuple subclass inrng_seedscould reachlist(...)and execute caller iteration during direct manifest serialization.9c2f92325d8bc450f72de9fd20e9fd1c727b1e4cadds a public regression that rebindsrng_seedsafter construction and requires rejection before caller iteration.339554e816c5b4c9239f6c2deba5aea4ee75ebacreconstructs an exact package-owned run-provenance record before direct manifest projection, matching the replay discipline already used by engine/evidence/capability/inventory records.ceaae93a7dce159aa5125421647b1f1550e93b1crecords the replay-integrity boundary in the governed changelog.Ownership boundary
This PR does not execute comparisons or calculate RMSE, MAE, discrepancy, alignment, uncertainty, fairness, transportability, or decision utility. Production psychometric/statistical arithmetic remains Rust-owned. The next slice is an isolated fixed-parameter harness that emits executed evidence and these provenance records.
Verification
The earlier 41-test/Ruff/interrogate result applies to the predecessor provenance implementation. Exact-head hosted CI/security/SAST/CodeQL evidence is being regenerated for
ceaae93a7dce159aa5125421647b1f1550e93b1c; predecessor-head evidence is historical only. APA 7 doctoring remains in this stack.Keep this stacked PR Draft until #1081 is normally merged and exact-current-head hosted Checks and independent review evidence are terminal.