feat(evidence): add deterministic exact snapshot manifests by wolfiesch · Pull Request #44 · SynthGL/ExcelBench

wolfiesch · 2026-08-31T00:18:03Z

Summary

add a versioned, deterministic exact-file evidence manifest contract
bind snapshots to an explicit source commit, observation time, and optional package/artifact subjects
fail closed on symlinks, non-UTF-8/portable-path collisions, in-flight file or namespace mutation, oversized inputs, duplicate JSON keys, missing/extra/changed files, and non-canonical manifests
publish manifests atomically with no-clobber default behavior
expose excelbench evidence-manifest and excelbench verify-evidence
document the signing/attestation boundary and ship a JSON Schema

Why

ExcelBench currently has several dated result directories but no single machine-verifiable envelope proving exactly which files, source commit, and tested package identities a claim refers to. This is the first step toward making ExcelBench the canonical evidence authority while keeping product repositories limited to signed receipts.

The manifest is deliberately deterministic: observed_at is explicit, absolute paths never enter the payload, and the aggregate digest is over the canonical sorted artifact inventory.

Verification

local focused suite: 35 evidence-manifest tests pass
local focused Ruff and strict mypy checks pass
exact-head CI at a16160ab7d7b: lint/type checking, Python 3.11 and 3.12 suites (1,260 passed, 68 skipped, 6 xfailed on each), benchmark smoke, and Rust smoke all pass (run 225)
exact-head Codex review completed with no findings; all review threads are resolved

Follow-ups

have release workflows sign/attest the canonical manifest
move large immutable result payloads to release/object artifacts while retaining this manifest in product repositories
bind public claim publication to a verified, fresh manifest

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

chatgpt-codex-connector · 2026-08-31T00:18:15Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-08-31T02:42:09.017475Z	`a16160a`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4fea8bfcd1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

wolfiesch · 2026-08-31T00:33:44Z

@codex review

Please re-review the current head after the destination-boundary and schema-hardening fixes; the full CI matrix is green.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 621f0032ed

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

wolfiesch · 2026-08-31T00:48:24Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7ad5311479

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

wolfiesch · 2026-08-31T00:54:55Z

@codex review

Please review the current exact head 9420e6c8; CI run 216 is green, and the final change extends the same portable-path contract to the manifest filename itself.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9420e6c8f2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

wolfiesch · 2026-08-31T01:08:44Z

@codex review

Please re-review exact head adfb5bfe; full CI run 218 is green after the stable-descriptor hashing and manifest-key reservation fixes.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: adfb5bfe20

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

wolfiesch · 2026-08-31T01:25:17Z

@codex review

Please re-review exact head 95d5c41e; CI run 219 is fully green after the stable second-inventory pass and explicit non-UTF-8 path refusal.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 95d5c41eed

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

wolfiesch · 2026-08-31T01:52:09Z

@codex review

Please re-review exact head bbc5c3e0. Exact-head CI run 220 is fully green, all prior threads are resolved, and the focused evidence-manifest suite is 20/20.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bbc5c3e0f4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

wolfiesch · 2026-08-31T02:08:02Z

@codex review

Please re-review current exact head f5735a24. The publish-window and manifest-size findings are fixed/resolved. Local exact-content verification: 22 focused tests passed; Ruff passed; Mypy passed for both implementation files; git diff --check passed.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f5735a2459

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

wolfiesch · 2026-08-31T02:27:22Z

@codex review

Please review exact head dbce5565f19d041e6a11ab0029946ae657c35790. The three prior P2 findings are fixed and resolved with strict serialized-text UTF-8 validation, symmetric manifest inventory limits, and descriptor-bound no-follow reads with replacement detection.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dbce5565f1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

wolfiesch · 2026-08-31T02:36:53Z

@codex review

Please re-review exact head 4ee48cb99b5532e65a61741aacbd691a93330b4a. The latest P1/P2 findings are fixed and resolved: the manifest is re-read after inventory, destination aliases are reserved during document validation, and deeply nested JSON follows an iterative/controlled error boundary.

wolfiesch · 2026-08-31T02:40:39Z

@codex review

Please review exact head a16160ab7d7be70724b33123d7e1682ce9b22da7. This test-only follow-up makes the deep-JSON refusal regression portable across Python 3.11 (decoder rejects) and 3.12 (iterative validator rejects), while asserting the same controlled CLI boundary.

chatgpt-codex-connector · 2026-08-31T02:42:07Z

Codex Review: Didn't find any major issues. 👍

Reviewed commit: a16160ab7d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

feat(evidence): add deterministic exact snapshot manifests

4fea8bf

fix(evidence): use collections ABC imports

a8aaac5

chatgpt-codex-connector Bot reviewed Aug 31, 2026

View reviewed changes

Comment thread src/excelbench/evidence_manifest.py Outdated

Comment thread src/excelbench/evidence_manifest.py

Comment thread src/excelbench/evidence_manifest.py Outdated

wolfiesch added 4 commits August 30, 2026 17:23

fix(evidence): report empty snapshots as exact mismatches

e819e4f

fix(evidence): make manifest helper type explicit

b44b7d8

fix(evidence): harden manifest replacement boundaries

6636682

test(evidence): cover destination and schema attacks

621f003

chatgpt-codex-connector Bot reviewed Aug 31, 2026

View reviewed changes

Comment thread src/excelbench/evidence_manifest.py

Comment thread src/excelbench/evidence_manifest.py

Comment thread README.md Outdated

wolfiesch added 3 commits August 30, 2026 17:44

fix(evidence): enforce portable artifact paths

4842924

test(evidence): cover path aliases and decoder limits

d9b77c6

docs(evidence): render the CLI example as a code block

7ad5311

wolfiesch added 2 commits August 30, 2026 17:49

fix(evidence): require portable manifest filenames

e6f2c9d

test(evidence): reject nonportable manifest names

9420e6c

chatgpt-codex-connector Bot reviewed Aug 31, 2026

View reviewed changes

Comment thread src/excelbench/evidence_manifest.py Outdated

Comment thread src/excelbench/evidence_manifest.py

chatgpt-codex-connector Bot reviewed Aug 31, 2026

View reviewed changes

Comment thread src/excelbench/evidence_manifest.py

wolfiesch added 2 commits August 30, 2026 18:04

fix(evidence): reject unstable and manifest-aliased artifacts

1d54474

test(evidence): cover unstable files and name aliases

adfb5bf

chatgpt-codex-connector Bot reviewed Aug 31, 2026

View reviewed changes

Comment thread src/excelbench/evidence_manifest.py

Comment thread src/excelbench/evidence_manifest.py

fix(evidence): require a stable portable inventory

95d5c41

chatgpt-codex-connector Bot reviewed Aug 31, 2026

View reviewed changes

Comment thread src/excelbench/evidence_manifest.py Outdated

fix(evidence): bind hashes to the initial walk

bbc5c3e

chatgpt-codex-connector Bot reviewed Aug 31, 2026

View reviewed changes

Comment thread src/excelbench/cli.py

Comment thread src/excelbench/evidence_manifest.py Outdated

fix(evidence): align manifest generation and read limits

a1c8859

wolfiesch added 2 commits August 30, 2026 19:07

fix(evidence): verify snapshot after publication

05a75a7

test(evidence): cover publish races and size bounds

f5735a2

chatgpt-codex-connector Bot reviewed Aug 31, 2026

View reviewed changes

Comment thread src/excelbench/evidence_manifest.py

Comment thread src/excelbench/evidence_manifest.py

Comment thread src/excelbench/evidence_manifest.py Outdated

fix(evidence): harden manifest validation and reads

dbce556

chatgpt-codex-connector Bot reviewed Aug 31, 2026

View reviewed changes

Comment thread src/excelbench/evidence_manifest.py

Comment thread src/excelbench/evidence_manifest.py Outdated

Comment thread src/excelbench/evidence_manifest.py

fix(evidence): close verification race windows

4ee48cb

test(evidence): keep deep refusal cross-version

a16160a

Uh oh!

Conversation

wolfiesch commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Verification

Follow-ups

Uh oh!

chatgpt-codex-connector Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wolfiesch commented Aug 31, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wolfiesch commented Aug 31, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

wolfiesch commented Aug 31, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

wolfiesch commented Aug 31, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

wolfiesch commented Aug 31, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

wolfiesch commented Aug 31, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

wolfiesch commented Aug 31, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wolfiesch commented Aug 31, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

wolfiesch commented Aug 31, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Aug 31, 2026 •

edited

Loading