Skip to content

feat(tests): name the offending files when the fixture provenance guard aborts - #128

Merged
codeasier merged 1 commit into
mainfrom
feat/issue-118-provenance-guard-diagnostics
Sep 17, 2026
Merged

codeasier merged 1 commit into
mainfrom
feat/issue-118-provenance-guard-diagnostics

Conversation

@codeasier

Copy link
Copy Markdown
Owner

📝 Description

tests/conftest.py deliberately aborts the whole pytest session before collection when tests/fixtures/snapshots/ holds a .pkl/.pickle that SHA256SUMS does not list, so an unreviewed pickle can never be deserialized by accident. That guard stays exactly as strict — no override flag is added. What changes is the diagnostic:

before
Exit: Executable fixture provenance check failed: Executable fixture set does not match the reviewed manifest

after
Exit: Executable fixture provenance check failed: Executable fixture set does not match the reviewed manifest in tests/fixtures/snapshots.
  unexpected (not in SHA256SUMS): memory0.pickle
  missing (in SHA256SUMS but absent): <none>
Move unreviewed pickles out of tests/fixtures/snapshots/ (for example into .tmp/), or complete the provenance review in PROVENANCE.md and update SHA256SUMS before running the suite.

The old message was hard to act on: it did not say which file was the problem, and the directory's .gitignore rule for *.pickle makes it look like a safe scratch location for local captures. On a real checkout this cost a source-code dive to discover two local pickles had to be moved.

Changes:

  • tests/_fixture_provenance.py: verify_snapshot_fixtures() reports the sorted unexpected and missing names separately with guidance; a mismatch between SHA256SUMS and the hard-coded size table is now reported as its own error (only in SHA256SUMS / only in size table) instead of being folded into the generic message. Only file names are reported — the guard still never reads pickle content.
  • The function takes fixture_dir, manifest, and expected_sizes as defaulted parameters (defaults are the existing module constants, so conftest.py and CI behavior are unchanged). This lets the diagnostics be unit-tested against a temporary directory without touching the real allowlist.
  • tests/AGENTS.md and tests/fixtures/AGENTS.md state that any stray pickle in the fixture directory aborts the entire session, that the message lists the names, that .gitignore does not make such files acceptable, and that local snapshots belong elsewhere (e.g. .tmp/).

🔗 Related Issue

Fixes #118

🧪 Testing

  • Tests added/updated
  • Manual testing completed

New tests in tests/test_fixture_provenance.py (5) use arbitrary bytes — never real pickles — in a tmp_path fixture directory: unexpected file is named with guidance and non-pickle files are ignored; missing manifest entry is named; multiple unexpected names are sorted and complete; manifest/size-table disagreement is reported; checksum and size mismatches still fail and a consistent directory passes.

pytest tests/test_fixture_provenance.py: 7 passed. Full suite (macOS, Python 3.13.11): 774 passed, 1 xfailed. Manual check: dropping a stray memory0.pickle (plain text) into the fixture directory produces the "after" message above and exits before collection; removing it restores the green run. ruff check . / black --check . clean.

✅ Checklist

  • Code follows project guidelines
  • Documentation updated (if applicable)
  • No new warnings introduced
  • Changes tested locally

Snapshot Runtime Provenance

  • Snapshot provenance evaluation completed when snapshot runtime files change
    Snapshot provenance decision: no-update
    Snapshot provenance no-update reason: This PR changes the test-side fixture guard, its tests, and two tests AGENTS.md files; no files under src/pt_snap_cli/snapshot/ are touched, and the reviewed fixture allowlist (SHA256SUMS, PROVENANCE.md) is unchanged.

…rd aborts

The session-start guard in tests/conftest.py intentionally stops pytest
before collection whenever tests/fixtures/snapshots/ contains a pickle
that SHA256SUMS does not list, so an unreviewed object can never be
deserialized by accident. Its only message, however, was "Executable
fixture set does not match the reviewed manifest": it did not say which
file was unexpected or missing, nor what to do about it, and the
directory's .gitignore rule for *.pickle makes it look like a safe place
for local captures.

verify_snapshot_fixtures() now reports the sorted unexpected and missing
names separately, points at the fix (move the file out, e.g. to .tmp/, or
complete provenance review), and reports SHA256SUMS/size-table drift as
its own error. The guard still never reads pickle content. The function
takes the fixture directory, manifest, and size table as defaulted
parameters so the diagnostics are unit-tested against a temporary
directory without touching the real allowlist. tests/AGENTS.md and
tests/fixtures/AGENTS.md document that any stray pickle aborts the whole
session and that there is deliberately no override.

Fixes #118
@codeasier
codeasier merged commit 890ce57 into main Sep 17, 2026
6 checks passed
@codeasier
codeasier deleted the feat/issue-118-provenance-guard-diagnostics branch October 3, 2026 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] fixture 溯源守卫失败时列出具体的多出/缺失文件,并在文档中说明本地 pickle 会中止整个测试会话

1 participant