Repository navigation
feat(tests): name the offending files when the fixture provenance guard aborts - #128
Merged
Merged
Conversation
…rd aborts The session-start guard in tests/conftest.py intentionally stops pytest before collection whenever tests/fixtures/snapshots/ contains a pickle that SHA256SUMS does not list, so an unreviewed object can never be deserialized by accident. Its only message, however, was "Executable fixture set does not match the reviewed manifest": it did not say which file was unexpected or missing, nor what to do about it, and the directory's .gitignore rule for *.pickle makes it look like a safe place for local captures. verify_snapshot_fixtures() now reports the sorted unexpected and missing names separately, points at the fix (move the file out, e.g. to .tmp/, or complete provenance review), and reports SHA256SUMS/size-table drift as its own error. The guard still never reads pickle content. The function takes the fixture directory, manifest, and size table as defaulted parameters so the diagnostics are unit-tested against a temporary directory without touching the real allowlist. tests/AGENTS.md and tests/fixtures/AGENTS.md document that any stray pickle aborts the whole session and that there is deliberately no override. Fixes #118
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📝 Description
tests/conftest.pydeliberately aborts the whole pytest session before collection whentests/fixtures/snapshots/holds a.pkl/.picklethatSHA256SUMSdoes not list, so an unreviewed pickle can never be deserialized by accident. That guard stays exactly as strict — no override flag is added. What changes is the diagnostic:The old message was hard to act on: it did not say which file was the problem, and the directory's
.gitignorerule for*.picklemakes it look like a safe scratch location for local captures. On a real checkout this cost a source-code dive to discover two local pickles had to be moved.Changes:
tests/_fixture_provenance.py:verify_snapshot_fixtures()reports the sorted unexpected and missing names separately with guidance; a mismatch betweenSHA256SUMSand the hard-coded size table is now reported as its own error (only in SHA256SUMS/only in size table) instead of being folded into the generic message. Only file names are reported — the guard still never reads pickle content.fixture_dir,manifest, andexpected_sizesas defaulted parameters (defaults are the existing module constants, soconftest.pyand CI behavior are unchanged). This lets the diagnostics be unit-tested against a temporary directory without touching the real allowlist.tests/AGENTS.mdandtests/fixtures/AGENTS.mdstate that any stray pickle in the fixture directory aborts the entire session, that the message lists the names, that.gitignoredoes not make such files acceptable, and that local snapshots belong elsewhere (e.g..tmp/).🔗 Related Issue
Fixes #118
🧪 Testing
New tests in
tests/test_fixture_provenance.py(5) use arbitrary bytes — never real pickles — in atmp_pathfixture directory: unexpected file is named with guidance and non-pickle files are ignored; missing manifest entry is named; multiple unexpected names are sorted and complete; manifest/size-table disagreement is reported; checksum and size mismatches still fail and a consistent directory passes.pytest tests/test_fixture_provenance.py: 7 passed. Full suite (macOS, Python 3.13.11):774 passed, 1 xfailed. Manual check: dropping a straymemory0.pickle(plain text) into the fixture directory produces the "after" message above and exits before collection; removing it restores the green run.ruff check ./black --check .clean.✅ Checklist
Snapshot Runtime Provenance
Snapshot provenance decision: no-update
Snapshot provenance no-update reason: This PR changes the test-side fixture guard, its tests, and two tests AGENTS.md files; no files under src/pt_snap_cli/snapshot/ are touched, and the reviewed fixture allowlist (SHA256SUMS, PROVENANCE.md) is unchanged.