Parent scope: automation.
test.yml runs for main pushes, main-targeted PR opened/synchronize/reopened/edited
events, workflow_call, nightly at 03:17 UTC, and workflow_dispatch. Provenance,
lint, browser and Python test-matrix jobs are independent. Checkouts use the PR
head SHA or the triggering SHA; manual dispatch checks the selected ref's SHA.
Concurrency is keyed by GitHub ref and superseded runs are cancelled.
| Job/step | Gate |
|---|---|
| Snapshot provenance | Full Git history; base/head comparison; PR decision required only in PR mode |
| Lint & Format | Python 3.13, editable dev install, Ruff, Black, basedpyright with the selected interpreter |
| Compatibility matrix | Python 3.10–3.12; fixture provenance first, then pytest -m "not slow and not nightly" |
| Full PR test tier | Python 3.13; pytest -m "not nightly" with branch coverage, preserving the existing slow tests |
| Overall coverage | pyproject.toml fail-under 90 with branches enabled |
| Critical runtime coverage | Ten independent line+branch floors, enforced by the fail-closed checker below |
| Python 3.13 packaging | Upload coverage, build wheel, run offline clean-wheel skill/CLI acceptance including both sharded formats |
| Nightly/manual dataset acceptance | Python 3.13; provenance, wheel build and audit, then nightly tests including the required installed-wheel CLI/API test |
Million-event deep-offset pagination tests are marked slow; their small
pagination contracts still run across the compatibility matrix.
After full Python 3.13 coverage, check_critical_coverage.py enforces this complete
inventory. A high average cannot compensate for a single file below its floor.
Runtime module (src/pt_snap_cli/) |
Minimum line + branch coverage |
|---|---|
core/snapshot_import_backend.py |
90% |
snapshot/simulate/simulated_caching_allocator.py |
88% |
snapshot/simulate/snapshot_mutator.py |
84% |
core/import_metadata_contract.py |
100% |
core/import_metadata.py |
90% |
core/split_service.py |
90% |
core/focus_service.py |
90% |
core/skill_service.py |
90% |
core/snapshot_cache.py |
90% |
snapshot/representation.py |
95% |
The checker requires .coverage, branch data, every listed source file, measured
coverage for each exact path, and executable statements. Missing/corrupt evidence
fails; percentages are compared before display rounding. It has no selector or
floor overrides. The overall 90% gate remains separate and unchanged. CI contract
mutation tests reject an omitted/conditional checker, alternate coverage commands,
ignored failures, reordered execution, and narrowed source/omit/exclusion settings;
tests/test_critical_coverage_gate.py separately protects the exact inventory and
exercises real coverage-data success/failure cases. Raise floors only with actual
CI evidence; do not delete code, add exclusions, or weaken assertions to meet them.
tests/test_ci_tiers.py checks parsed commands, matrix rows, marker truth tables,
conditions, independent file floors and gate ordering, not step display names or
complete shell text. tests/_ci_contract.py intentionally supports only a bounded
CI language: simple failure-propagating commands, shell quoting/continuations,
sequential steps or a terminal && chain, and explicit boolean conditions.
Unknown shell control flow, OR/pipes, failure suppression, selector overrides and
unrecognized conditions fail closed. Extend it with positive and negative
mutation tests when introducing new syntax; do not weaken it into substring
checks. Action version bumps, cache configuration, renaming, equivalent selectors
and step splits/merges do not change a gate.
The Linux browser job installs Chromium and sets PT_SNAP_REQUIRE_BROWSER=1.
Browser cases are always collected even if Playwright is absent. Optional matrix
and platform runs report each missing-dependency case as skipped plus an explicit
not executed summary; that is not browser-pass evidence. Required mode fails
for missing package/binary, any browser skip, empty/deselected collection or
incomplete execution. --collect-only remains a collection diagnostic and is
explicitly reported as not run. Broken installations and launch failures fail in
both modes. Browser results are not added to non-browser platform requirements.
Coverage reporting writes .coverage/coverage.xml; building writes dist/ and
intermediates. Dependency installation/build isolation may access the network.
The acceptance script uses temporary
paths but imports a reviewed executable fixture. These are more than YAML checks.
The nightly/manual job invokes run_dataset_acceptance.py with the single built
wheel. All nightly tests are selected automatically; the installed-wheel case
must appear exactly once. New mandatory gates also add their complete pytest
node IDs to REQUIRED_TEST_NODES in that runner. Do not register absent future
files. Missing wheels, missing/empty JUnit, missing required nodes, duplicate
identities, any skipped/failed/error test, or evidence for a different wheel fail
the gate. It uploads the pytest log, JUnit (including the test's evidence), and a
summary with complete executed/required test node IDs, actual checkout SHA and wheel
SHA-256, even after failure. PT_SNAP_ACCEPTANCE_EVIDENCE_DIR is available for
additional test-owned artifacts. Local invocation:
python .github/scripts/run_dataset_acceptance.py \
--wheel dist/<built-wheel>.whl --output-dir .tmp/dataset-acceptanceThe required installed-wheel case creates a clean venv without system packages
and runs its actual pt-snap console script from isolated HOME/CWD. Its trusted,
test-owned input has exactly 1,000,000 events; the default exporter produces two
full 500,000-event shards. All installed catalog entries are classified: currently
12 dataset-supported templates succeed with installed CLI/API equivalence and
independent known-result checks; event_frames remains the explicit
unsupported-dataset query. Both peak-memory and memory-tree reports succeed.
The allocation/free-request/free-completion boundary assertions remain covered,
alongside static occupancy, offset pages,
terminal survivors and unchanged source/manifest/member hashes. A second required
nightly case separately imports 500,002 events and proves the original final
short shard has exactly two real events, with the crossing completed lifetime
and the final survivor. Its evidence is stored in its own subdirectory so the
two gates cannot overwrite each other's command logs. This adds one half-million
event import plus five query subprocesses to the nightly job. Existing
synthetic scale/default-budget regressions remain separate source-level tests.
PT_SNAP_ACCEPTANCE_WHEELHOUSE optionally supplies dependencies for an offline
clean-venv install. Timings are recorded without machine-speed thresholds; this
does not claim live agents, live hardware capture or upstream GUI interoperability.
The ordinary PR tier excludes nightly tests; the required nightly/manual runner
must never skip them. Manually dispatch the exact published branch/commit and
verify the run's head SHA and summary; a PR run alone is not nightly evidence.
release.yml triggers on v* tag pushes and enforces:
quality → build → publish → github-release.
qualityinvokes./.github/workflows/test.ymlat the tagged commit.buildfetches full history for setuptools-scm, builds sdist/wheel, verifies the installed wheel's version matches the tag, runspt-snap --version, and extracts a nonempty matchingCHANGELOG.mdsection before uploading artifacts.publishuses the protectedpypienvironment and OIDC trusted publishing.github-releasepublishes the same distributions and extracted notes only after PyPI publication succeeds.
Do not substitute a different commit's CI result or publish before version/notes validation. For non-PR provenance checks, tag runs compare against the prior tag (or root commit fallback); pushes use the before SHA with a root fallback.
Review pyproject.toml, both workflow YAMLs, the provenance/acceptance scripts and
tests/test_release_workflow.py together when changing execution order. Validate
YAML and run pytest tests/test_governance.py tests/test_release_workflow.py tests/test_ci_tiers.py tests/test_ci_contract_mutations.py tests/test_browser_execution_contract.py tests/test_dataset_acceptance_runner.py tests/test_wheel_audit.py from
the repository root. Full CI tests the matrix and installed wheel; static tests
do not prove a release environment is configured or a publication occurred.
The scheduled/manual dataset-acceptance job also runs
run_differential_acceptance.py --seed-count 200 after the installed-wheel
nightly gate. This separate source-query step is required, with no skip or
continue-on-error escape. Its JUnit report must contain exactly 400 distinct
seed/format test IDs (200 per v1/v2), matching all per-seed coverage artifacts,
expected query counts/templates, zero unexpected errors and actual scratch use.
A single-seed environment override is removed by the runner. Logs, source SHA,
summary, JUnit and compact timeline/coverage/failure JSON are always uploaded from
.tmp/differential-acceptance/; regenerated SQLite files stay out of stored artifacts. The ordinary PR matrix retains the default
20-seed profile. This does not make the differential step installed-wheel or
upstream-producer evidence.