Skip to content

release-guide.md

Latest commit

 

History

History
292 lines (248 loc) · 15.3 KB

release-guide.md

File metadata and controls

292 lines (248 loc) · 15.3 KB
Raw

Cutting a BenchBox release

BenchBox releases follow a version-branch flow on a single repo (joeharris76/BenchBox) with two long-lived branches: develop (dev work) and release (release-only). This guide is the maintainer runbook.

The flow (2 commands)

git checkout develop && git pull
make release-cut VERSION=X.Y.Z
# review the PR; wait for validate-base and release-required-result
make release-finalize VERSION=X.Y.Z

That's the entire flow. The two Make targets do the rest. Precondition: fresh committed UAT release-gate evidence (≤21 days; see "UAT release-gate evidence" below) — without it validate-base fails the release PR.

Pre-merge release-required contract

Release PRs target release and must be opened from branches accepted by .github/workflows/validate-release-pr.yml (vX.Y.Z, optionally with a suffix). Before a release PR can merge, the release-only ruleset must require:

  • validate-base
  • release-required-result

release-required-result is the stable umbrella check in .github/workflows/test.yml. A green result means the release PR branch passed:

  • the required fast lane, test (ubuntu-latest, 3.12);
  • the bounded real-result correctness gate, make test-correctness-gate (DuckDB x TPC-H at SF=1 with the pinned reference qgen seed, through generate/load/execute with EXACT stored answer-set row-count validation of the 18 answer-stable TPC-H queries; Q11/Q16/Q18/Q20 are excluded for answer-set boundary sensitivity, and validation is cardinality-level, not value-level);
  • the credential-free integration-not-slow suite: tests/integration -m "integration and not (slow or stress or resource_heavy or live_integration)";
  • isolated exact-one-wheel package build/install smoke;
  • dependency upper-bound checks;
  • release-branch curation checks that confirm dev-only paths are absent.

It does not rerun the full stress matrix, live cloud integrations, or long-running UAT on the release PR itself. Slow/resource-heavy coverage is enforced through the freshness-based release canary below; long-running UAT is enforced through committed release-gate evidence (next sections).

Release canary and ruleset drift

Release PRs also depend on the validate-base workflow's release-readiness steps. That workflow queries the latest completed release-canary.yml run, reads the canary summary artifact, and fails the release PR when the canary is missing, red, older than 48 hours, or when the tested develop SHA recorded in the artifact is not an ancestor of the release PR head. The only bootstrap exception is the first release that introduces release-canary.yml before the workflow exists on the default branch; in that case validate-base runs the same non-fast canary suite and ruleset drift check inline, then later releases return to the scheduled/manual canary evidence path.

release-canary.yml runs daily and on manual dispatch. Scheduled runs execute from the default branch, but the workflow checks out develop before running release evidence and records that checked SHA in release-canary-summary.json. Its blocking canary suite is the credential-free non-fast family: (slow or resource_heavy) and not (stress or live_integration). The same workflow also runs scripts/ruleset_drift_check.py against docs/operations/repo-admin-settings.md, so ruleset drift makes the canary red instead of silently invalidating release assumptions. The ruleset drift check must use RULESET_DRIFT_TOKEN, a repository secret with enough ruleset visibility to expose bypass actors; the default GITHUB_TOKEN is insufficient for that part of the contract.

Stress tests and live cloud integrations remain advisory until their credential, cost, and flake policies are stable enough to make them release-blocking. Long-running UAT is not advisory: see the UAT release-gate evidence requirement below.

UAT release-gate evidence (required)

scripts/release_readiness_check.py also requires committed UAT evidence: _project/release-evidence/uat-gate-summary.json must have a green verdict, source_dirty: false, a source_commit_sha that is an ancestor of the release PR head, and be at most 21 days old (vs the canary's 48h: a full 3-stage sweep costs an operator-day and releases are cut every few weeks, so a 48h window would force redundant sweeps). Missing, red, stale, non-ancestor, or dirty evidence fails validate-base on the release PR. Because release trees curate _project/ away, CI reads the evidence from the fetched origin/develop ref. The gate activates on the first release whose base branch already carries this script version — i.e. the release after the one that ships it; commit evidence to develop before that second release.

Producing the evidence before cutting a release:

make uat-sweep CONFIG=tests/uat/configs/release-gate-01-native-dataframe.yaml
make uat-sweep CONFIG=tests/uat/configs/release-gate-02-docker-nonoltp.yaml   # after stage 1 completes
make uat-sweep CONFIG=tests/uat/configs/release-gate-03-docker-oltp.yaml     # after stage 2 completes
make uat-gate-check STAGE1=<run-dir> STAGE2=<run-dir> STAGE3=<run-dir>
# review, then commit _project/release-evidence/uat-gate-summary.json to develop

See docs/operations/uat-framework.md "Release-gate re-run" for stage ordering and the mechanized APPROVE/HOLD checklist. The same emergency override below covers this check.

Emergency override is admin-only: set repository variables RELEASE_READINESS_OVERRIDE_SHA to the exact release PR head SHA and RELEASE_READINESS_OVERRIDE_REASON to the incident/approval record. Remove both variables after the release. API outages, stale canaries, or ruleset drift must not be bypassed with an undocumented local change.

What release-cut does

  1. Cuts a vX.Y.Z branch off develop (develop itself is never modified).
  2. Bumps the 6 version sources via scripts/update_version.py and generates the CHANGELOG entry via scripts/generate_changelog_entry.py --since-ref origin/release. The release note boundary is the current release branch patch delta against release, not git log origin/release..HEAD ancestry and not the latest tag reachable from develop, because release-finalize intentionally does not replay release commits onto develop.
  3. Opens $EDITOR on CHANGELOG.md for hand-curation when a terminal is attached, then gates on generate_changelog_entry.py --check-curation, which inspects the drafted section's text: it rejects verbatim commit subjects (trailing (#NNNN)), the manual-edit placeholder, and more than 60 bullets. Curation is always required — the draft is the raw origin/release..HEAD delta, hundreds of commits whenever release lags develop. Headless runs stop here: hand-curate the section, then re-run make release-cut (see "Resuming or aborting a cut" below). RELEASE_ALLOW_RAW_CHANGELOG=1 accepts the raw draft deliberately.
  4. Curates the release branch — git rm's the dev-only and deferred release paths (_project/, _blog/, results explorer/data, agent configs, dev-tooling root files; full list in the release-cut: Makefile target and gated by scripts/check_release_curation.py). For v0.3.0, landing/ and docs/blog/ stay in the release tree so /prompts/ and promoted release posts ship; results-explorer/, results-data/, and explorer/results-data workflows do not.
  5. Commits a single Release vX.Y.Z commit on vX.Y.Z.
  6. Merges origin/release into vX.Y.Z with -s ours --allow-unrelated-histories. release and develop have unrelated roots and diverge permanently, so without this the release PR is unmergeable, GitHub cannot compute a merge ref, and no CI runs at all. -s ours keeps the curated release tree byte-for-byte (the merge is guarded on that) and only records release as a second parent, which release-finalize's squash-merge then collapses away.
  7. Pushes and opens a PR against release.
  8. Sweeps prior v* branches on origin (option-c lifecycle: keep until superseded, then auto-delete on the next release-cut).

Resuming or aborting a cut

Steps 1-3 are idempotent, so an interrupted cut is resumed by re-running the same command from the vX.Y.Z branch:

make release-cut VERSION=X.Y.Z      # reuses the branch, keeps the CHANGELOG section

The branch is reused rather than recreated, the version bump and uv lock re-apply to the same values, and an existing ## [X.Y.Z] section is left untouched — so a section you curated between runs survives. To throw the cut away instead:

make release-cut-abort VERSION=X.Y.Z   # reset, return to develop, delete the branch

release-cut-abort refuses once vX.Y.Z exists on origin, and refuses to run from any branch other than vX.Y.Z or develop. release-cut likewise refuses to resume a branch that already carries its Release vX.Y.Z commit.

Changelog summarization shells out to the claude CLI. It is skipped automatically inside a Claude Code session (where the nested call blocks until its 120s timeout) and whenever BENCHBOX_CHANGELOG_SUMMARIZE=0; set BENCHBOX_CHANGELOG_SUMMARIZE=1 to force it. Skipping only means the draft is raw commit subjects, which step 3 requires you to curate anyway.

What release-finalize does

  1. Finds the open release PR for vX.Y.Z.
  2. Checks the required PR status list once and refuses to continue unless both validate-base and release-required-result are present and green. Missing means the ruleset/workflow contract is broken; pending means wait in GitHub Actions and rerun the command. release-finalize does not poll.
  3. Squash-merges the PR. (Ruleset release-only also blocks the merge unless validate-base and release-required-result are green.)
  4. Fast-forwards release and tags vX.Y.Z.
  5. Pushes the tag — which fires .github/workflows/release.yml: dependency-bounds → build (with SOURCE_DATE_EPOCH from the tag commit) → publish (PyPI trusted publisher) → github-release → test-installation (cross-platform pip install verification).
  6. Leaves develop untouched. Dev-only paths persist on develop by design (per A3 in _project/decisions/single-repo-migration.md); the release squash on release does not need to be replayed onto develop.

Push-to-release jobs are post-merge signals. They may still start when release advances, but they are not pre-publish evidence: the tag push follows the successful release PR merge and .github/workflows/release.yml begins from that public tag. If a post-merge release check fails after the tag is pushed, handle it as a patch release or incident; do not treat the already-published release as if it had been blocked.

Recovering from common failures

  • validate-base or release-required-result is missing: stop. The release-only ruleset or release workflow contract is out of sync; do not finalize until both stable required contexts exist.
  • validate-base or release-required-result is pending or failed: wait for GitHub Actions or fix on a feature branch off develop, PR back to develop, then re-run make release-cut (the option-c sweep will delete the stale vX.Y.Z branch automatically).
  • UAT gate evidence is missing, stale, red, dirty, or non-ancestor: run the 3-stage release-gate sweep, make uat-gate-check, and commit the evidence file to develop (see "UAT release-gate evidence"). Use the emergency override variables only with an explicit incident/approval record.
  • Release canary is missing, stale, or red: inspect the latest release-canary.yml run. If the non-fast canary failed, fix through develop; if ruleset drift failed, update the live GitHub ruleset or this runbook so they match. Use the emergency override variables only with an explicit incident/approval record.
  • Wheel content is wrong: adjust pyproject.toml / MANIFEST.in excludes on develop, then cut a patch release. PyPI rejects re-uploads of an existing version, so always bump.
  • release.yml fails after tag push: investigate via gh run view, fix the underlying issue, bump to the next patch version, and re-cut. Do not force-push or re-tag; the tag is already public.

Recovering from a broken PyPI release

PyPI versions are immutable: once X.Y.Z is published you cannot re-upload or delete-and-replace it, even to fix a broken build. This is generalized guidance — _project/TODO/main/active/release-recovery-v0-3-1.yaml is the worked precedent (see below) for what happens when this bites in practice: v0.3.0 published 2026-05-16 with a broken clean import (ModuleNotFoundError: No module named 'pandas') and sat as PyPI-latest, uncaught, for over a month, because nightly's package-install test built a wheel from develop rather than checking the artifact actually on PyPI. (.github/workflows/release-canary.yml's pypi-latest-installability job now closes that specific detection gap going forward — see "Release canary and ruleset drift" above.)

Two response options once a bad version is confirmed on PyPI, and when to use each:

  • Yank the broken version. PyPI's "yank" (pip index / PyPI project UI) hides a release from default dependency resolution without deleting it — existing installs that already pinned the bad version are unaffected, but new pip install benchbox resolutions skip it and fall back to the latest non-yanked version. Use this when there is a safe prior version to fall back to and the fix is not ready yet.
  • Ship a .postN or the next patch. This repo's actual precedent is a full next patch release (v0.3.0 -> v0.3.1), not a .postN suffix, per release-recovery-v0-3-1. Use this when the fix is ready quickly and a monotonic forward release is cleaner than yanking (e.g. there is no good prior fallback version, or you want users who already resolved the bad version to get the fix on their next install/upgrade rather than silently falling back).

Either path is a judgment call based on the incident (is there a safe fallback version, is a fix already close) — it is not automated by the canary itself; a canary failure is a signal to make that call, not a trigger for an automatic rollback action.

The forward-fix path uses the same flow as any other release — there is no separate recovery procedure:

git checkout develop && git pull
make release-cut VERSION=X.Y.Z
# review the PR; wait for validate-base and release-required-result
make release-finalize VERSION=X.Y.Z

See release-recovery-v0-3-1 for the worked example of diagnosing a broken PyPI-latest release, confirming the fix on develop, and cutting the recovery version through this same flow.

Reference

  • Makefile targets: release-cut, release-finalize.
  • Workflow: .github/workflows/release.yml.
  • Canary workflow: .github/workflows/release-canary.yml.
  • Release-readiness gate: scripts/release_readiness_check.py.
  • Ruleset drift gate: scripts/ruleset_drift_check.py.
  • Curation drift guard: scripts/check_release_curation.py (runs in lint.yml on every PR).
  • Version updater: scripts/update_version.py.
  • Changelog generator: scripts/generate_changelog_entry.py.
  • Architecture record: _project/decisions/single-repo-migration.md (D5 / A3 / A4 / A5, plus the amendments for the 2-command flow, the develop-not-modified rule, and the v0.3.0 release-scope curation).