Skip to content

fix(release): let two prunes of the same build tags both succeed - #594

Merged
Vivswan merged 2 commits into
mainfrom
fix/release-prune-race
Nov 12, 2026
Merged

Vivswan merged 2 commits into
mainfrom
fix/release-prune-race

Conversation

@Vivswan

@Vivswan Vivswan commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

Before

list build tags -> one delete push -> refused -> job fails, latest not moved
$ bun .github/scripts/release-pipeline.ts package-commit
To /tmp/fixture/origin.git
 * [new tag]         eb47538eab0fa3cedaa248845683be0b1e41402d -> build/13.f23dfd7
remote: error: cannot lock ref 'refs/tags/build/2.9443df7': unable to resolve reference 'refs/tags/build/2.9443df7'
remote: error: cannot lock ref 'refs/tags/build/3.8228faf': unable to resolve reference 'refs/tags/build/3.8228faf'
To /tmp/fixture/origin.git
 ! [remote rejected] build/2.9443df7 (reference does not exist)
 ! [remote rejected] build/3.8228faf (reference does not exist)
error: failed to push some refs to '/tmp/fixture/origin.git'
release-pipeline package-commit: git push origin :refs/tags/build/3.8228faf :refs/tags/build/2.9443df7 failed: remote: error: cannot lock ref 'refs/tags/build/2.9443df7': unable to resolve reference 'refs/tags/build/2.9443df7'
remote: error: cannot lock ref 'refs/tags/build/3.8228faf': unable to resolve reference 'refs/tags/build/3.8228faf'
To /tmp/fixture/origin.git
 ! [remote rejected] build/2.9443df7 (reference does not exist)
 ! [remote rejected] build/3.8228faf (reference does not exist)
error: failed to push some refs to '/tmp/fixture/origin.git'
exit 1

the two tags still on origin: none
build tags on origin: 10
latest on origin: absent

After

list build tags -> one delete push -> refused? -> no:  done
                                               -> yes: read origin's build tags -> every named tag gone? -> yes: done
                                                                                                         -> no:  job fails with the refusal
$ bun .github/scripts/release-pipeline.ts package-commit
To /tmp/fixture/origin.git
 * [new tag]         eb47538eab0fa3cedaa248845683be0b1e41402d -> build/13.f23dfd7
remote: error: cannot lock ref 'refs/tags/build/2.9443df7': unable to resolve reference 'refs/tags/build/2.9443df7'
remote: error: cannot lock ref 'refs/tags/build/3.8228faf': unable to resolve reference 'refs/tags/build/3.8228faf'
To /tmp/fixture/origin.git
 ! [remote rejected] build/2.9443df7 (reference does not exist)
 ! [remote rejected] build/3.8228faf (reference does not exist)
error: failed to push some refs to '/tmp/fixture/origin.git'
To /tmp/fixture/origin.git
 * [new tag]         eb47538eab0fa3cedaa248845683be0b1e41402d -> latest
refs/tags/build/13.f23dfd7: created at eb47538eab0fa3cedaa248845683be0b1e41402d; pruned refs/tags/build/3.8228faf, refs/tags/build/2.9443df7; refs/tags/latest: moved to eb47538eab0fa3cedaa248845683be0b1e41402d
exit 0

the two tags still on origin: none
build tags on origin: 10
latest on origin: eb47538eab0fa3cedaa248845683be0b1e41402d refs/tags/latest

A rival run took the same two tags after this run's delete push read origin's refs.

Not covered: runs whose lists differ. If the server rejects the whole push, a tag only one run named stays and that job fails.

How

  • A refused delete push is judged by origin's refs, read back once, never by git's words.
  • Every named tag gone means done, and the run goes on to move latest.
  • A named tag still on origin fails the job with the refusal as git worded it.
  • A push git accepts is trusted as before, with no extra read.

Proof

  • Tests: 170 pass in the four release-pipeline test files, the race and sealed-environment cases seen failing first.
  • Controls: the leftover-tag case fails under two wrong forms of the check.
  • Gates: typecheck, lint, lint:arch, knip, check:compat, and build:check pass.
Technical details
  • The race in the fixture: origin's pre-receive hook deletes the tags, which is after the push read origin's refs and before origin locks them.
  • Fixture isolation: the hook directory sits in the fixture root and is set as origin's own core.hooksPath.
  • Sealed git environment: while a file's tests run, the fixture takes every GIT_* variable out of the environment and points global and system config at /dev/null, then puts the caller's variables back.
  • Seal reach: the fixture's own git calls and the pipeline under test both inherit the process environment, so one seal covers both.
  • Seal test: with a decoy GIT_DIR and GIT_CONFIG in the environment before the fixture seeds, both decoys stay byte-identical and origin holds its own config.
  • Identity: unchanged, since each clone already carries the fixture's identity in its own config.
  • Shared fixture: the three step suites that install the same fixture pass too, 205 tests across the seven files.
  • Leftover-tag case: the hook takes one tag and origin's update hook declines the other, so one rejection names a tag that is gone and one a tag that stays.
  • The two wrong forms: a check that throws only when every named tag is left, and one that never throws.
  • Git's reason is not pinned: the leftover-tag test asserts that git rejected the ref the rival took, and leaves the reason in parentheses to git.
  • Unchanged: a rival's deletion that lands before the push still draws git's warning and a clean exit.
  • Unchanged: the two replayed permanent refusals still fail the delete push with git's own words.
  • Step log: a run that passes this way still shows git's rejection lines above its report line, which release-pipeline.ts words and this change leaves alone.
  • Result field: the pruned doc comment now says a rival may have done the deleting.
  • Library: none applies, since the change is how the release script judges one push's outcome.
  • Capture command: REPO=<tree> CAPTURE_OUT=<file> bun test ./capture.test.ts, the script below, once on a copy of origin/main and once on this branch.
  • Capture identity and dates: the script pins one identity and one instant for every fixture commit inside the test, so both runs name the same tags on any machine.
  • Capture scenario: main at position 13, build tags on positions 2 to 12, the step run for position 13, and a rival that acts on the delete push alone.
  • Last three lines of each block: the capture script's own read of origin after the step.
  • Redaction: the fixture root is shown as /tmp/fixture, and trailing spaces are trimmed.
  • Files: .github/scripts/release-pipeline/{tags,git}.ts, test/scripts/release-pipeline-{build.test,fixture,fixture.test}.ts.
// capture.test.ts: the real package-commit step in a fixture where a rival deletes the same two build tags
// inside this run's delete push. REPO names the tree whose pipeline and fixture harness run.
import { setDefaultTimeout, test } from "bun:test";
import { mkdirSync, writeFileSync } from "node:fs";
import { join } from "node:path";

const REPO = process.env.REPO ?? "";
const fixture = await import(join(REPO, "test/scripts/release-pipeline-fixture.ts"));
const { seedFixture, git, write, commitAll, pushGreenCommit, subcommand, installReleasePipelineFixture, buildTags } =
  fixture;
setDefaultTimeout(120_000);
installReleasePipelineFixture();

test("capture", async () => {
  // One identity and one instant for every fixture commit, set once the fixture's hooks ran, so both trees name
  // the same tags on any machine.
  Object.assign(process.env, {
    GIT_AUTHOR_NAME: "fixture",
    GIT_AUTHOR_EMAIL: "fixture@example.invalid",
    GIT_AUTHOR_DATE: "2026-01-01T00:00:00Z",
    GIT_COMMITTER_NAME: "fixture",
    GIT_COMMITTER_EMAIL: "fixture@example.invalid",
    GIT_COMMITTER_DATE: "2026-01-01T00:00:00Z",
  });
  const fx = seedFixture();
  // main grows to position 12, and positions 2..12 each carry a build tag: eleven tags, one over the window.
  const shas: string[] = [fx.mergeSha];
  for (let n = 3; n <= 12; n++) {
    write(fx.work, "src/marker.ts", `export const marker = ${n};\n`);
    shas.push(commitAll(fx.work, `feat: commit ${n}`));
  }
  git(fx.work, "push", "--quiet", "origin", "HEAD:refs/heads/main");
  const tags = shas.map((sha, i) => `refs/tags/build/${i + 2}.${sha.slice(0, 7)}`);
  git(fx.work, "push", "--quiet", "origin", ...shas.map((sha, i) => `${sha}:${tags[i]}`));
  // The next push to main, as its post-green run checks it out.
  const head = pushGreenCommit(fx, "third-push", "packaged-bundle-bytes-13\n");
  // The rival run: it deletes the same two oldest tags after this run's delete push read origin's refs.
  const hooks = join(fx.root, "origin-hooks");
  mkdirSync(hooks);
  writeFileSync(
    join(hooks, "pre-receive"),
    [
      "#!/bin/sh",
      "deleting=",
      'while read -r old new ref; do case "$new" in *[!0]*) ;; *) deleting=1 ;; esac; done',
      `if [ -n "$deleting" ]; then git update-ref -d ${tags[0]} && git update-ref -d ${tags[1]}; fi`,
      "",
    ].join("\n"),
    { mode: 0o755 },
  );
  git(fx.origin, "config", "core.hooksPath", hooks);
  const run = await subcommand(head.dir, { GITHUB_SHA: head.sha, RUN_URL: undefined }, "package-commit");
  const redact = (text: string) => text.replaceAll(fx.root, "/tmp/fixture").replaceAll(REPO, "/repo");
  const left = buildTags(fx).filter((ref: string) => ref === tags[0] || ref === tags[1]);
  const latest = git(fx.work, "ls-remote", "origin", "refs/tags/latest");
  writeFileSync(
    process.env.CAPTURE_OUT ?? "",
    [
      "$ bun .github/scripts/release-pipeline.ts package-commit",
      redact(run.stderr.trimEnd()),
      `exit ${run.status}`,
      "",
      `the two tags still on origin: ${left.length === 0 ? "none" : left.join(", ")}`,
      `build tags on origin: ${buildTags(fx).length}`,
      `latest on origin: ${latest === "" ? "absent" : latest.replace("\t", " ")}`,
      "",
    ].join("\n"),
  );
});

Two post-green runs can name the same build tags for deletion. When the rival's deletion lands after this run's delete push has read origin's refs, the server refuses the deletion of a ref that is already gone, git push exits non-zero, and the job failed with its build tag minted and latest not moved.

A refused delete push is now judged by the prune's postcondition: origin's build tags are read back, and the prune is done when none of the tags it named is left. A refusal that leaves a named tag on origin is thrown as before, in git's words. A push git accepts is trusted without a read-back.

The fixture gains origin-side hooks, which play a rival landing inside a push, after it read origin's refs and before origin writes any: the gap a push plan cannot reach. One test has the rival take every named tag there and expects the prune to succeed; another has it take one while origin declines the other and expects the refusal.
Copilot AI balanced review requested due to automatic review settings November 12, 2026 01:57
@github-actions

Copy link
Copy Markdown
Contributor

File size check

0 over a hard cap (fails), 6 warning(s).

File Size Tier Cap
src/engine/diff.ts:56 11 comment lines warn 10
src/github/api-error.ts:9 11 comment lines warn 10
src/problem.ts:73 18 comment lines warn 10
src/sections/contract/errors.ts:13 11 comment lines warn 10
src/sections/contract/module.ts:55 12 comment lines warn 10
src/sections/shared/list/decl.ts:265 14 comment lines warn 10

Split the file, wrap the line, shorten or exempt the comment, or list the path in .file-size-allow.local with a # reason.

5 managed file(s) skipped; repo-platform owns them.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Member shape check

3 finding(s).

File Line Rule
test/scripts/release-pipeline-fixture.test.ts 25 one-line-comment
test/scripts/release-pipeline-fixture.ts 53 one-line-comment
test/sections/contract/validate-hook.test.ts 80 all-or-none

Add the blank lines, make the comment one line directly above the item, comment every field or none, or list the path in .member-shape-allow.local with a # reason.

4 generated file(s) skipped.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues blocking the concurrent tag-pruning change.

No open issues 🎉

What changed in this PR

This PR lets concurrent release runs successfully prune the same build tags, allowing the release pipeline to continue updating latest when a rival has already deleted every targeted tag.

Changes:

  • Re-read origin after a refused deletion push; succeed only when all targeted tags are gone, otherwise preserve the push error.
  • Add isolated origin-hook fixtures and regression tests for complete and partial rival deletions.
  • Clarify comments describing push outcomes and tags deleted by rival runs.
File Description
test/​scripts/​release-pipeline-fixture.ts Adds isolated origin hooks to simulate changes during a push.
test/​scripts/​release-pipeline-build.test.ts Tests successful concurrent pruning and failure when a targeted tag remains.
.github/​scripts/​release-pipeline/​tags.ts Checks remote refs after refused prune pushes and clarifies result comments.
.github/​scripts/​release-pipeline/​git.ts Updates the push helper's documentation to describe completed deletions.

Review effort: Balanced

…onment

Every git the suites run, the fixture's own calls and the pipeline under test, inherits the process environment. With GIT_DIR or GIT_CONFIG in it (git exports GIT_DIR to a hook), a fixture command such as the origin's config write landed in the file or repository those variables name: the repository the tests were launched from.

The fixture now takes every GIT_* variable out of the environment where it already puts its shim on PATH, points global and system config at /dev/null, and puts the caller's variables back when the file's tests end. Identity needs nothing new: each clone carries the fixture's in its own config, which an exported author or committer would outrank.

One test runs the fixture with a decoy GIT_DIR and GIT_CONFIG in the environment. Both decoys stay byte-identical, origin holds its own config, the pipeline's tag lands on origin, and the caller's variables are back once the fixture ended.
Copilot AI balanced review requested due to automatic review settings November 12, 2026 02:38

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the change allowing overlapping release runs to prune the same build tags.

No open issues 🎉

Review effort: Balanced

@Vivswan
Vivswan marked this pull request as ready for review November 12, 2026 03:02
@Vivswan Vivswan added the merge-when-green Owner approved: merge once every gate is green label Nov 12, 2026
@Vivswan
Vivswan merged commit 4a7d4c6 into main Nov 12, 2026
31 checks passed
@Vivswan
Vivswan deleted the fix/release-prune-race branch November 12, 2026 04:43
This was referenced Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-when-green Owner approved: merge once every gate is green

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants