Skip to content

orchestrate: remote slice branch survives merge when local delete fails (worktree checked out) — remote cruft accumulates every run #284

Closed
Closed
orchestrate: remote slice branch survives merge when local delete fails (worktree checked out) — remote cruft accumulates every run#284
Labels
bugSomething isn't workinggrill-primedPrimed with an autonomous pre-grill triage

Description

@rodrigorjsf
rodrigorjsf
opened on Jun 4, 2026

Symptom

During /orchestrate 269 (runId prd269-20260604-105556), every slice merge printed:

failed to delete local branch orchestrate/slice-<N>: ... used by worktree at ...

That local-delete warning is expected and documented — the slice worktree still has the branch checked out, and the skill performs the authoritative local reclamation via the explicit git branch -D at step 9 (SKILL.md lines 953–954, 1017–1021).

The unexpected part: the remote slice branches orchestrate/slice-271..275 also survived on the remote and had to be deleted manually with git push origin --delete. Confirmed after the run via:

git ls-remote --heads origin 'orchestrate/slice-*'

Evidence this is systemic, not a one-off

The same ground-truth query today still shows stale slice branches from a prior run (PRD #241):

refs/heads/orchestrate/slice-144
refs/heads/orchestrate/slice-145
refs/heads/orchestrate/slice-146
refs/heads/orchestrate/slice-147
refs/heads/orchestrate/slice-148
refs/heads/orchestrate/slice-149
refs/heads/orchestrate/slice-150
refs/heads/orchestrate/slice-151
refs/heads/orchestrate/slice-152
refs/heads/orchestrate/slice-self-containment

So merged slice branches accumulate on the remote across runs — not a single-run anomaly.

Root-cause hypothesis

SKILL.md §8 reclaims a merged slice branch via gh pr merge <N> --squash --delete-branch and claims (lines 949–954):

The --delete-branch flag reclaims the remote slice branch as part of the merge; it may additionally warn or no-op on the local branch because the slice worktree still has it checked out — that warning is tolerated, not a slice failure.

And §9 restates (lines 1029–1031):

This completes incremental reclamation: the remote half was done by --delete-branch at step 8, the local half here.

The hypothesis: when gh pr merge --delete-branch's local-branch deletion fails (the worktree still has the branch checked out — which is always the case here, since the orchestrator removes the worktree only later at step 9), gh does not (or does not reliably) proceed to delete the remote head branch either. The result is that the "remote half done by --delete-branch" claim does not hold whenever a worktree is checked out — i.e. on every slice.

Impact

Remote-branch cruft accumulates on origin every run. The step-8 / step-9 SKILL.md guarantee ("remote half done by --delete-branch") is false in the normal path, so the documented incremental-reclamation invariant silently leaks remote branches.

Load-bearing references

  • plugins/orchestrate/skills/orchestrate/SKILL.md:949-954 — the "--delete-branch reclaims the remote slice branch" claim.
  • plugins/orchestrate/skills/orchestrate/SKILL.md:1017-1021 — the explicit git branch -D local reclamation at step 9.
  • plugins/orchestrate/skills/orchestrate/SKILL.md:1029-1031 — the "remote half was done by --delete-branch at step 8" claim.

Suggested fix (any one of these)

  1. Backstop in §9 (preferred — smallest, idempotent): after remove_worktree + git branch -D, add an explicit remote-delete backstop: 
    git push origin --delete orchestrate/slice-<N>
    This is idempotent — an already-absent remote branch is success (exit 0 or a tolerated "remote ref does not exist"), so it is safe on resumed runs where --delete-branch did succeed.
  2. Orchestrator verify-and-delete: after merge, have the orchestrator git ls-remote --heads origin orchestrate/slice-<N> and delete the remote head if it survived.
  3. Cleanup sweep extension: extend the /orchestrate clean sweep to prune merged remote slice branches.

If the backstop is added, the §8/§9 prose should be corrected: the "remote half is done by --delete-branch" claim only holds when no worktree has the branch checked out, which is never true in the normal path — so the remote reclamation must move to (or be backstopped at) step 9.

Lineage

Closed #232 ("orchestrate: cleanup enforcement — incremental slice-branch reclamation + failed-worktree retention/reclaim") introduced the incremental-reclamation design and the now-falsified "remote half done by --delete-branch" assumption. This issue is the field regression of that assumption — not a duplicate (#232 is closed and describes the design, not this failure mode).

Run: prd269-20260604-105556 · plugin: orchestrate

Activity

  1. rodrigorjsf
    added
    bugSomething isn't working
    on Jun 4, 2026

  2. rodrigorjsf commented on Jun 4, 2026

    @rodrigorjsf
    rodrigorjsf
    on Jun 4, 2026
    OwnerAuthor

    Autonomous pre-grill triage — AI-generated insights to seed a future grilling session. Not a decision.

    Evidence

    • The leak is real and empirically grounded. Running the issue's own ground-truth query today still returns stale remote slice branches from a prior run: git -C <repo> ls-remote --heads origin 'orchestrate/slice-*' returns orchestrate/slice-144 through slice-152 plus orchestrate/slice-self-containment (10 refs, the exact PRD PRD: orchestrate plugin hardening v2 — resilience, verification depth, cross-run isolation, issue lifecycle #241 set the issue cites). This is not a hypothesis — the branches are on origin right now. The "systemic accumulation across runs" claim holds.
    • The cited line numbers are stale. The issue cites SKILL.md:949-954, :1017-1021, :1029-1031, but plugins/orchestrate/skills/orchestrate/SKILL.md is now 428 lines total. The PRD PRD: orchestrate MCP-first decomposition — cut orchestrator at-load context to a judgment spine #269 / orchestrate: relocate procedural prose to phase-loaded references; shrink SKILL.md to the judgment spine #275 "procedural-prose relocation" (referenced in SKILL.md:19) moved this prose out of the spine into references/slice-pipeline.md. The exact prose the issue quotes now lives at:
      • references/slice-pipeline.md:273-278 — "The --delete-branch flag reclaims the remote slice branch as part of the merge; it may additionally warn or no-op on the local branch because the slice worktree still has it checked out."
      • references/slice-pipeline.md:340-365 — step 9, the git branch -D local reclaim and the "the remote half was done by --delete-branch at step 8" claim (:363).
        So the issue's diagnosis is accurate; only the file:line pointers are dead. An implementer following the "load-bearing references" verbatim hits non-existent lines.
    • finalize_slice post-merge is local-only — confirmed in source. finalize-slice.ts:524 (finalizePostMerge) contains no git push, no --delete, no remote op (verified by scanning the function body). Its remote param is explicitly documented as "Unused in 'post-merge' (local reclaim only)" at finalize-slice.ts:97. So per-slice, the remote reclaim depends entirely on gh pr merge --delete-branch — exactly as the issue states.
    • The cleanup sweep ALREADY deletes remote branches — but is gated. clean-runs.ts:390-408 (deleteBranch) calls both deleteLocalBranch and deleteRemoteBranch (clean-runs.ts:353-384, which runs git push origin --delete -- <branch>, treating an already-absent ref as success). removeRunFootprint invokes it on every non-preserved slice branch (clean-runs.ts:519-524). So the capability the issue's fix feat: convert agent files to universal reference documents for standalone skills #3 asks for is already built. The gap is when it fires, not that it can't.
    • Why the backstop rarely fires (a complementary root cause the issue does not name). clean_runs is status-gated: it acts only on a run whose final-PR verdict is merged AND whose run-state.json status === "completed" with a non-null finalPullRequest (clean-runs.ts:27-34, reasons run-not-completed / final-pr-open / no-verdict-from-orchestrator at :110-115). But the final integration PR is deliberately left unmerged for a human to review (wave-loop.md:185 "left unmerged"; the run sets status: "completed" at PR-open time, wave-loop.md:209-210). So in the window between final-PR-open and the human merging it, the sweep verdict is open → skipped → the per-slice --delete-branch is the only thing that could have reclaimed the remote. If it didn't (the issue's observation), nothing else does until a later run's start-of-run sweep sees a merged verdict — and only if that run's directory still exists to be swept.

    Ambiguities

    • The root-cause mechanism is asserted, not proven. The issue observes that remote branches survived (solid), but hypothesizes the mechanism: "when --delete-branch's local-branch deletion fails (worktree checked out), gh does not reliably proceed to delete the remote head." That causal chain is unverified. Public cli/cli issues are inconclusive on the ordering: #13380 reports the API merge succeeds first, then the local git checkout fails (implying remote delete may already be done), and #7897 ("leaves dangling remotes") turns out to mean stale local remote-tracking refs refs/remotes/origin/..., with the actual remote branch deleted ("✓ Deleted remote branch"). Whether gh's remote-delete API call happens before or after the local checkout/branch-delete that fails in a worktree is the decisive unknown — and it determines whether the issue's named mechanism is the cause or just one candidate.
    • "On every slice" vs. observed. The issue asserts the local-delete failure is "always the case here" and therefore the remote leak is too. But the empirical leak set for run 269 was slice-271..275 (all of them), while the impact claim leans on the general "always." If the mechanism is order-dependent in gh, the leak might be all-or-nothing per gh version, not per-slice — different framing for the fix's idempotency guarantee.
    • Which sweep was expected to catch PRD: orchestrate plugin hardening v2 — resilience, verification depth, cross-run isolation, issue lifecycle #241. The issue treats the surviving PRD: orchestrate plugin hardening v2 — resilience, verification depth, cross-run isolation, issue lifecycle #241 branches as proof the per-slice step failed. But they could equally have survived because PRD: orchestrate plugin hardening v2 — resilience, verification depth, cross-run isolation, issue lifecycle #241's final umbrella PR was never merged (sweep never saw merged) and/or its run directory was cleaned before any later merged-verdict sweep ran. The issue does not disambiguate these.

    Honest doubts — impact & real value delivery

    • The case against doing anything urgent: the symptom is cosmetic cruft on a remote namespace, not a correctness or data-loss bug. Stale orchestrate/slice-* heads on origin cost nothing at runtime, never collide (run-scoped slice IDs), and are trivially swept with one git push origin --delete or by /orchestrate clean once the final PR merges. No build, no run, no merge is blocked by them. A maintainer could legitimately rank this below any functional bug.
    • The strongest pro-value counter: the issue's real payload is not the cruft — it is that a documented invariant is false. slice-pipeline.md:363 flatly claims "the remote half was done by --delete-branch at step 8." If that is false in the normal path (worktree always checked out), the skill's prose actively misleads every future reader and every agent resuming a run. False documentation in a judgment-spine skill is worse than a missing line: it teaches the wrong model. That is the durable value of the fix — correctness of the contract, not of the namespace.
    • Who is NOT asking for this: no end user of the plugin filed this; it is dogfood self-observation from a run. There is no external pressure, no SLA. That lowers urgency but does not lower the doc-correctness stakes.
    • Risk of over-fixing: adding a per-slice git push origin --delete backstop runs a network op on every merged slice for a problem that the existing clean_runs sweep would also solve if the gating window were closed. Two overlapping remote-delete paths is itself a small maintenance smell.

    Alternative approaches

    1. Per-slice backstop inside finalize_slice post-merge (the issue's fix feat: Phase 1 - Create reference content for agent initializer skills #1, sharpened). Activate the already-present-but-dormant remote param (finalize-slice.ts:91-98) so finalizePostMerge runs git push <remote> --delete <branch> after the worktree removal and local branch -D, treating an absent remote ref as success (the isAbsentRefError helper at finalize-slice.ts already exists for exactly this). Trade-off: this is a pure git op, so it honors the finalize_slice "git-only, never shells gh" invariant (slice-pipeline.md:367-368) and sidesteps the entire gh --delete-branch ordering uncertainty — it does not matter whether gh deleted the remote or not, because this is idempotent. Costs one extra network round-trip per merged slice and a doc correction. Strongest candidate precisely because it does not depend on resolving the unknown mechanism.
    2. Close the sweep-gating window instead of touching the per-slice path. Leave --delete-branch as-is; make the backstop reliable. Either (a) have the orchestrator run the start-of-run sweep against runs whose final PR is now merged even if their dir was already cleaned (track a "pending remote-reclaim" list), or (b) extend clean_runs to prune any orchestrate/slice-* remote head with no open PR regardless of run status. Trade-off: centralizes reclaim in one place (no double-delete), and fixes the real reason cross-run accumulation happens (the unmerged-final-PR gate), but it reclaims late (next sweep), not incrementally — so the documented "incremental reclamation" invariant still needs its prose softened to "incremental local + swept remote."
    3. Orchestrator verify-and-delete after each merge (the issue's fix feat: Add asset templates for skills (Phase 2) #2). After gh pr merge, the orchestrator itself runs git ls-remote --heads origin orchestrate/slice-<N> and deletes the head if it survived. Trade-off: keeps the remote op in the spine (where all gh/remote ops already live per the single-writer invariant) rather than pushing a remote op into the "git-only" MCP tool — arguably the cleaner ownership boundary — but adds a verify+delete to the spine's hot path and duplicates logic that clean_runs already encapsulates. More prose in an already over-cap-exception spine.

    (These are genuinely distinct: #1 changes the MCP tool and is mechanism-agnostic; #2 changes the spine/orchestrator and keeps remote ops out of the tool; #3 changes when the existing sweep fires and touches neither hot path. They locate the fix in three different layers.)

    Gotchas

    • finalize_slice is "git-only, never shells gh" (slice-pipeline.md:367-368). Fix feat: Phase 1 - Create reference content for agent initializer skills #1 fits ONLY because git push --delete is git, not gh — do not let an implementer reach for gh pr ... inside the tool. But note the boundary tension: the orchestrator currently owns all remote/forge ops (single-writer invariant, SKILL.md:365-373), and pushing a remote-delete into the MCP tool slightly blurs that. Alternative feat: Add asset templates for skills (Phase 2) #2/feat: convert agent files to universal reference documents for standalone skills #3 keep the boundary cleaner. This is a real design-ownership decision, not a detail.
    • orchestrate-mcp/dist/ is committed. Any src/ change to finalize-slice.ts or clean-runs.ts requires npm run build — vitest tests run against src, so green tests do NOT mean the shipped bundle changed (project memory: orchestrate-mcp dist/ is committed). A fix that passes tests but skips the rebuild ships nothing.
    • Stale local remote-tracking refs are a separate problem. Even a perfect remote delete leaves refs/remotes/origin/orchestrate/slice-* dangling locally (cli/cli #7897). If the fix's success check uses git branch -a or a cached remote list rather than a fresh git ls-remote, it can false-positive "branch still there." Verify against the live remote, never the local cache.
    • Idempotency must tolerate the already-deleted case both ways. On a resumed run at subState: merged, or when --delete-branch did succeed, the backstop must treat "remote ref does not exist" as success. The isAbsentRefError matcher already covers several phrasings (finalize-slice.ts, clean-runs.ts:341) — reuse it, do not hand-roll.
    • Doc-in-sync is a P0 acceptance criterion here. The fix is primarily a doc-correctness fix. slice-pipeline.md:273-278 and :363 both assert the remote half is done at step 8; any code fix that does NOT also correct that prose leaves the invariant still-false and fails the project's docs-in-sync gate. The issue itself calls this out ("the §8/§9 prose should be corrected").
    • Reference-file line cap. slice-pipeline.md is 24 KB / ~430 lines already; .claude/rules/reference-files.md caps reference files at 200 lines (with a ## Contents ToC over 100). Adding prose to it may trip that rule — check before expanding it. (The SKILL.md spine carries a documented over-cap exception; reference files do not.)

    Points to clarify

    Other analytical lenses

    • Cost-of-delay: near zero. The cruft is harmless and self-healing once a later merged-verdict sweep runs. Nothing degrades while this sits open. This is a "fix when touching the area" item, not a drop-everything.
    • Reversibility: high. All three fixes are small, local, and idempotent; any can be reverted with no migration. Low-stakes to get wrong on the first try.
    • Second-order effect — the doc is the real artifact. Because this is a judgment-spine skill consumed by agents, a false invariant propagates into every future orchestration run's mental model. The second-order cost of leaving slice-pipeline.md:363 wrong dwarfs the first-order cost of remote cruft. Frame value around the contract, not the namespace.
    • Lineage / regression framing. The issue correctly notes it is the field regression of closed orchestrate: cleanup enforcement — incremental slice-branch reclamation + failed-worktree retention/reclaim #232's design assumption, not a duplicate. Worth confirming orchestrate: cleanup enforcement — incremental slice-branch reclamation + failed-worktree retention/reclaim #232's acceptance criteria did not already assert "remote reclaimed incrementally" as tested — if it did, there is a missing test, and the fix should add a regression test (route via /tdd per project convention) that asserts no orchestrate/slice-* survives on the remote after a merged slice.
    • Double-delete smell. If fix feat: Phase 1 - Create reference content for agent initializer skills #1 lands AND clean_runs keeps its remote-delete, two paths now delete the same remote branch. Harmless (idempotent) but worth a one-line comment so a future reader does not "consolidate" one away and reopen the leak.

    Suggested grilling agenda

    1. Resolve the mechanism, or design around it: does gh pr merge --delete-branch delete the remote before the worktree-induced local failure aborts the command? If this can't be pinned quickly, the grilling should conclude that the fix must be mechanism-agnostic (fix feat: Phase 1 - Create reference content for agent initializer skills #1's idempotent git push --delete), which collapses the most uncertainty in one move. Lead here.
    2. MCP-tool vs. spine ownership of the remote-delete — does putting git push --delete in finalize_slice violate the single-writer-of-remote-state invariant, or is "git-only" the right line? This decides which of the three alternatives you build.
    3. Incremental vs. eventual reclaim — is transient cruft during the human-review window (final PR unmerged) acceptable by design, making the real fix "close the sweep-gating window" (feat: Add asset templates for skills (Phase 2) #2 alt-b) rather than a per-slice backstop?
    4. Did PRD: orchestrate plugin hardening v2 — resilience, verification depth, cross-run isolation, issue lifecycle #241's final PR merge? — the one cheap query that tells you whether you are fixing the per-slice path, the sweep gate, or both.
    5. Scope of the doc fix — agree up front that correcting slice-pipeline.md:273-278 and :363 is in-scope and P0, and confirm the fix adds a /tdd regression assertion that no orchestrate/slice-* remote head survives a merged slice.
  3. rodrigorjsf
    added
    grill-primedPrimed with an autonomous pre-grill triage
    on Jun 4, 2026

  4. rodrigorjsf commented on Jun 4, 2026

    @rodrigorjsf
    rodrigorjsf
    on Jun 4, 2026
    OwnerAuthor

    Recurrence on a second repo + second PRD — and the tolerated exit 1 is masking this leak, not a cosmetic warning

    Confirming this is systemic across repos, not just across runs. Independent run prd1-20260604-162601 (orchestrate v1.4.0) on a different repo (no-bash-mcp) and a different PRD (PRD-001, 9 slices, issues #2–#10) reproduces the exact leak #284 reports for the toolkit repo's prd269.

    Ground truth — merged PR, remote branch survives

    All nine slice branches survived on the remote after the run, with the run's worktrees and local branches already cleaned (step-9 local reclaim worked; the remote half did not):

    $ git ls-remote --heads origin 'orchestrate/slice-*'
8aa0f4e…  refs/heads/orchestrate/slice-2
fdef06b…  refs/heads/orchestrate/slice-3
838748d…  refs/heads/orchestrate/slice-4
69a881a…  refs/heads/orchestrate/slice-5
17a18bb…  refs/heads/orchestrate/slice-6
2320e2c…  refs/heads/orchestrate/slice-7
e5d9f9a…  refs/heads/orchestrate/slice-8
31542f8…  refs/heads/orchestrate/slice-9
11ed5e1…  refs/heads/orchestrate/slice-10

$ git branch --list 'orchestrate/slice-*'      # empty — local already reclaimed
$ git worktree list                            # only the main checkout remains

    Pre-empting the "were those just unmerged PRs?" counter — the slice PR for orchestrate/slice-2 is MERGED, yet its remote head is still present:

    $ gh pr view 11 --json number,state,mergedAt,headRefName
{"number":11,"state":"MERGED","mergedAt":"2026-06-04T18:01:24Z","headRefName":"orchestrate/slice-2"}

    So: merged squash, remote branch survives. Same failure mode, second repo, second PRD.

    The exit code is the foot-gun, and the toleration is masking this leak

    A complementary angle on the same bug. On every one of the 9 slices, gh pr merge <pr> --squash --delete-branch exited 1 with:

    failed to delete local branch orchestrate/slice-<N>: failed to run git: error: cannot delete branch 'orchestrate/slice-<N>' used by worktree at '<path>'

    The squash-merge itself succeeded each time (verified via gh pr view <pr> --json state,mergedAt → MERGED). The skill correctly documents this exit 1 as "tolerated, not a slice failure" (the worktree still has the branch checked out). But two things compound here:

    1. The exit code cannot distinguish a tolerated local-delete warning from a real merge failure. A future automation (or a stricter spine) that trusts the exit code would read a successful merge as failed. The orchestrator only avoids this today by re-querying gh pr view after every merge.
    2. More importantly, the toleration is masking this very leak. The exit 1 is reported as "just the local-branch warning," but the same --delete-branch invocation is also the only remote-delete attempt (per this issue's own source analysis: finalize-slice.ts:524 is local-only). When gh aborts on the worktree-induced local failure, the remote head is left behind — and the "tolerated warning" framing hides that the remote half silently failed too. The exit 1 is not cosmetic; it is the observable signal of the leak this issue is about.

    Correction to a tempting "fix" — do NOT just drop --delete-branch

    A natural-looking fix is "drop --delete-branch; the merge already deletes the remote branch on GitHub's side." This run's evidence falsifies that: the remote branches above prove the remote is not auto-deleted. Since --delete-branch is the only remote-delete attempt, removing it as-written would turn the leak from intermittent into guaranteed.

    The two fixes must be paired:

    • (a) Fix the exit-code foot-gun by removing --delete-branch from the per-slice gh pr merge (so a successful merge returns exit 0 and the exit code regains its meaning), and
    • (b) move the remote reclaim to a step-9 backstop — exactly this issue's fix feat: Phase 1 - Create reference content for agent initializer skills #1 (git push origin --delete orchestrate/slice-<N>, idempotent, treating an absent ref as success). With (b) in place, dropping --delete-branch in (a) is safe; without it, do not drop it.

    Net: this run is recurrence evidence for the leak, and it sharpens the issue's framing — the per-slice exit 1 is not a cosmetic local-delete warning to be tolerated, it is the surfacing of the remote-reclaim failure, and the exit-code-meaning loss is a second cost of the same root cause.

    Run prd1-20260604-162601 · orchestrate v1.4.0 · repo no-bash-mcp · final umbrella PR #20 (OPEN, left for human review as designed).

  5. rodrigorjsf
    closed this as completedon Jun 5, 2026

  6. rodrigorjsf commented on Aug 3, 2026

    @rodrigorjsf
    rodrigorjsf
    on Aug 3, 2026
    OwnerAuthor

    Still reproducing after this issue was closed — tracked forward in #384

    Recurrence during orchestrate run prd352-20260803-015333 (PRD #352, orchestrate 1.6.0 + the PRD #352 umbrella). On slice #353, gh pr merge 366 --squash --delete-branch exited 1 with the familiar worktree-checked-out local-delete error, the squash landed (MERGED), and git ls-remote --heads origin orchestrate/slice-353 still returned the branch.

    This issue was closed on 2026-06-05 with no linked fix commit, and the state it diagnosed is unchanged today:

    • plugins/orchestrate/skills/orchestrate/references/slice-pipeline.md:330-335 — the "--delete-branch reclaims the remote slice branch … warning is tolerated" prose is still there.
    • plugins/orchestrate/skills/orchestrate/references/slice-pipeline.md:420 — "the remote half was done by --delete-branch at step 8" is still asserted.
    • plugins/orchestrate/orchestrate-mcp/src/tools/finalize-slice.ts — finalizePostMerge still performs no remote operation: checkpoint subState: merged, remove the worktree, deleteLocalBranch(...), return. No git push --delete in the phase.

    Not reopening this one. Filed forward as #384 with the fresh evidence and this issue's original analysis carried over.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinggrill-primedPrimed with an autonomous pre-grill triage

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions