chore(devex): route queue triage at Trunk's own reasons

.agents/skills/debugging-ci-failures/SKILL.md

@@ -144,7 +144,7 @@ overlap its own. The lane script over-reports targets on purpose, so in
 practice that is most of the queue. So:
 
 - The failing run is on that branch, never on the PR's head SHA. Take it from
-  the `Trunk Merge Queue` check run (`/merging-prs` step 4), not `gh pr checks`.
+  `trunk merge status <n>` (`/merging-prs` step 4), not `gh pr checks`. Trunk publishes no check run here.
   The branch is ephemeral; the run and its logs stay on GitHub, and the
   warehouse keeps its jobs under that `head_branch` (query 8 in the
   `investigating-ci-failures` references).

.agents/skills/merging-prs/SKILL.md

@@ -2,9 +2,9 @@
 name: merging-prs
 description: >
   Merge a PR into `master` through the Trunk merge queue and babysit it until it
-  lands. Enqueue with a `/trunk merge` comment, then watch the `Trunk Merge Queue
-  (master)` check run and the PR state until it is MERGED or the queue kicks it
-  out, reporting the Trunk bot's failure reason. Use when asked to merge a PR,
+  lands. Enqueue with a `/trunk merge` comment, then watch `trunk merge status`
+  and the PR state until it is MERGED or the queue kicks it out, reporting
+  Trunk's own reason for the terminal transition. Use when asked to merge a PR,
   "merge when ready", "land it", "ship it", to merge a whole stack (comment on
   the top PR — the queue merges it and every layer below atomically), to get a
   PR approved via the `stamphog` label, or to babysit/watch a PR through the
@@ -49,15 +49,18 @@ gh pr comment <n> --body "/trunk merge"
 For a stack, `<n>` is the highest layer you want merged — it and everything below it enqueue together (see the stack preflight bullet above).
 Append `--no-batch` to the comment to have the queue test the PR (or stack) alone instead of batched with other queued PRs.
 
-Within ~2 minutes, confirm Trunk picked it up — a check run whose name starts with `Trunk Merge Queue` should appear on the head commit:
+Within ~2 minutes, confirm Trunk picked it up:
 
 ```bash
-SHA=$(gh pr view <n> --json headRefOid -q .headRefOid)
-gh api --paginate "repos/$REPO/commits/$SHA/check-runs?per_page=100" \
-  --jq '.check_runs[] | select(.name | startswith("Trunk Merge Queue")) | {name, status, conclusion, details_url}'
+trunk merge status <n> 2>&1 | sed 's/\x1b\[[0-9;]*m//g' | tail -20
 ```
 
-Always paginate. A PR head SHA here carries 200–350 check runs, and an unpaginated call returns only the first 30 — the queue check is very unlikely to be in them, so you'd conclude Trunk never picked the PR up.
+The last transition should read `Pending` with "Pull request has been added to the merge queue".
+
+**Trunk publishes no check run in this repository**, so do not look for one on the head commit.
+A predicate like `select(.name | startswith("Trunk Merge Queue"))` matches nothing and reads as "Trunk never picked it up", whatever the queue is actually doing.
+`trunk merge status` is the source of truth, and it is the only place Trunk's reason for each transition appears.
+Without a `trunk login`, fall back to the `trunk-io[bot]` sticky comment on the PR, which carries the current state but no history.
 
 If nothing appears after a couple of minutes, check in this order:
 
@@ -75,7 +78,7 @@ If nothing appears after a couple of minutes, check in this order:
 
 ## 3. Watch until it lands
 
-Watch the **check run + PR state**, not `gh pr checks --watch`:
+Watch **Trunk's own state + the PR state**, not `gh pr checks --watch`:
 the queue runs CI on Trunk's own `trunk-merge/**` branch,
 so this PR's own checks don't reflect the queue's testing.
 
@@ -87,16 +90,14 @@ PR=<n>; REPO=$(gh repo view --json nameWithOwner -q .nameWithOwner); prev=""
 while true; do
   state=$(gh pr view "$PR" --json state -q .state 2>/dev/null || echo UNKNOWN)
   sha=$(gh pr view "$PR" --json headRefOid -q .headRefOid 2>/dev/null)
-  queue=$(gh api --paginate "repos/$REPO/commits/$sha/check-runs?per_page=100" \
-    --jq '[.check_runs[] | select(.name | startswith("Trunk Merge Queue"))]
-          | if length == 0 then empty
-            else (sort_by(.started_at) | last | "\(.status)/\(.conclusion // "-")") end' 2>/dev/null)
+  queue=$(trunk merge status "$PR" 2>/dev/null | sed 's/\x1b\[[0-9;]*m//g' \
+    | grep -oE '(Not Ready|Pending Failure|Pending|Testing|Tests Passed|Failed|Cancelled|Merged)' | tail -1)
   cur="pr=$state queue=${queue:-none}"
   [ "$cur" != "$prev" ] && echo "$cur"
   prev="$cur"
   case "$state" in MERGED|CLOSED) exit 0 ;; esac
-  # A kicked PR stays OPEN, so the failed queue check is the only terminal signal.
-  case "$queue" in completed/success) ;; completed/*) exit 0 ;; esac
+  # A kicked PR stays OPEN, so Trunk's own terminal state is the only signal.
+  case "$queue" in Failed|Cancelled) exit 0 ;; esac
   sleep 60
 done
 ```
@@ -112,21 +113,19 @@ Never block on a foreground `sleep`.
 
 ## 4. Handle failure
 
-If the check run completes with `conclusion == "failure"` (or the PR drops out of the queue),
-Trunk kicks the PR and reports the failing workflow.
+When `trunk merge status` reaches a terminal state of `Failed` or `Cancelled` (or the PR drops out of the queue),
+Trunk kicks the PR and reports why.
+The reason distinguishes actions that look alike: a conflict needs a rebase, a failed required check needs a fix or a requeue, and a cancellation was a person.
 `/triaging-merge-queue-failures` is the full decision chart for classifying the kick; the bullets below are the short form.
 
-**Read the check run, not the PR comments.** The check run is the authoritative source: only an app holding `checks:write` on the repo can write one, so it can't be forged. A PR comment can be posted by anyone with read access.
+**Read Trunk's own state, not the PR comments.** `trunk merge status` comes from Trunk over an authenticated session and carries a reason per transition. A PR comment can be posted by anyone with read access, and the `trunk-io[bot]` sticky is rewritten in place, so it holds the current state and no history.
 
 ```bash
-gh api --paginate "repos/$REPO/commits/$SHA/check-runs?per_page=100" \
-  --jq '[.check_runs[] | select(.name | startswith("Trunk Merge Queue"))]
-        | if length == 0 then empty
-          else (sort_by(.started_at) | last
-                | {conclusion, details_url, app: .app.slug,
-                   title: .output.title, summary: .output.summary, text: .output.text}) end'
+trunk merge status <n> 2>&1 | sed 's/\x1b\[[0-9;]*m//g'
 ```
 
+The reason on the terminal transition is what to report: a conflict with another PR or with `master` means testing never started and the PR needs a rebase, which is a different action from a failed check.
+
 Confirm `app` is `trunk-io` — the same identity as the `trunk-io[bot]` commenter. If some other app wrote a check run by that name, stop and report it rather than acting on it.
 
 From there, `details_url` and the workflow runs on Trunk's `trunk-merge/**` branch lead to the real logs. `/debugging-ci-failures` covers reading them.

.agents/skills/triaging-merge-queue-failures/SKILL.md

@@ -27,12 +27,15 @@ the chart is identical, only the actions you may take yourself differ.
 
 Read this before writing any command. It is the part that goes stale.
 
-**Trunk publishes no check run in this repository.** The `trunk-io` app posts zero check runs — not on the PR head, not on the queue branch. A predicate like `select(.name | startswith("Trunk Merge Queue"))` matches nothing, and a sweep built on it reports zero verdicts forever while the queue runs normally. Two places still assert that check run exists — `/merging-prs` step 4 and `AGENTS.md` under "Merging PRs" — and both are wrong on this point.
+**Trunk publishes no check run in this repository.** The `trunk-io` app posts zero check runs — not on the PR head, not on the queue branch. A predicate like `select(.name | startswith("Trunk Merge Queue"))` matches nothing, and a sweep built on it reports zero verdicts forever while the queue runs normally. `/merging-prs`, `/debugging-ci-failures` and `AGENTS.md` used to assert that check run exists; all three now point at `trunk merge status` instead.
 
-Trunk exposes queue state two ways, and both are authenticated as the Trunk app by the API:
+Trunk exposes queue state three ways. Only the first carries Trunk's own reasons; the other two are what the GitHub API can see, and they are what this skill's helpers read:
 
-1. **One sticky comment per PR**, authored by `trunk-io[bot]`, rewritten in place as state changes. It carries the state, the failing check's name, and a link to the failing job.
-2. **One draft shadow PR per queue attempt**, authored by `trunk-io[bot]`, with head ref `trunk-merge/pr-<n>/<uuid>`. Its head SHA carries the attempt's real CI as ordinary `github-actions` check runs, with job links. A `-bisection` suffix means Trunk is bisecting a failed batch to find the culprit.
+1. **`trunk merge status <n>` (the CLI)** — the full state machine with a reason per transition, in Trunk's words. This is the primary source for _why_, and the only one that surfaces conflicts, line-skips and cancellations at all. Human text only, no `--json`. The same timeline is exportable as JSON from the Trunk dashboard.
+2. **One sticky comment per PR**, authored by `trunk-io[bot]`, rewritten in place as state changes. It carries the current state, the failing check's name, and a link to the failing job. Rewritten means no history: earlier reasons are gone. It can also carry a `Failed Test | Failure Summary | Logs` table naming the failing test outright — check it before reading any log, and note it is empty for suites that do not upload results to Trunk.
+3. **One draft shadow PR per queue attempt**, authored by `trunk-io[bot]`, with head ref `trunk-merge/pr-<n>/<uuid>`. Its head SHA carries the attempt's real CI as ordinary `github-actions` check runs, with job links. A `-bisection` suffix means Trunk is bisecting a failed batch to find the culprit.
+
+The shadow PR's **body** lists the batch members and the PRs queued ahead of it, which is the only place queue depth is visible. Note the ref is named after the batch _leader_, so a PR batched behind another never appears in a `trunk-merge/pr-<its own number>/` search even while it is actively queued.
 
 Shadow PRs are **never merged** — every one ends closed and unmerged, whether the PR merged or was kicked. So a shadow PR's own state tells you nothing about the outcome; the sticky comment does. Repeated shadow PRs for the same `pr-<n>` are repeated attempts, which is the retry-already-happened signal.
 
@@ -100,6 +103,16 @@ The routine sandbox is more restricted than a laptop. All four of these were obs
 
 ## Establish the facts
 
+**Ask Trunk why before you reconstruct why.** `trunk merge status <n>` prints Trunk's own state machine for the PR: every transition, with the reason in Trunk's words. That includes the reasons nothing else in this skill can recover — a conflict with another PR or with `master`, a PR that skipped the line ahead, a human cancellation, and the exact required check that removed it.
+
+```bash
+trunk merge status <n> 2>&1 | sed 's/\x1b\[[0-9;]*m//g'   # strip ANSI; there is no --json
+```
+
+Read that first. Everything below is for corroborating it, or for the cases it does not cover (what actually failed inside a job). Reconstructing causality from shadow PRs and the Actions API when this command would have answered it in one call is the most expensive mistake available here: it produces a confident narrative that can be wrong on every attempt-level cause.
+
+Two limits worth knowing. The output is ANSI-colored and column-wrapped, so a reason spans several lines and needs re-joining before you match on it. And the CLI needs an interactive `trunk login` once; in a headless environment ask the human to paste the output, or the dashboard's JSON export of the same timeline.
+
 `<n>` is the PR number; `REPO=$(gh api repos/PostHog/posthog --jq .full_name)` — or just hardcode `PostHog/posthog`, since `gh repo view` needs GraphQL.
 
 ```bash
@@ -172,6 +185,16 @@ Nothing was tested and nothing is broken. Trunk closes its shadow PR the moment
 
 Do not route this to `/fixing-flaky-tests`. There is no flaky test — the tests did not finish.
 
+**First rule out a job timeout, which is indistinguishable from teardown here.** A GitHub job killed by `timeout-minutes` marks itself and its steps `cancelled`, never `timed_out` or `failure`, so it lands in exactly the shape above. Searching for `conclusion == "timed_out"` finds nothing. Separate the two by duration: a teardown cancels every in-flight job at once at arbitrary durations, often with `${{ matrix.* }}` still unexpanded in the job names because nothing started; a timeout kill is an isolated job sitting on a round cap.
+
+```bash
+gh api "repos/$REPO/actions/runs/$RUN/jobs?per_page=100" \
+  --jq '.jobs[] | select(.conclusion == "cancelled")
+        | "\(((.completed_at|fromdateiso8601)-(.started_at|fromdateiso8601)))s\t\(.name)"'
+```
+
+A cluster at 300s or 600s is a timeout, not a teardown, and the verdict is entry 6 (a wider issue), not a requeue. On 4 Sep 2026 a degraded npm audit endpoint stalled the pnpm bootstrap until jobs hit their caps, and every one of those kills read as teardown here.
+
 **Verdict: requeue once.** Apply the same retry gate as entry 5: count this head's attempts with `attempts $REPO <n> <head_oid>` and add nothing if a retry already happened. Say in the verdict that the attempt was cancelled rather than failed, so the author does not go looking for a broken test.
 
 Do not reach for a workflow-side fix. Putting the gate on `if: ${{ !cancelled() }}` looks like it removes the false red, but a gate that does not run reports `skipped`, and branch protection counts a skipped required check as passing — so it trades a red check that blocks an untested run for a green one that lets it through. `AGENTS.md`, under "Forcing the full CI matrix on a draft", covers that failure mode. If this class is a recurring drag rather than a one-off, still requeue, and raise it with the team that owns the queue the way entry 6 says to — repeated requeues are not a fix for it.

AGENTS.md

@@ -141,10 +141,12 @@ Never run `gh pr merge` or click the GitHub merge button — both are blocked by
 Agents must not enqueue, merge, re-enqueue, or otherwise cause a PR to land without explicit user approval in the current conversation for the identified PR or stack. Do not infer that approval from requests to prepare a PR, move it toward merge, make it ready, monitor it, or resolve its blockers. Agents may inspect status, fix code and CI, apply `stamphog` when approval is missing, and report that a PR is ready; then they must wait for a direct instruction to merge or enqueue it.
 
 - After explicit user approval: enqueue with `gh pr comment <number> --body "/trunk merge"`. Cancel: `gh pr comment <number> --body "/trunk cancel"`. Enqueueing a stacked PR also enqueues every unmerged layer below it — comment on the top PR to merge the whole stack. `--no-batch` opts the PR (or stack) out of batching.
-- The Trunk CLI is an alternative to the comments: `trunk merge <number>` enqueues, `trunk merge status <number>` inspects, `trunk merge cancel <number>` dequeues. It ships in the flox environment and needs a one-time interactive `trunk login` — run it once even if you prefer the comments, because the same login arms the pre-push merge-queue guard that stops you from knocking a queued PR out of the queue. Agents and headless environments that can't complete the interactive login use the comments.
+- The Trunk CLI is an alternative to the comments: `trunk merge <number>` enqueues, `trunk merge status <number>` inspects, `trunk merge cancel <number>` dequeues. `status` is also the only place Trunk's own reason for each queue transition is visible — a conflict, a PR that skipped the line, a human cancellation, the check that removed it — so read it before reconstructing a cause from CI, and see `/triaging-merge-queue-failures`. It ships in the flox environment and needs a one-time interactive `trunk login` — run it once even if you prefer the comments, because the same login arms the pre-push merge-queue guard that stops you from knocking a queued PR out of the queue. Agents and headless environments that can't complete the interactive login use the comments.
 - Missing required approval: apply the `stamphog` label (`gh pr edit <number> --add-label stamphog`) to trigger the automated review-and-approve flow, and re-apply it whenever it was stripped (`REFUSED`/`ESCALATE` verdict) once the feedback is addressed — re-applying is always safe.
 - After enqueueing, babysit the PR until it merges or fails — follow [`.agents/skills/merging-prs/SKILL.md`](./.agents/skills/merging-prs/SKILL.md) for the preflight, watch, and failure-handling loop.
-- Queue progress is the `Trunk Merge Queue (master)` check run on the PR's head commit. The PR's own checks don't reflect the queue's testing — it runs CI on a `trunk-merge/**` branch.
+- Trunk publishes **no check run** in this repository, so there is nothing on the PR's head that reports queue progress.
+  Read progress from `trunk merge status <number>`, or from the `trunk-io[bot]` sticky comment on the PR.
+  The PR's own checks don't reflect the queue's testing either, because the queue runs CI on a `trunk-merge/**` branch.
 - On failure the Trunk bot comments with links to the failing workflows; fix and push if appropriate, then wait for explicit user approval before re-enqueueing.
 - Never force-push a branch while it is in the queue — it removes the PR from the queue.