[Szer]

What was paging

PagerDuty got repeated Monitor: stalled — no run in 5h30m pushes between 2026-09-07 and 2026-09-11. Monitor never stopped scheduling. What happened each time:

1. One codex agent leg (agent (vahter) or agent (coupon)) wedged. The sibling leg in the same run finished in 1–3 min.
2. Since Monitor: 20-min timeout on agent legs + codex transcript artifact #471 the wedged leg is killed by timeout-minutes: 20. GitHub records a timed-out job as cancelled, and the whole run ends cancelled. finalize passed because it only checked for failure.
3. workflow-canary.yml excluded cancelled runs from its "last scheduled run" lookup, so the gap it measured jumped from ~4h to ~8h and it fired stalled for the next two polls until the following Monitor slot resolved it.

Monitor run (UTC)	wedged leg	run conclusion	canary said	PD triggers
09-07 12:40	coupon, runner died after 64 min	failure	failed	3
09-08 01:02	coupon, runner died after 62 min	failure	failed	2
09-08 12:39	vahter, 20 min timeout	cancelled	stalled	2
09-09 12:39	coupon, 20 min timeout	cancelled	stalled	2
09-10 08:36	vahter, 20 min timeout	cancelled	stalled	2
09-10 12:40	none — gather guard failed (coupon prometheus unreachable)	failure	failed	4
09-11 08:35	coupon, 20 min timeout	cancelled	stalled	2

Changes

agent-retry.yml (new) — one automatic re-run on a fresh runner

• workflow_run on completion of Monitor, Daily Project Assessment, Product Analysis (the three codex-action workflows Monitor: 20-min timeout on agent legs + codex transcript artifact #471 listed).
• Fires only when conclusion is failure or cancelled and run_attempt == 1, so at most one retry per run, ever. Attempt 2's completion re-triggers this workflow and is ignored by the guard.
• Calls POST …/actions/runs/{id}/rerun-failed-jobs. Only the failed/cancelled jobs and their dependants re-run; gather and its monitor-evidence artifact are reused, so no second VPN session and no duplicate baseline.sh append sample. Falls back to a full rerun only if that endpoint refuses.
• A cancelled job is retried only if its annotations contain exceeded the maximum execution time. A manual cancel or an aks-vpn concurrency-queue bump stays cancelled.
• Not a step-level retry on purpose: codex-action's safety-strategy: drop-sudo is one-way per runner (see monitor.yml header), so a second invocation on the same runner dies at ensurePasswordlessSudo. A job re-run is a fresh runner.

workflow-canary.yml — cancelled runs no longer vanish

• Stall check: any scheduled run, whatever its conclusion, counts as proof the cron fired.
• Failure check: a cancelled latest run whose job carries the timeout annotation is classified failed with conclusion reported as timed_out. Any other cancelled run is skipped in favour of the newest non-cancelled one, exactly as before.
• Table gains an attempt column and a ::notice is logged when the latest run needed attempt 2, so the wedge rate stays visible even though it no longer pages. Needs checks: read for the annotations call.

monitor.yml finalize

• needs.agent.result == "cancelled" (what a timed-out matrix leg reports — confirmed in run 34579969780's finalize log) now fails the run like failure.

Verified on a throwaway workflow (branch-only, deleted before this PR)

Job with timeout-minutes: 1 + sleep 300, followed by an if: always() finalize that exits 1 (run 34830783954):

• job timeout-leg → cancelled, finalize → failure, run conclusion → cancelled. So the finalize fix alone never turns a wedge red at run level; the canary change is required, not optional.
• POST rerun-failed-jobs → HTTP 201, and attempt 2 contained timeout-leg in progress. Timed-out jobs are included in "failed jobs".
• actionlint clean on all three files.

What this does not do

• It does not explain the wedge. The transcript artifact from Monitor: 20-min timeout on agent legs + codex transcript artifact #471 was only uploaded for the healthy leg in every timed-out run, so codex never wrote a final message before the timeout. The ::notice line in the canary log is the running tally until someone digs into it.
• It does not touch the 09-10 12:40 gather failure (Invalid format 'loki' / coupon prometheus unreachable), which recovered on its own the next slot. A retry would also cover that class, since gather is a normal failed job.

After merge, watch for

• First real attempt-2 on a Monitor run: check that the re-run leg finished and that PD stayed quiet.
• Agent Retry runs skipping with "deliberate cancel, not retrying" on runs you cancelled by hand.

🤖 Generated with Claude Code

https://claude.ai/code/session_01V4AMHh8hMfZh53Tnj3TSSn

[Szer]

End-to-end proof plan (owner asked for certainty today). workflow_run only listens on the default branch, so the branch now also carries a temporary _retry-probe.yml fixture (attempt 1 sleeps past a 1-min timeout; attempt 2 succeeds in recover mode or times out again in timeout-both mode) and lists it in agent-retry.yml. After merge:

1. dispatch probe recover → expect: attempt 1 cancelled by timeout, Agent Retry fires, attempt 2 green, Agent Retry on attempt 2 completion skipped by the guard;
2. dispatch probe timeout-both → expect exactly one retry, run ends cancelled;
3. run the canary in dry-run from the throwaway branch canary-probe-test (which adds the probe to the watch list; never merged, nothing reaches PagerDuty) → expect failed / timed_out for the probe after step 2, and the attempt 2 notice after step 1;
4. follow-up PR removes the probe and its list entry.

Results will be posted here.

[Szer]

End-to-end results (all on main after merge, 2026-09-14 10:23–10:31Z)

step	run	result
probe recover, attempt 1	34832907738	leg cancelled by timeout-minutes, finalize failure, run cancelled
Agent Retry on that	34833030624	fired 10 s after completion, took the rerun-failed-jobs path (no fallback)
probe attempt 2	same run	only leg + finalize re-ran, both green, run success
Agent Retry on attempt 2 completion	34833060677	skipped by the run_attempt == 1 guard
probe timeout-both, attempts 1+2	34833146134	exactly one retry (34833266456), attempt 2 cancelled again, no attempt 3, second Agent Retry (34833392417) skipped
canary vs recovered run	34833098162	row shows success, attempt 2, ::notice … needed attempt 2
canary vs double-timeout run	34833573075	row shows state failed, conclusion timed_out; summary Retry Probe (temporary): failed — <url>

One casualty: the first canary test run (34833098162) was meant to be a dry run but sent a real Retry Probe (temporary): stalled — no run in 26h warning to PagerDuty at 10:25Z. Cause: the pre-existing DRY_RUN: ${{ inputs.dry_run == 'true' }} compares a boolean input to a string and is always false. The incident was resolved via the Events API from the throwaway branch (34833513756, HTTP 202). Fix + fixture removal: #490.