Repository navigation
Monitor: 20-min timeout on agent legs + codex transcript artifact - #471
Merged
Merged
Conversation
Both coupon-leg runs on 2026-09-07/08 (34123128603, 34175324767) hit "hosted runner lost communication with the server" ~60min in with zero step logs; healthy legs finish in 1-3min. Adds job-level timeout-minutes: 20 to bound the blast radius, and uploads codex-action's output-file (final agent message) as a per-bot artifact so a future wedge still leaves a transcript behind. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Szer
added a commit
that referenced
this pull request
Sep 14, 2026
…s "stalled" Five Monitor slots in five days (2026-09-07..11) lost one codex agent leg to a wedge. Since PR #471 the leg dies at timeout-minutes, GitHub records that job — and the whole run — as "cancelled", and workflow-canary.yml dropped cancelled runs from its "last scheduled run" lookup. The gap it measured then doubled to ~8h and PagerDuty got two "Monitor: stalled — no run in 5h30m" pushes per wedge, although Monitor was scheduling fine. agent-retry.yml (new): on workflow_run completion of Monitor / Daily Project Assessment / Product Analysis with conclusion failure or cancelled and run_attempt == 1, call rerun-failed-jobs. Only the dead leg and its dependants re-run on a fresh runner; gather's evidence artifact is reused (no second VPN session, no duplicate baseline sample). Cancelled jobs without the timeout annotation (manual / concurrency-queue cancel) are not retried. Verified on a throwaway run: rerun-failed-jobs does include a job killed by timeout-minutes (attempt 2 re-ran it). workflow-canary.yml: any scheduled run now counts for the stall check; a cancelled latest run whose job carries the "exceeded the maximum execution time" annotation is classified failed (conclusion reported as timed_out), other cancels still fall back to the newest non-cancelled run. Table gains an attempt column and a ::notice when the latest run needed a retry, so the wedge rate stays visible. monitor.yml finalize: needs.agent.result == "cancelled" (what a timed-out leg reports) fails the run like "failure". Verified the run conclusion stays "cancelled" regardless, which is why the canary change above is needed rather than optional. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V4AMHh8hMfZh53Tnj3TSSn
Szer
added a commit
that referenced
this pull request
Sep 14, 2026
…s "stalled" Five Monitor slots in five days (2026-09-07..11) lost one codex agent leg to a wedge. Since PR #471 the leg dies at timeout-minutes, GitHub records that job — and the whole run — as "cancelled", and workflow-canary.yml dropped cancelled runs from its "last scheduled run" lookup. The gap it measured then doubled to ~8h and PagerDuty got two "Monitor: stalled — no run in 5h30m" pushes per wedge, although Monitor was scheduling fine. agent-retry.yml (new): on workflow_run completion of Monitor / Daily Project Assessment / Product Analysis with conclusion failure or cancelled and run_attempt == 1, call rerun-failed-jobs. Only the dead leg and its dependants re-run on a fresh runner; gather's evidence artifact is reused (no second VPN session, no duplicate baseline sample). Cancelled jobs without the timeout annotation (manual / concurrency-queue cancel) are not retried. Verified on a throwaway run: rerun-failed-jobs does include a job killed by timeout-minutes (attempt 2 re-ran it). workflow-canary.yml: any scheduled run now counts for the stall check; a cancelled latest run whose job carries the "exceeded the maximum execution time" annotation is classified failed (conclusion reported as timed_out), other cancels still fall back to the newest non-cancelled run. Table gains an attempt column and a ::notice when the latest run needed a retry, so the wedge rate stays visible. monitor.yml finalize: needs.agent.result == "cancelled" (what a timed-out leg reports) fails the run like "failure". Verified the run conclusion stays "cancelled" regardless, which is why the canary change above is needed rather than optional. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V4AMHh8hMfZh53Tnj3TSSn
Szer
added a commit
that referenced
this pull request
Sep 14, 2026
…s "stalled" (#489) * CI: retry wedged codex agent legs once; canary stops reporting them as "stalled" Five Monitor slots in five days (2026-09-07..11) lost one codex agent leg to a wedge. Since PR #471 the leg dies at timeout-minutes, GitHub records that job — and the whole run — as "cancelled", and workflow-canary.yml dropped cancelled runs from its "last scheduled run" lookup. The gap it measured then doubled to ~8h and PagerDuty got two "Monitor: stalled — no run in 5h30m" pushes per wedge, although Monitor was scheduling fine. agent-retry.yml (new): on workflow_run completion of Monitor / Daily Project Assessment / Product Analysis with conclusion failure or cancelled and run_attempt == 1, call rerun-failed-jobs. Only the dead leg and its dependants re-run on a fresh runner; gather's evidence artifact is reused (no second VPN session, no duplicate baseline sample). Cancelled jobs without the timeout annotation (manual / concurrency-queue cancel) are not retried. Verified on a throwaway run: rerun-failed-jobs does include a job killed by timeout-minutes (attempt 2 re-ran it). workflow-canary.yml: any scheduled run now counts for the stall check; a cancelled latest run whose job carries the "exceeded the maximum execution time" annotation is classified failed (conclusion reported as timed_out), other cancels still fall back to the newest non-cancelled run. Table gains an attempt column and a ::notice when the latest run needed a retry, so the wedge rate stays visible. monitor.yml finalize: needs.agent.result == "cancelled" (what a timed-out leg reports) fails the run like "failure". Verified the run conclusion stays "cancelled" regardless, which is why the canary change above is needed rather than optional. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V4AMHh8hMfZh53Tnj3TSSn * test: temporary Retry Probe workflow to prove agent-retry.yml end to end on main workflow_run only listens to workflows on the default branch, so the retry chain cannot be exercised from a PR branch. This fixture (attempt 1 sleeps past a 1-min timeout, attempt 2 succeeds or also times out depending on input) is dispatched after merge and removed in the follow-up PR. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V4AMHh8hMfZh53Tnj3TSSn --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Failure summary
On 2026-09-07 and 2026-09-08 the
agent (coupon)leg's GitHub-hosted runner died with "The hosted runner lost communication with the server":Both runs took 62-64 min before dying, vs 1-3 min for a healthy leg. Because the runner died mid-run, no step log was uploaded, so there is zero evidence of what codex was doing at the time. The only correlation found so far: both legs created a new issue (#468 / #469) and then went silent — no further signal available without a transcript.
The
agentjob has notimeout-minutes, andopenai/codex-action@v1exposes no timeout input of its own, so a wedged runner burns the full 6h GitHub Actions default with nothing to show for it.What each change buys
timeout-minutes: 20on theagentjob (.github/workflows/monitor.yml) — bounds the blast radius of a future wedge to 20 min instead of 6h. Healthy legs finish in 1-3 min, so 20 min is a generous ceiling with no expected false positives. Scoped to theagentjob only;gatherandfinalizeare untouched.id: run-agent"Run monitor agent" step anoutput-file: /tmp/codex-output/${{ matrix.bot }}.md. Per codex-action'saction.yml(confirmed by reading it directly),output-fileis "File where the final Codex message is written" — the same content exposed as the action'sfinal-messageoutput, just persisted to disk. The action does not expose a richer full transcript/log location, so this is the most we can capture short of instrumenting codex itself. A new "Upload codex transcript" step (actions/upload-artifact@v4,if: always(), namecodex-transcript-${{ matrix.bot }},path: /tmp/codex-output/,if-no-files-found: ignore,retention-days: 14) uploads it even if the step fails or times out. The artifact name includesmatrix.botso per-bot legs never collide with each other, and thecodex-transcript-prefix keeps it distinct from the existingmonitor-evidenceartifact.mkdir -p /tmp/codex-outputwas added to the existing "Load agent prompt and evidence" step so the directory exists before codex runs.Sibling workflows using the same codex-action pattern (not modified, candidates for the same treatment)
.github/workflows/_sre-agent.yml.github/workflows/project.yml.github/workflows/product.ymlValidation
actionlint .github/workflows/monitor.yml— no output (clean).python3 -c "import yaml; yaml.safe_load(open('.github/workflows/monitor.yml'))"—YAML_OK.🤖 Generated with Claude Code