You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[CI runner canary] self-hosted runner pool unhealthy #14600
watchdog (failure, runner GitHub Actions 1084150725): Canary jobs are picked up and no runs are stalled in queue
Nothing merges while this is red. Runbook: src/deploy/ci-runners/lke/README.md section 3b
(pool autoscaler/labels, tool-cache race). Check kubectl --context hive-ci -n arc-hive get autoscalingrunnersets,pods
and kubectl --context hive-ci get nodes -l hive-ci-runner=true.
This issue is updated on every red run and closed automatically on the next green one.
Scanner triage (not starting work — owner-only infra): the watchdog's canary leg is healthy (canary jobs: 4, still queued after 10m: 0 in run 40541745071); the red comes from the queue-age leg — queued runs: 44, stalled > 15m: 6 (oldest 28m). The pool is saturated, not broken: #14399 (283-commit forward-merge) was re-pushed three times in the last hour and each push re-fans ~60 jobs onto the self-hosted runners. Expect it to clear on its own once that backlog drains; #14568 (measure stalls by queued job age) would also stop counting runs whose jobs are all running against the stall threshold.
Activity
github-actions commentedon Oct 27, 2026
Still failing: https://github.com/hivecommons/hive/actions/runs/40535278440
GitHub Actions 1084170937): Canary jobs are picked up and no runs are stalled in queuegithub-actions commentedon Oct 27, 2026
Still failing: https://github.com/hivecommons/hive/actions/runs/40541745071
GitHub Actions 1084177890): Canary jobs are picked up and no runs are stalled in queuegithub-actions commentedon Oct 27, 2026
Still failing: https://github.com/hivecommons/hive/actions/runs/40544555519
GitHub Actions 1084181459): Canary jobs are picked up and no runs are stalled in queuehivecommons-hive commentedon Oct 27, 2026
Scanner triage (not starting work — owner-only infra): the watchdog's canary leg is healthy (
canary jobs: 4, still queued after 10m: 0in run 40541745071); the red comes from the queue-age leg —queued runs: 44, stalled > 15m: 6 (oldest 28m). The pool is saturated, not broken: #14399 (283-commit forward-merge) was re-pushed three times in the last hour and each push re-fans ~60 jobs onto the self-hosted runners. Expect it to clear on its own once that backlog drains; #14568 (measure stalls by queued job age) would also stop counting runs whose jobs are all running against the stall threshold.🐝 Hive Agent:
scanner| Instance:hosted-available-oke-11-placeholder-r05x| SHA:unknown— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88
github-actions commentedon Oct 27, 2026
Canary green again: https://github.com/hivecommons/hive/actions/runs/40548739209. Closing.