Skip to content

[CI runner canary] self-hosted runner pool unhealthy #14600

Closed

Description

@github-actions

The self-hosted CI runner pool failed its canary at 2026-10-27T16:11:31.077Z.

Run: https://github.com/hivecommons/hive/actions/runs/40530544403

  • watchdog (failure, runner GitHub Actions 1084150725): Canary jobs are picked up and no runs are stalled in queue

Nothing merges while this is red. Runbook: src/deploy/ci-runners/lke/README.md section 3b
(pool autoscaler/labels, tool-cache race). Check kubectl --context hive-ci -n arc-hive get autoscalingrunnersets,pods
and kubectl --context hive-ci get nodes -l hive-ci-runner=true.

This issue is updated on every red run and closed automatically on the next green one.

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug.
    priority/critical-urgentHighest priority. Must be actively worked on as someone's top priority right now.
    ciApproved by a Hive merger/owner for auto-merge on green CI
    ci-runner-canarySelf-hosted CI runner pool canary is red
    on Oct 27, 2026
  2. github-actions commented on Oct 27, 2026

    @github-actions
    ContributorAuthor

    Still failing: https://github.com/hivecommons/hive/actions/runs/40535278440

    • watchdog (failure, runner GitHub Actions 1084170937): Canary jobs are picked up and no runs are stalled in queue
  3. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Oct 27, 2026
  4. github-actions commented on Oct 27, 2026

    @github-actions
    ContributorAuthor

    Still failing: https://github.com/hivecommons/hive/actions/runs/40541745071

    • watchdog (failure, runner GitHub Actions 1084177890): Canary jobs are picked up and no runs are stalled in queue
  5. github-actions commented on Oct 27, 2026

    @github-actions
    ContributorAuthor

    Still failing: https://github.com/hivecommons/hive/actions/runs/40544555519

    • watchdog (failure, runner GitHub Actions 1084181459): Canary jobs are picked up and no runs are stalled in queue
  6. hivecommons-hive commented on Oct 27, 2026

    @hivecommons-hive
    Contributor

    Scanner triage (not starting work — owner-only infra): the watchdog's canary leg is healthy (canary jobs: 4, still queued after 10m: 0 in run 40541745071); the red comes from the queue-age leg — queued runs: 44, stalled > 15m: 6 (oldest 28m). The pool is saturated, not broken: #14399 (283-commit forward-merge) was re-pushed three times in the last hour and each push re-fans ~60 jobs onto the self-hosted runners. Expect it to clear on its own once that backlog drains; #14568 (measure stalls by queued job age) would also stop counting runs whose jobs are all running against the stall threshold.


    🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

    — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

  7. github-actions commented on Oct 27, 2026

    @github-actions
    ContributorAuthor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ciApproved by a Hive merger/owner for auto-merge on green CIci-failureci-runner-canarySelf-hosted CI runner pool canary is redhelp wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.kind/bugCategorizes issue or PR as related to a bug.priority/critical-urgentHighest priority. Must be actively worked on as someone's top priority right now.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions