Skip to content

checkGitHubCI blocks indefinitely on orphaned check suites stuck in QUEUED #1160

Description

@trivenay

Problem

checkGitHubCI() in pr_checker.js blocks landing with ✘ GitHub CI is still running when any GitHub Actions check suite has status !== 'COMPLETED'. However, GitHub occasionally creates check suites that remain stuck in QUEUED status with 0 check runs indefinitely — they are never dispatched and will never complete.

This has blocked at least two recent PRs on nodejs/node:

  • #64991 — Test macOS suite stuck 3+ days, landed manually
  • #64830 — same Test macOS suite stuck, landed manually

In both cases, the orphaned suite:

  • Has status: QUEUED, conclusion: null
  • Has 0 check runs (never started executing)
  • Has a valid workflowRun (Test macOS, workflow ID 858952)
  • Was created at the same time as sibling suites that completed fine
  • Never gets re-triggered by request-ci because GitHub sees it as already "queued"
  • The only workaround is a collaborator manually running git node land --yes

The commit-queue.sh pre-check (gh pr checks | grep pending) passes fine because it evaluates individual check runs (all completed), not suite containers.

Data

PR #64991, SHA 94a47d6f15b2:

  • 25 GitHub Actions suites total
  • 24 completed (success/skipped)
  • 1 orphaned: suite 84102428609, Test macOS, QUEUED, 0 check runs, created 2026-08-05T13:19:48Z

No newer run of Test macOS exists for this SHA — request-ci does not re-trigger a workflow GitHub considers already "queued."

Proposed fix

Consider a suite as orphaned and skip it if:

  • status !== 'COMPLETED'
  • checkRuns.nodes.length === 0 (nothing has ever executed inside it)
  • Suite age exceeds a configurable threshold (suggesting 3 hours since createdAt — legitimate suites dispatch runs within seconds)
if (status !== 'COMPLETED') {
  const runCount = checkRuns?.nodes?.length ?? 0;
  const ageMs = Date.now() - new Date(createdAt).getTime();
  if (runCount === 0 && ageMs > ORPHANED_SUITE_TIMEOUT) {
    cli.warn(`Ignoring orphaned check suite with no runs (status: ${status}, age: ${Math.round(ageMs / 3600000)}h)`);
    continue;
  }
  pendingJobs.push({ app: app.slug, status, conclusion });
  continue;
}

GraphQL change: add createdAt to the checkSuites query in PRCommits.gql.

Trade-offs

What this fixes: Orphaned suites no longer block the commit queue forever.

Risk: If someone adds commit-queue before request-ci has finished triggering all workflows, a suite that should have run but hasn't started yet (0 runs, within the threshold) could age past the timeout and get skipped — landing the PR without that test suite's results.

However, the current workaround (git node land --yes) already bypasses ALL checks — including real failures. A targeted skip of empty aged-out suites is strictly safer than the current manual workaround.

Activity

  1. trivenay commented on Aug 10, 2026

    @trivenay
    Author

    As a longer-term thought — would it make sense to consolidate CI validation into GitHub's required status checks (or rulesets), so the merge gate is enforced by GitHub itself rather than checked by git node land? That way the tool wouldn't need to iterate suites and distinguish orphaned from running — GitHub would answer "all required checks passed" or "not yet" definitively.

    I understand there may be reasons this isn't straightforward today (Jenkins coverage, commit-queue direct push model), but curious if there's a path toward it.

  2. trivenay commented on Aug 14, 2026

    @trivenay
    Author

    Opened a PR implementing the discussed approach: #1163

    It skips a non-completed check suite only when it has zero check runs and was created more than 3 hours ago (configurable), so orphaned suites that GitHub never dispatches no longer block landing, while genuinely pending or freshly-created suites still block. Validated against the two PRs that hit this (#64991, #64830) — the orphaned queued suite in each would be skipped. Feedback welcome on the approach and the timeout.

  3. trivikr commented on Aug 18, 2026

    @trivikr
    Member

    Because of the GitHub outage on 2026-08-17, many check suites were stuck today which blocked landing using commit queue. Fixing this issue will help avoid these situations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions