Skip to content

Release job's in-job test gate hangs on packages/ai for 30 minutes and fails the release #1888

Description

@code-yeongyu

Summary

The release job's in-job test gate hung for 30 minutes on packages/ai and failed the release. The same commit's normal CI was green, so this is a release-job-only failure mode, not a broken test.

Run 35521423483, db42de496a, Run release step.

Timeline

16:05:38  [run-workspaces] > packages/agent: npm run test
16:06:02  Test Files 87 passed (87) | Tests 992 passed | 1 skipped
16:06:02  [run-workspaces] > packages/ai (@earendil-works/pi-ai): npm run test
16:06:02  > vitest --run
16:06:02  RUN v4.1.11 /home/runner/work/senpi/senpi/packages/ai
          <30 minutes of no output at all>
16:36:17  job failed
16:36:17  Cleaning up orphan processes
16:36:17  Terminate orphan process: pid (28245) (senpi)
16:36:17  Terminate orphan process: pid (28253) (esbuild)
16:36:17  Terminate orphan process: pid (28303) (senpi)

vitest printed its banner and then produced no collection output, no test output and no summary. packages/agent immediately before it completed normally in 24 seconds.

Why this is not a test failure

Check and test on the same commit db42de496a is completed/success (CI run 35523569740 lineage; the green run for that sha finished 16:12:01). The identical suite passes in the normal CI workflow. Re-dispatching the release after that green run existed made decideTestGate take the skip path, and the release completed in about three minutes — 35523379185, success, tag v2026.9.20 pushed.

So the code is fine and the gate logic is fine. What fails is running that suite inside the release job.

Why it was reachable at all

decideTestGate skips the local gate only when HEAD already carries a green Check and test run. The release was dispatched at 16:02:12, seconds after the merge that created db42de496a; that commit's CI started 16:02:00 and went green only at 16:12:01. The release therefore lost the race, fell into the full in-job suite, and hung there.

This makes the failure mode systematically reachable: dispatching a release promptly after a merge is the normal thing to do, and it is exactly what routes the job into the path that hung.

Hypotheses, not conclusions

  • The hang is at collection/startup, since not a single test line was emitted.
  • The orphaned esbuild process suggests a wedged transform rather than a stuck test body.
  • Two orphaned senpi processes suggest a spawned CLI that never exited and was awaited.

None of these are verified. What is verified is the 30-minute silence and the surviving processes.

Expected / ideal state

  • The in-job test gate cannot silently consume the entire job budget: it carries its own timeout well under timeout-minutes: 50 and fails with the wedged workspace named, instead of 30 minutes of nothing.
  • A release that loses the CI race either waits for the in-flight Check and test run on the same sha and reuses it, or runs the suite under that bounded timeout.
  • Child processes the gate spawns are reaped when it exits, so the runner does not have to terminate orphans.

Impact

One failed release, roughly 34 minutes of runner time, and a version applied to 10 workspace manifests in a run that produced no tag. Recovery needed a second dispatch once CI had gone green independently.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions