Skip to content

fix(runner): fence shutdown to the original task claim - #189

Open
lhy-dcf wants to merge 1 commit into
CopilotKit:mainfrom
lhy-dcf:fix/runner-shutdown-claim-ownership
Open

lhy-dcf wants to merge 1 commit into
CopilotKit:mainfrom
lhy-dcf:fix/runner-shutdown-claim-ownership

Conversation

@lhy-dcf

@lhy-dcf lhy-dcf commented Nov 7, 2026

Copy link
Copy Markdown

Summary

Fence background-runner shutdown to the exact claim it originally acquired. A stale runner must not interrupt a replacement run merely because both runs belong to the same task ID.

Background

If a task is paused and explicitly retried while the original executor is still settling, another runner can acquire a new lease. The old Runner.stop() looked up the current database task and passed that new lease into Store.interrupt(). The store's ownership check consequently succeeded for the replacement run, changing it to interrupted and preventing its result from being committed.

This was found by source inspection on main (d4214aff). I searched issues and all-state PRs for runner/lease/shutdown/claim, including the earlier interruption work in #47. I found no matching fix; the open headless-continuity PR #176 changes a different path. No new issue was required by CONTRIBUTING.md.

Changes

  • Retain each active run's original Claim alongside its AbortController.
  • Pass the held claim to Store.interrupt() on shutdown, reusing its existing lease fence rather than reconstructing ownership from the latest task row.
  • Keep abort-by-ID and abort-all behavior, with no schema, dependency, configuration or UI changes.
  • Add a deterministic regression using two Runner instances and two real SQLite connections. Deferred executors reproduce the handover without sleeps; the replacement lease must remain running and its result must commit after the old runner stops and settles.

Testing

Windows, Node 24.22.1; locked dependencies installed with lifecycle scripts disabled.

  • RED, before the source fix: vitest run tests/runner.test.ts — 1 failed / 8 passed. The new test expected the replacement lease to remain running; it became interrupted with a null lease.
  • Targeted runner/store/timeout/shutdown suites — 27 passed across 4 files, including the existing graceful-shutdown control.
  • vitest run --maxWorkers=2 — 352 passed across 56 files, with unchanged timeouts.
  • The first default-concurrency full run was 351 passed / 1 failed, a 5-second telemetry subprocess timeout. The unchanged telemetry file passed 8/8 when rerun alone. That initial failure is retained here, not described as proven unrelated baseline behavior.
  • npm run check-format, npm run lint, npm run typecheck, npm run build — passed.
  • npm run knip and npm run knip:production — executed, exit 1 with unused dependency/export findings outside the changed files. These checks are report-only in current CI. I did not change those findings or independently rerun Knip on an untouched main checkout.
  • git diff --check — passed.

Additional Notes

AI-assisted implementation and verification. The reproduction exercises real local SQLite ownership transitions with synthetic executor results, not live Intelligence/model/computer services or a production incident. It establishes the stale-worker fencing invariant; it does not claim distributed deployment acceptance or end-to-end recovery of external side effects.

The existing normal-shutdown test still verifies that a runner interrupts work it actually owns and leaves it awaiting explicit review/retry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant