Skip to content

fix(server): drain live heartbeat runs on shutdown so same-agent checkout can reenter (#30)#203

Open
om952 wants to merge 1 commit into
OpenScanAI:masterfrom
om952:fix/issue-30
Open

fix(server): drain live heartbeat runs on shutdown so same-agent checkout can reenter (#30)#203
om952 wants to merge 1 commit into
OpenScanAI:masterfrom
om952:fix/issue-30

Conversation

@om952

@om952 om952 commented Jul 2, 2026

Copy link
Copy Markdown

Thinking Path

  • Paperclip orchestrates AI agents for zero-human companies
  • Heartbeat runs track the lifecycle of agent work on issues; issues.checkoutRunId and issues.executionRunId act as locks
  • When the server restarts (deploy, crash, scale-down), live heartbeat runs in queued, running, or scheduled_retry remain marked as active and continue holding those locks
  • A restarted same-agent process then hits a 409 on POST /api/issues/:id/checkout because adoptStaleCheckoutRun only considers terminal/missing runs as stale
  • Without an audit trail, operators cannot reconstruct when or why an issue's checkoutRunId silently changed ownership
  • This pull request drains live heartbeat runs on graceful shutdown and adds an issue.stale_lock_takeover activity-log row when a stale lock is adopted
  • The benefit is that server restarts no longer wedge issue locks and stale takeovers become observable

What Changed

  • Added server/src/services/heartbeat-shutdown-drain.ts with drainStaleHeartbeatRunsOnShutdown(db)
    • Cancels live heartbeat runs (queued, running, scheduled_retry)
    • Clears issues.checkoutRunId, issues.executionRunId, issues.executionAgentNameKey, and issues.executionLockedAt
    • Marks cancelled runs with errorCode: "server_shutdown_stale_lock_cleanup"
  • Wired the drain into the SIGINT/SIGTERM shutdown handler in server/src/index.ts
    • Bounded by a 5s Promise.race timeout
    • Best-effort: errors are logged at WARN and never block shutdown
  • Updated adoptStaleCheckoutRun in server/src/services/issues.ts
    • Added companyId to the .returning(...) clause
    • On successful adoption, inserts an issue.stale_lock_takeover activity-log row
    • Audit failure is wrapped in try/catch so it cannot block checkout adoption
  • Added tests:
    • server/src/__tests__/heartbeat-shutdown-drain.test.ts
    • server/src/__tests__/issues-checkout-stale-lock-takeover.test.ts

Verification

pnpm -r typecheck
pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && npx vitest run server/src/__tests__/heartbeat-shutdown-drain.test.ts server/src/__tests__/issues-checkout-stale-lock-takeover.test.ts
node scripts/run-vitest-stable.mjs --mode general --group general-server

closes #30

…kout can reenter

On a graceful SIGINT/SIGTERM shutdown, every still-live heartbeat-run row
(queued / running / scheduled_retry) is now cancelled and the issue locks
they hold (executionRunId, checkoutRunId) are cleared.

- New drainStaleHeartbeatRunsOnShutdown service, wired into index.ts shutdown handler
- Audit row issue.stale_lock_takeover on adoptStaleCheckoutRun success
- Tests for shutdown drain and stale-lock takeover audit behavior

Closes OpenScanAI#30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(heartbeat): port upstream PR #6712 — drain live heartbeat runs on shutdown for safe restart

1 participant