Skip to content

Drain task goroutines on shutdown and replay unsaved activities - #300

Merged
BaaRai merged 2 commits into
BaaRai:mainfrom
Protonmatter:protonmatter/shutdown-outbox
Nov 17, 2026
Merged

BaaRai merged 2 commits into
BaaRai:mainfrom
Protonmatter:protonmatter/shutdown-outbox

Conversation

@Protonmatter

Copy link
Copy Markdown
Contributor

Summary

  • Split from Keep control-plane state correct when a write or a restart fails #293.
  • Activities that cannot be inserted are queued in activity_outbox and replayed on startup. The migration is 0006 because 0005_traffic_rewrite_rules.sql is already on main.
  • Shutdown waits for the task goroutines and cancels them with AbortShutdown.
  • That cancel does not write a terminal task status. An in-flight intent claim stays running, so the next start resets it to open and resumes the task.

Test plan

  • loadMigrations accepts 0005_traffic_rewrite_rules.sql and 0006_activity_outbox.sql together.
  • Stop the process while a worker is inside an intent and confirm the task status is unchanged and the intent is still running, then open after restart.
  • Fail an activity insert and confirm the row is stored from the outbox on the next start.

…t be stored.

The outbox migration is 0006 because 0005 is already traffic rewrite rules. A shutdown cancel leaves an in-flight intent claim running, so the next start can reset it to open instead of recording the interrupt as done or blocked.

@BaaRai BaaRai left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the outbox and the shutdown work, the failure modes you target are the right ones.

1. Wait does not drain every task goroutine, and work is still admitted while it waits.

  • Engine.Wait only waits on rt.wg (server/engine.go:260-290). Detached chat runs are counted in e.inflight through beginTaskOperation, not in rt.wg: runDetachedIntent starts runIntent in a bare goroutine (server/engine.go:1430-1433).
  • beginTaskOperation does not check draining (server/engine_timeout.go:51-59). Only registerTaskRoutines does (server/engine.go:235).
  • srv.Wait runs before httpSrv.Shutdown (cmd/artex/main.go:131 and :138), so the listener keeps accepting requests during the 15 s drain. When run() returns, the deferred mgr.Close() (cmd/artex/main.go:89) closes the stores while a detached run can still be writing.

Fix: stop admission first (call httpSrv.Shutdown before draining, or make beginTaskOperation fail while draining is set), and make Wait also wait until every inflightCount reaches 0. Please add a test where a detached operation is running and Wait blocks until it ends.

2. The replay can store the same activity twice. In FlushActivityOutbox, AppendActivity (db/activity_outbox.go:80) and the DELETE (db/activity_outbox.go:84) are two separate statements. A crash between them leaves the row in the outbox, and the next start inserts the activity again.
Fix: do the insert and the delete in one transaction, or make the replay idempotent.

3. One permanently failing row blocks the outbox forever. If AppendActivity fails, the row gets UPDATE ... last_error and the flush returns (db/activity_outbox.go:80-83). The same row fails first on every start, no later row is ever replayed, and the table keeps growing because EnqueueActivity (server/engine.go:709) still adds rows.
Fix: bound the attempts per row (then drop it with a log line, or move it to a dead-letter table), or document explicitly that a stuck row blocks the queue.

4. Test. The outbox path has no test because it needs Postgres (FlushActivityOutbox is only called from server/server.go:304). Please add a Postgres-backed test: an activity that failed to store is replayed exactly once after a restart, and its outbox row is gone.

The listener closes before Wait, and a shutdown refuses new task operations. Wait also stays until detached runs finish. The activity insert and the outbox delete commit together, and a row that keeps failing is dropped after five attempts so it cannot block the rest.

@BaaRai BaaRai left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this addresses the review. Verified in the code: the listener closes before the drain, beginTaskOperation refuses work while draining under runtimeMu, Wait waits for the task groups and then for every inflight count, and the outbox replay now inserts and deletes in one transaction, keeps the detail, and drops a stuck row after five attempts instead of blocking the queue. The new tests cover the replay-once path and the stuck-row drop, and the migration gains the attempts column.

@BaaRai
BaaRai merged commit 03ac1c7 into BaaRai:main Nov 17, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants