Repository navigation
Drain task goroutines on shutdown and replay unsaved activities - #300
Conversation
…t be stored. The outbox migration is 0006 because 0005 is already traffic rewrite rules. A shutdown cancel leaves an in-flight intent claim running, so the next start can reset it to open instead of recording the interrupt as done or blocked.
BaaRai
left a comment
There was a problem hiding this comment.
Thanks for the outbox and the shutdown work, the failure modes you target are the right ones.
1. Wait does not drain every task goroutine, and work is still admitted while it waits.
Engine.Waitonly waits onrt.wg(server/engine.go:260-290). Detached chat runs are counted ine.inflightthroughbeginTaskOperation, not inrt.wg:runDetachedIntentstartsrunIntentin a bare goroutine (server/engine.go:1430-1433).beginTaskOperationdoes not checkdraining(server/engine_timeout.go:51-59). OnlyregisterTaskRoutinesdoes (server/engine.go:235).srv.Waitruns beforehttpSrv.Shutdown(cmd/artex/main.go:131 and :138), so the listener keeps accepting requests during the 15 s drain. Whenrun()returns, the deferredmgr.Close()(cmd/artex/main.go:89) closes the stores while a detached run can still be writing.
Fix: stop admission first (call httpSrv.Shutdown before draining, or make beginTaskOperation fail while draining is set), and make Wait also wait until every inflightCount reaches 0. Please add a test where a detached operation is running and Wait blocks until it ends.
2. The replay can store the same activity twice. In FlushActivityOutbox, AppendActivity (db/activity_outbox.go:80) and the DELETE (db/activity_outbox.go:84) are two separate statements. A crash between them leaves the row in the outbox, and the next start inserts the activity again.
Fix: do the insert and the delete in one transaction, or make the replay idempotent.
3. One permanently failing row blocks the outbox forever. If AppendActivity fails, the row gets UPDATE ... last_error and the flush returns (db/activity_outbox.go:80-83). The same row fails first on every start, no later row is ever replayed, and the table keeps growing because EnqueueActivity (server/engine.go:709) still adds rows.
Fix: bound the attempts per row (then drop it with a log line, or move it to a dead-letter table), or document explicitly that a stuck row blocks the queue.
4. Test. The outbox path has no test because it needs Postgres (FlushActivityOutbox is only called from server/server.go:304). Please add a Postgres-backed test: an activity that failed to store is replayed exactly once after a restart, and its outbox row is gone.
The listener closes before Wait, and a shutdown refuses new task operations. Wait also stays until detached runs finish. The activity insert and the outbox delete commit together, and a row that keeps failing is dropped after five attempts so it cannot block the rest.
BaaRai
left a comment
There was a problem hiding this comment.
Thanks, this addresses the review. Verified in the code: the listener closes before the drain, beginTaskOperation refuses work while draining under runtimeMu, Wait waits for the task groups and then for every inflight count, and the outbox replay now inserts and deletes in one transaction, keeps the detail, and drops a stuck row after five attempts instead of blocking the queue. The new tests cover the replay-once path and the stuck-row drop, and the migration gains the attempts column.
Summary
activity_outboxand replayed on startup. The migration is0006because0005_traffic_rewrite_rules.sqlis already on main.AbortShutdown.running, so the next start resets it to open and resumes the task.Test plan
loadMigrationsaccepts0005_traffic_rewrite_rules.sqland0006_activity_outbox.sqltogether.running, thenopenafter restart.