fix(platform): queue serializable task-comment writes per task - #3268
Merged
Conversation
Every comment on a task bumps the same rows — the task's comment_count and the discussion thread's next message slot — so serializable commenters that overlap all lose but the first, and a plain retry loses again whenever another commits first. `transactSerializable` gives up after five attempts, a count any burst larger than itself defeats; the audit chain head had the same storm (#3246) and the message slot under READ COMMITTED had its count-bound twin (#3267). Comment appends and deletes now run through `queuedOnTask`: the transaction-level advisory lock on `task-comment:<taskId>` queues a first attempt behind a retry that holds the key as a session lock from before its BEGIN, and a 40001/40P01 raised anywhere in the write is marked with the key so the caller's next attempt takes that session lock first. A commenter therefore wastes at most one attempt; READ COMMITTED callers pay only the lock. Edits touch no shared row and stay as they were. The integration proof gains a lane: twelve serializable commenters on one task, overlap widened, all land with one comment_count of twelve, twelve distinct message slots, and between thirteen and twenty-four attempts — every loser retried exactly once, queued.
A retry queued on one key drops the others: a task comment that lost at the org's audit chain head was re-marked with the task key by its outer queue, so its retry took the task's session lock alone, met the other tasks' commenters at the chain head again and lost there — the storm #3246 removed came back for cross-task bursts. Marks now nest. `markRetryQueueKey` prepends an outer key to the keys an inner callback already put on the error (never twice), so the list reads in the callback's own acquisition order; `beginQueued` takes the session locks in that order before BEGIN and unlocks in reverse; the retry loop adopts the whole list whenever a marked failure arrives. Every first attempt takes its transaction-level locks in the same order, so a queued retry and a first attempt never hold two keys in opposite orders. `retryQueueKeysOf` exposes the list; `retryQueueKeyOf` keeps answering the outermost key.
…laim The overdue ladder's level-2 rung stamps the task row and then comments on the task, while every commenter takes the task's comment key first and writes the row second — two lock orders for one pair of resources, which Postgres resolves after deadlock_timeout with a 40P01 that the READ COMMITTED commenters (agents, workflows, REST) surface as an error. The claim now takes the key as its first statement, the order everyone else uses; the unit test pins key-before-claim.
`queuedOnTask` marks a serialization failure with the task key on top of the chain-head key the audit write inside it already carries, and the shared retry queue now honours both, so a retry holds the task AND the org's chain head from before its BEGIN. The lock statement is exported (`lockTaskCommentQueue`) for transactions that write the task row before commenting, the meta row shape has a name, and unit tests pin the contract: lock first, task key in front of an inner audit key, other failures untouched. The integration proof gains the cross-task lane: twelve serializable commenters on twelve tasks of one org meet only at the chain head, all land, the chain verifies, and attempts stay within twice the burst.
The key list made `beginQueued` take several session locks in a row, but the loop sat outside the unlock guard: a failure on the second key (a cancelled statement, lock-table exhaustion) skipped the unlocks and returned the connection to the pool still holding the first key — every later transaction-level lock on that key would then wait until that pooled connection died. The loop now runs inside the guard, so the keys already held are unlocked in reverse on every exit and the connection is released clean. The retry loop also keeps its key list when a marked failure carries a subset of it: a queued attempt that loses to a writer outside its inner queues is re-marked with the outer key alone, and dropping the inner key is how a retry loses at that resource again.
The subset helper had landed between transactSerializable's JSDoc and the function; it now sits above with its own line. The cancelled-lock test also asserts that no second reservation happens, so a future widening of the transient table is caught here rather than trusted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Defect
addTaskCommentruns undertransactSerializable(task comment route, connector task store) and every comment on a task bumps the same rows: the task'scomment_countand the discussion thread's next message slot. Serializable commenters that overlap therefore all lose but the first, and a plain retry loses again whenever another commits first.transactSerializablegives up after five attempts — a count any burst larger than itself defeats — so a burst of comments on one task (several agents reporting on one task, a fan-out) fails legitimate comments with a 40001. This is the same storm #3246 fixed for the audit chain head, and the serializable twin of the count bound #3267 removes from the message slot under READ COMMITTED. Surfaced by the adversarial review of #3267 and priced there as a separate change.Fix
domains/tasks/comments.ts: comment appends and deletes run throughqueuedOnTask, the twin of the audit head'slockChainHead:task-comment:<taskId>(the sharedRETRY_QUEUE_LOCK_CLASS) is taken before the write's first read, so a first attempt queues behind a retry that holds the key as a session lock from before its BEGIN;markRetryQueueKey), which makes the caller's next attempt take that session lock first (transactSerializable's retry queue).A commenter wastes at most one attempt under contention. READ COMMITTED callers (the date nudge, the sandbox shim) pay only the lock, which orders one task's comments and marks nothing. Edits touch no shared row and stay as they were. Public signatures are unchanged.
Proof
tasks: a burst of serializable commenters on one task all land: twelve serializable commenters on one task with the overlap widened →landed=12/12 rejected=0 commentCount=12 distinctOrders=12 attempts=23— exactly N + (N−1): every loser retried once, queued, none twice.messages: concurrent appends each take their own slot(appends=12, failed=1) — the READ COMMITTED slot-count defect that fix(platform): claim message slots until a deadline, not a fixed count #3267 fixes; this branch is cut from main 4df487e, which predates it. With fix(platform): claim message slots until a deadline, not a fixed count #3267 merged the run is 526/526.tsc --noEmit,oxlint --type-aware,knip:check,oxfmtclean;vitest --project server backend/domains/tasks: 13 files / 80 tests green.Review round 1 (adversarial review: changes requested) — repaired in 9dba83b, b53b7a0, de4857a
Two real blockers, both fixed:
queuedOnTaskre-marked every 40001 with the task key, overwriting the marklockChainHeadhad put on a loss at the org's audit chain head, so a comment retry queued on the task key alone and met the other tasks' commenters at the chain head again — the fix(platform): harden erasure, legal-hold and audit-log paths #3246 storm returned for cross-task bursts in one org. The shared retry queue now carries a key list: marks nest (an outer mark is prepended to the inner ones, never twice),beginQueuedtakes the session locks in that order before BEGIN and unlocks in reverse, and the retry loop adopts the whole list. First attempts take their transaction-level locks in the same order (task, then chain head), so no two transactions ever hold the two keys in opposite orders.retryQueueKeysOfexposes the list;retryQueueKeyOfstill answers the outermost key.date-notifications.tsstamped the task row and then commented, while every commenter takes the task key first and writes the row second — a deadlock pair resolved only afterdeadlock_timeout. The claim now takeslockTaskCommentQueueas its first statement; the unit test pins key-before-claim.Also from the review:
CommentMetanamed instead of an inline type;queuedOnTaskand the retry-queue note now describe the nesting; new unit suitecomments.queue.test.ts(lock first, task key in front of an inner audit key, other failures untouched) andserializable.test.tscases for nested keys (19/19).Proof on the repaired tree: real-Postgres integration run 527/527 across 140/140 lanes (the message-slot lane happened to pass this time). New cross-task lane
tasks: a burst of serializable commenters across one org's tasks all land:landed=12/12 rejected=0 chainValid=true attempts=23— every loser at the chain head retried exactly once, queued on task + chain head; the same-task lane againattempts=23; the audit lane unchanged (8/8, attempts=15). Fullvitest --project server: 581 files / 6325 tests green; shared + platformtsc,oxlint,knip,oxfmtclean.Review round 2 (adversarial re-review: changes requested) — repaired in 3b12792
One real defect the round-1 repair introduced: with several keys,
beginQueuedlocked them OUTSIDE its unlock guard, so a failure on a laterpg_advisory_lock(a cancelled statement, lock-table exhaustion) returned the pooled connection with the first key still held as a session lock — every later lock on that key would wait until that connection died. The lock loop now runs inside the guard: the keys already held are unlocked in reverse on every exit. Unit test: a 57014 on the second lock → lock, lock, unlock(first) — no BEGIN, no second attempt, connection released.Also hardened (advisory): the retry loop keeps its key list when a marked failure carries a subset of it (a queued attempt that lost to a writer outside its inner queues), so a retry never drops the inner key; unit test covers it.
serializable.test.tsis now 21/21.attempts=23,attempts=23,attempts=15(deterministic, three runs in a row); the single red is fix(platform): claim message slots until a deadline, not a fixed count #3267's message-slot lane, which this branch's base predates.Review round 3 (adversarial re-review): approve — nits applied in 1aced44
Round 2's repair holds on every exit path (lock failure on any key, BEGIN failure, callback failure, COMMIT answering ROLLBACK, COMMIT throwing, unlock failure): exactly the acquired keys are unlocked in reverse and the connection is released once; the single-key (audit) statement sequence is byte-for-byte unchanged. Two nits fixed: the subset helper had landed between
transactSerializable's doc and the function; the cancelled-lock test now also asserts a single reservation. Reviewer's conclusion on the key-list rule: only two mark sites exist (lockChainHead,queuedOnTask) and bothqueuedOnTaskcallers queue on one task per transaction, so the reachable transitions are superset → replace and subset → keep; a disjoint mark is unreachable today and replace stays the conservative choice.