Repository navigation
fix: a JS-parsed ingest takes its turn in the table write queue (SLEEPER-PROD-8469) - #1
Closed
claude[bot] wants to merge 2 commits into
Closed
claude[bot] wants to merge 2 commits into
claude[bot] wants to merge 2 commits into
Conversation
… its batch never starts inside a running native shred's savepoint (SLEEPER-PROD-8469) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WS7KpoVVBG1REsUE1GkM2e
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WS7KpoVVBG1REsUE1GkM2e
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested via Slack thread
Before: Partitions that can't use the native shredder (
player-seasonandplayer-gamesin sleeperbot's player_stats store) were written by a synchronoustable.overwrite(), which skips the table's write queue. If another partition's native shred was still running, it heldSAVEPOINT nitro_shredon the same writer connection; nitro doesn't queueexecuteAsync, so nothing stopped the sync write. Its batch'sBEGIN EXCLUSIVEthen failed with "cannot start a transaction within a transaction" (SLEEPER-PROD-8469). The batch's ROLLBACK also aborted the running shred, which then failed with "no such savepoint: nitro_shred" (SLEEPER-PROD-846A). The partition write was dropped, and after 4 such contentions the store went fatal: it reopened and refetched, and in some cases moved to its in-memory database. All 5 executeBatch events in 8469 have a matching 846A event from the same user in the same second.After: The JS-parse path, along with
ingestRaw's fallback after a failed shred, goes throughshred(..., { inJs: true }). That path waits its turn in the table'sserialized()write queue like every other async write, so it never starts a batch while a native shred is still running.How:
RowTable.shredtakes an optional{ inJs }(src/table/types.ts). Insrc/table/sqlite.ts,shredOrParseskips the native shredder wheninJsis set but still runs insidewithDeferredIndexes(serialized(...)). Insrc/define_partitions.ts,ingestRaw'sinJsnow awaits that call instead of the syncoverwrite. Twodefine_partitionstests are updated, and a row_table regression test checks that a JS-parsed shred waits for a native shred that hasn't finished.tscis clean;yarn test: 721 passed;yarn test:prod: 698 passed.Not covered:
lib/is rebuilt in this PR, but shipping still needs a new gitpkg tag and a bump of the pin in sleeperbot'sclients/app-shared/package.json.cpp/sqliteExecuteBatch.cpp:63-64sends ROLLBACK even when its own BEGIN failed, so it rolls back the other caller's transaction; andexecuteAsyncis not queued byDatabaseQueue, sostartOperationSynccan't see it.executeAsyncevents grouped under 8469. The likely sources are the fire-and-forget ETag write and index DDL outside the queue; not confirmed.onContendedmessage ("usually means its dedicated reader never opened") is misleading here. The reader is enabled, and the collision is between two writes.🤖 Generated with Claude Code
https://claude.ai/code/session_01WS7KpoVVBG1REsUE1GkM2e