fix(queue): count a multi-family message's rows without crashing its batch - #9718
Merged
Conversation
…batch The consumer's opening log line summed `m.rows.length` over every valid message in the batch. A `chain-detail` message carries `families` and no `rows` at all -- `packMultiFamilyMessage` deletes the key -- so that read threw a TypeError above the per-message try/catch, failing the WHOLE batch: every lane co-batched with it, five retries, into the dead-letter queue. `syncBatchRowCount` was written for exactly this and was never imported. The reason it was invisible is the type: `rows` was declared required, and stayed required when `families` arrived, so `m.rows.length` typechecked on a shape that has no `rows`. It is now optional, `packSyncBatchMessages` returns a narrowed type that still guarantees it, and the one other reader (`syncBatchRows`) answers empty instead of throwing. Nothing routes chain-detail today, so this is a landmine rather than an outage -- but the lane cannot be cut over until it is gone, and the blast radius was every other lane sharing the batch. Closes metagraphed-infra#370.
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
metagraphed-registry-sync-api | 41750bf | Aug 07 2026, 07:01 AM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
metagraphed-data-api | 41750bf | Aug 07 2026, 07:01 AM |
This was referenced Aug 7, 2026
JSONbored
added a commit
that referenced
this pull request
Aug 7, 2026
…9776) The last lane in the epic's scope, and the one it was for. chain-detail is the largest D1 writer here and the only CONTINUOUS one: ~1,245 rows every 12 seconds is ~9M rows/day, against account-balances' ~1.5M. The bulk lanes are bursty; this one never stops. It could not travel raw at any producer setting. Its four families are posted together so a block and its extrinsics cannot land separately, so the batch is already ONE BLOCK -- and one block measured 476.6 KiB of JSON against a 128 KiB cap. Not a batching problem, a units problem, the same class as #360's rows-vs-bytes. Three things had to land, and all three are deployed: #9765 compression -- 476.6 KiB -> 40.5 KiB, 11.8x, with the budget measuring the compressed size #9718 the consumer's batch log read `rows` on a message that has none, above the per-message try/catch -- one families message would have taken the whole batch and the other four lanes into the DLQ infra#386 the producer posts one block per POST, not two. Two compress to 64.9 KiB against a 96 KiB budget, and that pairs the busiest block with a QUIET neighbour; two busy ones land near 81 KiB, ~16% margin, on a lane that wedges rather than degrades The poller is already running one block per tick -- 1 scanned, 1 written, 0 errors, ~1.7s, every ~5 minutes. Rollback is dropping the word: unlike account-balances this lane still has its inline write, so removing it restores the old path on the next tick. Closes #9775.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes metagraphed-infra#370. Prerequisite for metagraphed-infra#359.
The
sync-batchesconsumer opens with a batch summary:A
chain-detailmessage carriesfamiliesand norowsat all —packMultiFamilyMessagedeletes the key by construction (src/sync-batch-queue.ts). So that read throws aTypeError.Why a log line could take out four healthy lanes
It throws above the per-message
try/catch, so the failure is not scoped to the offending message. The whole batch of up to 10 fails, is retried five times, and dead-letters — every lane co-batched with it.account-balances,hotkey-alpha,validator-nominator-countsandnominator-positionsall share this consumer.syncBatchRowCountwas written for exactly this case (it already sums families) and was never imported.Why nothing caught it
SyncBatchMessage.rowswas declared required, and stayed required whenfamilieswas added alongside it in #9678.m.rows.lengththerefore typechecks on a shape that has norows. The only tests driving the realworker.queuecoverednominator-positionsandaccount-balances— both single-family — so the multi-family path had unit coverage of its parts and none of its handler.The fix
syncBatchRowCountrowsis now optional on the wire type, so the type system enforces the distinction rather than the reader remembering it.packSyncBatchMessagesreturns a narrowedSyncBatchRowsMessage[]that still guaranteesrows, so every existing caller keeps its non-null reads and the tests needed no!syncBatchRows— the only other reader — answers[]rather than throwingRegression test
tests/data-api-sync-queue-consumer.test.tsdrives a real multi-family message throughworker.queuealongside a single-family neighbour, and asserts the neighbour still acks and still writes. That neighbour is the property that was lost; a families-only test would pass against the broken code's intent and miss the blast radius.Verified the test reproduces the bug: reverting only the
syncBatchRowCountcall fails it withTypeError: Cannot read properties of undefined (reading 'length').Not fixed here
Nothing routes
chain-detailtoday (SYNC_QUEUE_LANESnames four lanes and this is not one), so this was a landmine, not an outage. Two things still stand between this and the #359 cutover, both recorded there rather than papered over here:packMultiFamilyMessagewould throw on the first tickpass_total, and the family writer discards the tally argumentValidation
Patch coverage measured by intersecting the diff's changed lines with a v8 report over both changed files: 0 uncovered statements, 0 uncovered branches.
Template Used