Skip to content

fix(queue): count a multi-family message's rows without crashing its batch - #9718

Merged
JSONbored merged 1 commit into
mainfrom
fix/queue-multi-family-batch-log
Aug 7, 2026
Merged

fix(queue): count a multi-family message's rows without crashing its batch#9718
JSONbored merged 1 commit into
mainfrom
fix/queue-multi-family-batch-log

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Closes metagraphed-infra#370. Prerequisite for metagraphed-infra#359.

The sync-batches consumer opens with a batch summary:

const { valid, invalid } = classifySyncBatch(batch.messages);
const rows = valid.reduce((n: number, m) => n + m.rows.length, 0);

A chain-detail message carries families and no rows at allpackMultiFamilyMessage deletes the key by construction (src/sync-batch-queue.ts). So that read throws a TypeError.

Why a log line could take out four healthy lanes

It throws above the per-message try/catch, so the failure is not scoped to the offending message. The whole batch of up to 10 fails, is retried five times, and dead-letters — every lane co-batched with it. account-balances, hotkey-alpha, validator-nominator-counts and nominator-positions all share this consumer.

syncBatchRowCount was written for exactly this case (it already sums families) and was never imported.

Why nothing caught it

SyncBatchMessage.rows was declared required, and stayed required when families was added alongside it in #9678. m.rows.length therefore typechecks on a shape that has no rows. The only tests driving the real worker.queue covered nominator-positions and account-balances — both single-family — so the multi-family path had unit coverage of its parts and none of its handler.

The fix

  • the count goes through syncBatchRowCount
  • rows is now optional on the wire type, so the type system enforces the distinction rather than the reader remembering it. packSyncBatchMessages returns a narrowed SyncBatchRowsMessage[] that still guarantees rows, so every existing caller keeps its non-null reads and the tests needed no !
  • syncBatchRows — the only other reader — answers [] rather than throwing

Regression test

tests/data-api-sync-queue-consumer.test.ts drives a real multi-family message through worker.queue alongside a single-family neighbour, and asserts the neighbour still acks and still writes. That neighbour is the property that was lost; a families-only test would pass against the broken code's intent and miss the blast radius.

Verified the test reproduces the bug: reverting only the syncBatchRowCount call fails it with TypeError: Cannot read properties of undefined (reading 'length').

Not fixed here

Nothing routes chain-detail today (SYNC_QUEUE_LANES names four lanes and this is not one), so this was a landmine, not an outage. Two things still stand between this and the #359 cutover, both recorded there rather than papered over here:

  • the producer posts ~350–662 KiB per chain-detail POST, over the 128 KB transport cap, so packMultiFamilyMessage would throw on the first tick
  • the lane declares no pass_total, and the family writer discards the tally argument

Validation

npm run typecheck        # clean
npm run lint             # clean
npx prettier --check     # clean on all four changed files
npx vitest run tests/sync-batch-queue.test.ts tests/data-api-sync-queue-consumer.test.ts \
  tests/data-api-{nominator-positions,account-balances,hotkey-alpha,neurons,validator-nominator-counts}-d1.test.ts \
  tests/chain-detail-sync-route.test.ts   # 232 passed

Patch coverage measured by intersecting the diff's changed lines with a v8 report over both changed files: 0 uncovered statements, 0 uncovered branches.

Template Used

  • Backend/code change

…batch

The consumer's opening log line summed `m.rows.length` over every valid
message in the batch. A `chain-detail` message carries `families` and no
`rows` at all -- `packMultiFamilyMessage` deletes the key -- so that read
threw a TypeError above the per-message try/catch, failing the WHOLE batch:
every lane co-batched with it, five retries, into the dead-letter queue.

`syncBatchRowCount` was written for exactly this and was never imported.

The reason it was invisible is the type: `rows` was declared required, and
stayed required when `families` arrived, so `m.rows.length` typechecked on a
shape that has no `rows`. It is now optional, `packSyncBatchMessages` returns
a narrowed type that still guarantees it, and the one other reader
(`syncBatchRows`) answers empty instead of throwing.

Nothing routes chain-detail today, so this is a landmine rather than an
outage -- but the lane cannot be cut over until it is gone, and the blast
radius was every other lane sharing the batch.

Closes metagraphed-infra#370.
@superagent-security

Copy link
Copy Markdown

Superagent didn't find any vulnerabilities or security issues in this PR.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
metagraphed-registry-sync-api 41750bf Aug 07 2026, 07:01 AM

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
metagraphed-data-api 41750bf Aug 07 2026, 07:01 AM

@JSONbored JSONbored self-assigned this Aug 7, 2026
@JSONbored
JSONbored merged commit 635e78c into main Aug 7, 2026
7 checks passed
@JSONbored
JSONbored deleted the fix/queue-multi-family-batch-log branch August 7, 2026 07:04
JSONbored added a commit that referenced this pull request Aug 7, 2026
…9776)

The last lane in the epic's scope, and the one it was for. chain-detail is
the largest D1 writer here and the only CONTINUOUS one: ~1,245 rows every 12
seconds is ~9M rows/day, against account-balances' ~1.5M. The bulk lanes are
bursty; this one never stops.

It could not travel raw at any producer setting. Its four families are posted
together so a block and its extrinsics cannot land separately, so the batch is
already ONE BLOCK -- and one block measured 476.6 KiB of JSON against a 128 KiB
cap. Not a batching problem, a units problem, the same class as #360's
rows-vs-bytes.

Three things had to land, and all three are deployed:

  #9765           compression -- 476.6 KiB -> 40.5 KiB, 11.8x, with the budget
                  measuring the compressed size
  #9718           the consumer's batch log read `rows` on a message that has
                  none, above the per-message try/catch -- one families message
                  would have taken the whole batch and the other four lanes
                  into the DLQ
  infra#386       the producer posts one block per POST, not two. Two compress
                  to 64.9 KiB against a 96 KiB budget, and that pairs the
                  busiest block with a QUIET neighbour; two busy ones land near
                  81 KiB, ~16% margin, on a lane that wedges rather than
                  degrades

The poller is already running one block per tick -- 1 scanned, 1 written, 0
errors, ~1.7s, every ~5 minutes.

Rollback is dropping the word: unlike account-balances this lane still has its
inline write, so removing it restores the old path on the next tick.

Closes #9775.
@JSONbored JSONbored mentioned this pull request Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant