Skip to content

chore(queue): cut chain-detail over, on three landed prerequisites - #9776

Merged
JSONbored merged 1 commit into
mainfrom
chore/cut-chain-detail-over
Aug 7, 2026
Merged

chore(queue): cut chain-detail over, on three landed prerequisites#9776
JSONbored merged 1 commit into
mainfrom
chore/cut-chain-detail-over

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Closes #9775. The last lane in metagraphed-infra#346's scope, and the one the migration was for.

chain-detail is the largest D1 writer on the platform and the only continuous one — ~1,245 rows every 12 seconds is ~9M rows/day, against account-balances' ~1.5M. The bulk lanes are bursty; this one never stops. #346's argument for one queue was global backpressure: "a lane left out is a lane that can still overwhelm the database the others are being polite about."

Why this is one word and three PRs of prerequisites

It could not travel the transport raw, at any producer setting. The four families are posted together so a block and its extrinsics cannot land separately, which makes the batch already one block — and one block measured 476.6 KiB of JSON (#8790494) against a 128 KiB cap. A units problem, not a batching one: the same class as metagraphed-infra#360's rows-vs-bytes.

deployed
#9765 compression — 476.6 KiB → 40.5 KiB (11.8×), budget measures the compressed size ✅ verified in wrangler deployments
#9718 the consumer's batch log read rows on a message that has none, above the per-message try/catch — one families message would have failed the whole batch into the DLQ, taking the other four lanes
metagraphed-infra#386 producer posts one block per POST ✅ container image swapped, poller ticking 1 scanned, 1 written, 0 error(s)

That third one is worth its own line: two blocks compress to 64.9 KiB against a 96 KiB budget — but that pairs the busiest block with a quiet neighbour (199 chain events vs 694). Two genuinely busy blocks land near 81 KiB, ~16% margin, on a lane where overflow does not degrade: the producer advances its cursor only on a POST that succeeded, so it wedges.

What to watch, and it is not a row count

This lane declares no pass_total, so there is no completeness tally. The signal is chain-detail in lane_health staying ok at its ~5-minute cadence, and chain_detail_blocks advancing. A wedge shows up as the same block retried — reachable by both the staleness watchdog and lane-alarm.

Rollback

Dropping the word. Unlike account-balances (#9747), this lane still has its inline D1 write, so removing it restores the old path on the next tick with no backfill. That asymmetry is deliberate and noted in the config.

Validation

npm run typecheck    # clean
npx prettier --check wrangler.data.jsonc   # clean
npx vitest run tests/chain-detail-sync-route.test.ts tests/sync-batch-queue.test.ts \
  tests/data-api-sync-queue-consumer.test.ts   # 82 passed

One var. The code paths it selects are already tested end to end — including a real multi-family message driven through worker.queue alongside a single-family neighbour.

Template Used

  • Backend/code change

The last lane in the epic's scope, and the one it was for. chain-detail is
the largest D1 writer here and the only CONTINUOUS one: ~1,245 rows every 12
seconds is ~9M rows/day, against account-balances' ~1.5M. The bulk lanes are
bursty; this one never stops.

It could not travel raw at any producer setting. Its four families are posted
together so a block and its extrinsics cannot land separately, so the batch is
already ONE BLOCK -- and one block measured 476.6 KiB of JSON against a 128 KiB
cap. Not a batching problem, a units problem, the same class as #360's
rows-vs-bytes.

Three things had to land, and all three are deployed:

  #9765           compression -- 476.6 KiB -> 40.5 KiB, 11.8x, with the budget
                  measuring the compressed size
  #9718           the consumer's batch log read `rows` on a message that has
                  none, above the per-message try/catch -- one families message
                  would have taken the whole batch and the other four lanes
                  into the DLQ
  infra#386       the producer posts one block per POST, not two. Two compress
                  to 64.9 KiB against a 96 KiB budget, and that pairs the
                  busiest block with a QUIET neighbour; two busy ones land near
                  81 KiB, ~16% margin, on a lane that wedges rather than
                  degrades

The poller is already running one block per tick -- 1 scanned, 1 written, 0
errors, ~1.7s, every ~5 minutes.

Rollback is dropping the word: unlike account-balances this lane still has its
inline write, so removing it restores the old path on the next tick.

Closes #9775.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 7, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
metagraphed-data-api 95b9a8b Aug 07 2026, 09:19 AM

@superagent-security

Copy link
Copy Markdown

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored JSONbored self-assigned this Aug 7, 2026
@JSONbored
JSONbored merged commit eda4839 into main Aug 7, 2026
6 checks passed
@JSONbored
JSONbored deleted the chore/cut-chain-detail-over branch August 7, 2026 09:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

queue: cut chain-detail over to sync-batches

1 participant