chore(queue): cut chain-detail over, on three landed prerequisites - #9776
Merged
Conversation
The last lane in the epic's scope, and the one it was for. chain-detail is the largest D1 writer here and the only CONTINUOUS one: ~1,245 rows every 12 seconds is ~9M rows/day, against account-balances' ~1.5M. The bulk lanes are bursty; this one never stops. It could not travel raw at any producer setting. Its four families are posted together so a block and its extrinsics cannot land separately, so the batch is already ONE BLOCK -- and one block measured 476.6 KiB of JSON against a 128 KiB cap. Not a batching problem, a units problem, the same class as #360's rows-vs-bytes. Three things had to land, and all three are deployed: #9765 compression -- 476.6 KiB -> 40.5 KiB, 11.8x, with the budget measuring the compressed size #9718 the consumer's batch log read `rows` on a message that has none, above the per-message try/catch -- one families message would have taken the whole batch and the other four lanes into the DLQ infra#386 the producer posts one block per POST, not two. Two compress to 64.9 KiB against a 96 KiB budget, and that pairs the busiest block with a QUIET neighbour; two busy ones land near 81 KiB, ~16% margin, on a lane that wedges rather than degrades The poller is already running one block per tick -- 1 scanned, 1 written, 0 errors, ~1.7s, every ~5 minutes. Rollback is dropping the word: unlike account-balances this lane still has its inline write, so removing it restores the old path on the next tick. Closes #9775.
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
metagraphed-data-api | 95b9a8b | Aug 07 2026, 09:19 AM |
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #9775. The last lane in metagraphed-infra#346's scope, and the one the migration was for.
chain-detailis the largest D1 writer on the platform and the only continuous one — ~1,245 rows every 12 seconds is ~9M rows/day, againstaccount-balances' ~1.5M. The bulk lanes are bursty; this one never stops. #346's argument for one queue was global backpressure: "a lane left out is a lane that can still overwhelm the database the others are being polite about."Why this is one word and three PRs of prerequisites
It could not travel the transport raw, at any producer setting. The four families are posted together so a block and its extrinsics cannot land separately, which makes the batch already one block — and one block measured 476.6 KiB of JSON (#8790494) against a 128 KiB cap. A units problem, not a batching one: the same class as metagraphed-infra#360's rows-vs-bytes.
wrangler deploymentsrowson a message that has none, above the per-message try/catch — one families message would have failed the whole batch into the DLQ, taking the other four lanes1 scanned, 1 written, 0 error(s)That third one is worth its own line: two blocks compress to 64.9 KiB against a 96 KiB budget — but that pairs the busiest block with a quiet neighbour (199 chain events vs 694). Two genuinely busy blocks land near 81 KiB, ~16% margin, on a lane where overflow does not degrade: the producer advances its cursor only on a POST that succeeded, so it wedges.
What to watch, and it is not a row count
This lane declares no
pass_total, so there is no completeness tally. The signal ischain-detailinlane_healthstayingokat its ~5-minute cadence, andchain_detail_blocksadvancing. A wedge shows up as the same block retried — reachable by both the staleness watchdog andlane-alarm.Rollback
Dropping the word. Unlike
account-balances(#9747), this lane still has its inline D1 write, so removing it restores the old path on the next tick with no backfill. That asymmetry is deliberate and noted in the config.Validation
One var. The code paths it selects are already tested end to end — including a real multi-family message driven through
worker.queuealongside a single-family neighbour.Template Used