The last lane in metagraphed-infra#346's scope, and the one the migration was for.
chain-detail is the largest D1 writer on the platform and the only continuous one: ~1,245 rows every 12 seconds is ~9M rows/day, against account-balances' ~1.5M. The bulk lanes are bursty; this one never stops. metagraphed-infra#346's argument for one queue was global backpressure — "a lane left out is a lane that can still overwhelm the database the others are being polite about" — and this was that lane.
Why it took three PRs to get here
It could not travel the transport raw, at any producer setting. Its four families are posted together so a block and its extrinsics cannot land separately, which means the batch is already one block — and one block measured 476.6 KiB of JSON (#8790494, the busiest captured) against a 128 KiB cap. Not a batching problem; a units problem, the same class as metagraphed-infra#360's rows-vs-bytes.
|
|
| #9765 |
compress the message — 476.6 KiB → 40.5 KiB (11.8×), and the budget now measures the compressed size |
| #9718 |
the consumer's batch log read rows on a message that has none, above the per-message try/catch — one families message would have failed the whole batch, five retries, into the DLQ, taking the other four lanes with it |
| metagraphed-infra#386 |
the producer posts one block per POST. Two compress to 64.9 KiB against a 96 KiB budget — and that pairs the busiest block with a quiet neighbour. Two genuinely busy blocks land near 81 KiB, ~16% margin, on a lane where overflow wedges rather than degrades |
All three are merged and deployed. The poller is running one block per tick — 1 scanned, 1 written, 0 error(s), ~1.7 s, every ~5 minutes.
What to watch, and it is not a row count
This lane declares no pass_total, so there is no completeness tally to read. The signal is chain-detail in lane_health staying ok at its cadence, and chain_detail_blocks continuing to advance. A wedge shows up as the same block retried — which the staleness watchdog and lane-alarm both reach.
Rollback
Dropping the word from SYNC_QUEUE_LANES. Unlike account-balances, this lane still has its inline D1 write, so removing it restores the old path on the next tick with no backfill.
Done when
A tick lands with the four families written together through the queue, and chain_detail_blocks has advanced past the cutover deploy.
The last lane in metagraphed-infra#346's scope, and the one the migration was for.
chain-detailis the largest D1 writer on the platform and the only continuous one: ~1,245 rows every 12 seconds is ~9M rows/day, againstaccount-balances' ~1.5M. The bulk lanes are bursty; this one never stops. metagraphed-infra#346's argument for one queue was global backpressure — "a lane left out is a lane that can still overwhelm the database the others are being polite about" — and this was that lane.Why it took three PRs to get here
It could not travel the transport raw, at any producer setting. Its four families are posted together so a block and its extrinsics cannot land separately, which means the batch is already one block — and one block measured 476.6 KiB of JSON (#8790494, the busiest captured) against a 128 KiB cap. Not a batching problem; a units problem, the same class as metagraphed-infra#360's rows-vs-bytes.
rowson a message that has none, above the per-message try/catch — one families message would have failed the whole batch, five retries, into the DLQ, taking the other four lanes with itAll three are merged and deployed. The poller is running one block per tick —
1 scanned, 1 written, 0 error(s), ~1.7 s, every ~5 minutes.What to watch, and it is not a row count
This lane declares no
pass_total, so there is no completeness tally to read. The signal ischain-detailinlane_healthstayingokat its cadence, andchain_detail_blockscontinuing to advance. A wedge shows up as the same block retried — which the staleness watchdog andlane-alarmboth reach.Rollback
Dropping the word from
SYNC_QUEUE_LANES. Unlikeaccount-balances, this lane still has its inline D1 write, so removing it restores the old path on the next tick with no backfill.Done when
A tick lands with the four families written together through the queue, and
chain_detail_blockshas advanced past the cutover deploy.