feat(queue): accept chain-detail's four row families in one message - #9678
Merged
Conversation
Closes metagraphed-infra#359. chain-detail posts blocks, extrinsics, chain events and account events in one request because they must land in one write: a block whose drill-down shows no calls is readable and WRONG. SyncBatchMessage carried one `rows` array, so this lane had no shape to move in. ## Widened, not split The issue offered one-message-per-family or a family map. The first loses the property the four are posted together FOR -- four messages retry independently -- so it is rejected on that basis rather than chosen for being easiest. `families` is OPTIONAL and MUTUALLY EXCLUSIVE with `rows`. Every existing lane carries one array and is untouched; only a lane in MULTI_FAMILY_LANES may send families, and only the names in MULTI_FAMILY_LANE_ROW_FAMILIES, so an unrecognised family is refused rather than written to a guessed table. A message carrying both shapes is refused too -- accepting both would leave the writer to guess which is authoritative and silently drop one. ## The row count is the sum passTallyFor counts families as well as rows. Counting only `rows` would credit zero and a declared pass would never close. ## It throws rather than splitting packMultiFamilyMessage refuses an oversize chunk instead of degrading into the split the shape exists to prevent. THAT CONSTRAINT BITES TODAY, and the test says so: the producer batches 2 blocks per POST at ~350-662 KiB against a 128 KB cap. So this lane CANNOT be cut over until the producer posts smaller batches. Wired and inert -- SYNC_QUEUE_LANES does not name it -- with the blocker recorded in a test rather than left to be rediscovered at cutover. - full suite 703 files / 17,065 tests green
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
metagraphed-data-api | 9a756ab | Aug 06 2026, 12:31 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
metagraphed-registry-sync-api | 9a756ab | Aug 06 2026, 12:31 PM |
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
This was referenced Aug 7, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes metagraphed-infra#359.
chain-detailposts blocks, extrinsics, chain events and account events in one request because they must land in one write: a block whose drill-down shows no calls is readable and wrong.SyncBatchMessagecarried a singlerowsarray, so this lane had no shape to move in.Widened, not split
The issue offered one-message-per-family or a family map. The first loses the property the four are posted together for — four messages retry independently — so it is rejected on that basis rather than chosen for being easiest.
familiesis optional and mutually exclusive withrows:MULTI_FAMILY_LANESmay send families, and only the names inMULTI_FAMILY_LANE_ROW_FAMILIES, so an unrecognised family is refused rather than written to a guessed tableThe row count is the sum
passTallyForcounts families as well as rows. Counting onlyrowswould credit zero and a declared pass would never close.It throws rather than splitting
packMultiFamilyMessagerefuses an oversize chunk instead of degrading into the split the shape exists to prevent.That constraint bites today, and a test says so. The producer batches 2 blocks per POST at ~350–662 KiB against a 128 KB cap — so this lane cannot be cut over until the producer posts smaller batches.
Shipped wired and inert (
SYNC_QUEUE_LANESdoes not name it), with the blocker recorded in a test rather than left to be rediscovered at cutover. That follow-up belongs on the producer side.On the Workflows alternative
metagraphed-infra#363 flags Workflows as a candidate here, and a step boundary would also give atomicity. This is the smaller change: it is contained to a shape three lanes already ignore, adds no new platform primitive, and does not block a Workflows evaluation later — if that lands, this becomes one fewer reason to need it.
Verification