Repository navigation
fix: the volume-test fixes (ACK ceiling, false send failures, forward-window re-key, BLE queue discards) - #583
Merged
Merged
Conversation
… deaf A frame more than the sender-ratchet forward distance ahead of the receiver failed as an ordinary decryption error. So did every later frame from that sender and every re-sealed resend, each further ahead, so the session never decrypted from that sender again. The 2026-10-19 volume test lost its iPhone-to-Android direction this way: 3,915 INVALID_CIPHERTEXT failures on freshly sealed frames, with the sender logging 12,152 retries against 2,434 messages the receiver decrypted. TooDistantInTheFuture now classifies as SessionDesync, which the engine answers with its rate-limited re-key. Unlike the epoch checks, this one is authenticated: OpenMLS decrypts the sender data under AEAD before it reads the generation. TooDistantInThePast stays Decryption, since the session is healthy and a re-sealed resend recovers it. The window rises from 1,000 to 8,192 generations, pinned at four times the engine's in-flight ceiling. It is receiver-local policy and changes nothing on the wire.
…of wedging the engine At 1,000 pending acknowledgements the retry path propagated the capacity error out of process(), which then returned before the timeout sweep on every tick. The one step that frees slots never ran again, and the engine never recovered. The direct send path returned the same error to the app after the frame was on the wire, so one phone's API accepted 2,748 messages while the other received 4,626. Every transmit path now asks for a slot before it writes. With none free the message goes to the outbox and the retry queue, the app gets message_deferred with reason ack_capacity, and the send returns its id. A held retry is re-queued without being re-sealed or counted as an attempt. process() runs every step on every tick and reports the first error at the end. The capacity error is now typed (AckCapacity). Two leaks that kept the table full are fixed with it. A refused registration marked every timed-out entry and discarded the list, and the sweep skips marked entries, so those messages were never retried. The prune of timed-out entries ran at a fixed five minutes, exactly the retry ladder's maximum delay, so a ninth retry found its slot gone and needed a new one at capacity. The prune age now derives from the configuration. A frame that times out while still in the BLE transport's queue gets its deadline extended, for up to a minute, instead of being retried. Each retry used to append a re-sealed copy behind the original, growing the queue, the round trip and the receiver's ratchet gap together.
…ssage Both outbound fragment queues discard everything queued for a peer when they reach their cap, and they reached it. Android's two-second poll pulled fragments from the core with no backpressure check at all, and both platforms' drains checked the backed-up mark only after the destructive pull. With a cap of 100 a single sealed 16 KB message was already more than the queue could hold, which is why 16 KB and 30 KB messages from Android never arrived. Both drains and the Android poll now stop before pulling while any peer is past the mark. The cap is 768, so the mark holds a whole 512-fragment message, the largest the core produces. iOS passed its shared inbound constant to the outbound queues, so they get their own. A Rust guard pins the two platforms' caps equal and above the message ceiling.
A blocked sender's frame was dropped without marking its id, on purpose, so a retry arriving after an unblock was delivered. A sender whose retry ladder was still running when the block was lifted delivered everything it sent during the block. That is not what a user who blocked someone expects, and it is the opposite of the post-unblock reset, which already drops our own queued messages to that peer. The id is now remembered. Nothing is answered while the block lasts, and a retry after it meets the duplicate path, which answers it, so the sender stops retrying.
…ked messages States the new invariants where each state machine is described: a frame is never transmitted without a slot to track it, a frame still queued locally gets a bounded extension instead of a retry, a receiver past the forward window re-keys (an authenticated trigger, so R2 is unchanged), and a message sent during a block is never delivered. Changelog entries for the volume-test fixes.
Sessions restored from storage never emit secure_session_established, the only thing the demo listened to, so after a restart every contact showed no lock. The context now seeds its session set from mlsListSessions once the protocol starts.
The device rerun of the volume test still lost the whole iPhone-to-Android direction to decrypt failures, 12,160 of them, with the forward window already widened to 8,192. It turns out every resend was re-sealed under a fresh ratchet generation, unconditionally. The receiver has to fast-forward over each one, and a resend of something it already holds is dropped as a duplicate without ever being decrypted. So when the link stopped carrying acknowledgements for 17 minutes, the retry ladder kept re-sealing at up to 200 frames a second: 20,919 of them. The receiver ended up past the window and could decrypt nothing. The queued-frame grace was supposed to stop exactly this, and didn't: the iOS bridge had already pulled those fragments out of the Rust queue into its own, where they expired. So bound the thing that matters directly. An acknowledgement for a frame proves the receiver decrypted at least that far, so a sender that never seals more than the window past the last acknowledged frame can never strand its peer. The MLS session layer now counts seals per session incarnation against the highest acknowledged one. Past a quarter of the window, resends go out verbatim, which costs nothing and still decrypts unless the receiver has moved more than the out-of-order tolerance past it. The next acknowledgement re-opens the budget. New messages always seal. Acknowledgements are unauthenticated, so a forged one can hand re-seals back. That is no worse than every sender before this.
In the device rerun the iPhone's writes to Android stopped at 16:41:38 and did not come back for 17 minutes. CoreBluetooth reported the write buffer full the whole time, while the same link kept delivering Android's notifications. So nothing ever said the link was down. Every fragment queued for Android expired, and every ack the iPhone owed went with them. It recovered only when a connection to Android's *next* advertising identity happened to complete. Main hit the same stall at 14:52 in the first run and got lucky sooner. Two things kept it stuck. A refused write-without-response just waits for a ready callback that, on such a link, never comes. And CoreBluetooth never times out a connect: the pending connect to the new identity sat there, and every later attempt was refused as "already connecting". The 10 s connection timeout constant has been declared and unused this whole time. This is not great. Let's fix both. A central link that refuses every write for 15 s, with no accepted write and no ready callback in between, is cancelled so it reconnects. Refusals alone don't count: a busy link refuses constantly while still moving. And a foreground connect that has not landed in 10 s is cancelled so the next discovery can try again. In the background a pending connect is how iOS reconnects a peer coming back into range, so it is left alone.
The third device rerun of the volume test lost the whole iPhone-to-Android direction for its last 25 minutes. The new write stall teardown fired, the iPhone reconnected to Android twice under Android's new advertising address and verified it there both times, and still never wrote another fragment. It turns out the registry answered "which peripheral is this peer on?" with the first match of a dictionary walk. After a rotation two identifiers map to the same device id: the verified, connected one, and the one Android just left, which nothing ever unmaps on an ordinary disconnect. The walk kept landing on the stale one. So every send found "no connected peripheral", re-dialled an address nobody advertised any more, and the connect timeout cancelled and re-armed it every 10 s. 709 attempts over 19 identifiers, with a perfectly good link to the same peer sitting idle the whole time. Confusion ensues. Let's fix it at all three levels. A send prefers a connected link, with ties broken on the identifier, so the answer no longer depends on hash order. Verifying a peer on one identifier retires every other disconnected one mapped to it, so neither the send path nor the retry closures dial it again, and a connect still pending to it is cancelled. And the connection monitor stops re-dialling a peripheral it never identified once it has not advertised for a minute; the dozen pending connects to Android's earlier addresses were only crowding out the one to where it went. A known peer that goes quiet is still redialled, since rotation is handled by retirement. Central-role mappings are left alone on purpose: they key inbound fragments on a link the peer opened, which a verification on our outbound link says nothing about.
In the third volume rerun Android resent 19,404 times against 5,000 messages, nearly all of it over the one direction that was already the bottleneck: notifies to the iPhone go out as indications, one confirmed fragment at a time, about 12 a second. The queued-frame grace exists for exactly this. A message whose acknowledgement times out while it is still waiting in the BLE queue gets its timeout extended instead of a second copy. But the core can only see what it still holds, and a message stops counting as queued the moment the bridge pulls its first fragment. Both bridges pulled until a peer held 576 fragments, three quarters of the cap, because the pull mark was derived from the cap. Android sat at exactly 576 for most of the run. So the core saw nothing queued, resent everything behind the first few seconds of link time, and the fragments it had already handed over aged out after 30 s, tearing their messages on the way. Give the mark its own value: 32 fragments, a few seconds of the slowest link, on both platforms. The rest of the backlog stays in the core, which costs nothing and is where the grace can see it. The 768 cap is untouched, and it still never tears a message. Not because the mark holds a whole 512-fragment message, which is what the old tests and guard claimed, but because every fragment in the queue arrives through a pop that checks the mark first, so a peer never holds more than the mark. The guard now pins what actually matters: the same cap and the same mark on both platforms, the cap above the largest message, and the mark below the cap.
A delivery acknowledgement is the thing that stops its sender's retry timer. The BLE carrier queued it like any other frame, at the back of one FIFO, behind every data frame and every resend already waiting for the bridge. On a link carrying less than it is offered that is exactly backwards. In run 4 of the 2026-10-19 volume test the round trip for an acknowledgement reached 196 seconds against a 10-second timeout, so every message was retried, and every retry the receiver saw cost it one *more* acknowledgement on the same saturated link. Confusion ensues. Queue an acknowledgement ahead of every queued data frame, behind earlier acknowledgements so they keep their order. The fragments of a message already being written are not overtaken: they are past this queue, and tearing a message to save one frame's latency would be silly.
Android writes every fragment to a subscribed central as an ATT indication, one outstanding at a time, and then waited another 20 ms before draining the next one. The pause exists for completions that only say *our* controller took the op: firing the next write instantly out-runs a slower peer and drops fragments mid-burst. An indication's completion is not that. It fires on the central's confirmation, so the peer already has the fragment and there is nothing left to wait for. On the indicate path the pause was dead time. It held the Android-to-iPhone direction to 12 to 16 fragments a second in the 2026-10-19 volume test, below the 15 it was offered, and that deficit is where the collapse started. Drain at once on a confirmed indication. Central writes keep the pacing.
Two phones sending 5 messages a second each way over Bluetooth LE delivered 70.5%. The link was up the whole time. It just carried 12 to 16 frames a second in one direction against 15 offered. It turns out the sender has no flow control at all. Every new frame, every resend and one acknowledgement per received frame go into one unbounded queue, on a retry timer that starts at enqueue. So the acknowledgement round trip passed the 10-second timeout after about two minutes, everything was retried, each resend cost the receiver another acknowledgement on the saturated path, and that path's rate *halved*. Both phones then filled their 1,000-slot acknowledgement tables, and every later send evicted a message as "Outbox capacity exceeded". 50,000 retries, 6,000 messages failed. The four earlier fixes on this branch each removed one amplifier of that loop, and the loop found the next one every time. This removes the loop. At most 16 frames that wait for an acknowledgement are in flight to one peer over a mesh carrier. The next one waits in the outbox, the app gets message_deferred with reason peer_window, and held messages go out oldest first as slots free. A new message never overtakes a held one, holding is not an attempt, and a resend of an in-flight frame always passes. That bounds the queue, and with it the round trip, the resends and the receiver's ratchet gap, at any link rate. The count is derived from the outbox rather than kept on the side, so no settle, give-up, expiry or eviction path can leak a slot and wedge a peer. Please don't turn it into a counter. Only the mesh carriers. A relay mailbox is store-and-forward, and holding a message back from it would strand it on a phone the user is about to close.
Every delivered frame got its own acknowledgement frame: a 264-byte unsealed Message, one BLE fragment. In the 2026-10-19 volume test that was a third of the frames each phone sent, on a link already carrying less than it was offered. 13,058 of the 25,133 fragments Android managed to push to the iPhone were acknowledgements. That's a lot of radio spent saying "got it". To a peer that advertises ack_versions entry 2, delivery acknowledgements for frames that arrived over a mesh carrier are now held for up to 500 ms and sent as one frame of up to five: the first id in ack_for, the rest in a new ack_more key. At a few messages a second that is two or three answers per frame. Five still fits in one 512-byte fragment. The receiving side settles each id exactly as if it had arrived in its own frame, every check included, so a coalesced acknowledgement can do nothing five single ones couldn't. It reads at most four extra ids, and ignores ack_more entirely next to ack_status: an undecryptable answer is not a delivery and is never coalesced. It's capability-gated because an older SDK reads only ack_for and would settle one id of five. A peer that never advertised entry 2 keeps getting one acknowledgement per frame. The relay isn't coalesced: a relay frame costs a round trip to a server, not a share of a radio link.
Past the re-seal budget a resend goes out verbatim, which spends no ratchet generation. That's the point of the budget. The catch is that a verbatim copy only decrypts while the receiver is within the out-of-order tolerance (32 generations) of it. It turns out a copy sealed thousands of generations back is refused outright (TooDistantInThePast), and on a mesh link nothing tells the sender. So it went out verbatim, failed, was retried, went out verbatim again, until its ladder ran out. In run 4 of the volume test 525 messages died exactly like that: 3,308 decrypt failures on the receiver and not one of them delivered. A resend that cannot decrypt is a wasted resend either way, so spend the generation and re-seal it. The exception stops at half the forward window, because the receiver going deaf is the failure the budget exists for, and a wasted resend is not. With the per-peer window this should essentially never fire on a mesh link. It's here for the paths the window does not govern.
Record the volume-test collapse and the five changes that fix it, and tell application teams the one thing they can see: a burst of more than 16 direct messages to a nearby peer now reports the rest as message_deferred with reason peer_window instead of message_sent, and sends them in order as acknowledgements return. An app that renders that deferral as a failure will look broken while working fine. Show it as queued.
When the outbox hit 500 entries it failed its least recently sent entry to admit any new one, whatever either of them was doing. In run 5 of the 2026-10-19 volume test a ten-minute Bluetooth outage turned that into about 2,500 terminal failures per phone, each one a message the app had shown as queued minutes earlier, thrown away so a *newer* message to the same unreachable peer could take its slot. Under load on a live link the same rule failed frames that were already on the wire, whose acknowledgements then settled nothing. That is backwards. Let's admit instead of evict. An app send to a full outbox is refused when its recipient holds the largest backlog, ties included: no entry, message_failed with "Outbox capacity exceeded" and zero attempts, and send still returns the id, exactly like the ack_capacity deferral. A send to a recipient with a smaller backlog evicts the heaviest recipient's oldest entry, so one dead peer's week of messages can't block a message to someone reachable. Nothing awaiting its acknowledgement, and nothing the per-peer window holds for a live mesh link, is ever evicted. Internal frames are never refused: they evict by the same rules and are admitted over the cap only while every entry is in delivery, which the acknowledgement table bounds. Media keeps its own cap. The existing eviction tests now force eviction through a lighter recipient, and the awaiting-ack case asserts the message survives and is delivered instead of being settled once.
Run 5 of the 2026-10-19 volume test lost ten minutes to a peer that was never gone. The iPhone held a link to Android's server; its random address rotated, Android saw a "new" device, dialled it, verified it as the iPhone and kept it. Two Link Layer connections between the same two radios. When the first one's write buffer stalled, the stall policy tore it down and the iPhone carried on over the second link for exactly ten seconds. Then the redial of the dead address hit the connect timeout. CoreBluetooth reports a cancelled connect as a disconnect, the identifier still mapped to Android, and the "wasn't connected" branch reported Android lost. The core refused every send to a peer the iPhone was connected to. It turns out no reconnect could have ended it either. Two devices hold one connection between them (Core 5.4, Vol 6 Part B, 1.1.1), and every iPhone dial came from the address Android's second link already held. Seven advertising addresses in a row timed out, and Android minted a new one every two minutes because its "forced BLE refresh" restarts the advertising set. The one dial that worked followed the iPhone's own address rotation. So: iOS never reports a peer lost, and never dials it, while the peer is connected and verified on another identifier or subscribed to our server from one; the dropped identifier is retired instead. Android treats a client link as a duplicate when the peer already holds a subscribed link to its server, and the periodic refresh restarts the advertisement only while no link is up. ADR 0030 records why both halves are needed. The write stall itself is still unexplained. It has only ever shown up with two links up, so the teardown now logs the peer's other live links and run 6 will say whether that suspicion holds.
Two Fixed entries for the false peer loss and the outbox admission
rule, and an upgrade note: a send to the recipient with the largest
backlog in a full outbox now gets message_failed ("Outbox capacity
exceeded", no attempts) for the id it returns, instead of an older
queued message failing to make room.
A full outbox refused every send to every recipient as soon as one slow mesh peer's held line filled it. The per-peer window parks the seventeenth message onward in the outbox, and those held entries were marked unevictable on a live link. So with 16 in flight and 484 held for one Bluetooth peer, a message to anyone else found nothing it was allowed to evict and was refused. That rule was protecting the wrong thing. A held message has never left the device. The thing that must never be evicted is a message awaiting its acknowledgement, because failing it leaves an ACK that settles nothing. Held messages are now evictable, from the back of their recipient's line, so the front (the part about to go out) survives and the conversation stays contiguous. The order comes from the line position, not a timestamp: a burst is held within a single millisecond, and ranking by time picked a random message. While at it, session control is no longer windowed. A key package, Welcome, confirmation probe or its answer is what heals the session the held frames are sealed under. Parking one behind sixteen frames the peer cannot decrypt waits for acknowledgements that will never come. That is a deadlock with extra steps. The MessageDeferred reference now lists the reasons it can carry, and says that a deferred message never gets message_sent.
The re-seal budget was a quarter of SENDER_RATCHET_MAXIMUM_FORWARD_ DISTANCE, 2,048 generations, on the assumption that every 1:1 receiver runs OpenMLS configured with 8,192. A leaf node does not. It runs mls-rs, and mls-rs hard-codes its receiver window at 1,024 (MAX_RATCHET_BACK_HISTORY in the secret tree) with no setting to change it. The group code even claimed the leaf was "configured with the same two numbers". It was not, and could not be. So a phone could legitimately burn 2,048 generations re-sealing for a leaf that gives up at 1,024, and walk it past its window with the very mechanism meant to prevent exactly that. This is not great. Name the leaf's window in the sealed layer, derive the budget and the dead-verbatim hard limit from the smaller of the two windows (256 and 512 now), and pin the number against mls-rs itself: the interop harness now sends a frame exactly 1,024 generations ahead, which must decrypt, and one 1,025 ahead, which must be refused. An mls-rs upgrade that moves the limit fails CI instead of a device. The smaller budget costs nothing in practice. The per-peer window caps unacknowledged growth at resends of sixteen frames, so the budget only trips after a whole retry ladder with no ACK, and the existing compile-time assertion still holds (176 < 256).
Review of the volume-test branch found that the bridge wiring, which CI only type-checks, undid several of its own fixes. These are all the same kind of bug, so they go in together, one paragraph each. The backed-up stop was global. Both drains stopped pulling from the core while *any* recipient sat at the 32-fragment mark, but the pop is one FIFO across peers. A peer whose link had just closed kept collecting fragments it could never send, and every other peer's egress waited until those expired, in 30-second rounds. Only a recipient with a link to drain into may stop the pull now. The iOS connect timeout captured only the peripheral. A timer from a dial that had already failed cancelled the healthy redial armed after it, and the cancel was reported as the peer lost. The timer is now bound to the attempt that armed it. A verified peer that walked away was redialled forever. The never-connected branch kept the identifier mapped, so the monitor dialled it every tick, the timeout cancelled it at 10 s, and every cycle re-reported the peer lost and restarted the advertisement. That branch now unmaps the identifier, so the monitor's 60-second advertisement window applies. The write-stall clock outlived the link it was timing. A refusal between the teardown verdict and the disconnect re-armed it, the identifier survives the reconnect, and the fresh link could be torn down on its first refusal. It is forgotten on connect and disconnect, and reset on stop. A subscribed central counted as a live link until didUnsubscribeFrom arrived. If that callback never came, the gate kept a departed peer a neighbour for good: it retired the real identifiers, refused every dial and reported nothing lost. A subscription counts only while its central has been heard from in the last minute. evictPeer had the opposite problem, evicting one of a peer's identifiers and reporting the peer lost while the other link stayed up. It now evicts them all. On Android the forced refresh never restarted a stopped advertisement while anything was linked, including an unverified central. It now goes through a small policy: restart when no identified peer is linked, or when the advertiser says it is not running. While at it: a failed indication is logged and paced instead of treated as a confirmation, the tiny-cap mark can no longer be zero, and the dead hasOtherClientLink is gone. A new Rust guard pins the wiring above by reading the Swift and Kotlin sources, since nothing else would notice it breaking.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. Seven device runs of the two-phone volume test (iPhone 11 and Galaxy M36, 5,000 DMs each way at 5 msg/s over BLE) drove this branch. Run 7, on the head commit with the three review fixes, delivered and acknowledged 10,000 of 10,000.
Device runs
sendMessagefailed sends already on the wire; Android went deaf past the forward window; BLE queues tore large messagesWhat changed
Engine
message_deferred(ack_capacity).process()runs every step on every tick. Two leaks that kept the table full are fixed.sendMessageno longer reports failure for a message it sent.TooDistantInTheFutureisSessionDesync; authenticated, so R2 is unchanged). The OpenMLS window rises to 8,192.message_deferred,peer_window) and sent in order. Session control is never held.ack_versionsentry 2 are coalesced, up to 5 per frame within 500 ms (ack_more).tools/mls-interoppins.Bridges
Demo app: shows the lock for restored sessions.
Review fixes (last three commits)
fix(protocol): let a full outbox evict never-sent held messages: one slow live peer could make every send to every other recipient a refusal.fix(protocol): size the re-seal budget for the leaf's receiver window: the budget assumed an 8,192 window that a leaf (mls-rs, fixed at 1,024) does not have.fix(bindings): stop the BLE bridges stalling and flapping peers: the global backed-up stop, the connect-timer binding, the redial loop, the stall clock, the stale-subscription gate,evictPeer, and the Android refresh. A Rust guard now pins this wiring.Verification
cargo fmt --all -- --check,cargo clippy --workspace -- -D warnings,cargo test --workspace --libandRUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-depspass; each of the three review commits passes the unit tests on its own.offline-protocol-sealedlints clean onthumbv8m.main-none-eabihf.tools/mls-interoppasses, including the two new leaf-window steps; with the constant set to 1,025 it fails.PeerStreamFramingTest, which cannot find its vector file there.Still owed: a three-device leg for the cross-recipient stall the two-phone harness cannot exercise. Open follow-ups: the iOS write-stall trigger (none fired in run 7), Android's indication-only egress ceiling, and the iPhone's futile dials to Android's rotating advertising address: the stack changed it 9 times in run 7 although the forced refresh kept the advertisement, and the iPhone timed out 92 dials to those addresses while already linked (no effect on delivery).
Checklist
cargo fmt --all -- --checkpassescargo clippy --workspace -- -D warningspassescargo test --workspacepassescargo-denyis satisfied (no dependency changes; not run locally)CHANGELOG.mdupdated where relevantunsafein core crates