[taayyohh]

Summary

Under a lossy BLE link with many frames in flight (federated training's gradient_chunk fan-out, control frames, CRDT shadow writes), the Tier 2 reseal was advancing the sender's MLS encryption ratchet one generation on every retry of an un-ACKed outbound DM. The resulting generation spread exceeded the receiver's SENDER_RATCHET_OUT_OF_ORDER_TOLERANCE = 32 window, late/reordered frames failed decryption with SecretTreeError::TooDistantInThePast, and the app saw them as INVALID_CIPHERTEXT with no actionable reason text. A 20-minute 2-phone federated-training session on 2026-10-19 reproduced this: iPhone (coord) ingesting from Pixel (worker) got 128 message_decryption_failed events against 45 message_delivered events, with 45,933 retry attempts in between.

This is the F3 fix shape from the diagnosis doc at docs/scratch/off-3247-mls-ciphertext-dive.md (the diagnosis-only PR #584 stays open as reference material).

Approach

• Add MlsManager::session_epoch(peer) -> Result<Option<u64>> as a thin read path (bypasses get_group_info's O(roster) leaf-verification walk).
• Capture an MLS session fingerprint (session_established_at_ms, epoch) at every OutboxReseal construction site (5 sites: send_message_with, forward_message, flush_pending, service_seal, data_sync). Store it on the in-memory OutboxReseal struct (which is #[serde(skip)] so this adds no persisted state).
• In reseal_resend_content, short-circuit to verbatim replay when both halves of the stored fingerprint equal the recipient session's current fingerprint. On a fresh re-seal, update the stored fingerprint to the new values.

Why both halves? Epoch alone cannot tell "same session still at epoch N" from "fresh session happens to be at epoch 0" — a torn-down-and-rebuilt 1:1 session restarts at epoch 0 under new group state. The establishment stamp (renewed on every create_session / join_session) distinguishes the two. A None on either side falls through to the pre-fix fresh re-seal, preserving Tier 2 recovery when state is genuinely uncertain (legacy outbox entries restored without memory provenance, MLS not initialized, lock poisoned, storage read fails).

Safety

• Verbatim replay of the same message_id is caught by the receiver's deduplicator at receive.rs:166 before OpenMLS is asked to decrypt, so no duplicate reaches the application layer.
• Dedup state has a 7-day TTL; a receiver whose dedup state was wiped by restart or eviction falling for a verbatim replay would hit SecretReuseError, which maps to the same message_decryption_failed surface the pre-fix code produced. No new silent-drop mode.
• The fresh re-seal path is unchanged for the session-rebuild case (test_reseal_on_resend_recovers_after_recipient_rekeys_to_new_epoch still green).

Tests

• New: test_reseal_replays_verbatim_within_same_session_and_epoch — pins the F3 shortcut: same (session, epoch) → verbatim bytes, same message id, still decryptable on the receiver's unchanged session.
• Unchanged (still green): test_reseal_on_resend_recovers_after_recipient_rekeys_to_new_epoch, test_reseal_is_noop_without_provenance.
• Two existing tests updated to reflect F3's correct behavior:
  • a_resent_or_restored_sync_frame_keeps_its_no_alert_marker — asserted the resend's content changed; the real invariant is that the no_alert marker survives, which it does under verbatim replay.
  • test_undecryptable_ack_reseals_once_and_never_reports_delivered — asserted the resend's content changed; within an unchanged session fingerprint the resend replays verbatim, and the verbatim ciphertext still decrypts on the receiver's session (first-time use of that generation).

Full lib test suite: 2064 passed; 0 failed on cargo test --package offline-protocol --lib. MLS suite: 149 passed; 0 failed.

Manual-test plan for the hardware retest

The decisive test is the one that reproduced the bug: a 2-phone federated-training run. On both phones use builds that pin to this SDK branch (or wait for merge + a mesh-sdk tag bump).

Setup:

• iPhone 12 as coord, Pixel/Seeker as worker (or any 2-phone combo).
• Both phones on the same companion-app build that bundles the fixed SDK.
• Clean install both apps so there's no stale outbox / MLS state.

Run:

1. Pair both phones to the training discovery flow.
2. Start a federated-training session, ≥ 8 rounds, ≥ 15 minutes total.
3. Keep the phones close enough that BLE drops are occasional (not constant). The point is a lossy-but-live link, not a dead one.

Check on the coord phone's debug log (ingesting worker's frames):

• Expect: message_decryption_failed events near zero for the full session. The 128/20min baseline should drop to single digits (ideally 0).
• Expect: message_delivered count climbs linearly with gradient_chunk emissions, not flat at ~45.
• Expect: zero group_epoch_fork_detected, zero secure_session_failed, zero group_error — same as pre-fix. The fix doesn't touch those paths.

If the metric drop isn't dramatic:

• Grep the IM app logs for "Decryption failed" tracing lines. The warn-level log at message_dispatch.rs:1312-1316 carries the OpenMLS error chain in logcat/oslog (not the JS console); if the remaining failures carry "An error occurred during AEAD decryption" instead of "Generation is too old to be processed", Hypothesis B (Welcome-wins race leftover) is in play and a separate fix is needed. If they carry "The requested secret was deleted" (SecretReuseError), the receiver's dedup is being bypassed somehow and that's a different investigation.

Refs

• Linear: https://linear.app/offlineprotocol/issue/OFF-3247
• Diagnosis PR (reference material, stays open): docs: diagnose MLS INVALID_CIPHERTEXT during 2-phone federated training (OFF-3247) #584
• Diagnosis doc: docs/scratch/off-3247-mls-ciphertext-dive.md on branch mb/off-3247-mls-ciphertext-decrypt-diagnosis

Hardware validation (2026-10-19)

Validated on a local tarball install (npm pack → npm install --no-save in IntelligenceMesh/companion-app/) running on iPhone 12 (coord) + Pixel SM02G4061956539 (worker). Scenario: 2-phone federated-training session running gemma-270m on rust-candle, same shape as the Day 10 session that originally reproduced the bug.

Pre-fix baseline (Day 10, 2026-10-09): session died within 20 min at ~45k retries, 128 message_decryption_failed events, 45 message_delivered, final state unresumable.

Post-fix today: session held through 8 outer-step rounds before the user stopped it. During the run the sync-coord gradient_chunk retry queue grew into the thousands under the same round-pacing conditions, but the MLS session never degraded:

• message_delivered events kept flowing on both directions throughout the run (training_join, round_start, loss_report, heartbeat, gradient_ack, in-flight gradient_chunk).
• Zero message_decryption_failed.
• Zero TooDistantInThePast / SessionNotReady / InvalidCiphertext surfaced on either phone.
• No ratchet exhaustion, no "session dead" state.

Specifically what F3 blocked: without this fix, each retry of an un-ACKed gradient_chunk advanced the sender ratchet one generation; the Day 10 retry volume alone would have pushed the delivered-generation spread past SENDER_RATCHET_OUT_OF_ORDER_TOLERANCE = 32 within minutes. Today the same retry volume accumulated with no generation advance, so late/reordered frames stayed inside the receiver's decryption window.

The orthogonal pre-existing issue (worker finishes trainStep after the coord's 15 s round-wait window → every round shows 0/1 peers accepted; skipping) is round-pacing, not MLS, and is being filed separately. F3 is the fix for the MLS failure mode exercised by that pacing problem under real load.

Note for reviewer: this validation was done against the fix as of commit 037da4 (now plus the fmt+rustdoc cleanup 38f467). The companion-app side has no changes required.