feat(backfill): peer-to-peer transcript backfill request/response (S1 of #47) - #1335
Open
joelteply wants to merge 1 commit into
Open
feat(backfill): peer-to-peer transcript backfill request/response (S1 of #47)#1335joelteply wants to merge 1 commit into
joelteply wants to merge 1 commit into
Conversation
… of #47) A message published while a peer is unreachable is LOST, not queued — measured three times on 2026-08-08, once inside the investigation of it. Replay resumes from the RECEIVER's cursor against the RECEIVER's store, so a frame that never arrived cannot be replayed and the receiver cannot know it missed one. PULL, not an outbox. An outbox needs new per-event durable state AND an eviction decision (a queue growing while a peer is away a week is its own incident). Pull needs neither: the sender already has the events in its durable transcript, the receiver already holds a per-room cursor. Backfill is the daemon's existing resume, crossing a peer boundary instead of a process one. DeliveryLedger cannot serve this — it is per-peer COUNTERS, so it knows a peer has not acked but not WHICH events it missed. Carried on the existing command bus (request/reply/await_reply over FrameKind::Command); nothing new invented on the wire. Receive-side dedup already exists, so an overlapping page costs bandwidth, never correctness — which is what lets the asker be sloppy about its cursor. Extracted command_bus::reply_addressing: every responder needs (reply_to, correlation_id) off a request, and each one re-deriving it from raw headers is how they drift. `truncated` is on the response because a bounded page presented as a complete one lets the gap survive the mechanism built to close it — worse than no backfill, because it looks fixed. NOTHING CALLS THIS YET. The reconnect watcher is S2; landing them separately keeps this slice incapable of changing existing behaviour. Validated on ALL targets per [[validate-lib-tests-before-push]], not just --lib. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
S1 of store-and-forward (#47). Nothing calls this automatically yet — landing it separately keeps this slice incapable of changing existing behaviour. The reconnect watcher is S2.
The failure
A message published while a peer is unreachable is lost, not queued.
airc msgreportsreached 0 of 76 enrolled peers — NONE currently connectedand that's the end of it. Measured four times on 2026-08-08 — twice by accident, once inside the investigation of it while the peer was down for anairc update, and once more during the blackout drill.Why lost rather than pending: replay resumes from the RECEIVER's cursor against the RECEIVER's store. That's exactly why a daemon restart loses nothing — the events are already local. Across a peer boundary they aren't, so a frame that never arrived can't be replayed, and the receiver has no way to know it missed one.
Pull, not an outbox
A sender-side outbox needs new per-event durable state and an eviction decision — a queue growing while a peer is away for a week is its own incident.
Pull needs neither. The sender already has every event in its durable transcript; the receiver already holds a per-room cursor. Backfill is "give me events on this channel since my cursor" — the daemon's existing resume, crossing a peer boundary instead of a process one. Nothing new is stored, so nothing new needs evicting.
DeliveryLedgercan't serve this role, incidentally: it's per-peer counters (attempts/acks/attempts-since-ack), so it knows a peer hasn't acked but not which events it missed. Useful as a trigger, useless as a queue.Ask always; don't detect
A receiver can't detect a gap whose frames never arrived — absence leaves no trace. So the contract is to ask on every peer-connect. When the cursor is current the answer is an empty page, which is cheap. Detection is the part with no signal; asking is the part that's always safe.
Duplicates are free: events carry a stable
event_idand the receive path already dedups. An overlapping page costs bandwidth, never correctness.Shape
Carried on the existing command bus (
request/reply/await_reply) — nothing new invented on the wire. Extractedcommand_bus::reply_addressingbecause every responder needs(reply_to, correlation_id)off a request, and each one re-deriving it from raw headers is how they drift.truncatedis on the response because a bounded page presented as a complete one lets the gap survive the mechanism built to close it — worse than no backfill, because it looks fixed.Validated on ALL targets, not just
--lib— that narrower scope is what let a regression reach CI earlier today.🤖 Generated with Claude Code
https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc