Proposal: an event journal for wallet backup and recovery

[dcorral]

1. Summary

Today a backup is an encrypted archive of the whole wallet directory. It protects the wallet only as of the moment it was taken, and its cost grows with the wallet's age, so it cannot be taken at every step. Two states lose funds for good if the device is lost before the next archive: a paid blind-receive invoice, and the change of a broadcast send. We propose that rgb-lib can additionally keep a journal of the small, immutable inputs behind every state change, hand each one to the consumer as it happens, and rebuild a wallet from scratch from seed + indexer + journal. Storage location, encryption and durability policy remain the consumer's decision.

2. Current backup and recovery

Backup zips the whole wallet directory (stash, DB, every transfer ever made, media), encrypts it and writes one file. Restore unzips it, then the consumer opens the wallet normally and the consistency check runs on going online.

Strengths to preserve: one call each way, encrypted, byte-exact restore, offline, full history, works with Wallet::load.

Limitations:

• Protect anything that happened after the last archive.
• Be taken often: every archive re-ships the whole stash and every transfer directory ever created.
• Tell a consumer what changed or which files matter.
• Be ordered against other state: backup_info() is only a yes/no.

Known defects, independent of this proposal: the archive is built without a lock, captures the runtime lock file (a restored wallet blocks for an hour) and a BDK sidecar, and marks the wallet as backed up before the copy is made. No upstream test restores a wallet in any non-terminal state.

3. Failure scenarios with the current mechanism

Alice receives. Alice creates a blind-receive invoice and sends it to Bob. The blinding secret that makes the incoming allocation hers is stored only in the stash file. Bob pays. Alice's phone is lost before her next backup, and her last archive predates the invoice. After restore the payment is on chain, but her wallet cannot reveal the seal: the asset shows up and can never be spent. No chain scan recovers a random secret.

Bob sends. Bob sends 30 of his 100 units to Carol. At broadcast his wallet consumes its own fascia, the only record of where the 70 units of change went. His laptop dies before the next backup. After restore the wallet knows the 100 were spent but has no transition for the 70: the change is gone. Carol's consignment carries her 30, not his 70.

A node or an auto-sync wallet. To be recoverable at every moment it must archive the whole directory after every operation: O(wallet size) inside the hot path, racy between overlapping uploads, and still taken after the broadcast, so it does not save Bob.

With a journal, Alice's secret is a 100-byte event emitted when the invoice is created; Bob's fascia is a few-kilobyte event emitted before the broadcast; the node stores each event once, anywhere.

4. Proposed mechanism

4.1 Journal the inputs, not the derived state

The stash and the DB are materialized views, rewritten wholesale and always growing. The inputs behind them are few and small:

State change	Input to journal	Size
issue asset	contract consignment	KBs
media, issue or receive	the bytes, by digest	any, once
blind_receive	blinding secret + invoice	bytes
witness_receive	invoice + revealed script	bytes
send, inflate, burn, link	fascia + signed PSBT + outgoing consignments	KBs
receive, once validated	validated consignment	KBs
broadcast, confirmation, settlement, utxo creation, sync, addresses	nothing, chain + descriptors

Replaying these into an empty stash uses primitives rgb-lib already has. Rebuilding the DB view from stash + chain is the one new piece.

4.2 Event model

An event carries: a per-wallet sequence number, the wallet fingerprint, a kind (ContractImported, Media, SecretSeal, InvoiceIssued, TransferPrepared, TransferValidated, Checkpoint), a versioned payload, its digest, and a durability hint (Critical or Normal).

• Written in the same DB transaction as the state change: an event exists if the change was committed.
• seq is per wallet, monotonic, gap-free. Replay is idempotent.

4.3 Delivery modes

• Pull: the consumer asks for events after a given sequence number, stores them, then acknowledges up to the last one stored.
• Push: the consumer installs a sink, and rgb-lib calls it with each event as it happens.

Events stay in the outbox until acknowledged, so a crash between receiving and storing just replays them.

4.4 Durability before irreversible actions

When a sink is installed, Critical events are handed over before the irreversible action. If the sink fails, the operation aborts cleanly and can be retried.

Operation	Event	Handed over before
issue	contract	returning the asset
blind / witness receive	secret + invoice	returning the invoice
send	fascia + PSBT + consignment	posting to the proxy, and again before broadcast
receive	validated consignment	sending the ACK

Without a sink, consumers pull and see the same hints; the ordering is then their contract.

4.5 Recovery procedure

recover() takes the seed, an indexer and the events and produces an ordinary wallet directory that Wallet::new opens. It does the following:

• Validate the journal (version, fingerprint, gap-free seq, digests).
• Replay into an empty stash; in-flight sends and pending receives are left pending, not forced.
• Rescan the descriptors, rebuild the DB view from stash + chain, run the consistency check.
• Online, because consignment import needs a resolver; the wallet needs a rescan anyway.

4.6 Optional checkpoints

A Checkpoint event holds the minimal materialized view (DB, stash, referenced media, in-flight transfer files). Never required. Useful to prune old events, speed up recovery, or keep the full transfer history. export_events() backfills wallets that predate the feature.

5. Outcome for the scenarios in section 3

Alice. Fixed when the consumer installs a blocking sink. The blinding secret is a Critical event handed to the sink before the invoice is returned, so no invoice can reach Bob unless its secret is already stored. On recovery the secret is replayed into the stash and the pending invoice is rebuilt; Bob's consignment is either already in the journal (if Alice had validated it) or fetched again from the proxy; the allocation is spendable. Not fixed with pull-only delivery: rgb-lib returns the invoice immediately, so a consumer that has not pulled yet still has a window. The dependency on the proxy still holding a not-yet-fetched consignment is the same as today.

Bob. Fixed when a sink is installed: the fascia, PSBT and consignment are handed over before the proxy post and again before the broadcast, so a broadcast transaction always has its fascia stored. On recovery the transaction is found on chain, the fascia is consumed and the 70 units of change are back. With pull-only delivery it is fixed for the split flow, where the consumer can pull between send_end and the refresh that broadcasts, but not for the single-call send, which broadcasts before returning.

Node or auto-sync wallet. Fixed in full with a sink. Each operation produces one small event instead of an archive of the whole directory, and the fascia is durable before the broadcast rather than after it. This is the case the proposal is designed for.

What the proposal does not change. Consumers that do not enable the journal keep today's behaviour. Recovery requires an indexer. The settled-transfer history is restored only from a checkpoint. A consignment that was paid but never fetched still depends on the proxy retaining it.

6. Benefits

Benefit	Effect
No funds lost between backups	Alice's secret and Bob's fascia are durable before the invoice leaves the wallet and before the broadcast. Both recover.
Cost independent of wallet age	One send = one event of a few KB, emitted once. A node doing 100 payments a day journals a few hundred KB a day and never re-uploads.
Any backend, the consumer's rules	Same events to Google Drive, iCloud, a local file, or a node's own remote store with its own keys. rgb-lib gains no dependency, runtime or crypto policy.
One ordered history next to other state	A node that persists RGB events before acting on them can never restore channel state that references missing RGB state.
Recoverable states become tested	A live-versus-replay test after every scenario turns today's unchecked cross-store invariants into tested ones, for the archive path too.
The archive improves later, for free	backup() can become "journal + optional checkpoint": consistent, small, no lock file inside. Separate, also additive.

7. Implementation scope

Piece	New?	Note
outbox table, emission at every mutation site	yes	every site that bumps backup_info today, plus the rust-only stash mutators
pull API, sink trait, durability hints	yes	sync, object-safe, FFI-friendly
sink placement before irreversible actions	yes	only active when a sink is installed
event authentication tag, keyed from the master key; verified by recover()	yes	see section 9
event encryption helper, keyed from the wallet keys	yes	optional for consumers with their own scheme
recover()	yes	replay = existing runtime primitives
DB view rebuild from stash + chain	yes, the core	asset rows from the stash's contracts; colorings from the stash's assignments over our UTXOs; pending rows from unsettled events
checkpoint, backfill	optional
create assets/ and media_files/ lazily	yes	a restored wallet without them fails its next issuance

Risks and mitigations. A missed emission or incomplete rebuild is a silent gap: caught by the live-versus-replay test. Critical events add one round-trip on the hot path: only for consumers that install a blocking sink. Recovery is online and slower than unzipping: the wallet needs a rescan after any restore anyway. The minimum journal restores funds and pending state, not the full settled history: a checkpoint does. Pending blind receives from before the feature can be backfilled only if the stash exposes its secrets: to verify. Multisig already replays through the hub: the journal must not double-apply.

8. Compatibility

• backup(), backup_info(), restore_backup(): same signatures, same behaviour.
• Journal off by default. A wallet that never enables it pays nothing.
• Directory layout, stash, DB schema (plus one outbox table), consistency check: unchanged.
• Consumers happy with an occasional archive change nothing.

9. Security

The journal carries the same sensitive material as today's archive (blinding secrets, fascias, signed PSBTs, invoices), so it is handled with the same care, built into the mechanism:

• Encryption. rgb-lib provides an encryption helper keyed from the wallet's own keys, so an event can be encrypted before it leaves the wallet without the consumer designing any cryptography. Consumers with their own scheme may use it instead; the event API documents that events are sensitive and must be stored encrypted.
• Authentication. Every event carries an authentication tag keyed from the wallet's master key. recover() verifies the tag of each event and refuses a journal that was altered, whatever store it came from. Forged or substituted contracts, consignments and fascias would in any case fail validation against the chain on replay; the tag turns that into an early, explicit rejection.
• Availability trade-off. With a blocking sink, the consumer's store is on the path of invoices, sends and acknowledgements: if the store is unavailable, those operations fail instead of proceeding without a backup. This is the intended behaviour for consumers that opt into it; pull mode has no such dependency.

Compared with the archive, the journal is stronger on three points:

• It never contains extended public keys or descriptors. The archive does, and a compromised archive exposes the wallet's entire address history; a compromised journal does not.
• A corrupted or compromised item affects one event, not the whole backup.
• Sequence numbers let a consumer detect a stale or partial restore, which a boolean backup flag cannot.

[cryptoquick]

Love this idea

There might be some overlap with some problems I'm trying to solve with Carbonado: bitmask-stack/carbonado#32