refactor: frame AOF and snapshot records with a checked length

ARCHITECTURE.md

@@ -171,7 +171,7 @@ persistence is per-shard:
 - each shard has its own snapshot file
 - recovery can run shard-by-shard in parallel
 
-the AOF format is binary and append-only. every record has a stable tag plus a CRC32 checksum. snapshots are also binary, include a header and footer checksum, and are written by streaming the live keyspace once.
+the AOF and snapshot files share one layout: a header with a version and flags, then one frame per record or entry. a frame holds its length twice, the second time inverted, then the body and a CRC32 of the body. that lets recovery tell a last frame cut short by a crash, which it drops, from damage elsewhere, which stops the start. snapshots store each key's expiry as a unix time, so a snapshot restored later doesn't bring back keys that expired in between. snapshots are written by streaming the live keyspace once.
 
 both snapshot writes and AOF rewrites use the usual safe pattern:
 

CHANGELOG.md

@@ -2,6 +2,10 @@
 
 ## unreleased
 
+### changed
+- new layout for AOF and snapshot files, version 5. every record and entry is a frame with its length stored twice and a CRC, so recovery tells a record cut short by a crash from damage in the middle of the file. a header flag marks encryption. files from earlier versions are no longer read (#450)
+- snapshots store each key's expiry as a unix time. they stored the TTL left and recovery subtracted the time since the file's mtime, so a snapshot restored from a backup brought back keys that had expired (#450)
+
 ### removed
 - concurrent mode: `--concurrent`, `EMBER_CONCURRENT`, the `concurrent` config key and the helm `ember.concurrent` value. the default mode is faster, serving about 35% more requests without pipelining and 2.6× more with a pipeline of 16 on a 32-core machine. concurrent mode also left strings out of the AOF and replication, rejected collection commands, and ran SET NX/XX and INCR as separate read and write steps (#446)
 - raft and the `openraft` dependency. raft only ever ran on the bootstrap node, and gossip with config epochs already decided slot ownership and failover, as in Redis Cluster. `raft-port-offset` and `--cluster-raft-port-offset` are still accepted and ignored, so existing configs keep working (#443)

crates/ember-core/src/shard/persistence.rs

@@ -93,10 +93,10 @@ pub(super) fn handle_serialize_snapshot(
 ) -> ShardResponse {
     let entries: Vec<SnapEntry> = keyspace
         .iter_entries()
-        .map(|(key, value, expire_ms)| SnapEntry {
+        .map(|(key, value, ttl_ms)| SnapEntry {
             key: key.to_owned(),
             value: value_to_snap(value),
-            expire_ms,
+            expires_at_ms: expires_at(ttl_ms),
         })
         .collect();
 
@@ -193,6 +193,14 @@ pub(super) fn handle_snapshot(
     }
 }
 
+/// Turns the TTL left, or -1 for none, into the unix ms time a snapshot
+/// stores, or 0 for none.
+fn expires_at(ttl_ms: i64) -> u64 {
+    u64::try_from(ttl_ms).map_or(0, |ms| {
+        ember_persistence::aof::unix_now_ms().saturating_add(ms)
+    })
+}
+
 /// Converts a `Value` reference to a `SnapValue` for serialization.
 pub(super) fn value_to_snap(value: &Value) -> SnapValue {
     match value {
@@ -298,7 +306,7 @@ pub(super) fn write_snapshot(
         writer.write_entry(&SnapEntry {
             key: key.to_owned(),
             value: value_to_snap(value),
-            expire_ms: ttl_ms,
+            expires_at_ms: expires_at(ttl_ms),
         })?;
         count += 1;
     }

crates/ember-persistence/README.md

@@ -6,34 +6,37 @@ each shard gets its own persistence files (`shard-{id}.aof` and `shard-{id}.snap
 
 ## what's in here
 
-- **aof** — append-only file writer/reader with CRC32 integrity checks. binary TLV format, configurable fsync (always, every-second, OS-managed). gracefully handles truncated records from mid-write crashes
-- **snapshot** — point-in-time serialization of an entire shard's keyspace. writes to a `.tmp` file first, then atomic rename to prevent partial snapshots from corrupting existing data
-- **recovery** — startup sequence: load snapshot, replay AOF tail, skip expired entries. handles corrupt files gracefully (logs warning, starts empty)
-- **encryption** — optional AES-256-GCM encryption for AOF records and snapshot files (enabled with `--features encryption`). accepts a 32-byte key file (raw bytes or hex). existing plaintext files are read transparently and migrated on the next rewrite/snapshot
-- **format** — low-level binary serialization helpers: length-prefixed bytes, integers, floats, checksums, header validation
+- **aof**: the append-only file writer and reader. the writer queues records in memory and writes each batch with one call. a write that fails part way is cut back to the last whole record. fsync runs after every write, once a second, or when the OS decides
+- **snapshot**: a point-in-time copy of one shard's keyspace. it is written to a `.tmp` file, fsynced, then renamed into place, so a crash while writing leaves the previous snapshot as it was
+- **recovery**: the startup sequence. it loads the snapshot without the keys that have expired, then replays the AOF records written after it. a file that can't be read stops the start with an error and is left untouched. a partial record at the end of the AOF, left by a crash, is cut off
+- **frame**: the file header and the record framing that both files share
+- **encryption**: optional AES-256-GCM (`--features encryption`) with a 32-byte key file, as raw bytes or hex. each file is encrypted with its own key, derived from the master key and a salt in the file's header. plaintext files are still read, and are rewritten encrypted at startup
+- **format**: binary helpers for length-prefixed bytes, integers, floats and checksums
 
 ## file formats
 
-**AOF** — `[EAOF magic][version][record...]` where each record is `[tag][payload][crc32]`
+every file starts with `[magic: 4B][version: 1B][flags: 1B]`, followed by a 16-byte salt when the encrypted flag is set. the current layout is version 5, and files in other versions are refused. records and snapshot entries are stored as frames:
 
-supported record types: SET, DEL, EXPIRE, PERSIST, PEXPIRE, INCR, DECR, INCRBY, DECRBY, APPEND, RENAME, COPY, LPUSH, RPUSH, LPOP, RPOP, ZADD, ZREM, HSET, HDEL, HINCRBY, SADD, SREM (plus VADD, VREM, PROTO.SET, PROTO.REGISTER when compiled with those features)
+```text
+[len: 4B][!len: 4B][body][crc32 of body: 4B]
+```
+
+the second copy of the length, inverted, lets a reader tell a last frame cut short by a crash, which it drops, from a damaged length, which is an error. with encryption, the body is `[nonce: 12B][ciphertext with its 16-byte tag]`.
 
-**snapshot (v2)** — `[ESNP magic][version][shard_id][entry_count][entries...][footer_crc32]` where entries are type-tagged (string=0, list=1, sorted set=2, hash=3, set=4). v1 snapshots (no type tags) are still readable.
+**AOF** (`EAOF`): the header, then one frame per record.
 
-**encrypted formats (v3)** — when an encryption key is configured, AOF records and snapshot entries are wrapped with AES-256-GCM: `[nonce (12 bytes)][ciphertext][auth tag (16 bytes)]`. the file headers and version bytes remain plaintext so the reader can detect encrypted files and select the right decoding path. plaintext v2 files are still readable even with a key configured.
+supported record types: SET, DEL, EXPIRE, PERSIST, PEXPIRE, INCR, DECR, INCRBY, DECRBY, APPEND, RENAME, COPY, LPUSH, RPUSH, LPOP, RPOP, ZADD, ZREM, HSET, HDEL, HINCRBY, SADD, SREM (plus VADD, VREM, PROTO.SET, PROTO.REGISTER when compiled with those features)
+
+**snapshot** (`ESNP`): `[header][shard_id: 2B][one frame per entry][empty frame][entry count: 4B][crc32: 4B]`. each entry holds the key, a type-tagged value (string=0, list=1, sorted set=2, hash=3, set=4, protobuf=5, vector=6) and the unix time in milliseconds when it expires, or 0 for never. the trailing CRC covers the entry frames and identifies the snapshot: the AOF's checkpoint record names the snapshot it follows by this CRC.
 
 ## usage
 
 ```rust
-use ember_persistence::aof::{AofWriter, AofRecord};
-use ember_persistence::snapshot::{SnapshotWriter, SnapshotReader, SnapEntry, SnapValue};
-use ember_persistence::recovery::recover_shard;
-
-// recovery is typically called by the shard on startup
-let result = recover_shard(data_dir, shard_id);
-for entry in result.entries {
-    // insert into keyspace...
-}
+use ember_persistence::recovery::{recover_shard, Recover};
+
+// the shard implements `Recover` with its keyspace, so recovered entries
+// and replayed records go through the same code as live writes
+let result = recover_shard(data_dir, shard_id, &mut keyspace)?;
 ```
 
 ## related crates