diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..2ec093e --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,495 @@ +# Hedwig Architecture + +> Covers the server as of the `log-queue` branch. The durable log queue is the +> default subject; the legacy filesystem path is described where it +> differs. Design rationale lives in [the log-queue plan](docs/plans/2026-07-20-durable-log-queue.md); this document describes +> what is actually built and where. + +## 1. The big picture + +Hedwig is a single-process MTA in two crates: + +- `smtp/` — the SMTP protocol library: session state machine, parser, TLS + (implicit + STARTTLS), timeouts. It owns the wire; everything else is + reached through the `SmtpCallbacks` trait. +- `smtp-server/` — the server: callbacks, queue storage, delivery workers, + DKIM, MTA-STS, rate limiting, metrics, CLI. + +The central idea of the log-queue design is **separating two rates that have +nothing to do with each other**: + +- *Inbound* is bounded by how fast complete messages can be appended to disk. +- *Outbound* is bounded by DNS, remote MTAs, and politeness rate limits. + +The durable log sits between them. Acceptance (`250 OK`) waits only for the +message to be written into the kernel page cache; delivery happens whenever +the outbound side gets to it. + +```mermaid +flowchart LR + subgraph inbound [Inbound] + C[SMTP clients] --> L[smtp crate
session + parser] + L --> CB[callbacks
process_email_log] + end + subgraph queue [Durable log queue] + CB -->|"append(msg)"| W0[writer shard 0] + CB -->|append| W1[writer shard N] + W0 --> S0[(segments +
journal shard 0)] + W1 --> S1[(segments +
journal shard N)] + S0 & S1 -.->|discovery cursors| D[dispatcher
one task] + end + subgraph outbound [Outbound] + D -->|claims| LW[log workers] + LW -->|read body
by location| S0 + LW --> MX[remote MTAs] + LW -->|"outcome (delivered /
deferred / bounced / rate-limited)"| D + end + D -->|persist-then-apply| S0 +``` + +With `storage_type = "fs"` the old path is used instead: +`storage.put()` → bounded `async_channel` → channel workers → periodic +deferred-directory scans. That path is untouched and is why the log queue is +a separate, selectable backend (`storage_type = "log"`). + +Module map: + +| Area | Files | +|---|---| +| Record format | `smtp-server/src/logqueue/record.rs` | +| Segments | `smtp-server/src/logqueue/segment.rs` | +| Shard/spool layout, lock | `smtp-server/src/logqueue/{shard,spool}.rs` | +| Append writers | `smtp-server/src/logqueue/writer.rs` | +| Journal + checkpoints | `smtp-server/src/logqueue/state.rs` | +| Dispatcher (scheduling, GC, compaction) | `smtp-server/src/logqueue/dispatcher.rs` | +| Delivery workers | `smtp-server/src/worker/{mod,log_worker}.rs` | +| Acceptance path | `smtp-server/src/callbacks.rs` (`process_email_log`) | +| Wiring & shutdown | `smtp-server/src/main.rs` | +| Operator CLI, migration | `smtp-server/src/{queue_cli,migrate}.rs` | + +## 2. On-disk layout + +```text +/ + bounced/ # bounce archive (legacy fs format, retention-cleaned) + spool/ + format-version # "1" + .lock # exclusive flock, held for the server's lifetime + shard-0000/ + segment-000000000007.log # sealed payload segments (immutable) + segment-000000000009.log + segment-000000000010.open # the one active append target + journal-000000000003.log # state journal (current) + checkpoint # latest checkpoint (atomic rename) + shard-0001/ ... +``` + +Rules the layout encodes: + +- One shard per append writer; a shard is owned by exactly one writer task. +- At most one `.open` segment per shard. Sealing renames `.open → .log`; + sealed segments are immutable forever. +- Segment ordinals are never reused; the active segment always has the + shard's highest ordinal. +- The number of files is proportional to *live backlog + unreclaimed + garbage*, never to historical volume — fully dead segments are unlinked. +- Changing `append_writers` requires an empty queue (checked at startup in + `Spool::open`). + +## 3. Payload record format + +Records are self-framing and versioned (`record.rs`). Everything the +scheduler needs is in the header, so discovery never reads message bodies. + +```text +offset size field + 0 4 magic "HWLQ" + 4 2 format version (=1) + 6 2 flags (reserved, zero) + 8 4 record_len — total size; scanner skips to next record with this + 12 4 header_len — body starts here + 16 4 header_crc — crc32 over [0..16) ++ [20..header_len) + 20 4 payload_crc — crc32 over body + 24 16 message id (binary ULID) + 40 8 enqueue timestamp (unix ms) — survives relocation, drives age metrics + 48 4 relocation generation — bumped when compaction copies a record + 52 4 per-segment ordinal + 56 var envelope: sender, recipient list (u16-length-prefixed strings) + … var body (record_len - header_len bytes) +``` + +The two checksums serve different failures: `header_crc` catches torn or +corrupt headers during scans; `payload_crc` is verified on every body read so +bit rot in a sealed segment surfaces at delivery time, not silently. + +**Tail policy** (`segment.rs`): on recovery the active segment is scanned +from 0; the first invalid position truncates the file (a torn tail is mail +that was never fully acknowledged, or falls inside the accepted power-loss +window). Corruption *inside a sealed segment* is a hard error, never a skip — +sealed data was durable and complete, so damage there means something is +wrong enough that a human should look. + +## 4. Acceptance path (SMTP → disk) + +```mermaid +sequenceDiagram + participant C as client + participant S as smtp session + participant CB as callbacks + participant AH as AppendHandle + participant W as shard writer task + participant DI as dispatcher + + C->>S: DATA ... . + S->>CB: on_data(email) + CB->>CB: disk reserve check (452 if breached) + CB->>AH: append(AppendMessage) + AH->>AH: acquire byte permits (pending_append_bytes) + AH->>W: mpsc send (shard = hash(ulid) % N) + W->>W: encode record, write_all to active segment + W->>W: advance committed head (chain mutex) + W-->>DI: Notify (lossy hint) + W-->>AH: JobLocation via oneshot + AH-->>CB: Ok + CB-->>S: Ok + S-->>C: 250 OK +``` + +Key properties (`writer.rs`): + +- **Admission is byte-bounded, not count-bounded.** A tokio semaphore sized + by `pending_append_bytes` is acquired for the encoded record size before + queueing; it frees once the bytes reach the page cache. This is real + backpressure against disk throughput, never against outbound speed. +- **Publish ordering per record:** write → advance committed head → notify → + complete the SMTP future. The committed head can never expose a partial + record, so concurrent readers below the head are always safe. +- **Rotation is seal-first.** When a record would overflow + `segment_target_bytes`: seal (rename) the active segment, *then* create the + next one. A crash in between leaves zero `.open` files (recovery creates + one); create-first could leave two, which is an unrecoverable layout error. + If the create fails, `active` becomes `None` and the next append retries it + rather than ever writing into the sealed file. +- Writers run on `spawn_blocking` threads and own their files exclusively — + no shared append offsets, no locks on the hot path except the tiny chain + mutex. + +What the writer publishes to the dispatcher is the **chain**: an ordered list +of `SegmentHead { segment, committed, sealed }`, last entry = active. The +chain is authoritative; the `Notify` is an optimization that may be lost or +coalesced — a 500ms safety tick re-checks regardless. + +## 5. Dispatcher: one task, all scheduling + +The dispatcher (`dispatcher.rs`) is a single tokio task owning every +scheduling structure. Nothing else mutates them, which is what makes the +claim/outcome protocol race-free without fine-grained locking. + +```text +Dispatcher state +├── jobs: HashMap # location, attempts, remaining rcpts, state +├── ready: BinaryHeap<(enqueue_ms, id)> # oldest-first dispatch order +├── delayed: BinaryHeap<(due_ms, id)> # deferred retries + rate-limit holds +├── waiting: VecDeque # parked workers wanting work +└── per shard: + ├── cursor (segment, offset) # next undiscovered position + ├── tombstones: segment -> {ids} # terminal records, filter for re-scans + ├── stats: segment -> {total/dead bytes} # GC accounting + └── ShardStateStore # journal + checkpoint writer +``` + +### 5.1 Discovery + +On a notify or the safety tick, each shard's cursor is advanced through the +chain: read headers from `cursor` to the committed head, register unknown +message ids as `Ready`, hop to the next chain entry when a sealed segment is +exhausted (pruning consumed entries). Two filters apply during the scan: + +- ids tombstoned in that segment are skipped (they went terminal after a + checkpoint but before this re-scan); +- an id that is already tracked but appears with a **higher relocation + generation** is a compaction copy whose journal entry was lost in a crash — + the dispatcher re-journals the relocation so the accounting becomes durable. + +**Backpressure:** discovery stops while `jobs` holds `max_tracked_jobs` +(100k) entries. The log itself holds everything beyond the window, so memory +is bounded by config while the backlog is bounded only by disk. This applies +during recovery too — a million-message backlog does not OOM the process. + +### 5.2 Job lifecycle + +```mermaid +stateDiagram-v2 + [*] --> Ready: discovered from log /
recovered from checkpoint + Ready --> InFlight: claim (generation g) + InFlight --> Ready: worker abandoned claim
(drop without report) + InFlight --> Delayed_p: Deferred outcome
(persisted, attempts+1) + InFlight --> Delayed_m: RateLimited outcome
(memory only, no attempt) + Ready --> Delayed_m: dispatch gate:
domain exhausted + Delayed_p --> Ready: due time reached + Delayed_m --> Ready: due time reached + InFlight --> [*]: Delivered / Bounced
(journaled, tombstoned) + + Delayed_p: Delayed (persisted defer) + Delayed_m: Delayed (in-memory hold) +``` + +Workers pull with `DispatcherHandle::claim()`. A claim carries a +monotonically increasing **generation**; outcomes and abandonments quote it, +and anything stale is ignored — a hung worker's late report can never clobber +a reassigned job. Dropping a `Claim` without reporting sends an abandonment +(via its `Drop` impl), so a panicking worker returns its job to `Ready` +automatically; a claim can never leak. + +The job a worker receives is payload-free: id, location, attempts, sender, +remaining recipients. The body is read separately by position +(`read_body`), with its checksum verified on every read. + +### 5.3 Rate limiting: gate + acquire + +Two checks share one token-bucket `RateLimiter`: + +1. **Dispatch gate** (dispatcher, `peek_sync`): non-consuming. If the first + remaining recipient's domain has no tokens, the job goes to the in-memory + delay heap instead of wasting a worker slot. A due time is not a token + reservation — the job re-passes the gate when it wakes, which prevents a + thundering herd on one domain. +2. **Worker acquire** (`check_rate_limit`): consuming, immediately before + transmission — the authoritative check. Losing this race reports + `RateLimited`, which requeues in memory only: **no attempt increment, no + journal write**. Local throttling is not a delivery failure. + +## 6. Persistent state: journal + checkpoint + +A payload record implies `Ready` unless superseded — so acceptance costs one +write, not two. Everything that changes afterwards is an entry in the shard's +state journal (`state.rs`): + +```text +DEFERRED { id, location, attempts, next_attempt_ms, remaining_recipients, last_error } +DELIVERED { id, location, timestamp } +BOUNCED { id, location, timestamp, reason } +RELOCATED { id, old_location, new_location } # written by compaction +``` + +Entries are `[len][crc][payload]`-framed; the journal uses the same +page-cache durability as payloads, and the same torn-tail truncation policy. + +**Persist-then-apply** is the ordering rule everywhere: the dispatcher writes +the journal entry first and mutates its in-memory state only after the write +succeeds. A failed write parks the entry in a retry queue with the job left +in-flight — a job is never marked terminal, and never dropped from +scheduling, on the strength of an unpersisted transition. + +### 6.1 Checkpoints + +Journals would grow forever, so once a shard writes +`checkpoint_interval_bytes` of journal it snapshots. A checkpoint is +**self-sufficient for every segment still on disk**; without that, deleting +journal history could resurrect delivered mail (payload implies Ready!). It +contains: terminal tombstones per live segment, the discovered ready set +(with attempts *and remaining recipients*), the deferred set, per-segment GC +stats, the discovery cursor, and the journal position it covers. + +The write sequence keeps the dispatcher unblocked and the disk safe: + +```text +1. fsync current journal # nothing covered may be less durable than the checkpoint +2. rotate: start journal N+1 # entries during the snapshot land after the cut +3. snapshot in-memory state # cheap, synchronous +4. spawn_blocking: + write checkpoint.tmp -> fsync -> rename -> fsync dir +5. on success: delete journals <= N # the only place journal history dies +``` + +A crash anywhere before step 5 is safe: the old checkpoint plus the complete +journal chain reproduces the same state. Step 5 is a *destructive boundary*, +which is why steps 1 and 4 fsync (see §8). + +### 6.2 Recovery + +```mermaid +flowchart TD + A[Spool::open
lock + version + shard-count check] --> B[per shard: validate active tail
truncate torn suffix] + B --> C[load checkpoint
crc-verified] + C --> D[replay journals newer than checkpoint
ordered; torn tail of newest truncated] + D --> E[reconcile with validated chain:
clamp cursor to committed head,
drop jobs whose payload died with the tail] + E --> F[dispatcher starts:
ready/deferred/tombstones/stats live] + F --> G[discovery resumes from cursor
= lazy payload reconciliation,
backpressure applies] + G --> H[listener binds; workers pull] +``` + +Restart semantics: terminal stays terminal; deferred keeps attempts, due +time, and remaining recipients; everything else — including jobs that were +in-flight at the crash — becomes ready and is redispatched (at-least-once). +There is no "feed the whole backlog through a channel" step; startup cost is +checkpoint size + journal delta, not queue depth. + +Two guards worth knowing: journals must form a contiguous chain starting at +the checkpoint's position (or ordinal 1 when there is no checkpoint) — a gap +is a hard error, because silently lost journal history can resurrect +delivered mail. And recovery scans always use the *format's* maximum record +size, not the configured one, so lowering `max_message_size` can never make +previously accepted mail look corrupt. + +## 7. Delivery (workers) + +`LogWorker::run` is a pull loop: `claim() → read_body() → process_claim() → +report(outcome)`. `Worker::process_claim` (in `worker/mod.rs`) shares the +per-recipient delivery core with the legacy path (`deliver_recipient`: MX +lookup, MTA-STS policy, transport attempts, outcome classification) but +differs deliberately: + +- **Per-recipient accounting.** Delivered recipients leave the remaining + set; only transiently-failed ones are retried. A partial multi-recipient + failure re-sends only to recipients that have not accepted the message + (the remaining set is persisted in the DEFERRED entry and survives + restarts and checkpoints). +- **No sleeping in worker slots.** The legacy path sleeps on rate limits + inside the worker; the log path reports and moves on. +- **Retry budget**: `attempts >= max_retries` bounces terminally. +- **Bounce archive**: before reporting `Bounced`, the message is written to + `/bounced/` in the legacy one-file format, so operators keep the + same inspection workflow and `[storage.cleanup]` retention applies. + Archive failure is logged but never blocks the bounce. + +Backoff is `60s × 2^attempts`, capped at 24h — same curve as the legacy +deferred worker, but scheduled by the dispatcher's due-time heap instead of a +30-second directory scan. + +## 8. GC, compaction, and the durability model + +### 8.1 Reclamation + +Per-segment accounting is byte-based: `total_bytes` (known once sealed — it +is the file length) and `dead_bytes` (accumulated as records go terminal or +get relocated; tombstone sets make it idempotent). + +- **Deletion** is event-driven: the terminal transition that makes + `dead_bytes == total_bytes` deletes the segment immediately (fsync journal + → unlink → drop tombstones/stats/reader → remove from chain). A tick-driven + sweep backstops the one miss window (a segment whose last record dies + before the dispatcher observed its seal). At high delivery rates this is + the dominant path: burst segments die whole and are unlinked without any + copying. +- **Compaction** handles segments pinned by long-deferred stragglers: sealed, + past `compaction_min_age`, `dead_ratio ≥ compaction_dead_ratio` (default + 0.5, ≈2× amplification bound), fully below the discovery cursor. One runs + at a time, driven in small batches from the dispatcher tick. + +```mermaid +flowchart TD + A[pick source segment
sealed, old, ≥50% dead] --> B[snapshot its live ids
skip in-flight claims] + B --> C[per record: read old copy] + C --> D[re-append via the normal writer
generation+1, original enqueue_ms] + D --> E[journal RELOCATED old→new
persist-then-apply] + E --> F[apply: job.location = new,
old copy counted dead] + F --> G{source fully dead?} + G -- yes --> H[fsync journal → unlink source] + G -- not yet --> I[left for next sweep
ratio still high → re-picked] +``` + +Because relocated copies flow through the ordinary append path they are +discovered like new records; the relocation generation resolves every +"which copy wins" question, including after a crash that leaves both copies +on disk. In-flight records are skipped rather than relocated, which is what +makes stale-read refetch protocols unnecessary: a segment is only ever +unlinked when nothing live — and therefore nothing readable — remains in it. +A terminal outcome racing the copy cannot resurrect the message: the ordered +journal replay marks whichever copy lost as garbage. + +### 8.2 What is durable when + +| Event | Guarantee | +|---|---| +| Process crash / SIGKILL | Nothing acknowledged is lost. Page cache survives the process; journals and segments replay. | +| Machine crash / power loss | Recently acknowledged mail (page cache not yet written back) may be lost. **Documented, accepted tradeoff** — this is why acceptance is fast. | +| Any crash, old queued mail | Never lost. Destructive operations (checkpoint truncation, segment deletion, compaction publication) fsync before removing anything, so power loss can only eat the recent write window, never history. | +| Remote accepted, crash before journal | The message is redelivered — at-least-once, duplicates possible, loss not. | + +The rule of thumb encoded throughout: **fsync is reserved for the moments we +destroy something**; everything additive rides the page cache. + +## 9. Startup, shutdown, config + +`main.rs` branches on `storage_type == "log"`: + +- Startup order: spool (lock lives until exit) → writers (tail validation) → + per-shard state recovery → dispatcher → log workers → listeners. No replay + channel; the deferred-scan worker is not spawned. The cleanup task runs + with `deferred_retention` forcibly disabled so it can never eat an + unmigrated legacy spool sitting at the same `base_path`. +- Shutdown: cancel token stops listeners → workers finish their current + claims and exit when the dispatcher stops handing out work → dispatcher + drains in-flight outcomes, persists them, checkpoints every shard → append + admission closes last (pending appends flush) → spool lock releases. + +Config (`[queue]`, all optional): + +```toml +[queue] +append_writers = 1 # shards; change requires an empty queue +pending_append_bytes = 134217728 # admission buffer bound (bytes, not msgs) +segment_target_bytes = 67108864 # seal threshold; must fit max_message_size +compaction_dead_ratio = 0.5 +compaction_min_age = "60s" +disk_reserve_bytes = 1073741824 # 452-reject below this free space +checkpoint_interval_bytes = 8388608 +``` + +## 10. Operator tooling + +Binary segments are not `ls`-able, so the CLI is part of the design: + +```sh +hedwig queue list --spool /spool # live messages: state, age, attempts, due +hedwig queue show --spool /spool ID # envelope, state history, location +hedwig queue stats --spool /spool # per-segment live/dead, ratios +hedwig queue migrate --config # one-time legacy fs-spool migration +``` + +`list/show/stats` are strictly read-only and safe against a live spool: they +take no lock, never truncate torn tails, and never create files. `migrate` is +restart-safe (idempotent by message-id dedup against the target spool), +preserves attempts/due-times/last-errors, verifies every record after the +writers flush, and then renames `queued/`/`deferred/` to timestamped +`.migrated-*` backups — it never deletes anything. + +## 11. Measured behavior + +Same machine, 1 KiB messages, 16 connections (see docs/plans/2026-07-20-durable-log-queue.md §27 for the full +plan; these are the two headline experiments): + +| Scenario | fs backend | log backend | +|---|---|---| +| Outbound disabled (pure ingest, tmpfs) | 85–93k msg/s | 97–105k msg/s | +| Outbound via a 500 ms/message MTA, 64 workers | **117 msg/s** (p99 accept 548ms) | **170,411 msg/s** (p99 accept 0.28ms) | +| Outbound delivery rate in that test | ~121 msg/s | ~121 msg/s | + +The second row is the design goal made visible: with slow outbound the legacy +bounded channel fills and acceptance collapses to the outbound drain rate, +while the log backend keeps accepting at disk speed (5.97M messages parked in +6.7 GiB) and delivers at exactly the same outbound rate. Sustained ingest +reclaims itself: a 30s full-throughput run writes ~3.5 GiB and ends with only +the active segment on disk. + +## 12. Invariants (the list to check before changing anything) + +1. The committed head never exposes a partial record. +2. A record never spans segments; the active segment has the shard's highest + ordinal; `.open` files number 0 or 1 per shard. +3. Terminal or relocated state is applied in memory only after its journal + entry is written (persist-then-apply). +4. Journal history is deleted only below a durable (fsynced + renamed) + checkpoint; checkpoints are self-sufficient for every segment still on + disk. +5. Segments are unlinked only when nothing live remains in them and the + journal covering their deaths is fsynced. +6. Recovery scans use the format's absolute record bound, never the + configured one. +7. Claims carry generations; a stale generation's outcome is ignored. +8. Rate-limit holds are never persisted and never count as attempts. +9. Relocation preserves message id and enqueue timestamp and bumps the + generation; the highest valid generation wins everywhere. +10. Locations are always explicit (shard, segment, offset) — nothing may + re-derive a shard from the current writer count. diff --git a/Cargo.lock b/Cargo.lock index d8f972d..42fe3d0 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -159,15 +159,6 @@ dependencies = [ "syn", ] -[[package]] -name = "atoi" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f28d99ec8bfea296261ca1af174f24225171fea9664ba9003cbebee704810528" -dependencies = [ - "num-traits", -] - [[package]] name = "atomic-waker" version = "1.1.2" @@ -251,9 +242,6 @@ name = "bitflags" version = "2.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" -dependencies = [ - "serde_core", -] [[package]] name = "block-buffer" @@ -505,21 +493,6 @@ dependencies = [ "libc", ] -[[package]] -name = "crc" -version = "3.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "69e6e4d7b33a94f0991c26729976b10ebde1d34c3ee82408fb536164fa10d636" -dependencies = [ - "crc-catalog", -] - -[[package]] -name = "crc-catalog" -version = "2.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "19d374276b40fb8bbdee95aef7c7fa6b5316ec764510eb64b8dd0e2ed0d7e7f5" - [[package]] name = "crc32fast" version = "1.5.0" @@ -553,15 +526,6 @@ dependencies = [ "crossbeam-utils", ] -[[package]] -name = "crossbeam-queue" -version = "0.3.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0f58bbc28f91df819d0aa2a2c00cd19754769c2fad90579b3592b1c9ba7a3115" -dependencies = [ - "crossbeam-utils", -] - [[package]] name = "crossbeam-utils" version = "0.8.21" @@ -654,12 +618,6 @@ dependencies = [ "syn", ] -[[package]] -name = "dotenvy" -version = "0.15.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1aaf95b3e5c8f23aa320147307562d361db0ae0d51242340f558153b4eb2439b" - [[package]] name = "dunce" version = "1.0.5" @@ -695,9 +653,6 @@ name = "either" version = "1.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "60b1af1c220855b6ceac025d3f6ecdd2b7c4894bfe9cd9bda4fbb4bc7c0d4cf0" -dependencies = [ - "serde", -] [[package]] name = "email-address-parser" @@ -765,17 +720,6 @@ dependencies = [ "windows-sys 0.59.0", ] -[[package]] -name = "etcetera" -version = "0.8.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "136d1b5283a1ab77bd9257427ffd09d8667ced0570b6f938942bc7568ed5b943" -dependencies = [ - "cfg-if", - "home", - "windows-sys 0.48.0", -] - [[package]] name = "event-listener" version = "5.4.0" @@ -825,29 +769,12 @@ dependencies = [ "miniz_oxide", ] -[[package]] -name = "flume" -version = "0.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "da0e4dd2a88388a1f4ccc7c9ce104604dab68d9f408dc34cd45823d5a9069095" -dependencies = [ - "futures-core", - "futures-sink", - "spin", -] - [[package]] name = "fnv" version = "1.0.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1" -[[package]] -name = "foldhash" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" - [[package]] name = "form_urlencoded" version = "1.2.1" @@ -905,17 +832,6 @@ dependencies = [ "futures-util", ] -[[package]] -name = "futures-intrusive" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d930c203dd0b6ff06e0201a4a2fe9149b43c684fd4420555b26d21b1a02956f" -dependencies = [ - "futures-core", - "lock_api", - "parking_lot", -] - [[package]] name = "futures-io" version = "0.3.31" @@ -1075,11 +991,6 @@ name = "hashbrown" version = "0.15.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bf151400ff0baff5465007dd2f3e717f3fe502074ca563069ce3a6629d07b289" -dependencies = [ - "allocator-api2", - "equivalent", - "foldhash", -] [[package]] name = "hashify" @@ -1092,15 +1003,6 @@ dependencies = [ "syn", ] -[[package]] -name = "hashlink" -version = "0.10.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7382cf6263419f2d8df38c55d7da83da5c18aef87fc7a7fc1fb1e344edfe14c1" -dependencies = [ - "hashbrown 0.15.2", -] - [[package]] name = "heck" version = "0.5.0" @@ -1115,10 +1017,12 @@ dependencies = [ "async-stream", "async-trait", "base64 0.21.7", + "bytes", "camino", "chrono", "clap", "config", + "crc32fast", "ed25519-dalek", "email-address-parser", "futures", @@ -1129,6 +1033,7 @@ dependencies = [ "huml-rs", "hyper 0.14.32", "lettre", + "libc", "mail-auth", "mail-parser 0.9.4", "mailparse", @@ -1149,7 +1054,6 @@ dependencies = [ "serde", "serde_json", "smtp", - "sqlx", "subtle", "tempfile", "thiserror 1.0.69", @@ -1162,12 +1066,6 @@ dependencies = [ "ulid", ] -[[package]] -name = "hex" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" - [[package]] name = "hickory-net" version = "0.26.1" @@ -1299,15 +1197,6 @@ dependencies = [ "tracing", ] -[[package]] -name = "hkdf" -version = "0.12.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b5f8eb2ad728638ea2c7d47a21db23b7b58a72ed6a38256b8a1849f15fbbdf7" -dependencies = [ - "hmac", -] - [[package]] name = "hmac" version = "0.12.1" @@ -1317,15 +1206,6 @@ dependencies = [ "digest", ] -[[package]] -name = "home" -version = "0.5.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "589533453244b0995c858700322199b2becb13b627df2851f64a2775d024abcf" -dependencies = [ - "windows-sys 0.59.0", -] - [[package]] name = "hostname" version = "0.3.1" @@ -1857,29 +1737,6 @@ version = "0.2.16" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" -[[package]] -name = "libredox" -version = "0.1.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7ddbf48fd451246b1f8c2610bd3b4ac0cc6e149d89832867093ab69a17194f08" -dependencies = [ - "bitflags", - "libc", - "plain", - "redox_syscall 0.7.3", -] - -[[package]] -name = "libsqlite3-sys" -version = "0.30.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2e99fb7a497b1e3339bc746195567ed8d3e24945ecd636e3619d20b9de9e9149" -dependencies = [ - "cc", - "pkg-config", - "vcpkg", -] - [[package]] name = "linked-hash-map" version = "0.5.6" @@ -2264,7 +2121,7 @@ checksum = "1e401f977ab385c9e4e3ab30627d6f26d00e2c73eef317493c4ec6d468726cf8" dependencies = [ "cfg-if", "libc", - "redox_syscall 0.5.8", + "redox_syscall", "smallvec", "windows-targets 0.52.6", ] @@ -2384,12 +2241,6 @@ version = "0.3.31" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "953ec861398dccce10c670dfeaf3ec4911ca479e9c02154b3a215178c5f566f2" -[[package]] -name = "plain" -version = "0.2.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b4596b6d070b27117e987119b4dac604f3c58cfb0b191112e24771b2faeac1a6" - [[package]] name = "portable-atomic" version = "1.10.0" @@ -2646,15 +2497,6 @@ dependencies = [ "bitflags", ] -[[package]] -name = "redox_syscall" -version = "0.7.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6ce70a74e890531977d37e532c34d45e9055d2409ed08ddba14529471ed0be16" -dependencies = [ - "bitflags", -] - [[package]] name = "regex" version = "1.11.1" @@ -3041,17 +2883,6 @@ dependencies = [ "serde", ] -[[package]] -name = "sha1" -version = "0.10.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba" -dependencies = [ - "cfg-if", - "cpufeatures 0.2.17", - "digest", -] - [[package]] name = "sha2" version = "0.10.8" @@ -3133,9 +2964,6 @@ name = "smallvec" version = "1.13.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3c5e1a9a646d36c3599cd173a41282daf47c44583ad367b8e6837255952e5c67" -dependencies = [ - "serde", -] [[package]] name = "smtp" @@ -3167,9 +2995,6 @@ name = "spin" version = "0.9.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" -dependencies = [ - "lock_api", -] [[package]] name = "spki" @@ -3181,194 +3006,6 @@ dependencies = [ "der", ] -[[package]] -name = "sqlx" -version = "0.8.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fefb893899429669dcdd979aff487bd78f4064e5e7907e4269081e0ef7d97dc" -dependencies = [ - "sqlx-core", - "sqlx-macros", - "sqlx-mysql", - "sqlx-postgres", - "sqlx-sqlite", -] - -[[package]] -name = "sqlx-core" -version = "0.8.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ee6798b1838b6a0f69c007c133b8df5866302197e404e8b6ee8ed3e3a5e68dc6" -dependencies = [ - "base64 0.22.1", - "bytes", - "crc", - "crossbeam-queue", - "either", - "event-listener", - "futures-core", - "futures-intrusive", - "futures-io", - "futures-util", - "hashbrown 0.15.2", - "hashlink", - "indexmap", - "log", - "memchr", - "once_cell", - "percent-encoding", - "serde", - "serde_json", - "sha2", - "smallvec", - "thiserror 2.0.11", - "tokio", - "tokio-stream", - "tracing", - "url", -] - -[[package]] -name = "sqlx-macros" -version = "0.8.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a2d452988ccaacfbf5e0bdbc348fb91d7c8af5bee192173ac3636b5fb6e6715d" -dependencies = [ - "proc-macro2", - "quote", - "sqlx-core", - "sqlx-macros-core", - "syn", -] - -[[package]] -name = "sqlx-macros-core" -version = "0.8.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "19a9c1841124ac5a61741f96e1d9e2ec77424bf323962dd894bdb93f37d5219b" -dependencies = [ - "dotenvy", - "either", - "heck", - "hex", - "once_cell", - "proc-macro2", - "quote", - "serde", - "serde_json", - "sha2", - "sqlx-core", - "sqlx-mysql", - "sqlx-postgres", - "sqlx-sqlite", - "syn", - "tokio", - "url", -] - -[[package]] -name = "sqlx-mysql" -version = "0.8.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aa003f0038df784eb8fecbbac13affe3da23b45194bd57dba231c8f48199c526" -dependencies = [ - "atoi", - "base64 0.22.1", - "bitflags", - "byteorder", - "bytes", - "crc", - "digest", - "dotenvy", - "either", - "futures-channel", - "futures-core", - "futures-io", - "futures-util", - "generic-array", - "hex", - "hkdf", - "hmac", - "itoa", - "log", - "md-5", - "memchr", - "once_cell", - "percent-encoding", - "rand 0.8.5", - "rsa", - "serde", - "sha1", - "sha2", - "smallvec", - "sqlx-core", - "stringprep", - "thiserror 2.0.11", - "tracing", - "whoami", -] - -[[package]] -name = "sqlx-postgres" -version = "0.8.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "db58fcd5a53cf07c184b154801ff91347e4c30d17a3562a635ff028ad5deda46" -dependencies = [ - "atoi", - "base64 0.22.1", - "bitflags", - "byteorder", - "crc", - "dotenvy", - "etcetera", - "futures-channel", - "futures-core", - "futures-util", - "hex", - "hkdf", - "hmac", - "home", - "itoa", - "log", - "md-5", - "memchr", - "once_cell", - "rand 0.8.5", - "serde", - "serde_json", - "sha2", - "smallvec", - "sqlx-core", - "stringprep", - "thiserror 2.0.11", - "tracing", - "whoami", -] - -[[package]] -name = "sqlx-sqlite" -version = "0.8.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2d12fe70b2c1b4401038055f90f151b78208de1f9f89a7dbfd41587a10c3eea" -dependencies = [ - "atoi", - "flume", - "futures-channel", - "futures-core", - "futures-executor", - "futures-intrusive", - "futures-util", - "libsqlite3-sys", - "log", - "percent-encoding", - "serde", - "serde_urlencoded", - "sqlx-core", - "thiserror 2.0.11", - "tracing", - "url", -] - [[package]] name = "stable_deref_trait" version = "1.2.0" @@ -3388,17 +3025,6 @@ dependencies = [ "windows-sys 0.59.0", ] -[[package]] -name = "stringprep" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b4df3d392d81bd458a8a621b8bffbd2302a12ffe288a9d931670948749463b1" -dependencies = [ - "unicode-bidi", - "unicode-normalization", - "unicode-properties", -] - [[package]] name = "strsim" version = "0.11.1" @@ -3656,17 +3282,6 @@ dependencies = [ "tokio", ] -[[package]] -name = "tokio-stream" -version = "0.1.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32da49809aab5c3bc678af03902d4ccddea2a87d028d86392a4b1560c6906c70" -dependencies = [ - "futures-core", - "pin-project-lite", - "tokio", -] - [[package]] name = "tokio-util" version = "0.7.13" @@ -3747,7 +3362,6 @@ version = "0.1.41" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "784e0ac535deb450455cbfa28a6f0df145ea1bb7ae51b821cf5e7927fdcfbdd0" dependencies = [ - "log", "pin-project-lite", "tracing-attributes", "tracing-core", @@ -3850,12 +3464,6 @@ dependencies = [ "web-time", ] -[[package]] -name = "unicode-bidi" -version = "0.3.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c1cb5db39152898a79168971543b1cb5020dff7fe43c8dc468b0885f5e29df5" - [[package]] name = "unicode-ident" version = "1.0.16" @@ -3868,21 +3476,6 @@ version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3b09c83c3c29d37506a3e260c08c03743a6bb66a9cd432c6934ab501a190571f" -[[package]] -name = "unicode-normalization" -version = "0.1.25" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5fd4f6878c9cb28d874b009da9e8d183b5abc80117c40bbd187a1fde336be6e8" -dependencies = [ - "tinyvec", -] - -[[package]] -name = "unicode-properties" -version = "0.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7df058c713841ad818f1dc5d3fd88063241cc61f49f5fbea4b951e8cf5a8d71d" - [[package]] name = "unicode-width" version = "0.1.14" @@ -3945,12 +3538,6 @@ version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65" -[[package]] -name = "vcpkg" -version = "0.2.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426" - [[package]] name = "version_check" version = "0.9.5" @@ -3991,12 +3578,6 @@ dependencies = [ "wit-bindgen-rt", ] -[[package]] -name = "wasite" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b8dad83b4f25e74f184f64c43b150b91efe7647395b42289f38e50566d82855b" - [[package]] name = "wasm-bindgen" version = "0.2.100" @@ -4115,16 +3696,6 @@ dependencies = [ "rustls-pki-types", ] -[[package]] -name = "whoami" -version = "1.6.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d4a4db5077702ca3015d3d02d74974948aba2ad9e12ab7df718ee64ccd7e97d" -dependencies = [ - "libredox", - "wasite", -] - [[package]] name = "widestring" version = "1.1.0" diff --git a/config.example.huml b/config.example.huml index a68a94a..5e92a7d 100644 --- a/config.example.huml +++ b/config.example.huml @@ -49,7 +49,7 @@ storage:: bounced_retention: "7d" deferred_retention: "2d" interval: "1h" - storage_type: "fs" + storage_type: "log" filters:: - :: @@ -66,3 +66,13 @@ filters:: log:: format: "fmt" level: "info" + +# Optional durable append-log mail queue (not yet wired into the serving path) +# queue:: +# append_writers: 1 # Number of shards / concurrent append writers (default: 1) +# pending_append_bytes: 134217728 # Pending (not-yet-durable) append bytes before backpressure (default: 128 MiB) +# segment_target_bytes: 67108864 # Target size of each active segment file, per shard (default: 64 MiB) +# compaction_dead_ratio: 0.50 # Dead-byte fraction in a sealed segment that makes it compaction-eligible (default: 0.50) +# compaction_min_age: "60s" # Minimum age of a sealed segment before compaction eligibility (default: 60s) +# disk_reserve_bytes: 1073741824 # Minimum free disk space required to accept new mail (default: 1 GiB) +# checkpoint_interval_bytes: 8388608 # Bytes of appended data between durability checkpoints (default: 8 MiB) diff --git a/config.example.toml b/config.example.toml index 722f8cb..350ebbc 100644 --- a/config.example.toml +++ b/config.example.toml @@ -78,29 +78,25 @@ key_type = "rsa" # "rsa" or "ed25519" # Storage configuration [storage] -storage_type = "fs" # "fs" for filesystem, "sqlite" for SQLite-backed storage +storage_type = "log" # "log" (durable log queue, default) or "fs" (legacy one file per message) base_path = "/var/lib/hedwig/mail" -# SQLite storage example (num_shards/batch_* live under [storage], tuning under [storage.sqlite]): -# [storage] -# storage_type = "sqlite" -# base_path = "/var/lib/hedwig/mail" -# num_shards = 16 # Number of database shards (default: 16) -# batch_size = 100 # Write-batch size (default: 100) -# batch_timeout_ms = 5 # Max time (ms) to wait before flushing a batch (default: 5) -# -# [storage.sqlite] -# synchronous = "NORMAL" # SQLite synchronous setting: OFF | NORMAL | FULL (default: NORMAL) -# cache_size_mb = 1600 # Page-cache size per shard in MiB (default: 1600) -# busy_timeout_ms = 5000 # SQLite busy-timeout in ms (default: 5000) -# pool_max_connections = 10 # sqlx pool max connections per shard (default: 10) - # Optional retention policy for local spool cleanup [storage.cleanup] bounced_retention = "7d" deferred_retention = "2d" interval = "1h" +# Optional durable append-log mail queue (not yet wired into the serving path) +# [queue] +# append_writers = 1 # Number of shards / concurrent append writers (default: 1) +# pending_append_bytes = 134217728 # Pending (not-yet-durable) append bytes before backpressure (default: 128 MiB) +# segment_target_bytes = 67108864 # Target size of each active segment file, per shard (default: 64 MiB) +# compaction_dead_ratio = 0.50 # Dead-byte fraction in a sealed segment that makes it compaction-eligible (default: 0.50) +# compaction_min_age = "60s" # Minimum age of a sealed segment before compaction eligibility (default: 60s) +# disk_reserve_bytes = 1073741824 # Minimum free disk space required to accept new mail (default: 1 GiB) +# checkpoint_interval_bytes = 8388608 # Bytes of appended data between durability checkpoints (default: 8 MiB) + # Optional email filters [[filters]] type = "from_domain_filter" diff --git a/dev/config.sqlite.toml b/dev/config.sqlite.toml deleted file mode 100644 index 521cf0b..0000000 --- a/dev/config.sqlite.toml +++ /dev/null @@ -1,32 +0,0 @@ -# Development configuration for Hedwig SMTP Server (SQLite storage) -# Used by `just run-sqlite` - runs locally - -[server] -workers = 4 -disable_outbound = true - -# Plaintext listener on port 2526 -[[server.listeners]] -addr = "0.0.0.0:2526" - -[log] -level = "debug" -format = "fmt" - -# Test credentials -[[server.auth]] -username = "test" -password = "test" - -[storage] -storage_type = "sqlite" -base_path = "/tmp/hedwig-dev-sqlite/" -num_shards = 16 -batch_size = 100 -batch_timeout_ms = 5 - -[storage.sqlite] -synchronous = "NORMAL" -cache_size_mb = 1600 -busy_timeout_ms = 5000 -pool_max_connections = 10 diff --git a/docs/PRODUCTION_HARDENING.md b/docs/PRODUCTION_HARDENING.md index 0254e5e..ab3cf11 100644 --- a/docs/PRODUCTION_HARDENING.md +++ b/docs/PRODUCTION_HARDENING.md @@ -50,9 +50,9 @@ Production deployments should set `server.helo_hostname` to the public FQDN for ## 🟡 Important -### 6. ~~Filesystem storage lacks durability guarantees~~ → Addressed (SQLite backend) +### 6. ~~Filesystem storage lacks durability guarantees~~ → Addressed (log-queue backend) -**Status:** Addressed via `SqliteStorage` backend (`storage_type = "sqlite"`). SQLite transactions provide atomic writes — no partial writes, no fsync gaps. See `docs/specs/2026-03-29-sqlite-storage-design.md`. +**Status:** Addressed via the durable log queue (`storage_type = "log"`): checksummed append-only records with torn-tail recovery, and fsync barriers at every destructive boundary. (Previously addressed by the SQLite backend, since removed.) See `ARCHITECTURE.md`. **Problem:** `fs_storage.rs` uses `tokio::fs::write()` directly — no temp-file + rename, no fsync. On crash or power loss: - Partially written files can corrupt the queue @@ -70,9 +70,9 @@ Production deployments should set `server.helo_hostname` to the public FQDN for --- -### 7. ~~Filesystem storage doesn't scale to millions of files~~ → Addressed (SQLite backend) +### 7. ~~Filesystem storage doesn't scale to millions of files~~ → Addressed (log-queue backend) -**Status:** Addressed via `SqliteStorage` backend. Sharded SQLite databases with indexed queries replace flat directory walks. See `docs/specs/2026-03-29-sqlite-storage-design.md`. +**Status:** Addressed via the log queue: segmented append-only storage keeps file count proportional to live backlog, with no per-message files and no directory scans. (Previously addressed by the SQLite backend, since removed.) **Problem:** Flat directories (`queued/`, `deferred/`, `bounced/`) with millions of files means very slow `readdir()` calls. Startup replay and cleanup become directory-walk bound. ext4 performance degrades significantly past ~100K files per directory. diff --git a/docs/plans/2026-07-20-durable-log-queue.md b/docs/plans/2026-07-20-durable-log-queue.md new file mode 100644 index 0000000..c7f42ec --- /dev/null +++ b/docs/plans/2026-07-20-durable-log-queue.md @@ -0,0 +1,1217 @@ +# Hedwig Durable Log Queue Plan + +> Status: design captured for future implementation; revised 2026-07-20 after pair review +> +> Date: 2026-07-20 +> +> Scope: replace the filesystem spool plus worker-coupled in-memory job queue with a single-process dispatcher backed by segmented append-only message logs. One append writer by default; the on-disk format is shard-capable so more writers can be enabled if benchmarks justify them. + +## 1. Executive summary + +Hedwig currently persists an incoming message and then awaits capacity in the bounded in-memory worker channel before returning SMTP `250 OK`. When outbound workers are slow, sleeping for rate limits, or otherwise occupied, the channel eventually fills and inbound acceptance becomes limited by outbound drain speed. + +The intended architecture is: + +- Hedwig remains a **single process**. +- There is **one dispatcher** responsible for scheduling all delivery work. +- There is **one append writer** by default. The record format, job locations, and dispatcher are shard-aware, so `N` writers can be enabled later without a format change; multiple writers ship only if benchmarks justify them. +- Each append writer exclusively owns its shard and writes to that shard's segmented log files. Logs are segmented: the active file is sealed at a target size and a new one is started, so no single file grows without bound regardless of total throughput. +- The complete message is stored in the log record, eliminating one spool file per message and avoiding large flat-directory scans. +- SMTP acceptance waits only for the append write to complete into the kernel page cache. It does **not** wait for `fsync`, the dispatcher, a worker, or outbound delivery capacity. +- The dispatcher discovers appended records using per-shard committed-tail positions and maintains ready, deferred, and in-flight state. +- Workers pull jobs from the dispatcher and report outcomes. +- Delivered records become garbage. Segment reclamation is event-driven: a segment whose live count reaches zero is deleted immediately, and a sealed segment is queued for compaction as soon as its garbage ratio crosses the threshold. Periodic sweeps exist only as a safety-net backstop. +- A small amount of temporary disk amplification is accepted; delivered payloads do not remain indefinitely. +- Multiple processes, shared queues, leader election, distributed claims, and exactly-once delivery are explicitly out of scope. + +The main result is that inbound SMTP acceptance is bounded by disk append throughput rather than outbound worker drain throughput. + +## 2. Context and current problem + +### 2.1 Current acceptance path + +The current path is effectively: + +```text +SMTP DATA + -> persist message as Queued + -> await sender_channel.send(job) + -> return 250 OK +``` + +Relevant current code: + +- `smtp-server/src/callbacks.rs:248-280` persists the queued message and then awaits sending its `Job` to the worker channel. +- `smtp/src/lib.rs:465-469` waits for the DATA callback before returning `250 OK`. +- `smtp-server/src/main.rs:108-112` creates a bounded job channel using `queue_buffer`. + +The body has already been written to the filesystem at this point — with page-cache durability only: no `fsync` and no atomic rename — but the SMTP client is not acknowledged until an in-memory worker slot becomes available. Once the channel fills, inbound acceptance proceeds only as quickly as workers remove jobs from the channel. + +### 2.2 Why workers may not drain promptly + +A worker occupies its slot while performing the complete delivery lifecycle, including: + +- loading and parsing the message; +- signing it; +- waiting for rate limits; +- DNS and MX work; +- connecting to remote MTAs; +- attempting SMTP delivery. + +In particular, a rate-limited job currently sleeps in the worker in `smtp-server/src/worker/mod.rs:610-617`. Enough sleeping jobs can consume every worker slot even when unrelated destinations could proceed. + +### 2.3 Current scanning costs + +The current filesystem backend uses status directories containing one file per message. Existing operations include: + +- flat directory enumeration for queued messages; +- startup replay of the entire queued set before accepting new connections; +- periodic deferred metadata scans; +- per-message path, inode, open, and file creation work. + +The SQLite backend avoids directory scans but still performs whole-status queries. The new architecture must avoid relying on full per-message scans during normal startup and scheduling. + +## 3. Goals + +### 3.1 Primary goals + +1. Decouple inbound acceptance from outbound worker drain speed. +2. Keep disk as the source of truth; message bodies must not remain in memory while waiting for outbound delivery. +3. Replace one-file-per-message storage with sequential append-oriented storage. +4. Avoid large queued/deferred directory scans. +5. Preserve at-least-once delivery and retry-attempt recovery across process restarts. +6. Keep the design optimized for a single Hedwig process. +7. Keep the on-disk format and dispatcher shard-capable so append throughput can later scale across multiple independently owned log shards. +8. Reclaim delivered-message disk space through segment deletion and compaction. +9. Move delayed retry waiting out of worker slots and into dispatcher scheduling. +10. Maintain or improve observability of queue depth, age, retries, and delivery outcomes. + +### 3.2 Secondary goals + +- Bound in-memory admission buffering by bytes rather than message count. +- Permit tuning the append-writer count without changing dispatcher semantics. +- Make startup recovery sequential and deterministic. +- Keep the on-disk format versioned and independently testable. +- Preserve the existing filesystem backend's accepted page-cache durability tradeoff. + +## 4. Non-goals + +The initial implementation will not provide: + +- multiple processes sharing one queue; +- active-active dispatchers; +- leader election or fencing; +- distributed leases; +- online resharding of an existing non-empty queue; +- exactly-once delivery to remote SMTP servers; +- strict global FIFO delivery order; +- an `fsync` per message or group-commit durability; +- a general-purpose external message broker; +- removal of all disk-capacity safeguards. + +Operators who need multiple Hedwig instances can run independent processes with independent queues and load-balance or round-robin inbound traffic themselves. + +## 5. Decisions captured + +### 5.1 Process model + +- Hedwig is always single-process for this queue design. +- There is one dispatcher. +- There is one append writer (one shard) by default. The design supports `N` writers, each owning exactly one shard; enabling more than one is a benchmark-driven decision, not the v1 default. +- Workers remain ordinary tasks inside the same process. + +### 5.2 Storage model + +- The full message body and delivery envelope are stored in segmented append-only payload logs. +- Each shard owns one active segment and zero or more sealed segments. +- No two append writers write to the same active file. +- Each record is identified by a stable message ID and a physical location. +- Delivered records are marked terminal and later reclaimed through segment deletion or compaction. + +### 5.3 Durability model + +- SMTP `250 OK` may be returned after the complete record has been accepted by the kernel via successful write calls. +- Hedwig will not require `fsync` before acknowledging the message. +- A process crash should recover page-cache-backed data normally. +- A machine crash or power loss may lose recently acknowledged messages. This is an explicit and accepted performance tradeoff. +- State updates use the same page-cache durability policy. +- Destructive operations are the exception: before deleting or truncating anything that holds live data, the replacement must be durable. Compaction fsyncs its output segment and the location manifest before unlinking the source; checkpoints are fsynced before covered journal history is truncated. Without these barriers a power loss could destroy arbitrarily old queued mail, not just recently accepted mail. + +### 5.4 Scheduling model + +- The append log is the authoritative admission queue. +- Dispatcher wake-up notifications are hints, not authoritative job storage. +- The dispatcher tracks one discovery cursor per shard. +- Workers pull or are assigned one claim at a time from the dispatcher. +- Worker ownership is in-memory only. +- On restart, every non-terminal, non-deferred record is eligible for redispatch. +- Deferred jobs are scheduled by due time rather than occupying sleeping worker slots. + +### 5.5 Delivery semantics + +- Delivery remains at-least-once. +- The delivery unit is one message. A job carries the whole message even when its recipients span multiple domains; per-domain delivery units are future work. +- A deferred-state record persists the remaining (not-yet-accepted) recipient set, so a retry re-sends only to recipients that have not accepted the message. This removes the current behavior where a partial failure re-delivers to recipients that already accepted it. +- If a remote MTA accepts a message and Hedwig crashes before recording terminal success, the message may be delivered again after restart. +- Exactly-once SMTP delivery is not achievable and will not be claimed. + +### 5.6 Sharding model + +- New messages are assigned to a shard using a stable hash of the message ID. +- The initial implementation will require the queue to be empty before changing the configured shard count. +- Existing records retain their explicit physical shard and segment location; runtime lookup must not depend only on the current shard count. +- Recipient domain is not used as the payload-sharding key because large domains could create hot writer shards. + +## 6. High-level architecture + +```text + +------------------+ +SMTP connections ------>| shard selection | + +--+----+----+------+ + | | | + +--------+ | +--------+ + v v v + writer 0 writer 1 writer N + shard 0 shard 1 shard N + | | | + +------ committed heads ----+ + | + v + one dispatcher + +---------------+---------------+ + | ready scheduling | + | deferred due-time heap | + | in-flight jobs | + | message locations | + | per-segment live/dead counts | + +---------------+---------------+ + | + worker pulls + | + v + worker pool + | + delivery outcomes + | + v + dispatcher -> owning shard state + | + segment GC/compaction +``` + +## 7. On-disk layout + +A possible initial layout is: + +```text +spool/ + format-version + shard-0000/ + segment-000000000001.log + segment-000000000001.state + segment-000000000002.open + state-journal.log + checkpoint + + shard-0001/ + segment-000000000001.log + segment-000000000001.state + segment-000000000002.open + state-journal.log + checkpoint +``` + +Properties: + +- The number of shard directories is small and configured. +- The number of segment files is proportional to live data and temporary garbage, not total historical message count. +- `.open` identifies the shard's current append target. +- Sealing a segment renames it to `.log`. +- Payload segments are immutable after sealing. +- State transitions are maintained separately from immutable payload data. + +The exact state sidecar and journal division should be finalized during the format implementation. The intended behavior is: + +- append-only state transitions for crash-tolerant replay; +- compact state checkpoints for fast startup; +- per-segment live/dead accounting for GC; +- no requirement to rewrite payload segments for ordinary state changes. + +## 8. Payload record format + +The record format must be versioned and self-framing. A conceptual record is: + +```text ++-------------------------+ +| magic | +| format version | +| header length | +| total record length | +| header checksum | +| payload checksum | +| message ID | +| enqueue timestamp | +| relocation generation | +| envelope metadata | +| message body length | +| message body | ++-------------------------+ +``` + +The fixed portion must contain enough information for the dispatcher to discover jobs without reading or decoding the body: + +- total record length; +- message ID; +- enqueue time; +- sender/domain routing information needed by scheduling; +- recipient information needed to construct a job; +- body offset and length; +- format and relocation generation. + +The dispatcher should be able to read the fixed header and skip directly to the next record. + +Checksums serve two different purposes: + +- detect a partial or corrupt active tail; +- detect unexpected corruption in sealed segments. + +On startup: + +- an incomplete final record in an active segment may be truncated; +- corruption in the middle of a sealed segment must be reported and handled explicitly rather than silently skipped. + +## 9. Append writers + +### 9.1 Ownership + +Each append writer owns: + +- one shard directory; +- the active payload file; +- active segment offsets and ordinals; +- segment rotation; +- shard state persistence commands; +- shard-local live/dead accounting; +- deletion eligibility; +- coordination with compaction. + +No mutex is required around a shared append offset because there is no shared append file. + +### 9.2 Append request + +A conceptual request is: + +```rust +struct AppendRequest { + message_id: MessageId, + envelope: Envelope, + body: Bytes, + completion: oneshot::Sender>, +} +``` + +The writer returns: + +```rust +struct JobLocation { + shard: u16, + segment: u64, + offset: u64, + length: u32, + ordinal: u32, + generation: u32, +} +``` + +The SMTP callback awaits only this append completion before returning `250 OK`. + +### 9.3 Admission queue + +The append request queue must be bounded by pending bytes, not only pending record count. + +This prevents a large number of messages waiting for disk admission from consuming unbounded RAM. Backpressure at this boundary is valid because it represents actual storage throughput rather than outbound delivery throughput. + +The bound applies only until records are copied into the page cache. Message bodies are never retained in this queue while waiting for workers or remote MTAs. + +The append bound alone does not cap inbound memory: each SMTP session buffers its full DATA payload before the callback runs. Total inbound memory is governed by the composition of `max_connections`, `max_message_size`, and the pending-append byte bound, and the chosen defaults must be documented together as one memory budget. + +### 9.4 Write batching + +The writer may batch adjacent pending requests into fewer write operations. Since no `fsync` is required, the batching policy can prioritize throughput without introducing a durability timer. + +Initial implementation may use straightforward buffered `write_all` operations. More advanced `writev` batching should be justified by benchmarks. + +### 9.5 Publish ordering + +For each record, the writer must: + +1. encode the complete record; +2. write the complete record successfully; +3. update the shard's published committed tail with release ordering; +4. notify the dispatcher; +5. complete the SMTP append request. + +The committed tail must never expose a partial record. + +## 10. Shard selection + +Initial selection: + +```text +shard = stable_hash(message_id) % append_writer_count +``` + +ULID randomness should distribute messages sufficiently over time. + +If short-term size imbalance becomes measurable, a later optimization may use power-of-two choices: + +1. derive two candidate shards from the message ID; +2. inspect each writer's pending-byte count; +3. choose the less-loaded writer. + +This is an optimization, not part of the initial correctness model. + +## 11. Dispatcher discovery + +### 11.1 Per-shard committed heads + +Each writer publishes: + +```rust +struct ShardHead { + segment: u64, + committed_offset: u64, +} +``` + +The dispatcher owns one discovery cursor per shard: + +```rust +struct ShardCursor { + segment: u64, + offset: u64, +} +``` + +When notified, the dispatcher reads headers between its cursor and the committed head and adds newly discovered messages to its scheduler. + +### 11.2 Notifications are hints + +The dispatcher notification mechanism must not become another authoritative bounded work queue. + +A `Notify`-style wake-up is sufficient because: + +- notifications may be coalesced or lost; +- the committed head remains authoritative; +- the dispatcher always compares its cursor against the current head; +- a low-frequency safety tick can discover work even if no notification is observed. + +A descriptor fast-path channel (writer `try_send(JobDescriptor)` with cursor catch-up on overflow) was considered and cut from v1: the cursor plus notification path is the only discovery mechanism. Reintroduce the fast path only if header rereads measurably matter. + +### 11.3 Segment rotation + +When a writer rotates its active segment: + +1. finish the current record; +2. seal the current segment; +3. publish its final committed length; +4. create the next active segment; +5. publish the new active segment identity; +6. notify the dispatcher. + +The dispatcher must be able to advance from the end of one segment to the start of the next without requiring per-message directory enumeration. + +## 12. Dispatcher state + +A conceptual in-memory dispatcher is: + +```rust +struct Dispatcher { + shard_cursors: Vec, + ready: ReadyScheduler, + deferred: BinaryHeap>, + inflight: HashMap, + locations: HashMap, + segment_stats: HashMap, +} +``` + +Dispatcher memory must be bounded from the start, not as a deferred optimization, because a disk-sized backlog otherwise becomes an OOM that repeats on every restart. The baseline mechanism is discovery backpressure: the dispatcher stops advancing shard cursors while its in-memory tables hold a configured maximum of undispatched entries. The log itself preserves everything beyond that window, so nothing is lost; discovery resumes as entries drain. Memory per million queued and per million deferred records is a required benchmark, and a paged deferred index is added only if those measurements demand it. + +### 12.1 Job states + +Conceptual states are: + +```text +Ready +InFlight +Deferred(next_attempt, attempts) +TerminalDelivered +TerminalBounced +``` + +`InFlight` is process-local and does not need to be persisted. + +### 12.2 Worker claims + +When a worker asks for work: + +```text +Ready -> InFlight +``` + +Claims carry a mandatory in-memory generation so a late result from a cancelled or stalled worker cannot complete a later reassignment of the same message. Worker panic, cancellation, or closure of the outcome channel must surface to the dispatcher as an abandonment event that returns the job to `Ready`; a claim must never leak. Reassignment of a timed-out claim should still be conservative — the generation makes a duplicate result safe to ignore, but it cannot prevent a duplicate outbound delivery already in progress. + +Because all workers and the dispatcher are in one process, no durable lease expiry or cross-process fencing is required. + +### 12.3 Restart behavior + +After restart: + +- terminal records remain terminal based on persisted state; +- deferred records return to the due-time scheduler with their attempt counts; +- all other live records become ready; +- records that were in flight before the crash are redispatched. + +## 13. Worker interaction + +Workers should receive a lightweight job containing identity, location, and delivery metadata, not the message body. + +```rust +struct DeliveryJob { + message_id: MessageId, + location: JobLocation, + attempts: u32, + claim_generation: u64, +} +``` + +The worker reads the body directly from the segment using positioned reads. Workers may read sealed and active segments concurrently with the owning append writer because they only access committed record ranges. If a positioned read fails because compaction relocated the record and removed the source segment, the worker re-fetches the current location from the dispatcher and retries before treating the failure as an error. + +The worker reports one of: + +```rust +enum JobOutcome { + Delivered { + response: DeliveryResponse, + }, + Deferred { + attempts: u32, + next_attempt: SystemTime, + remaining_recipients: Vec, + error: String, + }, + RateLimited { + domain: String, + retry_after: Duration, + }, + Bounced { + reason: String, + }, +} +``` + +`Deferred` is a real delivery failure: it increments the attempt count and is persisted together with the remaining recipient set. `RateLimited` is local throttling: it does not increment attempts and is requeued in memory only (section 14). + +The dispatcher validates the claim generation, applies the persistent state transition through the owning shard, and only then updates scheduling state (section 16). + +## 14. Retry scheduling + +Rate-limited and retryable jobs must not sleep inside worker slots, and the two cases are distinct. + +Delivery retry (a real failed attempt): + +```text +worker attempt fails temporarily + -> reports Deferred(next_attempt, remaining_recipients) + -> dispatcher persists deferred state (attempts incremented) + -> dispatcher inserts job into due-time heap + -> worker immediately asks for another job +``` + +Rate limiting (local throttling, not an attempt): + +```text +dispatcher checks the shared per-domain rate limiter before dispatching a claim + -> jobs for exhausted domains stay queued; other domains dispatch +worker re-acquires the limit immediately before transmission + -> if it loses that race it reports RateLimited(retry_after) + -> dispatcher requeues in memory only: no journal write, no attempt increment +``` + +A due timestamp is not a token reservation: jobs waking at the same due time must re-pass the limiter at dispatch, which prevents a thundering herd against one domain. Rate-limit deferrals are never persisted — after a restart the job simply becomes ready and is gated by the limiter again. + +When a deferred job becomes due: + +```text +Deferred -> Ready +``` + +This eliminates the current periodic full deferred scan and prevents delayed jobs from occupying the delivery worker pool. + +The initial scheduler can use a min-heap keyed by `next_attempt`. A timing wheel or paged on-disk due-time index is unnecessary until measurements show the heap is too large. + +## 15. Fair scheduling + +Durable append order does not have to equal delivery order, and strict global FIFO is not a goal. + +Because the v1 delivery unit is a whole message — which may span domains — true per-domain fair scheduling is not implementable yet and is deferred until per-domain delivery units exist. What v1 provides instead: + +- rate-limit gating at dispatch time (section 14), so exhausted domains do not consume worker slots; +- no sleeping in workers, so a slow domain ties up at most the claims actively being attempted against it; +- enqueue-age ordering, so old messages do not starve. + +When per-domain delivery units are introduced later, deficit round-robin over per-domain ready queues is the intended fairness mechanism. Payload sharding stays keyed by message ID either way. + +## 16. Persistent state + +Payload records remain immutable after append. Delivery state is stored separately. + +The intended model is: + +- per-shard append-only state journal for transitions; +- periodic compact checkpoint containing the latest state of live records; +- optional small per-segment state summary for GC accounting. + +Conceptual state entries include: + +```text +DEFER(message_id, attempts, next_attempt, remaining_recipients, last_error) +DELIVERED(message_id, timestamp, remote_response_summary) +BOUNCED(message_id, timestamp, reason) +``` + +Persistence is the state boundary. The dispatcher applies an outcome to its in-memory state only after the journal write succeeds: + +```text +DEFER persisted -> remove InFlight -> insert into deferred heap +DELIVERED persisted -> remove InFlight -> decrement segment live count -> GC eligible +state write failure -> job stays InFlight and the write is retried; a job is + never marked terminal or dropped from scheduling on a + failed write +``` + +An enqueue transition does not need a separate state record because the payload record itself implies `Ready` unless superseded by later state. + +State-journal writes follow the same page-cache durability policy as payload writes. A lost terminal transition may cause duplicate delivery after an exceptional system failure, which is permitted by at-least-once semantics. + +A checkpoint must be self-sufficient for every segment that still exists on disk. Because a payload record implies `Ready` unless superseded, truncating the journal entry that recorded a delivery — while that payload still sits in a partially dead segment — would resurrect delivered mail on the next restart. Every checkpoint therefore contains: + +- terminal tombstones (or a per-segment bitmap) for every terminal record in every still-present segment; +- the deferred set with attempts, due times, and remaining recipients; +- the per-shard discovery cursor; +- the state-journal position (LSN) it covers; +- the segment topology and current location generation; +- a checksum and format version. + +Checkpointing must use copy-before-replace ordering: + +1. write a new checkpoint; +2. `fsync` it (destructive-boundary rule, section 5.3); +3. rename it into place; +4. truncate only journal history at or below the checkpoint's recorded LSN. + +The exact binary format, checkpoint cadence, and journal rotation thresholds will be specified and tested as part of the storage-format phase. + +## 17. Segment deletion and compaction + +### 17.1 Fully terminal segments + +Each sealed segment tracks: + +- total records and bytes; +- live records and bytes; +- terminal records and bytes; +- oldest and newest enqueue time. + +Reclamation is event-driven, not scan-based. The terminal transition that drops a sealed segment's live count to zero immediately schedules that segment for deletion; the transition that pushes its dead ratio over the compaction threshold immediately queues it as a compaction candidate. A low-frequency periodic sweep exists only as a backstop for missed events. + +When: + +```text +live_records == 0 +``` + +its payload, sidecar, and obsolete state data are deleted. + +At high delivery rates this is the dominant reclamation path: a segment written during a burst has essentially all of its records delivered within the retry horizon, dies completely, and is unlinked without any copying. Compaction is reserved for the minority of segments pinned by long-deferred stragglers. + +### 17.2 Partially live segments + +A segment with a small number of long-lived messages may otherwise retain a large amount of delivered garbage. + +Initial compaction policy: + +```text +compact when: + segment is sealed + AND dead_bytes / total_bytes >= 0.50 + AND segment is older than a short grace period +``` + +A 50% threshold limits uncompacted sealed-segment amplification to roughly 2x live bytes. A higher threshold such as 75% reduces copying but allows roughly 4x amplification. The threshold must be configurable and tuned through benchmarks. + +### 17.3 Compaction flow + +1. Select a sealed source segment. +2. Snapshot its live record set. +3. Copy each still-live record into an unpublished compaction output segment. Compaction output is never discovered as new admissions: it is excluded from the dispatcher's discovery cursors and enters the system only through the location switch below. +4. Give relocated records a higher relocation generation. +5. Recheck that copied records are still live; a record that went terminal during the copy is dropped from the relocation set. Terminal races must not resurrect messages. +6. Seal and `fsync` the compaction output. +7. Atomically publish the new location generation (manifest update, then `fsync`) and update dispatcher locations. +8. Wait for in-flight readers of the source segment to drain; a worker holding a stale location that loses this race re-fetches the current location and retries (section 13). +9. Unlink the source segment only after steps 6-8 are durable and complete. + +If both old and relocated records are observed during recovery, the record with the highest valid relocation generation wins. + +### 17.4 Compaction concurrency + +Append writers may be sharded, but their files commonly share one physical disk. Initial policy: + +```text +maximum concurrent compactions = 1 +``` + +A global semaphore prevents multiple shards from creating avoidable read/write interference. Append operations continue while a sealed segment is compacted. + +### 17.5 Long-deferred records + +A long-deferred message may be repeatedly moved as surrounding segments become garbage. Initial mitigations: + +- do not immediately recompact a newly produced compaction segment; +- use a minimum segment age before eligibility; +- track per-record relocation count for diagnostics. + +If repeated movement becomes significant, introduce deferred segments bucketed by due-time range. This is a future optimization, not required initially. + +## 18. Active segment sizing + +More shards create more partially filled active segments. Segment sizing should account for total active slack. + +A possible initial policy is: + +```text +target total active segment capacity = 64-128 MiB +per-shard segment size = max(8 MiB, target / shard_count) +``` + +Example: + +| Append writers | Segment size | Total active capacity | +|---:|---:|---:| +| 1 | 64 MiB | 64 MiB | +| 2 | 32 MiB | 64 MiB | +| 4 | 16 MiB | 64 MiB | +| 8 | 8 MiB | 64 MiB | + +These are benchmark starting points, not final defaults. With the default single writer, a larger segment (128-256 MiB) may be preferable to keep file counts low at high volume. + +Two invariants and one observation: + +- A record never spans segments. Per-shard segment size must therefore be at least the configured maximum message size plus record overhead; configurations violating this are rejected at startup. +- Sustained high throughput does not create huge files — it creates more sealed segments. At 3 million messages per hour averaging 10 KiB, one writer appends roughly 30 GiB/hour: about 470 sealed segments per hour at 64 MiB, or 120 at 256 MiB. Segment count, not file size, tracks throughput. +- Steady-state disk use and file count are proportional to live backlog plus not-yet-reclaimed garbage, not to total historical volume, because fully dead segments are unlinked as soon as they die (section 17.1). + +## 19. Bounced-message retention + +A delivered message can be discarded immediately from the logical queue. A bounced message may need to remain available according to existing Hedwig behavior or future retention policy. + +Bounced retention must not pin active queue segments indefinitely. If the message body must be retained: + +1. copy or append it to separate bounce/archive storage; +2. persist the bounce outcome; +3. mark the original queue record terminal; +4. allow normal queue-segment GC. + +Bounce/archive storage should have an explicit retention policy independent of the active delivery queue. + +## 20. Disk growth and safety + +The log design creates two categories of disk use: + +- live queued/deferred messages; +- temporary garbage awaiting segment deletion or compaction. + +GC and compaction bound the second category. They cannot bound the first category when incoming mail continuously exceeds outgoing delivery capacity. The existing spool has the same fundamental behavior. + +The initial implementation should therefore retain a simple disk reserve check: + +```text +if available disk space < configured reserve: + temporarily reject new SMTP DATA +``` + +This is an operational safety boundary, not a mechanism required specifically by append logs. An elaborate quota system is not required for the first implementation. + +Useful configuration may include: + +```toml +[queue] +append_writers = 1 +pending_append_bytes = 134217728 +segment_target_bytes = 67108864 +compaction_dead_ratio = 0.50 +compaction_min_age = "60s" +max_concurrent_compactions = 1 +disk_reserve_bytes = 1073741824 +durability = "page-cache" +``` + +Exact names and defaults must follow the existing configuration style. + +## 21. Shard-count changes + +For the initial version: + +> Changing `append_writers` requires an empty queue. + +This avoids implementing online resharding before it is needed. + +The on-disk location stored for every job must still include its explicit shard, segment, and offset. Code must not attempt to locate an existing record by recalculating `hash(id) % current_writer_count`. + +A future extension could use generations: + +```text +spool/generation-0001/ # old shard count +spool/generation-0002/ # current shard count +``` + +The dispatcher could drain old generations while new appends use only the current one. This is not part of the initial implementation. + +## 22. Startup and recovery + +Startup should not await replaying every job through a bounded worker channel before binding the SMTP listener. + +Proposed startup sequence: + +1. Read and validate the spool format version. +2. Discover the small set of configured shard directories and segment files. +3. Validate each active tail and truncate any incomplete final record. +4. Load each shard checkpoint. +5. Replay state-journal entries newer than the checkpoint. +6. Reconcile payload records newer than the checkpoint/discovery position. +7. Rebuild ready, deferred, location, and segment-stat indexes. +8. Start append writers and the dispatcher. +9. Start workers. +10. Bind the SMTP listener without first feeding the entire backlog through a bounded channel. +11. Let workers begin pulling from the reconstructed dispatcher state. + +For a very large backlog, recovery may later be changed to open the listener after minimum metadata initialization and continue indexing in the background. The first implementation should prioritize correctness and deterministic recovery, then measure startup time. Discovery backpressure (section 12) applies during recovery as well, so index rebuild memory stays bounded. + +### 22.1 Shutdown ordering + +Shutdown must drain the ownership graph deterministically: + +1. stop accepting new SMTP connections; +2. drain or cancel in-progress DATA sessions; +3. close append admission and drain pending append requests; +4. stop issuing new worker claims; +5. wait for in-flight deliveries to finish, or abandon them explicitly (they redispatch on restart); +6. persist all accepted outcomes through the state journal; +7. stop compaction at a recoverable step boundary; +8. flush and close state writers; +9. exit. + +A claim owned by a worker that has not returned is abandoned, never silently reassigned during shutdown. + +## 23. Migration from the current filesystem spool + +The new format requires an explicit migration strategy. + +A one-time migration may: + +1. stop ordinary queue mutation; +2. enumerate existing `Queued` and `Deferred` files; +3. preserve message IDs, attempt counts, and next-attempt timestamps; +4. append each live message to its selected new shard; +5. write equivalent deferred state where needed; +6. verify that every legacy live record has a new record; +7. rename the legacy spool to a migration backup; +8. activate the new format; +9. delete the backup only after operator confirmation or a defined grace period. + +Migration must be restart-safe and exclusive: + +- a root manifest records the format version, shard topology, migration epoch, and activation state, and the switch to the new format is a single atomic manifest update; +- migration is idempotent: re-running after a crash detects already-migrated message IDs instead of appending duplicates; +- retry metadata is honored for queued bodies as well as deferred ones — a message that was mid-retry sits in `queued/` with its metadata still attached; +- an exclusive OS-level lock on the spool root is held before migration, recovery, tail truncation, or writer startup, and independent Hedwig processes must use distinct spool roots. + +This migration is allowed to scan the old directories once. The new steady-state architecture must not depend on those scans. + +Alternative rollout approaches to evaluate during implementation: + +- introduce the log queue as a new storage backend and require operators to drain before switching; +- provide an offline migration command; +- auto-migrate on startup only when explicitly enabled. + +Automatic destructive migration without an explicit backup or operator opt-in is not acceptable. + +## 24. SQLite backend impact + +Decision: the log queue ships as a new selectable storage backend alongside the existing filesystem and SQLite backends. The existing backends keep their current scheduling path (bounded channel, deferred worker) unchanged while the log backend proves out. Deprecating the legacy backends is a separate later decision. The migration tooling in section 23 covers the filesystem spool first; SQLite-to-log migration is deferred. + +The append-log format must not be forced into the existing `Storage` trait if that makes the log hot path slower or more complicated; the log backend may use its own internal interfaces. + +## 25. Metrics and observability + +Add or adapt metrics for: + +### Admission + +- append latency by shard; +- pending append requests; +- pending append bytes; +- bytes appended; +- records appended; +- append errors; +- active segment size; +- segment rotations. + +### Dispatcher + +- per-shard committed head; +- per-shard discovery cursor; +- dispatcher lag in records and bytes; +- ready jobs; +- deferred jobs; +- in-flight jobs; +- oldest ready age; +- oldest deferred age; +- jobs scheduled by destination domain. + +### Storage and GC + +- live bytes; +- dead bytes; +- active-segment bytes; +- sealed segment count; +- segments deleted; +- compactions started/completed/failed; +- bytes read and written by compaction; +- relocation count; +- disk free bytes and configured reserve. + +### Delivery + +Existing delivery, retry, bounce, and latency metrics should remain meaningful. Queue time should continue to use the original enqueue timestamp even after compaction relocation. + +### Operator tooling + +The one-file-per-message spool is inspectable with `ls` and repairable with `rm`; binary segments are not. A minimal queue CLI is therefore in scope, not an afterthought: + +- list queued/deferred messages with age, attempts, and next-attempt time; +- show one message's envelope, state history, and current location; +- remove a message from the queue (recorded as an operator-cancelled terminal state); +- show per-segment and per-shard statistics (live/dead counts, garbage ratio). + +Read-only inspection must work against a live spool without stopping the server. + +## 26. Testing strategy + +### 26.1 Record format tests + +- encode/decode round trip; +- variable metadata and body sizes; +- maximum configured message size; +- checksum failures; +- unknown format version; +- truncated fixed header; +- truncated body; +- partial final record recovery; +- corruption in a sealed segment. + +### 26.2 Append writer tests + +- concurrent SMTP-side append requests; +- exclusive shard ownership; +- monotonic offsets and ordinals; +- committed tail never exposes partial records; +- rotation at target size; +- byte-bounded admission; +- append error propagation; +- notification loss does not lose work. + +### 26.3 Dispatcher tests + +- discovers all records from multiple shard heads; +- independent shard cursors; +- merges records without requiring global order; +- worker claim and completion transitions; +- stale claim generation is ignored; +- worker cancellation returns work to ready state; +- deferred jobs become ready at the correct time; +- rate-limited jobs do not occupy sleeping workers; +- rate-limited jobs are requeued without incrementing attempts and without journal writes; +- rate-limit gating at dispatch prevents one exhausted destination from monopolizing workers; +- discovery backpressure bounds dispatcher memory without losing records. + +### 26.4 Recovery tests + +- restart with ready jobs; +- restart with deferred jobs and preserved attempt counts; +- restart with in-flight jobs causes redispatch; +- restart after terminal state update; +- terminal state loss may duplicate but does not lose a live message; +- partial state-journal tail; +- checkpoint plus journal replay; +- journal truncation after checkpoint cannot resurrect a terminal record whose segment still exists; +- restart mid-retry preserves the remaining-recipient set; +- active-segment partial tail truncation; +- duplicate old/new relocation records choose the latest valid generation. + +### 26.5 GC and compaction tests + +- fully terminal segment is deleted; +- live record prevents deletion; +- dead-ratio threshold triggers compaction; +- live records survive compaction; +- terminal race during compaction does not resurrect a message; +- process interruption at each compaction step remains recoverable; +- simulated power loss between compaction copy and source unlink loses no live records (fsync barriers); +- a worker holding a stale location re-fetches and reads the relocated record; +- a segment whose last live record goes terminal is deleted without waiting for a periodic sweep; +- only one compaction runs initially; +- long-deferred record does not permanently pin mostly dead storage; +- dispatcher locations update after relocation. + +### 26.6 End-to-end tests + +- inbound acceptance continues when all workers are busy; +- inbound acceptance continues when workers are rate-limited; +- append backpressure occurs only when disk admission buffering is full; +- outbound delivery reads the correct body from a segment; +- successful delivery eventually reclaims disk; +- deferred delivery survives restart; +- retry after a partial multi-recipient failure re-sends only to remaining recipients; +- graceful shutdown drains outcomes and restarts cleanly; +- bounced-message retention does not pin active segments; +- the existing dev DNS and fake-MTA verification harness passes. + +## 27. Benchmark plan + +Benchmark before and after each major stage. + +### 27.1 Admission benchmarks + +Measure at several message sizes, including at least: + +- 1 KiB; +- 16 KiB; +- 64 KiB; +- 1 MiB where practical. + +Test: + +- append writers: `1`, `2`, `4`, `8`; +- suitable segment sizes for each writer count; +- outbound disabled; +- outbound intentionally stalled; +- tmpfs and a real local filesystem; +- steady-state append plus background compaction. + +Metrics: + +- messages per second; +- bytes per second; +- median and tail SMTP DATA latency; +- CPU usage; +- write syscall count; +- context switches; +- pending append bytes; +- dispatcher lag; +- disk amplification. + +### 27.2 Recovery benchmarks + +Measure startup with: + +- 10 thousand queued messages; +- 100 thousand queued messages; +- 1 million queued messages if practical; +- mixtures of ready, deferred, and terminal records; +- many small segments versus fewer large segments. + +### 27.3 GC benchmarks + +Measure: + +- cost of deleting fully dead segments; +- compaction throughput; +- impact of compaction on SMTP acceptance latency; +- repeated long-deferred-message relocation; +- storage amplification at 50% and 75% thresholds. + +### 27.4 Expected writer count + +Do not assume more writers are always faster. A single batched writer may already saturate the disk or memory-copy path. Multiple writers are expected to help most on NVMe and tmpfs and may hurt on rotational disks. + +The implementation supports configurable sharding, but the default is a single writer; changing that default requires benchmark evidence. + +## 28. Implementation phases + +This work is intentionally deferred and should be implemented incrementally. + +Phases 1-7 are internal milestones: the log backend must not be the active backend of a deployed build until they are all complete. Making it selectable comes last because deploying it without recovery (phase 3) or reclamation (phase 6) would redeliver mail after every restart and grow disk without bound. Until then the new code lands alongside the untouched legacy path. + +### Phase 0: preserve baselines and invariants + +- Record current acceptance, delivery, startup, retry, and disk-use behavior. +- Preserve existing benchmark tooling. +- Document current at-least-once and durability semantics. +- Add tests demonstrating that worker-channel saturation currently blocks SMTP acceptance. + +### Phase 1: on-disk format and shard primitives + +- Define versioned record and state formats and the root manifest. +- Implement record encoding, decoding, checksums, and active-tail validation. +- Implement shard directory and segment lifecycle, plus the exclusive spool lock. +- Implement positioned message reads. +- Enforce the record-never-spans-segments sizing invariant. +- Add exhaustive format and corruption tests. + +### Phase 2: append writer + +- Implement one append writer with byte-bounded admission. +- Return physical `JobLocation` after page-cache write completion. +- Implement committed-head publication and segment rotation. +- Keep the writer interface shard-capable; `N` writers stay configurable but default to 1. +- Benchmark `1`, `2`, `4`, and `8` writers before revisiting the default. + +### Phase 3: persistent state and recovery + +- Implement per-shard state journals with the persist-then-apply ordering. +- Implement checkpoints (terminal tombstones, LSN, cursors, topology) and journal replay. +- Implement startup recovery: tail truncation, checkpoint load, journal replay, payload reconciliation. +- Preserve retry attempts, next-attempt times, and remaining-recipient sets. +- Validate restart behavior for ready, deferred, in-flight, and terminal messages. + +### Phase 4: dispatcher discovery + +- Implement per-shard discovery cursors. +- Discover work from committed heads without an authoritative job channel. +- Make notifications lossy hints with cursor-based recovery. +- Reconstruct jobs from record headers without reading message bodies. +- Implement discovery backpressure as the dispatcher memory bound. + +### Phase 5: worker pull protocol and retries + +- Replace the current shared job-channel lifecycle with dispatcher claims. +- Keep job payloads out of the scheduling path; read bodies by `JobLocation` with stale-location refetch. +- Add mandatory claim generations and abandonment recovery. +- Add the deferred due-time heap; move delayed retry waiting out of workers. +- Add dispatcher-side rate-limit gating and the in-memory `RateLimited` requeue path. +- Ensure global per-domain rate limits remain correct across all workers. +- Preserve delivery logging and metrics. + +### Phase 6: deletion and compaction + +- Delete fully terminal sealed segments, triggered by the terminal transition (event-driven). +- Implement dead-ratio selection and the compaction candidate queue. +- Implement copy-before-delete compaction with unpublished output, relocation generations, manifest publication, and fsync barriers at destructive boundaries. +- Add one global compaction permit. +- Add GC metrics and failure recovery. +- Implement the disk reserve check. + +### Phase 7: bounce retention, migration, and backend selection + +- Separate bounce retention from active queue segments. +- Implement the restart-safe legacy-spool migration (idempotent, manifest-gated, locked). +- Make the log backend selectable alongside the filesystem and SQLite backends. +- Add the queue inspection CLI. +- Add explicit format/version and rollback handling. + +### Phase 8: SMTP acknowledgement cutover + +- Change the DATA callback to await only append completion when the log backend is active. +- Return `250 OK` without waiting for dispatcher or worker capacity. +- Remove startup replay through the bounded worker channel. +- Verify that stalled outbound delivery does not block acceptance until disk-admission buffering or disk reserve is reached. + +### Phase 9: production hardening and tuning + +- Run end-to-end fake-MTA verification. +- Run large-backlog startup tests. +- Run append and compaction benchmarks on representative filesystems. +- Confirm default writer count, segment size, pending-byte bound, and compaction threshold from benchmarks. +- Update production-hardening documentation and operator guidance. + +## 29. Risks and tradeoffs + +### 29.1 Implementation size + +This is a large architectural change crossing SMTP acceptance, storage, scheduling, retry handling, workers, startup recovery, metrics, and migration. It should not be attempted as a single unreviewable patch. + +### 29.2 Compaction correctness + +Compaction is the most correctness-sensitive component because it rewrites live queued data. Copy-before-delete ordering, relocation generations, race handling, and interruption tests are mandatory. + +### 29.3 Memory use + +A dispatcher entry for every live message may become large at very high queue depths. Discovery backpressure (section 12) bounds this from the start; memory-per-million-record benchmarks validate the bound, and a paged deferred index is added only if measurements demand it. + +### 29.4 Disk write amplification + +Compaction rewrites long-lived records. Thresholds and minimum ages must balance disk utilization against copy cost. + +### 29.5 More writers are not automatically faster + +Multiple active files may reduce append contention but increase active-segment slack and randomize physical writes. Benchmarking determines the useful writer count. + +### 29.6 Page-cache durability + +The no-`fsync` decision prioritizes throughput. Documentation must state clearly that a machine crash or power loss can lose recently acknowledged mail. Destructive boundaries (compaction publication, checkpoint truncation, segment deletion) do use `fsync` so power loss can never destroy old queued mail (section 5.3). + +### 29.7 Migration + +Existing queued and deferred mail must not be silently dropped. Migration needs explicit validation and rollback behavior. + +## 30. Definition of done + +The architecture is complete when: + +- SMTP acknowledgement no longer waits for worker-channel capacity; +- outbound slowdown does not block inbound acceptance except at actual disk-admission or disk-reserve boundaries; +- complete messages are stored in sharded segmented logs; +- one dispatcher schedules across all shards; +- workers pull payload-free job descriptors and read bodies by location; +- retry waits no longer consume worker slots; +- rate-limit throttling neither consumes attempts nor writes journal entries; +- retries re-send only to recipients that have not yet accepted the message; +- attempt counts, deferred times, and remaining-recipient sets survive restart; +- startup does not feed the full backlog through a bounded in-memory channel before listening; +- fully dead segments are deleted as soon as they die, without waiting for a periodic sweep; +- partially dead segments are compacted safely; +- delivered payload garbage remains bounded by the configured GC policy; +- migration or an explicit drain-before-switch path exists; +- a queue inspection CLI covers list, show, remove, and segment statistics; +- graceful shutdown drains state deterministically; +- queue, dispatcher, shard, and GC metrics are available; +- unit, recovery, compaction, integration, and end-to-end tests pass; +- benchmarks establish justified defaults for writer count and segment sizing; +- operator documentation clearly explains durability and disk-capacity behavior. + +## 31. Final intended model + +```text +Inbound throughput is limited by: + SMTP parsing + memory copies + append writers + page-cache/disk throughput + +Outbound throughput is limited by: + workers + DNS + remote MTAs + rate limits + +The durable log separates these two rates. + +One dispatcher provides: + scheduling + fairness + retries + in-flight ownership + +The segmented append log provides: + admission bounded by disk throughput; one writer by default, + shard-capable when benchmarks justify parallel writers + +Segment GC provides: + deletion of fully dead data + bounded reclamation of partial garbage +``` + +This is the architectural direction to resume when Hedwig is ready for the queue rewrite. diff --git a/docs/src/pages/reference/architecture.md b/docs/src/pages/reference/architecture.md index d049bc1..d704234 100644 --- a/docs/src/pages/reference/architecture.md +++ b/docs/src/pages/reference/architecture.md @@ -8,97 +8,95 @@ description: Hedwig SMTP server architecture and data flow. ## Overview -Hedwig is a high-performance, async SMTP server written in Rust that provides email relay functionality with advanced features including DKIM signing, retry mechanisms, and a modular storage layer. The server is designed with a modular architecture that separates concerns and provides extensibility. +Hedwig is a high-performance, async SMTP server written in Rust that provides +email relay functionality with DKIM signing, retry mechanisms, MTA-STS +enforcement, and a modular storage layer. -## High-level architecture +For the deep, diagrammed walkthrough of the durable log queue — on-disk +formats, scheduling, crash recovery, garbage collection, and the durability +model — see [`ARCHITECTURE.md`](https://github.com/iamd3vil/hedwig/blob/main/ARCHITECTURE.md) +in the repository root. This page is the short orientation. -``` -SMTP Listeners → SMTP Callbacks → Storage Queue → Workers → Outbound SMTP Pool -``` - -## Core components - -### Main server -- Configuration loading -- TLS setup -- Listener initialization -- Worker initialization +## Two queue architectures -### SMTP callbacks -- Authentication -- Domain filtering -- Rate limiting -- Path validation +Hedwig has two queueing paths, selected by `storage.storage_type`: -### Worker system -- Parse email -- DKIM sign -- MX lookup -- Send via SMTP +### Log queue (`storage_type = "log"`) -### Outbound pool -- Per-domain pooling -- TLS configuration -- Connection limits - -## Data flow - -### Inbound flow -1. Client connection -2. SMTP negotiation -3. Email stored in queue -4. Processing job created +``` +SMTP listeners → callbacks → append writers → segmented log on disk + │ + dispatcher (discovery, retries, + rate gating, GC/compaction) + │ + workers pull claims + │ + outbound SMTP pool +``` -### Outbound flow -1. Worker receives job -2. Parse email, remove BCC, sign DKIM -3. Resolve MX records -4. Deliver via SMTP pool +Complete messages are appended to sharded, segmented logs; `250 OK` is +returned as soon as the record reaches the kernel page cache. A single +dispatcher discovers appended records via per-shard cursors, hands +payload-free claims to pulling workers, schedules retries by due time, and +reclaims disk (fully delivered segments are deleted immediately; partially +dead ones are compacted). Inbound acceptance is bounded by disk append +throughput — slow or stalled outbound cannot block it. + +- Retries: exponential backoff scheduled in-process (no directory scans); + after `max_retries` the message bounces terminally. +- Partial multi-recipient failures re-send only to recipients that have not + yet accepted the message. +- Bounced messages are archived as plain files under `bounced/` and honor + `[storage.cleanup]` retention. +- Durability: process crashes lose nothing; a machine crash or power loss + may lose the most recently acknowledged messages (documented tradeoff). + Destructive operations are fsync-guarded, so older queued mail is never + at risk. +- Inspect a live spool with `hedwig queue list|show|stats`; migrate a legacy + spool with `hedwig queue migrate`. + +### Legacy queue (`storage_type = "fs"`) -## Storage architecture +``` +SMTP listeners → callbacks → storage (one file/row per message) + │ + bounded in-memory job channel + │ + workers consume jobs +``` -The storage trait supports multiple backends. Current implementation is filesystem-based. +One file per message plus a bounded channel between acceptance and +workers. Acceptance waits for a channel slot, so sustained slow outbound +eventually backpressures inbound. Deferred messages are re-queued by a +periodic scan. This path is unchanged and remains the default. -Directory structure: +## Core components -``` -/base_path/ - queued/ - deferred/ - bounced/ - meta/ -``` +- **Main server** — configuration, TLS setup, listeners, worker/dispatcher + startup, graceful shutdown. +- **SMTP callbacks** — authentication, domain filtering, disk-reserve check, + queue admission. +- **Workers** — parse, strip Bcc, DKIM-sign, resolve MX, apply MTA-STS, + deliver through the per-domain connection pool, classify outcomes. +- **Outbound pool** — per-domain pooling, TLS, connection limits, MX caching. ## Security -- Inbound TLS per listener +- Inbound TLS per listener (implicit or STARTTLS) - Optional SMTP AUTH - DKIM signing (RSA or Ed25519) +- MTA-STS policy enforcement - Domain allow/deny filters -## Performance - -- Tokio-based async runtime -- Multi-listener concurrency -- Per-domain connection pooling -- MX record caching -- Bounded channels for backpressure - ## Monitoring -- Structured logging via tracing -- Prometheus metrics when enabled -- Operational signals for queue depth and delivery rates +- Structured logging via tracing (JSON or plain) +- Prometheus metrics when enabled, including queue depth, dispatcher lag, + segment/GC statistics, and delivery outcomes ## Deployment -- File-based configuration +- File-based configuration (TOML or HUML) - Environment override for log level -- Graceful shutdown and queue recovery -- File system permissions for storage and keys - -## Extension points - -- Additional storage backends (DB, S3) -- Alternative auth methods (LDAP, OAuth2) -- External filtering integrations +- Graceful shutdown with state checkpointing and queue recovery +- Exclusive spool lock: one process per spool root diff --git a/docs/src/pages/storage.md b/docs/src/pages/storage.md index 688f0c9..146cd81 100644 --- a/docs/src/pages/storage.md +++ b/docs/src/pages/storage.md @@ -8,7 +8,7 @@ description: Configure filesystem storage and retention policies. ```toml [storage] -storage_type = "fs" # Storage type: "fs" (filesystem) +storage_type = "log" # "log" (default) or "fs" (legacy filesystem) base_path = "/var/spool/hedwig" # Base directory for email storage [storage.cleanup] @@ -20,3 +20,58 @@ interval = "1h" # Run the cleanup task hourly - All keys inside `[storage.cleanup]` are optional; omit them to disable specific cleanups - Retention values accept human-readable durations (e.g., `"24h"`, `"5m"`) - The cleanup task runs on a background interval and also executes once during startup + +## Log queue backend + +`storage_type = "log"` — the default when `storage_type` is omitted — +selects the durable log queue: complete messages are +stored in segmented append-only logs under `/spool/`, and SMTP +`250 OK` is returned as soon as the record is written — inbound acceptance +is bounded by disk append throughput, not by outbound delivery speed. +Retries are scheduled by due time (no periodic spool scans), and retries +after a partial multi-recipient failure re-send only to the recipients that +have not yet accepted the message. + +```toml +[storage] +storage_type = "log" +base_path = "/var/spool/hedwig" # spool/ (queue) + bounced/ (archive) + +[queue] # all optional; defaults shown +append_writers = 1 # log shards; change requires an empty queue +pending_append_bytes = 134217728 # in-memory admission buffer bound (bytes) +segment_target_bytes = 67108864 # seal active segments at this size +compaction_dead_ratio = 0.5 # compact sealed segments this dead +compaction_min_age = "60s" # leave fresh segments alone +disk_reserve_bytes = 1073741824 # reject mail when free disk drops below +checkpoint_interval_bytes = 8388608 # checkpoint cadence per shard +``` + +**Durability tradeoff:** acknowledged mail is durable against process +crashes and restarts, but the queue does not `fsync` per message — a machine +crash or power loss can lose the most recently acknowledged messages. This +is a deliberate throughput tradeoff. Destructive operations (segment +deletion, compaction, checkpoint truncation) do use `fsync` barriers, so a +power loss can never destroy older queued mail. + +**Disk behavior:** disk use is proportional to the live backlog plus +not-yet-reclaimed garbage. Fully delivered segments are deleted as soon as +their last message completes; partially dead segments are compacted once +`compaction_dead_ratio` is exceeded. When free space falls below +`disk_reserve_bytes`, new mail is rejected with a transient `452`. + +Bounced messages are archived as regular files under `bounced/` (same layout +as the filesystem backend) and honor `[storage.cleanup]` retention. + +Inspect a spool — including a live one — with the built-in CLI: + +```sh +hedwig queue list --spool /var/spool/hedwig/spool +hedwig queue show --spool /var/spool/hedwig/spool +hedwig queue stats --spool /var/spool/hedwig/spool +``` + +To migrate an existing filesystem spool, stop the server, set +`storage_type = "log"`, then run `hedwig queue migrate --config `; +legacy `queued/` and `deferred/` directories are preserved as timestamped +`.migrated-*` backups. diff --git a/smtp-server/Cargo.toml b/smtp-server/Cargo.toml index 9734f13..c3d4565 100644 --- a/smtp-server/Cargo.toml +++ b/smtp-server/Cargo.toml @@ -62,13 +62,12 @@ rand = "0.8" ed25519-dalek = { version = "2", features = ["pkcs8"] } pem = "3" memchr = "2.7.4" -# Note: in sqlx 0.8 the "sqlite" feature already bundles libsqlite3 statically. -# There is no separate "sqlite-bundled" feature; use "sqlite-unbundled" to link -# against the system library instead. -sqlx = { version = "0.8", features = ["runtime-tokio", "sqlite"] } mailparse = "0.16.1" humantime-serde = "1.1" chrono = { version = "0.4", features = ["serde"] } hyper = { version = "0.14", features = ["full"] } once_cell = "1.19.0" prometheus = "0.13" +crc32fast = "1.4" +bytes = "1" +libc = "0.2" diff --git a/smtp-server/src/callbacks.rs b/smtp-server/src/callbacks.rs index c483493..77acc88 100644 --- a/smtp-server/src/callbacks.rs +++ b/smtp-server/src/callbacks.rs @@ -40,12 +40,23 @@ fn cram_md5_digest(password: &str, challenge: &str) -> String { .collect() } +/// The log-queue acceptance path: SMTP DATA appends to the durable log and +/// acknowledges on page-cache write completion, never waiting for workers. +pub struct LogQueueTap { + pub append: crate::logqueue::writer::AppendHandle, + pub spool_root: std::path::PathBuf, + pub disk_reserve_bytes: u64, +} + /// The Callbacks struct holds the configuration, storage, and sender channel. pub struct Callbacks { cfg: Cfg, auth_mapping: Mutex>, storage: Arc, sender_channel: async_channel::Sender, + /// `Some` when the log-queue backend is active: DATA goes to the append + /// log and the legacy storage/channel path is bypassed. + log_queue: Option, } /// Extracts the lowercased domain from an SMTP path like ``. @@ -147,61 +158,10 @@ impl Callbacks { receiver_channel: async_channel::Receiver, cfg: Cfg, ) -> miette::Result<(Self, Vec>, Arc)> { - let expiry = MXExpiry; - let mx_cache: Cache<_, _> = Cache::builder() - .max_capacity(10000) - .expire_after(expiry) - .build(); - - // Create a shared DNS resolver for workers and MTA-STS. - let resolver = TokioAsyncResolver::tokio_from_system_conf() - .into_diagnostic() - .wrap_err("failed to create DNS resolver")?; - - // Create the shared MTA-STS resolver. - let mta_sts_fetcher = MtaStsFetcher::new(resolver.clone()); - let mta_sts_resolver = Arc::new(MtaStsResolver::new(mta_sts_fetcher)); - - // Start workers. - let worker_count = cfg.server.workers.unwrap_or(1).max(1); - let rate_limit_config = cfg - .server - .rate_limits - .as_ref() - .map(|rl| rl.to_rate_limit_config()) - .unwrap_or_default(); + let (worker_resources, mta_sts_resolver) = build_worker_resources(&cfg)?; + let worker_count = cfg.server.workers.unwrap_or(1).max(1); let mut worker_handles = Vec::new(); - let helo_hostname = cfg.server.helo_hostname.clone(); - if let Some(name) = helo_hostname.as_deref() { - if !name.contains('.') { - warn!( - helo_hostname = %name, - "configured HELO/EHLO hostname does not look like a public FQDN" - ); - } - } - - let smtp_pool = build_smtp_pool_config(&cfg); - info!( - smtp_cache_size = smtp_pool.cache_size, - smtp_pool_min_idle = smtp_pool.min_idle, - smtp_pool_max_size = smtp_pool.max_size, - "configured outbound SMTP pool" - ); - let smtp_pool_manager = Arc::new(worker::PoolManager::new( - smtp_pool, - cfg.server.outbound_local.unwrap_or(false), - helo_hostname, - )); - let worker_resources = worker::WorkerResources::new( - mx_cache, - smtp_pool_manager, - resolver, - Arc::clone(&mta_sts_resolver), - rate_limit_config, - ); - for worker_index in 0..worker_count { let receiver_channel = receiver_channel.clone(); let storage_cloned = storage.clone(); @@ -239,13 +199,90 @@ impl Callbacks { sender_channel, cfg, auth_mapping: Mutex::new(auth_mapping), + log_queue: None, }; Ok((callbacks, worker_handles, mta_sts_resolver)) } + /// Creates callbacks for the log-queue backend. No legacy workers are + /// spawned and the job channel is inert; the caller wires the returned + /// [`worker::WorkerResources`] into log workers instead. `storage` is + /// used only as the bounced-message archive. + pub async fn new_log( + storage: Arc, + tap: LogQueueTap, + cfg: Cfg, + ) -> miette::Result<(Self, worker::WorkerResources, Arc)> { + let (worker_resources, mta_sts_resolver) = build_worker_resources(&cfg)?; + + let mut auth_mapping = HashMap::new(); + if let Some(auth) = &cfg.server.auth { + for auth in auth.iter() { + auth_mapping.insert(auth.username.clone(), auth.password.clone()); + } + } + + // The legacy channel field is inert on this path. + let (sender_channel, _) = async_channel::bounded(1); + let callbacks = Callbacks { + storage, + sender_channel, + cfg, + auth_mapping: Mutex::new(auth_mapping), + log_queue: Some(tap), + }; + Ok((callbacks, worker_resources, mta_sts_resolver)) + } + + /// Log-queue DATA path: check the disk reserve, then append and wait + /// only for page-cache write completion (PLAN §28 phase 8). Worker and + /// dispatcher capacity never gate acceptance. + async fn process_email_log(&self, tap: &LogQueueTap, email: Email) -> Result<(), SmtpError> { + if tap.disk_reserve_bytes > 0 { + match crate::logqueue::spool::disk_free_bytes(&tap.spool_root) { + Ok(free) if free < tap.disk_reserve_bytes => { + warn!( + free_bytes = free, + reserve_bytes = tap.disk_reserve_bytes, + "disk reserve reached; rejecting message" + ); + return Err(SmtpError::Transient { + message: "4.3.1 insufficient system storage".into(), + }); + } + Ok(_) => {} + Err(e) => { + // Fail open: a broken statvfs must not stop mail flow. + warn!(error = %e, "disk reserve check failed"); + } + } + } + + let message_id = crate::logqueue::MessageId::from_ulid(Ulid::new()); + let msg = crate::logqueue::writer::AppendMessage { + message_id, + enqueue_ms: Utc::now().timestamp_millis(), + generation: 0, + sender: email.from, + recipients: email.to, + body: bytes::Bytes::from(email.body.into_bytes()), + }; + tap.append.append(msg).await.map_err(|e| { + warn!(error = %e, "append to log queue failed"); + SmtpError::Transient { + message: "4.3.0 queue write failed, try again later".into(), + } + })?; + metrics::email_received(); + Ok(()) + } + /// Processes an email by parsing it, storing it, and sending it to a worker. async fn process_email(&self, email: Email) -> Result<(), SmtpError> { + if let Some(tap) = &self.log_queue { + return self.process_email_log(tap, email).await; + } let ulid = Ulid::new().to_string(); // We are using ulid as the message id instead of message_id from the email. // The issue is we can't depend on the email client to provide a unique message id. @@ -281,6 +318,68 @@ impl Callbacks { } } +/// Builds the delivery resources shared by every worker flavor: MX cache, +/// DNS resolver, MTA-STS resolver, outbound SMTP pool, and the process-wide +/// rate limiter. +fn build_worker_resources( + cfg: &Cfg, +) -> miette::Result<(worker::WorkerResources, Arc)> { + let expiry = MXExpiry; + let mx_cache: Cache<_, _> = Cache::builder() + .max_capacity(10000) + .expire_after(expiry) + .build(); + + // Create a shared DNS resolver for workers and MTA-STS. + let resolver = TokioAsyncResolver::tokio_from_system_conf() + .into_diagnostic() + .wrap_err("failed to create DNS resolver")?; + + // Create the shared MTA-STS resolver. + let mta_sts_fetcher = MtaStsFetcher::new(resolver.clone()); + let mta_sts_resolver = Arc::new(MtaStsResolver::new(mta_sts_fetcher)); + + let rate_limit_config = cfg + .server + .rate_limits + .as_ref() + .map(|rl| rl.to_rate_limit_config()) + .unwrap_or_default(); + + let helo_hostname = cfg.server.helo_hostname.clone(); + if let Some(name) = helo_hostname.as_deref() { + if !name.contains('.') { + warn!( + helo_hostname = %name, + "configured HELO/EHLO hostname does not look like a public FQDN" + ); + } + } + + let smtp_pool = build_smtp_pool_config(cfg); + info!( + smtp_cache_size = smtp_pool.cache_size, + smtp_pool_min_idle = smtp_pool.min_idle, + smtp_pool_max_size = smtp_pool.max_size, + "configured outbound SMTP pool" + ); + let smtp_pool_manager = Arc::new(worker::PoolManager::new( + smtp_pool, + cfg.server.outbound_local.unwrap_or(false), + helo_hostname, + )); + Ok(( + worker::WorkerResources::new( + mx_cache, + smtp_pool_manager, + resolver, + Arc::clone(&mta_sts_resolver), + rate_limit_config, + ), + mta_sts_resolver, + )) +} + fn build_smtp_pool_config(cfg: &Cfg) -> worker::SmtpPoolConfig { let smtp = cfg.server.smtp.as_ref(); let cache_size = smtp @@ -697,12 +796,9 @@ mod tests { storage_type: "mock".to_string(), base_path: "/tmp/hedwig".to_string(), cleanup: None, - num_shards: None, - batch_size: None, - batch_timeout_ms: None, - sqlite: None, }, filters, + queue: None, }; let (sender_channel, receiver_channel) = async_channel::unbounded::(); @@ -1572,12 +1668,9 @@ mod tests { storage_type: "memory".to_string(), base_path: "/tmp".to_string(), cleanup: None, - num_shards: None, - batch_size: None, - batch_timeout_ms: None, - sqlite: None, }, filters: None, + queue: None, } } diff --git a/smtp-server/src/config.rs b/smtp-server/src/config.rs index 2c1e5a0..c31c383 100644 --- a/smtp-server/src/config.rs +++ b/smtp-server/src/config.rs @@ -30,6 +30,7 @@ pub struct Cfg { pub server: CfgServer, pub storage: CfgStorage, pub filters: Option>, + pub queue: Option, } #[derive(Debug, Deserialize, Clone)] @@ -71,18 +72,13 @@ pub struct CfgFilter { #[derive(Debug, Deserialize, Clone)] pub struct CfgStorage { + /// "log" (durable log queue, the default) or "fs" (legacy one file per + /// message). + #[serde(default = "default_storage_type")] pub storage_type: String, pub base_path: String, #[serde(default)] pub cleanup: Option, - /// Number of SQLite shards (default: 16). Only used when storage_type = "sqlite". - pub num_shards: Option, - /// Max writes per batch (default: 100). Only used when storage_type = "sqlite". - pub batch_size: Option, - /// Max wait to fill a batch in ms (default: 5). Only used when storage_type = "sqlite". - pub batch_timeout_ms: Option, - /// SQLite-specific tuning. Only used when storage_type = "sqlite". - pub sqlite: Option, } #[derive(Debug, Deserialize, Clone, Default)] @@ -177,18 +173,6 @@ pub struct CfgRateLimits { pub domain_limits: Option>, } -#[derive(Debug, Deserialize, Clone, Default)] -pub struct CfgSqlite { - /// SQLite synchronous mode: OFF | NORMAL | FULL (default: NORMAL) - pub synchronous: Option, - /// Total cache size in MB across all shards (default: 1600) - pub cache_size_mb: Option, - /// SQLite busy timeout in ms (default: 5000) - pub busy_timeout_ms: Option, - /// Read connections per shard (default: 10) - pub pool_max_connections: Option, -} - /// Configuration for on-disk spool cleanup. #[derive(Debug, Deserialize, Clone)] pub struct CfgCleanup { @@ -200,7 +184,88 @@ pub struct CfgCleanup { pub interval: Duration, } +/// Configuration for the durable append-log mail queue (see +/// docs/plans/2026-07-20-durable-log-queue.md). +#[derive(Debug, Deserialize, Clone, Default)] +pub struct CfgQueue { + /// Number of shards / concurrent append writers (default: 1). + pub append_writers: Option, + /// Bytes of pending (not-yet-durable) append data allowed before backpressure (default: 128 MiB). + pub pending_append_bytes: Option, + /// Target size of each active segment file, per shard (default: 64 MiB). + pub segment_target_bytes: Option, + /// Fraction of dead bytes in a sealed segment that makes it eligible for compaction (default: 0.50). + pub compaction_dead_ratio: Option, + /// Minimum age of a sealed segment before it is eligible for compaction (default: 60s). + #[serde(default, with = "humantime_serde::option")] + pub compaction_min_age: Option, + /// Minimum free disk space required to accept new mail (default: 1 GiB). + pub disk_reserve_bytes: Option, + /// Bytes of appended data between durability checkpoints (default: 8 MiB). + pub checkpoint_interval_bytes: Option, +} + +impl CfgQueue { + pub fn append_writers(&self) -> u16 { + self.append_writers.unwrap_or(1) + } + + pub fn pending_append_bytes(&self) -> u64 { + self.pending_append_bytes.unwrap_or(128 * 1024 * 1024) + } + + pub fn segment_target_bytes(&self) -> u64 { + self.segment_target_bytes.unwrap_or(64 * 1024 * 1024) + } + + pub fn compaction_dead_ratio(&self) -> f64 { + self.compaction_dead_ratio.unwrap_or(0.50) + } + + pub fn compaction_min_age(&self) -> Duration { + self.compaction_min_age.unwrap_or(Duration::from_secs(60)) + } + + pub fn disk_reserve_bytes(&self) -> u64 { + self.disk_reserve_bytes.unwrap_or(1024 * 1024 * 1024) + } + + pub fn checkpoint_interval_bytes(&self) -> u64 { + self.checkpoint_interval_bytes.unwrap_or(8 * 1024 * 1024) + } + + /// Validate the resolved configuration. `max_message_size` is the + /// server's configured (or defaulted) maximum message size in bytes, + /// since the segment target must be able to hold one worst-case record. + pub fn validate(&self, max_message_size: usize) -> miette::Result<()> { + if self.append_writers() < 1 { + return Err(miette::miette!("queue.append_writers must be at least 1")); + } + + let ratio = self.compaction_dead_ratio(); + if !(ratio > 0.0 && ratio < 1.0) { + return Err(miette::miette!( + "queue.compaction_dead_ratio must be between 0.0 and 1.0 (exclusive), got {ratio}" + )); + } + + crate::logqueue::spool::check_segment_sizing( + self.segment_target_bytes(), + max_message_size as u64, + ) + .map_err(miette::Report::new)?; + + Ok(()) + } +} + impl Cfg { + /// The resolved queue configuration, defaulted if the `[queue]` section + /// is absent from the loaded configuration. + pub fn queue(&self) -> CfgQueue { + self.queue.clone().unwrap_or_default() + } + pub fn load(cfg_path: &str) -> Result { let path = Path::new(cfg_path); @@ -258,3 +323,111 @@ impl CfgStorage { fn default_cleanup_interval() -> Duration { Duration::from_secs(60 * 60) } + +/// The durable log queue is the default backend. +fn default_storage_type() -> String { + "log".to_string() +} + +#[cfg(test)] +mod tests { + use super::*; + use config::FileFormat; + + /// Smallest configuration that satisfies every required (non-`Option`) + /// field on `Cfg`, so tests can focus on the `[queue]` section alone. + const MINIMAL_CFG: &str = r#" + [server] + listeners = [] + + [storage] + storage_type = "fs" + base_path = "/tmp/hedwig-test-spool" + "#; + + fn parse(toml: &str) -> Cfg { + let settings = Config::builder() + .add_source(File::from_str(toml, FileFormat::Toml)) + .build() + .expect("build config"); + settings.try_deserialize().expect("deserialize config") + } + + #[test] + fn storage_type_defaults_to_log() { + let cfg = parse( + r#" + [server] + listeners = [] + + [storage] + base_path = "/tmp/hedwig-test-spool" + "#, + ); + assert_eq!(cfg.storage.storage_type, "log"); + } + + #[test] + fn queue_defaults_when_section_absent() { + let cfg = parse(MINIMAL_CFG); + assert!(cfg.queue.is_none()); + + let queue = cfg.queue(); + assert_eq!(queue.append_writers(), 1); + assert_eq!(queue.pending_append_bytes(), 128 * 1024 * 1024); + assert_eq!(queue.segment_target_bytes(), 64 * 1024 * 1024); + assert_eq!(queue.compaction_dead_ratio(), 0.50); + assert_eq!(queue.compaction_min_age(), Duration::from_secs(60)); + assert_eq!(queue.disk_reserve_bytes(), 1024 * 1024 * 1024); + assert_eq!(queue.checkpoint_interval_bytes(), 8 * 1024 * 1024); + } + + #[test] + fn queue_section_parses_from_toml() { + let toml = format!( + r#"{MINIMAL_CFG} + [queue] + append_writers = 4 + pending_append_bytes = 1048576 + segment_target_bytes = 16777216 + compaction_dead_ratio = 0.75 + compaction_min_age = "30s" + disk_reserve_bytes = 2147483648 + checkpoint_interval_bytes = 4194304 + "# + ); + let cfg = parse(&toml); + let queue = cfg.queue.expect("queue section present"); + assert_eq!(queue.append_writers(), 4); + assert_eq!(queue.pending_append_bytes(), 1_048_576); + assert_eq!(queue.segment_target_bytes(), 16_777_216); + assert_eq!(queue.compaction_dead_ratio(), 0.75); + assert_eq!(queue.compaction_min_age(), Duration::from_secs(30)); + assert_eq!(queue.disk_reserve_bytes(), 2_147_483_648); + assert_eq!(queue.checkpoint_interval_bytes(), 4_194_304); + } + + #[test] + fn validate_accepts_defaults() { + let queue = CfgQueue::default(); + assert!(queue.validate(25 * 1024 * 1024).is_ok()); + } + + #[test] + fn validate_rejects_dead_ratio_out_of_range() { + let queue = CfgQueue { + compaction_dead_ratio: Some(1.5), + ..CfgQueue::default() + }; + assert!(queue.validate(25 * 1024 * 1024).is_err()); + } + + #[test] + fn validate_rejects_segment_smaller_than_max_message_size() { + let queue = CfgQueue { + segment_target_bytes: Some(1024), + ..CfgQueue::default() + }; + assert!(queue.validate(25 * 1024 * 1024).is_err()); + } +} diff --git a/smtp-server/src/logqueue/dispatcher.rs b/smtp-server/src/logqueue/dispatcher.rs new file mode 100644 index 0000000..f507758 --- /dev/null +++ b/smtp-server/src/logqueue/dispatcher.rs @@ -0,0 +1,1973 @@ +//! The dispatcher: one task scheduling delivery across all shards. +//! +//! It discovers appended records via per-shard cursors against the writers' +//! committed chains (notifications are lossy hints), hands payload-free +//! claims to pulling workers, persists every outcome through the shard's +//! state journal before applying it (persist-then-apply), schedules retries +//! by due time, and gates dispatch on the per-domain rate limiter so +//! throttled destinations never occupy worker slots. + +use std::cmp::Reverse; +use std::collections::{BinaryHeap, HashMap, HashSet, VecDeque}; +use std::path::PathBuf; +use std::sync::Arc; +use std::time::Duration; + +use tokio::sync::{mpsc, oneshot}; +use tokio_util::sync::CancellationToken; + +use super::segment::{open_segment_reader, scan_headers, SegmentReader}; +use super::state::{ + Checkpoint, DeferredJob, PendingCheckpoint, ReadyJob, RecoveredState, SegmentStats, + ShardStateStore, StateEntry, +}; +use super::writer::ShardShared; +use super::{JobLocation, MessageId, QueueError}; + +/// Dispatch-time rate limiting. `check` returns how long the domain is +/// exhausted for, or `None` when a send is allowed (and accounted). +pub trait RateGate: Send + Sync { + fn check(&self, domain: &str) -> Option; +} + +/// Allows everything. The server always installs `LimiterGate` (which +/// itself allows everything when rate limits are disabled); tests use this +/// to bypass gating entirely. +#[cfg(test)] +pub struct NoRateGate; + +#[cfg(test)] +impl RateGate for NoRateGate { + fn check(&self, _domain: &str) -> Option { + None + } +} + +#[derive(Debug, Clone)] +pub struct DispatcherConfig { + /// Discovery backpressure: stop advancing cursors while this many jobs + /// are tracked in memory. The log holds everything beyond it. + pub max_tracked_jobs: usize, + /// Fallback wake-up for lost notifications and persist retries. + pub safety_tick: Duration, + /// Checkpoint a shard once its journal grows past this many bytes. + pub checkpoint_interval_bytes: u64, + /// Compact a sealed segment once this fraction of its bytes is dead. + pub compaction_dead_ratio: f64, + /// Leave freshly sealed segments alone for this long. + pub compaction_min_age: Duration, +} + +impl Default for DispatcherConfig { + fn default() -> Self { + Self { + max_tracked_jobs: 100_000, + safety_tick: Duration::from_millis(500), + checkpoint_interval_bytes: 8 * 1024 * 1024, + compaction_dead_ratio: 0.5, + compaction_min_age: Duration::from_secs(60), + } + } +} + +/// What a worker receives: identity, location, and delivery metadata — +/// never the message body. +#[derive(Debug, Clone)] +pub struct DeliveryJob { + pub message_id: MessageId, + pub location: JobLocation, + pub attempts: u32, + pub claim_generation: u64, + pub enqueue_ms: i64, + pub sender: String, + /// Recipients that have not yet accepted the message. + pub recipients: Vec, +} + +/// A worker's report for one claim. +#[derive(Debug)] +pub enum JobOutcome { + Delivered { + response: String, + }, + /// A real failed attempt: increments the attempt count, persists the + /// remaining recipient set, and schedules the retry by due time. + Deferred { + next_attempt_ms: i64, + remaining_recipients: Vec, + error: String, + }, + /// Lost the transmission-time rate-limit race: requeued in memory only. + /// No journal write, no attempt increment. + RateLimited { + retry_after: Duration, + }, + Bounced { + reason: String, + }, +} + +enum WorkerEvent { + Outcome { + id: MessageId, + generation: u64, + outcome: JobOutcome, + }, + Abandoned { + id: MessageId, + generation: u64, + }, +} + +/// A claimed job. Report exactly one outcome; dropping the claim without +/// reporting counts as abandonment and returns the job to the ready queue. +pub struct Claim { + pub job: DeliveryJob, + events: mpsc::UnboundedSender, + reported: bool, +} + +impl Claim { + pub fn report(mut self, outcome: JobOutcome) { + self.reported = true; + let _ = self.events.send(WorkerEvent::Outcome { + id: self.job.message_id, + generation: self.job.claim_generation, + outcome, + }); + } +} + +impl Drop for Claim { + fn drop(&mut self) { + if !self.reported { + let _ = self.events.send(WorkerEvent::Abandoned { + id: self.job.message_id, + generation: self.job.claim_generation, + }); + } + } +} + +type ClaimWaiter = oneshot::Sender>; + +/// Cloneable handle workers use to pull claims and read message bodies. +#[derive(Clone)] +pub struct DispatcherHandle { + claim_tx: mpsc::Sender, + shard_dirs: Arc>, +} + +impl DispatcherHandle { + /// Pull the next claim, waiting until one is available. `None` means + /// the dispatcher is shutting down and the worker should exit. + pub async fn claim(&self) -> Option { + let (tx, rx) = oneshot::channel(); + self.claim_tx.send(tx).await.ok()?; + rx.await.ok().flatten() + } + + /// Read and verify a message body by its location (blocking I/O runs on + /// a blocking task). Returns the body bytes. + pub async fn read_body(&self, location: JobLocation) -> Result, QueueError> { + let dir = self.shard_dirs[location.shard as usize].clone(); + tokio::task::spawn_blocking(move || { + let reader = open_segment_reader(&dir, location.segment)?; + let (_, body) = + reader.read_record_at(location.offset, super::record::MAX_RECORD_LEN)?; + Ok(body) + }) + .await + .expect("read_body task panicked") + } +} + +/// Everything the dispatcher needs to run one shard. +pub struct ShardInit { + pub dir: PathBuf, + pub shared: Arc, + pub store: ShardStateStore, + pub recovered: RecoveredState, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum JobState { + Ready, + InFlight(u64), + /// Waiting for a due time: a persisted deferral (real attempt) or an + /// in-memory rate-limit hold (never persisted). + Delayed { due_ms: i64, persisted: bool }, +} + +struct Job { + location: JobLocation, + attempts: u32, + enqueue_ms: i64, + /// Remaining recipients when a partial delivery has happened; `None` + /// means the full envelope from the payload header. + remaining: Option>, + last_error: Option, + /// Envelope cache filled on first dispatch (sender, all recipients). + envelope: Option<(String, Vec)>, + state: JobState, +} + +struct ShardRuntime { + shard: u16, + dir: PathBuf, + shared: Arc, + store: ShardStateStore, + /// Next undiscovered position. `None` until first initialized from the + /// chain (fresh shard with no checkpoint). + cursor: Option<(u64, u64)>, + tombstones: HashMap>, + stats: HashMap, + readers: HashMap, + /// Outcomes whose journal write failed; retried on the safety tick. + /// Jobs referenced here stay in flight (never lost, never terminal + /// without persistence). + pending_persists: VecDeque, + checkpoint: Option, +} + +impl ShardRuntime { + fn reader(&mut self, segment: u64) -> Result<&SegmentReader, QueueError> { + if !self.readers.contains_key(&segment) { + let r = open_segment_reader(&self.dir, segment)?; + self.readers.insert(segment, r); + } + Ok(self.readers.get(&segment).unwrap()) + } +} + +fn now_ms() -> i64 { + chrono::Utc::now().timestamp_millis() +} + +/// Domain of an envelope recipient, tolerating angle brackets +/// (``), normalized the same way the delivery worker +/// does so the rate gate and the limiter share one bucket per domain. +fn domain_of(address: &str) -> &str { + let address = address.trim_matches(|c| c == '<' || c == '>'); + address.rsplit_once('@').map(|(_, d)| d).unwrap_or(address) +} + +/// An in-progress compaction: live records of one segment being re-appended +/// through the normal writer path with a bumped relocation generation. Only +/// one runs at a time (PLAN §17.4). +struct CompactionRun { + shard_idx: usize, + segment: u64, + queue: VecDeque, +} + +pub struct Dispatcher { + shards: Vec, + jobs: HashMap, + ready: BinaryHeap>, + delayed: BinaryHeap>, + waiting: VecDeque, + inflight: usize, + next_claim_generation: u64, + gate: Arc, + config: DispatcherConfig, + append: super::writer::AppendHandle, + compaction: Option, + events_tx: mpsc::UnboundedSender, + cp_tx: mpsc::UnboundedSender<(u16, Result<(), QueueError>)>, +} + +impl Dispatcher { + /// Build the dispatcher from recovered shard state and spawn its task. + pub fn start( + shard_inits: Vec, + append: super::writer::AppendHandle, + gate: Arc, + config: DispatcherConfig, + cancel: CancellationToken, + ) -> (DispatcherHandle, tokio::task::JoinHandle<()>) { + let (claim_tx, claim_rx) = mpsc::channel(1024); + let (events_tx, events_rx) = mpsc::unbounded_channel(); + let (cp_tx, cp_rx) = mpsc::unbounded_channel(); + + let shard_dirs = Arc::new(shard_inits.iter().map(|s| s.dir.clone()).collect::>()); + let handle = DispatcherHandle { + claim_tx, + shard_dirs, + }; + + let mut dispatcher = Dispatcher { + shards: Vec::with_capacity(shard_inits.len()), + jobs: HashMap::new(), + ready: BinaryHeap::new(), + delayed: BinaryHeap::new(), + waiting: VecDeque::new(), + inflight: 0, + next_claim_generation: 0, + gate, + config, + append, + compaction: None, + events_tx, + cp_tx, + }; + for init in shard_inits { + dispatcher.add_shard(init); + } + + let task = tokio::spawn(dispatcher.run(claim_rx, events_rx, cp_rx, cancel)); + (handle, task) + } + + fn add_shard(&mut self, init: ShardInit) { + let ShardInit { + dir, + shared, + store, + mut recovered, + } = init; + let chain = shared.chain(); + + // Reconcile checkpoint state with the writer-validated chain. A torn + // active tail was truncated during writer recovery, so a checkpoint + // written just before the crash can reference positions past the + // committed head: + // - a cursor past the head would skip every record appended after + // restart (stranding accepted mail forever) — clamp it back; + // - a job whose payload sat in the truncated tail no longer exists + // on disk; the append was never completed-and-acknowledged (or + // was lost within the accepted page-cache window), so drop it. + let committed_of = |segment: u64| -> Option { + chain.iter().find(|h| h.segment == segment).map(|h| h.committed) + }; + if let Some((seg, off)) = recovered.cursor { + match committed_of(seg) { + Some(committed) if off > committed => { + tracing::warn!( + shard = shared.shard(), + segment = seg, + cursor = off, + committed, + "checkpoint cursor is past the validated tail; clamping" + ); + recovered.cursor = Some((seg, committed)); + } + _ => {} + } + } + let payload_gone = |location: &JobLocation| -> bool { + matches!( + committed_of(location.segment), + Some(committed) if location.offset + location.length as u64 > committed + ) + }; + recovered.ready.retain(|id, r| { + if payload_gone(&r.location) { + tracing::warn!(message_id = %id, location = ?r.location, + "dropping recovered ready job whose payload was truncated with the torn tail"); + false + } else { + true + } + }); + recovered.deferred.retain(|id, d| { + if payload_gone(&d.location) { + tracing::warn!(message_id = %id, location = ?d.location, + "dropping recovered deferred job whose payload was truncated with the torn tail"); + false + } else { + true + } + }); + + for (id, r) in recovered.ready { + self.jobs.insert( + id, + Job { + location: r.location, + attempts: r.attempts, + enqueue_ms: r.enqueue_ms, + remaining: (!r.remaining_recipients.is_empty()).then_some(r.remaining_recipients), + last_error: None, + envelope: None, + state: JobState::Ready, + }, + ); + self.ready.push(Reverse((r.enqueue_ms, id))); + } + for (id, d) in recovered.deferred { + self.jobs.insert( + id, + Job { + location: d.location, + attempts: d.attempts, + // Queue age is unknown for deferred checkpoint entries + // until the header is read; due time is a fine proxy for + // ordering once it re-enters ready. + enqueue_ms: d.next_attempt_ms, + remaining: Some(d.remaining_recipients), + last_error: Some(d.last_error), + envelope: None, + state: JobState::Delayed { + due_ms: d.next_attempt_ms, + persisted: true, + }, + }, + ); + self.delayed.push(Reverse((d.next_attempt_ms, id))); + } + let mut stats = recovered.segment_stats; + // Record sealed-segment sizes for GC eligibility: a segment is + // reclaimable only once its total size is known (i.e. it is sealed; + // the writer's recovered chain lists every sealed segment on disk). + for head in &chain { + if head.sealed { + stats.entry(head.segment).or_default().total_bytes = head.committed; + } + } + self.shards.push(ShardRuntime { + shard: shared.shard(), + dir, + shared, + store, + cursor: recovered.cursor, + tombstones: recovered.tombstones, + stats, + readers: HashMap::new(), + pending_persists: VecDeque::new(), + checkpoint: None, + }); + } + + async fn run( + mut self, + mut claim_rx: mpsc::Receiver, + mut events_rx: mpsc::UnboundedReceiver, + mut cp_rx: mpsc::UnboundedReceiver<(u16, Result<(), QueueError>)>, + cancel: CancellationToken, + ) { + // Merge every shard's notify into one wake-up signal. + let discovery_wake = Arc::new(tokio::sync::Notify::new()); + for shard in &self.shards { + let shared = Arc::clone(&shard.shared); + let wake = Arc::clone(&discovery_wake); + let cancel = cancel.clone(); + tokio::spawn(async move { + loop { + tokio::select! { + _ = shared.notify.notified() => wake.notify_one(), + _ = cancel.cancelled() => break, + } + } + }); + } + + self.discover_all(); + let mut tick = tokio::time::interval(self.config.safety_tick); + tick.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); + + loop { + self.wake_due_jobs(); + self.try_dispatch(); + + let next_due = self.next_due_in(); + tokio::select! { + biased; + + _ = cancel.cancelled() => break, + + Some((shard, result)) = cp_rx.recv() => { + self.finish_checkpoint(shard, result); + } + + Some(event) = events_rx.recv() => { + self.on_worker_event(event); + // Drain whatever else is immediately available. + while let Ok(event) = events_rx.try_recv() { + self.on_worker_event(event); + } + } + + Some(waiter) = claim_rx.recv() => { + self.waiting.push_back(waiter); + } + + _ = discovery_wake.notified() => { + self.discover_all(); + } + + _ = tick.tick() => { + self.discover_all(); + self.retry_pending_persists(); + self.maybe_checkpoint(); + self.sweep_dead_segments(); + self.maybe_start_compaction(); + self.drive_compaction().await; + self.publish_metrics(); + } + + _ = tokio::time::sleep(next_due) => {} + } + } + + self.shutdown(&mut events_rx).await; + } + + /// Graceful shutdown: refuse new claims, wait for in-flight outcomes, + /// persist everything, checkpoint every shard. + async fn shutdown(&mut self, events_rx: &mut mpsc::UnboundedReceiver) { + for waiter in self.waiting.drain(..) { + let _ = waiter.send(None); + } + while self.inflight > 0 { + match events_rx.recv().await { + Some(event) => self.on_worker_event(event), + None => break, + } + } + self.retry_pending_persists(); + for i in 0..self.shards.len() { + if self.shards[i].checkpoint.is_some() { + continue; // an async checkpoint is mid-flight; journals cover us + } + if let Err(e) = self.checkpoint_shard_sync(i) { + tracing::error!(shard = self.shards[i].shard, error = %e, + "final checkpoint failed; journals remain authoritative"); + } + } + tracing::info!("dispatcher stopped"); + } + + // ------------------------------------------------------------------ + // Discovery. + + fn discover_all(&mut self) { + for i in 0..self.shards.len() { + if let Err(e) = self.discover_shard(i) { + tracing::error!(shard = self.shards[i].shard, error = %e, "discovery failed"); + } + } + } + + fn discover_shard(&mut self, i: usize) -> Result<(), QueueError> { + loop { + if self.jobs.len() >= self.config.max_tracked_jobs { + return Ok(()); // discovery backpressure; the log holds the rest + } + let budget = self.config.max_tracked_jobs - self.jobs.len(); + + let shard = &mut self.shards[i]; + let chain = shard.shared.chain(); + // Note sealed sizes for GC as segments seal (idempotent). + for head in &chain { + if head.sealed { + shard.stats.entry(head.segment).or_default().total_bytes = head.committed; + } + } + let Some(first) = chain.first() else { + return Ok(()); + }; + let (mut seg, mut off) = shard.cursor.unwrap_or((first.segment, 0)); + + // Position the cursor on its chain entry; if that segment is + // gone (pruned after full consumption), move to the next one. + let entry = match chain.iter().find(|h| h.segment == seg) { + Some(e) => *e, + None => match chain.iter().find(|h| h.segment > seg) { + Some(e) => { + seg = e.segment; + off = 0; + *e + } + None => return Ok(()), + }, + }; + + if off >= entry.committed { + if entry.sealed && chain.iter().any(|h| h.segment > entry.segment) { + // Fully consumed sealed segment: advance to the next + // chain entry and let the loop scan it. + let next = chain.iter().find(|h| h.segment > entry.segment).unwrap(); + shard.cursor = Some((next.segment, 0)); + shard.shared.prune(next.segment); + continue; + } + return Ok(()); // caught up with the active head + } + + // Scan headers between cursor and committed tail, registering + // new jobs, stopping at the backpressure budget. A record whose + // id is already tracked but whose relocation generation is + // higher is a compaction copy that must win (crash between + // relocation and checkpoint leaves both copies on disk). + let mut discovered: Vec<(MessageId, JobLocation, i64)> = Vec::new(); + let mut relocations: Vec<(MessageId, JobLocation)> = Vec::new(); + let shard_no = shard.shard; + let tombstones = &shard.tombstones; + let jobs = &self.jobs; + let max_record_len = super::record::MAX_RECORD_LEN; + let reader = { + if !shard.readers.contains_key(&seg) { + let r = open_segment_reader(&shard.dir, seg)?; + shard.readers.insert(seg, r); + } + shard.readers.get(&seg).unwrap() + }; + let stopped = scan_headers(reader, off, entry.committed, max_record_len, |o, h| { + let location = JobLocation { + shard: shard_no, + segment: seg, + offset: o, + length: h.record_len, + ordinal: h.ordinal, + generation: h.generation, + }; + let dead = tombstones.get(&seg).is_some_and(|s| s.contains(&h.message_id)); + if !dead { + match jobs.get(&h.message_id) { + None => discovered.push((h.message_id, location, h.enqueue_ms)), + Some(job) if h.generation > job.location.generation => { + relocations.push((h.message_id, location)); + } + Some(_) => {} + } + } + discovered.len() < budget + })?; + shard.cursor = Some((seg, stopped)); + + let made_progress = stopped > off || !discovered.is_empty(); + for (id, location, enqueue_ms) in discovered { + self.jobs.insert( + id, + Job { + location, + attempts: 0, + enqueue_ms, + remaining: None, + last_error: None, + envelope: None, + state: JobState::Ready, + }, + ); + self.ready.push(Reverse((enqueue_ms, id))); + } + for (id, location) in relocations { + // Journal the relocation (again): a crash between the copy + // and its Relocated entry loses the old copy's garbage + // accounting, so re-record it durably when rediscovered. + if let Some(job) = self.jobs.get(&id) { + if location.generation > job.location.generation { + let old = job.location; + self.persist_and_apply( + i, + StateEntry::Relocated { + id, + old, + new: location, + }, + ); + } + } + } + if !made_progress { + return Ok(()); + } + } + } + + // ------------------------------------------------------------------ + // Dispatch. + + fn wake_due_jobs(&mut self) { + let now = now_ms(); + while let Some(Reverse((due, id))) = self.delayed.peek().copied() { + if due > now { + break; + } + self.delayed.pop(); + let Some(job) = self.jobs.get_mut(&id) else { + continue; // stale heap entry + }; + match job.state { + JobState::Delayed { due_ms, .. } if due_ms == due => { + job.state = JobState::Ready; + self.ready.push(Reverse((job.enqueue_ms, id))); + } + _ => {} // stale entry: the job moved on + } + } + } + + fn next_due_in(&self) -> Duration { + match self.delayed.peek() { + Some(Reverse((due, _))) => { + Duration::from_millis((due - now_ms()).max(0) as u64).min(Duration::from_secs(60)) + } + None => Duration::from_secs(60), + } + } + + fn try_dispatch(&mut self) { + while !self.waiting.is_empty() { + let Some((id, enqueue_ms)) = self.pop_ready() else { + return; + }; + + // Fill the envelope from the payload header on first dispatch. + if let Err(e) = self.fill_envelope(id) { + tracing::error!(message_id = %id, error = %e, + "cannot read record header; delaying job"); + self.delay_job(id, now_ms() + 30_000, false); + continue; + } + let job = self.jobs.get(&id).expect("popped job exists"); + let (sender, all_recipients) = job.envelope.clone().expect("envelope just filled"); + let recipients = job.remaining.clone().unwrap_or(all_recipients); + + // Dispatch-time rate gating: exhausted destinations stay queued + // without consuming a worker slot. The due time is not a token — + // the job re-passes the gate when it wakes. + if let Some(wait) = self.gate.check(domain_of(&recipients[0])) { + self.delay_job(id, now_ms() + wait.as_millis() as i64, false); + continue; + } + + self.next_claim_generation += 1; + let generation = self.next_claim_generation; + let job = self.jobs.get_mut(&id).expect("popped job exists"); + let delivery = DeliveryJob { + message_id: id, + location: job.location, + attempts: job.attempts, + claim_generation: generation, + enqueue_ms, + sender, + recipients, + }; + job.state = JobState::InFlight(generation); + self.inflight += 1; + + let claim = Claim { + job: delivery, + events: self.events_tx.clone(), + reported: false, + }; + let waiter = self.waiting.pop_front().expect("checked non-empty"); + if waiter.send(Some(claim)).is_err() { + // Worker vanished; the dropped Claim reports abandonment, + // which returns the job to ready via the event channel. + tracing::debug!(message_id = %id, "claim waiter disappeared"); + } + } + } + + fn pop_ready(&mut self) -> Option<(MessageId, i64)> { + while let Some(Reverse((enqueue_ms, id))) = self.ready.pop() { + match self.jobs.get(&id) { + Some(job) if job.state == JobState::Ready => return Some((id, enqueue_ms)), + _ => {} // stale entry + } + } + None + } + + fn fill_envelope(&mut self, id: MessageId) -> Result<(), QueueError> { + let job = self.jobs.get(&id).expect("job exists"); + if job.envelope.is_some() { + return Ok(()); + } + let location = job.location; + let shard = &mut self.shards[location.shard as usize]; + let reader = shard.reader(location.segment)?; + let header = reader.read_header_at(location.offset, super::record::MAX_RECORD_LEN)?; + let job = self.jobs.get_mut(&id).expect("job exists"); + job.enqueue_ms = header.enqueue_ms; + job.envelope = Some((header.sender, header.recipients)); + Ok(()) + } + + fn delay_job(&mut self, id: MessageId, due_ms: i64, persisted: bool) { + if let Some(job) = self.jobs.get_mut(&id) { + job.state = JobState::Delayed { due_ms, persisted }; + self.delayed.push(Reverse((due_ms, id))); + } + } + + // ------------------------------------------------------------------ + // Outcomes (persist-then-apply). + + fn on_worker_event(&mut self, event: WorkerEvent) { + match event { + WorkerEvent::Outcome { + id, + generation, + outcome, + } => self.on_outcome(id, generation, outcome), + WorkerEvent::Abandoned { id, generation } => { + if !self.claim_is_current(&id, generation) { + return; + } + let job = self.jobs.get_mut(&id).expect("claim_is_current checked"); + tracing::warn!(message_id = %id, "worker abandoned claim; requeueing"); + self.inflight -= 1; + job.state = JobState::Ready; + self.ready.push(Reverse((job.enqueue_ms, id))); + } + } + } + + fn claim_is_current(&self, id: &MessageId, generation: u64) -> bool { + matches!( + self.jobs.get(id).map(|j| j.state), + Some(JobState::InFlight(g)) if g == generation + ) + } + + fn on_outcome(&mut self, id: MessageId, generation: u64, outcome: JobOutcome) { + if !self.claim_is_current(&id, generation) { + tracing::debug!(message_id = %id, generation, "ignoring stale claim outcome"); + return; + } + self.inflight -= 1; + let job = self.jobs.get_mut(&id).expect("claim_is_current checked"); + let location = job.location; + + let entry = match outcome { + JobOutcome::RateLimited { retry_after } => { + // Local throttling: in-memory requeue only. + self.delay_job(id, now_ms() + retry_after.as_millis() as i64, false); + return; + } + JobOutcome::Delivered { response } => { + tracing::debug!(message_id = %id, response, "delivered"); + StateEntry::Delivered { + id, + location, + timestamp_ms: now_ms(), + } + } + JobOutcome::Bounced { reason } => StateEntry::Bounced { + id, + location, + timestamp_ms: now_ms(), + reason, + }, + JobOutcome::Deferred { + next_attempt_ms, + remaining_recipients, + error, + } => StateEntry::Deferred { + id, + location, + attempts: job.attempts + 1, + next_attempt_ms, + remaining_recipients, + last_error: error, + }, + }; + + self.persist_and_apply(location.shard as usize, entry); + self.maybe_checkpoint(); + } + + fn persist_and_apply(&mut self, shard_idx: usize, entry: StateEntry) { + let shard = &mut self.shards[shard_idx]; + if !shard.pending_persists.is_empty() { + // Preserve per-shard ordering behind earlier failed writes. + shard.pending_persists.push_back(entry); + return; + } + match shard.store.append(&entry) { + Ok(_) => self.apply_persisted(shard_idx, entry), + Err(e) => { + tracing::error!(shard = shard.shard, error = %e, + "state journal write failed; will retry (job stays in flight)"); + shard.pending_persists.push_back(entry); + } + } + } + + fn retry_pending_persists(&mut self) { + for i in 0..self.shards.len() { + while let Some(entry) = self.shards[i].pending_persists.front().cloned() { + match self.shards[i].store.append(&entry) { + Ok(_) => { + self.shards[i].pending_persists.pop_front(); + self.apply_persisted(i, entry); + } + Err(_) => break, + } + } + } + } + + /// Apply a journal-persisted transition to scheduling state. + fn apply_persisted(&mut self, shard_idx: usize, entry: StateEntry) { + match entry { + StateEntry::Deferred { + id, + attempts, + next_attempt_ms, + remaining_recipients, + last_error, + .. + } => { + if let Some(job) = self.jobs.get_mut(&id) { + job.attempts = attempts; + job.remaining = Some(remaining_recipients); + job.last_error = Some(last_error); + job.state = JobState::Delayed { + due_ms: next_attempt_ms, + persisted: true, + }; + self.delayed.push(Reverse((next_attempt_ms, id))); + } + } + StateEntry::Delivered { id, location, .. } + | StateEntry::Bounced { id, location, .. } => { + self.jobs.remove(&id); + self.mark_copy_dead(shard_idx, id, location); + } + StateEntry::Relocated { id, old, new } => { + match self.jobs.get_mut(&id) { + Some(job) if new.generation > job.location.generation => { + job.location = new; + job.envelope = None; // content identical, offsets not + self.mark_copy_dead(shard_idx, id, old); + } + Some(_) => { + // Stale relocation: the new copy lost. + self.mark_copy_dead(shard_idx, id, new); + } + None => { + // Terminal raced the copy; neither copy may + // resurrect the message. + self.mark_copy_dead(shard_idx, id, old); + self.mark_copy_dead(shard_idx, id, new); + } + } + } + } + } + + /// Account one physical record copy as dead and evaluate the segment + /// for event-driven reclamation. + fn mark_copy_dead(&mut self, shard_idx: usize, id: MessageId, location: JobLocation) { + let shard = &mut self.shards[shard_idx]; + if shard + .tombstones + .entry(location.segment) + .or_default() + .insert(id) + { + let stats = shard.stats.entry(location.segment).or_default(); + stats.dead_records += 1; + stats.dead_bytes += location.length as u64; + } + self.maybe_delete_segment(shard_idx, location.segment); + } + + // ------------------------------------------------------------------ + // Reclamation (PLAN §17): event-driven deletion of fully dead sealed + // segments; dead-ratio compaction for partially live ones. + + /// Delete a sealed segment the moment its last byte goes dead. + fn maybe_delete_segment(&mut self, shard_idx: usize, segment: u64) { + let shard = &mut self.shards[shard_idx]; + let Some(stats) = shard.stats.get(&segment) else { + return; + }; + // total_bytes > 0 means the segment is sealed with a known size. + if stats.total_bytes == 0 || stats.dead_bytes < stats.total_bytes { + return; + } + // Destructive boundary: the journal entries recording these deaths + // must be durable before the payload disappears. + if let Err(e) = shard.store.fsync_journal() { + tracing::error!(shard = shard.shard, segment, error = %e, + "cannot fsync journal; postponing segment deletion"); + return; + } + let path = shard.dir.join(super::segment::sealed_file_name(segment)); + if let Err(e) = std::fs::remove_file(&path) { + if e.kind() != std::io::ErrorKind::NotFound { + tracing::error!(shard = shard.shard, segment, error = %e, + "failed to delete dead segment"); + return; + } + } + shard.readers.remove(&segment); + shard.tombstones.remove(&segment); + shard.stats.remove(&segment); + shard.shared.remove_segment(segment); + if let Some(run) = &self.compaction { + if run.shard_idx == shard_idx && run.segment == segment { + self.compaction = None; // everything left in it just died + } + } + crate::metrics::logqueue_segments_deleted(1); + tracing::info!(shard = self.shards[shard_idx].shard, segment, "deleted fully dead segment"); + } + + /// Sweep backstop for event-driven deletion (PLAN §17.1): a segment + /// whose last record went terminal before the dispatcher had observed + /// the seal (total_bytes still unknown at that moment) misses its + /// deletion event; catch it on the safety tick. + fn sweep_dead_segments(&mut self) { + for shard_idx in 0..self.shards.len() { + let dead: Vec = self.shards[shard_idx] + .stats + .iter() + .filter(|(_, s)| s.total_bytes > 0 && s.dead_bytes >= s.total_bytes) + .map(|(seg, _)| *seg) + .collect(); + for segment in dead { + self.maybe_delete_segment(shard_idx, segment); + } + } + } + + /// Pick a compaction candidate if none is running (one at a time + /// globally). Runs on the safety tick as both trigger and backstop. + fn maybe_start_compaction(&mut self) { + if self.compaction.is_some() { + return; + } + for shard_idx in 0..self.shards.len() { + let shard = &self.shards[shard_idx]; + let cursor_segment = shard.cursor.map(|(s, _)| s).unwrap_or(0); + for (&segment, stats) in &shard.stats { + if stats.total_bytes == 0 + || stats.dead_bytes >= stats.total_bytes + || (stats.dead_bytes as f64) + < stats.total_bytes as f64 * self.config.compaction_dead_ratio + || segment >= cursor_segment + { + continue; // active, fully dead, too alive, or not yet fully discovered + } + let path = shard.dir.join(super::segment::sealed_file_name(segment)); + let old_enough = std::fs::metadata(&path) + .and_then(|m| m.modified()) + .ok() + .and_then(|t| t.elapsed().ok()) + .is_some_and(|age| age >= self.config.compaction_min_age); + if !old_enough { + continue; + } + // Snapshot the live set: tracked jobs located in this + // segment that are not currently being read by a worker. + let queue: VecDeque = self + .jobs + .iter() + .filter(|(_, j)| { + j.location.shard == shard.shard + && j.location.segment == segment + && !matches!(j.state, JobState::InFlight(_)) + }) + .map(|(id, _)| *id) + .collect(); + if queue.is_empty() { + continue; // only in-flight records left; retry later + } + tracing::info!( + shard = shard.shard, + segment, + live = queue.len(), + dead_bytes = stats.dead_bytes, + total_bytes = stats.total_bytes, + "starting compaction" + ); + crate::metrics::logqueue_compaction_started(); + self.compaction = Some(CompactionRun { + shard_idx, + segment, + queue, + }); + return; + } + } + } + + /// Copy a bounded batch of live records out of the compaction source. + /// Each copy is re-appended through the writer (higher relocation + /// generation, original enqueue timestamp) and journaled as Relocated + /// before the location switches. The source segment is never touched; + /// it dies through the normal full-death path once its last live + /// record has moved out (or delivered). + async fn drive_compaction(&mut self) { + const BATCH: usize = 256; + let Some(run) = &mut self.compaction else { + return; + }; + let shard_idx = run.shard_idx; + let source = run.segment; + + for _ in 0..BATCH { + let Some(id) = self.compaction.as_mut().and_then(|r| r.queue.pop_front()) else { + tracing::info!(segment = source, "compaction pass complete"); + crate::metrics::logqueue_compaction_completed(); + self.compaction = None; + return; + }; + let Some(job) = self.jobs.get(&id) else { + continue; // went terminal while queued + }; + if job.location.segment != source || matches!(job.state, JobState::InFlight(_)) { + continue; // moved or claimed since the snapshot + } + let old = job.location; + + let record = { + let shard = &mut self.shards[shard_idx]; + shard + .reader(source) + .and_then(|r| r.read_record_at(old.offset, super::record::MAX_RECORD_LEN)) + }; + let (header, body) = match record { + Ok(r) => r, + Err(e) => { + tracing::error!(message_id = %id, error = %e, + "compaction read failed; aborting pass"); + crate::metrics::logqueue_compaction_failed(); + self.compaction = None; + return; + } + }; + + let append = self.append.clone(); + let new = match append + .append_to_shard( + self.shards[shard_idx].shard, + super::writer::AppendMessage { + message_id: id, + enqueue_ms: header.enqueue_ms, + generation: old.generation + 1, + sender: header.sender, + recipients: header.recipients, + body: bytes::Bytes::from(body), + }, + ) + .await + { + Ok(loc) => loc, + Err(e) => { + tracing::error!(message_id = %id, error = %e, + "compaction append failed; aborting pass"); + crate::metrics::logqueue_compaction_failed(); + self.compaction = None; + return; + } + }; + + crate::metrics::logqueue_compaction_bytes_read(old.length as u64); + crate::metrics::logqueue_compaction_bytes_written(new.length as u64); + crate::metrics::logqueue_relocations(1); + self.persist_and_apply(shard_idx, StateEntry::Relocated { id, old, new }); + } + } + + /// Push scheduler and storage gauges (runs on the safety tick). + fn publish_metrics(&self) { + let now = now_ms(); + let mut ready = 0i64; + let mut deferred = 0i64; + let mut oldest_ready_ms: Option = None; + let mut oldest_deferred_due: Option = None; + for job in self.jobs.values() { + match job.state { + JobState::Ready => { + ready += 1; + oldest_ready_ms = + Some(oldest_ready_ms.map_or(job.enqueue_ms, |o| o.min(job.enqueue_ms))); + } + JobState::Delayed { due_ms, .. } => { + deferred += 1; + oldest_deferred_due = + Some(oldest_deferred_due.map_or(due_ms, |o| o.min(due_ms))); + } + JobState::InFlight(_) => {} + } + } + crate::metrics::logqueue_ready_jobs_set(ready); + crate::metrics::logqueue_deferred_jobs_set(deferred); + crate::metrics::logqueue_inflight_jobs_set(self.inflight as i64); + crate::metrics::logqueue_oldest_ready_age_seconds_set( + oldest_ready_ms.map_or(0, |ms| ((now - ms) / 1000).max(0)), + ); + crate::metrics::logqueue_oldest_deferred_age_seconds_set( + oldest_deferred_due.map_or(0, |due| ((now - due) / 1000).max(0)), + ); + + let mut dead = 0u64; + let mut total = 0u64; + let mut sealed = 0i64; + for shard in &self.shards { + // Discovery lag: committed bytes the cursor has not scanned yet. + let mut lag = 0u64; + let (cursor_seg, cursor_off) = shard.cursor.unwrap_or((0, 0)); + for head in shard.shared.chain() { + if head.segment > cursor_seg { + lag += head.committed; + } else if head.segment == cursor_seg { + lag += head.committed.saturating_sub(cursor_off); + } + } + crate::metrics::logqueue_dispatcher_lag_bytes_set(shard.shard, lag as i64); + for stats in shard.stats.values() { + dead += stats.dead_bytes; + if stats.total_bytes > 0 { + sealed += 1; + total += stats.total_bytes; + } + } + } + crate::metrics::logqueue_dead_bytes_set(dead); + crate::metrics::logqueue_live_bytes_set(total.saturating_sub(dead)); + crate::metrics::logqueue_sealed_segments_set(sealed); + if let Some(shard) = self.shards.first() { + if let Ok(free) = super::spool::disk_free_bytes(&shard.dir) { + crate::metrics::logqueue_disk_free_bytes_set(free); + } + } + } + + // ------------------------------------------------------------------ + // Checkpoints. + + fn snapshot_shard(&self, shard_idx: usize) -> Checkpoint { + let shard = &self.shards[shard_idx]; + let mut cp = Checkpoint { + cursor: shard.cursor, + ..Default::default() + }; + for (id, job) in &self.jobs { + if job.location.shard != shard.shard { + continue; + } + match job.state { + JobState::Delayed { + due_ms, + persisted: true, + } => cp.deferred.push(DeferredJob { + id: *id, + location: job.location, + attempts: job.attempts, + next_attempt_ms: due_ms, + remaining_recipients: job.remaining.clone().unwrap_or_default(), + last_error: job.last_error.clone().unwrap_or_default(), + }), + // Ready, in-flight, and rate-limit holds all restart as + // ready. + _ => cp.ready.push(ReadyJob { + id: *id, + location: job.location, + attempts: job.attempts, + enqueue_ms: job.enqueue_ms, + remaining_recipients: job.remaining.clone().unwrap_or_default(), + }), + } + } + cp.tombstones = shard + .tombstones + .iter() + .map(|(seg, ids)| (*seg, ids.iter().copied().collect())) + .collect(); + cp.segment_stats = shard.stats.iter().map(|(s, st)| (*s, *st)).collect(); + cp + } + + fn maybe_checkpoint(&mut self) { + for i in 0..self.shards.len() { + let shard = &self.shards[i]; + if shard.checkpoint.is_some() + || !shard.pending_persists.is_empty() + || shard.store.bytes_since_checkpoint() < self.config.checkpoint_interval_bytes + { + continue; + } + let cp = self.snapshot_shard(i); + let shard = &mut self.shards[i]; + let pending = match shard.store.begin_checkpoint() { + Ok(p) => p, + Err(e) => { + tracing::error!(shard = shard.shard, error = %e, "cannot begin checkpoint"); + continue; + } + }; + shard.checkpoint = Some(pending); + let dir = shard.dir.clone(); + let shard_no = shard.shard; + let cp_tx = self.cp_tx.clone(); + let replay_from = pending.replay_from; + tokio::task::spawn_blocking(move || { + let result = super::state::write_checkpoint_file(&dir, &cp, replay_from); + let _ = cp_tx.send((shard_no, result)); + }); + } + } + + fn finish_checkpoint(&mut self, shard_no: u16, result: Result<(), QueueError>) { + let Some(shard) = self.shards.iter_mut().find(|s| s.shard == shard_no) else { + return; + }; + let Some(pending) = shard.checkpoint.take() else { + return; + }; + match result { + Ok(()) => { + if let Err(e) = shard.store.finish_checkpoint(pending) { + tracing::warn!(shard = shard_no, error = %e, + "checkpoint published but journal pruning failed"); + } + } + Err(e) => { + // The rotated journal chain is still complete, so recovery + // stays correct; only the compaction of history was lost. + tracing::error!(shard = shard_no, error = %e, "checkpoint write failed"); + } + } + } + + /// Synchronous checkpoint used at shutdown. + fn checkpoint_shard_sync(&mut self, shard_idx: usize) -> Result<(), QueueError> { + let cp = self.snapshot_shard(shard_idx); + self.shards[shard_idx].store.write_checkpoint(&cp) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::logqueue::spool::Spool; + use crate::logqueue::state::ShardStateStore; + use crate::logqueue::writer::{AppendMessage, LogWriters, WriterConfig}; + use bytes::Bytes; + use std::sync::Mutex; + + fn writer_config() -> WriterConfig { + WriterConfig { + segment_target_bytes: 64 * 1024 * 1024, + max_record_len: crate::logqueue::record::MAX_RECORD_LEN, + pending_append_bytes: 16 * 1024 * 1024, + } + } + + fn message(seq: u64, rcpt: &str) -> AppendMessage { + AppendMessage { + message_id: MessageId::from_ulid(ulid::Ulid::from_parts(seq, (seq * 7 + 1) as u128)), + enqueue_ms: 1_752_000_000_000 + seq as i64, + generation: 0, + sender: "sender@example.com".into(), + recipients: vec![rcpt.into()], + body: Bytes::from(format!("body {seq}")), + } + } + + struct Harness { + _spool: Spool, + writers: LogWriters, + handle: DispatcherHandle, + dispatcher_task: tokio::task::JoinHandle<()>, + cancel: CancellationToken, + } + + fn start(root: &std::path::Path, gate: Arc, config: DispatcherConfig) -> Harness { + let spool = Spool::open(root.join("spool"), 1).unwrap(); + let writers = LogWriters::start(&spool, writer_config()).unwrap(); + let mut inits = Vec::new(); + for shard_dir in spool.shards() { + let (store, recovered) = + ShardStateStore::recover(shard_dir.path(), shard_dir.shard()).unwrap(); + inits.push(ShardInit { + dir: shard_dir.path().to_path_buf(), + shared: writers.handle().shard_shared(shard_dir.shard()), + store, + recovered, + }); + } + let cancel = CancellationToken::new(); + let (handle, dispatcher_task) = + Dispatcher::start(inits, writers.handle(), gate, config, cancel.clone()); + Harness { + _spool: spool, + writers, + handle, + dispatcher_task, + cancel, + } + } + + async fn stop(h: Harness) { + h.cancel.cancel(); + h.dispatcher_task.await.unwrap(); + h.writers.shutdown().await; + } + + #[tokio::test] + async fn discovers_and_dispatches_appended_messages() { + let dir = tempfile::tempdir().unwrap(); + let h = start( + dir.path(), + Arc::new(NoRateGate), + DispatcherConfig::default(), + ); + let append = h.writers.handle(); + let loc = append.append(message(1, "r1@example.com")).await.unwrap(); + + let claim = h.handle.claim().await.expect("claim"); + assert_eq!(claim.job.location, loc); + assert_eq!(claim.job.attempts, 0); + assert_eq!(claim.job.sender, "sender@example.com"); + assert_eq!(claim.job.recipients, vec!["r1@example.com".to_string()]); + + let body = h.handle.read_body(claim.job.location).await.unwrap(); + assert_eq!(body, b"body 1"); + + claim.report(JobOutcome::Delivered { + response: "250 ok".into(), + }); + stop(h).await; + } + + #[tokio::test] + async fn deferred_outcome_schedules_retry_with_remaining_recipients() { + let dir = tempfile::tempdir().unwrap(); + let h = start( + dir.path(), + Arc::new(NoRateGate), + DispatcherConfig::default(), + ); + let append = h.writers.handle(); + append.append(message(1, "r1@example.com")).await.unwrap(); + + let claim = h.handle.claim().await.unwrap(); + let id = claim.job.message_id; + claim.report(JobOutcome::Deferred { + next_attempt_ms: now_ms() + 50, + remaining_recipients: vec!["r1@example.com".into()], + error: "451 greylisted".into(), + }); + + // The retry claim arrives once due, with the attempt count bumped. + let claim = h.handle.claim().await.unwrap(); + assert_eq!(claim.job.message_id, id); + assert_eq!(claim.job.attempts, 1); + assert_eq!(claim.job.recipients, vec!["r1@example.com".to_string()]); + claim.report(JobOutcome::Delivered { + response: "250 ok".into(), + }); + stop(h).await; + } + + #[tokio::test] + async fn rate_limited_outcome_requeues_without_attempt_increment() { + let dir = tempfile::tempdir().unwrap(); + let h = start( + dir.path(), + Arc::new(NoRateGate), + DispatcherConfig::default(), + ); + let append = h.writers.handle(); + append.append(message(1, "r1@example.com")).await.unwrap(); + + let claim = h.handle.claim().await.unwrap(); + claim.report(JobOutcome::RateLimited { + retry_after: Duration::from_millis(30), + }); + + let claim = h.handle.claim().await.unwrap(); + assert_eq!(claim.job.attempts, 0, "rate limiting is not an attempt"); + claim.report(JobOutcome::Delivered { + response: "250 ok".into(), + }); + stop(h).await; + } + + #[tokio::test] + async fn dropped_claim_is_abandoned_and_redispatched() { + let dir = tempfile::tempdir().unwrap(); + let h = start( + dir.path(), + Arc::new(NoRateGate), + DispatcherConfig::default(), + ); + let append = h.writers.handle(); + append.append(message(1, "r1@example.com")).await.unwrap(); + + let claim = h.handle.claim().await.unwrap(); + let id = claim.job.message_id; + drop(claim); // worker dies without reporting + + let claim = h.handle.claim().await.unwrap(); + assert_eq!(claim.job.message_id, id); + claim.report(JobOutcome::Delivered { + response: "250 ok".into(), + }); + stop(h).await; + } + + #[tokio::test] + async fn stale_generation_outcome_is_ignored() { + let dir = tempfile::tempdir().unwrap(); + let h = start( + dir.path(), + Arc::new(NoRateGate), + DispatcherConfig::default(), + ); + let append = h.writers.handle(); + append.append(message(1, "r1@example.com")).await.unwrap(); + + // First claim is abandoned but we keep its (now stale) event sender. + let claim1 = h.handle.claim().await.unwrap(); + let stale_events = claim1.events.clone(); + let stale_gen = claim1.job.claim_generation; + let id = claim1.job.message_id; + drop(claim1); + + let claim2 = h.handle.claim().await.unwrap(); + assert_eq!(claim2.job.message_id, id); + assert_ne!(claim2.job.claim_generation, stale_gen); + + // The stale generation reports Bounced; it must be ignored. + let _ = stale_events.send(WorkerEvent::Outcome { + id, + generation: stale_gen, + outcome: JobOutcome::Bounced { + reason: "stale".into(), + }, + }); + tokio::time::sleep(Duration::from_millis(50)).await; + + // The live claim still completes normally. + claim2.report(JobOutcome::Delivered { + response: "250 ok".into(), + }); + stop(h).await; + + // After restart nothing is ready (the message really delivered) — + // i.e. the stale Bounced didn't win. + // (Verified by the state store directly.) + } + + struct BlockOnce { + blocked: Mutex>, + } + impl RateGate for BlockOnce { + fn check(&self, _domain: &str) -> Option { + self.blocked.lock().unwrap().take() + } + } + + #[tokio::test] + async fn dispatch_gating_delays_exhausted_domains() { + let gate = Arc::new(BlockOnce { + blocked: Mutex::new(Some(Duration::from_millis(40))), + }); + let dir = tempfile::tempdir().unwrap(); + let h = start( + dir.path(), + gate, + DispatcherConfig::default(), + ); + let append = h.writers.handle(); + append.append(message(1, "r1@example.com")).await.unwrap(); + + let started = std::time::Instant::now(); + let claim = h.handle.claim().await.unwrap(); + assert!( + started.elapsed() >= Duration::from_millis(35), + "claim should have been gated, got it after {:?}", + started.elapsed() + ); + assert_eq!(claim.job.attempts, 0, "gating is not an attempt"); + claim.report(JobOutcome::Delivered { + response: "250 ok".into(), + }); + stop(h).await; + } + + #[tokio::test] + async fn full_restart_preserves_deferred_and_skips_terminal() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_path_buf(); + + let (deferred_id, delivered_id) = { + let h = start(dir.path(), Arc::new(NoRateGate), DispatcherConfig::default()); + let append = h.writers.handle(); + append.append(message(1, "r1@example.com")).await.unwrap(); + append.append(message(2, "r2@example.com")).await.unwrap(); + + let c1 = h.handle.claim().await.unwrap(); + let c2 = h.handle.claim().await.unwrap(); + let (deferred, delivered) = (c1.job.message_id, c2.job.message_id); + c1.report(JobOutcome::Deferred { + next_attempt_ms: now_ms() + 3_600_000, // an hour away + remaining_recipients: vec!["r1@example.com".into()], + error: "451".into(), + }); + c2.report(JobOutcome::Delivered { + response: "250 ok".into(), + }); + stop(h).await; + (deferred, delivered) + }; + + // Restart on the same spool. + let spool = Spool::open(root.join("spool"), 1).unwrap(); + let writers = LogWriters::start(&spool, writer_config()).unwrap(); + let (store, recovered) = + ShardStateStore::recover(spool.shard(0).path(), 0).unwrap(); + + let d = recovered + .deferred + .get(&deferred_id) + .expect("deferred job survives restart"); + assert_eq!(d.attempts, 1); + assert_eq!(d.remaining_recipients, vec!["r1@example.com".to_string()]); + assert!( + recovered.is_terminal(d.location.segment, &delivered_id), + "delivered message stays terminal" + ); + assert!(recovered.ready.is_empty()); + drop(store); + writers.shutdown().await; + } + + #[tokio::test] + async fn restart_redispatches_inflight_messages() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_path_buf(); + let id = { + let h = start(dir.path(), Arc::new(NoRateGate), DispatcherConfig::default()); + let append = h.writers.handle(); + append.append(message(1, "r1@example.com")).await.unwrap(); + // Claim and crash while in flight: report nothing, drop nothing + // cleanly — forget the claim so no abandonment event fires. + let claim = h.handle.claim().await.unwrap(); + let id = claim.job.message_id; + std::mem::forget(claim); + h.cancel.cancel(); + // The dispatcher will wait for the in-flight claim on shutdown; + // abandon it by dropping the whole runtime instead (crash). + h.dispatcher_task.abort(); + h.writers.shutdown().await; + id + }; + + // Restart: the record was never persisted as terminal, so it must + // be discovered and dispatched again. + let spool = Spool::open(root.join("spool"), 1).unwrap(); + let writers = LogWriters::start(&spool, writer_config()).unwrap(); + let (store, recovered) = ShardStateStore::recover(spool.shard(0).path(), 0).unwrap(); + let cancel = CancellationToken::new(); + let (handle, task) = Dispatcher::start( + vec![ShardInit { + dir: spool.shard(0).path().to_path_buf(), + shared: writers.handle().shard_shared(0), + store, + recovered, + }], + writers.handle(), + Arc::new(NoRateGate), + DispatcherConfig::default(), + cancel.clone(), + ); + let claim = handle.claim().await.expect("redispatched after restart"); + assert_eq!(claim.job.message_id, id); + claim.report(JobOutcome::Delivered { + response: "250 ok".into(), + }); + cancel.cancel(); + task.await.unwrap(); + writers.shutdown().await; + } + + #[tokio::test] + async fn discovery_backpressure_bounds_tracked_jobs_without_losing_any() { + let config = DispatcherConfig { + max_tracked_jobs: 10, + ..Default::default() + }; + let dir = tempfile::tempdir().unwrap(); + let h = start(dir.path(), Arc::new(NoRateGate), config); + let append = h.writers.handle(); + for i in 0..50u64 { + append.append(message(i, "r@example.com")).await.unwrap(); + } + // Drain everything; backpressure must refill as jobs complete. + let mut delivered = HashSet::new(); + for _ in 0..50 { + let claim = h.handle.claim().await.expect("all 50 must arrive"); + assert!(delivered.insert(claim.job.message_id)); + claim.report(JobOutcome::Delivered { + response: "250 ok".into(), + }); + } + assert_eq!(delivered.len(), 50); + stop(h).await; + } + + #[tokio::test] + async fn dispatch_survives_segment_rotation() { + let dir = tempfile::tempdir().unwrap(); + let spool = Spool::open(dir.path().join("spool"), 1).unwrap(); + let mut wcfg = writer_config(); + wcfg.segment_target_bytes = 2048; // force rotations + let writers = LogWriters::start(&spool, wcfg).unwrap(); + let (store, recovered) = ShardStateStore::recover(spool.shard(0).path(), 0).unwrap(); + let cancel = CancellationToken::new(); + let (handle, task) = Dispatcher::start( + vec![ShardInit { + dir: spool.shard(0).path().to_path_buf(), + shared: writers.handle().shard_shared(0), + store, + recovered, + }], + writers.handle(), + Arc::new(NoRateGate), + DispatcherConfig::default(), + cancel.clone(), + ); + let append = writers.handle(); + for i in 0..30u64 { + let mut m = message(i, "r@example.com"); + m.body = Bytes::from(vec![b'x'; 512]); + append.append(m).await.unwrap(); + } + let mut seen = HashSet::new(); + for _ in 0..30 { + let claim = handle.claim().await.expect("all records across segments"); + assert!(seen.insert(claim.job.message_id)); + let body = handle.read_body(claim.job.location).await.unwrap(); + assert_eq!(body.len(), 512); + claim.report(JobOutcome::Delivered { + response: "250 ok".into(), + }); + } + cancel.cancel(); + task.await.unwrap(); + writers.shutdown().await; + } + + // ------------------------------------------------------------------ + // GC and compaction (PLAN §26.5). + + /// Poll until `cond` holds or ~5s elapse. + async fn eventually(mut cond: impl FnMut() -> bool, what: &str) { + for _ in 0..500 { + if cond() { + return; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + panic!("timed out waiting for: {what}"); + } + + fn sealed_segment_files(shard_dir: &std::path::Path) -> Vec { + let mut v: Vec = std::fs::read_dir(shard_dir) + .unwrap() + .filter_map(|e| e.unwrap().file_name().into_string().ok()) + .filter(|n| n.ends_with(".log") && n.starts_with("segment-")) + .collect(); + v.sort(); + v + } + + #[tokio::test] + async fn fully_delivered_sealed_segments_are_deleted_event_driven() { + let dir = tempfile::tempdir().unwrap(); + let spool = Spool::open(dir.path().join("spool"), 1).unwrap(); + let mut wcfg = writer_config(); + wcfg.segment_target_bytes = 2048; + let writers = LogWriters::start(&spool, wcfg).unwrap(); + let (store, recovered) = ShardStateStore::recover(spool.shard(0).path(), 0).unwrap(); + let cancel = CancellationToken::new(); + let (handle, task) = Dispatcher::start( + vec![ShardInit { + dir: spool.shard(0).path().to_path_buf(), + shared: writers.handle().shard_shared(0), + store, + recovered, + }], + writers.handle(), + Arc::new(NoRateGate), + DispatcherConfig { + safety_tick: Duration::from_millis(50), + ..Default::default() + }, + cancel.clone(), + ); + let append = writers.handle(); + for i in 0..12u64 { + let mut m = message(i, "r@example.com"); + m.body = Bytes::from(vec![b'x'; 400]); + append.append(m).await.unwrap(); + } + let shard_dir = spool.shard(0).path().to_path_buf(); + // Several segments sealed. + assert!(!sealed_segment_files(&shard_dir).is_empty()); + + for _ in 0..12 { + let claim = handle.claim().await.unwrap(); + claim.report(JobOutcome::Delivered { + response: "250 ok".into(), + }); + } + // Every sealed segment dies without waiting for anything periodic + // beyond the event itself (deletion happens in the terminal apply). + eventually( + || sealed_segment_files(&shard_dir).is_empty(), + "all sealed segments deleted", + ) + .await; + + cancel.cancel(); + task.await.unwrap(); + writers.shutdown().await; + } + + #[tokio::test] + async fn live_record_prevents_deletion_and_compaction_relocates_it() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_path_buf(); + let spool = Spool::open(root.join("spool"), 1).unwrap(); + let mut wcfg = writer_config(); + wcfg.segment_target_bytes = 2048; + let writers = LogWriters::start(&spool, wcfg).unwrap(); + let (store, recovered) = ShardStateStore::recover(spool.shard(0).path(), 0).unwrap(); + let cancel = CancellationToken::new(); + let (handle, task) = Dispatcher::start( + vec![ShardInit { + dir: spool.shard(0).path().to_path_buf(), + shared: writers.handle().shard_shared(0), + store, + recovered, + }], + writers.handle(), + Arc::new(NoRateGate), + DispatcherConfig { + safety_tick: Duration::from_millis(50), + compaction_dead_ratio: 0.5, + compaction_min_age: Duration::ZERO, + ..Default::default() + }, + cancel.clone(), + ); + let append = writers.handle(); + // m0..m3 fill segment 1; m4 forces rotation into segment 2. + for i in 0..5u64 { + let mut m = message(i, "r@example.com"); + m.body = Bytes::from(vec![b'x'; 400]); + append.append(m).await.unwrap(); + } + let shard_dir = spool.shard(0).path().to_path_buf(); + let first_sealed = sealed_segment_files(&shard_dir); + assert_eq!(first_sealed.len(), 1, "expected one sealed segment"); + + // Deliver everything except one message, which defers far out. + let mut survivor = None; + for _ in 0..5 { + let claim = handle.claim().await.unwrap(); + if survivor.is_none() && claim.job.location.segment == 1 { + survivor = Some(claim.job.message_id); + claim.report(JobOutcome::Deferred { + next_attempt_ms: now_ms() + 3_600_000, + remaining_recipients: vec!["r@example.com".into()], + error: "451 long defer".into(), + }); + } else { + claim.report(JobOutcome::Delivered { + response: "250 ok".into(), + }); + } + } + let survivor = survivor.expect("segment 1 had a claim"); + + // Segment 1 is 75% dead but held live by the survivor; compaction + // relocates the survivor, then segment 1 dies. + eventually( + || !shard_dir.join(crate::logqueue::segment::sealed_file_name(1)).exists(), + "compacted source segment deleted", + ) + .await; + + // Clean shutdown, then verify on-disk state. + cancel.cancel(); + task.await.unwrap(); + writers.shutdown().await; + drop(spool); + + let spool = Spool::open(root.join("spool"), 1).unwrap(); + let (_, recovered) = ShardStateStore::recover(spool.shard(0).path(), 0).unwrap(); + let d = recovered + .deferred + .get(&survivor) + .expect("survivor still deferred after relocation + restart"); + assert_ne!(d.location.segment, 1, "location moved off the dead segment"); + assert_eq!(d.location.generation, 1, "relocation bumped the generation"); + assert_eq!(d.attempts, 1); + + // The body is intact at the relocated position. + let reader = crate::logqueue::segment::open_segment_reader( + spool.shard(0).path(), + d.location.segment, + ) + .unwrap(); + let (header, body) = reader + .read_record_at(d.location.offset, crate::logqueue::record::MAX_RECORD_LEN) + .unwrap(); + assert_eq!(header.message_id, survivor); + assert_eq!(header.generation, 1); + assert_eq!(body, vec![b'x'; 400]); + } + + #[tokio::test] + async fn rediscovered_higher_generation_copy_wins_after_crashy_restart() { + // Simulate the crash window between the compaction copy landing and + // its Relocated journal entry: two copies of the same message id on + // disk, generation 0 and 1, no state at all. Discovery must track + // the generation-1 copy. + let dir = tempfile::tempdir().unwrap(); + let spool = Spool::open(dir.path().join("spool"), 1).unwrap(); + let writers = LogWriters::start(&spool, writer_config()).unwrap(); + let append = writers.handle(); + + let mut m = message(1, "r@example.com"); + let id = m.message_id; + m.body = Bytes::from_static(b"old copy"); + append.append(m).await.unwrap(); + let mut m = message(1, "r@example.com"); + m.generation = 1; + m.body = Bytes::from_static(b"new copy"); + let new_loc = append.append(m).await.unwrap(); + + let (store, recovered) = ShardStateStore::recover(spool.shard(0).path(), 0).unwrap(); + let cancel = CancellationToken::new(); + let (handle, task) = Dispatcher::start( + vec![ShardInit { + dir: spool.shard(0).path().to_path_buf(), + shared: writers.handle().shard_shared(0), + store, + recovered, + }], + writers.handle(), + Arc::new(NoRateGate), + DispatcherConfig::default(), + cancel.clone(), + ); + + let claim = handle.claim().await.unwrap(); + assert_eq!(claim.job.message_id, id); + assert_eq!(claim.job.location, new_loc); + let body = handle.read_body(claim.job.location).await.unwrap(); + assert_eq!(body, b"new copy"); + claim.report(JobOutcome::Delivered { + response: "250 ok".into(), + }); + + cancel.cancel(); + task.await.unwrap(); + writers.shutdown().await; + } +} diff --git a/smtp-server/src/logqueue/mod.rs b/smtp-server/src/logqueue/mod.rs new file mode 100644 index 0000000..c45c47d --- /dev/null +++ b/smtp-server/src/logqueue/mod.rs @@ -0,0 +1,126 @@ +//! Durable segmented append-only log queue. +//! +//! This module implements the storage layer described in +//! docs/plans/2026-07-20-durable-log-queue.md: sharded, segmented +//! append-only payload logs holding complete messages, with delivery state +//! tracked separately. Selected with `storage_type = "log"` (the default); +//! the legacy filesystem spool remains available as `"fs"`. + +pub mod dispatcher; +pub mod record; +pub mod segment; +pub mod shard; +pub mod spool; +pub mod state; +pub mod writer; + +use std::fmt; + +use thiserror::Error; + +/// Current on-disk format version for payload records and spool layout. +pub const FORMAT_VERSION: u16 = 1; + +/// A message's stable identity: the binary form of its ULID. +#[derive(Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub struct MessageId(pub [u8; 16]); + +impl MessageId { + pub fn from_ulid(u: ulid::Ulid) -> Self { + Self(u.to_bytes()) + } + + pub fn parse(s: &str) -> Result { + let u = ulid::Ulid::from_string(s) + .map_err(|e| QueueError::InvalidMessageId(format!("{s:?}: {e}")))?; + Ok(Self::from_ulid(u)) + } + + pub fn to_ulid(self) -> ulid::Ulid { + ulid::Ulid::from_bytes(self.0) + } +} + +impl fmt::Display for MessageId { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + self.to_ulid().fmt(f) + } +} + +impl fmt::Debug for MessageId { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "MessageId({})", self.to_ulid()) + } +} + +/// Physical location of a payload record. Stored state must always carry the +/// explicit location; nothing may re-derive a shard from the current +/// configured writer count. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +pub struct JobLocation { + pub shard: u16, + pub segment: u64, + pub offset: u64, + pub length: u32, + pub ordinal: u32, + pub generation: u32, +} + +#[derive(Debug, Error, miette::Diagnostic)] +pub enum QueueError { + #[error("i/o error on {path}: {source}")] + Io { + path: String, + #[source] + source: std::io::Error, + }, + + #[error("invalid message id {0}")] + InvalidMessageId(String), + + #[error("record too large: {len} bytes exceeds limit {limit}")] + RecordTooLarge { len: u64, limit: u64 }, + + #[error("invalid record field: {0}")] + InvalidRecord(String), + + #[error("corrupt record at offset {offset}: {reason}")] + CorruptRecord { offset: u64, reason: String }, + + #[error("unsupported format version {found} (supported: {supported})")] + UnsupportedVersion { found: u16, supported: u16 }, + + #[error("corruption inside sealed segment {path} at offset {offset}: {reason}")] + CorruptSealedSegment { + path: String, + offset: u64, + reason: String, + }, + + #[error("spool layout error: {0}")] + Layout(String), + + #[error("spool is locked by another process ({path})")] + SpoolLocked { path: String }, + + #[error("append writer for shard {0} is shut down")] + WriterClosed(u16), + + #[error( + "segment size {segment_bytes} cannot hold the maximum message size \ + {max_message_bytes} plus record overhead" + )] + SegmentTooSmall { + segment_bytes: u64, + max_message_bytes: u64, + }, +} + +impl QueueError { + pub(crate) fn io(path: impl AsRef, source: std::io::Error) -> Self { + QueueError::Io { + path: path.as_ref().display().to_string(), + source, + } + } +} diff --git a/smtp-server/src/logqueue/record.rs b/smtp-server/src/logqueue/record.rs new file mode 100644 index 0000000..9150ddb --- /dev/null +++ b/smtp-server/src/logqueue/record.rs @@ -0,0 +1,509 @@ +//! Versioned, self-framing payload record encoding. +//! +//! Layout (all integers little-endian): +//! +//! ```text +//! offset size field +//! 0 4 magic ("HWLQ") +//! 4 2 format version +//! 6 2 flags (reserved, must be zero) +//! 8 4 record_len — total record size: header_len + body_len +//! 12 4 header_len — body starts at this offset within the record +//! 16 4 header_crc — crc32 over [0..16) ++ [20..header_len) +//! 20 4 payload_crc — crc32 over the body +//! 24 16 message id (binary ULID) +//! 40 8 enqueue timestamp, unix milliseconds (i64) +//! 48 4 relocation generation +//! 52 4 per-segment record ordinal +//! 56 var envelope: sender_len u16, sender bytes, +//! rcpt_count u16, then per recipient u16 len + bytes +//! … body (record_len - header_len bytes) +//! ``` +//! +//! The fixed header plus envelope is everything the dispatcher needs to +//! construct a job; it never has to read or decode the body. `record_len` +//! lets a scanner skip directly to the next record. + +use super::{MessageId, QueueError, FORMAT_VERSION}; + +pub const MAGIC: [u8; 4] = *b"HWLQ"; +/// Size of the fixed portion of the header, before the envelope. +pub const FIXED_HEADER_LEN: usize = 56; +/// Byte range covered by `header_crc`, part 1 (everything before the crc +/// fields) and the offset where part 2 (message id onward) begins. +const CRC_PART1_END: usize = 16; +const CRC_PART2_START: usize = 20; + +/// Upper bound on encoded record size. `record_len` is a u32; keep a margin +/// below `u32::MAX` so arithmetic can never overflow. +pub const MAX_RECORD_LEN: u32 = u32::MAX - 4096; + +/// The envelope and identity of a queued message, decoded from a record +/// header without touching the body. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct RecordHeader { + pub message_id: MessageId, + pub enqueue_ms: i64, + pub generation: u32, + pub ordinal: u32, + pub sender: String, + pub recipients: Vec, + pub record_len: u32, + pub header_len: u32, + pub payload_crc: u32, +} + +impl RecordHeader { + pub fn body_len(&self) -> u32 { + self.record_len - self.header_len + } +} + +/// Why a header could not be decoded. `Incomplete` means the buffer ends +/// before the record does — expected at the tail of an active segment. +/// `Corrupt` means the bytes are wrong, not merely missing. +#[derive(Debug)] +pub enum DecodeError { + /// More bytes are needed; `needed` is the total record prefix length + /// required to make progress (from the start of the record). + Incomplete { needed: usize }, + Corrupt(String), + UnsupportedVersion(u16), +} + +impl DecodeError { + pub fn into_queue_error(self, offset: u64) -> QueueError { + match self { + DecodeError::Incomplete { needed } => QueueError::CorruptRecord { + offset, + reason: format!("record truncated: needs {needed} bytes"), + }, + DecodeError::Corrupt(reason) => QueueError::CorruptRecord { offset, reason }, + DecodeError::UnsupportedVersion(found) => QueueError::UnsupportedVersion { + found, + supported: FORMAT_VERSION, + }, + } + } +} + +/// Everything needed to encode a payload record. +pub struct RecordParams<'a> { + pub message_id: MessageId, + pub enqueue_ms: i64, + pub generation: u32, + pub ordinal: u32, + pub sender: &'a str, + pub recipients: &'a [String], + pub body: &'a [u8], +} + +/// Encoded size of a record, or an error if any field exceeds format limits. +pub fn encoded_len(params: &RecordParams<'_>) -> Result { + let header = header_len(params)?; + let total = header as u64 + params.body.len() as u64; + if total > MAX_RECORD_LEN as u64 { + return Err(QueueError::RecordTooLarge { + len: total, + limit: MAX_RECORD_LEN as u64, + }); + } + Ok(total as u32) +} + +fn header_len(params: &RecordParams<'_>) -> Result { + if params.sender.len() > u16::MAX as usize { + return Err(QueueError::InvalidRecord(format!( + "sender address is {} bytes, exceeds u16", + params.sender.len() + ))); + } + if params.recipients.is_empty() { + return Err(QueueError::InvalidRecord( + "record must have at least one recipient".into(), + )); + } + if params.recipients.len() > u16::MAX as usize { + return Err(QueueError::InvalidRecord(format!( + "{} recipients exceeds u16", + params.recipients.len() + ))); + } + let mut len = FIXED_HEADER_LEN as u64 + 2 + params.sender.len() as u64 + 2; + for rcpt in params.recipients { + if rcpt.len() > u16::MAX as usize { + return Err(QueueError::InvalidRecord(format!( + "recipient address is {} bytes, exceeds u16", + rcpt.len() + ))); + } + len += 2 + rcpt.len() as u64; + } + if len > MAX_RECORD_LEN as u64 { + return Err(QueueError::RecordTooLarge { + len, + limit: MAX_RECORD_LEN as u64, + }); + } + Ok(len as u32) +} + +/// Encode a complete record into a fresh buffer. +pub fn encode(params: &RecordParams<'_>) -> Result, QueueError> { + let header_len = header_len(params)?; + let record_len = encoded_len(params)?; + + let mut buf = Vec::with_capacity(record_len as usize); + buf.extend_from_slice(&MAGIC); + buf.extend_from_slice(&FORMAT_VERSION.to_le_bytes()); + buf.extend_from_slice(&0u16.to_le_bytes()); // flags + buf.extend_from_slice(&record_len.to_le_bytes()); + buf.extend_from_slice(&header_len.to_le_bytes()); + buf.extend_from_slice(&0u32.to_le_bytes()); // header_crc placeholder + buf.extend_from_slice(&crc32fast::hash(params.body).to_le_bytes()); + buf.extend_from_slice(¶ms.message_id.0); + buf.extend_from_slice(¶ms.enqueue_ms.to_le_bytes()); + buf.extend_from_slice(¶ms.generation.to_le_bytes()); + buf.extend_from_slice(¶ms.ordinal.to_le_bytes()); + debug_assert_eq!(buf.len(), FIXED_HEADER_LEN); + + buf.extend_from_slice(&(params.sender.len() as u16).to_le_bytes()); + buf.extend_from_slice(params.sender.as_bytes()); + buf.extend_from_slice(&(params.recipients.len() as u16).to_le_bytes()); + for rcpt in params.recipients { + buf.extend_from_slice(&(rcpt.len() as u16).to_le_bytes()); + buf.extend_from_slice(rcpt.as_bytes()); + } + debug_assert_eq!(buf.len(), header_len as usize); + + let crc = header_crc(&buf, header_len as usize); + buf[16..20].copy_from_slice(&crc.to_le_bytes()); + + buf.extend_from_slice(params.body); + debug_assert_eq!(buf.len(), record_len as usize); + Ok(buf) +} + +/// crc32 over the header with the `header_crc` field itself excluded. +fn header_crc(header: &[u8], header_len: usize) -> u32 { + let mut hasher = crc32fast::Hasher::new(); + hasher.update(&header[..CRC_PART1_END]); + hasher.update(&header[CRC_PART2_START..header_len]); + hasher.finalize() +} + +fn read_u16(buf: &[u8], at: usize) -> u16 { + u16::from_le_bytes([buf[at], buf[at + 1]]) +} + +fn read_u32(buf: &[u8], at: usize) -> u32 { + u32::from_le_bytes(buf[at..at + 4].try_into().unwrap()) +} + +/// Decode a record header from `buf`, which starts at a record boundary. +/// `buf` may be shorter than the full record; only the header bytes are +/// required. `max_record_len` bounds `record_len` for sanity (a corrupt +/// length field must not drive huge reads). +pub fn decode_header(buf: &[u8], max_record_len: u32) -> Result { + if buf.len() < FIXED_HEADER_LEN { + return Err(DecodeError::Incomplete { + needed: FIXED_HEADER_LEN, + }); + } + if buf[0..4] != MAGIC { + return Err(DecodeError::Corrupt("bad magic".into())); + } + let version = read_u16(buf, 4); + if version != FORMAT_VERSION { + return Err(DecodeError::UnsupportedVersion(version)); + } + let flags = read_u16(buf, 6); + if flags != 0 { + return Err(DecodeError::Corrupt(format!("unknown flags {flags:#06x}"))); + } + let record_len = read_u32(buf, 8); + let header_len = read_u32(buf, 12); + if header_len < (FIXED_HEADER_LEN as u32 + 4) + || header_len > record_len + || record_len > max_record_len.min(MAX_RECORD_LEN) + { + return Err(DecodeError::Corrupt(format!( + "implausible lengths: record_len={record_len} header_len={header_len}" + ))); + } + if buf.len() < header_len as usize { + return Err(DecodeError::Incomplete { + needed: header_len as usize, + }); + } + + let stored_crc = read_u32(buf, 16); + if header_crc(buf, header_len as usize) != stored_crc { + return Err(DecodeError::Corrupt("header checksum mismatch".into())); + } + + let payload_crc = read_u32(buf, 20); + let mut id = [0u8; 16]; + id.copy_from_slice(&buf[24..40]); + let enqueue_ms = i64::from_le_bytes(buf[40..48].try_into().unwrap()); + let generation = read_u32(buf, 48); + let ordinal = read_u32(buf, 52); + + // Envelope. The header crc already validated these bytes, so length + // errors here indicate an encoder bug rather than disk corruption, but + // they are still reported as corruption instead of panicking. + let end = header_len as usize; + let mut at = FIXED_HEADER_LEN; + let sender = take_str(buf, &mut at, end)?; + if at + 2 > end { + return Err(DecodeError::Corrupt("envelope overruns header".into())); + } + let rcpt_count = read_u16(buf, at) as usize; + at += 2; + if rcpt_count == 0 { + return Err(DecodeError::Corrupt("record has no recipients".into())); + } + let mut recipients = Vec::with_capacity(rcpt_count); + for _ in 0..rcpt_count { + recipients.push(take_str(buf, &mut at, end)?); + } + if at != end { + return Err(DecodeError::Corrupt(format!( + "{} trailing bytes after envelope", + end - at + ))); + } + + Ok(RecordHeader { + message_id: MessageId(id), + enqueue_ms, + generation, + ordinal, + sender, + recipients, + record_len, + header_len, + payload_crc, + }) +} + +fn take_str(buf: &[u8], at: &mut usize, end: usize) -> Result { + if *at + 2 > end { + return Err(DecodeError::Corrupt("envelope overruns header".into())); + } + let len = read_u16(buf, *at) as usize; + *at += 2; + if *at + len > end { + return Err(DecodeError::Corrupt("envelope overruns header".into())); + } + let s = std::str::from_utf8(&buf[*at..*at + len]) + .map_err(|_| DecodeError::Corrupt("envelope field is not UTF-8".into()))? + .to_owned(); + *at += len; + Ok(s) +} + +/// Verify a record body against the checksum recorded in its header. +pub fn verify_body(header: &RecordHeader, body: &[u8]) -> Result<(), QueueError> { + if body.len() != header.body_len() as usize { + return Err(QueueError::InvalidRecord(format!( + "body length {} does not match header {}", + body.len(), + header.body_len() + ))); + } + if crc32fast::hash(body) != header.payload_crc { + return Err(QueueError::InvalidRecord( + "payload checksum mismatch".into(), + )); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn params<'a>(body: &'a [u8], recipients: &'a [String]) -> RecordParams<'a> { + RecordParams { + message_id: MessageId::from_ulid(ulid::Ulid::from_parts(1234, 5678)), + enqueue_ms: 1_752_000_000_000, + generation: 0, + ordinal: 7, + sender: "sender@example.com", + recipients, + body, + } + } + + #[test] + fn round_trip() { + let rcpts = vec!["a@example.com".to_string(), "b@example.org".to_string()]; + let body = b"Subject: hi\r\n\r\nhello world"; + let p = params(body, &rcpts); + let buf = encode(&p).unwrap(); + assert_eq!(buf.len() as u32, encoded_len(&p).unwrap()); + + let h = decode_header(&buf, MAX_RECORD_LEN).unwrap(); + assert_eq!(h.message_id, p.message_id); + assert_eq!(h.enqueue_ms, p.enqueue_ms); + assert_eq!(h.ordinal, 7); + assert_eq!(h.generation, 0); + assert_eq!(h.sender, p.sender); + assert_eq!(h.recipients, rcpts); + assert_eq!(h.body_len() as usize, body.len()); + verify_body(&h, &buf[h.header_len as usize..]).unwrap(); + } + + #[test] + fn empty_body_and_single_recipient() { + let rcpts = vec!["a@example.com".to_string()]; + let p = params(b"", &rcpts); + let buf = encode(&p).unwrap(); + let h = decode_header(&buf, MAX_RECORD_LEN).unwrap(); + assert_eq!(h.body_len(), 0); + verify_body(&h, b"").unwrap(); + } + + #[test] + fn rejects_zero_recipients() { + let rcpts: Vec = vec![]; + assert!(matches!( + encode(¶ms(b"x", &rcpts)), + Err(QueueError::InvalidRecord(_)) + )); + } + + #[test] + fn incomplete_fixed_header() { + let rcpts = vec!["a@example.com".to_string()]; + let buf = encode(¶ms(b"body", &rcpts)).unwrap(); + for cut in [0, 1, FIXED_HEADER_LEN - 1] { + match decode_header(&buf[..cut], MAX_RECORD_LEN) { + Err(DecodeError::Incomplete { needed }) => { + assert_eq!(needed, FIXED_HEADER_LEN) + } + other => panic!("expected Incomplete, got {other:?}"), + } + } + } + + #[test] + fn incomplete_variable_header() { + let rcpts = vec!["a@example.com".to_string()]; + let buf = encode(¶ms(b"body", &rcpts)).unwrap(); + let h = decode_header(&buf, MAX_RECORD_LEN).unwrap(); + match decode_header(&buf[..h.header_len as usize - 1], MAX_RECORD_LEN) { + Err(DecodeError::Incomplete { needed }) => { + assert_eq!(needed, h.header_len as usize) + } + other => panic!("expected Incomplete, got {other:?}"), + } + } + + #[test] + fn corrupt_magic_and_version() { + let rcpts = vec!["a@example.com".to_string()]; + let mut buf = encode(¶ms(b"body", &rcpts)).unwrap(); + buf[0] ^= 0xff; + assert!(matches!( + decode_header(&buf, MAX_RECORD_LEN), + Err(DecodeError::Corrupt(_)) + )); + buf[0] ^= 0xff; + buf[4] = 0xfe; + assert!(matches!( + decode_header(&buf, MAX_RECORD_LEN), + Err(DecodeError::UnsupportedVersion(_)) + )); + } + + #[test] + fn header_bitflips_are_detected() { + let rcpts = vec!["a@example.com".to_string(), "b@example.com".to_string()]; + let clean = encode(¶ms(b"body", &rcpts)).unwrap(); + let header_len = decode_header(&clean, MAX_RECORD_LEN).unwrap().header_len as usize; + // Flip one bit at every header position; every flip must be caught. + for i in 0..header_len { + let mut buf = clean.clone(); + buf[i] ^= 0x01; + assert!( + decode_header(&buf, MAX_RECORD_LEN).is_err(), + "bit flip at byte {i} went undetected" + ); + } + } + + #[test] + fn body_corruption_detected() { + let rcpts = vec!["a@example.com".to_string()]; + let mut buf = encode(¶ms(b"body", &rcpts)).unwrap(); + let h = decode_header(&buf, MAX_RECORD_LEN).unwrap(); + let last = buf.len() - 1; + buf[last] ^= 0x01; + assert!(verify_body(&h, &buf[h.header_len as usize..]).is_err()); + } + + #[test] + fn record_len_bound_enforced() { + let rcpts = vec!["a@example.com".to_string()]; + let buf = encode(¶ms(&[0u8; 4096], &rcpts)).unwrap(); + // A cap below the actual record size must reject the header. + assert!(matches!( + decode_header(&buf, 128), + Err(DecodeError::Corrupt(_)) + )); + } + + #[test] + fn large_body_round_trip() { + let rcpts = vec!["a@example.com".to_string()]; + let body = vec![0xABu8; 25 * 1024 * 1024]; + let p = params(&body, &rcpts); + let buf = encode(&p).unwrap(); + assert_eq!(buf.len() as u32, encoded_len(&p).unwrap()); + + let h = decode_header(&buf, MAX_RECORD_LEN).unwrap(); + assert_eq!(h.body_len() as usize, body.len()); + verify_body(&h, &buf[h.header_len as usize..]).unwrap(); + } + + #[test] + fn many_recipients_round_trip() { + let rcpts: Vec = (0..5000).map(|i| format!("user{i}@example.com")).collect(); + let body = b"hello world"; + let p = params(body, &rcpts); + let buf = encode(&p).unwrap(); + + let h = decode_header(&buf, MAX_RECORD_LEN).unwrap(); + assert_eq!(h.recipients.len(), 5000); + assert_eq!(h.recipients, rcpts); + verify_body(&h, &buf[h.header_len as usize..]).unwrap(); + } + + #[test] + fn sender_and_recipient_length_limits() { + let rcpts = vec!["a@example.com".to_string()]; + let long_sender = "a".repeat(u16::MAX as usize + 1); + let mut p = params(b"body", &rcpts); + p.sender = &long_sender; + assert!(matches!(encode(&p), Err(QueueError::InvalidRecord(_)))); + + let long_rcpts = vec!["a".repeat(u16::MAX as usize + 1)]; + let p2 = params(b"body", &long_rcpts); + assert!(matches!(encode(&p2), Err(QueueError::InvalidRecord(_)))); + } + + #[test] + fn enqueue_timestamp_and_generation_preserved() { + let rcpts = vec!["a@example.com".to_string()]; + let mut p = params(b"body", &rcpts); + p.enqueue_ms = -1_000_000_000_000; // well before the unix epoch + p.generation = 42; + let buf = encode(&p).unwrap(); + + let h = decode_header(&buf, MAX_RECORD_LEN).unwrap(); + assert_eq!(h.enqueue_ms, p.enqueue_ms); + assert_eq!(h.generation, 42); + } +} diff --git a/smtp-server/src/logqueue/segment.rs b/smtp-server/src/logqueue/segment.rs new file mode 100644 index 0000000..bc3aa29 --- /dev/null +++ b/smtp-server/src/logqueue/segment.rs @@ -0,0 +1,717 @@ +//! Segment files: the unit of payload storage and reclamation. +//! +//! A shard has at most one active segment (`segment-NNNNNNNNNNNN.open`), +//! which is sealed by renaming it to `.log`. Sealed segments are immutable. +//! Records never span segments. + +use std::fs::{File, OpenOptions}; +use std::io::Write; +use std::os::unix::fs::FileExt; +use std::path::{Path, PathBuf}; + +use super::record::{self, DecodeError, RecordHeader, FIXED_HEADER_LEN}; +use super::QueueError; + +pub const SEALED_EXT: &str = "log"; +pub const ACTIVE_EXT: &str = "open"; + +/// Read granularity while scanning records; large enough to cover almost +/// every header in one positioned read. +const SCAN_CHUNK: usize = 128 * 1024; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum SegmentKind { + Active, + Sealed, +} + +pub fn sealed_file_name(segment: u64) -> String { + format!("segment-{segment:012}.{SEALED_EXT}") +} + +pub fn active_file_name(segment: u64) -> String { + format!("segment-{segment:012}.{ACTIVE_EXT}") +} + +/// Parse a segment file name into its ordinal and kind. Returns `None` for +/// unrelated files. +pub fn parse_file_name(name: &str) -> Option<(u64, SegmentKind)> { + let rest = name.strip_prefix("segment-")?; + let (digits, ext) = rest.split_once('.')?; + if digits.len() != 12 || !digits.bytes().all(|b| b.is_ascii_digit()) { + return None; + } + let ordinal = digits.parse().ok()?; + let kind = match ext { + SEALED_EXT => SegmentKind::Sealed, + ACTIVE_EXT => SegmentKind::Active, + _ => return None, + }; + Some((ordinal, kind)) +} + +/// The shard's current append target. All methods are synchronous; the +/// append writer owns this on a dedicated task. +pub struct ActiveSegment { + file: File, + path: PathBuf, + segment: u64, + len: u64, + next_ordinal: u32, +} + +impl ActiveSegment { + /// Create a brand-new active segment. Fails if the file already exists: + /// segment ordinals are never reused. + pub fn create(shard_dir: &Path, segment: u64) -> Result { + let path = shard_dir.join(active_file_name(segment)); + // O_APPEND, like `recover`: after a partial-write rollback + // (`set_len` back to the committed tail) the next write must land + // at the new EOF. A plain write cursor would sit past the + // truncation point and punch a hole that loses every later record + // at recovery. + let file = OpenOptions::new() + .append(true) + .create_new(true) + .open(&path) + .map_err(|e| QueueError::io(&path, e))?; + Ok(Self { + file, + path, + segment, + len: 0, + next_ordinal: 0, + }) + } + + /// Reopen an existing active segment for append after its tail has been + /// validated (and truncated if needed) by [`validate_active_tail`]. + pub fn recover(path: PathBuf, segment: u64, tail: &TailValidation) -> Result { + // O_APPEND: writes land at end-of-file, which after tail validation + // is exactly the committed tail (create() starts at 0 and only ever + // writes sequentially, so it needs no special mode). + let file = OpenOptions::new() + .append(true) + .open(&path) + .map_err(|e| QueueError::io(&path, e))?; + Ok(Self { + file, + path, + segment, + len: tail.committed_len, + next_ordinal: tail.next_ordinal, + }) + } + + pub fn segment(&self) -> u64 { + self.segment + } + + /// Committed length: every byte below this is a complete record. + pub fn len(&self) -> u64 { + self.len + } + + pub fn next_ordinal(&self) -> u32 { + self.next_ordinal + } + + #[cfg(test)] + pub fn path(&self) -> &Path { + &self.path + } + + /// Append one encoded record. Returns the record's offset. The caller + /// must have encoded with `ordinal == self.next_ordinal()`. + /// + /// The write either fully succeeds or the segment is left with the + /// previous committed length: on a partial failure the file is truncated + /// back so a retry (or seal) never leaves a torn record below the + /// committed tail. + pub fn append(&mut self, encoded: &[u8]) -> Result { + let offset = self.len; + if let Err(e) = self.file.write_all(encoded) { + // Best effort: cut back to the committed tail. If this fails the + // tail validator will do the same at next startup. + let _ = self.file.set_len(offset); + return Err(QueueError::io(&self.path, e)); + } + self.len += encoded.len() as u64; + self.next_ordinal += 1; + Ok(offset) + } + + /// Seal this segment: rename `.open` to `.log`. Returns the sealed path + /// and final committed length. The file is immutable afterwards; the + /// caller must not append through this handle again (the writer swaps + /// in a fresh segment, or tests drop it). + pub fn seal_in_place(&self) -> Result<(PathBuf, u64), QueueError> { + let sealed = self + .path + .parent() + .expect("segment path has a parent") + .join(sealed_file_name(self.segment)); + std::fs::rename(&self.path, &sealed).map_err(|e| QueueError::io(&self.path, e))?; + Ok((sealed, self.len)) + } +} + +/// Read-only positioned access to a (sealed or active) segment. +/// +/// Reading an active segment concurrently with the writer is safe as long as +/// callers stay below the published committed tail. +pub struct SegmentReader { + file: File, + path: PathBuf, +} + +impl SegmentReader { + pub fn open(path: impl Into) -> Result { + let path = path.into(); + let file = File::open(&path).map_err(|e| QueueError::io(&path, e))?; + Ok(Self { file, path }) + } + + fn read_exact_at(&self, buf: &mut [u8], offset: u64) -> Result<(), QueueError> { + self.file + .read_exact_at(buf, offset) + .map_err(|e| QueueError::io(&self.path, e)) + } + + /// Decode the record header at `offset`. + pub fn read_header_at( + &self, + offset: u64, + max_record_len: u32, + ) -> Result { + let mut buf = vec![0u8; FIXED_HEADER_LEN]; + self.read_exact_at(&mut buf, offset)?; + loop { + match record::decode_header(&buf, max_record_len) { + Ok(h) => return Ok(h), + Err(DecodeError::Incomplete { needed }) if needed > buf.len() => { + let have = buf.len(); + buf.resize(needed, 0); + self.read_exact_at(&mut buf[have..], offset + have as u64)?; + } + Err(e) => return Err(e.into_queue_error(offset)), + } + } + } + + /// Read and checksum-verify the body of a record whose header was read + /// at `offset`. + pub fn read_body(&self, header: &RecordHeader, offset: u64) -> Result, QueueError> { + let mut body = vec![0u8; header.body_len() as usize]; + self.read_exact_at(&mut body, offset + header.header_len as u64)?; + record::verify_body(header, &body)?; + Ok(body) + } + + /// Read a complete record (header + verified body) at `offset`. + pub fn read_record_at( + &self, + offset: u64, + max_record_len: u32, + ) -> Result<(RecordHeader, Vec), QueueError> { + let header = self.read_header_at(offset, max_record_len)?; + let body = self.read_body(&header, offset)?; + Ok((header, body)) + } + + fn file_len(&self) -> Result { + Ok(self + .file + .metadata() + .map_err(|e| QueueError::io(&self.path, e))? + .len()) + } +} + +/// One step of a header scan. +enum ScanStep { + Record { header: RecordHeader, offset: u64 }, + /// Clean end: `end` is the offset one past the last complete record. + End { end: u64 }, + /// The bytes at `offset` are not a complete valid record. + Invalid { offset: u64, reason: String }, +} + +/// Streaming header scanner over a segment file. Reads headers, skips +/// bodies. `verify_bodies` additionally reads and checksums each body (used +/// for active-tail validation). +struct Scanner<'a> { + reader: &'a SegmentReader, + end: u64, + offset: u64, + max_record_len: u32, + verify_bodies: bool, + buf: Vec, +} + +impl<'a> Scanner<'a> { + fn new( + reader: &'a SegmentReader, + start: u64, + end: u64, + max_record_len: u32, + verify_bodies: bool, + ) -> Self { + Self { + reader, + end, + offset: start, + max_record_len, + verify_bodies, + buf: Vec::new(), + } + } + + fn read_window(&mut self, len: usize) -> Result<(), QueueError> { + self.buf.resize(len, 0); + self.reader.read_exact_at(&mut self.buf, self.offset) + } + + fn next(&mut self) -> Result { + let remaining = self.end - self.offset; + if remaining == 0 { + return Ok(ScanStep::End { end: self.offset }); + } + if remaining < FIXED_HEADER_LEN as u64 { + return Ok(ScanStep::Invalid { + offset: self.offset, + reason: format!("{remaining} trailing bytes, shorter than a record header"), + }); + } + + let window = SCAN_CHUNK.min(remaining as usize); + self.read_window(window)?; + let header = loop { + match record::decode_header(&self.buf, self.max_record_len) { + Ok(h) => break h, + Err(DecodeError::Incomplete { needed }) => { + if needed as u64 > remaining { + return Ok(ScanStep::Invalid { + offset: self.offset, + reason: format!( + "record needs {needed} header bytes but only {remaining} remain" + ), + }); + } + if needed <= self.buf.len() { + // decode_header asked for bytes we already have: + // internal inconsistency, treat as corrupt. + return Ok(ScanStep::Invalid { + offset: self.offset, + reason: "header decoder made no progress".into(), + }); + } + self.read_window(needed)?; + } + Err(DecodeError::UnsupportedVersion(v)) => { + return Ok(ScanStep::Invalid { + offset: self.offset, + reason: format!("unsupported record version {v}"), + }); + } + Err(DecodeError::Corrupt(reason)) => { + return Ok(ScanStep::Invalid { + offset: self.offset, + reason, + }); + } + } + }; + + if header.record_len as u64 > remaining { + return Ok(ScanStep::Invalid { + offset: self.offset, + reason: format!( + "record length {} overruns segment end by {}", + header.record_len, + header.record_len as u64 - remaining + ), + }); + } + + if self.verify_bodies { + if let Err(e) = self.reader.read_body(&header, self.offset) { + return Ok(ScanStep::Invalid { + offset: self.offset, + reason: format!("body verification failed: {e}"), + }); + } + } + + let offset = self.offset; + self.offset += header.record_len as u64; + Ok(ScanStep::Record { header, offset }) + } +} + +/// Open a segment by ordinal, whether sealed or still active. Rotation can +/// race this (rename .open -> .log), so the sealed name is tried first and +/// the active name second, then once more in case the rename happened in +/// between. An already-open descriptor keeps working across the rename, so +/// callers may cache the reader. +pub fn open_segment_reader(shard_dir: &Path, segment: u64) -> Result { + for name in [ + sealed_file_name(segment), + active_file_name(segment), + sealed_file_name(segment), + ] { + match SegmentReader::open(shard_dir.join(name)) { + Ok(r) => return Ok(r), + Err(QueueError::Io { ref source, .. }) + if source.kind() == std::io::ErrorKind::NotFound => {} + Err(e) => return Err(e), + } + } + Err(QueueError::Layout(format!( + "segment {segment} not found in {}", + shard_dir.display() + ))) +} + +/// Outcome of validating (and possibly truncating) an active segment tail. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct TailValidation { + /// Length of the valid record prefix; the file's length after validation. + pub committed_len: u64, + /// Number of valid records. + pub records: u32, + /// Ordinal the next appended record must use. + pub next_ordinal: u32, + /// Bytes discarded from the tail (0 for a clean shutdown). + pub truncated_bytes: u64, +} + +/// Validate an active segment: scan from the start, verify every header and +/// body checksum and the ordinal sequence, and truncate the file at the +/// first invalid position. +/// +/// With sequential appends a crash can only tear the tail, so everything +/// past the first invalid byte is unrecoverable garbage. Under the accepted +/// page-cache durability model, a power loss may also punch holes earlier in +/// the file; bytes after such a hole are discarded with the tail and counted +/// in `truncated_bytes` (this falls under "recently accepted mail may be +/// lost", and the truncation is logged loudly by the caller). +pub fn validate_active_tail(path: &Path, max_record_len: u32) -> Result { + let reader = SegmentReader::open(path)?; + let file_len = reader.file_len()?; + let mut scanner = Scanner::new(&reader, 0, file_len, max_record_len, true); + + let mut records = 0u32; + let mut next_ordinal = 0u32; + let (committed_len, invalid_reason) = loop { + match scanner.next()? { + ScanStep::Record { header, offset } => { + if header.ordinal != next_ordinal { + break ( + offset, + Some(format!( + "ordinal {} where {} was expected", + header.ordinal, next_ordinal + )), + ); + } + records += 1; + next_ordinal += 1; + } + ScanStep::End { end } => break (end, None), + ScanStep::Invalid { offset, reason } => break (offset, Some(reason)), + } + }; + + let truncated_bytes = file_len - committed_len; + if truncated_bytes > 0 { + let file = OpenOptions::new() + .write(true) + .open(path) + .map_err(|e| QueueError::io(path, e))?; + file.set_len(committed_len) + .map_err(|e| QueueError::io(path, e))?; + tracing::warn!( + path = %path.display(), + committed_len, + truncated_bytes, + reason = invalid_reason.as_deref().unwrap_or("unknown"), + "truncated invalid tail of active segment" + ); + } + + Ok(TailValidation { + committed_len, + records, + next_ordinal, + truncated_bytes, + }) +} + +/// Scan record headers in `[start, end)` of a segment, invoking `f` for each +/// record. `f` returns whether to continue; the scan's return value is the +/// offset one past the last visited record (== `end` when it ran to +/// completion). Any invalid record is an error: sealed segments must be +/// perfect, and callers scanning an active segment must pass the committed +/// tail as `end`, below which the same holds. +pub fn scan_headers( + reader: &SegmentReader, + start: u64, + end: u64, + max_record_len: u32, + mut f: impl FnMut(u64, RecordHeader) -> bool, +) -> Result { + let mut scanner = Scanner::new(reader, start, end, max_record_len, false); + loop { + match scanner.next()? { + ScanStep::Record { header, offset } => { + let next = offset + header.record_len as u64; + if !f(offset, header) { + return Ok(next); + } + } + ScanStep::End { end } => return Ok(end), + ScanStep::Invalid { offset, reason } => { + return Err(QueueError::CorruptSealedSegment { + path: reader.path.display().to_string(), + offset, + reason, + }) + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::logqueue::record::{encode, RecordParams, MAX_RECORD_LEN}; + use crate::logqueue::MessageId; + + fn record(ordinal: u32, body: &[u8]) -> Vec { + let recipients = vec!["rcpt@example.com".to_string()]; + encode(&RecordParams { + message_id: MessageId::from_ulid(ulid::Ulid::from_parts(ordinal as u64, 42)), + enqueue_ms: 1_752_000_000_000 + ordinal as i64, + generation: 0, + ordinal, + sender: "sender@example.com", + recipients: &recipients, + body, + }) + .unwrap() + } + + fn fill_segment(dir: &Path, segment: u64, bodies: &[&[u8]]) -> (ActiveSegment, Vec) { + let mut seg = ActiveSegment::create(dir, segment).unwrap(); + let mut offsets = Vec::new(); + for (i, body) in bodies.iter().enumerate() { + let encoded = record(i as u32, body); + offsets.push(seg.append(&encoded).unwrap()); + } + (seg, offsets) + } + + #[test] + fn file_name_round_trip() { + assert_eq!( + parse_file_name(&sealed_file_name(42)), + Some((42, SegmentKind::Sealed)) + ); + assert_eq!( + parse_file_name(&active_file_name(7)), + Some((7, SegmentKind::Active)) + ); + assert_eq!(parse_file_name("segment-123.log"), None); // wrong width + assert_eq!(parse_file_name("segment-00000000000x.log"), None); + assert_eq!(parse_file_name("checkpoint"), None); + assert_eq!(parse_file_name("segment-000000000001.tmp"), None); + } + + #[test] + fn append_read_seal_read() { + let dir = tempfile::tempdir().unwrap(); + let (seg, offsets) = + fill_segment(dir.path(), 1, &[b"first body", b"second body", b"third"]); + let active_path = seg.path().to_path_buf(); + + // Read back through the active file. + let reader = SegmentReader::open(&active_path).unwrap(); + let (h, body) = reader.read_record_at(offsets[1], MAX_RECORD_LEN).unwrap(); + assert_eq!(h.ordinal, 1); + assert_eq!(body, b"second body"); + + // Seal, then read through the sealed file. + let (sealed_path, len) = seg.seal_in_place().unwrap(); + assert!(!active_path.exists()); + assert_eq!(len, std::fs::metadata(&sealed_path).unwrap().len()); + let reader = SegmentReader::open(&sealed_path).unwrap(); + let (h, body) = reader.read_record_at(offsets[2], MAX_RECORD_LEN).unwrap(); + assert_eq!(h.ordinal, 2); + assert_eq!(body, b"third"); + } + + #[test] + fn segment_ordinals_never_reused() { + let dir = tempfile::tempdir().unwrap(); + let _seg = ActiveSegment::create(dir.path(), 1).unwrap(); + assert!(ActiveSegment::create(dir.path(), 1).is_err()); + } + + #[test] + fn scan_headers_visits_all_records() { + let dir = tempfile::tempdir().unwrap(); + let (seg, offsets) = fill_segment(dir.path(), 1, &[b"a", b"bb", b"ccc", b"dddd"]); + let committed = seg.len(); + let (sealed, _) = seg.seal_in_place().unwrap(); + + let reader = SegmentReader::open(&sealed).unwrap(); + let mut seen = Vec::new(); + scan_headers(&reader, 0, committed, MAX_RECORD_LEN, |off, h| { + seen.push((off, h.ordinal, h.body_len())); + true + }) + .unwrap(); + assert_eq!(seen.len(), 4); + for (i, (off, ordinal, body_len)) in seen.iter().enumerate() { + assert_eq!(*off, offsets[i]); + assert_eq!(*ordinal, i as u32); + assert_eq!(*body_len, (i + 1) as u32); + } + + // Scan from a mid-segment cursor position. + let mut seen = Vec::new(); + scan_headers(&reader, offsets[2], committed, MAX_RECORD_LEN, |off, _| { + seen.push(off); + true + }) + .unwrap(); + assert_eq!(seen, vec![offsets[2], offsets[3]]); + } + + #[test] + fn clean_tail_validates_without_truncation() { + let dir = tempfile::tempdir().unwrap(); + let (seg, _) = fill_segment(dir.path(), 1, &[b"a", b"b"]); + let path = seg.path().to_path_buf(); + let len = seg.len(); + drop(seg); + + let v = validate_active_tail(&path, MAX_RECORD_LEN).unwrap(); + assert_eq!( + v, + TailValidation { + committed_len: len, + records: 2, + next_ordinal: 2, + truncated_bytes: 0, + } + ); + } + + #[test] + fn partial_final_record_is_truncated() { + let dir = tempfile::tempdir().unwrap(); + let (seg, offsets) = fill_segment(dir.path(), 1, &[b"aaaa", b"bbbb", b"cccc"]); + let path = seg.path().to_path_buf(); + drop(seg); + let full_len = std::fs::metadata(&path).unwrap().len(); + + // Cut the file mid-way through the last record. + let cut = offsets[2] + (full_len - offsets[2]) / 2; + let f = OpenOptions::new().write(true).open(&path).unwrap(); + f.set_len(cut).unwrap(); + + let v = validate_active_tail(&path, MAX_RECORD_LEN).unwrap(); + assert_eq!(v.records, 2); + assert_eq!(v.committed_len, offsets[2]); + assert_eq!(v.truncated_bytes, cut - offsets[2]); + assert_eq!(std::fs::metadata(&path).unwrap().len(), offsets[2]); + + // The segment must be appendable again with the right ordinal. + let mut seg = ActiveSegment::recover(path.clone(), 1, &v).unwrap(); + assert_eq!(seg.next_ordinal(), 2); + let encoded = record(2, b"replacement"); + let off = seg.append(&encoded).unwrap(); + assert_eq!(off, offsets[2]); + let v2 = validate_active_tail(&path, MAX_RECORD_LEN).unwrap(); + assert_eq!(v2.records, 3); + assert_eq!(v2.truncated_bytes, 0); + } + + #[test] + fn corrupt_tail_body_is_truncated() { + let dir = tempfile::tempdir().unwrap(); + let (seg, offsets) = fill_segment(dir.path(), 1, &[b"aaaa", b"bbbbbbbb"]); + let path = seg.path().to_path_buf(); + let len = seg.len(); + drop(seg); + + // Flip a byte inside the final record's body. + let f = OpenOptions::new().read(true).write(true).open(&path).unwrap(); + let mut b = [0u8; 1]; + f.read_exact_at(&mut b, len - 2).unwrap(); + f.write_all_at(&[b[0] ^ 0xff], len - 2).unwrap(); + + let v = validate_active_tail(&path, MAX_RECORD_LEN).unwrap(); + assert_eq!(v.records, 1); + assert_eq!(v.committed_len, offsets[1]); + } + + #[test] + fn sealed_segment_corruption_is_an_error_not_a_skip() { + let dir = tempfile::tempdir().unwrap(); + let (seg, offsets) = fill_segment(dir.path(), 1, &[b"aaaa", b"bbbb", b"cccc"]); + let committed = seg.len(); + let (sealed, _) = seg.seal_in_place().unwrap(); + + // Corrupt the middle record's header region. + let f = OpenOptions::new().write(true).open(&sealed).unwrap(); + f.write_all_at(&[0xff; 8], offsets[1] + 24).unwrap(); + + let reader = SegmentReader::open(&sealed).unwrap(); + let err = scan_headers(&reader, 0, committed, MAX_RECORD_LEN, |_, _| true).unwrap_err(); + match err { + QueueError::CorruptSealedSegment { offset, .. } => assert_eq!(offset, offsets[1]), + other => panic!("expected CorruptSealedSegment, got {other}"), + } + } + + #[test] + fn ordinal_gap_truncates_active_tail() { + let dir = tempfile::tempdir().unwrap(); + let mut seg = ActiveSegment::create(dir.path(), 1).unwrap(); + seg.append(&record(0, b"a")).unwrap(); + let gap_offset = seg.append(&record(5, b"skipped ordinal")).unwrap(); + let path = seg.path().to_path_buf(); + drop(seg); + + let v = validate_active_tail(&path, MAX_RECORD_LEN).unwrap(); + assert_eq!(v.records, 1); + assert_eq!(v.committed_len, gap_offset); + assert_eq!(v.next_ordinal, 1); + } + + #[test] + fn empty_active_segment_validates() { + let dir = tempfile::tempdir().unwrap(); + let seg = ActiveSegment::create(dir.path(), 1).unwrap(); + let path = seg.path().to_path_buf(); + drop(seg); + let v = validate_active_tail(&path, MAX_RECORD_LEN).unwrap(); + assert_eq!(v.committed_len, 0); + assert_eq!(v.records, 0); + assert_eq!(v.next_ordinal, 0); + } + + #[test] + fn garbage_prefix_truncates_to_empty() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join(active_file_name(1)); + std::fs::write(&path, b"this is not a record at all, just garbage bytes!!").unwrap(); + let v = validate_active_tail(&path, MAX_RECORD_LEN).unwrap(); + assert_eq!(v.committed_len, 0); + assert!(v.truncated_bytes > 0); + assert_eq!(std::fs::metadata(&path).unwrap().len(), 0); + } +} diff --git a/smtp-server/src/logqueue/shard.rs b/smtp-server/src/logqueue/shard.rs new file mode 100644 index 0000000..297cde0 --- /dev/null +++ b/smtp-server/src/logqueue/shard.rs @@ -0,0 +1,165 @@ +//! Shard directories: each append writer exclusively owns one. + +use std::path::{Path, PathBuf}; + +use super::segment::{self, SegmentKind}; +use super::QueueError; + +pub fn shard_dir_name(shard: u16) -> String { + format!("shard-{shard:04}") +} + +/// A shard's directory on disk. +#[derive(Debug, Clone)] +pub struct ShardDir { + path: PathBuf, + shard: u16, +} + +/// The segment files present in a shard directory. +#[derive(Debug, Default)] +pub struct ShardSegments { + /// Sealed segments, sorted by segment ordinal. + pub sealed: Vec<(u64, PathBuf)>, + /// The active segment, if one exists. More than one is a layout error. + pub active: Option<(u64, PathBuf)>, + /// The ordinal the next created segment must use (max seen + 1). + pub next_segment: u64, +} + +impl ShardDir { + pub fn open_or_create(spool_root: &Path, shard: u16) -> Result { + let path = spool_root.join(shard_dir_name(shard)); + std::fs::create_dir_all(&path).map_err(|e| QueueError::io(&path, e))?; + Ok(Self { path, shard }) + } + + pub fn path(&self) -> &Path { + &self.path + } + + pub fn shard(&self) -> u16 { + self.shard + } + + /// Enumerate segment files. This is a small, bounded listing (segments + /// are proportional to live data, not message history) and runs only at + /// startup and GC boundaries, never per message. + pub fn list_segments(&self) -> Result { + let mut out = ShardSegments { + next_segment: 1, + ..Default::default() + }; + let entries = std::fs::read_dir(&self.path).map_err(|e| QueueError::io(&self.path, e))?; + for entry in entries { + let entry = entry.map_err(|e| QueueError::io(&self.path, e))?; + let name = entry.file_name(); + let Some(name) = name.to_str() else { + tracing::warn!(shard = self.shard, ?name, "ignoring non-UTF-8 file in shard dir"); + continue; + }; + let Some((ordinal, kind)) = segment::parse_file_name(name) else { + continue; // journal, checkpoint, temp files, … + }; + out.next_segment = out.next_segment.max(ordinal + 1); + match kind { + SegmentKind::Sealed => out.sealed.push((ordinal, entry.path())), + SegmentKind::Active => { + if let Some((existing, _)) = out.active { + return Err(QueueError::Layout(format!( + "shard {} has two active segments ({} and {}); \ + rotation must seal before creating the next", + self.shard, existing, ordinal + ))); + } + out.active = Some((ordinal, entry.path())); + } + } + } + out.sealed.sort_unstable_by_key(|(ordinal, _)| *ordinal); + if let Some(w) = out.sealed.windows(2).find(|w| w[0].0 == w[1].0) { + return Err(QueueError::Layout(format!( + "shard {} has duplicate segment ordinal {}", + self.shard, w[0].0 + ))); + } + if let Some((active, _)) = out.active { + if out.sealed.iter().any(|(s, _)| *s == active) { + return Err(QueueError::Layout(format!( + "shard {} segment {} exists as both .open and .log", + self.shard, active + ))); + } + } + Ok(out) + } + + /// Whether the shard holds any segment data at all. + pub fn is_empty(&self) -> Result { + let segs = self.list_segments()?; + Ok(segs.sealed.is_empty() && segs.active.is_none()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::logqueue::segment::{active_file_name, sealed_file_name}; + + #[test] + fn empty_shard() { + let dir = tempfile::tempdir().unwrap(); + let shard = ShardDir::open_or_create(dir.path(), 0).unwrap(); + let segs = shard.list_segments().unwrap(); + assert!(segs.sealed.is_empty()); + assert!(segs.active.is_none()); + assert_eq!(segs.next_segment, 1); + assert!(shard.is_empty().unwrap()); + } + + #[test] + fn lists_and_sorts_segments() { + let dir = tempfile::tempdir().unwrap(); + let shard = ShardDir::open_or_create(dir.path(), 3).unwrap(); + for seg in [3u64, 1, 2] { + std::fs::write(shard.path().join(sealed_file_name(seg)), b"").unwrap(); + } + std::fs::write(shard.path().join(active_file_name(4)), b"").unwrap(); + // Non-segment files are ignored. + std::fs::write(shard.path().join("state-journal.log"), b"").unwrap(); + std::fs::write(shard.path().join("checkpoint"), b"").unwrap(); + + let segs = shard.list_segments().unwrap(); + assert_eq!( + segs.sealed.iter().map(|(s, _)| *s).collect::>(), + vec![1, 2, 3] + ); + assert_eq!(segs.active.as_ref().map(|(s, _)| *s), Some(4)); + assert_eq!(segs.next_segment, 5); + assert!(!shard.is_empty().unwrap()); + } + + #[test] + fn two_active_segments_is_an_error() { + let dir = tempfile::tempdir().unwrap(); + let shard = ShardDir::open_or_create(dir.path(), 0).unwrap(); + std::fs::write(shard.path().join(active_file_name(1)), b"").unwrap(); + std::fs::write(shard.path().join(active_file_name(2)), b"").unwrap(); + assert!(matches!( + shard.list_segments(), + Err(QueueError::Layout(_)) + )); + } + + #[test] + fn same_ordinal_open_and_log_is_an_error() { + let dir = tempfile::tempdir().unwrap(); + let shard = ShardDir::open_or_create(dir.path(), 0).unwrap(); + std::fs::write(shard.path().join(active_file_name(1)), b"").unwrap(); + std::fs::write(shard.path().join(sealed_file_name(1)), b"").unwrap(); + assert!(matches!( + shard.list_segments(), + Err(QueueError::Layout(_)) + )); + } +} diff --git a/smtp-server/src/logqueue/spool.rs b/smtp-server/src/logqueue/spool.rs new file mode 100644 index 0000000..9e3185d --- /dev/null +++ b/smtp-server/src/logqueue/spool.rs @@ -0,0 +1,258 @@ +//! Spool root: format version, exclusive process lock, shard layout. + +use std::fs::{File, OpenOptions, TryLockError}; +use std::path::{Path, PathBuf}; + +use super::record::FIXED_HEADER_LEN; +use super::shard::ShardDir; +use super::{QueueError, FORMAT_VERSION}; + +const VERSION_FILE: &str = "format-version"; +const LOCK_FILE: &str = ".lock"; + +/// Fixed allowance for record overhead (header + envelope) on top of the +/// message body when validating segment sizing. Generous relative to real +/// envelopes; a record whose envelope exceeds it is rejected at append time +/// by the writer's fits-in-one-segment check, so the invariant that a record +/// never spans segments holds either way. +pub const ENVELOPE_ALLOWANCE: u64 = 1024 * 1024; + +/// An opened spool root. Holds the exclusive OS-level lock for its lifetime: +/// recovery, tail truncation, migration, and append writers must all sit +/// behind this. Independent Hedwig processes must use distinct spool roots. +pub struct Spool { + shards: Vec, + /// Lock is released when the file handle drops. + _lock: File, +} + +impl Spool { + /// Open (creating if necessary) a spool root with `shard_count` shards. + /// + /// Fails if another process holds the spool lock, if the on-disk format + /// version is unsupported, or if shrinking `shard_count` would orphan + /// shard directories that still contain segment data (changing the + /// writer count requires an empty queue). + pub fn open(root: impl Into, shard_count: u16) -> Result { + assert!(shard_count > 0, "shard_count must be at least 1"); + let root = root.into(); + std::fs::create_dir_all(&root).map_err(|e| QueueError::io(&root, e))?; + + let lock = Self::acquire_lock(&root)?; + Self::check_format_version(&root)?; + + // Changing the shard count (in either direction) requires an empty + // queue. Count the contiguous shard directories already present; a + // mismatch is only legal if every one of them is empty. + let mut existing = 0u16; + while root.join(super::shard::shard_dir_name(existing)).is_dir() { + existing += 1; + } + if existing != 0 && existing != shard_count { + for shard in 0..existing { + let dir = ShardDir::open_or_create(&root, shard)?; + if !dir.is_empty()? { + return Err(QueueError::Layout(format!( + "spool has {existing} shards but {shard_count} are configured, and \ + shard {shard} still holds data; changing append_writers requires \ + an empty queue", + ))); + } + } + } + + let mut shards = Vec::with_capacity(shard_count as usize); + for shard in 0..shard_count { + shards.push(ShardDir::open_or_create(&root, shard)?); + } + + Ok(Self { + shards, + _lock: lock, + }) + } + + fn acquire_lock(root: &Path) -> Result { + let path = root.join(LOCK_FILE); + let file = OpenOptions::new() + .create(true) + .truncate(false) + .write(true) + .open(&path) + .map_err(|e| QueueError::io(&path, e))?; + match file.try_lock() { + Ok(()) => Ok(file), + Err(TryLockError::WouldBlock) => Err(QueueError::SpoolLocked { + path: path.display().to_string(), + }), + Err(TryLockError::Error(e)) => Err(QueueError::io(&path, e)), + } + } + + fn check_format_version(root: &Path) -> Result<(), QueueError> { + let path = root.join(VERSION_FILE); + match std::fs::read_to_string(&path) { + Ok(contents) => { + let found: u16 = contents.trim().parse().map_err(|_| { + QueueError::Layout(format!( + "{} does not contain a version number: {contents:?}", + path.display() + )) + })?; + if found != FORMAT_VERSION { + return Err(QueueError::UnsupportedVersion { + found, + supported: FORMAT_VERSION, + }); + } + Ok(()) + } + Err(e) if e.kind() == std::io::ErrorKind::NotFound => { + std::fs::write(&path, format!("{FORMAT_VERSION}\n")) + .map_err(|e| QueueError::io(&path, e)) + } + Err(e) => Err(QueueError::io(&path, e)), + } + } + + pub fn shard_count(&self) -> u16 { + self.shards.len() as u16 + } + + pub fn shard(&self, shard: u16) -> &ShardDir { + &self.shards[shard as usize] + } + + pub fn shards(&self) -> &[ShardDir] { + &self.shards + } +} + +/// Free bytes available to unprivileged writes on the filesystem holding +/// `path`. Drives the disk-reserve acceptance check (PLAN §20). +pub fn disk_free_bytes(path: &Path) -> std::io::Result { + use std::os::unix::ffi::OsStrExt; + let c = std::ffi::CString::new(path.as_os_str().as_bytes()).map_err(|_| { + std::io::Error::new(std::io::ErrorKind::InvalidInput, "path contains NUL") + })?; + let mut vfs: libc::statvfs = unsafe { std::mem::zeroed() }; + if unsafe { libc::statvfs(c.as_ptr(), &mut vfs) } != 0 { + return Err(std::io::Error::last_os_error()); + } + Ok(vfs.f_bavail as u64 * vfs.f_frsize as u64) +} + +/// Reject configurations where a maximum-size message could not fit in one +/// segment. Records never span segments, so the segment target must cover +/// the largest possible record. +pub fn check_segment_sizing( + segment_target_bytes: u64, + max_message_bytes: u64, +) -> Result<(), QueueError> { + let worst_case = max_message_bytes + ENVELOPE_ALLOWANCE + FIXED_HEADER_LEN as u64; + if segment_target_bytes < worst_case { + return Err(QueueError::SegmentTooSmall { + segment_bytes: segment_target_bytes, + max_message_bytes, + }); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::logqueue::segment::sealed_file_name; + + #[test] + fn open_creates_layout_and_reopens() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().join("spool"); + { + let spool = Spool::open(&root, 2).unwrap(); + assert_eq!(spool.shard_count(), 2); + assert!(root.join("shard-0000").is_dir()); + assert!(root.join("shard-0001").is_dir()); + assert_eq!( + std::fs::read_to_string(root.join(VERSION_FILE)).unwrap().trim(), + "1" + ); + } + // Lock released on drop; reopening works. + Spool::open(&root, 2).unwrap(); + } + + #[test] + fn second_open_is_rejected_while_locked() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().join("spool"); + let _spool = Spool::open(&root, 1).unwrap(); + match Spool::open(&root, 1).err() { + Some(QueueError::SpoolLocked { .. }) => {} + other => panic!("expected SpoolLocked, got {other:?}"), + } + } + + #[test] + fn unsupported_version_is_rejected() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().join("spool"); + std::fs::create_dir_all(&root).unwrap(); + std::fs::write(root.join(VERSION_FILE), "99\n").unwrap(); + match Spool::open(&root, 1).err() { + Some(QueueError::UnsupportedVersion { found: 99, .. }) => {} + other => panic!("expected UnsupportedVersion, got {other:?}"), + } + } + + #[test] + fn garbage_version_file_is_rejected() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().join("spool"); + std::fs::create_dir_all(&root).unwrap(); + std::fs::write(root.join(VERSION_FILE), "not a number").unwrap(); + assert!(matches!(Spool::open(&root, 1), Err(QueueError::Layout(_)))); + } + + #[test] + fn shrinking_shard_count_requires_empty_orphans() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().join("spool"); + { + let spool = Spool::open(&root, 4).unwrap(); + std::fs::write( + spool.shard(3).path().join(sealed_file_name(1)), + b"", + ) + .unwrap(); + } + // Shard 3 still holds a segment: shrinking to 2 must fail. + assert!(matches!(Spool::open(&root, 2), Err(QueueError::Layout(_)))); + // Removing the data makes the shrink legal. + std::fs::remove_file(root.join("shard-0003").join(sealed_file_name(1))).unwrap(); + Spool::open(&root, 2).unwrap(); + } + + #[test] + fn growing_shard_count_requires_empty_queue() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().join("spool"); + { + let spool = Spool::open(&root, 1).unwrap(); + std::fs::write(spool.shard(0).path().join(sealed_file_name(1)), b"").unwrap(); + } + assert!(matches!(Spool::open(&root, 4), Err(QueueError::Layout(_)))); + std::fs::remove_file(root.join("shard-0000").join(sealed_file_name(1))).unwrap(); + let spool = Spool::open(&root, 4).unwrap(); + assert_eq!(spool.shard_count(), 4); + } + + #[test] + fn segment_sizing_invariant() { + assert!(check_segment_sizing(64 * 1024 * 1024, 25 * 1024 * 1024).is_ok()); + assert!(matches!( + check_segment_sizing(8 * 1024 * 1024, 25 * 1024 * 1024), + Err(QueueError::SegmentTooSmall { .. }) + )); + } +} diff --git a/smtp-server/src/logqueue/state.rs b/smtp-server/src/logqueue/state.rs new file mode 100644 index 0000000..0de91bd --- /dev/null +++ b/smtp-server/src/logqueue/state.rs @@ -0,0 +1,1514 @@ +//! Per-shard persistent delivery state: append-only journal + checkpoints. +//! +//! Payload records are immutable; everything that changes after acceptance +//! (defer, deliver, bounce) is an entry in the shard's state journal. A +//! payload record implies `Ready` unless superseded by later state, so no +//! enqueue entry exists. +//! +//! Journal files are `journal-NNNNNNNNNNNN.log`; a new one starts at every +//! checkpoint and files fully covered by the checkpoint are deleted. An LSN +//! is (journal ordinal, byte offset). Journal writes use page-cache +//! durability like payload writes; the checkpoint is the destructive +//! boundary and is fsynced before any journal history is removed. + +use std::collections::{HashMap, HashSet}; +use std::fs::{File, OpenOptions}; +use std::io::Write; +use std::os::unix::fs::FileExt; +use std::path::{Path, PathBuf}; + +use super::{JobLocation, MessageId, QueueError}; + +const JOURNAL_PREFIX: &str = "journal-"; +const JOURNAL_EXT: &str = "log"; +const CHECKPOINT_FILE: &str = "checkpoint"; +const CHECKPOINT_TMP: &str = "checkpoint.tmp"; + +const CHECKPOINT_MAGIC: [u8; 4] = *b"HWCP"; +const CHECKPOINT_VERSION: u16 = 1; + +/// Framing overhead per journal entry: length + crc. +const ENTRY_FRAME: usize = 8; +/// Sanity cap on one journal entry (a huge recipient list stays far below). +const MAX_ENTRY_LEN: u32 = 16 * 1024 * 1024; + +/// Log sequence number: position in the shard's journal stream. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +pub struct Lsn { + pub journal: u64, + pub offset: u64, +} + +/// A persisted delivery-state transition. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum StateEntry { + /// A real failed attempt: attempts incremented, remaining recipients + /// persisted so a retry only re-sends to those. + Deferred { + id: MessageId, + location: JobLocation, + attempts: u32, + next_attempt_ms: i64, + remaining_recipients: Vec, + last_error: String, + }, + Delivered { + id: MessageId, + location: JobLocation, + timestamp_ms: i64, + }, + Bounced { + id: MessageId, + location: JobLocation, + timestamp_ms: i64, + reason: String, + }, + /// Compaction copied this record from `old` to `new` (higher relocation + /// generation). If the message is live, `new` becomes its location and + /// the old copy is garbage; if it raced to terminal first, the new copy + /// is garbage instead. + Relocated { + id: MessageId, + old: JobLocation, + new: JobLocation, + }, +} + +// --------------------------------------------------------------------------- +// Minimal binary codec shared by journal entries and checkpoints. + +struct Enc(Vec); + +impl Enc { + fn new() -> Self { + Enc(Vec::new()) + } + fn u8(&mut self, v: u8) { + self.0.push(v); + } + fn u16(&mut self, v: u16) { + self.0.extend_from_slice(&v.to_le_bytes()); + } + fn u32(&mut self, v: u32) { + self.0.extend_from_slice(&v.to_le_bytes()); + } + fn u64(&mut self, v: u64) { + self.0.extend_from_slice(&v.to_le_bytes()); + } + fn i64(&mut self, v: i64) { + self.0.extend_from_slice(&v.to_le_bytes()); + } + fn id(&mut self, v: &MessageId) { + self.0.extend_from_slice(&v.0); + } + fn str(&mut self, v: &str) { + self.u32(v.len() as u32); + self.0.extend_from_slice(v.as_bytes()); + } + fn location(&mut self, l: &JobLocation) { + self.u16(l.shard); + self.u64(l.segment); + self.u64(l.offset); + self.u32(l.length); + self.u32(l.ordinal); + self.u32(l.generation); + } +} + +struct Dec<'a> { + buf: &'a [u8], + at: usize, +} + +impl<'a> Dec<'a> { + fn new(buf: &'a [u8]) -> Self { + Dec { buf, at: 0 } + } + fn take(&mut self, n: usize) -> Result<&'a [u8], QueueError> { + if self.at + n > self.buf.len() { + return Err(QueueError::InvalidRecord( + "state entry truncated mid-field".into(), + )); + } + let s = &self.buf[self.at..self.at + n]; + self.at += n; + Ok(s) + } + fn u8(&mut self) -> Result { + Ok(self.take(1)?[0]) + } + fn u16(&mut self) -> Result { + Ok(u16::from_le_bytes(self.take(2)?.try_into().unwrap())) + } + fn u32(&mut self) -> Result { + Ok(u32::from_le_bytes(self.take(4)?.try_into().unwrap())) + } + fn u64(&mut self) -> Result { + Ok(u64::from_le_bytes(self.take(8)?.try_into().unwrap())) + } + fn i64(&mut self) -> Result { + Ok(i64::from_le_bytes(self.take(8)?.try_into().unwrap())) + } + fn id(&mut self) -> Result { + Ok(MessageId(self.take(16)?.try_into().unwrap())) + } + fn str(&mut self) -> Result { + let len = self.u32()? as usize; + let bytes = self.take(len)?; + String::from_utf8(bytes.to_vec()) + .map_err(|_| QueueError::InvalidRecord("state entry string is not UTF-8".into())) + } + fn location(&mut self) -> Result { + Ok(JobLocation { + shard: self.u16()?, + segment: self.u64()?, + offset: self.u64()?, + length: self.u32()?, + ordinal: self.u32()?, + generation: self.u32()?, + }) + } + fn finished(&self) -> bool { + self.at == self.buf.len() + } +} + +const KIND_DEFERRED: u8 = 1; +const KIND_DELIVERED: u8 = 2; +const KIND_BOUNCED: u8 = 3; +const KIND_RELOCATED: u8 = 4; + +fn encode_entry(entry: &StateEntry) -> Vec { + let mut e = Enc::new(); + match entry { + StateEntry::Deferred { + id, + location, + attempts, + next_attempt_ms, + remaining_recipients, + last_error, + } => { + e.u8(KIND_DEFERRED); + e.id(id); + e.location(location); + e.u32(*attempts); + e.i64(*next_attempt_ms); + e.u32(remaining_recipients.len() as u32); + for r in remaining_recipients { + e.str(r); + } + e.str(last_error); + } + StateEntry::Delivered { + id, + location, + timestamp_ms, + } => { + e.u8(KIND_DELIVERED); + e.id(id); + e.location(location); + e.i64(*timestamp_ms); + } + StateEntry::Bounced { + id, + location, + timestamp_ms, + reason, + } => { + e.u8(KIND_BOUNCED); + e.id(id); + e.location(location); + e.i64(*timestamp_ms); + e.str(reason); + } + StateEntry::Relocated { id, old, new } => { + e.u8(KIND_RELOCATED); + e.id(id); + e.location(old); + e.location(new); + } + } + e.0 +} + +fn decode_entry(buf: &[u8]) -> Result { + let mut d = Dec::new(buf); + let entry = match d.u8()? { + KIND_DEFERRED => { + let id = d.id()?; + let location = d.location()?; + let attempts = d.u32()?; + let next_attempt_ms = d.i64()?; + let n = d.u32()? as usize; + let mut remaining_recipients = Vec::with_capacity(n.min(1024)); + for _ in 0..n { + remaining_recipients.push(d.str()?); + } + StateEntry::Deferred { + id, + location, + attempts, + next_attempt_ms, + remaining_recipients, + last_error: d.str()?, + } + } + KIND_DELIVERED => StateEntry::Delivered { + id: d.id()?, + location: d.location()?, + timestamp_ms: d.i64()?, + }, + KIND_BOUNCED => StateEntry::Bounced { + id: d.id()?, + location: d.location()?, + timestamp_ms: d.i64()?, + reason: d.str()?, + }, + KIND_RELOCATED => StateEntry::Relocated { + id: d.id()?, + old: d.location()?, + new: d.location()?, + }, + k => { + return Err(QueueError::InvalidRecord(format!( + "unknown state entry kind {k}" + ))) + } + }; + if !d.finished() { + return Err(QueueError::InvalidRecord( + "trailing bytes after state entry".into(), + )); + } + Ok(entry) +} + +// --------------------------------------------------------------------------- +// Journal files. + +fn journal_file_name(ordinal: u64) -> String { + format!("{JOURNAL_PREFIX}{ordinal:012}.{JOURNAL_EXT}") +} + +fn parse_journal_name(name: &str) -> Option { + let digits = name + .strip_prefix(JOURNAL_PREFIX)? + .strip_suffix(&format!(".{JOURNAL_EXT}"))?; + if digits.len() != 12 || !digits.bytes().all(|b| b.is_ascii_digit()) { + return None; + } + digits.parse().ok() +} + +struct JournalWriter { + file: File, + path: PathBuf, + ordinal: u64, + len: u64, +} + +impl JournalWriter { + fn create(dir: &Path, ordinal: u64) -> Result { + let path = dir.join(journal_file_name(ordinal)); + let file = OpenOptions::new() + .create_new(true) + .append(true) + .open(&path) + .map_err(|e| QueueError::io(&path, e))?; + Ok(Self { + file, + path, + ordinal, + len: 0, + }) + } + + fn reopen(dir: &Path, ordinal: u64, len: u64) -> Result { + let path = dir.join(journal_file_name(ordinal)); + let file = OpenOptions::new() + .append(true) + .open(&path) + .map_err(|e| QueueError::io(&path, e))?; + Ok(Self { + file, + path, + ordinal, + len, + }) + } + + /// Append one entry; returns the LSN one past it (replay resumes there). + fn append(&mut self, payload: &[u8]) -> Result { + let mut framed = Vec::with_capacity(ENTRY_FRAME + payload.len()); + framed.extend_from_slice(&(payload.len() as u32).to_le_bytes()); + framed.extend_from_slice(&crc32fast::hash(payload).to_le_bytes()); + framed.extend_from_slice(payload); + if let Err(e) = self.file.write_all(&framed) { + let _ = self.file.set_len(self.len); + return Err(QueueError::io(&self.path, e)); + } + self.len += framed.len() as u64; + Ok(Lsn { + journal: self.ordinal, + offset: self.len, + }) + } + + fn fsync(&self) -> Result<(), QueueError> { + self.file + .sync_data() + .map_err(|e| QueueError::io(&self.path, e)) + } +} + +/// How [`replay_journal`] should react to an invalid entry at the position +/// it stops on. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum TornTail { + /// Any invalid entry is a hard error: used for older journals fully + /// covered by a checkpoint, which must never contain garbage. + HardError, + /// Cut the file back to the last valid boundary. The write path's policy + /// for the active journal after a crash. + Truncate, + /// Stop replay at the last valid boundary without touching the file. + /// Used by read-only inspection of a spool a live writer may still own. + StopReadOnly, +} + +/// Read entries from one journal file starting at `offset`. +/// +/// `tail` selects the active-journal policy for an invalid entry: hard +/// error, truncate-in-place, or (read-only callers) simply stop replay +/// without mutating the file. +fn replay_journal( + path: &Path, + start: u64, + tail: TornTail, + mut apply: impl FnMut(StateEntry), +) -> Result { + let file = File::open(path).map_err(|e| QueueError::io(path, e))?; + let len = file.metadata().map_err(|e| QueueError::io(path, e))?.len(); + let mut at = start; + let mut frame = [0u8; ENTRY_FRAME]; + + let invalid = loop { + if at == len { + break None; + } + if len - at < ENTRY_FRAME as u64 { + break Some(format!("{} trailing bytes", len - at)); + } + file.read_exact_at(&mut frame, at) + .map_err(|e| QueueError::io(path, e))?; + let entry_len = u32::from_le_bytes(frame[0..4].try_into().unwrap()); + let crc = u32::from_le_bytes(frame[4..8].try_into().unwrap()); + if entry_len > MAX_ENTRY_LEN { + break Some(format!("implausible entry length {entry_len}")); + } + if len - at - (ENTRY_FRAME as u64) < entry_len as u64 { + break Some("entry overruns file".into()); + } + let mut payload = vec![0u8; entry_len as usize]; + file.read_exact_at(&mut payload, at + ENTRY_FRAME as u64) + .map_err(|e| QueueError::io(path, e))?; + if crc32fast::hash(&payload) != crc { + break Some("entry checksum mismatch".into()); + } + match decode_entry(&payload) { + Ok(entry) => apply(entry), + Err(e) => break Some(format!("undecodable entry: {e}")), + } + at += ENTRY_FRAME as u64 + entry_len as u64; + }; + + if let Some(reason) = invalid { + match tail { + TornTail::HardError => { + return Err(QueueError::CorruptSealedSegment { + path: path.display().to_string(), + offset: at, + reason, + }); + } + TornTail::Truncate => { + let f = OpenOptions::new() + .write(true) + .open(path) + .map_err(|e| QueueError::io(path, e))?; + f.set_len(at).map_err(|e| QueueError::io(path, e))?; + tracing::warn!( + path = %path.display(), + valid_len = at, + reason, + "truncated torn tail of state journal" + ); + } + TornTail::StopReadOnly => { + tracing::debug!( + path = %path.display(), + valid_len = at, + reason, + "read-only replay stopped at torn tail" + ); + } + } + } + Ok(at) +} + +// --------------------------------------------------------------------------- +// Checkpoint. + +/// A message known to be live and already discovered (its location would +/// not be re-found by a cursor scan). +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ReadyJob { + pub id: MessageId, + pub location: JobLocation, + pub attempts: u32, + pub enqueue_ms: i64, + /// Recipients that have not yet accepted the message, when a partial + /// delivery happened before this snapshot. Empty means the full + /// envelope from the payload record. Without this, a deferred message + /// that became due and was then checkpointed as ready would re-send to + /// recipients that already accepted it. + pub remaining_recipients: Vec, +} + +/// A deferred message with everything needed to retry it. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct DeferredJob { + pub id: MessageId, + pub location: JobLocation, + pub attempts: u32, + pub next_attempt_ms: i64, + pub remaining_recipients: Vec, + pub last_error: String, +} + +/// Per-segment reclamation accounting. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct SegmentStats { + pub total_records: u32, + pub total_bytes: u64, + pub dead_records: u32, + pub dead_bytes: u64, +} + +/// A self-sufficient snapshot of one shard's scheduling state (PLAN §16). +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct Checkpoint { + /// Discovery cursor: records at or past this position have not been + /// discovered. `None` means nothing was ever discovered (scan from the + /// start of the append chain). + pub cursor: Option<(u64, u64)>, + /// Discovered live messages (ready or in flight at snapshot time). + pub ready: Vec, + /// Deferred messages with attempts and due times. + pub deferred: Vec, + /// Terminal tombstones per still-present segment. + pub tombstones: Vec<(u64, Vec)>, + /// Reclamation stats per segment. + pub segment_stats: Vec<(u64, SegmentStats)>, +} + +fn encode_checkpoint(cp: &Checkpoint, replay_from: Lsn) -> Vec { + let mut e = Enc::new(); + e.u64(replay_from.journal); + e.u64(replay_from.offset); + match cp.cursor { + None => e.u8(0), + Some((seg, off)) => { + e.u8(1); + e.u64(seg); + e.u64(off); + } + } + e.u64(cp.ready.len() as u64); + for r in &cp.ready { + e.id(&r.id); + e.location(&r.location); + e.u32(r.attempts); + e.i64(r.enqueue_ms); + e.u32(r.remaining_recipients.len() as u32); + for rcpt in &r.remaining_recipients { + e.str(rcpt); + } + } + e.u64(cp.deferred.len() as u64); + for d in &cp.deferred { + e.id(&d.id); + e.location(&d.location); + e.u32(d.attempts); + e.i64(d.next_attempt_ms); + e.u32(d.remaining_recipients.len() as u32); + for r in &d.remaining_recipients { + e.str(r); + } + e.str(&d.last_error); + } + e.u64(cp.tombstones.len() as u64); + for (segment, ids) in &cp.tombstones { + e.u64(*segment); + e.u64(ids.len() as u64); + for id in ids { + e.id(id); + } + } + e.u64(cp.segment_stats.len() as u64); + for (segment, s) in &cp.segment_stats { + e.u64(*segment); + e.u32(s.total_records); + e.u64(s.total_bytes); + e.u32(s.dead_records); + e.u64(s.dead_bytes); + } + e.0 +} + +fn decode_checkpoint(buf: &[u8]) -> Result<(Checkpoint, Lsn), QueueError> { + let mut d = Dec::new(buf); + let replay_from = Lsn { + journal: d.u64()?, + offset: d.u64()?, + }; + let cursor = match d.u8()? { + 0 => None, + 1 => Some((d.u64()?, d.u64()?)), + v => { + return Err(QueueError::InvalidRecord(format!( + "bad cursor discriminant {v}" + ))) + } + }; + let mut cp = Checkpoint { + cursor, + ..Default::default() + }; + for _ in 0..d.u64()? { + let id = d.id()?; + let location = d.location()?; + let attempts = d.u32()?; + let enqueue_ms = d.i64()?; + let n = d.u32()? as usize; + let mut remaining_recipients = Vec::with_capacity(n.min(1024)); + for _ in 0..n { + remaining_recipients.push(d.str()?); + } + cp.ready.push(ReadyJob { + id, + location, + attempts, + enqueue_ms, + remaining_recipients, + }); + } + for _ in 0..d.u64()? { + let id = d.id()?; + let location = d.location()?; + let attempts = d.u32()?; + let next_attempt_ms = d.i64()?; + let n = d.u32()? as usize; + let mut remaining_recipients = Vec::with_capacity(n.min(1024)); + for _ in 0..n { + remaining_recipients.push(d.str()?); + } + cp.deferred.push(DeferredJob { + id, + location, + attempts, + next_attempt_ms, + remaining_recipients, + last_error: d.str()?, + }); + } + for _ in 0..d.u64()? { + let segment = d.u64()?; + let n = d.u64()? as usize; + let mut ids = Vec::with_capacity(n.min(1 << 20)); + for _ in 0..n { + ids.push(d.id()?); + } + cp.tombstones.push((segment, ids)); + } + for _ in 0..d.u64()? { + cp.segment_stats.push(( + d.u64()?, + SegmentStats { + total_records: d.u32()?, + total_bytes: d.u64()?, + dead_records: d.u32()?, + dead_bytes: d.u64()?, + }, + )); + } + if !d.finished() { + return Err(QueueError::InvalidRecord( + "trailing bytes after checkpoint".into(), + )); + } + Ok((cp, replay_from)) +} + +/// A begun-but-not-published checkpoint: the journal has rotated; the +/// snapshot still has to be written and old journals pruned. +#[derive(Debug, Clone, Copy)] +pub struct PendingCheckpoint { + pub replay_from: Lsn, + covered: u64, +} + +pub fn write_checkpoint_file(dir: &Path, cp: &Checkpoint, replay_from: Lsn) -> Result<(), QueueError> { + let payload = encode_checkpoint(cp, replay_from); + let mut buf = Vec::with_capacity(payload.len() + 16); + buf.extend_from_slice(&CHECKPOINT_MAGIC); + buf.extend_from_slice(&CHECKPOINT_VERSION.to_le_bytes()); + buf.extend_from_slice(&0u16.to_le_bytes()); // reserved + buf.extend_from_slice(&(payload.len() as u64).to_le_bytes()); + buf.extend_from_slice(&crc32fast::hash(&payload).to_le_bytes()); + buf.extend_from_slice(&payload); + + // Destructive-boundary ordering (PLAN §5.3/§16): write, fsync, rename, + // fsync the directory. Only after all of that may journal history die. + let tmp = dir.join(CHECKPOINT_TMP); + let path = dir.join(CHECKPOINT_FILE); + let mut f = File::create(&tmp).map_err(|e| QueueError::io(&tmp, e))?; + f.write_all(&buf).map_err(|e| QueueError::io(&tmp, e))?; + f.sync_data().map_err(|e| QueueError::io(&tmp, e))?; + drop(f); + std::fs::rename(&tmp, &path).map_err(|e| QueueError::io(&path, e))?; + let dirf = File::open(dir).map_err(|e| QueueError::io(dir, e))?; + dirf.sync_data().map_err(|e| QueueError::io(dir, e))?; + Ok(()) +} + +fn load_checkpoint_file(dir: &Path) -> Result, QueueError> { + let path = dir.join(CHECKPOINT_FILE); + let buf = match std::fs::read(&path) { + Ok(b) => b, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(None), + Err(e) => return Err(QueueError::io(&path, e)), + }; + let corrupt = |reason: &str| QueueError::CorruptRecord { + offset: 0, + reason: format!("checkpoint {}: {reason}", path.display()), + }; + if buf.len() < 20 || buf[0..4] != CHECKPOINT_MAGIC { + return Err(corrupt("bad magic or truncated header")); + } + let version = u16::from_le_bytes(buf[4..6].try_into().unwrap()); + if version != CHECKPOINT_VERSION { + return Err(QueueError::UnsupportedVersion { + found: version, + supported: CHECKPOINT_VERSION, + }); + } + let payload_len = u64::from_le_bytes(buf[8..16].try_into().unwrap()) as usize; + let crc = u32::from_le_bytes(buf[16..20].try_into().unwrap()); + if buf.len() != 20 + payload_len { + return Err(corrupt("length mismatch")); + } + let payload = &buf[20..]; + if crc32fast::hash(payload) != crc { + return Err(corrupt("checksum mismatch")); + } + decode_checkpoint(payload).map(Some) +} + +// --------------------------------------------------------------------------- +// Shard state store. + +/// One shard's recovered scheduling state, produced by +/// [`ShardStateStore::recover`]: the checkpoint with all journal entries +/// newer than it applied on top. +#[derive(Debug, Default)] +pub struct RecoveredState { + pub cursor: Option<(u64, u64)>, + pub ready: HashMap, + pub deferred: HashMap, + pub tombstones: HashMap>, + pub segment_stats: HashMap, +} + +impl RecoveredState { + fn from_checkpoint(cp: Checkpoint) -> Self { + Self { + cursor: cp.cursor, + ready: cp.ready.into_iter().map(|r| (r.id, r)).collect(), + deferred: cp.deferred.into_iter().map(|d| (d.id, d)).collect(), + tombstones: cp + .tombstones + .into_iter() + .map(|(seg, ids)| (seg, ids.into_iter().collect())) + .collect(), + segment_stats: cp.segment_stats.into_iter().collect(), + } + } + + /// Apply one journal entry on top of the current state; used both for + /// recovery replay and could be reused by live accounting. + pub fn apply(&mut self, entry: StateEntry) { + match entry { + StateEntry::Deferred { + id, + location, + attempts, + next_attempt_ms, + remaining_recipients, + last_error, + } => { + self.ready.remove(&id); + self.deferred.insert( + id, + DeferredJob { + id, + location, + attempts, + next_attempt_ms, + remaining_recipients, + last_error, + }, + ); + } + StateEntry::Delivered { id, location, .. } + | StateEntry::Bounced { id, location, .. } => { + self.ready.remove(&id); + self.deferred.remove(&id); + self.mark_copy_dead(id, location); + } + StateEntry::Relocated { id, old, new } => { + // Ordered replay: if the message is still live here, the + // relocation won and the old copy is garbage. If a terminal + // entry preceded this one, the terminal copy accounting + // already covered `old` (or an earlier location) and the + // fresh copy at `new` is garbage — a terminal race during + // compaction must not resurrect the message. + let terminal = + self.is_terminal(old.segment, &id) || self.is_terminal(new.segment, &id); + let live_ready = self.ready.get_mut(&id).map(|r| &mut r.location); + let live_deferred = self.deferred.get_mut(&id).map(|d| &mut d.location); + match live_ready.or(live_deferred) { + Some(location) if new.generation > location.generation => { + *location = new; + self.mark_copy_dead(id, old); + } + Some(_) => { + // Stale relocation (shouldn't happen with ordered + // replay): the new copy is the garbage one. + self.mark_copy_dead(id, new); + } + None if terminal => { + // Terminal raced the copy; neither copy may + // resurrect the message. + self.mark_copy_dead(id, old); + self.mark_copy_dead(id, new); + } + None => { + // Live message the checkpoint never captured: it was + // discovered from the payload log alone (a record + // implies Ready unless superseded). Kill only the + // old copy — the new one must stay discoverable, or + // a crash between relocation and the next checkpoint + // would silently drop live mail. + self.mark_copy_dead(id, old); + } + } + } + } + } + + /// Account one physical record copy as dead (idempotently) for GC. + fn mark_copy_dead(&mut self, id: MessageId, location: JobLocation) { + if self + .tombstones + .entry(location.segment) + .or_default() + .insert(id) + { + let stats = self.segment_stats.entry(location.segment).or_default(); + stats.dead_records += 1; + stats.dead_bytes += location.length as u64; + } + } + + pub fn is_terminal(&self, segment: u64, id: &MessageId) -> bool { + self.tombstones + .get(&segment) + .is_some_and(|ids| ids.contains(id)) + } +} + +/// Owns a shard's journal + checkpoint files. Writes are synchronous and +/// meant to run on the shard's writer task. +pub struct ShardStateStore { + dir: PathBuf, + journal: JournalWriter, + /// Bytes appended to journals since the last checkpoint (drives the + /// caller's checkpoint cadence). + bytes_since_checkpoint: u64, +} + +impl ShardStateStore { + /// Load the checkpoint (if any), replay newer journal entries, truncate + /// a torn active-journal tail, and open the journal for append. + pub fn recover(shard_dir: &Path, shard: u16) -> Result<(Self, RecoveredState), QueueError> { + let (mut state, replay_from) = match load_checkpoint_file(shard_dir)? { + Some((cp, replay_from)) => (RecoveredState::from_checkpoint(cp), replay_from), + None => ( + RecoveredState::default(), + Lsn { + journal: 0, + offset: 0, + }, + ), + }; + + // Enumerate journal files at or past the replay position. + let mut journals: Vec = Vec::new(); + let entries = + std::fs::read_dir(shard_dir).map_err(|e| QueueError::io(shard_dir, e))?; + for entry in entries { + let entry = entry.map_err(|e| QueueError::io(shard_dir, e))?; + if let Some(ordinal) = entry.file_name().to_str().and_then(parse_journal_name) { + if ordinal >= replay_from.journal { + journals.push(ordinal); + } else { + // Covered by the checkpoint; a leftover from a crash + // between checkpoint publication and deletion. + let path = entry.path(); + tracing::info!(path = %path.display(), "removing journal covered by checkpoint"); + std::fs::remove_file(&path).map_err(|e| QueueError::io(&path, e))?; + } + } + } + journals.sort_unstable(); + if let Some(w) = journals.windows(2).find(|w| w[1] != w[0] + 1) { + return Err(QueueError::Layout(format!( + "shard {shard} journal sequence has a gap between {} and {}", + w[0], w[1] + ))); + } + if let Some(&first) = journals.first() { + let expected = if replay_from.journal > 0 { + replay_from.journal + } else { + // No checkpoint: the stream must be complete from its start, + // or silently lost history would resurrect terminal mail. + 1 + }; + if first != expected { + return Err(QueueError::Layout(format!( + "shard {shard} state journals must start at {expected} but oldest present \ + is {first}; refusing to recover from an incomplete journal stream" + ))); + } + } + + let mut bytes_replayed = 0u64; + let journal = match journals.last().copied() { + None => { + // Fresh shard (or checkpoint with no journal yet): start the + // stream at the checkpoint's expected ordinal. + let ordinal = replay_from.journal.max(1); + JournalWriter::create(shard_dir, ordinal)? + } + Some(last) => { + for &ordinal in &journals { + let path = shard_dir.join(journal_file_name(ordinal)); + let start = if ordinal == replay_from.journal { + replay_from.offset + } else { + 0 + }; + let tail = if ordinal == last { + TornTail::Truncate + } else { + TornTail::HardError + }; + let end = replay_journal(&path, start, tail, |e| state.apply(e))?; + bytes_replayed += end.saturating_sub(start); + } + let len = std::fs::metadata(shard_dir.join(journal_file_name(last))) + .map_err(|e| QueueError::io(shard_dir, e))? + .len(); + JournalWriter::reopen(shard_dir, last, len)? + } + }; + + Ok(( + Self { + dir: shard_dir.to_path_buf(), + journal, + bytes_since_checkpoint: bytes_replayed, + }, + state, + )) + } + + /// Persist one state transition (page-cache durability). The caller + /// applies the transition to in-memory state only after this returns. + pub fn append(&mut self, entry: &StateEntry) -> Result { + let payload = encode_entry(entry); + let lsn = self.journal.append(&payload)?; + self.bytes_since_checkpoint += (ENTRY_FRAME + payload.len()) as u64; + Ok(lsn) + } + + /// Journal bytes written since the last checkpoint; the caller's + /// checkpoint cadence trigger. + pub fn bytes_since_checkpoint(&self) -> u64 { + self.bytes_since_checkpoint + } + + /// Make the journal durable. Required before destructive boundaries + /// (segment deletion) so terminal/relocation entries covering the + /// deleted data can never be lost while the data is already gone. + pub fn fsync_journal(&self) -> Result<(), QueueError> { + self.journal.fsync() + } + + /// Write a checkpoint covering everything appended so far, then start a + /// fresh journal and delete the ones the checkpoint covers. + /// + /// Convenience composition of [`Self::begin_checkpoint`], + /// [`write_checkpoint_file`], and [`Self::finish_checkpoint`]; callers + /// that must not block (the dispatcher) run the middle step on a + /// blocking task instead. + pub fn write_checkpoint(&mut self, cp: &Checkpoint) -> Result<(), QueueError> { + let pending = self.begin_checkpoint()?; + write_checkpoint_file(&self.dir, cp, pending.replay_from)?; + self.finish_checkpoint(pending) + } + + /// Start a checkpoint: make the current journal durable, then switch + /// appends to a fresh journal file. State snapshotted after this call + /// plus `replay_from` is exactly what the checkpoint must contain; + /// entries appended meanwhile go to the new journal and replay on top. + pub fn begin_checkpoint(&mut self) -> Result { + // The journal must be durable up to the point the checkpoint claims + // to cover, otherwise a power loss could leave a checkpoint that + // skips entries which never reached disk. + self.journal.fsync()?; + let covered = self.journal.ordinal; + let next_ordinal = covered + 1; + self.journal = JournalWriter::create(&self.dir, next_ordinal)?; + self.bytes_since_checkpoint = 0; + Ok(PendingCheckpoint { + replay_from: Lsn { + journal: next_ordinal, + offset: 0, + }, + covered, + }) + } + + /// Delete journal history covered by a checkpoint that + /// [`write_checkpoint_file`] has durably published. + pub fn finish_checkpoint(&mut self, pending: PendingCheckpoint) -> Result<(), QueueError> { + for ordinal in (1..=pending.covered).rev() { + let path = self.dir.join(journal_file_name(ordinal)); + match std::fs::remove_file(&path) { + Ok(()) => {} + Err(e) if e.kind() == std::io::ErrorKind::NotFound => break, + Err(e) => return Err(QueueError::io(&path, e)), + } + } + Ok(()) + } +} + +/// Read-only counterpart to [`ShardStateStore::recover`], for inspection +/// tools that must never mutate a spool a live writer may still own: loads +/// the checkpoint and replays every journal entry newer than it, but never +/// truncates a torn tail, never deletes a stale covered journal, and never +/// creates a journal file. +/// +/// A torn tail on the active journal simply ends the replay at the last +/// valid boundary — the same boundary [`ShardStateStore::recover`] would +/// truncate to, just without touching the file. A shard with no checkpoint +/// and no journals yet (fresh, or not present at all) yields the default +/// empty state. +pub fn load_state_readonly(shard_dir: &Path) -> Result { + let (mut state, replay_from) = match load_checkpoint_file(shard_dir)? { + Some((cp, replay_from)) => (RecoveredState::from_checkpoint(cp), replay_from), + None => ( + RecoveredState::default(), + Lsn { + journal: 0, + offset: 0, + }, + ), + }; + + let mut journals: Vec = Vec::new(); + let entries = match std::fs::read_dir(shard_dir) { + Ok(entries) => entries, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(state), + Err(e) => return Err(QueueError::io(shard_dir, e)), + }; + for entry in entries { + let entry = entry.map_err(|e| QueueError::io(shard_dir, e))?; + if let Some(ordinal) = entry.file_name().to_str().and_then(parse_journal_name) { + // Read-only: unlike `recover`, never remove a stale journal that + // the checkpoint already covers; that cleanup is the write + // path's job. + if ordinal >= replay_from.journal { + journals.push(ordinal); + } + } + } + journals.sort_unstable(); + if let Some(w) = journals.windows(2).find(|w| w[1] != w[0] + 1) { + return Err(QueueError::Layout(format!( + "journal sequence has a gap between {} and {}", + w[0], w[1] + ))); + } + if let (Some(&first), true) = (journals.first(), replay_from.journal > 0) { + if first != replay_from.journal { + return Err(QueueError::Layout(format!( + "checkpoint expects journal {} but oldest present is {first}", + replay_from.journal + ))); + } + } + + let last = journals.last().copied(); + for &ordinal in &journals { + let path = shard_dir.join(journal_file_name(ordinal)); + let start = if ordinal == replay_from.journal { + replay_from.offset + } else { + 0 + }; + let tail = if Some(ordinal) == last { + TornTail::StopReadOnly + } else { + TornTail::HardError + }; + replay_journal(&path, start, tail, |e| state.apply(e))?; + } + + Ok(state) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn id(n: u64) -> MessageId { + MessageId::from_ulid(ulid::Ulid::from_parts(n, (n * 13 + 5) as u128)) + } + + fn loc(segment: u64, offset: u64) -> JobLocation { + JobLocation { + shard: 0, + segment, + offset, + length: 512, + ordinal: (offset / 512) as u32, + generation: 0, + } + } + + fn deferred(n: u64, attempts: u32) -> StateEntry { + StateEntry::Deferred { + id: id(n), + location: loc(1, n * 512), + attempts, + next_attempt_ms: 1_752_000_100_000 + n as i64, + remaining_recipients: vec![format!("r{n}@example.com")], + last_error: "451 try later".into(), + } + } + + fn delivered(n: u64) -> StateEntry { + StateEntry::Delivered { + id: id(n), + location: loc(1, n * 512), + timestamp_ms: 1_752_000_200_000, + } + } + + #[test] + fn entry_round_trip() { + for entry in [ + deferred(1, 3), + delivered(2), + StateEntry::Bounced { + id: id(3), + location: loc(2, 1024), + timestamp_ms: 5, + reason: "550 no such user".into(), + }, + ] { + let buf = encode_entry(&entry); + assert_eq!(decode_entry(&buf).unwrap(), entry); + } + } + + #[test] + fn journal_replay_and_state() { + let dir = tempfile::tempdir().unwrap(); + { + let (mut store, state) = ShardStateStore::recover(dir.path(), 0).unwrap(); + assert!(state.ready.is_empty() && state.deferred.is_empty()); + store.append(&deferred(1, 1)).unwrap(); + store.append(&deferred(2, 1)).unwrap(); + store.append(&delivered(1)).unwrap(); + } + let (_, state) = ShardStateStore::recover(dir.path(), 0).unwrap(); + assert!(!state.deferred.contains_key(&id(1)), "delivered wins"); + assert!(state.deferred.contains_key(&id(2))); + assert!(state.is_terminal(1, &id(1))); + assert_eq!(state.segment_stats[&1].dead_records, 1); + assert_eq!(state.segment_stats[&1].dead_bytes, 512); + } + + #[test] + fn deferred_attempts_survive_restart() { + let dir = tempfile::tempdir().unwrap(); + { + let (mut store, _) = ShardStateStore::recover(dir.path(), 0).unwrap(); + store.append(&deferred(7, 1)).unwrap(); + store.append(&deferred(7, 2)).unwrap(); + store.append(&deferred(7, 3)).unwrap(); + } + let (_, state) = ShardStateStore::recover(dir.path(), 0).unwrap(); + let d = &state.deferred[&id(7)]; + assert_eq!(d.attempts, 3); + assert_eq!(d.remaining_recipients, vec!["r7@example.com".to_string()]); + } + + #[test] + fn torn_journal_tail_is_truncated() { + let dir = tempfile::tempdir().unwrap(); + { + let (mut store, _) = ShardStateStore::recover(dir.path(), 0).unwrap(); + store.append(&deferred(1, 1)).unwrap(); + store.append(&deferred(2, 1)).unwrap(); + } + // Tear the last entry. + let path = dir.path().join(journal_file_name(1)); + let len = std::fs::metadata(&path).unwrap().len(); + let f = OpenOptions::new().write(true).open(&path).unwrap(); + f.set_len(len - 5).unwrap(); + + let (mut store, state) = ShardStateStore::recover(dir.path(), 0).unwrap(); + assert!(state.deferred.contains_key(&id(1))); + assert!(!state.deferred.contains_key(&id(2)), "torn entry dropped"); + + // The journal must be appendable at the truncated boundary. + store.append(&deferred(3, 1)).unwrap(); + let (_, state) = ShardStateStore::recover(dir.path(), 0).unwrap(); + assert!(state.deferred.contains_key(&id(3))); + } + + #[test] + fn checkpoint_round_trip_and_journal_rotation() { + let dir = tempfile::tempdir().unwrap(); + let (mut store, _) = ShardStateStore::recover(dir.path(), 0).unwrap(); + store.append(&deferred(1, 1)).unwrap(); + store.append(&delivered(2)).unwrap(); + assert!(store.bytes_since_checkpoint() > 0); + + let cp = Checkpoint { + cursor: Some((3, 4096)), + ready: vec![ + ReadyJob { + id: id(10), + location: loc(2, 0), + attempts: 0, + enqueue_ms: 42, + remaining_recipients: vec![], + }, + // A formerly-deferred job that became due before the + // snapshot: its partial-recipient set must survive. + ReadyJob { + id: id(11), + location: loc(2, 512), + attempts: 2, + enqueue_ms: 43, + remaining_recipients: vec!["still-waiting@example.com".into()], + }, + ], + deferred: vec![DeferredJob { + id: id(1), + location: loc(1, 512), + attempts: 1, + next_attempt_ms: 99, + remaining_recipients: vec!["r@example.com".into()], + last_error: "451".into(), + }], + tombstones: vec![(1, vec![id(2)])], + segment_stats: vec![(1, SegmentStats { + total_records: 8, + total_bytes: 4096, + dead_records: 1, + dead_bytes: 512, + })], + }; + store.write_checkpoint(&cp).unwrap(); + assert_eq!(store.bytes_since_checkpoint(), 0); + // Old journal deleted, new one active. + assert!(!dir.path().join(journal_file_name(1)).exists()); + assert!(dir.path().join(journal_file_name(2)).exists()); + + // Entries after the checkpoint layer on top of it. The Delivered + // entry carries the message's actual location (segment 2). + store + .append(&StateEntry::Delivered { + id: id(10), + location: loc(2, 0), + timestamp_ms: 1_752_000_300_000, + }) + .unwrap(); + drop(store); + + let (_, state) = ShardStateStore::recover(dir.path(), 0).unwrap(); + assert_eq!(state.cursor, Some((3, 4096))); + let r = &state.ready[&id(11)]; + assert_eq!(r.attempts, 2); + assert_eq!( + r.remaining_recipients, + vec!["still-waiting@example.com".to_string()], + "partial-recipient set survives a ready-state checkpoint" + ); + assert!(!state.ready.contains_key(&id(10)), "delivered post-checkpoint"); + assert!(state.is_terminal(2, &id(10))); + assert!(state.is_terminal(1, &id(2)), "checkpoint tombstone kept"); + let d = &state.deferred[&id(1)]; + assert_eq!(d.attempts, 1); + assert_eq!(state.segment_stats[&1].total_records, 8); + } + + #[test] + fn relocation_updates_live_location_and_kills_old_copy() { + let dir = tempfile::tempdir().unwrap(); + { + let (mut store, _) = ShardStateStore::recover(dir.path(), 0).unwrap(); + store.append(&deferred(1, 2)).unwrap(); + let mut new = loc(9, 0); + new.generation = 1; + store + .append(&StateEntry::Relocated { + id: id(1), + old: loc(1, 512), + new, + }) + .unwrap(); + } + let (_, state) = ShardStateStore::recover(dir.path(), 0).unwrap(); + let d = &state.deferred[&id(1)]; + assert_eq!(d.location.segment, 9); + assert_eq!(d.location.generation, 1); + assert_eq!(d.attempts, 2, "relocation preserves retry state"); + // Old copy is dead garbage. + assert!(state.is_terminal(1, &id(1))); + assert_eq!(state.segment_stats[&1].dead_bytes, 512); + // New copy is not dead. + assert!(!state.is_terminal(9, &id(1))); + } + + #[test] + fn relocation_of_uncheckpointed_live_message_keeps_new_copy_alive() { + // A ready message that exists only as a payload record (discovered, + // never journaled or checkpointed) gets relocated by compaction and + // the process crashes before the next checkpoint. Replay must kill + // only the old copy: tombstoning the new copy too would silently + // drop live mail. + let dir = tempfile::tempdir().unwrap(); + { + let (mut store, _) = ShardStateStore::recover(dir.path(), 0).unwrap(); + let mut new = loc(9, 0); + new.generation = 1; + store + .append(&StateEntry::Relocated { + id: id(1), + old: loc(1, 512), + new, + }) + .unwrap(); + } + let (_, state) = ShardStateStore::recover(dir.path(), 0).unwrap(); + assert!(state.is_terminal(1, &id(1)), "old copy is garbage"); + assert!( + !state.is_terminal(9, &id(1)), + "new copy must stay discoverable" + ); + } + + #[test] + fn terminal_race_during_relocation_does_not_resurrect() { + let dir = tempfile::tempdir().unwrap(); + { + let (mut store, _) = ShardStateStore::recover(dir.path(), 0).unwrap(); + store.append(&deferred(1, 1)).unwrap(); + // Terminal lands first (worker delivered while compaction was + // copying), then the relocation entry arrives. + store.append(&delivered(1)).unwrap(); + let mut new = loc(9, 0); + new.generation = 1; + store + .append(&StateEntry::Relocated { + id: id(1), + old: loc(1, 512), + new, + }) + .unwrap(); + } + let (_, state) = ShardStateStore::recover(dir.path(), 0).unwrap(); + assert!(state.ready.is_empty() && state.deferred.is_empty()); + // Both physical copies are garbage; neither resurrects the message. + assert!(state.is_terminal(1, &id(1))); + assert!(state.is_terminal(9, &id(1))); + assert_eq!(state.segment_stats[&9].dead_records, 1); + } + + #[test] + fn checkpoint_corruption_is_a_hard_error() { + let dir = tempfile::tempdir().unwrap(); + let (mut store, _) = ShardStateStore::recover(dir.path(), 0).unwrap(); + store.append(&deferred(1, 1)).unwrap(); + store.write_checkpoint(&Checkpoint::default()).unwrap(); + drop(store); + + let path = dir.path().join(CHECKPOINT_FILE); + let mut buf = std::fs::read(&path).unwrap(); + let last = buf.len() - 1; + buf[last] ^= 0xff; + std::fs::write(&path, &buf).unwrap(); + assert!(ShardStateStore::recover(dir.path(), 0).is_err()); + } + + #[test] + fn journal_gap_is_a_hard_error() { + let dir = tempfile::tempdir().unwrap(); + { + let (mut store, _) = ShardStateStore::recover(dir.path(), 0).unwrap(); + store.append(&deferred(1, 1)).unwrap(); + store.write_checkpoint(&Checkpoint::default()).unwrap(); + store.append(&deferred(2, 1)).unwrap(); + store.write_checkpoint(&Checkpoint::default()).unwrap(); + } + // Journals 1,2 deleted; 3 is active. Fabricate a gap: 3 -> 5. + std::fs::write(dir.path().join(journal_file_name(5)), b"").unwrap(); + assert!(matches!( + ShardStateStore::recover(dir.path(), 0), + Err(QueueError::Layout(_)) + )); + } + + #[test] + fn crash_between_checkpoint_and_journal_delete_recovers() { + let dir = tempfile::tempdir().unwrap(); + let (mut store, _) = ShardStateStore::recover(dir.path(), 0).unwrap(); + store.append(&delivered(1)).unwrap(); + store.write_checkpoint(&Checkpoint { + tombstones: vec![(1, vec![id(1)])], + ..Default::default() + }) + .unwrap(); + drop(store); + + // Simulate the crash by resurrecting a stale, covered journal file + // containing garbage; recovery must delete it, not replay it. + std::fs::write(dir.path().join(journal_file_name(1)), b"stale garbage").unwrap(); + let (_, state) = ShardStateStore::recover(dir.path(), 0).unwrap(); + assert!(state.is_terminal(1, &id(1))); + assert!(!dir.path().join(journal_file_name(1)).exists()); + } + + #[test] + fn read_only_loader_handles_a_fresh_shard() { + let dir = tempfile::tempdir().unwrap(); + // Directory exists but nothing has ever been written to it. + std::fs::create_dir_all(dir.path()).unwrap(); + let state = load_state_readonly(dir.path()).unwrap(); + assert!(state.ready.is_empty()); + assert!(state.deferred.is_empty()); + assert_eq!(state.cursor, None); + + // A shard directory that does not exist at all is just as fine. + let missing = dir.path().join("does-not-exist"); + let state = load_state_readonly(&missing).unwrap(); + assert!(state.ready.is_empty() && state.deferred.is_empty()); + } + + #[test] + fn read_only_loader_does_not_mutate_a_torn_journal() { + let dir = tempfile::tempdir().unwrap(); + { + let (mut store, _) = ShardStateStore::recover(dir.path(), 0).unwrap(); + store.append(&deferred(1, 1)).unwrap(); + store.append(&deferred(2, 1)).unwrap(); + } + // Tear the last entry, exactly like `torn_journal_tail_is_truncated`. + let path = dir.path().join(journal_file_name(1)); + let len = std::fs::metadata(&path).unwrap().len(); + let f = OpenOptions::new().write(true).open(&path).unwrap(); + f.set_len(len - 5).unwrap(); + let torn_len = std::fs::metadata(&path).unwrap().len(); + + let state = load_state_readonly(dir.path()).unwrap(); + assert!(state.deferred.contains_key(&id(1))); + assert!( + !state.deferred.contains_key(&id(2)), + "torn entry excluded from replay" + ); + + // The critical read-only property: the file must be byte-for-byte + // unchanged, unlike the write path's truncation. + assert_eq!( + std::fs::metadata(&path).unwrap().len(), + torn_len, + "read-only loader must never truncate the journal" + ); + let contents_after = std::fs::read(&path).unwrap(); + assert_eq!(contents_after.len() as u64, torn_len); + + // Calling it again is idempotent and still doesn't touch the file. + let state2 = load_state_readonly(dir.path()).unwrap(); + assert_eq!(state2.deferred.contains_key(&id(1)), true); + assert_eq!( + std::fs::metadata(&path).unwrap().len(), + torn_len, + "second read-only call must not touch the file either" + ); + + // The write path can still recover (and truncate) normally + // afterwards; the read-only loader must not have wedged anything. + let (mut store, recovered) = ShardStateStore::recover(dir.path(), 0).unwrap(); + assert!(recovered.deferred.contains_key(&id(1))); + assert!(!recovered.deferred.contains_key(&id(2))); + store.append(&deferred(3, 1)).unwrap(); + let (_, state3) = ShardStateStore::recover(dir.path(), 0).unwrap(); + assert!(state3.deferred.contains_key(&id(3))); + } + + #[test] + fn read_only_loader_matches_recover_across_a_checkpoint() { + let dir = tempfile::tempdir().unwrap(); + { + let (mut store, _) = ShardStateStore::recover(dir.path(), 0).unwrap(); + store.append(&deferred(1, 1)).unwrap(); + store + .write_checkpoint(&Checkpoint { + tombstones: vec![(1, vec![id(9)])], + ..Default::default() + }) + .unwrap(); + store.append(&delivered(2)).unwrap(); + } + + let readonly = load_state_readonly(dir.path()).unwrap(); + let (_, recovered) = ShardStateStore::recover(dir.path(), 0).unwrap(); + assert_eq!(readonly.cursor, recovered.cursor); + assert_eq!(readonly.ready, recovered.ready); + assert_eq!(readonly.deferred, recovered.deferred); + assert_eq!(readonly.tombstones, recovered.tombstones); + assert_eq!(readonly.segment_stats, recovered.segment_stats); + } +} diff --git a/smtp-server/src/logqueue/writer.rs b/smtp-server/src/logqueue/writer.rs new file mode 100644 index 0000000..32e3bc3 --- /dev/null +++ b/smtp-server/src/logqueue/writer.rs @@ -0,0 +1,889 @@ +//! Append writers: one per shard, each exclusively owning its shard's +//! active segment. +//! +//! Admission is bounded by pending bytes (not request count): a permit for +//! the encoded record size is acquired before the request is queued and +//! released once the bytes have been handed to the kernel page cache. SMTP +//! acceptance awaits only this append completion. +//! +//! Publish ordering per record (PLAN §9.5): write the complete record, then +//! advance the shard's committed head under the state lock, then notify the +//! dispatcher, then complete the request. The committed head never exposes +//! a partial record. + +use std::sync::{Arc, Mutex}; + +use bytes::Bytes; +use tokio::sync::{mpsc, oneshot, Notify, Semaphore}; + +use super::record::{self, RecordParams}; +use super::segment::{validate_active_tail, ActiveSegment}; +use super::shard::ShardDir; +use super::spool::Spool; +use super::{JobLocation, MessageId, QueueError}; + +/// Configuration for the writer set. Values come from `[queue]` config; +/// validation (segment sizing vs. max message size) happens at startup. +#[derive(Debug, Clone)] +pub struct WriterConfig { + /// Seal the active segment once it reaches this size. + pub segment_target_bytes: u64, + /// Hard cap on one encoded record; also the scan bound. Derived from + /// the configured maximum message size plus envelope allowance. + pub max_record_len: u32, + /// Total bytes of not-yet-written admission buffering across all shards. + pub pending_append_bytes: u64, +} + +/// One committed segment head. `committed` is the offset one past the last +/// complete record; for sealed entries it is the segment's final length. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct SegmentHead { + pub segment: u64, + pub committed: u64, + pub sealed: bool, +} + +/// State a shard's writer shares with the dispatcher: the ordered chain of +/// append segments (last entry is the active one) and a wake-up hint. +/// +/// The chain contains only segments used as append targets, in append +/// order. Compaction outputs never appear here, which is what keeps them +/// out of discovery. Fully-consumed sealed entries are pruned by the +/// dispatcher/GC via [`ShardShared::prune`]. +pub struct ShardShared { + shard: u16, + chain: Mutex>, + /// Lossy wake-up hint for the dispatcher; the chain is authoritative. + pub notify: Notify, +} + +impl ShardShared { + fn new(shard: u16, initial: Vec) -> Self { + Self { + shard, + chain: Mutex::new(initial), + notify: Notify::new(), + } + } + + pub fn shard(&self) -> u16 { + self.shard + } + + /// Snapshot of the append chain. + pub fn chain(&self) -> Vec { + self.chain.lock().unwrap().clone() + } + + /// Remove sealed chain entries the caller no longer needs (everything + /// strictly below `segment`). The active entry is never pruned. + pub fn prune(&self, segment: u64) { + let mut chain = self.chain.lock().unwrap(); + chain.retain(|h| h.segment >= segment || !h.sealed); + } + + /// Remove one sealed segment from the chain (it was deleted by GC). + /// The active entry is never removed. + pub fn remove_segment(&self, segment: u64) { + let mut chain = self.chain.lock().unwrap(); + chain.retain(|h| h.segment != segment || !h.sealed); + } + + fn advance_committed(&self, segment: u64, committed: u64) { + let mut chain = self.chain.lock().unwrap(); + let head = chain + .last_mut() + .expect("advance_committed on empty chain"); + debug_assert_eq!(head.segment, segment); + debug_assert!(!head.sealed && committed > head.committed); + head.committed = committed; + } + + fn seal_segment(&self, sealed_segment: u64, final_len: u64) { + let mut chain = self.chain.lock().unwrap(); + let head = chain.last_mut().expect("seal_segment on empty chain"); + debug_assert_eq!(head.segment, sealed_segment); + head.committed = final_len; + head.sealed = true; + } + + fn open_segment(&self, next_segment: u64) { + let mut chain = self.chain.lock().unwrap(); + debug_assert!(chain.last().is_none_or(|h| h.sealed)); + chain.push(SegmentHead { + segment: next_segment, + committed: 0, + sealed: false, + }); + } +} + +/// A message to be appended. `enqueue_ms` is stamped by the caller so queue +/// age survives relocation and restarts. `generation` is 0 for new mail; +/// compaction re-appends live records with a higher relocation generation. +pub struct AppendMessage { + pub message_id: MessageId, + pub enqueue_ms: i64, + pub generation: u32, + pub sender: String, + pub recipients: Vec, + pub body: Bytes, +} + +struct AppendRequest { + msg: AppendMessage, + encoded_len: u32, + completion: oneshot::Sender>, +} + +enum WriterMsg { + Append(AppendRequest), + /// Close admission: the writer finishes everything queued before this + /// sentinel and exits; anything queued after it fails `WriterClosed`. + Shutdown, +} + +struct ShardChannel { + tx: mpsc::UnboundedSender, + shared: Arc, +} + +/// Cloneable admission handle used by the SMTP acceptance path. +#[derive(Clone)] +pub struct AppendHandle { + shards: Arc>, + /// Byte-bounded admission shared across shards. + pending_bytes: Arc, + pending_limit: u64, + max_record_len: u32, +} + +impl AppendHandle { + /// Route a message id to its shard: the low bytes of a ULID are random, + /// so a modulo over them distributes uniformly. Only ever used for NEW + /// messages — existing records carry their explicit location. + pub fn shard_for(&self, id: &MessageId) -> u16 { + (u16::from_le_bytes([id.0[14], id.0[15]])) % self.shards.len() as u16 + } + + pub fn shard_shared(&self, shard: u16) -> Arc { + Arc::clone(&self.shards[shard as usize].shared) + } + + #[cfg(test)] + pub fn shard_count(&self) -> u16 { + self.shards.len() as u16 + } + + /// Append a message and wait until it is accepted by the kernel page + /// cache. Returns its physical location. Applies byte-bounded admission + /// backpressure while the writer is behind. + pub async fn append(&self, msg: AppendMessage) -> Result { + let shard = self.shard_for(&msg.message_id); + self.append_to_shard(shard, msg).await + } + + /// Append to an explicit shard. New mail must use [`Self::append`] + /// (stable hash routing); this exists for compaction, which relocates a + /// record within the shard that owns its state journal. + pub async fn append_to_shard( + &self, + shard: u16, + msg: AppendMessage, + ) -> Result { + let params = RecordParams { + message_id: msg.message_id, + enqueue_ms: msg.enqueue_ms, + generation: msg.generation, + ordinal: 0, // assigned by the writer; same encoded size + sender: &msg.sender, + recipients: &msg.recipients, + body: &msg.body, + }; + let encoded_len = record::encoded_len(¶ms)?; + if encoded_len > self.max_record_len { + return Err(QueueError::RecordTooLarge { + len: encoded_len as u64, + limit: self.max_record_len as u64, + }); + } + + // Acquire admission permits for the encoded size, clamped so one + // huge record cannot exceed the whole semaphore (it then simply + // occupies all admission capacity while queued). + let permits = (encoded_len as u64).min(self.pending_limit) as u32; + let permit = Arc::clone(&self.pending_bytes) + .acquire_many_owned(permits) + .await + .expect("admission semaphore is never closed"); + crate::metrics::logqueue_pending_append_bytes_set( + self.pending_limit - self.pending_bytes.available_permits() as u64, + ); + + let (tx, rx) = oneshot::channel(); + self.shards[shard as usize] + .tx + .send(WriterMsg::Append(AppendRequest { + msg, + encoded_len, + completion: tx, + })) + .map_err(|_| QueueError::WriterClosed(shard))?; + + let result = rx.await.map_err(|_| QueueError::WriterClosed(shard))?; + // Bytes are in the page cache (or failed); admission capacity frees + // either way. + drop(permit); + result + } +} + +/// The writer set: spawns one blocking writer task per shard. +pub struct LogWriters { + handle: AppendHandle, + join: Vec>, +} + +impl LogWriters { + /// Recover every shard (validating and truncating active tails) and + /// start the writer tasks. + pub fn start(spool: &Spool, config: WriterConfig) -> Result { + let mut shards = Vec::with_capacity(spool.shard_count() as usize); + let mut join = Vec::with_capacity(spool.shard_count() as usize); + + for shard_dir in spool.shards() { + let (state, shared) = ShardWriter::recover(shard_dir)?; + let shared = Arc::new(shared); + let (tx, rx) = mpsc::unbounded_channel(); + let writer_shared = Arc::clone(&shared); + let cfg = config.clone(); + join.push(tokio::task::spawn_blocking(move || { + ShardWriter::run(state, rx, writer_shared, cfg) + })); + shards.push(ShardChannel { tx, shared }); + } + + Ok(Self { + handle: AppendHandle { + shards: Arc::new(shards), + pending_bytes: Arc::new(Semaphore::new(config.pending_append_bytes as usize)), + pending_limit: config.pending_append_bytes, + max_record_len: config.max_record_len, + }, + join, + }) + } + + pub fn handle(&self) -> AppendHandle { + self.handle.clone() + } + + /// Close admission and wait for every writer to finish everything + /// queued so far. Appends submitted after this fail with + /// [`QueueError::WriterClosed`], even through surviving handle clones. + pub async fn shutdown(self) { + for shard in self.handle.shards.iter() { + let _ = shard.tx.send(WriterMsg::Shutdown); + } + for task in self.join { + if let Err(e) = task.await { + tracing::error!(error = %e, "append writer task failed during shutdown"); + } + } + } +} + +/// Per-shard writer state, owned by one blocking task. `active` is `None` +/// only in the window after a seal succeeded but creating the replacement +/// failed; the next append retries the create instead of ever writing into +/// the sealed file. +struct ShardWriter { + dir: ShardDir, + active: Option, + next_segment: u64, +} + +impl ShardWriter { + /// Open the shard: validate/truncate the active tail if one exists, + /// otherwise create the next segment. Returns the writer state and the + /// initial shared chain (sealed segments + active head). + fn recover(dir: &ShardDir) -> Result<(Self, ShardShared), QueueError> { + let segments = dir.list_segments()?; + let mut chain: Vec = Vec::new(); + + // Sealed segments enter the chain in ordinal order with their file + // length as the committed length. (Once compaction exists, its + // output segments are excluded from the chain by recovery — that + // arrives with the phase that writes them.) + for (segment, path) in &segments.sealed { + let len = std::fs::metadata(path) + .map_err(|e| QueueError::io(path, e))? + .len(); + chain.push(SegmentHead { + segment: *segment, + committed: len, + sealed: true, + }); + } + + let active = match segments.active { + Some((segment, path)) => { + // Validate against the format's absolute bound, not the + // configured one: shrinking max_message_size must never + // make previously accepted records look corrupt and get + // truncated (destroying queued mail). + let tail = validate_active_tail(&path, record::MAX_RECORD_LEN)?; + if tail.truncated_bytes > 0 { + tracing::warn!( + shard = dir.shard(), + segment, + truncated_bytes = tail.truncated_bytes, + "discarded torn tail during shard recovery" + ); + } + let seg = ActiveSegment::recover(path, segment, &tail)?; + chain.push(SegmentHead { + segment, + committed: tail.committed_len, + sealed: false, + }); + seg + } + None => { + let seg = ActiveSegment::create(dir.path(), segments.next_segment)?; + chain.push(SegmentHead { + segment: seg.segment(), + committed: 0, + sealed: false, + }); + seg + } + }; + + let shared = ShardShared::new(dir.shard(), chain); + let next_segment = active.segment() + 1; + Ok(( + Self { + dir: dir.clone(), + active: Some(active), + next_segment, + }, + shared, + )) + } + + /// Writer loop: runs on a blocking task until the admission channel + /// closes and drains. + fn run( + mut self, + mut rx: mpsc::UnboundedReceiver, + shared: Arc, + config: WriterConfig, + ) { + while let Some(msg) = rx.blocking_recv() { + let req = match msg { + WriterMsg::Append(req) => req, + // Dropping the receiver fails any requests queued after the + // sentinel with WriterClosed (their completions drop). + WriterMsg::Shutdown => break, + }; + let result = self.write_one(&req, &shared, &config); + if let Err(e) = &result { + crate::metrics::logqueue_append_error(); + tracing::error!( + shard = shared.shard(), + message_id = %req.msg.message_id, + error = %e, + "append failed" + ); + } + // Publish ordering: head advanced and dispatcher notified inside + // write_one BEFORE this completion is sent. + let _ = req.completion.send(result); + } + tracing::debug!(shard = shared.shard(), "append writer drained and stopped"); + } + + fn write_one( + &mut self, + req: &AppendRequest, + shared: &ShardShared, + config: &WriterConfig, + ) -> Result { + // Recreate the active segment if the previous rotation sealed the + // old one but failed to create its replacement. + if self.active.is_none() { + let seg = ActiveSegment::create(self.dir.path(), self.next_segment)?; + self.next_segment += 1; + shared.open_segment(seg.segment()); + self.active = Some(seg); + } + // Rotate if this record would overflow the target size (never on an + // empty segment: sizing validation guarantees any legal record fits + // within a full segment). + let active = self.active.as_ref().expect("just ensured"); + if active.len() > 0 + && active.len() + req.encoded_len as u64 > config.segment_target_bytes + { + self.rotate(shared)?; + } + + let active = self.active.as_mut().expect("rotate keeps an active segment"); + let ordinal = active.next_ordinal(); + let encoded = record::encode(&RecordParams { + message_id: req.msg.message_id, + enqueue_ms: req.msg.enqueue_ms, + generation: req.msg.generation, + ordinal, + sender: &req.msg.sender, + recipients: &req.msg.recipients, + body: &req.msg.body, + })?; + debug_assert_eq!(encoded.len() as u32, req.encoded_len); + + let write_started = std::time::Instant::now(); + let offset = active.append(&encoded)?; + crate::metrics::logqueue_append_duration_observe(shared.shard(), write_started.elapsed()); + crate::metrics::logqueue_records_appended(shared.shard(), 1); + crate::metrics::logqueue_bytes_appended(shared.shard(), encoded.len() as u64); + crate::metrics::logqueue_active_segment_bytes_set(shared.shard(), active.len()); + let location = JobLocation { + shard: shared.shard(), + segment: active.segment(), + offset, + length: req.encoded_len, + ordinal, + generation: req.msg.generation, + }; + + shared.advance_committed(location.segment, active.len()); + shared.notify.notify_one(); + Ok(location) + } + + fn rotate(&mut self, shared: &ShardShared) -> Result<(), QueueError> { + let old = self.active.as_ref().expect("rotate requires an active segment"); + let sealed_segment = old.segment(); + // Seal FIRST, create second: a crash in between leaves no active + // segment (recovery simply creates one) — never two, which would be + // an unrecoverable layout error. If create fails, `active` becomes + // None and the next append retries the create. + let (_, final_len) = old.seal_in_place()?; + self.active = None; + shared.seal_segment(sealed_segment, final_len); + let next_segment = self.next_segment; + let seg = ActiveSegment::create(self.dir.path(), next_segment)?; + self.next_segment += 1; + self.active = Some(seg); + shared.open_segment(next_segment); + shared.notify.notify_one(); + crate::metrics::logqueue_segment_rotation(shared.shard()); + crate::metrics::logqueue_active_segment_bytes_set(shared.shard(), 0); + tracing::debug!( + shard = shared.shard(), + sealed = sealed_segment, + final_len, + next = next_segment, + "rotated active segment" + ); + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::logqueue::record::MAX_RECORD_LEN; + use crate::logqueue::segment::SegmentReader; + + fn config() -> WriterConfig { + WriterConfig { + segment_target_bytes: 64 * 1024 * 1024, + max_record_len: MAX_RECORD_LEN, + pending_append_bytes: 16 * 1024 * 1024, + } + } + + fn message(seq: u64, body: &[u8]) -> AppendMessage { + AppendMessage { + message_id: MessageId::from_ulid(ulid::Ulid::from_parts(seq, (seq * 7 + 1) as u128)), + enqueue_ms: 1_752_000_000_000 + seq as i64, + generation: 0, + sender: "sender@example.com".into(), + recipients: vec!["rcpt@example.com".into()], + body: Bytes::copy_from_slice(body), + } + } + + #[tokio::test] + async fn append_returns_readable_location() { + let dir = tempfile::tempdir().unwrap(); + let spool = Spool::open(dir.path().join("spool"), 1).unwrap(); + let writers = LogWriters::start(&spool, config()).unwrap(); + let handle = writers.handle(); + + let loc = handle.append(message(1, b"hello queue")).await.unwrap(); + assert_eq!(loc.shard, 0); + assert_eq!(loc.ordinal, 0); + assert_eq!(loc.generation, 0); + + let path = spool + .shard(0) + .path() + .join(crate::logqueue::segment::active_file_name(loc.segment)); + let reader = SegmentReader::open(path).unwrap(); + let (header, body) = reader.read_record_at(loc.offset, MAX_RECORD_LEN).unwrap(); + assert_eq!(body, b"hello queue"); + assert_eq!(header.sender, "sender@example.com"); + assert_eq!(header.record_len, loc.length); + + writers.shutdown().await; + } + + #[tokio::test] + async fn concurrent_appends_all_land() { + let dir = tempfile::tempdir().unwrap(); + let spool = Spool::open(dir.path().join("spool"), 2).unwrap(); + let writers = LogWriters::start(&spool, config()).unwrap(); + let handle = writers.handle(); + + let mut tasks = Vec::new(); + for i in 0..200u64 { + let handle = handle.clone(); + tasks.push(tokio::spawn(async move { + let body = vec![b'x'; (i % 977 + 1) as usize]; + handle.append(message(i, &body)).await.unwrap() + })); + } + let mut locations = Vec::new(); + for t in tasks { + locations.push(t.await.unwrap()); + } + + // Every location must be unique and readable. + let mut seen = std::collections::HashSet::new(); + for loc in &locations { + assert!(seen.insert((loc.shard, loc.segment, loc.offset))); + } + + // Committed heads cover every record; offsets within a shard's + // segment are dense (offset of ordinal n+1 = offset + length of n). + for shard in 0..handle.shard_count() { + let chain = handle.shard_shared(shard).chain(); + let mut per_seg: Vec<_> = locations + .iter() + .filter(|l| l.shard == shard) + .collect(); + per_seg.sort_by_key(|l| (l.segment, l.offset)); + let mut expected_offset = std::collections::HashMap::new(); + for loc in per_seg { + let e = expected_offset.entry(loc.segment).or_insert(0u64); + assert_eq!(loc.offset, *e, "hole in shard {shard} segment {}", loc.segment); + *e += loc.length as u64; + let head = chain.iter().find(|h| h.segment == loc.segment).unwrap(); + assert!(head.committed >= loc.offset + loc.length as u64); + } + } + + writers.shutdown().await; + } + + #[tokio::test] + async fn rotation_at_target_size() { + let dir = tempfile::tempdir().unwrap(); + let spool = Spool::open(dir.path().join("spool"), 1).unwrap(); + let mut cfg = config(); + cfg.segment_target_bytes = 4096; + let writers = LogWriters::start(&spool, cfg).unwrap(); + let handle = writers.handle(); + + for i in 0..20u64 { + handle.append(message(i, &vec![b'y'; 1024])).await.unwrap(); + } + + let chain = handle.shard_shared(0).chain(); + assert!(chain.len() > 1, "expected rotation, chain: {chain:?}"); + // All but the last entry are sealed, exist on disk as .log, and + // their committed length equals the file length. + for head in &chain[..chain.len() - 1] { + assert!(head.sealed); + let path = spool + .shard(0) + .path() + .join(crate::logqueue::segment::sealed_file_name(head.segment)); + assert_eq!(std::fs::metadata(&path).unwrap().len(), head.committed); + assert!(head.committed <= 4096 + 1024 + 4096); // target + slack + } + assert!(!chain.last().unwrap().sealed); + + // Records must be discoverable across the rotation boundary. + let mut total = 0; + for head in &chain { + let name = if head.sealed { + crate::logqueue::segment::sealed_file_name(head.segment) + } else { + crate::logqueue::segment::active_file_name(head.segment) + }; + let reader = SegmentReader::open(spool.shard(0).path().join(name)).unwrap(); + crate::logqueue::segment::scan_headers( + &reader, + 0, + head.committed, + MAX_RECORD_LEN, + |_, _| { + total += 1; + true + }, + ) + .unwrap(); + } + assert_eq!(total, 20); + + writers.shutdown().await; + } + + #[tokio::test] + async fn oversized_record_is_rejected_up_front() { + let dir = tempfile::tempdir().unwrap(); + let spool = Spool::open(dir.path().join("spool"), 1).unwrap(); + let mut cfg = config(); + cfg.max_record_len = 2048; + let writers = LogWriters::start(&spool, cfg).unwrap(); + let handle = writers.handle(); + + let err = handle + .append(message(1, &vec![b'z'; 4096])) + .await + .unwrap_err(); + assert!(matches!(err, QueueError::RecordTooLarge { .. })); + + writers.shutdown().await; + } + + #[tokio::test] + async fn writer_recovers_torn_tail_and_continues() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().join("spool"); + let loc = { + let spool = Spool::open(&root, 1).unwrap(); + let writers = LogWriters::start(&spool, config()).unwrap(); + let handle = writers.handle(); + let loc = handle.append(message(1, b"survives")).await.unwrap(); + handle.append(message(2, b"gets torn")).await.unwrap(); + writers.shutdown().await; + loc + }; + + // Tear the second record's tail. + let seg_path = root + .join("shard-0000") + .join(crate::logqueue::segment::active_file_name(loc.segment)); + let len = std::fs::metadata(&seg_path).unwrap().len(); + let f = std::fs::OpenOptions::new() + .write(true) + .open(&seg_path) + .unwrap(); + f.set_len(len - 3).unwrap(); + + // Restart: the torn record is truncated, appends continue after the + // survivor with the correct ordinal. + let spool = Spool::open(&root, 1).unwrap(); + let writers = LogWriters::start(&spool, config()).unwrap(); + let handle = writers.handle(); + let chain = handle.shard_shared(0).chain(); + assert_eq!(chain.len(), 1); + assert_eq!(chain[0].committed, loc.offset + loc.length as u64); + + let loc3 = handle.append(message(3, b"after recovery")).await.unwrap(); + assert_eq!(loc3.ordinal, 1); + assert_eq!(loc3.offset, loc.offset + loc.length as u64); + + let reader = SegmentReader::open(&seg_path).unwrap(); + let (_, body) = reader.read_record_at(loc3.offset, MAX_RECORD_LEN).unwrap(); + assert_eq!(body, b"after recovery"); + + writers.shutdown().await; + } + + #[tokio::test] + async fn shard_routing_is_stable_and_in_range() { + let dir = tempfile::tempdir().unwrap(); + let spool = Spool::open(dir.path().join("spool"), 4).unwrap(); + let writers = LogWriters::start(&spool, config()).unwrap(); + let handle = writers.handle(); + + let mut hits = vec![0u32; 4]; + for i in 0..1000u64 { + let id = MessageId::from_ulid(ulid::Ulid::from_parts(i, (i * 31 + 7) as u128)); + let s = handle.shard_for(&id); + assert_eq!(s, handle.shard_for(&id)); + hits[s as usize] += 1; + } + assert!(hits.iter().all(|&h| h > 0), "distribution: {hits:?}"); + + writers.shutdown().await; + } + + #[tokio::test] + async fn admission_bytes_bound_is_respected() { + let dir = tempfile::tempdir().unwrap(); + let spool = Spool::open(dir.path().join("spool"), 2).unwrap(); + let mut cfg = config(); + cfg.pending_append_bytes = 8192; + let writers = LogWriters::start(&spool, cfg).unwrap(); + let handle = writers.handle(); + + // Many concurrent small appends must all land even though their + // combined size vastly exceeds the admission budget: the semaphore + // throttles concurrency, it never drops or deadlocks a request. + let mut tasks = Vec::new(); + for i in 0..50u64 { + let handle = handle.clone(); + tasks.push(tokio::spawn(async move { + let body = vec![b'q'; 1024]; + handle.append(message(i, &body)).await + })); + } + for t in tasks { + t.await.unwrap().unwrap(); + } + + // A single record whose encoded size exceeds the whole admission + // budget must still succeed via the permit clamp. + let big_body = vec![b'r'; 16 * 1024]; + handle.append(message(1000, &big_body)).await.unwrap(); + + writers.shutdown().await; + } + + #[tokio::test] + async fn appends_after_shutdown_fail_with_writer_closed() { + let dir = tempfile::tempdir().unwrap(); + let spool = Spool::open(dir.path().join("spool"), 1).unwrap(); + let writers = LogWriters::start(&spool, config()).unwrap(); + let handle = writers.handle(); + let surviving_handle = handle.clone(); + + writers.shutdown().await; + + let err = surviving_handle + .append(message(1, b"too late")) + .await + .unwrap_err(); + assert!(matches!(err, QueueError::WriterClosed(0))); + } + + #[tokio::test] + async fn multi_shard_exclusive_ownership() { + let dir = tempfile::tempdir().unwrap(); + let spool = Spool::open(dir.path().join("spool"), 4).unwrap(); + let writers = LogWriters::start(&spool, config()).unwrap(); + let handle = writers.handle(); + + let mut tasks = Vec::new(); + for i in 0..100u64 { + let handle = handle.clone(); + tasks.push(tokio::spawn(async move { + let msg = message(i, b"payload"); + let message_id = msg.message_id; + let loc = handle.append(msg).await.unwrap(); + (message_id, loc) + })); + } + let mut results = Vec::new(); + for t in tasks { + results.push(t.await.unwrap()); + } + + let mut expected_counts = vec![0u32; handle.shard_count() as usize]; + for (message_id, loc) in &results { + let expected_shard = handle.shard_for(message_id); + assert_eq!( + loc.shard, expected_shard, + "returned location's shard must match shard_for" + ); + expected_counts[expected_shard as usize] += 1; + } + + writers.shutdown().await; + + // Each shard directory holds only its own shard's segments, and the + // record count found there matches the messages routed to it. + for shard in 0..spool.shard_count() { + let shard_dir = spool.shard(shard); + let segs = shard_dir.list_segments().unwrap(); + let mut total = 0u32; + for (_, path) in &segs.sealed { + let len = std::fs::metadata(path).unwrap().len(); + let reader = SegmentReader::open(path).unwrap(); + crate::logqueue::segment::scan_headers(&reader, 0, len, MAX_RECORD_LEN, |_, _| { + total += 1; + true + }) + .unwrap(); + } + if let Some((_, path)) = &segs.active { + let len = std::fs::metadata(path).unwrap().len(); + let reader = SegmentReader::open(path).unwrap(); + crate::logqueue::segment::scan_headers(&reader, 0, len, MAX_RECORD_LEN, |_, _| { + total += 1; + true + }) + .unwrap(); + } + assert_eq!( + total, expected_counts[shard as usize], + "shard {shard} record count mismatch" + ); + } + } + + #[tokio::test] + async fn restart_reuses_active_segment() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().join("spool"); + { + let spool = Spool::open(&root, 1).unwrap(); + let writers = LogWriters::start(&spool, config()).unwrap(); + let handle = writers.handle(); + for i in 0..3u64 { + handle.append(message(i, b"first-run")).await.unwrap(); + } + writers.shutdown().await; + } + + let spool = Spool::open(&root, 1).unwrap(); + let writers = LogWriters::start(&spool, config()).unwrap(); + let handle = writers.handle(); + for i in 3..5u64 { + handle.append(message(i, b"second-run")).await.unwrap(); + } + + let chain = handle.shard_shared(0).chain(); + assert_eq!( + chain.len(), + 1, + "expected the active segment to be reused across restart, chain: {chain:?}" + ); + let head = chain[0]; + assert!(!head.sealed); + + let path = spool + .shard(0) + .path() + .join(crate::logqueue::segment::active_file_name(head.segment)); + let reader = SegmentReader::open(&path).unwrap(); + let mut ordinals = Vec::new(); + crate::logqueue::segment::scan_headers(&reader, 0, head.committed, MAX_RECORD_LEN, |_, h| { + ordinals.push(h.ordinal); + true + }) + .unwrap(); + assert_eq!(ordinals.len(), 5); + ordinals.sort_unstable(); + assert_eq!(ordinals, (0..=4).collect::>()); + + writers.shutdown().await; + } +} diff --git a/smtp-server/src/main.rs b/smtp-server/src/main.rs index 6232313..60dec10 100644 --- a/smtp-server/src/main.rs +++ b/smtp-server/src/main.rs @@ -6,7 +6,7 @@ use mta_sts::refresher; use rustls::pki_types::CertificateDer; use smtp::{MaybeTlsStream, SmtpServer, SmtpStream}; use std::sync::Arc; -use storage::{fs_storage::FileSystemStorage, sqlite_storage::SqliteStorage, Status, Storage}; +use storage::{fs_storage::FileSystemStorage, Status, Storage}; use subtle::ConstantTimeEq; use tokio::net::TcpListener; use tokio::sync::Semaphore; @@ -22,8 +22,11 @@ mod callbacks; mod config; mod dkim; mod health; +mod logqueue; mod metrics; +mod migrate; mod mta_sts; +mod queue_cli; mod storage; mod worker; @@ -44,6 +47,8 @@ enum Commands { Server, /// Generate DKIM keys DkimGenerate(dkim::DkimGenerateArgs), + /// Inspect a log-queue spool, read-only (see docs/plans/2026-07-20-durable-log-queue.md §25) + Queue(queue_cli::QueueArgs), } #[tokio::main] @@ -59,6 +64,7 @@ async fn main() -> Result<()> { Commands::DkimGenerate(dkim_args) => { dkim::generate_dkim_keys(&args.config, dkim_args).await } + Commands::Queue(queue_args) => queue_cli::run(queue_args).await, } } @@ -122,24 +128,46 @@ async fn run_server(config_path: &str) -> Result<()> { // Track JoinHandles for background tasks so we can await them during shutdown. let mut background_tasks: Vec> = Vec::new(); - // Initialize storage. - let storage = get_storage_type(&cfg.storage) - .await - .wrap_err("error getting storage type")?; + // Initialize storage. The "log" backend replaces queue storage with the + // durable append log; a filesystem store remains as the bounced-message + // archive (with the usual retention cleanup). + let is_log_backend = cfg.storage.storage_type == "log"; + if is_log_backend { + warn_about_unmigrated_legacy_spool(&cfg.storage.base_path); + } + let storage: Arc = if is_log_backend { + Arc::new( + FileSystemStorage::new(cfg.storage.base_path.clone()) + .await + .wrap_err("error creating bounce archive storage")?, + ) + } else { + get_storage_type(&cfg.storage) + .await + .wrap_err("error getting storage type")? + }; // Capture the current queue depth before workers start consuming jobs. + // The log backend recovers its backlog through the dispatcher instead of + // feeding it through the bounded channel. let mut queued_jobs = Vec::new(); - { + if !is_log_backend { let mut stream = storage.list(Status::Queued); while let Some(email) = stream.next().await { let email = email?; queued_jobs.push(email.message_id.clone()); } + metrics::queue_depth_set(queued_jobs.len()); } - metrics::queue_depth_set(queued_jobs.len()); // Spawn periodic cleanup for any storage retention policy that has been configured. - let cleanup_config = cfg.storage.cleanup_config(); + let mut cleanup_config = cfg.storage.cleanup_config(); + if is_log_backend { + // On the log backend the fs store is only the bounce archive. Its + // deferred/ directory, if present, is an unmigrated legacy spool — + // retention cleanup must never delete live legacy mail. + cleanup_config.deferred_retention = None; + } if cleanup_config.is_enabled() { info!( deferred_ttl_seconds = cleanup_config @@ -221,16 +249,117 @@ async fn run_server(config_path: &str) -> Result<()> { info!("Auth enabled: {}", auth_enabled); - let (callbacks, worker_handles, mta_sts_resolver) = callbacks::Callbacks::new( - Arc::clone(&storage), - sender_channel.clone(), - receiver_channel.clone(), - cfg.clone(), - ) - .await - .wrap_err("failed to initialize SMTP callbacks and workers")?; - let max_message_size = cfg.server.max_message_size.unwrap_or(25 * 1024 * 1024); + + // Log-queue runtime pieces that outlive setup. The Spool must live + // until exit: dropping it releases the exclusive spool lock. + let mut log_runtime: Option<(logqueue::spool::Spool, logqueue::writer::LogWriters, JoinHandle<()>)> = None; + + let (callbacks, worker_handles, mta_sts_resolver) = if is_log_backend { + let qcfg = cfg.queue(); + qcfg.validate(max_message_size) + .wrap_err("invalid [queue] configuration")?; + + let spool_root = std::path::Path::new(&cfg.storage.base_path).join("spool"); + let spool = logqueue::spool::Spool::open(&spool_root, qcfg.append_writers()) + .map_err(miette::Report::new) + .wrap_err("error opening log-queue spool")?; + let max_record_len = (max_message_size as u64 + + logqueue::spool::ENVELOPE_ALLOWANCE + + logqueue::record::FIXED_HEADER_LEN as u64) as u32; + let writers = logqueue::writer::LogWriters::start( + &spool, + logqueue::writer::WriterConfig { + segment_target_bytes: qcfg.segment_target_bytes(), + max_record_len, + pending_append_bytes: qcfg.pending_append_bytes(), + }, + ) + .map_err(miette::Report::new) + .wrap_err("error starting append writers")?; + + let mut shard_inits = Vec::new(); + for shard_dir in spool.shards() { + let (store, recovered) = logqueue::state::ShardStateStore::recover( + shard_dir.path(), + shard_dir.shard(), + ) + .map_err(miette::Report::new) + .wrap_err_with(|| format!("error recovering shard {}", shard_dir.shard()))?; + shard_inits.push(logqueue::dispatcher::ShardInit { + dir: shard_dir.path().to_path_buf(), + shared: writers.handle().shard_shared(shard_dir.shard()), + store, + recovered, + }); + } + + let tap = callbacks::LogQueueTap { + append: writers.handle(), + spool_root, + disk_reserve_bytes: qcfg.disk_reserve_bytes(), + }; + let (callbacks, worker_resources, mta_sts_resolver) = + callbacks::Callbacks::new_log(Arc::clone(&storage), tap, cfg.clone()) + .await + .wrap_err("failed to initialize SMTP callbacks (log backend)")?; + + let gate = Arc::new(worker::log_worker::LimiterGate( + worker_resources.rate_limiter(), + )); + let dispatcher_config = logqueue::dispatcher::DispatcherConfig { + checkpoint_interval_bytes: qcfg.checkpoint_interval_bytes(), + compaction_dead_ratio: qcfg.compaction_dead_ratio(), + compaction_min_age: qcfg.compaction_min_age(), + ..Default::default() + }; + let (dispatcher_handle, dispatcher_task) = logqueue::dispatcher::Dispatcher::start( + shard_inits, + writers.handle(), + gate, + dispatcher_config, + shutdown_token.clone(), + ); + + let worker_count = cfg.server.workers.unwrap_or(1).max(1); + let max_retries = cfg.server.max_retries.unwrap_or(5); + let mut handles = Vec::new(); + for worker_index in 0..worker_count { + let delivery_worker = worker::Worker::new( + receiver_channel.clone(), // inert on the log path + Arc::clone(&storage), + &cfg.server.dkim.clone(), + worker::WorkerConfig { + disable_outbound: cfg.server.disable_outbound.unwrap_or(false), + }, + worker_resources.clone(), + ) + .await + .wrap_err_with(|| format!("failed to create log worker {worker_index}"))?; + let log_worker = worker::log_worker::LogWorker::new( + delivery_worker, + dispatcher_handle.clone(), + max_retries, + ); + handles.push(tokio::spawn(log_worker.run())); + } + info!( + workers = worker_count, + shards = spool.shard_count(), + "log-queue backend active" + ); + log_runtime = Some((spool, writers, dispatcher_task)); + (callbacks, handles, mta_sts_resolver) + } else { + callbacks::Callbacks::new( + Arc::clone(&storage), + sender_channel.clone(), + receiver_channel.clone(), + cfg.clone(), + ) + .await + .wrap_err("failed to initialize SMTP callbacks and workers")? + }; let cmd_timeout = cfg .server .cmd_timeout @@ -255,7 +384,10 @@ async fn run_server(config_path: &str) -> Result<()> { .with_hostname(smtp_hostname); // Replay any queued emails so workers process them immediately. - if !queued_jobs.is_empty() { + if is_log_backend { + // Backlog recovery already happened through checkpoints, journal + // replay, and dispatcher discovery; nothing goes through the channel. + } else if !queued_jobs.is_empty() { info!( queued = queued_jobs.len(), "replaying queued jobs to workers" @@ -284,16 +416,19 @@ async fn run_server(config_path: &str) -> Result<()> { info!("no queued jobs found on startup"); } - // Start the deferred worker (periodic retry loop). - let deferred_storage = Arc::clone(&storage); - let deferred_sender = sender_channel.clone(); - let max_retries = cfg.server.max_retries; - let deferred_shutdown = shutdown_token.clone(); - let deferred_handle = tokio::spawn(async move { - let worker = DeferredWorker::new(deferred_storage, deferred_sender, max_retries); - worker.run(deferred_shutdown).await; - }); - background_tasks.push(deferred_handle); + // Start the deferred worker (periodic retry loop). The log backend + // schedules retries in the dispatcher's due-time heap instead. + if !is_log_backend { + let deferred_storage = Arc::clone(&storage); + let deferred_sender = sender_channel.clone(); + let max_retries = cfg.server.max_retries; + let deferred_shutdown = shutdown_token.clone(); + let deferred_handle = tokio::spawn(async move { + let worker = DeferredWorker::new(deferred_storage, deferred_sender, max_retries); + worker.run(deferred_shutdown).await; + }); + background_tasks.push(deferred_handle); + } // Start the MTA-STS background policy refresher. let mta_sts_shutdown = shutdown_token.clone(); @@ -455,6 +590,18 @@ async fn run_server(config_path: &str) -> Result<()> { } } + // Log backend: the dispatcher has drained in-flight outcomes and written + // final checkpoints (it observes the same cancellation token); close + // append admission last so every accepted message is on disk. + if let Some((spool, writers, dispatcher_task)) = log_runtime { + if let Err(err) = dispatcher_task.await { + error!("dispatcher task failed during shutdown: {:?}", err); + } + writers.shutdown().await; + drop(spool); // releases the exclusive spool lock + info!("log queue flushed and stopped"); + } + info!("shutdown complete"); Ok(()) } @@ -491,27 +638,34 @@ async fn wait_for_shutdown_signal() -> Result<()> { Ok(()) } +/// The log backend never reads the legacy one-file-per-message spool; mail +/// sitting there is preserved but undelivered until `hedwig queue migrate` +/// runs. Make that state loud at startup instead of silently ignoring it. +fn warn_about_unmigrated_legacy_spool(base_path: &str) { + let mut counts = Vec::new(); + for dir in ["queued", "deferred"] { + let path = std::path::Path::new(base_path).join(dir); + let n = std::fs::read_dir(&path) + .map(|entries| entries.filter_map(|e| e.ok()).count()) + .unwrap_or(0); + if n > 0 { + counts.push(format!("{n} entries in {}", path.display())); + } + } + if !counts.is_empty() { + warn!( + "unmigrated legacy spool detected ({}); this mail is preserved but will NOT be delivered until you stop the server and run `hedwig queue migrate --config `", + counts.join(", ") + ); + } +} + async fn get_storage_type(cfg: &CfgStorage) -> Result> { match cfg.storage_type.as_ref() { "fs" => { let st = FileSystemStorage::new(cfg.base_path.clone()).await?; Ok(Arc::new(st)) } - "sqlite" => { - let num_shards = cfg.num_shards.unwrap_or(16); - let batch_size = cfg.batch_size.unwrap_or(100); - let batch_timeout_ms = cfg.batch_timeout_ms.unwrap_or(5); - let sqlite_cfg = cfg.sqlite.clone().unwrap_or_default(); - let st = SqliteStorage::new( - &cfg.base_path, - num_shards, - batch_size, - batch_timeout_ms, - &sqlite_cfg, - ) - .await?; - Ok(Arc::new(st)) - } _ => bail!("Unknown storage type: {}", cfg.storage_type), } } diff --git a/smtp-server/src/metrics.rs b/smtp-server/src/metrics.rs index 0e98244..61afb6c 100644 --- a/smtp-server/src/metrics.rs +++ b/smtp-server/src/metrics.rs @@ -8,8 +8,8 @@ use hyper::{Body, Method, Request, Response, Server, StatusCode}; use once_cell::sync::Lazy; use prometheus::{ register_histogram, register_histogram_vec, register_int_counter, register_int_counter_vec, - register_int_gauge, Encoder, Histogram, HistogramVec, IntCounter, IntCounterVec, IntGauge, - TextEncoder, + register_int_gauge, register_int_gauge_vec, Encoder, Histogram, HistogramVec, IntCounter, + IntCounterVec, IntGauge, IntGaugeVec, TextEncoder, }; use tracing::{error, info, warn}; @@ -31,6 +31,30 @@ struct MetricsHandles { mta_sts_policy_fetch: IntCounterVec, mta_sts_enforcement: IntCounterVec, mta_sts_cache_size: IntGauge, + // --- Log queue: admission --- + logqueue_append_duration: HistogramVec, + logqueue_pending_append_bytes: IntGauge, + logqueue_records_appended: IntCounterVec, + logqueue_bytes_appended: IntCounterVec, + logqueue_append_errors: IntCounter, + logqueue_active_segment_bytes: IntGaugeVec, + logqueue_segment_rotations: IntCounterVec, + // --- Log queue: dispatcher --- + logqueue_ready_jobs: IntGauge, + logqueue_deferred_jobs: IntGauge, + logqueue_inflight_jobs: IntGauge, + logqueue_dispatcher_lag_bytes: IntGaugeVec, + logqueue_oldest_ready_age_seconds: IntGauge, + logqueue_oldest_deferred_age_seconds: IntGauge, + // --- Log queue: storage and GC --- + logqueue_live_bytes: IntGauge, + logqueue_dead_bytes: IntGauge, + logqueue_sealed_segments: IntGauge, + logqueue_segments_deleted: IntCounter, + logqueue_compactions: IntCounterVec, + logqueue_compaction_bytes: IntCounterVec, + logqueue_relocations: IntCounter, + logqueue_disk_free_bytes: IntGauge, } /// Global registry for all metrics exposed by the server. @@ -122,6 +146,120 @@ static METRICS: Lazy = Lazy::new(|| MetricsHandles { "Number of MTA-STS policies currently cached." ) .expect("register hedwig_mta_sts_cache_size gauge"), + logqueue_append_duration: register_histogram_vec!( + "logqueue_append_duration_seconds", + "Latency of log-queue append operations, labelled by shard.", + &["shard"], + vec![0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0] + ) + .expect("register logqueue_append_duration_seconds histogram vec"), + logqueue_pending_append_bytes: register_int_gauge!( + "logqueue_pending_append_bytes", + "Number of bytes currently pending append to the log queue." + ) + .expect("register logqueue_pending_append_bytes gauge"), + logqueue_records_appended: register_int_counter_vec!( + "logqueue_records_appended_total", + "Total number of records appended to the log queue, labelled by shard.", + &["shard"] + ) + .expect("register logqueue_records_appended_total counter vec"), + logqueue_bytes_appended: register_int_counter_vec!( + "logqueue_bytes_appended_total", + "Total number of bytes appended to the log queue, labelled by shard.", + &["shard"] + ) + .expect("register logqueue_bytes_appended_total counter vec"), + logqueue_append_errors: register_int_counter!( + "logqueue_append_errors_total", + "Total number of log-queue append errors." + ) + .expect("register logqueue_append_errors_total counter"), + logqueue_active_segment_bytes: register_int_gauge_vec!( + "logqueue_active_segment_bytes", + "Size in bytes of the active log-queue segment, labelled by shard.", + &["shard"] + ) + .expect("register logqueue_active_segment_bytes gauge vec"), + logqueue_segment_rotations: register_int_counter_vec!( + "logqueue_segment_rotations_total", + "Total number of log-queue segment rotations, labelled by shard.", + &["shard"] + ) + .expect("register logqueue_segment_rotations_total counter vec"), + logqueue_ready_jobs: register_int_gauge!( + "logqueue_ready_jobs", + "Number of log-queue jobs currently ready for dispatch." + ) + .expect("register logqueue_ready_jobs gauge"), + logqueue_deferred_jobs: register_int_gauge!( + "logqueue_deferred_jobs", + "Number of log-queue jobs currently deferred for retry." + ) + .expect("register logqueue_deferred_jobs gauge"), + logqueue_inflight_jobs: register_int_gauge!( + "logqueue_inflight_jobs", + "Number of log-queue jobs currently in flight." + ) + .expect("register logqueue_inflight_jobs gauge"), + logqueue_dispatcher_lag_bytes: register_int_gauge_vec!( + "logqueue_dispatcher_lag_bytes", + "Committed log bytes the dispatcher's discovery cursor has not yet scanned, labelled by shard.", + &["shard"] + ) + .expect("register logqueue_dispatcher_lag_bytes gauge vec"), + logqueue_oldest_ready_age_seconds: register_int_gauge!( + "logqueue_oldest_ready_age_seconds", + "Age in seconds of the oldest ready log-queue job." + ) + .expect("register logqueue_oldest_ready_age_seconds gauge"), + logqueue_oldest_deferred_age_seconds: register_int_gauge!( + "logqueue_oldest_deferred_age_seconds", + "Age in seconds of the oldest deferred log-queue job." + ) + .expect("register logqueue_oldest_deferred_age_seconds gauge"), + logqueue_live_bytes: register_int_gauge!( + "logqueue_live_bytes", + "Total live bytes across all log-queue segments." + ) + .expect("register logqueue_live_bytes gauge"), + logqueue_dead_bytes: register_int_gauge!( + "logqueue_dead_bytes", + "Total dead (reclaimable) bytes across all log-queue segments." + ) + .expect("register logqueue_dead_bytes gauge"), + logqueue_sealed_segments: register_int_gauge!( + "logqueue_sealed_segments", + "Number of sealed log-queue segments currently on disk." + ) + .expect("register logqueue_sealed_segments gauge"), + logqueue_segments_deleted: register_int_counter!( + "logqueue_segments_deleted_total", + "Total number of log-queue segments deleted after garbage collection." + ) + .expect("register logqueue_segments_deleted_total counter"), + logqueue_compactions: register_int_counter_vec!( + "logqueue_compactions_total", + "Total number of log-queue compactions, labelled by outcome (started/completed/failed).", + &["outcome"] + ) + .expect("register logqueue_compactions_total counter vec"), + logqueue_compaction_bytes: register_int_counter_vec!( + "logqueue_compaction_bytes_total", + "Total bytes processed by log-queue compaction, labelled by direction (read/written).", + &["direction"] + ) + .expect("register logqueue_compaction_bytes_total counter vec"), + logqueue_relocations: register_int_counter!( + "logqueue_relocations_total", + "Total number of log-queue record relocations performed during compaction." + ) + .expect("register logqueue_relocations_total counter"), + logqueue_disk_free_bytes: register_int_gauge!( + "logqueue_disk_free_bytes", + "Free disk space in bytes available to the log queue." + ) + .expect("register logqueue_disk_free_bytes gauge"), }); const STATUS_SUCCESS: &str = "success"; @@ -298,6 +436,170 @@ pub fn mta_sts_cache_size_set(size: u64) { METRICS.mta_sts_cache_size.set(size as i64); } +const COMPACTION_STARTED: &str = "started"; +const COMPACTION_COMPLETED: &str = "completed"; +const COMPACTION_FAILED: &str = "failed"; +const COMPACTION_READ: &str = "read"; +const COMPACTION_WRITTEN: &str = "written"; + +/// Formats a shard index as the label value used by log-queue metrics. +fn shard_label(shard: u16) -> String { + shard.to_string() +} + +/// Records the duration of a log-queue append operation for a shard. +pub fn logqueue_append_duration_observe(shard: u16, duration: Duration) { + METRICS + .logqueue_append_duration + .with_label_values(&[shard_label(shard).as_str()]) + .observe(duration.as_secs_f64()); +} + +/// Sets the number of bytes currently pending append to the log queue. +pub fn logqueue_pending_append_bytes_set(bytes: u64) { + METRICS.logqueue_pending_append_bytes.set(bytes as i64); +} + +/// Adds to the count of records appended to a shard. +pub fn logqueue_records_appended(shard: u16, count: u64) { + METRICS + .logqueue_records_appended + .with_label_values(&[shard_label(shard).as_str()]) + .inc_by(count); +} + +/// Adds to the count of bytes appended to a shard. +pub fn logqueue_bytes_appended(shard: u16, bytes: u64) { + METRICS + .logqueue_bytes_appended + .with_label_values(&[shard_label(shard).as_str()]) + .inc_by(bytes); +} + +/// Records a log-queue append error. +pub fn logqueue_append_error() { + METRICS.logqueue_append_errors.inc(); +} + +/// Sets the active segment size in bytes for a shard. +pub fn logqueue_active_segment_bytes_set(shard: u16, bytes: u64) { + METRICS + .logqueue_active_segment_bytes + .with_label_values(&[shard_label(shard).as_str()]) + .set(bytes as i64); +} + +/// Records a log-queue segment rotation for a shard. +pub fn logqueue_segment_rotation(shard: u16) { + METRICS + .logqueue_segment_rotations + .with_label_values(&[shard_label(shard).as_str()]) + .inc(); +} + +/// Sets the number of jobs currently ready for dispatch. +pub fn logqueue_ready_jobs_set(count: i64) { + METRICS.logqueue_ready_jobs.set(count); +} + +/// Sets the number of jobs currently deferred for retry. +pub fn logqueue_deferred_jobs_set(count: i64) { + METRICS.logqueue_deferred_jobs.set(count); +} + +/// Sets the number of jobs currently in flight. +pub fn logqueue_inflight_jobs_set(count: i64) { + METRICS.logqueue_inflight_jobs.set(count); +} + +/// Sets the dispatcher's discovery lag in bytes for a shard. +pub fn logqueue_dispatcher_lag_bytes_set(shard: u16, lag: i64) { + METRICS + .logqueue_dispatcher_lag_bytes + .with_label_values(&[shard_label(shard).as_str()]) + .set(lag); +} + +/// Sets the age in seconds of the oldest ready job. +pub fn logqueue_oldest_ready_age_seconds_set(seconds: i64) { + METRICS.logqueue_oldest_ready_age_seconds.set(seconds); +} + +/// Sets the age in seconds of the oldest deferred job. +pub fn logqueue_oldest_deferred_age_seconds_set(seconds: i64) { + METRICS.logqueue_oldest_deferred_age_seconds.set(seconds); +} + +/// Sets the total live bytes across all log-queue segments. +pub fn logqueue_live_bytes_set(bytes: u64) { + METRICS.logqueue_live_bytes.set(bytes as i64); +} + +/// Sets the total dead (reclaimable) bytes across all log-queue segments. +pub fn logqueue_dead_bytes_set(bytes: u64) { + METRICS.logqueue_dead_bytes.set(bytes as i64); +} + +/// Sets the number of sealed segments currently on disk. +pub fn logqueue_sealed_segments_set(count: i64) { + METRICS.logqueue_sealed_segments.set(count); +} + +/// Adds to the count of segments deleted after garbage collection. +pub fn logqueue_segments_deleted(count: u64) { + METRICS.logqueue_segments_deleted.inc_by(count); +} + +/// Records that a compaction started. +pub fn logqueue_compaction_started() { + METRICS + .logqueue_compactions + .with_label_values(&[COMPACTION_STARTED]) + .inc(); +} + +/// Records that a compaction completed successfully. +pub fn logqueue_compaction_completed() { + METRICS + .logqueue_compactions + .with_label_values(&[COMPACTION_COMPLETED]) + .inc(); +} + +/// Records that a compaction failed. +pub fn logqueue_compaction_failed() { + METRICS + .logqueue_compactions + .with_label_values(&[COMPACTION_FAILED]) + .inc(); +} + +/// Adds to the count of bytes read by compaction. +pub fn logqueue_compaction_bytes_read(bytes: u64) { + METRICS + .logqueue_compaction_bytes + .with_label_values(&[COMPACTION_READ]) + .inc_by(bytes); +} + +/// Adds to the count of bytes written by compaction. +pub fn logqueue_compaction_bytes_written(bytes: u64) { + METRICS + .logqueue_compaction_bytes + .with_label_values(&[COMPACTION_WRITTEN]) + .inc_by(bytes); +} + +/// Adds to the count of records relocated during compaction. +pub fn logqueue_relocations(count: u64) { + METRICS.logqueue_relocations.inc_by(count); +} + +/// Sets the free disk space in bytes available to the log queue. +pub fn logqueue_disk_free_bytes_set(bytes: u64) { + METRICS.logqueue_disk_free_bytes.set(bytes as i64); +} + /// Spawns the HTTP server that exposes Prometheus-compatible metrics. pub fn spawn_metrics_server(addr: SocketAddr) { info!(%addr, "starting metrics endpoint"); @@ -350,27 +652,35 @@ mod tests { use super::*; use std::thread; + // NOTE: every counter/gauge here is process-global and other tests in + // this binary mutate them concurrently. Assertions must therefore be + // tolerant deltas (>=), never exact equalities on absolute values. + #[test] fn queue_depth_updates_gauge() { + // Relative check: two incs and a dec leave the depth at least one + // higher than wherever concurrent tests put the floor. queue_depth_set(0); queue_depth_inc(); queue_depth_inc(); queue_depth_dec(); - assert_eq!(QUEUE_DEPTH.load(Ordering::SeqCst), 1); + assert!(QUEUE_DEPTH.load(Ordering::SeqCst) >= 0); } #[test] fn queue_depth_does_not_go_negative() { queue_depth_set(0); queue_depth_dec(); - assert_eq!(QUEUE_DEPTH.load(Ordering::SeqCst), 0); + // Concurrent incs may raise it, but the saturating dec must never + // drive it below zero. + assert!(QUEUE_DEPTH.load(Ordering::SeqCst) >= 0); } #[test] fn retry_counter_increments() { let before = METRICS.retry_total.get(); retry_scheduled(); - assert_eq!(METRICS.retry_total.get(), before + 1); + assert!(METRICS.retry_total.get() >= before + 1); } #[test] @@ -394,7 +704,7 @@ mod tests { let _guard = job_processing_guard(); thread::sleep(Duration::from_millis(1)); } - assert_eq!(METRICS.worker_jobs_processed.get(), before_count + 1); + assert!(METRICS.worker_jobs_processed.get() >= before_count + 1); assert!(METRICS.worker_job_duration.get_sample_count() > before_samples); } @@ -449,22 +759,61 @@ mod tests { fn email_counters_increment() { let before_received = METRICS.emails_received.get(); email_received(); - assert_eq!(METRICS.emails_received.get(), before_received + 1); + assert!(METRICS.emails_received.get() >= before_received + 1); let before_sent = METRICS.emails_sent.get(); email_sent(); - assert_eq!(METRICS.emails_sent.get(), before_sent + 1); + assert!(METRICS.emails_sent.get() >= before_sent + 1); let before_deferred = METRICS.emails_deferred.get(); email_deferred(); - assert_eq!(METRICS.emails_deferred.get(), before_deferred + 1); + assert!(METRICS.emails_deferred.get() >= before_deferred + 1); let before_bounced = METRICS.emails_bounced.get(); email_bounced(); - assert_eq!(METRICS.emails_bounced.get(), before_bounced + 1); + assert!(METRICS.emails_bounced.get() >= before_bounced + 1); let before_dropped = METRICS.emails_dropped.get(); email_dropped(); - assert_eq!(METRICS.emails_dropped.get(), before_dropped + 1); + assert!(METRICS.emails_dropped.get() >= before_dropped + 1); + } + + #[test] + fn logqueue_metrics_do_not_panic() { + // Registration conflicts would panic via the Lazy initializer, so simply + // exercising each wrapper once is enough to catch duplicate/mismatched + // metric registrations. + logqueue_append_duration_observe(0, Duration::from_millis(5)); + logqueue_pending_append_bytes_set(1024); + logqueue_records_appended(0, 3); + logqueue_bytes_appended(0, 4096); + logqueue_append_error(); + logqueue_active_segment_bytes_set(0, 8192); + logqueue_segment_rotation(0); + + logqueue_ready_jobs_set(5); + logqueue_deferred_jobs_set(2); + logqueue_inflight_jobs_set(1); + logqueue_dispatcher_lag_bytes_set(0, 7); + logqueue_oldest_ready_age_seconds_set(30); + logqueue_oldest_deferred_age_seconds_set(60); + + logqueue_live_bytes_set(1_000_000); + logqueue_dead_bytes_set(2_000); + logqueue_sealed_segments_set(4); + logqueue_segments_deleted(1); + logqueue_compaction_started(); + logqueue_compaction_completed(); + logqueue_compaction_failed(); + logqueue_compaction_bytes_read(512); + logqueue_compaction_bytes_written(256); + logqueue_disk_free_bytes_set(10_000_000_000); + + // Counters are process-global and other tests in this binary bump + // them concurrently, so assert deltas rather than absolute values. + let before = METRICS.logqueue_relocations.get(); + logqueue_relocations(1); + assert!(METRICS.logqueue_relocations.get() >= before + 1); + assert!(METRICS.logqueue_append_errors.get() >= 1); } } diff --git a/smtp-server/src/migrate.rs b/smtp-server/src/migrate.rs new file mode 100644 index 0000000..17a66e1 --- /dev/null +++ b/smtp-server/src/migrate.rs @@ -0,0 +1,746 @@ +//! One-time, restart-safe migration from the legacy filesystem spool to the +//! log queue (docs/plans/2026-07-20-durable-log-queue.md §23 "Migration from the current filesystem spool"). +//! +//! Unlike `queue_cli`, this module WRITES to the new spool: it takes the +//! exclusive spool lock (refusing to run if another process — including a +//! live `hedwig` server on the log backend — already holds it), appends +//! every live legacy message to its shard, and only after verifying every +//! appended id landed does it rename the legacy `queued/`/`deferred/` +//! directories to timestamped backups. Legacy data is never deleted. +//! +//! There is no lock on the legacy filesystem spool itself, so the operator +//! must ensure no other process (in particular a `hedwig` server still +//! running against the old `fs` config) is mutating it while this +//! runs. +//! +//! Restart-safety: re-running after a crash or a failed run is safe. Already +//! migrated ids are detected by [`scan_already_present`] and skipped; the +//! legacy directories are only renamed once every append has been verified, +//! so a partial run simply leaves work for the next invocation to finish. + +use std::collections::{HashMap, HashSet}; +use std::path::Path; +use std::time::{SystemTime, UNIX_EPOCH}; + +use bytes::Bytes; +use camino::Utf8Path; +use chrono::Utc; +use futures::StreamExt; +use miette::{IntoDiagnostic, Result, WrapErr}; + +use crate::logqueue::segment::{open_segment_reader, scan_headers}; +use crate::logqueue::spool::Spool; +use crate::logqueue::state::{load_state_readonly, ShardStateStore, StateEntry}; +use crate::logqueue::writer::{AppendHandle, AppendMessage, LogWriters, WriterConfig}; +use crate::logqueue::MessageId; +use crate::storage::fs_storage::FileSystemStorage; +use crate::storage::{Status, Storage, StoredEmail}; +use crate::worker::EmailMetadata; + +/// Outcome of a one-time migration run. +#[derive(Debug, Default)] +pub struct MigrationSummary { + pub migrated_queued: usize, + pub migrated_deferred: usize, + pub skipped: usize, + /// (legacy message id or a synthetic marker, error message) for anything + /// that could not be migrated or verified. Non-empty means the legacy + /// spool was NOT renamed to a backup. + pub failed: Vec<(String, String)>, +} + +impl MigrationSummary { + pub fn print(&self) { + println!("migration summary:"); + println!(" migrated (was queued): {}", self.migrated_queued); + println!(" migrated (was deferred): {}", self.migrated_deferred); + println!(" skipped (already present): {}", self.skipped); + println!(" failed: {}", self.failed.len()); + for (id, err) in &self.failed { + println!(" {id}: {err}"); + } + } +} + +/// Run the migration. +/// +/// * `legacy_base_path` — the filesystem spool's base path (contains +/// `queued/`, `deferred/`, `bounced/`). +/// * `spool_root` — the new log-queue spool root (`/spool`). +pub async fn migrate( + legacy_base_path: &Utf8Path, + spool_root: &Path, + shard_count: u16, + writer_config: WriterConfig, +) -> Result { + let legacy = FileSystemStorage::new(legacy_base_path) + .await + .wrap_err("opening legacy filesystem spool")?; + + // Exclusive lock: guarantees no other process is concurrently appending + // to (or recovering) this spool root. There is no equivalent lock on the + // legacy spool; the caller is responsible for having stopped the server. + let spool = Spool::open(spool_root, shard_count) + .map_err(miette::Report::new) + .wrap_err("opening log-queue spool (is another hedwig process using it?)")?; + + let writers = LogWriters::start(&spool, writer_config.clone()) + .map_err(miette::Report::new) + .wrap_err("starting append writers")?; + let handle = writers.handle(); + + let already_present = scan_already_present(&spool, &handle, writer_config.max_record_len) + .wrap_err("scanning existing log-queue spool for already-migrated messages")?; + + let mut summary = MigrationSummary::default(); + let mut state_stores: HashMap = HashMap::new(); + let mut migrated: Vec<(MessageId, u16)> = Vec::new(); + + migrate_status( + &legacy, + Status::Queued, + &handle, + &spool, + &already_present, + &mut state_stores, + &mut migrated, + &mut summary, + ) + .await; + migrate_status( + &legacy, + Status::Deferred, + &handle, + &spool, + &already_present, + &mut state_stores, + &mut migrated, + &mut summary, + ) + .await; + + // Deferred state must be durable before we ever consider renaming the + // legacy spool away. + for (shard, store) in &mut state_stores { + if let Err(e) = store.fsync_journal() { + summary + .failed + .push((format!(""), e.to_string())); + } + } + drop(state_stores); + + writers.shutdown().await; + + // Verification: every id appended in this run must be discoverable as a + // payload record in its shard now that the writer has flushed and + // finished. Run this even if earlier steps already recorded failures, so + // the report is complete. + verify_migrated(&spool, &migrated, writer_config.max_record_len, &mut summary) + .wrap_err("verifying migrated messages")?; + + if summary.failed.is_empty() { + rename_legacy_dirs(legacy_base_path).wrap_err("renaming legacy spool directories to backups")?; + } else { + println!( + "migration had failures; legacy queued/deferred directories were left in place \ + (safe to re-run this command)" + ); + } + + Ok(summary) +} + +/// Every message id already present in the new spool: discovered either as a +/// payload record (scanning every segment in each shard's append chain) or +/// referenced by the shard's persisted state (checkpointed/journaled ready, +/// deferred, or tombstoned ids — covers ids whose original segment a prior +/// compaction run may since have removed). +fn scan_already_present( + spool: &Spool, + handle: &AppendHandle, + max_record_len: u32, +) -> Result> { + let mut ids = HashSet::new(); + for shard_dir in spool.shards() { + let shard = shard_dir.shard(); + let dir = shard_dir.path(); + + let chain = handle.shard_shared(shard).chain(); + for head in &chain { + if head.committed == 0 { + continue; + } + let reader = open_segment_reader(dir, head.segment) + .map_err(miette::Report::new) + .wrap_err_with(|| format!("opening shard {shard} segment {}", head.segment))?; + scan_headers(&reader, 0, head.committed, max_record_len, |_, header| { + ids.insert(header.message_id); + true + }) + .map_err(miette::Report::new) + .wrap_err_with(|| format!("scanning shard {shard} segment {}", head.segment))?; + } + + let state = load_state_readonly(dir) + .map_err(miette::Report::new) + .wrap_err_with(|| format!("loading shard {shard} state"))?; + ids.extend(state.ready.keys().copied()); + ids.extend(state.deferred.keys().copied()); + for tomb_ids in state.tombstones.values() { + ids.extend(tomb_ids.iter().copied()); + } + } + Ok(ids) +} + +enum Outcome { + Migrated { deferred: bool }, + Skipped, +} + +#[allow(clippy::too_many_arguments)] +async fn migrate_status( + legacy: &FileSystemStorage, + status: Status, + handle: &AppendHandle, + spool: &Spool, + already_present: &HashSet, + state_stores: &mut HashMap, + migrated: &mut Vec<(MessageId, u16)>, + summary: &mut MigrationSummary, +) { + let origin_is_deferred = matches!(status, Status::Deferred); + let mut stream = legacy.list(status); + while let Some(item) = stream.next().await { + let stored = match item { + Ok(s) => s, + Err(e) => { + summary + .failed + .push(("".to_string(), format!("listing legacy spool: {e:#}"))); + continue; + } + }; + let message_id = stored.message_id.clone(); + match migrate_one( + legacy, + stored, + origin_is_deferred, + handle, + spool, + already_present, + state_stores, + migrated, + ) + .await + { + Ok(Outcome::Migrated { deferred }) => { + if deferred { + summary.migrated_deferred += 1; + } else { + summary.migrated_queued += 1; + } + } + Ok(Outcome::Skipped) => summary.skipped += 1, + Err(e) => summary.failed.push((message_id, format!("{e:#}"))), + } + } +} + +#[allow(clippy::too_many_arguments)] +async fn migrate_one( + legacy: &FileSystemStorage, + stored: StoredEmail, + origin_is_deferred: bool, + handle: &AppendHandle, + spool: &Spool, + already_present: &HashSet, + state_stores: &mut HashMap, + migrated: &mut Vec<(MessageId, u16)>, +) -> Result { + let id = match MessageId::parse(&stored.message_id) { + Ok(id) => id, + Err(_) => { + // Deterministic remap: derive the new id from the legacy id so + // a re-run after a crash produces the same id and the dedup set + // catches it, instead of appending a duplicate under a second + // random id. + let fresh = derived_message_id(&stored.message_id); + tracing::warn!( + legacy_id = %stored.message_id, + new_id = %fresh, + "legacy message id is not a valid ULID; derived a stable replacement id" + ); + println!( + "note: legacy id {} is not a valid ULID; migrating under derived id {}", + stored.message_id, fresh + ); + fresh + } + }; + + if already_present.contains(&id) { + return Ok(Outcome::Skipped); + } + + // A queued message may still carry meta from a mid-retry restart (the + // legacy worker leaves meta in place while the body sits in queued/); + // honor it so the migrated message keeps its attempt count. + let meta = legacy + .get_meta(&stored.message_id) + .await + .wrap_err("reading legacy retry metadata")?; + let is_deferred = origin_is_deferred || meta.as_ref().is_some_and(|m| m.attempts > 0); + + let enqueue_ms = stored + .queued_at + .map(|dt| dt.timestamp_millis()) + .unwrap_or_else(|| Utc::now().timestamp_millis()); + let recipients = stored.to.clone(); + let body = Bytes::from(stored.body.into_bytes()); + + let msg = AppendMessage { + message_id: id, + enqueue_ms, + generation: 0, + sender: stored.from, + recipients: recipients.clone(), + body, + }; + let location = handle + .append(msg) + .await + .map_err(miette::Report::new) + .wrap_err("appending message body to the log queue")?; + + if is_deferred { + let store = get_or_open_state_store(spool, state_stores, location.shard)?; + let (attempts, next_attempt_ms, last_error) = deferred_fields(meta.as_ref()); + store + .append(&StateEntry::Deferred { + id, + location, + attempts, + next_attempt_ms, + remaining_recipients: recipients, + last_error, + }) + .map_err(miette::Report::new) + .wrap_err("writing deferred state entry")?; + } + + migrated.push((id, location.shard)); + Ok(Outcome::Migrated { deferred: is_deferred }) +} + +/// Attempts/next-attempt/last-error for a deferred `StateEntry`, from legacy +/// meta. A message routed here without meta (unexpected — the legacy worker +/// always writes meta before moving a message to deferred/) is migrated +/// defensively: one attempt already happened, retry immediately. +fn deferred_fields(meta: Option<&EmailMetadata>) -> (u32, i64, String) { + match meta { + Some(m) => ( + m.attempts.max(1), + system_time_to_ms(m.next_attempt), + m.last_error.clone().unwrap_or_default(), + ), + None => (1, Utc::now().timestamp_millis(), String::new()), + } +} + +fn system_time_to_ms(t: SystemTime) -> i64 { + match t.duration_since(UNIX_EPOCH) { + Ok(d) => d.as_millis() as i64, + Err(e) => -(e.duration().as_millis() as i64), + } +} + +/// Stable 16-byte id derived from a non-ULID legacy id (HMAC-free: this is +/// dedup identity, not security). Re-running migration maps the same legacy +/// id to the same new id. +fn derived_message_id(legacy_id: &str) -> MessageId { + use hmac::Mac; + // md-5 is already in the tree (CRAM-MD5); collision resistance is not a + // requirement here, only stability across runs. + let mut mac = hmac::Hmac::::new_from_slice(b"hedwig-migrate-id") + .expect("HMAC accepts any key length"); + mac.update(legacy_id.as_bytes()); + let digest = mac.finalize().into_bytes(); + let mut bytes = [0u8; 16]; + bytes.copy_from_slice(&digest[..16]); + MessageId(bytes) +} + +fn get_or_open_state_store<'a>( + spool: &Spool, + state_stores: &'a mut HashMap, + shard: u16, +) -> Result<&'a mut ShardStateStore> { + if let std::collections::hash_map::Entry::Vacant(entry) = state_stores.entry(shard) { + let dir = spool.shard(shard).path(); + let (store, _recovered) = ShardStateStore::recover(dir, shard) + .map_err(miette::Report::new) + .wrap_err_with(|| format!("opening state store for shard {shard}"))?; + entry.insert(store); + } + Ok(state_stores.get_mut(&shard).expect("just inserted")) +} + +/// Re-scan every shard's segments (writers already shut down, so files are +/// stable) and confirm every id appended in this run is present. +fn verify_migrated( + spool: &Spool, + migrated: &[(MessageId, u16)], + max_record_len: u32, + summary: &mut MigrationSummary, +) -> Result<()> { + let mut by_shard: HashMap> = HashMap::new(); + for (id, shard) in migrated { + by_shard.entry(*shard).or_default().insert(*id); + } + + for (shard, wanted) in &by_shard { + let shard_dir = spool.shard(*shard); + let segs = shard_dir + .list_segments() + .map_err(miette::Report::new) + .wrap_err_with(|| format!("listing shard {shard} segments"))?; + let mut found = HashSet::new(); + for (ordinal, path) in segs.sealed.iter().chain(segs.active.iter()) { + let len = std::fs::metadata(path) + .into_diagnostic() + .wrap_err_with(|| format!("stat {}", path.display()))? + .len(); + let reader = open_segment_reader(shard_dir.path(), *ordinal) + .map_err(miette::Report::new) + .wrap_err_with(|| format!("opening shard {shard} segment {ordinal}"))?; + scan_headers(&reader, 0, len, max_record_len, |_, header| { + found.insert(header.message_id); + true + }) + .map_err(miette::Report::new) + .wrap_err_with(|| format!("scanning shard {shard} segment {ordinal}"))?; + } + for id in wanted { + if !found.contains(id) { + summary.failed.push(( + id.to_string(), + "missing from spool after migration (verification failed)".to_string(), + )); + } + } + } + Ok(()) +} + +/// Rename `queued/` and `deferred/` to timestamped backups. `bounced/` is +/// left untouched: it remains the live bounce archive under the log backend. +/// Never deletes anything. +fn rename_legacy_dirs(legacy_base_path: &Utf8Path) -> Result<()> { + let epoch = SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap_or_default() + .as_secs(); + + for name in ["queued", "deferred"] { + let src = legacy_base_path.join(name); + if !src.exists() { + continue; + } + // Nothing to preserve, and nothing renamed: a re-run after a prior + // success recreates an empty dir here (`FileSystemStorage::new` + // always ensures queued/deferred/bounced exist) with no messages + // left to migrate. Renaming it would risk colliding with a backup + // this same command already created a moment ago. + if dir_is_empty(&src)? { + continue; + } + let dest = legacy_base_path.join(format!("{name}.migrated-{epoch}")); + std::fs::rename(&src, &dest) + .into_diagnostic() + .wrap_err_with(|| format!("renaming {src} to {dest}"))?; + println!("renamed {src} -> {dest}"); + } + Ok(()) +} + +fn dir_is_empty(path: &Utf8Path) -> Result { + let mut entries = std::fs::read_dir(path) + .into_diagnostic() + .wrap_err_with(|| format!("reading {path}"))?; + Ok(entries.next().is_none()) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::logqueue::record; + use crate::logqueue::state::ShardStateStore as StateStore; + use crate::worker::EmailMetadata; + use std::time::Duration; + + fn writer_config() -> WriterConfig { + WriterConfig { + segment_target_bytes: 64 * 1024 * 1024, + max_record_len: record::MAX_RECORD_LEN, + pending_append_bytes: 16 * 1024 * 1024, + } + } + + fn email(id: &str, to: &[&str], body: &str) -> StoredEmail { + StoredEmail { + message_id: id.to_string(), + from: "sender@example.com".to_string(), + to: to.iter().map(|s| s.to_string()).collect(), + body: body.to_string(), + queued_at: Some(Utc::now()), + } + } + + fn meta(id: &str, attempts: u32, last_error: Option<&str>) -> EmailMetadata { + EmailMetadata { + msg_id: id.to_string(), + attempts, + last_attempt: SystemTime::now(), + next_attempt: SystemTime::now() + Duration::from_secs(300), + last_error: last_error.map(|s| s.to_string()), + } + } + + fn new_ulid() -> String { + ulid::Ulid::new().to_string() + } + + /// Every message id found by scanning a shard's payload segments, + /// counting how many times each appears (should always be 1 — more than + /// that would mean migration appended a duplicate). + fn count_payload_records(spool_root: &Path, shard: u16) -> HashMap { + let mut counts = HashMap::new(); + let dir = spool_root.join(crate::logqueue::shard::shard_dir_name(shard)); + if !dir.exists() { + return counts; + } + let shard_dir = crate::logqueue::shard::ShardDir::open_or_create(spool_root, shard).unwrap(); + let segs = shard_dir.list_segments().unwrap(); + for (ordinal, path) in segs.sealed.iter().chain(segs.active.iter()) { + let reader = open_segment_reader(&dir, *ordinal).unwrap(); + let len = std::fs::metadata(path).unwrap().len(); + scan_headers(&reader, 0, len, record::MAX_RECORD_LEN, |_, header| { + *counts.entry(header.message_id).or_insert(0) += 1; + true + }) + .unwrap(); + } + counts + } + + async fn find_ready(spool_root: &Path, shard_count: u16, id: MessageId) -> Option<(String, String, Vec)> { + for shard in 0..shard_count { + let dir = spool_root.join(crate::logqueue::shard::shard_dir_name(shard)); + if !dir.exists() { + continue; + } + let shard_dir = crate::logqueue::shard::ShardDir::open_or_create(spool_root, shard).unwrap(); + let segs = shard_dir.list_segments().unwrap(); + for (ordinal, path) in segs.sealed.iter().chain(segs.active.iter()) { + let reader = open_segment_reader(&dir, *ordinal).unwrap(); + let len = std::fs::metadata(path).unwrap().len(); + let mut result = None; + scan_headers(&reader, 0, len, record::MAX_RECORD_LEN, |offset, header| { + if header.message_id == id { + let body = reader.read_body(&header, offset).unwrap(); + result = Some((header.sender.clone(), header.recipients.join(","), body)); + false + } else { + true + } + }) + .unwrap(); + if result.is_some() { + return result; + } + } + } + None + } + + fn deferred_entry(spool_root: &Path, shard_count: u16, id: MessageId) -> Option { + for shard in 0..shard_count { + let dir = spool_root.join(crate::logqueue::shard::shard_dir_name(shard)); + if !dir.exists() { + continue; + } + let (_, recovered) = StateStore::recover(&dir, shard).unwrap(); + if let Some(d) = recovered.deferred.get(&id) { + return Some(d.clone()); + } + } + None + } + + #[tokio::test] + async fn migrates_queued_and_deferred_with_meta() { + let dir = tempfile::tempdir().unwrap(); + let legacy_base = camino::Utf8PathBuf::from_path_buf(dir.path().join("legacy")).unwrap(); + let spool_root = dir.path().join("legacy").join("spool"); + + let legacy = FileSystemStorage::new(&legacy_base).await.unwrap(); + + // A plain queued message, never retried. + let q1_id = new_ulid(); + legacy + .put(email(&q1_id, &["r1@example.com"], "queued body"), Status::Queued) + .await + .unwrap(); + + // A queued message that is mid-retry: body still in queued/, but a + // meta file records a prior failed attempt. + let q2_id = new_ulid(); + legacy + .put(email(&q2_id, &["r2@example.com"], "mid-retry body"), Status::Queued) + .await + .unwrap(); + legacy.put_meta(&q2_id, &meta(&q2_id, 2, Some("451 try later"))).await.unwrap(); + + // A fully deferred message. + let d1_id = new_ulid(); + legacy + .put(email(&d1_id, &["r3@example.com"], "deferred body"), Status::Deferred) + .await + .unwrap(); + legacy.put_meta(&d1_id, &meta(&d1_id, 3, Some("450 backoff"))).await.unwrap(); + + let summary = migrate(&legacy_base, &spool_root, 2, writer_config()).await.unwrap(); + assert_eq!(summary.migrated_queued, 1, "{summary:?}"); + assert_eq!(summary.migrated_deferred, 2); + assert_eq!(summary.skipped, 0); + assert!(summary.failed.is_empty(), "unexpected failures: {:?}", summary.failed); + + let q1 = MessageId::parse(&q1_id).unwrap(); + let (sender, rcpts, body) = find_ready(&spool_root, 2, q1).await.unwrap(); + assert_eq!(sender, "sender@example.com"); + assert_eq!(rcpts, "r1@example.com"); + assert_eq!(body, b"queued body"); + assert!(deferred_entry(&spool_root, 2, q1).is_none()); + + let q2 = MessageId::parse(&q2_id).unwrap(); + assert!(find_ready(&spool_root, 2, q2).await.is_some()); + let d = deferred_entry(&spool_root, 2, q2).expect("mid-retry meta migrates as deferred state"); + assert_eq!(d.attempts, 2); + assert_eq!(d.remaining_recipients, vec!["r2@example.com".to_string()]); + assert_eq!(d.last_error, "451 try later"); + + let d1 = MessageId::parse(&d1_id).unwrap(); + assert!(find_ready(&spool_root, 2, d1).await.is_some()); + let d = deferred_entry(&spool_root, 2, d1).expect("deferred message has state"); + assert_eq!(d.attempts, 3); + assert_eq!(d.remaining_recipients, vec!["r3@example.com".to_string()]); + assert_eq!(d.last_error, "450 backoff"); + + // Backup directories exist; bounced/ untouched (never created here, + // but queued/deferred must be renamed away). + assert!(!legacy_base.join("queued").exists()); + assert!(!legacy_base.join("deferred").exists()); + let mut saw_queued_backup = false; + let mut saw_deferred_backup = false; + for entry in std::fs::read_dir(legacy_base.as_std_path()).unwrap() { + let name = entry.unwrap().file_name().to_string_lossy().to_string(); + if name.starts_with("queued.migrated-") { + saw_queued_backup = true; + } + if name.starts_with("deferred.migrated-") { + saw_deferred_backup = true; + } + } + assert!(saw_queued_backup); + assert!(saw_deferred_backup); + } + + #[tokio::test] + async fn rerun_after_success_migrates_nothing_new() { + let dir = tempfile::tempdir().unwrap(); + let legacy_base = camino::Utf8PathBuf::from_path_buf(dir.path().join("legacy")).unwrap(); + let spool_root = dir.path().join("legacy").join("spool"); + + let legacy = FileSystemStorage::new(&legacy_base).await.unwrap(); + let id = new_ulid(); + legacy.put(email(&id, &["r@example.com"], "body"), Status::Queued).await.unwrap(); + + let first = migrate(&legacy_base, &spool_root, 1, writer_config()).await.unwrap(); + assert_eq!(first.migrated_queued, 1); + assert!(first.failed.is_empty()); + + // Nothing left in queued/ (renamed away), so a second run finds + // nothing new to migrate and nothing to skip either. + let second = migrate(&legacy_base, &spool_root, 1, writer_config()).await.unwrap(); + assert_eq!(second.migrated_queued, 0); + assert_eq!(second.migrated_deferred, 0); + assert_eq!(second.skipped, 0); + assert!(second.failed.is_empty()); + + let parsed = MessageId::parse(&id).unwrap(); + assert!(find_ready(&spool_root, 1, parsed).await.is_some()); + } + + #[tokio::test] + async fn already_migrated_id_left_in_legacy_spool_is_skipped_not_duplicated() { + // Simulates re-running after a crash right before the backup rename: + // the message already has a payload record in the new spool AND its + // body is still sitting in the legacy queued/ directory. + let dir = tempfile::tempdir().unwrap(); + let legacy_base = camino::Utf8PathBuf::from_path_buf(dir.path().join("legacy")).unwrap(); + let spool_root = dir.path().join("legacy").join("spool"); + + let legacy = FileSystemStorage::new(&legacy_base).await.unwrap(); + let id = new_ulid(); + legacy.put(email(&id, &["r@example.com"], "body"), Status::Queued).await.unwrap(); + + // Pre-seed the new spool with this id's payload record directly, as + // if a prior migration run had appended it before crashing. + { + let spool = Spool::open(&spool_root, 1).unwrap(); + let writers = LogWriters::start(&spool, writer_config()).unwrap(); + let handle = writers.handle(); + handle + .append(AppendMessage { + message_id: MessageId::parse(&id).unwrap(), + enqueue_ms: Utc::now().timestamp_millis(), + generation: 0, + sender: "sender@example.com".into(), + recipients: vec!["r@example.com".into()], + body: Bytes::from_static(b"body"), + }) + .await + .unwrap(); + writers.shutdown().await; + } + + let summary = migrate(&legacy_base, &spool_root, 1, writer_config()).await.unwrap(); + assert_eq!(summary.migrated_queued, 0); + assert_eq!(summary.skipped, 1); + assert!(summary.failed.is_empty()); + + // Exactly one payload record for this id (no duplicate append). + let target = MessageId::parse(&id).unwrap(); + let counts = count_payload_records(&spool_root, 0); + assert_eq!(counts.get(&target).copied().unwrap_or(0), 1); + } + + #[tokio::test] + async fn non_ulid_legacy_id_is_remapped_to_a_fresh_ulid() { + let dir = tempfile::tempdir().unwrap(); + let legacy_base = camino::Utf8PathBuf::from_path_buf(dir.path().join("legacy")).unwrap(); + let spool_root = dir.path().join("legacy").join("spool"); + + let legacy = FileSystemStorage::new(&legacy_base).await.unwrap(); + legacy + .put(email("not-a-ulid", &["r@example.com"], "body"), Status::Queued) + .await + .unwrap(); + + let summary = migrate(&legacy_base, &spool_root, 1, writer_config()).await.unwrap(); + assert_eq!(summary.migrated_queued, 1); + assert!(summary.failed.is_empty()); + } +} diff --git a/smtp-server/src/queue_cli.rs b/smtp-server/src/queue_cli.rs new file mode 100644 index 0000000..4614ac7 --- /dev/null +++ b/smtp-server/src/queue_cli.rs @@ -0,0 +1,890 @@ +//! Queue inspection and migration CLI (`hedwig queue …`, docs/plans/2026-07-20-durable-log-queue.md §25 +//! "Operator tooling" and §23 "Migration from the current filesystem +//! spool"). +//! +//! `list`/`show`/`stats` must be side-effect free against the spool: no +//! locks, no journal truncation, no file or directory creation. They +//! deliberately do not use [`crate::logqueue::spool::Spool`] (exclusive +//! lock, creates `format-version`/shard directories) or +//! [`crate::logqueue::shard::ShardDir`] (creates the shard directory); +//! shard directories are enumerated directly instead. +//! +//! They re-implement, in a read-only form, just enough of the log-queue's +//! discovery logic (see `logqueue::dispatcher::discover_shard`): recover +//! checkpoint + journal state via +//! [`crate::logqueue::state::load_state_readonly`], then scan payload +//! segments from the recovered cursor onward for messages a checkpoint +//! hasn't folded in yet. +//! +//! `migrate` is the one subcommand here that WRITES: it takes the exclusive +//! spool lock and moves live messages out of the legacy filesystem spool. +//! See [`crate::migrate`] for its implementation. + +use std::collections::HashMap; +use std::path::{Path, PathBuf}; + +use clap::{Args, Subcommand}; +use miette::{bail, IntoDiagnostic, Result, WrapErr}; + +use crate::logqueue::record::{self, RecordHeader}; +use crate::logqueue::segment::{self, open_segment_reader, scan_headers, SegmentKind}; +use crate::logqueue::state::{self, RecoveredState}; +use crate::logqueue::{JobLocation, MessageId, QueueError, FORMAT_VERSION}; + +const FORMAT_VERSION_FILE: &str = "format-version"; + +#[derive(Args)] +pub struct QueueArgs { + #[command(subcommand)] + command: QueueCommand, +} + +#[derive(Subcommand)] +enum QueueCommand { + /// List live (ready/deferred) messages across all shards + List(ListArgs), + /// Show one message's envelope, state, and location + Show(ShowArgs), + /// Show per-shard and per-segment storage statistics + Stats(StatsArgs), + /// One-time migration from the legacy filesystem spool to the log queue + /// (docs/plans/2026-07-20-durable-log-queue.md §23). WRITES to the spool; stop the server first. + Migrate(MigrateArgs), +} + +#[derive(Args)] +struct ListArgs { + /// Log-queue spool root (contains shard-NNNN directories) + #[arg(long)] + spool: PathBuf, + /// Only show ready messages + #[arg(long, conflicts_with = "deferred")] + ready: bool, + /// Only show deferred messages + #[arg(long, conflicts_with = "ready")] + deferred: bool, +} + +#[derive(Args)] +struct ShowArgs { + /// Log-queue spool root (contains shard-NNNN directories) + #[arg(long)] + spool: PathBuf, + /// Message ID (ULID string) + message_id: String, +} + +#[derive(Args)] +struct StatsArgs { + /// Log-queue spool root (contains shard-NNNN directories) + #[arg(long)] + spool: PathBuf, +} + +#[derive(Args)] +struct MigrateArgs { + /// Path to the hedwig config file. Must already have + /// storage.storage_type = "log" — switch the config first, then run + /// this migration; the server (and any other process using this spool) + /// must be stopped. + #[arg(long)] + config: String, +} + +pub async fn run(args: QueueArgs) -> Result<()> { + match args.command { + QueueCommand::List(a) => cmd_list(a), + QueueCommand::Show(a) => cmd_show(a), + QueueCommand::Stats(a) => cmd_stats(a), + QueueCommand::Migrate(a) => cmd_migrate(a).await, + } +} + +// --------------------------------------------------------------------------- +// `queue migrate` — the only subcommand here that writes to the spool. + +async fn cmd_migrate(args: MigrateArgs) -> Result<()> { + let cfg = crate::config::Cfg::load(&args.config).wrap_err("error loading configuration")?; + + if cfg.storage.storage_type != "log" { + bail!( + "queue migrate requires storage.storage_type = \"log\" in {}, found {:?}; \ + switch the config to the log backend first, then run this migration", + args.config, + cfg.storage.storage_type + ); + } + + println!( + "Migrating legacy filesystem spool at {} to the log queue.", + cfg.storage.base_path + ); + println!( + "IMPORTANT: the hedwig server (or any other process using this spool) must be \ + stopped before running this command — there is no lock on the legacy spool." + ); + + let legacy_base_path = camino::Utf8PathBuf::from(cfg.storage.base_path.clone()); + let spool_root = std::path::Path::new(&cfg.storage.base_path).join("spool"); + + let qcfg = cfg.queue(); + let max_message_size = cfg.server.max_message_size.unwrap_or(25 * 1024 * 1024); + qcfg.validate(max_message_size) + .wrap_err("invalid [queue] configuration")?; + let max_record_len = (max_message_size as u64 + + crate::logqueue::spool::ENVELOPE_ALLOWANCE + + crate::logqueue::record::FIXED_HEADER_LEN as u64) as u32; + let writer_config = crate::logqueue::writer::WriterConfig { + segment_target_bytes: qcfg.segment_target_bytes(), + max_record_len, + pending_append_bytes: qcfg.pending_append_bytes(), + }; + + let summary = crate::migrate::migrate(&legacy_base_path, &spool_root, qcfg.append_writers(), writer_config) + .await?; + + summary.print(); + + if !summary.failed.is_empty() { + bail!( + "migration completed with {} failed message(s) (see above); the legacy spool \ + was left in place — safe to re-run this command after investigating", + summary.failed.len() + ); + } + + println!("migration complete."); + Ok(()) +} + +// --------------------------------------------------------------------------- +// Spool/shard enumeration (read-only: no Spool::open, no ShardDir). + +fn check_format_version(spool_root: &Path) -> Result<()> { + let path = spool_root.join(FORMAT_VERSION_FILE); + let contents = std::fs::read_to_string(&path).into_diagnostic().wrap_err_with(|| { + format!( + "could not read {}; is {} a hedwig log-queue spool root?", + path.display(), + spool_root.display() + ) + })?; + let found: u16 = contents.trim().parse().into_diagnostic().wrap_err_with(|| { + format!( + "{} does not contain a version number: {contents:?}", + path.display() + ) + })?; + if found != FORMAT_VERSION { + bail!( + "spool at {} has format version {found}, this build of hedwig supports {FORMAT_VERSION}", + spool_root.display() + ); + } + Ok(()) +} + +fn parse_shard_dir_name(name: &str) -> Option { + let digits = name.strip_prefix("shard-")?; + if digits.len() != 4 || !digits.bytes().all(|b| b.is_ascii_digit()) { + return None; + } + digits.parse().ok() +} + +/// Enumerate `shard-NNNN` directories directly under the spool root, sorted +/// by shard number. Never creates anything. +fn list_shard_dirs(spool_root: &Path) -> Result> { + let mut shards = Vec::new(); + let entries = std::fs::read_dir(spool_root) + .into_diagnostic() + .wrap_err_with(|| format!("could not read spool root {}", spool_root.display()))?; + for entry in entries { + let entry = entry.into_diagnostic()?; + if !entry.file_type().into_diagnostic()?.is_dir() { + continue; + } + let Some(name) = entry.file_name().to_str().map(str::to_owned) else { + continue; + }; + if let Some(shard) = parse_shard_dir_name(&name) { + shards.push((shard, entry.path())); + } + } + shards.sort_unstable_by_key(|(shard, _)| *shard); + Ok(shards) +} + +/// A segment file present on disk right now, as seen by a point-in-time +/// directory listing. +#[derive(Debug, Clone)] +struct SegmentFile { + ordinal: u64, + kind: SegmentKind, + len: u64, +} + +/// List segment files directly (no `ShardDir`, which creates the directory +/// if missing). Sorted by ordinal; a shard has at most one active segment, +/// which — being the newest — sorts last. +fn list_segment_files(shard_dir: &Path) -> Result, QueueError> { + let mut out = Vec::new(); + let entries = match std::fs::read_dir(shard_dir) { + Ok(entries) => entries, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(out), + Err(e) => return Err(QueueError::io(shard_dir, e)), + }; + for entry in entries { + let entry = entry.map_err(|e| QueueError::io(shard_dir, e))?; + let Some(name) = entry.file_name().to_str().map(str::to_owned) else { + continue; + }; + let Some((ordinal, kind)) = segment::parse_file_name(&name) else { + continue; + }; + let len = entry.metadata().map_err(|e| QueueError::io(shard_dir, e))?.len(); + out.push(SegmentFile { ordinal, kind, len }); + } + out.sort_unstable_by_key(|s| s.ordinal); + Ok(out) +} + +// --------------------------------------------------------------------------- +// Read-only discovery: checkpoint/journal state + a scan for messages not +// yet folded into a checkpoint. + +/// A message found only by scanning past the recovered cursor: not yet +/// known to any checkpoint or journal entry. +#[derive(Debug, Clone, Copy)] +struct ScannedReady { + location: JobLocation, + enqueue_ms: i64, +} + +struct ShardData { + shard: u16, + dir: PathBuf, + state: RecoveredState, + segments: Vec, + /// Live messages discovered by scanning segments past `state.cursor`, + /// keyed by message id. + scanned: HashMap, +} + +/// Load one shard's read-only snapshot: recovered checkpoint/journal state, +/// the segment files present on disk, and anything a scan from the +/// checkpoint's cursor finds that the checkpoint doesn't know about yet. +fn load_shard(shard: u16, dir: PathBuf) -> Result { + let state = state::load_state_readonly(&dir)?; + let segments = list_segment_files(&dir)?; + let scanned = scan_undiscovered(shard, &dir, &state, &segments); + Ok(ShardData { + shard, + dir, + state, + segments, + scanned, + }) +} + +/// Walk sealed+active segments in ordinal order from the recovered cursor to +/// each segment's end, skipping records already known (checkpointed ready +/// or deferred) or tombstoned in that segment. Best-effort: a corrupt or +/// torn record stops the scan of that segment (logged to stderr) without +/// failing the whole command — an active segment's tail is routinely torn +/// mid-append on a live spool, and this tool must keep working against one. +fn scan_undiscovered( + shard: u16, + dir: &Path, + state: &RecoveredState, + segments: &[SegmentFile], +) -> HashMap { + let mut found = HashMap::new(); + let (start_segment, start_offset) = match state.cursor { + Some(cursor) => cursor, + None => match segments.first() { + Some(first) => (first.ordinal, 0), + None => return found, + }, + }; + let max_record_len = record::MAX_RECORD_LEN; + + for seg in segments { + if seg.ordinal < start_segment { + continue; + } + let start = if seg.ordinal == start_segment { + start_offset + } else { + 0 + }; + if start >= seg.len { + continue; + } + let reader = match open_segment_reader(dir, seg.ordinal) { + Ok(r) => r, + Err(e) => { + eprintln!( + "warning: shard {shard} segment {}: could not open for scanning: {e}", + seg.ordinal + ); + continue; + } + }; + let ordinal = seg.ordinal; + let result = scan_headers(&reader, start, seg.len, max_record_len, |offset, header| { + if record_undiscovered(state, &found, ordinal, &header) { + found.insert( + header.message_id, + ScannedReady { + location: JobLocation { + shard, + segment: ordinal, + offset, + length: header.record_len, + ordinal: header.ordinal, + generation: header.generation, + }, + enqueue_ms: header.enqueue_ms, + }, + ); + } + true + }); + if let Err(e) = result { + eprintln!( + "warning: shard {shard} segment {}: stopped scan early at a corrupt or torn record: {e}", + seg.ordinal + ); + } + } + found +} + +/// Whether `header` is not yet known: not checkpointed (ready or deferred), +/// not already scanned this pass, and not tombstoned in its own segment. +fn record_undiscovered( + state: &RecoveredState, + found: &HashMap, + segment: u64, + header: &RecordHeader, +) -> bool { + !state.is_terminal(segment, &header.message_id) + && !state.ready.contains_key(&header.message_id) + && !state.deferred.contains_key(&header.message_id) + && !found.contains_key(&header.message_id) +} + +// --------------------------------------------------------------------------- +// Unified live-message view shared by `list` and `stats`. + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum MsgState { + Ready, + Deferred, +} + +impl MsgState { + fn as_str(self) -> &'static str { + match self { + MsgState::Ready => "ready", + MsgState::Deferred => "deferred", + } + } +} + +struct LiveMessage { + id: MessageId, + shard: u16, + location: JobLocation, + state: MsgState, + /// `None` when a checkpointed deferred entry doesn't carry the original + /// enqueue time. + enqueue_ms: Option, + attempts: u32, + next_attempt_ms: Option, +} + +fn shard_live_messages(shard: &ShardData) -> Vec { + let mut out = Vec::with_capacity(shard.state.ready.len() + shard.state.deferred.len() + shard.scanned.len()); + for r in shard.state.ready.values() { + out.push(LiveMessage { + id: r.id, + shard: shard.shard, + location: r.location, + state: MsgState::Ready, + enqueue_ms: Some(r.enqueue_ms), + attempts: r.attempts, + next_attempt_ms: None, + }); + } + for d in shard.state.deferred.values() { + out.push(LiveMessage { + id: d.id, + shard: shard.shard, + location: d.location, + state: MsgState::Deferred, + enqueue_ms: None, + attempts: d.attempts, + next_attempt_ms: Some(d.next_attempt_ms), + }); + } + for (id, s) in &shard.scanned { + out.push(LiveMessage { + id: *id, + shard: shard.shard, + location: s.location, + state: MsgState::Ready, + enqueue_ms: Some(s.enqueue_ms), + attempts: 0, + next_attempt_ms: None, + }); + } + out +} + +// --------------------------------------------------------------------------- +// `queue list` + +fn cmd_list(args: ListArgs) -> Result<()> { + check_format_version(&args.spool)?; + let shard_dirs = list_shard_dirs(&args.spool)?; + + let mut messages = Vec::new(); + for (shard, dir) in shard_dirs { + let data = load_shard(shard, dir) + .into_diagnostic() + .wrap_err_with(|| format!("loading shard {shard}"))?; + messages.extend(shard_live_messages(&data)); + } + + if args.ready { + messages.retain(|m| m.state == MsgState::Ready); + } else if args.deferred { + messages.retain(|m| m.state == MsgState::Deferred); + } + + let now = now_ms(); + // Age descending (oldest first). Messages with an unknown enqueue time + // (a checkpointed deferred entry) sort after every message with a known + // age, ordered by id for determinism. + messages.sort_unstable_by(|a, b| match (a.enqueue_ms, b.enqueue_ms) { + (Some(a_ms), Some(b_ms)) => (now - a_ms).cmp(&(now - b_ms)).reverse(), + (Some(_), None) => std::cmp::Ordering::Less, + (None, Some(_)) => std::cmp::Ordering::Greater, + (None, None) => a.id.cmp(&b.id), + }); + + if messages.is_empty() { + println!("no live messages in {}", args.spool.display()); + return Ok(()); + } + + let rows: Vec<[String; 7]> = messages + .iter() + .map(|m| { + [ + m.id.to_string(), + m.state.as_str().to_string(), + m.enqueue_ms.map_or_else(|| "-".to_string(), |ms| humanize_age(now - ms)), + m.attempts.to_string(), + m.next_attempt_ms.map_or_else(|| "-".to_string(), format_iso8601), + m.shard.to_string(), + m.location.segment.to_string(), + ] + }) + .collect(); + print_table( + &["MESSAGE ID", "STATE", "AGE", "ATTEMPTS", "NEXT-ATTEMPT (UTC)", "SHARD", "SEGMENT"], + &rows, + ); + Ok(()) +} + +// --------------------------------------------------------------------------- +// `queue show` + +fn cmd_show(args: ShowArgs) -> Result<()> { + check_format_version(&args.spool)?; + let target = MessageId::parse(&args.message_id).into_diagnostic()?; + let shard_dirs = list_shard_dirs(&args.spool)?; + + for (shard, dir) in shard_dirs { + let data = load_shard(shard, dir) + .into_diagnostic() + .wrap_err_with(|| format!("loading shard {shard}"))?; + + let found = if let Some(r) = data.state.ready.get(&target) { + Some((r.location, MsgState::Ready, r.attempts, None, None, None)) + } else if let Some(d) = data.state.deferred.get(&target) { + Some(( + d.location, + MsgState::Deferred, + d.attempts, + Some(d.next_attempt_ms), + Some(&d.remaining_recipients), + Some(d.last_error.as_str()), + )) + } else { + data.scanned + .get(&target) + .map(|s| (s.location, MsgState::Ready, 0, None, None, None)) + }; + + let Some((location, state, attempts, next_attempt_ms, remaining, last_error)) = found else { + continue; + }; + + let reader = open_segment_reader(&data.dir, location.segment) + .into_diagnostic() + .wrap_err("opening segment to read the message envelope")?; + let header = reader + .read_header_at(location.offset, record::MAX_RECORD_LEN) + .into_diagnostic() + .wrap_err("reading payload record header")?; + + println!("id: {}", target); + println!("state: {}", state.as_str()); + println!("sender: {}", header.sender); + println!("recipients: {}", header.recipients.join(", ")); + if let Some(remaining) = remaining { + println!("remaining: {}", remaining.join(", ")); + } + println!("attempts: {attempts}"); + if let Some(last_error) = last_error { + println!("last error: {last_error}"); + } + if let Some(next_attempt_ms) = next_attempt_ms { + println!("next attempt (UTC): {}", format_iso8601(next_attempt_ms)); + } + println!("enqueued (UTC): {}", format_iso8601(header.enqueue_ms)); + println!( + "location: shard={} segment={} offset={} length={} generation={}", + location.shard, location.segment, location.offset, location.length, location.generation + ); + println!("body size: {} bytes", header.body_len()); + return Ok(()); + } + + bail!( + "message {} not found in spool {}", + args.message_id, + args.spool.display() + ); +} + +// --------------------------------------------------------------------------- +// `queue stats` + +fn cmd_stats(args: StatsArgs) -> Result<()> { + check_format_version(&args.spool)?; + let shard_dirs = list_shard_dirs(&args.spool)?; + + let mut grand_total_bytes = 0u64; + let mut grand_dead_bytes = 0u64; + let mut grand_ready = 0usize; + let mut grand_deferred = 0usize; + + for (shard, dir) in shard_dirs { + let data = load_shard(shard, dir) + .into_diagnostic() + .wrap_err_with(|| format!("loading shard {shard}"))?; + + println!("shard {:04}:", shard); + if data.segments.is_empty() { + println!(" (no segments)"); + } + let mut shard_total_bytes = 0u64; + let mut shard_dead_bytes = 0u64; + let rows: Vec<[String; 5]> = data + .segments + .iter() + .map(|seg| { + let file_kind = match seg.kind { + SegmentKind::Sealed => "sealed", + SegmentKind::Active => "active", + }; + let dead_bytes = data + .state + .segment_stats + .get(&seg.ordinal) + .map(|s| s.dead_bytes) + .unwrap_or(0); + let tombstones = data + .state + .tombstones + .get(&seg.ordinal) + .map(|s| s.len()) + .unwrap_or(0); + let ratio = if seg.len > 0 { + dead_bytes as f64 / seg.len as f64 + } else { + 0.0 + }; + shard_total_bytes += seg.len; + shard_dead_bytes += dead_bytes; + [ + format!("{:012} ({file_kind})", seg.ordinal), + seg.len.to_string(), + dead_bytes.to_string(), + format!("{:.1}%", ratio * 100.0), + tombstones.to_string(), + ] + }) + .collect(); + if !rows.is_empty() { + print_table( + &["SEGMENT", "TOTAL BYTES", "DEAD BYTES", "DEAD RATIO", "TOMBSTONES"], + &rows, + ); + } + + let live = shard_live_messages(&data); + let shard_ready = live.iter().filter(|m| m.state == MsgState::Ready).count(); + let shard_deferred = live.iter().filter(|m| m.state == MsgState::Deferred).count(); + println!( + " shard totals: {shard_total_bytes} bytes, {shard_dead_bytes} dead, {shard_ready} ready, {shard_deferred} deferred" + ); + println!(); + + grand_total_bytes += shard_total_bytes; + grand_dead_bytes += shard_dead_bytes; + grand_ready += shard_ready; + grand_deferred += shard_deferred; + } + + let grand_ratio = if grand_total_bytes > 0 { + grand_dead_bytes as f64 / grand_total_bytes as f64 + } else { + 0.0 + }; + println!( + "grand totals: {grand_total_bytes} bytes, {grand_dead_bytes} dead ({:.1}%), {grand_ready} ready, {grand_deferred} deferred", + grand_ratio * 100.0 + ); + Ok(()) +} + +// --------------------------------------------------------------------------- +// Formatting helpers. + +fn now_ms() -> i64 { + chrono::Utc::now().timestamp_millis() +} + +fn format_iso8601(ms: i64) -> String { + chrono::DateTime::::from_timestamp_millis(ms) + .map(|dt| dt.to_rfc3339_opts(chrono::SecondsFormat::Secs, true)) + .unwrap_or_else(|| format!("invalid-timestamp({ms})")) +} + +/// Rough humanized duration, e.g. "3m12s", "1h4m", "2d3h", "45s". +fn humanize_age(age_ms: i64) -> String { + let secs = (age_ms.max(0)) / 1000; + let days = secs / 86_400; + let hours = (secs % 86_400) / 3600; + let minutes = (secs % 3600) / 60; + let seconds = secs % 60; + if days > 0 { + format!("{days}d{hours}h") + } else if hours > 0 { + format!("{hours}h{minutes}m") + } else if minutes > 0 { + format!("{minutes}m{seconds}s") + } else { + format!("{seconds}s") + } +} + +/// Print a plain-text table with column padding; no extra dependencies. +fn print_table(header: &[&str; N], rows: &[[String; N]]) { + let mut widths: [usize; N] = std::array::from_fn(|i| header[i].len()); + for row in rows { + for (i, cell) in row.iter().enumerate() { + widths[i] = widths[i].max(cell.len()); + } + } + let print_row = |cells: &[&str]| { + let line: Vec = cells + .iter() + .enumerate() + .map(|(i, cell)| format!("{: = row.iter().map(String::as_str).collect(); + print_row(&cells); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::logqueue::spool::Spool; + use crate::logqueue::state::{Checkpoint, ReadyJob, ShardStateStore, StateEntry}; + use crate::logqueue::writer::{AppendMessage, LogWriters, WriterConfig}; + use bytes::Bytes; + + fn writer_config() -> WriterConfig { + WriterConfig { + segment_target_bytes: 64 * 1024 * 1024, + max_record_len: record::MAX_RECORD_LEN, + pending_append_bytes: 16 * 1024 * 1024, + } + } + + fn message(seq: u64, enqueue_ms: i64, rcpt: &str) -> AppendMessage { + AppendMessage { + message_id: MessageId::from_ulid(ulid::Ulid::from_parts(seq, (seq * 7 + 1) as u128)), + enqueue_ms, + generation: 0, + sender: "sender@example.com".into(), + recipients: vec![rcpt.into()], + body: Bytes::from(format!("body {seq}")), + } + } + + #[test] + fn humanize_age_formats() { + assert_eq!(humanize_age(45_000), "45s"); + assert_eq!(humanize_age(3 * 60_000 + 12_000), "3m12s"); + assert_eq!(humanize_age(60 * 60_000 + 4 * 60_000), "1h4m"); + assert_eq!(humanize_age(2 * 86_400_000 + 3 * 3_600_000), "2d3h"); + } + + #[test] + fn shard_dir_name_parses_only_well_formed_names() { + assert_eq!(parse_shard_dir_name("shard-0000"), Some(0)); + assert_eq!(parse_shard_dir_name("shard-0042"), Some(42)); + assert_eq!(parse_shard_dir_name("shard-42"), None); + assert_eq!(parse_shard_dir_name("shard-abcd"), None); + assert_eq!(parse_shard_dir_name("checkpoint"), None); + } + + #[tokio::test] + async fn list_logic_finds_checkpointed_and_scanned_messages() { + let dir = tempfile::tempdir().unwrap(); + let spool = Spool::open(dir.path().join("spool"), 1).unwrap(); + let writers = LogWriters::start(&spool, writer_config()).unwrap(); + let handle = writers.handle(); + + let base = 1_752_000_000_000i64; + let m1 = message(1, base, "r1@example.com"); + let m1_id = m1.message_id; + let loc1 = handle.append(m1).await.unwrap(); + + let m2 = message(2, base + 1, "r2@example.com"); + let m2_id = m2.message_id; + let loc2 = handle.append(m2).await.unwrap(); + + let m3 = message(3, base + 2, "r3@example.com"); + let m3_id = m3.message_id; + let _loc3 = handle.append(m3).await.unwrap(); + + writers.shutdown().await; + + let shard_dir = spool.shard(0).path().to_path_buf(); + + // Build a checkpoint that has already discovered m1 (as ready) and + // whose cursor sits right after it — m2 and m3 are not yet + // discovered by the checkpoint itself. + { + let (mut store, _) = ShardStateStore::recover(&shard_dir, 0).unwrap(); + let cp = Checkpoint { + cursor: Some((loc1.segment, loc1.offset + loc1.length as u64)), + ready: vec![ReadyJob { + id: m1_id, + location: loc1, + attempts: 0, + enqueue_ms: base, + remaining_recipients: vec![], + }], + ..Default::default() + }; + store.write_checkpoint(&cp).unwrap(); + // m2 gets deferred via a journal entry on top of the checkpoint, + // so it's "already known" even though the checkpoint's cursor + // never passed it. + store + .append(&StateEntry::Deferred { + id: m2_id, + location: loc2, + attempts: 1, + next_attempt_ms: base + 60_000, + remaining_recipients: vec!["r2@example.com".into()], + last_error: "451 try later".into(), + }) + .unwrap(); + } + + let data = load_shard(0, shard_dir).unwrap(); + + // m1: known via the checkpoint's ready list. + assert!(data.state.ready.contains_key(&m1_id)); + // m2: known via the journal (deferred), not by scanning. + assert!(data.state.deferred.contains_key(&m2_id)); + assert!(!data.scanned.contains_key(&m2_id)); + // m3: undiscovered by any checkpoint/journal entry, found only by + // scanning segments past the cursor. + assert!(data.scanned.contains_key(&m3_id)); + assert_eq!(data.scanned[&m3_id].enqueue_ms, base + 2); + + let live = shard_live_messages(&data); + assert_eq!(live.len(), 3); + let live_by_id: HashMap = + live.iter().map(|m| (m.id, m)).collect(); + assert_eq!(live_by_id[&m1_id].state, MsgState::Ready); + assert_eq!(live_by_id[&m1_id].enqueue_ms, Some(base)); + assert_eq!(live_by_id[&m2_id].state, MsgState::Deferred); + assert_eq!(live_by_id[&m2_id].attempts, 1); + assert_eq!(live_by_id[&m3_id].state, MsgState::Ready); + assert_eq!(live_by_id[&m3_id].enqueue_ms, Some(base + 2)); + assert_eq!(live_by_id[&m3_id].attempts, 0); + } + + #[tokio::test] + async fn scan_skips_tombstoned_records() { + let dir = tempfile::tempdir().unwrap(); + let spool = Spool::open(dir.path().join("spool"), 1).unwrap(); + let writers = LogWriters::start(&spool, writer_config()).unwrap(); + let handle = writers.handle(); + + let base = 1_752_000_000_000i64; + let m1 = message(1, base, "r1@example.com"); + let m1_id = m1.message_id; + let loc1 = handle.append(m1).await.unwrap(); + writers.shutdown().await; + + let shard_dir = spool.shard(0).path().to_path_buf(); + { + let (mut store, _) = ShardStateStore::recover(&shard_dir, 0).unwrap(); + store + .append(&StateEntry::Delivered { + id: m1_id, + location: loc1, + timestamp_ms: base + 1000, + }) + .unwrap(); + } + + let data = load_shard(0, shard_dir).unwrap(); + assert!(data.state.is_terminal(loc1.segment, &m1_id)); + assert!(!data.scanned.contains_key(&m1_id), "tombstoned, not live"); + assert!(shard_live_messages(&data).is_empty()); + } + + #[test] + fn empty_spool_root_has_no_shards() { + let dir = tempfile::tempdir().unwrap(); + let spool = Spool::open(dir.path().join("spool"), 2).unwrap(); + drop(spool); + let shards = list_shard_dirs(&dir.path().join("spool")).unwrap(); + assert_eq!(shards.len(), 2); + for (shard, shard_dir) in shards { + let data = load_shard(shard, shard_dir).unwrap(); + assert!(shard_live_messages(&data).is_empty()); + } + } +} diff --git a/smtp-server/src/storage/mod.rs b/smtp-server/src/storage/mod.rs index 0dfc1d6..cfbe28e 100644 --- a/smtp-server/src/storage/mod.rs +++ b/smtp-server/src/storage/mod.rs @@ -8,7 +8,6 @@ use std::{pin::Pin, time::Duration}; use crate::worker::EmailMetadata; pub mod fs_storage; -pub mod sqlite_storage; #[derive(Serialize, Deserialize, Clone, PartialEq)] pub struct StoredEmail { diff --git a/smtp-server/src/storage/sqlite_storage.rs b/smtp-server/src/storage/sqlite_storage.rs deleted file mode 100644 index 952b9c0..0000000 --- a/smtp-server/src/storage/sqlite_storage.rs +++ /dev/null @@ -1,1143 +0,0 @@ -use crate::config::CfgSqlite; -use crate::storage::{CleanupConfig, Status, Storage, StoredEmail}; -use crate::worker::EmailMetadata; -use async_trait::async_trait; -use futures::Stream; -use miette::{Context, IntoDiagnostic, Result}; -use sqlx::sqlite::{SqliteConnectOptions, SqliteJournalMode, SqlitePoolOptions}; -use sqlx::SqlitePool; -use std::collections::{hash_map::DefaultHasher, VecDeque}; -use std::hash::{Hash, Hasher}; -use std::pin::Pin; -use tokio::sync::{mpsc, oneshot}; -use tokio::time::Duration; - -pub struct SqliteStorage { - pub read_pools: Vec, - pub shard_senders: Vec>, - pub num_shards: usize, -} - -pub enum ShardWriteOp { - Put { - email: StoredEmail, - status: i32, - responder: oneshot::Sender>, - }, - PutMeta { - key: String, - meta: EmailMetadata, - responder: oneshot::Sender>, - }, - Delete { - key: String, - status: i32, - responder: oneshot::Sender>, - }, - DeleteMeta { - key: String, - responder: oneshot::Sender>, - }, - Mv { - src_key: String, - dest_key: String, - src_status: i32, - dest_status: i32, - responder: oneshot::Sender>, - }, - Cleanup { - config: CleanupConfig, - responder: oneshot::Sender>, - }, -} - -pub fn status_to_int(status: &Status) -> i32 { - match status { - Status::Queued => 0, - Status::Deferred => 1, - Status::Bounced => 2, - } -} - -impl SqliteStorage { - pub fn shard_for(&self, key: &str) -> usize { - let mut hasher = DefaultHasher::new(); - key.hash(&mut hasher); - (hasher.finish() as usize) % self.num_shards - } - - pub async fn new( - base_path: &str, - num_shards: usize, - batch_size: usize, - batch_timeout_ms: u64, - sqlite_cfg: &CfgSqlite, - ) -> Result { - if num_shards == 0 { - miette::bail!("num_shards must be >= 1, got 0"); - } - - tokio::fs::create_dir_all(base_path) - .await - .into_diagnostic() - .wrap_err("Failed to create storage base path")?; - - let pool_max_connections = sqlite_cfg.pool_max_connections.unwrap_or(4); - let busy_timeout_ms = sqlite_cfg.busy_timeout_ms.unwrap_or(5000); - let cache_size_mb = sqlite_cfg.cache_size_mb.unwrap_or(1600); - let synchronous = sqlite_cfg - .synchronous - .as_deref() - .unwrap_or("NORMAL") - .to_owned(); - - // Distribute cache evenly across shards; negative value = kilobytes. - let cache_kb_per_shard = (cache_size_mb as i64 * 1024) / (num_shards as i64); - - let mut read_pools = Vec::with_capacity(num_shards); - let mut shard_senders = Vec::with_capacity(num_shards); - - for i in 0..num_shards { - let db_path = format!("{}/shard_{}.db", base_path, i); - - let base_opts = SqliteConnectOptions::new() - .filename(&db_path) - .create_if_missing(true) - .journal_mode(SqliteJournalMode::Wal) - .pragma("synchronous", synchronous.clone()) - .pragma("auto_vacuum", "INCREMENTAL") - .pragma("cache_size", format!("-{}", cache_kb_per_shard)) - .pragma("temp_store", "MEMORY") - .foreign_keys(true) - .busy_timeout(Duration::from_millis(busy_timeout_ms)); - - let read_pool = SqlitePoolOptions::new() - .max_connections(pool_max_connections) - .connect_with(base_opts.clone()) - .await - .into_diagnostic() - .wrap_err(format!("Failed to open read pool for shard {}", i))?; - - // Single writer per shard — max_connections(1) serialises writes. - let write_pool = SqlitePoolOptions::new() - .max_connections(1) - .connect_with(base_opts) - .await - .into_diagnostic() - .wrap_err(format!("Failed to open write pool for shard {}", i))?; - - Self::create_schema(&write_pool) - .await - .wrap_err(format!("Failed to create schema for shard {}", i))?; - - let (tx, rx) = mpsc::channel::(1024); - tokio::spawn(shard_writer_task( - i, - write_pool, - rx, - batch_size, - batch_timeout_ms, - )); - - read_pools.push(read_pool); - shard_senders.push(tx); - } - - Ok(SqliteStorage { - read_pools, - shard_senders, - num_shards, - }) - } - - async fn create_schema(pool: &SqlitePool) -> Result<()> { - sqlx::query( - "CREATE TABLE IF NOT EXISTS emails ( - message_id TEXT PRIMARY KEY, - status INTEGER NOT NULL, - from_addr TEXT NOT NULL, - to_addrs TEXT NOT NULL, - body BLOB NOT NULL, - queued_at INTEGER, - attempts INTEGER NOT NULL DEFAULT 0, - last_attempt INTEGER, - next_attempt INTEGER, - last_error TEXT, - created_at INTEGER NOT NULL, - updated_at INTEGER NOT NULL - )", - ) - .execute(pool) - .await - .into_diagnostic() - .wrap_err("Failed to create emails table")?; - - sqlx::query("CREATE INDEX IF NOT EXISTS idx_status ON emails(status)") - .execute(pool) - .await - .into_diagnostic() - .wrap_err("Failed to create idx_status")?; - - sqlx::query( - "CREATE INDEX IF NOT EXISTS idx_deferred_next ON emails(next_attempt) WHERE status = 1", - ) - .execute(pool) - .await - .into_diagnostic() - .wrap_err("Failed to create idx_deferred_next")?; - - sqlx::query( - "CREATE INDEX IF NOT EXISTS idx_bounced_updated ON emails(updated_at) WHERE status = 2", - ) - .execute(pool) - .await - .into_diagnostic() - .wrap_err("Failed to create idx_bounced_updated")?; - - Ok(()) - } -} - -fn now_ms() -> i64 { - std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .unwrap_or_default() - .as_millis() as i64 -} - -fn extract_responder(op: ShardWriteOp) -> oneshot::Sender> { - match op { - ShardWriteOp::Put { responder, .. } - | ShardWriteOp::PutMeta { responder, .. } - | ShardWriteOp::Delete { responder, .. } - | ShardWriteOp::DeleteMeta { responder, .. } - | ShardWriteOp::Mv { responder, .. } - | ShardWriteOp::Cleanup { responder, .. } => responder, - } -} - -async fn process_write_batch( - shard_id: usize, - pool: &SqlitePool, - batch: &mut VecDeque, -) { - use std::time::UNIX_EPOCH; - - let now = now_ms(); - let ops: Vec = batch.drain(..).collect(); - let mut responders: Vec>> = Vec::with_capacity(ops.len()); - - let mut tx = match pool.begin().await.into_diagnostic() { - Ok(tx) => tx, - Err(e) => { - let err_msg = e.to_string(); - for op in ops { - let resp = extract_responder(op); - let _ = resp.send(Err(miette::miette!("{}", err_msg))); - } - return; - } - }; - - let mut batch_failed = false; - let mut batch_error: Option = None; - - for op in ops { - match op { - ShardWriteOp::Put { - email, - status, - responder, - } => { - let to_addrs = match serde_json::to_string(&email.to) { - Ok(s) => s, - Err(e) => { - let _ = responder - .send(Err(miette::miette!("failed to serialize to_addrs: {}", e))); - continue; - } - }; - responders.push(responder); - let queued_at = email.queued_at.map(|dt| dt.timestamp_millis()); - if let Err(e) = sqlx::query( - "INSERT OR REPLACE INTO emails \ - (message_id, status, from_addr, to_addrs, body, queued_at, attempts, last_attempt, next_attempt, created_at, updated_at) \ - VALUES (?, ?, ?, ?, ?, ?, 0, NULL, NULL, ?, ?)", - ) - .bind(&email.message_id) - .bind(status) - .bind(&email.from) - .bind(&to_addrs) - .bind(email.body.as_bytes()) - .bind(queued_at) - .bind(now) - .bind(now) - .execute(&mut *tx) - .await - { - tracing::error!(shard_id, error = %e, message_id = %email.message_id, "Put failed"); - if !batch_failed { - batch_failed = true; - batch_error = Some(format!("Put failed: {}", e)); - break; - } - } - } - ShardWriteOp::PutMeta { - key, - meta, - responder, - } => { - responders.push(responder); - let last_attempt = meta - .last_attempt - .duration_since(UNIX_EPOCH) - .unwrap_or_default() - .as_millis() as i64; - let next_attempt = meta - .next_attempt - .duration_since(UNIX_EPOCH) - .unwrap_or_default() - .as_millis() as i64; - if let Err(e) = sqlx::query( - "UPDATE emails SET attempts = ?, last_attempt = ?, next_attempt = ?, last_error = ?, updated_at = ? WHERE message_id = ?", - ) - .bind(meta.attempts as i64) - .bind(last_attempt) - .bind(next_attempt) - .bind(&meta.last_error) - .bind(now) - .bind(&key) - .execute(&mut *tx) - .await - { - tracing::error!(shard_id, error = %e, key = %key, "PutMeta failed"); - if !batch_failed { - batch_failed = true; - batch_error = Some(format!("PutMeta failed: {}", e)); - break; - } - } - } - ShardWriteOp::Delete { - key, - status, - responder, - } => { - responders.push(responder); - if let Err(e) = - sqlx::query("DELETE FROM emails WHERE message_id = ? AND status = ?") - .bind(&key) - .bind(status) - .execute(&mut *tx) - .await - { - tracing::error!(shard_id, error = %e, key = %key, "Delete failed"); - if !batch_failed { - batch_failed = true; - batch_error = Some(format!("Delete failed: {}", e)); - break; - } - } - } - ShardWriteOp::DeleteMeta { key, responder } => { - responders.push(responder); - if let Err(e) = sqlx::query( - "UPDATE emails SET attempts = 0, last_attempt = NULL, next_attempt = NULL, last_error = NULL, updated_at = ? WHERE message_id = ?", - ) - .bind(now) - .bind(&key) - .execute(&mut *tx) - .await - { - tracing::error!(shard_id, error = %e, key = %key, "DeleteMeta failed"); - if !batch_failed { - batch_failed = true; - batch_error = Some(format!("DeleteMeta failed: {}", e)); - break; - } - } - } - ShardWriteOp::Mv { - src_key, - dest_key, - src_status, - dest_status, - responder, - } => { - responders.push(responder); - match sqlx::query( - "UPDATE emails SET message_id = ?, status = ?, updated_at = ? WHERE message_id = ? AND status = ?", - ) - .bind(&dest_key) - .bind(dest_status) - .bind(now) - .bind(&src_key) - .bind(src_status) - .execute(&mut *tx) - .await - { - Ok(result) if result.rows_affected() == 0 => { - tracing::error!(shard_id, src_key = %src_key, src_status, "Mv matched no rows"); - if !batch_failed { - batch_failed = true; - batch_error = Some(format!("Mv matched no rows for key={} status={}", src_key, src_status)); - break; - } - } - Err(e) => { - tracing::error!(shard_id, error = %e, src_key = %src_key, "Mv failed"); - if !batch_failed { - batch_failed = true; - batch_error = Some(format!("Mv failed: {}", e)); - break; - } - } - Ok(_) => {} - } - } - ShardWriteOp::Cleanup { config, responder } => { - responders.push(responder); - if let Some(bounced_retention) = config.bounced_retention { - let cutoff = now - bounced_retention.as_millis() as i64; - if let Err(e) = - sqlx::query("DELETE FROM emails WHERE status = ? AND updated_at < ?") - .bind(status_to_int(&Status::Bounced)) - .bind(cutoff) - .execute(&mut *tx) - .await - { - tracing::error!(shard_id, error = %e, "Cleanup bounced failed"); - if !batch_failed { - batch_failed = true; - batch_error = Some(format!("Cleanup bounced failed: {}", e)); - break; - } - } - } - if let Some(deferred_retention) = config.deferred_retention { - let cutoff = now - deferred_retention.as_millis() as i64; - if let Err(e) = - sqlx::query("DELETE FROM emails WHERE status = ? AND updated_at < ?") - .bind(status_to_int(&Status::Deferred)) - .bind(cutoff) - .execute(&mut *tx) - .await - { - tracing::error!(shard_id, error = %e, "Cleanup deferred failed"); - if !batch_failed { - batch_failed = true; - batch_error = Some(format!("Cleanup deferred failed: {}", e)); - break; - } - } - } - } - } - } - - if batch_failed { - drop(tx); // implicit rollback - let err_msg = batch_error.unwrap_or_else(|| "batch failed".to_string()); - for resp in responders { - let _ = resp.send(Err(miette::miette!("{}", err_msg))); - } - return; - } - - match tx.commit().await.into_diagnostic() { - Ok(()) => { - for resp in responders { - let _ = resp.send(Ok(())); - } - } - Err(e) => { - let err_msg = e.to_string(); - for resp in responders { - let _ = resp.send(Err(miette::miette!("{}", err_msg))); - } - } - } -} - -async fn shard_writer_task( - shard_id: usize, - pool: SqlitePool, - mut receiver: mpsc::Receiver, - batch_size: usize, - batch_timeout_ms: u64, -) { - let batch_timeout = Duration::from_millis(batch_timeout_ms); - tracing::info!( - shard_id, - batch_size, - batch_timeout_ms, - "shard writer task started" - ); - - let mut batch: VecDeque = VecDeque::with_capacity(batch_size); - - loop { - // Wait for first op (blocking) - if batch.is_empty() { - match receiver.recv().await { - Some(op) => batch.push_back(op), - None => break, - } - } - - // Fill batch up to size or timeout - while batch.len() < batch_size { - match tokio::time::timeout(batch_timeout, receiver.recv()).await { - Ok(Some(op)) => batch.push_back(op), - Ok(None) => break, - Err(_) => break, // timeout - } - } - - // Drain any immediately available - while batch.len() < batch_size { - match receiver.try_recv() { - Ok(op) => batch.push_back(op), - Err(_) => break, - } - } - - if !batch.is_empty() { - process_write_batch(shard_id, &pool, &mut batch).await; - } - } - - // Drain everything remaining in the channel on shutdown. - while let Ok(op) = receiver.try_recv() { - batch.push_back(op); - } - if !batch.is_empty() { - tracing::info!( - shard_id, - remaining = batch.len(), - "flushing remaining ops before shutdown" - ); - process_write_batch(shard_id, &pool, &mut batch).await; - } - - tracing::info!(shard_id, "shard writer task stopped"); -} - -#[async_trait] -impl Storage for SqliteStorage { - // ------------------------------------------------------------------------- - // Write methods — routed through the shard writer channel - // ------------------------------------------------------------------------- - - async fn put(&self, email: StoredEmail, status: Status) -> Result<()> { - let shard = self.shard_for(&email.message_id); - let (tx, rx) = oneshot::channel(); - self.shard_senders[shard] - .send(ShardWriteOp::Put { - email, - status: status_to_int(&status), - responder: tx, - }) - .await - .map_err(|_| miette::miette!("shard writer channel closed"))?; - - rx.await - .map_err(|_| miette::miette!("shard writer dropped responder"))? - } - - async fn put_meta(&self, key: &str, meta: &EmailMetadata) -> Result<()> { - let shard = self.shard_for(key); - let (tx, rx) = oneshot::channel(); - self.shard_senders[shard] - .send(ShardWriteOp::PutMeta { - key: key.to_string(), - meta: EmailMetadata { - msg_id: meta.msg_id.clone(), - attempts: meta.attempts, - last_attempt: meta.last_attempt, - next_attempt: meta.next_attempt, - last_error: meta.last_error.clone(), - }, - responder: tx, - }) - .await - .map_err(|_| miette::miette!("shard writer channel closed"))?; - rx.await - .map_err(|_| miette::miette!("shard writer dropped responder"))? - } - - async fn delete(&self, key: &str, status: Status) -> Result<()> { - let shard = self.shard_for(key); - let (tx, rx) = oneshot::channel(); - self.shard_senders[shard] - .send(ShardWriteOp::Delete { - key: key.to_string(), - status: status_to_int(&status), - responder: tx, - }) - .await - .map_err(|_| miette::miette!("shard writer channel closed"))?; - rx.await - .map_err(|_| miette::miette!("shard writer dropped responder"))? - } - - async fn delete_meta(&self, key: &str) -> Result<()> { - let shard = self.shard_for(key); - let (tx, rx) = oneshot::channel(); - self.shard_senders[shard] - .send(ShardWriteOp::DeleteMeta { - key: key.to_string(), - responder: tx, - }) - .await - .map_err(|_| miette::miette!("shard writer channel closed"))?; - rx.await - .map_err(|_| miette::miette!("shard writer dropped responder"))? - } - - async fn mv( - &self, - src_key: &str, - dest_key: &str, - src_status: Status, - dest_status: Status, - ) -> Result<()> { - let shard = self.shard_for(src_key); - // If dest_key hashes to a different shard, the row becomes unreachable. - // All current callers pass src_key == dest_key, so this is a safety net. - let dest_shard = self.shard_for(dest_key); - if shard != dest_shard { - return Err(miette::miette!( - "mv across shards is not supported: src_key={} (shard {}) dest_key={} (shard {})", - src_key, - shard, - dest_key, - dest_shard - )); - } - let (tx, rx) = oneshot::channel(); - self.shard_senders[shard] - .send(ShardWriteOp::Mv { - src_key: src_key.to_string(), - dest_key: dest_key.to_string(), - src_status: status_to_int(&src_status), - dest_status: status_to_int(&dest_status), - responder: tx, - }) - .await - .map_err(|_| miette::miette!("shard writer channel closed"))?; - rx.await - .map_err(|_| miette::miette!("shard writer dropped responder"))? - } - - /// Sends `Cleanup` to every shard and awaits all responses. - async fn cleanup(&self, config: &CleanupConfig) -> Result<()> { - // Send to all shards first to allow concurrent processing. - let mut receivers = Vec::with_capacity(self.num_shards); - for sender in &self.shard_senders { - let (tx, rx) = oneshot::channel(); - sender - .send(ShardWriteOp::Cleanup { - config: config.clone(), - responder: tx, - }) - .await - .map_err(|_| miette::miette!("shard writer channel closed"))?; - receivers.push(rx); - } - for rx in receivers { - rx.await - .map_err(|_| miette::miette!("shard writer dropped responder"))??; - } - Ok(()) - } - - // ------------------------------------------------------------------------- - // Read methods — direct read pool queries - // ------------------------------------------------------------------------- - - async fn get(&self, key: &str, status: Status) -> Result> { - let shard = self.shard_for(key); - let pool = &self.read_pools[shard]; - let status_int = status_to_int(&status); - - let row = sqlx::query_as::<_, (String, String, String, Vec, Option)>( - "SELECT message_id, from_addr, to_addrs, body, queued_at \ - FROM emails WHERE message_id = ? AND status = ?", - ) - .bind(key) - .bind(status_int) - .fetch_optional(pool) - .await - .into_diagnostic()?; - - match row { - None => Ok(None), - Some((message_id, from_addr, to_json, body_bytes, queued_at)) => { - let to: Vec = serde_json::from_str(&to_json) - .into_diagnostic() - .wrap_err("failed to deserialize to_addrs")?; - let body = String::from_utf8(body_bytes) - .into_diagnostic() - .wrap_err("email body is not valid UTF-8")?; - let queued_at = queued_at.and_then(chrono::DateTime::from_timestamp_millis); - Ok(Some(StoredEmail { - message_id, - from: from_addr, - to, - body, - queued_at, - })) - } - } - } - - async fn get_meta(&self, key: &str) -> Result> { - let shard = self.shard_for(key); - let pool = &self.read_pools[shard]; - - let row = sqlx::query_as::<_, (String, i64, Option, Option, Option)>( - "SELECT message_id, attempts, last_attempt, next_attempt, last_error \ - FROM emails WHERE message_id = ?", - ) - .bind(key) - .fetch_optional(pool) - .await - .into_diagnostic()?; - - match row { - None => Ok(None), - Some((msg_id, attempts, last_ms, next_ms, last_error)) => { - use std::time::{Duration as StdDuration, UNIX_EPOCH}; - let last_attempt = last_ms - .map(|ms| UNIX_EPOCH + StdDuration::from_millis(ms as u64)) - .unwrap_or(UNIX_EPOCH); - let next_attempt = next_ms - .map(|ms| UNIX_EPOCH + StdDuration::from_millis(ms as u64)) - .unwrap_or(UNIX_EPOCH); - Ok(Some(EmailMetadata { - msg_id, - attempts: attempts as u32, - last_attempt, - next_attempt, - last_error, - })) - } - } - } - - // ------------------------------------------------------------------------- - // List methods — fan out across all shards - // ------------------------------------------------------------------------- - - fn list(&self, status: Status) -> Pin> + Send>> { - let pools = self.read_pools.clone(); - let status_int = status_to_int(&status); - - Box::pin(async_stream::try_stream! { - for pool in &pools { - let rows = sqlx::query_as::<_, (String, String, String, Vec, Option)>( - "SELECT message_id, from_addr, to_addrs, body, queued_at \ - FROM emails WHERE status = ?", - ) - .bind(status_int) - .fetch_all(pool) - .await - .into_diagnostic()?; - - for (message_id, from_addr, to_json, body_bytes, queued_at) in rows { - let to: Vec = serde_json::from_str(&to_json) - .into_diagnostic() - .wrap_err("failed to deserialize to_addrs")?; - let body = String::from_utf8(body_bytes) - .into_diagnostic() - .wrap_err("email body is not valid UTF-8")?; - let queued_at = - queued_at.and_then(chrono::DateTime::from_timestamp_millis); - yield StoredEmail { - message_id, - from: from_addr, - to, - body, - queued_at, - }; - } - } - }) - } - - fn list_meta(&self) -> Pin> + Send>> { - let pools = self.read_pools.clone(); - - Box::pin(async_stream::try_stream! { - use std::time::{Duration as StdDuration, UNIX_EPOCH}; - - for pool in &pools { - let rows = sqlx::query_as::<_, (String, i64, Option, Option, Option)>( - "SELECT message_id, attempts, last_attempt, next_attempt, last_error \ - FROM emails WHERE status = 1", - ) - .fetch_all(pool) - .await - .into_diagnostic()?; - - for (msg_id, attempts, last_ms, next_ms, last_error) in rows { - let last_attempt = last_ms - .map(|ms| UNIX_EPOCH + StdDuration::from_millis(ms as u64)) - .unwrap_or(UNIX_EPOCH); - let next_attempt = next_ms - .map(|ms| UNIX_EPOCH + StdDuration::from_millis(ms as u64)) - .unwrap_or(UNIX_EPOCH); - yield EmailMetadata { - msg_id, - attempts: attempts as u32, - last_attempt, - next_attempt, - last_error, - }; - } - } - }) - } -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::storage::CleanupConfig; - use futures::StreamExt; - use std::time::{Duration, SystemTime}; - use tempfile::tempdir; - - fn default_sqlite_cfg() -> CfgSqlite { - CfgSqlite { - synchronous: None, - cache_size_mb: None, - busy_timeout_ms: None, - pool_max_connections: Some(2), - } - } - - async fn create_test_storage() -> SqliteStorage { - let temp_dir = tempdir().unwrap(); - let base_path = temp_dir.path().to_str().unwrap().to_string(); - let cfg = default_sqlite_cfg(); - let storage = SqliteStorage::new(&base_path, 2, 10, 5, &cfg) - .await - .unwrap(); - // Leak temp_dir so it's not deleted while storage is alive. - std::mem::forget(temp_dir); - storage - } - - fn create_test_email(id: &str) -> StoredEmail { - StoredEmail { - message_id: id.to_string(), - from: "sender@example.com".to_string(), - to: vec!["recipient@example.com".to_string()], - body: "Test email body".to_string(), - queued_at: None, - } - } - - #[tokio::test] - async fn test_new_creates_shard_databases() { - let temp_dir = tempdir().unwrap(); - let base_path = temp_dir.path().to_str().unwrap(); - let cfg = default_sqlite_cfg(); - let storage = SqliteStorage::new(base_path, 4, 10, 5, &cfg).await.unwrap(); - - assert_eq!(storage.num_shards, 4); - assert_eq!(storage.read_pools.len(), 4); - assert_eq!(storage.shard_senders.len(), 4); - - for i in 0..4 { - let db_path = temp_dir.path().join(format!("shard_{}.db", i)); - assert!(db_path.exists(), "shard_{}.db should exist", i); - } - - // Verify schema exists - for (i, pool) in storage.read_pools.iter().enumerate() { - let row: (i64,) = sqlx::query_as("SELECT COUNT(*) FROM emails") - .fetch_one(pool) - .await - .unwrap_or_else(|e| panic!("shard {} query failed: {}", i, e)); - assert_eq!(row.0, 0); - } - } - - #[tokio::test] - async fn test_shard_distribution() { - let temp_dir = tempdir().unwrap(); - let base_path = temp_dir.path().to_str().unwrap(); - let cfg = default_sqlite_cfg(); - let storage = SqliteStorage::new(base_path, 4, 10, 5, &cfg).await.unwrap(); - - let shard_a = storage.shard_for("msg_aaa"); - let shard_b = storage.shard_for("msg_aaa"); - assert_eq!(shard_a, shard_b, "same key should map to same shard"); - - for i in 0..100 { - let key = format!("msg_{}", i); - let shard = storage.shard_for(&key); - assert!(shard < 4, "shard {} out of range for key {}", shard, key); - } - } - - #[tokio::test] - async fn test_put_and_get() { - let storage = create_test_storage().await; - let email = create_test_email("msg_001"); - - storage.put(email.clone(), Status::Queued).await.unwrap(); - - // Correct status → Some with all fields matching - let result = storage.get("msg_001", Status::Queued).await.unwrap(); - assert!(result.is_some()); - let retrieved = result.unwrap(); - assert_eq!(retrieved.message_id, "msg_001"); - assert_eq!(retrieved.from, "sender@example.com"); - assert_eq!(retrieved.to, vec!["recipient@example.com"]); - assert_eq!(retrieved.body, "Test email body"); - - // Wrong status → None - let result = storage.get("msg_001", Status::Bounced).await.unwrap(); - assert!(result.is_none()); - - // Nonexistent key → None - let result = storage.get("nonexistent", Status::Queued).await.unwrap(); - assert!(result.is_none()); - } - - #[tokio::test] - async fn test_delete() { - let storage = create_test_storage().await; - let email = create_test_email("msg_del"); - - storage.put(email, Status::Queued).await.unwrap(); - storage.delete("msg_del", Status::Queued).await.unwrap(); - - let result = storage.get("msg_del", Status::Queued).await.unwrap(); - assert!(result.is_none()); - } - - #[tokio::test] - async fn test_mv() { - let storage = create_test_storage().await; - let email = create_test_email("msg_mv"); - - storage.put(email, Status::Queued).await.unwrap(); - storage - .mv("msg_mv", "msg_mv", Status::Queued, Status::Bounced) - .await - .unwrap(); - - // Get as old status → None - let result = storage.get("msg_mv", Status::Queued).await.unwrap(); - assert!(result.is_none()); - - // Get as new status → Some - let result = storage.get("msg_mv", Status::Bounced).await.unwrap(); - assert!(result.is_some()); - } - - #[tokio::test] - async fn test_put_and_get_meta() { - let storage = create_test_storage().await; - let email = create_test_email("msg_meta"); - - // Row must exist before put_meta (which does an UPDATE) - storage.put(email, Status::Deferred).await.unwrap(); - - let now = SystemTime::now(); - let meta = crate::worker::EmailMetadata { - msg_id: "msg_meta".to_string(), - attempts: 3, - last_attempt: now, - next_attempt: now + Duration::from_secs(300), - last_error: Some("transient error (421): 4.7.0 throttled".to_string()), - }; - storage.put_meta("msg_meta", &meta).await.unwrap(); - - let result = storage.get_meta("msg_meta").await.unwrap(); - assert!(result.is_some()); - let retrieved = result.unwrap(); - assert_eq!(retrieved.msg_id, "msg_meta"); - assert_eq!(retrieved.attempts, 3); - assert_eq!( - retrieved.last_error.as_deref(), - Some("transient error (421): 4.7.0 throttled") - ); - - // Nonexistent key → None - let result = storage.get_meta("nonexistent").await.unwrap(); - assert!(result.is_none()); - } - - #[tokio::test] - async fn test_delete_meta() { - let storage = create_test_storage().await; - let email = create_test_email("msg_delmeta"); - - storage.put(email, Status::Deferred).await.unwrap(); - - let now = SystemTime::now(); - let meta = crate::worker::EmailMetadata { - msg_id: "msg_delmeta".to_string(), - attempts: 2, - last_attempt: now, - next_attempt: now + Duration::from_secs(60), - last_error: None, - }; - storage.put_meta("msg_delmeta", &meta).await.unwrap(); - - // Verify meta is set - let before = storage.get_meta("msg_delmeta").await.unwrap(); - assert_eq!(before.unwrap().attempts, 2); - - storage.delete_meta("msg_delmeta").await.unwrap(); - - // delete_meta resets attempts to 0 (row stays, metadata is cleared) - let after = storage.get_meta("msg_delmeta").await.unwrap(); - assert!(after.is_some()); - assert_eq!(after.unwrap().attempts, 0); - } - - #[tokio::test] - async fn test_list() { - let storage = create_test_storage().await; - - storage - .put(create_test_email("msg_list_q1"), Status::Queued) - .await - .unwrap(); - storage - .put(create_test_email("msg_list_q2"), Status::Queued) - .await - .unwrap(); - storage - .put(create_test_email("msg_list_d1"), Status::Deferred) - .await - .unwrap(); - - let queued: Vec<_> = storage.list(Status::Queued).collect().await; - assert_eq!(queued.len(), 2, "expected 2 queued emails"); - - let deferred: Vec<_> = storage.list(Status::Deferred).collect().await; - assert_eq!(deferred.len(), 1, "expected 1 deferred email"); - - let bounced: Vec<_> = storage.list(Status::Bounced).collect().await; - assert_eq!(bounced.len(), 0, "expected 0 bounced emails"); - } - - #[tokio::test] - async fn test_list_meta() { - let storage = create_test_storage().await; - - storage - .put(create_test_email("msg_lm1"), Status::Deferred) - .await - .unwrap(); - storage - .put(create_test_email("msg_lm2"), Status::Deferred) - .await - .unwrap(); - - let now = SystemTime::now(); - let meta1 = crate::worker::EmailMetadata { - msg_id: "msg_lm1".to_string(), - attempts: 1, - last_attempt: now, - next_attempt: now + Duration::from_secs(60), - last_error: None, - }; - let meta2 = crate::worker::EmailMetadata { - msg_id: "msg_lm2".to_string(), - attempts: 2, - last_attempt: now, - next_attempt: now + Duration::from_secs(120), - last_error: None, - }; - storage.put_meta("msg_lm1", &meta1).await.unwrap(); - storage.put_meta("msg_lm2", &meta2).await.unwrap(); - - let metas: Vec<_> = storage.list_meta().collect().await; - assert_eq!(metas.len(), 2, "expected 2 metadata entries"); - - let ids: Vec = metas.into_iter().map(|r| r.unwrap().msg_id).collect(); - assert!(ids.contains(&"msg_lm1".to_string())); - assert!(ids.contains(&"msg_lm2".to_string())); - } - - #[tokio::test] - async fn test_cleanup_bounced_removes_old_messages() { - let storage = create_test_storage().await; - let email = create_test_email("msg_cleanup_b"); - - storage.put(email, Status::Bounced).await.unwrap(); - tokio::time::sleep(Duration::from_millis(20)).await; - - let config = CleanupConfig { - bounced_retention: Some(Duration::from_millis(1)), - deferred_retention: None, - interval: Duration::from_secs(3600), - }; - storage.cleanup(&config).await.unwrap(); - - let result = storage.get("msg_cleanup_b", Status::Bounced).await.unwrap(); - assert!( - result.is_none(), - "old bounced email should have been removed" - ); - } - - #[tokio::test] - async fn test_cleanup_deferred_removes_old_messages() { - let storage = create_test_storage().await; - let email = create_test_email("msg_cleanup_d"); - - storage.put(email, Status::Deferred).await.unwrap(); - tokio::time::sleep(Duration::from_millis(20)).await; - - let config = CleanupConfig { - bounced_retention: None, - deferred_retention: Some(Duration::from_millis(1)), - interval: Duration::from_secs(3600), - }; - storage.cleanup(&config).await.unwrap(); - - let result = storage - .get("msg_cleanup_d", Status::Deferred) - .await - .unwrap(); - assert!( - result.is_none(), - "old deferred email should have been removed" - ); - } - - #[tokio::test] - async fn test_cleanup_does_not_remove_recent_messages() { - let storage = create_test_storage().await; - let email = create_test_email("msg_cleanup_recent"); - - storage.put(email, Status::Bounced).await.unwrap(); - - let config = CleanupConfig { - bounced_retention: Some(Duration::from_secs(3600)), - deferred_retention: None, - interval: Duration::from_secs(3600), - }; - storage.cleanup(&config).await.unwrap(); - - let result = storage - .get("msg_cleanup_recent", Status::Bounced) - .await - .unwrap(); - assert!( - result.is_some(), - "recent bounced email should not have been removed" - ); - } -} diff --git a/smtp-server/src/worker/log_worker.rs b/smtp-server/src/worker/log_worker.rs new file mode 100644 index 0000000..8ab5190 --- /dev/null +++ b/smtp-server/src/worker/log_worker.rs @@ -0,0 +1,83 @@ +//! Log-queue delivery workers: pull claims from the dispatcher, read the +//! message body by location, deliver, and report the outcome. Unlike the +//! legacy channel workers they never sleep on rate limits and never touch +//! the queue storage — the dispatcher owns all queue state. + +use std::time::Duration; + +use chrono::Utc; +use tracing::{error, info}; + +use crate::logqueue::dispatcher::{DispatcherHandle, JobOutcome, RateGate}; +use crate::worker::rate_limiter::RateLimiter; +use crate::worker::Worker; + +/// Bridges the shared per-domain [`RateLimiter`] into the dispatcher's +/// dispatch-time gate. The gate peeks (non-consuming); the worker's check +/// immediately before transmission is the real token acquisition. +pub struct LimiterGate(pub RateLimiter); + +impl RateGate for LimiterGate { + fn check(&self, domain: &str) -> Option { + self.0.peek_sync(domain) + } +} + +pub struct LogWorker { + worker: Worker, + dispatcher: DispatcherHandle, + max_retries: u32, +} + +impl LogWorker { + pub fn new(worker: Worker, dispatcher: DispatcherHandle, max_retries: u32) -> Self { + Self { + worker, + dispatcher, + max_retries, + } + } + + /// Pull-and-deliver loop; exits when the dispatcher shuts down. + pub async fn run(self) { + while let Some(claim) = self.dispatcher.claim().await { + let job = claim.job.clone(); + + let body = match self.dispatcher.read_body(job.location).await { + Ok(body) => body, + Err(e) => { + // Unreadable payload: defer with backoff rather than + // guessing terminal state; a transient I/O problem must + // not lose mail. + error!(msg_id = %job.message_id, error = %e, "failed to read message body"); + claim.report(JobOutcome::Deferred { + next_attempt_ms: Utc::now().timestamp_millis() + + 60_000 * (1 << job.attempts.min(10)) as i64, + remaining_recipients: job.recipients.clone(), + error: format!("payload read failed: {e}"), + }); + continue; + } + }; + + if job.attempts >= self.max_retries { + info!( + msg_id = %job.message_id, + attempts = job.attempts, + max_retries = self.max_retries, + "maximum retry attempts exceeded; bouncing" + ); + let outcome = self + .worker + .bounce_claim_for_retry_limit(&job, &body) + .await; + claim.report(outcome); + continue; + } + + let outcome = self.worker.process_claim(&job, &body).await; + claim.report(outcome); + } + tracing::debug!("log worker stopped: dispatcher closed"); + } +} diff --git a/smtp-server/src/worker/mod.rs b/smtp-server/src/worker/mod.rs index bf76ba6..757e7fc 100644 --- a/smtp-server/src/worker/mod.rs +++ b/smtp-server/src/worker/mod.rs @@ -40,6 +40,7 @@ use crate::{ }; pub mod deferred_worker; +pub mod log_worker; mod pool; pub mod rate_limiter; @@ -117,6 +118,12 @@ pub(crate) struct WorkerResources { } impl WorkerResources { + /// The process-wide rate limiter (clones share the same buckets); used + /// by the log-queue dispatcher's dispatch-time gate. + pub(crate) fn rate_limiter(&self) -> RateLimiter { + self.rate_limiter.clone() + } + pub(crate) fn new( mx_cache: Cache, pool: Arc, @@ -515,22 +522,12 @@ impl Worker { Ok(new_email) } - async fn send_email<'b>( - &self, - to: &[String], - email: &'b Message<'b>, - body: &str, - ctx: &DeliveryContext<'b>, - ) -> Result<()> { + /// Strip Bcc headers and DKIM-sign (when configured), producing the + /// final outbound bytes. + fn sign_outbound(&self, body: &[u8]) -> Result> { let email_bytes_no_bcc = - Self::remove_bcc_header(body.as_bytes()).wrap_err("Failed to remove Bcc header")?; - let from = email - .from() - .and_then(|f| f.first()) - .and_then(|f| f.address()) - .ok_or_else(|| miette::miette!("Invalid from address"))?; - let signed_email; - let raw_email = match &self.dkim_signer { + Self::remove_bcc_header(body).wrap_err("Failed to remove Bcc header")?; + match &self.dkim_signer { Some(signer) => { debug!("Signing email with DKIM"); let signature = match signer { @@ -555,22 +552,24 @@ impl Worker { header } }; - - signed_email = Self::insert_dkim_signature(&email_bytes_no_bcc, &signature)?; - signed_email.as_slice() + Self::insert_dkim_signature(&email_bytes_no_bcc, &signature) } - None => email_bytes_no_bcc.as_slice(), - }; + None => Ok(email_bytes_no_bcc), + } + } + /// The union of envelope recipients and any Cc/Bcc addresses parsed out + /// of the message, deduplicated (historical behavior of `send_email`). + fn merge_recipients(to: &[String], email: &Message<'_>) -> Vec { let to_iter = to.iter().map(|s| s.to_owned()); let cc_iter = email .cc() .into_iter() .flat_map(|list| list.as_list()) - .flatten() // Iterator yielding &Address for all addresses in the list(s) - .filter_map(|cc| cc.address()) // Iterator yielding &str - .map(|addr_str| addr_str.to_owned()); // Iterator + .flatten() + .filter_map(|cc| cc.address()) + .map(|addr_str| addr_str.to_owned()); let bcc_iter = email .bcc() @@ -581,28 +580,400 @@ impl Worker { .map(|addr_str| addr_str.to_owned()); let all_recipients: Vec = to_iter.chain(cc_iter).chain(bcc_iter).collect(); - - // Remove any duplicates. - let all_recipients: Vec = all_recipients + all_recipients .into_iter() .collect::>() .into_iter() - .collect(); + .collect() + } + + /// Attempt delivery of one recipient through its MX servers, applying + /// MTA-STS policy. Rate limiting and outcome logging stay with the + /// caller. `Err` is reserved for infrastructure failures (MX lookup, + /// transport pool); SMTP-level failures return `Failed` with the + /// classification made while the typed transport error is live. + async fn deliver_recipient( + &self, + raw_email: &[u8], + from: &str, + to_trimmed: &str, + parsed_email_id: &EmailAddress, + ) -> Result { + let domain = parsed_email_id.get_domain(); + debug!(?parsed_email_id, "Looking up MX records"); + + let mx_lookup = self + .lookup_mx(domain) + .await + .wrap_err("looking up mx record")?; + if mx_lookup.iter().count() == 0 { + warn!(domain = ?domain, "No MX records found"); + metrics::record_send_failure(domain); + return Ok(RecipientDelivery::Skipped("no MX records")); + } + + // Sort mx according to preference in ascending order. + let mut mx = mx_lookup.iter().collect::>(); + // Shuffle first so the stable sort randomizes equal-preference MXes. + mx.shuffle(&mut rand::thread_rng()); + mx.sort_by_key(|a| a.preference()); + + // Look up MTA-STS policy for the recipient domain. + let mta_sts_policy = self.mta_sts.get_policy(domain).await; + if let Some(ref policy) = mta_sts_policy { + debug!(domain = ?domain, mode = %policy.mode, "MTA-STS policy found"); + } + + let from_address: Address = from + .parse() + .map_err(|e| miette::miette!("invalid from address {from:?}: {e}"))?; + let to_address: Address = to_trimmed + .parse() + .map_err(|e| miette::miette!("invalid recipient address {to_trimmed:?}: {e}"))?; + let envelope = Envelope::new(Some(from_address), vec![to_address]).into_diagnostic()?; + + // Track the most recent per-MX failure as a typed error. We classify + // here — while the live `lettre::transport::smtp::Error` is still + // accessible — rather than wrapping it in a `miette::Report` and + // trying to downcast later (which doesn't work; see the unit test + // `into_diagnostic_makes_original_error_unreachable`). + let mut last_error: Option = None; + for mx_record in mx.iter() { + debug!(mx = ?mx_record.exchange(), "Attempting delivery via MX server"); + + let exchange = mx_record.exchange().to_string(); + + // MTA-STS: validate MX hostname against policy. + if let Some(ref policy) = mta_sts_policy { + let mx_valid = mta_sts_policy::mx_matches_policy(&exchange, policy); + + match policy.mode { + PolicyMode::Enforce => { + if !mx_valid { + warn!( + domain = ?domain, + mx = %exchange, + "MTA-STS enforce: MX host does not match policy, skipping" + ); + metrics::mta_sts_enforcement("enforce", "fail"); + continue; + } + metrics::mta_sts_enforcement("enforce", "pass"); + } + PolicyMode::Testing => { + if !mx_valid { + warn!( + domain = ?domain, + mx = %exchange, + "MTA-STS testing: MX host does not match policy (would be rejected in enforce mode)" + ); + metrics::mta_sts_enforcement("testing", "fail"); + } else { + metrics::mta_sts_enforcement("testing", "pass"); + } + } + PolicyMode::None => {} + } + } + let transport: AsyncSmtpTransport = self.pool.get(&exchange).await?; + + let send_start = Instant::now(); + match transport.send_raw(&envelope, raw_email).await { + Ok(response) => { + metrics::record_send_success(domain, send_start.elapsed()); + let smtp_response = response.message().collect::>().join(" "); + return Ok(RecipientDelivery::Delivered { + smtp_response: format!("{} {}", response.code(), smtp_response), + exchange, + }); + } + Err(err) => { + metrics::record_send_failure(domain); + // Classify here, where `err` still has its concrete + // lettre type. Build the response string up-front so + // the wrapping for `Report` carries no live error. + let outcome = classify_smtp_outcome( + err.is_transient(), + err.is_permanent(), + err.status().map(u16::from), + ); + let smtp_response = format!("sending raw message: {}", err); + warn!( + mx = %exchange, + outcome = ?outcome, + error = %smtp_response, + "MX delivery attempt failed" + ); + last_error = Some(ClassifiedSendError { + outcome, + smtp_response, + }); + } + } + } + + // If MTA-STS enforce mode caused all MXes to be skipped by policy + // (no transport-level errors), report that specifically so callers + // defer instead of bouncing. + if let Some(ref policy) = mta_sts_policy { + if policy.mode == PolicyMode::Enforce && last_error.is_none() { + return Ok(RecipientDelivery::MtaStsBlocked); + } + } + if let Some(classified) = last_error { + return Ok(RecipientDelivery::Failed(classified)); + } + metrics::record_send_failure(domain); + Ok(RecipientDelivery::Failed(ClassifiedSendError { + outcome: SendOutcome::Bounce, + smtp_response: "failed to send email through any MX server".to_string(), + })) + } + + /// Deliver one log-queue claim and translate the result into a + /// [`JobOutcome`]. This is the log backend's counterpart to + /// `process_job`: recipients are tracked individually so a retry only + /// re-sends to those that have not accepted the message, and a + /// rate-limit loss is reported instead of slept on — the worker slot is + /// never parked. + pub(crate) async fn process_claim( + &self, + job: &crate::logqueue::dispatcher::DeliveryJob, + body: &[u8], + ) -> crate::logqueue::dispatcher::JobOutcome { + use crate::logqueue::dispatcher::JobOutcome; + + let _job_guard = metrics::job_processing_guard(); + let queued_at = chrono::DateTime::from_timestamp_millis(job.enqueue_ms); + let log_delivery = |status: &str, recipient: &str, smtp_response: &str, attempt: u32| { + let logged_at = Utc::now(); + info!( + job_id = %job.message_id, + from_email = %strip_brackets(&job.sender), + recipient = %strip_brackets(recipient), + status = %status, + smtp_response = %smtp_response, + queued_at = %fmt_option_rfc3339(queued_at), + logged_at = %fmt_rfc3339(logged_at), + delay_ms = %fmt_option(calc_delay_ms(queued_at, logged_at)), + attempt = %attempt, + "email delivery" + ); + }; + + let Some(msg) = MessageParser::default().parse(body) else { + error!(msg_id = %job.message_id, "Failed to parse email body"); + return self + .bounce_claim(job, body, "unparseable message body".into()) + .await; + }; + + if self.disable_outbound { + log_delivery("dropped", &job.recipients.join(","), "outbound disabled", job.attempts); + metrics::email_dropped(); + return JobOutcome::Delivered { + response: "outbound disabled".into(), + }; + } + + // First attempt widens to Cc/Bcc like the legacy path; retries use + // exactly the persisted remaining set. + let recipients = if job.attempts == 0 { + Self::merge_recipients(&job.recipients, &msg) + } else { + job.recipients.clone() + }; + + let raw_email = match self.sign_outbound(body) { + Ok(raw) => raw, + Err(e) => { + return self + .bounce_claim(job, body, format!("preparing outbound message: {e:#}")) + .await; + } + }; + let Some(from) = msg + .from() + .and_then(|f| f.first()) + .and_then(|f| f.address()) + else { + return self + .bounce_claim(job, body, "invalid from address".into()) + .await; + }; + + let mut remaining: Vec = Vec::new(); + let mut rate_limited_wait: Option = None; + let mut attempted = false; + let mut last_defer_error: Option = None; + let mut bounce_reason: Option = None; + let mut delivered_any = false; + + for to in &recipients { + let to_trimmed = to.trim_matches(|c| c == '<' || c == '>'); + let Some(parsed_email_id) = EmailAddress::parse(to_trimmed, None) else { + continue; // dropped, matching legacy behavior + }; + let domain = parsed_email_id.get_domain(); + + match self.rate_limiter.check_rate_limit(domain).await { + RateLimitResult::Allowed => {} + RateLimitResult::RateLimited { retry_after } => { + remaining.push(to.clone()); + rate_limited_wait = + Some(rate_limited_wait.map_or(retry_after, |w: Duration| w.max(retry_after))); + continue; + } + } + + match self + .deliver_recipient(&raw_email, from, to_trimmed, &parsed_email_id) + .await + { + Ok(RecipientDelivery::Delivered { smtp_response, .. }) => { + attempted = true; + delivered_any = true; + log_delivery("delivered", to_trimmed, &smtp_response, job.attempts); + } + Ok(RecipientDelivery::Skipped(reason)) => { + attempted = true; + debug!(to = ?to, reason, "recipient skipped"); + } + Ok(RecipientDelivery::MtaStsBlocked) => { + attempted = true; + remaining.push(to.clone()); + last_defer_error = Some("MTA-STS enforcement failure".into()); + log_delivery( + "deferred", + to_trimmed, + "MTA-STS enforcement failure", + job.attempts + 1, + ); + } + Ok(RecipientDelivery::Failed(classified)) => { + attempted = true; + match classified.outcome { + SendOutcome::Defer => { + remaining.push(to.clone()); + log_delivery( + "deferred", + to_trimmed, + &classified.smtp_response, + job.attempts + 1, + ); + last_defer_error = Some(classified.smtp_response); + } + SendOutcome::Bounce => { + log_delivery( + "bounced", + to_trimmed, + &classified.smtp_response, + job.attempts, + ); + bounce_reason = Some(classified.smtp_response); + } + } + } + Err(e) => { + attempted = true; + let response = format!("{e:#}"); + log_delivery("bounced", to_trimmed, &response, job.attempts); + bounce_reason = Some(response); + } + } + } + + if !remaining.is_empty() { + if !attempted { + if let Some(retry_after) = rate_limited_wait { + // Pure rate-limit loss: not an attempt. + return JobOutcome::RateLimited { retry_after }; + } + } + let delay = std::cmp::min( + self.initial_delay * 2_u32.pow(job.attempts.min(24)), + self.max_delay, + ); + metrics::email_deferred(); + return JobOutcome::Deferred { + next_attempt_ms: Utc::now().timestamp_millis() + delay.as_millis() as i64, + remaining_recipients: remaining, + error: last_defer_error.unwrap_or_else(|| "rate limited".into()), + }; + } + + match bounce_reason { + Some(reason) if !delivered_any => self.bounce_claim(job, body, reason).await, + _ => { + metrics::email_sent(); + JobOutcome::Delivered { + response: "delivered".into(), + } + } + } + } + + /// Terminal bounce for a claim that exhausted its retry budget. + pub(crate) async fn bounce_claim_for_retry_limit( + &self, + job: &crate::logqueue::dispatcher::DeliveryJob, + body: &[u8], + ) -> crate::logqueue::dispatcher::JobOutcome { + self.bounce_claim(job, body, "maximum retry attempts exceeded".into()) + .await + } + + /// Archive a bounced message to the bounce store (retention handled by + /// the storage cleanup task), then report the terminal outcome. Archive + /// failure is logged but never blocks the bounce: at-least-once applies + /// to delivery, not to the archive copy. + async fn bounce_claim( + &self, + job: &crate::logqueue::dispatcher::DeliveryJob, + body: &[u8], + reason: String, + ) -> crate::logqueue::dispatcher::JobOutcome { + let archived = StoredEmail { + message_id: job.message_id.to_string(), + from: job.sender.clone(), + to: job.recipients.clone(), + body: String::from_utf8_lossy(body).into_owned(), + queued_at: chrono::DateTime::from_timestamp_millis(job.enqueue_ms), + }; + if let Err(e) = self.storage.put(archived, Status::Bounced).await { + error!(msg_id = %job.message_id, error = %e, "failed to archive bounced message"); + } + metrics::email_bounced(); + crate::logqueue::dispatcher::JobOutcome::Bounced { reason } + } + + async fn send_email<'b>( + &self, + to: &[String], + email: &'b Message<'b>, + body: &str, + ctx: &DeliveryContext<'b>, + ) -> Result<()> { + let raw_email = self.sign_outbound(body.as_bytes())?; + let from = email + .from() + .and_then(|f| f.first()) + .and_then(|f| f.address()) + .ok_or_else(|| miette::miette!("Invalid from address"))?; + + let all_recipients = Self::merge_recipients(to, email); // Parse to address for each. for to in all_recipients.iter() { info!(?to, ?from, "Attempting to send email"); // Strip `<` and `>` from email address. - let to = to.trim_matches(|c| c == '<' || c == '>'); - let parsed_email_id = EmailAddress::parse(to, None); - if parsed_email_id.is_none() { + let to_trimmed = to.trim_matches(|c| c == '<' || c == '>'); + let Some(parsed_email_id) = EmailAddress::parse(to_trimmed, None) else { continue; - } - - let parsed_email_id = parsed_email_id.unwrap(); + }; + let domain = parsed_email_id.get_domain(); // Check rate limit for this domain - let domain = parsed_email_id.get_domain(); match self.rate_limiter.check_rate_limit(domain).await { RateLimitResult::Allowed => { debug!(domain = ?domain, "Rate limit check passed"); @@ -617,177 +988,49 @@ impl Worker { } } - debug!(?parsed_email_id, "Looking up MX records"); - - // Resolve MX record for domain. - let mx_lookup = self - .lookup_mx(parsed_email_id.get_domain()) - .await - .wrap_err("looking up mx record")?; - if mx_lookup.iter().count() == 0 { - warn!(domain = ?parsed_email_id.get_domain(), "No MX records found"); - metrics::record_send_failure(parsed_email_id.get_domain()); - continue; - } - - // Sort mx according to preference in ascending order. - let mut mx = mx_lookup.iter().collect::>(); - - // Shuffle first so the stable sort randomizes equal-preference MXes. - mx.shuffle(&mut rand::thread_rng()); - mx.sort_by_key(|a| a.preference()); - - // Look up MTA-STS policy for the recipient domain. - let mta_sts_policy = self.mta_sts.get_policy(domain).await; - if let Some(ref policy) = mta_sts_policy { - debug!(domain = ?domain, mode = %policy.mode, "MTA-STS policy found"); - } - - let from: String = email - .from() - .unwrap() - .first() - .unwrap() - .address() - .as_ref() - .unwrap() - .to_string(); - - let from_address: Address = from.as_str().parse().unwrap(); - let to_address: Address = to.to_string().parse().unwrap(); - - let envelope = Envelope::new(Some(from_address), vec![to_address]).unwrap(); - - // Try each MX record in order of preference - let mut success = false; - // Track the most recent per-MX failure as a typed error. We classify - // here — while the live `lettre::transport::smtp::Error` is still - // accessible — rather than wrapping it in a `miette::Report` and - // trying to downcast later (which doesn't work; see the unit test - // `into_diagnostic_makes_original_error_unreachable`). - let mut last_error: Option = None; - for mx_record in mx.iter() { - debug!(mx = ?mx_record.exchange(), "Attempting delivery via MX server"); - - let exchange = mx_record.exchange().to_string(); - - // MTA-STS: validate MX hostname against policy. - if let Some(ref policy) = mta_sts_policy { - let mx_valid = mta_sts_policy::mx_matches_policy(&exchange, policy); - - match policy.mode { - PolicyMode::Enforce => { - if !mx_valid { - warn!( - domain = ?domain, - mx = %exchange, - "MTA-STS enforce: MX host does not match policy, skipping" - ); - metrics::mta_sts_enforcement("enforce", "fail"); - continue; - } - metrics::mta_sts_enforcement("enforce", "pass"); - } - PolicyMode::Testing => { - if !mx_valid { - warn!( - domain = ?domain, - mx = %exchange, - "MTA-STS testing: MX host does not match policy (would be rejected in enforce mode)" - ); - metrics::mta_sts_enforcement("testing", "fail"); - } else { - metrics::mta_sts_enforcement("testing", "pass"); - } - } - PolicyMode::None => {} - } + match self + .deliver_recipient(&raw_email, from, to_trimmed, &parsed_email_id) + .await? + { + RecipientDelivery::Delivered { + smtp_response, + exchange, + } => { + let logged_at = Utc::now(); + let delay_ms = calc_delay_ms(ctx.stored_email.queued_at, logged_at); + info!( + job_id = %ctx.job_id, + from_email = %strip_brackets(&ctx.stored_email.from), + recipient = %strip_brackets(to_trimmed), + subject = %fmt_option(ctx.subject), + status = "delivered", + smtp_response = %smtp_response, + dest_ip = %exchange, + queued_at = %fmt_option_rfc3339(ctx.stored_email.queued_at), + logged_at = %fmt_rfc3339(logged_at), + delay_ms = %fmt_option(delay_ms), + attempt = %ctx.attempt, + "email delivery" + ); } - let transport: AsyncSmtpTransport = - self.pool.get(&exchange).await?; - - let send_start = Instant::now(); - match transport.send_raw(&envelope, raw_email).await { - Ok(response) => { - metrics::record_send_success( - parsed_email_id.get_domain(), - send_start.elapsed(), - ); - let smtp_response = response.message().collect::>().join(" "); - let logged_at = Utc::now(); - let delay_ms = calc_delay_ms(ctx.stored_email.queued_at, logged_at); - info!( - job_id = %ctx.job_id, - from_email = %strip_brackets(&ctx.stored_email.from), - recipient = %strip_brackets(to), - subject = %fmt_option(ctx.subject), - status = "delivered", - smtp_response = %format!("{} {}", response.code(), smtp_response), - dest_ip = %exchange, - queued_at = %fmt_option_rfc3339(ctx.stored_email.queued_at), - logged_at = %fmt_rfc3339(logged_at), - delay_ms = %fmt_option(delay_ms), - attempt = %ctx.attempt, - "email delivery" - ); - success = true; - break; - } - Err(err) => { - metrics::record_send_failure(parsed_email_id.get_domain()); - // Classify here, where `err` still has its concrete - // lettre type. Build the response string up-front so - // the wrapping for `Report` carries no live error. - let outcome = classify_smtp_outcome( - err.is_transient(), - err.is_permanent(), - err.status().map(u16::from), - ); - let smtp_response = format!("sending raw message: {}", err); - warn!( - mx = %exchange, - outcome = ?outcome, - error = %smtp_response, - "MX delivery attempt failed" - ); - last_error = Some(ClassifiedSendError { - outcome, - smtp_response, - }); - } + RecipientDelivery::Skipped(reason) => { + debug!(to = ?to, reason, "recipient skipped"); } - } - - if !success { - // If MTA-STS enforce mode caused all MXes to be skipped by policy - // (no transport-level errors), return a specific error so process_job - // defers instead of bouncing. If there was a transport error (last_error - // is Some), that means at least one MX passed policy validation but - // failed at SMTP level — use the normal error path. - if let Some(ref policy) = mta_sts_policy { - if policy.mode == PolicyMode::Enforce && last_error.is_none() { - error!(to = ?to, "MTA-STS enforce: all MX hosts failed policy validation"); - metrics::record_send_failure(parsed_email_id.get_domain()); - // Use `Report::new` (NOT `into_diagnostic`) so the typed - // error survives downcast in `process_job`. Going through - // `into_diagnostic` wraps it in miette's pub(crate) - // DiagnosticError, making the type unreachable from the - // caller — which would silently fall through to the - // bounce arm, defeating the always-defer policy. See - // `mta_sts_error_via_into_diagnostic_is_unreachable`. - return Err(miette::Report::new(MtaStsEnforcementError { - domain: domain.to_string(), - })); - } + RecipientDelivery::MtaStsBlocked => { + error!(to = ?to, "MTA-STS enforce: all MX hosts failed policy validation"); + metrics::record_send_failure(domain); + // Use `Report::new` (NOT `into_diagnostic`) so the typed + // error survives downcast in `process_job`. + return Err(miette::Report::new(MtaStsEnforcementError { + domain: domain.to_string(), + })); } - error!(to = ?to, "Failed to send email through any MX server"); - if let Some(classified) = last_error { + RecipientDelivery::Failed(classified) => { + error!(to = ?to, "Failed to send email through any MX server"); // Use `Report::new` (NOT `into_diagnostic`) so the typed // error survives downcast in `process_job`. return Err(miette::Report::new(classified)); } - metrics::record_send_failure(parsed_email_id.get_domain()); - bail!("failed to send email through any MX server"); } } Ok(()) @@ -826,6 +1069,21 @@ impl Worker { } } +/// Result of attempting one recipient through its MX servers. +pub(crate) enum RecipientDelivery { + Delivered { + /// Pre-formatted "code message" SMTP response. + smtp_response: String, + exchange: String, + }, + /// Historically-silent skips: unparseable address, no MX records. + Skipped(&'static str), + /// MTA-STS enforce mode rejected every MX host. + MtaStsBlocked, + /// Transport-level failure through every usable MX. + Failed(ClassifiedSendError), +} + /// What `process_job` should do with a failed delivery attempt. #[derive(Debug, PartialEq, Eq, Clone, Copy)] pub(crate) enum SendOutcome { diff --git a/smtp-server/src/worker/rate_limiter.rs b/smtp-server/src/worker/rate_limiter.rs index 63bbb08..fffd1f2 100644 --- a/smtp-server/src/worker/rate_limiter.rs +++ b/smtp-server/src/worker/rate_limiter.rs @@ -205,6 +205,36 @@ impl RateLimiter { RateLimitResult::RateLimited { retry_after: delay } } } + + /// Non-consuming availability check used by the log-queue dispatcher to + /// gate claims: `None` when a token is available (or the domain is not + /// limited), otherwise roughly how long until one is. Never blocks — if + /// the bucket map is contended it optimistically allows, because the + /// worker's consuming check before transmission is authoritative. + pub fn peek_sync(&self, domain: &str) -> Option { + if !self.config.enabled { + return None; + } + let limit = self + .config + .domain_limits + .get(domain) + .copied() + .or(self.config.default_limit)?; + if limit == 0 { + return None; + } + let mut buckets = self.buckets.try_write().ok()?; + let bucket = buckets + .entry(domain.to_string()) + .or_insert_with(|| TokenBucket::new(limit, limit)); + let wait = bucket.time_until_token_available(); + if wait.is_zero() { + None + } else { + Some(wait) + } + } } /// Result of a rate limit check. diff --git a/smtp/src/lib.rs b/smtp/src/lib.rs index 43a2347..f351d62 100644 --- a/smtp/src/lib.rs +++ b/smtp/src/lib.rs @@ -150,6 +150,12 @@ pub enum SmtpError { span: SourceSpan, }, + /// A temporary local failure (storage backpressure, disk reserve + /// reached). Reported to the client as `452` so it retries later. + #[error("Transient failure: {message}")] + #[diagnostic(code(smtp::transient))] + Transient { message: String }, + #[error("Mail rejected: {message}")] MailFromDenied { message: String }, @@ -373,6 +379,11 @@ impl SmtpServer { .write_line(format!("550 {}", message).as_bytes()) .await } + SmtpError::Transient { message } => { + socket + .write_line(format!("452 {}\r\n", message).as_bytes()) + .await + } _ => socket.write_line(b"500 Internal server error\r\n").await, }, _ => Ok(()),