Conversation
Gives callback implementations a way to reject a message temporarily (storage backpressure, disk reserve reached) without tearing the session down with a 500. The session error handler maps the new variant to "452 <message>" so well-behaved clients queue and retry. Needed by the upcoming log-queue backend, whose DATA path must be able to shed load at the disk-admission boundary.
Adds the PLAN.md §25 metric set under the logqueue_ prefix: admission (append duration/bytes/records per shard, pending bytes, errors, active segment size, rotations), dispatcher (ready/deferred/in-flight gauges, per-shard lag, oldest ready/deferred age), and storage/GC (live/dead bytes, sealed segments, deletions, compaction outcomes and volume, relocations, disk free). Thin wrapper functions follow the existing conventions; they are wired up by the log-queue engine in the following commits.
Implements the storage and scheduling core from the log-queue design (PLAN.md): sharded append-only payload segments holding complete messages, per-shard state journals with compact checkpoints, and a single dispatcher that schedules delivery across all shards. - record: versioned self-framing records (crc-guarded header carrying the full envelope, so discovery never reads bodies) - segment: active/sealed segment lifecycle, positioned reads, torn-tail validation and truncation; sealed corruption is a hard error - shard/spool: shard directories, format version, exclusive spool lock, shard-count change guard, disk-free probe - writer: one append task per shard with byte-bounded admission; SMTP-facing append completes on page-cache write, publish order write -> committed head -> notify -> completion; seal-before-create rotation so a crash can never leave two active segments - state: DEFER/DELIVERED/BOUNCED/RELOCATED journal entries with persist-then-apply replay, fsync-then-rename checkpoints that carry tombstones, the discovered ready/deferred sets (including partial recipient sets) and the discovery cursor; journal history is only deleted once a covering checkpoint is durable - dispatcher: cursor-based discovery with lossy notification hints and backpressure, claim generations with abandonment recovery, due-time retry heap, dispatch-time rate gating, event-driven deletion of fully dead segments and dead-ratio compaction that relocates live records through the append path under fsync barriers Durability model is page-cache acknowledgement by design; fsync guards only destructive boundaries (checkpoint truncation, segment deletion, compaction publication). Recovery scans use the format's absolute record bound so config changes can never orphan accepted mail. The engine lands dormant: nothing selects it yet.
Extracts the per-recipient delivery core (MX lookup, MTA-STS policy, transport attempts, outcome classification) out of send_email into deliver_recipient, keeping the legacy channel worker's behavior unchanged, and builds the log-queue path on top of it: - Worker::process_claim delivers one dispatcher claim and reports a JobOutcome. Recipients are tracked individually so a retry re-sends only to those that have not accepted the message; a lost rate-limit race is reported as RateLimited (no attempt charged, no journal write) instead of sleeping in the worker slot. - LogWorker pulls claims, reads bodies by location, enforces the retry budget, and archives bounced messages through the storage layer before reporting the terminal outcome. - LimiterGate bridges the shared per-domain RateLimiter into the dispatcher's dispatch-time gate via a new non-consuming peek_sync; the worker's consuming check right before transmission stays authoritative. Envelope From parsing failures now surface as errors (bounce) instead of panicking the worker task.
New optional CfgQueue block (append_writers, pending_append_bytes, segment_target_bytes, compaction_dead_ratio, compaction_min_age, max_concurrent_compactions, disk_reserve_bytes, checkpoint_interval_bytes) with defaults resolved via accessor methods and a validate() step that rejects a writer count of zero, an out-of-range dead ratio, and segment sizes that could not hold a maximum-size message (records never span segments). Documents the section in both example configs; existing test fixtures gain the new field.
storage_type = "log" activates the durable log queue. The DATA path then checks the disk reserve (452 on breach), appends to the log, and returns 250 OK as soon as the record reaches the kernel page cache — acceptance no longer waits for worker-channel capacity, so stalled or rate-limited outbound cannot block inbound mail (PLAN.md phase 8). Startup wires the spool (exclusive lock held for the server lifetime), per-shard append writers, state recovery, the dispatcher gated by the shared rate limiter, and claim-pulling log workers. The legacy replay through the bounded channel and the deferred-scan worker are skipped: recovery happens through checkpoints, journal replay, and dispatcher discovery. Shutdown drains in-flight outcomes, checkpoints every shard, and closes append admission last so accepted mail is on disk. The filesystem store remains as the bounced-message archive with the usual retention cleanup; deferred-retention cleanup is disabled in log mode so it can never delete an unmigrated legacy spool. Existing fs and sqlite backends keep their current scheduling path unchanged.
hedwig queue list/show/stats inspect a log-queue spool read-only — including a live one: state is loaded without taking the lock, truncating torn tails, or creating files, and not-yet-discovered records are found by scanning segments from the recovered cursor. hedwig queue migrate performs the one-time filesystem-spool migration (PLAN.md §23): idempotent via a dedup scan of the target spool, preserves message ids, attempt counts, next-attempt times and last-error strings, derives stable replacement ids for non-ULID legacy ids so re-runs cannot duplicate, verifies every migrated record is present after the writers flush, and only then renames queued/ and deferred/ to timestamped .migrated-* backups. Nothing is ever deleted; bounced/ stays in place as the live bounce archive.
Covers backend selection, the [queue] tuning knobs with defaults, the page-cache durability tradeoff (machine crash may lose the most recent acknowledgements; destructive operations are fsync-guarded so older queued mail is never at risk), disk growth and the reserve check, bounce retention, the read-only inspection CLI, and the migration procedure from the filesystem spool.
Event-driven deletion fires on the terminal transition, but a segment whose last record goes terminal before the dispatcher has observed the seal (its total size is unknown at that instant) misses the event and was never reclaimed — a 30s full-throughput bench left 1.4 GiB of 100%-dead sealed segments behind. Add the PLAN.md §17.1 safety-net sweep on the dispatcher tick; the same bench now leaves only the active segment on disk.
End-to-end description of the implemented system: crate/module map, on-disk layout and record format, the acceptance path and its publish ordering, dispatcher scheduling (discovery, job lifecycle, claim generations, rate gating), journal/checkpoint persistence and the recovery sequence, GC/compaction with the durability table, startup and shutdown ordering, operator tooling, measured benchmark behavior, and the invariant list to check before changing anything. Diagrams in mermaid so they render on GitHub.
The reference architecture page still described only the legacy channel/worker flow. Rewrite it as a short orientation covering both queue paths (log and legacy), their tradeoffs, and operational properties, deferring to the repository-root ARCHITECTURE.md for the detailed diagrams and durability model.
Drops SqliteStorage, its config surface (num_shards, batch_size, batch_timeout_ms, [storage.sqlite]), the sqlx dependency, the dev config, and the documentation references. The backend saw no use, and the problems it addressed over the filesystem spool (atomic writes, no per-message files, indexed lookups) are solved better by the durable log queue. BREAKING CHANGE: storage_type = "sqlite" is no longer accepted; the server exits with "Unknown storage type" at startup. Existing SQLite spools should be drained before upgrading (or their remaining mail re-injected), then switch to storage_type = "log" (recommended) or "fs". Unknown [storage] keys in existing configs are ignored, so leftover sqlite tuning entries are harmless.
…ools storage_type may now be omitted; the durable log queue is the default backend. On log-backend startup, a non-empty legacy queued/ or deferred/ directory at the base path logs a loud warning: that mail is preserved (log mode never reads or cleans it) but will not be delivered until `hedwig queue migrate` runs, and silence would make it easy to forget.
The metrics test module asserts on process-global Prometheus counters that other tests in the same binary bump in parallel, which made several tests flake (a pre-existing race, more visible now that the log-queue tests also touch these counters). Convert exact-value assertions to delta/floor assertions; 8 consecutive runs stay green.
Deploying hedwig with
|
| Latest commit: |
580086c
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://2e8b5a67.hedwig-1mq.pages.dev |
| Branch Preview URL: | https://log-queue.hedwig-1mq.pages.dev |
The log-queue design plan is a dated design document, not a live top-level file; docs/plans/ is where the repository keeps those (alongside the sqlite and deliverability plans). Renamed to docs/plans/2026-07-20-durable-log-queue.md and path references in ARCHITECTURE.md and the CLI updated. ARCHITECTURE.md remains the current description of what is built; the plan stays as rationale.
The logqueue module and CfgQueue carried allow(dead_code) plus 'not yet wired into the serving path' comments from the incremental build; the backend has been live for a while. Remove the blanket allows and what they were hiding: leftover helpers with no callers (StateEntry accessors, is_tombstoned, Spool::root, store accessors, an unused DispatcherHandle field) and the max_concurrent_compactions config knob that nothing consumed (compaction concurrency is fixed at one by design). Test-only API keeps targeted allows with a note.
Zero allow(dead_code) remain in the tree. The two genuinely unused metric wrappers are now real: pending_append_bytes is published from the admission semaphore on every append, and the unmeasurable dispatcher_lag_records metric became dispatcher_lag_bytes, computed per shard from the committed chain minus the discovery cursor on the metrics tick. Test-only helpers (ActiveSegment::path, AppendHandle::shard_count, NoRateGate) are cfg(test)-gated instead of allowed, and the consuming ActiveSegment::seal wrapper is gone — tests use seal_in_place like the writer does.
iamd3vil
added a commit
that referenced
this pull request
Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the durable log queue designed in docs/plans/2026-07-20-durable-log-queue.md (all phases): complete messages are stored in sharded, segmented append-only logs; SMTP
250 OKis returned once the record reaches the kernel page cache; a single dispatcher schedules delivery, retries, and reclamation. Inbound acceptance is bounded by disk append throughput instead of outbound drain speed.See
ARCHITECTURE.md(added here) for the full design walkthrough with diagrams, anddocs/src/pages/storage.mdfor operator-facing behavior.What's in it
smtp-server/src/logqueue/): versioned crc-guarded record format; active/sealed segment lifecycle with torn-tail recovery; per-shard append writers with byte-bounded admission and seal-first rotation; per-shard state journals (defer/deliver/bounce/relocate, persist-then-apply) with fsynced checkpoints; one dispatcher (cursor discovery with backpressure, claim generations, due-time retry heap, dispatch-time rate gating); event-driven segment deletion + dead-ratio compaction via relocation generations, with fsync barriers at every destructive boundary.bounced/archive (existing retention applies).storage_type = "log"(now the default) selects the backend; DATA path checks the disk reserve (452) and acks on append completion; startup recovers via checkpoint + journal replay + discovery (no replay channel); graceful shutdown drains outcomes and checkpoints.hedwig queue list|show|stats(read-only, live-spool safe),hedwig queue migrate(idempotent legacy-spool migration, renames to backups, never deletes), startup warning for unmigrated legacy spools, PLAN §25 metrics, docs.Breaking changes
storage_type = "sqlite"no longer accepted). Drain such spools before upgrading; leftover config keys are ignored.storage_typenow defaults to"log"when omitted (previously required).Durability model
Acknowledged mail survives process crashes; a machine crash/power loss may lose the most recently acknowledged messages (documented tradeoff, same spirit as the fs backend today). Destructive operations (checkpoint truncation, segment deletion, compaction publication) are fsync-guarded, so older queued mail is never at risk.
Verification
Benchmarks (1 KiB, 16 conns)
Both delivered outbound at the same ~121 msg/s in the second test; the fs channel collapses acceptance to drain speed, the log backend keeps accepting at disk speed. A 30 s full-throughput run writes ~3.5 GiB and reclaims down to the active segment.