feat(jobs): transactional outbox — opt-in atomic event dispatch - #60
Merged
Conversation
After-response dispatch (4-arg with_repo_errors) is best-effort by design: a process dying between the DB commit and Jobs::submit loses the event. For must-not-lose events (money mail, reconciled webhooks) add the outbox: - migrations/010_outbox.sql: outbox table (kind, jsonb payload, claimed_at, attempts, last_error) + partial index over the unclaimed backlog - src/jobs/Outbox.hpp/.cpp: Outbox::enqueue(txn, kind, payload) INSERTs in the caller's open transaction (commit = event durable, rollback = gone); Outbox::drain(batch) claims via UPDATE ... FOR UPDATE SKIP LOCKED, relays to Jobs::submit, DELETEs on success, releases with attempts+1 + last_error on failure; stale claims (dead drainer) re-claimable after kStaleClaimSec — at-least-once end to end - Core schedules the drain every outbox.drain_interval_sec seconds (OUTBOX_DRAIN_INTERVAL_SEC, default 0 = off — the pattern is opt-in; existing email/webhook flows keep their after-response path unchanged) - docs: CONVENTIONS gotcha 20 (outbox vs after-response decision + 3-line billing-fork example), CONFIG.md row, module-deps edge jobs -> database - tests/integration/test_outbox.cpp: commit->drain->job-in-Redis, rollback invisibility, failed-submit retry accounting, concurrent-drain exactly-once partition
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After-response dispatch loses the event if the process dies between the DB commit and the job submit. The template now ships a transactional outbox for the flows where that's not acceptable.
Outbox::enqueue(txn, kind, payload)inserts in the caller's open transaction — commit makes the domain write and the event durable atomically; rollback erases both (proven by theRolledBackEnqueueIsInvisibletest).Outbox::drain: short claim txn (FOR UPDATE SKIP LOCKED, no Redis I/O under row locks) →Jobs::submitper row → one finalize txn (delete successes, release+count failures). A drainer dying mid-flight can't strand rows — stale claims are re-claimed after 300 s; honest at-least-once, documented (handlers must tolerate duplicates anyway due to the jobs lease path).outbox.drain_interval_secdefaults to 0 (off), existing email/webhook flows untouched — behavior bit-for-bit. The decision guide (when after-response is fine vs when you need the outbox) lives in the header doc-comment and as CONVENTIONS gotcha 20, with a 3-line billing-fork example inside the wallet transaction.Gates green incl. module-deps (63 edges — the one new
jobs→databaseedge declared) and config-sync (141 triples). Compile + tests: this CI run.