type = improvement: Design durable outbox storage and relay

[bartul]

Related decision: #76
Related work: #129

type = improvement
status = proposed

Context/problem

Durable/web deployments need to prevent state committed, message lost failures. The terminal sandbox is currently in-memory and fire-and-forget, so an in-memory outbox would add ceremony without the durability guarantee.

Once durable persistence is selected, Rondel state changes and outbound messages must be committed atomically, and a relay must deliver pending messages with at-least-once semantics.

Proposed change

Design the durable outbox storage model and relay behavior for the selected persistence backend.

The design should cover:

• OutboxMessage identity and payload metadata.
• Destination classification for domain events and outbound bounded-context commands.
• GameId and per-game ordering/sequence requirements.
• Attempt count, timestamps, retry/backoff, processed status, and dead-letter/poison-message handling.
• Atomic commit semantics for state plus outbox messages.
• Relay delivery and acknowledgment semantics.
• Observability needed to diagnose stuck messages and repeated delivery failures.

Acceptance criteria

• Persistence transaction constraints are documented for the chosen backend.
• State and outbox messages are committed atomically, or the design explicitly rejects the backend as unsuitable.
• Relay leaves failed deliveries pending for retry.
• Relay acknowledges messages only after successful delivery.
• Duplicate delivery after relay crash or acknowledge failure is documented as expected behavior.
• Per-GameId ordering requirements are specified and tested.
• Tests cover atomic commit success, rollback/no partial commit, retry after delivery failure, and duplicate delivery after acknowledge failure.

Consequences

This provides the durable half of the outbox/inbox strategy. It intentionally does not claim exactly-once delivery by itself; consumers still need idempotent inbox or business-level idempotency.

[bartul]

Research Report: Durable Outbox Storage and Relay

Related decision: #76
Related work: #129, #131, #49

Status: Research complete; no implementation performed.

Problem Summary

Issue #130 asks for the durable producer-side half of the outbox/inbox strategy selected in #76. Durable or web deployments must prevent the failure mode where Rondel state is committed but the corresponding integration event or outbound bounded-context command is lost.

The current terminal sandbox is intentionally in-memory and enqueue-ack only. It remains out of scope for durable outbox work. The durable design must instead plug into the explicit commit boundary introduced by #129:

type RondelEffects =
    { State: RondelState option
      IntegrationEvents: RondelEvent list
      OutboundCommands: RondelOutboundCommand list }

type CommitRondelEffects = RondelEffects -> Async<unit>

type RondelDependencies =
    { Load: LoadRondelState
      Commit: CommitRondelEffects }

The durable commit must atomically persist:

1. the new Rondel state, when present;
2. outbox messages derived from IntegrationEvents;
3. outbox messages derived from OutboundCommands.

A relay then delivers pending outbox messages with at-least-once semantics. The relay must acknowledge only after successful delivery. Duplicate delivery after relay crash, acknowledgement failure, or retry is expected behavior, and #131 is responsible for inbox or business-level idempotency on the consumer side.

Classification: architecture improvement / durable infrastructure design.

Authoritative Context

Issue and comment chain treated as authority:

• Plan transactional outbox/retry strategy for save-publish-dispatch consistency (deferred) #76 approved transactional outbox plus idempotent inbox for durable/web deployments and explicitly rejected an in-memory terminal outbox.
• type = improvement: Introduce explicit Rondel effect commit boundary #129 completed the explicit Rondel effect commit boundary and made CommitRondelEffects the extension point.
• type = improvement: Design durable outbox storage and relay #130 is the durable outbox storage and relay design.
• type = improvement: Add inbox and processed-message tracking for inbound delivery #131 is the consumer-side inbox / processed-message tracking design.
• Implement Stateful Accounting Domain with Balance Tracking #49 will make Accounting stateful and idempotent by BillingId.
• Design host Execute acknowledgement semantics (deferred decision) #74 decided host Execute semantics are enqueue acknowledgment only.
• Plan JSON file persistence adapter for terminal store (deferred) #75 closed JSON terminal persistence as not near future.

Architecture Impact

Current Rondel Write Path

Code facts verified locally:

• Rondel.fsi exposes RondelEffects, CommitRondelEffects, and RondelDependencies = { Load; Commit }.
• Rondel.execute and Rondel.handle route to internal handlers, receive one RondelEffects record, and call deps.Commit once.
• Internal Rondel handlers load state, run pure execution modules, and return effects. They do not perform IO.
• The terminal host composes Commit with RondelDirectCommit.create store.Save bus.Publish dispatch.
• RondelDirectCommit is explicitly non-durable and preserves old ordering: save state, publish integration events, dispatch outbound commands.
• RondelHost.Execute and AccountingHost.Execute enqueue mailbox messages and return immediately.
• The terminal bus suppresses subscriber exceptions and is not a durable delivery mechanism.
• Accounting is currently stateless and auto-approves charges.
• The web host is currently a placeholder.
• docs/future_decisions.md lists web storage and bus candidates but does not select a backend.

Current durable-relevant path:

RondelHost.Execute cmd
  -> mailbox.Post
    -> Rondel.execute deps cmd
      -> internal handler loads state
      -> pure execution returns (state, events, outbound commands)
      -> handler wraps tuple as RondelEffects
      -> router calls deps.Commit effects
        -> terminal direct commit: save -> publish -> dispatch

For #130, the durable web host should keep the same domain shape:

Rondel.execute { Load = durableLoad; Commit = durableCommit } cmd
  -> durableCommit effects
    -> begin transaction
    -> save state
    -> insert outbox messages for events and outbound commands
    -> commit transaction

OutboxRelay
  -> claim due pending messages
  -> deliver message
  -> mark processed only after delivery succeeds
  -> leave failed messages pending for retry, or dead-letter after policy limit

Affected Areas

Directly affected by durable outbox design:

• Future durable/web infrastructure package or host.
• Future persistent Rondel store implementation.
• Outbox message schema and relay service.
• Destination mapping from RondelEvent and RondelOutboundCommand.
• Observability and operations for stuck messages.
• Tests for atomic commit, retry, duplicate delivery, and per-game ordering.

Indirectly affected:

• Accounting and other consumers must be idempotent through type = improvement: Add inbox and processed-message tracking for inbound delivery #131 and Implement Stateful Accounting Domain with Balance Tracking #49.
• Host model choice may change if Wolverine, MassTransit, Rebus, or Akka.NET becomes the web host.
• Web API command endpoints should preserve 202 Accepted semantics after synchronous contract validation.

Unaffected by #130:

• Pure Rondel move rules.
• Contract transformations, except serializers/mappers reused for outbox payloads.
• Terminal in-memory bus/store behavior.
• Query handlers, except future read model decisions.

External Research Findings

Canonical Patterns

The transactional outbox pattern stores messages in the same database transaction as business state. A relay later publishes them. This prevents state committed, message lost, but delivery is at-least-once, so duplicates are normal.

The idempotent consumer pattern tracks processed message IDs per consumer or applies business-level idempotency. This is the consumer-side complement to outbox delivery.

Sources:

• https://microservices.io/patterns/data/transactional-outbox.html
• https://microservices.io/patterns/communication-style/idempotent-consumer.html

Storage and Transaction Candidates

Marten:

• PostgreSQL document/event store.
• Supports unit-of-work style sessions and transactionally saving changes.
• Pairs strongly with Wolverine, which has Marten-integrated durability.

Sources:

• https://martendb.io/documents/sessions.html
• https://wolverinefx.io/guide/durability/marten/outbox

EF Core:

• SaveChanges is transactional by default for most providers.
• Explicit transactions can combine multiple operations in one unit.
• Pairs strongly with MassTransit and Wolverine outbox integrations.

Source:

• https://learn.microsoft.com/en-us/ef/core/saving/transactions

Raw Npgsql / Dapper:

• Both can participate in explicit ADO.NET database transactions.
• Dapper is mostly an SQL mapper; it does not provide outbox semantics by itself.
• Raw Npgsql or Dapper can implement the cleanest custom outbox, at the cost of owning all relay behavior.

Sources:

• https://www.npgsql.org/doc/basic-usage.html
• https://www.learndapper.com/misc/transaction

PostgreSQL / SQL Server queue mechanics:

• PostgreSQL can use row-level locks with FOR UPDATE SKIP LOCKED for work-queue style claiming.
• SQL Server supports READPAST and update locks for work-queue patterns.
• Both PostgreSQL and SQL Server support sequence objects, but outbox ordering should usually use a per-game sequence allocated transactionally with state/outbox insertion rather than global message ID alone.

Sources:

• https://www.postgresql.org/docs/current/sql-select.html
• https://www.postgresql.org/docs/current/sql-createsequence.html
• https://learn.microsoft.com/en-us/sql/t-sql/queries/hints-transact-sql-table
• https://learn.microsoft.com/en-us/sql/t-sql/statements/create-sequence-transact-sql

Framework Candidates

Wolverine:

• Durable messaging framework with inbox/outbox support.
• Integrates with Marten and EF Core.
• Provides retries, dead-letter storage/replay, health checks, logging, metrics, and OpenTelemetry.
• Can be used as infrastructure behind CommitRondelEffects, or as a broader HTTP/message host model.

Sources:

• https://wolverinefx.net/guide/durability/
• https://wolverinefx.io/guide/durability/marten/outbox
• https://wolverinefx.net/guide/durability/efcore/outbox-and-inbox
• https://wolverinefx.net/guide/logging

MassTransit:

• Mature .NET service bus framework.
• Provides transactional outbox support, including EF Core outbox and bus/consumer outbox concepts.
• Provides retry, delayed redelivery, error queues, faults, and OpenTelemetry.
• Can be used as a commit adapter, but is most natural when bounded-context handlers become message consumers.

Sources:

• https://masstransit.io/documentation/configuration/middleware/outbox
• https://masstransit.io/documentation/concepts/exceptions
• https://masstransit.io/documentation/configuration/observability

Rebus:

• Lightweight .NET service bus.
• Official SQL Server outbox support exists.
• Provides retry and error queue handling.
• SQL Server being acceptable makes Rebus viable for this project.
• Can be a commit adapter or broader handler host, though PostgreSQL/Marten would weaken the fit.

Sources:

• https://github-wiki-see.page/m/rebus-org/Rebus/wiki/Outbox
• https://github-wiki-see.page/m/rebus-org/Rebus/wiki/Automatic-retries-and-error-handling

Akka.NET:

• Actor model framework.
• Default actor delivery is at-most-once; stronger delivery requires explicit reliable-delivery patterns.
• The older persistence at-least-once delivery API is obsolete in favor of Akka.Delivery.
• Akka.Delivery and Akka.Cluster.Sharding.Delivery can use EventSourcedProducerQueue as a durable producer-side transport queue.
• EventSourcedProducerQueue durability is not an atomic business commit with Rondel state persistence.
• Akka.NET is viable if the project intentionally chooses actor-hosted bounded contexts, especially per-game actors.
• Since Rondel should remain snapshot/state based rather than event-sourced, the preferred Akka model is RondelState snapshot plus pending dispatch records in one authoritative commit, followed by Akka.Delivery as transport.
• As only a CommitRondelEffects adapter, Akka.NET adds little unless backed by the same durable state+outbox storage.

Sources:

• https://getakka.net/articles/concepts/message-delivery-reliability.html
• https://getakka.net/articles/actors/reliable-delivery.html
• https://getakka.net/articles/clustering/cluster-sharding-delivery.html
• https://getakka.net/api/Akka.Persistence.Delivery.EventSourcedProducerQueue.html
• https://getakka.net/articles/persistence/at-least-once-delivery.html

Hypothesis Evolution

H1: Custom SQL outbox under CommitRondelEffects

Initial status: active.

Current status: active, strong as reference design.

Supporting evidence:

• Fits type = improvement: Introduce explicit Rondel effect commit boundary #129 exactly: one CommitRondelEffects implementation can own the database transaction.
• Database transaction boundaries are explicit and testable.
• Per-GameId ordering can be specified precisely.
• Does not force a host model decision.

Contradicting evidence:

• Relay, retry, dead-lettering, observability, and operational tooling must be built and maintained.
• The design becomes more complex if cross-BC delivery later needs external brokers.

Confidence: medium-high. Would rise if the project chooses a SQL store but no framework host.

H2: Wolverine as integrated durable host/outbox

Initial status: active.

Current status: active, strong.

Supporting evidence:

• Strong fit with Marten and EF Core.
• Durable messaging, outbox/inbox, retry, dead-letter, and observability are framework-supported.
• More F#-friendly than many .NET messaging frameworks.
• Can preserve the current effect boundary or become the broader host.

Contradicting evidence:

• Framework adoption changes composition and operational model.
• If the project chooses SQL Server/Rebus or actor hosting, Wolverine may be less compelling.

Confidence: medium-high. Would rise if Marten/PostgreSQL becomes the web storage decision.

H3: MassTransit + EF Core

Initial status: active.

Current status: active, viable.

Supporting evidence:

• Mature .NET ecosystem and service bus abstractions.
• EF Core transactional outbox is a conventional fit.
• Strong retry, redelivery, error queue, and telemetry features.

Contradicting evidence:

• Heavier and more C#-centric than Wolverine.
• More naturally message-consumer-first than a thin CommitRondelEffects adapter.

Confidence: medium. Would rise if EF Core plus brokered messaging becomes the standard web stack.

H4: Rebus + SQL Server outbox

Initial status: weakened because SQL Server had not been established.

Current status: active, viable.

Supporting evidence:

• SQL Server is acceptable.
• Rebus is simpler than MassTransit and has SQL Server outbox support.
• Works as both commit adapter and broader handler host.

Contradicting evidence:

• SQL Server outbox support is the natural fit; PostgreSQL/Marten would weaken it.
• Ecosystem breadth and integrated features are narrower than MassTransit.

Confidence: medium. Would rise if SQL Server is chosen and simplicity is prioritized.

H5: Akka.NET actor-hosted bounded contexts

Initial status: weakened as a pure outbox adapter.

Current status: active, viable as broader host model.

Supporting evidence:

• Per-game actor serialization maps naturally to per-GameId command ordering.
• Akka.Persistence plus Akka.Delivery can model durable actors and reliable delivery.
• Actor hosting could fit Rondel and future BCs if the project wants a distributed actor architecture.

Contradicting evidence:

• Default actor messaging is not a transactional outbox.
• Actor persistence plus delivery is a broader architecture decision, not a small adapter.
• Migration cost is highest.

Confidence: medium. Would rise if actor-hosting becomes the preferred web direction.

Decision Criteria

Ranked for issue #130:

1. Atomic state plus outbox commit guarantee.
2. Explicit per-GameId ordering and sequence semantics.
3. Fit with the current RondelEffects -> Async<unit> seam.
4. Relay behavior: claim, deliver, acknowledge, retry, backoff, dead-letter.
5. Observability: stuck messages, repeated failures, age, attempts, destination, correlation.
6. F# and codebase consistency.
7. Broader host-model viability for Rondel and other BCs.
8. Migration cost and reversibility.
9. Testability against the issue acceptance criteria.

Common Outbox Model

All recommended approaches should satisfy this conceptual model, even if a framework maps it to its own tables/envelopes.

[<RequireQualifiedAccess>]
type OutboxDestination =
    | DomainEvent of topic: string
    | BoundedContextCommand of target: string

[<RequireQualifiedAccess>]
type OutboxStatus =
    | Pending
    | Processing
    | Processed
    | DeadLettered

type OutboxMessage =
    { Id: Guid
      GameId: Id
      GameSequence: int64
      OccurredAt: DateTimeOffset
      Destination: OutboxDestination
      MessageType: string
      PayloadJson: string
      Attempts: int
      NextAttemptAt: DateTimeOffset
      LastAttemptAt: DateTimeOffset option
      ProcessedAt: DateTimeOffset option
      DeadLetteredAt: DateTimeOffset option
      LastError: string option
      CorrelationId: Guid option
      CausationId: Guid option }

Required invariants:

• Id is stable and globally unique.
• GameId is mandatory for all Rondel outbox messages.
• GameSequence is monotonic per GameId.
• (GameId, GameSequence) is unique.
• Messages for the same GameId are delivered in sequence unless the final design explicitly permits later messages to bypass dead-lettered earlier messages.
• PayloadJson stores serialized contract/domain message payload plus enough type metadata to deserialize and route.
• Attempts, NextAttemptAt, LastError, and status fields support retry and operations.
• Successful delivery is acknowledged after delivery only.
• Failed delivery leaves a message pending or dead-lettered according to policy.

Destination mapping for current Rondel effects:

let eventDestination (event: RondelEvent) =
    OutboxDestination.DomainEvent "rondel.events"

let commandDestination (command: RondelOutboundCommand) =
    match command with
    | ChargeMovement _ -> OutboxDestination.BoundedContextCommand "accounting"
    | VoidCharge _ -> OutboxDestination.BoundedContextCommand "accounting"

Approach A: Custom SQL Outbox Under Current Host Model

Summary

Implement an explicit durable SQL outbox behind CommitRondelEffects. This preserves the existing domain and host shape while making state plus outbox insertion atomic.

Type Definitions

type SaveRondelStateInTransaction =
    DbTransaction -> RondelState -> Async<unit>

type InsertOutboxMessages =
    DbTransaction -> OutboxMessage list -> Async<unit>

type AllocateGameSequences =
    DbTransaction -> Id -> int -> Async<int64 list>

type SerializeOutboxPayload =
    OutboxDestination -> obj -> string

type ClaimDueMessages =
    int -> DateTimeOffset -> Async<OutboxMessage list>

type MarkDelivered =
    Guid -> DateTimeOffset -> Async<unit>

type MarkFailed =
    Guid -> error: string -> nextAttemptAt: DateTimeOffset -> Async<unit>

type DeadLetter =
    Guid -> error: string -> DateTimeOffset -> Async<unit>

Public Surface Sketch

No domain API change:

val createRondelOutboxCommit:
    connectionFactory: (unit -> Task<DbConnection>) ->
    saveState: SaveRondelStateInTransaction ->
    insertOutbox: InsertOutboxMessages ->
    allocateSequences: AllocateGameSequences ->
    serialize: SerializeOutboxPayload ->
    now: (unit -> DateTimeOffset) ->
    CommitRondelEffects

Implementation Sketch

let toOutboxMessages allocate serialize now tx effects =
    async {
        let messages =
            [ for event in effects.IntegrationEvents ->
                  OutboxDestination.DomainEvent "rondel.events", box event
              for command in effects.OutboundCommands ->
                  OutboxDestination.BoundedContextCommand "accounting", box command ]

        match effects.State, messages with
        | None, [] -> return []
        | Some state, _
        | None, ({ } :: _ as _) ->
            let gameId =
                effects.State
                |> Option.map _.GameId
                |> Option.defaultWith (fun () ->
                    // Current Rondel events/commands all include GameId.
                    // Real implementation should extract through typed helpers.
                    failwith "GameId extraction required for stateless effects")

            let! sequences = allocate tx gameId messages.Length
            let occurredAt = now ()

            return
                (messages, sequences)
                ||> List.map2 (fun (destination, payload) sequence ->
                    { Id = Guid.NewGuid()
                      GameId = gameId
                      GameSequence = sequence
                      OccurredAt = occurredAt
                      Destination = destination
                      MessageType = payload.GetType().FullName
                      PayloadJson = serialize destination payload
                      Attempts = 0
                      NextAttemptAt = occurredAt
                      LastAttemptAt = None
                      ProcessedAt = None
                      DeadLetteredAt = None
                      LastError = None
                      CorrelationId = None
                      CausationId = None })
    }

let createRondelOutboxCommit connectionFactory saveState insertOutbox allocate serialize now : CommitRondelEffects =
    fun effects ->
        async {
            use! connection = Async.AwaitTask (connectionFactory ())
            use! tx = Async.AwaitTask (connection.BeginTransactionAsync())

            try
                match effects.State with
                | Some state -> do! saveState tx state
                | None -> ()

                let! messages = toOutboxMessages allocate serialize now tx effects
                do! insertOutbox tx messages

                do! Async.AwaitTask (tx.CommitAsync())
            with ex ->
                do! Async.AwaitTask (tx.RollbackAsync())
                raise ex
        }

Relay sketch:

type DeliverOutboxMessage = OutboxMessage -> Async<Result<unit, string>>

let runOnce claim markDelivered markFailed deadLetter deliver now maxAttempts backoff =
    async {
        let! messages = claim 100 (now ())

        for message in messages do
            let! result = deliver message

            match result with
            | Ok () ->
                do! markDelivered message.Id (now ())
            | Error error when message.Attempts + 1 >= maxAttempts ->
                do! deadLetter message.Id error (now ())
            | Error error ->
                let next = backoff (message.Attempts + 1) (now ())
                do! markFailed message.Id error next
    }

Caller Example

let deps : RondelDependencies =
    { Load = durableLoadRondelState
      Commit =
        createRondelOutboxCommit
            connectionFactory
            saveRondelStateInTx
            insertOutboxMessages
            allocateGameSequences
            serializeOutboxPayload
            clock.UtcNow }

Test Sketch

testCase "commits state and outbox atomically" <| fun _ ->
    let effects =
        { State = Some sampleState
          IntegrationEvents = [ sampleEvent ]
          OutboundCommands = [ sampleCommand ] }

    commit effects |> Async.RunSynchronously

    Expect.isSome (loadState sampleState.GameId) "state saved"
    Expect.equal (loadOutbox sampleState.GameId).Length 2 "event and command persisted"

testCase "rollback leaves no partial state or messages" <| fun _ ->
    let failingInsert _ _ = async { failwith "insert failed" }
    let commit = createCommitWith failingInsert

    Expect.throws (fun () -> commit sampleEffects |> Async.RunSynchronously) "commit fails"
    Expect.isNone (loadState sampleGameId) "state rolled back"
    Expect.isEmpty (loadOutbox sampleGameId) "outbox rolled back"

testCase "relay leaves failed delivery pending for retry" <| fun _ ->
    enqueue sampleMessage
    runOnce claim markDelivered markFailed deadLetter (fun _ -> async { return Error "down" }) now 3 backoff
    let message = loadMessage sampleMessage.Id
    Expect.equal message.Status Pending "message remains pending"
    Expect.equal message.Attempts 1 "attempt counted"

testCase "crash after delivery before acknowledgement redelivers" <| fun _ ->
    let delivered = ResizeArray<Guid>()
    let deliver msg = async { delivered.Add msg.Id; return Ok () }
    let failingAck _ _ = async { failwith "ack failed" }

    Expect.throws (fun () -> runOnce claim failingAck markFailed deadLetter deliver now 3 backoff |> Async.RunSynchronously) "ack fails"
    runOnce claim markDelivered markFailed deadLetter deliver now 3 backoff |> Async.RunSynchronously

    Expect.equal delivered.Count 2 "message delivered twice"

Pro/Con Analysis

Architecture fit: excellent for the existing seam. CommitRondelEffects remains the only durable boundary.

Complexity: high operational and implementation complexity. The codebase owns schema design, relay claiming, sequencing, retry policy, and metrics.

Risk: medium. SQL transaction semantics are straightforward, but relay locking and per-game ordering must be implemented carefully.

Testability: high. The guarantee is local and can be covered with integration tests against the chosen database.

Migration: moderate. No domain change, but a new durable infrastructure layer is required.

Extensibility: high. New destinations and bounded contexts can add mappings and payload types.

F# idiom score: 8/10. Pure effect data plus explicit commit function is idiomatic. SQL plumbing is imperative but contained.

Codebase consistency: 10/10. This is the cleanest continuation of #129.

Confidence: medium-high.

Approach B: Wolverine as Commit Adapter and Broader Host Model

Summary

Use Wolverine durable messaging with Marten or EF Core. It can be a CommitRondelEffects adapter while keeping the current router shape, or it can become the durable web host model for HTTP/message handlers.

Type Definitions

type WolverineMessage =
    | PublishRondelEvent of RondelEvent
    | SendAccountingCommand of RondelOutboundCommand

type WolverineRondelStore =
    { Load: LoadRondelState
      StoreState: IDocumentSession -> RondelState -> unit }

type WolverineRondelCommit =
    IDocumentSession -> IMessageBus -> CommitRondelEffects

Public Surface Sketch

Commit-adapter mode:

val createWolverineCommit:
    storeState: (IDocumentSession -> RondelState -> unit) ->
    bus: IMessageBus ->
    session: IDocumentSession ->
    CommitRondelEffects

Host-model mode:

type ExecuteRondelCommand =
    RondelCommand -> Task

type HandleRondelInboundEvent =
    RondelInboundEvent -> Task

Implementation Sketch - Commit Adapter

let createWolverineCommit storeState (bus: IMessageBus) (session: IDocumentSession) : CommitRondelEffects =
    fun effects ->
        async {
            match effects.State with
            | Some state -> storeState session state
            | None -> ()

            for event in effects.IntegrationEvents do
                do! bus.PublishAsync(event) |> Async.AwaitTask

            for command in effects.OutboundCommands do
                match command with
                | ChargeMovement _
                | VoidCharge _ ->
                    do! bus.SendAsync(command) |> Async.AwaitTask

            do! session.SaveChangesAsync() |> Async.AwaitTask
        }

The exact implementation depends on Wolverine/Marten configuration. The design requirement is that outgoing Wolverine envelopes are enlisted in the same durable session as state changes.

Implementation Sketch - Host Model

type RondelCommandHandler(load: LoadRondelState, commit: CommitRondelEffects) =
    member _.Handle(command: RondelCommand) : Task =
        execute { Load = load; Commit = commit } command
        |> Async.StartAsTask
        :> Task

type RondelInboundEventHandler(load: LoadRondelState, commit: CommitRondelEffects) =
    member _.Handle(event: RondelInboundEvent) : Task =
        handle { Load = load; Commit = commit } event
        |> Async.StartAsTask
        :> Task

In full host-model mode, Wolverine receives HTTP commands or messages, applies transactional middleware, persists state through Marten/EF, and stores outgoing messages through its outbox.

Caller Example

let depsForWolverine session bus =
    { Load = loadFromMarten session
      Commit = createWolverineCommit storeRondelState bus session }

Test Sketch

testCase "wolverine transaction stores state and outgoing envelopes" <| fun _ ->
    let handler = RondelCommandHandler(load, commit)
    handler.Handle(Move sampleMove) |> Async.AwaitTask |> Async.RunSynchronously

    Expect.isSome (loadState sampleGameId) "state saved"
    Expect.isNonEmpty (loadOutgoingEnvelopes sampleGameId) "outgoing messages stored"

testCase "handler failure rolls back state and outgoing messages" <| fun _ ->
    let commit = createFailingWolverineCommit()
    Expect.throwsTask (fun () -> handlerWith commit sampleCommand) "handler fails"
    Expect.isNone (loadState sampleGameId) "state rolled back"
    Expect.isEmpty (loadOutgoingEnvelopes sampleGameId) "outbox rolled back"

Pro/Con Analysis

Architecture fit: strong. Adapter mode preserves #129. Host mode is a bigger shift but still routes through domain routers.

Complexity: medium. Framework configuration replaces much custom relay code.

Risk: medium. Most reliability features are framework-owned, but correctness depends on configuration and storage integration.

Testability: medium-high. Framework integration tests are needed.

Migration: medium. Adapter mode can be incremental; host mode changes web composition.

Extensibility: high. Durable messaging and handler hosting scale to other BCs.

F# idiom score: 7/10. The domain stays idiomatic F#; framework integration is .NET object-oriented but workable.

Codebase consistency: 8/10 adapter mode, 7/10 host mode.

Confidence: medium-high.

Approach C: MassTransit + EF Core as Commit Adapter and Host Model

Summary

Use MassTransit transactional outbox with EF Core. It can be configured behind CommitRondelEffects or adopted as the durable message-consumer host.

Type Definitions

type RondelDbContext =
    inherit DbContext
    abstract RondelStates: DbSet<RondelStateEntity>

type MassTransitRondelCommit =
    RondelDbContext -> IPublishEndpoint -> ISendEndpointProvider -> CommitRondelEffects

type RondelStateEntity =
    { GameId: Guid
      PayloadJson: string
      Version: int64 }

Public Surface Sketch

val createMassTransitCommit:
    mapState: (RondelState -> RondelStateEntity) ->
    publishEndpoint: IPublishEndpoint ->
    sendEndpointProvider: ISendEndpointProvider ->
    db: RondelDbContext ->
    CommitRondelEffects

Consumer host model:

type MoveConsumer =
    interface IConsumer<Contract.Rondel.MoveCommand>

type AccountingCommandConsumer =
    interface IConsumer<Contract.Accounting.AccountingCommand>

Implementation Sketch - Commit Adapter

let createMassTransitCommit mapState publish send db : CommitRondelEffects =
    fun effects ->
        async {
            match effects.State with
            | Some state ->
                db.UpsertRondelState(mapState state)
            | None -> ()

            for event in effects.IntegrationEvents do
                let contract = RondelEvent.toContract event
                do! publish.Publish(contract) |> Async.AwaitTask

            for command in effects.OutboundCommands do
                let! endpoint = send.GetSendEndpoint(Uri("queue:accounting")) |> Async.AwaitTask

                match command with
                | ChargeMovement cmd ->
                    do! endpoint.Send(ChargeMovementOutboundCommand.toContract cmd) |> Async.AwaitTask
                | VoidCharge cmd ->
                    do! endpoint.Send(VoidChargeOutboundCommand.toContract cmd) |> Async.AwaitTask

            do! db.SaveChangesAsync() |> Async.AwaitTask |> Async.Ignore
        }

The EF outbox must be configured so MassTransit sends/publishes are written to the outbox and delivered later.

Implementation Sketch - Host Model

type MoveConsumer(depsFactory: ConsumeContext -> RondelDependencies) =
    interface IConsumer<Contract.Rondel.MoveCommand> with
        member _.Consume context =
            task {
                match MoveCommand.fromContract context.Message with
                | Error e -> invalidOp e
                | Ok cmd ->
                    let deps = depsFactory context
                    do! execute deps (RondelCommand.Move cmd) |> Async.StartAsTask
            }

Caller Example

let deps : RondelDependencies =
    { Load = loadFromEf db
      Commit = createMassTransitCommit mapState publishEndpoint sendEndpointProvider db }

Test Sketch

testCase "ef outbox rollback prevents partial commit" <| fun _ ->
    let db = createDb()
    let commit = createMassTransitCommit mapState failingPublish send db

    Expect.throws (fun () -> commit sampleEffects |> Async.RunSynchronously) "commit fails"
    Expect.isNone (loadState db sampleGameId) "state rolled back"
    Expect.isEmpty (loadOutboxRows db) "outbox rolled back"

testCase "consumer duplicate delivery relies on inbox/idempotency" <| fun _ ->
    deliverSameMessageTwice()
    Expect.equal (loadAccountingDebitCount billingId) 1 "consumer idempotency required"

Pro/Con Analysis

Architecture fit: good. Adapter mode fits current seam; host mode is a conventional message-consumer architecture.

Complexity: medium-high. MassTransit is powerful but configuration-heavy.

Risk: medium. Mature tooling, but outbox correctness depends on EF and MassTransit configuration.

Testability: medium-high. Requires integration tests with EF provider and possibly broker/test harness.

Migration: medium. Adapter mode is moderate; host mode shifts command delivery and hosting.

Extensibility: high. Strong support for multiple BCs and external messaging.

F# idiom score: 6/10. Domain stays idiomatic; consumer/configuration layer is C#-style.

Codebase consistency: 7/10 adapter mode, 6/10 host mode.

Confidence: medium.

Approach D: Rebus + SQL Server Outbox as Commit Adapter and Host Model

Summary

Use Rebus with SQL Server outbox support. Since SQL Server is acceptable, this is a viable simpler service-bus path.

Type Definitions

type RebusRondelCommit =
    SqlConnection -> IBus -> CommitRondelEffects

type SqlServerRondelStateRow =
    { GameId: Guid
      PayloadJson: string
      Version: int64 }

type RebusMessage =
    | RebusRondelEvent of Contract.Rondel.RondelEvent
    | RebusAccountingCommand of Contract.Accounting.AccountingCommand

Public Surface Sketch

val createRebusCommit:
    saveState: (SqlConnection -> SqlTransaction -> RondelState -> Async<unit>) ->
    bus: Rebus.Bus.IBus ->
    connectionFactory: (unit -> Task<SqlConnection>) ->
    CommitRondelEffects

Host model:

type RebusRondelCommandHandler =
    interface IHandleMessages<Contract.Rondel.MoveCommand>

Implementation Sketch - Commit Adapter

let createRebusCommit saveState (bus: Rebus.Bus.IBus) connectionFactory : CommitRondelEffects =
    fun effects ->
        async {
            use! connection = Async.AwaitTask (connectionFactory ())
            use! tx = Async.AwaitTask (connection.BeginTransactionAsync())

            try
                match effects.State with
                | Some state -> do! saveState connection tx state
                | None -> ()

                for event in effects.IntegrationEvents do
                    do! bus.Publish(RondelEvent.toContract event) |> Async.AwaitTask

                for command in effects.OutboundCommands do
                    match command with
                    | ChargeMovement cmd ->
                        do! bus.Send(ChargeMovementOutboundCommand.toContract cmd) |> Async.AwaitTask
                    | VoidCharge cmd ->
                        do! bus.Send(VoidChargeOutboundCommand.toContract cmd) |> Async.AwaitTask

                do! Async.AwaitTask (tx.CommitAsync())
            with ex ->
                do! Async.AwaitTask (tx.RollbackAsync())
                raise ex
        }

The concrete implementation should use Rebus SQL Server outbox transaction context instead of manually assuming bus.Send is enlisted. The sketch shows the shape the adapter must provide.

Implementation Sketch - Host Model

type MoveHandler(depsFactory: unit -> RondelDependencies) =
    interface IHandleMessages<Contract.Rondel.MoveCommand> with
        member _.Handle message =
            task {
                match MoveCommand.fromContract message with
                | Error e -> invalidOp e
                | Ok cmd ->
                    do! execute (depsFactory ()) (RondelCommand.Move cmd) |> Async.StartAsTask
            }

Caller Example

let deps : RondelDependencies =
    { Load = loadFromSqlServer
      Commit = createRebusCommit saveRondelState bus sqlConnectionFactory }

Test Sketch

testCase "rebus sql outbox stores outgoing messages with state" <| fun _ ->
    commit sampleEffects |> Async.RunSynchronously
    Expect.isSome (loadState sampleGameId) "state saved"
    Expect.isNonEmpty (loadRebusOutboxRows()) "messages stored"

testCase "rebus retries then moves poison message to error queue" <| fun _ ->
    enqueueOutboxMessage failingMessage
    runRebusUntilRetriesExhausted()
    Expect.isTrue (existsInErrorQueue failingMessage.Id) "poison message visible"

Pro/Con Analysis

Architecture fit: good if SQL Server is selected. Adapter mode is possible; host mode is simple.

Complexity: medium. Rebus is lighter than MassTransit but still framework-owned.

Risk: medium. Strongest around SQL Server; weaker if backend changes.

Testability: medium. Integration tests must verify SQL Server outbox and retry behavior.

Migration: medium. Simpler host migration than MassTransit in some cases, but still changes composition.

Extensibility: medium-high. Good for BC messaging; fewer advanced ecosystem features than MassTransit.

F# idiom score: 6/10. Domain remains idiomatic; host integration is .NET/OOP style.

Codebase consistency: 7/10 adapter mode, 7/10 host mode.

Confidence: medium.

Approach E: Akka.NET as Commit Adapter and Actor-Hosted BC Model

Summary

Evaluate Akka.NET in both modes. As only a CommitRondelEffects adapter, it still needs durable storage/outbox underneath and adds limited value. As a broader actor-host model, it is viable and may fit per-game serialization well.

Important revision: Rondel should remain snapshot/state based, not event-sourced. Therefore Akka.NET should not be evaluated as "Rondel actor journal as outbox" for this project. The preferred Akka actor-host model is:

RondelState snapshot + PendingDispatch records in one authoritative commit
then Akka.Delivery or Akka.Cluster.Sharding.Delivery as transport

Akka.Delivery's EventSourcedProducerQueue can make already-submitted producer messages durable, but it cannot recover messages that were never queued because the process crashed after saving state and before queue submission. It is transport durability, not the atomic outbox record.

Type Definitions

Adapter mode:

type CommitRequest =
    { Effects: RondelEffects
      ReplyTo: IActorRef }

type CommitResponse =
    | Committed
    | CommitFailed of string

type AkkaRondelCommit =
    IActorRef -> CommitRondelEffects

Actor-host mode:

type RondelActorMessage =
    | ExecuteCommand of RondelCommand
    | HandleInboundEvent of RondelInboundEvent
    | DeliverPendingOutbox

type PendingDispatchStatus =
    | Pending
    | Queued
    | TransportConfirmed
    | Failed

type PendingDispatch =
    { DispatchId: Guid
      GameId: Id
      Sequence: int64
      Payload: OutboxMessage
      Status: PendingDispatchStatus }

type RondelDurableRecord =
    { GameId: Id
      Version: int64
      State: RondelState option
      PendingDispatches: Map<Guid, PendingDispatch> }

Public Surface Sketch

Adapter:

val createAkkaCommitAdapter:
    commitActor: IActorRef ->
    timeout: TimeSpan ->
    CommitRondelEffects

Host model:

val rondelActorProps:
    deliver: DeliverOutboxMessage ->
    Props

Implementation Sketch - Commit Adapter

let createAkkaCommitAdapter (commitActor: IActorRef) timeout : CommitRondelEffects =
    fun effects ->
        async {
            let request = { Effects = effects; ReplyTo = ActorRefs.NoSender }
            let! response = commitActor.Ask(request, timeout) |> Async.AwaitTask

            match response with
            | :? CommitResponse as Committed -> ()
            | :? CommitResponse as CommitFailed error -> failwith error
            | _ -> failwith "Unexpected Akka commit response."
        }

This is only useful if the commit actor itself writes state plus outbox to a durable store transactionally.

Implementation Sketch - Actor Host Model

let toPendingDispatches gameId nextSequence effects =
    let events =
        effects.IntegrationEvents
        |> List.mapi (fun i event -> eventToOutbox gameId (nextSequence + int64 i) event)

    let commands =
        effects.OutboundCommands
        |> List.mapi (fun i command -> commandToOutbox gameId (nextSequence + int64 events.Length + int64 i) command)

    events @ commands

let commitEffects store gameId currentVersion nextSequence effects =
    async {
        let dispatches = toPendingDispatches gameId nextSequence effects

        // One authoritative commit:
        //   save effects.State as the current snapshot/state record
        //   save pending dispatches derived from events and outbound commands
        do! store.SaveStateAndDispatches(gameId, currentVersion, effects.State, dispatches)
    }

A production Akka implementation should not save only the final RondelState and rely on EventSourcedProducerQueue as the only durable outbound record. It should save state and pending dispatches together. After that commit, the actor or a relay can submit pending dispatches to ProducerController, ShardingProducerController, or another transport.

EventSourcedProducerQueue is still valuable:

• it can persist already-submitted messages;
• AskNextTo can confirm that the queue stored a dispatch;
• it supports ordered delivery and backpressure;
• it works with Akka.Cluster.Sharding.Delivery for sharded recipients.

But it does not remove the state+dispatch commit requirement.

Why final-state-only persistence is insufficient:

• simple paid moves may be derivable because PendingMovement stores Nation, TargetSpace, and BillingId;
• superseding unpaid movements require old pending movement details for VoidCharge and MoveToActionSpaceRejected;
• after saving only the final state, old billing IDs and rejection targets may be gone;
• integration events such as ActionDetermined also need durable dispatch records if they must be delivered.

Caller Example

let gameActor = shardRegion.ForEntity(gameId)
gameActor <! ExecuteCommand(RondelCommand.Move move)

Test Sketch

testCase "actor stores state and dispatches before delivery" <| fun _ ->
    actor <! ExecuteCommand sampleMove
    let record = loadDurableRecord sampleGameId
    Expect.isSome record.State "state snapshot stored"
    Expect.isNonEmpty record.PendingDispatches "pending dispatch tracked"

testCase "actor restart resends pending dispatches" <| fun _ ->
    actor <! ExecuteCommand samplePaidMove
    stopActor actor
    let restarted = startActor gameId
    restarted <! DeliverPendingOutbox
    Expect.isTrue (deliveryAttempted billingCommandId) "pending dispatch delivered after restart"

testCase "superseded pending movement preserves void dispatch" <| fun _ ->
    actor <! ExecuteCommand sampleSupersedingMove
    let record = loadDurableRecord sampleGameId
    Expect.contains (dispatchKinds record.PendingDispatches) "VoidCharge" "old charge void is durable"
    Expect.contains (dispatchKinds record.PendingDispatches) "MoveToActionSpaceRejected" "rejection event is durable"

Pro/Con Analysis

Architecture fit: weak as a thin adapter; strong if the web host moves to actor-hosted per-game BCs.

Complexity: high. Requires actor lifecycle, persistence, sharding decisions, reliable delivery, and idempotent consumers.

Risk: high as an incremental outbox implementation; medium as a deliberate architecture choice.

Testability: medium. Actor persistence/restart tests are more complex than SQL transaction tests.

Migration: high. Host model and operational model change significantly.

Akka-specific conclusion: choose Akka.NET only for actor-hosting reasons. If selected, keep the commit model snapshot based:

state snapshot + pending dispatches first
Akka.Delivery / ShardingProducerController second

Extensibility: high if actors become the chosen web architecture. Per-game ordering becomes natural.

F# idiom score: 6/10. Actors can be functional internally, but Akka.NET APIs are strongly object-oriented.

Codebase consistency: 4/10 adapter mode, 7/10 host mode if chosen as a separate architecture decision.

Confidence: medium.

Comparison Matrix

Scores: 1 weak, 5 strong.

Criterion	Custom SQL	Wolverine	MassTransit	Rebus	Akka.NET
Atomic commit guarantee	5	5	5	4	3 adapter / 4 host
Current seam fit	5	4	4	4	2
Broader host-model fit	2	5	4	4	5
Per-GameId ordering clarity	5	4	4	4	5
Retry/backoff/dead-letter	2	5	5	4	4
Observability	2	5	5	4	3
F# / codebase consistency	5	4	3	3	3
Migration cost	3	3	3	3	1
Testability	5	4	4	4	3

Recommendation

The recommended design is to specify a canonical durable outbox contract independent of any one framework, then choose the implementation path based on the future host decision.

Primary recommendation:

1. Define the type = improvement: Design durable outbox storage and relay #130 design around the CommitRondelEffects seam and the common outbox model.
2. Require all implementations to provide the same guarantees:
   • state and outbox inserted atomically;
   • failed transaction rolls back both;
   • relay acknowledges only after delivery;
   • failed delivery remains pending or dead-lettered by policy;
   • duplicates are expected and delegated to type = improvement: Add inbox and processed-message tracking for inbound delivery #131;
   • per-GameId ordering is explicit and tested.
3. Treat custom SQL outbox as the reference implementation model because it is the clearest expression of the invariant.
4. Prefer Wolverine if the project chooses Marten/PostgreSQL or wants a framework-hosted web architecture.
5. Prefer MassTransit if the project chooses EF Core plus a mature brokered messaging stack.
6. Prefer Rebus if the project chooses SQL Server and values a simpler bus.
7. Prefer Akka.NET only if a separate architecture decision selects actor-hosted bounded contexts; do not choose Akka.NET merely to implement outbox behind the current seam.
   • If Akka.NET is selected, preserve Rondel's snapshot-based state model: commit RondelState plus pending dispatches atomically, then use Akka.Delivery or Akka.Cluster.Sharding.Delivery as transport.

Practical recommendation for issue #130:

Record the durable outbox storage model in framework-neutral terms, then explicitly list backend-specific mappings:

• Custom SQL: direct outbox table plus relay.
• Wolverine: Wolverine envelopes/outbox tables must satisfy the same state+message atomicity and per-game ordering requirements.
• MassTransit: EF outbox tables must satisfy the same requirements.
• Rebus: SQL Server outbox must satisfy the same requirements.
• Akka.NET: snapshot/state storage plus pending dispatches must satisfy the same requirements if actor hosting is selected; Akka.Delivery and Akka.Cluster.Sharding.Delivery are transport options after that commit.

This keeps #130 useful regardless of the later backend decision and prevents framework adoption from hiding the actual consistency requirements.

Risks and Mitigations

Risk: framework outbox hides ordering semantics.

Mitigation: require explicit tests for per-GameId sequencing and delivery order in every selected implementation.

Risk: relay crash after delivery but before ack creates duplicate delivery.

Mitigation: document this as expected behavior and implement #131 before relying on durable retry in production.

Risk: poison message blocks later messages for the same game.

Mitigation: choose and document policy: either preserve strict ordering and block until operator intervention, or dead-letter poison messages and allow later messages after an explicit status transition. Recommendation: strict ordering while pending; allow bypass only after dead-letter with observable operator-visible status.

Risk: custom SQL relay grows into a poorly maintained mini-bus.

Mitigation: keep custom outbox small, limited to BC messages, and revisit Wolverine/MassTransit/Rebus if relay needs expand.

Risk: Akka.NET actor direction changes the problem scope.

Mitigation: create a separate host-architecture decision before implementing Akka actor hosting. That decision should explicitly reject event-sourcing Rondel merely to get outbox behavior, unless the domain model itself changes. The durable Akka path should be snapshot state plus pending dispatches in one commit.

Risk: Accounting is not idempotent yet.

Mitigation: keep #131 and #49 as prerequisites for production durable retry.

Migration Path

Step 1: Record #130 design decision.

• Add an issue comment or follow-up issue describing the common outbox schema, relay behavior, ordering policy, and observability requirements.
• Do not implement terminal in-memory outbox.

Step 2: Choose durable backend/host.

• Decide among custom SQL, Wolverine, MassTransit, Rebus, or Akka.NET.
• If Akka.NET is selected, create a separate actor-hosting decision before implementation and require the snapshot-state plus pending-dispatch commit model.

Step 3: Add durable storage model.

• Add state storage for Rondel.
• Add outbox storage or framework envelope storage.
• Add transaction boundary that saves state and messages together.

Step 4: Implement CommitRondelEffects.

• Keep domain unchanged.
• Wire web/durable host with { Load; Commit }.
• Preserve terminal RondelDirectCommit unchanged.

Step 5: Implement relay.

• Claim due messages safely.
• Deliver by destination.
• Ack only after success.
• Retry failed messages with backoff.
• Dead-letter poison messages after policy limit.
• Expose stuck/failed metrics and diagnostics.

Step 6: Implement consumer idempotency.

• Complete type = improvement: Add inbox and processed-message tracking for inbound delivery #131 inbox or processed-message tracking.
• Complete Implement Stateful Accounting Domain with Balance Tracking #49 stateful Accounting idempotency by BillingId.

Step 7: Wire web API.

• Validate contracts synchronously.
• Return 400 for transformation errors.
• Enqueue/execute accepted commands and return 202.
• Let durable outbox and relay handle completion asynchronously.

Test Strategy

Commit Tests

• Atomic success: state and all outbox messages appear together.
• Rollback: injected outbox insert failure leaves no state and no messages.
• Empty effects: no state and no messages is a no-op.
• Event-only effects: valid if handler returns no state but emits events.
• Command-only effects: valid if handler returns no state but dispatches commands.
• Serialization: each message has destination, type, payload, GameId, and sequence.

Ordering Tests

• Multiple messages from one RondelEffects receive contiguous per-game sequence numbers.
• Messages for one GameId are claimed in sequence order.
• Concurrent relay workers do not claim the same message.
• Later messages for the same GameId do not overtake a pending earlier message unless the earlier message is dead-lettered by explicit policy.

Relay Tests

• Successful delivery marks processed only after delivery succeeds.
• Failed delivery increments attempts and schedules next retry.
• Repeated failure dead-letters after policy limit.
• Relay crash after delivery before ack causes redelivery.
• Dead-letter messages include last error and attempt metadata.

Observability Tests

• Stuck pending message age can be queried.
• Repeated delivery failures surface destination, type, attempts, last error, and next attempt time.
• Dead-letter messages are queryable.
• Correlation/causation IDs are preserved when available.

Consumer Idempotency Tests

These belong primarily to #131 and #49 but must be called out as prerequisites:

• Duplicate RondelInvoicePaid does not emit duplicate completed-move effects.
• Duplicate RondelInvoicePaymentFailed does not emit duplicate rejection effects.
• Duplicate Accounting charge with the same BillingId does not debit twice.
• Duplicate Accounting void is safe.
• Failed-charge retry behavior is explicitly tested as terminal or retryable.

Final Position

Issue #130 should not choose a framework by accident. It should define the durable outbox contract and relay behavior in terms of the existing RondelEffects seam. Frameworks are implementation strategies that must satisfy that contract.

Recommended ranking today:

1. Wolverine, if choosing Marten/PostgreSQL or a framework-hosted durable web model.
2. Custom SQL, if preserving the current architecture with minimal host-model change.
3. MassTransit, if choosing EF Core and a mature brokered messaging stack.
4. Rebus, if choosing SQL Server and a simpler service bus.
5. Akka.NET, if choosing actor-hosted bounded contexts as a separate architecture decision.

Research is complete. No implementation has been performed.