PayFlow

PayFlow processes payments across distributed systems. The core system demonstrates transactional correctness, event-driven architecture, and idempotent processing.

System components

Five components form the system:

1. Payment API: Handles idempotent creation and refunds backed by a double-entry ledger.
2. Ledger: Maintains immutable double-entry accounting. All transactions net to zero.
3. Outbox and Kafka publisher: Handles event publishing without dual-write problems.
4. Consumer with retry and DLQ: Delivers events at least once and sends failed events to a dead-letter queue.
5. Settlement engine: Runs scheduled batch operations per merchant.

How does PayFlow prevent accounting errors? Every transaction requires balanced debit and credit entries.

The scope excludes wallets, merchant onboarding, fraud detection, and admin dashboards.

Architecture

Component diagram

flowchart LR
    Client[Client / Postman] -->|REST + Idempotency-Key| API[Payment API<br/>Spring Boot]
    API -->|same DB txn| DB[(PostgreSQL)]
    DB --- Ledger[ledger_entries]
    DB --- Outbox[(outbox_events)]
    Outbox -->|poll unpublished rows| Publisher[Outbox Publisher]
    Publisher --> Topic[[Kafka: payment.events]]
    Topic --> Worker[Consumer]
    Worker -->|success| DB
    Worker -->|retries exhausted| DLQ[[Kafka: payments.dlq]]
    Scheduler[Spring @Scheduled] -->|daily trigger| Settlement[Settlement Engine]
    Settlement --> DB
    Settlement --> Batches[(settlement_batches)]

Loading

The outbox pattern prevents dual-write issues. The API writes ledger entries and outbox events in one database transaction. A system failure between Postgres and Kafka cannot drop or duplicate events. A poller reads unpublished rows, publishes them to Kafka, and marks rows published after receiving broker acknowledgements.

Payment lifecycle

sequenceDiagram
    participant C as Client
    participant A as Payment API
    participant D as Postgres
    participant P as Outbox Publisher
    participant K as Kafka
    participant W as Consumer

    C->>A: POST /payments (Idempotency-Key: X)
    A->>D: check idempotency_key (unique constraint)
    alt key already exists
        A-->>C: return original result (no reprocessing)
    else new key
        A->>D: BEGIN TXN
        A->>D: SELECT FOR UPDATE accounts (lock payer row)
        A->>D: check balance >= amount
        alt insufficient funds
            A->>D: ROLLBACK
            A-->>C: 422 Unprocessable Entity
        else sufficient funds
            A->>D: INSERT ledger_entries (debit payer, credit payee)
            A->>D: INSERT outbox_events (payment.created)
            A->>D: COMMIT
            A-->>C: 201 Created
            P->>D: poll unpublished outbox rows
            P->>K: publish payment.created
            K->>W: consume
            W->>W: process (retry with backoff on failure)
            alt success
                W->>D: mark event processed
            else retries exhausted
                W->>K: publish to payments.dlq
            end
        end
    end

Loading

Data model

Table	Key columns	Notes
accounts	id, owner_type, owner_ref, currency	Values for owner_type include payer, merchant, and platform.
ledger_entries	id, account_id, payment_id, amount, direction	Immutable insert-only table. Entries per payment net to zero.
payments	id, idempotency_key (unique), amount, status, payer_account_id, payee_account_id, settlement_batch_id	Unique idempotency keys prevent duplicate processing.
outbox_events	id, aggregate_id, event_type, payload, published_at	Null values in published_at mark unpublished events.
settlement_batches	id, merchant_id, period, status, UNIQUE(merchant_id, period)	Unique constraints stop duplicate settlement runs for a period.

Tech stack

Java 21, Spring Boot 4, PostgreSQL, jOOQ, Flyway, Apache Kafka, Docker Compose, JUnit 5, Testcontainers, Mockito, springdoc-openapi.

The design decision section details the reasoning for each technology choice.

Local execution

Prerequisites: JDK 21 and Docker Desktop.

Run the infrastructure and application:

docker compose up -d
./mvnw spring-boot:run

Flyway runs database migrations during application startup.

Check health status: curl http://localhost:8080/actuator/health

Open API documentation: http://localhost:8080/swagger-ui.html

Interactive API Demo UI: http://localhost:8080/index.html

• Interactive 9-scenario test suite in the browser: account setup, payments, insufficient funds, self-payment rejection, refunds, duplicate refund guards, idempotency replays, and real-time parallel race-condition execution.

API

Method	Path	Description
POST	/accounts	Creates a payer or merchant account
POST	/payments	Creates a payment using the Idempotency-Key header
GET	/payments/{id}	Returns payment details
POST	/payments/{id}/refund	Reverses ledger entries for a confirmed payment

Testing

Run unit and integration tests:

./mvnw test

Tests execute against PostgreSQL via Testcontainers and Apache Kafka using @EmbeddedKafka. The test suite avoids mock objects for infrastructure components:

• Verifies transactional atomicity and append-only ledger zero-sum invariants.
• Proves race-condition safety under multi-threaded concurrency (CountDownLatch), ensuring parallel debits exceeding balance cannot double-spend.
• Verifies consumer retry and dead-letter queue (DLQ) behavior.
• Confirms idempotent settlement batch processing.

Design decisions

• Pessimistic row locking (SELECT ... FOR UPDATE) on the payer account serializes concurrent debits within the database transaction. If simultaneous requests exceed available funds, exactly one succeeds and the other is safely rejected with InsufficientFundsException (HTTP 422), eliminating double-spending without optimistic locking retry overhead.
• Database unique constraints enforce idempotency instead of application checks. Application checks create race conditions during concurrent requests.
• The ledger system uses insert-only operations. Rows never update. The system calculates account balances by summing transaction entries.
• The outbox publisher marks rows processed after receiving Kafka broker acknowledgements. Asynchronous publishing requires confirmed receipts to prevent data loss.
• A single DefaultErrorHandler bean configures retry and dead-letter queues across all @KafkaListener annotations.
• Settlement idempotency uses two protection layers. Completed settlement periods skip processing, and a UNIQUE(merchant_id, period) database constraint blocks concurrent execution.