Buyer-visible gap
Durable ETL job intake is not commercially complete until accepted jobs can be claimed, executed, recovered after process failure, and driven to terminal state across horizontally scaled replicas. Pending request payloads must not remain indefinitely solely because no worker exists.
Vertical slice
Add a database-owned durable worker lifecycle:
- claim one eligible
etl_job_records row with PostgreSQL FOR UPDATE SKIP LOCKED;
- use a unique
lease_claim_id and stable lease_owner_id to fence every execution and terminal transition;
- recover an expired running lease after the owning transaction or connection is gone;
- execute through the existing durable idempotent ETL path so target rows, response ledger, and
succeeded state commit atomically;
- retry stable transient failures with a bounded attempt count;
- mark deterministic or exhausted work
failed using stable non-sensitive codes;
- clear
request_payload atomically on every terminal transition;
- expose succeeded and failed status through the existing owner-scoped resource API;
- poll with bounded fixed-delay scheduling while PostgreSQL, not scheduler uniqueness, owns cross-replica distribution.
Required guarantees
- no two live workers can commit target effects for the same job;
- a stale lease owner cannot commit terminal state;
- a crash cannot leave committed target rows without durable success metadata;
- expired/exhausted jobs reach a terminal failure and clear payloads;
- retryable work cannot loop without a configured upper bound;
- logs, metrics, database failure codes, and API responses never expose payloads, raw principals, raw idempotency keys, SQL, or exception messages;
- every production method and record has complete Javadoc;
- every added statement and branch has focused tests with no skipped project tests;
- migration, state machine, claiming, fencing, retries, privacy, operations, and rollback are documented and recorded in
CHANGELOG.md.
Standards evidence
- RFC 9110
202 Accepted status-monitor semantics;
- PostgreSQL 18
SELECT ... FOR UPDATE SKIP LOCKED guidance for queue-like consumers;
- Spring fixed-delay scheduling, with database claiming as the correctness boundary.
Non-goals
Cancellation, priority queues, recurring schedules, operator replay, and a dead-letter UI remain separate state-machine and product slices.
Buyer-visible gap
Durable ETL job intake is not commercially complete until accepted jobs can be claimed, executed, recovered after process failure, and driven to terminal state across horizontally scaled replicas. Pending request payloads must not remain indefinitely solely because no worker exists.
Vertical slice
Add a database-owned durable worker lifecycle:
etl_job_recordsrow with PostgreSQLFOR UPDATE SKIP LOCKED;lease_claim_idand stablelease_owner_idto fence every execution and terminal transition;succeededstate commit atomically;failedusing stable non-sensitive codes;request_payloadatomically on every terminal transition;Required guarantees
CHANGELOG.md.Standards evidence
202 Acceptedstatus-monitor semantics;SELECT ... FOR UPDATE SKIP LOCKEDguidance for queue-like consumers;Non-goals
Cancellation, priority queues, recurring schedules, operator replay, and a dead-letter UI remain separate state-machine and product slices.