feat(etl): add lease-fenced durable job worker

CHANGELOG.md

@@ -20,7 +20,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Updated existing pull-request candidates now carry their captured pre-agent head into the deterministic publisher, which rejects destructive ancestry, more than 50 agent-introduced files, and any agent-introduced `.github/**` or `CODEOWNERS` change before exposing the updated pull request or authorizing checks.
 - The hourly OpenCode workflow now uses the current free NVIDIA `deepseek-ai/deepseek-v4-pro` endpoint for long-context coding and agentic tool use instead of the deprecated Qwen3 Coder free endpoint; model or endpoint rejection fails visibly without a non-NVIDIA, partner-only, or automatic fallback.
 - The managed Jackson component set now uses the patched 2.21.5 BOM, closing CVE-2026-54515, CVE-2026-59889, and GHSA-mhm7-754m-9p8w while keeping core, annotations, datatype, and module artifacts aligned.
-- Durable `POST /api/etl/jobs` submissions now return RFC 9110 `202 Accepted`, a stable pending-job representation, `Location` status-monitor metadata, and explicit replay metadata without changing the synchronous `/api/etl/process` contract. The incomplete intake controller is fail-closed and requires explicit `xtrmetl.etl.jobs.intake-enabled=true` operator opt-in until worker execution and terminal payload clearing are implemented.
+- Durable `POST /api/etl/jobs` submissions now return RFC 9110 `202 Accepted`, a stable pending-job representation, `Location` status-monitor metadata, and explicit replay metadata without changing the synchronous `/api/etl/process` contract. Durable intake remains independently opt-in through the preferred `mightyetl.etl.jobs.intake-enabled=true` property or supported legacy alias, while worker activation is a separate disabled-by-default control.
+- Durable ETL jobs now execute through an opt-in lease-fenced worker: PostgreSQL `FOR UPDATE SKIP LOCKED` claims one eligible job per poll across replicas, exact lease-owner/token fencing prevents stale transitions, target writes plus terminal success commit transactionally, terminal states clear retained payloads, retries are bounded with stable non-sensitive failure codes, and fixed-cardinality worker telemetry records `idle`, `succeeded`, `retried`, `failed`, and `stale` outcomes.
 - Concurrent requests using the same authenticated-principal-scoped semantic idempotency key now return immediate RFC 9457 `409 etl_idempotency_request_in_progress` responses through PostgreSQL `pg_try_advisory_xact_lock`; retries after completion still replay the committed response.
 - `POST /api/etl/process` now supports optional authenticated-principal-scoped `Idempotency-Key` retries with atomic target writes, durable response replay, payload-conflict rejection, and explicit replay response metadata.
 - `Idempotency-Key` now prefers the quoted RFC 9651 Structured Field String representation while retaining and normalizing the legacy raw representation to the same durable ledger key.

docs/etl/durable-job-intake.md

@@ -2,17 +2,21 @@
 
 ## Scope
 
-`POST /api/etl/jobs` creates a durable, authenticated-principal-scoped ETL job resource. This
-bounded intake slice persists accepted work and exposes its status monitor; it does not execute jobs
-yet. Execution, PostgreSQL `FOR UPDATE SKIP LOCKED` claiming, lease fencing, bounded attempts,
-terminal payload clearing, and crash recovery belong to the following worker and lease-fencing slice.
-
-The incomplete intake surface is disabled by default. It is absent unless an operator explicitly
-sets the preferred `mightyetl.etl.jobs.intake-enabled=true` property, its supported legacy alias
-`xtrmetl.etl.jobs.intake-enabled=true`, or `ETL_JOB_INTAKE_ENABLED=true`. When both full namespaces
-are supplied, `mightyetl.*` wins. Enabling intake accepts the temporary boundary that submitted
-payloads remain retained in `PENDING` jobs until the worker and terminal payload-clearing slice is
-implemented. Deployments that cannot accept that retention boundary must leave the setting false.
+`POST /api/etl/jobs` creates a durable, authenticated-principal-scoped ETL job resource. Durable
+execution is now implemented as a separate opt-in worker boundary: mightyETL executes at most one
+eligible durable job per worker poll, while PostgreSQL owns cross-replica claim arbitration through
+`FOR UPDATE SKIP LOCKED`, lease fencing, bounded attempts, and exact conditional lifecycle
+transitions.
+
+Both externally reachable intake and background execution remain disabled by default. Durable job
+intake is disabled by default and is absent unless an operator explicitly sets the preferred
+`mightyetl.etl.jobs.intake-enabled=true` property, its supported legacy alias
+`xtrmetl.etl.jobs.intake-enabled=true`, or `ETL_JOB_INTAKE_ENABLED=true`. The worker is disabled by
+default and does not poll until an operator explicitly enables either the preferred
+`mightyetl.etl.jobs.worker.enabled=true` property or its supported legacy alias
+`xtrmetl.etl.jobs.worker.enabled=true`. When both full namespaces are supplied, `mightyetl.*` wins.
+Keeping intake and execution as separate opt-ins allows an operator to stage schema and API rollout
+without silently starting background target writes.
 
 The existing synchronous `POST /api/etl/process` endpoint remains unchanged.
 
@@ -79,8 +83,58 @@ return `404 etl_job_not_found`; callers cannot use this endpoint to probe anothe
 existence.
 
 The response excludes the request payload, raw principal, raw submission key, and all internal
-hashes. Timestamps are explicit ISO-8601 strings. Before worker execution is implemented, newly
-accepted jobs remain `PENDING` with an `attemptCount` of zero.
+hashes. Timestamps are explicit ISO-8601 strings. A newly accepted job begins as `PENDING`. When the
+worker claims it, the database records `RUNNING`, a lease owner, an opaque lease token, lease expiry,
+and the incremented attempt count. Terminal success or failure clears the retained request payload.
+
+## Worker execution and lease fencing
+
+The scheduled worker claims at most one eligible row per poll. The claim repository serializes the
+selection in PostgreSQL with `FOR UPDATE SKIP LOCKED`, so concurrent replicas do not wait on or
+execute the same currently claimable row. Eligibility includes new `PENDING` work and reclaimable
+expired `RUNNING` work while the configured maximum attempt count has not been exhausted.
+
+Every mutable lifecycle transition is fenced by the exact claim identity. Success, retry release,
+and terminal failure must still match the job record, lease owner, opaque lease token, and valid
+lease boundary expected by the caller. A stale or superseded worker therefore cannot overwrite a
+newer owner's state. Stale transitions surface as a finite `stale` worker outcome rather than being
+silently accepted.
+
+`EtlJobExecutionService` is transactional. It validates the retained job and durable response-ledger
+identity, executes or replays the target operation, and then marks the exact live lease `SUCCEEDED`
+in the same transaction. If the success transition is stale, Spring rolls back the target and
+response-ledger effects from that execution attempt.
+
+Transient Spring data-access failures are released for retry only while attempts remain. Exhausted
+transient failures become `etl_target_unavailable`; non-transient data-access failures become
+`etl_target_failure`; deterministic request and integrity failures retain their stable application
+error codes; and unexpected runtime failures become `etl_internal_error`. The worker does not place
+raw payloads, principals, submission keys, job identifiers, lease identifiers, SQL, exception class
+names, or exception messages into metric labels.
+
+Worker telemetry uses the fixed terminal outcome vocabulary `idle`, `succeeded`, `retried`, `failed`,
+and `stale`. Each completed poll records exactly one outcome and one matching duration sample,
+including idle polls and database failures while persisting retry or terminal state.
+
+## Worker configuration
+
+The production worker remains fail-closed until explicitly activated. Supported keys include:
+
+- `mightyetl.etl.jobs.worker.enabled` / `xtrmetl.etl.jobs.worker.enabled`;
+- `mightyetl.etl.jobs.worker.fixed-delay-milliseconds` /
+  `xtrmetl.etl.jobs.worker.fixed-delay-milliseconds`;
+- `mightyetl.etl.jobs.worker.initial-delay-milliseconds` /
+  `xtrmetl.etl.jobs.worker.initial-delay-milliseconds`;
+- `mightyetl.etl.jobs.worker.lease-duration-seconds` /
+  `xtrmetl.etl.jobs.worker.lease-duration-seconds`;
+- `mightyetl.etl.jobs.worker.max-attempts` / `xtrmetl.etl.jobs.worker.max-attempts`; and
+- `mightyetl.etl.jobs.worker.lease-owner-id` / `xtrmetl.etl.jobs.worker.lease-owner-id`.
+
+Defaults are bounded in production configuration, and property validation rejects non-positive delay,
+lease-duration, and attempt settings as well as blank or oversized lease-owner identifiers. Operators
+should assign a stable, non-secret owner identifier per worker instance and size lease duration above
+normal execution latency while retaining enough margin for crash recovery through expired-lease
+reclamation.
 
 ## Validation and persistence
 
@@ -89,43 +143,51 @@ bounds used by synchronous ETL admission. The complete body must be a JSON array
 fields are rejected, every element must be an object with a safe textual `id`, and normalized field
 names must remain unique.
 
-Flyway migration `V2__create_etl_job_records.sql` creates `etl_job_records`. All schema objects use
+Flyway migration `V2__create_etl_job_records.sql` creates `etl_job_records`; later worker migrations
+add lease-fencing columns and a partial eligibility index for claim scans. All schema objects use
 descriptive multi-word `snake_case` names. The database stores:
 
 - an opaque UUID job identifier;
 - SHA-256 hashes of the principal scope, semantic submission key, and exact JSON text;
-- the request payload needed by the future worker while status is `PENDING` or `RUNNING`;
-- status, attempt, failure, and timestamp fields.
+- the request payload only while the job remains nonterminal;
+- status, attempt, failure, lease-owner/token/expiry, and lifecycle timestamp fields.
 
-The schema reserves the stable lifecycle vocabulary `PENDING`, `RUNNING`, `SUCCEEDED`, and `FAILED`.
-A database check requires a non-null request payload only for the two nonterminal states and requires
-that payload to be null for both terminal states. This makes terminal payload clearing an enforced
-persistence invariant rather than a documentation-only convention.
+The stable lifecycle vocabulary is `PENDING`, `RUNNING`, `SUCCEEDED`, and `FAILED`. Database checks
+require a non-null request payload only for nonterminal states and require the payload to be null for
+terminal states. Terminal payload clearing is therefore a persistence invariant, not merely an
+application convention.
 
-Raw authenticated principal names and raw idempotency keys are never persisted. The request payload
-is sensitive operational data and must inherit the classification of its source records. Until the
-worker slice reaches a terminal state and clears it, operators must apply database access control,
-encryption, backup, and retention policy accordingly.
+Raw authenticated principal names and raw idempotency keys are never persisted. The retained request
+payload is sensitive operational data and inherits the classification of its source records. While a
+job is `PENDING` or `RUNNING`, operators must protect it with database access control, encryption,
+backup, and retention policy appropriate to the underlying records.
 
 ## Operational boundary
 
-This slice deliberately does not advertise job completion or background execution. The controller is
-disabled by default; setting an activation property to `true` is an explicit operator opt-in to
-durable intake without execution. Deployments that need completed asynchronous processing must wait
-for the worker and lease-fencing slice. The next slice must claim jobs safely across replicas, fence
-stale lease owners, commit target effects and terminal success atomically, reclaim expired leases,
-bound attempts, publish stable failure codes, and clear the stored request payload at terminal state.
+Enabling intake alone still does not start background processing; enabling the worker alone does not
+create externally submitted jobs. A production deployment that wants asynchronous execution must
+intentionally enable both surfaces and apply the worker migrations first. Rollback is operationally
+safe by disabling the worker property: existing durable rows remain in PostgreSQL and expired
+`RUNNING` leases become reclaimable when a compatible worker is enabled again. Operators must not
+manually rewrite lease tokens or terminal status to manufacture recovery.
+
+This slice establishes durable execution, lease fencing, bounded retries, terminal payload clearing,
+and finite worker telemetry. Higher-level job-list pagination, polling advisories, conditional status
+reads, cancellation, and replay remain separate later stack items and must not be represented as part
+of this boundary until their own exact-head gates pass.
 
 ## Standards basis
 
 - RFC 9110 Section 15.3.3 defines `202 Accepted` as noncommittal and recommends that the response
   describe current status and point to a status monitor.
-- RFC 9457 supplies the problem-details representation used by deterministic submission and lookup
-  failures.
+- RFC 9457 supplies the problem-details representation used by deterministic submission, lookup, and
+  execution failures.
 - RFC 9651 defines the current Structured Fields String syntax accepted for `Idempotency-Key`.
 - The expired IETF HTTPAPI `Idempotency-Key` draft-07 is used only as work-in-progress design
   evidence for unique client keys, request fingerprints, `422` payload conflicts, and tenant-isolation
   security concerns. It expired on April 18, 2026 and is not represented as a published RFC.
+- PostgreSQL row locking and `SKIP LOCKED` semantics are the database authority for concurrent claim
+  behavior; the worker does not attempt to replace that arbitration with process-local locking.
 
 ### References
 
@@ -138,3 +200,5 @@ bound attempts, publish stable failure codes, and clear the stored request paylo
   https://www.rfc-editor.org/rfc/rfc9457
 - Nottingham, M., & Kamp, P. (2024). *Structured field values for HTTP* (RFC 9651). RFC Editor.
   https://www.rfc-editor.org/rfc/rfc9651
+- PostgreSQL Global Development Group. (2026). *SELECT*. PostgreSQL 18 documentation.
+  https://www.postgresql.org/docs/18/sql-select.html

docs/operations/durable-job-claim-index-rollout.md

@@ -0,0 +1,95 @@
+# Durable-job claim index rollout
+
+## Purpose
+
+The durable worker queries `etl_job_records` for the oldest eligible `PENDING` row or expired
+`RUNNING` row. The descriptive `etl_job_claim_eligibility_index` supports that queue-like access
+path without changing the lease-fencing state machine.
+
+The table can already receive job submissions when this index is introduced. PostgreSQL's ordinary
+`CREATE INDEX` permits reads but blocks `INSERT`, `UPDATE`, and `DELETE` until the build completes.
+For a production ETL control plane that write outage is not acceptable. Migration
+`V4__add_etl_job_claim_eligibility_index.sql` therefore uses `CREATE INDEX CONCURRENTLY`.
+
+## Flyway execution boundary
+
+PostgreSQL rejects `CREATE INDEX CONCURRENTLY` inside a transaction block. The companion script
+configuration file
+`V4__add_etl_job_claim_eligibility_index.sql.conf` contains:
+
+```properties
+executeInTransaction=false
+```
+
+Flyway's PostgreSQL transactional advisory lock is also disabled with:
+
+```properties
+spring.flyway.postgresql.transactional-lock=false
+```
+
+This causes Flyway to use the PostgreSQL integration's non-transactional lock mode required for
+concurrent index DDL. The transactional V3 migration remains responsible only for lease columns,
+legacy-data repair, and lifecycle constraints. Isolating the index in V4 prevents an index-build
+failure from partially committing those schema invariants.
+
+## Deployment procedure
+
+1. Keep durable-job intake and worker execution disabled while validating the migration package.
+2. Confirm no other concurrent index build or schema migration is active on `etl_job_records`.
+3. Apply V3 and verify the lease columns and lifecycle constraints.
+4. Apply V4 and monitor `pg_stat_progress_create_index`, database I/O, and transaction latency.
+5. Verify `pg_index.indisvalid` is true for `etl_job_claim_eligibility_index`.
+6. Run the claim-selection plan against production-equivalent data and confirm the expected index is
+   available without forcing it through planner settings.
+7. Enable one worker canary only after schema history, catalog state, and application health agree.
+
+`CREATE INDEX CONCURRENTLY` performs more work and can take longer than a regular build. It preserves
+normal writes, but it still adds CPU, memory, and I/O load and allows only one concurrent index build
+per table.
+
+## Failed migration and invalid-index recovery
+
+A failed concurrent build can leave an **invalid index** in the PostgreSQL catalog. Do not mark the
+Flyway migration successful merely because an index name exists.
+
+Recovery is fail-closed:
+
+1. Keep worker execution disabled and inspect the failed Flyway record plus `pg_index.indisvalid`.
+2. Preserve database and migration logs under incident-response controls.
+3. Correct the underlying resource, permission, duplicate-build, or transaction-mode cause.
+4. Remove an unusable index without blocking normal table access:
+
+   ```sql
+   DROP INDEX CONCURRENTLY etl_job_claim_eligibility_index;
+   ```
+
+5. Use Flyway repair only after catalog inspection and operator approval, then rerun the unchanged,
+   checksum-verified migration.
+6. Reconfirm index validity and query-plan evidence before enabling workers.
+
+Do not edit an applied versioned migration or create a same-name replacement with different SQL.
+
+## Rollback boundary
+
+Application rollback does not require removing the index; an unused valid index is compatible with
+older binaries, although it adds write-maintenance overhead. Remove it only after every deployed
+worker version no longer relies on it and a controlled change has verified the performance impact:
+
+```sql
+DROP INDEX CONCURRENTLY etl_job_claim_eligibility_index;
+```
+
+`DROP INDEX CONCURRENTLY` must also run outside a transaction block. Lease columns and constraints
+require a later forward compensating migration after all compatible binaries have been removed; they
+must not be rolled back by editing V3.
+
+## Standards and primary documentation
+
+PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: Building indexes
+concurrently*. https://www.postgresql.org/docs/18/sql-createindex.html#SQL-CREATEINDEX-CONCURRENTLY
+
+Redgate Software. (2026). *Flyway script configuration*.
+https://documentation.red-gate.com/flyway/reference/script-configuration
+
+Redgate Software. (2026). *Flyway PostgreSQL transactional lock setting*.
+https://documentation.red-gate.com/fd/flyway-postgresql-transactional-lock-setting-277579114.html

etl-service/src/main/java/com/xtrmetl/etl/config/MightyEtlConfigAliasEnvironmentPostProcessor.java

@@ -28,6 +28,12 @@ public class MightyEtlConfigAliasEnvironmentPostProcessor implements Environment
             "etl.max-payload-bytes",
             "etl.max-batch-records",
             "etl.jobs.intake-enabled",
+            "etl.jobs.worker.enabled",
+            "etl.jobs.worker.fixed-delay-milliseconds",
+            "etl.jobs.worker.initial-delay-milliseconds",
+            "etl.jobs.worker.lease-duration-seconds",
+            "etl.jobs.worker.max-attempts",
+            "etl.jobs.worker.lease-owner-id",
             "connectors.databricks.enabled",
             "connectors.snowflake.enabled",
             "connectors.qlik-sense.enabled"