From 956b9eef22160325936194119dec1c58d60212c6 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:29:46 +0900 Subject: [PATCH 01/92] docs: design lease-fenced durable job worker --- ...6-08-05-durable-job-lease-worker-design.md | 109 ++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-05-durable-job-lease-worker-design.md diff --git a/docs/superpowers/specs/2026-08-05-durable-job-lease-worker-design.md b/docs/superpowers/specs/2026-08-05-durable-job-lease-worker-design.md new file mode 100644 index 00000000..2ead4c57 --- /dev/null +++ b/docs/superpowers/specs/2026-08-05-durable-job-lease-worker-design.md @@ -0,0 +1,109 @@ +# Durable ETL Job Lease Worker Design + +## Status + +Accepted implementation design for issue #120. This is a bounded follow-on to the durable asynchronous intake merged in PR #119 and is stacked on PR #121 until that workflow-security prerequisite reaches `develop`. + +## Product outcome + +Accepted asynchronous ETL jobs must progress from `PENDING` to a terminal state without depending on the client connection or on one service replica. The worker must distribute work across replicas through PostgreSQL row locking, fence stale owners, bound retry attempts, atomically couple target effects with terminal success, clear retained payloads at terminal state, and expose only stable non-sensitive status metadata through the existing owner-scoped API. + +## Scope + +This slice adds: + +- a PostgreSQL-owned claim operation using deterministic ordering and `FOR UPDATE SKIP LOCKED`; +- process-lifetime `lease_owner_id` and per-claim `lease_claim_id` fencing; +- lease expiry and reclaim; +- bounded attempts with deterministic terminal failure codes; +- fixed-delay polling that is disabled by default; +- atomic ETL target writes plus conditional `SUCCEEDED` transition; +- retry and failure transitions that require the exact live lease; +- finite-cardinality execution metrics; +- migration, rollback, privacy, operations, and failure-recovery documentation. + +Cancellation, priorities, recurring schedules, manual replay, result-body persistence, and a dead-letter user interface remain out of scope. + +## Data model + +Flyway migration `V3__add_etl_job_lease_fencing.sql` adds the following descriptive `snake_case` columns to `etl_job_records`: + +- `lease_claim_id UUID` — unique token generated for every claim or reclaim; +- `lease_owner_id VARCHAR(128)` — stable non-sensitive identifier for one worker process; +- `lease_expires_at TIMESTAMPTZ` — database-time expiry boundary. + +A lifecycle constraint requires all three lease columns for `RUNNING` rows and requires all three to be null for every other state. A failure lifecycle constraint requires `failure_code` only for `FAILED` rows. The existing terminal-payload constraint remains authoritative. An eligibility index covers `job_status`, `lease_expires_at`, `created_at`, and `job_record_id`. + +## Claim protocol + +`EtlJobLeaseRepository.claimNext` runs in one transaction: + +1. Terminalize eligible rows whose `attempt_count` has reached the configured maximum. Clear `request_payload` and all lease columns and assign `etl_worker_attempts_exhausted`. +2. Select one `PENDING` row or one expired `RUNNING` row with `attempt_count < max_attempts`, ordered by `created_at, job_record_id`, using `FETCH FIRST 1 ROW ONLY FOR UPDATE SKIP LOCKED`. +3. Read `CURRENT_TIMESTAMP` from the database in the same statement and derive the next expiry from that database time. +4. Generate a new `lease_claim_id`, increment `attempt_count`, set `RUNNING`, set the owner and expiry, clear any prior failure code, and commit. + +The scheduler does not provide uniqueness. The database row lock and state predicate are the cross-replica authority. PostgreSQL documents `SKIP LOCKED` as suitable for avoiding contention among multiple consumers of a queue-like table, while warning that it is not a general-purpose consistent view; that limitation is appropriate here because each worker needs one exclusive claim rather than a complete snapshot. + +## Execution and fencing + +`EtlJobExecutionService.execute` starts a new transaction, calls the existing `EtlService.processData` through a separate Spring bean, then conditionally transitions the job to `SUCCEEDED` only when all of the following still match: + +- `job_record_id`; +- `job_status = 'RUNNING'`; +- exact `lease_claim_id`; +- exact `lease_owner_id`; +- `lease_expires_at > CURRENT_TIMESTAMP`. + +If the conditional update affects no row, `StaleEtlJobLeaseException` is thrown. The exception rolls back the same transaction, including all target writes, so an expired or superseded worker cannot commit target effects. + +## Failure policy + +The polling coordinator catches execution failures after the execution transaction rolls back and performs a separate exact-lease transition: + +- `TransientDataAccessException`: return to `PENDING` when attempts remain; otherwise terminal `FAILED` with `etl_target_unavailable`; +- `EtlRequestException`: terminal `FAILED` with the existing stable request `errorCode`; +- other `DataAccessException`: terminal `FAILED` with `etl_target_failure`; +- other `RuntimeException`: terminal `FAILED` with `etl_internal_error`; +- `StaleEtlJobLeaseException`: make no state change because another owner or expiry boundary is authoritative. + +Every retry or failure update repeats the exact-live-lease predicate. A zero-row update is treated as stale evidence, not as success. + +## Scheduling and activation + +Spring fixed-delay scheduling is used because the next delay is measured after completion of the previous invocation. `xtrmetl.etl.jobs.worker.enabled` defaults to `false`; operators must explicitly enable both intake and worker execution. Configurable values are bounded and validated: + +- `fixed-delay-milliseconds` > 0; +- `initial-delay-milliseconds` >= 0; +- `lease-duration-seconds` > 0; +- `max-attempts` between 1 and 100; +- `lease-owner-id` is 8–128 safe ASCII characters and defaults to a process-lifetime generated identifier. + +One polling invocation claims at most one job. Horizontal throughput is achieved by replicas and repeated fixed-delay invocations rather than unbounded in-process fan-out. + +## Observability and privacy + +The worker emits a duration timer and a finite outcome counter for `claimed`, `succeeded`, `retried`, `failed`, and `stale`. Metric tags never include payloads, principals, idempotency keys, job identifiers, SQL, lease identifiers, or exception messages. Logs follow the same rule. Database client instrumentation should retain the stable OpenTelemetry SQL semantic conventions and avoid opting raw query text into telemetry unless the deployment has separately assessed that exposure. + +## Testing strategy + +- Migration tests enforce descriptive names, lifecycle constraints, index shape, and rollback instructions. +- Repository integration tests use H2's supported `FOR UPDATE SKIP LOCKED` syntax to prove one live claim, deterministic ordering, expiry reclaim, attempt increment, and exhaustion terminalization. +- Execution integration tests prove target rows and `SUCCEEDED` commit together and prove a stale claim rolls target writes back. +- Coordinator tests cover every failure classification, retry bound, zero-work poll, metrics outcome, and stale transition. +- Property tests cover every validation boundary and generated owner identifier. +- Documentation and coverage policy tests require complete public Javadoc and zero missed instruction, line, method, and branch coverage for the durable-job package. + +## Rollback + +Before application rollback, stop all workers and disable intake. Allow active leases to expire, confirm no `RUNNING` rows remain, and decide whether pending payloads will be drained or explicitly failed. Roll back the application first. The three lease columns and eligibility index may be removed only after all rows are non-running and no deployed binary reads them. Flyway versioned migrations are not edited or deleted after publication; a forward compensating migration must perform any production schema reversal. + +## Standards and primary documentation + +Fielding, R., Nottingham, M., & Reschke, J. (2022). *HTTP semantics* (RFC 9110). Internet Engineering Task Force. https://www.rfc-editor.org/rfc/rfc9110.html + +OpenTelemetry Authors. (2026). *Semantic conventions for database calls and systems*. Cloud Native Computing Foundation. https://opentelemetry.io/docs/specs/semconv/db/ + +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: SELECT*. https://www.postgresql.org/docs/18/sql-select.html + +Spring Authors. (2026). *Task execution and scheduling*. Broadcom. https://docs.spring.io/spring-framework/reference/integration/scheduling.html From 4f170ecc8a4f40a6ac3f4fcca21e7c0d6810d8eb Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:30:17 +0900 Subject: [PATCH 02/92] docs: plan lease-fenced durable job worker --- .../2026-08-05-durable-job-lease-worker.md | 108 ++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-05-durable-job-lease-worker.md diff --git a/docs/superpowers/plans/2026-08-05-durable-job-lease-worker.md b/docs/superpowers/plans/2026-08-05-durable-job-lease-worker.md new file mode 100644 index 00000000..cde8d2b0 --- /dev/null +++ b/docs/superpowers/plans/2026-08-05-durable-job-lease-worker.md @@ -0,0 +1,108 @@ +# Durable ETL Job Lease Worker Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Execute accepted asynchronous ETL jobs safely across replicas with PostgreSQL claim locking, exact lease fencing, bounded retries, atomic success, terminal payload clearing, and operator-safe evidence. + +**Architecture:** A transaction-scoped repository owns claim and state transitions; a separate transactional execution service couples existing ETL target writes with an exact-live-lease success update; a fixed-delay coordinator classifies failures and performs retry or terminal transitions in a new transaction. PostgreSQL row state is the distribution and fencing authority, while scheduling only supplies repeated polling. + +**Tech Stack:** Java 25, Spring Boot, Spring JDBC transactions, Spring scheduling, PostgreSQL 18 SQL, Flyway, Micrometer, JUnit 5, Mockito, H2 compatibility tests, Maven/Jacoco. + +## Global Constraints + +- Preserve standalone operation and modular MSA compatibility with ContextualWisdomLab/.github, naruon, and other CWL services. +- Database objects contain at least two descriptive words and use `snake_case`. +- Worker activation is fail-closed and disabled by default. +- Every public production type and method has beginner-readable Javadoc. +- Added durable-job production code must have zero missed instruction, line, method, or branch coverage. +- Payloads, principals, idempotency keys, job identifiers, lease identifiers, SQL, and exception messages never enter metrics or logs. +- A stale or expired lease cannot commit target effects or state transitions. +- Versioned Flyway migrations are immutable after publication; rollback uses a forward compensating migration. + +--- + +### Task 1: Lock the schema and configuration contracts + +**Files:** +- Create: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseMigrationTest.java` +- Create: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerPropertiesTest.java` +- Create: `etl-service/src/main/resources/db/migration/V3__add_etl_job_lease_fencing.sql` +- Create: `etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorkerProperties.java` +- Modify: `etl-service/src/main/java/com/xtrmetl/etl/EtlApplication.java` +- Modify: `etl-service/src/main/resources/application.yml` + +**Interfaces:** +- Produces: `EtlJobWorkerProperties` with `enabled`, `fixedDelayMilliseconds`, `initialDelayMilliseconds`, `leaseDurationSeconds`, `maxAttempts`, and `leaseOwnerId`. + +- [ ] Write migration and property tests first. Require the three lease columns, lifecycle constraints, claim index, fail-closed defaults, safe owner profile, and all numeric boundaries. +- [ ] Run `./mvnw -B -pl etl-service -Dtest=EtlJobLeaseMigrationTest,EtlJobWorkerPropertiesTest test` and record the expected missing-file/type failure. +- [ ] Add the migration, properties, application registration, and environment-backed defaults. +- [ ] Re-run the focused tests and commit. + +### Task 2: Add exclusive claim and exact transition persistence + +**Files:** +- Create: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryIntegrationTest.java` +- Create: `etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLease.java` +- Create: `etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java` +- Create: `etl-service/src/main/java/com/xtrmetl/etl/job/StaleEtlJobLeaseException.java` + +**Interfaces:** +- Produces: `Optional claimNext(String leaseOwnerId, Duration leaseDuration, int maxAttempts)`. +- Produces: `markSucceeded`, `releaseForRetry`, and `markFailed`, each returning only after an exact, unexpired lease transition or throwing `StaleEtlJobLeaseException`. + +- [ ] Write H2 integration tests for deterministic order, simultaneous single claim, expired reclaim, exhausted terminalization, success, retry, failure, and stale update refusal. +- [ ] Run the focused test and record the missing-type failure. +- [ ] Implement the two-statement lock-and-update claim transaction using `FOR UPDATE SKIP LOCKED` and database `CURRENT_TIMESTAMP`. +- [ ] Implement exact-live-lease transition predicates and stable failure validation. +- [ ] Re-run the focused test and commit. + +### Task 3: Couple ETL target effects to terminal success + +**Files:** +- Create: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobExecutionServiceIntegrationTest.java` +- Create: `etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobExecutionService.java` + +**Interfaces:** +- Consumes: `EtlJobLease`, `EtlService.processData`, `EtlJobLeaseRepository.markSucceeded`. +- Produces: `void execute(EtlJobLease lease)` in one Spring transaction. + +- [ ] Write integration tests proving target rows and `SUCCEEDED` commit together. +- [ ] Add a stale-lease test that changes the claim before execution and asserts both the exception and zero committed target rows. +- [ ] Run the focused test and record the missing-type failure. +- [ ] Implement the minimal transactional service and re-run the tests. +- [ ] Commit. + +### Task 4: Add bounded fixed-delay coordination and evidence + +**Files:** +- Create: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java` +- Create: `etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorker.java` + +**Interfaces:** +- Consumes: repository claim/transitions, execution service, worker properties, `MeterRegistry`. +- Produces: one `pollOnce()` invocation that claims at most one job and records finite outcomes. + +- [ ] Write tests for no work, success, transient retry, exhausted transient failure, deterministic request failure, non-transient target failure, unexpected failure, and stale evidence. +- [ ] Run the focused test and record the missing-type failure. +- [ ] Implement the conditional worker bean, fixed-delay method, failure classification, retry bound, duration timer, and finite-cardinality outcome counter. +- [ ] Re-run the tests and commit. + +### Task 5: Complete operations, privacy, compatibility, and release evidence + +**Files:** +- Modify: `docs/etl/durable-job-intake.md` +- Create: `docs/operations/durable-job-worker.md` +- Modify: `CHANGELOG.md` +- Modify: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobMigrationDocumentationTest.java` +- Modify: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCoveragePolicyTest.java` + +**Interfaces:** +- Produces: authoritative activation, SLO, metrics, failure-code, recovery, retention, and rollback guidance. + +- [ ] Add documentation-first tests requiring activation pairs, privacy boundaries, exact failure codes, rollback ordering, and standards references. +- [ ] Update the authoritative docs and changelog. +- [ ] Run `./mvnw -B -pl etl-service test`. +- [ ] Run `./mvnw -B test` across the full reactor. +- [ ] Inspect Jacoco for zero missed durable-job instructions, lines, methods, and branches. +- [ ] Open a stacked draft PR against `ci/hourly-opencode-nvidia-nim`, inspect every review and exact-head check, and mark ready only after all gates pass. From e4960fe9cc67d62bff0d3d96ff0eea6e0901dd0e Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:30:37 +0900 Subject: [PATCH 03/92] test: specify durable job lease schema --- .../etl/job/EtlJobLeaseMigrationTest.java | 87 +++++++++++++++++++ 1 file changed, 87 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseMigrationTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseMigrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseMigrationTest.java new file mode 100644 index 00000000..8d73c761 --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseMigrationTest.java @@ -0,0 +1,87 @@ +package com.xtrmetl.etl.job; + +import org.junit.jupiter.api.Test; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; + +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Specifies the additive Flyway contract for exact durable-job lease fencing. + */ +class EtlJobLeaseMigrationTest { + + @Test + void addsDescriptiveLeaseColumnsAndEligibilityIndex() throws IOException { + String migration = readMigration(); + + assertTrue(migration.contains("ADD COLUMN lease_claim_id UUID")); + assertTrue(migration.contains("ADD COLUMN lease_owner_id VARCHAR(128)")); + assertTrue(migration.contains("ADD COLUMN lease_expires_at TIMESTAMPTZ")); + assertTrue(migration.contains("CREATE INDEX etl_job_claim_eligibility_index")); + assertTrue(migration.contains( + "(job_status, lease_expires_at, created_at, job_record_id)" + )); + assertFalse(migration.contains(" ADD COLUMN owner ")); + assertFalse(migration.contains(" ADD COLUMN lease ")); + } + + @Test + void requiresLeaseFieldsOnlyForRunningRows() throws IOException { + String migration = normalize(readMigration()); + + assertTrue(migration.contains("CONSTRAINT etl_job_lease_lifecycle_check")); + assertTrue(migration.contains( + "job_status = 'RUNNING' AND lease_claim_id IS NOT NULL AND lease_owner_id IS NOT NULL AND lease_expires_at IS NOT NULL" + )); + assertTrue(migration.contains( + "job_status <> 'RUNNING' AND lease_claim_id IS NULL AND lease_owner_id IS NULL AND lease_expires_at IS NULL" + )); + } + + @Test + void requiresFailureCodesOnlyForFailedRows() throws IOException { + String migration = normalize(readMigration()); + + assertTrue(migration.contains("CONSTRAINT etl_job_failure_lifecycle_check")); + assertTrue(migration.contains("job_status = 'FAILED' AND failure_code IS NOT NULL")); + assertTrue(migration.contains("job_status <> 'FAILED' AND failure_code IS NULL")); + } + + private static String readMigration() throws IOException { + return Files.readString( + projectRoot().resolve( + "etl-service/src/main/resources/db/migration/" + + "V3__add_etl_job_lease_fencing.sql" + ), + StandardCharsets.UTF_8 + ); + } + + private static String normalize(String value) { + return value.replaceAll("\\s+", " ").trim(); + } + + private static Path projectRoot() { + Path current = Paths.get(System.getProperty("user.dir")).toAbsolutePath(); + Path lastPomParent = null; + while (current != null) { + if (Files.exists(current.resolve(".git"))) { + return current; + } + if (Files.exists(current.resolve("pom.xml"))) { + lastPomParent = current; + } + current = current.getParent(); + } + if (lastPomParent != null) { + return lastPomParent; + } + throw new IllegalStateException("Could not find project root"); + } +} From 8f526301d1e6555a87a2c3c5601fb1be72c68a77 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:30:57 +0900 Subject: [PATCH 04/92] test: specify worker configuration bounds --- .../etl/job/EtlJobWorkerPropertiesTest.java | 94 +++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerPropertiesTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerPropertiesTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerPropertiesTest.java new file mode 100644 index 00000000..603eef51 --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerPropertiesTest.java @@ -0,0 +1,94 @@ +package com.xtrmetl.etl.job; + +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Specifies fail-closed activation and bounded durable-job worker configuration. + */ +class EtlJobWorkerPropertiesTest { + + @Test + void defaultsToDisabledBoundedPollingWithGeneratedSafeOwner() { + EtlJobWorkerProperties properties = new EtlJobWorkerProperties(); + + assertFalse(properties.isEnabled()); + assertEquals(5_000L, properties.getFixedDelayMilliseconds()); + assertEquals(5_000L, properties.getInitialDelayMilliseconds()); + assertEquals(300L, properties.getLeaseDurationSeconds()); + assertEquals(3, properties.getMaxAttempts()); + assertTrue(properties.getLeaseOwnerId().matches("[A-Za-z0-9._:-]{8,128}")); + + EtlJobWorkerProperties another = new EtlJobWorkerProperties(); + assertNotEquals(properties.getLeaseOwnerId(), another.getLeaseOwnerId()); + } + + @Test + void acceptsEverySupportedBoundary() { + EtlJobWorkerProperties properties = new EtlJobWorkerProperties(); + + properties.setEnabled(true); + properties.setFixedDelayMilliseconds(1L); + properties.setInitialDelayMilliseconds(0L); + properties.setLeaseDurationSeconds(1L); + properties.setMaxAttempts(1); + properties.setLeaseOwnerId("worker-01"); + + assertTrue(properties.isEnabled()); + assertEquals(1L, properties.getFixedDelayMilliseconds()); + assertEquals(0L, properties.getInitialDelayMilliseconds()); + assertEquals(1L, properties.getLeaseDurationSeconds()); + assertEquals(1, properties.getMaxAttempts()); + assertEquals("worker-01", properties.getLeaseOwnerId()); + + properties.setMaxAttempts(100); + properties.setLeaseOwnerId("w".repeat(128)); + assertEquals(100, properties.getMaxAttempts()); + assertEquals(128, properties.getLeaseOwnerId().length()); + } + + @Test + void rejectsUnsafeNumericConfiguration() { + EtlJobWorkerProperties properties = new EtlJobWorkerProperties(); + + assertThrows( + IllegalArgumentException.class, + () -> properties.setFixedDelayMilliseconds(0L) + ); + assertThrows( + IllegalArgumentException.class, + () -> properties.setInitialDelayMilliseconds(-1L) + ); + assertThrows( + IllegalArgumentException.class, + () -> properties.setLeaseDurationSeconds(0L) + ); + assertThrows(IllegalArgumentException.class, () -> properties.setMaxAttempts(0)); + assertThrows(IllegalArgumentException.class, () -> properties.setMaxAttempts(101)); + } + + @Test + void rejectsMissingShortLongOrUnsafeOwnerIdentifiers() { + EtlJobWorkerProperties properties = new EtlJobWorkerProperties(); + + assertThrows(NullPointerException.class, () -> properties.setLeaseOwnerId(null)); + assertThrows(IllegalArgumentException.class, () -> properties.setLeaseOwnerId("short")); + assertThrows( + IllegalArgumentException.class, + () -> properties.setLeaseOwnerId("w".repeat(129)) + ); + assertThrows( + IllegalArgumentException.class, + () -> properties.setLeaseOwnerId("worker identifier") + ); + assertThrows( + IllegalArgumentException.class, + () -> properties.setLeaseOwnerId("worker/identifier") + ); + } +} From 6155eb807743ac0f1cf8fd2e72c56dd156d49aae Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:35:31 +0900 Subject: [PATCH 05/92] feat(etl): add durable job lease schema --- .../V3__add_etl_job_lease_fencing.sql | 51 +++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 etl-service/src/main/resources/db/migration/V3__add_etl_job_lease_fencing.sql diff --git a/etl-service/src/main/resources/db/migration/V3__add_etl_job_lease_fencing.sql b/etl-service/src/main/resources/db/migration/V3__add_etl_job_lease_fencing.sql new file mode 100644 index 00000000..d54bf6df --- /dev/null +++ b/etl-service/src/main/resources/db/migration/V3__add_etl_job_lease_fencing.sql @@ -0,0 +1,51 @@ +ALTER TABLE etl_job_records + ADD COLUMN lease_claim_id UUID, + ADD COLUMN lease_owner_id VARCHAR(128), + ADD COLUMN lease_expires_at TIMESTAMPTZ; + +-- Repair legacy rows before enforcing the stronger failure lifecycle invariant. +UPDATE etl_job_records +SET failure_code = 'etl_legacy_failure' +WHERE job_status = 'FAILED' + AND failure_code IS NULL; + +UPDATE etl_job_records +SET failure_code = NULL +WHERE job_status <> 'FAILED' + AND failure_code IS NOT NULL; + +ALTER TABLE etl_job_records + ADD CONSTRAINT etl_job_lease_lifecycle_check CHECK ( + ( + job_status = 'RUNNING' + AND lease_claim_id IS NOT NULL + AND lease_owner_id IS NOT NULL + AND lease_expires_at IS NOT NULL + ) + OR + ( + job_status <> 'RUNNING' + AND lease_claim_id IS NULL + AND lease_owner_id IS NULL + AND lease_expires_at IS NULL + ) + ), + ADD CONSTRAINT etl_job_failure_lifecycle_check CHECK ( + ( + job_status = 'FAILED' + AND failure_code IS NOT NULL + ) + OR + ( + job_status <> 'FAILED' + AND failure_code IS NULL + ) + ); + +CREATE INDEX etl_job_claim_eligibility_index + ON etl_job_records ( + job_status, + lease_expires_at, + created_at, + job_record_id + ); From 0325b863140867517d1bd91465c090b3b667171b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:35:54 +0900 Subject: [PATCH 06/92] feat(etl): define fail-closed worker properties --- .../etl/job/EtlJobWorkerProperties.java | 172 ++++++++++++++++++ 1 file changed, 172 insertions(+) create mode 100644 etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorkerProperties.java diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorkerProperties.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorkerProperties.java new file mode 100644 index 00000000..64ff386b --- /dev/null +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorkerProperties.java @@ -0,0 +1,172 @@ +package com.xtrmetl.etl.job; + +import org.springframework.boot.context.properties.ConfigurationProperties; + +import java.util.Objects; +import java.util.UUID; +import java.util.regex.Pattern; + +/** + * Holds bounded, fail-closed configuration for durable ETL job execution. + * + *

The worker is disabled unless an operator explicitly enables it. A process-lifetime lease + * owner identifier is generated when no external value is supplied. The identifier is deliberately + * restricted to a short safe ASCII profile because it is persisted as operational metadata and + * must never become a free-form log or database injection surface.

+ */ +@ConfigurationProperties(prefix = "xtrmetl.etl.jobs.worker") +public class EtlJobWorkerProperties { + + private static final Pattern SAFE_LEASE_OWNER_PATTERN = Pattern.compile( + "[A-Za-z0-9._:-]{8,128}" + ); + + private boolean enabled; + private long fixedDelayMilliseconds = 5_000L; + private long initialDelayMilliseconds = 5_000L; + private long leaseDurationSeconds = 300L; + private int maxAttempts = 3; + private String leaseOwnerId = "worker-" + UUID.randomUUID(); + + /** + * Creates disabled worker configuration with bounded production-safe defaults. + */ + public EtlJobWorkerProperties() { + // Spring Boot binds through the public setters while preserving generated defaults. + } + + /** + * Reports whether scheduled durable-job execution is explicitly enabled. + * + * @return {@code true} only when an operator enabled the worker + */ + public boolean isEnabled() { + return enabled; + } + + /** + * Enables or disables scheduled durable-job execution. + * + * @param enabled whether the worker should run + */ + public void setEnabled(boolean enabled) { + this.enabled = enabled; + } + + /** + * Returns the delay measured after one polling invocation completes. + * + * @return positive fixed delay in milliseconds + */ + public long getFixedDelayMilliseconds() { + return fixedDelayMilliseconds; + } + + /** + * Sets the delay measured after one polling invocation completes. + * + * @param fixedDelayMilliseconds positive fixed delay in milliseconds + * @throws IllegalArgumentException when the delay is zero or negative + */ + public void setFixedDelayMilliseconds(long fixedDelayMilliseconds) { + if (fixedDelayMilliseconds <= 0L) { + throw new IllegalArgumentException("fixedDelayMilliseconds must be positive"); + } + this.fixedDelayMilliseconds = fixedDelayMilliseconds; + } + + /** + * Returns the delay before the first polling invocation after application startup. + * + * @return non-negative initial delay in milliseconds + */ + public long getInitialDelayMilliseconds() { + return initialDelayMilliseconds; + } + + /** + * Sets the delay before the first polling invocation after application startup. + * + * @param initialDelayMilliseconds non-negative initial delay in milliseconds + * @throws IllegalArgumentException when the delay is negative + */ + public void setInitialDelayMilliseconds(long initialDelayMilliseconds) { + if (initialDelayMilliseconds < 0L) { + throw new IllegalArgumentException("initialDelayMilliseconds must not be negative"); + } + this.initialDelayMilliseconds = initialDelayMilliseconds; + } + + /** + * Returns how long one database claim remains valid without renewal. + * + * @return positive lease duration in seconds + */ + public long getLeaseDurationSeconds() { + return leaseDurationSeconds; + } + + /** + * Sets how long one database claim remains valid without renewal. + * + * @param leaseDurationSeconds positive lease duration in seconds + * @throws IllegalArgumentException when the duration is zero or negative + */ + public void setLeaseDurationSeconds(long leaseDurationSeconds) { + if (leaseDurationSeconds <= 0L) { + throw new IllegalArgumentException("leaseDurationSeconds must be positive"); + } + this.leaseDurationSeconds = leaseDurationSeconds; + } + + /** + * Returns the maximum number of claims permitted before terminal failure. + * + * @return maximum attempt count from 1 through 100 + */ + public int getMaxAttempts() { + return maxAttempts; + } + + /** + * Sets the maximum number of claims permitted before terminal failure. + * + * @param maxAttempts maximum attempt count from 1 through 100 + * @throws IllegalArgumentException when the value is outside the supported range + */ + public void setMaxAttempts(int maxAttempts) { + if (maxAttempts < 1 || maxAttempts > 100) { + throw new IllegalArgumentException("maxAttempts must be between 1 and 100"); + } + this.maxAttempts = maxAttempts; + } + + /** + * Returns the non-sensitive process identifier persisted on active leases. + * + * @return safe process-lifetime lease owner identifier + */ + public String getLeaseOwnerId() { + return leaseOwnerId; + } + + /** + * Sets the non-sensitive process identifier persisted on active leases. + * + * @param leaseOwnerId 8-to-128-character safe ASCII process identifier + * @throws NullPointerException when the identifier is {@code null} + * @throws IllegalArgumentException when the identifier is too short, too long, or unsafe + */ + public void setLeaseOwnerId(String leaseOwnerId) { + String requiredOwnerId = Objects.requireNonNull( + leaseOwnerId, + "leaseOwnerId must not be null" + ); + if (!SAFE_LEASE_OWNER_PATTERN.matcher(requiredOwnerId).matches()) { + throw new IllegalArgumentException( + "leaseOwnerId must match [A-Za-z0-9._:-]{8,128}" + ); + } + this.leaseOwnerId = requiredOwnerId; + } +} From a12933356e943af57bbd59aeebd97328d919cf34 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:36:05 +0900 Subject: [PATCH 07/92] feat(etl): register worker configuration --- .../src/main/java/com/xtrmetl/etl/EtlApplication.java | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/etl-service/src/main/java/com/xtrmetl/etl/EtlApplication.java b/etl-service/src/main/java/com/xtrmetl/etl/EtlApplication.java index c6796c2b..17213106 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/EtlApplication.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/EtlApplication.java @@ -1,6 +1,7 @@ package com.xtrmetl.etl; import com.xtrmetl.etl.connector.ConnectorProperties; +import com.xtrmetl.etl.job.EtlJobWorkerProperties; import com.xtrmetl.etl.service.EtlBatchProperties; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; @@ -16,7 +17,11 @@ @EnableDiscoveryClient @EnableAspectJAutoProxy(proxyTargetClass = true) @EnableRetry -@EnableConfigurationProperties({ConnectorProperties.class, EtlBatchProperties.class}) +@EnableConfigurationProperties({ + ConnectorProperties.class, + EtlBatchProperties.class, + EtlJobWorkerProperties.class +}) public class EtlApplication { /** From aeeeb380a1900e63738a2f2e4d5508d16914b026 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:36:22 +0900 Subject: [PATCH 08/92] feat(etl): add fail-closed worker defaults --- etl-service/src/main/resources/application.yml | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/etl-service/src/main/resources/application.yml b/etl-service/src/main/resources/application.yml index dbb4ebcb..642675d3 100644 --- a/etl-service/src/main/resources/application.yml +++ b/etl-service/src/main/resources/application.yml @@ -27,9 +27,15 @@ xtrmetl: max-payload-bytes: ${ETL_MAX_PAYLOAD_BYTES:1048576} max-batch-records: ${ETL_MAX_BATCH_RECORDS:1000} jobs: - # Intake persists validated payloads but does not execute them in this bounded slice. - # Keep disabled until an operator explicitly accepts the temporary retention boundary. + # Intake persists validated payloads for durable worker execution. intake-enabled: ${ETL_JOB_INTAKE_ENABLED:false} + worker: + # Execution remains fail-closed until an operator enables the worker explicitly. + enabled: ${ETL_JOB_WORKER_ENABLED:false} + fixed-delay-milliseconds: ${ETL_JOB_WORKER_FIXED_DELAY_MILLISECONDS:5000} + initial-delay-milliseconds: ${ETL_JOB_WORKER_INITIAL_DELAY_MILLISECONDS:5000} + lease-duration-seconds: ${ETL_JOB_WORKER_LEASE_DURATION_SECONDS:300} + max-attempts: ${ETL_JOB_WORKER_MAX_ATTEMPTS:3} # Warehouse/BI targets: SPI + config binding + validation + catalog; writes remain SCAFFOLD. connectors: databricks: From faab9debd45724fe59b080de98e8e5ffe1e19595 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:36:49 +0900 Subject: [PATCH 09/92] test(etl): align claim index contract --- .../db/migration/V3__add_etl_job_lease_fencing.sql | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/etl-service/src/main/resources/db/migration/V3__add_etl_job_lease_fencing.sql b/etl-service/src/main/resources/db/migration/V3__add_etl_job_lease_fencing.sql index d54bf6df..0b5a5555 100644 --- a/etl-service/src/main/resources/db/migration/V3__add_etl_job_lease_fencing.sql +++ b/etl-service/src/main/resources/db/migration/V3__add_etl_job_lease_fencing.sql @@ -43,9 +43,4 @@ ALTER TABLE etl_job_records ); CREATE INDEX etl_job_claim_eligibility_index - ON etl_job_records ( - job_status, - lease_expires_at, - created_at, - job_record_id - ); + ON etl_job_records (job_status, lease_expires_at, created_at, job_record_id); From f1eac6e2263408015ecdd1f75a0adedc77359167 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:38:37 +0900 Subject: [PATCH 10/92] test(etl): specify lease-fenced job persistence --- .../EtlJobLeaseRepositoryIntegrationTest.java | 430 ++++++++++++++++++ 1 file changed, 430 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryIntegrationTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryIntegrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryIntegrationTest.java new file mode 100644 index 00000000..773c5936 --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryIntegrationTest.java @@ -0,0 +1,430 @@ +package com.xtrmetl.etl.job; + +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.jdbc.datasource.DataSourceTransactionManager; +import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseBuilder; +import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseType; +import org.springframework.test.context.junit.jupiter.SpringJUnitConfig; +import org.springframework.transaction.PlatformTransactionManager; +import org.springframework.transaction.annotation.EnableTransactionManagement; + +import javax.sql.DataSource; +import java.time.Duration; +import java.time.Instant; +import java.util.List; +import java.util.Optional; +import java.util.UUID; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Proves exclusive durable-job claims and exact-live-lease state transitions against SQL. + */ +@SpringJUnitConfig(EtlJobLeaseRepositoryIntegrationTest.TestConfiguration.class) +class EtlJobLeaseRepositoryIntegrationTest { + + private static final Duration LEASE_DURATION = Duration.ofMinutes(5); + private static final String OWNER_ALPHA = "worker-alpha"; + private static final String OWNER_BETA = "worker-beta"; + private static final String PAYLOAD = "[{\"id\":\"record_alpha\"}]"; + + private final EtlJobLeaseRepository repository; + private final JdbcTemplate jdbcTemplate; + private ExecutorService executorService; + + @Autowired + EtlJobLeaseRepositoryIntegrationTest( + EtlJobLeaseRepository repository, + JdbcTemplate jdbcTemplate + ) { + this.repository = repository; + this.jdbcTemplate = jdbcTemplate; + } + + @BeforeEach + void createJobTable() { + executorService = Executors.newFixedThreadPool(2); + jdbcTemplate.execute("DROP TABLE IF EXISTS etl_job_records"); + jdbcTemplate.execute(""" + CREATE TABLE etl_job_records ( + job_record_id UUID PRIMARY KEY, + principal_scope_hash CHAR(64) NOT NULL, + submission_key_hash CHAR(64) NOT NULL, + request_digest CHAR(64) NOT NULL, + request_payload CLOB, + job_status VARCHAR(32) NOT NULL, + attempt_count INTEGER NOT NULL DEFAULT 0, + failure_code VARCHAR(128), + lease_claim_id UUID, + lease_owner_id VARCHAR(128), + lease_expires_at TIMESTAMP WITH TIME ZONE, + created_at TIMESTAMP WITH TIME ZONE NOT NULL, + updated_at TIMESTAMP WITH TIME ZONE NOT NULL + ) + """); + } + + @AfterEach + void closeExecutor() { + executorService.close(); + } + + @Test + void claimsTheOldestEligibleJobAndIncrementsItsAttempt() { + UUID newerJobId = insertPending(Instant.parse("2026-08-05T00:01:00Z"), 0); + UUID olderJobId = insertPending(Instant.parse("2026-08-05T00:00:00Z"), 1); + + EtlJobLease lease = repository.claimNext(OWNER_ALPHA, LEASE_DURATION, 3).orElseThrow(); + + assertEquals(olderJobId, lease.jobRecordId()); + assertEquals(OWNER_ALPHA, lease.leaseOwnerId()); + assertEquals(PAYLOAD, lease.requestPayload()); + assertEquals(2, lease.attemptCount()); + assertNotNull(lease.leaseClaimId()); + assertTrue(lease.leaseExpiresAt().isAfter(Instant.now())); + assertEquals("RUNNING", textColumn(olderJobId, "job_status")); + assertEquals(2, integerColumn(olderJobId, "attempt_count")); + assertEquals("PENDING", textColumn(newerJobId, "job_status")); + } + + @Test + void skipsLiveClaimsAndReclaimsExpiredClaimsWithFreshFencing() { + UUID liveJobId = insertRunning( + Instant.parse("2026-08-05T00:00:00Z"), + 1, + OWNER_ALPHA, + UUID.randomUUID(), + Instant.now().plusSeconds(300) + ); + UUID priorClaimId = UUID.randomUUID(); + UUID expiredJobId = insertRunning( + Instant.parse("2026-08-05T00:01:00Z"), + 1, + OWNER_ALPHA, + priorClaimId, + Instant.now().minusSeconds(60) + ); + + EtlJobLease reclaimed = repository.claimNext(OWNER_BETA, LEASE_DURATION, 3).orElseThrow(); + + assertEquals(expiredJobId, reclaimed.jobRecordId()); + assertEquals(OWNER_BETA, reclaimed.leaseOwnerId()); + assertNotEquals(priorClaimId, reclaimed.leaseClaimId()); + assertEquals(2, reclaimed.attemptCount()); + assertEquals("RUNNING", textColumn(liveJobId, "job_status")); + assertEquals(OWNER_ALPHA, textColumn(liveJobId, "lease_owner_id")); + } + + @Test + void returnsEmptyWhenEveryClaimIsLive() { + insertRunning( + Instant.now(), + 1, + OWNER_ALPHA, + UUID.randomUUID(), + Instant.now().plusSeconds(300) + ); + + Optional lease = repository.claimNext(OWNER_BETA, LEASE_DURATION, 3); + + assertTrue(lease.isEmpty()); + } + + @Test + void onlyOneConcurrentWorkerCanClaimOnePendingJob() throws Exception { + UUID jobRecordId = insertPending(Instant.now(), 0); + CountDownLatch start = new CountDownLatch(1); + + Future> alpha = executorService.submit(() -> { + start.await(); + return repository.claimNext(OWNER_ALPHA, LEASE_DURATION, 3); + }); + Future> beta = executorService.submit(() -> { + start.await(); + return repository.claimNext(OWNER_BETA, LEASE_DURATION, 3); + }); + start.countDown(); + + List claims = List.of(alpha.get(), beta.get()).stream() + .flatMap(Optional::stream) + .toList(); + + assertEquals(1, claims.size()); + assertEquals(jobRecordId, claims.getFirst().jobRecordId()); + assertEquals(1, integerColumn(jobRecordId, "attempt_count")); + } + + @Test + void terminalizesExhaustedEligibleRowsBeforeLookingForWork() { + UUID pendingJobId = insertPending(Instant.now(), 3); + UUID expiredJobId = insertRunning( + Instant.now().plusSeconds(1), + 3, + OWNER_ALPHA, + UUID.randomUUID(), + Instant.now().minusSeconds(60) + ); + + Optional lease = repository.claimNext(OWNER_BETA, LEASE_DURATION, 3); + + assertTrue(lease.isEmpty()); + assertTerminalExhaustion(pendingJobId); + assertTerminalExhaustion(expiredJobId); + } + + @Test + void exactLiveLeaseCanSucceedRetryOrFailAndClearsTheRightFields() { + UUID successJobId = insertPending(Instant.parse("2026-08-05T00:00:00Z"), 0); + EtlJobLease successLease = repository.claimNext(OWNER_ALPHA, LEASE_DURATION, 3) + .orElseThrow(); + repository.markSucceeded(successLease); + assertEquals(successJobId, successLease.jobRecordId()); + assertEquals("SUCCEEDED", textColumn(successJobId, "job_status")); + assertNull(textColumn(successJobId, "request_payload")); + assertNull(textColumn(successJobId, "failure_code")); + assertNull(textColumn(successJobId, "lease_owner_id")); + + UUID retryJobId = insertPending(Instant.parse("2026-08-05T00:01:00Z"), 0); + EtlJobLease retryLease = repository.claimNext(OWNER_ALPHA, LEASE_DURATION, 3) + .orElseThrow(); + repository.releaseForRetry(retryLease, 3); + assertEquals(retryJobId, retryLease.jobRecordId()); + assertEquals("PENDING", textColumn(retryJobId, "job_status")); + assertEquals(PAYLOAD, textColumn(retryJobId, "request_payload")); + assertNull(textColumn(retryJobId, "failure_code")); + assertNull(textColumn(retryJobId, "lease_owner_id")); + + UUID failedJobId = insertPending(Instant.parse("2026-08-05T00:02:00Z"), 2); + EtlJobLease failedLease = repository.claimNext(OWNER_ALPHA, LEASE_DURATION, 3) + .orElseThrow(); + repository.markFailed(failedLease, "etl_target_failure"); + assertEquals(failedJobId, failedLease.jobRecordId()); + assertEquals("FAILED", textColumn(failedJobId, "job_status")); + assertNull(textColumn(failedJobId, "request_payload")); + assertEquals("etl_target_failure", textColumn(failedJobId, "failure_code")); + assertNull(textColumn(failedJobId, "lease_owner_id")); + } + + @Test + void rejectsExpiredSupersededOrExhaustedTransitions() { + UUID jobRecordId = insertPending(Instant.now(), 0); + EtlJobLease lease = repository.claimNext(OWNER_ALPHA, LEASE_DURATION, 1).orElseThrow(); + jdbcTemplate.update( + "UPDATE etl_job_records SET lease_expires_at = ? WHERE job_record_id = ?", + Instant.now().minusSeconds(1), + jobRecordId + ); + + assertThrows(StaleEtlJobLeaseException.class, () -> repository.markSucceeded(lease)); + assertThrows( + StaleEtlJobLeaseException.class, + () -> repository.releaseForRetry(lease, 1) + ); + assertThrows( + StaleEtlJobLeaseException.class, + () -> repository.markFailed(lease, "etl_target_failure") + ); + + jdbcTemplate.update( + """ + UPDATE etl_job_records + SET lease_claim_id = ?, lease_expires_at = ? + WHERE job_record_id = ? + """, + UUID.randomUUID(), + Instant.now().plusSeconds(300), + jobRecordId + ); + assertThrows(StaleEtlJobLeaseException.class, () -> repository.markSucceeded(lease)); + } + + @Test + void rejectsInvalidPublicArgumentsBeforeSqlExecution() { + assertThrows( + NullPointerException.class, + () -> repository.claimNext(null, LEASE_DURATION, 3) + ); + assertThrows( + NullPointerException.class, + () -> repository.claimNext(OWNER_ALPHA, null, 3) + ); + assertThrows( + IllegalArgumentException.class, + () -> repository.claimNext("short", LEASE_DURATION, 3) + ); + assertThrows( + IllegalArgumentException.class, + () -> repository.claimNext(OWNER_ALPHA, Duration.ZERO, 3) + ); + assertThrows( + IllegalArgumentException.class, + () -> repository.claimNext(OWNER_ALPHA, LEASE_DURATION, 0) + ); + assertThrows( + NullPointerException.class, + () -> repository.markSucceeded(null) + ); + assertThrows( + NullPointerException.class, + () -> repository.releaseForRetry(null, 3) + ); + assertThrows( + IllegalArgumentException.class, + () -> repository.releaseForRetry(sampleLease(), 0) + ); + assertThrows( + NullPointerException.class, + () -> repository.markFailed(null, "etl_target_failure") + ); + assertThrows( + NullPointerException.class, + () -> repository.markFailed(sampleLease(), null) + ); + assertThrows( + IllegalArgumentException.class, + () -> repository.markFailed(sampleLease(), "UNSAFE FAILURE") + ); + } + + private UUID insertPending(Instant createdAt, int attemptCount) { + UUID jobRecordId = UUID.randomUUID(); + jdbcTemplate.update( + """ + INSERT INTO etl_job_records ( + job_record_id, principal_scope_hash, submission_key_hash, + request_digest, request_payload, job_status, attempt_count, + created_at, updated_at + ) VALUES (?, ?, ?, ?, ?, 'PENDING', ?, ?, ?) + """, + jobRecordId, + "a".repeat(64), + UUID.randomUUID().toString().replace("-", "").repeat(2), + "b".repeat(64), + PAYLOAD, + attemptCount, + createdAt, + createdAt + ); + return jobRecordId; + } + + private UUID insertRunning( + Instant createdAt, + int attemptCount, + String ownerId, + UUID claimId, + Instant expiresAt + ) { + UUID jobRecordId = UUID.randomUUID(); + jdbcTemplate.update( + """ + INSERT INTO etl_job_records ( + job_record_id, principal_scope_hash, submission_key_hash, + request_digest, request_payload, job_status, attempt_count, + lease_claim_id, lease_owner_id, lease_expires_at, + created_at, updated_at + ) VALUES (?, ?, ?, ?, ?, 'RUNNING', ?, ?, ?, ?, ?, ?) + """, + jobRecordId, + "c".repeat(64), + UUID.randomUUID().toString().replace("-", "").repeat(2), + "d".repeat(64), + PAYLOAD, + attemptCount, + claimId, + ownerId, + expiresAt, + createdAt, + createdAt + ); + return jobRecordId; + } + + private void assertTerminalExhaustion(UUID jobRecordId) { + assertEquals("FAILED", textColumn(jobRecordId, "job_status")); + assertEquals( + "etl_worker_attempts_exhausted", + textColumn(jobRecordId, "failure_code") + ); + assertNull(textColumn(jobRecordId, "request_payload")); + assertNull(textColumn(jobRecordId, "lease_owner_id")); + } + + private String textColumn(UUID jobRecordId, String columnName) { + return jdbcTemplate.queryForObject( + "SELECT " + columnName + " FROM etl_job_records WHERE job_record_id = ?", + String.class, + jobRecordId + ); + } + + private int integerColumn(UUID jobRecordId, String columnName) { + Integer value = jdbcTemplate.queryForObject( + "SELECT " + columnName + " FROM etl_job_records WHERE job_record_id = ?", + Integer.class, + jobRecordId + ); + return value == null ? -1 : value; + } + + private static EtlJobLease sampleLease() { + return new EtlJobLease( + UUID.randomUUID(), + UUID.randomUUID(), + OWNER_ALPHA, + PAYLOAD, + 1, + Instant.now().plusSeconds(300) + ); + } + + /** + * Minimal transaction-enabled SQL context for durable lease persistence tests. + */ + @Configuration + @EnableTransactionManagement + static class TestConfiguration { + + @Bean + DataSource dataSource() { + return new EmbeddedDatabaseBuilder() + .generateUniqueName(true) + .setType(EmbeddedDatabaseType.H2) + .build(); + } + + @Bean + JdbcTemplate jdbcTemplate(DataSource dataSource) { + return new JdbcTemplate(dataSource); + } + + @Bean + PlatformTransactionManager transactionManager(DataSource dataSource) { + return new DataSourceTransactionManager(dataSource); + } + + @Bean + EtlJobLeaseRepository etlJobLeaseRepository( + JdbcTemplate jdbcTemplate, + PlatformTransactionManager transactionManager + ) { + return new EtlJobLeaseRepository(jdbcTemplate, transactionManager); + } + } +} From d037bf1c4a4ece20f85fca92b9445897de89ef98 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:38:59 +0900 Subject: [PATCH 11/92] test(etl): specify lease value invariants --- .../xtrmetl/etl/job/EtlJobLeaseModelTest.java | 79 +++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseModelTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseModelTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseModelTest.java new file mode 100644 index 00000000..bf677eb1 --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseModelTest.java @@ -0,0 +1,79 @@ +package com.xtrmetl.etl.job; + +import org.junit.jupiter.api.Test; + +import java.time.Instant; +import java.util.UUID; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; + +/** + * Specifies the immutable value contract carried from a database claim into execution. + */ +class EtlJobLeaseModelTest { + + private static final UUID JOB_RECORD_ID = UUID.randomUUID(); + private static final UUID LEASE_CLAIM_ID = UUID.randomUUID(); + private static final String OWNER_ID = "worker-alpha"; + private static final String PAYLOAD = "[{\"id\":\"record_alpha\"}]"; + private static final Instant EXPIRY = Instant.parse("2026-08-05T01:00:00Z"); + + @Test + void retainsEveryValidatedClaimField() { + EtlJobLease lease = new EtlJobLease( + JOB_RECORD_ID, + LEASE_CLAIM_ID, + OWNER_ID, + PAYLOAD, + 2, + EXPIRY + ); + + assertEquals(JOB_RECORD_ID, lease.jobRecordId()); + assertEquals(LEASE_CLAIM_ID, lease.leaseClaimId()); + assertEquals(OWNER_ID, lease.leaseOwnerId()); + assertEquals(PAYLOAD, lease.requestPayload()); + assertEquals(2, lease.attemptCount()); + assertEquals(EXPIRY, lease.leaseExpiresAt()); + } + + @Test + void rejectsMissingUnsafeOrImpossibleFields() { + assertThrows( + NullPointerException.class, + () -> new EtlJobLease(null, LEASE_CLAIM_ID, OWNER_ID, PAYLOAD, 1, EXPIRY) + ); + assertThrows( + NullPointerException.class, + () -> new EtlJobLease(JOB_RECORD_ID, null, OWNER_ID, PAYLOAD, 1, EXPIRY) + ); + assertThrows( + NullPointerException.class, + () -> new EtlJobLease(JOB_RECORD_ID, LEASE_CLAIM_ID, null, PAYLOAD, 1, EXPIRY) + ); + assertThrows( + IllegalArgumentException.class, + () -> new EtlJobLease( + JOB_RECORD_ID, + LEASE_CLAIM_ID, + "unsafe owner", + PAYLOAD, + 1, + EXPIRY + ) + ); + assertThrows( + NullPointerException.class, + () -> new EtlJobLease(JOB_RECORD_ID, LEASE_CLAIM_ID, OWNER_ID, null, 1, EXPIRY) + ); + assertThrows( + IllegalArgumentException.class, + () -> new EtlJobLease(JOB_RECORD_ID, LEASE_CLAIM_ID, OWNER_ID, PAYLOAD, 0, EXPIRY) + ); + assertThrows( + NullPointerException.class, + () -> new EtlJobLease(JOB_RECORD_ID, LEASE_CLAIM_ID, OWNER_ID, PAYLOAD, 1, null) + ); + } +} From f1d066f2c01fb3678fea8a998bcd46db129a7655 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:39:12 +0900 Subject: [PATCH 12/92] feat(etl): add immutable lease value --- .../java/com/xtrmetl/etl/job/EtlJobLease.java | 52 +++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLease.java diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLease.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLease.java new file mode 100644 index 00000000..23ca99b1 --- /dev/null +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLease.java @@ -0,0 +1,52 @@ +package com.xtrmetl.etl.job; + +import java.time.Instant; +import java.util.Objects; +import java.util.UUID; +import java.util.regex.Pattern; + +/** + * Carries one immutable, fenced database claim into ETL execution. + * + * @param jobRecordId durable job identifier + * @param leaseClaimId unique token generated for this exact claim or reclaim + * @param leaseOwnerId non-sensitive process-lifetime worker identifier + * @param requestPayload validated JSON payload retained while the job is non-terminal + * @param attemptCount one-based claim attempt count after this claim was persisted + * @param leaseExpiresAt database-derived instant after which this claim is stale + */ +public record EtlJobLease( + UUID jobRecordId, + UUID leaseClaimId, + String leaseOwnerId, + String requestPayload, + int attemptCount, + Instant leaseExpiresAt +) { + + private static final Pattern SAFE_LEASE_OWNER_PATTERN = Pattern.compile( + "[A-Za-z0-9._:-]{8,128}" + ); + + /** + * Validates every field needed for exact lease fencing and deterministic execution. + */ + public EtlJobLease { + Objects.requireNonNull(jobRecordId, "jobRecordId must not be null"); + Objects.requireNonNull(leaseClaimId, "leaseClaimId must not be null"); + String requiredOwnerId = Objects.requireNonNull( + leaseOwnerId, + "leaseOwnerId must not be null" + ); + if (!SAFE_LEASE_OWNER_PATTERN.matcher(requiredOwnerId).matches()) { + throw new IllegalArgumentException( + "leaseOwnerId must match [A-Za-z0-9._:-]{8,128}" + ); + } + Objects.requireNonNull(requestPayload, "requestPayload must not be null"); + if (attemptCount < 1) { + throw new IllegalArgumentException("attemptCount must be positive"); + } + Objects.requireNonNull(leaseExpiresAt, "leaseExpiresAt must not be null"); + } +} From b04a10549ec73ed7b698651ce14f6fc81edfb609 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:39:39 +0900 Subject: [PATCH 13/92] feat(etl): define stale lease failure --- .../etl/job/StaleEtlJobLeaseException.java | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 etl-service/src/main/java/com/xtrmetl/etl/job/StaleEtlJobLeaseException.java diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/StaleEtlJobLeaseException.java b/etl-service/src/main/java/com/xtrmetl/etl/job/StaleEtlJobLeaseException.java new file mode 100644 index 00000000..d7320870 --- /dev/null +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/StaleEtlJobLeaseException.java @@ -0,0 +1,17 @@ +package com.xtrmetl.etl.job; + +/** + * Signals that a worker no longer owns the exact live lease required for a state transition. + * + *

The exception intentionally carries no job, claim, owner, payload, SQL, or timestamp values so + * accidental logging cannot disclose operational identifiers or retained customer data.

+ */ +public class StaleEtlJobLeaseException extends RuntimeException { + + /** + * Creates the stable non-sensitive stale-lease signal. + */ + public StaleEtlJobLeaseException() { + super("The durable ETL job lease is stale or no longer owned"); + } +} From de7d29ddcc514f1c2233a1e39d63fd6aa56e1ee7 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:40:21 +0900 Subject: [PATCH 14/92] feat(etl): persist exclusive fenced claims --- .../etl/job/EtlJobLeaseRepository.java | 351 ++++++++++++++++++ 1 file changed, 351 insertions(+) create mode 100644 etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java new file mode 100644 index 00000000..328bb9ca --- /dev/null +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java @@ -0,0 +1,351 @@ +package com.xtrmetl.etl.job; + +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.stereotype.Repository; +import org.springframework.transaction.PlatformTransactionManager; +import org.springframework.transaction.support.TransactionTemplate; + +import java.time.Duration; +import java.time.Instant; +import java.time.OffsetDateTime; +import java.time.ZoneOffset; +import java.util.List; +import java.util.Objects; +import java.util.Optional; +import java.util.UUID; +import java.util.regex.Pattern; + +/** + * Owns PostgreSQL-backed durable-job claims and exact lease-fenced state transitions. + * + *

Every claim transaction first terminalizes eligible exhausted rows, then locks at most one + * oldest eligible row with {@code FOR UPDATE SKIP LOCKED}, and finally writes a fresh claim token, + * owner, expiry, and incremented attempt count before commit. State transitions repeat the exact + * claim token, owner, running status, and database-time expiry predicates so stale workers cannot + * mutate lifecycle state.

+ */ +@Repository +public class EtlJobLeaseRepository { + + /** Stable terminal code assigned when no additional claim is permitted. */ + public static final String ATTEMPTS_EXHAUSTED_FAILURE_CODE = + "etl_worker_attempts_exhausted"; + + private static final Pattern SAFE_OWNER_PATTERN = Pattern.compile( + "[A-Za-z0-9._:-]{8,128}" + ); + private static final Pattern SAFE_FAILURE_CODE_PATTERN = Pattern.compile( + "[a-z][a-z0-9_]{2,127}" + ); + + private static final String TERMINALIZE_EXHAUSTED_SQL = """ + UPDATE etl_job_records + SET job_status = 'FAILED', + request_payload = NULL, + failure_code = ?, + lease_claim_id = NULL, + lease_owner_id = NULL, + lease_expires_at = NULL, + updated_at = CURRENT_TIMESTAMP + WHERE attempt_count >= ? + AND ( + job_status = 'PENDING' + OR ( + job_status = 'RUNNING' + AND lease_expires_at <= CURRENT_TIMESTAMP + ) + ) + """; + + private static final String SELECT_CANDIDATE_SQL = """ + SELECT job_record_id, + request_payload, + attempt_count, + CURRENT_TIMESTAMP AS database_now + FROM etl_job_records + WHERE attempt_count < ? + AND ( + job_status = 'PENDING' + OR ( + job_status = 'RUNNING' + AND lease_expires_at <= CURRENT_TIMESTAMP + ) + ) + ORDER BY created_at, job_record_id + FETCH FIRST 1 ROW ONLY + FOR UPDATE SKIP LOCKED + """; + + private static final String CLAIM_CANDIDATE_SQL = """ + UPDATE etl_job_records + SET job_status = 'RUNNING', + attempt_count = attempt_count + 1, + failure_code = NULL, + lease_claim_id = ?, + lease_owner_id = ?, + lease_expires_at = ?, + updated_at = CURRENT_TIMESTAMP + WHERE job_record_id = ? + AND attempt_count = ? + AND ( + job_status = 'PENDING' + OR ( + job_status = 'RUNNING' + AND lease_expires_at <= CURRENT_TIMESTAMP + ) + ) + """; + + private static final String MARK_SUCCEEDED_SQL = """ + UPDATE etl_job_records + SET job_status = 'SUCCEEDED', + request_payload = NULL, + failure_code = NULL, + lease_claim_id = NULL, + lease_owner_id = NULL, + lease_expires_at = NULL, + updated_at = CURRENT_TIMESTAMP + WHERE job_record_id = ? + AND job_status = 'RUNNING' + AND lease_claim_id = ? + AND lease_owner_id = ? + AND lease_expires_at > CURRENT_TIMESTAMP + """; + + private static final String RELEASE_FOR_RETRY_SQL = """ + UPDATE etl_job_records + SET job_status = 'PENDING', + failure_code = NULL, + lease_claim_id = NULL, + lease_owner_id = NULL, + lease_expires_at = NULL, + updated_at = CURRENT_TIMESTAMP + WHERE job_record_id = ? + AND job_status = 'RUNNING' + AND lease_claim_id = ? + AND lease_owner_id = ? + AND lease_expires_at > CURRENT_TIMESTAMP + AND attempt_count < ? + """; + + private static final String MARK_FAILED_SQL = """ + UPDATE etl_job_records + SET job_status = 'FAILED', + request_payload = NULL, + failure_code = ?, + lease_claim_id = NULL, + lease_owner_id = NULL, + lease_expires_at = NULL, + updated_at = CURRENT_TIMESTAMP + WHERE job_record_id = ? + AND job_status = 'RUNNING' + AND lease_claim_id = ? + AND lease_owner_id = ? + AND lease_expires_at > CURRENT_TIMESTAMP + """; + + private final JdbcTemplate jdbcTemplate; + private final TransactionTemplate transactionTemplate; + + /** + * Creates lease persistence using one JDBC adapter and one transaction authority. + * + * @param jdbcTemplate JDBC operations for the durable-job table + * @param transactionManager transaction manager that owns row locks and claim commits + */ + public EtlJobLeaseRepository( + JdbcTemplate jdbcTemplate, + PlatformTransactionManager transactionManager + ) { + this.jdbcTemplate = Objects.requireNonNull( + jdbcTemplate, + "jdbcTemplate must not be null" + ); + this.transactionTemplate = new TransactionTemplate(Objects.requireNonNull( + transactionManager, + "transactionManager must not be null" + )); + } + + /** + * Claims at most one oldest eligible job for one worker process. + * + * @param leaseOwnerId safe non-sensitive process identifier + * @param leaseDuration positive duration applied to database claim time + * @param maxAttempts maximum permitted claim count from 1 through 100 + * @return a fresh claim, or an empty result when no row is eligible + * @throws NullPointerException when an argument is {@code null} + * @throws IllegalArgumentException when an argument violates its bounded contract + * @throws IllegalStateException when a locked candidate unexpectedly cannot be claimed + */ + public Optional claimNext( + String leaseOwnerId, + Duration leaseDuration, + int maxAttempts + ) { + String validatedOwnerId = requireSafeOwnerId(leaseOwnerId); + Duration validatedDuration = requirePositiveDuration(leaseDuration); + int validatedMaxAttempts = requireMaxAttempts(maxAttempts); + + return Objects.requireNonNull(transactionTemplate.execute(transactionStatus -> { + jdbcTemplate.update( + TERMINALIZE_EXHAUSTED_SQL, + ATTEMPTS_EXHAUSTED_FAILURE_CODE, + validatedMaxAttempts + ); + List candidates = jdbcTemplate.query( + SELECT_CANDIDATE_SQL, + (resultSet, rowNumber) -> new ClaimCandidate( + resultSet.getObject("job_record_id", UUID.class), + resultSet.getString("request_payload"), + resultSet.getInt("attempt_count"), + resultSet.getObject("database_now", OffsetDateTime.class).toInstant() + ), + validatedMaxAttempts + ); + if (candidates.isEmpty()) { + return Optional.empty(); + } + + ClaimCandidate candidate = candidates.getFirst(); + UUID leaseClaimId = UUID.randomUUID(); + Instant leaseExpiresAt = candidate.databaseNow().plus(validatedDuration); + int updatedRows = jdbcTemplate.update( + CLAIM_CANDIDATE_SQL, + leaseClaimId, + validatedOwnerId, + OffsetDateTime.ofInstant(leaseExpiresAt, ZoneOffset.UTC), + candidate.jobRecordId(), + candidate.attemptCount() + ); + if (updatedRows != 1) { + throw new IllegalStateException("Locked ETL job candidate could not be claimed"); + } + return Optional.of(new EtlJobLease( + candidate.jobRecordId(), + leaseClaimId, + validatedOwnerId, + candidate.requestPayload(), + candidate.attemptCount() + 1, + leaseExpiresAt + )); + }), "claim transaction must return a result"); + } + + /** + * Commits terminal success only for the exact live claim. + * + * @param lease exact claim whose target effects completed in the same transaction + * @throws NullPointerException when the lease is {@code null} + * @throws StaleEtlJobLeaseException when the claim is expired or no longer authoritative + */ + public void markSucceeded(EtlJobLease lease) { + EtlJobLease requiredLease = Objects.requireNonNull(lease, "lease must not be null"); + requireTransition(jdbcTemplate.update( + MARK_SUCCEEDED_SQL, + requiredLease.jobRecordId(), + requiredLease.leaseClaimId(), + requiredLease.leaseOwnerId() + )); + } + + /** + * Returns a failed execution to pending only while attempts remain and the claim is exact. + * + * @param lease exact live claim to release + * @param maxAttempts maximum permitted claim count from 1 through 100 + * @throws NullPointerException when the lease is {@code null} + * @throws IllegalArgumentException when the maximum is outside the supported range + * @throws StaleEtlJobLeaseException when the claim is stale or no retry remains + */ + public void releaseForRetry(EtlJobLease lease, int maxAttempts) { + EtlJobLease requiredLease = Objects.requireNonNull(lease, "lease must not be null"); + int validatedMaxAttempts = requireMaxAttempts(maxAttempts); + requireTransition(jdbcTemplate.update( + RELEASE_FOR_RETRY_SQL, + requiredLease.jobRecordId(), + requiredLease.leaseClaimId(), + requiredLease.leaseOwnerId(), + validatedMaxAttempts + )); + } + + /** + * Commits terminal failure and clears the retained payload for the exact live claim. + * + * @param lease exact live claim to fail + * @param failureCode stable non-sensitive machine-readable failure classification + * @throws NullPointerException when an argument is {@code null} + * @throws IllegalArgumentException when the failure code is unsafe + * @throws StaleEtlJobLeaseException when the claim is expired or no longer authoritative + */ + public void markFailed(EtlJobLease lease, String failureCode) { + EtlJobLease requiredLease = Objects.requireNonNull(lease, "lease must not be null"); + String validatedFailureCode = requireSafeFailureCode(failureCode); + requireTransition(jdbcTemplate.update( + MARK_FAILED_SQL, + validatedFailureCode, + requiredLease.jobRecordId(), + requiredLease.leaseClaimId(), + requiredLease.leaseOwnerId() + )); + } + + private static String requireSafeOwnerId(String leaseOwnerId) { + String requiredOwnerId = Objects.requireNonNull( + leaseOwnerId, + "leaseOwnerId must not be null" + ); + if (!SAFE_OWNER_PATTERN.matcher(requiredOwnerId).matches()) { + throw new IllegalArgumentException( + "leaseOwnerId must match [A-Za-z0-9._:-]{8,128}" + ); + } + return requiredOwnerId; + } + + private static Duration requirePositiveDuration(Duration leaseDuration) { + Duration requiredDuration = Objects.requireNonNull( + leaseDuration, + "leaseDuration must not be null" + ); + if (requiredDuration.isZero() || requiredDuration.isNegative()) { + throw new IllegalArgumentException("leaseDuration must be positive"); + } + return requiredDuration; + } + + private static int requireMaxAttempts(int maxAttempts) { + if (maxAttempts < 1 || maxAttempts > 100) { + throw new IllegalArgumentException("maxAttempts must be between 1 and 100"); + } + return maxAttempts; + } + + private static String requireSafeFailureCode(String failureCode) { + String requiredFailureCode = Objects.requireNonNull( + failureCode, + "failureCode must not be null" + ); + if (!SAFE_FAILURE_CODE_PATTERN.matcher(requiredFailureCode).matches()) { + throw new IllegalArgumentException( + "failureCode must match [a-z][a-z0-9_]{2,127}" + ); + } + return requiredFailureCode; + } + + private static void requireTransition(int updatedRows) { + if (updatedRows != 1) { + throw new StaleEtlJobLeaseException(); + } + } + + private record ClaimCandidate( + UUID jobRecordId, + String requestPayload, + int attemptCount, + Instant databaseNow + ) { + } +} From c118dc51a26468829c38650ecc8243272a7061a2 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:42:32 +0900 Subject: [PATCH 15/92] test(etl): specify atomic leased execution --- ...EtlJobExecutionServiceIntegrationTest.java | 256 ++++++++++++++++++ 1 file changed, 256 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobExecutionServiceIntegrationTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobExecutionServiceIntegrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobExecutionServiceIntegrationTest.java new file mode 100644 index 00000000..6b376423 --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobExecutionServiceIntegrationTest.java @@ -0,0 +1,256 @@ +package com.xtrmetl.etl.job; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.xtrmetl.etl.service.EtlBatchProperties; +import com.xtrmetl.etl.service.EtlRequestLock; +import com.xtrmetl.etl.service.EtlService; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.jdbc.datasource.DataSourceTransactionManager; +import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseBuilder; +import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseType; +import org.springframework.test.context.junit.jupiter.SpringJUnitConfig; +import org.springframework.transaction.PlatformTransactionManager; +import org.springframework.transaction.annotation.EnableTransactionManagement; + +import javax.sql.DataSource; +import java.time.Duration; +import java.time.Instant; +import java.util.UUID; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; + +/** + * Proves that target writes and exact-live-lease success commit or roll back together. + */ +@SpringJUnitConfig(EtlJobExecutionServiceIntegrationTest.TestConfiguration.class) +class EtlJobExecutionServiceIntegrationTest { + + private static final String OWNER_ID = "worker-alpha"; + private static final String PAYLOAD = """ + [{"id":"record_alpha","name":"accepted","email":"USER@EXAMPLE.COM"}] + """; + + private final EtlJobExecutionService executionService; + private final EtlJobLeaseRepository leaseRepository; + private final JdbcTemplate jdbcTemplate; + + @Autowired + EtlJobExecutionServiceIntegrationTest( + EtlJobExecutionService executionService, + EtlJobLeaseRepository leaseRepository, + JdbcTemplate jdbcTemplate + ) { + this.executionService = executionService; + this.leaseRepository = leaseRepository; + this.jdbcTemplate = jdbcTemplate; + } + + @BeforeEach + void createTables() { + jdbcTemplate.execute("DROP TABLE IF EXISTS processed_data"); + jdbcTemplate.execute("DROP TABLE IF EXISTS etl_job_records"); + jdbcTemplate.execute(""" + CREATE TABLE etl_job_records ( + job_record_id UUID PRIMARY KEY, + principal_scope_hash CHAR(64) NOT NULL, + submission_key_hash CHAR(64) NOT NULL, + request_digest CHAR(64) NOT NULL, + request_payload CLOB, + job_status VARCHAR(32) NOT NULL, + attempt_count INTEGER NOT NULL DEFAULT 0, + failure_code VARCHAR(128), + lease_claim_id UUID, + lease_owner_id VARCHAR(128), + lease_expires_at TIMESTAMP WITH TIME ZONE, + created_at TIMESTAMP WITH TIME ZONE NOT NULL, + updated_at TIMESTAMP WITH TIME ZONE NOT NULL + ) + """); + jdbcTemplate.execute(""" + CREATE TABLE processed_data ( + processed_record_id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + data VARCHAR(8192) NOT NULL + ) + """); + } + + @Test + void commitsTargetRowsAndTerminalSuccessInOneTransaction() { + UUID jobRecordId = insertPendingJob(); + EtlJobLease lease = leaseRepository.claimNext( + OWNER_ID, + Duration.ofMinutes(5), + 3 + ).orElseThrow(); + + executionService.execute(lease); + + assertEquals(jobRecordId, lease.jobRecordId()); + assertEquals(1, processedRowCount()); + assertEquals( + "ID:record_alpha,NAME:ACCEPTED,EMAIL:user@example.com,", + jdbcTemplate.queryForObject("SELECT data FROM processed_data", String.class) + ); + assertEquals("SUCCEEDED", jobStatus(jobRecordId)); + assertEquals(0, retainedPayloadCount(jobRecordId)); + } + + @Test + void rollsBackTargetRowsWhenTheClaimWasSuperseded() { + UUID jobRecordId = insertPendingJob(); + EtlJobLease lease = leaseRepository.claimNext( + OWNER_ID, + Duration.ofMinutes(5), + 3 + ).orElseThrow(); + jdbcTemplate.update( + "UPDATE etl_job_records SET lease_claim_id = ? WHERE job_record_id = ?", + UUID.randomUUID(), + jobRecordId + ); + + assertThrows(StaleEtlJobLeaseException.class, () -> executionService.execute(lease)); + + assertEquals(0, processedRowCount()); + assertEquals("RUNNING", jobStatus(jobRecordId)); + assertEquals(1, retainedPayloadCount(jobRecordId)); + } + + @Test + void rejectsMissingCollaboratorsOrLease() { + assertThrows( + NullPointerException.class, + () -> new EtlJobExecutionService(null, leaseRepository) + ); + assertThrows( + NullPointerException.class, + () -> new EtlJobExecutionService(executionService.etlService(), null) + ); + assertThrows(NullPointerException.class, () -> executionService.execute(null)); + } + + private UUID insertPendingJob() { + UUID jobRecordId = UUID.randomUUID(); + Instant now = Instant.now(); + jdbcTemplate.update( + """ + INSERT INTO etl_job_records ( + job_record_id, principal_scope_hash, submission_key_hash, + request_digest, request_payload, job_status, attempt_count, + created_at, updated_at + ) VALUES (?, ?, ?, ?, ?, 'PENDING', 0, ?, ?) + """, + jobRecordId, + "a".repeat(64), + "b".repeat(64), + "c".repeat(64), + PAYLOAD, + now, + now + ); + return jobRecordId; + } + + private int processedRowCount() { + Integer count = jdbcTemplate.queryForObject( + "SELECT COUNT(*) FROM processed_data", + Integer.class + ); + return count == null ? 0 : count; + } + + private String jobStatus(UUID jobRecordId) { + return jdbcTemplate.queryForObject( + "SELECT job_status FROM etl_job_records WHERE job_record_id = ?", + String.class, + jobRecordId + ); + } + + private int retainedPayloadCount(UUID jobRecordId) { + Integer count = jdbcTemplate.queryForObject( + """ + SELECT COUNT(*) + FROM etl_job_records + WHERE job_record_id = ? + AND request_payload IS NOT NULL + """, + Integer.class, + jobRecordId + ); + return count == null ? 0 : count; + } + + /** + * Transaction-enabled execution context using one database for source state and target effects. + */ + @Configuration + @EnableTransactionManagement + static class TestConfiguration { + + @Bean + DataSource dataSource() { + return new EmbeddedDatabaseBuilder() + .generateUniqueName(true) + .setType(EmbeddedDatabaseType.H2) + .build(); + } + + @Bean + JdbcTemplate jdbcTemplate(DataSource dataSource) { + return new JdbcTemplate(dataSource); + } + + @Bean + PlatformTransactionManager transactionManager(DataSource dataSource) { + return new DataSourceTransactionManager(dataSource); + } + + @Bean + EtlBatchProperties etlBatchProperties() { + return new EtlBatchProperties(); + } + + @Bean + ObjectMapper objectMapper() { + return new ObjectMapper(); + } + + @Bean + EtlRequestLock etlRequestLock() { + return idempotencyKeyHash -> true; + } + + @Bean + EtlService etlService( + JdbcTemplate jdbcTemplate, + ObjectMapper objectMapper, + EtlBatchProperties properties, + EtlRequestLock requestLock + ) { + return new EtlService(jdbcTemplate, objectMapper, properties, requestLock); + } + + @Bean + EtlJobLeaseRepository etlJobLeaseRepository( + JdbcTemplate jdbcTemplate, + PlatformTransactionManager transactionManager + ) { + return new EtlJobLeaseRepository(jdbcTemplate, transactionManager); + } + + @Bean + EtlJobExecutionService etlJobExecutionService( + EtlService etlService, + EtlJobLeaseRepository leaseRepository + ) { + return new EtlJobExecutionService(etlService, leaseRepository); + } + } +} From 72fbed4f507a2998a32f7aafcb19fcf3d843bd7c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:42:46 +0900 Subject: [PATCH 16/92] feat(etl): couple target writes to lease success --- .../etl/job/EtlJobExecutionService.java | 55 +++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100644 etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobExecutionService.java diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobExecutionService.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobExecutionService.java new file mode 100644 index 00000000..7286dfc1 --- /dev/null +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobExecutionService.java @@ -0,0 +1,55 @@ +package com.xtrmetl.etl.job; + +import com.xtrmetl.etl.service.EtlService; +import org.springframework.stereotype.Service; +import org.springframework.transaction.annotation.Transactional; + +import java.util.Objects; + +/** + * Executes one claimed ETL payload and commits terminal success in the same transaction. + * + *

The existing {@link EtlService} performs validated target writes. The subsequent conditional + * success transition must match the exact unexpired claim. If that transition reports a stale + * lease, {@link StaleEtlJobLeaseException} escapes and Spring rolls back every target write made by + * the same transaction.

+ */ +@Service +public class EtlJobExecutionService { + + private final EtlService etlService; + private final EtlJobLeaseRepository leaseRepository; + + /** + * Creates the atomic durable-job execution boundary. + * + * @param etlService validated ETL target writer + * @param leaseRepository exact lease-fenced lifecycle persistence + */ + public EtlJobExecutionService( + EtlService etlService, + EtlJobLeaseRepository leaseRepository + ) { + this.etlService = Objects.requireNonNull(etlService, "etlService must not be null"); + this.leaseRepository = Objects.requireNonNull( + leaseRepository, + "leaseRepository must not be null" + ); + } + + /** + * Processes the retained payload and marks the exact live claim successful atomically. + * + * @param lease exact database claim to execute + * @throws NullPointerException when the lease is {@code null} + * @throws com.xtrmetl.etl.service.EtlRequestException when the retained request is invalid + * @throws org.springframework.dao.DataAccessException when a target write fails + * @throws StaleEtlJobLeaseException when the claim expires or is superseded before success + */ + @Transactional + public void execute(EtlJobLease lease) { + EtlJobLease requiredLease = Objects.requireNonNull(lease, "lease must not be null"); + etlService.processData(requiredLease.requestPayload()); + leaseRepository.markSucceeded(requiredLease); + } +} From 8fffb6bd9726c57e36969c0ae0a303791810f0a0 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:43:23 +0900 Subject: [PATCH 17/92] test(etl): keep execution collaborators encapsulated --- .../etl/job/EtlJobExecutionServiceIntegrationTest.java | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobExecutionServiceIntegrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobExecutionServiceIntegrationTest.java index 6b376423..3f29c0a6 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobExecutionServiceIntegrationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobExecutionServiceIntegrationTest.java @@ -38,16 +38,19 @@ class EtlJobExecutionServiceIntegrationTest { private final EtlJobExecutionService executionService; private final EtlJobLeaseRepository leaseRepository; + private final EtlService etlService; private final JdbcTemplate jdbcTemplate; @Autowired EtlJobExecutionServiceIntegrationTest( EtlJobExecutionService executionService, EtlJobLeaseRepository leaseRepository, + EtlService etlService, JdbcTemplate jdbcTemplate ) { this.executionService = executionService; this.leaseRepository = leaseRepository; + this.etlService = etlService; this.jdbcTemplate = jdbcTemplate; } @@ -130,7 +133,7 @@ void rejectsMissingCollaboratorsOrLease() { ); assertThrows( NullPointerException.class, - () -> new EtlJobExecutionService(executionService.etlService(), null) + () -> new EtlJobExecutionService(etlService, null) ); assertThrows(NullPointerException.class, () -> executionService.execute(null)); } From a3b7af40143a637ee4a7c6299d9be9e357eabd8f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:44:41 +0900 Subject: [PATCH 18/92] test(etl): specify bounded worker outcomes --- .../com/xtrmetl/etl/job/EtlJobWorkerTest.java | 248 ++++++++++++++++++ 1 file changed, 248 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java new file mode 100644 index 00000000..cda9971a --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java @@ -0,0 +1,248 @@ +package com.xtrmetl.etl.job; + +import com.xtrmetl.etl.service.EtlRequestError; +import com.xtrmetl.etl.service.EtlRequestException; +import io.micrometer.core.instrument.simple.SimpleMeterRegistry; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.springframework.dao.CannotAcquireLockException; +import org.springframework.dao.DataIntegrityViolationException; + +import java.time.Instant; +import java.util.Optional; +import java.util.UUID; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyInt; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +/** + * Specifies one-job polling, bounded retries, stable failures, stale fencing, and safe metrics. + */ +@ExtendWith(MockitoExtension.class) +class EtlJobWorkerTest { + + private static final String OWNER_ID = "worker-alpha"; + private static final String PAYLOAD = "[{\"id\":\"record_alpha\"}]"; + + @Mock + private EtlJobLeaseRepository leaseRepository; + + @Mock + private EtlJobExecutionService executionService; + + private EtlJobWorkerProperties properties; + private SimpleMeterRegistry meterRegistry; + private EtlJobWorker worker; + + @BeforeEach + void createWorker() { + properties = new EtlJobWorkerProperties(); + properties.setEnabled(true); + properties.setLeaseOwnerId(OWNER_ID); + properties.setLeaseDurationSeconds(120L); + properties.setMaxAttempts(3); + meterRegistry = new SimpleMeterRegistry(); + worker = new EtlJobWorker( + leaseRepository, + executionService, + properties, + meterRegistry + ); + } + + @Test + void recordsIdleWithoutExecutingWhenNoJobIsEligible() { + when(leaseRepository.claimNext(anyString(), any(), anyInt())) + .thenReturn(Optional.empty()); + + worker.pollOnce(); + + verify(executionService, never()).execute(any()); + assertMetric("idle", 0.0, 1L); + } + + @Test + void executesAtMostOneClaimAndRecordsClaimedAndSucceeded() { + EtlJobLease lease = lease(1); + when(leaseRepository.claimNext(anyString(), any(), anyInt())) + .thenReturn(Optional.of(lease)); + + worker.pollOnce(); + + verify(executionService).execute(lease); + assertMetric("claimed", 1.0, 0L); + assertMetric("succeeded", 1.0, 1L); + } + + @Test + void releasesTransientFailureWhenAttemptsRemain() { + EtlJobLease lease = lease(2); + when(leaseRepository.claimNext(anyString(), any(), anyInt())) + .thenReturn(Optional.of(lease)); + doThrow(new CannotAcquireLockException("temporary")) + .when(executionService).execute(lease); + + worker.pollOnce(); + + verify(leaseRepository).releaseForRetry(lease, 3); + verify(leaseRepository, never()).markFailed(any(), anyString()); + assertMetric("retried", 1.0, 1L); + } + + @Test + void terminalizesTransientFailureAtTheAttemptLimit() { + EtlJobLease lease = lease(3); + when(leaseRepository.claimNext(anyString(), any(), anyInt())) + .thenReturn(Optional.of(lease)); + doThrow(new CannotAcquireLockException("temporary")) + .when(executionService).execute(lease); + + worker.pollOnce(); + + verify(leaseRepository).markFailed(lease, "etl_target_unavailable"); + verify(leaseRepository, never()).releaseForRetry(any(), anyInt()); + assertMetric("failed", 1.0, 1L); + } + + @Test + void terminalizesDeterministicRequestFailureWithItsStableCode() { + EtlJobLease lease = lease(1); + when(leaseRepository.claimNext(anyString(), any(), anyInt())) + .thenReturn(Optional.of(lease)); + doThrow(new EtlRequestException(EtlRequestError.INVALID_JSON)) + .when(executionService).execute(lease); + + worker.pollOnce(); + + verify(leaseRepository).markFailed(lease, "etl_invalid_json"); + assertMetric("failed", 1.0, 1L); + } + + @Test + void terminalizesNonTransientTargetFailureWithoutDiagnosticText() { + EtlJobLease lease = lease(1); + when(leaseRepository.claimNext(anyString(), any(), anyInt())) + .thenReturn(Optional.of(lease)); + doThrow(new DataIntegrityViolationException("sensitive SQL")) + .when(executionService).execute(lease); + + worker.pollOnce(); + + verify(leaseRepository).markFailed(lease, "etl_target_failure"); + assertMetric("failed", 1.0, 1L); + } + + @Test + void terminalizesUnexpectedRuntimeFailureWithGenericCode() { + EtlJobLease lease = lease(1); + when(leaseRepository.claimNext(anyString(), any(), anyInt())) + .thenReturn(Optional.of(lease)); + doThrow(new IllegalStateException("sensitive implementation detail")) + .when(executionService).execute(lease); + + worker.pollOnce(); + + verify(leaseRepository).markFailed(lease, "etl_internal_error"); + assertMetric("failed", 1.0, 1L); + } + + @Test + void treatsAnExecutionOrTransitionFenceFailureAsStaleEvidence() { + EtlJobLease executionStaleLease = lease(1); + when(leaseRepository.claimNext(anyString(), any(), anyInt())) + .thenReturn(Optional.of(executionStaleLease)); + doThrow(new StaleEtlJobLeaseException()) + .when(executionService).execute(executionStaleLease); + + worker.pollOnce(); + + verify(leaseRepository, never()).markFailed(any(), anyString()); + verify(leaseRepository, never()).releaseForRetry(any(), anyInt()); + assertMetric("stale", 1.0, 1L); + } + + @Test + void treatsAStaleRetryTransitionAsStaleEvidence() { + EtlJobLease lease = lease(1); + when(leaseRepository.claimNext(anyString(), any(), anyInt())) + .thenReturn(Optional.of(lease)); + doThrow(new CannotAcquireLockException("temporary")) + .when(executionService).execute(lease); + doThrow(new StaleEtlJobLeaseException()) + .when(leaseRepository).releaseForRetry(lease, 3); + + worker.pollOnce(); + + assertMetric("stale", 1.0, 1L); + assertMetric("retried", 0.0, 0L); + } + + @Test + void recordsClaimDatabaseFailureWithoutLeakingOrExecuting() { + when(leaseRepository.claimNext(anyString(), any(), anyInt())) + .thenThrow(new CannotAcquireLockException("sensitive SQL")); + + worker.pollOnce(); + + verify(executionService, never()).execute(any()); + assertMetric("failed", 1.0, 1L); + } + + @Test + void rejectsMissingCollaborators() { + assertThrows( + NullPointerException.class, + () -> new EtlJobWorker(null, executionService, properties, meterRegistry) + ); + assertThrows( + NullPointerException.class, + () -> new EtlJobWorker(leaseRepository, null, properties, meterRegistry) + ); + assertThrows( + NullPointerException.class, + () -> new EtlJobWorker(leaseRepository, executionService, null, meterRegistry) + ); + assertThrows( + NullPointerException.class, + () -> new EtlJobWorker(leaseRepository, executionService, properties, null) + ); + } + + private void assertMetric(String outcome, double counterCount, long timerCount) { + assertEquals( + counterCount, + meterRegistry.find("etl.jobs.worker.outcomes") + .tag("outcome", outcome) + .counter() + .count() + ); + assertEquals( + timerCount, + meterRegistry.find("etl.jobs.execution.duration") + .tag("outcome", outcome) + .timer() + .count() + ); + } + + private static EtlJobLease lease(int attemptCount) { + return new EtlJobLease( + UUID.randomUUID(), + UUID.randomUUID(), + OWNER_ID, + PAYLOAD, + attemptCount, + Instant.now().plusSeconds(300) + ); + } +} From de3e5e1759777a8ef2d2acc2971772da0daefa94 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:45:19 +0900 Subject: [PATCH 19/92] feat(etl): execute one bounded leased job per poll --- .../com/xtrmetl/etl/job/EtlJobWorker.java | 203 ++++++++++++++++++ 1 file changed, 203 insertions(+) create mode 100644 etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorker.java diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorker.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorker.java new file mode 100644 index 00000000..d83f2537 --- /dev/null +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorker.java @@ -0,0 +1,203 @@ +package com.xtrmetl.etl.job; + +import com.xtrmetl.etl.service.EtlRequestException; +import io.micrometer.core.instrument.Counter; +import io.micrometer.core.instrument.MeterRegistry; +import io.micrometer.core.instrument.Timer; +import org.springframework.boot.autoconfigure.condition.ConditionalOnBooleanProperty; +import org.springframework.dao.DataAccessException; +import org.springframework.dao.TransientDataAccessException; +import org.springframework.scheduling.annotation.Scheduled; +import org.springframework.stereotype.Component; + +import java.time.Duration; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Optional; + +/** + * Polls and executes at most one durable ETL job per fixed-delay invocation. + * + *

The database claim repository, not the scheduler, distributes work across replicas. The + * worker classifies failures into stable non-sensitive codes, retries only transient database + * failures while attempts remain, and treats every failed exact-lease transition as stale evidence. + * Metrics use a fixed outcome vocabulary and never tag payloads, principals, keys, job identifiers, + * lease identifiers, SQL, exception classes, or exception messages.

+ */ +@Component +@ConditionalOnBooleanProperty( + prefix = "xtrmetl.etl.jobs.worker", + name = "enabled", + havingValue = true, + matchIfMissing = false +) +public class EtlJobWorker { + + /** Stable target-unavailability code used after transient attempts are exhausted. */ + public static final String TARGET_UNAVAILABLE_FAILURE_CODE = "etl_target_unavailable"; + + /** Stable non-transient database failure code. */ + public static final String TARGET_FAILURE_CODE = "etl_target_failure"; + + /** Stable unexpected implementation failure code. */ + public static final String INTERNAL_FAILURE_CODE = "etl_internal_error"; + + private static final String METRIC_OUTCOMES = "etl.jobs.worker.outcomes"; + private static final String METRIC_DURATION = "etl.jobs.execution.duration"; + private static final String IDLE_OUTCOME = "idle"; + private static final String CLAIMED_OUTCOME = "claimed"; + private static final String SUCCEEDED_OUTCOME = "succeeded"; + private static final String RETRIED_OUTCOME = "retried"; + private static final String FAILED_OUTCOME = "failed"; + private static final String STALE_OUTCOME = "stale"; + private static final List FINITE_OUTCOMES = List.of( + IDLE_OUTCOME, + CLAIMED_OUTCOME, + SUCCEEDED_OUTCOME, + RETRIED_OUTCOME, + FAILED_OUTCOME, + STALE_OUTCOME + ); + + private final EtlJobLeaseRepository leaseRepository; + private final EtlJobExecutionService executionService; + private final EtlJobWorkerProperties properties; + private final MeterRegistry meterRegistry; + private final Map outcomeCounters; + private final Map outcomeTimers; + + /** + * Creates one fail-closed worker and pre-registers its finite metric vocabulary. + * + * @param leaseRepository database claim and transition authority + * @param executionService atomic target-write and success boundary + * @param properties bounded worker configuration + * @param meterRegistry metrics registry for finite-cardinality evidence + */ + public EtlJobWorker( + EtlJobLeaseRepository leaseRepository, + EtlJobExecutionService executionService, + EtlJobWorkerProperties properties, + MeterRegistry meterRegistry + ) { + this.leaseRepository = Objects.requireNonNull( + leaseRepository, + "leaseRepository must not be null" + ); + this.executionService = Objects.requireNonNull( + executionService, + "executionService must not be null" + ); + this.properties = Objects.requireNonNull(properties, "properties must not be null"); + this.meterRegistry = Objects.requireNonNull( + meterRegistry, + "meterRegistry must not be null" + ); + + Map counters = new LinkedHashMap<>(); + Map timers = new LinkedHashMap<>(); + for (String outcome : FINITE_OUTCOMES) { + counters.put( + outcome, + Counter.builder(METRIC_OUTCOMES) + .description("Durable ETL worker outcomes") + .tag("outcome", outcome) + .register(this.meterRegistry) + ); + timers.put( + outcome, + Timer.builder(METRIC_DURATION) + .description("Duration of one durable ETL worker poll") + .tag("outcome", outcome) + .register(this.meterRegistry) + ); + } + this.outcomeCounters = Map.copyOf(counters); + this.outcomeTimers = Map.copyOf(timers); + } + + /** + * Claims and handles at most one eligible durable job. + * + *

Fixed delay is measured after this invocation completes. A database outage during claim is + * converted into a finite failed metric without copying diagnostic text into application logs or + * telemetry. Spring invokes the method only when worker activation is explicitly enabled.

+ */ + @Scheduled( + fixedDelayString = "${xtrmetl.etl.jobs.worker.fixed-delay-milliseconds:5000}", + initialDelayString = "${xtrmetl.etl.jobs.worker.initial-delay-milliseconds:5000}" + ) + public void pollOnce() { + Timer.Sample sample = Timer.start(meterRegistry); + String finalOutcome = runOnePoll(); + sample.stop(outcomeTimers.get(finalOutcome)); + } + + private String runOnePoll() { + final Optional claimedLease; + try { + claimedLease = leaseRepository.claimNext( + properties.getLeaseOwnerId(), + Duration.ofSeconds(properties.getLeaseDurationSeconds()), + properties.getMaxAttempts() + ); + } catch (DataAccessException exception) { + increment(FAILED_OUTCOME); + return FAILED_OUTCOME; + } + + if (claimedLease.isEmpty()) { + return IDLE_OUTCOME; + } + + EtlJobLease lease = claimedLease.orElseThrow(); + increment(CLAIMED_OUTCOME); + try { + executionService.execute(lease); + increment(SUCCEEDED_OUTCOME); + return SUCCEEDED_OUTCOME; + } catch (StaleEtlJobLeaseException exception) { + increment(STALE_OUTCOME); + return STALE_OUTCOME; + } catch (TransientDataAccessException exception) { + return handleTransientFailure(lease); + } catch (EtlRequestException exception) { + return markFailedOrStale(lease, exception.error().errorCode()); + } catch (DataAccessException exception) { + return markFailedOrStale(lease, TARGET_FAILURE_CODE); + } catch (RuntimeException exception) { + return markFailedOrStale(lease, INTERNAL_FAILURE_CODE); + } + } + + private String handleTransientFailure(EtlJobLease lease) { + if (lease.attemptCount() < properties.getMaxAttempts()) { + try { + leaseRepository.releaseForRetry(lease, properties.getMaxAttempts()); + increment(RETRIED_OUTCOME); + return RETRIED_OUTCOME; + } catch (StaleEtlJobLeaseException exception) { + increment(STALE_OUTCOME); + return STALE_OUTCOME; + } + } + return markFailedOrStale(lease, TARGET_UNAVAILABLE_FAILURE_CODE); + } + + private String markFailedOrStale(EtlJobLease lease, String failureCode) { + try { + leaseRepository.markFailed(lease, failureCode); + increment(FAILED_OUTCOME); + return FAILED_OUTCOME; + } catch (StaleEtlJobLeaseException exception) { + increment(STALE_OUTCOME); + return STALE_OUTCOME; + } + } + + private void increment(String outcome) { + outcomeCounters.get(outcome).increment(); + } +} From 19d30854953089299261c2acee3c970c8c469811 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:45:34 +0900 Subject: [PATCH 20/92] feat(etl): enable conditional worker scheduling --- etl-service/src/main/java/com/xtrmetl/etl/EtlApplication.java | 2 ++ 1 file changed, 2 insertions(+) diff --git a/etl-service/src/main/java/com/xtrmetl/etl/EtlApplication.java b/etl-service/src/main/java/com/xtrmetl/etl/EtlApplication.java index 17213106..961b3d7d 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/EtlApplication.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/EtlApplication.java @@ -9,6 +9,7 @@ import org.springframework.cloud.client.discovery.EnableDiscoveryClient; import org.springframework.context.annotation.EnableAspectJAutoProxy; import org.springframework.retry.annotation.EnableRetry; +import org.springframework.scheduling.annotation.EnableScheduling; /** * Bootstraps the mightyETL transformation and loading service. @@ -17,6 +18,7 @@ @EnableDiscoveryClient @EnableAspectJAutoProxy(proxyTargetClass = true) @EnableRetry +@EnableScheduling @EnableConfigurationProperties({ ConnectorProperties.class, EtlBatchProperties.class, From bd48d4aa00637e1c6049da20f6875759a1e80944 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:49:11 +0900 Subject: [PATCH 21/92] test(etl): make lease transition order deterministic --- .../etl/job/EtlJobLeaseRepositoryIntegrationTest.java | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryIntegrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryIntegrationTest.java index 773c5936..5fcdb5a9 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryIntegrationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryIntegrationTest.java @@ -26,7 +26,6 @@ import java.util.concurrent.Future; import static org.junit.jupiter.api.Assertions.assertEquals; -import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertNotEquals; import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.assertNull; @@ -210,7 +209,7 @@ void exactLiveLeaseCanSucceedRetryOrFailAndClearsTheRightFields() { assertNull(textColumn(retryJobId, "failure_code")); assertNull(textColumn(retryJobId, "lease_owner_id")); - UUID failedJobId = insertPending(Instant.parse("2026-08-05T00:02:00Z"), 2); + UUID failedJobId = insertPending(Instant.parse("2026-08-05T00:00:30Z"), 2); EtlJobLease failedLease = repository.claimNext(OWNER_ALPHA, LEASE_DURATION, 3) .orElseThrow(); repository.markFailed(failedLease, "etl_target_failure"); @@ -272,10 +271,18 @@ void rejectsInvalidPublicArgumentsBeforeSqlExecution() { IllegalArgumentException.class, () -> repository.claimNext(OWNER_ALPHA, Duration.ZERO, 3) ); + assertThrows( + IllegalArgumentException.class, + () -> repository.claimNext(OWNER_ALPHA, Duration.ofSeconds(-1), 3) + ); assertThrows( IllegalArgumentException.class, () -> repository.claimNext(OWNER_ALPHA, LEASE_DURATION, 0) ); + assertThrows( + IllegalArgumentException.class, + () -> repository.claimNext(OWNER_ALPHA, LEASE_DURATION, 101) + ); assertThrows( NullPointerException.class, () -> repository.markSucceeded(null) From 379d5cbd723eb49df07e2652559d349250f15337 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:50:11 +0900 Subject: [PATCH 22/92] test(etl): cover stale terminal transitions --- .../com/xtrmetl/etl/job/EtlJobWorkerTest.java | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java index cda9971a..6659fea1 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java @@ -157,7 +157,7 @@ void terminalizesUnexpectedRuntimeFailureWithGenericCode() { } @Test - void treatsAnExecutionOrTransitionFenceFailureAsStaleEvidence() { + void treatsAnExecutionFenceFailureAsStaleEvidence() { EtlJobLease executionStaleLease = lease(1); when(leaseRepository.claimNext(anyString(), any(), anyInt())) .thenReturn(Optional.of(executionStaleLease)); @@ -187,6 +187,22 @@ void treatsAStaleRetryTransitionAsStaleEvidence() { assertMetric("retried", 0.0, 0L); } + @Test + void treatsAStaleTerminalTransitionAsStaleEvidence() { + EtlJobLease lease = lease(1); + when(leaseRepository.claimNext(anyString(), any(), anyInt())) + .thenReturn(Optional.of(lease)); + doThrow(new EtlRequestException(EtlRequestError.INVALID_JSON)) + .when(executionService).execute(lease); + doThrow(new StaleEtlJobLeaseException()) + .when(leaseRepository).markFailed(lease, "etl_invalid_json"); + + worker.pollOnce(); + + assertMetric("stale", 1.0, 1L); + assertMetric("failed", 0.0, 0L); + } + @Test void recordsClaimDatabaseFailureWithoutLeakingOrExecuting() { when(leaseRepository.claimNext(anyString(), any(), anyInt())) From 6fe41622ac9da56ef6685d06c23dd9c4fc955043 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:51:24 +0900 Subject: [PATCH 23/92] test(config): specify durable worker aliases --- ...nfigAliasEnvironmentPostProcessorTest.java | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/config/MightyEtlConfigAliasEnvironmentPostProcessorTest.java b/etl-service/src/test/java/com/xtrmetl/etl/config/MightyEtlConfigAliasEnvironmentPostProcessorTest.java index 285b24d7..389dda7f 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/config/MightyEtlConfigAliasEnvironmentPostProcessorTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/config/MightyEtlConfigAliasEnvironmentPostProcessorTest.java @@ -45,6 +45,38 @@ void mirrorsModernDurableIntakeFlagToTheLegacyControllerCondition() { assertEquals("true", aliases.get("xtrmetl.etl.jobs.intake-enabled")); } + @Test + void mirrorsEveryModernDurableWorkerSettingToLegacyConsumers() { + MockEnvironment env = new MockEnvironment(); + env.setProperty("mightyetl.etl.jobs.worker.enabled", "true"); + env.setProperty("mightyetl.etl.jobs.worker.fixed-delay-milliseconds", "2500"); + env.setProperty("mightyetl.etl.jobs.worker.initial-delay-milliseconds", "1000"); + env.setProperty("mightyetl.etl.jobs.worker.lease-duration-seconds", "120"); + env.setProperty("mightyetl.etl.jobs.worker.max-attempts", "5"); + env.setProperty("mightyetl.etl.jobs.worker.lease-owner-id", "worker-primary"); + + Map aliases = MightyEtlConfigAliasEnvironmentPostProcessor.buildAliases(env); + + assertEquals("true", aliases.get("xtrmetl.etl.jobs.worker.enabled")); + assertEquals( + "2500", + aliases.get("xtrmetl.etl.jobs.worker.fixed-delay-milliseconds") + ); + assertEquals( + "1000", + aliases.get("xtrmetl.etl.jobs.worker.initial-delay-milliseconds") + ); + assertEquals( + "120", + aliases.get("xtrmetl.etl.jobs.worker.lease-duration-seconds") + ); + assertEquals("5", aliases.get("xtrmetl.etl.jobs.worker.max-attempts")); + assertEquals( + "worker-primary", + aliases.get("xtrmetl.etl.jobs.worker.lease-owner-id") + ); + } + @Test void mirrorsLegacyBatchLimitForModernTooling() { MockEnvironment env = new MockEnvironment(); From aa90cd365598128b377ba49607328936fa9efb5e Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:51:39 +0900 Subject: [PATCH 24/92] feat(config): dual-read durable worker settings --- .../MightyEtlConfigAliasEnvironmentPostProcessor.java | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/etl-service/src/main/java/com/xtrmetl/etl/config/MightyEtlConfigAliasEnvironmentPostProcessor.java b/etl-service/src/main/java/com/xtrmetl/etl/config/MightyEtlConfigAliasEnvironmentPostProcessor.java index 5462db4b..16a9ad00 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/config/MightyEtlConfigAliasEnvironmentPostProcessor.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/config/MightyEtlConfigAliasEnvironmentPostProcessor.java @@ -28,6 +28,12 @@ public class MightyEtlConfigAliasEnvironmentPostProcessor implements Environment "etl.max-payload-bytes", "etl.max-batch-records", "etl.jobs.intake-enabled", + "etl.jobs.worker.enabled", + "etl.jobs.worker.fixed-delay-milliseconds", + "etl.jobs.worker.initial-delay-milliseconds", + "etl.jobs.worker.lease-duration-seconds", + "etl.jobs.worker.max-attempts", + "etl.jobs.worker.lease-owner-id", "connectors.databricks.enabled", "connectors.snowflake.enabled", "connectors.qlik-sense.enabled" From f988caa86a8bb133d5bfcf2275a582c916d0309d Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:55:51 +0900 Subject: [PATCH 25/92] test(etl): specify stored execution identity --- .../xtrmetl/etl/job/EtlJobLeaseModelTest.java | 112 ++++++++++++++---- 1 file changed, 90 insertions(+), 22 deletions(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseModelTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseModelTest.java index bf677eb1..b221d95c 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseModelTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseModelTest.java @@ -16,23 +16,22 @@ class EtlJobLeaseModelTest { private static final UUID JOB_RECORD_ID = UUID.randomUUID(); private static final UUID LEASE_CLAIM_ID = UUID.randomUUID(); private static final String OWNER_ID = "worker-alpha"; + private static final String PRINCIPAL_SCOPE_HASH = "a".repeat(64); + private static final String SUBMISSION_KEY_HASH = "b".repeat(64); + private static final String REQUEST_DIGEST = "c".repeat(64); private static final String PAYLOAD = "[{\"id\":\"record_alpha\"}]"; private static final Instant EXPIRY = Instant.parse("2026-08-05T01:00:00Z"); @Test void retainsEveryValidatedClaimField() { - EtlJobLease lease = new EtlJobLease( - JOB_RECORD_ID, - LEASE_CLAIM_ID, - OWNER_ID, - PAYLOAD, - 2, - EXPIRY - ); + EtlJobLease lease = validLease(); assertEquals(JOB_RECORD_ID, lease.jobRecordId()); assertEquals(LEASE_CLAIM_ID, lease.leaseClaimId()); assertEquals(OWNER_ID, lease.leaseOwnerId()); + assertEquals(PRINCIPAL_SCOPE_HASH, lease.principalScopeHash()); + assertEquals(SUBMISSION_KEY_HASH, lease.submissionKeyHash()); + assertEquals(REQUEST_DIGEST, lease.requestDigest()); assertEquals(PAYLOAD, lease.requestPayload()); assertEquals(2, lease.attemptCount()); assertEquals(EXPIRY, lease.leaseExpiresAt()); @@ -42,38 +41,107 @@ void retainsEveryValidatedClaimField() { void rejectsMissingUnsafeOrImpossibleFields() { assertThrows( NullPointerException.class, - () -> new EtlJobLease(null, LEASE_CLAIM_ID, OWNER_ID, PAYLOAD, 1, EXPIRY) + () -> lease(null, LEASE_CLAIM_ID, OWNER_ID, PRINCIPAL_SCOPE_HASH, + SUBMISSION_KEY_HASH, REQUEST_DIGEST, PAYLOAD, 1, EXPIRY) + ); + assertThrows( + NullPointerException.class, + () -> lease(JOB_RECORD_ID, null, OWNER_ID, PRINCIPAL_SCOPE_HASH, + SUBMISSION_KEY_HASH, REQUEST_DIGEST, PAYLOAD, 1, EXPIRY) ); assertThrows( NullPointerException.class, - () -> new EtlJobLease(JOB_RECORD_ID, null, OWNER_ID, PAYLOAD, 1, EXPIRY) + () -> lease(JOB_RECORD_ID, LEASE_CLAIM_ID, null, PRINCIPAL_SCOPE_HASH, + SUBMISSION_KEY_HASH, REQUEST_DIGEST, PAYLOAD, 1, EXPIRY) + ); + assertThrows( + IllegalArgumentException.class, + () -> lease(JOB_RECORD_ID, LEASE_CLAIM_ID, "unsafe owner", + PRINCIPAL_SCOPE_HASH, SUBMISSION_KEY_HASH, REQUEST_DIGEST, + PAYLOAD, 1, EXPIRY) + ); + assertThrows( + NullPointerException.class, + () -> lease(JOB_RECORD_ID, LEASE_CLAIM_ID, OWNER_ID, null, + SUBMISSION_KEY_HASH, REQUEST_DIGEST, PAYLOAD, 1, EXPIRY) + ); + assertThrows( + IllegalArgumentException.class, + () -> lease(JOB_RECORD_ID, LEASE_CLAIM_ID, OWNER_ID, "A".repeat(64), + SUBMISSION_KEY_HASH, REQUEST_DIGEST, PAYLOAD, 1, EXPIRY) ); assertThrows( NullPointerException.class, - () -> new EtlJobLease(JOB_RECORD_ID, LEASE_CLAIM_ID, null, PAYLOAD, 1, EXPIRY) + () -> lease(JOB_RECORD_ID, LEASE_CLAIM_ID, OWNER_ID, PRINCIPAL_SCOPE_HASH, + null, REQUEST_DIGEST, PAYLOAD, 1, EXPIRY) ); assertThrows( IllegalArgumentException.class, - () -> new EtlJobLease( - JOB_RECORD_ID, - LEASE_CLAIM_ID, - "unsafe owner", - PAYLOAD, - 1, - EXPIRY - ) + () -> lease(JOB_RECORD_ID, LEASE_CLAIM_ID, OWNER_ID, PRINCIPAL_SCOPE_HASH, + "short", REQUEST_DIGEST, PAYLOAD, 1, EXPIRY) ); assertThrows( NullPointerException.class, - () -> new EtlJobLease(JOB_RECORD_ID, LEASE_CLAIM_ID, OWNER_ID, null, 1, EXPIRY) + () -> lease(JOB_RECORD_ID, LEASE_CLAIM_ID, OWNER_ID, PRINCIPAL_SCOPE_HASH, + SUBMISSION_KEY_HASH, null, PAYLOAD, 1, EXPIRY) ); assertThrows( IllegalArgumentException.class, - () -> new EtlJobLease(JOB_RECORD_ID, LEASE_CLAIM_ID, OWNER_ID, PAYLOAD, 0, EXPIRY) + () -> lease(JOB_RECORD_ID, LEASE_CLAIM_ID, OWNER_ID, PRINCIPAL_SCOPE_HASH, + SUBMISSION_KEY_HASH, "g".repeat(64), PAYLOAD, 1, EXPIRY) ); assertThrows( NullPointerException.class, - () -> new EtlJobLease(JOB_RECORD_ID, LEASE_CLAIM_ID, OWNER_ID, PAYLOAD, 1, null) + () -> lease(JOB_RECORD_ID, LEASE_CLAIM_ID, OWNER_ID, PRINCIPAL_SCOPE_HASH, + SUBMISSION_KEY_HASH, REQUEST_DIGEST, null, 1, EXPIRY) + ); + assertThrows( + IllegalArgumentException.class, + () -> lease(JOB_RECORD_ID, LEASE_CLAIM_ID, OWNER_ID, PRINCIPAL_SCOPE_HASH, + SUBMISSION_KEY_HASH, REQUEST_DIGEST, PAYLOAD, 0, EXPIRY) + ); + assertThrows( + NullPointerException.class, + () -> lease(JOB_RECORD_ID, LEASE_CLAIM_ID, OWNER_ID, PRINCIPAL_SCOPE_HASH, + SUBMISSION_KEY_HASH, REQUEST_DIGEST, PAYLOAD, 1, null) + ); + } + + private static EtlJobLease validLease() { + return lease( + JOB_RECORD_ID, + LEASE_CLAIM_ID, + OWNER_ID, + PRINCIPAL_SCOPE_HASH, + SUBMISSION_KEY_HASH, + REQUEST_DIGEST, + PAYLOAD, + 2, + EXPIRY + ); + } + + private static EtlJobLease lease( + UUID jobRecordId, + UUID leaseClaimId, + String leaseOwnerId, + String principalScopeHash, + String submissionKeyHash, + String requestDigest, + String requestPayload, + int attemptCount, + Instant leaseExpiresAt + ) { + return new EtlJobLease( + jobRecordId, + leaseClaimId, + leaseOwnerId, + principalScopeHash, + submissionKeyHash, + requestDigest, + requestPayload, + attemptCount, + leaseExpiresAt ); } } From 171670184865faf8fd32954dc558e95c56c30f03 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:58:10 +0900 Subject: [PATCH 26/92] feat(etl): carry durable execution identity in leases --- .../java/com/xtrmetl/etl/job/EtlJobLease.java | 25 ++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLease.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLease.java index 23ca99b1..321bab22 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLease.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLease.java @@ -6,11 +6,18 @@ import java.util.regex.Pattern; /** - * Carries one immutable, fenced database claim into ETL execution. + * Carries one immutable, fenced database claim into durable idempotent ETL execution. + * + *

The three lowercase SHA-256 values are non-reversible persistence identifiers copied from the + * accepted job row. They let the worker reuse the durable response ledger without retaining or + * reconstructing raw authenticated principals or raw client idempotency keys.

* * @param jobRecordId durable job identifier * @param leaseClaimId unique token generated for this exact claim or reclaim * @param leaseOwnerId non-sensitive process-lifetime worker identifier + * @param principalScopeHash SHA-256 hash of the authenticated principal namespace + * @param submissionKeyHash SHA-256 hash of the normalized durable submission key + * @param requestDigest SHA-256 digest of the exact retained request payload * @param requestPayload validated JSON payload retained while the job is non-terminal * @param attemptCount one-based claim attempt count after this claim was persisted * @param leaseExpiresAt database-derived instant after which this claim is stale @@ -19,6 +26,9 @@ public record EtlJobLease( UUID jobRecordId, UUID leaseClaimId, String leaseOwnerId, + String principalScopeHash, + String submissionKeyHash, + String requestDigest, String requestPayload, int attemptCount, Instant leaseExpiresAt @@ -27,6 +37,7 @@ public record EtlJobLease( private static final Pattern SAFE_LEASE_OWNER_PATTERN = Pattern.compile( "[A-Za-z0-9._:-]{8,128}" ); + private static final Pattern SHA256_HEX_PATTERN = Pattern.compile("[0-9a-f]{64}"); /** * Validates every field needed for exact lease fencing and deterministic execution. @@ -43,10 +54,22 @@ public record EtlJobLease( "leaseOwnerId must match [A-Za-z0-9._:-]{8,128}" ); } + requireSha256Hex(principalScopeHash, "principalScopeHash"); + requireSha256Hex(submissionKeyHash, "submissionKeyHash"); + requireSha256Hex(requestDigest, "requestDigest"); Objects.requireNonNull(requestPayload, "requestPayload must not be null"); if (attemptCount < 1) { throw new IllegalArgumentException("attemptCount must be positive"); } Objects.requireNonNull(leaseExpiresAt, "leaseExpiresAt must not be null"); } + + private static void requireSha256Hex(String value, String fieldName) { + String requiredValue = Objects.requireNonNull(value, fieldName + " must not be null"); + if (!SHA256_HEX_PATTERN.matcher(requiredValue).matches()) { + throw new IllegalArgumentException( + fieldName + " must be lowercase 64-character SHA-256 hex" + ); + } + } } From 108dc128911fafec4972091c665b68f313896ed0 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:58:56 +0900 Subject: [PATCH 27/92] feat(etl): claim durable execution identity --- .../com/xtrmetl/etl/job/EtlJobLeaseRepository.java | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java index 328bb9ca..776cb48c 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java @@ -59,6 +59,9 @@ public class EtlJobLeaseRepository { private static final String SELECT_CANDIDATE_SQL = """ SELECT job_record_id, + principal_scope_hash, + submission_key_hash, + request_digest, request_payload, attempt_count, CURRENT_TIMESTAMP AS database_now @@ -197,6 +200,9 @@ public Optional claimNext( SELECT_CANDIDATE_SQL, (resultSet, rowNumber) -> new ClaimCandidate( resultSet.getObject("job_record_id", UUID.class), + resultSet.getString("principal_scope_hash"), + resultSet.getString("submission_key_hash"), + resultSet.getString("request_digest"), resultSet.getString("request_payload"), resultSet.getInt("attempt_count"), resultSet.getObject("database_now", OffsetDateTime.class).toInstant() @@ -225,6 +231,9 @@ public Optional claimNext( candidate.jobRecordId(), leaseClaimId, validatedOwnerId, + candidate.principalScopeHash(), + candidate.submissionKeyHash(), + candidate.requestDigest(), candidate.requestPayload(), candidate.attemptCount() + 1, leaseExpiresAt @@ -343,6 +352,9 @@ private static void requireTransition(int updatedRows) { private record ClaimCandidate( UUID jobRecordId, + String principalScopeHash, + String submissionKeyHash, + String requestDigest, String requestPayload, int attemptCount, Instant databaseNow From f43d3697ba0dbbbf066fe94bd77f1fe85e9bf5d7 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:59:46 +0900 Subject: [PATCH 28/92] test(etl): provide worker execution hashes --- .../src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java index 6659fea1..9381fd8e 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java @@ -32,6 +32,9 @@ class EtlJobWorkerTest { private static final String OWNER_ID = "worker-alpha"; + private static final String PRINCIPAL_SCOPE_HASH = "a".repeat(64); + private static final String SUBMISSION_KEY_HASH = "b".repeat(64); + private static final String REQUEST_DIGEST = "c".repeat(64); private static final String PAYLOAD = "[{\"id\":\"record_alpha\"}]"; @Mock @@ -256,6 +259,9 @@ private static EtlJobLease lease(int attemptCount) { UUID.randomUUID(), UUID.randomUUID(), OWNER_ID, + PRINCIPAL_SCOPE_HASH, + SUBMISSION_KEY_HASH, + REQUEST_DIGEST, PAYLOAD, attemptCount, Instant.now().plusSeconds(300) From 960f6f66567fe814cc6f955a83dab8dd72d6bfd7 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:01:58 +0900 Subject: [PATCH 29/92] test(etl): verify claimed execution identity --- .../EtlJobLeaseRepositoryIntegrationTest.java | 86 ++++++------------- 1 file changed, 25 insertions(+), 61 deletions(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryIntegrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryIntegrationTest.java index 5fcdb5a9..f9604681 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryIntegrationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryIntegrationTest.java @@ -41,6 +41,8 @@ class EtlJobLeaseRepositoryIntegrationTest { private static final Duration LEASE_DURATION = Duration.ofMinutes(5); private static final String OWNER_ALPHA = "worker-alpha"; private static final String OWNER_BETA = "worker-beta"; + private static final String PRINCIPAL_SCOPE_HASH = "a".repeat(64); + private static final String REQUEST_DIGEST = "b".repeat(64); private static final String PAYLOAD = "[{\"id\":\"record_alpha\"}]"; private final EtlJobLeaseRepository repository; @@ -93,6 +95,9 @@ void claimsTheOldestEligibleJobAndIncrementsItsAttempt() { assertEquals(olderJobId, lease.jobRecordId()); assertEquals(OWNER_ALPHA, lease.leaseOwnerId()); + assertEquals(PRINCIPAL_SCOPE_HASH, lease.principalScopeHash()); + assertTrue(lease.submissionKeyHash().matches("[0-9a-f]{64}")); + assertEquals(REQUEST_DIGEST, lease.requestDigest()); assertEquals(PAYLOAD, lease.requestPayload()); assertEquals(2, lease.attemptCount()); assertNotNull(lease.leaseClaimId()); @@ -255,58 +260,19 @@ void rejectsExpiredSupersededOrExhaustedTransitions() { @Test void rejectsInvalidPublicArgumentsBeforeSqlExecution() { - assertThrows( - NullPointerException.class, - () -> repository.claimNext(null, LEASE_DURATION, 3) - ); - assertThrows( - NullPointerException.class, - () -> repository.claimNext(OWNER_ALPHA, null, 3) - ); - assertThrows( - IllegalArgumentException.class, - () -> repository.claimNext("short", LEASE_DURATION, 3) - ); - assertThrows( - IllegalArgumentException.class, - () -> repository.claimNext(OWNER_ALPHA, Duration.ZERO, 3) - ); - assertThrows( - IllegalArgumentException.class, - () -> repository.claimNext(OWNER_ALPHA, Duration.ofSeconds(-1), 3) - ); - assertThrows( - IllegalArgumentException.class, - () -> repository.claimNext(OWNER_ALPHA, LEASE_DURATION, 0) - ); - assertThrows( - IllegalArgumentException.class, - () -> repository.claimNext(OWNER_ALPHA, LEASE_DURATION, 101) - ); - assertThrows( - NullPointerException.class, - () -> repository.markSucceeded(null) - ); - assertThrows( - NullPointerException.class, - () -> repository.releaseForRetry(null, 3) - ); - assertThrows( - IllegalArgumentException.class, - () -> repository.releaseForRetry(sampleLease(), 0) - ); - assertThrows( - NullPointerException.class, - () -> repository.markFailed(null, "etl_target_failure") - ); - assertThrows( - NullPointerException.class, - () -> repository.markFailed(sampleLease(), null) - ); - assertThrows( - IllegalArgumentException.class, - () -> repository.markFailed(sampleLease(), "UNSAFE FAILURE") - ); + assertThrows(NullPointerException.class, () -> repository.claimNext(null, LEASE_DURATION, 3)); + assertThrows(NullPointerException.class, () -> repository.claimNext(OWNER_ALPHA, null, 3)); + assertThrows(IllegalArgumentException.class, () -> repository.claimNext("short", LEASE_DURATION, 3)); + assertThrows(IllegalArgumentException.class, () -> repository.claimNext(OWNER_ALPHA, Duration.ZERO, 3)); + assertThrows(IllegalArgumentException.class, () -> repository.claimNext(OWNER_ALPHA, Duration.ofSeconds(-1), 3)); + assertThrows(IllegalArgumentException.class, () -> repository.claimNext(OWNER_ALPHA, LEASE_DURATION, 0)); + assertThrows(IllegalArgumentException.class, () -> repository.claimNext(OWNER_ALPHA, LEASE_DURATION, 101)); + assertThrows(NullPointerException.class, () -> repository.markSucceeded(null)); + assertThrows(NullPointerException.class, () -> repository.releaseForRetry(null, 3)); + assertThrows(IllegalArgumentException.class, () -> repository.releaseForRetry(sampleLease(), 0)); + assertThrows(NullPointerException.class, () -> repository.markFailed(null, "etl_target_failure")); + assertThrows(NullPointerException.class, () -> repository.markFailed(sampleLease(), null)); + assertThrows(IllegalArgumentException.class, () -> repository.markFailed(sampleLease(), "UNSAFE FAILURE")); } private UUID insertPending(Instant createdAt, int attemptCount) { @@ -320,9 +286,9 @@ INSERT INTO etl_job_records ( ) VALUES (?, ?, ?, ?, ?, 'PENDING', ?, ?, ?) """, jobRecordId, - "a".repeat(64), + PRINCIPAL_SCOPE_HASH, UUID.randomUUID().toString().replace("-", "").repeat(2), - "b".repeat(64), + REQUEST_DIGEST, PAYLOAD, attemptCount, createdAt, @@ -365,10 +331,7 @@ INSERT INTO etl_job_records ( private void assertTerminalExhaustion(UUID jobRecordId) { assertEquals("FAILED", textColumn(jobRecordId, "job_status")); - assertEquals( - "etl_worker_attempts_exhausted", - textColumn(jobRecordId, "failure_code") - ); + assertEquals("etl_worker_attempts_exhausted", textColumn(jobRecordId, "failure_code")); assertNull(textColumn(jobRecordId, "request_payload")); assertNull(textColumn(jobRecordId, "lease_owner_id")); } @@ -395,15 +358,16 @@ private static EtlJobLease sampleLease() { UUID.randomUUID(), UUID.randomUUID(), OWNER_ALPHA, + PRINCIPAL_SCOPE_HASH, + "e".repeat(64), + REQUEST_DIGEST, PAYLOAD, 1, Instant.now().plusSeconds(300) ); } - /** - * Minimal transaction-enabled SQL context for durable lease persistence tests. - */ + /** Minimal transaction-enabled SQL context for durable lease persistence tests. */ @Configuration @EnableTransactionManagement static class TestConfiguration { From 9ac089024233c086c9b4fbd12b44e30aa5b2b245 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:02:43 +0900 Subject: [PATCH 30/92] test(etl): specify durable execution ledger --- ...lJobIdempotencyServiceIntegrationTest.java | 265 ++++++++++++++++++ 1 file changed, 265 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobIdempotencyServiceIntegrationTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobIdempotencyServiceIntegrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobIdempotencyServiceIntegrationTest.java new file mode 100644 index 00000000..2e0ed2b7 --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobIdempotencyServiceIntegrationTest.java @@ -0,0 +1,265 @@ +package com.xtrmetl.etl.job; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.xtrmetl.etl.service.EtlBatchProperties; +import com.xtrmetl.etl.service.EtlRequestLock; +import com.xtrmetl.etl.service.EtlService; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; +import org.springframework.dao.CannotAcquireLockException; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.jdbc.datasource.DataSourceTransactionManager; +import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseBuilder; +import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseType; +import org.springframework.test.context.junit.jupiter.SpringJUnitConfig; +import org.springframework.transaction.PlatformTransactionManager; +import org.springframework.transaction.annotation.EnableTransactionManagement; + +import javax.sql.DataSource; +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.time.Instant; +import java.util.HexFormat; +import java.util.UUID; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.reset; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +/** + * Proves that durable jobs reuse the response ledger without retaining raw identity values. + */ +@SpringJUnitConfig(EtlJobIdempotencyServiceIntegrationTest.TestConfiguration.class) +class EtlJobIdempotencyServiceIntegrationTest { + + private static final String OWNER_ID = "worker-alpha"; + private static final String PRINCIPAL_SCOPE_HASH = "a".repeat(64); + private static final String SUBMISSION_KEY_HASH = "b".repeat(64); + private static final String PAYLOAD = """ + [{"id":"record_alpha","name":"accepted","email":"USER@EXAMPLE.COM"}] + """; + + private final EtlJobIdempotencyService idempotencyService; + private final EtlService etlService; + private final EtlRequestLock requestLock; + private final JdbcTemplate jdbcTemplate; + + @Autowired + EtlJobIdempotencyServiceIntegrationTest( + EtlJobIdempotencyService idempotencyService, + EtlService etlService, + EtlRequestLock requestLock, + JdbcTemplate jdbcTemplate + ) { + this.idempotencyService = idempotencyService; + this.etlService = etlService; + this.requestLock = requestLock; + this.jdbcTemplate = jdbcTemplate; + } + + @BeforeEach + void createTables() { + reset(requestLock); + when(requestLock.tryLock(anyString())).thenReturn(true); + jdbcTemplate.execute("DROP TABLE IF EXISTS processed_data"); + jdbcTemplate.execute("DROP TABLE IF EXISTS etl_idempotency_records"); + jdbcTemplate.execute(""" + CREATE TABLE processed_data ( + processed_record_id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + data VARCHAR(8192) NOT NULL + ) + """); + jdbcTemplate.execute(""" + CREATE TABLE etl_idempotency_records ( + idempotency_key_hash CHAR(64) PRIMARY KEY, + request_digest CHAR(64) NOT NULL, + response_body CLOB NOT NULL, + created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP + ) + """); + } + + @Test + void writesTargetAndLedgerThenReplaysWithoutASecondTargetWrite() { + EtlJobLease lease = lease(PAYLOAD, sha256(PAYLOAD)); + + String firstResponse = idempotencyService.process(lease); + String replayedResponse = idempotencyService.process(lease); + + assertEquals("Processed: record_alpha", firstResponse); + assertEquals(firstResponse, replayedResponse); + assertEquals(1, count("processed_data")); + assertEquals(1, count("etl_idempotency_records")); + verify(requestLock, org.mockito.Mockito.times(2)).tryLock(anyString()); + } + + @Test + void rejectsPayloadDigestMismatchBeforeLockOrWrites() { + EtlJobLease lease = lease(PAYLOAD, "c".repeat(64)); + + EtlJobIntegrityException exception = assertThrows( + EtlJobIntegrityException.class, + () -> idempotencyService.process(lease) + ); + + assertEquals("etl_job_integrity_failure", exception.failureCode()); + verify(requestLock, never()).tryLock(anyString()); + assertEquals(0, count("processed_data")); + assertEquals(0, count("etl_idempotency_records")); + } + + @Test + void rejectsConflictingStoredDigestWithoutAnotherTargetWrite() { + EtlJobLease lease = lease(PAYLOAD, sha256(PAYLOAD)); + idempotencyService.process(lease); + jdbcTemplate.update( + "UPDATE etl_idempotency_records SET request_digest = ?", + "d".repeat(64) + ); + + assertThrows(EtlJobIntegrityException.class, () -> idempotencyService.process(lease)); + + assertEquals(1, count("processed_data")); + assertEquals(1, count("etl_idempotency_records")); + } + + @Test + void reportsBusyLedgerAsTransientWithoutWrites() { + when(requestLock.tryLock(anyString())).thenReturn(false); + EtlJobLease lease = lease(PAYLOAD, sha256(PAYLOAD)); + + assertThrows(CannotAcquireLockException.class, () -> idempotencyService.process(lease)); + + assertEquals(0, count("processed_data")); + assertEquals(0, count("etl_idempotency_records")); + } + + @Test + void failsClosedWithoutARealTransaction() { + EtlJobIdempotencyService directService = new EtlJobIdempotencyService( + jdbcTemplate, + etlService, + requestLock + ); + + assertThrows(IllegalStateException.class, () -> directService.process( + lease(PAYLOAD, sha256(PAYLOAD)) + )); + verify(requestLock, never()).tryLock(anyString()); + } + + @Test + void rejectsMissingCollaboratorsAndLease() { + assertThrows( + NullPointerException.class, + () -> new EtlJobIdempotencyService(null, etlService, requestLock) + ); + assertThrows( + NullPointerException.class, + () -> new EtlJobIdempotencyService(jdbcTemplate, null, requestLock) + ); + assertThrows( + NullPointerException.class, + () -> new EtlJobIdempotencyService(jdbcTemplate, etlService, null) + ); + assertThrows(NullPointerException.class, () -> idempotencyService.process(null)); + } + + private int count(String tableName) { + Integer count = jdbcTemplate.queryForObject( + "SELECT COUNT(*) FROM " + tableName, + Integer.class + ); + return count == null ? 0 : count; + } + + private static EtlJobLease lease(String payload, String requestDigest) { + return new EtlJobLease( + UUID.randomUUID(), + UUID.randomUUID(), + OWNER_ID, + PRINCIPAL_SCOPE_HASH, + SUBMISSION_KEY_HASH, + requestDigest, + payload, + 1, + Instant.now().plusSeconds(300) + ); + } + + private static String sha256(String value) { + try { + MessageDigest digest = MessageDigest.getInstance("SHA-256"); + return HexFormat.of().formatHex(digest.digest(value.getBytes(StandardCharsets.UTF_8))); + } catch (NoSuchAlgorithmException exception) { + throw new AssertionError(exception); + } + } + + /** Transaction-enabled service context backed by an isolated H2 database. */ + @Configuration + @EnableTransactionManagement + static class TestConfiguration { + + @Bean + DataSource dataSource() { + return new EmbeddedDatabaseBuilder() + .generateUniqueName(true) + .setType(EmbeddedDatabaseType.H2) + .build(); + } + + @Bean + JdbcTemplate jdbcTemplate(DataSource dataSource) { + return new JdbcTemplate(dataSource); + } + + @Bean + PlatformTransactionManager transactionManager(DataSource dataSource) { + return new DataSourceTransactionManager(dataSource); + } + + @Bean + EtlBatchProperties etlBatchProperties() { + return new EtlBatchProperties(); + } + + @Bean + ObjectMapper objectMapper() { + return new ObjectMapper(); + } + + @Bean + EtlRequestLock etlRequestLock() { + return mock(EtlRequestLock.class); + } + + @Bean + EtlService etlService( + JdbcTemplate jdbcTemplate, + ObjectMapper objectMapper, + EtlBatchProperties properties, + EtlRequestLock requestLock + ) { + return new EtlService(jdbcTemplate, objectMapper, properties, requestLock); + } + + @Bean + EtlJobIdempotencyService etlJobIdempotencyService( + JdbcTemplate jdbcTemplate, + EtlService etlService, + EtlRequestLock requestLock + ) { + return new EtlJobIdempotencyService(jdbcTemplate, etlService, requestLock); + } + } +} From dcdeb64a80c0c99723aa05d5bc33130d702a8daf Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:03:07 +0900 Subject: [PATCH 31/92] feat(etl): define durable job integrity failure --- .../etl/job/EtlJobIntegrityException.java | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobIntegrityException.java diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobIntegrityException.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobIntegrityException.java new file mode 100644 index 00000000..7aa04390 --- /dev/null +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobIntegrityException.java @@ -0,0 +1,30 @@ +package com.xtrmetl.etl.job; + +/** + * Signals that persisted durable-job execution identity no longer matches retained or ledger data. + * + *

The exception exposes only one stable machine-readable failure code and a non-sensitive + * message. It deliberately omits payloads, hashes, identifiers, SQL, timestamps, and stored + * response bodies so accidental logging does not disclose customer or operational data.

+ */ +public class EtlJobIntegrityException extends RuntimeException { + + /** Stable terminal failure code for payload or response-ledger integrity mismatches. */ + public static final String FAILURE_CODE = "etl_job_integrity_failure"; + + /** + * Creates a non-sensitive integrity failure signal. + */ + public EtlJobIntegrityException() { + super("Durable ETL job execution identity failed integrity validation"); + } + + /** + * Returns the stable machine-readable terminal failure classification. + * + * @return {@value #FAILURE_CODE} + */ + public String failureCode() { + return FAILURE_CODE; + } +} From c37618931c06167b75a6280d7bcda416d7e50fc5 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:03:19 +0900 Subject: [PATCH 32/92] feat(etl): centralize SHA-256 digesting --- .../com/xtrmetl/etl/service/Sha256Digest.java | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 etl-service/src/main/java/com/xtrmetl/etl/service/Sha256Digest.java diff --git a/etl-service/src/main/java/com/xtrmetl/etl/service/Sha256Digest.java b/etl-service/src/main/java/com/xtrmetl/etl/service/Sha256Digest.java new file mode 100644 index 00000000..0979bd6c --- /dev/null +++ b/etl-service/src/main/java/com/xtrmetl/etl/service/Sha256Digest.java @@ -0,0 +1,39 @@ +package com.xtrmetl.etl.service; + +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.util.HexFormat; +import java.util.Objects; + +/** + * Produces lowercase SHA-256 hexadecimal digests for durable ETL persistence identities. + * + *

SHA-256 is required by the Java platform. The defensive exception branch therefore indicates + * a broken runtime rather than invalid customer input.

+ */ +public final class Sha256Digest { + + private Sha256Digest() { + // Utility class. + } + + /** + * Hashes one UTF-8 string into lowercase 64-character SHA-256 hexadecimal text. + * + * @param value text to hash + * @return lowercase SHA-256 hexadecimal digest + * @throws NullPointerException when the value is {@code null} + * @throws IllegalStateException when the Java runtime lacks mandatory SHA-256 support + */ + public static String digest(String value) { + String requiredValue = Objects.requireNonNull(value, "value must not be null"); + try { + MessageDigest messageDigest = MessageDigest.getInstance("SHA-256"); + byte[] digest = messageDigest.digest(requiredValue.getBytes(StandardCharsets.UTF_8)); + return HexFormat.of().formatHex(digest); + } catch (NoSuchAlgorithmException exception) { + throw new IllegalStateException("SHA-256 is required by the Java platform", exception); + } + } +} From 262d53b43676fcbcec0e4c856b319ed519558640 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:03:40 +0900 Subject: [PATCH 33/92] feat(etl): reuse response ledger for durable jobs --- .../etl/job/EtlJobIdempotencyService.java | 136 ++++++++++++++++++ 1 file changed, 136 insertions(+) create mode 100644 etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobIdempotencyService.java diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobIdempotencyService.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobIdempotencyService.java new file mode 100644 index 00000000..aa120a49 --- /dev/null +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobIdempotencyService.java @@ -0,0 +1,136 @@ +package com.xtrmetl.etl.job; + +import com.xtrmetl.etl.service.EtlRequestLock; +import com.xtrmetl.etl.service.EtlService; +import com.xtrmetl.etl.service.Sha256Digest; +import org.springframework.dao.CannotAcquireLockException; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.stereotype.Service; +import org.springframework.transaction.annotation.Transactional; +import org.springframework.transaction.support.TransactionSynchronizationManager; + +import java.util.List; +import java.util.Objects; + +/** + * Reuses the durable ETL response ledger with hashed job identity only. + * + *

The accepted job stores independent hashes of its authenticated principal and normalized + * submission key. This service domain-separates and hashes those values into one response-ledger + * key, verifies the exact retained payload digest, serializes execution with the existing + * transaction-level request lock, replays an existing matching response, or writes the target and + * response ledger in the surrounding transaction. Raw principals and client keys are neither + * required nor reconstructed.

+ */ +@Service +public class EtlJobIdempotencyService { + + private static final String LEDGER_KEY_DOMAIN = "mightyetl:durable-job:v1:"; + private static final String SELECT_LEDGER_SQL = """ + SELECT request_digest, response_body + FROM etl_idempotency_records + WHERE idempotency_key_hash = ? + """; + private static final String INSERT_LEDGER_SQL = """ + INSERT INTO etl_idempotency_records ( + idempotency_key_hash, + request_digest, + response_body + ) VALUES (?, ?, ?) + """; + + private final JdbcTemplate jdbcTemplate; + private final EtlService etlService; + private final EtlRequestLock requestLock; + + /** + * Creates the hashed durable-job response-ledger adapter. + * + * @param jdbcTemplate parameterized response-ledger database access + * @param etlService validated ETL target writer + * @param requestLock transaction-lifetime response-ledger lock + */ + public EtlJobIdempotencyService( + JdbcTemplate jdbcTemplate, + EtlService etlService, + EtlRequestLock requestLock + ) { + this.jdbcTemplate = Objects.requireNonNull( + jdbcTemplate, + "jdbcTemplate must not be null" + ); + this.etlService = Objects.requireNonNull(etlService, "etlService must not be null"); + this.requestLock = Objects.requireNonNull(requestLock, "requestLock must not be null"); + } + + /** + * Executes or replays one durable job inside a real database transaction. + * + * @param lease exact live claim carrying hashed execution identity and retained payload + * @return newly generated or replayed stable response body + * @throws NullPointerException when the lease is {@code null} + * @throws IllegalStateException when invoked without an actual transaction + * @throws EtlJobIntegrityException when retained payload or ledger identity conflicts + * @throws CannotAcquireLockException when another transaction owns the execution ledger key + */ + @Transactional + public String process(EtlJobLease lease) { + EtlJobLease requiredLease = Objects.requireNonNull(lease, "lease must not be null"); + requireActiveTransaction(); + if (!Sha256Digest.digest(requiredLease.requestPayload()).equals( + requiredLease.requestDigest() + )) { + throw new EtlJobIntegrityException(); + } + + String ledgerKeyHash = Sha256Digest.digest( + LEDGER_KEY_DOMAIN + + requiredLease.principalScopeHash() + + ':' + + requiredLease.submissionKeyHash() + ); + if (!requestLock.tryLock(ledgerKeyHash)) { + throw new CannotAcquireLockException("Durable ETL execution ledger is busy"); + } + + List storedResponses = jdbcTemplate.query( + SELECT_LEDGER_SQL, + (resultSet, rowNumber) -> new StoredResponse( + resultSet.getString("request_digest"), + resultSet.getString("response_body") + ), + ledgerKeyHash + ); + if (!storedResponses.isEmpty()) { + StoredResponse storedResponse = storedResponses.getFirst(); + if (!storedResponse.requestDigest().equals(requiredLease.requestDigest())) { + throw new EtlJobIntegrityException(); + } + return storedResponse.responseBody(); + } + + String responseBody = etlService.processData(requiredLease.requestPayload()); + jdbcTemplate.update( + INSERT_LEDGER_SQL, + ledgerKeyHash, + requiredLease.requestDigest(), + responseBody + ); + return responseBody; + } + + private static void requireActiveTransaction() { + if (!TransactionSynchronizationManager.isActualTransactionActive()) { + throw new IllegalStateException( + "Durable ETL job execution requires an active transaction" + ); + } + } + + private record StoredResponse(String requestDigest, String responseBody) { + private StoredResponse { + Objects.requireNonNull(requestDigest, "requestDigest must not be null"); + Objects.requireNonNull(responseBody, "responseBody must not be null"); + } + } +} From b072dacbbb445a03e1a4dc06e0af6dd123f62b2d Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:03:59 +0900 Subject: [PATCH 34/92] feat(etl): commit ledger and target with lease success --- .../etl/job/EtlJobExecutionService.java | 30 +++++++++++-------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobExecutionService.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobExecutionService.java index 7286dfc1..74b0192a 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobExecutionService.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobExecutionService.java @@ -1,36 +1,39 @@ package com.xtrmetl.etl.job; -import com.xtrmetl.etl.service.EtlService; import org.springframework.stereotype.Service; import org.springframework.transaction.annotation.Transactional; import java.util.Objects; /** - * Executes one claimed ETL payload and commits terminal success in the same transaction. + * Executes one claimed ETL payload and commits ledger, target, and terminal success atomically. * - *

The existing {@link EtlService} performs validated target writes. The subsequent conditional - * success transition must match the exact unexpired claim. If that transition reports a stale - * lease, {@link StaleEtlJobLeaseException} escapes and Spring rolls back every target write made by - * the same transaction.

+ *

{@link EtlJobIdempotencyService} verifies the retained payload identity, acquires the durable + * execution-ledger lock, replays or writes the response ledger, and writes target rows. The + * subsequent conditional success transition must match the exact unexpired claim. If that + * transition reports a stale lease, {@link StaleEtlJobLeaseException} escapes and Spring rolls back + * every target and response-ledger write made by the same transaction.

*/ @Service public class EtlJobExecutionService { - private final EtlService etlService; + private final EtlJobIdempotencyService idempotencyService; private final EtlJobLeaseRepository leaseRepository; /** * Creates the atomic durable-job execution boundary. * - * @param etlService validated ETL target writer + * @param idempotencyService hashed response-ledger and target execution service * @param leaseRepository exact lease-fenced lifecycle persistence */ public EtlJobExecutionService( - EtlService etlService, + EtlJobIdempotencyService idempotencyService, EtlJobLeaseRepository leaseRepository ) { - this.etlService = Objects.requireNonNull(etlService, "etlService must not be null"); + this.idempotencyService = Objects.requireNonNull( + idempotencyService, + "idempotencyService must not be null" + ); this.leaseRepository = Objects.requireNonNull( leaseRepository, "leaseRepository must not be null" @@ -38,18 +41,19 @@ public EtlJobExecutionService( } /** - * Processes the retained payload and marks the exact live claim successful atomically. + * Processes or replays the retained job and marks the exact live claim successful atomically. * * @param lease exact database claim to execute * @throws NullPointerException when the lease is {@code null} + * @throws EtlJobIntegrityException when persisted job or ledger identity conflicts * @throws com.xtrmetl.etl.service.EtlRequestException when the retained request is invalid - * @throws org.springframework.dao.DataAccessException when a target write fails + * @throws org.springframework.dao.DataAccessException when locking or a database write fails * @throws StaleEtlJobLeaseException when the claim expires or is superseded before success */ @Transactional public void execute(EtlJobLease lease) { EtlJobLease requiredLease = Objects.requireNonNull(lease, "lease must not be null"); - etlService.processData(requiredLease.requestPayload()); + idempotencyService.process(requiredLease); leaseRepository.markSucceeded(requiredLease); } } From 081093731d59ba40d3c2fefb1472672ca6568bd9 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:04:40 +0900 Subject: [PATCH 35/92] test(etl): prove atomic ledger target and success --- ...EtlJobExecutionServiceIntegrationTest.java | 51 +++++++++++++------ 1 file changed, 36 insertions(+), 15 deletions(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobExecutionServiceIntegrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobExecutionServiceIntegrationTest.java index 3f29c0a6..de034e77 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobExecutionServiceIntegrationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobExecutionServiceIntegrationTest.java @@ -4,6 +4,7 @@ import com.xtrmetl.etl.service.EtlBatchProperties; import com.xtrmetl.etl.service.EtlRequestLock; import com.xtrmetl.etl.service.EtlService; +import com.xtrmetl.etl.service.Sha256Digest; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import org.springframework.beans.factory.annotation.Autowired; @@ -26,7 +27,7 @@ import static org.junit.jupiter.api.Assertions.assertThrows; /** - * Proves that target writes and exact-live-lease success commit or roll back together. + * Proves that ledger, target writes, and exact-live-lease success commit or roll back together. */ @SpringJUnitConfig(EtlJobExecutionServiceIntegrationTest.TestConfiguration.class) class EtlJobExecutionServiceIntegrationTest { @@ -38,25 +39,26 @@ class EtlJobExecutionServiceIntegrationTest { private final EtlJobExecutionService executionService; private final EtlJobLeaseRepository leaseRepository; - private final EtlService etlService; + private final EtlJobIdempotencyService idempotencyService; private final JdbcTemplate jdbcTemplate; @Autowired EtlJobExecutionServiceIntegrationTest( EtlJobExecutionService executionService, EtlJobLeaseRepository leaseRepository, - EtlService etlService, + EtlJobIdempotencyService idempotencyService, JdbcTemplate jdbcTemplate ) { this.executionService = executionService; this.leaseRepository = leaseRepository; - this.etlService = etlService; + this.idempotencyService = idempotencyService; this.jdbcTemplate = jdbcTemplate; } @BeforeEach void createTables() { jdbcTemplate.execute("DROP TABLE IF EXISTS processed_data"); + jdbcTemplate.execute("DROP TABLE IF EXISTS etl_idempotency_records"); jdbcTemplate.execute("DROP TABLE IF EXISTS etl_job_records"); jdbcTemplate.execute(""" CREATE TABLE etl_job_records ( @@ -81,10 +83,18 @@ CREATE TABLE processed_data ( data VARCHAR(8192) NOT NULL ) """); + jdbcTemplate.execute(""" + CREATE TABLE etl_idempotency_records ( + idempotency_key_hash CHAR(64) PRIMARY KEY, + request_digest CHAR(64) NOT NULL, + response_body CLOB NOT NULL, + created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP + ) + """); } @Test - void commitsTargetRowsAndTerminalSuccessInOneTransaction() { + void commitsLedgerTargetRowsAndTerminalSuccessInOneTransaction() { UUID jobRecordId = insertPendingJob(); EtlJobLease lease = leaseRepository.claimNext( OWNER_ID, @@ -95,7 +105,8 @@ void commitsTargetRowsAndTerminalSuccessInOneTransaction() { executionService.execute(lease); assertEquals(jobRecordId, lease.jobRecordId()); - assertEquals(1, processedRowCount()); + assertEquals(1, tableCount("processed_data")); + assertEquals(1, tableCount("etl_idempotency_records")); assertEquals( "ID:record_alpha,NAME:ACCEPTED,EMAIL:user@example.com,", jdbcTemplate.queryForObject("SELECT data FROM processed_data", String.class) @@ -105,7 +116,7 @@ void commitsTargetRowsAndTerminalSuccessInOneTransaction() { } @Test - void rollsBackTargetRowsWhenTheClaimWasSuperseded() { + void rollsBackLedgerAndTargetRowsWhenTheClaimWasSuperseded() { UUID jobRecordId = insertPendingJob(); EtlJobLease lease = leaseRepository.claimNext( OWNER_ID, @@ -120,7 +131,8 @@ void rollsBackTargetRowsWhenTheClaimWasSuperseded() { assertThrows(StaleEtlJobLeaseException.class, () -> executionService.execute(lease)); - assertEquals(0, processedRowCount()); + assertEquals(0, tableCount("processed_data")); + assertEquals(0, tableCount("etl_idempotency_records")); assertEquals("RUNNING", jobStatus(jobRecordId)); assertEquals(1, retainedPayloadCount(jobRecordId)); } @@ -133,7 +145,7 @@ void rejectsMissingCollaboratorsOrLease() { ); assertThrows( NullPointerException.class, - () -> new EtlJobExecutionService(etlService, null) + () -> new EtlJobExecutionService(idempotencyService, null) ); assertThrows(NullPointerException.class, () -> executionService.execute(null)); } @@ -152,7 +164,7 @@ INSERT INTO etl_job_records ( jobRecordId, "a".repeat(64), "b".repeat(64), - "c".repeat(64), + Sha256Digest.digest(PAYLOAD), PAYLOAD, now, now @@ -160,9 +172,9 @@ INSERT INTO etl_job_records ( return jobRecordId; } - private int processedRowCount() { + private int tableCount(String tableName) { Integer count = jdbcTemplate.queryForObject( - "SELECT COUNT(*) FROM processed_data", + "SELECT COUNT(*) FROM " + tableName, Integer.class ); return count == null ? 0 : count; @@ -191,7 +203,7 @@ SELECT COUNT(*) } /** - * Transaction-enabled execution context using one database for source state and target effects. + * Transaction-enabled execution context using one database for job, ledger, and target effects. */ @Configuration @EnableTransactionManagement @@ -240,6 +252,15 @@ EtlService etlService( return new EtlService(jdbcTemplate, objectMapper, properties, requestLock); } + @Bean + EtlJobIdempotencyService etlJobIdempotencyService( + JdbcTemplate jdbcTemplate, + EtlService etlService, + EtlRequestLock requestLock + ) { + return new EtlJobIdempotencyService(jdbcTemplate, etlService, requestLock); + } + @Bean EtlJobLeaseRepository etlJobLeaseRepository( JdbcTemplate jdbcTemplate, @@ -250,10 +271,10 @@ EtlJobLeaseRepository etlJobLeaseRepository( @Bean EtlJobExecutionService etlJobExecutionService( - EtlService etlService, + EtlJobIdempotencyService idempotencyService, EtlJobLeaseRepository leaseRepository ) { - return new EtlJobExecutionService(etlService, leaseRepository); + return new EtlJobExecutionService(idempotencyService, leaseRepository); } } } From de85194524d44b91f10720d168ad3b5f2e43e022 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:05:29 +0900 Subject: [PATCH 36/92] feat(etl): expose stable integrity failures --- etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorker.java | 2 ++ 1 file changed, 2 insertions(+) diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorker.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorker.java index d83f2537..ce8b625f 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorker.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorker.java @@ -163,6 +163,8 @@ private String runOnePoll() { return STALE_OUTCOME; } catch (TransientDataAccessException exception) { return handleTransientFailure(lease); + } catch (EtlJobIntegrityException exception) { + return markFailedOrStale(lease, exception.failureCode()); } catch (EtlRequestException exception) { return markFailedOrStale(lease, exception.error().errorCode()); } catch (DataAccessException exception) { From c3752dbacc524c4c08cd2043ab7fbc188751223c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:06:10 +0900 Subject: [PATCH 37/92] test(etl): cover integrity failure classification --- .../java/com/xtrmetl/etl/job/EtlJobWorkerTest.java | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java index 9381fd8e..4c7f0d3b 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java @@ -117,6 +117,19 @@ void terminalizesTransientFailureAtTheAttemptLimit() { assertMetric("failed", 1.0, 1L); } + @Test + void terminalizesIntegrityFailureWithItsStableCode() { + EtlJobLease lease = lease(1); + when(leaseRepository.claimNext(anyString(), any(), anyInt())) + .thenReturn(Optional.of(lease)); + doThrow(new EtlJobIntegrityException()).when(executionService).execute(lease); + + worker.pollOnce(); + + verify(leaseRepository).markFailed(lease, "etl_job_integrity_failure"); + assertMetric("failed", 1.0, 1L); + } + @Test void terminalizesDeterministicRequestFailureWithItsStableCode() { EtlJobLease lease = lease(1); From 8c31ed993bcea14238f878d5cbc3b542bd9503e9 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:07:01 +0900 Subject: [PATCH 38/92] test(etl): verify SHA-256 digest utility --- .../xtrmetl/etl/service/Sha256DigestTest.java | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/service/Sha256DigestTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/service/Sha256DigestTest.java b/etl-service/src/test/java/com/xtrmetl/etl/service/Sha256DigestTest.java new file mode 100644 index 00000000..907a3680 --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/service/Sha256DigestTest.java @@ -0,0 +1,25 @@ +package com.xtrmetl.etl.service; + +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; + +/** + * Verifies deterministic lowercase SHA-256 text identities used by durable ETL persistence. + */ +class Sha256DigestTest { + + @Test + void producesThePublishedSha256Vector() { + assertEquals( + "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad", + Sha256Digest.digest("abc") + ); + } + + @Test + void rejectsMissingInput() { + assertThrows(NullPointerException.class, () -> Sha256Digest.digest(null)); + } +} From e536c9bc377dd06396b3564aa066422c405b631d Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:07:58 +0900 Subject: [PATCH 39/92] docs: add durable job worker runbook --- docs/operations/durable-job-worker.md | 166 ++++++++++++++++++++++++++ 1 file changed, 166 insertions(+) create mode 100644 docs/operations/durable-job-worker.md diff --git a/docs/operations/durable-job-worker.md b/docs/operations/durable-job-worker.md new file mode 100644 index 00000000..04ab521a --- /dev/null +++ b/docs/operations/durable-job-worker.md @@ -0,0 +1,166 @@ +# Durable ETL job worker operations + +## Purpose and safety boundary + +The durable worker moves accepted `etl_job_records` from `PENDING` through `RUNNING` to +`SUCCEEDED` or `FAILED`. PostgreSQL row state is the distribution and fencing authority. Spring's +fixed-delay scheduler only initiates polls; it does not establish exclusivity across replicas. + +The worker is fail-closed. Both of the following product switches must be reviewed deliberately: + +```text +mightyetl.etl.jobs.intake-enabled=true +mightyetl.etl.jobs.worker.enabled=true +``` + +The supported legacy aliases are `xtrmetl.etl.jobs.intake-enabled` and +`xtrmetl.etl.jobs.worker.enabled`. Environment variables are `ETL_JOB_INTAKE_ENABLED` and +`ETL_JOB_WORKER_ENABLED`. When both full namespaces are configured, `mightyetl.*` wins. + +Enable intake without the worker only for controlled maintenance windows where retained `PENDING` +payloads are acceptable. Enable the worker without intake only to drain already accepted work. + +## Configuration + +| Preferred property | Environment variable | Default | Constraint | +| --- | --- | ---: | --- | +| `mightyetl.etl.jobs.worker.enabled` | `ETL_JOB_WORKER_ENABLED` | `false` | explicit opt-in | +| `mightyetl.etl.jobs.worker.fixed-delay-milliseconds` | `ETL_JOB_WORKER_FIXED_DELAY_MILLISECONDS` | `5000` | greater than zero | +| `mightyetl.etl.jobs.worker.initial-delay-milliseconds` | `ETL_JOB_WORKER_INITIAL_DELAY_MILLISECONDS` | `5000` | zero or greater | +| `mightyetl.etl.jobs.worker.lease-duration-seconds` | `ETL_JOB_WORKER_LEASE_DURATION_SECONDS` | `300` | greater than zero | +| `mightyetl.etl.jobs.worker.max-attempts` | `ETL_JOB_WORKER_MAX_ATTEMPTS` | `3` | 1 through 100 | +| `mightyetl.etl.jobs.worker.lease-owner-id` | deployment-specific | generated | 8–128 safe ASCII characters | + +Set an explicit `lease-owner-id` only when the deployment platform can guarantee one stable, +non-sensitive value per process. Never use a hostname containing customer data, a pod annotation +containing credentials, an email address, a tenant identifier, or a raw infrastructure token. + +Choose a lease duration longer than the normal high-percentile execution time plus database and +network variance. The current slice does not renew leases. A lease that expires during execution +causes the final success transition to fail and rolls back target and response-ledger writes. + +## Claim, execution, and recovery + +Each poll handles at most one job: + +1. Eligible rows at or above `max-attempts` become terminal `FAILED`; their payload and lease fields + are cleared with `etl_worker_attempts_exhausted`. +2. The worker selects the oldest `PENDING` row or expired `RUNNING` row below the attempt limit using + `FOR UPDATE SKIP LOCKED`. +3. The claim writes a new `lease_claim_id`, the process `lease_owner_id`, database-derived expiry, + and incremented attempt count. +4. The execution transaction verifies the retained payload digest, acquires the domain-separated + response-ledger lock, replays or writes `etl_idempotency_records`, writes target rows, and then + conditionally marks the exact live lease `SUCCEEDED`. +5. A stale, superseded, or expired lease cannot commit target rows, response-ledger rows, or terminal + state. The whole execution transaction rolls back. + +An expired `RUNNING` job is reclaimed with a new claim identifier. The earlier worker may continue +using CPU, but its target and lifecycle writes cannot commit after losing the exact live lease. + +## Stable failure codes + +| Failure code | Meaning | Operator response | +| --- | --- | --- | +| `etl_worker_attempts_exhausted` | an eligible row had no remaining claim attempt | inspect target availability and payload validity before any future replay feature | +| `etl_target_unavailable` | transient database failures consumed the attempt limit | restore database service and retain evidence for incident review | +| `etl_target_failure` | non-transient database write failure | inspect schema, constraints, permissions, and target compatibility | +| `etl_job_integrity_failure` | retained payload or response-ledger identity conflicted | stop affected workers, preserve database evidence, investigate tampering or inconsistent migration | +| `etl_internal_error` | unexpected non-database runtime failure | inspect sanitized application diagnostics and open a defect | +| existing `etl_*` request codes | retained request failed deterministic ETL validation | correct the producer or migration source; do not blindly retry | + +Terminal states clear `request_payload` in the same state transition. The status API exposes only the +stable failure code, attempt count, lifecycle state, and timestamps to the authenticated owner. + +## Observability and SLO evidence + +The worker publishes finite-cardinality metrics only: + +- `etl.jobs.worker.outcomes{outcome=idle|claimed|succeeded|retried|failed|stale}`; +- `etl.jobs.execution.duration{outcome=idle|succeeded|retried|failed|stale}`. + +Do not add payloads, raw principals, raw idempotency keys, hashes, job identifiers, lease identifiers, +SQL text, exception messages, or unbounded exception classes as metric tags or log fields. + +Recommended initial service-level indicators are: + +- accepted-to-terminal latency by status; +- oldest eligible `PENDING` age; +- expired `RUNNING` count; +- terminal success ratio; +- retry and stale outcome rates; +- exhausted-attempt and integrity-failure counts; +- database connection-pool saturation and transaction latency. + +A production SLO must be calibrated from representative load and recovery tests. Do not claim a +numerical availability or latency SLO until monitoring, alert thresholds, and retained evidence have +been validated in the buyer's deployment topology. + +For OpenTelemetry database telemetry, use the stable SQL/PostgreSQL semantic conventions where the +instrumentation supports them. Prefer low-cardinality `db.query.summary`; treat raw `db.query.text` +and query parameters as opt-in sensitive telemetry requiring a separate privacy assessment. + +## Incident procedures + +### Backlog growth + +1. Confirm intake and worker switches independently. +2. Check database connectivity, pool saturation, lock waits, and worker failure outcomes. +3. Compare oldest eligible `PENDING` age with execution duration. +4. Add replicas only after confirming the database can support the additional claim and target-write + concurrency. +5. Do not update lifecycle fields manually while workers are active. + +### Repeated stale outcomes + +1. Compare the configured lease duration with high-percentile transaction duration. +2. Check clock-independent database latency and long-running statements; lease decisions use database + time. +3. Verify every process has a safe, distinct lease owner identifier. +4. Increase the lease duration only after confirming that crash recovery delay remains acceptable. + +### Integrity failure + +1. Disable the worker while preserving intake only if continued payload retention is acceptable. +2. Snapshot the affected database under incident-response controls. +3. Compare the job's stored request digest with a digest of the retained payload and compare the + domain-separated response-ledger row. +4. Review migration, restore, replication, and unauthorized-write evidence. +5. Do not disclose hashes or payloads in tickets, chat, dashboards, or ordinary logs. + +## Deployment and rollback + +Before enabling the worker: + +1. Apply and validate Flyway migration `V3__add_etl_job_lease_fencing.sql`. +2. Confirm the application principal has only the required table and advisory-lock permissions. +3. Run migration, claim-contention, stale-lease rollback, response-replay, and target compatibility + tests against a production-equivalent PostgreSQL environment. +4. Deploy with the worker disabled, inspect health and schema evidence, then enable a canary replica. +5. Verify target, response-ledger, and terminal state atomicity before widening rollout. + +Rollback order is fail-closed: + +1. Disable intake when new accepted work must stop. +2. Disable all workers and wait for active transactions to complete or roll back. +3. Confirm no `RUNNING` rows remain; allow leases to expire if necessary. +4. Decide whether `PENDING` payloads will be drained by the current version or retained under an + approved data-retention exception. +5. Roll back application binaries before any schema compensation. +6. Never edit or delete an applied Flyway versioned migration. Use a new forward compensating + migration only after every deployed binary no longer reads the lease columns. + +## Standards and primary documentation + +Fielding, R., Nottingham, M., & Reschke, J. (2022). *HTTP semantics* (RFC 9110). RFC Editor. +https://www.rfc-editor.org/rfc/rfc9110 + +OpenTelemetry Authors. (2026). *OpenTelemetry semantic conventions 1.43.0: Semantic conventions for +SQL databases client operations*. Cloud Native Computing Foundation. +https://opentelemetry.io/docs/specs/semconv/db/sql/ + +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: SELECT*. +https://www.postgresql.org/docs/18/sql-select.html + +Spring Authors. (2026). *Task execution and scheduling*. Broadcom. +https://docs.spring.io/spring-framework/reference/integration/scheduling.html From 1b7417fb4fc53387065dbb7d035e5d9d753d8273 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:08:33 +0900 Subject: [PATCH 40/92] docs: connect durable intake to lease worker --- docs/etl/durable-job-intake.md | 193 ++++++++++++++++++--------------- 1 file changed, 108 insertions(+), 85 deletions(-) diff --git a/docs/etl/durable-job-intake.md b/docs/etl/durable-job-intake.md index 7e29a2d2..dc54f365 100644 --- a/docs/etl/durable-job-intake.md +++ b/docs/etl/durable-job-intake.md @@ -1,24 +1,25 @@ -# Durable asynchronous ETL job intake +# Durable asynchronous ETL jobs -## Scope +## Scope and activation -`POST /api/etl/jobs` creates a durable, authenticated-principal-scoped ETL job resource. This -bounded intake slice persists accepted work and exposes its status monitor; it does not execute jobs -yet. Execution, PostgreSQL `FOR UPDATE SKIP LOCKED` claiming, lease fencing, bounded attempts, -terminal payload clearing, and crash recovery belong to the following worker and lease-fencing slice. +`POST /api/etl/jobs` creates a durable, authenticated-principal-scoped ETL job resource. A separate +lease-fenced worker claims accepted jobs across replicas, replays or writes the durable response +ledger, writes target rows, and commits terminal state atomically. -The incomplete intake surface is disabled by default. It is absent unless an operator explicitly -sets the preferred `mightyetl.etl.jobs.intake-enabled=true` property, its supported legacy alias -`xtrmetl.etl.jobs.intake-enabled=true`, or `ETL_JOB_INTAKE_ENABLED=true`. When both full namespaces -are supplied, `mightyetl.*` wins. Enabling intake accepts the temporary boundary that submitted -payloads remain retained in `PENDING` jobs until the worker and terminal payload-clearing slice is -implemented. Deployments that cannot accept that retention boundary must leave the setting false. +Both capabilities are fail-closed: -The existing synchronous `POST /api/etl/process` endpoint remains unchanged. +```text +mightyetl.etl.jobs.intake-enabled=false +mightyetl.etl.jobs.worker.enabled=false +``` -## Submit a job +The supported legacy aliases use the `xtrmetl.*` namespace. Environment variables are +`ETL_JOB_INTAKE_ENABLED` and `ETL_JOB_WORKER_ENABLED`. When both full namespaces are supplied, +`mightyetl.*` wins. Intake may be enabled alone for a controlled retention window, and the worker may +be enabled alone to drain accepted work. The existing synchronous `POST /api/etl/process` endpoint +remains unchanged. -A client sends: +## Submit a job ```http POST /api/etl/jobs HTTP/1.1 @@ -29,18 +30,13 @@ Idempotency-Key: "550e8400-e29b-41d4-a716-446655440000" [{"id":"record_alpha","name":"accepted"}] ``` -The service requires: - -- the same authenticated principal for every retry; -- the same semantic `Idempotency-Key`; and -- byte-for-byte same JSON text for every retry of that key. - -The preferred header representation is an RFC 9651 quoted String. The legacy raw safe-ASCII profile -remains accepted for compatibility and normalizes to the same semantic key. +The service requires the same authenticated principal, the same semantic idempotency key, and +byte-for-byte identical JSON text for every retry. The preferred header representation is an RFC +9651 quoted String. The legacy raw safe-ASCII profile remains accepted and normalizes to the same +semantic key. -A new or replayed durable submission returns RFC 9110 `202 Accepted` because acceptance does not mean -that processing has completed. The representation describes the current state and the `Location` -header identifies the status monitor: +A new or replayed submission returns RFC 9110 `202 Accepted`; acceptance does not mean processing is +complete. The `Location` header identifies the owner-scoped status monitor: ```http HTTP/1.1 202 Accepted @@ -56,15 +52,10 @@ Content-Type: application/json } ``` -All successful and covered problem responses for durable job resources include -`Cache-Control: no-store`. These authenticated operational resources must not be retained by shared -or private caches. - -A retry that resolves to the same durable resource returns the same job identifier and -`Idempotency-Replayed: true`. Reusing the same principal-scoped key with different JSON text returns -`422 etl_job_submission_key_reused`. A concurrent creation attempt that cannot acquire the -transaction-level submission lock returns `409 etl_job_submission_in_progress` rather than waiting -without a client-visible bound. +All successful and covered problem responses include `Cache-Control: no-store`. A replay returns the +same job identifier and `Idempotency-Replayed: true`. Reusing one principal-scoped key with different +JSON returns `422 etl_job_submission_key_reused`. A concurrent creation attempt that cannot acquire +the transaction-level submission lock returns `409 etl_job_submission_in_progress`. ## Read job status @@ -73,68 +64,100 @@ GET /api/etl/jobs/{job_record_id} HTTP/1.1 Authorization: Basic ``` -The service hashes the current authenticated principal and queries by both principal scope and job -identifier. A malformed or missing identifier and an identifier owned by another principal all -return `404 etl_job_not_found`; callers cannot use this endpoint to probe another tenant's job -existence. +The query binds the current principal hash and job identifier. A malformed, missing, or foreign-owned +identifier returns the same `404 etl_job_not_found`, preventing tenant-existence probing. + +The representation exposes only the opaque job identifier, stable lifecycle state, bounded attempt +count, stable failure code where applicable, status URL, and timestamps. It excludes request payload, +raw principal, raw submission key, internal hashes, lease identifiers, SQL, and response-ledger data. + +## Lifecycle and distribution + +Flyway migrations create descriptive multi-word `snake_case` objects: -The response excludes the request payload, raw principal, raw submission key, and all internal -hashes. Timestamps are explicit ISO-8601 strings. Before worker execution is implemented, newly -accepted jobs remain `PENDING` with an `attemptCount` of zero. +- `V2__create_etl_job_records.sql` creates `etl_job_records` and the submission uniqueness contract; +- `V3__add_etl_job_lease_fencing.sql` adds `lease_claim_id`, `lease_owner_id`, + `lease_expires_at`, lifecycle constraints, and `etl_job_claim_eligibility_index`. -## Validation and persistence +The stable lifecycle is `PENDING`, `RUNNING`, `SUCCEEDED`, and `FAILED`. -Before lock or table access, mightyETL enforces the same configured UTF-8 payload and record-count -bounds used by synchronous ETL admission. The complete body must be a JSON array, duplicate JSON -fields are rejected, every element must be an object with a safe textual `id`, and normalized field -names must remain unique. +Each fixed-delay poll handles at most one job. PostgreSQL, not scheduler uniqueness, distributes work: -Flyway migration `V2__create_etl_job_records.sql` creates `etl_job_records`. All schema objects use -descriptive multi-word `snake_case` names. The database stores: +1. rows at the attempt limit are terminalized and their payloads are cleared; +2. one oldest eligible `PENDING` row or expired `RUNNING` row is selected with + `FOR UPDATE SKIP LOCKED`; +3. a fresh claim identifier, process owner identifier, database-derived expiry, and incremented + attempt count are persisted; +4. execution verifies the retained payload digest and acquires a domain-separated response-ledger + lock derived only from stored hashes; +5. an existing matching response is replayed, or target rows and `etl_idempotency_records` are written; +6. `SUCCEEDED` is committed only for the exact unexpired claim in the same transaction. -- an opaque UUID job identifier; -- SHA-256 hashes of the principal scope, semantic submission key, and exact JSON text; -- the request payload needed by the future worker while status is `PENDING` or `RUNNING`; -- status, attempt, failure, and timestamp fields. +If the lease is expired or superseded, the final transition fails and rolls back target and ledger +writes. An expired row can be reclaimed with a new claim identifier. A stale worker therefore cannot +commit duplicate target effects or terminalize a newer owner's work. -The schema reserves the stable lifecycle vocabulary `PENDING`, `RUNNING`, `SUCCEEDED`, and `FAILED`. -A database check requires a non-null request payload only for the two nonterminal states and requires -that payload to be null for both terminal states. This makes terminal payload clearing an enforced -persistence invariant rather than a documentation-only convention. +## Retry and failure behavior -Raw authenticated principal names and raw idempotency keys are never persisted. The request payload -is sensitive operational data and must inherit the classification of its source records. Until the -worker slice reaches a terminal state and clears it, operators must apply database access control, -encryption, backup, and retention policy accordingly. +Transient database failures return the job to `PENDING` while attempts remain. At the configured +limit they become `FAILED` with `etl_target_unavailable`. Non-transient database failures use +`etl_target_failure`. Persisted payload or ledger identity conflicts use +`etl_job_integrity_failure`. Unexpected runtime failures use `etl_internal_error`. Deterministic ETL +validation retains its existing stable `etl_*` request code. Eligible rows already at the attempt +limit use `etl_worker_attempts_exhausted`. -## Operational boundary +Every retry or terminal transition repeats the exact live lease predicate. A zero-row transition is +stale evidence and does not overwrite the authoritative owner. -This slice deliberately does not advertise job completion or background execution. The controller is -disabled by default; setting an activation property to `true` is an explicit operator opt-in to -durable intake without execution. Deployments that need completed asynchronous processing must wait -for the worker and lease-fencing slice. The next slice must claim jobs safely across replicas, fence -stale lease owners, commit target effects and terminal success atomically, reclaim expired leases, -bound attempts, publish stable failure codes, and clear the stored request payload at terminal state. +## Validation, privacy, and retention + +Before submission lock or table access, mightyETL enforces configured UTF-8 payload and record-count +bounds. The complete body must be a JSON array, duplicate JSON fields are rejected, every element +must be an object with a safe textual `id`, and normalized field names must remain unique. + +The database stores an opaque UUID, SHA-256 hashes of principal scope, semantic submission key, and +exact JSON text, the retained request payload while nonterminal, lifecycle and attempt fields, and +lease metadata while running. Raw principal names and raw idempotency keys are never persisted. + +The request payload inherits the source records' data classification. Database constraints require a +payload for nonterminal rows and require it to be null for terminal rows. Success, deterministic +failure, attempts exhaustion, and non-retryable failure clear the payload in their terminal +transition. Apply least privilege, encryption, backup, restore, and retention controls while data is +retained. + +Metrics and ordinary logs must not include payloads, principals, client keys, hashes, job or lease +identifiers, SQL, exception messages, or unbounded error classes. Operational procedures and metric +contracts are authoritative in `docs/operations/durable-job-worker.md`. ## Standards basis -- RFC 9110 Section 15.3.3 defines `202 Accepted` as noncommittal and recommends that the response - describe current status and point to a status monitor. -- RFC 9457 supplies the problem-details representation used by deterministic submission and lookup - failures. -- RFC 9651 defines the current Structured Fields String syntax accepted for `Idempotency-Key`. -- The expired IETF HTTPAPI `Idempotency-Key` draft-07 is used only as work-in-progress design - evidence for unique client keys, request fingerprints, `422` payload conflicts, and tenant-isolation - security concerns. It expired on April 18, 2026 and is not represented as a published RFC. +- RFC 9110 Section 15.3.3 defines `202 Accepted` as noncommittal and recommends a current-status + representation and status monitor. +- RFC 9457 supplies deterministic problem-details representations. +- RFC 9651 defines the accepted Structured Fields String syntax. +- PostgreSQL 18 documents `SKIP LOCKED` as unsuitable for a general consistent view but useful for + avoiding contention among multiple consumers of a queue-like table. +- Spring fixed-delay scheduling measures each delay from completion of the preceding invocation. +- OpenTelemetry SQL/PostgreSQL semantic conventions define stable database telemetry fields; raw + query text and parameters remain privacy-sensitive opt-in data. ### References -- Fielding, R., Nottingham, M., & Reschke, J. (2022). *HTTP semantics* (RFC 9110). RFC Editor. - https://www.rfc-editor.org/rfc/rfc9110 -- Jena, J., & Dalal, S. (2025). *The Idempotency-Key HTTP header field* - (draft-ietf-httpapi-idempotency-key-header-07, expired April 18, 2026). Internet Engineering Task - Force. https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/ -- Nottingham, M., & Wilde, E. (2023). *Problem details for HTTP APIs* (RFC 9457). RFC Editor. - https://www.rfc-editor.org/rfc/rfc9457 -- Nottingham, M., & Kamp, P. (2024). *Structured field values for HTTP* (RFC 9651). RFC Editor. - https://www.rfc-editor.org/rfc/rfc9651 +Fielding, R., Nottingham, M., & Reschke, J. (2022). *HTTP semantics* (RFC 9110). RFC Editor. +https://www.rfc-editor.org/rfc/rfc9110 + +Nottingham, M., & Wilde, E. (2023). *Problem details for HTTP APIs* (RFC 9457). RFC Editor. +https://www.rfc-editor.org/rfc/rfc9457 + +Nottingham, M., & Kamp, P. (2024). *Structured field values for HTTP* (RFC 9651). RFC Editor. +https://www.rfc-editor.org/rfc/rfc9651 + +OpenTelemetry Authors. (2026). *OpenTelemetry semantic conventions 1.43.0: Semantic conventions for +SQL databases client operations*. Cloud Native Computing Foundation. +https://opentelemetry.io/docs/specs/semconv/db/sql/ + +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: SELECT*. +https://www.postgresql.org/docs/18/sql-select.html + +Spring Authors. (2026). *Task execution and scheduling*. Broadcom. +https://docs.spring.io/spring-framework/reference/integration/scheduling.html From fcefbf428a805dad6c165d7a053b0516375f07d3 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:09:34 +0900 Subject: [PATCH 41/92] docs: align lease worker design with response ledger --- ...6-08-05-durable-job-lease-worker-design.md | 139 ++++++++++++++---- 1 file changed, 107 insertions(+), 32 deletions(-) diff --git a/docs/superpowers/specs/2026-08-05-durable-job-lease-worker-design.md b/docs/superpowers/specs/2026-08-05-durable-job-lease-worker-design.md index 2ead4c57..ee347f50 100644 --- a/docs/superpowers/specs/2026-08-05-durable-job-lease-worker-design.md +++ b/docs/superpowers/specs/2026-08-05-durable-job-lease-worker-design.md @@ -2,11 +2,17 @@ ## Status -Accepted implementation design for issue #120. This is a bounded follow-on to the durable asynchronous intake merged in PR #119 and is stacked on PR #121 until that workflow-security prerequisite reaches `develop`. +Accepted implementation design for issue #120. This is a bounded follow-on to the durable +asynchronous intake merged in PR #119 and is stacked on PR #121 until that workflow-security +prerequisite reaches `develop`. ## Product outcome -Accepted asynchronous ETL jobs must progress from `PENDING` to a terminal state without depending on the client connection or on one service replica. The worker must distribute work across replicas through PostgreSQL row locking, fence stale owners, bound retry attempts, atomically couple target effects with terminal success, clear retained payloads at terminal state, and expose only stable non-sensitive status metadata through the existing owner-scoped API. +Accepted asynchronous ETL jobs must progress from `PENDING` to a terminal state without depending on +the client connection or one service replica. The worker must distribute work across replicas through +PostgreSQL row locking, fence stale owners, bound retry attempts, atomically couple response-ledger +and target effects with terminal success, clear retained payloads at terminal state, and expose only +stable non-sensitive status metadata through the existing owner-scoped API. ## Scope @@ -17,37 +23,70 @@ This slice adds: - lease expiry and reclaim; - bounded attempts with deterministic terminal failure codes; - fixed-delay polling that is disabled by default; -- atomic ETL target writes plus conditional `SUCCEEDED` transition; +- hashed durable execution identity copied from the accepted job; +- response-ledger replay or creation, target writes, and conditional `SUCCEEDED` in one transaction; - retry and failure transitions that require the exact live lease; - finite-cardinality execution metrics; - migration, rollback, privacy, operations, and failure-recovery documentation. -Cancellation, priorities, recurring schedules, manual replay, result-body persistence, and a dead-letter user interface remain out of scope. +Cancellation, priorities, recurring schedules, manual replay, result-body exposure, and a dead-letter +user interface remain out of scope. ## Data model -Flyway migration `V3__add_etl_job_lease_fencing.sql` adds the following descriptive `snake_case` columns to `etl_job_records`: +Flyway migration `V3__add_etl_job_lease_fencing.sql` adds the following descriptive `snake_case` +columns to `etl_job_records`: - `lease_claim_id UUID` — unique token generated for every claim or reclaim; - `lease_owner_id VARCHAR(128)` — stable non-sensitive identifier for one worker process; - `lease_expires_at TIMESTAMPTZ` — database-time expiry boundary. -A lifecycle constraint requires all three lease columns for `RUNNING` rows and requires all three to be null for every other state. A failure lifecycle constraint requires `failure_code` only for `FAILED` rows. The existing terminal-payload constraint remains authoritative. An eligibility index covers `job_status`, `lease_expires_at`, `created_at`, and `job_record_id`. +A lifecycle constraint requires all three lease columns for `RUNNING` rows and requires all three to +be null for every other state. A failure lifecycle constraint requires `failure_code` only for +`FAILED` rows. The existing terminal-payload constraint remains authoritative. An eligibility index +covers `job_status`, `lease_expires_at`, `created_at`, and `job_record_id`. + +The claim also carries the accepted row's `principal_scope_hash`, `submission_key_hash`, and +`request_digest`. These independent lowercase SHA-256 values support durable execution without +persisting or reconstructing raw authenticated principals or raw client idempotency keys. ## Claim protocol `EtlJobLeaseRepository.claimNext` runs in one transaction: -1. Terminalize eligible rows whose `attempt_count` has reached the configured maximum. Clear `request_payload` and all lease columns and assign `etl_worker_attempts_exhausted`. -2. Select one `PENDING` row or one expired `RUNNING` row with `attempt_count < max_attempts`, ordered by `created_at, job_record_id`, using `FETCH FIRST 1 ROW ONLY FOR UPDATE SKIP LOCKED`. -3. Read `CURRENT_TIMESTAMP` from the database in the same statement and derive the next expiry from that database time. -4. Generate a new `lease_claim_id`, increment `attempt_count`, set `RUNNING`, set the owner and expiry, clear any prior failure code, and commit. +1. Terminalize eligible rows whose `attempt_count` has reached the configured maximum. Clear + `request_payload` and all lease columns and assign `etl_worker_attempts_exhausted`. +2. Select one `PENDING` row or one expired `RUNNING` row with `attempt_count < max_attempts`, ordered + by `created_at, job_record_id`, using `FETCH FIRST 1 ROW ONLY FOR UPDATE SKIP LOCKED`. +3. Read `CURRENT_TIMESTAMP` from the database in the same statement and derive the next expiry from + that database time. +4. Generate a new `lease_claim_id`, increment `attempt_count`, set `RUNNING`, set the owner and + expiry, clear any prior failure code, and commit. + +The scheduler does not provide uniqueness. The database row lock and state predicate are the +cross-replica authority. PostgreSQL documents `SKIP LOCKED` as suitable for avoiding contention among +multiple consumers of a queue-like table while warning that it is not a general-purpose consistent +view. That limitation is appropriate because each worker needs one exclusive claim rather than a +complete snapshot. + +## Durable idempotent execution and fencing + +`EtlJobExecutionService.execute` starts one transaction and delegates to +`EtlJobIdempotencyService` before attempting terminal success. -The scheduler does not provide uniqueness. The database row lock and state predicate are the cross-replica authority. PostgreSQL documents `SKIP LOCKED` as suitable for avoiding contention among multiple consumers of a queue-like table, while warning that it is not a general-purpose consistent view; that limitation is appropriate here because each worker needs one exclusive claim rather than a complete snapshot. +The idempotency service: -## Execution and fencing +1. requires an actual Spring transaction; +2. recomputes the SHA-256 digest of `request_payload` and compares it with the stored + `request_digest` before lock or table access; +3. domain-separates and hashes `principal_scope_hash` plus `submission_key_hash` into a response + ledger key without recovering raw identity values; +4. acquires the existing transaction-lifetime `EtlRequestLock` for that key; +5. replays a matching `etl_idempotency_records` response or calls the existing validated + `EtlService.processData` target writer and inserts the response ledger row. -`EtlJobExecutionService.execute` starts a new transaction, calls the existing `EtlService.processData` through a separate Spring bean, then conditionally transitions the job to `SUCCEEDED` only when all of the following still match: +The execution service then conditionally transitions the job to `SUCCEEDED` only when all of the +following still match: - `job_record_id`; - `job_status = 'RUNNING'`; @@ -55,55 +94,91 @@ The scheduler does not provide uniqueness. The database row lock and state predi - exact `lease_owner_id`; - `lease_expires_at > CURRENT_TIMESTAMP`. -If the conditional update affects no row, `StaleEtlJobLeaseException` is thrown. The exception rolls back the same transaction, including all target writes, so an expired or superseded worker cannot commit target effects. +If the conditional update affects no row, `StaleEtlJobLeaseException` is thrown. The exception rolls +back the same transaction, including target and response-ledger writes. An expired or superseded +worker therefore cannot commit duplicate target effects, create a misleading response ledger, or +terminalize a newer owner's job. ## Failure policy -The polling coordinator catches execution failures after the execution transaction rolls back and performs a separate exact-lease transition: +The polling coordinator catches execution failures after the execution transaction rolls back and +performs a separate exact-lease transition: -- `TransientDataAccessException`: return to `PENDING` when attempts remain; otherwise terminal `FAILED` with `etl_target_unavailable`; +- `TransientDataAccessException`: return to `PENDING` when attempts remain; otherwise terminal + `FAILED` with `etl_target_unavailable`; +- `EtlJobIntegrityException`: terminal `FAILED` with `etl_job_integrity_failure`; - `EtlRequestException`: terminal `FAILED` with the existing stable request `errorCode`; - other `DataAccessException`: terminal `FAILED` with `etl_target_failure`; - other `RuntimeException`: terminal `FAILED` with `etl_internal_error`; -- `StaleEtlJobLeaseException`: make no state change because another owner or expiry boundary is authoritative. +- `StaleEtlJobLeaseException`: make no state change because another owner or expiry boundary is + authoritative. -Every retry or failure update repeats the exact-live-lease predicate. A zero-row update is treated as stale evidence, not as success. +Every retry or failure update repeats the exact-live-lease predicate. A zero-row update is treated as +stale evidence, not as success. ## Scheduling and activation -Spring fixed-delay scheduling is used because the next delay is measured after completion of the previous invocation. `xtrmetl.etl.jobs.worker.enabled` defaults to `false`; operators must explicitly enable both intake and worker execution. Configurable values are bounded and validated: +Spring fixed-delay scheduling is used because the next delay is measured after completion of the +previous invocation. `mightyetl.etl.jobs.worker.enabled` and its supported `xtrmetl.*` alias default +to `false`. Configurable values are bounded and validated: - `fixed-delay-milliseconds` > 0; - `initial-delay-milliseconds` >= 0; - `lease-duration-seconds` > 0; - `max-attempts` between 1 and 100; -- `lease-owner-id` is 8–128 safe ASCII characters and defaults to a process-lifetime generated identifier. +- `lease-owner-id` is 8–128 safe ASCII characters and defaults to a process-lifetime generated + identifier. -One polling invocation claims at most one job. Horizontal throughput is achieved by replicas and repeated fixed-delay invocations rather than unbounded in-process fan-out. +One polling invocation claims at most one job. Horizontal throughput is achieved by replicas and +repeated fixed-delay invocations rather than unbounded in-process fan-out. ## Observability and privacy -The worker emits a duration timer and a finite outcome counter for `claimed`, `succeeded`, `retried`, `failed`, and `stale`. Metric tags never include payloads, principals, idempotency keys, job identifiers, SQL, lease identifiers, or exception messages. Logs follow the same rule. Database client instrumentation should retain the stable OpenTelemetry SQL semantic conventions and avoid opting raw query text into telemetry unless the deployment has separately assessed that exposure. +The worker emits a duration timer and a finite outcome counter for `idle`, `claimed`, `succeeded`, +`retried`, `failed`, and `stale`. Metric tags never include payloads, principals, idempotency keys, +hashes, job identifiers, SQL, lease identifiers, exception classes, or exception messages. Logs +follow the same rule. Database client instrumentation should retain stable OpenTelemetry +SQL/PostgreSQL semantic conventions and avoid opting raw query text or parameters into telemetry +unless the deployment has separately assessed that exposure. ## Testing strategy -- Migration tests enforce descriptive names, lifecycle constraints, index shape, and rollback instructions. -- Repository integration tests use H2's supported `FOR UPDATE SKIP LOCKED` syntax to prove one live claim, deterministic ordering, expiry reclaim, attempt increment, and exhaustion terminalization. -- Execution integration tests prove target rows and `SUCCEEDED` commit together and prove a stale claim rolls target writes back. -- Coordinator tests cover every failure classification, retry bound, zero-work poll, metrics outcome, and stale transition. +- Migration tests enforce descriptive names, lifecycle constraints, index shape, and rollback + instructions. +- Repository integration tests use H2's supported `FOR UPDATE SKIP LOCKED` syntax to prove one live + claim, deterministic ordering, expiry reclaim, attempt increment, execution identity, and + exhaustion terminalization. +- Idempotency integration tests prove first execution, response replay without duplicate target + writes, payload digest rejection before locking, ledger conflict rejection, transient lock + contention, and fail-closed transaction requirements. +- Execution integration tests prove target rows, response ledger, and `SUCCEEDED` commit together and + prove a stale claim rolls all three effects back. +- Coordinator tests cover every failure classification, retry bound, zero-work poll, metrics outcome, + and stale transition. - Property tests cover every validation boundary and generated owner identifier. -- Documentation and coverage policy tests require complete public Javadoc and zero missed instruction, line, method, and branch coverage for the durable-job package. +- Documentation and coverage policy tests require complete public Javadoc and zero missed + instruction, line, method, and branch coverage for the durable-job package. ## Rollback -Before application rollback, stop all workers and disable intake. Allow active leases to expire, confirm no `RUNNING` rows remain, and decide whether pending payloads will be drained or explicitly failed. Roll back the application first. The three lease columns and eligibility index may be removed only after all rows are non-running and no deployed binary reads them. Flyway versioned migrations are not edited or deleted after publication; a forward compensating migration must perform any production schema reversal. +Before application rollback, stop all workers and disable intake. Allow active leases to expire, +confirm no `RUNNING` rows remain, and decide whether pending payloads will be drained or retained +under an approved exception. Roll back the application first. The three lease columns and eligibility +index may be removed only after all rows are non-running and no deployed binary reads them. Flyway +versioned migrations are not edited or deleted after publication; a forward compensating migration +must perform any production schema reversal. ## Standards and primary documentation -Fielding, R., Nottingham, M., & Reschke, J. (2022). *HTTP semantics* (RFC 9110). Internet Engineering Task Force. https://www.rfc-editor.org/rfc/rfc9110.html +Fielding, R., Nottingham, M., & Reschke, J. (2022). *HTTP semantics* (RFC 9110). RFC Editor. +https://www.rfc-editor.org/rfc/rfc9110.html -OpenTelemetry Authors. (2026). *Semantic conventions for database calls and systems*. Cloud Native Computing Foundation. https://opentelemetry.io/docs/specs/semconv/db/ +OpenTelemetry Authors. (2026). *OpenTelemetry semantic conventions 1.43.0: Semantic conventions for +SQL databases client operations*. Cloud Native Computing Foundation. +https://opentelemetry.io/docs/specs/semconv/db/sql/ -PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: SELECT*. https://www.postgresql.org/docs/18/sql-select.html +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: SELECT*. +https://www.postgresql.org/docs/18/sql-select.html -Spring Authors. (2026). *Task execution and scheduling*. Broadcom. https://docs.spring.io/spring-framework/reference/integration/scheduling.html +Spring Authors. (2026). *Task execution and scheduling*. Broadcom. +https://docs.spring.io/spring-framework/reference/integration/scheduling.html From a727b87e1400a8131f42f0992f7213f88a48679d Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:10:13 +0900 Subject: [PATCH 42/92] docs: record lease-fenced durable execution --- CHANGELOG.md | 316 ++++++++++++--------------------------------------- 1 file changed, 75 insertions(+), 241 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 86c4082b..2666f769 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,257 +1,91 @@ # Changelog -All notable changes to this project will be documented in this file. +All notable changes to mightyETL are documented in this file. -The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), -and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses +[Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] -### Changed - -- The hourly pull-request disposition loop now requires at least one non-author approval anchored to the exact current head SHA; stale approvals, comment-only reviews, and the mere absence of requested changes cannot authorize unattended merge. -- The hourly OpenCode workflow now scopes repository write permissions to its sole maintenance job, replaces the npm installation command with the immutable OpenCode 1.18.13 Linux release archive plus pinned SHA-256 validation, and uses a removable repository-local GitHub CLI credential helper instead of storing an encoded authorization header while retaining `persist-credentials: false`. -- The managed Jackson component set now uses the patched 2.21.5 BOM, closing CVE-2026-54515, CVE-2026-59889, and GHSA-mhm7-754m-9p8w while keeping core, annotations, datatype, and module artifacts aligned. -- Durable `POST /api/etl/jobs` submissions now return RFC 9110 `202 Accepted`, a stable pending-job representation, `Location` status-monitor metadata, and explicit replay metadata without changing the synchronous `/api/etl/process` contract. The incomplete intake controller is fail-closed and requires explicit `xtrmetl.etl.jobs.intake-enabled=true` operator opt-in until worker execution and terminal payload clearing are implemented. -- Concurrent requests using the same authenticated-principal-scoped semantic idempotency key now return immediate RFC 9457 `409 etl_idempotency_request_in_progress` responses through PostgreSQL `pg_try_advisory_xact_lock`; retries after completion still replay the committed response. -- `POST /api/etl/process` now supports optional authenticated-principal-scoped `Idempotency-Key` retries with atomic target writes, durable response replay, payload-conflict rejection, and explicit replay response metadata. -- `Idempotency-Key` now prefers the quoted RFC 9651 Structured Field String representation while retaining and normalizing the legacy raw representation to the same durable ledger key. -- ETL request errors now use RFC 9457 `application/problem+json` responses with a stable `errorCode`, fixed type URI, explicit 400/401/404/409/413/422/503/500 taxonomy, and no internal exception text in client responses. -- ETL requests now enforce bounded UTF-8 payload and record-count limits, prevalidate and transform the complete batch before the first JDBC call, and commit accepted records inside one Spring transaction. -- ETL transformations now preserve comma/colon-bearing values, use locale-independent text conversion and deterministic `BigDecimal` amount formatting, and retry only transient Spring data-access failures. -- Product branding: user-facing docs and suggested image tags use **mightyETL** (formerly xtrmETL). - - Legacy Java packages (`com.xtrmetl.*`), Maven `artifactId` `xtrmETL`, and some env/topic defaults remain for compatibility. - - See `docs/rebrand-name-matrix.md`. - ### Added -- A separate fail-closed hourly OpenCode maintenance workflow pinned to OpenCode 1.18.13 and `nvidia/qwen/qwen3-coder-480b-a35b-instruct`, using only the existing `NVIDIA_NIM_API_KEY` through OpenCode's `NVIDIA_API_KEY` provider variable while preserving the independent review agent and deterministic merge-disposition workflow. -- Principal-scoped durable asynchronous ETL job intake and owner-scoped status resources, Flyway `etl_job_records` migration, deterministic replay/conflict coverage, and the explicit worker boundary in `docs/etl/durable-job-intake.md`. -- Durable idempotency ledger migration, PostgreSQL transaction advisory-lock adapter, deterministic concurrency/rollback coverage, and the operator/client contract `docs/etl/idempotent-retries.md`. -- ETL problem-details client and operator contract: `docs/api/problem-details.md`. -- Operator-configurable ETL admission limits under `mightyetl.etl.*` / `xtrmetl.etl.*`, backed by `ETL_MAX_PAYLOAD_BYTES` and `ETL_MAX_BATCH_RECORDS` environment variables with hard safety ceilings. -- ETL transaction rollback integration coverage and the operator runbook `docs/etl/bounded-atomic-batches.md`. -- Connector scaffolds (contracts + docs only): Qlik Sense, Databricks, Snowflake under `docs/connectors/` and `etl-service` SPI stubs. -- Any-to-any CDC design notes and source SPI scaffold: `docs/cdc/any-to-any-cdc.md`, `cdc-service` SPI stubs. -- CDC operations notes: `docs/cdc/ops-and-reliability.md`. -- Product upgrade progress tracker: `docs/mightyETL-product-upgrade-progress.md`. -- CDC status/sources API: `GET /api/cdc/status`, `GET /api/cdc/sources` (no secrets). -- `DebeziumChangeRecordMapper` + `CanonicalChangeRecord` (mapper unit-tested; not on live publish path). -- CDC target SPI registry (`kafka`, `jdbc-replica`) for any-to-any routing scaffold. -- `etl-service` `xtrmetl.connectors.*` disabled config keys for Databricks/Snowflake/Qlik. -- Dual-read config aliases: `mightyetl.*` preferred → `xtrmetl.*` (`MightyEtlConfigAliasEnvironmentPostProcessor`). -- Configurable replica tables (`xtrmetl.replica.tables`) for `(id,data)`-shaped tables. -- Optional CDC canonical-map counters (`xtrmetl.cdc.canonical-map-enabled`). -- ETL connector catalog API `GET /api/etl/connectors` + scaffold enable guard. -- CDC replication slot lag probe on `GET /api/cdc/status` (`ReplicationSlotProbe`). -- CDC multi-source config list + `CdcSourceFactory` (declarative; single live engine). -- `GET /api/cdc/targets` for target SPI discovery. -- Actuator `cdcEngine` health indicator (engine running + slot details). -- SPI lifecycle: `PostgresDebeziumCdcSource.start/stop` delegates to `CdcService`. -- Scaffold CDC sources: `mysql-debezium`, `sqlserver-debezium` (discovery only). -- Root POM `mightyETL` (artifactId remains `xtrmETL`). -- README honest “Supported today” matrix; compose file product-name header. +- Lease-fenced durable ETL execution across replicas with deterministic PostgreSQL + `FOR UPDATE SKIP LOCKED` claiming, process and per-claim fencing, expiry reclaim, bounded attempts, + exact-live-lease transitions, and terminal payload clearing. +- Hashed durable execution identity and domain-separated reuse of `etl_idempotency_records`, so + response replay, target writes, and `SUCCEEDED` commit atomically without retaining or + reconstructing raw principals or raw client idempotency keys. +- Stable durable-worker failure classifications, finite-cardinality outcome and duration metrics, + migration/rollback evidence, contention and stale-lease rollback tests, and the operator runbook + `docs/operations/durable-job-worker.md`. +- A separate fail-closed hourly OpenCode maintenance workflow pinned to OpenCode 1.18.13 and + `nvidia/qwen/qwen3-coder-480b-a35b-instruct`, using the existing `NVIDIA_NIM_API_KEY` through + OpenCode's `NVIDIA_API_KEY` provider variable while preserving the independent review agent and + deterministic merge-disposition workflow. +- Principal-scoped durable asynchronous ETL job intake and owner-scoped status resources, Flyway + `etl_job_records` migration, deterministic replay/conflict coverage, and the authoritative + contract `docs/etl/durable-job-intake.md`. +- Durable synchronous idempotency ledger migration, PostgreSQL transaction advisory-lock adapter, + deterministic concurrency/rollback coverage, and `docs/etl/idempotent-retries.md`. +- RFC 9457 ETL problem-details contract in `docs/api/problem-details.md`. +- Operator-configurable ETL admission limits under `mightyetl.etl.*` and supported `xtrmetl.etl.*` + aliases, backed by bounded environment variables. +- ETL transaction rollback integration coverage and `docs/etl/bounded-atomic-batches.md`. +- Connector contract and documentation scaffolds for Qlik Sense, Databricks, and Snowflake. +- Any-to-any CDC design notes, source and target SPI scaffolds, status/source/target APIs, replication + slot lag evidence, health indicators, and operations documentation. +- Product upgrade progress tracking and the preferred `mightyetl.*` configuration namespace with + supported compatibility aliases. -### Added (historical) +### Changed -- Comprehensive documentation suite (2026-01-08) - - `README.md`: Quick start guide and project overview - - `PRD.md`: Product Requirements Document with detailed specifications - - `ARCHITECTURE.md`: System architecture and technical diagrams - - `SUMMARY_KR.md`: Korean language summary - - `CHANGELOG.md`: This file +- The hourly pull-request disposition loop now requires at least one non-author approval anchored to + the exact current head SHA; stale approvals, comment-only reviews, and absence of requested changes + cannot authorize unattended merge. +- The hourly OpenCode workflow now scopes write permissions to its maintenance job, installs an + immutable checksum-pinned release archive, validates the exact archive shape, and uses a removable + repository-local GitHub CLI credential helper while retaining `persist-credentials: false`. +- The managed Jackson component set now uses the patched 2.21.5 BOM, closing CVE-2026-54515, + CVE-2026-59889, and GHSA-mhm7-754m-9p8w while keeping managed artifacts aligned. +- Durable `POST /api/etl/jobs` submissions return RFC 9110 `202 Accepted`, `Location` status-monitor + metadata, stable replay metadata, and fail-closed intake activation without changing synchronous + `/api/etl/process` behavior. +- The durable job worker and all worker configuration aliases remain disabled by default. Operators + may independently enable intake or execution for controlled drain and maintenance procedures. +- Concurrent synchronous idempotency requests use PostgreSQL transaction advisory locks and return + deterministic RFC 9457 conflict responses instead of waiting without a client-visible bound. +- `POST /api/etl/process` supports authenticated-principal-scoped idempotency keys, atomic target and + response-ledger writes, response replay, and payload-conflict rejection. +- `Idempotency-Key` prefers the RFC 9651 quoted Structured Field String representation while retaining + the normalized legacy safe-ASCII representation. +- ETL request errors use non-sensitive RFC 9457 `application/problem+json` responses with stable + error codes and explicit HTTP taxonomy. +- ETL admission validates the complete bounded UTF-8 batch before the first JDBC write, preserves + punctuation-bearing values, uses locale-independent conversion and deterministic decimal + formatting, and retries only transient data-access failures. +- User-facing documentation and recommended image tags use **mightyETL**. Legacy Java packages, + Maven artifact identifiers, and selected environment/topic defaults remain compatibility surfaces + documented in `docs/rebrand-name-matrix.md`. + +### Security + +- Durable-worker telemetry excludes payloads, raw principals, raw idempotency keys, internal hashes, + job and lease identifiers, SQL, exception messages, and unbounded exception labels. +- Exact payload-digest and response-ledger conflicts fail closed with + `etl_job_integrity_failure`; stale workers cannot commit target, ledger, or terminal-state effects. ## [1.0.0] - 2026-01-08 -### Project Documentation Initiative - -This release focuses on reverse-engineering and documenting the -existing xtrmETL platform. - -#### Added Documentation - -1. **README.md** (478 lines) - - Project overview and value proposition - - Quick start guide with prerequisites - - Service descriptions for all microservices - - Authentication flow and API examples - - Database setup scripts - - Testing instructions - - Monitoring setup with Zipkin - - Technology stack reference - - Development guidelines - -2. **PRD.md** (608 lines) - - Executive summary and product vision - - Problem statement analysis - - Solution overview with core capabilities - - Functional requirements (FR-CDC-1 through FR-GATE-1) - - Non-functional requirements (Performance, Reliability, Security, etc.) - - Complete data model specifications - - API specifications with examples - - Deployment architecture - - Use cases and scenarios - - Future enhancements roadmap - - Success metrics and KPIs - - Risk assessment and mitigation strategies - - Comprehensive glossary - -3. **ARCHITECTURE.md** (633 lines) - - High-level system architecture diagrams - - Service communication patterns (synchronous/asynchronous) - - Detailed data flow diagrams for: - - ETL processing - - CDC event capture - - Authentication flow - - Service discovery and registration - - Security architecture - - Monitoring and observability stack - - Deployment architectures (single-node and multi-node) - - Debezium integration details - - Spring Retry mechanism - - Network and port configuration - - Scalability considerations - -4. **SUMMARY_KR.md** (206 lines) - - Korean language summary for stakeholders - - Project purpose and goals - - Key features overview - - System architecture summary - - Technology stack - - Use cases - - API specifications - - Quick start guide - - Future improvements - - Technical debt assessment - -#### Project Understanding - -Through code analysis, identified the platform as: - -- **Enterprise ETL and CDC Platform** -- Microservices-based architecture using Spring Cloud -- Real-time Change Data Capture using Debezium -- Data transformation pipelines with parallel processing -- JWT-based security with role-based access control -- Event streaming via Apache Kafka -- Service discovery with Netflix Eureka -- Distributed tracing with Zipkin - -#### Key Components Documented - -1. **CDC Service** (Port 8001) - - PostgreSQL change data capture - - Debezium embedded engine - - Kafka event publishing - - Real-time monitoring capabilities - -2. **ETL Service** (Port 8000) - - JSON data processing - - Parallel record processing - - Configurable transformations - - Automatic retry mechanism - - Target database loading - -3. **Zuul Gateway** (Port 8080) - - API Gateway with routing - - JWT authentication filter - - Load balancing - - Request routing to services - -4. **Eureka Server** (Port 8761) - - Service discovery - - Service registration - - Health monitoring - -5. **Config Server** (Port 8888) - - Centralized configuration (planned) - -6. **Zipkin** (Port 9412) - - Distributed tracing - - Performance monitoring - -#### Technology Stack Documented - -- Java 25 -- Spring Boot 2.7.14 -- Spring Cloud 2021.0.8 -- Debezium 2.3.x - 2.5.x -- PostgreSQL 12+ -- Apache Kafka -- Netflix Zuul -- Netflix Eureka -- Maven - -#### Identified Technical Debt - -- Common module referenced but not implemented -- MyBatis dependencies present but unused -- Redis integration configured but not utilized -- Config Server implemented but not actively used -- Missing Spring Boot Actuator health checks - -#### Future Enhancements Documented - -- Multi-database CDC support (MySQL, Oracle, SQL Server) -- Custom transformation functions -- Data quality validation -- Web UI for configuration and monitoring -- Schema registry integration -- Dead Letter Queue for failed messages -- Enhanced metrics dashboard - -### Files Changed - -- `CHANGELOG.md` (new) -- `README.md` (new) -- `PRD.md` (new) -- `ARCHITECTURE.md` (new) -- `SUMMARY_KR.md` (new) - -### Issue Resolved - -This release addresses the GitHub issue requesting reverse-engineering of the program's purpose and PRD creation. The issue noted: "이 프로그램이 무엇을 하고 싶었던 프로그램인지 역추적하고 PRD 작성. 아마도 데이터베이스 CDC 프로그램이었던 것 같음." - -**Confirmation**: Yes, this is a database CDC (Change Data Capture) program, specifically an enterprise-grade ETL and CDC platform for real-time data integration. - -### Documentation Statistics - -- Total lines of documentation: 1,925 -- Total files created: 4 -- Total size: ~75 KB -- Languages: English (primary), Korean (summary) - -### Related Documents - -For more information, see: - -- [README.md](README.md) - Quick start guide -- [PRD.md](PRD.md) - Product Requirements Document -- [ARCHITECTURE.md](ARCHITECTURE.md) - Technical architecture -- [SUMMARY_KR.md](SUMMARY_KR.md) - Korean summary -- Original design notes (Korean) in project files - ---- - -## Notes on Versioning - -Since this is documentation work on an existing codebase: - -- Version 1.0.0 represents the first documented release -- The actual codebase existed before this documentation -- Future versions will track both code and documentation changes - -## Changelog Maintenance - -This changelog will be updated: +### Added -- When new features are added -- When bugs are fixed -- When documentation is significantly updated -- For each release or milestone +- Initial reverse-engineered product documentation: `README.md`, `PRD.md`, `ARCHITECTURE.md`, and + `SUMMARY_KR.md`. +- Baseline documentation for the Java/Spring microservice architecture, PostgreSQL ETL path, + Debezium-based CDC path, Kafka publication, service discovery, gateway routing, and Zipkin tracing. ---- +### Known baseline limitations -**Changelog Version**: 1.0 -**Last Updated**: 2026-08-04 -**Maintained By**: Development Team \ No newline at end of file +- Several connector and multi-source capabilities were documented or scaffolded rather than live. +- Config Server, Redis, and selected dependencies were present without complete production usage. +- Operational health, security, idempotency, bounded admission, and durable asynchronous execution + required the later unreleased hardening documented above. From 1c44790fc18640617a7efc6757bfb89d3115462e Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:14:28 +0900 Subject: [PATCH 43/92] docs: preserve stacked changelog history --- CHANGELOG.md | 324 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 251 insertions(+), 73 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2666f769..1b8f46be 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,91 +1,269 @@ # Changelog -All notable changes to mightyETL are documented in this file. +All notable changes to this project will be documented in this file. -The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses -[Semantic Versioning](https://semver.org/spec/v2.0.0.html). +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] -### Added +### Changed -- Lease-fenced durable ETL execution across replicas with deterministic PostgreSQL - `FOR UPDATE SKIP LOCKED` claiming, process and per-claim fencing, expiry reclaim, bounded attempts, - exact-live-lease transitions, and terminal payload clearing. -- Hashed durable execution identity and domain-separated reuse of `etl_idempotency_records`, so - response replay, target writes, and `SUCCEEDED` commit atomically without retaining or - reconstructing raw principals or raw client idempotency keys. -- Stable durable-worker failure classifications, finite-cardinality outcome and duration metrics, - migration/rollback evidence, contention and stale-lease rollback tests, and the operator runbook - `docs/operations/durable-job-worker.md`. -- A separate fail-closed hourly OpenCode maintenance workflow pinned to OpenCode 1.18.13 and - `nvidia/qwen/qwen3-coder-480b-a35b-instruct`, using the existing `NVIDIA_NIM_API_KEY` through - OpenCode's `NVIDIA_API_KEY` provider variable while preserving the independent review agent and - deterministic merge-disposition workflow. -- Principal-scoped durable asynchronous ETL job intake and owner-scoped status resources, Flyway - `etl_job_records` migration, deterministic replay/conflict coverage, and the authoritative - contract `docs/etl/durable-job-intake.md`. -- Durable synchronous idempotency ledger migration, PostgreSQL transaction advisory-lock adapter, - deterministic concurrency/rollback coverage, and `docs/etl/idempotent-retries.md`. -- RFC 9457 ETL problem-details contract in `docs/api/problem-details.md`. -- Operator-configurable ETL admission limits under `mightyetl.etl.*` and supported `xtrmetl.etl.*` - aliases, backed by bounded environment variables. -- ETL transaction rollback integration coverage and `docs/etl/bounded-atomic-batches.md`. -- Connector contract and documentation scaffolds for Qlik Sense, Databricks, and Snowflake. -- Any-to-any CDC design notes, source and target SPI scaffolds, status/source/target APIs, replication - slot lag evidence, health indicators, and operations documentation. -- Product upgrade progress tracking and the preferred `mightyetl.*` configuration namespace with - supported compatibility aliases. +- Durable asynchronous ETL jobs now progress from `PENDING` through lease-fenced execution to `SUCCEEDED` or `FAILED`; PostgreSQL owns cross-replica claiming, stale workers cannot commit target or lifecycle effects, and intake and execution remain independently fail-closed. +- The hourly pull-request disposition loop now requires at least one non-author approval anchored to the exact current head SHA; stale approvals, comment-only reviews, and the mere absence of requested changes cannot authorize unattended merge. +- The hourly OpenCode workflow now scopes repository write permissions to its sole maintenance job, replaces the npm installation command with the immutable OpenCode 1.18.13 Linux release archive plus pinned SHA-256 validation, requires exactly one regular-file archive member before private-directory extraction, rejects non-regular or symbolic-link output, and uses a removable repository-local GitHub CLI credential helper instead of storing an encoded authorization header while retaining `persist-credentials: false`. +- The hourly OpenCode workflow now uses the current free NVIDIA `deepseek-ai/deepseek-v4-pro` endpoint for long-context coding and agentic tool use instead of the deprecated Qwen3 Coder free endpoint; model or endpoint rejection fails visibly without a non-NVIDIA, partner-only, or automatic fallback. +- The managed Jackson component set now uses the patched 2.21.5 BOM, closing CVE-2026-54515, CVE-2026-59889, and GHSA-mhm7-754m-9p8w while keeping core, annotations, datatype, and module artifacts aligned. +- Durable `POST /api/etl/jobs` submissions now return RFC 9110 `202 Accepted`, a stable job representation, `Location` status-monitor metadata, and explicit replay metadata without changing the synchronous `/api/etl/process` contract. +- Concurrent requests using the same authenticated-principal-scoped semantic idempotency key now return immediate RFC 9457 `409 etl_idempotency_request_in_progress` responses through PostgreSQL `pg_try_advisory_xact_lock`; retries after completion still replay the committed response. +- `POST /api/etl/process` now supports optional authenticated-principal-scoped `Idempotency-Key` retries with atomic target writes, durable response replay, payload-conflict rejection, and explicit replay response metadata. +- `Idempotency-Key` now prefers the quoted RFC 9651 Structured Field String representation while retaining and normalizing the legacy raw representation to the same durable ledger key. +- ETL request errors now use RFC 9457 `application/problem+json` responses with a stable `errorCode`, fixed type URI, explicit 400/401/404/409/413/422/503/500 taxonomy, and no internal exception text in client responses. +- ETL requests now enforce bounded UTF-8 payload and record-count limits, prevalidate and transform the complete batch before the first JDBC call, and commit accepted records inside one Spring transaction. +- ETL transformations now preserve comma/colon-bearing values, use locale-independent text conversion and deterministic `BigDecimal` amount formatting, and retry only transient Spring data-access failures. +- Product branding: user-facing docs and suggested image tags use **mightyETL** (formerly xtrmETL). + - Legacy Java packages (`com.xtrmetl.*`), Maven `artifactId` `xtrmETL`, and some env/topic defaults remain for compatibility. + - See `docs/rebrand-name-matrix.md`. -### Changed +### Added -- The hourly pull-request disposition loop now requires at least one non-author approval anchored to - the exact current head SHA; stale approvals, comment-only reviews, and absence of requested changes - cannot authorize unattended merge. -- The hourly OpenCode workflow now scopes write permissions to its maintenance job, installs an - immutable checksum-pinned release archive, validates the exact archive shape, and uses a removable - repository-local GitHub CLI credential helper while retaining `persist-credentials: false`. -- The managed Jackson component set now uses the patched 2.21.5 BOM, closing CVE-2026-54515, - CVE-2026-59889, and GHSA-mhm7-754m-9p8w while keeping managed artifacts aligned. -- Durable `POST /api/etl/jobs` submissions return RFC 9110 `202 Accepted`, `Location` status-monitor - metadata, stable replay metadata, and fail-closed intake activation without changing synchronous - `/api/etl/process` behavior. -- The durable job worker and all worker configuration aliases remain disabled by default. Operators - may independently enable intake or execution for controlled drain and maintenance procedures. -- Concurrent synchronous idempotency requests use PostgreSQL transaction advisory locks and return - deterministic RFC 9457 conflict responses instead of waiting without a client-visible bound. -- `POST /api/etl/process` supports authenticated-principal-scoped idempotency keys, atomic target and - response-ledger writes, response replay, and payload-conflict rejection. -- `Idempotency-Key` prefers the RFC 9651 quoted Structured Field String representation while retaining - the normalized legacy safe-ASCII representation. -- ETL request errors use non-sensitive RFC 9457 `application/problem+json` responses with stable - error codes and explicit HTTP taxonomy. -- ETL admission validates the complete bounded UTF-8 batch before the first JDBC write, preserves - punctuation-bearing values, uses locale-independent conversion and deterministic decimal - formatting, and retries only transient data-access failures. -- User-facing documentation and recommended image tags use **mightyETL**. Legacy Java packages, - Maven artifact identifiers, and selected environment/topic defaults remain compatibility surfaces - documented in `docs/rebrand-name-matrix.md`. +- PostgreSQL `FOR UPDATE SKIP LOCKED` durable-job claiming, per-process and per-claim lease fencing, expiry reclaim, bounded attempts, exact-live-lease transitions, terminal payload clearing, stable failure codes, and finite-cardinality worker metrics. +- Hashed durable execution identity and domain-separated reuse of `etl_idempotency_records`, coupling response replay or creation, target writes, and terminal `SUCCEEDED` in one transaction without retaining or reconstructing raw principals or raw client idempotency keys. +- Deterministic migration, concurrency, expiry, exhaustion, response-replay, integrity, stale-lease rollback, privacy, configuration-boundary, and operator-recovery tests plus `docs/operations/durable-job-worker.md`. +- A separate fail-closed hourly OpenCode maintenance workflow pinned to OpenCode 1.18.13 and `nvidia/deepseek-ai/deepseek-v4-pro`, using only the existing `NVIDIA_NIM_API_KEY` through OpenCode's `NVIDIA_API_KEY` provider variable while preserving the independent review agent and deterministic merge-disposition workflow. +- Supply-chain doctoring evidence for checksum binding, exact archive-member and entry-type validation, private extraction, post-extraction file checks, test-first regression evidence, and rollback in `docs/doctoring/opencode-archive-extraction-evidence.md`. +- NVIDIA model-selection doctoring evidence for endpoint availability, deprecated-endpoint rejection, capability and context evidence, no-fallback semantics, test-first regression evidence, and replacement procedure in `docs/doctoring/nvidia-opencode-model-selection-evidence.md`. +- Principal-scoped durable asynchronous ETL job intake and owner-scoped status resources, Flyway `etl_job_records` migration, deterministic replay/conflict coverage, and the authoritative lifecycle contract in `docs/etl/durable-job-intake.md`. +- Durable idempotency ledger migration, PostgreSQL transaction advisory-lock adapter, deterministic concurrency/rollback coverage, and the operator/client contract `docs/etl/idempotent-retries.md`. +- ETL problem-details client and operator contract: `docs/api/problem-details.md`. +- Operator-configurable ETL admission limits under `mightyetl.etl.*` / `xtrmetl.etl.*`, backed by `ETL_MAX_PAYLOAD_BYTES` and `ETL_MAX_BATCH_RECORDS` environment variables with hard safety ceilings. +- ETL transaction rollback integration coverage and the operator runbook `docs/etl/bounded-atomic-batches.md`. +- Connector scaffolds (contracts + docs only): Qlik Sense, Databricks, Snowflake under `docs/connectors/` and `etl-service` SPI stubs. +- Any-to-any CDC design notes and source SPI scaffold: `docs/cdc/any-to-any-cdc.md`, `cdc-service` SPI stubs. +- CDC operations notes: `docs/cdc/ops-and-reliability.md`. +- Product upgrade progress tracker: `docs/mightyETL-product-upgrade-progress.md`. +- CDC status/sources API: `GET /api/cdc/status`, `GET /api/cdc/sources` (no secrets). +- `DebeziumChangeRecordMapper` + `CanonicalChangeRecord` (mapper unit-tested; not on live publish path). +- CDC target SPI registry (`kafka`, `jdbc-replica`) for any-to-any routing scaffold. +- `etl-service` `xtrmetl.connectors.*` disabled config keys for Databricks/Snowflake/Qlik. +- Dual-read config aliases: `mightyetl.*` preferred → `xtrmetl.*` (`MightyEtlConfigAliasEnvironmentPostProcessor`). +- Configurable replica tables (`xtrmetl.replica.tables`) for `(id,data)`-shaped tables. +- Optional CDC canonical-map counters (`xtrmetl.cdc.canonical-map-enabled`). +- ETL connector catalog API `GET /api/etl/connectors` + scaffold enable guard. +- CDC replication slot lag probe on `GET /api/cdc/status` (`ReplicationSlotProbe`). +- CDC multi-source config list + `CdcSourceFactory` (declarative; single live engine). +- `GET /api/cdc/targets` for target SPI discovery. +- Actuator `cdcEngine` health indicator (engine running + slot details). +- SPI lifecycle: `PostgresDebeziumCdcSource.start/stop` delegates to `CdcService`. +- Scaffold CDC sources: `mysql-debezium`, `sqlserver-debezium` (discovery only). +- Root POM `mightyETL` (artifactId remains `xtrmETL`). +- README honest “Supported today” matrix; compose file product-name header. ### Security -- Durable-worker telemetry excludes payloads, raw principals, raw idempotency keys, internal hashes, - job and lease identifiers, SQL, exception messages, and unbounded exception labels. -- Exact payload-digest and response-ledger conflicts fail closed with - `etl_job_integrity_failure`; stale workers cannot commit target, ledger, or terminal-state effects. +- Durable-worker metrics and ordinary logs exclude payloads, raw principals, raw idempotency keys, hashes, job and lease identifiers, SQL, exception messages, and unbounded exception labels. +- Retained payload or response-ledger identity conflicts fail closed with `etl_job_integrity_failure`; an expired or superseded lease rolls back target, ledger, and terminal-state effects. + +### Added (historical) + +- Comprehensive documentation suite (2026-01-08) + - `README.md`: Quick start guide and project overview + - `PRD.md`: Product Requirements Document with detailed specifications + - `ARCHITECTURE.md`: System architecture and technical diagrams + - `SUMMARY_KR.md`: Korean language summary + - `CHANGELOG.md`: This file ## [1.0.0] - 2026-01-08 -### Added +### Project Documentation Initiative + +This release focuses on reverse-engineering and documenting the +existing xtrmETL platform. + +#### Added Documentation + +1. **README.md** (478 lines) + - Project overview and value proposition + - Quick start guide with prerequisites + - Service descriptions for all microservices + - Authentication flow and API examples + - Database setup scripts + - Testing instructions + - Monitoring setup with Zipkin + - Technology stack reference + - Development guidelines + +2. **PRD.md** (608 lines) + - Executive summary and product vision + - Problem statement analysis + - Solution overview with core capabilities + - Functional requirements (FR-CDC-1 through FR-GATE-1) + - Non-functional requirements (Performance, Reliability, Security, etc.) + - Complete data model specifications + - API specifications with examples + - Deployment architecture + - Use cases and scenarios + - Future enhancements roadmap + - Success metrics and KPIs + - Risk assessment and mitigation strategies + - Comprehensive glossary + +3. **ARCHITECTURE.md** (633 lines) + - High-level system architecture diagrams + - Service communication patterns (synchronous/asynchronous) + - Detailed data flow diagrams for: + - ETL processing + - CDC event capture + - Authentication flow + - Service discovery and registration + - Security architecture + - Monitoring and observability stack + - Deployment architectures (single-node and multi-node) + - Debezium integration details + - Spring Retry mechanism + - Network and port configuration + - Scalability considerations + +4. **SUMMARY_KR.md** (206 lines) + - Korean language summary for stakeholders + - Project purpose and goals + - Key features overview + - System architecture summary + - Technology stack + - Use cases + - API specifications + - Quick start guide + - Future improvements + - Technical debt assessment + +#### Project Understanding + +Through code analysis, identified the platform as: + +- **Enterprise ETL and CDC Platform** +- Microservices-based architecture using Spring Cloud +- Real-time Change Data Capture using Debezium +- Data transformation pipelines with parallel processing +- JWT-based security with role-based access control +- Event streaming via Apache Kafka +- Service discovery with Netflix Eureka +- Distributed tracing with Zipkin + +#### Key Components Documented + +1. **CDC Service** (Port 8001) + - PostgreSQL change data capture + - Debezium embedded engine + - Kafka event publishing + - Real-time monitoring capabilities + +2. **ETL Service** (Port 8000) + - JSON data processing + - Parallel record processing + - Configurable transformations + - Automatic retry mechanism + - Target database loading + +3. **Zuul Gateway** (Port 8080) + - API Gateway with routing + - JWT authentication filter + - Load balancing + - Request routing to services + +4. **Eureka Server** (Port 8761) + - Service discovery + - Service registration + - Health monitoring + +5. **Config Server** (Port 8888) + - Centralized configuration (planned) + +6. **Zipkin** (Port 9412) + - Distributed tracing + - Performance monitoring + +#### Technology Stack Documented + +- Java 25 +- Spring Boot 2.7.14 +- Spring Cloud 2021.0.8 +- Debezium 2.3.x - 2.5.x +- PostgreSQL 12+ +- Apache Kafka +- Netflix Zuul +- Netflix Eureka +- Maven + +#### Identified Technical Debt + +- Common module referenced but not implemented +- MyBatis dependencies present but unused +- Redis integration configured but not utilized +- Config Server implemented but not actively used +- Missing Spring Boot Actuator health checks + +#### Future Enhancements Documented + +- Multi-database CDC support (MySQL, Oracle, SQL Server) +- Custom transformation functions +- Data quality validation +- Web UI for configuration and monitoring +- Schema registry integration +- Dead Letter Queue for failed messages +- Enhanced metrics dashboard + +### Files Changed + +- `CHANGELOG.md` (new) +- `README.md` (new) +- `PRD.md` (new) +- `ARCHITECTURE.md` (new) +- `SUMMARY_KR.md` (new) + +### Issue Resolved + +This release addresses the GitHub issue requesting reverse-engineering of the program's purpose and PRD creation. The issue noted: "이 프로그램이 무엇을 하고 싶었던 프로그램인지 역추적하고 PRD 작성. 아마도 데이터베이스 CDC 프로그램이었던 것 같음." + +**Confirmation**: Yes, this is a database CDC (Change Data Capture) program, specifically an enterprise-grade ETL and CDC platform for real-time data integration. + +### Documentation Statistics + +- Total lines of documentation: 1,925 +- Total files created: 4 +- Total size: ~75 KB +- Languages: English (primary), Korean (summary) + +### Related Documents + +For more information, see: + +- [README.md](README.md) - Quick start guide +- [PRD.md](PRD.md) - Product Requirements Document +- [ARCHITECTURE.md](ARCHITECTURE.md) - Technical architecture +- [SUMMARY_KR.md](SUMMARY_KR.md) - Korean summary +- Original design notes (Korean) in project files + +--- + +## Notes on Versioning + +Since this is documentation work on an existing codebase: + +- Version 1.0.0 represents the first documented release +- The actual codebase existed before this documentation +- Future versions will track both code and documentation changes + +## Changelog Maintenance + +This changelog will be updated: -- Initial reverse-engineered product documentation: `README.md`, `PRD.md`, `ARCHITECTURE.md`, and - `SUMMARY_KR.md`. -- Baseline documentation for the Java/Spring microservice architecture, PostgreSQL ETL path, - Debezium-based CDC path, Kafka publication, service discovery, gateway routing, and Zipkin tracing. +- When new features are added +- When bugs are fixed +- When documentation is significantly updated +- For each release or milestone -### Known baseline limitations +--- -- Several connector and multi-source capabilities were documented or scaffolded rather than live. -- Config Server, Redis, and selected dependencies were present without complete production usage. -- Operational health, security, idempotency, bounded admission, and durable asynchronous execution - required the later unreleased hardening documented above. +**Changelog Version**: 1.0 +**Last Updated**: 2026-08-05 +**Maintained By**: Development Team \ No newline at end of file From cafedcafafcc170fbb69a707c536afb85f4070fc Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:15:58 +0900 Subject: [PATCH 44/92] test(etl): enforce complete durable execution coverage --- etl-service/pom.xml | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/etl-service/pom.xml b/etl-service/pom.xml index 00695de2..9e0e56b5 100644 --- a/etl-service/pom.xml +++ b/etl-service/pom.xml @@ -98,12 +98,6 @@ org.jacoco jacoco-maven-plugin 0.8.15 - - - com.xtrmetl.etl.job.* - com.xtrmetl.etl.controller.EtlJobController* - - prepare-durable-job-coverage @@ -118,6 +112,13 @@ report + + + com/xtrmetl/etl/job/*.class + com/xtrmetl/etl/controller/EtlJobController*.class + com/xtrmetl/etl/service/Sha256Digest*.class + + check-durable-job-coverage @@ -126,13 +127,24 @@ check + + com/xtrmetl/etl/job/*.class + com/xtrmetl/etl/controller/EtlJobController*.class + com/xtrmetl/etl/service/Sha256Digest*.class + + + BUNDLE + + + INSTRUCTION + TOTALCOUNT + 1 + + + CLASS - - com.xtrmetl.etl.job.* - com.xtrmetl.etl.controller.EtlJobController* - INSTRUCTION From 956a392c28d1aaf04992af963300211c80892f6e Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:16:37 +0900 Subject: [PATCH 45/92] test(etl): verify complete execution coverage policy --- .../etl/job/EtlJobCoveragePolicyTest.java | 203 +++++++++++++++--- 1 file changed, 176 insertions(+), 27 deletions(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCoveragePolicyTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCoveragePolicyTest.java index 4a728d75..675498ed 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCoveragePolicyTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCoveragePolicyTest.java @@ -1,55 +1,204 @@ package com.xtrmetl.etl.job; import org.junit.jupiter.api.Test; +import org.w3c.dom.Document; +import org.w3c.dom.Element; +import org.w3c.dom.Node; +import org.w3c.dom.NodeList; +import org.xml.sax.SAXException; +import javax.xml.parsers.DocumentBuilderFactory; +import javax.xml.parsers.ParserConfigurationException; import java.io.IOException; -import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; +import java.util.HashSet; +import java.util.Set; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.assertTrue; /** - * Keeps the durable-job production slice bound to an executable 100% coverage policy. + * Keeps the durable-job production slice bound to an executable, non-empty 100% coverage policy. * - *

The policy is intentionally scoped to the production classes introduced by the durable-job - * intake slice. It requires current Java-compatible JaCoCo instrumentation and zero missed - * instructions, lines, methods, or branches while the ordinary {@code mvn test} lifecycle runs.

+ *

JaCoCo's agent instrumentation filters and Maven report filters consume different name + * forms. A plugin-wide dotted include can therefore match neither compiled class-file paths nor + * the names seen by the agent, creating a report with zero analyzed classes that still satisfies + * zero-missed rules vacuously. This contract requires unrestricted test instrumentation, + * execution-specific class-file filters, and an explicit non-empty bundle check before the + * zero-missed instruction, line, method, and branch rules can pass.

*/ class EtlJobCoveragePolicyTest { + private static final Set DURABLE_JOB_CLASS_FILES = Set.of( + "com/xtrmetl/etl/job/*.class", + "com/xtrmetl/etl/controller/EtlJobController*.class", + "com/xtrmetl/etl/service/Sha256Digest*.class" + ); + /** - * Requires the ETL module build to fail when any durable-job production path is untested. + * Requires the ETL module build to analyze at least one intended production class and fail + * when any analyzed durable-job path is untested. * * @throws IOException when the module build descriptor cannot be read + * @throws ParserConfigurationException when the JDK XML parser cannot be created + * @throws SAXException when the Maven descriptor is not well-formed XML */ @Test - void etlModuleEnforcesCompleteInstructionAndBranchCoverageForTheDurableJobSlice() - throws IOException { - String modulePom = read("etl-service/pom.xml"); - - assertTrue(modulePom.contains("jacoco-maven-plugin")); - assertTrue(modulePom.contains("0.8.15")); - assertTrue(modulePom.contains("initialize")); - assertTrue(modulePom.contains("prepare-agent")); - assertTrue(modulePom.contains("test")); - assertTrue(modulePom.contains("report")); - assertTrue(modulePom.contains("check")); - assertTrue(modulePom.contains("com.xtrmetl.etl.job.*")); - assertTrue(modulePom.contains( + void etlModuleEnforcesNonEmptyCompleteCoverageForTheDurableJobSlice() + throws IOException, ParserConfigurationException, SAXException { + Document modulePom = parseModulePom(); + Element jacocoPlugin = findPlugin(modulePom, "jacoco-maven-plugin"); + + assertEquals("0.8.15", directText(jacocoPlugin, "version")); + Element pluginConfiguration = directChild(jacocoPlugin, "configuration"); + assertTrue( + pluginConfiguration == null || directChild(pluginConfiguration, "includes") == null, + "JaCoCo includes must not be shared across agent and report goals" + ); + + Element prepareExecution = findExecution(jacocoPlugin, "prepare-durable-job-coverage"); + assertEquals("initialize", directText(prepareExecution, "phase")); + assertTrue(goalNames(prepareExecution).contains("prepare-agent")); + Element prepareConfiguration = directChild(prepareExecution, "configuration"); + assertTrue( + prepareConfiguration == null || directChild(prepareConfiguration, "includes") == null, + "The test agent must instrument all application classes; report filtering is separate" + ); + + Element reportExecution = findExecution(jacocoPlugin, "report-durable-job-coverage"); + assertEquals("test", directText(reportExecution, "phase")); + assertTrue(goalNames(reportExecution).contains("report")); + assertEquals(DURABLE_JOB_CLASS_FILES, configuredIncludes(reportExecution)); + + Element checkExecution = findExecution(jacocoPlugin, "check-durable-job-coverage"); + assertEquals("test", directText(checkExecution, "phase")); + assertTrue(goalNames(checkExecution).contains("check")); + assertEquals(DURABLE_JOB_CLASS_FILES, configuredIncludes(checkExecution)); + assertTrue(hasLimit(checkExecution, "BUNDLE", "INSTRUCTION", "TOTALCOUNT", "minimum", "1")); + + for (String counter : Set.of("INSTRUCTION", "LINE", "METHOD", "BRANCH")) { + assertTrue( + hasLimit(checkExecution, "CLASS", counter, "MISSEDCOUNT", "maximum", "0"), + () -> "Missing zero-missed class rule for " + counter + ); + } + + String serializedPom = Files.readString(projectRoot().resolve("etl-service/pom.xml")); + assertFalse(serializedPom.contains("com.xtrmetl.etl.job.*")); + assertFalse(serializedPom.contains( "com.xtrmetl.etl.controller.EtlJobController*" )); - assertTrue(modulePom.contains("INSTRUCTION")); - assertTrue(modulePom.contains("LINE")); - assertTrue(modulePom.contains("METHOD")); - assertTrue(modulePom.contains("BRANCH")); - assertTrue(modulePom.contains("MISSEDCOUNT")); - assertTrue(modulePom.contains("0")); + assertFalse(serializedPom.contains( + "com.xtrmetl.etl.service.Sha256Digest*" + )); + } + + private static Document parseModulePom() + throws ParserConfigurationException, IOException, SAXException { + DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance(); + factory.setFeature("http://apache.org/xml/features/disallow-doctype-decl", true); + factory.setFeature("http://xml.org/sax/features/external-general-entities", false); + factory.setFeature("http://xml.org/sax/features/external-parameter-entities", false); + factory.setXIncludeAware(false); + factory.setExpandEntityReferences(false); + return factory.newDocumentBuilder().parse(projectRoot().resolve("etl-service/pom.xml").toFile()); + } + + private static Element findPlugin(Document document, String artifactId) { + NodeList plugins = document.getElementsByTagName("plugin"); + for (int index = 0; index < plugins.getLength(); index++) { + Element plugin = (Element) plugins.item(index); + if (artifactId.equals(directText(plugin, "artifactId"))) { + return plugin; + } + } + throw new AssertionError("Missing Maven plugin " + artifactId); + } + + private static Element findExecution(Element plugin, String executionId) { + NodeList executions = plugin.getElementsByTagName("execution"); + for (int index = 0; index < executions.getLength(); index++) { + Element execution = (Element) executions.item(index); + if (executionId.equals(directText(execution, "id"))) { + return execution; + } + } + throw new AssertionError("Missing JaCoCo execution " + executionId); + } + + private static Set goalNames(Element execution) { + Element goals = directChild(execution, "goals"); + assertNotNull(goals, "Every JaCoCo execution must declare goals"); + return directTexts(goals, "goal"); + } + + private static Set configuredIncludes(Element execution) { + Element configuration = directChild(execution, "configuration"); + assertNotNull(configuration, "Report and check executions require explicit configuration"); + Element includes = directChild(configuration, "includes"); + assertNotNull(includes, "Report and check executions require class-file include patterns"); + return directTexts(includes, "include"); + } + + private static boolean hasLimit( + Element execution, + String elementName, + String counter, + String value, + String boundName, + String boundValue + ) { + Element configuration = directChild(execution, "configuration"); + assertNotNull(configuration); + NodeList rules = configuration.getElementsByTagName("rule"); + for (int ruleIndex = 0; ruleIndex < rules.getLength(); ruleIndex++) { + Element rule = (Element) rules.item(ruleIndex); + if (!elementName.equals(directText(rule, "element"))) { + continue; + } + NodeList limits = rule.getElementsByTagName("limit"); + for (int limitIndex = 0; limitIndex < limits.getLength(); limitIndex++) { + Element limit = (Element) limits.item(limitIndex); + if (counter.equals(directText(limit, "counter")) + && value.equals(directText(limit, "value")) + && boundValue.equals(directText(limit, boundName))) { + return true; + } + } + } + return false; + } + + private static Set directTexts(Element parent, String childName) { + Set values = new HashSet<>(); + NodeList children = parent.getChildNodes(); + for (int index = 0; index < children.getLength(); index++) { + Node child = children.item(index); + if (child instanceof Element element && childName.equals(element.getTagName())) { + values.add(element.getTextContent().trim()); + } + } + return Set.copyOf(values); } - private static String read(String relativePath) throws IOException { - return Files.readString(projectRoot().resolve(relativePath), StandardCharsets.UTF_8); + private static String directText(Element parent, String childName) { + Element child = directChild(parent, childName); + return child == null ? null : child.getTextContent().trim(); + } + + private static Element directChild(Element parent, String childName) { + NodeList children = parent.getChildNodes(); + for (int index = 0; index < children.getLength(); index++) { + Node child = children.item(index); + if (child instanceof Element element && childName.equals(element.getTagName())) { + return element; + } + } + return null; } private static Path projectRoot() { From 5090dfaf02ad3e442b831b49980d786d7c4a49ac Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:21:19 +0900 Subject: [PATCH 46/92] test(etl): specify SHA-256 failure coverage --- .../xtrmetl/etl/service/Sha256DigestTest.java | 34 +++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/service/Sha256DigestTest.java b/etl-service/src/test/java/com/xtrmetl/etl/service/Sha256DigestTest.java index 907a3680..55401a48 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/service/Sha256DigestTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/service/Sha256DigestTest.java @@ -2,11 +2,15 @@ import org.junit.jupiter.api.Test; +import java.lang.reflect.Constructor; +import java.security.NoSuchAlgorithmException; + import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; import static org.junit.jupiter.api.Assertions.assertThrows; /** - * Verifies deterministic lowercase SHA-256 text identities used by durable ETL persistence. + * Verifies deterministic lowercase SHA-256 text identities and fail-closed runtime handling. */ class Sha256DigestTest { @@ -19,7 +23,33 @@ void producesThePublishedSha256Vector() { } @Test - void rejectsMissingInput() { + void rejectsMissingInputOrFactory() { assertThrows(NullPointerException.class, () -> Sha256Digest.digest(null)); + assertThrows( + NullPointerException.class, + () -> Sha256Digest.digest("abc", null) + ); + } + + @Test + void convertsMissingMandatoryAlgorithmIntoBrokenRuntimeSignal() { + NoSuchAlgorithmException missingAlgorithm = new NoSuchAlgorithmException("missing"); + + IllegalStateException exception = assertThrows( + IllegalStateException.class, + () -> Sha256Digest.digest("abc", () -> { + throw missingAlgorithm; + }) + ); + + assertEquals("SHA-256 is required by the Java platform", exception.getMessage()); + assertInstanceOf(NoSuchAlgorithmException.class, exception.getCause()); + } + + @Test + void utilityConstructorCannotBeCalledNormallyButRemainsCovered() throws Exception { + Constructor constructor = Sha256Digest.class.getDeclaredConstructor(); + constructor.setAccessible(true); + constructor.newInstance(); } } From 02353f36cb92428bc1e5007e014ffbc9a26ef5fe Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:21:32 +0900 Subject: [PATCH 47/92] feat(etl): make digest failure path testable --- .../com/xtrmetl/etl/service/Sha256Digest.java | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/etl-service/src/main/java/com/xtrmetl/etl/service/Sha256Digest.java b/etl-service/src/main/java/com/xtrmetl/etl/service/Sha256Digest.java index 0979bd6c..4603b431 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/service/Sha256Digest.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/service/Sha256Digest.java @@ -27,13 +27,27 @@ private Sha256Digest() { * @throws IllegalStateException when the Java runtime lacks mandatory SHA-256 support */ public static String digest(String value) { + return digest(value, () -> MessageDigest.getInstance("SHA-256")); + } + + static String digest(String value, MessageDigestFactory messageDigestFactory) { String requiredValue = Objects.requireNonNull(value, "value must not be null"); + MessageDigestFactory requiredFactory = Objects.requireNonNull( + messageDigestFactory, + "messageDigestFactory must not be null" + ); try { - MessageDigest messageDigest = MessageDigest.getInstance("SHA-256"); + MessageDigest messageDigest = requiredFactory.create(); byte[] digest = messageDigest.digest(requiredValue.getBytes(StandardCharsets.UTF_8)); return HexFormat.of().formatHex(digest); } catch (NoSuchAlgorithmException exception) { throw new IllegalStateException("SHA-256 is required by the Java platform", exception); } } + + @FunctionalInterface + interface MessageDigestFactory { + + MessageDigest create() throws NoSuchAlgorithmException; + } } From e02e87369cc92e4a098a97b95ccf2e1bfd49f92f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:22:11 +0900 Subject: [PATCH 48/92] test(etl): compare expanded coverage includes deterministically --- .../com/xtrmetl/etl/job/EtlJobCoveragePolicyTest.java | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCoveragePolicyTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCoveragePolicyTest.java index 675498ed..74fa5e0a 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCoveragePolicyTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCoveragePolicyTest.java @@ -72,12 +72,12 @@ void etlModuleEnforcesNonEmptyCompleteCoverageForTheDurableJobSlice() Element reportExecution = findExecution(jacocoPlugin, "report-durable-job-coverage"); assertEquals("test", directText(reportExecution, "phase")); assertTrue(goalNames(reportExecution).contains("report")); - assertEquals(DURABLE_JOB_CLASS_FILES, configuredIncludes(reportExecution)); + assertConfiguredIncludes(reportExecution); Element checkExecution = findExecution(jacocoPlugin, "check-durable-job-coverage"); assertEquals("test", directText(checkExecution, "phase")); assertTrue(goalNames(checkExecution).contains("check")); - assertEquals(DURABLE_JOB_CLASS_FILES, configuredIncludes(checkExecution)); + assertConfiguredIncludes(checkExecution); assertTrue(hasLimit(checkExecution, "BUNDLE", "INSTRUCTION", "TOTALCOUNT", "minimum", "1")); for (String counter : Set.of("INSTRUCTION", "LINE", "METHOD", "BRANCH")) { @@ -97,6 +97,13 @@ void etlModuleEnforcesNonEmptyCompleteCoverageForTheDurableJobSlice() )); } + private static void assertConfiguredIncludes(Element execution) { + assertEquals( + DURABLE_JOB_CLASS_FILES.stream().sorted().toList(), + configuredIncludes(execution).stream().sorted().toList() + ); + } + private static Document parseModulePom() throws ParserConfigurationException, IOException, SAXException { DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance(); From 92bc0d27db52283d64bafdf8d1bc236409d2dd97 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 12:05:11 +0900 Subject: [PATCH 49/92] test(etl): reject unbounded worker timing configuration --- .../etl/job/EtlJobWorkerPropertiesTest.java | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerPropertiesTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerPropertiesTest.java index 603eef51..8a5239f3 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerPropertiesTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerPropertiesTest.java @@ -13,6 +13,9 @@ */ class EtlJobWorkerPropertiesTest { + private static final long MAXIMUM_SCHEDULER_DELAY_MILLISECONDS = 86_400_000L; + private static final long MAXIMUM_LEASE_DURATION_SECONDS = 86_400L; + @Test void defaultsToDisabledBoundedPollingWithGeneratedSafeOwner() { EtlJobWorkerProperties properties = new EtlJobWorkerProperties(); @@ -46,8 +49,14 @@ void acceptsEverySupportedBoundary() { assertEquals(1, properties.getMaxAttempts()); assertEquals("worker-01", properties.getLeaseOwnerId()); + properties.setFixedDelayMilliseconds(MAXIMUM_SCHEDULER_DELAY_MILLISECONDS); + properties.setInitialDelayMilliseconds(MAXIMUM_SCHEDULER_DELAY_MILLISECONDS); + properties.setLeaseDurationSeconds(MAXIMUM_LEASE_DURATION_SECONDS); properties.setMaxAttempts(100); properties.setLeaseOwnerId("w".repeat(128)); + assertEquals(MAXIMUM_SCHEDULER_DELAY_MILLISECONDS, properties.getFixedDelayMilliseconds()); + assertEquals(MAXIMUM_SCHEDULER_DELAY_MILLISECONDS, properties.getInitialDelayMilliseconds()); + assertEquals(MAXIMUM_LEASE_DURATION_SECONDS, properties.getLeaseDurationSeconds()); assertEquals(100, properties.getMaxAttempts()); assertEquals(128, properties.getLeaseOwnerId().length()); } @@ -60,14 +69,30 @@ void rejectsUnsafeNumericConfiguration() { IllegalArgumentException.class, () -> properties.setFixedDelayMilliseconds(0L) ); + assertThrows( + IllegalArgumentException.class, + () -> properties.setFixedDelayMilliseconds( + MAXIMUM_SCHEDULER_DELAY_MILLISECONDS + 1L + ) + ); assertThrows( IllegalArgumentException.class, () -> properties.setInitialDelayMilliseconds(-1L) ); + assertThrows( + IllegalArgumentException.class, + () -> properties.setInitialDelayMilliseconds( + MAXIMUM_SCHEDULER_DELAY_MILLISECONDS + 1L + ) + ); assertThrows( IllegalArgumentException.class, () -> properties.setLeaseDurationSeconds(0L) ); + assertThrows( + IllegalArgumentException.class, + () -> properties.setLeaseDurationSeconds(MAXIMUM_LEASE_DURATION_SECONDS + 1L) + ); assertThrows(IllegalArgumentException.class, () -> properties.setMaxAttempts(0)); assertThrows(IllegalArgumentException.class, () -> properties.setMaxAttempts(101)); } From 8d243acca81e6f0084450f47753794ce842953c7 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 12:05:50 +0900 Subject: [PATCH 50/92] fix(etl): bound worker delays and lease duration --- .../etl/job/EtlJobWorkerProperties.java | 58 +++++++++++++------ 1 file changed, 39 insertions(+), 19 deletions(-) diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorkerProperties.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorkerProperties.java index 64ff386b..03047429 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorkerProperties.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorkerProperties.java @@ -9,14 +9,22 @@ /** * Holds bounded, fail-closed configuration for durable ETL job execution. * - *

The worker is disabled unless an operator explicitly enables it. A process-lifetime lease - * owner identifier is generated when no external value is supplied. The identifier is deliberately - * restricted to a short safe ASCII profile because it is persisted as operational metadata and - * must never become a free-form log or database injection surface.

+ *

The worker is disabled unless an operator explicitly enables it. Polling delays and lease + * durations are capped at one day so a malformed environment value cannot create effectively + * permanent scheduling gaps, arithmetic overflow, or a lease that prevents timely crash recovery. + * A process-lifetime lease owner identifier is generated when no external value is supplied. The + * identifier is deliberately restricted to a short safe ASCII profile because it is persisted as + * operational metadata and must never become a free-form log or database injection surface.

*/ @ConfigurationProperties(prefix = "xtrmetl.etl.jobs.worker") public class EtlJobWorkerProperties { + /** Maximum supported fixed or initial scheduler delay: one day in milliseconds. */ + public static final long MAXIMUM_SCHEDULER_DELAY_MILLISECONDS = 86_400_000L; + + /** Maximum supported durable-job lease duration: one day in seconds. */ + public static final long MAXIMUM_LEASE_DURATION_SECONDS = 86_400L; + private static final Pattern SAFE_LEASE_OWNER_PATTERN = Pattern.compile( "[A-Za-z0-9._:-]{8,128}" ); @@ -56,7 +64,7 @@ public void setEnabled(boolean enabled) { /** * Returns the delay measured after one polling invocation completes. * - * @return positive fixed delay in milliseconds + * @return fixed delay from one millisecond through one day */ public long getFixedDelayMilliseconds() { return fixedDelayMilliseconds; @@ -65,12 +73,16 @@ public long getFixedDelayMilliseconds() { /** * Sets the delay measured after one polling invocation completes. * - * @param fixedDelayMilliseconds positive fixed delay in milliseconds - * @throws IllegalArgumentException when the delay is zero or negative + * @param fixedDelayMilliseconds delay from one millisecond through one day + * @throws IllegalArgumentException when the delay is outside the supported range */ public void setFixedDelayMilliseconds(long fixedDelayMilliseconds) { - if (fixedDelayMilliseconds <= 0L) { - throw new IllegalArgumentException("fixedDelayMilliseconds must be positive"); + if (fixedDelayMilliseconds < 1L + || fixedDelayMilliseconds > MAXIMUM_SCHEDULER_DELAY_MILLISECONDS) { + throw new IllegalArgumentException( + "fixedDelayMilliseconds must be between 1 and " + + MAXIMUM_SCHEDULER_DELAY_MILLISECONDS + ); } this.fixedDelayMilliseconds = fixedDelayMilliseconds; } @@ -78,7 +90,7 @@ public void setFixedDelayMilliseconds(long fixedDelayMilliseconds) { /** * Returns the delay before the first polling invocation after application startup. * - * @return non-negative initial delay in milliseconds + * @return initial delay from zero milliseconds through one day */ public long getInitialDelayMilliseconds() { return initialDelayMilliseconds; @@ -87,12 +99,16 @@ public long getInitialDelayMilliseconds() { /** * Sets the delay before the first polling invocation after application startup. * - * @param initialDelayMilliseconds non-negative initial delay in milliseconds - * @throws IllegalArgumentException when the delay is negative + * @param initialDelayMilliseconds delay from zero milliseconds through one day + * @throws IllegalArgumentException when the delay is outside the supported range */ public void setInitialDelayMilliseconds(long initialDelayMilliseconds) { - if (initialDelayMilliseconds < 0L) { - throw new IllegalArgumentException("initialDelayMilliseconds must not be negative"); + if (initialDelayMilliseconds < 0L + || initialDelayMilliseconds > MAXIMUM_SCHEDULER_DELAY_MILLISECONDS) { + throw new IllegalArgumentException( + "initialDelayMilliseconds must be between 0 and " + + MAXIMUM_SCHEDULER_DELAY_MILLISECONDS + ); } this.initialDelayMilliseconds = initialDelayMilliseconds; } @@ -100,7 +116,7 @@ public void setInitialDelayMilliseconds(long initialDelayMilliseconds) { /** * Returns how long one database claim remains valid without renewal. * - * @return positive lease duration in seconds + * @return lease duration from one second through one day */ public long getLeaseDurationSeconds() { return leaseDurationSeconds; @@ -109,12 +125,16 @@ public long getLeaseDurationSeconds() { /** * Sets how long one database claim remains valid without renewal. * - * @param leaseDurationSeconds positive lease duration in seconds - * @throws IllegalArgumentException when the duration is zero or negative + * @param leaseDurationSeconds duration from one second through one day + * @throws IllegalArgumentException when the duration is outside the supported range */ public void setLeaseDurationSeconds(long leaseDurationSeconds) { - if (leaseDurationSeconds <= 0L) { - throw new IllegalArgumentException("leaseDurationSeconds must be positive"); + if (leaseDurationSeconds < 1L + || leaseDurationSeconds > MAXIMUM_LEASE_DURATION_SECONDS) { + throw new IllegalArgumentException( + "leaseDurationSeconds must be between 1 and " + + MAXIMUM_LEASE_DURATION_SECONDS + ); } this.leaseDurationSeconds = leaseDurationSeconds; } From 027375310770d0b3ae3cd8fa5d074d86253e5046 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 12:06:08 +0900 Subject: [PATCH 51/92] test(etl): reject overlong repository leases --- .../EtlJobLeaseRepositoryValidationTest.java | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryValidationTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryValidationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryValidationTest.java new file mode 100644 index 00000000..16f4ba5a --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryValidationTest.java @@ -0,0 +1,39 @@ +package com.xtrmetl.etl.job; + +import org.junit.jupiter.api.Test; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.transaction.PlatformTransactionManager; + +import java.time.Duration; + +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verifyNoInteractions; + +/** + * Proves that the lease repository rejects unsafe public arguments before database access. + */ +class EtlJobLeaseRepositoryValidationTest { + + @Test + void rejectsLeaseDurationsAboveTheOperationalSafetyCeilingBeforeDatabaseAccess() { + JdbcTemplate jdbcTemplate = mock(JdbcTemplate.class); + PlatformTransactionManager transactionManager = mock(PlatformTransactionManager.class); + EtlJobLeaseRepository repository = new EtlJobLeaseRepository( + jdbcTemplate, + transactionManager + ); + + assertThrows( + IllegalArgumentException.class, + () -> repository.claimNext( + "worker-alpha", + Duration.ofSeconds( + EtlJobWorkerProperties.MAXIMUM_LEASE_DURATION_SECONDS + 1L + ), + 3 + ) + ); + verifyNoInteractions(jdbcTemplate, transactionManager); + } +} From 1377f6efffdd784c54873a64d349fdad9f399de7 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 12:07:18 +0900 Subject: [PATCH 52/92] fix(etl): enforce repository lease ceiling --- .../xtrmetl/etl/job/EtlJobLeaseRepository.java | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java index 776cb48c..7842d248 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java @@ -22,7 +22,8 @@ * oldest eligible row with {@code FOR UPDATE SKIP LOCKED}, and finally writes a fresh claim token, * owner, expiry, and incremented attempt count before commit. State transitions repeat the exact * claim token, owner, running status, and database-time expiry predicates so stale workers cannot - * mutate lifecycle state.

+ * mutate lifecycle state. Public callers cannot create leases longer than the worker's one-day + * operational ceiling, even when they bypass Spring configuration binding.

*/ @Repository public class EtlJobLeaseRepository { @@ -174,7 +175,7 @@ public EtlJobLeaseRepository( * Claims at most one oldest eligible job for one worker process. * * @param leaseOwnerId safe non-sensitive process identifier - * @param leaseDuration positive duration applied to database claim time + * @param leaseDuration duration from one second through one day * @param maxAttempts maximum permitted claim count from 1 through 100 * @return a fresh claim, or an empty result when no row is eligible * @throws NullPointerException when an argument is {@code null} @@ -318,8 +319,15 @@ private static Duration requirePositiveDuration(Duration leaseDuration) { leaseDuration, "leaseDuration must not be null" ); - if (requiredDuration.isZero() || requiredDuration.isNegative()) { - throw new IllegalArgumentException("leaseDuration must be positive"); + Duration maximumDuration = Duration.ofSeconds( + EtlJobWorkerProperties.MAXIMUM_LEASE_DURATION_SECONDS + ); + if (requiredDuration.isZero() + || requiredDuration.isNegative() + || requiredDuration.compareTo(maximumDuration) > 0) { + throw new IllegalArgumentException( + "leaseDuration must be between one second and one day" + ); } return requiredDuration; } From 51d120232cedbf09bb0f012b68094477d9777c6a Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 12:08:08 +0900 Subject: [PATCH 53/92] docs(etl): document bounded worker timing controls --- docs/operations/durable-job-worker.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/docs/operations/durable-job-worker.md b/docs/operations/durable-job-worker.md index 04ab521a..bc0b31be 100644 --- a/docs/operations/durable-job-worker.md +++ b/docs/operations/durable-job-worker.md @@ -25,19 +25,26 @@ payloads are acceptable. Enable the worker without intake only to drain already | Preferred property | Environment variable | Default | Constraint | | --- | --- | ---: | --- | | `mightyetl.etl.jobs.worker.enabled` | `ETL_JOB_WORKER_ENABLED` | `false` | explicit opt-in | -| `mightyetl.etl.jobs.worker.fixed-delay-milliseconds` | `ETL_JOB_WORKER_FIXED_DELAY_MILLISECONDS` | `5000` | greater than zero | -| `mightyetl.etl.jobs.worker.initial-delay-milliseconds` | `ETL_JOB_WORKER_INITIAL_DELAY_MILLISECONDS` | `5000` | zero or greater | -| `mightyetl.etl.jobs.worker.lease-duration-seconds` | `ETL_JOB_WORKER_LEASE_DURATION_SECONDS` | `300` | greater than zero | +| `mightyetl.etl.jobs.worker.fixed-delay-milliseconds` | `ETL_JOB_WORKER_FIXED_DELAY_MILLISECONDS` | `5000` | 1 through 86,400,000 | +| `mightyetl.etl.jobs.worker.initial-delay-milliseconds` | `ETL_JOB_WORKER_INITIAL_DELAY_MILLISECONDS` | `5000` | 0 through 86,400,000 | +| `mightyetl.etl.jobs.worker.lease-duration-seconds` | `ETL_JOB_WORKER_LEASE_DURATION_SECONDS` | `300` | 1 through 86,400 | | `mightyetl.etl.jobs.worker.max-attempts` | `ETL_JOB_WORKER_MAX_ATTEMPTS` | `3` | 1 through 100 | | `mightyetl.etl.jobs.worker.lease-owner-id` | deployment-specific | generated | 8–128 safe ASCII characters | +Scheduler delays and lease durations have a one-day safety ceiling. Configuration binding and the +lease repository enforce the same limit, so direct repository callers cannot bypass it. Values above +the ceiling fail application binding or claim validation rather than creating an effectively +permanent polling pause, arithmetic overflow, or multi-day stale-work recovery delay. + Set an explicit `lease-owner-id` only when the deployment platform can guarantee one stable, non-sensitive value per process. Never use a hostname containing customer data, a pod annotation containing credentials, an email address, a tenant identifier, or a raw infrastructure token. Choose a lease duration longer than the normal high-percentile execution time plus database and network variance. The current slice does not renew leases. A lease that expires during execution -causes the final success transition to fail and rolls back target and response-ledger writes. +causes the final success transition to fail and rolls back target and response-ledger writes. If a +normal execution can exceed one day, do not increase the ceiling silently; implement and validate +lease renewal as a separate fenced capability first. ## Claim, execution, and recovery @@ -117,7 +124,8 @@ and query parameters as opt-in sensitive telemetry requiring a separate privacy 2. Check clock-independent database latency and long-running statements; lease decisions use database time. 3. Verify every process has a safe, distinct lease owner identifier. -4. Increase the lease duration only after confirming that crash recovery delay remains acceptable. +4. Increase the lease duration only within the one-day ceiling and only after confirming that crash + recovery delay remains acceptable; implement lease renewal instead of exceeding the ceiling. ### Integrity failure From 5af809f64ece959adaffcc37ec9ec03ed9f92f63 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 12:08:59 +0900 Subject: [PATCH 54/92] docs(config): expose worker timing ceilings --- etl-service/src/main/resources/application.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/etl-service/src/main/resources/application.yml b/etl-service/src/main/resources/application.yml index 642675d3..f5b566d5 100644 --- a/etl-service/src/main/resources/application.yml +++ b/etl-service/src/main/resources/application.yml @@ -32,8 +32,10 @@ xtrmetl: worker: # Execution remains fail-closed until an operator enables the worker explicitly. enabled: ${ETL_JOB_WORKER_ENABLED:false} + # Scheduler delays are bounded to one day (86,400,000 milliseconds). fixed-delay-milliseconds: ${ETL_JOB_WORKER_FIXED_DELAY_MILLISECONDS:5000} initial-delay-milliseconds: ${ETL_JOB_WORKER_INITIAL_DELAY_MILLISECONDS:5000} + # Leases are bounded to one day (86,400 seconds); longer jobs require lease renewal. lease-duration-seconds: ${ETL_JOB_WORKER_LEASE_DURATION_SECONDS:300} max-attempts: ${ETL_JOB_WORKER_MAX_ATTEMPTS:3} # Warehouse/BI targets: SPI + config binding + validation + catalog; writes remain SCAFFOLD. From 8e91f69316a3db5698f42e59a5d8d4c4d4c68bda Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 12:18:26 +0900 Subject: [PATCH 55/92] test(etl): align runbook contract with lease-fenced execution --- .../job/EtlJobMigrationDocumentationTest.java | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobMigrationDocumentationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobMigrationDocumentationTest.java index 46aa44bc..698acc9c 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobMigrationDocumentationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobMigrationDocumentationTest.java @@ -51,21 +51,23 @@ void migrationReservesStableWorkerStatesAndRequiresTerminalPayloadClearing() thr } @Test - void runbookDocumentsAcceptedSemanticsOwnershipAndTheWorkerBoundary() throws IOException { + void runbookDocumentsAcceptedSemanticsOwnershipAndLeaseFencedExecution() throws IOException { String runbook = read("docs/etl/durable-job-intake.md").replaceAll("\\s+", " "); assertTrue(runbook.contains("202 Accepted")); assertTrue(runbook.contains("Location: /api/etl/jobs/{job_record_id}")); assertTrue(runbook.contains("same authenticated principal")); - assertTrue(runbook.contains("byte-for-byte same JSON text")); - assertTrue(runbook.contains("does not execute jobs yet")); + assertTrue(runbook.contains("byte-for-byte identical JSON text")); + assertTrue(runbook.contains("lease-fenced worker claims accepted jobs")); + assertTrue(runbook.contains("PostgreSQL, not scheduler uniqueness, distributes work")); + assertTrue(runbook.contains("same transaction")); assertTrue(runbook.contains("request payload")); - assertTrue(runbook.contains("worker and lease-fencing slice")); assertTrue(runbook.contains("Cache-Control: no-store")); assertTrue(runbook.contains("422 etl_job_submission_key_reused")); - assertTrue(runbook.contains("disabled by default")); - assertTrue(runbook.contains("mightyetl.etl.jobs.intake-enabled=true")); - assertTrue(runbook.contains("xtrmetl.etl.jobs.intake-enabled=true")); + assertTrue(runbook.contains("fail-closed")); + assertTrue(runbook.contains("mightyetl.etl.jobs.intake-enabled=false")); + assertTrue(runbook.contains("mightyetl.etl.jobs.worker.enabled=false")); + assertTrue(runbook.contains("xtrmetl.*")); } private static String read(String relativePath) throws IOException { From 5bafdd508c656f4656cb06c38b4db367cc391a3a Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 12:21:18 +0900 Subject: [PATCH 56/92] test(etl): cover impossible locked-claim transition --- .../EtlJobLeaseRepositoryValidationTest.java | 55 ++++++++++++++++++- 1 file changed, 54 insertions(+), 1 deletion(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryValidationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryValidationTest.java index 16f4ba5a..b56f85ae 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryValidationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryValidationTest.java @@ -2,16 +2,28 @@ import org.junit.jupiter.api.Test; import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.jdbc.core.RowMapper; import org.springframework.transaction.PlatformTransactionManager; +import org.springframework.transaction.TransactionDefinition; +import org.springframework.transaction.TransactionStatus; +import java.sql.ResultSet; import java.time.Duration; +import java.time.OffsetDateTime; +import java.time.ZoneOffset; +import java.util.List; +import java.util.UUID; import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.doAnswer; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.verifyNoInteractions; +import static org.mockito.Mockito.when; /** - * Proves that the lease repository rejects unsafe public arguments before database access. + * Proves that the lease repository rejects unsafe arguments and impossible claim transitions. */ class EtlJobLeaseRepositoryValidationTest { @@ -36,4 +48,45 @@ void rejectsLeaseDurationsAboveTheOperationalSafetyCeilingBeforeDatabaseAccess() ); verifyNoInteractions(jdbcTemplate, transactionManager); } + + @Test + void failsClosedWhenALockedCandidateCannotBeUpdated() throws Exception { + JdbcTemplate jdbcTemplate = mock(JdbcTemplate.class); + PlatformTransactionManager transactionManager = mock(PlatformTransactionManager.class); + TransactionStatus transactionStatus = mock(TransactionStatus.class); + ResultSet resultSet = mock(ResultSet.class); + UUID jobRecordId = UUID.randomUUID(); + + when(transactionManager.getTransaction(any(TransactionDefinition.class))) + .thenReturn(transactionStatus); + when(jdbcTemplate.update(anyString(), any(Object[].class))).thenReturn(0); + when(resultSet.getObject("job_record_id", UUID.class)).thenReturn(jobRecordId); + when(resultSet.getString("principal_scope_hash")).thenReturn("a".repeat(64)); + when(resultSet.getString("submission_key_hash")).thenReturn("b".repeat(64)); + when(resultSet.getString("request_digest")).thenReturn("c".repeat(64)); + when(resultSet.getString("request_payload")) + .thenReturn("[{\"id\":\"record_alpha\"}]"); + when(resultSet.getInt("attempt_count")).thenReturn(0); + when(resultSet.getObject("database_now", OffsetDateTime.class)) + .thenReturn(OffsetDateTime.of(2026, 8, 5, 0, 0, 0, 0, ZoneOffset.UTC)); + doAnswer(invocation -> { + @SuppressWarnings("unchecked") + RowMapper rowMapper = (RowMapper) invocation.getArgument(1); + return List.of(rowMapper.mapRow(resultSet, 0)); + }).when(jdbcTemplate).query( + anyString(), + any(RowMapper.class), + any(Object[].class) + ); + + EtlJobLeaseRepository repository = new EtlJobLeaseRepository( + jdbcTemplate, + transactionManager + ); + + assertThrows( + IllegalStateException.class, + () -> repository.claimNext("worker-alpha", Duration.ofMinutes(5), 3) + ); + } } From 95ed508e88846b443efc4e8ed0e3b34421fb3c2b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 13:04:32 +0900 Subject: [PATCH 57/92] test(etl): require complete worker outcome evidence --- .../com/xtrmetl/etl/job/EtlJobWorkerTest.java | 35 ++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java index 4c7f0d3b..45600a0e 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java @@ -71,7 +71,7 @@ void recordsIdleWithoutExecutingWhenNoJobIsEligible() { worker.pollOnce(); verify(executionService, never()).execute(any()); - assertMetric("idle", 0.0, 1L); + assertMetric("idle", 1.0, 1L); } @Test @@ -203,6 +203,23 @@ void treatsAStaleRetryTransitionAsStaleEvidence() { assertMetric("retried", 0.0, 0L); } + @Test + void recordsRetryTransitionDatabaseFailureAsFailedEvidence() { + EtlJobLease lease = lease(1); + when(leaseRepository.claimNext(anyString(), any(), anyInt())) + .thenReturn(Optional.of(lease)); + doThrow(new CannotAcquireLockException("temporary")) + .when(executionService).execute(lease); + doThrow(new DataIntegrityViolationException("sensitive SQL")) + .when(leaseRepository).releaseForRetry(lease, 3); + + worker.pollOnce(); + + assertMetric("failed", 1.0, 1L); + assertMetric("retried", 0.0, 0L); + assertMetric("stale", 0.0, 0L); + } + @Test void treatsAStaleTerminalTransitionAsStaleEvidence() { EtlJobLease lease = lease(1); @@ -219,6 +236,22 @@ void treatsAStaleTerminalTransitionAsStaleEvidence() { assertMetric("failed", 0.0, 0L); } + @Test + void recordsTerminalTransitionDatabaseFailureAsFailedEvidence() { + EtlJobLease lease = lease(1); + when(leaseRepository.claimNext(anyString(), any(), anyInt())) + .thenReturn(Optional.of(lease)); + doThrow(new EtlRequestException(EtlRequestError.INVALID_JSON)) + .when(executionService).execute(lease); + doThrow(new DataIntegrityViolationException("sensitive SQL")) + .when(leaseRepository).markFailed(lease, "etl_invalid_json"); + + worker.pollOnce(); + + assertMetric("failed", 1.0, 1L); + assertMetric("stale", 0.0, 0L); + } + @Test void recordsClaimDatabaseFailureWithoutLeakingOrExecuting() { when(leaseRepository.claimNext(anyString(), any(), anyInt())) From 0f77c21847402c0e4645c935990a4f62fbf0356e Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 13:04:58 +0900 Subject: [PATCH 58/92] fix(etl): complete durable worker outcome evidence --- .../java/com/xtrmetl/etl/job/EtlJobWorker.java | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorker.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorker.java index ce8b625f..1ffee55e 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorker.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorker.java @@ -24,7 +24,9 @@ * worker classifies failures into stable non-sensitive codes, retries only transient database * failures while attempts remain, and treats every failed exact-lease transition as stale evidence. * Metrics use a fixed outcome vocabulary and never tag payloads, principals, keys, job identifiers, - * lease identifiers, SQL, exception classes, or exception messages.

+ * lease identifiers, SQL, exception classes, or exception messages. Every completed poll records + * one terminal outcome counter and one matching duration sample, including idle polls and database + * failures while persisting retry or terminal transitions.

*/ @Component @ConditionalOnBooleanProperty( @@ -121,9 +123,10 @@ public EtlJobWorker( /** * Claims and handles at most one eligible durable job. * - *

Fixed delay is measured after this invocation completes. A database outage during claim is - * converted into a finite failed metric without copying diagnostic text into application logs or - * telemetry. Spring invokes the method only when worker activation is explicitly enabled.

+ *

Fixed delay is measured after this invocation completes. A database outage during claim or + * a database failure while persisting retry or terminal state is converted into a finite failed + * outcome without copying diagnostic text into application logs or telemetry. Spring invokes the + * method only when worker activation is explicitly enabled.

*/ @Scheduled( fixedDelayString = "${xtrmetl.etl.jobs.worker.fixed-delay-milliseconds:5000}", @@ -149,6 +152,7 @@ private String runOnePoll() { } if (claimedLease.isEmpty()) { + increment(IDLE_OUTCOME); return IDLE_OUTCOME; } @@ -183,6 +187,9 @@ private String handleTransientFailure(EtlJobLease lease) { } catch (StaleEtlJobLeaseException exception) { increment(STALE_OUTCOME); return STALE_OUTCOME; + } catch (DataAccessException exception) { + increment(FAILED_OUTCOME); + return FAILED_OUTCOME; } } return markFailedOrStale(lease, TARGET_UNAVAILABLE_FAILURE_CODE); @@ -196,6 +203,9 @@ private String markFailedOrStale(EtlJobLease lease, String failureCode) { } catch (StaleEtlJobLeaseException exception) { increment(STALE_OUTCOME); return STALE_OUTCOME; + } catch (DataAccessException exception) { + increment(FAILED_OUTCOME); + return FAILED_OUTCOME; } } From 4a589078b764da2d062a9fd9e7f7bb38fdf0e8d5 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 13:05:50 +0900 Subject: [PATCH 59/92] docs(etl): define complete worker outcome accounting --- docs/operations/durable-job-worker.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/operations/durable-job-worker.md b/docs/operations/durable-job-worker.md index bc0b31be..8b63c246 100644 --- a/docs/operations/durable-job-worker.md +++ b/docs/operations/durable-job-worker.md @@ -86,6 +86,12 @@ The worker publishes finite-cardinality metrics only: - `etl.jobs.worker.outcomes{outcome=idle|claimed|succeeded|retried|failed|stale}`; - `etl.jobs.execution.duration{outcome=idle|succeeded|retried|failed|stale}`. +Every completed poll increments one terminal outcome counter and records one matching duration +sample. `claimed` is an additional progress counter for polls that acquired work. Idle polls count as +`idle`; a database failure while persisting a retry or terminal transition counts as `failed`, leaves +the fenced row recoverable through lease expiry, and is never mislabeled as `retried`, `succeeded`, +or `stale`. + Do not add payloads, raw principals, raw idempotency keys, hashes, job identifiers, lease identifiers, SQL text, exception messages, or unbounded exception classes as metric tags or log fields. From 5b7e6d04bc071eadc5ede615bbf0eb7584277dfa Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 13:07:25 +0900 Subject: [PATCH 60/92] docs(changelog): record complete worker outcome accounting --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1b8f46be..b7d0be44 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed - Durable asynchronous ETL jobs now progress from `PENDING` through lease-fenced execution to `SUCCEEDED` or `FAILED`; PostgreSQL owns cross-replica claiming, stale workers cannot commit target or lifecycle effects, and intake and execution remain independently fail-closed. +- Durable-worker observability now records one terminal outcome counter and one matching duration sample for every completed poll, including idle polls and database failures while persisting retry or terminal transitions. - The hourly pull-request disposition loop now requires at least one non-author approval anchored to the exact current head SHA; stale approvals, comment-only reviews, and the mere absence of requested changes cannot authorize unattended merge. - The hourly OpenCode workflow now scopes repository write permissions to its sole maintenance job, replaces the npm installation command with the immutable OpenCode 1.18.13 Linux release archive plus pinned SHA-256 validation, requires exactly one regular-file archive member before private-directory extraction, rejects non-regular or symbolic-link output, and uses a removable repository-local GitHub CLI credential helper instead of storing an encoded authorization header while retaining `persist-credentials: false`. - The hourly OpenCode workflow now uses the current free NVIDIA `deepseek-ai/deepseek-v4-pro` endpoint for long-context coding and agentic tool use instead of the deprecated Qwen3 Coder free endpoint; model or endpoint rejection fails visibly without a non-NVIDIA, partner-only, or automatic fallback. @@ -36,7 +37,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Principal-scoped durable asynchronous ETL job intake and owner-scoped status resources, Flyway `etl_job_records` migration, deterministic replay/conflict coverage, and the authoritative lifecycle contract in `docs/etl/durable-job-intake.md`. - Durable idempotency ledger migration, PostgreSQL transaction advisory-lock adapter, deterministic concurrency/rollback coverage, and the operator/client contract `docs/etl/idempotent-retries.md`. - ETL problem-details client and operator contract: `docs/api/problem-details.md`. -- Operator-configurable ETL admission limits under `mightyetl.etl.*` / `xtrmetl.etl.*`, backed by `ETL_MAX_PAYLOAD_BYTES` and `ETL_MAX_BATCH_RECORDS` environment variables with hard safety ceilings. +- Operator-configurable ETL admission limits under `mightyetl.etl.*` / `xtrmetl.*`, backed by `ETL_MAX_PAYLOAD_BYTES` and `ETL_MAX_BATCH_RECORDS` environment variables with hard safety ceilings. - ETL transaction rollback integration coverage and the operator runbook `docs/etl/bounded-atomic-batches.md`. - Connector scaffolds (contracts + docs only): Qlik Sense, Databricks, Snowflake under `docs/connectors/` and `etl-service` SPI stubs. - Any-to-any CDC design notes and source SPI scaffold: `docs/cdc/any-to-any-cdc.md`, `cdc-service` SPI stubs. From 38a16757fec62e499420e86990bcfbec3422a9ca Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 13:08:16 +0900 Subject: [PATCH 61/92] fix(changelog): preserve ETL alias namespace --- CHANGELOG.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b7d0be44..2c530b27 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,7 +37,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Principal-scoped durable asynchronous ETL job intake and owner-scoped status resources, Flyway `etl_job_records` migration, deterministic replay/conflict coverage, and the authoritative lifecycle contract in `docs/etl/durable-job-intake.md`. - Durable idempotency ledger migration, PostgreSQL transaction advisory-lock adapter, deterministic concurrency/rollback coverage, and the operator/client contract `docs/etl/idempotent-retries.md`. - ETL problem-details client and operator contract: `docs/api/problem-details.md`. -- Operator-configurable ETL admission limits under `mightyetl.etl.*` / `xtrmetl.*`, backed by `ETL_MAX_PAYLOAD_BYTES` and `ETL_MAX_BATCH_RECORDS` environment variables with hard safety ceilings. +- Operator-configurable ETL admission limits under `mightyetl.etl.*` / `xtrmetl.etl.*`, backed by `ETL_MAX_PAYLOAD_BYTES` and `ETL_MAX_BATCH_RECORDS` environment variables with hard safety ceilings. - ETL transaction rollback integration coverage and the operator runbook `docs/etl/bounded-atomic-batches.md`. - Connector scaffolds (contracts + docs only): Qlik Sense, Databricks, Snowflake under `docs/connectors/` and `etl-service` SPI stubs. - Any-to-any CDC design notes and source SPI scaffold: `docs/cdc/any-to-any-cdc.md`, `cdc-service` SPI stubs. @@ -131,7 +131,7 @@ existing xtrmETL platform. - Key features overview - System architecture summary - Technology stack - - Use cases + - Use cases and scenarios - API specifications - Quick start guide - Future improvements From 6d53965228625ff0081e99eec21d28c0e2c279f7 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 13:09:32 +0900 Subject: [PATCH 62/92] fix(changelog): keep historical summary unchanged --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c530b27..f206858f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -131,7 +131,7 @@ existing xtrmETL platform. - Key features overview - System architecture summary - Technology stack - - Use cases and scenarios + - Use cases - API specifications - Quick start guide - Future improvements From 5ac536f52bc59c878a2f3286dda35936bb19e252 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 14:07:49 +0900 Subject: [PATCH 63/92] test(etl): require one terminal worker outcome per poll --- .../EtlJobWorkerOutcomeAccountingTest.java | 99 +++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerOutcomeAccountingTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerOutcomeAccountingTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerOutcomeAccountingTest.java new file mode 100644 index 00000000..0581bbb3 --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerOutcomeAccountingTest.java @@ -0,0 +1,99 @@ +package com.xtrmetl.etl.job; + +import io.micrometer.core.instrument.Counter; +import io.micrometer.core.instrument.Timer; +import io.micrometer.core.instrument.simple.SimpleMeterRegistry; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.time.Instant; +import java.util.Optional; +import java.util.UUID; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyInt; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.when; + +/** + * Verifies that one completed worker poll contributes exactly one terminal outcome observation. + * + *

The outcome counter and duration timer are intended to share one finite terminal vocabulary. + * A successful poll therefore records only {@code succeeded}; an intermediate claim event must not + * be counted as a second outcome because summing the outcome series is an operator-facing poll-rate + * and service-level evidence boundary.

+ */ +@ExtendWith(MockitoExtension.class) +class EtlJobWorkerOutcomeAccountingTest { + + private static final String OWNER_ID = "worker-alpha"; + private static final String HASH_A = "a".repeat(64); + private static final String HASH_B = "b".repeat(64); + private static final String HASH_C = "c".repeat(64); + + @Mock + private EtlJobLeaseRepository leaseRepository; + + @Mock + private EtlJobExecutionService executionService; + + @Test + void successfulPollRecordsExactlyOneTerminalCounterAndDuration() { + EtlJobWorkerProperties properties = new EtlJobWorkerProperties(); + properties.setEnabled(true); + properties.setLeaseOwnerId(OWNER_ID); + properties.setLeaseDurationSeconds(120L); + properties.setMaxAttempts(3); + SimpleMeterRegistry meterRegistry = new SimpleMeterRegistry(); + EtlJobLease lease = new EtlJobLease( + UUID.randomUUID(), + UUID.randomUUID(), + OWNER_ID, + HASH_A, + HASH_B, + HASH_C, + "[{\"id\":\"record_alpha\"}]", + 1, + Instant.now().plusSeconds(300) + ); + when(leaseRepository.claimNext(anyString(), any(), anyInt())) + .thenReturn(Optional.of(lease)); + + new EtlJobWorker( + leaseRepository, + executionService, + properties, + meterRegistry + ).pollOnce(); + + double totalOutcomeCount = meterRegistry.find("etl.jobs.worker.outcomes") + .counters() + .stream() + .mapToDouble(Counter::count) + .sum(); + long totalDurationCount = meterRegistry.find("etl.jobs.execution.duration") + .timers() + .stream() + .mapToLong(Timer::count) + .sum(); + + assertEquals(1.0, totalOutcomeCount); + assertEquals(1L, totalDurationCount); + assertEquals( + 1.0, + meterRegistry.find("etl.jobs.worker.outcomes") + .tag("outcome", "succeeded") + .counter() + .count() + ); + assertNull( + meterRegistry.find("etl.jobs.worker.outcomes") + .tag("outcome", "claimed") + .counter() + ); + } +} From c66d052459ea400a2fb7b132c1250876fa2314c7 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 14:08:25 +0900 Subject: [PATCH 64/92] fix(etl): count one terminal worker outcome per poll --- .../main/java/com/xtrmetl/etl/job/EtlJobWorker.java | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorker.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorker.java index 1ffee55e..ef3737c2 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorker.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorker.java @@ -23,10 +23,10 @@ *

The database claim repository, not the scheduler, distributes work across replicas. The * worker classifies failures into stable non-sensitive codes, retries only transient database * failures while attempts remain, and treats every failed exact-lease transition as stale evidence. - * Metrics use a fixed outcome vocabulary and never tag payloads, principals, keys, job identifiers, - * lease identifiers, SQL, exception classes, or exception messages. Every completed poll records - * one terminal outcome counter and one matching duration sample, including idle polls and database - * failures while persisting retry or terminal transitions.

+ * Metrics use a fixed terminal-outcome vocabulary and never tag payloads, principals, keys, job + * identifiers, lease identifiers, SQL, exception classes, or exception messages. Every completed + * poll records exactly one terminal outcome counter and one matching duration sample, including + * idle polls and database failures while persisting retry or terminal transitions.

*/ @Component @ConditionalOnBooleanProperty( @@ -49,14 +49,12 @@ public class EtlJobWorker { private static final String METRIC_OUTCOMES = "etl.jobs.worker.outcomes"; private static final String METRIC_DURATION = "etl.jobs.execution.duration"; private static final String IDLE_OUTCOME = "idle"; - private static final String CLAIMED_OUTCOME = "claimed"; private static final String SUCCEEDED_OUTCOME = "succeeded"; private static final String RETRIED_OUTCOME = "retried"; private static final String FAILED_OUTCOME = "failed"; private static final String STALE_OUTCOME = "stale"; private static final List FINITE_OUTCOMES = List.of( IDLE_OUTCOME, - CLAIMED_OUTCOME, SUCCEEDED_OUTCOME, RETRIED_OUTCOME, FAILED_OUTCOME, @@ -104,7 +102,7 @@ public EtlJobWorker( counters.put( outcome, Counter.builder(METRIC_OUTCOMES) - .description("Durable ETL worker outcomes") + .description("Durable ETL worker terminal outcomes") .tag("outcome", outcome) .register(this.meterRegistry) ); @@ -157,7 +155,6 @@ private String runOnePoll() { } EtlJobLease lease = claimedLease.orElseThrow(); - increment(CLAIMED_OUTCOME); try { executionService.execute(lease); increment(SUCCEEDED_OUTCOME); From 3f1e23a18228751df1890cd1c95bdfc592c6f56f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 14:09:10 +0900 Subject: [PATCH 65/92] test(etl): align worker metric expectations with terminal outcomes --- .../src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java index 45600a0e..eaf0fbfa 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobWorkerTest.java @@ -75,7 +75,7 @@ void recordsIdleWithoutExecutingWhenNoJobIsEligible() { } @Test - void executesAtMostOneClaimAndRecordsClaimedAndSucceeded() { + void executesAtMostOneClaimAndRecordsSucceeded() { EtlJobLease lease = lease(1); when(leaseRepository.claimNext(anyString(), any(), anyInt())) .thenReturn(Optional.of(lease)); @@ -83,7 +83,6 @@ void executesAtMostOneClaimAndRecordsClaimedAndSucceeded() { worker.pollOnce(); verify(executionService).execute(lease); - assertMetric("claimed", 1.0, 0L); assertMetric("succeeded", 1.0, 1L); } From a68d84cac4f50834a1f6fd9e84cf3b791442280e Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 20:12:04 +0900 Subject: [PATCH 66/92] test(etl): require nonblocking claim index rollout --- .../job/EtlJobClaimIndexMigrationTest.java | 107 ++++++++++++++++++ 1 file changed, 107 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobClaimIndexMigrationTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobClaimIndexMigrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobClaimIndexMigrationTest.java new file mode 100644 index 00000000..4f740f57 --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobClaimIndexMigrationTest.java @@ -0,0 +1,107 @@ +package com.xtrmetl.etl.job; + +import org.junit.jupiter.api.Test; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; + +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Guards the production rollout contract for the durable-job claim eligibility index. + * + *

The table can already contain accepted work when lease fencing is introduced. PostgreSQL's + * regular index build blocks inserts, updates, and deletes, so the claim index must be isolated in + * a non-transactional concurrent migration while the lease columns and constraints remain in the + * transactional V3 migration.

+ */ +class EtlJobClaimIndexMigrationTest { + + private static final String V3_MIGRATION = + "etl-service/src/main/resources/db/migration/V3__add_etl_job_lease_fencing.sql"; + private static final String V4_MIGRATION = + "etl-service/src/main/resources/db/migration/V4__add_etl_job_claim_eligibility_index.sql"; + private static final String V4_CONFIGURATION = V4_MIGRATION + ".conf"; + + @Test + void keepsTransactionalLeaseSchemaSeparateFromConcurrentIndexBuild() throws IOException { + String leaseMigration = normalize(read(V3_MIGRATION)); + String indexMigration = normalize(read(V4_MIGRATION)); + + assertFalse( + leaseMigration.contains("CREATE INDEX"), + "the transactional lease migration must not contain a production index build" + ); + assertTrue(indexMigration.contains( + "CREATE INDEX CONCURRENTLY etl_job_claim_eligibility_index" + )); + assertTrue(indexMigration.contains( + "ON etl_job_records (job_status, lease_expires_at, created_at, job_record_id)" + )); + } + + @Test + void disablesFlywayTransactionsAndPostgresqlTransactionalLocksForConcurrentDdl() + throws IOException { + Path configurationPath = projectRoot().resolve(V4_CONFIGURATION); + String application = normalize( + read("etl-service/src/main/resources/application.yml") + ); + + assertTrue( + Files.exists(configurationPath), + "the concurrent migration requires a matching Flyway script configuration" + ); + assertTrue( + Files.readString(configurationPath, StandardCharsets.UTF_8) + .contains("executeInTransaction=false") + ); + assertTrue(application.contains("postgresql: transactional-lock: false")); + } + + @Test + void runbookDocumentsConcurrentFailureRecoveryAndRollback() throws IOException { + String runbook = normalize(read("docs/operations/durable-job-worker.md")); + + assertTrue(runbook.contains("V4__add_etl_job_claim_eligibility_index.sql")); + assertTrue(runbook.contains("CREATE INDEX CONCURRENTLY")); + assertTrue(runbook.contains("invalid index")); + assertTrue(runbook.contains( + "DROP INDEX CONCURRENTLY etl_job_claim_eligibility_index" + )); + assertTrue(runbook.contains("executeInTransaction=false")); + } + + private static String read(String relativePath) throws IOException { + return Files.readString(projectRoot().resolve(relativePath), StandardCharsets.UTF_8); + } + + private static String normalize(String value) { + return value.replaceAll("\\s+", " ").trim(); + } + + /** + * Finds the reactor root from either repository-root or module-local Maven execution. + */ + private static Path projectRoot() { + Path current = Paths.get(System.getProperty("user.dir")).toAbsolutePath(); + Path lastPomParent = null; + while (current != null) { + if (Files.exists(current.resolve(".git"))) { + return current; + } + if (Files.exists(current.resolve("pom.xml"))) { + lastPomParent = current; + } + current = current.getParent(); + } + if (lastPomParent != null) { + return lastPomParent; + } + throw new IllegalStateException("Could not find project root"); + } +} From a6232dce0094e6644d4c129544b75bc5bd0cabeb Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 20:12:33 +0900 Subject: [PATCH 67/92] fix(etl): keep lease schema migration transactional --- .../resources/db/migration/V3__add_etl_job_lease_fencing.sql | 3 --- 1 file changed, 3 deletions(-) diff --git a/etl-service/src/main/resources/db/migration/V3__add_etl_job_lease_fencing.sql b/etl-service/src/main/resources/db/migration/V3__add_etl_job_lease_fencing.sql index 0b5a5555..2469acc7 100644 --- a/etl-service/src/main/resources/db/migration/V3__add_etl_job_lease_fencing.sql +++ b/etl-service/src/main/resources/db/migration/V3__add_etl_job_lease_fencing.sql @@ -41,6 +41,3 @@ ALTER TABLE etl_job_records AND failure_code IS NULL ) ); - -CREATE INDEX etl_job_claim_eligibility_index - ON etl_job_records (job_status, lease_expires_at, created_at, job_record_id); From d1d1444fa4f1190b642940bc667bd0dba1a0dbd6 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 20:12:50 +0900 Subject: [PATCH 68/92] fix(etl): build claim index without blocking writers --- .../V4__add_etl_job_claim_eligibility_index.sql | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 etl-service/src/main/resources/db/migration/V4__add_etl_job_claim_eligibility_index.sql diff --git a/etl-service/src/main/resources/db/migration/V4__add_etl_job_claim_eligibility_index.sql b/etl-service/src/main/resources/db/migration/V4__add_etl_job_claim_eligibility_index.sql new file mode 100644 index 00000000..a7939f34 --- /dev/null +++ b/etl-service/src/main/resources/db/migration/V4__add_etl_job_claim_eligibility_index.sql @@ -0,0 +1,11 @@ +-- Support oldest-first claim selection across pending and expired-running durable jobs. +-- CONCURRENTLY preserves inserts, updates, and deletes while PostgreSQL builds the index. +-- The companion .sql.conf disables Flyway's per-migration transaction because PostgreSQL +-- rejects CREATE INDEX CONCURRENTLY inside a transaction block. +CREATE INDEX CONCURRENTLY etl_job_claim_eligibility_index + ON etl_job_records ( + job_status, + lease_expires_at, + created_at, + job_record_id + ); From e44ea4aa0f765a160716a3ed7ef0a8371debadc6 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 20:13:07 +0900 Subject: [PATCH 69/92] build(etl): run claim index migration outside transaction --- .../migration/V4__add_etl_job_claim_eligibility_index.sql.conf | 1 + 1 file changed, 1 insertion(+) create mode 100644 etl-service/src/main/resources/db/migration/V4__add_etl_job_claim_eligibility_index.sql.conf diff --git a/etl-service/src/main/resources/db/migration/V4__add_etl_job_claim_eligibility_index.sql.conf b/etl-service/src/main/resources/db/migration/V4__add_etl_job_claim_eligibility_index.sql.conf new file mode 100644 index 00000000..73bd53a1 --- /dev/null +++ b/etl-service/src/main/resources/db/migration/V4__add_etl_job_claim_eligibility_index.sql.conf @@ -0,0 +1 @@ +executeInTransaction=false From 9835495ad6d18f6889129cc51ad270892a9c459c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 20:14:05 +0900 Subject: [PATCH 70/92] fix(etl): use session Flyway lock for concurrent indexes --- etl-service/src/main/resources/application.properties | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 etl-service/src/main/resources/application.properties diff --git a/etl-service/src/main/resources/application.properties b/etl-service/src/main/resources/application.properties new file mode 100644 index 00000000..918dd4ad --- /dev/null +++ b/etl-service/src/main/resources/application.properties @@ -0,0 +1,2 @@ +# PostgreSQL concurrent index migrations cannot use Flyway's transactional advisory lock. +spring.flyway.postgresql.transactional-lock=false From 2797795b0eeee43912583434af1fb5977f31fa08 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 20:14:46 +0900 Subject: [PATCH 71/92] test(etl): verify concurrent Flyway lock configuration --- .../xtrmetl/etl/job/EtlJobClaimIndexMigrationTest.java | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobClaimIndexMigrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobClaimIndexMigrationTest.java index 4f740f57..104719a6 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobClaimIndexMigrationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobClaimIndexMigrationTest.java @@ -40,7 +40,7 @@ void keepsTransactionalLeaseSchemaSeparateFromConcurrentIndexBuild() throws IOEx "CREATE INDEX CONCURRENTLY etl_job_claim_eligibility_index" )); assertTrue(indexMigration.contains( - "ON etl_job_records (job_status, lease_expires_at, created_at, job_record_id)" + "ON etl_job_records ( job_status, lease_expires_at, created_at, job_record_id )" )); } @@ -48,8 +48,8 @@ void keepsTransactionalLeaseSchemaSeparateFromConcurrentIndexBuild() throws IOEx void disablesFlywayTransactionsAndPostgresqlTransactionalLocksForConcurrentDdl() throws IOException { Path configurationPath = projectRoot().resolve(V4_CONFIGURATION); - String application = normalize( - read("etl-service/src/main/resources/application.yml") + String applicationProperties = read( + "etl-service/src/main/resources/application.properties" ); assertTrue( @@ -60,7 +60,9 @@ void disablesFlywayTransactionsAndPostgresqlTransactionalLocksForConcurrentDdl() Files.readString(configurationPath, StandardCharsets.UTF_8) .contains("executeInTransaction=false") ); - assertTrue(application.contains("postgresql: transactional-lock: false")); + assertTrue(applicationProperties.contains( + "spring.flyway.postgresql.transactional-lock=false" + )); } @Test From 7bde8e1b204ae01d21f4675169fe30a291d54ed2 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 20:16:27 +0900 Subject: [PATCH 72/92] docs(etl): document nonblocking claim index rollout --- .../durable-job-claim-index-rollout.md | 95 +++++++++++++++++++ 1 file changed, 95 insertions(+) create mode 100644 docs/operations/durable-job-claim-index-rollout.md diff --git a/docs/operations/durable-job-claim-index-rollout.md b/docs/operations/durable-job-claim-index-rollout.md new file mode 100644 index 00000000..79adde34 --- /dev/null +++ b/docs/operations/durable-job-claim-index-rollout.md @@ -0,0 +1,95 @@ +# Durable-job claim index rollout + +## Purpose + +The durable worker queries `etl_job_records` for the oldest eligible `PENDING` row or expired +`RUNNING` row. The descriptive `etl_job_claim_eligibility_index` supports that queue-like access +path without changing the lease-fencing state machine. + +The table can already receive job submissions when this index is introduced. PostgreSQL's ordinary +`CREATE INDEX` permits reads but blocks `INSERT`, `UPDATE`, and `DELETE` until the build completes. +For a production ETL control plane that write outage is not acceptable. Migration +`V4__add_etl_job_claim_eligibility_index.sql` therefore uses `CREATE INDEX CONCURRENTLY`. + +## Flyway execution boundary + +PostgreSQL rejects `CREATE INDEX CONCURRENTLY` inside a transaction block. The companion script +configuration file +`V4__add_etl_job_claim_eligibility_index.sql.conf` contains: + +```properties +executeInTransaction=false +``` + +Flyway's PostgreSQL transactional advisory lock is also disabled with: + +```properties +spring.flyway.postgresql.transactional-lock=false +``` + +This causes Flyway to use the PostgreSQL integration's non-transactional lock mode required for +concurrent index DDL. The transactional V3 migration remains responsible only for lease columns, +legacy-data repair, and lifecycle constraints. Isolating the index in V4 prevents an index-build +failure from partially committing those schema invariants. + +## Deployment procedure + +1. Keep durable-job intake and worker execution disabled while validating the migration package. +2. Confirm no other concurrent index build or schema migration is active on `etl_job_records`. +3. Apply V3 and verify the lease columns and lifecycle constraints. +4. Apply V4 and monitor `pg_stat_progress_create_index`, database I/O, and transaction latency. +5. Verify `pg_index.indisvalid` is true for `etl_job_claim_eligibility_index`. +6. Run the claim-selection plan against production-equivalent data and confirm the expected index is + available without forcing it through planner settings. +7. Enable one worker canary only after schema history, catalog state, and application health agree. + +`CREATE INDEX CONCURRENTLY` performs more work and can take longer than a regular build. It preserves +normal writes, but it still adds CPU, memory, and I/O load and allows only one concurrent index build +per table. + +## Failed migration and invalid-index recovery + +A failed concurrent build can leave an **invalid index** in the PostgreSQL catalog. Do not mark the +Flyway migration successful merely because an index name exists. + +Recovery is fail-closed: + +1. Keep worker execution disabled and inspect the failed Flyway record plus `pg_index.indisvalid`. +2. Preserve database and migration logs under incident-response controls. +3. Correct the underlying resource, permission, duplicate-build, or transaction-mode cause. +4. Remove an unusable index without blocking normal table access: + + ```sql + DROP INDEX CONCURRENTLY etl_job_claim_eligibility_index; + ``` + +5. Use Flyway repair only after catalog inspection and operator approval, then rerun the unchanged, + checksum-verified migration. +6. Reconfirm index validity and query-plan evidence before enabling workers. + +Do not edit an applied versioned migration or create a same-name replacement with different SQL. + +## Rollback boundary + +Application rollback does not require removing the index; an unused valid index is compatible with +older binaries, although it adds write-maintenance overhead. Remove it only after every deployed +worker version no longer relies on it and a controlled change has verified the performance impact: + +```sql +DROP INDEX CONCURRENTLY etl_job_claim_eligibility_index; +``` + +`DROP INDEX CONCURRENTLY` must also run outside a transaction block. Lease columns and constraints +require a later forward compensating migration after all compatible binaries have been removed; they +must not be rolled back by editing V3. + +## Standards and primary documentation + +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: Building indexes +concurrently*. https://www.postgresql.org/docs/18/sql-createindex.html#SQL-CREATEINDEX-CONCURRENTLY + +Redgate Software. (2026). *Flyway script configuration*. +https://documentation.red-gate.com/flyway/reference/script-configuration + +Redgate Software. (2026). *Flyway PostgreSQL transactional lock setting*. +https://documentation.red-gate.com/fd/flyway-postgresql-transactional-lock-setting-277579114.html From c447bfaac0635039327a30831f1af877efdaca72 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 20:17:09 +0900 Subject: [PATCH 73/92] test(etl): bind claim index evidence to focused runbook --- .../com/xtrmetl/etl/job/EtlJobClaimIndexMigrationTest.java | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobClaimIndexMigrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobClaimIndexMigrationTest.java index 104719a6..a3bc6479 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobClaimIndexMigrationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobClaimIndexMigrationTest.java @@ -67,7 +67,9 @@ void disablesFlywayTransactionsAndPostgresqlTransactionalLocksForConcurrentDdl() @Test void runbookDocumentsConcurrentFailureRecoveryAndRollback() throws IOException { - String runbook = normalize(read("docs/operations/durable-job-worker.md")); + String runbook = normalize(read( + "docs/operations/durable-job-claim-index-rollout.md" + )); assertTrue(runbook.contains("V4__add_etl_job_claim_eligibility_index.sql")); assertTrue(runbook.contains("CREATE INDEX CONCURRENTLY")); From ec140c58214d890df260e04a05a92840f5f56279 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 20:17:48 +0900 Subject: [PATCH 74/92] test(etl): keep lease schema migration focused --- .../xtrmetl/etl/job/EtlJobLeaseMigrationTest.java | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseMigrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseMigrationTest.java index 8d73c761..6491616c 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseMigrationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseMigrationTest.java @@ -12,21 +12,18 @@ import static org.junit.jupiter.api.Assertions.assertTrue; /** - * Specifies the additive Flyway contract for exact durable-job lease fencing. + * Specifies the transactional Flyway contract for exact durable-job lease fencing. */ class EtlJobLeaseMigrationTest { @Test - void addsDescriptiveLeaseColumnsAndEligibilityIndex() throws IOException { + void addsDescriptiveLeaseColumnsWithoutNonTransactionalIndexDdl() throws IOException { String migration = readMigration(); assertTrue(migration.contains("ADD COLUMN lease_claim_id UUID")); assertTrue(migration.contains("ADD COLUMN lease_owner_id VARCHAR(128)")); assertTrue(migration.contains("ADD COLUMN lease_expires_at TIMESTAMPTZ")); - assertTrue(migration.contains("CREATE INDEX etl_job_claim_eligibility_index")); - assertTrue(migration.contains( - "(job_status, lease_expires_at, created_at, job_record_id)" - )); + assertFalse(migration.contains("CREATE INDEX")); assertFalse(migration.contains(" ADD COLUMN owner ")); assertFalse(migration.contains(" ADD COLUMN lease ")); } @@ -67,6 +64,9 @@ private static String normalize(String value) { return value.replaceAll("\\s+", " ").trim(); } + /** + * Finds the reactor root from either repository-root or module-local Maven execution. + */ private static Path projectRoot() { Path current = Paths.get(System.getProperty("user.dir")).toAbsolutePath(); Path lastPomParent = null; From 1b53a9873c510a317921639aa051f72fc6b537d6 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 20:22:52 +0900 Subject: [PATCH 75/92] docs(etl): align worker metrics and migration rollout --- docs/operations/durable-job-worker.md | 39 +++++++++++++++++++-------- 1 file changed, 28 insertions(+), 11 deletions(-) diff --git a/docs/operations/durable-job-worker.md b/docs/operations/durable-job-worker.md index 8b63c246..34723549 100644 --- a/docs/operations/durable-job-worker.md +++ b/docs/operations/durable-job-worker.md @@ -83,14 +83,15 @@ stable failure code, attempt count, lifecycle state, and timestamps to the authe The worker publishes finite-cardinality metrics only: -- `etl.jobs.worker.outcomes{outcome=idle|claimed|succeeded|retried|failed|stale}`; +- `etl.jobs.worker.outcomes{outcome=idle|succeeded|retried|failed|stale}`; - `etl.jobs.execution.duration{outcome=idle|succeeded|retried|failed|stale}`. -Every completed poll increments one terminal outcome counter and records one matching duration -sample. `claimed` is an additional progress counter for polls that acquired work. Idle polls count as -`idle`; a database failure while persisting a retry or terminal transition counts as `failed`, leaves -the fenced row recoverable through lease expiry, and is never mislabeled as `retried`, `succeeded`, -or `stale`. +Every completed poll increments exactly one terminal outcome counter and records exactly one matching +duration sample. Idle polls count as `idle`; a database failure while persisting a retry or terminal +transition counts as `failed`, leaves the fenced row recoverable through lease expiry, and is never +mislabeled as `retried`, `succeeded`, or `stale`. Claim acquisition is an internal phase, not a second +outcome series, so summing the outcome counters yields the completed poll count without double +counting work-bearing polls. Do not add payloads, raw principals, raw idempotency keys, hashes, job identifiers, lease identifiers, SQL text, exception messages, or unbounded exception classes as metric tags or log fields. @@ -146,12 +147,19 @@ and query parameters as opt-in sensitive telemetry requiring a separate privacy Before enabling the worker: -1. Apply and validate Flyway migration `V3__add_etl_job_lease_fencing.sql`. -2. Confirm the application principal has only the required table and advisory-lock permissions. -3. Run migration, claim-contention, stale-lease rollback, response-replay, and target compatibility +1. Apply the transactional `V3__add_etl_job_lease_fencing.sql` migration and then the nonblocking + `V4__add_etl_job_claim_eligibility_index.sql` migration. +2. Confirm V4 uses `CREATE INDEX CONCURRENTLY`, its companion configuration contains + `executeInTransaction=false`, Flyway PostgreSQL transactional locking is disabled, and + `etl_job_claim_eligibility_index` is valid in the PostgreSQL catalog. +3. Confirm the application principal has only the required table and advisory-lock permissions. +4. Run migration, claim-contention, stale-lease rollback, response-replay, and target compatibility tests against a production-equivalent PostgreSQL environment. -4. Deploy with the worker disabled, inspect health and schema evidence, then enable a canary replica. -5. Verify target, response-ledger, and terminal state atomicity before widening rollout. +5. Deploy with the worker disabled, inspect health and schema evidence, then enable a canary replica. +6. Verify target, response-ledger, and terminal state atomicity before widening rollout. + +The dedicated rollout, invalid-index recovery, and concurrent rollback procedure is +`docs/operations/durable-job-claim-index-rollout.md`. Rollback order is fail-closed: @@ -176,5 +184,14 @@ https://opentelemetry.io/docs/specs/semconv/db/sql/ PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: SELECT*. https://www.postgresql.org/docs/18/sql-select.html +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: CREATE INDEX*. +https://www.postgresql.org/docs/18/sql-createindex.html + +Redgate Software. (2026). *Flyway PostgreSQL transactional lock setting*. +https://documentation.red-gate.com/fd/flyway-postgresql-transactional-lock-setting-277579114.html + +Redgate Software. (2026). *Flyway script configuration*. +https://documentation.red-gate.com/flyway/reference/script-configuration + Spring Authors. (2026). *Task execution and scheduling*. Broadcom. https://docs.spring.io/spring-framework/reference/integration/scheduling.html From c2f0802a728a0382b78feb67cf67455583f70f98 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 20:24:48 +0900 Subject: [PATCH 76/92] docs(changelog): record nonblocking claim index rollout --- CHANGELOG.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f206858f..3dd3e853 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- The durable-job claim eligibility index now builds in a separate PostgreSQL `CREATE INDEX CONCURRENTLY` migration with Flyway non-transactional script configuration and session-level PostgreSQL migration locking, preserving normal job writes during rollout while keeping lease columns and constraints transactional. - Durable asynchronous ETL jobs now progress from `PENDING` through lease-fenced execution to `SUCCEEDED` or `FAILED`; PostgreSQL owns cross-replica claiming, stale workers cannot commit target or lifecycle effects, and intake and execution remain independently fail-closed. - Durable-worker observability now records one terminal outcome counter and one matching duration sample for every completed poll, including idle polls and database failures while persisting retry or terminal transitions. - The hourly pull-request disposition loop now requires at least one non-author approval anchored to the exact current head SHA; stale approvals, comment-only reviews, and the mere absence of requested changes cannot authorize unattended merge. @@ -28,6 +29,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- A production rollout and invalid-index recovery runbook for the nonblocking durable-job claim index: `docs/operations/durable-job-claim-index-rollout.md`. - PostgreSQL `FOR UPDATE SKIP LOCKED` durable-job claiming, per-process and per-claim lease fencing, expiry reclaim, bounded attempts, exact-live-lease transitions, terminal payload clearing, stable failure codes, and finite-cardinality worker metrics. - Hashed durable execution identity and domain-separated reuse of `etl_idempotency_records`, coupling response replay or creation, target writes, and terminal `SUCCEEDED` in one transaction without retaining or reconstructing raw principals or raw client idempotency keys. - Deterministic migration, concurrency, expiry, exhaustion, response-replay, integrity, stale-lease rollback, privacy, configuration-boundary, and operator-recovery tests plus `docs/operations/durable-job-worker.md`. @@ -267,4 +269,4 @@ This changelog will be updated: **Changelog Version**: 1.0 **Last Updated**: 2026-08-05 -**Maintained By**: Development Team \ No newline at end of file +**Maintained By**: Development Team From 9d6f4ba8e237c2251b73df2e1f151e26a56442ea Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 20:27:04 +0900 Subject: [PATCH 77/92] fix(build): preserve explicit patched Jackson BOM import --- pom.xml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/pom.xml b/pom.xml index 94bda043..66720d8f 100644 --- a/pom.xml +++ b/pom.xml @@ -34,6 +34,13 @@ + + com.fasterxml.jackson + jackson-bom + ${jackson-bom.version} + pom + import + org.springframework.boot spring-boot-dependencies From 36313a6278cc5ff3fc9dc41c46b056c713a0a3de Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 20:59:56 +0900 Subject: [PATCH 78/92] test(etl): define durable worker retry boundary --- .../EtlJobIdempotencyRetryBoundaryTest.java | 100 ++++++++++++++++++ 1 file changed, 100 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobIdempotencyRetryBoundaryTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobIdempotencyRetryBoundaryTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobIdempotencyRetryBoundaryTest.java new file mode 100644 index 00000000..cc266ede --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobIdempotencyRetryBoundaryTest.java @@ -0,0 +1,100 @@ +package com.xtrmetl.etl.job; + +import com.xtrmetl.etl.service.EtlRequestLock; +import com.xtrmetl.etl.service.EtlService; +import com.xtrmetl.etl.service.Sha256Digest; +import org.junit.jupiter.api.Test; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.jdbc.datasource.DataSourceTransactionManager; +import org.springframework.jdbc.datasource.embedded.EmbeddedDatabase; +import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseBuilder; +import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseType; +import org.springframework.transaction.support.TransactionTemplate; + +import java.time.Instant; +import java.util.UUID; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +/** + * Guards the retry boundary between durable database attempts and synchronous request retries. + * + *

A durable worker already owns bounded, persisted retries through {@code attempt_count}. It + * must therefore invoke an ETL entry point that joins the current lease transaction exactly once, + * rather than the synchronous {@code @Retryable} API whose advice is designed to wrap a fresh + * transaction for each HTTP-request attempt.

+ */ +class EtlJobIdempotencyRetryBoundaryTest { + + private static final String PAYLOAD = "[{\"id\":\"record_alpha\"}]"; + private static final String RESPONSE = "Processed: record_alpha"; + + /** + * Proves one durable attempt uses only the non-retrying current-transaction ETL entry point. + */ + @Test + void durableAttemptDoesNotInvokeTheSynchronousRetryableEntryPoint() { + EmbeddedDatabase database = new EmbeddedDatabaseBuilder() + .generateUniqueName(true) + .setType(EmbeddedDatabaseType.H2) + .build(); + try { + JdbcTemplate jdbcTemplate = new JdbcTemplate(database); + jdbcTemplate.execute(""" + CREATE TABLE etl_idempotency_records ( + idempotency_key_hash CHAR(64) PRIMARY KEY, + request_digest CHAR(64) NOT NULL, + response_body CLOB NOT NULL, + created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP + ) + """); + + EtlService etlService = mock(EtlService.class); + EtlRequestLock requestLock = mock(EtlRequestLock.class); + when(requestLock.tryLock(anyString())).thenReturn(true); + when(etlService.processDataInExistingTransaction(PAYLOAD)).thenReturn(RESPONSE); + EtlJobIdempotencyService service = new EtlJobIdempotencyService( + jdbcTemplate, + etlService, + requestLock + ); + TransactionTemplate transactionTemplate = new TransactionTemplate( + new DataSourceTransactionManager(database) + ); + + String result = transactionTemplate.execute(status -> service.process(lease())); + + assertEquals(RESPONSE, result); + verify(etlService).processDataInExistingTransaction(PAYLOAD); + verify(etlService, never()).processData(anyString()); + assertEquals( + 1, + jdbcTemplate.queryForObject( + "SELECT COUNT(*) FROM etl_idempotency_records", + Integer.class + ) + ); + } finally { + database.shutdown(); + } + } + + private static EtlJobLease lease() { + return new EtlJobLease( + UUID.randomUUID(), + UUID.randomUUID(), + "worker-alpha", + "a".repeat(64), + "b".repeat(64), + Sha256Digest.digest(PAYLOAD), + PAYLOAD, + 1, + Instant.now().plusSeconds(300) + ); + } +} From e8ec36dc5eef65a6809e92d46e65f72638ffd662 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 21:00:35 +0900 Subject: [PATCH 79/92] test(etl): require an active durable execution transaction --- ...iceIdempotencyTransactionBoundaryTest.java | 36 ++++++++++++++++--- 1 file changed, 32 insertions(+), 4 deletions(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/service/EtlServiceIdempotencyTransactionBoundaryTest.java b/etl-service/src/test/java/com/xtrmetl/etl/service/EtlServiceIdempotencyTransactionBoundaryTest.java index faa6ee1d..15c078e0 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/service/EtlServiceIdempotencyTransactionBoundaryTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/service/EtlServiceIdempotencyTransactionBoundaryTest.java @@ -14,10 +14,10 @@ * Verifies that durable idempotency preserves its transaction and admission-control boundaries. * *

Spring applies {@code @Transactional} through a proxy. A directly constructed service does - * not receive that proxy, so allowing the idempotent method to continue would release a PostgreSQL - * transaction advisory lock too early and could separate target writes from the response ledger. - * The production method therefore has to fail before lock or database access when no transaction - * is active.

+ * not receive that proxy, so allowing an idempotent or durable-worker entry point to continue + * would release PostgreSQL transaction locks too early and could separate target writes from the + * response ledger. Transaction-scoped entry points therefore fail before database access when no + * actual transaction is active.

* *

Keyed requests must also enforce key and payload admission before computing their durable * decision through the request lock or ledger. This preserves the same zero-database-work boundary @@ -55,6 +55,34 @@ void refusesIdempotentProcessingWithoutAnActiveTransactionBeforeDatabaseWork() { verifyNoInteractions(requestLock, jdbcTemplate); } + /** + * Proves the durable-worker ETL entry point cannot silently execute outside its lease transaction. + */ + @Test + void refusesDurableWorkerProcessingWithoutAnActiveTransactionBeforeDatabaseWork() { + JdbcTemplate jdbcTemplate = mock(JdbcTemplate.class); + EtlRequestLock requestLock = mock(EtlRequestLock.class); + EtlService etlService = new EtlService( + jdbcTemplate, + new ObjectMapper(), + new EtlBatchProperties(), + requestLock + ); + + IllegalStateException exception = assertThrows( + IllegalStateException.class, + () -> etlService.processDataInExistingTransaction( + "[{\"id\":\"record_alpha\"}]" + ) + ); + + assertEquals( + "Durable ETL execution requires an active transaction", + exception.getMessage() + ); + verifyNoInteractions(requestLock, jdbcTemplate); + } + /** * Proves a missing key is rejected before transaction, request-lock, or JDBC work. */ From b734b99f56d6257df342ba244a75604a05a4dcc3 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 21:03:33 +0900 Subject: [PATCH 80/92] fix(etl): separate durable and synchronous retry boundaries --- .../com/xtrmetl/etl/service/EtlService.java | 37 +++++++++++++++---- 1 file changed, 30 insertions(+), 7 deletions(-) diff --git a/etl-service/src/main/java/com/xtrmetl/etl/service/EtlService.java b/etl-service/src/main/java/com/xtrmetl/etl/service/EtlService.java index 6bb32a15..28b86bb2 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/service/EtlService.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/service/EtlService.java @@ -38,6 +38,11 @@ * batch back rather than leaving committed prefix records. The service intentionally avoids the * JVM common pool and one-task-per-record fan-out.

* + *

Synchronous request entry points own Spring Retry outside their transaction boundaries. A + * durable worker instead uses {@link #processDataInExistingTransaction(String)} exactly once per + * persisted attempt so the lease, target rows, response ledger, and terminal state remain in one + * database transaction and retry accounting stays in the durable job record.

+ * *

Callers may optionally use {@link #processDataIdempotently(String, String, String)}. That * method attempts a principal-scoped transaction lock without waiting, replays a prior successful * response, and commits the ETL rows and durable ledger entry in the same transaction.

@@ -142,8 +147,8 @@ public EtlService( /** * Processes one JSON-array request as a prevalidated transaction-scoped batch. * - *

Only transient Spring data-access failures are retried. Typed input failures and - * deterministic target constraints fail immediately instead of repeating the same work.

+ *

Only transient Spring data-access failures are retried. Retry advice wraps the transaction + * advice so every synchronous request attempt receives a fresh transaction.

* * @param data UTF-8 JSON array payload * @return one {@code Processed: } line per record, in input order @@ -160,6 +165,26 @@ public String processData(@Nullable String data) { return processDataInCurrentTransaction(data); } + /** + * Processes one durable-job payload exactly once inside the caller's existing transaction. + * + *

This method deliberately has neither {@link Retryable} nor {@link Transactional}. The + * durable worker owns its persisted retry count and supplies the transaction that also contains + * its lease-fenced terminal transition and response-ledger write. An in-process retry inside + * that outer transaction could reuse an already failed transaction and would not represent a + * new durable attempt.

+ * + * @param data retained UTF-8 JSON array payload + * @return one {@code Processed: } line per record, in input order + * @throws IllegalStateException when the caller did not establish an actual transaction + * @throws EtlRequestException when the retained request violates a deterministic contract + * @throws org.springframework.dao.DataAccessException when the target database rejects work + */ + public String processDataInExistingTransaction(@Nullable String data) { + requireActiveTransaction("Durable ETL execution requires an active transaction"); + return processDataInCurrentTransaction(data); + } + /** * Processes or replays one principal-scoped idempotent ETL request. * @@ -200,7 +225,7 @@ public EtlIdempotencyResult processDataIdempotently( if (data == null) { throw new EtlRequestException(EtlRequestError.INVALID_JSON); } - requireActiveTransaction(); + requireActiveTransaction("Idempotent ETL processing requires an active transaction"); enforcePayloadLimit(data); String idempotencyKeyHash = sha256( @@ -288,11 +313,9 @@ private static String validatePrincipalScope(@Nullable String principalScope) { return principalScope; } - private static void requireActiveTransaction() { + private static void requireActiveTransaction(String failureMessage) { if (!TransactionSynchronizationManager.isActualTransactionActive()) { - throw new IllegalStateException( - "Idempotent ETL processing requires an active transaction" - ); + throw new IllegalStateException(failureMessage); } } From d1405db3ccb77a3f4e2b1b9aadc962d415057803 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 21:04:21 +0900 Subject: [PATCH 81/92] fix(etl): keep durable retries outside the lease transaction --- .../com/xtrmetl/etl/job/EtlJobIdempotencyService.java | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobIdempotencyService.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobIdempotencyService.java index aa120a49..531a31e5 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobIdempotencyService.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobIdempotencyService.java @@ -21,6 +21,10 @@ * transaction-level request lock, replays an existing matching response, or writes the target and * response ledger in the surrounding transaction. Raw principals and client keys are neither * required nor reconstructed.

+ * + *

One invocation represents exactly one persisted durable attempt. It therefore calls the + * non-retrying ETL entry point that joins the current lease transaction; transient failures escape + * to {@link EtlJobWorker}, which owns bounded retry accounting in {@code attempt_count}.

*/ @Service public class EtlJobIdempotencyService { @@ -109,7 +113,9 @@ public String process(EtlJobLease lease) { return storedResponse.responseBody(); } - String responseBody = etlService.processData(requiredLease.requestPayload()); + String responseBody = etlService.processDataInExistingTransaction( + requiredLease.requestPayload() + ); jdbcTemplate.update( INSERT_LEDGER_SQL, ledgerKeyHash, From 1a5d9c0d2e1df391439dbf6d77e14e269ae740e3 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 21:05:46 +0900 Subject: [PATCH 82/92] docs(etl): record durable retry-boundary evidence --- .../durable-job-retry-boundary-evidence.md | 71 +++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 docs/doctoring/durable-job-retry-boundary-evidence.md diff --git a/docs/doctoring/durable-job-retry-boundary-evidence.md b/docs/doctoring/durable-job-retry-boundary-evidence.md new file mode 100644 index 00000000..359e7850 --- /dev/null +++ b/docs/doctoring/durable-job-retry-boundary-evidence.md @@ -0,0 +1,71 @@ +# Durable-job retry-boundary doctoring evidence + +## Decision + +One durable worker claim represents exactly one persisted execution attempt. The worker must call +`EtlService.processDataInExistingTransaction` rather than the synchronous `processData` entry point. + +The synchronous endpoint keeps `@Retryable` outside `@Transactional` so each request retry can own a +fresh transaction. The durable worker already owns bounded retries through `attempt_count`, claim +renewal by a later poll, and lease-fenced lifecycle transitions. Its target writes, response-ledger +write, and exact-live-lease success transition must remain inside one caller-owned transaction. + +## Failure mode prevented + +Calling the synchronous `@Retryable` method from an already active durable execution transaction can +place retry advice inside an outer transaction that it did not create. After a transactional database +failure, another in-process invocation can reuse a rollback-only or otherwise failed transaction +instead of starting the fresh transaction assumed by Spring Retry. It also performs retries that are +not represented by the durable job's `attempt_count`. + +The fail-closed contract is therefore: + +```text +one database claim +→ one existing execution transaction +→ one ETL invocation +→ one success transition or one escaped failure +→ worker-owned durable retry decision +``` + +No in-process retry happens inside the lease transaction. A transient exception escapes to +`EtlJobWorker`, which either returns the exact live lease to `PENDING` while attempts remain or records +a stable terminal failure after the configured maximum. + +## Test-first evidence + +The regression contract was added before the production entry point existed: + +- `EtlJobIdempotencyRetryBoundaryTest` required durable execution to call + `processDataInExistingTransaction` exactly once and never call `processData`; +- `EtlServiceIdempotencyTransactionBoundaryTest` required the new entry point to reject direct use + without an actual transaction before JDBC or request-lock access. + +Production then added the non-retrying, transaction-requiring entry point and changed +`EtlJobIdempotencyService` to use it. Existing synchronous processing retains its retry behavior. + +## Review and operational evidence + +Reviewers should confirm all of the following on the exact current head: + +1. `processDataInExistingTransaction` has neither `@Retryable` nor `@Transactional`; +2. it fails closed when no actual Spring transaction is active; +3. `EtlJobIdempotencyService` invokes only that entry point for a newly executed job; +4. response replay does not invoke ETL target writes; +5. transient exceptions escape to `EtlJobWorker` and affect durable attempt accounting once; +6. target rows, response ledger, and `SUCCEEDED` remain atomic with the lease-fenced transition; +7. statement and branch coverage gates remain at 100% for the configured production scope. + +## Rollback + +If this change must be reverted, disable the durable worker first. Do not revert to calling the +synchronous retryable entry point while the worker is enabled. A safe replacement must preserve one +persisted attempt per claim and prove fresh-transaction semantics independently before deployment. + +## References — APA 7th + +Spring Retry Authors. (2026). *EnableRetry.java* [Source code]. GitHub. +https://github.com/spring-projects/spring-retry/blob/main/src/main/java/org/springframework/retry/annotation/EnableRetry.java + +Spring Retry Authors. (2026). *RetryOperationsInterceptor.java* [Source code]. GitHub. +https://github.com/spring-projects/spring-retry/blob/main/src/main/java/org/springframework/retry/interceptor/RetryOperationsInterceptor.java From 0b3faf216d59e80ee4b897cff02d7205ba3a306a Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 22:16:01 +0900 Subject: [PATCH 83/92] test(etl): require transactional success fencing --- ...obLeaseSuccessTransactionBoundaryTest.java | 61 +++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseSuccessTransactionBoundaryTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseSuccessTransactionBoundaryTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseSuccessTransactionBoundaryTest.java new file mode 100644 index 00000000..a1f740b0 --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseSuccessTransactionBoundaryTest.java @@ -0,0 +1,61 @@ +package com.xtrmetl.etl.job; + +import org.junit.jupiter.api.Test; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.transaction.PlatformTransactionManager; + +import java.time.Instant; +import java.util.UUID; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verifyNoInteractions; + +/** + * Guards the atomicity boundary of the durable-job success transition. + * + *

Marking a job successful is valid only in the database transaction that also contains the + * target effects and response-ledger write. A direct repository call without an actual transaction + * could otherwise publish a false terminal success after unrelated or absent target work.

+ */ +class EtlJobLeaseSuccessTransactionBoundaryTest { + + /** + * Proves success fails closed before JDBC when no caller-owned transaction is active. + */ + @Test + void refusesSuccessWithoutAnActiveTransactionBeforeDatabaseAccess() { + JdbcTemplate jdbcTemplate = mock(JdbcTemplate.class); + PlatformTransactionManager transactionManager = mock(PlatformTransactionManager.class); + EtlJobLeaseRepository repository = new EtlJobLeaseRepository( + jdbcTemplate, + transactionManager + ); + + IllegalStateException exception = assertThrows( + IllegalStateException.class, + () -> repository.markSucceeded(sampleLease()) + ); + + assertEquals( + "Durable ETL success requires an active transaction", + exception.getMessage() + ); + verifyNoInteractions(jdbcTemplate, transactionManager); + } + + private static EtlJobLease sampleLease() { + return new EtlJobLease( + UUID.randomUUID(), + UUID.randomUUID(), + "worker-alpha", + "a".repeat(64), + "b".repeat(64), + "c".repeat(64), + "[{\"id\":\"record_alpha\"}]", + 1, + Instant.now().plusSeconds(300) + ); + } +} From 9c1670ee4502d2f4f09caa936f4189f94cdcf0b4 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 22:19:15 +0900 Subject: [PATCH 84/92] fix(etl): require atomic success transaction --- .../xtrmetl/etl/job/EtlJobLeaseRepository.java | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java index 7842d248..f88f50b6 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java @@ -3,6 +3,7 @@ import org.springframework.jdbc.core.JdbcTemplate; import org.springframework.stereotype.Repository; import org.springframework.transaction.PlatformTransactionManager; +import org.springframework.transaction.support.TransactionSynchronizationManager; import org.springframework.transaction.support.TransactionTemplate; import java.time.Duration; @@ -24,6 +25,11 @@ * claim token, owner, running status, and database-time expiry predicates so stale workers cannot * mutate lifecycle state. Public callers cannot create leases longer than the worker's one-day * operational ceiling, even when they bypass Spring configuration binding.

+ * + *

Terminal success is intentionally stricter than retry or failure bookkeeping: it is accepted + * only inside the caller-owned transaction that also contains the target effects and durable + * response-ledger write. This prevents a direct repository call from publishing false success + * independently of the data it claims to have committed.

*/ @Repository public class EtlJobLeaseRepository { @@ -243,14 +249,16 @@ public Optional claimNext( } /** - * Commits terminal success only for the exact live claim. + * Commits terminal success only for the exact live claim in the atomic execution transaction. * * @param lease exact claim whose target effects completed in the same transaction * @throws NullPointerException when the lease is {@code null} + * @throws IllegalStateException when no actual Spring transaction is active * @throws StaleEtlJobLeaseException when the claim is expired or no longer authoritative */ public void markSucceeded(EtlJobLease lease) { EtlJobLease requiredLease = Objects.requireNonNull(lease, "lease must not be null"); + requireActiveSuccessTransaction(); requireTransition(jdbcTemplate.update( MARK_SUCCEEDED_SQL, requiredLease.jobRecordId(), @@ -352,6 +360,14 @@ private static String requireSafeFailureCode(String failureCode) { return requiredFailureCode; } + private static void requireActiveSuccessTransaction() { + if (!TransactionSynchronizationManager.isActualTransactionActive()) { + throw new IllegalStateException( + "Durable ETL success requires an active transaction" + ); + } + } + private static void requireTransition(int updatedRows) { if (updatedRows != 1) { throw new StaleEtlJobLeaseException(); From a7e16e62d3055c1e87f155b0d2e25681822446c1 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 22:21:56 +0900 Subject: [PATCH 85/92] test(etl): execute success fencing in its atomic transaction --- .../xtrmetl/etl/job/EtlJobLeaseRepositoryIntegrationTest.java | 3 +++ 1 file changed, 3 insertions(+) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryIntegrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryIntegrationTest.java index f9604681..231d2340 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryIntegrationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobLeaseRepositoryIntegrationTest.java @@ -13,6 +13,7 @@ import org.springframework.test.context.junit.jupiter.SpringJUnitConfig; import org.springframework.transaction.PlatformTransactionManager; import org.springframework.transaction.annotation.EnableTransactionManagement; +import org.springframework.transaction.annotation.Transactional; import javax.sql.DataSource; import java.time.Duration; @@ -193,6 +194,7 @@ void terminalizesExhaustedEligibleRowsBeforeLookingForWork() { } @Test + @Transactional void exactLiveLeaseCanSucceedRetryOrFailAndClearsTheRightFields() { UUID successJobId = insertPending(Instant.parse("2026-08-05T00:00:00Z"), 0); EtlJobLease successLease = repository.claimNext(OWNER_ALPHA, LEASE_DURATION, 3) @@ -226,6 +228,7 @@ void exactLiveLeaseCanSucceedRetryOrFailAndClearsTheRightFields() { } @Test + @Transactional void rejectsExpiredSupersededOrExhaustedTransitions() { UUID jobRecordId = insertPending(Instant.now(), 0); EtlJobLease lease = repository.claimNext(OWNER_ALPHA, LEASE_DURATION, 1).orElseThrow(); From b4fb3600f143a9f59bed1aaf93c44c2cf78bf925 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 22:23:10 +0900 Subject: [PATCH 86/92] docs(etl): record atomic success transaction gate --- .../durable-job-retry-boundary-evidence.md | 64 ++++++++++++++----- 1 file changed, 47 insertions(+), 17 deletions(-) diff --git a/docs/doctoring/durable-job-retry-boundary-evidence.md b/docs/doctoring/durable-job-retry-boundary-evidence.md index 359e7850..e40ddfdf 100644 --- a/docs/doctoring/durable-job-retry-boundary-evidence.md +++ b/docs/doctoring/durable-job-retry-boundary-evidence.md @@ -1,4 +1,4 @@ -# Durable-job retry-boundary doctoring evidence +# Durable-job retry and success-boundary doctoring evidence ## Decision @@ -10,7 +10,13 @@ fresh transaction. The durable worker already owns bounded retries through `atte renewal by a later poll, and lease-fenced lifecycle transitions. Its target writes, response-ledger write, and exact-live-lease success transition must remain inside one caller-owned transaction. -## Failure mode prevented +`EtlJobLeaseRepository.markSucceeded` therefore rejects direct invocation when no actual Spring +transaction is active. Retry and failure transitions remain independently persistable after an +execution exception, but success can never be published separately from the effects it certifies. + +## Failure modes prevented + +### Retry advice inside an existing durable transaction Calling the synchronous `@Retryable` method from an already active durable execution transaction can place retry advice inside an outer transaction that it did not create. After a transactional database @@ -18,14 +24,26 @@ failure, another in-process invocation can reuse a rollback-only or otherwise fa instead of starting the fresh transaction assumed by Spring Retry. It also performs retries that are not represented by the durable job's `attempt_count`. -The fail-closed contract is therefore: +### False success outside the atomic execution transaction + +A public success-transition method that can run in autocommit mode allows accidental callers to mark +a job `SUCCEEDED` without the target rows and response ledger being committed in the same unit of +work. Even an exact lease predicate cannot prove those effects exist. Requiring an actual transaction +before the success SQL executes makes the repository fail closed at its public boundary. + +The complete contract is: ```text one database claim → one existing execution transaction -→ one ETL invocation -→ one success transition or one escaped failure -→ worker-owned durable retry decision +→ one non-retrying ETL invocation +→ target rows + response ledger + exact-live-lease success +→ one atomic commit + +or + +one escaped failure +→ worker-owned durable retry / terminal-failure decision ``` No in-process retry happens inside the lease transaction. A transient exception escapes to @@ -34,15 +52,20 @@ a stable terminal failure after the configured maximum. ## Test-first evidence -The regression contract was added before the production entry point existed: +The regression contracts were added before their production behavior: - `EtlJobIdempotencyRetryBoundaryTest` required durable execution to call `processDataInExistingTransaction` exactly once and never call `processData`; -- `EtlServiceIdempotencyTransactionBoundaryTest` required the new entry point to reject direct use - without an actual transaction before JDBC or request-lock access. +- `EtlServiceIdempotencyTransactionBoundaryTest` required the durable ETL entry point to reject + direct use without an actual transaction before JDBC or request-lock access; +- `EtlJobLeaseSuccessTransactionBoundaryTest` required `markSucceeded` to reject use without an + actual transaction before JDBC access; +- `EtlJobLeaseRepositoryIntegrationTest` executes successful lease fencing inside a real Spring test + transaction and still verifies expiry and supersession rejection. -Production then added the non-retrying, transaction-requiring entry point and changed -`EtlJobIdempotencyService` to use it. Existing synchronous processing retains its retry behavior. +Production then added the non-retrying, transaction-requiring ETL entry point, changed +`EtlJobIdempotencyService` to use it, and added the active-transaction guard to the public success +transition. Existing synchronous processing retains its retry behavior. ## Review and operational evidence @@ -52,15 +75,19 @@ Reviewers should confirm all of the following on the exact current head: 2. it fails closed when no actual Spring transaction is active; 3. `EtlJobIdempotencyService` invokes only that entry point for a newly executed job; 4. response replay does not invoke ETL target writes; -5. transient exceptions escape to `EtlJobWorker` and affect durable attempt accounting once; -6. target rows, response ledger, and `SUCCEEDED` remain atomic with the lease-fenced transition; -7. statement and branch coverage gates remain at 100% for the configured production scope. +5. `EtlJobLeaseRepository.markSucceeded` fails before JDBC without an actual transaction; +6. `EtlJobExecutionService.execute` owns the transaction containing ETL, ledger, and success; +7. transient exceptions escape to `EtlJobWorker` and affect durable attempt accounting once; +8. retry and terminal-failure transitions remain exact-lease fenced; +9. target rows, response ledger, and `SUCCEEDED` roll back together when success fencing fails; +10. statement and branch coverage gates remain at 100% for the configured production scope. ## Rollback -If this change must be reverted, disable the durable worker first. Do not revert to calling the -synchronous retryable entry point while the worker is enabled. A safe replacement must preserve one -persisted attempt per claim and prove fresh-transaction semantics independently before deployment. +Disable the durable worker before reverting either boundary. Do not restore synchronous retry advice +inside the lease transaction, and do not allow `markSucceeded` to run in autocommit mode. A safe +replacement must preserve one persisted attempt per claim and prove that target effects, the response +ledger, and terminal success commit or roll back together. ## References — APA 7th @@ -69,3 +96,6 @@ https://github.com/spring-projects/spring-retry/blob/main/src/main/java/org/spri Spring Retry Authors. (2026). *RetryOperationsInterceptor.java* [Source code]. GitHub. https://github.com/spring-projects/spring-retry/blob/main/src/main/java/org/springframework/retry/interceptor/RetryOperationsInterceptor.java + +Spring Framework Authors. (2026). *TransactionSynchronizationManager.java* [Source code]. GitHub. +https://github.com/spring-projects/spring-framework/blob/main/spring-tx/src/main/java/org/springframework/transaction/support/TransactionSynchronizationManager.java From 26d110763d1d53c5bb068f28e15ad37ce9eb01d1 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 22:42:43 +0900 Subject: [PATCH 87/92] test(etl): forbid idempotency-owned execution transactions --- ...bIdempotencyTransactionAnnotationTest.java | 36 +++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobIdempotencyTransactionAnnotationTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobIdempotencyTransactionAnnotationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobIdempotencyTransactionAnnotationTest.java new file mode 100644 index 00000000..42fe752a --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobIdempotencyTransactionAnnotationTest.java @@ -0,0 +1,36 @@ +package com.xtrmetl.etl.job; + +import org.junit.jupiter.api.Test; +import org.springframework.transaction.annotation.Transactional; + +import java.lang.reflect.Method; + +import static org.junit.jupiter.api.Assertions.assertNull; + +/** + * Guards the durable response-ledger service from creating its own execution transaction. + * + *

The lease-fenced execution service owns the transaction that contains target effects, the + * response ledger, and terminal success. A transactional annotation on the public ledger method + * would let a direct Spring-proxy caller commit target and ledger effects without the success fence.

+ */ +class EtlJobIdempotencyTransactionAnnotationTest { + + /** + * Requires the public processing method to join, rather than create, the caller transaction. + * + * @throws NoSuchMethodException when the public durable processing contract is missing + */ + @Test + void processDoesNotCreateAStandaloneTransaction() throws NoSuchMethodException { + Method processMethod = EtlJobIdempotencyService.class.getMethod( + "process", + EtlJobLease.class + ); + + assertNull( + processMethod.getAnnotation(Transactional.class), + "Durable job idempotency must not own a transaction" + ); + } +} From c10bfdff61008523fa8da5ced0ee8c899e64ac49 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 22:44:51 +0900 Subject: [PATCH 88/92] fix(etl): join the lease-fenced execution transaction --- .../etl/job/EtlJobIdempotencyService.java | 22 ++++++++++++------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobIdempotencyService.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobIdempotencyService.java index 531a31e5..cc9d3965 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobIdempotencyService.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobIdempotencyService.java @@ -6,7 +6,6 @@ import org.springframework.dao.CannotAcquireLockException; import org.springframework.jdbc.core.JdbcTemplate; import org.springframework.stereotype.Service; -import org.springframework.transaction.annotation.Transactional; import org.springframework.transaction.support.TransactionSynchronizationManager; import java.util.List; @@ -19,12 +18,15 @@ * submission key. This service domain-separates and hashes those values into one response-ledger * key, verifies the exact retained payload digest, serializes execution with the existing * transaction-level request lock, replays an existing matching response, or writes the target and - * response ledger in the surrounding transaction. Raw principals and client keys are neither - * required nor reconstructed.

+ * response ledger in the caller-owned lease-fenced transaction. Raw principals and client keys are + * neither required nor reconstructed.

* - *

One invocation represents exactly one persisted durable attempt. It therefore calls the - * non-retrying ETL entry point that joins the current lease transaction; transient failures escape - * to {@link EtlJobWorker}, which owns bounded retry accounting in {@code attempt_count}.

+ *

One invocation represents exactly one persisted durable attempt. This service deliberately + * creates no transaction and performs no in-process retry. It joins the transaction owned by + * {@link EtlJobExecutionService}, calls the non-retrying ETL entry point once, and lets transient + * failures escape to {@link EtlJobWorker}, which owns bounded retry accounting in + * {@code attempt_count}. Direct Spring-proxy invocation without an existing transaction fails + * before lock or JDBC access.

*/ @Service public class EtlJobIdempotencyService { @@ -68,7 +70,12 @@ public EtlJobIdempotencyService( } /** - * Executes or replays one durable job inside a real database transaction. + * Executes or replays one durable job inside the caller's exact lease-fenced transaction. + * + *

The method has neither transaction-creation nor retry advice. The caller must establish + * the transaction that also contains terminal success fencing; otherwise execution fails + * before any request lock, target write, or response-ledger access. This prevents a direct + * proxy caller from committing durable effects without the lease-success predicate.

* * @param lease exact live claim carrying hashed execution identity and retained payload * @return newly generated or replayed stable response body @@ -77,7 +84,6 @@ public EtlJobIdempotencyService( * @throws EtlJobIntegrityException when retained payload or ledger identity conflicts * @throws CannotAcquireLockException when another transaction owns the execution ledger key */ - @Transactional public String process(EtlJobLease lease) { EtlJobLease requiredLease = Objects.requireNonNull(lease, "lease must not be null"); requireActiveTransaction(); From 3a7aa6350beea3cdeba49bf3d22f63f13faf6a5c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 22:46:29 +0900 Subject: [PATCH 89/92] test(etl): prove expired leases roll back durable effects --- ...EtlJobExecutionServiceIntegrationTest.java | 31 ++++++++++++++++--- 1 file changed, 27 insertions(+), 4 deletions(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobExecutionServiceIntegrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobExecutionServiceIntegrationTest.java index de034e77..9fd231e1 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobExecutionServiceIntegrationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobExecutionServiceIntegrationTest.java @@ -131,10 +131,26 @@ void rollsBackLedgerAndTargetRowsWhenTheClaimWasSuperseded() { assertThrows(StaleEtlJobLeaseException.class, () -> executionService.execute(lease)); - assertEquals(0, tableCount("processed_data")); - assertEquals(0, tableCount("etl_idempotency_records")); - assertEquals("RUNNING", jobStatus(jobRecordId)); - assertEquals(1, retainedPayloadCount(jobRecordId)); + assertUncommittedDurableEffects(jobRecordId); + } + + @Test + void rollsBackLedgerAndTargetRowsWhenTheLeaseExpiredBeforeExecution() { + UUID jobRecordId = insertPendingJob(); + EtlJobLease lease = leaseRepository.claimNext( + OWNER_ID, + Duration.ofMinutes(5), + 3 + ).orElseThrow(); + jdbcTemplate.update( + "UPDATE etl_job_records SET lease_expires_at = ? WHERE job_record_id = ?", + Instant.now().minusSeconds(1), + jobRecordId + ); + + assertThrows(StaleEtlJobLeaseException.class, () -> executionService.execute(lease)); + + assertUncommittedDurableEffects(jobRecordId); } @Test @@ -150,6 +166,13 @@ void rejectsMissingCollaboratorsOrLease() { assertThrows(NullPointerException.class, () -> executionService.execute(null)); } + private void assertUncommittedDurableEffects(UUID jobRecordId) { + assertEquals(0, tableCount("processed_data")); + assertEquals(0, tableCount("etl_idempotency_records")); + assertEquals("RUNNING", jobStatus(jobRecordId)); + assertEquals(1, retainedPayloadCount(jobRecordId)); + } + private UUID insertPendingJob() { UUID jobRecordId = UUID.randomUUID(); Instant now = Instant.now(); From cee009681ea0b1ce5f97fd63cd4fe37f18cbeef9 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 23:03:34 +0900 Subject: [PATCH 90/92] test(etl): run idempotency integration cases in transactions --- .../etl/job/EtlJobIdempotencyServiceIntegrationTest.java | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobIdempotencyServiceIntegrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobIdempotencyServiceIntegrationTest.java index 2e0ed2b7..adeb0867 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobIdempotencyServiceIntegrationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobIdempotencyServiceIntegrationTest.java @@ -17,6 +17,7 @@ import org.springframework.test.context.junit.jupiter.SpringJUnitConfig; import org.springframework.transaction.PlatformTransactionManager; import org.springframework.transaction.annotation.EnableTransactionManagement; +import org.springframework.transaction.annotation.Transactional; import javax.sql.DataSource; import java.nio.charset.StandardCharsets; @@ -89,6 +90,7 @@ request_digest CHAR(64) NOT NULL, } @Test + @Transactional void writesTargetAndLedgerThenReplaysWithoutASecondTargetWrite() { EtlJobLease lease = lease(PAYLOAD, sha256(PAYLOAD)); @@ -103,6 +105,7 @@ void writesTargetAndLedgerThenReplaysWithoutASecondTargetWrite() { } @Test + @Transactional void rejectsPayloadDigestMismatchBeforeLockOrWrites() { EtlJobLease lease = lease(PAYLOAD, "c".repeat(64)); @@ -118,6 +121,7 @@ void rejectsPayloadDigestMismatchBeforeLockOrWrites() { } @Test + @Transactional void rejectsConflictingStoredDigestWithoutAnotherTargetWrite() { EtlJobLease lease = lease(PAYLOAD, sha256(PAYLOAD)); idempotencyService.process(lease); @@ -133,6 +137,7 @@ void rejectsConflictingStoredDigestWithoutAnotherTargetWrite() { } @Test + @Transactional void reportsBusyLedgerAsTransientWithoutWrites() { when(requestLock.tryLock(anyString())).thenReturn(false); EtlJobLease lease = lease(PAYLOAD, sha256(PAYLOAD)); From f57256bcfb89c443415709a4af30ce7b190684e1 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 11:25:49 +0900 Subject: [PATCH 91/92] fix(stack): restore complete #121 tree in durable worker Rebuild the worker tree from GitHub's conflict-free synthetic merge of the prior #121 base and worker head, then apply the final authority-separation delta. This restores the updated workflow contract tests that were omitted by the earlier hand-built merge tree. --- ...exact-head-check-authorization-evidence.md | 280 +++++++++--------- .../operations/hourly-opencode-maintenance.md | 242 +++++++-------- ...-08-04-hourly-opencode-maintenance-plan.md | 244 ++++++--------- ...8-04-hourly-opencode-maintenance-design.md | 230 +++++++------- ...HourlyOpenCodeMaintenanceWorkflowTest.java | 246 +++++++-------- ...CodeRequiredWorkflowAuthorizationTest.java | 13 +- 6 files changed, 554 insertions(+), 701 deletions(-) diff --git a/docs/doctoring/github-token-exact-head-check-authorization-evidence.md b/docs/doctoring/github-token-exact-head-check-authorization-evidence.md index 96b6983a..221f4546 100644 --- a/docs/doctoring/github-token-exact-head-check-authorization-evidence.md +++ b/docs/doctoring/github-token-exact-head-check-authorization-evidence.md @@ -1,64 +1,105 @@ -# GitHub-token exact-head check authorization evidence +# GitHub-token publication and exact-head authorization evidence ## Decision -The hourly development workflow uses the repository-scoped `GITHUB_TOKEN` rather than a personal -access token or GitHub App installation token. When OpenCode creates or updates a same-repository -pull request with that token, mightyETL must explicitly authorize the resulting approval-required -pull-request workflow runs before treating the agent's work as ready for review. +The hourly development loop uses the repository-scoped `GITHUB_TOKEN`, but it does not expose the same GitHub authority to the model, the pull-request publisher, and the workflow-run authorizer. -Authorization is limited to starting validation for the exact current head. It is not pull-request -approval, review, merge authority, branch-protection bypass, or evidence that any check succeeded. -The OpenCode process itself never receives Actions write authority. +The deployed contract is: -## Platform behavior +```text +model execution and branch push + ≠ +draft pull-request publication + ≠ +workflow-run authorization + ≠ +independent review and merge +``` + +This separation addresses two concrete failure modes found during exact-head review: + +1. `pull-requests: write` on the OpenCode job allowed the model process to call review and merge endpoints even when the prompt prohibited it. +2. authorizing workflow runs by `head_sha` alone could authorize a run associated with another pull request that referenced the same commit. + +## Primary-source finding + +OpenCode 1.18.13 exposes two relevant execution paths. + +- `opencode github run` includes GitHub lifecycle behavior and can call the pull-request creation endpoint after committing and pushing scheduled work. +- `opencode run` is the plain non-interactive model runner and does not itself own GitHub pull-request publication. + +mightyETL therefore uses `opencode run --model ... --auto` under a read-only pull-request token. Deterministic jobs validate and publish the resulting branch separately. + +## Authority topology + +```mermaid +flowchart TB + M[maintain-repository] -->|candidate JSON| P[publish-agent-pull-request] + P -->|PR number, head ref, exact SHA| A[authorize-exact-head-checks] + A --> C[required CI and security workflows] + C --> R[independent OpenCode and Noema reviews] + R --> D[expected-head merge disposition] +``` + +### Model job + +`maintain-repository` is the only job that checks out source or runs OpenCode. It has: + +```text +actions: read +checks: read +contents: write +issues: write +pull-requests: read +security-events: read +statuses: read +``` + +The model can inspect pull requests and push one branch. It cannot create, update, approve, close, or merge a pull request through the token. The prompt prohibition remains defense in depth rather than the primary authorization boundary. + +### Deterministic publisher -GitHub prevents most events created with `GITHUB_TOKEN` from recursively starting another workflow. -For pull requests opened, synchronized, reopened, or updated through GitHub Actions, GitHub creates -pull-request workflow runs in an approval-required state rather than granting an automated writer an -unreviewed recursive execution path. The workflow-run approval endpoint requires Actions write -permission. +`publish-agent-pull-request` has: -Without a bounded authorization step, a scheduled OpenCode run could push a valid fix while leaving -the new exact head without CI, SAST, SBOM, dependency, or security execution. That would break the -required review → fix → exact-head revalidation loop even though the source change itself was valid. +```text +contents: read +pull-requests: write +``` + +It never checks out or executes repository code and never receives `NVIDIA_API_KEY`. It accepts only one structured candidate emitted by the model job. A new branch must: -A second timing problem also matters: GitHub materializes the several pull-request workflows -asynchronously. Stopping as soon as the first run appears can authorize CI while a later Security Scan -or SAST run remains approval-required indefinitely. The authorization job must therefore observe the -complete named workflow set, authorizing newly visible waiting runs on every discovery pass. +- match `automation/opencode-YYYYMMDDTHHMMSSZ-short-slug`; +- retain the captured exact SHA; +- be ahead of `develop`; +- change at most 50 files; +- avoid `.github/**` and all `CODEOWNERS` paths. -## Split-job authority model +The publisher creates a draft with a fixed JSON payload. It contains no pull-request review or merge endpoint. -The workflow separates mutable development from run authorization: +### Exact-head run authorizer + +`authorize-exact-head-checks` has: ```text -maintain-repository job - ├─ checks out protected default-branch source - ├─ installs and executes OpenCode - ├─ may prepare a feature branch and pull request - ├─ has actions: read - └─ outputs only the pre-agent pull-request head map - - job output boundary - ↓ - -authorize-exact-head-checks job - ├─ never checks out or executes repository code - ├─ receives no NVIDIA model credential - ├─ has actions: write, contents: read, pull-requests: read - ├─ compares before/after exact heads - └─ authorizes only approval-required exact-head workflow runs +actions: write +contents: read +pull-requests: read ``` -This prevents generated code, repository scripts, or the OpenCode process from using Actions write -permission. The sole privileged job consumes only GitHub API metadata and a compact JSON map of pull -request numbers to commit SHAs produced before the agent starts. +It never checks out source and never receives the model credential. Before authorizing a run, it requires: -## Required workflow set +```text +run.event == pull_request +run.head_sha == expected_head +any(run.pull_requests; number == expected_pull_request_number) +live pull request head == expected_head +``` + +The `pull_requests` association is mandatory. A commit SHA is not a unique pull-request identity: more than one pull request can reference the same commit. -For a direct pull request to `develop`, exact-head validation is incomplete until all of these -pull-request workflows have materialized: +## Complete workflow materialization + +GitHub can materialize pull-request workflows asynchronously. The authorizer repeats bounded discovery and authorizes newly visible `action_required` or `waiting` runs on every pass. It succeeds only after this complete workflow-name set is associated with the exact pull request and exact SHA: ```text CI @@ -68,117 +109,62 @@ SAST Semgrep Security Scan ``` -These are workflow names, not conclusions. Their presence proves only that validation was created for -the head. Every run and named check must still complete successfully through the ordinary repository -policy before merge. A workflow rename or required-workflow change must update the contract test, -this evidence document, and the authorization list in one reviewed change. - -## Fail-closed authorization algorithm - -The protected default-branch workflow performs the following steps: - -1. Before OpenCode starts, snapshot the exact heads of all same-repository pull requests targeting - `develop` and expose that compact object as the maintenance job's output. -2. After the maintenance job completes or fails without cancellation, start the isolated - authorization job. -3. Require the prior output to exist and parse as a JSON object. -4. Enumerate the same pull-request set again. -5. Select only a new pull request or a pull request whose head changed during this run. -6. Refuse automatic run authorization when the pull request changes any path below `.github/` or - any `CODEOWNERS` file. Those policy changes require explicit human authorization. -7. Read the still-current pull-request head and require it to equal the selected expected SHA. -8. Repeatedly discover only `pull_request` workflow runs whose `head_sha` equals that expected SHA. -9. On each discovery pass, authorize every exact-head run still in `action_required` or `waiting` - state. -10. Compare the unique observed workflow names with the complete required workflow set. -11. Continue bounded discovery until every required workflow has materialized or the discovery - window expires. -12. Fail visibly when no run appears or any required workflow remains absent. -13. Re-read the pull-request head immediately before declaring authorization complete and reject a - moved head. -14. Leave check execution, review, mergeability, branch protection, and expected-head merge - disposition to their existing independent gates. - -The isolated job runs even when OpenCode fails, provided the workflow was not cancelled. This covers -a partial agent session that pushed a branch before later failing. If the pre-run snapshot is absent, -run discovery fails, the head moves, a policy file changed, the named workflow set is incomplete, or -GitHub rejects authorization, the workflow fails instead of reporting successful revalidation. - -## Authority and credential boundary - -The workflow-level permission remains `contents: read`. The maintenance job retains only the -minimum branch, pull-request, issue, check, status, security-read, and Actions-read permissions needed -for development. The separate authorization job receives the workflow's only `actions: write` -permission plus read-only contents and pull-request metadata access. No personal token, GitHub App -token, OIDC token, or additional repository secret is introduced. - -The authorization job never checks out the repository, never executes repository files, and never -receives `NVIDIA_API_KEY`. Its Actions write permission is used solely for the workflow-run approval -endpoint. The implementation contains no pull-request review approval command and no merge API call. -The OpenCode prompt continues to forbid approval, merge, protected-branch push, branch-protection -bypass, review-agent modification, and unauthorized workflow-policy changes. +Name presence proves only that a run was created. Every run must still complete successfully before merge. + +## Time-of-check/time-of-use controls + +The loop validates state at several points: + +1. snapshot `develop`, open pull-request heads, and prior automation branches before model execution; +2. require `develop` to remain unchanged after the model exits; +3. select at most one changed existing pull request or strict automation branch; +4. re-read the branch or pull request before draft publication; +5. re-read the pull request before workflow-run discovery; +6. re-read the exact head on every discovery pass; +7. re-read it once more before declaring authorization complete. + +A moved head, multiple candidate, invalid namespace, policy-file change, absent required workflow, or GitHub authorization rejection fails closed. ## Test-first evidence -`HourlyOpenCodeMaintenanceWorkflowTest` first required the following contracts before production -implemented them: - -- a pre-agent exact-head snapshot exported as a job output; -- the absence of Actions write authority from the OpenCode job; -- exactly one isolated authorization job with Actions write permission; -- no checkout or NVIDIA credential in that authorization job; -- same-repository and `develop` targeting; -- refusal of `.github/**` and `CODEOWNERS` changes; -- exact-head workflow-run discovery; -- two head-SHA time-of-check/time-of-use validations; -- explicit failure when no run materializes; -- authorization through the workflow-run endpoint only; -- continued absence of pull-request approval and merge operations. - -`HourlyOpenCodeRequiredWorkflowAuthorizationTest` then required the complete five-workflow set, -repeated bounded discovery, repeated waiting-run authorization, observed/missing workflow evidence, -and explicit failure when the set remains incomplete. Production implemented those contracts with -job outputs, `gh api`, canonical JSON processing through `jq`, and exact SHA and workflow-name arrays -passed as data rather than interpolated into jq source. - -## Verification checklist - -Reviewers must verify on the exact current pull-request head that: - -- the scheduler still runs only from protected default-branch workflow source; -- the snapshot precedes the OpenCode process and is the only cross-job mutable evidence; -- the maintenance job has `actions: read`, not `actions: write`; -- the authorization job is the only job with `actions: write`; -- the authorization job has no checkout, repository-code execution, or NVIDIA credential; -- only heads changed by that run are considered; -- `.github/**` and all `CODEOWNERS` paths are excluded from automatic authorization; -- both current-head reads equal the expected head; -- run discovery filters `event=pull_request` and the exact head SHA; -- waiting or action-required runs are authorized on every discovery pass; -- CI, Dependency Review, SBOM, SAST, and Security Scan all materialize before success is reported; -- absent runs, missing named workflows, and authorization failures make the workflow fail; -- no review approval, merge, protected-branch push, or secret fallback was added; -- every authorized run must still complete successfully before merge disposition can proceed. +`HourlyOpenCodeMaintenanceWorkflowTest` was changed before production to require: + +- plain `opencode run`, not the GitHub lifecycle handler; +- `pull-requests: read` on the model job; +- exactly one non-checkout publisher with `pull-requests: write`; +- exactly one non-checkout authorizer with `actions: write`; +- no NVIDIA credential in either privileged deterministic job; +- one strict publication candidate; +- draft-only deterministic publication; +- `.github/**` and `CODEOWNERS` exclusion; +- exact pull-request association for every workflow run; +- complete required-workflow materialization; +- continued absence of review and merge endpoints. + +The initial test commit intentionally made the existing workflow contract fail. Production was then changed to satisfy the new authority and association requirements. + +## Residual risks and controls + +- `contents: write` remains necessary for a branch push. Protected-branch rules remain authoritative for `develop` and `main`. +- The deterministic publisher necessarily has coarse pull-request write permission. It has no checkout, model input, or executable repository source and its script exposes only draft creation or metadata validation. +- Workflow-run approval is an Actions control, not a successful check or pull-request approval. +- A workflow or `CODEOWNERS` change always requires explicit human authorization. +- Independent exact-head approval remains mandatory after every new commit. ## Rollback -If GitHub changes the approval-required run model or the endpoint becomes unavailable, disable the -hourly development workflow. Do not remove the exact-head authorization contract while leaving the -agent able to push changes with `GITHUB_TOKEN`, because that recreates unvalidated agent heads. +Disable the hourly workflow if the platform's `GITHUB_TOKEN` recursion or workflow-run approval model changes. Do not restore pull-request write authority to the model job and do not revert to SHA-only run authorization. -Do not move `actions: write` back into the OpenCode job. A replacement based on a GitHub App may -remove the isolated authorization job only after its installation permissions, recursive-trigger -behavior, actor identity, secret lifecycle, exact-head workflow evidence, complete required-run -materialization, and independent review boundary are documented and tested through a separate pull -request. +A GitHub App or endpoint proxy may replace the publisher only after its endpoint allowlist, actor identity, installation scope, token lifetime, audit log, and exact-head behavior are independently tested and documented. ## References — APA 7th -GitHub, Inc. (2026). *GITHUB_TOKEN*. GitHub Docs. -https://docs.github.com/en/actions/concepts/security/github_token +Anomaly. (2026). *GitHub handler (Version 1.18.13)* [Source code]. GitHub. https://github.com/anomalyco/opencode/blob/v1.18.13/packages/opencode/src/cli/cmd/github.handler.ts + +Anomaly. (2026). *Run command (Version 1.18.13)* [Source code]. GitHub. https://github.com/anomalyco/opencode/blob/v1.18.13/packages/opencode/src/cli/cmd/run.ts + +GitHub, Inc. (2026). *GITHUB_TOKEN*. GitHub Docs. https://docs.github.com/en/actions/concepts/security/github_token -GitHub, Inc. (2026). *Triggering a workflow*. GitHub Docs. -https://docs.github.com/en/actions/using-workflows/triggering-a-workflow +GitHub, Inc. (2026). *REST API endpoints for workflow runs*. GitHub Docs. https://docs.github.com/en/rest/actions/workflow-runs -GitHub, Inc. (2026). *REST API endpoints for workflow runs*. GitHub Docs. -https://docs.github.com/en/rest/actions/workflow-runs +GitHub, Inc. (2026). *Security hardening for GitHub Actions*. GitHub Docs. https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions diff --git a/docs/operations/hourly-opencode-maintenance.md b/docs/operations/hourly-opencode-maintenance.md index 2654f836..6d72a119 100644 --- a/docs/operations/hourly-opencode-maintenance.md +++ b/docs/operations/hourly-opencode-maintenance.md @@ -2,35 +2,35 @@ ## Purpose -`.github/workflows/hourly-opencode-maintenance.yml` runs at minute 43 of every hour, in UTC. It uses a checksum-pinned OpenCode 1.18.13 executable and the repository's existing `NVIDIA_NIM_API_KEY` to repair one dependency-eligible development pull request or prepare one bounded buyer-visible product improvement. +`.github/workflows/hourly-opencode-maintenance.yml` runs at minute 43 of every UTC hour. It uses a checksum-pinned OpenCode 1.18.13 executable and the existing `NVIDIA_NIM_API_KEY` repository secret to repair one dependency-eligible development branch or prepare one bounded buyer-visible improvement. -The workflow is separate from independent review and deterministic merge disposition. It never approves or merges a pull request, pushes to `develop` or `main`, weakens branch protection, changes the review agent's credential path, or publishes a release. +The workflow is not a reviewer or merger. Independent review, required checks, branch protection, the central CWL review workflows, and deterministic expected-head merge disposition remain authoritative. -## Required secret and model +## Credential and model boundary -The only model secret referenced by this workflow is: +The only model secret referenced by the workflow is: ```text NVIDIA_NIM_API_KEY ``` -The OpenCode step maps it to: +It is mapped only inside the model-execution step to: ```text NVIDIA_API_KEY ``` -A step-level environment variable is visible to that step's Bash shell and every child process, including OpenCode. The workflow has no GitHub Copilot, Anthropic, OpenAI, partner-only NVIDIA, or automatic model fallback. A missing secret fails before the agent starts. +The selected model is `nvidia/deepseek-ai/deepseek-v4-pro`. There is no GitHub Copilot credential, `COPILOT_GITHUB_TOKEN`, Anthropic or OpenAI key, partner-only endpoint, or automatic provider fallback. A missing NVIDIA credential fails before model execution. -The pinned provider/model is: +The workflow invokes the plain non-interactive command: ```text -nvidia/deepseek-ai/deepseek-v4-pro +opencode run --model "${MODEL}" --auto ``` -Current model-selection evidence and replacement rules are recorded in `docs/doctoring/nvidia-opencode-model-selection-evidence.md`. +It deliberately does not invoke `opencode github run`. OpenCode's GitHub schedule handler can create a pull request itself, which would require giving the model process coarse `pull-requests: write` authority. Plain `opencode run` lets the model edit, test, commit, and push one branch while deterministic non-model jobs own publication and workflow-run authorization. -## Immutable execution contract +## Immutable OpenCode installation | Control | Value | | --- | --- | @@ -40,58 +40,81 @@ Current model-selection evidence and replacement rules are recorded in `docs/doc | SHA-256 | `8d500b20fed2d26e537e221895b1a575476571b4f0089bb29fb13eeb8eb9e937` | | Archive shape | exactly one regular-file member named `opencode` | | Checkout | full-SHA-pinned `actions/checkout` | -| OpenCode timeout | `TERM` at 45 minutes, `KILL` after 30 seconds | +| OpenCode timeout | `TERM` after 45 minutes; `KILL` after 30 seconds | | Job timeout | 50 minutes | -| Overlap | serialized, active run not cancelled | -| Session sharing | disabled | +| Overlap | serialized; an active run is not cancelled | -The installer downloads only the immutable GitHub release asset over HTTPS, verifies its checksum, requires exactly one member, requires GNU tar's regular-file type before extraction, extracts into a fresh mode-`0700` directory without restoring archive ownership or permissions, refuses overwrite, rejects symbolic links and non-regular output, and verifies the exact executable version. It does not use npm installation, a floating package version, or a mutable OpenCode action reference. Detailed evidence is in `docs/doctoring/opencode-archive-extraction-evidence.md`. +The installer verifies the immutable archive checksum, member count, member name, GNU tar regular-file type, private mode-`0700` extraction directory, overwrite refusal, non-symbolic-link output, and exact executable version before running OpenCode. Checkout keeps `persist-credentials: false`. -## Direct-token Git bootstrap +## Three-job authority separation -`USE_GITHUB_TOKEN=true` makes OpenCode use the repository-scoped token rather than an OpenCode App or OIDC token. Checkout keeps `persist-credentials: false`. Before OpenCode starts, the maintenance job: +```mermaid +flowchart LR + A[maintain-repository
model + checked-out source] -->|candidate JSON only| B[publish-agent-pull-request
no checkout, no model secret] + B -->|PR number + exact SHA| C[authorize-exact-head-checks
no checkout, no model secret] + C --> D[CI and security checks] + D --> E[independent review and merge disposition] +``` -1. requires `GITHUB_TOKEN` and `GH_TOKEN`; -2. removes inherited repository-local GitHub credential helpers; -3. installs a repository-local `!gh auth git-credential` helper; -4. sets the local author to `opencode-agent[bot]`; -5. registers an `EXIT` trap that removes the helper after success, failure, or timeout. +### `maintain-repository` -No plaintext or encoded token is stored in Git configuration. No personal token or additional repository secret is introduced. +This is the only job that executes checked-out repository code and OpenCode. Its pull-request permission is read-only: -## Split-job permission model +```text +actions: read +checks: read +contents: write +issues: write +pull-requests: read +security-events: read +statuses: read +``` -The workflow-level permission is only `contents: read`. +`contents: write` permits one feature-branch push. It does not grant pull-request review or merge endpoints. The model prompt additionally forbids direct pull-request creation, update, approval, closure, or merge, but the permission map—not the prompt—is the primary authorization boundary. -### `maintain-repository` +Before OpenCode starts, the job snapshots: -This job checks out and executes repository and agent code. It receives: +- the `develop` head; +- every same-repository open pull request targeting `develop` and its exact head; +- every existing `automation/opencode-*` branch and its exact head. -- `actions: read` -- `checks: read` -- `contents: write` -- `issues: write` -- `pull-requests: write` -- `security-events: read` -- `statuses: read` +After OpenCode exits, the job requires `develop` to remain unchanged and selects at most one candidate: -It does **not** receive Actions write permission. Therefore OpenCode, generated code, and checked-out repository scripts cannot authorize workflow runs. +- one existing pull request whose head moved; or +- one new or advanced branch matching `automation/opencode-YYYYMMDDTHHMMSSZ-short-slug`. -### `authorize-exact-head-checks` +Multiple candidates fail closed. No candidate is represented as a no-op, not as successful product development. -This separate job receives: +### `publish-agent-pull-request` + +This job has `contents: read` and the workflow's sole `pull-requests: write` grant. It never checks out or executes repository code and never receives `NVIDIA_API_KEY`. + +For an existing pull request, it only re-reads and validates repository, base branch, head branch, state, and exact SHA. For a new branch, it additionally requires: + +- the strict `automation/opencode-*` namespace; +- the live branch SHA to equal the candidate SHA; +- at least one commit ahead of `develop`; +- no more than 50 changed files; +- no `.github/**` or `CODEOWNERS` change; +- a same-repository branch and `develop` base. + +It then creates one draft pull request through a fixed script-generated payload. Commit text may supply the title, but no untrusted source is executed. The job contains no review submission or merge endpoint. + +### `authorize-exact-head-checks` -- `actions: write` -- `contents: read` -- `pull-requests: read` +This job has `actions: write`, `contents: read`, and `pull-requests: read`. It never checks out repository code and never receives the model credential. Its only write operation is approval of GitHub Actions workflow runs that GitHub has placed in `action_required` or `waiting` state. -It never checks out or executes repository code and never receives `NVIDIA_API_KEY`. Its only write operation is the GitHub workflow-run approval endpoint for an approval-required run bound to a verified exact head. It contains no pull-request review approval or merge operation. +Every eligible run must satisfy all of the following: -## Why exact-head run authorization is required +1. event is `pull_request`; +2. `head_sha` equals the publisher's exact SHA; +3. the run's `pull_requests` association contains the exact pull-request number; +4. the pull request still has the expected repository, base, head branch, and SHA; +5. the pull request does not change `.github/**` or `CODEOWNERS`. -GitHub prevents most events created with `GITHUB_TOKEN` from recursively triggering workflows. Pull-request runs created after an Actions-authored open or synchronize event can remain approval-required. Without explicit authorization, an agent could push a valid repair whose exact head never receives CI, dependency review, SBOM, SAST, or security scans. +The pull-request association check matters because two pull requests can reference the same commit SHA. SHA-only filtering could authorize another pull request's waiting workflow. -GitHub materializes those workflows asynchronously. Seeing and authorizing only the first run is not sufficient: later workflows can appear after the authorization step exits and remain waiting forever. The isolated job therefore requires this complete workflow-name set for a direct `develop` pull request: +The job repeatedly discovers and authorizes runs until all of the following names materialize or the bounded wait expires: ```text CI @@ -101,125 +124,76 @@ SAST Semgrep Security Scan ``` -These names prove only that the runs exist for the head. They do not make a run successful and do not replace the named check requirements enforced by merge disposition. +Run authorization starts validation only. It does not make a check successful, approve the pull request, or permit merge. -The maintenance job snapshots same-repository pull requests targeting `develop` before OpenCode starts and exports only the compact pull-request-number-to-head-SHA map. The isolated authorization job runs after the maintenance job succeeds or fails without cancellation and: +## Agent development contract -1. requires the snapshot to exist and parse as a JSON object; -2. enumerates the same pull-request set after the agent run; -3. selects only a new pull request or a head changed by that run; -4. refuses automatic authorization for any `.github/**` or `CODEOWNERS` change; -5. verifies that the current head equals the expected SHA; -6. repeatedly discovers only `pull_request` workflow runs for that exact SHA; -7. authorizes every visible run still in `action_required` or `waiting` state on each pass; -8. compares observed names with the complete five-workflow set; -9. continues bounded discovery until all five materialize; -10. fails if no run appears or any required workflow remains missing; -11. verifies the head again immediately before authorization is declared complete. +When an eligible pull request exists, OpenCode may update only that same-repository head branch. When none exists, it may create exactly one strict `automation/opencode-*` branch. It must work test-first, preserve 100% configured production statement and branch coverage, add beginner-readable public documentation, update `CHANGELOG.md`, use descriptive multi-word `snake_case` database names, and record current primary standards or peer-reviewed evidence in APA 7th form where material. -The expected SHA and workflow-name arrays are passed to `jq` as data, not interpolated into jq source. Authorization only starts validation. Every check must still complete successfully, all review threads must be resolved, a non-author approval must be anchored to the same head, and branch protection and expected-head merge disposition must still permit merge. +The model must not: -Test-first, least-privilege, complete-materialization, time-of-check/time-of-use, and rollback evidence is recorded in `docs/doctoring/github-token-exact-head-check-authorization-evidence.md`. - -## Agent authority boundary - -The agent may inspect open pull requests and their exact heads, fix one dependency-eligible development pull request, or create one bounded product pull request when no development pull request exists. It must use current authoritative standards and primary documentation, add APA 7th references where material, preserve modular MSA operation, use descriptive multiword `snake_case` database names, add beginner-readable production documentation, maintain deterministic statement and branch coverage, update `CHANGELOG.md`, and report incomplete gates truthfully. +- mutate pull-request lifecycle state; +- push to `develop` or `main`; +- bypass checks, reviews, security gates, or branch protection; +- modify the existing review agent or its credential names; +- inspect or disclose secret values; +- modify `.github/**` or `CODEOWNERS` without a specifically authorized automation-maintenance issue; +- publish a release. -The agent must not: +## Failure behavior -- approve or merge a pull request; -- push directly to protected branches; -- treat pending, absent, cancelled, skipped-required, stale, neutral-required, or failed checks as passing; -- bypass review, security, coverage, or branch-protection policy; -- inspect or disclose secret values; -- change the existing review agent, its provider, workflow, credential flow, or secret names; -- create a second development pull request while one is dependency-eligible; -- modify workflow policy without a specifically authorized `automation-maintenance` issue; -- publish a release without a separately authorized release workflow and complete acceptance evidence. - -Even with an authorized automation issue, `.github/**` and `CODEOWNERS` changes are excluded from automatic run authorization and require human action. - -## Normal sequence - -1. GitHub starts the scheduled workflow from protected default-branch source. -2. OpenCode is installed from the immutable checksum-pinned archive. -3. The maintenance job snapshots current same-repository `develop` pull-request heads. -4. The NVIDIA and GitHub credential boundaries are validated and the removable Git helper is installed. -5. OpenCode reviews the current queue, executes one bounded test-first change, and leaves a feature branch or pull request. -6. The credential cleanup trap removes the local helper. -7. The isolated authorization job compares before and after heads. -8. Policy-changing pull requests are rejected from automatic run authorization. -9. Exact-head approval-required runs are repeatedly authorized while all five required workflow names materialize. -10. CI, security, coverage, independent review, and merge disposition operate separately. - -## Failure handling - -| Failure | Result | Required response | -| --- | --- | --- | -| NVIDIA secret missing | Fail before OpenCode | Restore `NVIDIA_NIM_API_KEY`; add no fallback | -| Repository token or Git helper unavailable | Fail visibly | Preserve `persist-credentials: false`; inspect runner tooling | -| Archive unavailable, checksum mismatch, unexpected member/type, or version mismatch | Fail before execution | Treat as supply-chain review; never relax the pin silently | -| Model endpoint unavailable or deprecated | Fail without fallback | Research a current NVIDIA endpoint and submit a reviewed test-first change | -| OpenCode exceeds 45 minutes | TERM then KILL; cleanup trap executes | Reduce slice size and inspect partial branch state | -| Pre-agent head output missing or invalid | Authorization job fails | Never authorize from an unknown baseline | -| Agent changes `.github/**` or `CODEOWNERS` | Automatic authorization refused | Require explicit human workflow-run authorization | -| Head moves during discovery or authorization | Authorization refused | Re-evaluate the new exact head | -| No exact-head run materializes | Workflow fails | Diagnose event and Actions policy; do not merge the head | -| A required workflow name remains absent | Workflow fails and lists missing names | Diagnose trigger filters or renamed workflows; update the contract only through review | -| Workflow-run approval is rejected | Workflow fails | Verify repository policy and token permissions; add no personal-token workaround | -| Checks or review fail | Pull request remains blocked | Fix the exact head without weakening the gate | -| A prior hourly run remains active | New run waits | Investigate only if the prior run is stuck | - -A failed run can leave a reviewable feature branch or pull request, but it cannot claim successful validation, approval, merge, or release. +| Failure | Result | +| --- | --- | +| NVIDIA credential missing | fail before model execution | +| OpenCode archive or checksum mismatch | fail before extraction or execution | +| protected `develop` head moves during the run | fail as indeterminate publication evidence | +| more than one candidate branch or PR changes | fail as ambiguous model output | +| candidate branch is not ahead of `develop` | refuse publication | +| candidate changes `.github/**` or `CODEOWNERS` | require explicit human handling | +| live PR or branch SHA differs from the captured SHA | fail closed | +| workflow run lacks exact PR association | exclude it from authorization | +| required workflow name never materializes | fail and list missing names | +| exact PR head moves during discovery | fail and require re-evaluation | +| any check or independent review fails | leave the PR unmerged | + +A failed OpenCode step is preserved as a failed job after candidate evidence is captured. A deterministic publisher may still expose a valid partial branch as a draft for review, but the workflow never calls that a successful maintenance run. ## Rollback -Disable **Hourly OpenCode maintenance** to stop execution immediately. Permanent rollback must revert the workflow, contract tests, operations document, doctoring evidence, plan/design documents, and `CHANGELOG.md` through a reviewed pull request. +Disable **Hourly OpenCode maintenance** to stop the schedule immediately. Permanent rollback must revert the workflow, contract tests, this operations document, doctoring evidence, design and plan records, and corresponding changelog material through an independently reviewed pull request. -Do not remove exact-head authorization while retaining agent writes through `GITHUB_TOKEN`, and do not move `actions: write` into the OpenCode job. A GitHub App replacement requires separately reviewed evidence for installation permissions, recursive triggers, actor identity, secret lifecycle, complete exact-head validation, and independent review. +Do not restore `pull-requests: write` to the model job. Do not remove exact pull-request association from workflow-run authorization. A replacement GitHub App or token broker requires separate evidence for endpoint-level capability, actor identity, secret lifecycle, exact-head binding, and independent review. ## Verification checklist -Before merge, verify on the exact current head: +Before merge, verify on the exact head: -- Ubuntu, macOS, and Windows CI succeeded; -- dependency review, SBOM, Semgrep, Trivy, OSV, Scorecard, and required security gates succeeded; -- no current unresolved review thread or requested change remains; -- a non-author approval is anchored to the exact head; -- required workflow-change labels are present; +- Ubuntu, macOS, and Windows CI succeed; +- dependency review, SBOM, Semgrep, Trivy, OSV, Scorecard, and required security gates succeed; +- all current review threads are resolved; +- a non-author approval is anchored to the exact current SHA; - only `NVIDIA_NIM_API_KEY` is referenced as a model secret; -- the current NVIDIA endpoint and immutable OpenCode pin remain valid; -- archive member, type, private extraction, overwrite, output-type, and version checks remain intact; -- OpenCode retains no Actions write authority; -- the isolated authorization job is the only holder of `actions: write` and performs no checkout; -- the before/after head output, `.github/**` and `CODEOWNERS` exclusion, exact-head run filter, double head check, and visible absent-run failure remain intact; -- all five required pull-request workflows must materialize before the authorization job reports completion; -- the development workflow contains no review approval, merge, protected-branch push, fallback credential, or review-agent modification. +- the immutable OpenCode archive and action pins remain unchanged; +- the model job has `pull-requests: read`, not write; +- the publisher is the only holder of `pull-requests: write` and performs no checkout; +- the authorizer is the only holder of `actions: write` and performs no checkout; +- every authorized run is associated with the exact pull-request number and SHA; +- the workflow contains no pull-request review or merge operation. ## References — APA 7th Anomaly. (2026). *GitHub handler (Version 1.18.13)* [Source code]. GitHub. https://github.com/anomalyco/opencode/blob/v1.18.13/packages/opencode/src/cli/cmd/github.handler.ts -Anomaly. (2026). *OpenCode release v1.18.13* [Software release]. GitHub. https://github.com/anomalyco/opencode/releases/tag/v1.18.13 - -Anomaly. (2026). *GitHub integration*. OpenCode. https://opencode.ai/docs/github/ +Anomaly. (2026). *Run command (Version 1.18.13)* [Source code]. GitHub. https://github.com/anomalyco/opencode/blob/v1.18.13/packages/opencode/src/cli/cmd/run.ts -Anomaly. (2026). *Providers*. OpenCode. https://opencode.ai/docs/providers/ +Anomaly. (2026). *OpenCode release v1.18.13* [Software release]. GitHub. https://github.com/anomalyco/opencode/releases/tag/v1.18.13 Free Software Foundation. (2023). *GNU tar 1.35: Security*. https://www.gnu.org/software/tar/manual/html_section/Security.html -Free Software Foundation. (2026). *timeout: Run a command with a time limit*. GNU Coreutils 9.11. https://www.gnu.org/software/coreutils/manual/html_node/timeout-invocation.html - GitHub, Inc. (2026). *GITHUB_TOKEN*. GitHub Docs. https://docs.github.com/en/actions/concepts/security/github_token -GitHub, Inc. (2026). *GitHub CLI manual: gh auth git-credential*. GitHub CLI Manual. https://cli.github.com/manual/gh_auth_git-credential - GitHub, Inc. (2026). *REST API endpoints for workflow runs*. GitHub Docs. https://docs.github.com/en/rest/actions/workflow-runs GitHub, Inc. (2026). *Security hardening for GitHub Actions*. GitHub Docs. https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions -GitHub, Inc. (2026). *Triggering a workflow*. GitHub Docs. https://docs.github.com/en/actions/using-workflows/triggering-a-workflow - NVIDIA Corporation. (2026). *DeepSeek V4 Pro*. NVIDIA NIM API catalog. https://build.nvidia.com/deepseek-ai/deepseek-v4-pro - -NVIDIA Corporation. (2026). *DeepSeek AI / DeepSeek V4 Pro*. NVIDIA NIM API reference. https://docs.api.nvidia.com/nim/reference/deepseek-ai-deepseek-v4-pro diff --git a/docs/superpowers/plans/2026-08-04-hourly-opencode-maintenance-plan.md b/docs/superpowers/plans/2026-08-04-hourly-opencode-maintenance-plan.md index 26978a27..f3f17985 100644 --- a/docs/superpowers/plans/2026-08-04-hourly-opencode-maintenance-plan.md +++ b/docs/superpowers/plans/2026-08-04-hourly-opencode-maintenance-plan.md @@ -1,190 +1,128 @@ # Hourly OpenCode Maintenance Agent Implementation Plan -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. +> **Execution rule:** implement each authority boundary test-first, then verify the final exact head through repository CI, security, and independent review. -**Goal:** Add a fail-closed hourly OpenCode development agent that uses `NVIDIA_NIM_API_KEY`, prepares one bounded development pull request, starts validation for an agent-written exact head, and never changes the independent review agent or deterministic merge authority. +**Goal:** Run one bounded NVIDIA NIM-backed development loop every hour while ensuring model execution cannot create, approve, close, or merge pull requests and cannot authorize GitHub Actions runs. -**Architecture:** A protected default-branch GitHub Actions workflow has two jobs. `maintain-repository` installs a checksum-verified immutable OpenCode release and may prepare one feature branch without Actions write permission. `authorize-exact-head-checks` never checks out repository code, receives no model secret, and uses the sole Actions write grant to authorize only approval-required pull-request workflow runs for an unchanged exact head. Existing review, security, branch protection, and merge-disposition automation remain independent and authoritative. - -**Tech Stack:** GitHub Actions, OpenCode 1.18.13 immutable release archive, NVIDIA NIM, DeepSeek V4 Pro, GitHub CLI credential helper, GNU Coreutils, GNU tar, Bash, jq, Maven, JUnit 5. +**Architecture:** Three jobs separate model execution, deterministic draft-PR publication, and exact-head workflow-run authorization. Independent OpenCode/Noema review and expected-head merge disposition remain outside all three jobs. ## Global constraints -- Keep the review-agent provider, workflow, credential flow, and secret names unchanged. -- Preserve `.github/workflows/hourly-pr-disposition.yml` as the independent exact-head merge boundary. -- Use only `${{ secrets.NVIDIA_NIM_API_KEY }}` for the model credential and expose it as `NVIDIA_API_KEY` only to the OpenCode step. -- Pin `MODEL: nvidia/deepseek-ai/deepseek-v4-pro`; reject deprecated or non-NVIDIA fallback identifiers. -- Pin executable content and third-party workflow sources immutably. -- Verify OpenCode 1.18.13 Linux x64 SHA-256 `8d500b20fed2d26e537e221895b1a575476571b4f0089bb29fb13eeb8eb9e937` before extraction. -- Require exactly one regular archive entry named `opencode`; use a fresh mode-`0700` directory, refuse overwrites, and reject non-regular or symbolic-link output. -- Keep checkout credential persistence disabled; use only a repository-local GitHub CLI helper removed by an `EXIT` trap. -- Keep workflow-level permissions read-only. -- Keep Actions write authority out of the OpenCode job and in one isolated non-checkout job. -- Refuse automatic workflow-run authorization for `.github/**` and `CODEOWNERS` changes. -- Bind run discovery and authorization to the exact still-current head SHA. -- Bound OpenCode with `TERM` after 45 minutes, `KILL` after a 30-second grace period, and a 50-minute job timeout. -- The agent may create or update one pull request but may never approve, merge, bypass protection, publish, or push to `develop` or `main`. -- Preserve standalone operation and modular CWL service compatibility. +- Use only `${{ secrets.NVIDIA_NIM_API_KEY }}` as the model secret. +- Do not introduce `COPILOT_GITHUB_TOKEN` or alter an existing review-agent secret. +- Invoke plain `opencode run`, not the GitHub lifecycle handler. +- Give the model job `pull-requests: read`, never write. +- Give exactly one non-checkout publisher `pull-requests: write`. +- Give exactly one non-checkout authorizer `actions: write`. +- Bind every workflow-run authorization to both exact SHA and exact PR number. +- Keep all `.github/**` and `CODEOWNERS` changes outside automatic publication and authorization. +- Preserve immutable OpenCode installation, branch protection, 100% configured production statement/branch coverage, public docstrings, APA 7th doctoring, and `CHANGELOG.md` maintenance. --- -### Task 1: Add fail-closed workflow contracts - -**Files:** -- Create: `etl-service/src/test/java/com/xtrmetl/etl/documentation/HourlyOpenCodeMaintenanceWorkflowTest.java` -- Create: `etl-service/src/test/java/com/xtrmetl/etl/documentation/HourlyOpenCodeArchiveValidationTest.java` - -- [x] **Step 1: Write the initial missing-workflow test** - -Require workflow existence, hourly schedule, serialized concurrency, bounded timeout, immutable checkout, no persisted credentials, NVIDIA-only credentials, private sharing, and prompt prohibitions. - -- [x] **Step 2: Verify initial RED** - -```bash -./mvnw -pl etl-service -Dtest=HourlyOpenCodeMaintenanceWorkflowTest test -``` - -Observed: the existence assertion failed before the workflow was created. - -- [x] **Step 3: Add immutable-installation and direct-token contracts** - -Require exact release URL and checksum, no npm install, graceful and forced timeout, removable GitHub CLI helper, bot author, cleanup trap, and no encoded authorization header. - -- [x] **Step 4: Add archive-member and entry-type contracts** - -Require one member named `opencode`, locale-stable GNU tar metadata, regular-file type `-`, validation before extraction, private directory, overwrite refusal, and post-extraction file checks. - -- [x] **Step 5: Add current-model availability contract** - -Require `nvidia/deepseek-ai/deepseek-v4-pro` and reject the deprecated Qwen3 Coder endpoint. - -- [x] **Step 6: Add exact-head revalidation contracts** - -Require a pre-agent head snapshot, isolated Actions-write job, no checkout or NVIDIA secret in that job, `.github/**` and `CODEOWNERS` exclusion, exact-run discovery, double SHA validation, absent-run failure, and no review or merge operation. - -- [x] **Step 7: Verify RED cycles** - -The workflow-existence, archive-member, archive-type, model-selection, exact-head authorization, policy-path exclusion, and Actions-write isolation contracts were committed before their corresponding production behavior. - -### Task 2: Implement the bounded NVIDIA OpenCode job - -**Files:** -- Create: `.github/workflows/hourly-opencode-maintenance.yml` - -- [x] **Step 1: Add protected scheduling and authority boundary** - -Configure `43 * * * *`, omit manual dispatch, serialize concurrency, set a 50-minute job timeout, checkout protected default-branch source, and prohibit approval, merge, protected-branch push, review-agent modification, secret disclosure, duplicate PR creation, and release publication. +## Task 1 — Test the authority boundaries before production changes -- [x] **Step 2: Add immutable OpenCode installation** +**File:** `etl-service/src/test/java/com/xtrmetl/etl/documentation/HourlyOpenCodeMaintenanceWorkflowTest.java` -Download the immutable `v1.18.13` Linux x64 archive over HTTPS, validate its SHA-256, require one regular entry named `opencode`, extract privately without restoring archive ownership or permissions, refuse overwrite, reject non-regular or symbolic-link output, and verify exact version. +- [x] Require hourly serialized execution and bounded TERM/KILL handling. +- [x] Require the immutable OpenCode 1.18.13 archive, SHA-256, regular-file member, private extraction, and exact version. +- [x] Require `NVIDIA_NIM_API_KEY` as the sole model secret and `nvidia/deepseek-ai/deepseek-v4-pro` as the selected model. +- [x] Require plain `opencode run --model "${MODEL}" --auto` and reject `opencode github run`. +- [x] Require `pull-requests: read` on `maintain-repository`. +- [x] Require one non-checkout publisher with the workflow's sole `pull-requests: write` grant. +- [x] Require one non-checkout authorizer with the workflow's sole `actions: write` grant. +- [x] Require one strict existing-PR or `automation/opencode-*` candidate. +- [x] Require draft publication, policy-path exclusion, and a branch ahead of `develop`. +- [x] Require workflow-run association with both PR number and exact SHA. +- [x] Require complete CI, Dependency Review, SBOM, Semgrep, and Security Scan materialization. -- [x] **Step 3: Add direct-token Git bootstrap** +The test-only head intentionally made the preceding workflow fail before implementation. -Fail closed on missing token aliases, reset inherited helpers locally, install `!gh auth git-credential`, set `opencode-agent[bot]` local identity, and remove the helper through an `EXIT` trap. +## Task 2 — Implement the model job without PR write authority -- [x] **Step 4: Select the current NVIDIA coding endpoint** +**File:** `.github/workflows/hourly-opencode-maintenance.yml` -Set `NVIDIA_API_KEY`, `MODEL`, `SHARE=false`, and `USE_GITHUB_TOKEN=true` without provider or model fallback. +- [x] Run at `43 * * * *` only from protected default-branch source. +- [x] Keep top-level `contents: read`. +- [x] Set model-job permissions to Actions/checks/PR/security/status read, issues write, and contents write. +- [x] Install OpenCode from the pinned immutable archive. +- [x] Configure the removable repository-local `gh auth git-credential` helper and bot author. +- [x] Pipe the bounded prompt into plain `opencode run`. +- [x] Permit an existing eligible branch update or exactly one strict automation branch. +- [x] Preserve the model step's failure after capturing any reviewable branch evidence. -- [x] **Step 5: Remove Actions write from the OpenCode job** +## Task 3 — Detect exactly one candidate -Give `maintain-repository` Actions read plus only the branch, issue, PR, check, security-read, and status permissions required for bounded maintenance. +**File:** `.github/workflows/hourly-opencode-maintenance.yml` -### Task 3: Implement isolated exact-head run authorization +- [x] Snapshot `develop`, current direct PR heads, and prior automation branch heads before model execution. +- [x] Require `develop` to remain unchanged afterward. +- [x] Detect one changed existing PR head or one strict automation branch. +- [x] Exclude an automation branch already represented by the changed PR. +- [x] Fail on multiple candidates. +- [x] Emit compact candidate JSON as a job output. -**Files:** -- Modify: `.github/workflows/hourly-opencode-maintenance.yml` -- Modify: `etl-service/src/test/java/com/xtrmetl/etl/documentation/HourlyOpenCodeMaintenanceWorkflowTest.java` - -- [x] **Step 1: Export the pre-agent head map** - -Snapshot same-repository pull requests targeting `develop` before OpenCode and expose compact JSON as a maintenance-job output. - -- [x] **Step 2: Add the isolated authorization job** - -Run after maintenance success or failure unless cancelled. Give it only `actions: write`, `contents: read`, and `pull-requests: read`. Do not checkout repository code or pass the NVIDIA credential. - -- [x] **Step 3: Add policy-path and TOCTOU gates** +## Task 4 — Publish deterministically without executing source -Reject `.github/**` and `CODEOWNERS` changes, verify the exact head before discovery and immediately before authorization, and pass the expected SHA to jq as data. +**File:** `.github/workflows/hourly-opencode-maintenance.yml` -- [x] **Step 4: Authorize only exact approval-required runs** +- [x] Add `publish-agent-pull-request` with `contents: read` and `pull-requests: write` only. +- [x] Do not checkout source or pass the NVIDIA credential. +- [x] Re-read existing PR repository, state, base, branch, and exact SHA. +- [x] For a new branch, require the strict namespace, live exact SHA, positive `ahead_by`, at most 50 files, and no policy path. +- [x] Create one draft PR from a fixed JSON payload. +- [x] Expose PR number, head ref, and exact SHA for the authorizer. +- [x] Contain no review or merge endpoint. -Discover `pull_request` runs by exact `head_sha`, fail when no run appears, and call the approval endpoint only for `action_required` or `waiting` runs. Do not approve or merge the pull request. +## Task 5 — Authorize only PR-associated exact-head runs -- [ ] **Step 5: Verify focused GREEN on the final exact head** +**File:** `.github/workflows/hourly-opencode-maintenance.yml` -```bash -./mvnw -pl etl-service \ - -Dtest='HourlyOpenCodeMaintenanceWorkflowTest,HourlyOpenCodeArchiveValidationTest' test -``` +- [x] Add `authorize-exact-head-checks` with `actions: write`, `contents: read`, and `pull-requests: read`. +- [x] Do not checkout source or pass the model credential. +- [x] Reject `.github/**` and `CODEOWNERS` changes. +- [x] Re-read the live PR before discovery and on every bounded pass. +- [x] Filter each run by `event=pull_request`, exact SHA, and `pull_requests[].number`. +- [x] Authorize only `action_required` or `waiting` runs. +- [x] Require all five named workflows to materialize. +- [x] Fail with missing names or any head movement. -Expected: zero failures, errors, and skipped project tests. - -### Task 4: Complete evidence and release notes +## Task 6 — Align operations, design, doctoring, and changelog evidence **Files:** -- Create: `docs/operations/hourly-opencode-maintenance.md` -- Create: `docs/doctoring/opencode-archive-extraction-evidence.md` -- Create: `docs/doctoring/nvidia-opencode-model-selection-evidence.md` -- Create: `docs/doctoring/github-token-exact-head-check-authorization-evidence.md` -- Modify: `docs/superpowers/specs/2026-08-04-hourly-opencode-maintenance-design.md` -- Modify: `docs/superpowers/plans/2026-08-04-hourly-opencode-maintenance-plan.md` -- Modify: `CHANGELOG.md` - -- [x] **Step 1: Document archive and credential boundaries** - -Record immutable release, checksum, exact member and type, extraction controls, direct-token helper lifecycle, timeout escalation, authority restrictions, failure behavior, and rollback. - -- [x] **Step 2: Document model selection** - -Record deprecated-endpoint rejection, current NVIDIA endpoint evidence, no-fallback semantics, RED evidence, and APA 7 references. - -- [x] **Step 3: Document exact-head run authorization** - -Record GitHub-token recursive-trigger behavior, split-job authority, before/after SHA evidence, policy-path exclusion, Actions-write isolation, TOCTOU validation, failure behavior, rollback, and APA 7 references. - -- [x] **Step 4: Align `CHANGELOG.md`** - -Record the NVIDIA model, immutable installation, isolated exact-head workflow-run authorization, and all doctoring evidence files under `Unreleased`. - -- [ ] **Step 5: Run full reactor verification** - -```bash -./mvnw -B test -``` - -Expected: all modules succeed; no project test is skipped. - -### Task 5: Verify and integrate the protected workflow-change pull request - -**Files:** -- No additional source files. - -- [ ] **Step 1: Verify exact branch head and diff** - -```bash -git status --short -git rev-parse HEAD -git diff develop...HEAD --check -./mvnw -B test -``` -Expected: clean tree, no whitespace errors, successful build. +- `docs/operations/hourly-opencode-maintenance.md` +- `docs/doctoring/github-token-exact-head-check-authorization-evidence.md` +- `docs/superpowers/specs/2026-08-04-hourly-opencode-maintenance-design.md` +- `docs/superpowers/plans/2026-08-04-hourly-opencode-maintenance-plan.md` +- `CHANGELOG.md` -- [x] **Step 2: Open and label the pull request** +- [x] Document the three-job authority topology. +- [x] Record why plain OpenCode run replaces the GitHub lifecycle handler. +- [x] Record strict candidate and draft publication controls. +- [x] Record exact PR-number plus SHA association. +- [x] Record residual coarse permissions and compensating controls. +- [x] Preserve APA 7th references to OpenCode and GitHub primary sources. +- [ ] Confirm the final root changelog wording against the final exact head before merge. -Use title `ci: schedule NVIDIA OpenCode maintenance agent`; apply `automerge-workflow`, and retain `manual-merge` until exact-head checks and non-author approval exist. +## Task 7 — Final verification and integration -- [ ] **Step 3: Reinspect all feedback on the final exact head** +- [ ] Run focused workflow contract tests on the exact final head. +- [ ] Run the complete Maven reactor with no skipped project test. +- [ ] Confirm Ubuntu, macOS, and Windows CI. +- [ ] Confirm Dependency Review and CycloneDX SBOM. +- [ ] Confirm Semgrep, Trivy, OSV, Scorecard, and all required security evidence. +- [ ] Confirm zero unresolved current review thread. +- [ ] Obtain non-author approval anchored to the exact final SHA. +- [ ] Remove `manual-merge` only immediately before an expected-head squash merge. +- [ ] Merge #121, then retarget and revalidate #122 and every successor in stack order. -Inspect human, CodeRabbit, GitHub Advanced Security, Dependabot, and automated feedback. Resolve only findings addressed by the current head. +## References — APA 7th -- [ ] **Step 4: Verify every final exact-head gate** +Anomaly. (2026). *GitHub handler (Version 1.18.13)* [Source code]. GitHub. https://github.com/anomalyco/opencode/blob/v1.18.13/packages/opencode/src/cli/cmd/github.handler.ts -Require successful Ubuntu, macOS, Windows, Dependency Review, SBOM, Semgrep, Trivy, OSV, Scorecard, combined status, mergeability, and zero unresolved current threads. Pending, queued, cancelled, neutral-required, skipped-required, stale, or absent evidence is not passing. +Anomaly. (2026). *Run command (Version 1.18.13)* [Source code]. GitHub. https://github.com/anomalyco/opencode/blob/v1.18.13/packages/opencode/src/cli/cmd/run.ts -- [ ] **Step 5: Require independent exact-head approval and merge** +GitHub, Inc. (2026). *GITHUB_TOKEN*. GitHub Docs. https://docs.github.com/en/actions/concepts/security/github_token -Do not self-approve. After a non-author approval whose commit ID equals the current head and every exact-head gate succeeds, remove `manual-merge` and squash-merge using the expected head SHA. Otherwise retain the hold and identify the exact external gate. +GitHub, Inc. (2026). *REST API endpoints for workflow runs*. GitHub Docs. https://docs.github.com/en/rest/actions/workflow-runs diff --git a/docs/superpowers/specs/2026-08-04-hourly-opencode-maintenance-design.md b/docs/superpowers/specs/2026-08-04-hourly-opencode-maintenance-design.md index e8ab9c14..4fad4ee0 100644 --- a/docs/superpowers/specs/2026-08-04-hourly-opencode-maintenance-design.md +++ b/docs/superpowers/specs/2026-08-04-hourly-opencode-maintenance-design.md @@ -2,185 +2,153 @@ ## Purpose -mightyETL needs a scheduled development loop that can inspect current repository state, remediate one bounded item, and open or update a pull request without weakening independent review, exact-head validation, or merge gates. The agent must use OpenCode with `NVIDIA_NIM_API_KEY`; it must not use GitHub Copilot or change the existing review agent's credentials, provider, workflow, or authority. +mightyETL requires an hourly development loop that can inspect the live pull-request queue, repair one dependency-eligible branch, or prepare one bounded buyer-visible improvement without granting model-generated code review, merge, or workflow-administration authority. -## Decision +The model credential is `NVIDIA_NIM_API_KEY`. GitHub Copilot credentials and changes to the existing review agents are outside this design. -Add `.github/workflows/hourly-opencode-maintenance.yml` as a two-job workflow: +## Architecture decision -```text -maintain-repository - ├─ protected default-branch checkout - ├─ pinned OpenCode + NVIDIA NIM - ├─ branch / issue / pull-request preparation - └─ actions: read - - exact pre-agent head map - ↓ - -authorize-exact-head-checks - ├─ no checkout or repository-code execution - ├─ no model credential - ├─ actions: write - └─ exact-head workflow-run authorization only +Use three physically separated GitHub Actions jobs: + +```mermaid +flowchart LR + M[maintain-repository
OpenCode + source checkout] -->|one candidate record| P[publish-agent-pull-request
no checkout] + P -->|PR number + exact SHA| A[authorize-exact-head-checks
no checkout] + A --> V[CI and security validation] + V --> R[independent review and merge disposition] ``` -Preserve `.github/workflows/hourly-pr-disposition.yml` as the deterministic fail-closed merge boundary and require a non-author approval anchored to the exact current head. +### Model execution job -The workflow: +`maintain-repository` uses plain `opencode run`, not `opencode github run`. The GitHub lifecycle handler can create pull requests and therefore requires pull-request write authority. Plain run separates model execution from PR publication. -- runs only from the protected default branch at minute 43 of every hour; -- omits manual dispatch so a feature branch or tag cannot become scheduler source; -- pins `actions/checkout` by full SHA with persisted credentials disabled; -- installs OpenCode 1.18.13 from an immutable release asset verified by SHA-256; -- accepts exactly one regular archive member named `opencode` before extraction; -- extracts into a fresh mode-`0700` directory without restoring archive ownership or permissions and with overwrites disabled; -- rejects non-regular or symbolic-link output and verifies the exact executable version; -- maps only `${{ secrets.NVIDIA_NIM_API_KEY }}` to `NVIDIA_API_KEY`; -- selects `nvidia/deepseek-ai/deepseek-v4-pro` with no automatic model or provider fallback; -- uses the repository-scoped GitHub token without OpenCode OIDC exchange; -- bootstraps a repository-local GitHub CLI credential helper and bot author because OpenCode 1.18.13 skips internal Git setup in direct-token mode; -- removes the helper through an `EXIT` trap; -- omits the ineffective `AGENT` environment variable, allowing repository `default_agent` or OpenCode's `build` fallback; -- disables public session sharing; -- caps OpenCode with a 45-minute `TERM` timeout, 30-second `KILL` escalation, and 50-minute job timeout; -- exports a compact pre-agent pull-request head map to the isolated authorization job; -- authorizes only approval-required `pull_request` workflow runs for a still-current exact head; -- refuses automatic authorization for `.github/**` and `CODEOWNERS` changes; -- never approves or merges a pull request. +The job receives: -## Model selection boundary +```text +actions: read +checks: read +contents: write +issues: write +pull-requests: read +security-events: read +statuses: read +``` -The previous Qwen3 Coder free endpoint is marked deprecated in NVIDIA's current catalog. DeepSeek V4 Pro is selected because NVIDIA currently exposes it through a free endpoint and documents coding, agentic AI, tool use, structured output, function calling, software-engineering use cases, and long context. +It may inspect pull requests and push one feature branch. It cannot create, update, approve, close, or merge a pull request with its token. -The model identifier is explicit and test guarded. No fallback runs after a partial agent session because another model could operate on non-deterministic workspace state, create duplicate branches, or generate conflicting pull requests. Endpoint rejection fails visibly and requires a separate test-first model-selection change. The repository does not claim NVIDIA benchmarks as mightyETL performance. +### Deterministic publisher -## Supply-chain boundary +`publish-agent-pull-request` has `contents: read` and `pull-requests: write`. It has no checkout, model credential, or repository-code execution. It validates one candidate and either identifies an already-open updated PR or creates one draft PR from a strict `automation/opencode-*` branch. -An exact npm version is not a content identity. The workflow consumes the immutable upstream release asset directly. +### Workflow-run authorizer -The installer: +`authorize-exact-head-checks` has `actions: write`, `contents: read`, and `pull-requests: read`. It has no checkout or model credential. It may approve only workflow runs that are: -1. downloads only over HTTPS with failure handling and TLS 1.2 minimum; -2. verifies SHA-256 `8d500b20fed2d26e537e221895b1a575476571b4f0089bb29fb13eeb8eb9e937`; -3. requires exactly one archive member named `opencode`; -4. under `LC_ALL=C`, requires GNU tar's regular-file type character `-` before extraction; -5. recreates a private mode-`0700` directory and refuses overwrites; -6. extracts without restoring archive ownership or permissions; -7. requires a regular non-symbolic-link output file; -8. applies executable mode only to that file; -9. verifies version `1.18.13` before adding it to `GITHUB_PATH`. +- `pull_request` events; +- bound to the exact expected SHA; +- associated with the exact expected pull-request number; +- in `action_required` or `waiting` state. -Checksum binding, pre-extraction name and type checks, private extraction, overwrite refusal, and post-extraction checks are cumulative controls. They avoid mutable package-manager bootstrapping and unconstrained archive extraction. +It does not approve a PR or merge code. -## Direct-token credential lifecycle +## Schedule and supply-chain contract -OpenCode 1.18.13 uses `GITHUB_TOKEN` for GitHub API access when `USE_GITHUB_TOKEN=true` and skips its internal Git configuration. With `persist-credentials: false`, ordinary `git commit` and `git push` need explicit local author and HTTPS helper configuration. +- cron: `43 * * * *`; +- no manual dispatch; +- serialized concurrency; +- immutable `actions/checkout` full SHA; +- `persist-credentials: false`; +- OpenCode 1.18.13 immutable Linux archive; +- SHA-256 `8d500b20fed2d26e537e221895b1a575476571b4f0089bb29fb13eeb8eb9e937`; +- exactly one regular archive entry named `opencode`; +- private mode-`0700` extraction without archived ownership or permissions; +- 45-minute `TERM`, 30-second `KILL` escalation, 50-minute job timeout; +- model `nvidia/deepseek-ai/deepseek-v4-pro` with no provider fallback. -The maintenance job therefore: +## Candidate-state contract -1. fails closed if `NVIDIA_API_KEY`, `GITHUB_TOKEN`, or `GH_TOKEN` is empty; -2. resets inherited GitHub credential helpers in repository-local configuration; -3. installs `!gh auth git-credential`, which reads the ephemeral token from `GH_TOKEN` only when Git requests credentials; -4. sets local author identity to `opencode-agent[bot]`; -5. removes the helper through an `EXIT` trap after success, failure, timeout, or forced termination. +Before OpenCode starts, snapshot: -No encoded token, personal token, OIDC permission, tracked credential file, alternate model credential, or review-agent secret is introduced. +1. protected `develop` head; +2. every same-repository open `develop` PR and head SHA; +3. every existing `automation/opencode-*` branch and head SHA. -## Permission and code-execution boundary +After OpenCode exits: -The workflow-level permission is only `contents: read`. +1. require `develop` to be unchanged; +2. identify existing PR heads changed during the run; +3. identify new or advanced strict automation branches; +4. exclude an automation branch already represented by the changed PR candidate; +5. require zero or one candidate; +6. fail if the output is ambiguous. -The `maintain-repository` job receives Actions read plus the minimum check, branch, issue, pull-request, security-read, and status-read permissions required for bounded development. It executes OpenCode and repository tests but has no Actions write authority. +A new branch must match: -The `authorize-exact-head-checks` job receives Actions write plus read-only contents and pull-request metadata. It never checks out or executes repository code and never receives `NVIDIA_API_KEY`. Its only write operation is authorizing an approval-required workflow run after exact-head verification. +```text +^automation/opencode-[0-9]{8}T[0-9]{6}Z-[a-z0-9][a-z0-9-]{0,48}$ +``` -This separation prevents OpenCode, generated code, and checked-out repository scripts from using Actions write permission while still closing the `GITHUB_TOKEN` recursive-trigger gap. +The publisher additionally requires a live matching SHA, at least one commit ahead of `develop`, at most 50 changed files, and no `.github/**` or `CODEOWNERS` path. -## Exact-head authorization algorithm +## Exact-head workflow contract -1. Before OpenCode, snapshot all same-repository pull requests targeting `develop` as `{pull_request_number: head_sha}`. -2. Export that compact object as the maintenance job output. -3. Run the authorization job after maintenance success or failure unless the workflow was cancelled. -4. Require the output to exist and parse as a JSON object. -5. Enumerate the same pull-request set after OpenCode. -6. Select only a new pull request or a changed head. -7. Refuse automatic authorization when any `.github/**` or `CODEOWNERS` path changed. -8. Verify the still-current pull-request head equals the expected SHA. -9. Discover only `pull_request` workflow runs for that SHA. -10. Fail if no exact-head run materializes. -11. Verify the current head again immediately before authorization. -12. Authorize only `action_required` or `waiting` runs. +GitHub creates pull-request workflows asynchronously. The authorizer performs bounded repeated discovery and succeeds only when all five names are associated with the exact PR number and SHA: -Expected SHA values are passed to `jq` as data rather than interpolated into jq source. Authorization begins validation but conveys no review, merge, or success decision. +```text +CI +Dependency Review +SBOM (CycloneDX) +SAST Semgrep +Security Scan +``` -## Agent authority boundary +A SHA-only filter is insufficient because multiple pull requests can reference the same commit. Every selected workflow run must satisfy: -The prompt is part of the security boundary. The agent may inspect, test, edit, commit, push one feature branch, update one dependency-eligible development pull request, or open one pull request. It must not: +```text +run.head_sha == expected_head +and any(run.pull_requests; number == expected_pull_request_number) +``` + +The live PR head is checked before discovery, on every discovery pass, and after the complete set materializes. -- approve or merge a pull request; -- push directly to `develop` or `main`; -- bypass checks, branch protection, security gates, coverage, or independent review; -- alter review-agent workflows, providers, credentials, secret names, `CODEOWNERS`, branch protection, or repository secrets; -- modify workflow policy unless a specifically labeled issue authorizes the bounded change; -- expose secret values or sensitive ETL payload information; -- create a second development pull request while another is dependency-eligible; -- publish a release without separate release authorization and every acceptance gate. +## Git credential lifecycle -The deterministic disposition workflow independently evaluates reviews, current threads, named checks, status contexts, labels, mergeability, and expected head SHA. +The model job uses an ephemeral repository-local `!gh auth git-credential` helper because checkout credentials remain disabled. It clears inherited local helpers, configures the bot author, and removes the helper through an `EXIT` trap after success, failure, or timeout. No token is written into Git configuration. -## Data flow +## Agent product contract -1. GitHub starts the schedule from the protected default branch. -2. The maintenance job checks out source without persisting credentials. -3. The installer validates and installs OpenCode. -4. The job snapshots direct `develop` pull-request heads and exports the map. -5. The shell validates model and repository credentials, installs local Git identity and helper, and registers cleanup. -6. OpenCode calls `nvidia/deepseek-ai/deepseek-v4-pro`, inspects the queue, and performs one bounded test-first slice. -7. OpenCode leaves one branch and pull request but does not approve or merge. -8. The helper cleanup trap runs. -9. The isolated authorization job compares before and after heads and rejects policy-changing pull requests. -10. It authorizes only approval-required runs for an unchanged exact head. -11. CI, security, independent review, branch protection, and deterministic disposition evaluate that exact head separately. +When a dependency-eligible PR exists, the model may update only that branch. Otherwise it may push exactly one strict automation branch. It must work test-first, preserve configured production statement and branch coverage at 100%, maintain public documentation and `CHANGELOG.md`, use descriptive multi-word `snake_case` database names, and record material primary standards or peer-reviewed evidence in APA 7th form. -## Failure behavior +The model may not mutate PR lifecycle state, protected branches, workflow policy, review-agent credentials, repository secrets, or releases. -Missing credentials, deprecated or rejected model, NVIDIA outage, download failure, checksum mismatch, archive mismatch, non-regular entry, extracted-file mismatch, version mismatch, Git bootstrap failure, timeout, test failure, missing pre-agent output, policy-file change, head movement, absent workflow run, authorization rejection, or permission denial fails visibly. No provider fallback or partial-success claim is allowed. Concurrency is serialized. +## Failure semantics + +Missing credentials, archive mismatch, model failure, protected-branch movement, multiple candidates, invalid branch namespace, policy-file changes, candidate SHA movement, absent workflow association, incomplete workflow-name materialization, or any rejected API mutation fails visibly. Partial branch evidence can be exposed only as a draft by the deterministic publisher; it is not reported as a successful run. ## Verification -Repository tests fail unless they prove: +Tests must prove: -- hourly serialized scheduling and bounded forced termination; -- immutable checkout and OpenCode content pins; -- exact pre-extraction member name and regular-file type; -- private extraction, overwrite refusal, and regular non-symbolic-link output; -- exclusive use of `NVIDIA_NIM_API_KEY` and the selected NVIDIA model; -- direct-token Git bootstrap and cleanup without stored authorization data; -- workflow-level read-only permissions; -- Actions write absent from the OpenCode job; -- exactly one isolated non-checkout authorization job with Actions write; -- before/after exact-head evidence, `.github/**` and `CODEOWNERS` exclusion, exact-run filtering, and double SHA validation; -- prompt prohibitions against approval, merge, protected-branch writes, review-agent changes, self-modification, and duplicate pull requests. +- hourly bounded execution and immutable installation; +- exclusive NVIDIA NIM model credential; +- plain OpenCode run instead of GitHub lifecycle execution; +- PR read-only authority in the model job; +- exactly one non-checkout PR publisher; +- exactly one non-checkout Actions authorizer; +- strict single-candidate publication; +- policy-file exclusion; +- exact PR-number plus SHA run association; +- complete five-workflow materialization; +- no PR review or merge endpoint in the workflow. ## References — APA 7th Anomaly. (2026). *GitHub handler (Version 1.18.13)* [Source code]. GitHub. https://github.com/anomalyco/opencode/blob/v1.18.13/packages/opencode/src/cli/cmd/github.handler.ts -Anomaly. (2026). *OpenCode release v1.18.13* [Software release]. GitHub. https://github.com/anomalyco/opencode/releases/tag/v1.18.13 - -Free Software Foundation. (2023). *GNU tar 1.35: Security*. https://www.gnu.org/software/tar/manual/html_section/Security.html +Anomaly. (2026). *Run command (Version 1.18.13)* [Source code]. GitHub. https://github.com/anomalyco/opencode/blob/v1.18.13/packages/opencode/src/cli/cmd/run.ts GitHub, Inc. (2026). *GITHUB_TOKEN*. GitHub Docs. https://docs.github.com/en/actions/concepts/security/github_token -GitHub, Inc. (2026). *GitHub CLI manual: gh auth git-credential*. GitHub CLI Manual. https://cli.github.com/manual/gh_auth_git-credential - GitHub, Inc. (2026). *REST API endpoints for workflow runs*. GitHub Docs. https://docs.github.com/en/rest/actions/workflow-runs -GitHub, Inc. (2026). *Triggering a workflow*. GitHub Docs. https://docs.github.com/en/actions/using-workflows/triggering-a-workflow - -GitHub, Inc. (2026). *Workflow syntax for GitHub Actions*. GitHub Docs. https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax - NVIDIA Corporation. (2026). *DeepSeek V4 Pro*. NVIDIA NIM API catalog. https://build.nvidia.com/deepseek-ai/deepseek-v4-pro - -NVIDIA Corporation. (2026). *DeepSeek AI / DeepSeek V4 Pro*. NVIDIA NIM API reference. https://docs.api.nvidia.com/nim/reference/deepseek-ai-deepseek-v4-pro diff --git a/etl-service/src/test/java/com/xtrmetl/etl/documentation/HourlyOpenCodeMaintenanceWorkflowTest.java b/etl-service/src/test/java/com/xtrmetl/etl/documentation/HourlyOpenCodeMaintenanceWorkflowTest.java index 903f8b37..f5f48fa6 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/documentation/HourlyOpenCodeMaintenanceWorkflowTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/documentation/HourlyOpenCodeMaintenanceWorkflowTest.java @@ -18,13 +18,14 @@ import static org.junit.jupiter.api.Assertions.assertTrue; /** - * Guards the credential, authority, supply-chain, and exact-head validation boundaries of the - * scheduled OpenCode maintenance workflow. + * Guards the credential, authority, supply-chain, publication, and exact-head validation + * boundaries of the scheduled OpenCode maintenance workflow. * - *

The scheduled agent is allowed to prepare feature-branch pull requests. It is not a reviewer - * or merger. These tests make that separation visible to beginners and prevent a later workflow - * edit from silently adding a fallback provider, mutable tool version, protected-branch push, - * self-approval path, policy-code auto-authorization, or unvalidated agent-generated head.

+ *

The model may edit and push one bounded feature branch. It never receives pull-request write + * authority. A deterministic non-checkout publisher may create one draft pull request, while a + * second isolated non-checkout job may authorize only approval-required workflow runs associated + * with that exact pull request and exact head. These tests keep those authorities physically + * separated from model and repository-code execution.

*/ class HourlyOpenCodeMaintenanceWorkflowTest { @@ -35,9 +36,7 @@ class HourlyOpenCodeMaintenanceWorkflowTest { private static String workflow; /** - * Reads the workflow once after first producing an ordinary assertion failure when the - * production workflow has not yet been implemented. Line endings are normalized so the same - * structural contracts run deterministically on Windows, Linux, and macOS checkouts. + * Reads the workflow once and normalizes line endings for deterministic cross-platform tests. * * @throws IOException when the workflow exists but cannot be read as UTF-8 text */ @@ -46,18 +45,12 @@ static void readWorkflow() throws IOException { Path workflowPath = projectRoot().resolve( ".github/workflows/hourly-opencode-maintenance.yml" ); - assertTrue( - Files.exists(workflowPath), - "The hourly OpenCode maintenance workflow must exist" - ); + assertTrue(Files.exists(workflowPath), "The hourly OpenCode workflow must exist"); workflow = Files.readString(workflowPath, StandardCharsets.UTF_8) .replace("\r\n", "\n"); } - /** - * Verifies that runs are offset from the top of the hour, serialized, and time bounded, - * including forced termination when an agent ignores the graceful termination signal. - */ + /** Verifies one serialized, bounded run every hour. */ @Test void schedulesOneBoundedNonOverlappingRunPerHour() { assertTrue(workflow.contains("cron: \"43 * * * *\"")); @@ -65,15 +58,11 @@ void schedulesOneBoundedNonOverlappingRunPerHour() { assertTrue(workflow.contains("cancel-in-progress: false")); assertTrue(workflow.contains("timeout-minutes: 50")); assertTrue(workflow.contains( - "timeout --signal=TERM --kill-after=30s 45m opencode github run" + "timeout --signal=TERM --kill-after=30s 45m opencode run" )); } - /** - * Verifies that repository source and the OpenCode executable are pinned by immutable - * content identifiers without retaining checkout credentials that a generated process could - * reuse implicitly. - */ + /** Verifies immutable checkout and OpenCode installation without persisted credentials. */ @Test void pinsCheckoutAndOpenCodeWithoutPersistedCredentials() { assertTrue(workflow.contains( @@ -98,16 +87,7 @@ void pinsCheckoutAndOpenCodeWithoutPersistedCredentials() { assertFalse(workflow.contains("anomalyco/opencode/github@")); } - /** - * Verifies that direct-token mode can actually create commits and push a feature branch. - * - *

OpenCode 1.18.13 intentionally skips its internal Git credential and author setup when - * {@code USE_GITHUB_TOKEN=true}. Because checkout credentials remain disabled, the workflow - * must install a repository-local GitHub CLI credential helper and local author identity - * before starting OpenCode, then remove the helper even when the process fails or times out. - * The helper reads the short-lived token from {@code GH_TOKEN}; no encoded token is written - * to Git configuration.

- */ + /** Verifies ephemeral Git credentials for branch pushes and deterministic cleanup. */ @Test void bootstrapsAndRemovesDirectTokenGitCredentials() { assertTrue(workflow.contains("GH_TOKEN: ${{ github.token }}")); @@ -131,151 +111,158 @@ void bootstrapsAndRemovesDirectTokenGitCredentials() { + "\"opencode-agent[bot]@users.noreply.github.com\"" )); assertFalse(workflow.contains("AUTHORIZATION: basic")); - assertFalse(workflow.contains("AGENT: build")); } - /** - * Verifies that the repository's NVIDIA NIM secret is the only model credential and is - * mapped to the environment variable documented by OpenCode's NVIDIA provider. - */ + /** Verifies NVIDIA NIM is the sole model credential and plain OpenCode owns no PR lifecycle. */ @Test - void usesOnlyTheNvidiaNimCredential() { + void usesOnlyNvidiaNimWithPlainOpenCodeRun() { assertEquals(Set.of("NVIDIA_NIM_API_KEY"), referencedSecrets()); assertTrue(workflow.contains( "NVIDIA_API_KEY: ${{ secrets.NVIDIA_NIM_API_KEY }}" )); - assertTrue(workflow.contains("SHARE: \"false\"")); - assertTrue(workflow.contains("USE_GITHUB_TOKEN: \"true\"")); - - String lowerCaseWorkflow = workflow.toLowerCase(Locale.ROOT); - assertFalse(lowerCaseWorkflow.contains("copilot")); + assertTrue(workflow.contains("MODEL: nvidia/deepseek-ai/deepseek-v4-pro")); + assertTrue(workflow.contains("opencode run --model \"${MODEL}\" --auto")); + assertFalse(workflow.contains("opencode github run")); + assertFalse(workflow.contains("USE_GITHUB_TOKEN")); + assertFalse(workflow.toLowerCase(Locale.ROOT).contains("copilot")); assertFalse(workflow.contains("ANTHROPIC_API_KEY")); assertFalse(workflow.contains("OPENAI_API_KEY")); } /** - * Requires a currently available free NVIDIA endpoint suited to repository-scale coding. - * - *

The previous Qwen3 Coder trial endpoint was deprecated by NVIDIA. DeepSeek V4 Pro is - * exposed by NVIDIA as a free endpoint with long-context coding and tool-use capabilities, so - * the workflow pins that provider/model identifier and fails instead of silently falling back.

+ * Proves the model has read-only pull-request authority and both PR-writing jobs are + * deterministic non-checkout jobs without the NVIDIA credential. */ @Test - void usesCurrentFreeAgenticCodingModel() { - assertTrue(workflow.contains("MODEL: nvidia/deepseek-ai/deepseek-v4-pro")); - assertFalse(workflow.contains("qwen/qwen3-coder-480b-a35b-instruct")); - } + void isolatesPullRequestAndActionsWriteAuthorityFromTheAgent() { + String maintenance = maintenanceJob(); + String publisher = publicationJob(); + String authorizer = authorizationJob(); - /** - * Verifies that the OpenCode process never receives Actions write authority. - * - *

The maintenance job owns only branch and pull-request preparation permissions. A separate - * job, which never checks out or executes repository code, receives the sole occurrence of - * {@code actions: write} needed to authorize approval-required exact-head workflow runs.

- */ - @Test - void isolatesActionsWriteFromTheAgentProcess() { assertTrue(workflow.contains("permissions:\n contents: read\n\njobs:")); - assertTrue(workflow.contains( - "maintain-repository:\n" - + " permissions:\n" - + " actions: read\n" - + " checks: read\n" - + " contents: write\n" - + " issues: write\n" - + " pull-requests: write\n" - + " security-events: read\n" - + " statuses: read" - )); - assertTrue(workflow.contains( - "authorize-exact-head-checks:\n" - + " needs: maintain-repository\n" - + " if: ${{ always() && !cancelled() }}\n" - + " permissions:\n" - + " actions: write\n" - + " contents: read\n" - + " pull-requests: read" - )); + assertTrue(maintenance.contains("actions: read")); + assertTrue(maintenance.contains("contents: write")); + assertTrue(maintenance.contains("pull-requests: read")); + assertFalse(maintenance.contains("pull-requests: write")); + assertFalse(maintenance.contains("actions: write")); + + assertTrue(publisher.contains("pull-requests: write")); + assertTrue(publisher.contains("contents: read")); + assertFalse(publisher.contains("actions/checkout@")); + assertFalse(publisher.contains("NVIDIA_API_KEY")); + assertFalse(publisher.contains("/reviews")); + assertFalse(publisher.contains("/merge")); + + assertTrue(authorizer.contains("actions: write")); + assertTrue(authorizer.contains("pull-requests: read")); + assertFalse(authorizer.contains("actions/checkout@")); + assertFalse(authorizer.contains("NVIDIA_API_KEY")); + assertFalse(authorizer.contains("pull-requests: write")); + assertEquals(1, countOccurrences(workflow, "actions: write")); assertEquals(1, countOccurrences(workflow, "contents: write")); - assertFalse(authorizationJob().contains("actions/checkout@")); - assertFalse(authorizationJob().contains("NVIDIA_API_KEY")); + assertEquals(1, countOccurrences(workflow, "pull-requests: write")); assertFalse(workflow.contains("id-token:")); assertFalse(workflow.contains("security-events: write")); } - /** - * Requires exact-head CI authorization after the agent creates or updates a same-repository PR. - * - *

GitHub prevents ordinary events produced with {@code GITHUB_TOKEN} from recursively - * starting workflows. Current GitHub behavior creates {@code opened}, {@code synchronize}, and - * {@code reopened} PR runs in an approval-required state instead. The trusted default-branch - * workflow must snapshot heads before the agent, detect only heads changed by this run, refuse - * every {@code .github} or CODEOWNERS policy change, bind every decision to the still-current - * SHA, and authorize only those exact-head runs in a separate non-checkout job. This starts - * validation; it does not approve or merge the pull request.

- */ + /** Verifies one strict branch or existing PR is selected before deterministic publication. */ @Test - void authorizesExactHeadChecksForAgentChangedPullRequests() { - assertTrue(workflow.contains("id: snapshot_heads")); - assertTrue(workflow.contains("open_pr_heads_before: ${{ steps.snapshot_heads.outputs.open_pr_heads }}")); - assertTrue(workflow.contains("open-pr-heads-before.json")); - assertTrue(workflow.contains("BEFORE_HEADS: ${{ needs.maintain-repository.outputs.open_pr_heads_before }}")); + void publishesOnlyOneValidatedAgentCandidate() { + assertTrue(workflow.contains( + "automation_branch_heads_before: " + + "${{ steps.snapshot_heads.outputs.automation_branch_heads }}" + )); assertTrue(workflow.contains( - "name: Authorize exact-head checks for agent-updated pull requests" + "agent_candidate: ${{ steps.detect_candidate.outputs.agent_candidate }}" )); - assertTrue(workflow.contains("head.repo.full_name == $repo")); - assertTrue(workflow.contains(".base.ref == \"develop\"")); + assertTrue(workflow.contains("automation/opencode-")); + assertTrue(workflow.contains("Multiple agent publication candidates were detected")); + assertTrue(workflow.contains("kind: \"existing_pr\"")); + assertTrue(workflow.contains("kind: \"new_branch\"")); + assertTrue(workflow.contains("draft: true")); assertTrue(workflow.contains("startswith(\".github/\")")); assertTrue(workflow.contains("CODEOWNERS")); - assertTrue(workflow.contains("head_sha=${expected_head}")); + assertTrue(workflow.contains("Agent branch is not ahead of develop")); + } + + /** + * Requires every workflow-run decision to bind the event, exact SHA, and associated pull + * request number before the isolated job can authorize a waiting run. + */ + @Test + void authorizesOnlyRunsAssociatedWithTheExactPullRequestHead() { + assertTrue(workflow.contains("required_workflow_names=")); assertTrue(workflow.contains("event=pull_request")); - assertTrue(workflow.contains("--arg expected_head \"${expected_head}\"")); + assertTrue(workflow.contains("head_sha=${expected_head}")); + assertTrue(workflow.contains("--argjson pull_request_number \"${number}\"")); + assertTrue(workflow.contains( + "any(.pull_requests[]?; .number == $pull_request_number)" + )); assertTrue(workflow.contains(".head_sha == $expected_head")); assertTrue(workflow.contains("/actions/runs/${run_id}/approve")); - assertTrue(workflow.contains("current_head")); - assertTrue(workflow.contains("expected_head")); - assertTrue(workflow.contains("No pull-request workflow run materialized")); + assertTrue(workflow.contains("Missing required exact-head workflows")); + assertTrue(workflow.contains("PR #${number} moved")); assertFalse(workflow.contains("gh pr review --approve")); assertFalse(workflow.contains("/pulls/${number}/merge")); } - /** - * Verifies that the model prompt preserves independent review and deterministic merge - * authority instead of granting the development agent governance powers. - */ + /** Verifies the prompt itself mirrors the hard authority boundary and bounded branch contract. */ @Test - void promptForbidsReviewMergeAndProtectedBranchBypass() { + void promptForbidsPullRequestMutationMergeAndProtectedBranchPushes() { assertTrue(workflow.contains("Start every run by inspecting every open pull request")); assertTrue(workflow.contains("exact current head")); - assertTrue(workflow.contains("Never approve or merge a pull request")); + assertTrue(workflow.contains( + "Do not create, update, approve, close, or merge a pull request directly" + )); assertTrue(workflow.contains("Never push directly to develop or main")); + assertTrue(workflow.contains("exactly one automation/opencode-")); assertTrue(workflow.contains("Do not bypass branch protection")); assertTrue(workflow.contains("Do not alter the existing review agent")); assertTrue(workflow.contains("Do not change any review-agent secret name")); assertTrue(workflow.contains("Do not modify .github/workflows/")); assertTrue(workflow.contains("automation-maintenance")); - assertTrue(workflow.contains("Do not create a second development pull request")); assertTrue(workflow.contains("Do not print, echo, summarize, or expose secret values")); } + /** @return workflow text for the OpenCode execution job only */ + private static String maintenanceJob() { + return jobSection(" maintain-repository:", " publish-agent-pull-request:"); + } + + /** @return workflow text for the deterministic draft-PR publisher only */ + private static String publicationJob() { + return jobSection(" publish-agent-pull-request:", " authorize-exact-head-checks:"); + } + + /** @return workflow text for the exact-head workflow-run authorizer */ + private static String authorizationJob() { + int start = workflow.indexOf(" authorize-exact-head-checks:"); + assertTrue(start >= 0, "The isolated exact-head authorization job must exist"); + return workflow.substring(start); + } + /** - * Returns the text of the isolated workflow-run authorization job. + * Extracts one job section between two top-level job keys. * - * @return workflow suffix beginning at the authorization job + * @param startMarker first job marker + * @param endMarker following job marker + * @return exact workflow section */ - private static String authorizationJob() { - int jobStart = workflow.indexOf(" authorize-exact-head-checks:"); - assertTrue(jobStart >= 0, "The isolated exact-head authorization job must exist"); - return workflow.substring(jobStart); + private static String jobSection(String startMarker, String endMarker) { + int start = workflow.indexOf(startMarker); + int end = workflow.indexOf(endMarker); + assertTrue(start >= 0, "Missing workflow job: " + startMarker); + assertTrue(end > start, "Invalid workflow job order for: " + startMarker); + return workflow.substring(start, end); } /** - * Counts non-overlapping occurrences of one literal fragment. + * Counts non-overlapping literal occurrences. * - * @param text complete text to inspect + * @param text complete text * @param fragment non-empty literal fragment - * @return number of non-overlapping occurrences + * @return occurrence count */ private static int countOccurrences(String text, String fragment) { int count = 0; @@ -287,11 +274,7 @@ private static int countOccurrences(String text, String fragment) { return count; } - /** - * Extracts every repository-secret name referenced by the workflow. - * - * @return immutable set of referenced GitHub Actions secret identifiers - */ + /** @return immutable set of referenced repository-secret names */ private static Set referencedSecrets() { Matcher matcher = SECRET_REFERENCE.matcher(workflow); Set secretNames = new java.util.HashSet<>(); @@ -302,10 +285,9 @@ private static Set referencedSecrets() { } /** - * Finds the reactor root from either root or module-local Maven execution. + * Finds the repository root from either reactor-root or module-local execution. * - * @return absolute path that contains the root Maven project - * @throws IllegalStateException when no repository or Maven root can be found + * @return absolute repository root */ private static Path projectRoot() { Path current = Paths.get(System.getProperty("user.dir")).toAbsolutePath(); diff --git a/etl-service/src/test/java/com/xtrmetl/etl/documentation/HourlyOpenCodeRequiredWorkflowAuthorizationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/documentation/HourlyOpenCodeRequiredWorkflowAuthorizationTest.java index 2e54f001..342e01b1 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/documentation/HourlyOpenCodeRequiredWorkflowAuthorizationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/documentation/HourlyOpenCodeRequiredWorkflowAuthorizationTest.java @@ -21,7 +21,8 @@ class HourlyOpenCodeRequiredWorkflowAuthorizationTest { /** - * Requires the authorization loop to wait for, and account for, every required workflow. + * Requires the authorization loop to wait for, associate, and account for every required + * workflow without coupling the test to incidental shell-variable or log-message wording. * * @throws IOException when the production workflow cannot be read */ @@ -38,11 +39,15 @@ void waitsForEveryRequiredExactHeadWorkflow() throws IOException { )); assertTrue(workflow.contains("observed_workflow_names")); assertTrue(workflow.contains("missing_workflow_names")); - assertTrue(workflow.contains("missing_workflow_count")); assertTrue(workflow.contains("for _ in $(seq 1 18); do")); + assertTrue(workflow.contains(".head_sha == $expected_head")); + assertTrue(workflow.contains( + "any(.pull_requests[]?; .number == $pull_request_number)" + )); assertTrue(workflow.contains("/actions/runs/${run_id}/approve")); - assertTrue(workflow.contains("All required exact-head pull-request workflows materialized")); - assertTrue(workflow.contains("Required exact-head pull-request workflows did not materialize")); + assertTrue(workflow.contains("jq 'length' <<<\"${missing_workflow_names}\"")); + assertTrue(workflow.contains("Missing required exact-head workflows for PR")); + assertTrue(workflow.contains("Authorized exact-head pull-request checks for PR")); } /** From 05f0ec8ea3cdc715747966812462516d7b85c94e Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 21:16:57 +0900 Subject: [PATCH 92/92] docs(changelog): reconcile exact-head workflow evidence --- CHANGELOG.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d4898847..397c6e1d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- Pull-request CI and CycloneDX SBOM jobs now check out the literal current source head, immediately assert `git rev-parse HEAD` against `github.event.pull_request.head.sha`, and disable checkout credential persistence; generated merge revisions remain useful compatibility previews but no longer masquerade as direct exact-head source evidence. - The durable-job claim eligibility index now builds in a separate PostgreSQL `CREATE INDEX CONCURRENTLY` migration with Flyway non-transactional script configuration and session-level PostgreSQL migration locking, preserving normal job writes during rollout while keeping lease columns and constraints transactional. - Durable asynchronous ETL jobs now progress from `PENDING` through lease-fenced execution to `SUCCEEDED` or `FAILED`; PostgreSQL owns cross-replica claiming, stale workers cannot commit target or lifecycle effects, and intake and execution remain independently fail-closed. - Durable-worker observability now records one terminal outcome counter and one matching duration sample for every completed poll, including idle polls and database failures while persisting retry or terminal transitions. @@ -31,6 +32,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- Permanent fail-first exact-head workflow contracts and authoritative evidence in `docs/doctoring/exact-head-source-workflow-evidence.md`, including the observed synthetic merge checkout, cross-platform source identity assertions, least-privilege boundary, stack invalidation rule, rollback prohibition, and APA 7th GitHub references. - A production rollout and invalid-index recovery runbook for the nonblocking durable-job claim index: `docs/operations/durable-job-claim-index-rollout.md`. - PostgreSQL `FOR UPDATE SKIP LOCKED` durable-job claiming, per-process and per-claim lease fencing, expiry reclaim, bounded attempts, exact-live-lease transitions, terminal payload clearing, stable failure codes, and finite-cardinality worker metrics. - Hashed durable execution identity and domain-separated reuse of `etl_idempotency_records`, coupling response replay or creation, target writes, and terminal `SUCCEEDED` in one transaction without retaining or reconstructing raw principals or raw client idempotency keys. @@ -269,5 +271,5 @@ This changelog will be updated: --- **Changelog Version**: 1.0 -**Last Updated**: 2026-08-06 +**Last Updated**: 2026-08-07 **Maintained By**: Development Team