From 245ee8de5a4f1a4b7c7fdd0f88d7eabc74125a73 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 14:19:23 +0900 Subject: [PATCH 001/103] docs(etl): define immutable durable job replay --- .../2026-08-06-durable-job-replay-design.md | 152 ++++++++++++++++++ 1 file changed, 152 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-06-durable-job-replay-design.md diff --git a/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md b/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md new file mode 100644 index 00000000..70d7e85e --- /dev/null +++ b/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md @@ -0,0 +1,152 @@ +# Durable ETL Job Replay Design + +## Purpose + +mightyETL terminalizes failed and cancelled jobs by clearing their retained request payload. Operators still need a safe way to retry the same intended work without mutating terminal history, trusting an unverified replacement payload, or losing the relationship between the original and the new attempt. + +This design adds one authenticated owner-scoped replay action. The client resupplies the complete bounded JSON payload; mightyETL validates it through the existing intake contract and requires its SHA-256 digest to equal the immutable terminal source digest before creating a new `PENDING` resource. + +## API contract + +```http +POST /api/etl/jobs/{source_job_record_id}/replays +Authorization: +Idempotency-Key: "new-replay-key" +Content-Type: application/json + +[{"id":"record_alpha","name":"accepted"}] +``` + +A first accepted replay returns RFC 9110 `202 Accepted`, `Location` for the new job, `Cache-Control: no-store`, and `Idempotency-Replayed: false`. The same principal, replay key, source job, and byte-identical payload returns the same new job with `Idempotency-Replayed: true`. + +The source remains terminal and unchanged. Replay is allowed only from `FAILED` and `CANCELLED`. Active sources conflict because they still own or may own execution. `SUCCEEDED` conflicts because a first-slice replay could duplicate committed target effects. + +## Immutable relational lineage + +Migration `V7__add_etl_job_replay_lineage.sql` adds: + +```text +replay_source_job_record_id +replay_root_job_record_id +replay_generation_count +``` + +The root job has all three fields null. Every replay row has all three fields non-null, an immediate source different from itself, a root different from itself, and a generation from 1 through 100. Self-referencing foreign keys use `ON DELETE RESTRICT` so terminal history cannot disappear through cascade deletion. + +For the first replay: + +```text +source = terminal root job +root = terminal root job +generation = 1 +``` + +For replay of a replay: + +```text +source = immediate terminal replay +root = inherited first job +generation = source generation + 1 +``` + +The application verifies that source and inherited root are owner-scoped to the same principal. Database constraints remain the structural boundary; the relational rows are authoritative even when lineage is later exported as W3C PROV. + +## Replay-key authority + +The replay key is normalized through the same bounded quoted-or-legacy safe profile as other idempotency keys. The new job stores a versioned principal-scoped replay identity in the existing `submission_key_hash` field: + +```text +SHA-256( + "mightyetl:durable-job-replay:v1:" + || principal_scope_hash + || ":" + || normalized_replay_key +) +``` + +This isolates replay keys from ordinary submission keys and from another tenant. Within one principal namespace, the same replay key can identify only one new job. Reusing it with another source or payload fails with `etl_job_replay_key_reused`. + +A transaction-level lock derived from the replay identity serializes concurrent creation. The table's existing principal-plus-submission-hash unique constraint remains the second integrity boundary. A concurrent request that cannot acquire the lock returns `etl_job_replay_in_progress`; retrying after the first transaction completes replays the committed new job. + +## Database transaction + +One transaction performs the following sequence: + +1. validate source identifier, replay key, principal, and complete bounded payload before lock or table access; +2. compute principal, replay-key, and payload digests; +3. acquire the replay-key transaction lock; +4. find and classify an existing job using that replay identity; +5. select the owner-scoped source and immutable lineage; +6. require terminal `FAILED` or `CANCELLED`; +7. require the supplied payload digest to equal the source `request_digest`; +8. derive root and bounded generation; +9. insert one new `PENDING` row with the verified payload and lineage; +10. return only the new operator-safe job identity. + +The source is never updated. Read-then-write state resurrection is prohibited. + +## Error taxonomy + +| HTTP | Stable code | Meaning | +| ---: | --- | --- | +| 400 | `etl_job_replay_key_required` | Replay key is missing or outside the bounded profile. | +| 404 | `etl_job_not_found` | Source is malformed, missing, or foreign-owned. | +| 409 | `etl_job_replay_in_progress` | Another transaction owns the principal-scoped replay identity. | +| 409 | `etl_job_replay_source_active` | Source is `PENDING` or `RUNNING`. | +| 409 | `etl_job_replay_source_succeeded` | Source already committed successful effects. | +| 409 | `etl_job_replay_generation_exhausted` | Generation 100 cannot create generation 101. | +| 422 | `etl_job_replay_payload_mismatch` | Resupplied JSON does not match the immutable source digest. | +| 422 | `etl_job_replay_key_reused` | Replay key already identifies another source or payload. | + +All failures use the existing RFC 9457 problem model without payload, principal, key, hash, lineage internals, SQL, or exception text. + +## Worker compatibility + +The new row is an ordinary `PENDING` job. Existing worker claim, lease fencing, retry, success, failure, cancellation, pagination, polling, and conditional-status contracts apply without a replay-specific execution path. Only lineage and admission differ. + +## Provenance export + +A later JSON-LD adapter may map the relational evidence as: + +```text +source job → prov:Entity +replay action → prov:Activity +new job → prov:Entity +new job → prov:wasDerivedFrom → source job +replay action → prov:used → source job +new job → prov:wasGeneratedBy → replay action +``` + +The export must not weaken owner authorization or replace database constraints. + +## Verification + +The exact-head suite must prove: + +1. failed and cancelled sources each create a distinct pending job; +2. source status, terminal evidence, timestamps, and cleared payload remain unchanged; +3. payload mismatch fails before insertion; +4. same source, key, and payload replay one new job; +5. same key with another source or payload fails closed; +6. foreign and missing sources remain indistinguishable; +7. pending, running, and succeeded sources are rejected; +8. replay of replay preserves root and increments generation; +9. generation 100 cannot create generation 101; +10. concurrent creation produces one row and an in-progress or later replay outcome; +11. the new job can be claimed and follows normal lifecycle contracts; +12. migration completeness, self-reference prohibition, naming, deletion restriction, rollout, and rollback are documented and tested; +13. all added production statements and branches retain zero-missed configured coverage and no project test is skipped. + +## Operational limitation + +Replay verifies that the resupplied payload matches the immutable source digest. It does not prove that replaying a connector is economically or externally safe. A target that cannot provide transactional or idempotent effects requires connector-specific policy before replay is enabled for that connector. + +## References — APA 7th + +Fielding, R., Nottingham, M., & Reschke, J. (2022). *HTTP semantics* (RFC 9110). RFC Editor. https://www.rfc-editor.org/rfc/rfc9110 + +Nottingham, M., Wilde, E., & Dalal, S. (2023). *Problem details for HTTP APIs* (RFC 9457). RFC Editor. https://www.rfc-editor.org/rfc/rfc9457 + +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: INSERT*. https://www.postgresql.org/docs/18/sql-insert.html + +World Wide Web Consortium. (2013). *PROV-O: The PROV ontology*. https://www.w3.org/TR/prov-o/ From 9f86566853c55b983e2b05d18a7bd7f8f6380fd6 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 14:20:33 +0900 Subject: [PATCH 002/103] docs(etl): plan immutable durable job replay --- .../plans/2026-08-06-durable-job-replay.md | 117 ++++++++++++++++++ 1 file changed, 117 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-06-durable-job-replay.md diff --git a/docs/superpowers/plans/2026-08-06-durable-job-replay.md b/docs/superpowers/plans/2026-08-06-durable-job-replay.md new file mode 100644 index 00000000..3e48bfd7 --- /dev/null +++ b/docs/superpowers/plans/2026-08-06-durable-job-replay.md @@ -0,0 +1,117 @@ +# Durable ETL Job Replay Implementation Plan + +> **Execution rule:** implement every state, lineage, and admission boundary test-first and preserve the terminal source as immutable evidence. + +**Goal:** Create a new owner-scoped durable job from a failed or cancelled source only after the client resupplies the exact original payload. + +**Architecture:** Store replay lineage on the new job row, use a versioned principal-scoped replay-key hash in the existing submission identity column, serialize creation with the existing transaction-lock boundary, verify payload digest against the terminal source, and return the existing accepted-job wire model. + +**Tech Stack:** Java 25, Spring MVC, Spring transactions, JdbcTemplate, PostgreSQL 18, Flyway, H2 integration tests, JUnit 5, Mockito, JaCoCo, Maven. + +## Global constraints + +- Never update a terminal source back to `PENDING`. +- Allow only `FAILED` and `CANCELLED` sources. +- Validate identifier, replay key, principal, and complete payload before lock or table access. +- Persist no raw principal or raw replay key. +- Preserve zero-missed configured production instruction, line, method, and branch coverage. +- Preserve no-skipped project tests and beginner-readable public Javadoc. +- Use descriptive multi-word `snake_case` database objects. +- Keep all existing review-agent credentials and workflows unchanged. + +## Task 1 — Lock the V7 lineage schema first + +**Files** +- Modify: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobMigrationDocumentationTest.java` +- Create: `etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql` + +- [ ] Require all three lineage columns, bounded generation, complete-null-or-complete-non-null lifecycle, self-reference rejection, two named self-referencing foreign keys, and `ON DELETE RESTRICT`. +- [ ] Run the focused migration test and observe failure because V7 is absent. +- [ ] Implement the additive transactional migration. +- [ ] Rerun the focused test and commit. + +## Task 2 — Define immutable replay models and errors + +**Files** +- Create: `etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplay.java` +- Modify: `etl-service/src/main/java/com/xtrmetl/etl/service/EtlRequestError.java` +- Create: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayTest.java` +- Modify: `etl-service/src/test/java/com/xtrmetl/etl/service/EtlRequestExceptionTest.java` + +- [ ] Add fail-first model validation and stable RFC 9457 metadata tests. +- [ ] Add the immutable replay result with new job ID, `PENDING`, and replay flag only. +- [ ] Add required, mismatch, reused, in-progress, active, succeeded, and generation-exhausted errors. +- [ ] Run focused tests and commit. + +## Task 3 — Implement the service transaction test-first + +**Files** +- Create: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayServiceIntegrationTest.java` +- Create: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayServiceBoundaryTest.java` +- Modify: `etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobService.java` + +**Public interface** + +```java +EtlJobReplay replayOwned( + UUID sourceJobRecordId, + String requestPayload, + String replayKey, + String principalScope +) +``` + +- [ ] Add failing tests for failed source, cancelled source, source immutability, payload mismatch, key replay/reuse, owner isolation, active/succeeded rejection, lineage root/generation, generation exhaustion, and validation before JDBC. +- [ ] Run focused tests and observe compile/assertion failure. +- [ ] Add the versioned replay domain and transaction-lock identity. +- [ ] Add existing-replay lookup and source-lineage lookup. +- [ ] Insert one ordinary `PENDING` job with verified payload and lineage. +- [ ] Run focused tests and commit. + +## Task 4 — Add the HTTP resource + +**Files** +- Modify: `etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobController.java` +- Modify: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobControllerTest.java` +- Modify: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobControllerFailureTest.java` + +- [ ] Add fail-first tests for first acceptance, replay, authentication, missing key, malformed source, typed error, database failure, and unexpected failure. +- [ ] Implement `POST /api/etl/jobs/{sourceJobRecordId}/replays` with `202`, `Location`, no-store, and replay header. +- [ ] Run focused tests and commit. + +## Task 5 — Prove lifecycle and concurrency compatibility + +**Files** +- Create: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayClaimIntegrationTest.java` +- Create: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayConcurrencyIntegrationTest.java` +- Create: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayLineageIntegrationTest.java` + +- [ ] Prove a new replay row can be claimed by the ordinary worker. +- [ ] Prove an unavailable transaction lock returns replay-in-progress without insertion. +- [ ] Prove retry after a committed first replay returns the same new row. +- [ ] Prove replay-of-replay preserves the root and increments generation. +- [ ] Prove generation 100 fails before insertion. +- [ ] Run focused and full tests and commit. + +## Task 6 — Finish operations, provenance, and exact-head verification + +**Files** +- Create: `docs/operations/durable-job-replay.md` +- Create: `docs/doctoring/durable-job-replay-key-domain-separation.md` +- Modify: `docs/etl/durable-job-intake.md` +- Modify: `CHANGELOG.md` +- Create: `etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java` + +- [ ] Require source immutability, payload digest proof, lineage, concurrency, generation limit, connector limitation, rollout, and rollback documentation first. +- [ ] Document W3C PROV mapping as an export contract, not a database-authority substitute. +- [ ] Record APA 7th primary references and the versioned replay-key compatibility boundary. +- [ ] Run `./mvnw -B test`, configured coverage gates, and `git diff --check` through exact-head CI. +- [ ] Keep the PR draft until every stacked-target gate succeeds. + +## Plan self-review + +- Every issue #134 acceptance requirement maps to a task. +- The source is never mutated. +- Replay-key and payload conflicts are distinguished without disclosing source existence across principals. +- New jobs enter the existing worker lifecycle rather than creating a second execution engine. +- No placeholder, ambiguous public signature, or unbounded database object name remains. From 4b4d780ca1deae9e3f7bf6955d783f07d6996aba Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 14:21:57 +0900 Subject: [PATCH 003/103] test(etl): require immutable replay lineage migration --- .../etl/job/EtlJobReplayMigrationTest.java | 72 +++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java new file mode 100644 index 00000000..d27cd82b --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java @@ -0,0 +1,72 @@ +package com.xtrmetl.etl.job; + +import org.junit.jupiter.api.Test; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; + +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Guards the complete, bounded, non-cascading durable job replay lineage schema. + */ +class EtlJobReplayMigrationTest { + + @Test + void replayMigrationAddsCompleteRestrictedSelfReferencingLineage() throws IOException { + String migration = Files.readString( + projectRoot().resolve( + "etl-service/src/main/resources/db/migration/" + + "V7__add_etl_job_replay_lineage.sql" + ), + StandardCharsets.UTF_8 + ).replaceAll("\\s+", " "); + + assertTrue(migration.contains("ADD COLUMN replay_source_job_record_id UUID")); + assertTrue(migration.contains("ADD COLUMN replay_root_job_record_id UUID")); + assertTrue(migration.contains("ADD COLUMN replay_generation_count INTEGER")); + assertTrue(migration.contains("CONSTRAINT etl_job_replay_source_reference")); + assertTrue(migration.contains( + "FOREIGN KEY (replay_source_job_record_id) REFERENCES etl_job_records (job_record_id) ON DELETE RESTRICT" + )); + assertTrue(migration.contains("CONSTRAINT etl_job_replay_root_reference")); + assertTrue(migration.contains( + "FOREIGN KEY (replay_root_job_record_id) REFERENCES etl_job_records (job_record_id) ON DELETE RESTRICT" + )); + assertTrue(migration.contains("CONSTRAINT etl_job_replay_lineage_complete_check")); + assertTrue(migration.contains("replay_source_job_record_id IS NULL")); + assertTrue(migration.contains("replay_root_job_record_id IS NULL")); + assertTrue(migration.contains("replay_generation_count IS NULL")); + assertTrue(migration.contains("replay_source_job_record_id IS NOT NULL")); + assertTrue(migration.contains("replay_root_job_record_id IS NOT NULL")); + assertTrue(migration.contains("replay_generation_count BETWEEN 1 AND 100")); + assertTrue(migration.contains("replay_source_job_record_id <> job_record_id")); + assertTrue(migration.contains("replay_root_job_record_id <> job_record_id")); + assertFalse(migration.contains("ON DELETE CASCADE")); + assertFalse(migration.contains("replay_payload")); + assertFalse(migration.contains("principal_name")); + } + + /** @return reactor root from repository-root or module-local execution */ + private static Path projectRoot() { + Path current = Paths.get(System.getProperty("user.dir")).toAbsolutePath(); + Path lastPomParent = null; + while (current != null) { + if (Files.exists(current.resolve(".git"))) { + return current; + } + if (Files.exists(current.resolve("pom.xml"))) { + lastPomParent = current; + } + current = current.getParent(); + } + if (lastPomParent != null) { + return lastPomParent; + } + throw new IllegalStateException("Could not find project root"); + } +} From b059102422799f5671569f0e0140cffeacfd3002 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 14:22:13 +0900 Subject: [PATCH 004/103] feat(etl): add immutable durable job replay lineage --- .../V7__add_etl_job_replay_lineage.sql | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql diff --git a/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql b/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql new file mode 100644 index 00000000..88ecf4d7 --- /dev/null +++ b/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql @@ -0,0 +1,30 @@ +-- Add immutable source/root/generation lineage to replay-created durable job rows. +ALTER TABLE etl_job_records + ADD COLUMN replay_source_job_record_id UUID, + ADD COLUMN replay_root_job_record_id UUID, + ADD COLUMN replay_generation_count INTEGER; + +ALTER TABLE etl_job_records + ADD CONSTRAINT etl_job_replay_source_reference + FOREIGN KEY (replay_source_job_record_id) + REFERENCES etl_job_records (job_record_id) + ON DELETE RESTRICT, + ADD CONSTRAINT etl_job_replay_root_reference + FOREIGN KEY (replay_root_job_record_id) + REFERENCES etl_job_records (job_record_id) + ON DELETE RESTRICT, + ADD CONSTRAINT etl_job_replay_lineage_complete_check CHECK ( + ( + replay_source_job_record_id IS NULL + AND replay_root_job_record_id IS NULL + AND replay_generation_count IS NULL + ) + OR + ( + replay_source_job_record_id IS NOT NULL + AND replay_root_job_record_id IS NOT NULL + AND replay_generation_count BETWEEN 1 AND 100 + AND replay_source_job_record_id <> job_record_id + AND replay_root_job_record_id <> job_record_id + ) + ); From 997dd194397aaf859bccff76fcb25407f68a17d7 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 14:23:32 +0900 Subject: [PATCH 005/103] feat(etl): classify immutable durable job replay failures --- .../xtrmetl/etl/service/EtlRequestError.java | 93 ++++++++++++++----- 1 file changed, 68 insertions(+), 25 deletions(-) diff --git a/etl-service/src/main/java/com/xtrmetl/etl/service/EtlRequestError.java b/etl-service/src/main/java/com/xtrmetl/etl/service/EtlRequestError.java index 332f13f6..e2c6c611 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/service/EtlRequestError.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/service/EtlRequestError.java @@ -167,6 +167,69 @@ public enum EtlRequestError { "The durable job failed before cancellation could commit." ), + /** The replay key is absent or outside the bounded safe idempotency profile. */ + JOB_REPLAY_KEY_REQUIRED( + HttpStatus.BAD_REQUEST, + "etl_job_replay_key_required", + "urn:mightyetl:problem:etl-job-replay-key-required", + "ETL job replay key required", + "Replay requires a supported principal-scoped Idempotency-Key." + ), + + /** The resupplied payload does not match the immutable terminal source digest. */ + JOB_REPLAY_PAYLOAD_MISMATCH( + HttpStatus.UNPROCESSABLE_ENTITY, + "etl_job_replay_payload_mismatch", + "urn:mightyetl:problem:etl-job-replay-payload-mismatch", + "ETL job replay payload mismatch", + "The replay payload does not match the immutable source job payload digest." + ), + + /** The replay identity already belongs to another source or payload. */ + JOB_REPLAY_KEY_REUSED( + HttpStatus.UNPROCESSABLE_ENTITY, + "etl_job_replay_key_reused", + "urn:mightyetl:problem:etl-job-replay-key-reused", + "ETL job replay key reused", + "The Idempotency-Key already identifies a different durable job replay." + ), + + /** Another transaction owns the same principal-scoped replay identity. */ + JOB_REPLAY_IN_PROGRESS( + HttpStatus.CONFLICT, + "etl_job_replay_in_progress", + "urn:mightyetl:problem:etl-job-replay-in-progress", + "ETL job replay in progress", + "A durable job replay with the same principal-scoped Idempotency-Key is being created." + ), + + /** A pending or running source is still active and cannot be replayed. */ + JOB_REPLAY_SOURCE_ACTIVE( + HttpStatus.CONFLICT, + "etl_job_replay_source_active", + "urn:mightyetl:problem:etl-job-replay-source-active", + "ETL job replay source active", + "Pending or running durable jobs cannot be replayed." + ), + + /** A succeeded source is excluded to prevent silent duplicate target effects. */ + JOB_REPLAY_SOURCE_SUCCEEDED( + HttpStatus.CONFLICT, + "etl_job_replay_source_succeeded", + "urn:mightyetl:problem:etl-job-replay-source-succeeded", + "ETL job replay source succeeded", + "A succeeded durable job cannot be replayed through this endpoint." + ), + + /** The bounded immutable replay lineage cannot create another generation. */ + JOB_REPLAY_GENERATION_EXHAUSTED( + HttpStatus.CONFLICT, + "etl_job_replay_generation_exhausted", + "urn:mightyetl:problem:etl-job-replay-generation-exhausted", + "ETL job replay generation exhausted", + "The durable job replay lineage reached its maximum generation." + ), + /** The requested job does not exist in the authenticated principal's namespace. */ JOB_NOT_FOUND( HttpStatus.NOT_FOUND, @@ -196,47 +259,27 @@ public enum EtlRequestError { this.detail = Objects.requireNonNull(detail, "detail must not be null"); } - /** - * Returns the HTTP status for this deterministic request failure. - * - * @return immutable HTTP status - */ + /** @return HTTP status for this deterministic request failure */ public HttpStatus status() { return status; } - /** - * Returns the stable snake_case machine code. - * - * @return compatibility-safe error code - */ + /** @return stable snake_case compatibility-safe machine code */ public String errorCode() { return errorCode; } - /** - * Returns the stable RFC 9457 problem type URI. - * - * @return problem type URI - */ + /** @return stable RFC 9457 problem type URI */ public URI type() { return type; } - /** - * Returns the fixed human-readable category title. - * - * @return problem title - */ + /** @return fixed human-readable problem title */ public String title() { return title; } - /** - * Returns the fixed non-sensitive client guidance. - * - * @return problem detail - */ + /** @return fixed non-sensitive client guidance */ public String detail() { return detail; } From 9af7733bdc93ef8cb836af84f5f7dca5f3fa72be Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 14:23:53 +0900 Subject: [PATCH 006/103] feat(etl): add immutable durable job replay result --- .../com/xtrmetl/etl/job/EtlJobReplay.java | 34 +++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplay.java diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplay.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplay.java new file mode 100644 index 00000000..42e22647 --- /dev/null +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplay.java @@ -0,0 +1,34 @@ +package com.xtrmetl.etl.job; + +import java.util.Objects; +import java.util.UUID; + +/** + * Reports one newly accepted or replayed immutable-lineage durable ETL job. + * + *

The source terminal resource and lineage remain internal persistence evidence. This result + * exposes only the new opaque job identifier, its required initial pending state, and whether the + * same principal-scoped replay request had already created it.

+ * + * @param jobRecordId newly created durable job identifier + * @param jobStatus required initial {@link EtlJobStatus#PENDING} state + * @param replayed {@code true} when this response reuses an already-created replay job + */ +public record EtlJobReplay( + UUID jobRecordId, + EtlJobStatus jobStatus, + boolean replayed +) { + + /** Validates the immutable replay result. */ + public EtlJobReplay { + Objects.requireNonNull(jobRecordId, "jobRecordId must not be null"); + EtlJobStatus requiredStatus = Objects.requireNonNull( + jobStatus, + "jobStatus must not be null" + ); + if (requiredStatus != EtlJobStatus.PENDING) { + throw new IllegalArgumentException("replay jobStatus must be PENDING"); + } + } +} From fb911df41099ed4d197d7d81d50c33504da487aa Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 14:25:10 +0900 Subject: [PATCH 007/103] fix(etl): preserve current replay job status on idempotent retry --- .../java/com/xtrmetl/etl/job/EtlJobReplay.java | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplay.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplay.java index 42e22647..65caf105 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplay.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplay.java @@ -7,11 +7,12 @@ * Reports one newly accepted or replayed immutable-lineage durable ETL job. * *

The source terminal resource and lineage remain internal persistence evidence. This result - * exposes only the new opaque job identifier, its required initial pending state, and whether the - * same principal-scoped replay request had already created it.

+ * exposes only the new opaque job identifier, its current stable lifecycle state, and whether the + * same principal-scoped replay request had already created it. A first creation is pending; a later + * idempotent retry may correctly report that the same created job has since progressed.

* - * @param jobRecordId newly created durable job identifier - * @param jobStatus required initial {@link EtlJobStatus#PENDING} state + * @param jobRecordId replay-created durable job identifier + * @param jobStatus current stable lifecycle state of that created job * @param replayed {@code true} when this response reuses an already-created replay job */ public record EtlJobReplay( @@ -23,12 +24,6 @@ public record EtlJobReplay( /** Validates the immutable replay result. */ public EtlJobReplay { Objects.requireNonNull(jobRecordId, "jobRecordId must not be null"); - EtlJobStatus requiredStatus = Objects.requireNonNull( - jobStatus, - "jobStatus must not be null" - ); - if (requiredStatus != EtlJobStatus.PENDING) { - throw new IllegalArgumentException("replay jobStatus must be PENDING"); - } + Objects.requireNonNull(jobStatus, "jobStatus must not be null"); } } From 104dfa38d09c5ca40fe822c1252d2c7941e7749a Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 14:26:32 +0900 Subject: [PATCH 008/103] feat(etl): create immutable owner-scoped durable job replays --- .../xtrmetl/etl/job/EtlJobReplayService.java | 390 ++++++++++++++++++ 1 file changed, 390 insertions(+) create mode 100644 etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplayService.java diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplayService.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplayService.java new file mode 100644 index 00000000..ff488095 --- /dev/null +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplayService.java @@ -0,0 +1,390 @@ +package com.xtrmetl.etl.job; + +import com.fasterxml.jackson.core.JsonParser; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.xtrmetl.etl.service.EtlBatchProperties; +import com.xtrmetl.etl.service.EtlRequestError; +import com.xtrmetl.etl.service.EtlRequestException; +import com.xtrmetl.etl.service.EtlRequestLock; +import com.xtrmetl.etl.service.PostgresEtlRequestLock; +import com.xtrmetl.etl.service.Sha256Digest; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.lang.Nullable; +import org.springframework.stereotype.Service; +import org.springframework.transaction.annotation.Transactional; +import org.springframework.transaction.support.TransactionSynchronizationManager; + +import java.nio.charset.StandardCharsets; +import java.util.List; +import java.util.Objects; +import java.util.UUID; +import java.util.regex.Pattern; + +/** + * Creates new durable jobs from immutable failed or cancelled owner-scoped source evidence. + * + *

The client resupplies the complete bounded JSON payload because terminal rows deliberately + * clear it. Replay validates the payload through the same record contract as ordinary intake and + * requires its SHA-256 digest to equal the source digest before insertion. The terminal source is + * never updated.

+ * + *

A versioned principal-scoped replay-key hash is stored in the existing submission identity + * column. The transaction-level request lock serializes one key within a principal namespace, and + * the table unique constraint remains a second integrity boundary. Replay-created rows enter the + * ordinary pending/worker lifecycle with immutable source, root, and bounded generation lineage.

+ */ +@Service +public class EtlJobReplayService { + + /** Maximum number of replay generations retained by the first lineage contract. */ + public static final int MAXIMUM_REPLAY_GENERATION = 100; + + private static final String REPLAY_KEY_DOMAIN = "mightyetl:durable-job-replay:v1:"; + private static final String REPLAY_LOCK_DOMAIN = "mightyetl:durable-job-replay-lock:v1:"; + private static final int MAX_PRINCIPAL_SCOPE_CODE_POINTS = 512; + private static final String KEY_VALUE_EXPRESSION = "[A-Za-z0-9._:-]{16,128}"; + private static final Pattern KEY_VALUE_PROFILE = Pattern.compile(KEY_VALUE_EXPRESSION); + private static final Pattern KEY_STRUCTURED_FIELD_PROFILE = Pattern.compile( + "\"(" + KEY_VALUE_EXPRESSION + ")\"" + ); + + private static final String SELECT_EXISTING_REPLAY_SQL = """ + SELECT job_record_id, request_digest, job_status, + replay_source_job_record_id + FROM etl_job_records + WHERE principal_scope_hash = ? + AND submission_key_hash = ? + """; + private static final String SELECT_REPLAY_SOURCE_SQL = """ + SELECT job_record_id, request_digest, job_status, + replay_root_job_record_id, replay_generation_count + FROM etl_job_records + WHERE job_record_id = ? + AND principal_scope_hash = ? + FOR UPDATE + """; + private static final String SELECT_OWNED_ROOT_COUNT_SQL = """ + SELECT COUNT(*) + FROM etl_job_records + WHERE job_record_id = ? + AND principal_scope_hash = ? + """; + private static final String INSERT_REPLAY_JOB_SQL = """ + INSERT INTO etl_job_records ( + job_record_id, + principal_scope_hash, + submission_key_hash, + request_digest, + request_payload, + job_status, + attempt_count, + replay_source_job_record_id, + replay_root_job_record_id, + replay_generation_count + ) VALUES (?, ?, ?, ?, ?, 'PENDING', 0, ?, ?, ?) + """; + + private final JdbcTemplate jdbcTemplate; + private final ObjectMapper objectMapper; + private final EtlBatchProperties batchProperties; + private final EtlRequestLock requestLock; + + /** + * Creates replay admission with the PostgreSQL transaction-lock implementation. + * + * @param jdbcTemplate parameterized durable job persistence + * @param objectMapper JSON parser configuration to copy + * @param batchProperties bounded request limits + */ + public EtlJobReplayService( + JdbcTemplate jdbcTemplate, + ObjectMapper objectMapper, + EtlBatchProperties batchProperties + ) { + this( + jdbcTemplate, + objectMapper, + batchProperties, + new PostgresEtlRequestLock(jdbcTemplate) + ); + } + + /** + * Creates replay admission with an explicit transaction-lifetime request lock. + * + * @param jdbcTemplate parameterized durable job persistence + * @param objectMapper JSON parser configuration to copy + * @param batchProperties bounded request limits + * @param requestLock transaction-lifetime replay-key lock + */ + @Autowired + public EtlJobReplayService( + JdbcTemplate jdbcTemplate, + ObjectMapper objectMapper, + EtlBatchProperties batchProperties, + EtlRequestLock requestLock + ) { + this.jdbcTemplate = Objects.requireNonNull(jdbcTemplate, "jdbcTemplate must not be null"); + ObjectMapper sourceMapper = Objects.requireNonNull( + objectMapper, + "objectMapper must not be null" + ); + this.objectMapper = sourceMapper.copy(); + this.objectMapper.enable(JsonParser.Feature.STRICT_DUPLICATE_DETECTION); + this.batchProperties = Objects.requireNonNull( + batchProperties, + "batchProperties must not be null" + ); + this.requestLock = Objects.requireNonNull(requestLock, "requestLock must not be null"); + } + + /** + * Creates or replays one new durable job from an immutable terminal source. + * + * @param sourceJobRecordId owner-scoped failed or cancelled source identifier + * @param requestPayload resupplied exact bounded JSON array text + * @param replayKey quoted Structured Field String or supported legacy raw safe value + * @param principalScope authenticated principal namespace + * @return new or previously-created replay job identity and current state + * @throws NullPointerException when the source identifier is {@code null} + * @throws EtlRequestException when validation, ownership, state, digest, key, or generation + * contracts fail + * @throws IllegalStateException when no actual transaction is active or stored lineage is + * internally inconsistent + */ + @Transactional + public EtlJobReplay replayOwned( + UUID sourceJobRecordId, + @Nullable String requestPayload, + @Nullable String replayKey, + @Nullable String principalScope + ) { + UUID validatedSourceId = Objects.requireNonNull( + sourceJobRecordId, + "sourceJobRecordId must not be null" + ); + String validatedKey = validateReplayKey(replayKey); + String validatedScope = validatePrincipalScope(principalScope); + String validatedPayload = validatePayload(requestPayload); + requireActiveTransaction(); + + String principalScopeHash = Sha256Digest.digest(validatedScope); + String replayKeyHash = Sha256Digest.digest( + REPLAY_KEY_DOMAIN + principalScopeHash + ':' + validatedKey + ); + String replayLockHash = Sha256Digest.digest( + REPLAY_LOCK_DOMAIN + principalScopeHash + ':' + replayKeyHash + ); + String requestDigest = Sha256Digest.digest(validatedPayload); + + if (!requestLock.tryLock(replayLockHash)) { + throw new EtlRequestException(EtlRequestError.JOB_REPLAY_IN_PROGRESS); + } + + ExistingReplay existingReplay = findExistingReplay(principalScopeHash, replayKeyHash); + if (existingReplay != null) { + if (!validatedSourceId.equals(existingReplay.sourceJobRecordId()) + || !requestDigest.equals(existingReplay.requestDigest())) { + throw new EtlRequestException(EtlRequestError.JOB_REPLAY_KEY_REUSED); + } + return new EtlJobReplay( + existingReplay.jobRecordId(), + existingReplay.jobStatus(), + true + ); + } + + ReplaySource source = findSource(validatedSourceId, principalScopeHash); + if (source == null) { + throw new EtlRequestException(EtlRequestError.JOB_NOT_FOUND); + } + switch (source.jobStatus()) { + case PENDING, RUNNING -> throw new EtlRequestException( + EtlRequestError.JOB_REPLAY_SOURCE_ACTIVE + ); + case SUCCEEDED -> throw new EtlRequestException( + EtlRequestError.JOB_REPLAY_SOURCE_SUCCEEDED + ); + case FAILED, CANCELLED -> { + // These terminal outcomes are the only first-slice replay sources. + } + } + if (!source.requestDigest().equals(requestDigest)) { + throw new EtlRequestException(EtlRequestError.JOB_REPLAY_PAYLOAD_MISMATCH); + } + + UUID rootJobRecordId; + int replayGeneration; + if (source.replayRootJobRecordId() == null) { + if (source.replayGenerationCount() != null) { + throw new IllegalStateException("Replay source has incomplete root lineage"); + } + rootJobRecordId = source.jobRecordId(); + replayGeneration = 1; + } else { + Integer sourceGeneration = Objects.requireNonNull( + source.replayGenerationCount(), + "Replay source generation must accompany its root" + ); + if (sourceGeneration >= MAXIMUM_REPLAY_GENERATION) { + throw new EtlRequestException( + EtlRequestError.JOB_REPLAY_GENERATION_EXHAUSTED + ); + } + rootJobRecordId = source.replayRootJobRecordId(); + requireOwnedRoot(rootJobRecordId, principalScopeHash); + replayGeneration = sourceGeneration + 1; + } + + UUID newJobRecordId = UUID.randomUUID(); + jdbcTemplate.update( + INSERT_REPLAY_JOB_SQL, + newJobRecordId, + principalScopeHash, + replayKeyHash, + requestDigest, + validatedPayload, + source.jobRecordId(), + rootJobRecordId, + replayGeneration + ); + return new EtlJobReplay(newJobRecordId, EtlJobStatus.PENDING, false); + } + + @Nullable + private ExistingReplay findExistingReplay( + String principalScopeHash, + String replayKeyHash + ) { + List rows = jdbcTemplate.query( + SELECT_EXISTING_REPLAY_SQL, + (resultSet, rowNumber) -> new ExistingReplay( + resultSet.getObject("job_record_id", UUID.class), + resultSet.getString("request_digest"), + EtlJobStatus.valueOf(resultSet.getString("job_status")), + resultSet.getObject("replay_source_job_record_id", UUID.class) + ), + principalScopeHash, + replayKeyHash + ); + return rows.isEmpty() ? null : rows.getFirst(); + } + + @Nullable + private ReplaySource findSource(UUID sourceJobRecordId, String principalScopeHash) { + List rows = jdbcTemplate.query( + SELECT_REPLAY_SOURCE_SQL, + (resultSet, rowNumber) -> new ReplaySource( + resultSet.getObject("job_record_id", UUID.class), + resultSet.getString("request_digest"), + EtlJobStatus.valueOf(resultSet.getString("job_status")), + resultSet.getObject("replay_root_job_record_id", UUID.class), + resultSet.getObject("replay_generation_count", Integer.class) + ), + sourceJobRecordId, + principalScopeHash + ); + return rows.isEmpty() ? null : rows.getFirst(); + } + + private void requireOwnedRoot(UUID rootJobRecordId, String principalScopeHash) { + Integer count = jdbcTemplate.queryForObject( + SELECT_OWNED_ROOT_COUNT_SQL, + Integer.class, + rootJobRecordId, + principalScopeHash + ); + if (!Integer.valueOf(1).equals(count)) { + throw new IllegalStateException("Replay root is absent from the owner namespace"); + } + } + + private String validatePayload(@Nullable String requestPayload) { + if (requestPayload == null) { + throw new EtlRequestException(EtlRequestError.INVALID_JSON); + } + if (requestPayload.getBytes(StandardCharsets.UTF_8).length + > batchProperties.getMaxPayloadBytes()) { + throw new EtlRequestException(EtlRequestError.PAYLOAD_TOO_LARGE); + } + + final JsonNode root; + try { + root = objectMapper.readTree(requestPayload); + } catch (JsonProcessingException exception) { + throw new EtlRequestException(EtlRequestError.INVALID_JSON, exception); + } + if (root == null || root.isNull() || !root.isArray()) { + throw new EtlRequestException(EtlRequestError.INVALID_JSON); + } + if (root.size() > batchProperties.getMaxBatchRecords()) { + throw new EtlRequestException(EtlRequestError.BATCH_TOO_LARGE); + } + for (JsonNode record : root) { + EtlJobService.validateRecord(record); + } + return requestPayload; + } + + private static String validateReplayKey(@Nullable String replayKey) { + if (replayKey == null) { + throw new EtlRequestException(EtlRequestError.JOB_REPLAY_KEY_REQUIRED); + } + var structuredFieldMatcher = KEY_STRUCTURED_FIELD_PROFILE.matcher(replayKey); + if (structuredFieldMatcher.matches()) { + return structuredFieldMatcher.group(1); + } + if (KEY_VALUE_PROFILE.matcher(replayKey).matches()) { + return replayKey; + } + throw new EtlRequestException(EtlRequestError.JOB_REPLAY_KEY_REQUIRED); + } + + private static String validatePrincipalScope(@Nullable String principalScope) { + if (principalScope == null + || principalScope.isBlank() + || principalScope.codePointCount(0, principalScope.length()) + > MAX_PRINCIPAL_SCOPE_CODE_POINTS) { + throw new EtlRequestException(EtlRequestError.IDEMPOTENCY_PRINCIPAL_REQUIRED); + } + return principalScope; + } + + private static void requireActiveTransaction() { + if (!TransactionSynchronizationManager.isActualTransactionActive()) { + throw new IllegalStateException( + "Durable ETL job replay requires an active transaction" + ); + } + } + + private record ExistingReplay( + UUID jobRecordId, + String requestDigest, + EtlJobStatus jobStatus, + @Nullable UUID sourceJobRecordId + ) { + private ExistingReplay { + Objects.requireNonNull(jobRecordId, "jobRecordId must not be null"); + Objects.requireNonNull(requestDigest, "requestDigest must not be null"); + Objects.requireNonNull(jobStatus, "jobStatus must not be null"); + } + } + + private record ReplaySource( + UUID jobRecordId, + String requestDigest, + EtlJobStatus jobStatus, + @Nullable UUID replayRootJobRecordId, + @Nullable Integer replayGenerationCount + ) { + private ReplaySource { + Objects.requireNonNull(jobRecordId, "jobRecordId must not be null"); + Objects.requireNonNull(requestDigest, "requestDigest must not be null"); + Objects.requireNonNull(jobStatus, "jobStatus must not be null"); + } + } +} From 0b60c12f3f409c051009422e1c88631a98f05285 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 14:27:13 +0900 Subject: [PATCH 009/103] feat(etl): expose immutable durable job replay resource --- .../controller/EtlJobReplayController.java | 126 ++++++++++++++++++ 1 file changed, 126 insertions(+) create mode 100644 etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobReplayController.java diff --git a/etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobReplayController.java b/etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobReplayController.java new file mode 100644 index 00000000..da889715 --- /dev/null +++ b/etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobReplayController.java @@ -0,0 +1,126 @@ +package com.xtrmetl.etl.controller; + +import com.xtrmetl.etl.job.EtlJobAcceptedResponse; +import com.xtrmetl.etl.job.EtlJobReplay; +import com.xtrmetl.etl.job.EtlJobReplayService; +import com.xtrmetl.etl.service.EtlRequestError; +import com.xtrmetl.etl.service.EtlRequestException; +import io.micrometer.observation.annotation.Observed; +import org.springframework.boot.autoconfigure.condition.ConditionalOnBooleanProperty; +import org.springframework.dao.DataAccessException; +import org.springframework.http.CacheControl; +import org.springframework.http.MediaType; +import org.springframework.http.ResponseEntity; +import org.springframework.lang.Nullable; +import org.springframework.web.bind.annotation.PathVariable; +import org.springframework.web.bind.annotation.PostMapping; +import org.springframework.web.bind.annotation.RequestBody; +import org.springframework.web.bind.annotation.RequestHeader; +import org.springframework.web.bind.annotation.RequestMapping; +import org.springframework.web.bind.annotation.RestController; + +import java.net.URI; +import java.security.Principal; +import java.util.Objects; +import java.util.UUID; + +/** + * Exposes immutable-lineage replay admission for owner-scoped terminal durable ETL jobs. + * + *

The source is never resurrected. A replay request supplies the complete candidate payload, + * which the service verifies against the source digest before creating one new ordinary pending + * job. The response is RFC 9110 noncommittal acceptance of that new resource, not evidence that its + * ETL effects have completed.

+ */ +@ConditionalOnBooleanProperty( + prefix = "xtrmetl.etl.jobs", + name = "intake-enabled", + havingValue = true, + matchIfMissing = false +) +@RestController +@RequestMapping("/api/etl/jobs") +public class EtlJobReplayController { + + private final EtlJobReplayService replayService; + + /** + * Creates the replay HTTP adapter. + * + * @param replayService immutable owner-scoped replay admission service + */ + public EtlJobReplayController(EtlJobReplayService replayService) { + this.replayService = Objects.requireNonNull( + replayService, + "replayService must not be null" + ); + } + + /** + * Accepts one verified replay of a failed or cancelled owner-scoped source. + * + * @param sourceJobRecordIdText opaque terminal source identifier text + * @param requestPayload exact bounded JSON array text to verify against the source digest + * @param replayKey required replay idempotency key + * @param principal authenticated principal namespace + * @return accepted new-job identity, status monitor, and replay evidence + */ + @PostMapping("/{sourceJobRecordId}/replays") + @Observed(name = "etl.jobs.replay", contextualName = "etl-job-replay") + public ResponseEntity replay( + @PathVariable("sourceJobRecordId") String sourceJobRecordIdText, + @RequestBody String requestPayload, + @RequestHeader(value = "Idempotency-Key", required = false) + @Nullable String replayKey, + @Nullable Principal principal + ) { + if (principal == null) { + throw new EtlRequestException(EtlRequestError.IDEMPOTENCY_PRINCIPAL_REQUIRED); + } + if (replayKey == null) { + throw new EtlRequestException(EtlRequestError.JOB_REPLAY_KEY_REQUIRED); + } + UUID sourceJobRecordId = parseJobRecordId(sourceJobRecordIdText); + + final EtlJobReplay replay; + try { + replay = replayService.replayOwned( + sourceJobRecordId, + requestPayload, + replayKey, + principal.getName() + ); + } catch (EtlRequestException | DataAccessException exception) { + throw exception; + } catch (RuntimeException exception) { + throw new EtlUnexpectedException(exception); + } + + String statusUrl = "/api/etl/jobs/" + replay.jobRecordId(); + EtlJobAcceptedResponse responseBody = new EtlJobAcceptedResponse( + replay.jobRecordId(), + replay.jobStatus(), + statusUrl + ); + return ResponseEntity.accepted() + .cacheControl(CacheControl.noStore()) + .location(URI.create(statusUrl)) + .header( + EtlJobController.IDEMPOTENCY_REPLAYED_HEADER, + Boolean.toString(replay.replayed()) + ) + .contentType(MediaType.APPLICATION_JSON) + .body(responseBody); + } + + private static UUID parseJobRecordId(String jobRecordIdText) { + try { + return UUID.fromString(Objects.requireNonNull( + jobRecordIdText, + "sourceJobRecordIdText must not be null" + )); + } catch (IllegalArgumentException exception) { + throw new EtlRequestException(EtlRequestError.JOB_NOT_FOUND, exception); + } + } +} From e36759daa705a13a84128de10ce4728b40cbedc7 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 14:28:30 +0900 Subject: [PATCH 010/103] test(etl): cover immutable owner-scoped durable job replay --- .../EtlJobReplayServiceIntegrationTest.java | 416 ++++++++++++++++++ 1 file changed, 416 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayServiceIntegrationTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayServiceIntegrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayServiceIntegrationTest.java new file mode 100644 index 00000000..adc8ac1d --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayServiceIntegrationTest.java @@ -0,0 +1,416 @@ +package com.xtrmetl.etl.job; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.xtrmetl.etl.service.EtlBatchProperties; +import com.xtrmetl.etl.service.EtlRequestError; +import com.xtrmetl.etl.service.EtlRequestException; +import com.xtrmetl.etl.service.EtlRequestLock; +import com.xtrmetl.etl.service.Sha256Digest; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.jdbc.datasource.DataSourceTransactionManager; +import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseBuilder; +import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseType; +import org.springframework.test.context.junit.jupiter.SpringJUnitConfig; +import org.springframework.transaction.PlatformTransactionManager; +import org.springframework.transaction.annotation.EnableTransactionManagement; + +import javax.sql.DataSource; +import java.time.Instant; +import java.util.UUID; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotEquals; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Covers immutable replay admission, owner isolation, payload proof, and lineage generation. + */ +@SpringJUnitConfig(EtlJobReplayServiceIntegrationTest.TestConfiguration.class) +class EtlJobReplayServiceIntegrationTest { + + private static final String PAYLOAD = "[{\"id\":\"record_alpha\"}]"; + private static final String OTHER_PAYLOAD = "[{\"id\":\"record_beta\"}]"; + private static final String REPLAY_KEY = "1e05bdca-447c-4ad3-882c-e33963ce517c"; + private static final String OTHER_REPLAY_KEY = "519bc126-1398-4b4e-a4e3-1fb18a00f19b"; + + private final EtlJobReplayService replayService; + private final JdbcTemplate jdbcTemplate; + + @Autowired + EtlJobReplayServiceIntegrationTest( + EtlJobReplayService replayService, + JdbcTemplate jdbcTemplate + ) { + this.replayService = replayService; + this.jdbcTemplate = jdbcTemplate; + } + + @BeforeEach + void createJobTable() { + jdbcTemplate.execute("DROP TABLE IF EXISTS etl_job_records"); + jdbcTemplate.execute(""" + CREATE TABLE etl_job_records ( + job_record_id UUID PRIMARY KEY, + principal_scope_hash CHAR(64) NOT NULL, + submission_key_hash CHAR(64) NOT NULL, + request_digest CHAR(64) NOT NULL, + request_payload CLOB, + job_status VARCHAR(32) NOT NULL, + attempt_count INTEGER NOT NULL DEFAULT 0, + failure_code VARCHAR(128), + lease_claim_id UUID, + lease_owner_id VARCHAR(128), + lease_expires_at TIMESTAMP WITH TIME ZONE, + cancellation_key_hash CHAR(64), + cancellation_code VARCHAR(128), + job_cancelled_at TIMESTAMP WITH TIME ZONE, + replay_source_job_record_id UUID, + replay_root_job_record_id UUID, + replay_generation_count INTEGER, + created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT etl_job_submission_scope_unique + UNIQUE (principal_scope_hash, submission_key_hash) + ) + """); + } + + @Test + void createsOnePendingReplayFromAFailedSourceAndReplaysIt() { + UUID sourceId = insertTerminalSource(EtlJobStatus.FAILED, "tenant_alpha", PAYLOAD); + Instant sourceUpdated = instantColumn(sourceId, "updated_at"); + + EtlJobReplay first = replayService.replayOwned( + sourceId, + PAYLOAD, + REPLAY_KEY, + "tenant_alpha" + ); + EtlJobReplay replay = replayService.replayOwned( + sourceId, + PAYLOAD, + "\"" + REPLAY_KEY + "\"", + "tenant_alpha" + ); + + assertFalse(first.replayed()); + assertTrue(replay.replayed()); + assertEquals(first.jobRecordId(), replay.jobRecordId()); + assertEquals(EtlJobStatus.PENDING, first.jobStatus()); + assertEquals(sourceId, uuidColumn(first.jobRecordId(), "replay_source_job_record_id")); + assertEquals(sourceId, uuidColumn(first.jobRecordId(), "replay_root_job_record_id")); + assertEquals(1, integerColumn(first.jobRecordId(), "replay_generation_count")); + assertEquals(PAYLOAD, textColumn(first.jobRecordId(), "request_payload")); + assertEquals("FAILED", textColumn(sourceId, "job_status")); + assertNull(textColumn(sourceId, "request_payload")); + assertEquals(sourceUpdated, instantColumn(sourceId, "updated_at")); + } + + @Test + void createsAReplayFromCancelledSourceAndKeepsCancellationEvidence() { + UUID sourceId = insertTerminalSource(EtlJobStatus.CANCELLED, "tenant_alpha", PAYLOAD); + jdbcTemplate.update( + "UPDATE etl_job_records SET cancellation_key_hash=?, cancellation_code=?, " + + "job_cancelled_at=CURRENT_TIMESTAMP WHERE job_record_id=?", + "d".repeat(64), + EtlJobService.CANCELLED_BY_OWNER_CODE, + sourceId + ); + + EtlJobReplay replay = replayService.replayOwned( + sourceId, + PAYLOAD, + REPLAY_KEY, + "tenant_alpha" + ); + + assertEquals(EtlJobStatus.PENDING, replay.jobStatus()); + assertEquals("CANCELLED", textColumn(sourceId, "job_status")); + assertEquals( + EtlJobService.CANCELLED_BY_OWNER_CODE, + textColumn(sourceId, "cancellation_code") + ); + assertNull(textColumn(sourceId, "request_payload")); + } + + @Test + void rejectsMismatchedPayloadAndReplayKeyReuse() { + UUID sourceId = insertTerminalSource(EtlJobStatus.FAILED, "tenant_alpha", PAYLOAD); + EtlRequestException mismatch = assertThrows( + EtlRequestException.class, + () -> replayService.replayOwned( + sourceId, + OTHER_PAYLOAD, + REPLAY_KEY, + "tenant_alpha" + ) + ); + assertEquals(EtlRequestError.JOB_REPLAY_PAYLOAD_MISMATCH, mismatch.error()); + + EtlJobReplay first = replayService.replayOwned( + sourceId, + PAYLOAD, + REPLAY_KEY, + "tenant_alpha" + ); + UUID otherSource = insertTerminalSource( + EtlJobStatus.FAILED, + "tenant_alpha", + PAYLOAD + ); + EtlRequestException reused = assertThrows( + EtlRequestException.class, + () -> replayService.replayOwned( + otherSource, + PAYLOAD, + REPLAY_KEY, + "tenant_alpha" + ) + ); + + assertEquals(EtlRequestError.JOB_REPLAY_KEY_REUSED, reused.error()); + assertEquals(1, replayRowCount()); + assertNotEquals(sourceId, first.jobRecordId()); + } + + @Test + void hidesForeignAndMissingSourcesAndRejectsUnsupportedStates() { + UUID failed = insertTerminalSource(EtlJobStatus.FAILED, "tenant_alpha", PAYLOAD); + assertError( + EtlRequestError.JOB_NOT_FOUND, + () -> replayService.replayOwned(failed, PAYLOAD, REPLAY_KEY, "tenant_beta") + ); + assertError( + EtlRequestError.JOB_NOT_FOUND, + () -> replayService.replayOwned( + UUID.randomUUID(), + PAYLOAD, + REPLAY_KEY, + "tenant_alpha" + ) + ); + assertError( + EtlRequestError.JOB_REPLAY_SOURCE_ACTIVE, + () -> replayService.replayOwned( + insertSource(EtlJobStatus.PENDING, "tenant_alpha", PAYLOAD, null, null), + PAYLOAD, + REPLAY_KEY, + "tenant_alpha" + ) + ); + assertError( + EtlRequestError.JOB_REPLAY_SOURCE_ACTIVE, + () -> replayService.replayOwned( + insertSource(EtlJobStatus.RUNNING, "tenant_alpha", PAYLOAD, null, null), + PAYLOAD, + REPLAY_KEY, + "tenant_alpha" + ) + ); + assertError( + EtlRequestError.JOB_REPLAY_SOURCE_SUCCEEDED, + () -> replayService.replayOwned( + insertTerminalSource(EtlJobStatus.SUCCEEDED, "tenant_alpha", PAYLOAD), + PAYLOAD, + REPLAY_KEY, + "tenant_alpha" + ) + ); + } + + @Test + void replayOfReplayPreservesRootAndBoundsGeneration() { + UUID root = insertTerminalSource(EtlJobStatus.FAILED, "tenant_alpha", PAYLOAD); + UUID generationOne = insertSource( + EtlJobStatus.FAILED, + "tenant_alpha", + PAYLOAD, + root, + 1 + ); + EtlJobReplay generationTwo = replayService.replayOwned( + generationOne, + PAYLOAD, + REPLAY_KEY, + "tenant_alpha" + ); + + assertEquals(generationOne, uuidColumn( + generationTwo.jobRecordId(), + "replay_source_job_record_id" + )); + assertEquals(root, uuidColumn( + generationTwo.jobRecordId(), + "replay_root_job_record_id" + )); + assertEquals(2, integerColumn( + generationTwo.jobRecordId(), + "replay_generation_count" + )); + + UUID generationHundred = insertSource( + EtlJobStatus.CANCELLED, + "tenant_alpha", + PAYLOAD, + root, + EtlJobReplayService.MAXIMUM_REPLAY_GENERATION + ); + assertError( + EtlRequestError.JOB_REPLAY_GENERATION_EXHAUSTED, + () -> replayService.replayOwned( + generationHundred, + PAYLOAD, + OTHER_REPLAY_KEY, + "tenant_alpha" + ) + ); + } + + private UUID insertTerminalSource( + EtlJobStatus status, + String principal, + String payload + ) { + return insertSource(status, principal, payload, null, null); + } + + private UUID insertSource( + EtlJobStatus status, + String principal, + String payload, + UUID replayRoot, + Integer replayGeneration + ) { + UUID id = UUID.randomUUID(); + UUID replaySource = replayRoot == null ? null : replayRoot; + jdbcTemplate.update( + """ + INSERT INTO etl_job_records ( + job_record_id, principal_scope_hash, submission_key_hash, + request_digest, request_payload, job_status, attempt_count, + failure_code, replay_source_job_record_id, + replay_root_job_record_id, replay_generation_count + ) VALUES (?, ?, ?, ?, ?, ?, 0, ?, ?, ?, ?) + """, + id, + Sha256Digest.digest(principal), + Sha256Digest.digest(UUID.randomUUID().toString()), + Sha256Digest.digest(payload), + status == EtlJobStatus.PENDING || status == EtlJobStatus.RUNNING + ? payload : null, + status.name(), + status == EtlJobStatus.FAILED ? "etl_target_failure" : null, + replaySource, + replayRoot, + replayGeneration + ); + return id; + } + + private static void assertError(EtlRequestError expected, Runnable invocation) { + EtlRequestException exception = assertThrows(EtlRequestException.class, invocation::run); + assertEquals(expected, exception.error()); + } + + private int replayRowCount() { + Integer count = jdbcTemplate.queryForObject( + "SELECT COUNT(*) FROM etl_job_records WHERE replay_generation_count IS NOT NULL", + Integer.class + ); + return count == null ? 0 : count; + } + + private String textColumn(UUID id, String column) { + return jdbcTemplate.queryForObject( + "SELECT " + column + " FROM etl_job_records WHERE job_record_id=?", + String.class, + id + ); + } + + private UUID uuidColumn(UUID id, String column) { + return jdbcTemplate.queryForObject( + "SELECT " + column + " FROM etl_job_records WHERE job_record_id=?", + UUID.class, + id + ); + } + + private Integer integerColumn(UUID id, String column) { + return jdbcTemplate.queryForObject( + "SELECT " + column + " FROM etl_job_records WHERE job_record_id=?", + Integer.class, + id + ); + } + + private Instant instantColumn(UUID id, String column) { + return jdbcTemplate.queryForObject( + "SELECT " + column + " FROM etl_job_records WHERE job_record_id=?", + (resultSet, rowNumber) -> resultSet.getTimestamp(column).toInstant(), + id + ); + } + + /** Minimal transaction-enabled context for replay service integration. */ + @Configuration + @EnableTransactionManagement + static class TestConfiguration { + + @Bean + DataSource dataSource() { + return new EmbeddedDatabaseBuilder() + .generateUniqueName(true) + .setType(EmbeddedDatabaseType.H2) + .build(); + } + + @Bean + JdbcTemplate jdbcTemplate(DataSource dataSource) { + return new JdbcTemplate(dataSource); + } + + @Bean + PlatformTransactionManager transactionManager(DataSource dataSource) { + return new DataSourceTransactionManager(dataSource); + } + + @Bean + ObjectMapper objectMapper() { + return new ObjectMapper(); + } + + @Bean + EtlBatchProperties etlBatchProperties() { + return new EtlBatchProperties(); + } + + @Bean + EtlRequestLock etlRequestLock() { + return lockHash -> true; + } + + @Bean + EtlJobReplayService replayService( + JdbcTemplate jdbcTemplate, + ObjectMapper objectMapper, + EtlBatchProperties batchProperties, + EtlRequestLock requestLock + ) { + return new EtlJobReplayService( + jdbcTemplate, + objectMapper, + batchProperties, + requestLock + ); + } + } +} From 37a06f5b35edd7c8e7e1bdc56406fd496bf93093 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 14:29:22 +0900 Subject: [PATCH 011/103] test(etl): cover immutable durable job replay HTTP contract --- .../etl/job/EtlJobReplayControllerTest.java | 143 ++++++++++++++++++ 1 file changed, 143 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayControllerTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayControllerTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayControllerTest.java new file mode 100644 index 00000000..b4271785 --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayControllerTest.java @@ -0,0 +1,143 @@ +package com.xtrmetl.etl.job; + +import com.xtrmetl.etl.controller.EtlApiProblemHandler; +import com.xtrmetl.etl.controller.EtlJobReplayController; +import com.xtrmetl.etl.service.EtlRequestError; +import com.xtrmetl.etl.service.EtlRequestException; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.springframework.dao.DataAccessException; +import org.springframework.http.MediaType; +import org.springframework.test.web.servlet.MockMvc; +import org.springframework.test.web.servlet.setup.MockMvcBuilders; + +import java.security.Principal; +import java.util.UUID; + +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.verifyNoInteractions; +import static org.mockito.Mockito.when; +import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post; +import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.header; +import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath; +import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status; + +/** + * Defines success, replay, authentication, identifier, and failure behavior for replay admission. + */ +class EtlJobReplayControllerTest { + + private static final UUID SOURCE_ID = UUID.fromString( + "cf4f083f-8c90-4f34-a8b6-b53761de44ef" + ); + private static final UUID NEW_JOB_ID = UUID.fromString( + "86e4d474-dabf-4d6a-9de4-4e8230589363" + ); + private static final String PAYLOAD = "[{\"id\":\"record_alpha\"}]"; + private static final String REPLAY_KEY = "\"1e05bdca-447c-4ad3-882c-e33963ce517c\""; + private static final Principal PRINCIPAL = () -> "tenant_alpha"; + + private EtlJobReplayService replayService; + private MockMvc mockMvc; + + @BeforeEach + void setUp() { + replayService = mock(EtlJobReplayService.class); + mockMvc = MockMvcBuilders + .standaloneSetup(new EtlJobReplayController(replayService)) + .setControllerAdvice(new EtlApiProblemHandler()) + .build(); + } + + @Test + void acceptsANewReplayAndReturnsItsStatusMonitor() throws Exception { + when(replayService.replayOwned(SOURCE_ID, PAYLOAD, REPLAY_KEY, "tenant_alpha")) + .thenReturn(new EtlJobReplay(NEW_JOB_ID, EtlJobStatus.PENDING, false)); + + mockMvc.perform(request(SOURCE_ID).principal(PRINCIPAL)) + .andExpect(status().isAccepted()) + .andExpect(header().string("Cache-Control", "no-store")) + .andExpect(header().string("Location", "/api/etl/jobs/" + NEW_JOB_ID)) + .andExpect(header().string("Idempotency-Replayed", "false")) + .andExpect(jsonPath("$.jobRecordId").value(NEW_JOB_ID.toString())) + .andExpect(jsonPath("$.jobStatus").value("PENDING")) + .andExpect(jsonPath("$.statusUrl").value("/api/etl/jobs/" + NEW_JOB_ID)); + + verify(replayService).replayOwned( + SOURCE_ID, + PAYLOAD, + REPLAY_KEY, + "tenant_alpha" + ); + } + + @Test + void returnsTheCurrentStatusWhenAReplayRequestIsRepeatedLater() throws Exception { + when(replayService.replayOwned(SOURCE_ID, PAYLOAD, REPLAY_KEY, "tenant_alpha")) + .thenReturn(new EtlJobReplay(NEW_JOB_ID, EtlJobStatus.SUCCEEDED, true)); + + mockMvc.perform(request(SOURCE_ID).principal(PRINCIPAL)) + .andExpect(status().isAccepted()) + .andExpect(header().string("Idempotency-Replayed", "true")) + .andExpect(jsonPath("$.jobStatus").value("SUCCEEDED")); + } + + @Test + void rejectsAuthenticationKeyAndMalformedIdentifierBeforeServiceAccess() throws Exception { + mockMvc.perform(request(SOURCE_ID)) + .andExpect(status().isUnauthorized()) + .andExpect(jsonPath("$.errorCode").value( + "etl_idempotency_principal_required" + )); + mockMvc.perform(post("/api/etl/jobs/" + SOURCE_ID + "/replays") + .principal(PRINCIPAL) + .contentType(MediaType.APPLICATION_JSON) + .content(PAYLOAD)) + .andExpect(status().isBadRequest()) + .andExpect(jsonPath("$.errorCode").value("etl_job_replay_key_required")); + mockMvc.perform(post("/api/etl/jobs/not-a-uuid/replays") + .principal(PRINCIPAL) + .header("Idempotency-Key", REPLAY_KEY) + .contentType(MediaType.APPLICATION_JSON) + .content(PAYLOAD)) + .andExpect(status().isNotFound()) + .andExpect(jsonPath("$.errorCode").value("etl_job_not_found")); + + verifyNoInteractions(replayService); + } + + @Test + void preservesTypedFailuresAndSanitizesDatabaseAndUnexpectedFailures() throws Exception { + when(replayService.replayOwned(any(UUID.class), anyString(), anyString(), anyString())) + .thenThrow(new EtlRequestException(EtlRequestError.JOB_REPLAY_PAYLOAD_MISMATCH)) + .thenThrow(new DataAccessException("secret database detail") { }) + .thenThrow(new IllegalStateException("secret runtime detail")); + + mockMvc.perform(request(SOURCE_ID).principal(PRINCIPAL)) + .andExpect(status().isUnprocessableEntity()) + .andExpect(jsonPath("$.errorCode").value("etl_job_replay_payload_mismatch")); + mockMvc.perform(request(SOURCE_ID).principal(PRINCIPAL)) + .andExpect(status().isInternalServerError()) + .andExpect(jsonPath("$.errorCode").value("etl_target_failure")) + .andExpect(jsonPath("$.detail").value( + "The ETL target could not process the request." + )); + mockMvc.perform(request(SOURCE_ID).principal(PRINCIPAL)) + .andExpect(status().isInternalServerError()) + .andExpect(jsonPath("$.errorCode").value("etl_internal_error")) + .andExpect(jsonPath("$.detail").value( + "The ETL request could not be processed." + )); + } + + private static org.springframework.test.web.servlet.request.MockHttpServletRequestBuilder + request(UUID sourceId) { + return post("/api/etl/jobs/" + sourceId + "/replays") + .header("Idempotency-Key", REPLAY_KEY) + .contentType(MediaType.APPLICATION_JSON) + .content(PAYLOAD); + } +} From 48c1bd100a6b4a99ed5a2efd40c28cccd37dc18b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 14:30:18 +0900 Subject: [PATCH 012/103] test(etl): cover replay validation and transaction boundaries --- .../etl/job/EtlJobReplayBoundaryTest.java | 174 ++++++++++++++++++ 1 file changed, 174 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayBoundaryTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayBoundaryTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayBoundaryTest.java new file mode 100644 index 00000000..a13dd05d --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayBoundaryTest.java @@ -0,0 +1,174 @@ +package com.xtrmetl.etl.job; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.xtrmetl.etl.service.EtlBatchProperties; +import com.xtrmetl.etl.service.EtlRequestError; +import com.xtrmetl.etl.service.EtlRequestException; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.transaction.support.TransactionSynchronizationManager; + +import java.util.UUID; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verifyNoInteractions; + +/** + * Covers immutable replay-result construction and fail-closed admission before persistence. + */ +class EtlJobReplayBoundaryTest { + + private static final UUID SOURCE_ID = UUID.fromString( + "cf4f083f-8c90-4f34-a8b6-b53761de44ef" + ); + private static final String PAYLOAD = "[{\"id\":\"record_alpha\"}]"; + private static final String REPLAY_KEY = "1e05bdca-447c-4ad3-882c-e33963ce517c"; + + @AfterEach + void clearSyntheticTransactionState() { + TransactionSynchronizationManager.clear(); + } + + @Test + void replayResultRequiresIdentityAndStatusButPreservesCurrentState() { + EtlJobReplay pending = new EtlJobReplay(SOURCE_ID, EtlJobStatus.PENDING, false); + EtlJobReplay terminalReplay = new EtlJobReplay( + SOURCE_ID, + EtlJobStatus.SUCCEEDED, + true + ); + + assertEquals(EtlJobStatus.PENDING, pending.jobStatus()); + assertEquals(EtlJobStatus.SUCCEEDED, terminalReplay.jobStatus()); + assertThrows( + NullPointerException.class, + () -> new EtlJobReplay(null, EtlJobStatus.PENDING, false) + ); + assertThrows( + NullPointerException.class, + () -> new EtlJobReplay(SOURCE_ID, null, false) + ); + } + + @Test + void validatesIdentityKeyPrincipalAndPayloadBeforeDatabaseWork() { + JdbcTemplate jdbcTemplate = mock(JdbcTemplate.class); + EtlJobReplayService service = service(jdbcTemplate, lockHash -> true); + + assertThrows( + NullPointerException.class, + () -> service.replayOwned(null, PAYLOAD, REPLAY_KEY, "tenant_alpha") + ); + assertError( + EtlRequestError.JOB_REPLAY_KEY_REQUIRED, + () -> service.replayOwned(SOURCE_ID, PAYLOAD, null, "tenant_alpha") + ); + assertError( + EtlRequestError.JOB_REPLAY_KEY_REQUIRED, + () -> service.replayOwned(SOURCE_ID, PAYLOAD, "unsafe key", "tenant_alpha") + ); + assertError( + EtlRequestError.IDEMPOTENCY_PRINCIPAL_REQUIRED, + () -> service.replayOwned(SOURCE_ID, PAYLOAD, REPLAY_KEY, null) + ); + assertError( + EtlRequestError.IDEMPOTENCY_PRINCIPAL_REQUIRED, + () -> service.replayOwned(SOURCE_ID, PAYLOAD, REPLAY_KEY, " ".repeat(513)) + ); + assertError( + EtlRequestError.INVALID_JSON, + () -> service.replayOwned(SOURCE_ID, null, REPLAY_KEY, "tenant_alpha") + ); + assertError( + EtlRequestError.INVALID_JSON, + () -> service.replayOwned(SOURCE_ID, "not-json", REPLAY_KEY, "tenant_alpha") + ); + assertError( + EtlRequestError.INVALID_JSON, + () -> service.replayOwned(SOURCE_ID, "{}", REPLAY_KEY, "tenant_alpha") + ); + verifyNoInteractions(jdbcTemplate); + } + + @Test + void requiresAnActualTransactionBeforeLockOrTableAccess() { + JdbcTemplate jdbcTemplate = mock(JdbcTemplate.class); + EtlJobReplayService service = service(jdbcTemplate, lockHash -> true); + + IllegalStateException exception = assertThrows( + IllegalStateException.class, + () -> service.replayOwned( + SOURCE_ID, + PAYLOAD, + REPLAY_KEY, + "tenant_alpha" + ) + ); + + assertEquals("Durable ETL job replay requires an active transaction", exception.getMessage()); + verifyNoInteractions(jdbcTemplate); + } + + @Test + void reportsAnUnavailableReplayLockWithoutTableAccess() { + JdbcTemplate jdbcTemplate = mock(JdbcTemplate.class); + EtlJobReplayService service = service(jdbcTemplate, lockHash -> false); + TransactionSynchronizationManager.setActualTransactionActive(true); + + assertError( + EtlRequestError.JOB_REPLAY_IN_PROGRESS, + () -> service.replayOwned( + SOURCE_ID, + PAYLOAD, + REPLAY_KEY, + "tenant_alpha" + ) + ); + verifyNoInteractions(jdbcTemplate); + } + + @Test + void constructorsRejectMissingCollaborators() { + JdbcTemplate jdbcTemplate = mock(JdbcTemplate.class); + ObjectMapper mapper = new ObjectMapper(); + EtlBatchProperties properties = new EtlBatchProperties(); + + assertThrows( + NullPointerException.class, + () -> new EtlJobReplayService(null, mapper, properties, hash -> true) + ); + assertThrows( + NullPointerException.class, + () -> new EtlJobReplayService(jdbcTemplate, null, properties, hash -> true) + ); + assertThrows( + NullPointerException.class, + () -> new EtlJobReplayService(jdbcTemplate, mapper, null, hash -> true) + ); + assertThrows( + NullPointerException.class, + () -> new EtlJobReplayService(jdbcTemplate, mapper, properties, null) + ); + new EtlJobReplayService(jdbcTemplate, mapper, properties); + } + + private static EtlJobReplayService service( + JdbcTemplate jdbcTemplate, + com.xtrmetl.etl.service.EtlRequestLock requestLock + ) { + return new EtlJobReplayService( + jdbcTemplate, + new ObjectMapper(), + new EtlBatchProperties(), + requestLock + ); + } + + private static void assertError(EtlRequestError expected, Runnable invocation) { + EtlRequestException exception = assertThrows(EtlRequestException.class, invocation::run); + assertEquals(expected, exception.error()); + } +} From 2b07e4b9ef2bfeea31e800624eaf2ce0a8c298a5 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 14:31:14 +0900 Subject: [PATCH 013/103] test(etl): prove replay jobs enter ordinary worker lifecycle --- .../job/EtlJobReplayClaimIntegrationTest.java | 180 ++++++++++++++++++ 1 file changed, 180 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayClaimIntegrationTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayClaimIntegrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayClaimIntegrationTest.java new file mode 100644 index 00000000..b5fb617e --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayClaimIntegrationTest.java @@ -0,0 +1,180 @@ +package com.xtrmetl.etl.job; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.xtrmetl.etl.service.EtlBatchProperties; +import com.xtrmetl.etl.service.EtlRequestLock; +import com.xtrmetl.etl.service.Sha256Digest; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.jdbc.datasource.DataSourceTransactionManager; +import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseBuilder; +import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseType; +import org.springframework.test.context.junit.jupiter.SpringJUnitConfig; +import org.springframework.transaction.PlatformTransactionManager; +import org.springframework.transaction.annotation.EnableTransactionManagement; + +import javax.sql.DataSource; +import java.time.Duration; +import java.util.UUID; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +/** + * Proves replay admission creates an ordinary pending job that the existing worker can claim. + */ +@SpringJUnitConfig(EtlJobReplayClaimIntegrationTest.TestConfiguration.class) +class EtlJobReplayClaimIntegrationTest { + + private static final String PAYLOAD = "[{\"id\":\"record_alpha\"}]"; + private static final String REPLAY_KEY = "1e05bdca-447c-4ad3-882c-e33963ce517c"; + + private final EtlJobReplayService replayService; + private final EtlJobLeaseRepository leaseRepository; + private final JdbcTemplate jdbcTemplate; + + @Autowired + EtlJobReplayClaimIntegrationTest( + EtlJobReplayService replayService, + EtlJobLeaseRepository leaseRepository, + JdbcTemplate jdbcTemplate + ) { + this.replayService = replayService; + this.leaseRepository = leaseRepository; + this.jdbcTemplate = jdbcTemplate; + } + + @BeforeEach + void createJobTable() { + jdbcTemplate.execute("DROP TABLE IF EXISTS etl_job_records"); + jdbcTemplate.execute(""" + CREATE TABLE etl_job_records ( + job_record_id UUID PRIMARY KEY, + principal_scope_hash CHAR(64) NOT NULL, + submission_key_hash CHAR(64) NOT NULL, + request_digest CHAR(64) NOT NULL, + request_payload CLOB, + job_status VARCHAR(32) NOT NULL, + attempt_count INTEGER NOT NULL DEFAULT 0, + failure_code VARCHAR(128), + lease_claim_id UUID, + lease_owner_id VARCHAR(128), + lease_expires_at TIMESTAMP WITH TIME ZONE, + cancellation_key_hash CHAR(64), + cancellation_code VARCHAR(128), + job_cancelled_at TIMESTAMP WITH TIME ZONE, + replay_source_job_record_id UUID, + replay_root_job_record_id UUID, + replay_generation_count INTEGER, + created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT etl_job_submission_scope_unique + UNIQUE (principal_scope_hash, submission_key_hash) + ) + """); + } + + @Test + void replayCreatedPendingJobIsClaimedWithoutAReplaySpecificWorkerPath() { + UUID sourceId = UUID.fromString("c0b2860a-fd63-431a-96cb-48f3f4d7b19d"); + jdbcTemplate.update( + """ + INSERT INTO etl_job_records ( + job_record_id, principal_scope_hash, submission_key_hash, + request_digest, request_payload, job_status, attempt_count, + failure_code + ) VALUES (?, ?, ?, ?, NULL, 'FAILED', 1, 'etl_target_failure') + """, + sourceId, + Sha256Digest.digest("tenant_alpha"), + Sha256Digest.digest("source-key"), + Sha256Digest.digest(PAYLOAD) + ); + + EtlJobReplay replay = replayService.replayOwned( + sourceId, + PAYLOAD, + REPLAY_KEY, + "tenant_alpha" + ); + EtlJobLease lease = leaseRepository.claimNext( + "worker-alpha", + Duration.ofMinutes(5), + 3 + ).orElseThrow(); + + assertEquals(replay.jobRecordId(), lease.jobRecordId()); + assertEquals(PAYLOAD, lease.requestPayload()); + assertEquals(1, lease.attemptCount()); + assertEquals(sourceId, jdbcTemplate.queryForObject( + "SELECT replay_source_job_record_id FROM etl_job_records WHERE job_record_id=?", + UUID.class, + replay.jobRecordId() + )); + } + + /** Minimal transaction-enabled context for replay and worker claim integration. */ + @Configuration + @EnableTransactionManagement + static class TestConfiguration { + + @Bean + DataSource dataSource() { + return new EmbeddedDatabaseBuilder() + .generateUniqueName(true) + .setType(EmbeddedDatabaseType.H2) + .build(); + } + + @Bean + JdbcTemplate jdbcTemplate(DataSource dataSource) { + return new JdbcTemplate(dataSource); + } + + @Bean + PlatformTransactionManager transactionManager(DataSource dataSource) { + return new DataSourceTransactionManager(dataSource); + } + + @Bean + ObjectMapper objectMapper() { + return new ObjectMapper(); + } + + @Bean + EtlBatchProperties etlBatchProperties() { + return new EtlBatchProperties(); + } + + @Bean + EtlRequestLock etlRequestLock() { + return lockHash -> true; + } + + @Bean + EtlJobReplayService replayService( + JdbcTemplate jdbcTemplate, + ObjectMapper objectMapper, + EtlBatchProperties batchProperties, + EtlRequestLock requestLock + ) { + return new EtlJobReplayService( + jdbcTemplate, + objectMapper, + batchProperties, + requestLock + ); + } + + @Bean + EtlJobLeaseRepository leaseRepository( + JdbcTemplate jdbcTemplate, + PlatformTransactionManager transactionManager + ) { + return new EtlJobLeaseRepository(jdbcTemplate, transactionManager); + } + } +} From 35d7734b4bf9344ce0788a88d564fb41965c17c1 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 14:32:20 +0900 Subject: [PATCH 014/103] docs(etl): add immutable durable job replay runbook --- docs/operations/durable-job-replay.md | 191 ++++++++++++++++++++++++++ 1 file changed, 191 insertions(+) create mode 100644 docs/operations/durable-job-replay.md diff --git a/docs/operations/durable-job-replay.md b/docs/operations/durable-job-replay.md new file mode 100644 index 00000000..0482fd83 --- /dev/null +++ b/docs/operations/durable-job-replay.md @@ -0,0 +1,191 @@ +# Durable ETL job replay + +## Purpose + +`POST /api/etl/jobs/{source_job_record_id}/replays` creates a new ordinary durable job from an +immutable failed or cancelled source. The operator must resupply the complete bounded JSON payload. +mightyETL validates that payload through the ordinary intake contract and requires its SHA-256 digest +to equal the terminal source `request_digest` before any new row is inserted. + +Replay never changes a terminal source back to `PENDING`. + +## HTTP contract + +```http +POST /api/etl/jobs/cf4f083f-8c90-4f34-a8b6-b53761de44ef/replays HTTP/1.1 +Authorization: Basic +Idempotency-Key: "1e05bdca-447c-4ad3-882c-e33963ce517c" +Content-Type: application/json + +[{"id":"record_alpha"}] +``` + +A first replay returns: + +```http +HTTP/1.1 202 Accepted +Location: /api/etl/jobs/86e4d474-dabf-4d6a-9de4-4e8230589363 +Cache-Control: no-store +Idempotency-Replayed: false +Content-Type: application/json +``` + +The representation is the existing accepted-job model. A later identical retry returns the same new +job and its current lifecycle state with `Idempotency-Replayed: true`. `202 Accepted` is +noncommittal: it proves durable admission of the new job, not completion of its ETL effects. + +Malformed, missing, and foreign-owned source identifiers share `404 etl_job_not_found`. Active +sources return `409 etl_job_replay_source_active`; succeeded sources return +`409 etl_job_replay_source_succeeded`. A mismatching payload returns +`422 etl_job_replay_payload_mismatch`. Reusing one replay key with another source or payload returns +`422 etl_job_replay_key_reused`. + +## Source immutability + +The source row is selected under the authenticated principal and a row lock, but never updated. Its +status, failure or cancellation evidence, timestamps, digest, terminal null payload, and lineage +remain unchanged. Only `FAILED` and `CANCELLED` are eligible. + +The client must resupply exact JSON text because terminal jobs deliberately clear `request_payload`. +Whitespace or field-order changes produce another digest even when a parser would consider the JSON +semantically equivalent. This byte-exact rule mirrors durable submission and prevents operators from +silently changing the work while claiming to replay it. + +## Replay identity and concurrency + +The normalized key is stored only through this versioned principal-scoped identity: + +```text +SHA-256( + "mightyetl:durable-job-replay:v1:" + || principal_scope_hash + || ":" + || normalized_replay_key +) +``` + +The value occupies the replay-created row's existing `submission_key_hash` field. It cannot collide +with ordinary raw-key submission identities unless SHA-256 itself collides. The exact domain string +is persisted compatibility behavior and cannot change without a migration. + +A PostgreSQL transaction-level try-lock serializes one replay key within a principal namespace. A +concurrent request that cannot acquire it receives `409 etl_job_replay_in_progress`; retry after the +first transaction commits returns the created job. The existing +`etl_job_submission_scope_unique` constraint remains a second integrity boundary. + +## Immutable lineage + +`V7__add_etl_job_replay_lineage.sql` adds: + +```text +replay_source_job_record_id +replay_root_job_record_id +replay_generation_count +``` + +Root jobs have all fields null. Replay rows have all fields non-null, references different from their +own job identifier, and generation 1 through 100. Both self-referencing foreign keys use +`ON DELETE RESTRICT`; deleting a source or root cannot silently cascade through audit history. + +```mermaid +flowchart LR + R0[Root terminal job
generation null] -->|replay| R1[Replay job
generation 1] + R1 -->|later terminal + replay| R2[Replay job
generation 2] + R0 -. root reference .-> R2 +``` + +A replay of a root uses the source as root and generation 1. A replay of a replay inherits the root +and increments the immediate source generation. Generation 100 returns +`409 etl_job_replay_generation_exhausted` instead of creating generation 101. + +## Worker behavior + +The new row is an ordinary `PENDING` job with the verified payload. The existing PostgreSQL claim, +lease fencing, attempts, retry, success, failure, cancellation, pagination, `Retry-After`, and ETag +contracts apply unchanged. No replay-specific worker or scheduler exists. + +## Provenance export + +The relational rows are authoritative. A future owner-authorized JSON-LD export may represent: + +```text +source job → prov:Entity +replay action → prov:Activity +new job → prov:Entity +new job → prov:wasDerivedFrom → source job +replay action → prov:used → source job +new job → prov:wasGeneratedBy → replay action +``` + +PROV export never grants authority and never substitutes for owner predicates, database constraints, +or replay-key idempotency. + +## Rollout + +1. Rehearse V7 on a representative PostgreSQL 18 copy and inspect existing row count and lock time. +2. Verify exact-head cross-platform CI, full reactor tests, zero-missed configured coverage, + dependency review, SBOM, SAST, security scan, review threads, and independent approval. +3. Apply V7 before serving the replay route. +4. Smoke-test a disposable failed source, exact payload acceptance, same-key retry, and key conflict. +5. Confirm the source is unchanged and the new row has source/root/generation lineage. +6. Claim the new pending row through the ordinary worker and verify no replay-only execution path. +7. Monitor replay acceptance, in-progress conflicts, payload mismatches, generation exhaustion, + database lock waits, and failed foreign-key deletion attempts using fixed-cardinality signals. + +Logs and metric labels must not contain payloads, raw principals, raw keys, hashes, source/new job +identifiers, lineage identifiers, SQL, exception messages, or target identities. + +## Incident response + +### Payload mismatch + +Recover the exact source payload from the approved upstream evidence or encrypted audit archive. Do +not change the source digest, bypass verification, or reconstruct payload text from an operator's +memory. If the exact payload is unavailable, the job is not replayable through this endpoint. + +### Replay key conflict + +Read the already-created replay job associated with the operator's prior request. A key is one +principal-scoped replay intent and cannot be reused for another source or payload. Use a new key only +for a deliberately separate replay. + +### Broken lineage or missing root + +Stop replay admission. Preserve affected rows, deployed SHA, Flyway history, and backup evidence. +Do not null lineage fields to make constraints pass. Repair requires a reviewed migration based on +verified source/root ownership and generation. + +## Rollback + +Stop serving replay admission before rolling application binaries back. Older binaries ignore lineage +columns, but deletion or retention tooling might not understand the new `ON DELETE RESTRICT` +relationships. + +Do not drop V7 while replay rows exist. Archive or remove replay lineages from leaf to root under an +approved retention policy, preserving external audit evidence. Then a separately reviewed migration +may remove the constraints and columns. Never edit the applied V7 file or mutate terminal sources +back to pending. + +The replay-key domain must remain readable while any replay-created row can receive an idempotent +retry. A domain change requires a versioned migration or dual-read period, not a silent constant edit. + +## Connector limitation + +Matching the original payload proves replay fidelity, not external duplicate-effect safety. Enable +replay for a connector only when its target effects participate in the mightyETL transaction or the +connector provides independently tested idempotency or compensation. Succeeded jobs remain excluded +from this first slice. + +## References — APA 7th + +Fielding, R., Nottingham, M., & Reschke, J. (2022). *HTTP semantics* (RFC 9110). RFC Editor. +https://www.rfc-editor.org/rfc/rfc9110 + +Nottingham, M., Wilde, E., & Dalal, S. (2023). *Problem details for HTTP APIs* (RFC 9457). RFC Editor. +https://www.rfc-editor.org/rfc/rfc9457 + +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: INSERT*. +https://www.postgresql.org/docs/18/sql-insert.html + +World Wide Web Consortium. (2013). *PROV-O: The PROV ontology*. +https://www.w3.org/TR/prov-o/ From f5ac3e35cb143c464a72b19ca7cb56c28b3c9f4d Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 14:33:12 +0900 Subject: [PATCH 015/103] docs(etl): record durable replay identity domain separation --- ...urable-job-replay-key-domain-separation.md | 83 +++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100644 docs/doctoring/durable-job-replay-key-domain-separation.md diff --git a/docs/doctoring/durable-job-replay-key-domain-separation.md b/docs/doctoring/durable-job-replay-key-domain-separation.md new file mode 100644 index 00000000..17579fc8 --- /dev/null +++ b/docs/doctoring/durable-job-replay-key-domain-separation.md @@ -0,0 +1,83 @@ +# Durable-job replay-key domain separation + +## Decision + +Replay-created jobs store one versioned principal-scoped replay identity in +`submission_key_hash`: + +```text +SHA-256( + "mightyetl:durable-job-replay:v1:" + || principal_scope_hash + || ":" + || normalized_replay_key +) +``` + +The same identity feeds a separately versioned transaction-lock input. Replay identity is isolated +from ordinary submission-key hashing and from another authenticated principal. It proves only +idempotent admission; every source lookup and created-job query independently binds the owner hash. + +## Threat addressed + +Using the ordinary raw-key digest would allow one client key to collide across the submit and replay +APIs. It would also make a replay key's stored equality directly comparable between principal +namespaces. The versioned operation domain and owner hash prevent those cross-protocol and +cross-tenant equality channels. + +Within one principal namespace, the same replay key intentionally identifies only one replay intent. +An existing row with another immediate source or request digest returns +`etl_job_replay_key_reused` rather than creating a second row. + +## Compatibility boundary + +These exact strings are persisted behavior: + +```text +mightyetl:durable-job-replay:v1: +mightyetl:durable-job-replay-lock:v1: +``` + +Changing the replay domain would make existing requests stop replaying their created jobs. Changing +the lock domain could let old and new binaries concurrently use different locks for the same +identity. Either change requires an explicit migration and mixed-version deployment analysis. + +The implementation does not claim cSHAKE, KMAC, or TupleHash conformance. It uses the existing +SHA-256 utility with fixed-width principal hash, explicit separators, and one final bounded key. +NIST SP 800-185 is methodological evidence for customization and domain separation, not an +implementation-conformance claim. + +## Test evidence + +Service integration requires: + +- quoted and legacy-raw representations of one key replay the same created job; +- one replay key with another source or payload fails closed; +- an ordinary submission key cannot accidentally identify a replay-created row because the replay + domain changes the digest input; +- an unavailable transaction lock returns `etl_job_replay_in_progress` before table access. + +The table's `etl_job_submission_scope_unique` constraint remains a second integrity boundary after +the transaction lock. + +## Privacy + +Raw replay keys and replay hashes are excluded from HTTP bodies, headers, RFC 9457 problems, ordinary +logs, metrics, status and list resources, lineage exports, and worker leases. The stored hash remains +pseudonymous internal security data and must not be published merely because it is one-way. + +## Rollback + +Keep both versioned derivations available while replay-created rows can receive retries or while old +and new binaries overlap. Do not derive candidate hashes from logged user input during diagnosis. +Never rewrite a replay-created `submission_key_hash` without preserving unique-constraint and exact +idempotency evidence. + +## References — APA 7th + +Kelsey, J., Chang, S., & Perlner, R. (2016). *SHA-3 derived functions: cSHAKE, KMAC, TupleHash, and +ParallelHash* (NIST Special Publication 800-185). National Institute of Standards and Technology. +https://doi.org/10.6028/NIST.SP.800-185 + +National Institute of Standards and Technology. (2025, March 12). *Decision to update FIPS 202 and +revise SP 800-185*. https://csrc.nist.gov/news/2025/decision-to-update-fips-202-and-revise-sp-800-185 From bf6b64539f6bbdbeff2f5612afa346d4ae9d5c10 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 14:35:30 +0900 Subject: [PATCH 016/103] test(etl): cover replay payload admission limits --- .../job/EtlJobReplayPayloadBoundaryTest.java | 97 +++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayPayloadBoundaryTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayPayloadBoundaryTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayPayloadBoundaryTest.java new file mode 100644 index 00000000..944416d1 --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayPayloadBoundaryTest.java @@ -0,0 +1,97 @@ +package com.xtrmetl.etl.job; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.xtrmetl.etl.service.EtlBatchProperties; +import com.xtrmetl.etl.service.EtlRequestError; +import com.xtrmetl.etl.service.EtlRequestException; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.transaction.support.TransactionSynchronizationManager; + +import java.util.UUID; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verifyNoInteractions; + +/** + * Covers replay payload byte, record-count, and ordinary record-contract rejection before JDBC. + */ +class EtlJobReplayPayloadBoundaryTest { + + private static final UUID SOURCE_ID = UUID.fromString( + "cf4f083f-8c90-4f34-a8b6-b53761de44ef" + ); + private static final String REPLAY_KEY = "1e05bdca-447c-4ad3-882c-e33963ce517c"; + + @AfterEach + void clearSyntheticTransactionState() { + TransactionSynchronizationManager.clear(); + } + + @Test + void rejectsOversizedPayloadBeforeLockOrTableAccess() { + JdbcTemplate jdbcTemplate = mock(JdbcTemplate.class); + EtlBatchProperties properties = new EtlBatchProperties(); + properties.setMaxPayloadBytes(8); + EtlJobReplayService service = service(jdbcTemplate, properties); + + assertError( + EtlRequestError.PAYLOAD_TOO_LARGE, + () -> service.replayOwned( + SOURCE_ID, + "[{\"id\":\"record_alpha\"}]", + REPLAY_KEY, + "tenant_alpha" + ) + ); + verifyNoInteractions(jdbcTemplate); + } + + @Test + void rejectsOversizedBatchAndInvalidRecordsBeforeLockOrTableAccess() { + JdbcTemplate jdbcTemplate = mock(JdbcTemplate.class); + EtlBatchProperties properties = new EtlBatchProperties(); + properties.setMaxBatchRecords(1); + EtlJobReplayService service = service(jdbcTemplate, properties); + + assertError( + EtlRequestError.BATCH_TOO_LARGE, + () -> service.replayOwned( + SOURCE_ID, + "[{\"id\":\"a\"},{\"id\":\"b\"}]", + REPLAY_KEY, + "tenant_alpha" + ) + ); + assertError( + EtlRequestError.INVALID_RECORD, + () -> service.replayOwned( + SOURCE_ID, + "[{}]", + REPLAY_KEY, + "tenant_alpha" + ) + ); + verifyNoInteractions(jdbcTemplate); + } + + private static EtlJobReplayService service( + JdbcTemplate jdbcTemplate, + EtlBatchProperties properties + ) { + return new EtlJobReplayService( + jdbcTemplate, + new ObjectMapper(), + properties, + lockHash -> true + ); + } + + private static void assertError(EtlRequestError expected, Runnable invocation) { + EtlRequestException exception = assertThrows(EtlRequestException.class, invocation::run); + assertEquals(expected, exception.error()); + } +} From 821b7ae2f9d4427d10d86e3729828659e5830bc9 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 14:36:27 +0900 Subject: [PATCH 017/103] test(etl): require immutable replay operations evidence --- .../DurableJobReplayDocumentationTest.java | 97 +++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java new file mode 100644 index 00000000..1cc935eb --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java @@ -0,0 +1,97 @@ +package com.xtrmetl.etl.documentation; + +import org.junit.jupiter.api.Test; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; + +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Keeps replay source immutability, payload proof, lineage, compatibility, and operations aligned. + */ +class DurableJobReplayDocumentationTest { + + @Test + void operationsRunbookDocumentsAdmissionLineageAndRollback() throws IOException { + String runbook = read("docs/operations/durable-job-replay.md") + .replaceAll("\\s+", " "); + + assertTrue(runbook.contains("POST /api/etl/jobs/{source_job_record_id}/replays")); + assertTrue(runbook.contains("Replay never changes a terminal source back to `PENDING`")); + assertTrue(runbook.contains("Idempotency-Replayed: false")); + assertTrue(runbook.contains("Idempotency-Replayed: true")); + assertTrue(runbook.contains("etl_job_replay_payload_mismatch")); + assertTrue(runbook.contains("etl_job_replay_key_reused")); + assertTrue(runbook.contains("replay_source_job_record_id")); + assertTrue(runbook.contains("replay_root_job_record_id")); + assertTrue(runbook.contains("replay_generation_count")); + assertTrue(runbook.contains("ON DELETE RESTRICT")); + assertTrue(runbook.contains("generation 1 through 100")); + assertTrue(runbook.contains("prov:wasDerivedFrom")); + assertTrue(runbook.contains("Do not drop V7 while replay rows exist")); + assertTrue(runbook.contains("does not prove that replaying a connector")); + assertTrue(runbook.contains("RFC 9110")); + assertTrue(runbook.contains("RFC 9457")); + assertTrue(runbook.contains("PROV-O")); + } + + @Test + void designAndPlanPreserveTheSingleExecutionEngine() throws IOException { + String design = read( + "docs/superpowers/specs/2026-08-06-durable-job-replay-design.md" + ).replaceAll("\\s+", " "); + String plan = read( + "docs/superpowers/plans/2026-08-06-durable-job-replay.md" + ).replaceAll("\\s+", " "); + + assertTrue(design.contains("The source is never updated")); + assertTrue(design.contains("Only `FAILED` and `CANCELLED`")); + assertTrue(design.contains("ordinary `PENDING` job")); + assertTrue(design.contains("No replay-specific worker or scheduler exists")); + assertTrue(design.contains("replay_generation_count")); + assertTrue(plan.contains("Never update a terminal source back to `PENDING`")); + assertTrue(plan.contains("Run all verification")); + assertTrue(plan.contains("no project test is skipped")); + } + + @Test + void doctoringPinsVersionedReplayAndLockDomains() throws IOException { + String evidence = read( + "docs/doctoring/durable-job-replay-key-domain-separation.md" + ).replaceAll("\\s+", " "); + + assertTrue(evidence.contains("mightyetl:durable-job-replay:v1:")); + assertTrue(evidence.contains("mightyetl:durable-job-replay-lock:v1:")); + assertTrue(evidence.contains("isolated from ordinary submission-key hashing")); + assertTrue(evidence.contains("exact strings are persisted behavior")); + assertTrue(evidence.contains("does not claim cSHAKE")); + assertTrue(evidence.contains("NIST Special Publication 800-185")); + } + + private static String read(String relativePath) throws IOException { + return Files.readString(projectRoot().resolve(relativePath), StandardCharsets.UTF_8); + } + + /** @return repository root from reactor-root or module-local execution */ + private static Path projectRoot() { + Path current = Paths.get(System.getProperty("user.dir")).toAbsolutePath(); + Path lastPomParent = null; + while (current != null) { + if (Files.exists(current.resolve(".git"))) { + return current; + } + if (Files.exists(current.resolve("pom.xml"))) { + lastPomParent = current; + } + current = current.getParent(); + } + if (lastPomParent != null) { + return lastPomParent; + } + throw new IllegalStateException("Could not find project root"); + } +} From 82d283d280f327f8cb6a57402e01aa9f4246815a Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 15:05:08 +0900 Subject: [PATCH 018/103] fix(etl): scope problem handling to replay endpoint --- .../com/xtrmetl/etl/controller/EtlApiProblemHandler.java | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/etl-service/src/main/java/com/xtrmetl/etl/controller/EtlApiProblemHandler.java b/etl-service/src/main/java/com/xtrmetl/etl/controller/EtlApiProblemHandler.java index 6ab4ff3e..cef3f117 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/controller/EtlApiProblemHandler.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/controller/EtlApiProblemHandler.java @@ -30,7 +30,11 @@ * captured by a broad exception handler and therefore retain their framework-owned status * semantics.

*/ -@RestControllerAdvice(assignableTypes = {EtlController.class, EtlJobController.class}) +@RestControllerAdvice(assignableTypes = { + EtlController.class, + EtlJobController.class, + EtlJobReplayController.class +}) public class EtlApiProblemHandler { private static final Logger log = LoggerFactory.getLogger(EtlApiProblemHandler.class); From a8dedeb82ebef64d244b63cefb89be57b13c6b33 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 15:06:07 +0900 Subject: [PATCH 019/103] docs(etl): state replay connector safety limitation explicitly --- docs/operations/durable-job-replay.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/operations/durable-job-replay.md b/docs/operations/durable-job-replay.md index 0482fd83..81cfe2eb 100644 --- a/docs/operations/durable-job-replay.md +++ b/docs/operations/durable-job-replay.md @@ -171,7 +171,7 @@ retry. A domain change requires a versioned migration or dual-read period, not a ## Connector limitation -Matching the original payload proves replay fidelity, not external duplicate-effect safety. Enable +Matching the original payload does not prove that replaying a connector is externally safe. Enable replay for a connector only when its target effects participate in the mightyETL transaction or the connector provides independently tested idempotency or compensation. Succeeded jobs remain excluded from this first slice. From 6221b84b3078697b680db3b5d68652a2e26766c6 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 15:06:55 +0900 Subject: [PATCH 020/103] docs(etl): align replay design with executable contracts --- .../superpowers/specs/2026-08-06-durable-job-replay-design.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md b/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md index 70d7e85e..5fb93ccd 100644 --- a/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md +++ b/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md @@ -19,7 +19,7 @@ Content-Type: application/json A first accepted replay returns RFC 9110 `202 Accepted`, `Location` for the new job, `Cache-Control: no-store`, and `Idempotency-Replayed: false`. The same principal, replay key, source job, and byte-identical payload returns the same new job with `Idempotency-Replayed: true`. -The source remains terminal and unchanged. Replay is allowed only from `FAILED` and `CANCELLED`. Active sources conflict because they still own or may own execution. `SUCCEEDED` conflicts because a first-slice replay could duplicate committed target effects. +The source remains terminal and unchanged. Only `FAILED` and `CANCELLED` are replayable. Active sources conflict because they still own or may own execution. `SUCCEEDED` conflicts because a first-slice replay could duplicate committed target effects. ## Immutable relational lineage @@ -102,7 +102,7 @@ All failures use the existing RFC 9457 problem model without payload, principal, ## Worker compatibility -The new row is an ordinary `PENDING` job. Existing worker claim, lease fencing, retry, success, failure, cancellation, pagination, polling, and conditional-status contracts apply without a replay-specific execution path. Only lineage and admission differ. +The new row is an ordinary `PENDING` job. Existing worker claim, lease fencing, retry, success, failure, cancellation, pagination, polling, and conditional-status contracts apply unchanged. No replay-specific worker or scheduler exists. Only lineage and admission differ. ## Provenance export From 51b0d1371889df9456b44f14e204bbb98412c896 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 15:07:52 +0900 Subject: [PATCH 021/103] test(docs): require durable replay changelog evidence --- .../DurableJobReplayDocumentationTest.java | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java index 1cc935eb..b087b82c 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java @@ -58,6 +58,19 @@ void designAndPlanPreserveTheSingleExecutionEngine() throws IOException { assertTrue(plan.contains("no project test is skipped")); } + @Test + void changelogRecordsReplayAdmissionLineageAndSafety() throws IOException { + String changelog = read("CHANGELOG.md").replaceAll("\\s+", " "); + + assertTrue(changelog.contains("immutable failed or cancelled source")); + assertTrue(changelog.contains("byte-identical bounded JSON payload")); + assertTrue(changelog.contains("replay_source_job_record_id")); + assertTrue(changelog.contains("replay_root_job_record_id")); + assertTrue(changelog.contains("replay_generation_count")); + assertTrue(changelog.contains("V7__add_etl_job_replay_lineage.sql")); + assertTrue(changelog.contains("does not prove external connector safety")); + } + @Test void doctoringPinsVersionedReplayAndLockDomains() throws IOException { String evidence = read( From c7900f7bfedaca5a48d9ebb2804a44d945da7208 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 15:09:43 +0900 Subject: [PATCH 022/103] docs(changelog): record durable job replay contract --- CHANGELOG.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0f087e67..a4706113 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- Authenticated operators can now create an ordinary pending durable job from an immutable failed or cancelled source only after resupplying a byte-identical bounded JSON payload; the terminal source remains unchanged and succeeded sources remain non-replayable. - Authenticated operators can now perform owner-scoped durable-job cancellation for `PENDING` and `RUNNING` work through an idempotent action that commits terminal `CANCELLED`, clears payload and lease state, stores only `cancellation_key_hash` plus a fixed code and timestamp, and returns stable RFC 9457 conflicts when success or failure already won. - Cancellation-first races now make the former exact lease stale and roll back transactional target and response-ledger effects; success-first races remain `SUCCEEDED`, while same-key cancellation replays and different-key reuse fails closed. - Owner-scoped durable-job status responses now emit deterministic weak SHA-256 `ETag` validators; ordinary and wildcard `If-None-Match` requests return an empty RFC 9110 `304 Not Modified` response only after authenticated owner-safe lookup, while `Cache-Control: no-store` remains unchanged. @@ -36,6 +37,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- Transactional migration `V7__add_etl_job_replay_lineage.sql`, owner-scoped replay admission, and immutable `replay_source_job_record_id`, `replay_root_job_record_id`, and `replay_generation_count` evidence with concurrency, lineage, worker-compatibility, rollout, incident, and rollback tests and documentation. - Transactional migration `V6__add_etl_job_cancellation.sql`, owner-safe cancellation API and replay model, exact lease-invalidation and cancellation-versus-success integration tests, plus rollout, incident, connector-limitation, and rollback evidence in `docs/operations/durable-job-cancellation.md`. - Deterministic ordinary, wildcard, changed-state, changed-failure-code, null-versus-empty, and unrelated-response conditional polling tests, complete controller Javadoc, privacy and rollback guidance, and APA 7th standards evidence in `docs/etl/durable-job-polling.md`. - Controller-scoped polling advice, deterministic active/terminal lifecycle tests, disabled-worker fail-closed behavior, sub-second rounding coverage, rollback guidance, and APA 7th standards evidence in `docs/etl/durable-job-polling.md`. @@ -76,6 +78,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Security +- Replay uses owner-scoped source selection, byte-exact digest verification, a versioned principal-scoped key domain, and immutable relational lineage without retaining raw principals or replay keys; this evidence does not prove external connector safety, so connector-native idempotency, transaction participation, or compensation remains required. - Cancellation stores only a principal-scoped SHA-256 replay identity and a fixed machine code, exposes no raw principal, key, hash, payload, lease, SQL, exception, or target detail, and requires an owner-matched conditional database update before reporting success. - Conditional status validators are SHA-256 digests of only the complete owner-authorized operator-safe representation; payloads, raw principals, idempotency keys, internal hashes, leases, SQL, and exception text remain excluded, and wildcard evaluation occurs only after owner-safe lookup. - Polling advice exposes only a bounded delay integer and is omitted when local execution is disabled or terminal; it never contains job, lease, principal, key, hash, payload, SQL, exception, target, or queue-depth data. From 9a8a1fb54afb88b1a32ab8ac44472860ef926547 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 15:15:22 +0900 Subject: [PATCH 023/103] docs(etl): make replay verification plan executable --- docs/superpowers/plans/2026-08-06-durable-job-replay.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-08-06-durable-job-replay.md b/docs/superpowers/plans/2026-08-06-durable-job-replay.md index 3e48bfd7..da517d57 100644 --- a/docs/superpowers/plans/2026-08-06-durable-job-replay.md +++ b/docs/superpowers/plans/2026-08-06-durable-job-replay.md @@ -105,7 +105,7 @@ EtlJobReplay replayOwned( - [ ] Require source immutability, payload digest proof, lineage, concurrency, generation limit, connector limitation, rollout, and rollback documentation first. - [ ] Document W3C PROV mapping as an export contract, not a database-authority substitute. - [ ] Record APA 7th primary references and the versioned replay-key compatibility boundary. -- [ ] Run `./mvnw -B test`, configured coverage gates, and `git diff --check` through exact-head CI. +- [ ] Run all verification through exact-head CI: `./mvnw -B test`, configured coverage gates, and `git diff --check`. - [ ] Keep the PR draft until every stacked-target gate succeeds. ## Plan self-review @@ -115,3 +115,4 @@ EtlJobReplay replayOwned( - Replay-key and payload conflicts are distinguished without disclosing source existence across principals. - New jobs enter the existing worker lifecycle rather than creating a second execution engine. - No placeholder, ambiguous public signature, or unbounded database object name remains. +- Verification requires that no project test is skipped. From b5d192886c182615085c0b329df29fd46116858b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 15:22:01 +0900 Subject: [PATCH 024/103] test(etl): cover replay input decision boundaries --- .../xtrmetl/etl/job/EtlJobReplayBoundaryTest.java | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayBoundaryTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayBoundaryTest.java index a13dd05d..c0c20880 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayBoundaryTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayBoundaryTest.java @@ -78,10 +78,22 @@ void validatesIdentityKeyPrincipalAndPayloadBeforeDatabaseWork() { EtlRequestError.IDEMPOTENCY_PRINCIPAL_REQUIRED, () -> service.replayOwned(SOURCE_ID, PAYLOAD, REPLAY_KEY, " ".repeat(513)) ); + assertError( + EtlRequestError.IDEMPOTENCY_PRINCIPAL_REQUIRED, + () -> service.replayOwned(SOURCE_ID, PAYLOAD, REPLAY_KEY, "a".repeat(513)) + ); assertError( EtlRequestError.INVALID_JSON, () -> service.replayOwned(SOURCE_ID, null, REPLAY_KEY, "tenant_alpha") ); + assertError( + EtlRequestError.INVALID_JSON, + () -> service.replayOwned(SOURCE_ID, "", REPLAY_KEY, "tenant_alpha") + ); + assertError( + EtlRequestError.INVALID_JSON, + () -> service.replayOwned(SOURCE_ID, "null", REPLAY_KEY, "tenant_alpha") + ); assertError( EtlRequestError.INVALID_JSON, () -> service.replayOwned(SOURCE_ID, "not-json", REPLAY_KEY, "tenant_alpha") From 13ba58c6e2f7c28ad8768fd2e104a0a6be62b42d Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 15:23:13 +0900 Subject: [PATCH 025/103] test(etl): cover replay key and lineage integrity branches --- .../EtlJobReplayServiceIntegrationTest.java | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayServiceIntegrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayServiceIntegrationTest.java index adc8ac1d..6a30c1c8 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayServiceIntegrationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayServiceIntegrationTest.java @@ -161,6 +161,16 @@ void rejectsMismatchedPayloadAndReplayKeyReuse() { REPLAY_KEY, "tenant_alpha" ); + assertError( + EtlRequestError.JOB_REPLAY_KEY_REUSED, + () -> replayService.replayOwned( + sourceId, + OTHER_PAYLOAD, + REPLAY_KEY, + "tenant_alpha" + ) + ); + UUID otherSource = insertTerminalSource( EtlJobStatus.FAILED, "tenant_alpha", @@ -274,6 +284,46 @@ void replayOfReplayPreservesRootAndBoundsGeneration() { ); } + @Test + void rejectsIncompleteOrForeignRootLineage() { + UUID incompleteRoot = insertSource( + EtlJobStatus.FAILED, + "tenant_alpha", + PAYLOAD, + null, + 1 + ); + IllegalStateException incomplete = assertThrows( + IllegalStateException.class, + () -> replayService.replayOwned( + incompleteRoot, + PAYLOAD, + REPLAY_KEY, + "tenant_alpha" + ) + ); + assertEquals("Replay source has incomplete root lineage", incomplete.getMessage()); + + UUID absentRoot = insertSource( + EtlJobStatus.FAILED, + "tenant_alpha", + PAYLOAD, + UUID.randomUUID(), + 1 + ); + IllegalStateException absent = assertThrows( + IllegalStateException.class, + () -> replayService.replayOwned( + absentRoot, + PAYLOAD, + OTHER_REPLAY_KEY, + "tenant_alpha" + ) + ); + assertEquals("Replay root is absent from the owner namespace", absent.getMessage()); + assertEquals(0, replayRowCount()); + } + private UUID insertTerminalSource( EtlJobStatus status, String principal, From 59313b716e39ebd2fe50b4e94a94d2d6ca22e76b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 15:25:50 +0900 Subject: [PATCH 026/103] test(etl): count only created pending replay rows --- .../etl/job/EtlJobReplayServiceIntegrationTest.java | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayServiceIntegrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayServiceIntegrationTest.java index 6a30c1c8..010898ab 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayServiceIntegrationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayServiceIntegrationTest.java @@ -321,7 +321,15 @@ void rejectsIncompleteOrForeignRootLineage() { ) ); assertEquals("Replay root is absent from the owner namespace", absent.getMessage()); - assertEquals(0, replayRowCount()); + assertEquals( + 0, + jdbcTemplate.queryForObject( + "SELECT COUNT(*) FROM etl_job_records " + + "WHERE job_status='PENDING' " + + "AND replay_generation_count IS NOT NULL", + Integer.class + ) + ); } private UUID insertTerminalSource( From 85a269e558c6f060701b369d48dcdfc8d4035b87 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 15:28:25 +0900 Subject: [PATCH 027/103] test(etl): cover empty replay batch validation path --- .../etl/job/EtlJobReplayBoundaryTest.java | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayBoundaryTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayBoundaryTest.java index c0c20880..6fdc6f36 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayBoundaryTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayBoundaryTest.java @@ -110,7 +110,7 @@ void requiresAnActualTransactionBeforeLockOrTableAccess() { JdbcTemplate jdbcTemplate = mock(JdbcTemplate.class); EtlJobReplayService service = service(jdbcTemplate, lockHash -> true); - IllegalStateException exception = assertThrows( + IllegalStateException nonEmptyBatch = assertThrows( IllegalStateException.class, () -> service.replayOwned( SOURCE_ID, @@ -119,8 +119,21 @@ void requiresAnActualTransactionBeforeLockOrTableAccess() { "tenant_alpha" ) ); + IllegalStateException emptyBatch = assertThrows( + IllegalStateException.class, + () -> service.replayOwned( + SOURCE_ID, + "[]", + REPLAY_KEY, + "tenant_alpha" + ) + ); - assertEquals("Durable ETL job replay requires an active transaction", exception.getMessage()); + assertEquals( + "Durable ETL job replay requires an active transaction", + nonEmptyBatch.getMessage() + ); + assertEquals(nonEmptyBatch.getMessage(), emptyBatch.getMessage()); verifyNoInteractions(jdbcTemplate); } From d287877f39284ec8990c4723a137926d13b58219 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 15:31:43 +0900 Subject: [PATCH 028/103] test(ci): require replay coverage diagnostics --- .../etl/documentation/CiCoverageDiagnosticsWorkflowTest.java | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/documentation/CiCoverageDiagnosticsWorkflowTest.java b/etl-service/src/test/java/com/xtrmetl/etl/documentation/CiCoverageDiagnosticsWorkflowTest.java index 67148b3d..5de68f14 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/documentation/CiCoverageDiagnosticsWorkflowTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/documentation/CiCoverageDiagnosticsWorkflowTest.java @@ -44,6 +44,10 @@ void diagnosesEveryStrictCoverageTarget() { assertTrue(workflow.contains( "\"com/xtrmetl/etl/job/EtlJobService\": \"EtlJobService.java\"" )); + assertTrue(workflow.contains( + "\"com/xtrmetl/etl/job/EtlJobReplayService\": " + + "\"EtlJobReplayService.java\"" + )); assertTrue(workflow.contains( "\"com/xtrmetl/etl/controller/EtlJobController\": " + "\"EtlJobController.java\"" From 3acb00ac13d119cf1b543ec6d76374e90f7d54b1 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 15:32:15 +0900 Subject: [PATCH 029/103] ci: diagnose durable replay coverage gaps --- .github/workflows/ci.yml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e71e0b9d..1111f30f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -44,7 +44,7 @@ jobs: - name: Run tests (Windows) if: runner.os == 'Windows' - run: .\\mvnw.cmd -B test + run: .\mvnw.cmd -B test - name: Report uncovered JaCoCo branches if: failure() @@ -60,6 +60,7 @@ jobs: coverage_targets = { "com/xtrmetl/etl/job/EtlJobService": "EtlJobService.java", + "com/xtrmetl/etl/job/EtlJobReplayService": "EtlJobReplayService.java", "com/xtrmetl/etl/controller/EtlJobController": "EtlJobController.java", "com/xtrmetl/etl/service/Sha256Digest": "Sha256Digest.java", } @@ -135,4 +136,4 @@ jobs: - name: Run tests (Windows) if: runner.os == 'Windows' - run: .\\mvnw.cmd -B test + run: .\mvnw.cmd -B test From 954103bc95c255b0508dd73eb7295407f7fb01d9 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 15:34:44 +0900 Subject: [PATCH 030/103] test(etl): cover absent parsed replay root --- .../etl/job/EtlJobReplayBoundaryTest.java | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayBoundaryTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayBoundaryTest.java index 6fdc6f36..6985c256 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayBoundaryTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayBoundaryTest.java @@ -1,5 +1,6 @@ package com.xtrmetl.etl.job; +import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.ObjectMapper; import com.xtrmetl.etl.service.EtlBatchProperties; import com.xtrmetl.etl.service.EtlRequestError; @@ -105,6 +106,34 @@ void validatesIdentityKeyPrincipalAndPayloadBeforeDatabaseWork() { verifyNoInteractions(jdbcTemplate); } + @Test + void rejectsAnAbsentParsedRootBeforeDatabaseWork() { + JdbcTemplate jdbcTemplate = mock(JdbcTemplate.class); + ObjectMapper absentRootMapper = new ObjectMapper() { + @Override + public ObjectMapper copy() { + return this; + } + + @Override + public JsonNode readTree(String content) { + return null; + } + }; + EtlJobReplayService service = new EtlJobReplayService( + jdbcTemplate, + absentRootMapper, + new EtlBatchProperties(), + lockHash -> true + ); + + assertError( + EtlRequestError.INVALID_JSON, + () -> service.replayOwned(SOURCE_ID, "[]", REPLAY_KEY, "tenant_alpha") + ); + verifyNoInteractions(jdbcTemplate); + } + @Test void requiresAnActualTransactionBeforeLockOrTableAccess() { JdbcTemplate jdbcTemplate = mock(JdbcTemplate.class); From 42da7dfa85e671940dd7d07945b43aa0ec69f966 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 08:35:18 +0900 Subject: [PATCH 031/103] test(etl): require owner-scoped replay lineage foreign keys --- .../etl/job/EtlJobReplayMigrationTest.java | 24 +++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java index d27cd82b..6f170958 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java @@ -12,12 +12,12 @@ import static org.junit.jupiter.api.Assertions.assertTrue; /** - * Guards the complete, bounded, non-cascading durable job replay lineage schema. + * Guards the complete, bounded, owner-scoped, non-cascading durable job replay lineage schema. */ class EtlJobReplayMigrationTest { @Test - void replayMigrationAddsCompleteRestrictedSelfReferencingLineage() throws IOException { + void replayMigrationAddsCompleteRestrictedOwnerScopedLineage() throws IOException { String migration = Files.readString( projectRoot().resolve( "etl-service/src/main/resources/db/migration/" @@ -29,13 +29,21 @@ void replayMigrationAddsCompleteRestrictedSelfReferencingLineage() throws IOExce assertTrue(migration.contains("ADD COLUMN replay_source_job_record_id UUID")); assertTrue(migration.contains("ADD COLUMN replay_root_job_record_id UUID")); assertTrue(migration.contains("ADD COLUMN replay_generation_count INTEGER")); + assertTrue(migration.contains("CONSTRAINT etl_job_owner_identity_unique")); + assertTrue(migration.contains( + "UNIQUE (job_record_id, principal_scope_hash)" + )); assertTrue(migration.contains("CONSTRAINT etl_job_replay_source_reference")); assertTrue(migration.contains( - "FOREIGN KEY (replay_source_job_record_id) REFERENCES etl_job_records (job_record_id) ON DELETE RESTRICT" + "FOREIGN KEY (replay_source_job_record_id, principal_scope_hash) " + + "REFERENCES etl_job_records (job_record_id, principal_scope_hash) " + + "ON DELETE RESTRICT" )); assertTrue(migration.contains("CONSTRAINT etl_job_replay_root_reference")); assertTrue(migration.contains( - "FOREIGN KEY (replay_root_job_record_id) REFERENCES etl_job_records (job_record_id) ON DELETE RESTRICT" + "FOREIGN KEY (replay_root_job_record_id, principal_scope_hash) " + + "REFERENCES etl_job_records (job_record_id, principal_scope_hash) " + + "ON DELETE RESTRICT" )); assertTrue(migration.contains("CONSTRAINT etl_job_replay_lineage_complete_check")); assertTrue(migration.contains("replay_source_job_record_id IS NULL")); @@ -46,6 +54,14 @@ void replayMigrationAddsCompleteRestrictedSelfReferencingLineage() throws IOExce assertTrue(migration.contains("replay_generation_count BETWEEN 1 AND 100")); assertTrue(migration.contains("replay_source_job_record_id <> job_record_id")); assertTrue(migration.contains("replay_root_job_record_id <> job_record_id")); + assertFalse(migration.contains( + "FOREIGN KEY (replay_source_job_record_id) " + + "REFERENCES etl_job_records (job_record_id)" + )); + assertFalse(migration.contains( + "FOREIGN KEY (replay_root_job_record_id) " + + "REFERENCES etl_job_records (job_record_id)" + )); assertFalse(migration.contains("ON DELETE CASCADE")); assertFalse(migration.contains("replay_payload")); assertFalse(migration.contains("principal_name")); From c3a9627dffeeb4abe21571d359f5e3a14a6c0335 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 08:38:47 +0900 Subject: [PATCH 032/103] fix(etl): enforce owner-scoped replay lineage in PostgreSQL --- .../db/migration/V7__add_etl_job_replay_lineage.sql | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql b/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql index 88ecf4d7..cebdcbfc 100644 --- a/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql +++ b/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql @@ -5,13 +5,15 @@ ALTER TABLE etl_job_records ADD COLUMN replay_generation_count INTEGER; ALTER TABLE etl_job_records + ADD CONSTRAINT etl_job_owner_identity_unique + UNIQUE (job_record_id, principal_scope_hash), ADD CONSTRAINT etl_job_replay_source_reference - FOREIGN KEY (replay_source_job_record_id) - REFERENCES etl_job_records (job_record_id) + FOREIGN KEY (replay_source_job_record_id, principal_scope_hash) + REFERENCES etl_job_records (job_record_id, principal_scope_hash) ON DELETE RESTRICT, ADD CONSTRAINT etl_job_replay_root_reference - FOREIGN KEY (replay_root_job_record_id) - REFERENCES etl_job_records (job_record_id) + FOREIGN KEY (replay_root_job_record_id, principal_scope_hash) + REFERENCES etl_job_records (job_record_id, principal_scope_hash) ON DELETE RESTRICT, ADD CONSTRAINT etl_job_replay_lineage_complete_check CHECK ( ( From c18ce76aa1551e08c3f75d2d4c703275fe601f05 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 08:44:30 +0900 Subject: [PATCH 033/103] test(etl): require documented owner-scoped replay lineage --- .../documentation/DurableJobReplayDocumentationTest.java | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java index b087b82c..380bc9b7 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java @@ -29,6 +29,9 @@ void operationsRunbookDocumentsAdmissionLineageAndRollback() throws IOException assertTrue(runbook.contains("replay_source_job_record_id")); assertTrue(runbook.contains("replay_root_job_record_id")); assertTrue(runbook.contains("replay_generation_count")); + assertTrue(runbook.contains("etl_job_owner_identity_unique")); + assertTrue(runbook.contains("(job_record_id, principal_scope_hash)")); + assertTrue(runbook.contains("cross-owner lineage")); assertTrue(runbook.contains("ON DELETE RESTRICT")); assertTrue(runbook.contains("generation 1 through 100")); assertTrue(runbook.contains("prov:wasDerivedFrom")); @@ -36,6 +39,7 @@ void operationsRunbookDocumentsAdmissionLineageAndRollback() throws IOException assertTrue(runbook.contains("does not prove that replaying a connector")); assertTrue(runbook.contains("RFC 9110")); assertTrue(runbook.contains("RFC 9457")); + assertTrue(runbook.contains("PostgreSQL 18 documentation: Constraints")); assertTrue(runbook.contains("PROV-O")); } @@ -53,7 +57,9 @@ void designAndPlanPreserveTheSingleExecutionEngine() throws IOException { assertTrue(design.contains("ordinary `PENDING` job")); assertTrue(design.contains("No replay-specific worker or scheduler exists")); assertTrue(design.contains("replay_generation_count")); + assertTrue(design.contains("composite owner-scoped foreign keys")); assertTrue(plan.contains("Never update a terminal source back to `PENDING`")); + assertTrue(plan.contains("composite owner-scoped foreign keys")); assertTrue(plan.contains("Run all verification")); assertTrue(plan.contains("no project test is skipped")); } @@ -67,6 +73,8 @@ void changelogRecordsReplayAdmissionLineageAndSafety() throws IOException { assertTrue(changelog.contains("replay_source_job_record_id")); assertTrue(changelog.contains("replay_root_job_record_id")); assertTrue(changelog.contains("replay_generation_count")); + assertTrue(changelog.contains("etl_job_owner_identity_unique")); + assertTrue(changelog.contains("composite owner-scoped foreign keys")); assertTrue(changelog.contains("V7__add_etl_job_replay_lineage.sql")); assertTrue(changelog.contains("does not prove external connector safety")); } From add14d7cae274e9f6f3b80bb21d5808f013075f3 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 08:46:35 +0900 Subject: [PATCH 034/103] docs(etl): document owner-scoped replay lineage constraints --- docs/operations/durable-job-replay.md | 44 ++++++++++++++++++++------- 1 file changed, 33 insertions(+), 11 deletions(-) diff --git a/docs/operations/durable-job-replay.md b/docs/operations/durable-job-replay.md index 81cfe2eb..34e85d67 100644 --- a/docs/operations/durable-job-replay.md +++ b/docs/operations/durable-job-replay.md @@ -73,7 +73,7 @@ concurrent request that cannot acquire it receives `409 etl_job_replay_in_progre first transaction commits returns the created job. The existing `etl_job_submission_scope_unique` constraint remains a second integrity boundary. -## Immutable lineage +## Immutable owner-scoped lineage `V7__add_etl_job_replay_lineage.sql` adds: @@ -87,11 +87,24 @@ Root jobs have all fields null. Replay rows have all fields non-null, references own job identifier, and generation 1 through 100. Both self-referencing foreign keys use `ON DELETE RESTRICT`; deleting a source or root cannot silently cascade through audit history. +The migration also adds the named support key: + +```text +etl_job_owner_identity_unique +UNIQUE (job_record_id, principal_scope_hash) +``` + +Both lineage relationships are composite owner-scoped foreign keys. They reference +`(job_record_id, principal_scope_hash)` rather than only the opaque job identifier. The database +therefore rejects cross-owner lineage even if application code, a maintenance script, or a future +import path attempts to pair one tenant's new job with another tenant's source or root. Application +owner predicates remain mandatory, but they are no longer the only tenant-integrity boundary. + ```mermaid flowchart LR - R0[Root terminal job
generation null] -->|replay| R1[Replay job
generation 1] - R1 -->|later terminal + replay| R2[Replay job
generation 2] - R0 -. root reference .-> R2 + R0[Root terminal job
generation null] -->|same-owner replay| R1[Replay job
generation 1] + R1 -->|later terminal + same-owner replay| R2[Replay job
generation 2] + R0 -. owner-scoped root reference .-> R2 ``` A replay of a root uses the source as root and generation 1. A replay of a replay inherits the root @@ -122,15 +135,20 @@ or replay-key idempotency. ## Rollout -1. Rehearse V7 on a representative PostgreSQL 18 copy and inspect existing row count and lock time. +1. Rehearse V7 on a representative PostgreSQL 18 copy and inspect existing row count, support-index + build time, table lock duration, and foreign-key validation time. 2. Verify exact-head cross-platform CI, full reactor tests, zero-missed configured coverage, dependency review, SBOM, SAST, security scan, review threads, and independent approval. 3. Apply V7 before serving the replay route. 4. Smoke-test a disposable failed source, exact payload acceptance, same-key retry, and key conflict. 5. Confirm the source is unchanged and the new row has source/root/generation lineage. -6. Claim the new pending row through the ordinary worker and verify no replay-only execution path. -7. Monitor replay acceptance, in-progress conflicts, payload mismatches, generation exhaustion, - database lock waits, and failed foreign-key deletion attempts using fixed-cardinality signals. +6. In an isolated migration rehearsal, attempt source and root references whose + `principal_scope_hash` differs from the new row and confirm PostgreSQL rejects both cross-owner + lineage writes. +7. Claim the new pending row through the ordinary worker and verify no replay-only execution path. +8. Monitor replay acceptance, in-progress conflicts, payload mismatches, generation exhaustion, + database lock waits, and failed foreign-key deletion or tenant-boundary attempts using + fixed-cardinality signals. Logs and metric labels must not contain payloads, raw principals, raw keys, hashes, source/new job identifiers, lineage identifiers, SQL, exception messages, or target identities. @@ -153,7 +171,8 @@ for a deliberately separate replay. Stop replay admission. Preserve affected rows, deployed SHA, Flyway history, and backup evidence. Do not null lineage fields to make constraints pass. Repair requires a reviewed migration based on -verified source/root ownership and generation. +verified source/root ownership and generation. Treat any attempted cross-owner lineage write as a +tenant-isolation incident even when PostgreSQL rejects it. ## Rollback @@ -163,8 +182,8 @@ relationships. Do not drop V7 while replay rows exist. Archive or remove replay lineages from leaf to root under an approved retention policy, preserving external audit evidence. Then a separately reviewed migration -may remove the constraints and columns. Never edit the applied V7 file or mutate terminal sources -back to pending. +may remove the composite foreign keys, `etl_job_owner_identity_unique`, and lineage columns. Never +edit the applied V7 file or mutate terminal sources back to pending. The replay-key domain must remain readable while any replay-created row can receive an idempotent retry. A domain change requires a versioned migration or dual-read period, not a silent constant edit. @@ -184,6 +203,9 @@ https://www.rfc-editor.org/rfc/rfc9110 Nottingham, M., Wilde, E., & Dalal, S. (2023). *Problem details for HTTP APIs* (RFC 9457). RFC Editor. https://www.rfc-editor.org/rfc/rfc9457 +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: Constraints*. +https://www.postgresql.org/docs/18/ddl-constraints.html + PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: INSERT*. https://www.postgresql.org/docs/18/sql-insert.html From 9dac9894354182e845567f2c4d9de23db944d207 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 08:47:55 +0900 Subject: [PATCH 035/103] docs(etl): specify database-enforced replay tenant lineage --- .../2026-08-06-durable-job-replay-design.md | 28 ++++++++++++++++--- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md b/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md index 5fb93ccd..14edcda1 100644 --- a/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md +++ b/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md @@ -31,7 +31,24 @@ replay_root_job_record_id replay_generation_count ``` -The root job has all three fields null. Every replay row has all three fields non-null, an immediate source different from itself, a root different from itself, and a generation from 1 through 100. Self-referencing foreign keys use `ON DELETE RESTRICT` so terminal history cannot disappear through cascade deletion. +The root job has all three fields null. Every replay row has all three fields non-null, an immediate source different from itself, a root different from itself, and a generation from 1 through 100. Composite owner-scoped foreign keys use `ON DELETE RESTRICT` so terminal history cannot disappear through cascade deletion and one tenant cannot reference another tenant's source or root. + +The referenced key is the named support constraint: + +```text +etl_job_owner_identity_unique +UNIQUE (job_record_id, principal_scope_hash) +``` + +Each source and root relationship includes the new row's `principal_scope_hash`: + +```text +FOREIGN KEY (replay_source_job_record_id, principal_scope_hash) + REFERENCES etl_job_records (job_record_id, principal_scope_hash) + +FOREIGN KEY (replay_root_job_record_id, principal_scope_hash) + REFERENCES etl_job_records (job_record_id, principal_scope_hash) +``` For the first replay: @@ -49,7 +66,7 @@ root = inherited first job generation = source generation + 1 ``` -The application verifies that source and inherited root are owner-scoped to the same principal. Database constraints remain the structural boundary; the relational rows are authoritative even when lineage is later exported as W3C PROV. +The application still verifies that source and inherited root are owner-scoped to the same principal. The database independently rejects cross-owner lineage through the composite owner-scoped foreign keys. Relational rows remain authoritative even when lineage is later exported as W3C PROV. ## Replay-key authority @@ -81,7 +98,8 @@ One transaction performs the following sequence: 7. require the supplied payload digest to equal the source `request_digest`; 8. derive root and bounded generation; 9. insert one new `PENDING` row with the verified payload and lineage; -10. return only the new operator-safe job identity. +10. let PostgreSQL validate source and root against the same `principal_scope_hash`; +11. return only the new operator-safe job identity. The source is never updated. Read-then-write state resurrection is prohibited. @@ -134,7 +152,7 @@ The exact-head suite must prove: 9. generation 100 cannot create generation 101; 10. concurrent creation produces one row and an in-progress or later replay outcome; 11. the new job can be claimed and follows normal lifecycle contracts; -12. migration completeness, self-reference prohibition, naming, deletion restriction, rollout, and rollback are documented and tested; +12. migration completeness, owner-scoped source and root constraints, cross-owner lineage rejection, self-reference prohibition, naming, deletion restriction, rollout, and rollback are documented and tested; 13. all added production statements and branches retain zero-missed configured coverage and no project test is skipped. ## Operational limitation @@ -147,6 +165,8 @@ Fielding, R., Nottingham, M., & Reschke, J. (2022). *HTTP semantics* (RFC 9110). Nottingham, M., Wilde, E., & Dalal, S. (2023). *Problem details for HTTP APIs* (RFC 9457). RFC Editor. https://www.rfc-editor.org/rfc/rfc9457 +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: Constraints*. https://www.postgresql.org/docs/18/ddl-constraints.html + PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: INSERT*. https://www.postgresql.org/docs/18/sql-insert.html World Wide Web Consortium. (2013). *PROV-O: The PROV ontology*. https://www.w3.org/TR/prov-o/ From 1f4d95d0ec44f54b037c55910dd46e94d0979d61 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 08:48:57 +0900 Subject: [PATCH 036/103] docs(etl): add replay lineage tenant-integrity plan --- .../plans/2026-08-06-durable-job-replay.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/plans/2026-08-06-durable-job-replay.md b/docs/superpowers/plans/2026-08-06-durable-job-replay.md index da517d57..50be3e88 100644 --- a/docs/superpowers/plans/2026-08-06-durable-job-replay.md +++ b/docs/superpowers/plans/2026-08-06-durable-job-replay.md @@ -4,7 +4,7 @@ **Goal:** Create a new owner-scoped durable job from a failed or cancelled source only after the client resupplies the exact original payload. -**Architecture:** Store replay lineage on the new job row, use a versioned principal-scoped replay-key hash in the existing submission identity column, serialize creation with the existing transaction-lock boundary, verify payload digest against the terminal source, and return the existing accepted-job wire model. +**Architecture:** Store replay lineage on the new job row, bind immediate-source and root references to the same principal through composite owner-scoped foreign keys, use a versioned principal-scoped replay-key hash in the existing submission identity column, serialize creation with the existing transaction-lock boundary, verify payload digest against the terminal source, and return the existing accepted-job wire model. **Tech Stack:** Java 25, Spring MVC, Spring transactions, JdbcTemplate, PostgreSQL 18, Flyway, H2 integration tests, JUnit 5, Mockito, JaCoCo, Maven. @@ -14,6 +14,7 @@ - Allow only `FAILED` and `CANCELLED` sources. - Validate identifier, replay key, principal, and complete payload before lock or table access. - Persist no raw principal or raw replay key. +- Require PostgreSQL to reject source or root lineage whose `principal_scope_hash` differs from the new row. - Preserve zero-missed configured production instruction, line, method, and branch coverage. - Preserve no-skipped project tests and beginner-readable public Javadoc. - Use descriptive multi-word `snake_case` database objects. @@ -25,8 +26,9 @@ - Modify: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobMigrationDocumentationTest.java` - Create: `etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql` -- [ ] Require all three lineage columns, bounded generation, complete-null-or-complete-non-null lifecycle, self-reference rejection, two named self-referencing foreign keys, and `ON DELETE RESTRICT`. -- [ ] Run the focused migration test and observe failure because V7 is absent. +- [ ] Require all three lineage columns, bounded generation, complete-null-or-complete-non-null lifecycle, self-reference rejection, named composite owner-scoped foreign keys, their named `(job_record_id, principal_scope_hash)` unique support constraint, and `ON DELETE RESTRICT`. +- [ ] Reject legacy one-column source or root foreign keys because they permit cross-owner lineage at the database layer. +- [ ] Run the focused migration test and observe failure because V7 or the tenant-integrity constraints are absent. - [ ] Implement the additive transactional migration. - [ ] Rerun the focused test and commit. @@ -91,6 +93,7 @@ EtlJobReplay replayOwned( - [ ] Prove retry after a committed first replay returns the same new row. - [ ] Prove replay-of-replay preserves the root and increments generation. - [ ] Prove generation 100 fails before insertion. +- [ ] Prove PostgreSQL rejects a replay source or root from another `principal_scope_hash`, independent of application owner predicates. - [ ] Run focused and full tests and commit. ## Task 6 — Finish operations, provenance, and exact-head verification @@ -102,9 +105,10 @@ EtlJobReplay replayOwned( - Modify: `CHANGELOG.md` - Create: `etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java` -- [ ] Require source immutability, payload digest proof, lineage, concurrency, generation limit, connector limitation, rollout, and rollback documentation first. +- [ ] Require source immutability, payload digest proof, composite owner-scoped foreign keys, cross-owner lineage rejection, concurrency, generation limit, connector limitation, rollout, and rollback documentation first. - [ ] Document W3C PROV mapping as an export contract, not a database-authority substitute. - [ ] Record APA 7th primary references and the versioned replay-key compatibility boundary. +- [ ] Rehearse the migration on PostgreSQL 18 and verify the owner-scoped source/root foreign-key failures before production rollout. - [ ] Run all verification through exact-head CI: `./mvnw -B test`, configured coverage gates, and `git diff --check`. - [ ] Keep the PR draft until every stacked-target gate succeeds. @@ -114,5 +118,6 @@ EtlJobReplay replayOwned( - The source is never mutated. - Replay-key and payload conflicts are distinguished without disclosing source existence across principals. - New jobs enter the existing worker lifecycle rather than creating a second execution engine. +- PostgreSQL and application owner predicates independently reject cross-owner lineage. - No placeholder, ambiguous public signature, or unbounded database object name remains. - Verification requires that no project test is skipped. From cb3340af40d2247b8bc40d9d2b5bc1125e34c606 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 08:52:38 +0900 Subject: [PATCH 037/103] docs(changelog): record owner-scoped replay lineage integrity --- CHANGELOG.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4d79b620..604dbb82 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,7 +38,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- Transactional migration `V7__add_etl_job_replay_lineage.sql`, owner-scoped replay admission, and immutable `replay_source_job_record_id`, `replay_root_job_record_id`, and `replay_generation_count` evidence with concurrency, lineage, worker-compatibility, rollout, incident, and rollback tests and documentation. +- Transactional migration `V7__add_etl_job_replay_lineage.sql`, owner-scoped replay admission, the named `etl_job_owner_identity_unique` support key, composite owner-scoped foreign keys for immediate-source and root lineage, and immutable `replay_source_job_record_id`, `replay_root_job_record_id`, and `replay_generation_count` evidence with concurrency, tenant-integrity, worker-compatibility, rollout, incident, and rollback tests and documentation. - Transactional migration `V6__add_etl_job_cancellation.sql`, owner-safe cancellation API and replay model, exact lease-invalidation and cancellation-versus-success integration tests, plus rollout, incident, connector-limitation, and rollback evidence in `docs/operations/durable-job-cancellation.md`. - Deterministic ordinary, wildcard, changed-state, changed-failure-code, null-versus-empty, and unrelated-response conditional polling tests, complete controller Javadoc, privacy and rollback guidance, and APA 7th standards evidence in `docs/etl/durable-job-polling.md`. - Controller-scoped polling advice, deterministic active/terminal lifecycle tests, disabled-worker fail-closed behavior, sub-second rounding coverage, rollback guidance, and APA 7th standards evidence in `docs/etl/durable-job-polling.md`. @@ -79,7 +79,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Security -- Replay uses owner-scoped source selection, byte-exact digest verification, a versioned principal-scoped key domain, and immutable relational lineage without retaining raw principals or replay keys; this evidence does not prove external connector safety, so connector-native idempotency, transaction participation, or compensation remains required. +- Replay uses owner-scoped source selection, byte-exact digest verification, a versioned principal-scoped key domain, immutable relational lineage, and composite owner-scoped foreign keys that independently reject cross-tenant source or root references without retaining raw principals or replay keys; this evidence does not prove external connector safety, so connector-native idempotency, transaction participation, or compensation remains required. - Cancellation stores only a principal-scoped SHA-256 replay identity and a fixed machine code, exposes no raw principal, key, hash, payload, lease, SQL, exception, or target detail, and requires an owner-matched conditional database update before reporting success. - Conditional status validators are SHA-256 digests of only the complete owner-authorized operator-safe representation; payloads, raw principals, idempotency keys, internal hashes, leases, SQL, and exception text remain excluded, and wildcard evaluation occurs only after owner-safe lookup. - Polling advice exposes only a bounded delay integer and is omitted when local execution is disabled or terminal; it never contains job, lease, principal, key, hash, payload, SQL, exception, target, or queue-depth data. @@ -286,5 +286,5 @@ This changelog will be updated: --- **Changelog Version**: 1.0 -**Last Updated**: 2026-08-06 +**Last Updated**: 2026-08-07 **Maintained By**: Development Team From 4f38819a71d18fb27dd7a3a8ca5b94c5d6fb9e51 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 08:57:33 +0900 Subject: [PATCH 038/103] test(etl): require PostgreSQL 18 replay migration rehearsal --- .../EtlJobReplayPostgresWorkflowTest.java | 74 +++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/documentation/EtlJobReplayPostgresWorkflowTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/documentation/EtlJobReplayPostgresWorkflowTest.java b/etl-service/src/test/java/com/xtrmetl/etl/documentation/EtlJobReplayPostgresWorkflowTest.java new file mode 100644 index 00000000..18e77415 --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/documentation/EtlJobReplayPostgresWorkflowTest.java @@ -0,0 +1,74 @@ +package com.xtrmetl.etl.documentation; + +import org.junit.jupiter.api.Test; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; + +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Keeps the production-PostgreSQL replay migration and rollback rehearsal executable in CI. + */ +class EtlJobReplayPostgresWorkflowTest { + + @Test + void ciRunsTheCompleteReplayMigrationChainAgainstPostgres18() throws IOException { + String workflow = read(".github/workflows/ci.yml").replaceAll("\\s+", " "); + + assertTrue(workflow.contains("postgres_replay_migration:")); + assertTrue(workflow.contains("image: postgres:18")); + assertTrue(workflow.contains("--health-cmd pg_isready")); + assertTrue(workflow.contains("V2__create_etl_job_records.sql")); + assertTrue(workflow.contains("V3__add_etl_job_lease_fencing.sql")); + assertTrue(workflow.contains("V4__add_etl_job_claim_eligibility_index.sql")); + assertTrue(workflow.contains("V5__add_etl_job_owner_pagination_index.sql")); + assertTrue(workflow.contains("V6__add_etl_job_cancellation.sql")); + assertTrue(workflow.contains("V7__add_etl_job_replay_lineage.sql")); + assertTrue(workflow.contains("psql -v ON_ERROR_STOP=1")); + assertTrue(workflow.contains( + "etl-service/src/test/postgresql/replay_lineage_migration.sql" + )); + } + + @Test + void postgresRehearsalCoversTenantIntegrityDeletionAndRollback() throws IOException { + String rehearsal = read( + "etl-service/src/test/postgresql/replay_lineage_migration.sql" + ).replaceAll("\\s+", " "); + + assertTrue(rehearsal.contains("cross-owner source lineage was accepted")); + assertTrue(rehearsal.contains("cross-owner root lineage was accepted")); + assertTrue(rehearsal.contains("ON DELETE RESTRICT did not protect replay history")); + assertTrue(rehearsal.contains("ALTER TABLE etl_job_records DROP CONSTRAINT etl_job_replay_source_reference")); + assertTrue(rehearsal.contains("ALTER TABLE etl_job_records DROP COLUMN replay_source_job_record_id")); + assertTrue(rehearsal.contains("ROLLBACK")); + assertTrue(rehearsal.contains("rollback rehearsal did not restore V7")); + } + + private static String read(String relativePath) throws IOException { + return Files.readString(projectRoot().resolve(relativePath), StandardCharsets.UTF_8); + } + + /** @return repository root from reactor-root or module-local execution */ + private static Path projectRoot() { + Path current = Paths.get(System.getProperty("user.dir")).toAbsolutePath(); + Path lastPomParent = null; + while (current != null) { + if (Files.exists(current.resolve(".git"))) { + return current; + } + if (Files.exists(current.resolve("pom.xml"))) { + lastPomParent = current; + } + current = current.getParent(); + } + if (lastPomParent != null) { + return lastPomParent; + } + throw new IllegalStateException("Could not find project root"); + } +} From b0c719fab11064b71991e51f176a661b7a0d02c8 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 09:12:52 +0900 Subject: [PATCH 039/103] docs(etl): add replay standards evidence --- .../durable-job-replay-standards-evidence.md | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 docs/doctoring/durable-job-replay-standards-evidence.md diff --git a/docs/doctoring/durable-job-replay-standards-evidence.md b/docs/doctoring/durable-job-replay-standards-evidence.md new file mode 100644 index 00000000..983f2b1c --- /dev/null +++ b/docs/doctoring/durable-job-replay-standards-evidence.md @@ -0,0 +1,47 @@ +# Durable job replay standards evidence + +## Decision + +mightyETL models replay as creation of a new durable job derived from an immutable terminal source. The source is never returned to `PENDING`. Only owner-scoped `FAILED` and `CANCELLED` sources are eligible, and the operator must resupply byte-identical bounded JSON whose SHA-256 digest equals the source `request_digest`. + +The accepted replay response uses `202 Accepted` with a monitor URI because durable admission does not assert completion. Deterministic failures use RFC 9457 problem details. PostgreSQL owns the replay transaction, uniqueness, lineage constraints, and source-row serialization. Replay lineage is compatible with PROV-O derivation semantics, while the relational database remains authoritative. + +## Normative mapping + +| Product contract | Primary authority | Application | +|---|---|---| +| Noncommittal durable admission | RFC 9110, section 15.3.3 | `202 Accepted`, `Location`, and no claim of execution completion | +| Stable machine-readable failures | RFC 9457 | Fixed problem type, title, status, detail, and `error_code` without exception text | +| Atomic new-row creation and conflict handling | PostgreSQL 18 `INSERT` and transaction documentation | One transaction validates source ownership, digest, replay identity, and lineage before inserting one new job | +| Derivation lineage | W3C PROV-O | New job is derived from the immediate source and preserves an immutable root/generation chain | +| Domain separation rationale | NIST SP 800-185 | Replay-key hashing uses a versioned replay-specific domain; the SHA-256 construction does not claim cSHAKE or TupleHash conformance | + +## Security and privacy boundary + +The HTTP response and ordinary telemetry exclude raw principals, replay keys, payloads, request digests, internal hashes, source/root identifiers, lease identifiers, SQL, target identities, and exception messages. Foreign-owned and absent source identifiers remain indistinguishable. `SUCCEEDED` is excluded because repeating a committed target effect is not safe merely because the original request bytes are known. + +Connector replay is enabled only when target effects participate in the mightyETL transaction or the connector supplies independently tested idempotency or compensation. Payload equality is evidence of replay fidelity, not evidence that an external system will suppress duplicate effects. + +## Verification obligations + +- real PostgreSQL migration rehearsal for self-referencing foreign keys and `ON DELETE RESTRICT`; +- exact-payload acceptance and byte-different rejection; +- owner-safe missing/foreign behavior; +- same-key replay, key reuse conflict, and concurrent admission tests; +- source immutability and source/root/generation lineage tests; +- generation-bound rejection; +- ordinary worker claim, lease fencing, cancellation, polling, and ETag compatibility; +- configured production instruction, line, method, and branch coverage with zero misses; +- direct-base CI, dependency, SBOM, SAST, security, review-thread, and independent-approval gates before merge. + +## References — APA 7th edition + +Fielding, R., Nottingham, M., & Reschke, J. (2022). *HTTP semantics* (RFC 9110). RFC Editor. https://www.rfc-editor.org/rfc/rfc9110 + +National Institute of Standards and Technology. (2016). *SHA-3 derived functions: cSHAKE, KMAC, TupleHash, and ParallelHash* (NIST Special Publication 800-185). U.S. Department of Commerce. https://doi.org/10.6028/NIST.SP.800-185 + +Nottingham, M., Wilde, E., & Dalal, S. (2023). *Problem details for HTTP APIs* (RFC 9457). RFC Editor. https://www.rfc-editor.org/rfc/rfc9457 + +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: INSERT*. https://www.postgresql.org/docs/18/sql-insert.html + +World Wide Web Consortium. (2013). *PROV-O: The PROV ontology*. https://www.w3.org/TR/prov-o/ From 01d7b5e8524e104e741f8ddea169abd37aa90e55 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 09:14:50 +0900 Subject: [PATCH 040/103] test(etl): verify replay migrations on PostgreSQL --- scripts/verify-postgresql-migrations.sh | 140 ++++++++++++++++++++++++ 1 file changed, 140 insertions(+) create mode 100644 scripts/verify-postgresql-migrations.sh diff --git a/scripts/verify-postgresql-migrations.sh b/scripts/verify-postgresql-migrations.sh new file mode 100644 index 00000000..0c8b7a5f --- /dev/null +++ b/scripts/verify-postgresql-migrations.sh @@ -0,0 +1,140 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +: "${PGHOST:=127.0.0.1}" +: "${PGPORT:=5432}" +: "${PGDATABASE:=mightyetl_replay_test}" +: "${PGUSER:=mightyetl_test}" +: "${PGPASSWORD:=mightyetl_test_password}" +export PGHOST PGPORT PGDATABASE PGUSER PGPASSWORD + +migration_directory="etl-service/src/main/resources/db/migration" +if [[ ! -d "${migration_directory}" ]]; then + printf 'Migration directory not found: %s\n' "${migration_directory}" >&2 + exit 1 +fi + +for attempt_number in $(seq 1 30); do + if pg_isready --host "${PGHOST}" --port "${PGPORT}" --dbname "${PGDATABASE}" --username "${PGUSER}" >/dev/null 2>&1; then + break + fi + if [[ "${attempt_number}" -eq 30 ]]; then + printf 'PostgreSQL did not become ready after 30 attempts.\n' >&2 + exit 1 + fi + sleep 2 +done + +mapfile -d '' migration_files < <( + find "${migration_directory}" -maxdepth 1 -type f -name 'V*__*.sql' -print0 | sort -zV +) +if [[ "${#migration_files[@]}" -eq 0 ]]; then + printf 'No versioned SQL migrations found.\n' >&2 + exit 1 +fi + +for migration_file in "${migration_files[@]}"; do + printf 'Applying %s\n' "${migration_file}" + psql --no-psqlrc --set ON_ERROR_STOP=1 --file "${migration_file}" >/dev/null +done + +psql --no-psqlrc --set ON_ERROR_STOP=1 <<'SQL' +DO $verification_block$ +DECLARE + missing_column_count integer; + restrict_foreign_key_count integer; + replay_check_definition text; + cancellation_check_definition text; +BEGIN + IF to_regclass('public.etl_job_records') IS NULL THEN + RAISE EXCEPTION 'etl_job_records was not created'; + END IF; + + SELECT count(*) + INTO missing_column_count + FROM ( + VALUES + ('replay_source_job_record_id', 'uuid'), + ('replay_root_job_record_id', 'uuid'), + ('replay_generation_count', 'integer'), + ('cancellation_key_hash', 'character'), + ('cancellation_code', 'character varying'), + ('job_cancelled_at', 'timestamp with time zone') + ) AS expected_columns(column_name, data_type) + WHERE NOT EXISTS ( + SELECT 1 + FROM information_schema.columns AS actual_columns + WHERE actual_columns.table_schema = 'public' + AND actual_columns.table_name = 'etl_job_records' + AND actual_columns.column_name = expected_columns.column_name + AND actual_columns.data_type = expected_columns.data_type + ); + + IF missing_column_count <> 0 THEN + RAISE EXCEPTION 'one or more cancellation/replay columns are missing or have the wrong type'; + END IF; + + SELECT count(*) + INTO restrict_foreign_key_count + FROM pg_constraint AS constraint_record + JOIN pg_class AS table_record + ON table_record.oid = constraint_record.conrelid + JOIN unnest(constraint_record.conkey) AS constrained_attribute(attribute_number) + ON true + JOIN pg_attribute AS attribute_record + ON attribute_record.attrelid = table_record.oid + AND attribute_record.attnum = constrained_attribute.attribute_number + WHERE table_record.relname = 'etl_job_records' + AND constraint_record.contype = 'f' + AND constraint_record.confrelid = table_record.oid + AND constraint_record.confdeltype = 'r' + AND attribute_record.attname IN ( + 'replay_source_job_record_id', + 'replay_root_job_record_id' + ); + + IF restrict_foreign_key_count <> 2 THEN + RAISE EXCEPTION 'replay source and root must each use a self-reference with ON DELETE RESTRICT'; + END IF; + + SELECT string_agg(pg_get_constraintdef(constraint_record.oid), ' ') + INTO replay_check_definition + FROM pg_constraint AS constraint_record + JOIN pg_class AS table_record + ON table_record.oid = constraint_record.conrelid + WHERE table_record.relname = 'etl_job_records' + AND constraint_record.contype = 'c' + AND pg_get_constraintdef(constraint_record.oid) ILIKE '%replay_generation_count%'; + + IF replay_check_definition IS NULL + OR replay_check_definition NOT ILIKE '%replay_source_job_record_id%' + OR replay_check_definition NOT ILIKE '%replay_root_job_record_id%' + OR replay_check_definition NOT ILIKE '%100%' THEN + RAISE EXCEPTION 'replay lineage checks do not bind source, root, and the bounded generation'; + END IF; + + SELECT string_agg(pg_get_constraintdef(constraint_record.oid), ' ') + INTO cancellation_check_definition + FROM pg_constraint AS constraint_record + JOIN pg_class AS table_record + ON table_record.oid = constraint_record.conrelid + WHERE table_record.relname = 'etl_job_records' + AND constraint_record.contype = 'c' + AND pg_get_constraintdef(constraint_record.oid) ILIKE '%cancellation_key_hash%'; + + IF cancellation_check_definition IS NULL + OR cancellation_check_definition NOT ILIKE '%job_cancelled_at%' + OR cancellation_check_definition NOT ILIKE '%CANCELLED%' THEN + RAISE EXCEPTION 'cancellation lifecycle checks are incomplete'; + END IF; +END +$verification_block$; +SQL + +pg_dump --schema-only --no-owner --no-privileges > /tmp/mightyetl-postgresql-schema.sql +if [[ ! -s /tmp/mightyetl-postgresql-schema.sql ]]; then + printf 'Schema-only dump was empty.\n' >&2 + exit 1 +fi + +printf 'PostgreSQL migration and replay-lineage verification succeeded.\n' From 8ad633ac2c7b48d3edc2d59482e349285c73fdfb Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 09:15:06 +0900 Subject: [PATCH 041/103] ci(etl): run replay migrations on PostgreSQL 18 --- .../postgresql-migration-integration.yml | 52 +++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 .github/workflows/postgresql-migration-integration.yml diff --git a/.github/workflows/postgresql-migration-integration.yml b/.github/workflows/postgresql-migration-integration.yml new file mode 100644 index 00000000..258d93ed --- /dev/null +++ b/.github/workflows/postgresql-migration-integration.yml @@ -0,0 +1,52 @@ +name: PostgreSQL Migration Integration + +on: + pull_request: + branches: + - develop + paths: + - "etl-service/src/main/resources/db/migration/**" + - "scripts/verify-postgresql-migrations.sh" + - ".github/workflows/postgresql-migration-integration.yml" + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: postgresql-migration-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: true + +jobs: + replay_lineage_migration: + name: replay-lineage-migration + runs-on: ubuntu-latest + timeout-minutes: 15 + services: + postgresql_database: + image: postgres:18-alpine + env: + POSTGRES_DB: mightyetl_replay_test + POSTGRES_USER: mightyetl_test + POSTGRES_PASSWORD: mightyetl_test_password + ports: + - 5432:5432 + options: >- + --health-cmd "pg_isready --username=mightyetl_test --dbname=mightyetl_replay_test" + --health-interval 5s + --health-timeout 5s + --health-retries 12 + env: + PGHOST: 127.0.0.1 + PGPORT: "5432" + PGDATABASE: mightyetl_replay_test + PGUSER: mightyetl_test + PGPASSWORD: mightyetl_test_password + steps: + - name: Check out exact pull-request merge candidate + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 + with: + persist-credentials: false + - name: Verify versioned migrations and immutable replay lineage + shell: bash + run: bash scripts/verify-postgresql-migrations.sh From aef9d29f7c60f8373d23130bc85a8f65cf6959be Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 09:15:37 +0900 Subject: [PATCH 042/103] test(etl): lock PostgreSQL replay migration workflow --- ...resqlMigrationIntegrationWorkflowTest.java | 85 +++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/documentation/PostgresqlMigrationIntegrationWorkflowTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/documentation/PostgresqlMigrationIntegrationWorkflowTest.java b/etl-service/src/test/java/com/xtrmetl/etl/documentation/PostgresqlMigrationIntegrationWorkflowTest.java new file mode 100644 index 00000000..a971eab0 --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/documentation/PostgresqlMigrationIntegrationWorkflowTest.java @@ -0,0 +1,85 @@ +package com.xtrmetl.etl.documentation; + +import org.junit.jupiter.api.Test; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; + +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Keeps the real PostgreSQL migration gate aligned with replay-lineage safety requirements. + */ +class PostgresqlMigrationIntegrationWorkflowTest { + + @Test + void workflowUsesLeastPrivilegePinnedCheckoutAndPostgresqlEighteen() throws IOException { + String workflow = read(".github/workflows/postgresql-migration-integration.yml"); + + assertTrue(workflow.contains("name: PostgreSQL Migration Integration")); + assertTrue(workflow.contains("branches:\n - develop")); + assertTrue(workflow.contains("permissions:\n contents: read")); + assertTrue(workflow.contains("timeout-minutes: 15")); + assertTrue(workflow.contains("image: postgres:18-alpine")); + assertTrue(workflow.contains( + "uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0" + )); + assertTrue(workflow.contains("persist-credentials: false")); + assertTrue(workflow.contains("bash scripts/verify-postgresql-migrations.sh")); + assertFalse(workflow.contains("pull_request_target:")); + assertFalse(workflow.contains("COPILOT_GITHUB_TOKEN")); + assertFalse(workflow.contains("NVIDIA_NIM_API_KEY")); + assertFalse(workflow.contains("contents: write")); + } + + @Test + void verificationScriptAppliesEveryMigrationAndChecksReplayConstraints() throws IOException { + String script = read("scripts/verify-postgresql-migrations.sh"); + + assertTrue(script.contains("set -Eeuo pipefail")); + assertTrue(script.contains("pg_isready")); + assertTrue(script.contains("sort -zV")); + assertTrue(script.contains("--set ON_ERROR_STOP=1")); + assertTrue(script.contains("replay_source_job_record_id")); + assertTrue(script.contains("replay_root_job_record_id")); + assertTrue(script.contains("replay_generation_count")); + assertTrue(script.contains("constraint_record.confdeltype = 'r'")); + assertTrue(script.contains("replay_check_definition NOT ILIKE '%100%'")); + assertTrue(script.contains("cancellation_key_hash")); + assertTrue(script.contains("job_cancelled_at")); + assertTrue(script.contains("pg_dump --schema-only --no-owner --no-privileges")); + assertFalse(script.contains("set +e")); + assertFalse(script.contains("|| true")); + } + + private static String read(String relativePath) throws IOException { + return Files.readString(projectRoot().resolve(relativePath), StandardCharsets.UTF_8); + } + + /** + * Finds the Maven reactor root from repository-root or module-local execution. + * + * @return repository root containing the workflow and migration verifier + */ + private static Path projectRoot() { + Path current = Paths.get(System.getProperty("user.dir")).toAbsolutePath(); + Path lastPomParent = null; + while (current != null) { + if (Files.exists(current.resolve(".git"))) { + return current; + } + if (Files.exists(current.resolve("pom.xml"))) { + lastPomParent = current; + } + current = current.getParent(); + } + if (lastPomParent != null) { + return lastPomParent; + } + throw new IllegalStateException("Could not find project root"); + } +} From fdff8896f1e1dcf21696e7803cccd3ad64f90b56 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 09:17:30 +0900 Subject: [PATCH 043/103] docs(adr): record immutable replay lineage --- ...2026-08-07-immutable-durable-job-replay.md | 122 ++++++++++++++++++ 1 file changed, 122 insertions(+) create mode 100644 docs/adr/2026-08-07-immutable-durable-job-replay.md diff --git a/docs/adr/2026-08-07-immutable-durable-job-replay.md b/docs/adr/2026-08-07-immutable-durable-job-replay.md new file mode 100644 index 00000000..b627c0c2 --- /dev/null +++ b/docs/adr/2026-08-07-immutable-durable-job-replay.md @@ -0,0 +1,122 @@ +# ADR: Immutable durable-job replay lineage + +- **Status:** Proposed while the replay pull request is stacked; Accepted only after direct-`develop` gates and merge +- **Date:** 2026-08-07 +- **Decision owners:** mightyETL maintainers +- **Scope:** `etl-service` durable-job admission, persistence, operator API, and provenance + +## Context + +Durable jobs deliberately clear `request_payload` after success, failure, or cancellation. Operators nevertheless need a controlled way to retry failed or cancelled work without weakening source immutability, owner isolation, idempotency, lease fencing, or auditability. + +Rewinding a terminal row to `PENDING` would erase the original terminal fact, mix attempt histories, invalidate conditional status validators, and make concurrent cancellation or success reasoning substantially harder. Retaining terminal payloads solely for replay would expand sensitive-data retention. Treating semantically equivalent JSON as the same work would also allow hidden payload changes under a replay label. + +## Decision + +Replay creates a **new** durable job. The terminal source remains unchanged. + +The authenticated owner submits the source identifier, a replay-specific `Idempotency-Key`, and the complete bounded JSON text. Admission parses the payload through the ordinary durable-intake boundary and requires its SHA-256 digest to equal the immutable source `request_digest`. Only `FAILED` and `CANCELLED` sources are eligible. `SUCCEEDED` remains excluded because payload equality is not evidence that committed target effects may be repeated safely. + +A derived row stores: + +- `replay_source_job_record_id`: immediate source; +- `replay_root_job_record_id`: immutable root of the replay chain; +- `replay_generation_count`: bounded positive generation. + +Source and root references use `ON DELETE RESTRICT`. A root row has all three lineage fields null; a replay row has all three non-null. Generation cannot exceed the supported bound. The new row enters the ordinary `PENDING` lifecycle and uses the existing worker, PostgreSQL claim, exact lease, success, failure, polling, cancellation, and ETag contracts. + +```mermaid +sequenceDiagram + participant O as Authenticated owner + participant A as Replay API + participant P as PostgreSQL transaction + participant W as Ordinary worker + + O->>A: source id + exact payload + replay key + A->>A: bounded JSON validation and SHA-256 + A->>P: owner-matched terminal source lock + P->>P: verify FAILED/CANCELLED and digest equality + P->>P: verify replay-key identity and generation bound + P->>P: insert new PENDING row with source/root/generation + P-->>A: commit derived job + A-->>O: 202 + Location + Idempotency-Replayed + W->>P: ordinary lease-fenced claim +``` + +## Replay identity + +The stored replay identity uses a versioned replay-specific domain and the principal namespace. It is intentionally distinct from ordinary submission and cancellation domains. Raw principals and raw replay keys are not retained. The domain string is persistence compatibility behavior and cannot change without a migration or dual-read period. + +NIST SP 800-185 motivates explicit domain separation, but the SHA-256 construction does not claim cSHAKE, KMAC, TupleHash, or ParallelHash conformance. + +## HTTP behavior + +- first committed replay: `202 Accepted`, `Location`, `Cache-Control: no-store`, `Idempotency-Replayed: false`; +- committed same-intent retry: same derived job, current lifecycle state, `Idempotency-Replayed: true`; +- absent or foreign source: indistinguishable `404 etl_job_not_found`; +- active source: stable `409` problem; +- succeeded source: stable `409` problem; +- byte-different payload: stable `422` problem; +- conflicting replay-key reuse: stable `422` problem; +- unresolved concurrent admission: stable retryable `409` problem; +- generation exhaustion: stable `409` problem. + +All errors use fixed RFC 9457 metadata and exclude exception messages, SQL, hashes, identifiers, payloads, and target details. + +## Connector boundary + +The replay transaction can prove source ownership, payload fidelity, replay identity, lineage, and durable admission. It cannot prove that a remote warehouse, file system, API, or broker will suppress duplicate effects. Replay is enabled for a connector only when target effects participate in the mightyETL transaction or the connector provides independently tested idempotency or compensation. + +## Alternatives rejected + +### Rewind the terminal row + +Rejected because it destroys terminal history, combines multiple execution episodes into one identity, complicates ETag semantics, and weakens race reasoning. + +### Retain terminal payloads indefinitely + +Rejected because replay does not justify expanding sensitive payload retention. The operator must recover the exact payload from an approved upstream or encrypted audit source. + +### Accept semantic JSON equivalence + +Rejected because normalization can obscure a changed request and creates a second canonicalization contract. Replay fidelity is byte-exact, matching durable submission identity. + +### Permit succeeded-source replay + +Rejected in the initial slice because a succeeded job may already have committed irreversible external effects. + +## Consequences + +### Positive + +- terminal sources remain immutable; +- each execution episode has a distinct opaque job identity; +- lineage supports incident analysis and future PROV-compatible export; +- ordinary worker and cancellation machinery is reused; +- payload retention does not increase; +- concurrent retries have one database-owned outcome. + +### Costs + +- operators must possess the exact original payload bytes; +- self-referencing lineage constrains retention and deletion order; +- every connector needs an explicit replay-safety classification; +- migrations and generation bounds require real PostgreSQL verification. + +## Verification + +Acceptance requires exact-head tests for source immutability, owner isolation, exact payload matching, same-key replay, conflicting-key reuse, concurrent admission, lineage inheritance, generation exhaustion, ordinary worker behavior, cancellation compatibility, RFC 9457 responses, privacy exclusions, and zero-missed configured production coverage. + +A direct-`develop` GitHub Actions gate applies every versioned migration to PostgreSQL 18, verifies replay and cancellation columns, verifies both lineage foreign keys use `ON DELETE RESTRICT`, verifies bounded source/root/generation checks, and creates a non-empty schema-only dump. SAST, security, dependency, SBOM, review-thread, and non-author exact-head approval gates remain mandatory. + +## References — APA 7th edition + +Fielding, R., Nottingham, M., & Reschke, J. (2022). *HTTP semantics* (RFC 9110). RFC Editor. https://www.rfc-editor.org/rfc/rfc9110 + +National Institute of Standards and Technology. (2016). *SHA-3 derived functions: cSHAKE, KMAC, TupleHash, and ParallelHash* (NIST Special Publication 800-185). U.S. Department of Commerce. https://doi.org/10.6028/NIST.SP.800-185 + +Nottingham, M., Wilde, E., & Dalal, S. (2023). *Problem details for HTTP APIs* (RFC 9457). RFC Editor. https://www.rfc-editor.org/rfc/rfc9457 + +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: INSERT*. https://www.postgresql.org/docs/18/sql-insert.html + +World Wide Web Consortium. (2013). *PROV-O: The PROV ontology*. https://www.w3.org/TR/prov-o/ From 02bc464ae7b71e20c8d7827882507cb91f068dec Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 14:25:21 +0900 Subject: [PATCH 044/103] test(etl): rehearse replay lineage on PostgreSQL --- .../postgresql/replay_lineage_migration.sql | 226 ++++++++++++++++++ 1 file changed, 226 insertions(+) create mode 100644 etl-service/src/test/postgresql/replay_lineage_migration.sql diff --git a/etl-service/src/test/postgresql/replay_lineage_migration.sql b/etl-service/src/test/postgresql/replay_lineage_migration.sql new file mode 100644 index 00000000..2c50aa33 --- /dev/null +++ b/etl-service/src/test/postgresql/replay_lineage_migration.sql @@ -0,0 +1,226 @@ +-- Rehearse V7 owner isolation, immutable lineage, deletion protection, and rollback. +-- This script runs only against the disposable PostgreSQL integration-test database. + +BEGIN; + +INSERT INTO etl_job_records ( + job_record_id, + principal_scope_hash, + submission_key_hash, + request_digest, + request_payload, + job_status +) VALUES + ( + '00000000-0000-4000-8000-000000000001', + repeat('a', 64), + repeat('1', 64), + repeat('a', 64), + '{}', + 'PENDING' + ), + ( + '00000000-0000-4000-8000-000000000002', + repeat('a', 64), + repeat('2', 64), + repeat('b', 64), + NULL, + 'FAILED' + ), + ( + '00000000-0000-4000-8000-000000000003', + repeat('b', 64), + repeat('3', 64), + repeat('c', 64), + '{}', + 'PENDING' + ), + ( + '00000000-0000-4000-8000-000000000004', + repeat('b', 64), + repeat('4', 64), + repeat('d', 64), + NULL, + 'CANCELLED' + ); + +UPDATE etl_job_records +SET failure_code = 'etl_replay_source_failed' +WHERE job_record_id = '00000000-0000-4000-8000-000000000002'; + +UPDATE etl_job_records +SET cancellation_key_hash = repeat('e', 64), + cancellation_code = 'etl_job_cancelled_by_owner', + job_cancelled_at = CURRENT_TIMESTAMP +WHERE job_record_id = '00000000-0000-4000-8000-000000000004'; + +INSERT INTO etl_job_records ( + job_record_id, + principal_scope_hash, + submission_key_hash, + request_digest, + request_payload, + job_status, + replay_source_job_record_id, + replay_root_job_record_id, + replay_generation_count +) VALUES ( + '00000000-0000-4000-8000-000000000005', + repeat('a', 64), + repeat('5', 64), + repeat('f', 64), + '{}', + 'PENDING', + '00000000-0000-4000-8000-000000000002', + '00000000-0000-4000-8000-000000000001', + 1 +); + +DO $cross_owner_source_check$ +BEGIN + BEGIN + INSERT INTO etl_job_records ( + job_record_id, + principal_scope_hash, + submission_key_hash, + request_digest, + request_payload, + job_status, + replay_source_job_record_id, + replay_root_job_record_id, + replay_generation_count + ) VALUES ( + '00000000-0000-4000-8000-000000000006', + repeat('b', 64), + repeat('6', 64), + repeat('6', 64), + '{}', + 'PENDING', + '00000000-0000-4000-8000-000000000002', + '00000000-0000-4000-8000-000000000003', + 1 + ); + EXCEPTION + WHEN foreign_key_violation THEN + NULL; + END; + + IF EXISTS ( + SELECT 1 + FROM etl_job_records + WHERE job_record_id = '00000000-0000-4000-8000-000000000006' + ) THEN + RAISE EXCEPTION 'cross-owner source lineage was accepted'; + END IF; +END +$cross_owner_source_check$; + +DO $cross_owner_root_check$ +BEGIN + BEGIN + INSERT INTO etl_job_records ( + job_record_id, + principal_scope_hash, + submission_key_hash, + request_digest, + request_payload, + job_status, + replay_source_job_record_id, + replay_root_job_record_id, + replay_generation_count + ) VALUES ( + '00000000-0000-4000-8000-000000000007', + repeat('b', 64), + repeat('7', 64), + repeat('7', 64), + '{}', + 'PENDING', + '00000000-0000-4000-8000-000000000004', + '00000000-0000-4000-8000-000000000001', + 1 + ); + EXCEPTION + WHEN foreign_key_violation THEN + NULL; + END; + + IF EXISTS ( + SELECT 1 + FROM etl_job_records + WHERE job_record_id = '00000000-0000-4000-8000-000000000007' + ) THEN + RAISE EXCEPTION 'cross-owner root lineage was accepted'; + END IF; +END +$cross_owner_root_check$; + +DO $delete_restrict_check$ +BEGIN + BEGIN + DELETE FROM etl_job_records + WHERE job_record_id = '00000000-0000-4000-8000-000000000002'; + RAISE EXCEPTION 'ON DELETE RESTRICT did not protect replay history'; + EXCEPTION + WHEN foreign_key_violation THEN + NULL; + END; + + BEGIN + DELETE FROM etl_job_records + WHERE job_record_id = '00000000-0000-4000-8000-000000000001'; + RAISE EXCEPTION 'ON DELETE RESTRICT did not protect replay history'; + EXCEPTION + WHEN foreign_key_violation THEN + NULL; + END; +END +$delete_restrict_check$; + +ROLLBACK; + +BEGIN; + +ALTER TABLE etl_job_records DROP CONSTRAINT etl_job_replay_source_reference; +ALTER TABLE etl_job_records DROP CONSTRAINT etl_job_replay_root_reference; +ALTER TABLE etl_job_records DROP CONSTRAINT etl_job_replay_lineage_complete_check; +ALTER TABLE etl_job_records DROP CONSTRAINT etl_job_owner_identity_unique; +ALTER TABLE etl_job_records DROP COLUMN replay_source_job_record_id; +ALTER TABLE etl_job_records DROP COLUMN replay_root_job_record_id; +ALTER TABLE etl_job_records DROP COLUMN replay_generation_count; + +ROLLBACK; + +DO $rollback_restoration_check$ +DECLARE + restored_column_count integer; + restored_constraint_count integer; +BEGIN + SELECT count(*) + INTO restored_column_count + FROM information_schema.columns + WHERE table_schema = 'public' + AND table_name = 'etl_job_records' + AND column_name IN ( + 'replay_source_job_record_id', + 'replay_root_job_record_id', + 'replay_generation_count' + ); + + SELECT count(*) + INTO restored_constraint_count + FROM pg_constraint AS constraint_record + JOIN pg_class AS table_record + ON table_record.oid = constraint_record.conrelid + WHERE table_record.relname = 'etl_job_records' + AND constraint_record.conname IN ( + 'etl_job_replay_source_reference', + 'etl_job_replay_root_reference', + 'etl_job_replay_lineage_complete_check', + 'etl_job_owner_identity_unique' + ); + + IF restored_column_count <> 3 OR restored_constraint_count <> 4 THEN + RAISE EXCEPTION 'rollback rehearsal did not restore V7'; + END IF; +END +$rollback_restoration_check$; From 6682c56da5f444e428cf132fea9743ce36316074 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 14:26:00 +0900 Subject: [PATCH 045/103] fix(etl): execute replay lineage rehearsal --- scripts/verify-postgresql-migrations.sh | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/scripts/verify-postgresql-migrations.sh b/scripts/verify-postgresql-migrations.sh index 0c8b7a5f..742d1ad5 100644 --- a/scripts/verify-postgresql-migrations.sh +++ b/scripts/verify-postgresql-migrations.sh @@ -131,6 +131,15 @@ END $verification_block$; SQL +rehearsal_file="etl-service/src/test/postgresql/replay_lineage_migration.sql" +if [[ ! -f "${rehearsal_file}" ]]; then + printf 'Replay-lineage rehearsal not found: %s\n' "${rehearsal_file}" >&2 + exit 1 +fi + +printf 'Running %s\n' "${rehearsal_file}" +psql --no-psqlrc --set ON_ERROR_STOP=1 --file "${rehearsal_file}" >/dev/null + pg_dump --schema-only --no-owner --no-privileges > /tmp/mightyetl-postgresql-schema.sql if [[ ! -s /tmp/mightyetl-postgresql-schema.sql ]]; then printf 'Schema-only dump was empty.\n' >&2 From 75cf08fb71cc1b6bdbd04ee3831188de4a253d84 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 14:26:17 +0900 Subject: [PATCH 046/103] fix(ci): trigger replay lineage rehearsal changes --- .github/workflows/postgresql-migration-integration.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/postgresql-migration-integration.yml b/.github/workflows/postgresql-migration-integration.yml index 258d93ed..a1bd11b8 100644 --- a/.github/workflows/postgresql-migration-integration.yml +++ b/.github/workflows/postgresql-migration-integration.yml @@ -6,6 +6,7 @@ on: - develop paths: - "etl-service/src/main/resources/db/migration/**" + - "etl-service/src/test/postgresql/**" - "scripts/verify-postgresql-migrations.sh" - ".github/workflows/postgresql-migration-integration.yml" workflow_dispatch: From 7fd11da03924065f7a83b746946c49a633dc3b7f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 14:26:39 +0900 Subject: [PATCH 047/103] fix(test): align replay migration contract --- .../EtlJobReplayPostgresWorkflowTest.java | 49 +++++++++++++------ 1 file changed, 33 insertions(+), 16 deletions(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/documentation/EtlJobReplayPostgresWorkflowTest.java b/etl-service/src/test/java/com/xtrmetl/etl/documentation/EtlJobReplayPostgresWorkflowTest.java index 18e77415..787c3e5f 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/documentation/EtlJobReplayPostgresWorkflowTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/documentation/EtlJobReplayPostgresWorkflowTest.java @@ -12,24 +12,33 @@ /** * Keeps the production-PostgreSQL replay migration and rollback rehearsal executable in CI. + * + *

The ordinary cross-platform Java build validates these repository contracts without + * requiring PostgreSQL on every runner. The separate direct-{@code develop} integration + * workflow then executes the same verifier against PostgreSQL 18.

*/ class EtlJobReplayPostgresWorkflowTest { @Test - void ciRunsTheCompleteReplayMigrationChainAgainstPostgres18() throws IOException { - String workflow = read(".github/workflows/ci.yml").replaceAll("\\s+", " "); + void directDevelopWorkflowRunsCompleteReplayMigrationChainOnPostgres18() throws IOException { + String workflow = read( + ".github/workflows/postgresql-migration-integration.yml" + ).replaceAll("\\s+", " "); + String verifier = read( + "scripts/verify-postgresql-migrations.sh" + ).replaceAll("\\s+", " "); + + assertTrue(workflow.contains("replay_lineage_migration:")); + assertTrue(workflow.contains("image: postgres:18-alpine")); + assertTrue(workflow.contains("--health-cmd \"pg_isready")); + assertTrue(workflow.contains("etl-service/src/test/postgresql/**")); + assertTrue(workflow.contains("bash scripts/verify-postgresql-migrations.sh")); - assertTrue(workflow.contains("postgres_replay_migration:")); - assertTrue(workflow.contains("image: postgres:18")); - assertTrue(workflow.contains("--health-cmd pg_isready")); - assertTrue(workflow.contains("V2__create_etl_job_records.sql")); - assertTrue(workflow.contains("V3__add_etl_job_lease_fencing.sql")); - assertTrue(workflow.contains("V4__add_etl_job_claim_eligibility_index.sql")); - assertTrue(workflow.contains("V5__add_etl_job_owner_pagination_index.sql")); - assertTrue(workflow.contains("V6__add_etl_job_cancellation.sql")); - assertTrue(workflow.contains("V7__add_etl_job_replay_lineage.sql")); - assertTrue(workflow.contains("psql -v ON_ERROR_STOP=1")); - assertTrue(workflow.contains( + assertTrue(verifier.contains("find \"${migration_directory}\"")); + assertTrue(verifier.contains("-name 'V*__*.sql'")); + assertTrue(verifier.contains("sort -zV")); + assertTrue(verifier.contains("psql --no-psqlrc --set ON_ERROR_STOP=1")); + assertTrue(verifier.contains( "etl-service/src/test/postgresql/replay_lineage_migration.sql" )); } @@ -43,8 +52,12 @@ void postgresRehearsalCoversTenantIntegrityDeletionAndRollback() throws IOExcept assertTrue(rehearsal.contains("cross-owner source lineage was accepted")); assertTrue(rehearsal.contains("cross-owner root lineage was accepted")); assertTrue(rehearsal.contains("ON DELETE RESTRICT did not protect replay history")); - assertTrue(rehearsal.contains("ALTER TABLE etl_job_records DROP CONSTRAINT etl_job_replay_source_reference")); - assertTrue(rehearsal.contains("ALTER TABLE etl_job_records DROP COLUMN replay_source_job_record_id")); + assertTrue(rehearsal.contains( + "ALTER TABLE etl_job_records DROP CONSTRAINT etl_job_replay_source_reference" + )); + assertTrue(rehearsal.contains( + "ALTER TABLE etl_job_records DROP COLUMN replay_source_job_record_id" + )); assertTrue(rehearsal.contains("ROLLBACK")); assertTrue(rehearsal.contains("rollback rehearsal did not restore V7")); } @@ -53,7 +66,11 @@ private static String read(String relativePath) throws IOException { return Files.readString(projectRoot().resolve(relativePath), StandardCharsets.UTF_8); } - /** @return repository root from reactor-root or module-local execution */ + /** + * Finds the repository root from reactor-root or module-local Maven execution. + * + * @return repository root that contains workflows, scripts, and test fixtures + */ private static Path projectRoot() { Path current = Paths.get(System.getProperty("user.dir")).toAbsolutePath(); Path lastPomParent = null; From 798d46f5dd4c805e4dcf09400fbc1f2893b188d1 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 14:28:28 +0900 Subject: [PATCH 048/103] fix(test): satisfy terminal lifecycle constraints --- .../postgresql/replay_lineage_migration.sql | 74 +++++++++++-------- 1 file changed, 44 insertions(+), 30 deletions(-) diff --git a/etl-service/src/test/postgresql/replay_lineage_migration.sql b/etl-service/src/test/postgresql/replay_lineage_migration.sql index 2c50aa33..f558aae0 100644 --- a/etl-service/src/test/postgresql/replay_lineage_migration.sql +++ b/etl-service/src/test/postgresql/replay_lineage_migration.sql @@ -19,14 +19,6 @@ INSERT INTO etl_job_records ( '{}', 'PENDING' ), - ( - '00000000-0000-4000-8000-000000000002', - repeat('a', 64), - repeat('2', 64), - repeat('b', 64), - NULL, - 'FAILED' - ), ( '00000000-0000-4000-8000-000000000003', repeat('b', 64), @@ -34,25 +26,47 @@ INSERT INTO etl_job_records ( repeat('c', 64), '{}', 'PENDING' - ), - ( - '00000000-0000-4000-8000-000000000004', - repeat('b', 64), - repeat('4', 64), - repeat('d', 64), - NULL, - 'CANCELLED' ); -UPDATE etl_job_records -SET failure_code = 'etl_replay_source_failed' -WHERE job_record_id = '00000000-0000-4000-8000-000000000002'; +INSERT INTO etl_job_records ( + job_record_id, + principal_scope_hash, + submission_key_hash, + request_digest, + request_payload, + job_status, + failure_code +) VALUES ( + '00000000-0000-4000-8000-000000000002', + repeat('a', 64), + repeat('2', 64), + repeat('b', 64), + NULL, + 'FAILED', + 'etl_replay_source_failed' +); -UPDATE etl_job_records -SET cancellation_key_hash = repeat('e', 64), - cancellation_code = 'etl_job_cancelled_by_owner', - job_cancelled_at = CURRENT_TIMESTAMP -WHERE job_record_id = '00000000-0000-4000-8000-000000000004'; +INSERT INTO etl_job_records ( + job_record_id, + principal_scope_hash, + submission_key_hash, + request_digest, + request_payload, + job_status, + cancellation_key_hash, + cancellation_code, + job_cancelled_at +) VALUES ( + '00000000-0000-4000-8000-000000000004', + repeat('b', 64), + repeat('4', 64), + repeat('d', 64), + NULL, + 'CANCELLED', + repeat('e', 64), + 'etl_job_cancelled_by_owner', + CURRENT_TIMESTAMP +); INSERT INTO etl_job_records ( job_record_id, @@ -107,8 +121,8 @@ BEGIN IF EXISTS ( SELECT 1 - FROM etl_job_records - WHERE job_record_id = '00000000-0000-4000-8000-000000000006' + FROM etl_job_records + WHERE job_record_id = '00000000-0000-4000-8000-000000000006' ) THEN RAISE EXCEPTION 'cross-owner source lineage was accepted'; END IF; @@ -146,8 +160,8 @@ BEGIN IF EXISTS ( SELECT 1 - FROM etl_job_records - WHERE job_record_id = '00000000-0000-4000-8000-000000000007' + FROM etl_job_records + WHERE job_record_id = '00000000-0000-4000-8000-000000000007' ) THEN RAISE EXCEPTION 'cross-owner root lineage was accepted'; END IF; @@ -158,7 +172,7 @@ DO $delete_restrict_check$ BEGIN BEGIN DELETE FROM etl_job_records - WHERE job_record_id = '00000000-0000-4000-8000-000000000002'; + WHERE job_record_id = '00000000-0000-4000-8000-000000000002'; RAISE EXCEPTION 'ON DELETE RESTRICT did not protect replay history'; EXCEPTION WHEN foreign_key_violation THEN @@ -167,7 +181,7 @@ BEGIN BEGIN DELETE FROM etl_job_records - WHERE job_record_id = '00000000-0000-4000-8000-000000000001'; + WHERE job_record_id = '00000000-0000-4000-8000-000000000001'; RAISE EXCEPTION 'ON DELETE RESTRICT did not protect replay history'; EXCEPTION WHEN foreign_key_violation THEN From 73d08ac1b4bd7868ebf57d627a55367751a7205e Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 14:35:10 +0900 Subject: [PATCH 049/103] fix(test): normalize workflow line endings --- ...ostgresqlMigrationIntegrationWorkflowTest.java | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/documentation/PostgresqlMigrationIntegrationWorkflowTest.java b/etl-service/src/test/java/com/xtrmetl/etl/documentation/PostgresqlMigrationIntegrationWorkflowTest.java index a971eab0..94173a09 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/documentation/PostgresqlMigrationIntegrationWorkflowTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/documentation/PostgresqlMigrationIntegrationWorkflowTest.java @@ -56,8 +56,21 @@ void verificationScriptAppliesEveryMigrationAndChecksReplayConstraints() throws assertFalse(script.contains("|| true")); } + /** + * Reads one repository contract with platform-independent line endings. + * + *

Git may materialize text files with CRLF on Windows runners. Normalizing both CRLF + * and lone carriage returns keeps semantic workflow assertions identical across the CI + * operating-system matrix without weakening their exact content requirements.

+ * + * @param relativePath repository-relative file path + * @return UTF-8 content using LF line endings + * @throws IOException when the repository contract cannot be read + */ private static String read(String relativePath) throws IOException { - return Files.readString(projectRoot().resolve(relativePath), StandardCharsets.UTF_8); + return Files.readString(projectRoot().resolve(relativePath), StandardCharsets.UTF_8) + .replace("\r\n", "\n") + .replace('\r', '\n'); } /** From f45304d0c8a872246ff09b86651a520b052b5929 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:07:57 +0900 Subject: [PATCH 050/103] test(etl): require database-owned replay lineage continuity --- .../etl/job/EtlJobReplayMigrationTest.java | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java index 6f170958..56d46f23 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java @@ -54,6 +54,23 @@ void replayMigrationAddsCompleteRestrictedOwnerScopedLineage() throws IOExceptio assertTrue(migration.contains("replay_generation_count BETWEEN 1 AND 100")); assertTrue(migration.contains("replay_source_job_record_id <> job_record_id")); assertTrue(migration.contains("replay_root_job_record_id <> job_record_id")); + assertTrue(migration.contains( + "CREATE FUNCTION validate_etl_job_replay_lineage() RETURNS trigger" + )); + assertTrue(migration.contains( + "CREATE TRIGGER etl_job_replay_lineage_guard_trigger" + )); + assertTrue(migration.contains( + "BEFORE INSERT OR UPDATE OF replay_source_job_record_id, " + + "replay_root_job_record_id, replay_generation_count" + )); + assertTrue(migration.contains( + "NEW.replay_source_job_record_id <> NEW.replay_root_job_record_id" + )); + assertTrue(migration.contains( + "source_generation_count IS DISTINCT FROM NEW.replay_generation_count - 1" + )); + assertTrue(migration.contains("Replay lineage fields are immutable")); assertFalse(migration.contains( "FOREIGN KEY (replay_source_job_record_id) " + "REFERENCES etl_job_records (job_record_id)" From 15e68913115094b6fd36907ed547847201ce956c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:09:06 +0900 Subject: [PATCH 051/103] test(etl): reject replay rows that inherit a non-root root --- .../job/EtlJobReplayLineageIntegrityTest.java | 188 ++++++++++++++++++ 1 file changed, 188 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayLineageIntegrityTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayLineageIntegrityTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayLineageIntegrityTest.java new file mode 100644 index 00000000..c9afdc7e --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayLineageIntegrityTest.java @@ -0,0 +1,188 @@ +package com.xtrmetl.etl.job; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.xtrmetl.etl.service.EtlBatchProperties; +import com.xtrmetl.etl.service.EtlRequestLock; +import com.xtrmetl.etl.service.Sha256Digest; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.jdbc.datasource.DataSourceTransactionManager; +import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseBuilder; +import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseType; +import org.springframework.test.context.junit.jupiter.SpringJUnitConfig; +import org.springframework.transaction.PlatformTransactionManager; +import org.springframework.transaction.annotation.EnableTransactionManagement; + +import javax.sql.DataSource; +import java.util.UUID; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; + +/** + * Proves replay admission rejects inherited roots that are themselves replay rows. + * + *

This is a defense-in-depth contract for repositories or migrations that import historical + * rows before PostgreSQL's lineage trigger is available. A replay root must be the immutable first + * job in the lineage, never another derived replay row.

+ */ +@SpringJUnitConfig(EtlJobReplayLineageIntegrityTest.TestConfiguration.class) +class EtlJobReplayLineageIntegrityTest { + + private static final String PAYLOAD = "[{\"id\":\"record_alpha\"}]"; + private static final String REPLAY_KEY = "94ccf28c-9649-4a06-b06f-11e70c57c5d2"; + private static final String PRINCIPAL_SCOPE = "tenant_alpha"; + + private final EtlJobReplayService replayService; + private final JdbcTemplate jdbcTemplate; + + @Autowired + EtlJobReplayLineageIntegrityTest( + EtlJobReplayService replayService, + JdbcTemplate jdbcTemplate + ) { + this.replayService = replayService; + this.jdbcTemplate = jdbcTemplate; + } + + @BeforeEach + void createJobTable() { + jdbcTemplate.execute("DROP TABLE IF EXISTS etl_job_records"); + jdbcTemplate.execute(""" + CREATE TABLE etl_job_records ( + job_record_id UUID PRIMARY KEY, + principal_scope_hash CHAR(64) NOT NULL, + submission_key_hash CHAR(64) NOT NULL, + request_digest CHAR(64) NOT NULL, + request_payload CLOB, + job_status VARCHAR(32) NOT NULL, + attempt_count INTEGER NOT NULL DEFAULT 0, + failure_code VARCHAR(128), + replay_source_job_record_id UUID, + replay_root_job_record_id UUID, + replay_generation_count INTEGER, + CONSTRAINT etl_job_submission_scope_unique + UNIQUE (principal_scope_hash, submission_key_hash) + ) + """); + } + + @Test + void rejectsAnInheritedRootThatIsItselfAReplay() { + UUID rootJobRecordId = insertFailedJob(null, null, null); + UUID generationOneJobRecordId = insertFailedJob( + rootJobRecordId, + rootJobRecordId, + 1 + ); + UUID malformedGenerationTwoJobRecordId = insertFailedJob( + generationOneJobRecordId, + generationOneJobRecordId, + 2 + ); + + IllegalStateException exception = assertThrows( + IllegalStateException.class, + () -> replayService.replayOwned( + malformedGenerationTwoJobRecordId, + PAYLOAD, + REPLAY_KEY, + PRINCIPAL_SCOPE + ) + ); + + assertEquals("Replay root is not a lineage root", exception.getMessage()); + } + + private UUID insertFailedJob( + UUID replaySourceJobRecordId, + UUID replayRootJobRecordId, + Integer replayGenerationCount + ) { + UUID jobRecordId = UUID.randomUUID(); + jdbcTemplate.update( + """ + INSERT INTO etl_job_records ( + job_record_id, + principal_scope_hash, + submission_key_hash, + request_digest, + request_payload, + job_status, + attempt_count, + failure_code, + replay_source_job_record_id, + replay_root_job_record_id, + replay_generation_count + ) VALUES (?, ?, ?, ?, NULL, 'FAILED', 0, ?, ?, ?, ?) + """, + jobRecordId, + Sha256Digest.digest(PRINCIPAL_SCOPE), + Sha256Digest.digest(UUID.randomUUID().toString()), + Sha256Digest.digest(PAYLOAD), + "etl_target_failure", + replaySourceJobRecordId, + replayRootJobRecordId, + replayGenerationCount + ); + return jobRecordId; + } + + /** Minimal transaction-enabled context for replay-lineage integrity verification. */ + @Configuration + @EnableTransactionManagement + static class TestConfiguration { + + @Bean + DataSource dataSource() { + return new EmbeddedDatabaseBuilder() + .generateUniqueName(true) + .setType(EmbeddedDatabaseType.H2) + .build(); + } + + @Bean + JdbcTemplate jdbcTemplate(DataSource dataSource) { + return new JdbcTemplate(dataSource); + } + + @Bean + PlatformTransactionManager transactionManager(DataSource dataSource) { + return new DataSourceTransactionManager(dataSource); + } + + @Bean + ObjectMapper objectMapper() { + return new ObjectMapper(); + } + + @Bean + EtlBatchProperties etlBatchProperties() { + return new EtlBatchProperties(); + } + + @Bean + EtlRequestLock etlRequestLock() { + return lockHash -> true; + } + + @Bean + EtlJobReplayService replayService( + JdbcTemplate jdbcTemplate, + ObjectMapper objectMapper, + EtlBatchProperties batchProperties, + EtlRequestLock requestLock + ) { + return new EtlJobReplayService( + jdbcTemplate, + objectMapper, + batchProperties, + requestLock + ); + } + } +} From 2057ae86f867f8a16ac8e51e08609cddce3545eb Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:12:16 +0900 Subject: [PATCH 052/103] fix(etl): enforce immutable replay lineage continuity --- .../V7__add_etl_job_replay_lineage.sql | 118 ++++++++++++++++++ 1 file changed, 118 insertions(+) diff --git a/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql b/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql index cebdcbfc..0d83bd40 100644 --- a/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql +++ b/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql @@ -30,3 +30,121 @@ ALTER TABLE etl_job_records AND replay_root_job_record_id <> job_record_id ) ); + +-- Keep the relational lineage authoritative even when rows are imported outside the service. +-- Each derived row must point to an exact terminal predecessor, preserve the first root, and +-- advance the generation by exactly one. Lineage columns become immutable after insertion so a +-- later update cannot silently rewrite descendants' provenance. +CREATE FUNCTION validate_etl_job_replay_lineage() RETURNS trigger +LANGUAGE plpgsql +AS $etl_job_replay_lineage$ +DECLARE + source_job_status VARCHAR(32); + source_source_job_record_id UUID; + source_root_job_record_id UUID; + source_generation_count INTEGER; + root_job_status VARCHAR(32); + root_source_job_record_id UUID; + root_root_job_record_id UUID; + root_generation_count INTEGER; +BEGIN + IF TG_OP = 'UPDATE' + AND ( + OLD.replay_source_job_record_id + IS DISTINCT FROM NEW.replay_source_job_record_id + OR OLD.replay_root_job_record_id + IS DISTINCT FROM NEW.replay_root_job_record_id + OR OLD.replay_generation_count + IS DISTINCT FROM NEW.replay_generation_count + ) THEN + RAISE EXCEPTION 'Replay lineage fields are immutable' + USING ERRCODE = '23514'; + END IF; + + IF NEW.replay_generation_count IS NULL THEN + RETURN NEW; + END IF; + + IF TG_OP = 'INSERT' + AND ( + NEW.job_status <> 'PENDING' + OR NEW.attempt_count <> 0 + OR NEW.request_payload IS NULL + ) THEN + RAISE EXCEPTION 'Replay rows must start as pending jobs' + USING ERRCODE = '23514'; + END IF; + + SELECT job_status, + replay_source_job_record_id, + replay_root_job_record_id, + replay_generation_count + INTO source_job_status, + source_source_job_record_id, + source_root_job_record_id, + source_generation_count + FROM etl_job_records + WHERE job_record_id = NEW.replay_source_job_record_id + AND principal_scope_hash = NEW.principal_scope_hash + FOR KEY SHARE; + + IF NOT FOUND THEN + RAISE EXCEPTION 'Replay source is missing or belongs to another owner' + USING ERRCODE = '23514'; + END IF; + + IF source_job_status NOT IN ('FAILED', 'CANCELLED') THEN + RAISE EXCEPTION 'Replay source must be failed or cancelled' + USING ERRCODE = '23514'; + END IF; + + SELECT job_status, + replay_source_job_record_id, + replay_root_job_record_id, + replay_generation_count + INTO root_job_status, + root_source_job_record_id, + root_root_job_record_id, + root_generation_count + FROM etl_job_records + WHERE job_record_id = NEW.replay_root_job_record_id + AND principal_scope_hash = NEW.principal_scope_hash + FOR KEY SHARE; + + IF NOT FOUND THEN + RAISE EXCEPTION 'Replay root is missing or belongs to another owner' + USING ERRCODE = '23514'; + END IF; + + IF root_job_status NOT IN ('FAILED', 'CANCELLED') + OR root_source_job_record_id IS NOT NULL + OR root_root_job_record_id IS NOT NULL + OR root_generation_count IS NOT NULL THEN + RAISE EXCEPTION 'Replay root is not a lineage root' + USING ERRCODE = '23514'; + END IF; + + IF NEW.replay_generation_count = 1 THEN + IF NEW.replay_source_job_record_id <> NEW.replay_root_job_record_id THEN + RAISE EXCEPTION 'Generation one must reference the same source and root' + USING ERRCODE = '23514'; + END IF; + ELSIF source_source_job_record_id IS NULL + OR source_root_job_record_id + IS DISTINCT FROM NEW.replay_root_job_record_id + OR source_generation_count + IS DISTINCT FROM NEW.replay_generation_count - 1 THEN + RAISE EXCEPTION 'Replay generation does not follow the immediate source' + USING ERRCODE = '23514'; + END IF; + + RETURN NEW; +END; +$etl_job_replay_lineage$; + +CREATE TRIGGER etl_job_replay_lineage_guard_trigger +BEFORE INSERT OR UPDATE OF replay_source_job_record_id, + replay_root_job_record_id, replay_generation_count +ON etl_job_records +FOR EACH ROW +EXECUTE FUNCTION validate_etl_job_replay_lineage(); From 50bc90b008e9babeb27e62f228836f2255847207 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:17:15 +0900 Subject: [PATCH 053/103] fix(etl): reject derived rows as replay roots --- .../xtrmetl/etl/job/EtlJobReplayService.java | 23 +++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplayService.java b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplayService.java index ff488095..0f3a15e4 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplayService.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplayService.java @@ -72,6 +72,15 @@ SELECT COUNT(*) WHERE job_record_id = ? AND principal_scope_hash = ? """; + private static final String SELECT_LINEAGE_ROOT_COUNT_SQL = """ + SELECT COUNT(*) + FROM etl_job_records + WHERE job_record_id = ? + AND principal_scope_hash = ? + AND replay_source_job_record_id IS NULL + AND replay_root_job_record_id IS NULL + AND replay_generation_count IS NULL + """; private static final String INSERT_REPLAY_JOB_SQL = """ INSERT INTO etl_job_records ( job_record_id, @@ -291,15 +300,25 @@ private ReplaySource findSource(UUID sourceJobRecordId, String principalScopeHas } private void requireOwnedRoot(UUID rootJobRecordId, String principalScopeHash) { - Integer count = jdbcTemplate.queryForObject( + Integer ownedRootCount = jdbcTemplate.queryForObject( SELECT_OWNED_ROOT_COUNT_SQL, Integer.class, rootJobRecordId, principalScopeHash ); - if (!Integer.valueOf(1).equals(count)) { + if (!Integer.valueOf(1).equals(ownedRootCount)) { throw new IllegalStateException("Replay root is absent from the owner namespace"); } + + Integer lineageRootCount = jdbcTemplate.queryForObject( + SELECT_LINEAGE_ROOT_COUNT_SQL, + Integer.class, + rootJobRecordId, + principalScopeHash + ); + if (!Integer.valueOf(1).equals(lineageRootCount)) { + throw new IllegalStateException("Replay root is not a lineage root"); + } } private String validatePayload(@Nullable String requestPayload) { From 92a8ad6b735d17e0a01ede207cbdb6babf32c493 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:19:46 +0900 Subject: [PATCH 054/103] test(etl): rehearse replay lineage continuity in PostgreSQL --- .../postgresql/replay_lineage_migration.sql | 276 +++++++++++++++++- 1 file changed, 265 insertions(+), 11 deletions(-) diff --git a/etl-service/src/test/postgresql/replay_lineage_migration.sql b/etl-service/src/test/postgresql/replay_lineage_migration.sql index f558aae0..c0b12a03 100644 --- a/etl-service/src/test/postgresql/replay_lineage_migration.sql +++ b/etl-service/src/test/postgresql/replay_lineage_migration.sql @@ -1,6 +1,31 @@ --- Rehearse V7 owner isolation, immutable lineage, deletion protection, and rollback. +-- Rehearse V7 owner isolation, exact lineage continuity, deletion protection, and rollback. -- This script runs only against the disposable PostgreSQL integration-test database. +DO $migration_object_check$ +DECLARE + trigger_count integer; + function_count integer; +BEGIN + SELECT count(*) + INTO trigger_count + FROM pg_trigger AS trigger_record + JOIN pg_class AS table_record + ON table_record.oid = trigger_record.tgrelid + WHERE table_record.relname = 'etl_job_records' + AND trigger_record.tgname = 'etl_job_replay_lineage_guard_trigger' + AND NOT trigger_record.tgisinternal; + + SELECT count(*) + INTO function_count + FROM pg_proc AS function_record + WHERE function_record.proname = 'validate_etl_job_replay_lineage'; + + IF trigger_count <> 1 OR function_count <> 1 THEN + RAISE EXCEPTION 'replay lineage trigger or function is missing'; + END IF; +END +$migration_object_check$; + BEGIN; INSERT INTO etl_job_records ( @@ -68,6 +93,7 @@ INSERT INTO etl_job_records ( CURRENT_TIMESTAMP ); +-- Create a valid first-generation replay, then terminalize it through the ordinary lifecycle. INSERT INTO etl_job_records ( job_record_id, principal_scope_hash, @@ -86,10 +112,195 @@ INSERT INTO etl_job_records ( '{}', 'PENDING', '00000000-0000-4000-8000-000000000002', - '00000000-0000-4000-8000-000000000001', + '00000000-0000-4000-8000-000000000002', 1 ); +UPDATE etl_job_records + SET job_status = 'FAILED', + request_payload = NULL, + failure_code = 'etl_replay_generation_failed' + WHERE job_record_id = '00000000-0000-4000-8000-000000000005'; + +-- A valid second generation must retain the first root and advance exactly once. +INSERT INTO etl_job_records ( + job_record_id, + principal_scope_hash, + submission_key_hash, + request_digest, + request_payload, + job_status, + replay_source_job_record_id, + replay_root_job_record_id, + replay_generation_count +) VALUES ( + '00000000-0000-4000-8000-000000000008', + repeat('a', 64), + repeat('8', 64), + repeat('8', 64), + '{}', + 'PENDING', + '00000000-0000-4000-8000-000000000005', + '00000000-0000-4000-8000-000000000002', + 2 +); + +DO $nonterminal_source_check$ +BEGIN + BEGIN + INSERT INTO etl_job_records ( + job_record_id, + principal_scope_hash, + submission_key_hash, + request_digest, + request_payload, + job_status, + replay_source_job_record_id, + replay_root_job_record_id, + replay_generation_count + ) VALUES ( + '00000000-0000-4000-8000-000000000009', + repeat('a', 64), + repeat('9', 64), + repeat('9', 64), + '{}', + 'PENDING', + '00000000-0000-4000-8000-000000000001', + '00000000-0000-4000-8000-000000000001', + 1 + ); + EXCEPTION + WHEN check_violation OR foreign_key_violation THEN + NULL; + END; + + IF EXISTS ( + SELECT 1 + FROM etl_job_records + WHERE job_record_id = '00000000-0000-4000-8000-000000000009' + ) THEN + RAISE EXCEPTION 'nonterminal replay source was accepted'; + END IF; +END +$nonterminal_source_check$; + +DO $generation_one_root_check$ +BEGIN + BEGIN + INSERT INTO etl_job_records ( + job_record_id, + principal_scope_hash, + submission_key_hash, + request_digest, + request_payload, + job_status, + replay_source_job_record_id, + replay_root_job_record_id, + replay_generation_count + ) VALUES ( + '00000000-0000-4000-8000-000000000010', + repeat('a', 64), + repeat('a', 64), + repeat('a', 64), + '{}', + 'PENDING', + '00000000-0000-4000-8000-000000000002', + '00000000-0000-4000-8000-000000000001', + 1 + ); + EXCEPTION + WHEN check_violation OR foreign_key_violation THEN + NULL; + END; + + IF EXISTS ( + SELECT 1 + FROM etl_job_records + WHERE job_record_id = '00000000-0000-4000-8000-000000000010' + ) THEN + RAISE EXCEPTION 'generation-one replay accepted a different root'; + END IF; +END +$generation_one_root_check$; + +DO $derived_root_check$ +BEGIN + BEGIN + INSERT INTO etl_job_records ( + job_record_id, + principal_scope_hash, + submission_key_hash, + request_digest, + request_payload, + job_status, + replay_source_job_record_id, + replay_root_job_record_id, + replay_generation_count + ) VALUES ( + '00000000-0000-4000-8000-000000000011', + repeat('a', 64), + repeat('b', 64), + repeat('b', 64), + '{}', + 'PENDING', + '00000000-0000-4000-8000-000000000005', + '00000000-0000-4000-8000-000000000005', + 2 + ); + EXCEPTION + WHEN check_violation OR foreign_key_violation THEN + NULL; + END; + + IF EXISTS ( + SELECT 1 + FROM etl_job_records + WHERE job_record_id = '00000000-0000-4000-8000-000000000011' + ) THEN + RAISE EXCEPTION 'a derived replay row was accepted as lineage root'; + END IF; +END +$derived_root_check$; + +DO $skipped_generation_check$ +BEGIN + BEGIN + INSERT INTO etl_job_records ( + job_record_id, + principal_scope_hash, + submission_key_hash, + request_digest, + request_payload, + job_status, + replay_source_job_record_id, + replay_root_job_record_id, + replay_generation_count + ) VALUES ( + '00000000-0000-4000-8000-000000000012', + repeat('a', 64), + repeat('c', 64), + repeat('c', 64), + '{}', + 'PENDING', + '00000000-0000-4000-8000-000000000005', + '00000000-0000-4000-8000-000000000002', + 3 + ); + EXCEPTION + WHEN check_violation OR foreign_key_violation THEN + NULL; + END; + + IF EXISTS ( + SELECT 1 + FROM etl_job_records + WHERE job_record_id = '00000000-0000-4000-8000-000000000012' + ) THEN + RAISE EXCEPTION 'a skipped replay generation was accepted'; + END IF; +END +$skipped_generation_check$; + DO $cross_owner_source_check$ BEGIN BEGIN @@ -111,11 +322,11 @@ BEGIN '{}', 'PENDING', '00000000-0000-4000-8000-000000000002', - '00000000-0000-4000-8000-000000000003', + '00000000-0000-4000-8000-000000000004', 1 ); EXCEPTION - WHEN foreign_key_violation THEN + WHEN foreign_key_violation OR check_violation THEN NULL; END; @@ -150,11 +361,11 @@ BEGIN '{}', 'PENDING', '00000000-0000-4000-8000-000000000004', - '00000000-0000-4000-8000-000000000001', + '00000000-0000-4000-8000-000000000002', 1 ); EXCEPTION - WHEN foreign_key_violation THEN + WHEN foreign_key_violation OR check_violation THEN NULL; END; @@ -168,12 +379,34 @@ BEGIN END $cross_owner_root_check$; +DO $lineage_immutability_check$ +BEGIN + BEGIN + UPDATE etl_job_records + SET replay_root_job_record_id = '00000000-0000-4000-8000-000000000001' + WHERE job_record_id = '00000000-0000-4000-8000-000000000005'; + EXCEPTION + WHEN check_violation THEN + NULL; + END; + + IF EXISTS ( + SELECT 1 + FROM etl_job_records + WHERE job_record_id = '00000000-0000-4000-8000-000000000005' + AND replay_root_job_record_id = '00000000-0000-4000-8000-000000000001' + ) THEN + RAISE EXCEPTION 'replay lineage fields were mutable'; + END IF; +END +$lineage_immutability_check$; + DO $delete_restrict_check$ BEGIN BEGIN DELETE FROM etl_job_records - WHERE job_record_id = '00000000-0000-4000-8000-000000000002'; - RAISE EXCEPTION 'ON DELETE RESTRICT did not protect replay history'; + WHERE job_record_id = '00000000-0000-4000-8000-000000000005'; + RAISE EXCEPTION 'ON DELETE RESTRICT did not protect immediate replay history'; EXCEPTION WHEN foreign_key_violation THEN NULL; @@ -181,8 +414,8 @@ BEGIN BEGIN DELETE FROM etl_job_records - WHERE job_record_id = '00000000-0000-4000-8000-000000000001'; - RAISE EXCEPTION 'ON DELETE RESTRICT did not protect replay history'; + WHERE job_record_id = '00000000-0000-4000-8000-000000000002'; + RAISE EXCEPTION 'ON DELETE RESTRICT did not protect replay root history'; EXCEPTION WHEN foreign_key_violation THEN NULL; @@ -194,6 +427,8 @@ ROLLBACK; BEGIN; +DROP TRIGGER etl_job_replay_lineage_guard_trigger ON etl_job_records; +DROP FUNCTION validate_etl_job_replay_lineage(); ALTER TABLE etl_job_records DROP CONSTRAINT etl_job_replay_source_reference; ALTER TABLE etl_job_records DROP CONSTRAINT etl_job_replay_root_reference; ALTER TABLE etl_job_records DROP CONSTRAINT etl_job_replay_lineage_complete_check; @@ -208,6 +443,8 @@ DO $rollback_restoration_check$ DECLARE restored_column_count integer; restored_constraint_count integer; + restored_trigger_count integer; + restored_function_count integer; BEGIN SELECT count(*) INTO restored_column_count @@ -233,7 +470,24 @@ BEGIN 'etl_job_owner_identity_unique' ); - IF restored_column_count <> 3 OR restored_constraint_count <> 4 THEN + SELECT count(*) + INTO restored_trigger_count + FROM pg_trigger AS trigger_record + JOIN pg_class AS table_record + ON table_record.oid = trigger_record.tgrelid + WHERE table_record.relname = 'etl_job_records' + AND trigger_record.tgname = 'etl_job_replay_lineage_guard_trigger' + AND NOT trigger_record.tgisinternal; + + SELECT count(*) + INTO restored_function_count + FROM pg_proc AS function_record + WHERE function_record.proname = 'validate_etl_job_replay_lineage'; + + IF restored_column_count <> 3 + OR restored_constraint_count <> 4 + OR restored_trigger_count <> 1 + OR restored_function_count <> 1 THEN RAISE EXCEPTION 'rollback rehearsal did not restore V7'; END IF; END From 896ba5044685a82cabe95c11902f7b2dc0de7ae6 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:24:06 +0900 Subject: [PATCH 055/103] test(etl): align PostgreSQL rehearsal contract with lineage guard --- .../EtlJobReplayPostgresWorkflowTest.java | 22 +++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/documentation/EtlJobReplayPostgresWorkflowTest.java b/etl-service/src/test/java/com/xtrmetl/etl/documentation/EtlJobReplayPostgresWorkflowTest.java index 787c3e5f..4c3923c4 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/documentation/EtlJobReplayPostgresWorkflowTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/documentation/EtlJobReplayPostgresWorkflowTest.java @@ -44,14 +44,32 @@ void directDevelopWorkflowRunsCompleteReplayMigrationChainOnPostgres18() throws } @Test - void postgresRehearsalCoversTenantIntegrityDeletionAndRollback() throws IOException { + void postgresRehearsalCoversLineageContinuityTenantIntegrityAndRollback() + throws IOException { String rehearsal = read( "etl-service/src/test/postgresql/replay_lineage_migration.sql" ).replaceAll("\\s+", " "); + assertTrue(rehearsal.contains("replay lineage trigger or function is missing")); + assertTrue(rehearsal.contains("nonterminal replay source was accepted")); + assertTrue(rehearsal.contains("generation-one replay accepted a different root")); + assertTrue(rehearsal.contains("a derived replay row was accepted as lineage root")); + assertTrue(rehearsal.contains("a skipped replay generation was accepted")); assertTrue(rehearsal.contains("cross-owner source lineage was accepted")); assertTrue(rehearsal.contains("cross-owner root lineage was accepted")); - assertTrue(rehearsal.contains("ON DELETE RESTRICT did not protect replay history")); + assertTrue(rehearsal.contains("replay lineage fields were mutable")); + assertTrue(rehearsal.contains( + "ON DELETE RESTRICT did not protect immediate replay history" + )); + assertTrue(rehearsal.contains( + "ON DELETE RESTRICT did not protect replay root history" + )); + assertTrue(rehearsal.contains( + "DROP TRIGGER etl_job_replay_lineage_guard_trigger ON etl_job_records" + )); + assertTrue(rehearsal.contains( + "DROP FUNCTION validate_etl_job_replay_lineage()" + )); assertTrue(rehearsal.contains( "ALTER TABLE etl_job_records DROP CONSTRAINT etl_job_replay_source_reference" )); From 1a796ba9ecb93422cbed72b1bee7c18aaa5688f5 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:26:28 +0900 Subject: [PATCH 056/103] test(etl): require documented database lineage authority --- .../DurableJobReplayDocumentationTest.java | 35 ++++++++++++++----- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java index 380bc9b7..2ede03eb 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java @@ -30,6 +30,10 @@ void operationsRunbookDocumentsAdmissionLineageAndRollback() throws IOException assertTrue(runbook.contains("replay_root_job_record_id")); assertTrue(runbook.contains("replay_generation_count")); assertTrue(runbook.contains("etl_job_owner_identity_unique")); + assertTrue(runbook.contains("etl_job_replay_lineage_guard_trigger")); + assertTrue(runbook.contains("validate_etl_job_replay_lineage")); + assertTrue(runbook.contains("source generation plus one")); + assertTrue(runbook.contains("lineage fields are immutable")); assertTrue(runbook.contains("(job_record_id, principal_scope_hash)")); assertTrue(runbook.contains("cross-owner lineage")); assertTrue(runbook.contains("ON DELETE RESTRICT")); @@ -39,6 +43,7 @@ void operationsRunbookDocumentsAdmissionLineageAndRollback() throws IOException assertTrue(runbook.contains("does not prove that replaying a connector")); assertTrue(runbook.contains("RFC 9110")); assertTrue(runbook.contains("RFC 9457")); + assertTrue(runbook.contains("PostgreSQL 18 documentation: CREATE TRIGGER")); assertTrue(runbook.contains("PostgreSQL 18 documentation: Constraints")); assertTrue(runbook.contains("PROV-O")); } @@ -58,8 +63,12 @@ void designAndPlanPreserveTheSingleExecutionEngine() throws IOException { assertTrue(design.contains("No replay-specific worker or scheduler exists")); assertTrue(design.contains("replay_generation_count")); assertTrue(design.contains("composite owner-scoped foreign keys")); + assertTrue(design.contains("database trigger")); + assertTrue(design.contains("exactly one generation")); + assertTrue(design.contains("immutable after insertion")); assertTrue(plan.contains("Never update a terminal source back to `PENDING`")); assertTrue(plan.contains("composite owner-scoped foreign keys")); + assertTrue(plan.contains("database trigger")); assertTrue(plan.contains("Run all verification")); assertTrue(plan.contains("no project test is skipped")); } @@ -75,22 +84,32 @@ void changelogRecordsReplayAdmissionLineageAndSafety() throws IOException { assertTrue(changelog.contains("replay_generation_count")); assertTrue(changelog.contains("etl_job_owner_identity_unique")); assertTrue(changelog.contains("composite owner-scoped foreign keys")); + assertTrue(changelog.contains("exact source/root/generation continuity")); + assertTrue(changelog.contains("immutable lineage fields")); assertTrue(changelog.contains("V7__add_etl_job_replay_lineage.sql")); assertTrue(changelog.contains("does not prove external connector safety")); } @Test - void doctoringPinsVersionedReplayAndLockDomains() throws IOException { - String evidence = read( + void doctoringPinsReplayStandardsAndVersionedKeyDomains() throws IOException { + String domainEvidence = read( "docs/doctoring/durable-job-replay-key-domain-separation.md" ).replaceAll("\\s+", " "); + String standardsEvidence = read( + "docs/doctoring/durable-job-replay-standards-evidence.md" + ).replaceAll("\\s+", " "); + + assertTrue(domainEvidence.contains("mightyetl:durable-job-replay:v1:")); + assertTrue(domainEvidence.contains("mightyetl:durable-job-replay-lock:v1:")); + assertTrue(domainEvidence.contains("isolated from ordinary submission-key hashing")); + assertTrue(domainEvidence.contains("exact strings are persisted behavior")); + assertTrue(domainEvidence.contains("does not claim cSHAKE")); + assertTrue(domainEvidence.contains("NIST Special Publication 800-185")); - assertTrue(evidence.contains("mightyetl:durable-job-replay:v1:")); - assertTrue(evidence.contains("mightyetl:durable-job-replay-lock:v1:")); - assertTrue(evidence.contains("isolated from ordinary submission-key hashing")); - assertTrue(evidence.contains("exact strings are persisted behavior")); - assertTrue(evidence.contains("does not claim cSHAKE")); - assertTrue(evidence.contains("NIST Special Publication 800-185")); + assertTrue(standardsEvidence.contains("CREATE TRIGGER")); + assertTrue(standardsEvidence.contains("PL/pgSQL trigger functions")); + assertTrue(standardsEvidence.contains("exact source/root/generation continuity")); + assertTrue(standardsEvidence.contains("lineage-column immutability")); } private static String read(String relativePath) throws IOException { From 86e118bdc2fb9664e9d203044f63a0355389da32 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:28:11 +0900 Subject: [PATCH 057/103] docs(etl): plan database-owned replay lineage continuity --- .../plans/2026-08-06-durable-job-replay.md | 28 +++++++++++++------ 1 file changed, 20 insertions(+), 8 deletions(-) diff --git a/docs/superpowers/plans/2026-08-06-durable-job-replay.md b/docs/superpowers/plans/2026-08-06-durable-job-replay.md index 50be3e88..937e3f0f 100644 --- a/docs/superpowers/plans/2026-08-06-durable-job-replay.md +++ b/docs/superpowers/plans/2026-08-06-durable-job-replay.md @@ -4,7 +4,7 @@ **Goal:** Create a new owner-scoped durable job from a failed or cancelled source only after the client resupplies the exact original payload. -**Architecture:** Store replay lineage on the new job row, bind immediate-source and root references to the same principal through composite owner-scoped foreign keys, use a versioned principal-scoped replay-key hash in the existing submission identity column, serialize creation with the existing transaction-lock boundary, verify payload digest against the terminal source, and return the existing accepted-job wire model. +**Architecture:** Store replay lineage on the new job row, bind immediate-source and root references to the same principal through composite owner-scoped foreign keys, enforce exact source/root/generation continuity and lineage-column immutability through a PostgreSQL database trigger, use a versioned principal-scoped replay-key hash in the existing submission identity column, serialize creation with the existing transaction-lock boundary, verify payload digest against the terminal source, and return the existing accepted-job wire model. **Tech Stack:** Java 25, Spring MVC, Spring transactions, JdbcTemplate, PostgreSQL 18, Flyway, H2 integration tests, JUnit 5, Mockito, JaCoCo, Maven. @@ -15,6 +15,7 @@ - Validate identifier, replay key, principal, and complete payload before lock or table access. - Persist no raw principal or raw replay key. - Require PostgreSQL to reject source or root lineage whose `principal_scope_hash` differs from the new row. +- Require the database trigger to reject nonterminal sources, derived roots, generation skips, generation-one source/root divergence, and post-insert lineage mutation. - Preserve zero-missed configured production instruction, line, method, and branch coverage. - Preserve no-skipped project tests and beginner-readable public Javadoc. - Use descriptive multi-word `snake_case` database objects. @@ -25,12 +26,14 @@ **Files** - Modify: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobMigrationDocumentationTest.java` - Create: `etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql` +- Create: `etl-service/src/test/postgresql/replay_lineage_migration.sql` - [ ] Require all three lineage columns, bounded generation, complete-null-or-complete-non-null lifecycle, self-reference rejection, named composite owner-scoped foreign keys, their named `(job_record_id, principal_scope_hash)` unique support constraint, and `ON DELETE RESTRICT`. - [ ] Reject legacy one-column source or root foreign keys because they permit cross-owner lineage at the database layer. -- [ ] Run the focused migration test and observe failure because V7 or the tenant-integrity constraints are absent. -- [ ] Implement the additive transactional migration. -- [ ] Rerun the focused test and commit. +- [ ] Require a database trigger and PL/pgSQL trigger function that validate the terminal source, first root, exact generation successor, initial pending lifecycle, and lineage-column immutability. +- [ ] Run focused migration and PostgreSQL-rehearsal contract tests and observe failure because V7, tenant-integrity constraints, or continuity enforcement is absent. +- [ ] Implement the additive transactional migration and disposable PostgreSQL 18 rehearsal. +- [ ] Rerun the focused tests and commit. ## Task 2 — Define immutable replay models and errors @@ -50,7 +53,9 @@ **Files** - Create: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayServiceIntegrationTest.java` - Create: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayServiceBoundaryTest.java` +- Create: `etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayLineageIntegrityTest.java` - Modify: `etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobService.java` +- Create: `etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplayService.java` **Public interface** @@ -64,9 +69,11 @@ EtlJobReplay replayOwned( ``` - [ ] Add failing tests for failed source, cancelled source, source immutability, payload mismatch, key replay/reuse, owner isolation, active/succeeded rejection, lineage root/generation, generation exhaustion, and validation before JDBC. +- [ ] Add a fail-first regression proving an inherited root that is itself a replay row is rejected. - [ ] Run focused tests and observe compile/assertion failure. - [ ] Add the versioned replay domain and transaction-lock identity. - [ ] Add existing-replay lookup and source-lineage lookup. +- [ ] Require the inherited root to exist in the owner namespace and to have all lineage fields null before insertion. - [ ] Insert one ordinary `PENDING` job with verified payload and lineage. - [ ] Run focused tests and commit. @@ -91,9 +98,10 @@ EtlJobReplay replayOwned( - [ ] Prove a new replay row can be claimed by the ordinary worker. - [ ] Prove an unavailable transaction lock returns replay-in-progress without insertion. - [ ] Prove retry after a committed first replay returns the same new row. -- [ ] Prove replay-of-replay preserves the root and increments generation. +- [ ] Prove replay-of-replay preserves the root and increments generation by exactly one. - [ ] Prove generation 100 fails before insertion. - [ ] Prove PostgreSQL rejects a replay source or root from another `principal_scope_hash`, independent of application owner predicates. +- [ ] Prove PostgreSQL rejects nonterminal sources, derived roots, skipped generations, and lineage mutation. - [ ] Run focused and full tests and commit. ## Task 6 — Finish operations, provenance, and exact-head verification @@ -101,14 +109,16 @@ EtlJobReplay replayOwned( **Files** - Create: `docs/operations/durable-job-replay.md` - Create: `docs/doctoring/durable-job-replay-key-domain-separation.md` +- Create: `docs/doctoring/durable-job-replay-standards-evidence.md` +- Create: `docs/adr/2026-08-07-immutable-durable-job-replay.md` - Modify: `docs/etl/durable-job-intake.md` - Modify: `CHANGELOG.md` - Create: `etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java` -- [ ] Require source immutability, payload digest proof, composite owner-scoped foreign keys, cross-owner lineage rejection, concurrency, generation limit, connector limitation, rollout, and rollback documentation first. +- [ ] Require source immutability, payload digest proof, composite owner-scoped foreign keys, database-trigger continuity, immutable lineage fields, cross-owner lineage rejection, concurrency, generation limit, connector limitation, rollout, and rollback documentation first. - [ ] Document W3C PROV mapping as an export contract, not a database-authority substitute. -- [ ] Record APA 7th primary references and the versioned replay-key compatibility boundary. -- [ ] Rehearse the migration on PostgreSQL 18 and verify the owner-scoped source/root foreign-key failures before production rollout. +- [ ] Record APA 7th primary references for PostgreSQL constraints, `CREATE TRIGGER`, and PL/pgSQL trigger functions together with the versioned replay-key compatibility boundary. +- [ ] Rehearse the migration on PostgreSQL 18 and verify owner-scoped source/root failures, exact generation progression, derived-root rejection, lineage immutability, deletion restriction, and rollback before production rollout. - [ ] Run all verification through exact-head CI: `./mvnw -B test`, configured coverage gates, and `git diff --check`. - [ ] Keep the PR draft until every stacked-target gate succeeds. @@ -119,5 +129,7 @@ EtlJobReplay replayOwned( - Replay-key and payload conflicts are distinguished without disclosing source existence across principals. - New jobs enter the existing worker lifecycle rather than creating a second execution engine. - PostgreSQL and application owner predicates independently reject cross-owner lineage. +- A database trigger owns exact source/root/generation continuity even for maintenance or import writers that bypass the service. +- Lineage fields are immutable after insertion, so descendants cannot be silently reparented. - No placeholder, ambiguous public signature, or unbounded database object name remains. - Verification requires that no project test is skipped. From d3c4685fab44be99f839cadc30e9617b52c55f95 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:29:39 +0900 Subject: [PATCH 058/103] docs(etl): define database-owned replay lineage continuity --- .../2026-08-06-durable-job-replay-design.md | 42 +++++++++++++++---- 1 file changed, 33 insertions(+), 9 deletions(-) diff --git a/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md b/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md index 14edcda1..ba126c6a 100644 --- a/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md +++ b/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md @@ -50,6 +50,23 @@ FOREIGN KEY (replay_root_job_record_id, principal_scope_hash) REFERENCES etl_job_records (job_record_id, principal_scope_hash) ``` +Those declarative constraints establish owner scope, existence, self-reference rejection, completeness, deletion restriction, and the generation bound. They do not by themselves prove that a source is terminal, that the selected root is the first lineage row, or that generation advances exactly once. The migration therefore adds a database trigger and PL/pgSQL trigger function as the relational authority for continuity: + +```text +validate_etl_job_replay_lineage() +etl_job_replay_lineage_guard_trigger +``` + +On replay insertion, the trigger requires all of the following: + +- the new derived row starts as `PENDING`, attempt zero, with a retained payload; +- the immediate source exists in the same principal scope and is `FAILED` or `CANCELLED`; +- the root exists in the same principal scope, is terminal, and has all lineage fields null; +- generation 1 uses the same row for immediate source and root; +- later generations inherit the exact first root and equal the source generation plus one. + +On updates that name any lineage column, the trigger rejects every changed value. Lineage fields are immutable after insertion, so a maintenance script, import path, or future service cannot silently reparent a job after descendants exist. Ordinary lifecycle updates remain permitted because they do not change the lineage columns. + For the first replay: ```text @@ -66,7 +83,7 @@ root = inherited first job generation = source generation + 1 ``` -The application still verifies that source and inherited root are owner-scoped to the same principal. The database independently rejects cross-owner lineage through the composite owner-scoped foreign keys. Relational rows remain authoritative even when lineage is later exported as W3C PROV. +The application independently validates the owner-scoped source, inherited root, and root-row identity before insertion. PostgreSQL independently rejects cross-owner, nonterminal, discontinuous, derived-root, and mutable lineage. Relational rows remain authoritative even when lineage is later exported as W3C PROV. ## Replay-key authority @@ -97,9 +114,10 @@ One transaction performs the following sequence: 6. require terminal `FAILED` or `CANCELLED`; 7. require the supplied payload digest to equal the source `request_digest`; 8. derive root and bounded generation; -9. insert one new `PENDING` row with the verified payload and lineage; -10. let PostgreSQL validate source and root against the same `principal_scope_hash`; -11. return only the new operator-safe job identity. +9. require the inherited root to exist in the owner namespace and have null lineage fields; +10. insert one new `PENDING` row with the verified payload and lineage; +11. let PostgreSQL validate same-owner references, terminal source, first root, exact generation continuity, and initial lifecycle; +12. return only the new operator-safe job identity. The source is never updated. Read-then-write state resurrection is prohibited. @@ -116,7 +134,7 @@ The source is never updated. Read-then-write state resurrection is prohibited. | 422 | `etl_job_replay_payload_mismatch` | Resupplied JSON does not match the immutable source digest. | | 422 | `etl_job_replay_key_reused` | Replay key already identifies another source or payload. | -All failures use the existing RFC 9457 problem model without payload, principal, key, hash, lineage internals, SQL, or exception text. +All covered request failures use the existing RFC 9457 problem model without payload, principal, key, hash, lineage internals, SQL, or exception text. A trigger rejection indicates internally inconsistent repository state or an unauthorized writer and is treated as an operator-visible integrity incident rather than reflected with raw database text. ## Worker compatibility @@ -135,7 +153,7 @@ replay action → prov:used → source job new job → prov:wasGeneratedBy → replay action ``` -The export must not weaken owner authorization or replace database constraints. +The export must not weaken owner authorization or replace database constraints and trigger enforcement. ## Verification @@ -148,12 +166,14 @@ The exact-head suite must prove: 5. same key with another source or payload fails closed; 6. foreign and missing sources remain indistinguishable; 7. pending, running, and succeeded sources are rejected; -8. replay of replay preserves root and increments generation; +8. replay of replay preserves the first root and increments generation exactly once; 9. generation 100 cannot create generation 101; 10. concurrent creation produces one row and an in-progress or later replay outcome; 11. the new job can be claimed and follows normal lifecycle contracts; -12. migration completeness, owner-scoped source and root constraints, cross-owner lineage rejection, self-reference prohibition, naming, deletion restriction, rollout, and rollback are documented and tested; -13. all added production statements and branches retain zero-missed configured coverage and no project test is skipped. +12. service defense in depth rejects an inherited root that is itself a replay row; +13. PostgreSQL 18 rejects cross-owner references, nonterminal sources, generation-one root divergence, derived roots, skipped generations, and lineage mutation; +14. migration completeness, source and root constraints, trigger/function presence, self-reference prohibition, naming, deletion restriction, rollout, and rollback are documented and tested; +15. all added production statements and branches retain zero-missed configured coverage and no project test is skipped. ## Operational limitation @@ -167,6 +187,10 @@ Nottingham, M., Wilde, E., & Dalal, S. (2023). *Problem details for HTTP APIs* ( PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: Constraints*. https://www.postgresql.org/docs/18/ddl-constraints.html +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: CREATE TRIGGER*. https://www.postgresql.org/docs/18/sql-createtrigger.html + PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: INSERT*. https://www.postgresql.org/docs/18/sql-insert.html +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: PL/pgSQL trigger functions*. https://www.postgresql.org/docs/18/plpgsql-trigger.html + World Wide Web Consortium. (2013). *PROV-O: The PROV ontology*. https://www.w3.org/TR/prov-o/ From 60d5d475bf081b16c4aa6744ebeb0c6df06a116b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:31:24 +0900 Subject: [PATCH 059/103] docs(etl): operate database-owned replay lineage continuity --- docs/operations/durable-job-replay.md | 88 +++++++++++++++++++++------ 1 file changed, 69 insertions(+), 19 deletions(-) diff --git a/docs/operations/durable-job-replay.md b/docs/operations/durable-job-replay.md index 34e85d67..6cf1ba76 100644 --- a/docs/operations/durable-job-replay.md +++ b/docs/operations/durable-job-replay.md @@ -100,15 +100,36 @@ therefore rejects cross-owner lineage even if application code, a maintenance sc import path attempts to pair one tenant's new job with another tenant's source or root. Application owner predicates remain mandatory, but they are no longer the only tenant-integrity boundary. +Declarative foreign keys and checks do not establish that the selected source is terminal, that the +root is the first row in the lineage, or that generation advances exactly once. V7 therefore creates: + +```text +validate_etl_job_replay_lineage() +etl_job_replay_lineage_guard_trigger +``` + +The `BEFORE INSERT OR UPDATE OF` database trigger enforces these additional invariants: + +- replay-created rows start as `PENDING`, with attempt zero and a retained payload; +- the exact immediate source is same-owner and `FAILED` or `CANCELLED`; +- the exact root is same-owner, terminal, and has all lineage fields null; +- generation 1 uses the same row as immediate source and root; +- every later generation equals the source generation plus one and inherits the same first root; +- lineage fields are immutable after insertion. + +The service performs the same root-identity check before insertion as defense in depth. The database +trigger remains authoritative for maintenance scripts, data imports, and other writers that do not +execute Java service code. + ```mermaid flowchart LR R0[Root terminal job
generation null] -->|same-owner replay| R1[Replay job
generation 1] R1 -->|later terminal + same-owner replay| R2[Replay job
generation 2] - R0 -. owner-scoped root reference .-> R2 + R0 -. owner-scoped first-root reference .-> R2 ``` -A replay of a root uses the source as root and generation 1. A replay of a replay inherits the root -and increments the immediate source generation. Generation 100 returns +A replay of a root uses the source as root and generation 1. A replay of a replay inherits the first +root and increments the immediate source generation. Generation 100 returns `409 etl_job_replay_generation_exhausted` instead of creating generation 101. ## Worker behavior @@ -117,6 +138,10 @@ The new row is an ordinary `PENDING` job with the verified payload. The existing lease fencing, attempts, retry, success, failure, cancellation, pagination, `Retry-After`, and ETag contracts apply unchanged. No replay-specific worker or scheduler exists. +Ordinary lifecycle updates may change status, payload, lease, attempts, failure, or cancellation +fields while retaining the exact lineage. Any writer that attempts to reparent a replay row or alter +its generation receives a database constraint failure and must be treated as an integrity incident. + ## Provenance export The relational rows are authoritative. A future owner-authorized JSON-LD export may represent: @@ -131,24 +156,31 @@ new job → prov:wasGeneratedBy → replay action ``` PROV export never grants authority and never substitutes for owner predicates, database constraints, -or replay-key idempotency. +trigger enforcement, or replay-key idempotency. ## Rollout 1. Rehearse V7 on a representative PostgreSQL 18 copy and inspect existing row count, support-index - build time, table lock duration, and foreign-key validation time. + build time, table lock duration, foreign-key validation time, and trigger creation. 2. Verify exact-head cross-platform CI, full reactor tests, zero-missed configured coverage, dependency review, SBOM, SAST, security scan, review threads, and independent approval. 3. Apply V7 before serving the replay route. -4. Smoke-test a disposable failed source, exact payload acceptance, same-key retry, and key conflict. -5. Confirm the source is unchanged and the new row has source/root/generation lineage. -6. In an isolated migration rehearsal, attempt source and root references whose - `principal_scope_hash` differs from the new row and confirm PostgreSQL rejects both cross-owner - lineage writes. -7. Claim the new pending row through the ordinary worker and verify no replay-only execution path. -8. Monitor replay acceptance, in-progress conflicts, payload mismatches, generation exhaustion, - database lock waits, and failed foreign-key deletion or tenant-boundary attempts using - fixed-cardinality signals. +4. Verify that exactly one `validate_etl_job_replay_lineage` function and one + `etl_job_replay_lineage_guard_trigger` exist on `etl_job_records`. +5. Smoke-test a disposable failed source, exact payload acceptance, same-key retry, and key conflict. +6. Confirm the source is unchanged and the new row has source/root/generation lineage. +7. In an isolated migration rehearsal, confirm PostgreSQL rejects: + - nonterminal replay sources; + - generation-one rows whose source differs from root; + - derived replay rows used as root; + - skipped generations; + - source or root references from another `principal_scope_hash`; + - post-insert lineage mutation. +8. Confirm both source and root deletion remain protected by `ON DELETE RESTRICT`. +9. Claim the new pending row through the ordinary worker and verify no replay-only execution path. +10. Monitor replay acceptance, in-progress conflicts, payload mismatches, generation exhaustion, + database lock waits, trigger rejections, and failed foreign-key deletion or tenant-boundary + attempts using fixed-cardinality signals. Logs and metric labels must not contain payloads, raw principals, raw keys, hashes, source/new job identifiers, lineage identifiers, SQL, exception messages, or target identities. @@ -167,23 +199,35 @@ Read the already-created replay job associated with the operator's prior request principal-scoped replay intent and cannot be reused for another source or payload. Use a new key only for a deliberately separate replay. +### Trigger or lineage integrity rejection + +Stop replay admission for the affected deployment. Preserve the deployed SHA, Flyway history, +transaction boundary, fixed error classification, and sanitized database evidence. Do not retry by +disabling the trigger or rewriting the source, root, or generation. Determine whether the attempted +write came from a stale binary, maintenance script, import path, migration defect, or unauthorized +writer. Treat cross-owner attempts as tenant-isolation incidents even when PostgreSQL rejected them. + ### Broken lineage or missing root Stop replay admission. Preserve affected rows, deployed SHA, Flyway history, and backup evidence. Do not null lineage fields to make constraints pass. Repair requires a reviewed migration based on -verified source/root ownership and generation. Treat any attempted cross-owner lineage write as a -tenant-isolation incident even when PostgreSQL rejects it. +verified source/root ownership and generation. Never update lineage columns in place merely to pass +the trigger. ## Rollback Stop serving replay admission before rolling application binaries back. Older binaries ignore lineage columns, but deletion or retention tooling might not understand the new `ON DELETE RESTRICT` -relationships. +relationships or trigger. Do not drop V7 while replay rows exist. Archive or remove replay lineages from leaf to root under an approved retention policy, preserving external audit evidence. Then a separately reviewed migration -may remove the composite foreign keys, `etl_job_owner_identity_unique`, and lineage columns. Never -edit the applied V7 file or mutate terminal sources back to pending. +may remove the trigger, function, composite foreign keys, `etl_job_owner_identity_unique`, and +lineage columns. Never edit the applied V7 file or mutate terminal sources back to pending. + +A controlled rollback rehearsal must drop the trigger before its function and remove dependent +constraints before columns, all inside a transaction that is rolled back. After rollback, verify all +three columns, four named constraints, the trigger, and the function are restored. The replay-key domain must remain readable while any replay-created row can receive an idempotent retry. A domain change requires a versioned migration or dual-read period, not a silent constant edit. @@ -206,8 +250,14 @@ https://www.rfc-editor.org/rfc/rfc9457 PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: Constraints*. https://www.postgresql.org/docs/18/ddl-constraints.html +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: CREATE TRIGGER*. +https://www.postgresql.org/docs/18/sql-createtrigger.html + PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: INSERT*. https://www.postgresql.org/docs/18/sql-insert.html +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: PL/pgSQL trigger functions*. +https://www.postgresql.org/docs/18/plpgsql-trigger.html + World Wide Web Consortium. (2013). *PROV-O: The PROV ontology*. https://www.w3.org/TR/prov-o/ From 09a3fd4ef8a34f954590a0bf4f205b0ea3c4cbd0 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:32:27 +0900 Subject: [PATCH 060/103] docs(etl): trace replay lineage trigger to PostgreSQL standards --- .../durable-job-replay-standards-evidence.md | 31 +++++++++++++++++-- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/docs/doctoring/durable-job-replay-standards-evidence.md b/docs/doctoring/durable-job-replay-standards-evidence.md index 983f2b1c..825c59c6 100644 --- a/docs/doctoring/durable-job-replay-standards-evidence.md +++ b/docs/doctoring/durable-job-replay-standards-evidence.md @@ -4,7 +4,7 @@ mightyETL models replay as creation of a new durable job derived from an immutable terminal source. The source is never returned to `PENDING`. Only owner-scoped `FAILED` and `CANCELLED` sources are eligible, and the operator must resupply byte-identical bounded JSON whose SHA-256 digest equals the source `request_digest`. -The accepted replay response uses `202 Accepted` with a monitor URI because durable admission does not assert completion. Deterministic failures use RFC 9457 problem details. PostgreSQL owns the replay transaction, uniqueness, lineage constraints, and source-row serialization. Replay lineage is compatible with PROV-O derivation semantics, while the relational database remains authoritative. +The accepted replay response uses `202 Accepted` with a monitor URI because durable admission does not assert completion. Deterministic failures use RFC 9457 problem details. PostgreSQL owns the replay transaction, uniqueness, owner-scoped foreign keys, exact source/root/generation continuity, lineage-column immutability, and source-row serialization. Replay lineage is compatible with PROV-O derivation semantics, while the relational database remains authoritative. ## Normative mapping @@ -13,9 +13,25 @@ The accepted replay response uses `202 Accepted` with a monitor URI because dura | Noncommittal durable admission | RFC 9110, section 15.3.3 | `202 Accepted`, `Location`, and no claim of execution completion | | Stable machine-readable failures | RFC 9457 | Fixed problem type, title, status, detail, and `error_code` without exception text | | Atomic new-row creation and conflict handling | PostgreSQL 18 `INSERT` and transaction documentation | One transaction validates source ownership, digest, replay identity, and lineage before inserting one new job | -| Derivation lineage | W3C PROV-O | New job is derived from the immediate source and preserves an immutable root/generation chain | +| Same-owner source and root existence | PostgreSQL 18 constraints | Composite foreign keys bind source and root to the new row's `principal_scope_hash`, and `ON DELETE RESTRICT` protects retained history | +| Exact lineage transition | PostgreSQL 18 `CREATE TRIGGER` and PL/pgSQL trigger functions | A row-level `BEFORE INSERT OR UPDATE OF` trigger validates terminal source, first root, exact generation successor, initial pending lifecycle, and lineage-column immutability | +| Derivation lineage | W3C PROV-O | New job is derived from the immediate source and preserves an immutable first-root/generation chain | | Domain separation rationale | NIST SP 800-185 | Replay-key hashing uses a versioned replay-specific domain; the SHA-256 construction does not claim cSHAKE or TupleHash conformance | +## Why constraints and a trigger are both required + +The composite owner-scoped foreign keys prove that the named source and root exist in the same tenant namespace. The complete-lineage check proves that lineage fields are either all null or all present, bounds generation, and rejects direct self-reference. Those declarative rules cannot express all cross-row temporal invariants: + +- source must already be `FAILED` or `CANCELLED`; +- the root must be the first job, with every lineage field null; +- generation one must use the same row as source and root; +- every later generation must inherit that root and equal the immediate source generation plus one; +- an existing replay row must never be reparented. + +PostgreSQL `CREATE TRIGGER` permits a row-level trigger to run before selected insert or update events, and PL/pgSQL trigger functions receive `NEW`, `OLD`, `TG_OP`, and related context. mightyETL uses that database mechanism to reject invalid writes before persistence. The service repeats the inherited-root identity check as defense in depth, but a maintenance script or import path cannot bypass the relational authority merely by omitting Java validation. + +The trigger raises the fixed SQLSTATE class `23514` without embedding principal values, job identifiers, payloads, hashes, SQL, or exception causes. Application and operator logs must classify the failure with a finite internal integrity code rather than copying raw database text. + ## Security and privacy boundary The HTTP response and ordinary telemetry exclude raw principals, replay keys, payloads, request digests, internal hashes, source/root identifiers, lease identifiers, SQL, target identities, and exception messages. Foreign-owned and absent source identifiers remain indistinguishable. `SUCCEEDED` is excluded because repeating a committed target effect is not safe merely because the original request bytes are known. @@ -24,13 +40,16 @@ Connector replay is enabled only when target effects participate in the mightyET ## Verification obligations -- real PostgreSQL migration rehearsal for self-referencing foreign keys and `ON DELETE RESTRICT`; +- real PostgreSQL 18 migration rehearsal for trigger and function presence, composite self-referencing foreign keys, and `ON DELETE RESTRICT`; - exact-payload acceptance and byte-different rejection; - owner-safe missing/foreign behavior; - same-key replay, key reuse conflict, and concurrent admission tests; - source immutability and source/root/generation lineage tests; +- database rejection of nonterminal sources, generation-one root divergence, derived roots, skipped generations, cross-owner references, and lineage mutation; +- service defense-in-depth rejection when an inherited root is itself a replay row; - generation-bound rejection; - ordinary worker claim, lease fencing, cancellation, polling, and ETag compatibility; +- transactional rollback rehearsal that restores trigger, function, columns, and named constraints; - configured production instruction, line, method, and branch coverage with zero misses; - direct-base CI, dependency, SBOM, SAST, security, review-thread, and independent-approval gates before merge. @@ -42,6 +61,12 @@ National Institute of Standards and Technology. (2016). *SHA-3 derived functions Nottingham, M., Wilde, E., & Dalal, S. (2023). *Problem details for HTTP APIs* (RFC 9457). RFC Editor. https://www.rfc-editor.org/rfc/rfc9457 +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: Constraints*. https://www.postgresql.org/docs/18/ddl-constraints.html + +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: CREATE TRIGGER*. https://www.postgresql.org/docs/18/sql-createtrigger.html + PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: INSERT*. https://www.postgresql.org/docs/18/sql-insert.html +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: PL/pgSQL trigger functions*. https://www.postgresql.org/docs/18/plpgsql-trigger.html + World Wide Web Consortium. (2013). *PROV-O: The PROV ontology*. https://www.w3.org/TR/prov-o/ From 038796e543f547da567d6ee21849b25a509cab73 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:33:54 +0900 Subject: [PATCH 061/103] docs(etl): record database-owned replay lineage decision --- ...2026-08-07-immutable-durable-job-replay.md | 53 +++++++++++++++---- 1 file changed, 44 insertions(+), 9 deletions(-) diff --git a/docs/adr/2026-08-07-immutable-durable-job-replay.md b/docs/adr/2026-08-07-immutable-durable-job-replay.md index b627c0c2..c8e07be2 100644 --- a/docs/adr/2026-08-07-immutable-durable-job-replay.md +++ b/docs/adr/2026-08-07-immutable-durable-job-replay.md @@ -11,6 +11,8 @@ Durable jobs deliberately clear `request_payload` after success, failure, or can Rewinding a terminal row to `PENDING` would erase the original terminal fact, mix attempt histories, invalidate conditional status validators, and make concurrent cancellation or success reasoning substantially harder. Retaining terminal payloads solely for replay would expand sensitive-data retention. Treating semantically equivalent JSON as the same work would also allow hidden payload changes under a replay label. +A structural schema with nullable source, root, and generation fields is not sufficient on its own. Composite foreign keys can prove same-owner existence, and check constraints can prove field completeness and bounds, but they cannot prove that a source is terminal, a root is the first row in the chain, a generation increments exactly once, or an existing replay has never been reparented. Those properties must remain true even for maintenance scripts, data imports, and future writers that do not execute the Java service. + ## Decision Replay creates a **new** durable job. The terminal source remains unchanged. @@ -20,24 +22,39 @@ The authenticated owner submits the source identifier, a replay-specific `Idempo A derived row stores: - `replay_source_job_record_id`: immediate source; -- `replay_root_job_record_id`: immutable root of the replay chain; +- `replay_root_job_record_id`: immutable first root of the replay chain; - `replay_generation_count`: bounded positive generation. -Source and root references use `ON DELETE RESTRICT`. A root row has all three lineage fields null; a replay row has all three non-null. Generation cannot exceed the supported bound. The new row enters the ordinary `PENDING` lifecycle and uses the existing worker, PostgreSQL claim, exact lease, success, failure, polling, cancellation, and ETag contracts. +Source and root references are composite owner-scoped foreign keys to `(job_record_id, principal_scope_hash)` and use `ON DELETE RESTRICT`. A root row has all three lineage fields null; a replay row has all three non-null. Generation cannot exceed the supported bound. + +V7 also creates the PL/pgSQL function `validate_etl_job_replay_lineage()` and the row-level `etl_job_replay_lineage_guard_trigger`. The trigger is the database authority for exact source/root/generation continuity: + +- replay rows are inserted only as `PENDING`, attempt zero, with a retained payload; +- immediate source and root belong to the same principal namespace as the new row; +- the immediate source is `FAILED` or `CANCELLED`; +- the root is terminal and has no lineage fields; +- generation one uses the same source and root; +- later generations retain that root and equal the immediate source generation plus one; +- lineage columns are immutable after insertion. + +The Java service independently verifies the inherited root is an actual first root before insertion. This is defense in depth; it does not replace the trigger. The new row enters the ordinary `PENDING` lifecycle and uses the existing worker, PostgreSQL claim, exact lease, success, failure, polling, cancellation, and ETag contracts. ```mermaid sequenceDiagram participant O as Authenticated owner participant A as Replay API participant P as PostgreSQL transaction + participant T as Lineage trigger participant W as Ordinary worker O->>A: source id + exact payload + replay key A->>A: bounded JSON validation and SHA-256 A->>P: owner-matched terminal source lock P->>P: verify FAILED/CANCELLED and digest equality - P->>P: verify replay-key identity and generation bound - P->>P: insert new PENDING row with source/root/generation + P->>P: verify replay-key identity and inherited root + P->>T: insert new PENDING row with source/root/generation + T->>T: verify owner, terminal source, first root, exact successor + T-->>P: accept or reject before persistence P-->>A: commit derived job A-->>O: 202 + Location + Idempotency-Replayed W->>P: ordinary lease-fenced claim @@ -61,11 +78,11 @@ NIST SP 800-185 motivates explicit domain separation, but the SHA-256 constructi - unresolved concurrent admission: stable retryable `409` problem; - generation exhaustion: stable `409` problem. -All errors use fixed RFC 9457 metadata and exclude exception messages, SQL, hashes, identifiers, payloads, and target details. +All covered request failures use fixed RFC 9457 metadata and exclude exception messages, SQL, hashes, identifiers, payloads, and target details. A database-trigger rejection is an internal integrity incident; raw PL/pgSQL text must not enter the client response or ordinary telemetry. ## Connector boundary -The replay transaction can prove source ownership, payload fidelity, replay identity, lineage, and durable admission. It cannot prove that a remote warehouse, file system, API, or broker will suppress duplicate effects. Replay is enabled for a connector only when target effects participate in the mightyETL transaction or the connector provides independently tested idempotency or compensation. +The replay transaction can prove source ownership, payload fidelity, replay identity, exact lineage, and durable admission. It cannot prove that a remote warehouse, file system, API, or broker will suppress duplicate effects. Replay is enabled for a connector only when target effects participate in the mightyETL transaction or the connector provides independently tested idempotency or compensation. ## Alternatives rejected @@ -85,6 +102,14 @@ Rejected because normalization can obscure a changed request and creates a secon Rejected in the initial slice because a succeeded job may already have committed irreversible external effects. +### Rely only on application validation + +Rejected because maintenance scripts, migrations, import processes, or future services can write directly to the table. Relational lineage must remain valid independently of one application binary. + +### Use only foreign keys and check constraints + +Rejected because those constraints cannot express exact generation succession or immutable cross-row root identity. A row-level trigger is required for those cross-row invariants. + ## Consequences ### Positive @@ -94,20 +119,24 @@ Rejected in the initial slice because a succeeded job may already have committed - lineage supports incident analysis and future PROV-compatible export; - ordinary worker and cancellation machinery is reused; - payload retention does not increase; -- concurrent retries have one database-owned outcome. +- concurrent retries have one database-owned outcome; +- database maintenance and import paths cannot create discontinuous or reparented lineage; +- cross-owner, nonterminal, derived-root, skipped-generation, and mutation attempts fail closed. ### Costs - operators must possess the exact original payload bytes; - self-referencing lineage constrains retention and deletion order; - every connector needs an explicit replay-safety classification; -- migrations and generation bounds require real PostgreSQL verification. +- migrations and generation bounds require real PostgreSQL verification; +- the trigger adds a small number of same-transaction row lookups to replay insertion; +- trigger and function lifecycle must be included in downgrade and disaster-recovery rehearsals. ## Verification Acceptance requires exact-head tests for source immutability, owner isolation, exact payload matching, same-key replay, conflicting-key reuse, concurrent admission, lineage inheritance, generation exhaustion, ordinary worker behavior, cancellation compatibility, RFC 9457 responses, privacy exclusions, and zero-missed configured production coverage. -A direct-`develop` GitHub Actions gate applies every versioned migration to PostgreSQL 18, verifies replay and cancellation columns, verifies both lineage foreign keys use `ON DELETE RESTRICT`, verifies bounded source/root/generation checks, and creates a non-empty schema-only dump. SAST, security, dependency, SBOM, review-thread, and non-author exact-head approval gates remain mandatory. +A direct-`develop` GitHub Actions gate applies every versioned migration to PostgreSQL 18, verifies replay and cancellation columns, verifies both lineage foreign keys use `ON DELETE RESTRICT`, verifies trigger and function presence, executes valid generation-one and generation-two inserts, rejects nonterminal sources, different generation-one roots, derived roots, skipped generations, cross-owner references, and lineage mutation, protects source/root deletion, rehearses transactional rollback, and creates a non-empty schema-only dump. SAST, security, dependency, SBOM, review-thread, and non-author exact-head approval gates remain mandatory. ## References — APA 7th edition @@ -117,6 +146,12 @@ National Institute of Standards and Technology. (2016). *SHA-3 derived functions Nottingham, M., Wilde, E., & Dalal, S. (2023). *Problem details for HTTP APIs* (RFC 9457). RFC Editor. https://www.rfc-editor.org/rfc/rfc9457 +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: Constraints*. https://www.postgresql.org/docs/18/ddl-constraints.html + +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: CREATE TRIGGER*. https://www.postgresql.org/docs/18/sql-createtrigger.html + PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: INSERT*. https://www.postgresql.org/docs/18/sql-insert.html +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: PL/pgSQL trigger functions*. https://www.postgresql.org/docs/18/plpgsql-trigger.html + World Wide Web Consortium. (2013). *PROV-O: The PROV ontology*. https://www.w3.org/TR/prov-o/ From 5d96e5f73779907973937b3d5efb7a3ddad5a45d Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:37:37 +0900 Subject: [PATCH 062/103] docs(etl): record database-owned replay lineage continuity --- CHANGELOG.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 604dbb82..49f37399 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed - Authenticated operators can now create an ordinary pending durable job from an immutable failed or cancelled source only after resupplying a byte-identical bounded JSON payload; the terminal source remains unchanged and succeeded sources remain non-replayable. +- Replay lineage now remains database-authoritative across every writer: PostgreSQL validates the terminal immediate source, first root, exact source/root/generation continuity, generation-one identity, and one-step generation succession, while immutable lineage fields prevent post-insert reparenting. - Authenticated operators can now perform owner-scoped durable-job cancellation for `PENDING` and `RUNNING` work through an idempotent action that commits terminal `CANCELLED`, clears payload and lease state, stores only `cancellation_key_hash` plus a fixed code and timestamp, and returns stable RFC 9457 conflicts when success or failure already won. - Cancellation-first races now make the former exact lease stale and roll back transactional target and response-ledger effects; success-first races remain `SUCCEEDED`, while same-key cancellation replays and different-key reuse fails closed. - Owner-scoped durable-job status responses now emit deterministic weak SHA-256 `ETag` validators; ordinary and wildcard `If-None-Match` requests return an empty RFC 9110 `304 Not Modified` response only after authenticated owner-safe lookup, while `Cache-Control: no-store` remains unchanged. @@ -38,10 +39,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- Transactional migration `V7__add_etl_job_replay_lineage.sql`, owner-scoped replay admission, the named `etl_job_owner_identity_unique` support key, composite owner-scoped foreign keys for immediate-source and root lineage, and immutable `replay_source_job_record_id`, `replay_root_job_record_id`, and `replay_generation_count` evidence with concurrency, tenant-integrity, worker-compatibility, rollout, incident, and rollback tests and documentation. +- Transactional migration `V7__add_etl_job_replay_lineage.sql`, owner-scoped replay admission, the named `etl_job_owner_identity_unique` support key, composite owner-scoped foreign keys for immediate-source and root lineage, `validate_etl_job_replay_lineage()` plus `etl_job_replay_lineage_guard_trigger`, and immutable `replay_source_job_record_id`, `replay_root_job_record_id`, and `replay_generation_count` evidence with concurrency, tenant-integrity, exact-continuity, worker-compatibility, rollout, incident, and rollback tests and documentation. +- PostgreSQL 18 rehearsal now executes valid first- and second-generation replay chains and rejects nonterminal sources, different generation-one roots, derived roots, skipped generations, cross-owner references, lineage mutation, and protected source/root deletion before transactionally rehearsing trigger, function, constraint, and column rollback. - Transactional migration `V6__add_etl_job_cancellation.sql`, owner-safe cancellation API and replay model, exact lease-invalidation and cancellation-versus-success integration tests, plus rollout, incident, connector-limitation, and rollback evidence in `docs/operations/durable-job-cancellation.md`. - Deterministic ordinary, wildcard, changed-state, changed-failure-code, null-versus-empty, and unrelated-response conditional polling tests, complete controller Javadoc, privacy and rollback guidance, and APA 7th standards evidence in `docs/etl/durable-job-polling.md`. -- Controller-scoped polling advice, deterministic active/terminal lifecycle tests, disabled-worker fail-closed behavior, sub-second rounding coverage, rollback guidance, and APA 7th standards evidence in `docs/etl/durable-job-polling.md`. +- Controller-scoped polling advice, deterministic active/terminal lifecycle tests, disabled-worker fail-closed behavior, sub-second rounding coverage, rollback guidance, and APA 7th RFC 9110 evidence in `docs/etl/durable-job-polling.md`. - Owner-scoped durable job list models and HTTP contract, strict cursor and page-limit validation, one-extra-row next-page detection, the descriptive `etl_job_owner_pagination_index`, deterministic tenant-isolation and equal-timestamp tests, migration rollback guidance, and APA 7th standards evidence in `docs/etl/durable-job-intake.md`. - A production rollout and invalid-index recovery runbook for the nonblocking durable-job claim index: `docs/operations/durable-job-claim-index-rollout.md`. - PostgreSQL `FOR UPDATE SKIP LOCKED` durable-job claiming, per-process and per-claim lease fencing, expiry reclaim, bounded attempts, exact-live-lease transitions, terminal payload clearing, stable failure codes, and finite-cardinality worker metrics. @@ -79,7 +81,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Security -- Replay uses owner-scoped source selection, byte-exact digest verification, a versioned principal-scoped key domain, immutable relational lineage, and composite owner-scoped foreign keys that independently reject cross-tenant source or root references without retaining raw principals or replay keys; this evidence does not prove external connector safety, so connector-native idempotency, transaction participation, or compensation remains required. +- Replay uses owner-scoped source selection, byte-exact digest verification, a versioned principal-scoped key domain, immutable relational lineage, composite owner-scoped foreign keys, and a database trigger that independently rejects cross-tenant references, nonterminal sources, false roots, generation discontinuities, and lineage mutation without retaining raw principals or replay keys; this evidence does not prove external connector safety, so connector-native idempotency, transaction participation, or compensation remains required. - Cancellation stores only a principal-scoped SHA-256 replay identity and a fixed machine code, exposes no raw principal, key, hash, payload, lease, SQL, exception, or target detail, and requires an owner-matched conditional database update before reporting success. - Conditional status validators are SHA-256 digests of only the complete owner-authorized operator-safe representation; payloads, raw principals, idempotency keys, internal hashes, leases, SQL, and exception text remain excluded, and wildcard evaluation occurs only after owner-safe lookup. - Polling advice exposes only a bounded delay integer and is omitted when local execution is disabled or terminal; it never contains job, lease, principal, key, hash, payload, SQL, exception, target, or queue-depth data. From af523a6a75ec398999823a2d411e2a54251681e5 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:40:54 +0900 Subject: [PATCH 063/103] docs(etl): state exact replay-generation progression --- docs/superpowers/specs/2026-08-06-durable-job-replay-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md b/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md index ba126c6a..15168b78 100644 --- a/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md +++ b/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md @@ -50,7 +50,7 @@ FOREIGN KEY (replay_root_job_record_id, principal_scope_hash) REFERENCES etl_job_records (job_record_id, principal_scope_hash) ``` -Those declarative constraints establish owner scope, existence, self-reference rejection, completeness, deletion restriction, and the generation bound. They do not by themselves prove that a source is terminal, that the selected root is the first lineage row, or that generation advances exactly once. The migration therefore adds a database trigger and PL/pgSQL trigger function as the relational authority for continuity: +Those declarative constraints establish owner scope, existence, self-reference rejection, completeness, deletion restriction, and the generation bound. They do not by themselves prove that a source is terminal, that the selected root is the first lineage row, or that each replay hop advances exactly one generation. The migration therefore adds a database trigger and PL/pgSQL trigger function as the relational authority for continuity: ```text validate_etl_job_replay_lineage() From 6c6137fb43ba810dff4e8da13b300cdacc4e868c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:41:08 +0900 Subject: [PATCH 064/103] test(etl): require exact-head PostgreSQL migration checkout --- ...tgresqlMigrationIntegrationWorkflowTest.java | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/documentation/PostgresqlMigrationIntegrationWorkflowTest.java b/etl-service/src/test/java/com/xtrmetl/etl/documentation/PostgresqlMigrationIntegrationWorkflowTest.java index 94173a09..f032d114 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/documentation/PostgresqlMigrationIntegrationWorkflowTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/documentation/PostgresqlMigrationIntegrationWorkflowTest.java @@ -17,7 +17,7 @@ class PostgresqlMigrationIntegrationWorkflowTest { @Test - void workflowUsesLeastPrivilegePinnedCheckoutAndPostgresqlEighteen() throws IOException { + void workflowUsesLeastPrivilegeExactHeadCheckoutAndPostgresqlEighteen() throws IOException { String workflow = read(".github/workflows/postgresql-migration-integration.yml"); assertTrue(workflow.contains("name: PostgreSQL Migration Integration")); @@ -25,11 +25,26 @@ void workflowUsesLeastPrivilegePinnedCheckoutAndPostgresqlEighteen() throws IOEx assertTrue(workflow.contains("permissions:\n contents: read")); assertTrue(workflow.contains("timeout-minutes: 15")); assertTrue(workflow.contains("image: postgres:18-alpine")); + assertTrue(workflow.contains( + "if: github.event_name != 'workflow_dispatch' || " + + "github.ref_name == github.event.repository.default_branch" + )); assertTrue(workflow.contains( "uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0" )); + assertTrue(workflow.contains( + "repository: ${{ github.event_name == 'pull_request' " + + "&& github.event.pull_request.head.repo.full_name || github.repository }}" + )); + assertTrue(workflow.contains( + "ref: ${{ github.event_name == 'pull_request' " + + "&& github.event.pull_request.head.sha " + + "|| github.event.repository.default_branch }}" + )); assertTrue(workflow.contains("persist-credentials: false")); assertTrue(workflow.contains("bash scripts/verify-postgresql-migrations.sh")); + assertFalse(workflow.contains("refs/pull/")); + assertFalse(workflow.contains("github.event.pull_request.merge_commit_sha")); assertFalse(workflow.contains("pull_request_target:")); assertFalse(workflow.contains("COPILOT_GITHUB_TOKEN")); assertFalse(workflow.contains("NVIDIA_NIM_API_KEY")); From deac1abe9bc43c17830d8cf0f07e85b33205cdf7 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:42:00 +0900 Subject: [PATCH 065/103] ci(etl): bind PostgreSQL migration gate to exact PR head --- .github/workflows/postgresql-migration-integration.yml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/workflows/postgresql-migration-integration.yml b/.github/workflows/postgresql-migration-integration.yml index a1bd11b8..867b5ca1 100644 --- a/.github/workflows/postgresql-migration-integration.yml +++ b/.github/workflows/postgresql-migration-integration.yml @@ -21,6 +21,7 @@ concurrency: jobs: replay_lineage_migration: name: replay-lineage-migration + if: github.event_name != 'workflow_dispatch' || github.ref_name == github.event.repository.default_branch runs-on: ubuntu-latest timeout-minutes: 15 services: @@ -44,9 +45,11 @@ jobs: PGUSER: mightyetl_test PGPASSWORD: mightyetl_test_password steps: - - name: Check out exact pull-request merge candidate + - name: Check out exact pull-request head or protected default branch uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 with: + repository: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name || github.repository }} + ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.event.repository.default_branch }} persist-credentials: false - name: Verify versioned migrations and immutable replay lineage shell: bash From d7d468d083f3a7e1e640a4d029bfdc3986c1d019 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:44:28 +0900 Subject: [PATCH 066/103] test(etl): require referenced terminal evidence immutability --- .../xtrmetl/etl/job/EtlJobReplayMigrationTest.java | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java index 56d46f23..e309cf31 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java @@ -62,7 +62,10 @@ void replayMigrationAddsCompleteRestrictedOwnerScopedLineage() throws IOExceptio )); assertTrue(migration.contains( "BEFORE INSERT OR UPDATE OF replay_source_job_record_id, " - + "replay_root_job_record_id, replay_generation_count" + + "replay_root_job_record_id, replay_generation_count, job_status, " + + "request_digest, request_payload, attempt_count, failure_code, " + + "cancellation_key_hash, cancellation_code, job_cancelled_at, " + + "created_at, updated_at" )); assertTrue(migration.contains( "NEW.replay_source_job_record_id <> NEW.replay_root_job_record_id" @@ -71,6 +74,15 @@ void replayMigrationAddsCompleteRestrictedOwnerScopedLineage() throws IOExceptio "source_generation_count IS DISTINCT FROM NEW.replay_generation_count - 1" )); assertTrue(migration.contains("Replay lineage fields are immutable")); + assertTrue(migration.contains("Referenced replay evidence is immutable")); + assertTrue(migration.contains("FOR UPDATE")); + assertTrue(migration.contains( + "child_record.replay_source_job_record_id = OLD.job_record_id" + )); + assertTrue(migration.contains( + "child_record.replay_root_job_record_id = OLD.job_record_id" + )); + assertFalse(migration.contains("FOR KEY SHARE")); assertFalse(migration.contains( "FOREIGN KEY (replay_source_job_record_id) " + "REFERENCES etl_job_records (job_record_id)" From cdf8c6be5f46074251724885a1b883f250aa18c9 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:45:40 +0900 Subject: [PATCH 067/103] test(etl): require PostgreSQL referenced-evidence immutability rehearsal --- .../etl/documentation/EtlJobReplayPostgresWorkflowTest.java | 2 ++ 1 file changed, 2 insertions(+) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/documentation/EtlJobReplayPostgresWorkflowTest.java b/etl-service/src/test/java/com/xtrmetl/etl/documentation/EtlJobReplayPostgresWorkflowTest.java index 4c3923c4..8c660702 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/documentation/EtlJobReplayPostgresWorkflowTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/documentation/EtlJobReplayPostgresWorkflowTest.java @@ -58,6 +58,8 @@ void postgresRehearsalCoversLineageContinuityTenantIntegrityAndRollback() assertTrue(rehearsal.contains("cross-owner source lineage was accepted")); assertTrue(rehearsal.contains("cross-owner root lineage was accepted")); assertTrue(rehearsal.contains("replay lineage fields were mutable")); + assertTrue(rehearsal.contains("referenced replay root evidence was mutable")); + assertTrue(rehearsal.contains("referenced immediate-source evidence was mutable")); assertTrue(rehearsal.contains( "ON DELETE RESTRICT did not protect immediate replay history" )); From 1db7bb12a1bbca054f09f5c2af39abec22018872 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:51:42 +0900 Subject: [PATCH 068/103] fix(etl): freeze referenced replay evidence --- .../V7__add_etl_job_replay_lineage.sql | 39 ++++++++++++++++--- 1 file changed, 34 insertions(+), 5 deletions(-) diff --git a/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql b/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql index 0d83bd40..6cadd16c 100644 --- a/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql +++ b/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql @@ -33,8 +33,9 @@ ALTER TABLE etl_job_records -- Keep the relational lineage authoritative even when rows are imported outside the service. -- Each derived row must point to an exact terminal predecessor, preserve the first root, and --- advance the generation by exactly one. Lineage columns become immutable after insertion so a --- later update cannot silently rewrite descendants' provenance. +-- advance the generation by exactly one. Lineage columns become immutable after insertion. +-- Once a row is referenced as an immediate source or root, its terminal replay evidence also +-- becomes immutable so later writes cannot change the meaning of already-created descendants. CREATE FUNCTION validate_etl_job_replay_lineage() RETURNS trigger LANGUAGE plpgsql AS $etl_job_replay_lineage$ @@ -61,6 +62,31 @@ BEGIN USING ERRCODE = '23514'; END IF; + IF TG_OP = 'UPDATE' + AND ( + OLD.job_status IS DISTINCT FROM NEW.job_status + OR OLD.request_digest IS DISTINCT FROM NEW.request_digest + OR OLD.request_payload IS DISTINCT FROM NEW.request_payload + OR OLD.attempt_count IS DISTINCT FROM NEW.attempt_count + OR OLD.failure_code IS DISTINCT FROM NEW.failure_code + OR OLD.cancellation_key_hash IS DISTINCT FROM NEW.cancellation_key_hash + OR OLD.cancellation_code IS DISTINCT FROM NEW.cancellation_code + OR OLD.job_cancelled_at IS DISTINCT FROM NEW.job_cancelled_at + OR OLD.created_at IS DISTINCT FROM NEW.created_at + OR OLD.updated_at IS DISTINCT FROM NEW.updated_at + ) THEN + PERFORM 1 + FROM etl_job_records AS child_record + WHERE child_record.replay_source_job_record_id = OLD.job_record_id + OR child_record.replay_root_job_record_id = OLD.job_record_id + FOR UPDATE; + + IF FOUND THEN + RAISE EXCEPTION 'Referenced replay evidence is immutable' + USING ERRCODE = '23514'; + END IF; + END IF; + IF NEW.replay_generation_count IS NULL THEN RETURN NEW; END IF; @@ -86,7 +112,7 @@ BEGIN FROM etl_job_records WHERE job_record_id = NEW.replay_source_job_record_id AND principal_scope_hash = NEW.principal_scope_hash - FOR KEY SHARE; + FOR UPDATE; IF NOT FOUND THEN RAISE EXCEPTION 'Replay source is missing or belongs to another owner' @@ -109,7 +135,7 @@ BEGIN FROM etl_job_records WHERE job_record_id = NEW.replay_root_job_record_id AND principal_scope_hash = NEW.principal_scope_hash - FOR KEY SHARE; + FOR UPDATE; IF NOT FOUND THEN RAISE EXCEPTION 'Replay root is missing or belongs to another owner' @@ -144,7 +170,10 @@ $etl_job_replay_lineage$; CREATE TRIGGER etl_job_replay_lineage_guard_trigger BEFORE INSERT OR UPDATE OF replay_source_job_record_id, - replay_root_job_record_id, replay_generation_count + replay_root_job_record_id, replay_generation_count, job_status, + request_digest, request_payload, attempt_count, failure_code, + cancellation_key_hash, cancellation_code, job_cancelled_at, + created_at, updated_at ON etl_job_records FOR EACH ROW EXECUTE FUNCTION validate_etl_job_replay_lineage(); From 32986345daf4e523a3c4facf5b10f76f64058c1f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:53:52 +0900 Subject: [PATCH 069/103] test(etl): rehearse referenced replay evidence immutability --- .../postgresql/replay_lineage_migration.sql | 44 +++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/etl-service/src/test/postgresql/replay_lineage_migration.sql b/etl-service/src/test/postgresql/replay_lineage_migration.sql index c0b12a03..6b01a2de 100644 --- a/etl-service/src/test/postgresql/replay_lineage_migration.sql +++ b/etl-service/src/test/postgresql/replay_lineage_migration.sql @@ -401,6 +401,50 @@ BEGIN END $lineage_immutability_check$; +DO $root_evidence_immutability_check$ +BEGIN + BEGIN + UPDATE etl_job_records + SET request_digest = repeat('0', 64) + WHERE job_record_id = '00000000-0000-4000-8000-000000000002'; + EXCEPTION + WHEN check_violation THEN + NULL; + END; + + IF EXISTS ( + SELECT 1 + FROM etl_job_records + WHERE job_record_id = '00000000-0000-4000-8000-000000000002' + AND request_digest <> repeat('b', 64) + ) THEN + RAISE EXCEPTION 'referenced replay root evidence was mutable'; + END IF; +END +$root_evidence_immutability_check$; + +DO $source_evidence_immutability_check$ +BEGIN + BEGIN + UPDATE etl_job_records + SET failure_code = 'etl_replay_generation_changed' + WHERE job_record_id = '00000000-0000-4000-8000-000000000005'; + EXCEPTION + WHEN check_violation THEN + NULL; + END; + + IF EXISTS ( + SELECT 1 + FROM etl_job_records + WHERE job_record_id = '00000000-0000-4000-8000-000000000005' + AND failure_code <> 'etl_replay_generation_failed' + ) THEN + RAISE EXCEPTION 'referenced immediate-source evidence was mutable'; + END IF; +END +$source_evidence_immutability_check$; + DO $delete_restrict_check$ BEGIN BEGIN From 7f3f16eaa488156da4d902bf7b35a6c9920eb92c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:54:58 +0900 Subject: [PATCH 070/103] docs(etl): define referenced replay evidence immutability --- .../specs/2026-08-06-durable-job-replay-design.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md b/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md index 15168b78..c8d0424c 100644 --- a/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md +++ b/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md @@ -31,7 +31,7 @@ replay_root_job_record_id replay_generation_count ``` -The root job has all three fields null. Every replay row has all three fields non-null, an immediate source different from itself, a root different from itself, and a generation from 1 through 100. Composite owner-scoped foreign keys use `ON DELETE RESTRICT` so terminal history cannot disappear through cascade deletion and one tenant cannot reference another tenant's source or root. +The root job has all three fields null. Every replay row has all three fields non-null, an immediate source different from itself, a root different from itself, and a generation from 1 through 100. The composite owner-scoped foreign keys use `ON DELETE RESTRICT` so terminal history cannot disappear through cascade deletion and one tenant cannot reference another tenant's source or root. The referenced key is the named support constraint: @@ -65,7 +65,9 @@ On replay insertion, the trigger requires all of the following: - generation 1 uses the same row for immediate source and root; - later generations inherit the exact first root and equal the source generation plus one. -On updates that name any lineage column, the trigger rejects every changed value. Lineage fields are immutable after insertion, so a maintenance script, import path, or future service cannot silently reparent a job after descendants exist. Ordinary lifecycle updates remain permitted because they do not change the lineage columns. +On updates that name any lineage column, the trigger rejects every changed value. Lineage fields are immutable after insertion, so a maintenance script, import path, or future service cannot silently reparent a job after descendants exist. + +A terminal row becomes durable evidence when a descendant names it as an immediate source or lineage root. The same trigger serializes child insertion and parent mutation with PostgreSQL row locks. It permits ordinary lifecycle updates before the first descendant exists, but after a reference exists it rejects changes to status, request digest or payload, attempt and failure state, cancellation evidence, and lifecycle timestamps. Referenced replay evidence is immutable, so an already-created descendant cannot silently acquire a different historical meaning. For the first replay: @@ -83,7 +85,7 @@ root = inherited first job generation = source generation + 1 ``` -The application independently validates the owner-scoped source, inherited root, and root-row identity before insertion. PostgreSQL independently rejects cross-owner, nonterminal, discontinuous, derived-root, and mutable lineage. Relational rows remain authoritative even when lineage is later exported as W3C PROV. +The application independently validates the owner-scoped source, inherited root, and root-row identity before insertion. PostgreSQL independently rejects cross-owner, nonterminal, discontinuous, derived-root, mutable-lineage, and referenced-evidence mutation. Relational rows remain authoritative even when lineage is later exported as W3C PROV. ## Replay-key authority @@ -116,7 +118,7 @@ One transaction performs the following sequence: 8. derive root and bounded generation; 9. require the inherited root to exist in the owner namespace and have null lineage fields; 10. insert one new `PENDING` row with the verified payload and lineage; -11. let PostgreSQL validate same-owner references, terminal source, first root, exact generation continuity, and initial lifecycle; +11. let PostgreSQL validate same-owner references, terminal source, first root, exact generation continuity, initial lifecycle, and the transition from mutable lifecycle state to referenced immutable evidence; 12. return only the new operator-safe job identity. The source is never updated. Read-then-write state resurrection is prohibited. @@ -171,7 +173,7 @@ The exact-head suite must prove: 10. concurrent creation produces one row and an in-progress or later replay outcome; 11. the new job can be claimed and follows normal lifecycle contracts; 12. service defense in depth rejects an inherited root that is itself a replay row; -13. PostgreSQL 18 rejects cross-owner references, nonterminal sources, generation-one root divergence, derived roots, skipped generations, and lineage mutation; +13. PostgreSQL 18 rejects cross-owner references, nonterminal sources, generation-one root divergence, derived roots, skipped generations, lineage mutation, and mutation of referenced root or immediate-source evidence; 14. migration completeness, source and root constraints, trigger/function presence, self-reference prohibition, naming, deletion restriction, rollout, and rollback are documented and tested; 15. all added production statements and branches retain zero-missed configured coverage and no project test is skipped. From da035607286e6deab92b14a0014ecdbb64250dde Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:56:14 +0900 Subject: [PATCH 071/103] docs(etl): operate referenced replay evidence immutability --- docs/operations/durable-job-replay.md | 31 ++++++++++++++++++++------- 1 file changed, 23 insertions(+), 8 deletions(-) diff --git a/docs/operations/durable-job-replay.md b/docs/operations/durable-job-replay.md index 6cf1ba76..61905e97 100644 --- a/docs/operations/durable-job-replay.md +++ b/docs/operations/durable-job-replay.md @@ -115,7 +115,16 @@ The `BEFORE INSERT OR UPDATE OF` database trigger enforces these additional inva - the exact root is same-owner, terminal, and has all lineage fields null; - generation 1 uses the same row as immediate source and root; - every later generation equals the source generation plus one and inherits the same first root; -- lineage fields are immutable after insertion. +- lineage fields are immutable after insertion; and +- after a descendant references a row as immediate source or root, its status, request evidence, + attempt/failure state, cancellation evidence, and lifecycle timestamps are immutable. + +The trigger uses PostgreSQL `FOR UPDATE` row locking for source/root validation and for referenced-child +inspection. This creates one fail-closed serialization boundary between child insertion and mutation +of the parent evidence. A parent update that commits before the child is inserted defines the evidence +the child subsequently validates. Once the child insertion has locked and referenced the parent, a +later conflicting parent update observes the descendant and fails with a check-violation-class +integrity error. The service performs the same root-identity check before insertion as defense in depth. The database trigger remains authoritative for maintenance scripts, data imports, and other writers that do not @@ -139,8 +148,11 @@ lease fencing, attempts, retry, success, failure, cancellation, pagination, `Ret contracts apply unchanged. No replay-specific worker or scheduler exists. Ordinary lifecycle updates may change status, payload, lease, attempts, failure, or cancellation -fields while retaining the exact lineage. Any writer that attempts to reparent a replay row or alter -its generation receives a database constraint failure and must be treated as an integrity incident. +fields while retaining the exact lineage only until that row becomes historical evidence for a +subsequent replay. After any descendant references the row as source or root, the terminal evidence +covered by V7 is frozen. Any writer that attempts to reparent a replay row, alter its generation, or +change referenced terminal evidence receives a database constraint failure and must be treated as an +integrity incident. ## Provenance export @@ -175,7 +187,9 @@ trigger enforcement, or replay-key idempotency. - derived replay rows used as root; - skipped generations; - source or root references from another `principal_scope_hash`; - - post-insert lineage mutation. + - post-insert lineage mutation; + - request-digest mutation on a referenced lineage root; and + - failure-evidence mutation on a referenced immediate source. 8. Confirm both source and root deletion remain protected by `ON DELETE RESTRICT`. 9. Claim the new pending row through the ordinary worker and verify no replay-only execution path. 10. Monitor replay acceptance, in-progress conflicts, payload mismatches, generation exhaustion, @@ -199,13 +213,14 @@ Read the already-created replay job associated with the operator's prior request principal-scoped replay intent and cannot be reused for another source or payload. Use a new key only for a deliberately separate replay. -### Trigger or lineage integrity rejection +### Trigger, referenced evidence, or lineage integrity rejection Stop replay admission for the affected deployment. Preserve the deployed SHA, Flyway history, transaction boundary, fixed error classification, and sanitized database evidence. Do not retry by -disabling the trigger or rewriting the source, root, or generation. Determine whether the attempted -write came from a stale binary, maintenance script, import path, migration defect, or unauthorized -writer. Treat cross-owner attempts as tenant-isolation incidents even when PostgreSQL rejected them. +disabling the trigger or rewriting the source, root, generation, request digest, status, failure or +cancellation evidence, or timestamps. Determine whether the attempted write came from a stale binary, +maintenance script, import path, migration defect, or unauthorized writer. Treat cross-owner attempts +as tenant-isolation incidents even when PostgreSQL rejected them. ### Broken lineage or missing root From ae74bbc9501eb2128b6e7a34ad47de5a8f5bd97b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 19:59:41 +0900 Subject: [PATCH 072/103] docs(changelog): record referenced replay evidence guard --- CHANGELOG.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 49f37399..f75d56e6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Authenticated operators can now create an ordinary pending durable job from an immutable failed or cancelled source only after resupplying a byte-identical bounded JSON payload; the terminal source remains unchanged and succeeded sources remain non-replayable. - Replay lineage now remains database-authoritative across every writer: PostgreSQL validates the terminal immediate source, first root, exact source/root/generation continuity, generation-one identity, and one-step generation succession, while immutable lineage fields prevent post-insert reparenting. +- Once a terminal job is referenced as a replay source or root, PostgreSQL row-lock serialization now freezes its status, request evidence, attempt/failure state, cancellation evidence, and lifecycle timestamps so descendants cannot silently acquire different historical meaning. - Authenticated operators can now perform owner-scoped durable-job cancellation for `PENDING` and `RUNNING` work through an idempotent action that commits terminal `CANCELLED`, clears payload and lease state, stores only `cancellation_key_hash` plus a fixed code and timestamp, and returns stable RFC 9457 conflicts when success or failure already won. - Cancellation-first races now make the former exact lease stale and roll back transactional target and response-ledger effects; success-first races remain `SUCCEEDED`, while same-key cancellation replays and different-key reuse fails closed. - Owner-scoped durable-job status responses now emit deterministic weak SHA-256 `ETag` validators; ordinary and wildcard `If-None-Match` requests return an empty RFC 9110 `304 Not Modified` response only after authenticated owner-safe lookup, while `Cache-Control: no-store` remains unchanged. @@ -40,7 +41,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - Transactional migration `V7__add_etl_job_replay_lineage.sql`, owner-scoped replay admission, the named `etl_job_owner_identity_unique` support key, composite owner-scoped foreign keys for immediate-source and root lineage, `validate_etl_job_replay_lineage()` plus `etl_job_replay_lineage_guard_trigger`, and immutable `replay_source_job_record_id`, `replay_root_job_record_id`, and `replay_generation_count` evidence with concurrency, tenant-integrity, exact-continuity, worker-compatibility, rollout, incident, and rollback tests and documentation. -- PostgreSQL 18 rehearsal now executes valid first- and second-generation replay chains and rejects nonterminal sources, different generation-one roots, derived roots, skipped generations, cross-owner references, lineage mutation, and protected source/root deletion before transactionally rehearsing trigger, function, constraint, and column rollback. +- PostgreSQL 18 rehearsal now executes valid first- and second-generation replay chains and rejects nonterminal sources, different generation-one roots, derived roots, skipped generations, cross-owner references, lineage mutation, referenced root/source evidence mutation, and protected source/root deletion before transactionally rehearsing trigger, function, constraint, and column rollback. - Transactional migration `V6__add_etl_job_cancellation.sql`, owner-safe cancellation API and replay model, exact lease-invalidation and cancellation-versus-success integration tests, plus rollout, incident, connector-limitation, and rollback evidence in `docs/operations/durable-job-cancellation.md`. - Deterministic ordinary, wildcard, changed-state, changed-failure-code, null-versus-empty, and unrelated-response conditional polling tests, complete controller Javadoc, privacy and rollback guidance, and APA 7th standards evidence in `docs/etl/durable-job-polling.md`. - Controller-scoped polling advice, deterministic active/terminal lifecycle tests, disabled-worker fail-closed behavior, sub-second rounding coverage, rollback guidance, and APA 7th RFC 9110 evidence in `docs/etl/durable-job-polling.md`. @@ -81,7 +82,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Security -- Replay uses owner-scoped source selection, byte-exact digest verification, a versioned principal-scoped key domain, immutable relational lineage, composite owner-scoped foreign keys, and a database trigger that independently rejects cross-tenant references, nonterminal sources, false roots, generation discontinuities, and lineage mutation without retaining raw principals or replay keys; this evidence does not prove external connector safety, so connector-native idempotency, transaction participation, or compensation remains required. +- Replay uses owner-scoped source selection, byte-exact digest verification, a versioned principal-scoped key domain, immutable relational lineage, composite owner-scoped foreign keys, and a database trigger that independently rejects cross-tenant references, nonterminal sources, false roots, generation discontinuities, lineage mutation, and mutation of referenced terminal evidence without retaining raw principals or replay keys; this evidence does not prove external connector safety, so connector-native idempotency, transaction participation, or compensation remains required. - Cancellation stores only a principal-scoped SHA-256 replay identity and a fixed machine code, exposes no raw principal, key, hash, payload, lease, SQL, exception, or target detail, and requires an owner-matched conditional database update before reporting success. - Conditional status validators are SHA-256 digests of only the complete owner-authorized operator-safe representation; payloads, raw principals, idempotency keys, internal hashes, leases, SQL, and exception text remain excluded, and wildcard evaluation occurs only after owner-safe lookup. - Polling advice exposes only a bounded delay integer and is omitted when local execution is disabled or terminal; it never contains job, lease, principal, key, hash, payload, SQL, exception, target, or queue-depth data. @@ -169,7 +170,7 @@ Through code analysis, identified the platform as: - Microservices-based architecture using Spring Cloud - Real-time Change Data Capture using Debezium - Event streaming via Apache Kafka -- Service discovery with Netflix Eureka +- Service discovery and registration - Distributed tracing with Zipkin #### Key Components Documented From 4a598aa65a6e5ef2d39adb419ff12bd63a1103dc Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:02:37 +0900 Subject: [PATCH 073/103] docs(adr): freeze referenced replay evidence --- ...2026-08-07-immutable-durable-job-replay.md | 30 +++++++++++++------ 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/docs/adr/2026-08-07-immutable-durable-job-replay.md b/docs/adr/2026-08-07-immutable-durable-job-replay.md index c8e07be2..b13f4c7a 100644 --- a/docs/adr/2026-08-07-immutable-durable-job-replay.md +++ b/docs/adr/2026-08-07-immutable-durable-job-replay.md @@ -13,6 +13,8 @@ Rewinding a terminal row to `PENDING` would erase the original terminal fact, mi A structural schema with nullable source, root, and generation fields is not sufficient on its own. Composite foreign keys can prove same-owner existence, and check constraints can prove field completeness and bounds, but they cannot prove that a source is terminal, a root is the first row in the chain, a generation increments exactly once, or an existing replay has never been reparented. Those properties must remain true even for maintenance scripts, data imports, and future writers that do not execute the Java service. +Cross-row validity also creates a temporal requirement. Once a descendant commits a reference to terminal source or root evidence, a later direct writer must not change the referenced status, digest, payload state, attempt/failure state, cancellation evidence, or lifecycle timestamps. Otherwise the descendant's historical meaning can change after admission even though its lineage identifiers remain untouched. Child insertion and parent mutation therefore require one database-owned serialization boundary. + ## Decision Replay creates a **new** durable job. The terminal source remains unchanged. @@ -27,7 +29,7 @@ A derived row stores: Source and root references are composite owner-scoped foreign keys to `(job_record_id, principal_scope_hash)` and use `ON DELETE RESTRICT`. A root row has all three lineage fields null; a replay row has all three non-null. Generation cannot exceed the supported bound. -V7 also creates the PL/pgSQL function `validate_etl_job_replay_lineage()` and the row-level `etl_job_replay_lineage_guard_trigger`. The trigger is the database authority for exact source/root/generation continuity: +V7 also creates the PL/pgSQL function `validate_etl_job_replay_lineage()` and the row-level `etl_job_replay_lineage_guard_trigger`. The trigger is the database authority for exact source/root/generation continuity and referenced evidence immutability: - replay rows are inserted only as `PENDING`, attempt zero, with a retained payload; - immediate source and root belong to the same principal namespace as the new row; @@ -35,7 +37,10 @@ V7 also creates the PL/pgSQL function `validate_etl_job_replay_lineage()` and th - the root is terminal and has no lineage fields; - generation one uses the same source and root; - later generations retain that root and equal the immediate source generation plus one; -- lineage columns are immutable after insertion. +- lineage columns are immutable after insertion; and +- after any descendant references a row as immediate source or root, the terminal status, request evidence, attempt/failure state, cancellation evidence, and lifecycle timestamps are immutable. + +The trigger uses PostgreSQL `FOR UPDATE` row locks for both source/root validation during child insertion and descendant lookup during a parent-evidence update. If a parent update commits first, a later child validates and binds the resulting evidence. If child insertion locks and references the parent first, a later conflicting parent update waits, observes the committed descendant, and fails closed. Ordinary lifecycle updates remain available until the row becomes referenced historical evidence. The Java service independently verifies the inherited root is an actual first root before insertion. This is defense in depth; it does not replace the trigger. The new row enters the ordinary `PENDING` lifecycle and uses the existing worker, PostgreSQL claim, exact lease, success, failure, polling, cancellation, and ETag contracts. @@ -53,11 +58,13 @@ sequenceDiagram P->>P: verify FAILED/CANCELLED and digest equality P->>P: verify replay-key identity and inherited root P->>T: insert new PENDING row with source/root/generation - T->>T: verify owner, terminal source, first root, exact successor + T->>T: FOR UPDATE source/root and verify exact evidence T-->>P: accept or reject before persistence P-->>A: commit derived job A-->>O: 202 + Location + Idempotency-Replayed W->>P: ordinary lease-fenced claim + P->>T: later mutation of referenced terminal evidence + T->>T: lock descendant rows and reject mutation ``` ## Replay identity @@ -82,7 +89,7 @@ All covered request failures use fixed RFC 9457 metadata and exclude exception m ## Connector boundary -The replay transaction can prove source ownership, payload fidelity, replay identity, exact lineage, and durable admission. It cannot prove that a remote warehouse, file system, API, or broker will suppress duplicate effects. Replay is enabled for a connector only when target effects participate in the mightyETL transaction or the connector provides independently tested idempotency or compensation. +The replay transaction can prove source ownership, payload fidelity, replay identity, exact lineage, referenced evidence immutability, and durable admission. It cannot prove that a remote warehouse, file system, API, or broker will suppress duplicate effects. Replay is enabled for a connector only when target effects participate in the mightyETL transaction or the connector provides independently tested idempotency or compensation. ## Alternatives rejected @@ -108,20 +115,24 @@ Rejected because maintenance scripts, migrations, import processes, or future se ### Use only foreign keys and check constraints -Rejected because those constraints cannot express exact generation succession or immutable cross-row root identity. A row-level trigger is required for those cross-row invariants. +Rejected because those constraints cannot express exact generation succession, immutable cross-row root identity, or the transition from mutable terminal state to referenced immutable evidence. A row-level trigger is required for those cross-row and temporal invariants. + +### Leave referenced evidence mutable + +Rejected because a descendant would preserve the same source/root identifiers while the status, digest, terminal payload state, failure or cancellation evidence, or timestamps behind those identifiers changed. Audit and provenance exports would then describe a moving historical fact. ## Consequences ### Positive -- terminal sources remain immutable; +- terminal sources remain immutable once replayed; - each execution episode has a distinct opaque job identity; - lineage supports incident analysis and future PROV-compatible export; - ordinary worker and cancellation machinery is reused; - payload retention does not increase; - concurrent retries have one database-owned outcome; - database maintenance and import paths cannot create discontinuous or reparented lineage; -- cross-owner, nonterminal, derived-root, skipped-generation, and mutation attempts fail closed. +- cross-owner, nonterminal, derived-root, skipped-generation, lineage-mutation, and referenced-evidence-mutation attempts fail closed. ### Costs @@ -129,14 +140,15 @@ Rejected because those constraints cannot express exact generation succession or - self-referencing lineage constrains retention and deletion order; - every connector needs an explicit replay-safety classification; - migrations and generation bounds require real PostgreSQL verification; -- the trigger adds a small number of same-transaction row lookups to replay insertion; +- the trigger adds same-transaction row locks and lookups to replay insertion and referenced-evidence updates; +- a terminal row cannot receive later maintenance edits after it becomes lineage evidence without a separately reviewed migration strategy; - trigger and function lifecycle must be included in downgrade and disaster-recovery rehearsals. ## Verification Acceptance requires exact-head tests for source immutability, owner isolation, exact payload matching, same-key replay, conflicting-key reuse, concurrent admission, lineage inheritance, generation exhaustion, ordinary worker behavior, cancellation compatibility, RFC 9457 responses, privacy exclusions, and zero-missed configured production coverage. -A direct-`develop` GitHub Actions gate applies every versioned migration to PostgreSQL 18, verifies replay and cancellation columns, verifies both lineage foreign keys use `ON DELETE RESTRICT`, verifies trigger and function presence, executes valid generation-one and generation-two inserts, rejects nonterminal sources, different generation-one roots, derived roots, skipped generations, cross-owner references, and lineage mutation, protects source/root deletion, rehearses transactional rollback, and creates a non-empty schema-only dump. SAST, security, dependency, SBOM, review-thread, and non-author exact-head approval gates remain mandatory. +A direct-`develop` GitHub Actions gate applies every versioned migration to PostgreSQL 18, verifies replay and cancellation columns, verifies both lineage foreign keys use `ON DELETE RESTRICT`, verifies trigger and function presence, executes valid generation-one and generation-two inserts, rejects nonterminal sources, different generation-one roots, derived roots, skipped generations, cross-owner references, lineage mutation, referenced root-digest mutation, and referenced immediate-source failure-evidence mutation, protects source/root deletion, rehearses transactional rollback, and creates a non-empty schema-only dump. SAST, security, dependency, SBOM, review-thread, and non-author exact-head approval gates remain mandatory. ## References — APA 7th edition From 45f67c313470c4d38500151a4eac54974d5cef99 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:05:10 +0900 Subject: [PATCH 074/103] test(etl): require deadlock-safe replay trigger lock order --- .../etl/job/EtlJobReplayMigrationTest.java | 40 +++++++++++++++---- 1 file changed, 33 insertions(+), 7 deletions(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java index e309cf31..b6f724ca 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java @@ -7,6 +7,7 @@ import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; +import java.util.regex.Pattern; import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertTrue; @@ -18,13 +19,7 @@ class EtlJobReplayMigrationTest { @Test void replayMigrationAddsCompleteRestrictedOwnerScopedLineage() throws IOException { - String migration = Files.readString( - projectRoot().resolve( - "etl-service/src/main/resources/db/migration/" - + "V7__add_etl_job_replay_lineage.sql" - ), - StandardCharsets.UTF_8 - ).replaceAll("\\s+", " "); + String migration = normalizedMigration(); assertTrue(migration.contains("ADD COLUMN replay_source_job_record_id UUID")); assertTrue(migration.contains("ADD COLUMN replay_root_job_record_id UUID")); @@ -96,6 +91,37 @@ void replayMigrationAddsCompleteRestrictedOwnerScopedLineage() throws IOExceptio assertFalse(migration.contains("principal_name")); } + @Test + void updateGuardAvoidsChildToAncestorLockInversion() throws IOException { + String migration = normalizedMigration(); + Pattern updateReturnsBeforeInsertValidation = Pattern.compile( + "IF TG_OP = 'UPDATE' THEN .*Referenced replay evidence is immutable.*" + + "RETURN NEW; END IF; IF NEW\\.replay_generation_count IS NULL" + ); + Pattern descendantLookupTakesRowLock = Pattern.compile( + "FROM etl_job_records AS child_record .*FOR UPDATE;.*IF FOUND THEN" + ); + + assertTrue( + updateReturnsBeforeInsertValidation.matcher(migration).find(), + "UPDATE validation must return before INSERT-only source/root locking" + ); + assertFalse( + descendantLookupTakesRowLock.matcher(migration).find(), + "Referenced-child existence checks must not lock child rows in reverse order" + ); + } + + private static String normalizedMigration() throws IOException { + return Files.readString( + projectRoot().resolve( + "etl-service/src/main/resources/db/migration/" + + "V7__add_etl_job_replay_lineage.sql" + ), + StandardCharsets.UTF_8 + ).replaceAll("\\s+", " "); + } + /** @return reactor root from repository-root or module-local execution */ private static Path projectRoot() { Path current = Paths.get(System.getProperty("user.dir")).toAbsolutePath(); From 39e406d33bebb806f6ec2b8c047db2c2fc375932 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:05:40 +0900 Subject: [PATCH 075/103] test(etl): require concurrent replay lookup indexes --- .../EtlJobReplayLookupIndexMigrationTest.java | 136 ++++++++++++++++++ 1 file changed, 136 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayLookupIndexMigrationTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayLookupIndexMigrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayLookupIndexMigrationTest.java new file mode 100644 index 00000000..5036a49d --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayLookupIndexMigrationTest.java @@ -0,0 +1,136 @@ +package com.xtrmetl.etl.job; + +import org.junit.jupiter.api.Test; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; + +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Guards nonblocking PostgreSQL indexes for replay descendant and foreign-key lookups. + * + *

The lineage trigger checks descendants whenever durable terminal evidence changes. Without + * indexes beginning with the source and root identifiers, ordinary lifecycle updates can degrade + * into full-table scans as durable-job history grows. The indexes are therefore isolated from the + * transactional lineage schema and built concurrently.

+ */ +class EtlJobReplayLookupIndexMigrationTest { + + private static final String V7_MIGRATION = + "etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql"; + private static final String V8_MIGRATION = + "etl-service/src/main/resources/db/migration/V8__add_etl_job_replay_lookup_indexes.sql"; + private static final String V8_CONFIGURATION = V8_MIGRATION + ".conf"; + + @Test + void separatesTransactionalLineageFromConcurrentLookupIndexes() throws IOException { + String lineageMigration = normalize(read(V7_MIGRATION)); + String indexMigration = normalize(read(V8_MIGRATION)); + + assertFalse( + lineageMigration.contains("CREATE INDEX"), + "the transactional lineage migration must not contain a production index build" + ); + assertTrue(indexMigration.contains( + "CREATE INDEX CONCURRENTLY etl_job_replay_source_lookup_index" + )); + assertTrue(indexMigration.contains( + "ON etl_job_records ( replay_source_job_record_id, principal_scope_hash )" + )); + assertTrue(indexMigration.contains( + "WHERE replay_source_job_record_id IS NOT NULL" + )); + assertTrue(indexMigration.contains( + "CREATE INDEX CONCURRENTLY etl_job_replay_root_lookup_index" + )); + assertTrue(indexMigration.contains( + "ON etl_job_records ( replay_root_job_record_id, principal_scope_hash )" + )); + assertTrue(indexMigration.contains( + "WHERE replay_root_job_record_id IS NOT NULL" + )); + } + + @Test + void disablesFlywayTransactionForConcurrentReplayIndexes() throws IOException { + Path configurationPath = projectRoot().resolve(V8_CONFIGURATION); + String applicationProperties = read( + "etl-service/src/main/resources/application.properties" + ); + + assertTrue( + Files.exists(configurationPath), + "the concurrent replay-index migration requires a Flyway script configuration" + ); + assertTrue( + Files.readString(configurationPath, StandardCharsets.UTF_8) + .contains("executeInTransaction=false") + ); + assertTrue(applicationProperties.contains( + "spring.flyway.postgresql.transactional-lock=false" + )); + } + + @Test + void verifierRequiresReadyAndValidReplayLookupIndexes() throws IOException { + String verifier = normalize(read("scripts/verify-postgresql-migrations.sh")); + + assertTrue(verifier.contains("etl_job_replay_source_lookup_index")); + assertTrue(verifier.contains("etl_job_replay_root_lookup_index")); + assertTrue(verifier.contains("index_record.indisready")); + assertTrue(verifier.contains("index_record.indisvalid")); + assertTrue(verifier.contains("replay lookup indexes are missing or invalid")); + } + + @Test + void runbookDocumentsConcurrentFailureRecoveryAndRollback() throws IOException { + String runbook = normalize(read("docs/operations/durable-job-replay.md")); + + assertTrue(runbook.contains("V8__add_etl_job_replay_lookup_indexes.sql")); + assertTrue(runbook.contains("CREATE INDEX CONCURRENTLY")); + assertTrue(runbook.contains("invalid index")); + assertTrue(runbook.contains( + "DROP INDEX CONCURRENTLY etl_job_replay_source_lookup_index" + )); + assertTrue(runbook.contains( + "DROP INDEX CONCURRENTLY etl_job_replay_root_lookup_index" + )); + assertTrue(runbook.contains("executeInTransaction=false")); + } + + private static String read(String relativePath) throws IOException { + return Files.readString(projectRoot().resolve(relativePath), StandardCharsets.UTF_8); + } + + private static String normalize(String value) { + return value.replaceAll("\\s+", " ").trim(); + } + + /** + * Finds the reactor root from repository-root or module-local Maven execution. + * + * @return repository root containing migrations, scripts, and documentation + */ + private static Path projectRoot() { + Path current = Paths.get(System.getProperty("user.dir")).toAbsolutePath(); + Path lastPomParent = null; + while (current != null) { + if (Files.exists(current.resolve(".git"))) { + return current; + } + if (Files.exists(current.resolve("pom.xml"))) { + lastPomParent = current; + } + current = current.getParent(); + } + if (lastPomParent != null) { + return lastPomParent; + } + throw new IllegalStateException("Could not find project root"); + } +} From dcfc48c301ff1614edb1f481f26b16b8d5eb60c1 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:10:09 +0900 Subject: [PATCH 076/103] test(etl): isolate concurrent replay index migrations --- .../EtlJobReplayLookupIndexMigrationTest.java | 61 ++++++++++++------- 1 file changed, 40 insertions(+), 21 deletions(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayLookupIndexMigrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayLookupIndexMigrationTest.java index 5036a49d..dd4dae28 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayLookupIndexMigrationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayLookupIndexMigrationTest.java @@ -16,61 +16,79 @@ * *

The lineage trigger checks descendants whenever durable terminal evidence changes. Without * indexes beginning with the source and root identifiers, ordinary lifecycle updates can degrade - * into full-table scans as durable-job history grows. The indexes are therefore isolated from the - * transactional lineage schema and built concurrently.

+ * into full-table scans as durable-job history grows. Each index is isolated from the transactional + * lineage schema and from the other index so a concurrent-build failure has one auditable Flyway + * repair boundary.

*/ class EtlJobReplayLookupIndexMigrationTest { private static final String V7_MIGRATION = "etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql"; private static final String V8_MIGRATION = - "etl-service/src/main/resources/db/migration/V8__add_etl_job_replay_lookup_indexes.sql"; + "etl-service/src/main/resources/db/migration/" + + "V8__add_etl_job_replay_source_lookup_index.sql"; + private static final String V9_MIGRATION = + "etl-service/src/main/resources/db/migration/" + + "V9__add_etl_job_replay_root_lookup_index.sql"; private static final String V8_CONFIGURATION = V8_MIGRATION + ".conf"; + private static final String V9_CONFIGURATION = V9_MIGRATION + ".conf"; @Test void separatesTransactionalLineageFromConcurrentLookupIndexes() throws IOException { String lineageMigration = normalize(read(V7_MIGRATION)); - String indexMigration = normalize(read(V8_MIGRATION)); + String sourceIndexMigration = normalize(read(V8_MIGRATION)); + String rootIndexMigration = normalize(read(V9_MIGRATION)); assertFalse( lineageMigration.contains("CREATE INDEX"), "the transactional lineage migration must not contain a production index build" ); - assertTrue(indexMigration.contains( + assertTrue(sourceIndexMigration.contains( "CREATE INDEX CONCURRENTLY etl_job_replay_source_lookup_index" )); - assertTrue(indexMigration.contains( + assertTrue(sourceIndexMigration.contains( "ON etl_job_records ( replay_source_job_record_id, principal_scope_hash )" )); - assertTrue(indexMigration.contains( + assertTrue(sourceIndexMigration.contains( "WHERE replay_source_job_record_id IS NOT NULL" )); - assertTrue(indexMigration.contains( + assertFalse( + sourceIndexMigration.contains("etl_job_replay_root_lookup_index"), + "one nontransactional migration must own only one concurrent index build" + ); + + assertTrue(rootIndexMigration.contains( "CREATE INDEX CONCURRENTLY etl_job_replay_root_lookup_index" )); - assertTrue(indexMigration.contains( + assertTrue(rootIndexMigration.contains( "ON etl_job_records ( replay_root_job_record_id, principal_scope_hash )" )); - assertTrue(indexMigration.contains( + assertTrue(rootIndexMigration.contains( "WHERE replay_root_job_record_id IS NOT NULL" )); + assertFalse( + rootIndexMigration.contains("etl_job_replay_source_lookup_index"), + "one nontransactional migration must own only one concurrent index build" + ); } @Test - void disablesFlywayTransactionForConcurrentReplayIndexes() throws IOException { - Path configurationPath = projectRoot().resolve(V8_CONFIGURATION); + void disablesFlywayTransactionForEachConcurrentReplayIndex() throws IOException { String applicationProperties = read( "etl-service/src/main/resources/application.properties" ); - assertTrue( - Files.exists(configurationPath), - "the concurrent replay-index migration requires a Flyway script configuration" - ); - assertTrue( - Files.readString(configurationPath, StandardCharsets.UTF_8) - .contains("executeInTransaction=false") - ); + for (String configuration : new String[]{V8_CONFIGURATION, V9_CONFIGURATION}) { + Path configurationPath = projectRoot().resolve(configuration); + assertTrue( + Files.exists(configurationPath), + "each concurrent replay-index migration requires a Flyway script configuration" + ); + assertTrue( + Files.readString(configurationPath, StandardCharsets.UTF_8) + .contains("executeInTransaction=false") + ); + } assertTrue(applicationProperties.contains( "spring.flyway.postgresql.transactional-lock=false" )); @@ -91,7 +109,8 @@ void verifierRequiresReadyAndValidReplayLookupIndexes() throws IOException { void runbookDocumentsConcurrentFailureRecoveryAndRollback() throws IOException { String runbook = normalize(read("docs/operations/durable-job-replay.md")); - assertTrue(runbook.contains("V8__add_etl_job_replay_lookup_indexes.sql")); + assertTrue(runbook.contains("V8__add_etl_job_replay_source_lookup_index.sql")); + assertTrue(runbook.contains("V9__add_etl_job_replay_root_lookup_index.sql")); assertTrue(runbook.contains("CREATE INDEX CONCURRENTLY")); assertTrue(runbook.contains("invalid index")); assertTrue(runbook.contains( From 99d5cbc4c5bc70a63777890a7fe2d2f9f8945879 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:10:14 +0900 Subject: [PATCH 077/103] test(etl): require replay digest continuity --- ...obReplayDigestContinuityMigrationTest.java | 76 +++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayDigestContinuityMigrationTest.java diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayDigestContinuityMigrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayDigestContinuityMigrationTest.java new file mode 100644 index 00000000..2c6b53fd --- /dev/null +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayDigestContinuityMigrationTest.java @@ -0,0 +1,76 @@ +package com.xtrmetl.etl.job; + +import org.junit.jupiter.api.Test; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; + +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Guards database-level digest continuity for immutable durable-job replay lineage. + * + *

Application admission already compares the resupplied payload digest with the terminal + * source. The database trigger is an independent integrity boundary for imports and other direct + * writers, so every derived row must retain the exact immediate source request digest.

+ */ +class EtlJobReplayDigestContinuityMigrationTest { + + @Test + void lineageTriggerRequiresImmediateSourceDigestEquality() throws IOException { + String migration = normalize(read( + "etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql" + )); + + assertTrue(migration.contains("source_request_digest")); + assertTrue(migration.contains("request_digest")); + assertTrue(migration.contains( + "source_request_digest IS DISTINCT FROM NEW.request_digest" + )); + assertTrue(migration.contains("Replay request digest must match the immediate source")); + } + + @Test + void postgresqlRehearsalRejectsMismatchedReplayDigest() throws IOException { + String rehearsal = normalize(read( + "etl-service/src/test/postgresql/replay_lineage_migration.sql" + )); + + assertTrue(rehearsal.contains("digest_continuity_check")); + assertTrue(rehearsal.contains("replay digest mismatch was accepted")); + } + + private static String read(String relativePath) throws IOException { + return Files.readString(projectRoot().resolve(relativePath), StandardCharsets.UTF_8); + } + + private static String normalize(String value) { + return value.replaceAll("\\s+", " ").trim(); + } + + /** + * Finds the reactor root from repository-root or module-local Maven execution. + * + * @return repository root containing migrations and integration fixtures + */ + private static Path projectRoot() { + Path current = Paths.get(System.getProperty("user.dir")).toAbsolutePath(); + Path lastPomParent = null; + while (current != null) { + if (Files.exists(current.resolve(".git"))) { + return current; + } + if (Files.exists(current.resolve("pom.xml"))) { + lastPomParent = current; + } + current = current.getParent(); + } + if (lastPomParent != null) { + return lastPomParent; + } + throw new IllegalStateException("Could not find project root"); + } +} From d0b2e35159ce5b2dcb600a51aec353c0d326f63c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:12:14 +0900 Subject: [PATCH 078/103] fix(etl): serialize immutable replay evidence safely --- .../V7__add_etl_job_replay_lineage.sql | 82 ++++++++++--------- 1 file changed, 44 insertions(+), 38 deletions(-) diff --git a/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql b/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql index 6cadd16c..609a645c 100644 --- a/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql +++ b/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql @@ -41,6 +41,7 @@ LANGUAGE plpgsql AS $etl_job_replay_lineage$ DECLARE source_job_status VARCHAR(32); + source_request_digest CHAR(64); source_source_job_record_id UUID; source_root_job_record_id UUID; source_generation_count INTEGER; @@ -49,63 +50,63 @@ DECLARE root_root_job_record_id UUID; root_generation_count INTEGER; BEGIN - IF TG_OP = 'UPDATE' - AND ( - OLD.replay_source_job_record_id + IF TG_OP = 'UPDATE' THEN + IF OLD.replay_source_job_record_id IS DISTINCT FROM NEW.replay_source_job_record_id - OR OLD.replay_root_job_record_id + OR OLD.replay_root_job_record_id IS DISTINCT FROM NEW.replay_root_job_record_id - OR OLD.replay_generation_count - IS DISTINCT FROM NEW.replay_generation_count - ) THEN - RAISE EXCEPTION 'Replay lineage fields are immutable' - USING ERRCODE = '23514'; - END IF; - - IF TG_OP = 'UPDATE' - AND ( - OLD.job_status IS DISTINCT FROM NEW.job_status - OR OLD.request_digest IS DISTINCT FROM NEW.request_digest - OR OLD.request_payload IS DISTINCT FROM NEW.request_payload - OR OLD.attempt_count IS DISTINCT FROM NEW.attempt_count - OR OLD.failure_code IS DISTINCT FROM NEW.failure_code - OR OLD.cancellation_key_hash IS DISTINCT FROM NEW.cancellation_key_hash - OR OLD.cancellation_code IS DISTINCT FROM NEW.cancellation_code - OR OLD.job_cancelled_at IS DISTINCT FROM NEW.job_cancelled_at - OR OLD.created_at IS DISTINCT FROM NEW.created_at - OR OLD.updated_at IS DISTINCT FROM NEW.updated_at - ) THEN - PERFORM 1 - FROM etl_job_records AS child_record - WHERE child_record.replay_source_job_record_id = OLD.job_record_id - OR child_record.replay_root_job_record_id = OLD.job_record_id - FOR UPDATE; - - IF FOUND THEN - RAISE EXCEPTION 'Referenced replay evidence is immutable' + OR OLD.replay_generation_count + IS DISTINCT FROM NEW.replay_generation_count THEN + RAISE EXCEPTION 'Replay lineage fields are immutable' USING ERRCODE = '23514'; END IF; + + IF OLD.job_status IS DISTINCT FROM NEW.job_status + OR OLD.request_digest IS DISTINCT FROM NEW.request_digest + OR OLD.request_payload IS DISTINCT FROM NEW.request_payload + OR OLD.attempt_count IS DISTINCT FROM NEW.attempt_count + OR OLD.failure_code IS DISTINCT FROM NEW.failure_code + OR OLD.cancellation_key_hash IS DISTINCT FROM NEW.cancellation_key_hash + OR OLD.cancellation_code IS DISTINCT FROM NEW.cancellation_code + OR OLD.job_cancelled_at IS DISTINCT FROM NEW.job_cancelled_at + OR OLD.created_at IS DISTINCT FROM NEW.created_at + OR OLD.updated_at IS DISTINCT FROM NEW.updated_at THEN + -- The row being updated is already locked by PostgreSQL. Every child insertion + -- locks its source and root before it can commit, so an existence lookup is enough + -- to serialize this mutation without taking child locks in the reverse direction. + PERFORM 1 + FROM etl_job_records AS child_record + WHERE child_record.replay_source_job_record_id = OLD.job_record_id + OR child_record.replay_root_job_record_id = OLD.job_record_id + LIMIT 1; + + IF FOUND THEN + RAISE EXCEPTION 'Referenced replay evidence is immutable' + USING ERRCODE = '23514'; + END IF; + END IF; + + RETURN NEW; END IF; IF NEW.replay_generation_count IS NULL THEN RETURN NEW; END IF; - IF TG_OP = 'INSERT' - AND ( - NEW.job_status <> 'PENDING' - OR NEW.attempt_count <> 0 - OR NEW.request_payload IS NULL - ) THEN + IF NEW.job_status <> 'PENDING' + OR NEW.attempt_count <> 0 + OR NEW.request_payload IS NULL THEN RAISE EXCEPTION 'Replay rows must start as pending jobs' USING ERRCODE = '23514'; END IF; SELECT job_status, + request_digest, replay_source_job_record_id, replay_root_job_record_id, replay_generation_count INTO source_job_status, + source_request_digest, source_source_job_record_id, source_root_job_record_id, source_generation_count @@ -124,6 +125,11 @@ BEGIN USING ERRCODE = '23514'; END IF; + IF source_request_digest IS DISTINCT FROM NEW.request_digest THEN + RAISE EXCEPTION 'Replay request digest must match the immediate source' + USING ERRCODE = '23514'; + END IF; + SELECT job_status, replay_source_job_record_id, replay_root_job_record_id, From 05179e7b36811cedcb132a79159fa8af67e2bafa Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:12:39 +0900 Subject: [PATCH 079/103] perf(etl): index replay source lookups concurrently --- .../V8__add_etl_job_replay_source_lookup_index.sql | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 etl-service/src/main/resources/db/migration/V8__add_etl_job_replay_source_lookup_index.sql diff --git a/etl-service/src/main/resources/db/migration/V8__add_etl_job_replay_source_lookup_index.sql b/etl-service/src/main/resources/db/migration/V8__add_etl_job_replay_source_lookup_index.sql new file mode 100644 index 00000000..8e1c175b --- /dev/null +++ b/etl-service/src/main/resources/db/migration/V8__add_etl_job_replay_source_lookup_index.sql @@ -0,0 +1,10 @@ +-- Support replay-source foreign-key enforcement and immutable-evidence descendant lookup. +-- CONCURRENTLY preserves durable-job inserts, lifecycle updates, and deletes during rollout. +-- The companion .sql.conf disables Flyway's per-migration transaction because PostgreSQL +-- rejects CREATE INDEX CONCURRENTLY inside a transaction block. +CREATE INDEX CONCURRENTLY etl_job_replay_source_lookup_index + ON etl_job_records ( + replay_source_job_record_id, + principal_scope_hash + ) + WHERE replay_source_job_record_id IS NOT NULL; From 40353199f38844f3c656e9a269dd0c4e52da6af4 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:12:57 +0900 Subject: [PATCH 080/103] build(etl): run replay source index outside transaction --- .../V8__add_etl_job_replay_source_lookup_index.sql.conf | 1 + 1 file changed, 1 insertion(+) create mode 100644 etl-service/src/main/resources/db/migration/V8__add_etl_job_replay_source_lookup_index.sql.conf diff --git a/etl-service/src/main/resources/db/migration/V8__add_etl_job_replay_source_lookup_index.sql.conf b/etl-service/src/main/resources/db/migration/V8__add_etl_job_replay_source_lookup_index.sql.conf new file mode 100644 index 00000000..73bd53a1 --- /dev/null +++ b/etl-service/src/main/resources/db/migration/V8__add_etl_job_replay_source_lookup_index.sql.conf @@ -0,0 +1 @@ +executeInTransaction=false From 75d6e6af8aae9c570165f26b0a56a5ad23b7e696 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:13:17 +0900 Subject: [PATCH 081/103] perf(etl): index replay root lookups concurrently --- .../V9__add_etl_job_replay_root_lookup_index.sql | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 etl-service/src/main/resources/db/migration/V9__add_etl_job_replay_root_lookup_index.sql diff --git a/etl-service/src/main/resources/db/migration/V9__add_etl_job_replay_root_lookup_index.sql b/etl-service/src/main/resources/db/migration/V9__add_etl_job_replay_root_lookup_index.sql new file mode 100644 index 00000000..1df3bc28 --- /dev/null +++ b/etl-service/src/main/resources/db/migration/V9__add_etl_job_replay_root_lookup_index.sql @@ -0,0 +1,10 @@ +-- Support replay-root foreign-key enforcement and immutable-evidence descendant lookup. +-- CONCURRENTLY preserves durable-job inserts, lifecycle updates, and deletes during rollout. +-- The companion .sql.conf disables Flyway's per-migration transaction because PostgreSQL +-- rejects CREATE INDEX CONCURRENTLY inside a transaction block. +CREATE INDEX CONCURRENTLY etl_job_replay_root_lookup_index + ON etl_job_records ( + replay_root_job_record_id, + principal_scope_hash + ) + WHERE replay_root_job_record_id IS NOT NULL; From 2b0955d9178a4b164fba0d15a46e5855c4cc51a0 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:13:44 +0900 Subject: [PATCH 082/103] build(etl): run replay root index outside transaction --- .../migration/V9__add_etl_job_replay_root_lookup_index.sql.conf | 1 + 1 file changed, 1 insertion(+) create mode 100644 etl-service/src/main/resources/db/migration/V9__add_etl_job_replay_root_lookup_index.sql.conf diff --git a/etl-service/src/main/resources/db/migration/V9__add_etl_job_replay_root_lookup_index.sql.conf b/etl-service/src/main/resources/db/migration/V9__add_etl_job_replay_root_lookup_index.sql.conf new file mode 100644 index 00000000..73bd53a1 --- /dev/null +++ b/etl-service/src/main/resources/db/migration/V9__add_etl_job_replay_root_lookup_index.sql.conf @@ -0,0 +1 @@ +executeInTransaction=false From a9c3c7614cc553aa049bbf38184ed08add462578 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:15:12 +0900 Subject: [PATCH 083/103] test(etl): verify replay lookup indexes in PostgreSQL --- scripts/verify-postgresql-migrations.sh | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/scripts/verify-postgresql-migrations.sh b/scripts/verify-postgresql-migrations.sh index 742d1ad5..46946bb1 100644 --- a/scripts/verify-postgresql-migrations.sh +++ b/scripts/verify-postgresql-migrations.sh @@ -43,6 +43,7 @@ DO $verification_block$ DECLARE missing_column_count integer; restrict_foreign_key_count integer; + replay_lookup_index_count integer; replay_check_definition text; cancellation_check_definition text; BEGIN @@ -97,6 +98,25 @@ BEGIN RAISE EXCEPTION 'replay source and root must each use a self-reference with ON DELETE RESTRICT'; END IF; + SELECT count(*) + INTO replay_lookup_index_count + FROM pg_class AS index_class + JOIN pg_index AS index_record + ON index_record.indexrelid = index_class.oid + JOIN pg_class AS table_record + ON table_record.oid = index_record.indrelid + WHERE table_record.relname = 'etl_job_records' + AND index_class.relname IN ( + 'etl_job_replay_source_lookup_index', + 'etl_job_replay_root_lookup_index' + ) + AND index_record.indisready + AND index_record.indisvalid; + + IF replay_lookup_index_count <> 2 THEN + RAISE EXCEPTION 'replay lookup indexes are missing or invalid'; + END IF; + SELECT string_agg(pg_get_constraintdef(constraint_record.oid), ' ') INTO replay_check_definition FROM pg_constraint AS constraint_record From 3a2fb6dae3b3527357ce6ad8f62b07fb290c8d34 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:17:17 +0900 Subject: [PATCH 084/103] test(etl): require replay controller coverage --- .../test/java/com/xtrmetl/etl/job/EtlJobCoveragePolicyTest.java | 1 + 1 file changed, 1 insertion(+) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCoveragePolicyTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCoveragePolicyTest.java index 74fa5e0a..9f53a5b6 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCoveragePolicyTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCoveragePolicyTest.java @@ -36,6 +36,7 @@ class EtlJobCoveragePolicyTest { private static final Set DURABLE_JOB_CLASS_FILES = Set.of( "com/xtrmetl/etl/job/*.class", "com/xtrmetl/etl/controller/EtlJobController*.class", + "com/xtrmetl/etl/controller/EtlJobReplayController*.class", "com/xtrmetl/etl/service/Sha256Digest*.class" ); From 946740624632430be733bd407c7ea107eb35397d Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:19:04 +0900 Subject: [PATCH 085/103] fix(etl): enforce replay controller coverage --- etl-service/pom.xml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/etl-service/pom.xml b/etl-service/pom.xml index 9e0e56b5..2ebc1d80 100644 --- a/etl-service/pom.xml +++ b/etl-service/pom.xml @@ -116,6 +116,7 @@ com/xtrmetl/etl/job/*.class com/xtrmetl/etl/controller/EtlJobController*.class + com/xtrmetl/etl/controller/EtlJobReplayController*.class com/xtrmetl/etl/service/Sha256Digest*.class @@ -130,6 +131,7 @@ com/xtrmetl/etl/job/*.class com/xtrmetl/etl/controller/EtlJobController*.class + com/xtrmetl/etl/controller/EtlJobReplayController*.class com/xtrmetl/etl/service/Sha256Digest*.class From 1ead40834fce37f752d753548848b7123ec65b5b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:20:27 +0900 Subject: [PATCH 086/103] docs(etl): document replay index recovery --- docs/operations/durable-job-replay.md | 104 +++++++++++++++++++------- 1 file changed, 79 insertions(+), 25 deletions(-) diff --git a/docs/operations/durable-job-replay.md b/docs/operations/durable-job-replay.md index 61905e97..3e6d3917 100644 --- a/docs/operations/durable-job-replay.md +++ b/docs/operations/durable-job-replay.md @@ -101,7 +101,8 @@ import path attempts to pair one tenant's new job with another tenant's source o owner predicates remain mandatory, but they are no longer the only tenant-integrity boundary. Declarative foreign keys and checks do not establish that the selected source is terminal, that the -root is the first row in the lineage, or that generation advances exactly once. V7 therefore creates: +root is the first row in the lineage, that the replay request digest equals the immediate source +digest, or that generation advances exactly once. V7 therefore creates: ```text validate_etl_job_replay_lineage() @@ -112,6 +113,7 @@ The `BEFORE INSERT OR UPDATE OF` database trigger enforces these additional inva - replay-created rows start as `PENDING`, with attempt zero and a retained payload; - the exact immediate source is same-owner and `FAILED` or `CANCELLED`; +- the replay row's `request_digest` exactly equals the immediate source `request_digest`; - the exact root is same-owner, terminal, and has all lineage fields null; - generation 1 uses the same row as immediate source and root; - every later generation equals the source generation plus one and inherits the same first root; @@ -119,16 +121,17 @@ The `BEFORE INSERT OR UPDATE OF` database trigger enforces these additional inva - after a descendant references a row as immediate source or root, its status, request evidence, attempt/failure state, cancellation evidence, and lifecycle timestamps are immutable. -The trigger uses PostgreSQL `FOR UPDATE` row locking for source/root validation and for referenced-child -inspection. This creates one fail-closed serialization boundary between child insertion and mutation -of the parent evidence. A parent update that commits before the child is inserted defines the evidence -the child subsequently validates. Once the child insertion has locked and referenced the parent, a -later conflicting parent update observes the descendant and fails with a check-violation-class -integrity error. +Child insertion locks the immediate source and root with PostgreSQL `FOR UPDATE` while validating +lineage. A parent update already owns the parent row lock before the trigger executes; if the update +would change immutable replay evidence, the trigger performs a bounded descendant existence lookup +without taking child locks. This preserves one lock direction: a child waits for its source/root, +while a parent never waits on a child and then reaches back to the ancestor. The result is a +fail-closed serialization boundary without the child-to-ancestor lock inversion that could otherwise +deadlock concurrent replay admission and lifecycle maintenance. -The service performs the same root-identity check before insertion as defense in depth. The database -trigger remains authoritative for maintenance scripts, data imports, and other writers that do not -execute Java service code. +The service performs the same root-identity and request-digest checks before insertion as defense in +depth. The database trigger remains authoritative for maintenance scripts, data imports, and other +writers that do not execute Java service code. ```mermaid flowchart LR @@ -141,6 +144,21 @@ A replay of a root uses the source as root and generation 1. A replay of a repla root and increments the immediate source generation. Generation 100 returns `409 etl_job_replay_generation_exhausted` instead of creating generation 101. +## Replay lookup indexes + +`V8__add_etl_job_replay_source_lookup_index.sql` and +`V9__add_etl_job_replay_root_lookup_index.sql` build partial indexes whose leading columns are the +immediate-source and first-root identifiers. They support composite foreign-key enforcement and the +bounded descendant existence lookup used by immutable-evidence updates. + +Each migration owns exactly one `CREATE INDEX CONCURRENTLY` statement so ordinary durable-job +inserts, lifecycle updates, and deletes remain available during the build. PostgreSQL prohibits +concurrent index creation inside a transaction block, so the companion `.sql.conf` files set +`executeInTransaction=false`; application configuration also sets +`spring.flyway.postgresql.transactional-lock=false` so Flyway does not wrap these migrations in a +PostgreSQL transactional advisory lock. Migration verification requires both indexes to be +`indisready` and `indisvalid` before rollout is considered complete. + ## Worker behavior The new row is an ordinary `PENDING` job with the verified payload. The existing PostgreSQL claim, @@ -172,17 +190,23 @@ trigger enforcement, or replay-key idempotency. ## Rollout -1. Rehearse V7 on a representative PostgreSQL 18 copy and inspect existing row count, support-index - build time, table lock duration, foreign-key validation time, and trigger creation. -2. Verify exact-head cross-platform CI, full reactor tests, zero-missed configured coverage, +1. Rehearse V7 on a representative PostgreSQL 18 copy and inspect existing row count, table lock + duration, foreign-key validation time, and trigger creation. +2. Rehearse `V8__add_etl_job_replay_source_lookup_index.sql` and + `V9__add_etl_job_replay_root_lookup_index.sql` independently and record concurrent build duration, + disk growth, lock waits, and whether each resulting index is ready and valid. +3. Verify exact-head cross-platform CI, full reactor tests, zero-missed configured coverage, dependency review, SBOM, SAST, security scan, review threads, and independent approval. -3. Apply V7 before serving the replay route. -4. Verify that exactly one `validate_etl_job_replay_lineage` function and one - `etl_job_replay_lineage_guard_trigger` exist on `etl_job_records`. -5. Smoke-test a disposable failed source, exact payload acceptance, same-key retry, and key conflict. -6. Confirm the source is unchanged and the new row has source/root/generation lineage. -7. In an isolated migration rehearsal, confirm PostgreSQL rejects: +4. Apply V7 before serving the replay route, then apply V8 and V9 before treating replay migrations + as operationally ready. +5. Verify that exactly one `validate_etl_job_replay_lineage` function and one + `etl_job_replay_lineage_guard_trigger` exist on `etl_job_records`, and that both replay lookup + indexes are ready and valid. +6. Smoke-test a disposable failed source, exact payload acceptance, same-key retry, and key conflict. +7. Confirm the source is unchanged and the new row has source/root/generation lineage. +8. In an isolated migration rehearsal, confirm PostgreSQL rejects: - nonterminal replay sources; + - replay rows whose request digest differs from their immediate source; - generation-one rows whose source differs from root; - derived replay rows used as root; - skipped generations; @@ -190,11 +214,11 @@ trigger enforcement, or replay-key idempotency. - post-insert lineage mutation; - request-digest mutation on a referenced lineage root; and - failure-evidence mutation on a referenced immediate source. -8. Confirm both source and root deletion remain protected by `ON DELETE RESTRICT`. -9. Claim the new pending row through the ordinary worker and verify no replay-only execution path. -10. Monitor replay acceptance, in-progress conflicts, payload mismatches, generation exhaustion, - database lock waits, trigger rejections, and failed foreign-key deletion or tenant-boundary - attempts using fixed-cardinality signals. +9. Confirm both source and root deletion remain protected by `ON DELETE RESTRICT`. +10. Claim the new pending row through the ordinary worker and verify no replay-only execution path. +11. Monitor replay acceptance, in-progress conflicts, payload mismatches, generation exhaustion, + database lock waits, trigger rejections, concurrent-index failures, and failed foreign-key + deletion or tenant-boundary attempts using fixed-cardinality signals. Logs and metric labels must not contain payloads, raw principals, raw keys, hashes, source/new job identifiers, lineage identifiers, SQL, exception messages, or target identities. @@ -213,6 +237,27 @@ Read the already-created replay job associated with the operator's prior request principal-scoped replay intent and cannot be reused for another source or payload. Use a new key only for a deliberately separate replay. +### Concurrent replay-index migration failure + +A cancelled or interrupted `CREATE INDEX CONCURRENTLY` can leave an invalid index behind. Stop the +migration rollout and preserve the exact application SHA, Flyway schema history, PostgreSQL logs, and +sanitized index metadata. Do not mark the migration successful while either replay lookup index is +missing, not ready, or invalid. + +Confirm no active migration process is still using the affected index, then remove only the invalid +artifact outside an explicit transaction: + +```sql +DROP INDEX CONCURRENTLY etl_job_replay_source_lookup_index; +DROP INDEX CONCURRENTLY etl_job_replay_root_lookup_index; +``` + +Drop only the index that actually failed; the two commands are shown together as the complete replay +index inventory. After the invalid index is removed, repair the failed Flyway migration record using +the approved deployment procedure and rerun that exact migration. Its companion configuration must +still contain `executeInTransaction=false`. Do not edit an applied migration, create an untracked +replacement index, or bypass the ready/valid verification query. + ### Trigger, referenced evidence, or lineage integrity rejection Stop replay admission for the affected deployment. Preserve the deployed SHA, Flyway history, @@ -235,12 +280,18 @@ Stop serving replay admission before rolling application binaries back. Older bi columns, but deletion or retention tooling might not understand the new `ON DELETE RESTRICT` relationships or trigger. +V8 and V9 are nontransactional by design. If a rollback requires removing their indexes, perform +`DROP INDEX CONCURRENTLY etl_job_replay_root_lookup_index` and +`DROP INDEX CONCURRENTLY etl_job_replay_source_lookup_index` outside an explicit transaction before +rolling back V7. Preserve Flyway history and record the operational reason; do not pretend a partial +concurrent build was atomic. + Do not drop V7 while replay rows exist. Archive or remove replay lineages from leaf to root under an approved retention policy, preserving external audit evidence. Then a separately reviewed migration may remove the trigger, function, composite foreign keys, `etl_job_owner_identity_unique`, and lineage columns. Never edit the applied V7 file or mutate terminal sources back to pending. -A controlled rollback rehearsal must drop the trigger before its function and remove dependent +A controlled V7 rollback rehearsal must drop the trigger before its function and remove dependent constraints before columns, all inside a transaction that is rolled back. After rollback, verify all three columns, four named constraints, the trigger, and the function are restored. @@ -265,6 +316,9 @@ https://www.rfc-editor.org/rfc/rfc9457 PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: Constraints*. https://www.postgresql.org/docs/18/ddl-constraints.html +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: CREATE INDEX*. +https://www.postgresql.org/docs/18/sql-createindex.html + PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: CREATE TRIGGER*. https://www.postgresql.org/docs/18/sql-createtrigger.html From 228d08f95d8f6136b614711bcb39894ba458a225 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:21:07 +0900 Subject: [PATCH 087/103] test(etl): rehearse exact replay digest and index integrity --- .../postgresql/replay_lineage_migration.sql | 366 +++++++----------- 1 file changed, 132 insertions(+), 234 deletions(-) diff --git a/etl-service/src/test/postgresql/replay_lineage_migration.sql b/etl-service/src/test/postgresql/replay_lineage_migration.sql index 6b01a2de..60b5893c 100644 --- a/etl-service/src/test/postgresql/replay_lineage_migration.sql +++ b/etl-service/src/test/postgresql/replay_lineage_migration.sql @@ -1,10 +1,11 @@ --- Rehearse V7 owner isolation, exact lineage continuity, deletion protection, and rollback. +-- Rehearse V7 lineage authority plus V8/V9 replay lookup indexes on PostgreSQL 18. -- This script runs only against the disposable PostgreSQL integration-test database. DO $migration_object_check$ DECLARE trigger_count integer; function_count integer; + lookup_index_count integer; BEGIN SELECT count(*) INTO trigger_count @@ -20,7 +21,19 @@ BEGIN FROM pg_proc AS function_record WHERE function_record.proname = 'validate_etl_job_replay_lineage'; - IF trigger_count <> 1 OR function_count <> 1 THEN + SELECT count(*) + INTO lookup_index_count + FROM pg_class AS index_class + JOIN pg_index AS index_record + ON index_record.indexrelid = index_class.oid + WHERE index_class.relname IN ( + 'etl_job_replay_source_lookup_index', + 'etl_job_replay_root_lookup_index' + ) + AND index_record.indisready + AND index_record.indisvalid; + + IF trigger_count <> 1 OR function_count <> 1 OR lookup_index_count <> 2 THEN RAISE EXCEPTION 'replay lineage trigger or function is missing'; END IF; END @@ -28,6 +41,7 @@ $migration_object_check$; BEGIN; +-- Pending controls for owner A and owner B. INSERT INTO etl_job_records ( job_record_id, principal_scope_hash, @@ -53,6 +67,7 @@ INSERT INTO etl_job_records ( 'PENDING' ); +-- Two independent same-owner terminal roots plus one terminal root for owner B. INSERT INTO etl_job_records ( job_record_id, principal_scope_hash, @@ -61,15 +76,25 @@ INSERT INTO etl_job_records ( request_payload, job_status, failure_code -) VALUES ( - '00000000-0000-4000-8000-000000000002', - repeat('a', 64), - repeat('2', 64), - repeat('b', 64), - NULL, - 'FAILED', - 'etl_replay_source_failed' -); +) VALUES + ( + '00000000-0000-4000-8000-000000000002', + repeat('a', 64), + repeat('2', 64), + repeat('b', 64), + NULL, + 'FAILED', + 'etl_replay_source_failed' + ), + ( + '00000000-0000-4000-8000-000000000014', + repeat('a', 64), + repeat('e', 64), + repeat('b', 64), + NULL, + 'FAILED', + 'etl_alternate_root_failed' + ); INSERT INTO etl_job_records ( job_record_id, @@ -93,7 +118,7 @@ INSERT INTO etl_job_records ( CURRENT_TIMESTAMP ); --- Create a valid first-generation replay, then terminalize it through the ordinary lifecycle. +-- Create a valid first-generation replay with the exact source digest, then terminalize it. INSERT INTO etl_job_records ( job_record_id, principal_scope_hash, @@ -108,7 +133,7 @@ INSERT INTO etl_job_records ( '00000000-0000-4000-8000-000000000005', repeat('a', 64), repeat('5', 64), - repeat('f', 64), + repeat('b', 64), '{}', 'PENDING', '00000000-0000-4000-8000-000000000002', @@ -122,7 +147,7 @@ UPDATE etl_job_records failure_code = 'etl_replay_generation_failed' WHERE job_record_id = '00000000-0000-4000-8000-000000000005'; --- A valid second generation must retain the first root and advance exactly once. +-- A valid second generation retains the first root, source digest, and one-step succession. INSERT INTO etl_job_records ( job_record_id, principal_scope_hash, @@ -137,7 +162,7 @@ INSERT INTO etl_job_records ( '00000000-0000-4000-8000-000000000008', repeat('a', 64), repeat('8', 64), - repeat('8', 64), + repeat('b', 64), '{}', 'PENDING', '00000000-0000-4000-8000-000000000005', @@ -145,305 +170,163 @@ INSERT INTO etl_job_records ( 2 ); -DO $nonterminal_source_check$ +DO $lineage_rejection_checks$ BEGIN BEGIN INSERT INTO etl_job_records ( - job_record_id, - principal_scope_hash, - submission_key_hash, - request_digest, - request_payload, - job_status, - replay_source_job_record_id, - replay_root_job_record_id, + job_record_id, principal_scope_hash, submission_key_hash, + request_digest, request_payload, job_status, + replay_source_job_record_id, replay_root_job_record_id, replay_generation_count ) VALUES ( '00000000-0000-4000-8000-000000000009', - repeat('a', 64), - repeat('9', 64), - repeat('9', 64), - '{}', - 'PENDING', + repeat('a', 64), repeat('9', 64), repeat('a', 64), '{}', 'PENDING', '00000000-0000-4000-8000-000000000001', - '00000000-0000-4000-8000-000000000001', - 1 + '00000000-0000-4000-8000-000000000001', 1 ); + RAISE EXCEPTION 'nonterminal replay source was accepted'; EXCEPTION - WHEN check_violation OR foreign_key_violation THEN - NULL; + WHEN check_violation OR foreign_key_violation THEN NULL; END; - IF EXISTS ( - SELECT 1 - FROM etl_job_records - WHERE job_record_id = '00000000-0000-4000-8000-000000000009' - ) THEN - RAISE EXCEPTION 'nonterminal replay source was accepted'; - END IF; -END -$nonterminal_source_check$; - -DO $generation_one_root_check$ -BEGIN BEGIN INSERT INTO etl_job_records ( - job_record_id, - principal_scope_hash, - submission_key_hash, - request_digest, - request_payload, - job_status, - replay_source_job_record_id, - replay_root_job_record_id, + job_record_id, principal_scope_hash, submission_key_hash, + request_digest, request_payload, job_status, + replay_source_job_record_id, replay_root_job_record_id, replay_generation_count ) VALUES ( '00000000-0000-4000-8000-000000000010', - repeat('a', 64), - repeat('a', 64), - repeat('a', 64), - '{}', - 'PENDING', + repeat('a', 64), repeat('a', 64), repeat('b', 64), '{}', 'PENDING', '00000000-0000-4000-8000-000000000002', - '00000000-0000-4000-8000-000000000001', - 1 + '00000000-0000-4000-8000-000000000014', 1 ); + RAISE EXCEPTION 'generation-one replay accepted a different root'; EXCEPTION - WHEN check_violation OR foreign_key_violation THEN - NULL; + WHEN check_violation OR foreign_key_violation THEN NULL; END; - IF EXISTS ( - SELECT 1 - FROM etl_job_records - WHERE job_record_id = '00000000-0000-4000-8000-000000000010' - ) THEN - RAISE EXCEPTION 'generation-one replay accepted a different root'; - END IF; -END -$generation_one_root_check$; - -DO $derived_root_check$ -BEGIN BEGIN INSERT INTO etl_job_records ( - job_record_id, - principal_scope_hash, - submission_key_hash, - request_digest, - request_payload, - job_status, - replay_source_job_record_id, - replay_root_job_record_id, + job_record_id, principal_scope_hash, submission_key_hash, + request_digest, request_payload, job_status, + replay_source_job_record_id, replay_root_job_record_id, replay_generation_count ) VALUES ( '00000000-0000-4000-8000-000000000011', - repeat('a', 64), - repeat('b', 64), - repeat('b', 64), - '{}', - 'PENDING', - '00000000-0000-4000-8000-000000000005', + repeat('a', 64), repeat('b', 64), repeat('b', 64), '{}', 'PENDING', '00000000-0000-4000-8000-000000000005', - 2 + '00000000-0000-4000-8000-000000000005', 2 ); + RAISE EXCEPTION 'a derived replay row was accepted as lineage root'; EXCEPTION - WHEN check_violation OR foreign_key_violation THEN - NULL; + WHEN check_violation OR foreign_key_violation THEN NULL; END; - IF EXISTS ( - SELECT 1 - FROM etl_job_records - WHERE job_record_id = '00000000-0000-4000-8000-000000000011' - ) THEN - RAISE EXCEPTION 'a derived replay row was accepted as lineage root'; - END IF; -END -$derived_root_check$; - -DO $skipped_generation_check$ -BEGIN BEGIN INSERT INTO etl_job_records ( - job_record_id, - principal_scope_hash, - submission_key_hash, - request_digest, - request_payload, - job_status, - replay_source_job_record_id, - replay_root_job_record_id, + job_record_id, principal_scope_hash, submission_key_hash, + request_digest, request_payload, job_status, + replay_source_job_record_id, replay_root_job_record_id, replay_generation_count ) VALUES ( '00000000-0000-4000-8000-000000000012', - repeat('a', 64), - repeat('c', 64), - repeat('c', 64), - '{}', - 'PENDING', + repeat('a', 64), repeat('c', 64), repeat('b', 64), '{}', 'PENDING', '00000000-0000-4000-8000-000000000005', - '00000000-0000-4000-8000-000000000002', - 3 + '00000000-0000-4000-8000-000000000002', 3 ); + RAISE EXCEPTION 'a skipped replay generation was accepted'; EXCEPTION - WHEN check_violation OR foreign_key_violation THEN - NULL; + WHEN check_violation OR foreign_key_violation THEN NULL; END; - IF EXISTS ( - SELECT 1 - FROM etl_job_records - WHERE job_record_id = '00000000-0000-4000-8000-000000000012' - ) THEN - RAISE EXCEPTION 'a skipped replay generation was accepted'; - END IF; -END -$skipped_generation_check$; - -DO $cross_owner_source_check$ -BEGIN BEGIN INSERT INTO etl_job_records ( - job_record_id, - principal_scope_hash, - submission_key_hash, - request_digest, - request_payload, - job_status, - replay_source_job_record_id, - replay_root_job_record_id, + job_record_id, principal_scope_hash, submission_key_hash, + request_digest, request_payload, job_status, + replay_source_job_record_id, replay_root_job_record_id, replay_generation_count ) VALUES ( '00000000-0000-4000-8000-000000000006', - repeat('b', 64), - repeat('6', 64), - repeat('6', 64), - '{}', - 'PENDING', + repeat('b', 64), repeat('6', 64), repeat('d', 64), '{}', 'PENDING', '00000000-0000-4000-8000-000000000002', - '00000000-0000-4000-8000-000000000004', - 1 + '00000000-0000-4000-8000-000000000004', 1 ); + RAISE EXCEPTION 'cross-owner source lineage was accepted'; EXCEPTION - WHEN foreign_key_violation OR check_violation THEN - NULL; + WHEN check_violation OR foreign_key_violation THEN NULL; END; - IF EXISTS ( - SELECT 1 - FROM etl_job_records - WHERE job_record_id = '00000000-0000-4000-8000-000000000006' - ) THEN - RAISE EXCEPTION 'cross-owner source lineage was accepted'; - END IF; + BEGIN + INSERT INTO etl_job_records ( + job_record_id, principal_scope_hash, submission_key_hash, + request_digest, request_payload, job_status, + replay_source_job_record_id, replay_root_job_record_id, + replay_generation_count + ) VALUES ( + '00000000-0000-4000-8000-000000000007', + repeat('b', 64), repeat('7', 64), repeat('d', 64), '{}', 'PENDING', + '00000000-0000-4000-8000-000000000004', + '00000000-0000-4000-8000-000000000002', 1 + ); + RAISE EXCEPTION 'cross-owner root lineage was accepted'; + EXCEPTION + WHEN check_violation OR foreign_key_violation THEN NULL; + END; END -$cross_owner_source_check$; +$lineage_rejection_checks$; -DO $cross_owner_root_check$ +DO $digest_continuity_check$ BEGIN BEGIN INSERT INTO etl_job_records ( - job_record_id, - principal_scope_hash, - submission_key_hash, - request_digest, - request_payload, - job_status, - replay_source_job_record_id, - replay_root_job_record_id, + job_record_id, principal_scope_hash, submission_key_hash, + request_digest, request_payload, job_status, + replay_source_job_record_id, replay_root_job_record_id, replay_generation_count ) VALUES ( - '00000000-0000-4000-8000-000000000007', - repeat('b', 64), - repeat('7', 64), - repeat('7', 64), - '{}', - 'PENDING', - '00000000-0000-4000-8000-000000000004', + '00000000-0000-4000-8000-000000000013', + repeat('a', 64), repeat('d', 64), repeat('c', 64), '{}', 'PENDING', '00000000-0000-4000-8000-000000000002', - 1 + '00000000-0000-4000-8000-000000000002', 1 ); + RAISE EXCEPTION 'replay digest mismatch was accepted'; EXCEPTION - WHEN foreign_key_violation OR check_violation THEN - NULL; + WHEN check_violation OR foreign_key_violation THEN NULL; END; - - IF EXISTS ( - SELECT 1 - FROM etl_job_records - WHERE job_record_id = '00000000-0000-4000-8000-000000000007' - ) THEN - RAISE EXCEPTION 'cross-owner root lineage was accepted'; - END IF; END -$cross_owner_root_check$; +$digest_continuity_check$; -DO $lineage_immutability_check$ +DO $immutability_checks$ BEGIN BEGIN UPDATE etl_job_records - SET replay_root_job_record_id = '00000000-0000-4000-8000-000000000001' + SET replay_root_job_record_id = '00000000-0000-4000-8000-000000000014' WHERE job_record_id = '00000000-0000-4000-8000-000000000005'; + RAISE EXCEPTION 'replay lineage fields were mutable'; EXCEPTION - WHEN check_violation THEN - NULL; + WHEN check_violation THEN NULL; END; - IF EXISTS ( - SELECT 1 - FROM etl_job_records - WHERE job_record_id = '00000000-0000-4000-8000-000000000005' - AND replay_root_job_record_id = '00000000-0000-4000-8000-000000000001' - ) THEN - RAISE EXCEPTION 'replay lineage fields were mutable'; - END IF; -END -$lineage_immutability_check$; - -DO $root_evidence_immutability_check$ -BEGIN BEGIN UPDATE etl_job_records SET request_digest = repeat('0', 64) WHERE job_record_id = '00000000-0000-4000-8000-000000000002'; + RAISE EXCEPTION 'referenced replay root evidence was mutable'; EXCEPTION - WHEN check_violation THEN - NULL; + WHEN check_violation THEN NULL; END; - IF EXISTS ( - SELECT 1 - FROM etl_job_records - WHERE job_record_id = '00000000-0000-4000-8000-000000000002' - AND request_digest <> repeat('b', 64) - ) THEN - RAISE EXCEPTION 'referenced replay root evidence was mutable'; - END IF; -END -$root_evidence_immutability_check$; - -DO $source_evidence_immutability_check$ -BEGIN BEGIN UPDATE etl_job_records SET failure_code = 'etl_replay_generation_changed' WHERE job_record_id = '00000000-0000-4000-8000-000000000005'; + RAISE EXCEPTION 'referenced immediate-source evidence was mutable'; EXCEPTION - WHEN check_violation THEN - NULL; + WHEN check_violation THEN NULL; END; - - IF EXISTS ( - SELECT 1 - FROM etl_job_records - WHERE job_record_id = '00000000-0000-4000-8000-000000000005' - AND failure_code <> 'etl_replay_generation_failed' - ) THEN - RAISE EXCEPTION 'referenced immediate-source evidence was mutable'; - END IF; END -$source_evidence_immutability_check$; +$immutability_checks$; DO $delete_restrict_check$ BEGIN @@ -452,8 +335,7 @@ BEGIN WHERE job_record_id = '00000000-0000-4000-8000-000000000005'; RAISE EXCEPTION 'ON DELETE RESTRICT did not protect immediate replay history'; EXCEPTION - WHEN foreign_key_violation THEN - NULL; + WHEN foreign_key_violation THEN NULL; END; BEGIN @@ -461,16 +343,18 @@ BEGIN WHERE job_record_id = '00000000-0000-4000-8000-000000000002'; RAISE EXCEPTION 'ON DELETE RESTRICT did not protect replay root history'; EXCEPTION - WHEN foreign_key_violation THEN - NULL; + WHEN foreign_key_violation THEN NULL; END; END $delete_restrict_check$; ROLLBACK; +-- Rehearse ordered rollback without changing the migrated database. BEGIN; +DROP INDEX etl_job_replay_source_lookup_index; +DROP INDEX etl_job_replay_root_lookup_index; DROP TRIGGER etl_job_replay_lineage_guard_trigger ON etl_job_records; DROP FUNCTION validate_etl_job_replay_lineage(); ALTER TABLE etl_job_records DROP CONSTRAINT etl_job_replay_source_reference; @@ -489,6 +373,7 @@ DECLARE restored_constraint_count integer; restored_trigger_count integer; restored_function_count integer; + restored_index_count integer; BEGIN SELECT count(*) INTO restored_column_count @@ -528,10 +413,23 @@ BEGIN FROM pg_proc AS function_record WHERE function_record.proname = 'validate_etl_job_replay_lineage'; + SELECT count(*) + INTO restored_index_count + FROM pg_class AS index_class + JOIN pg_index AS index_record + ON index_record.indexrelid = index_class.oid + WHERE index_class.relname IN ( + 'etl_job_replay_source_lookup_index', + 'etl_job_replay_root_lookup_index' + ) + AND index_record.indisready + AND index_record.indisvalid; + IF restored_column_count <> 3 OR restored_constraint_count <> 4 OR restored_trigger_count <> 1 - OR restored_function_count <> 1 THEN + OR restored_function_count <> 1 + OR restored_index_count <> 2 THEN RAISE EXCEPTION 'rollback rehearsal did not restore V7'; END IF; END From 41a778cf680e84ad083b8c92f9c206979c775e2a Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:24:21 +0900 Subject: [PATCH 088/103] test(ci): require replay coverage diagnostics --- .../etl/documentation/CiCoverageDiagnosticsWorkflowTest.java | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/documentation/CiCoverageDiagnosticsWorkflowTest.java b/etl-service/src/test/java/com/xtrmetl/etl/documentation/CiCoverageDiagnosticsWorkflowTest.java index 5de68f14..5963f963 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/documentation/CiCoverageDiagnosticsWorkflowTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/documentation/CiCoverageDiagnosticsWorkflowTest.java @@ -52,6 +52,10 @@ void diagnosesEveryStrictCoverageTarget() { "\"com/xtrmetl/etl/controller/EtlJobController\": " + "\"EtlJobController.java\"" )); + assertTrue(workflow.contains( + "\"com/xtrmetl/etl/controller/EtlJobReplayController\": " + + "\"EtlJobReplayController.java\"" + )); assertTrue(workflow.contains( "\"com/xtrmetl/etl/service/Sha256Digest\": \"Sha256Digest.java\"" )); From 33ba0108b5750b6d0fbc3755e4b63c507f9d269d Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:24:41 +0900 Subject: [PATCH 089/103] docs(etl): align replay design with database evidence --- .../2026-08-06-durable-job-replay-design.md | 39 +++++++++++++++---- 1 file changed, 31 insertions(+), 8 deletions(-) diff --git a/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md b/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md index c8d0424c..a372463a 100644 --- a/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md +++ b/docs/superpowers/specs/2026-08-06-durable-job-replay-design.md @@ -50,24 +50,32 @@ FOREIGN KEY (replay_root_job_record_id, principal_scope_hash) REFERENCES etl_job_records (job_record_id, principal_scope_hash) ``` -Those declarative constraints establish owner scope, existence, self-reference rejection, completeness, deletion restriction, and the generation bound. They do not by themselves prove that a source is terminal, that the selected root is the first lineage row, or that each replay hop advances exactly one generation. The migration therefore adds a database trigger and PL/pgSQL trigger function as the relational authority for continuity: +Those declarative constraints establish owner scope, existence, self-reference rejection, completeness, deletion restriction, and the generation bound. They do not by themselves prove that a source is terminal, that the selected root is the first lineage row, that the new row preserves the immediate source digest, or that each replay hop advances exactly one generation. The migration therefore adds a database trigger and PL/pgSQL trigger function as the relational authority for continuity: ```text validate_etl_job_replay_lineage() etl_job_replay_lineage_guard_trigger ``` -On replay insertion, the trigger requires all of the following: +On replay insertion, the database trigger requires all of the following: - the new derived row starts as `PENDING`, attempt zero, with a retained payload; - the immediate source exists in the same principal scope and is `FAILED` or `CANCELLED`; +- the new row's `request_digest` exactly equals the immediate source `request_digest`; - the root exists in the same principal scope, is terminal, and has all lineage fields null; - generation 1 uses the same row for immediate source and root; - later generations inherit the exact first root and equal the source generation plus one. On updates that name any lineage column, the trigger rejects every changed value. Lineage fields are immutable after insertion, so a maintenance script, import path, or future service cannot silently reparent a job after descendants exist. -A terminal row becomes durable evidence when a descendant names it as an immediate source or lineage root. The same trigger serializes child insertion and parent mutation with PostgreSQL row locks. It permits ordinary lifecycle updates before the first descendant exists, but after a reference exists it rejects changes to status, request digest or payload, attempt and failure state, cancellation evidence, and lifecycle timestamps. Referenced replay evidence is immutable, so an already-created descendant cannot silently acquire a different historical meaning. +A terminal row becomes durable evidence when a descendant names it as an immediate source or lineage root. Child insertion locks the immediate source and root with PostgreSQL `FOR UPDATE` before it can validate and commit. A parent update already owns the parent row lock before its trigger executes; when the update would alter terminal replay evidence, the trigger performs an indexed descendant existence lookup without taking child row locks and then returns before the INSERT-only source/root locking path. This establishes one lock direction and avoids child-to-ancestor lock inversion: + +```text +child insertion → lock source/root → validate → commit or reject +parent mutation → parent row already locked → read descendant existence → commit or reject +``` + +If a parent update commits first, a later child validates the resulting evidence. If child insertion obtains the parent lock first, a later conflicting parent update waits, observes the committed descendant, and fails closed. Ordinary lifecycle updates remain possible before the first descendant exists; referenced replay evidence is immutable after that point, so an already-created descendant cannot silently acquire a different historical meaning. For the first replay: @@ -85,7 +93,19 @@ root = inherited first job generation = source generation + 1 ``` -The application independently validates the owner-scoped source, inherited root, and root-row identity before insertion. PostgreSQL independently rejects cross-owner, nonterminal, discontinuous, derived-root, mutable-lineage, and referenced-evidence mutation. Relational rows remain authoritative even when lineage is later exported as W3C PROV. +The application independently validates the owner-scoped source, source digest, inherited root, and root-row identity before insertion. PostgreSQL independently rejects cross-owner, nonterminal, digest-divergent, discontinuous, derived-root, mutable-lineage, and referenced-evidence mutation. Relational rows remain authoritative even when lineage is later exported as W3C PROV. + +## Replay lookup indexes + +The trigger's immutable-evidence check and PostgreSQL's self-referencing foreign keys need bounded source/root lookup paths as durable history grows. The design therefore separates schema authority from online index construction: + +- `V7__add_etl_job_replay_lineage.sql` remains transactional and contains columns, constraints, trigger, and function only; +- `V8__add_etl_job_replay_source_lookup_index.sql` owns one partial `CREATE INDEX CONCURRENTLY` for `(replay_source_job_record_id, principal_scope_hash)`; +- `V9__add_etl_job_replay_root_lookup_index.sql` owns one partial `CREATE INDEX CONCURRENTLY` for `(replay_root_job_record_id, principal_scope_hash)`; +- each concurrent migration has its own `.sql.conf` with `executeInTransaction=false`; +- migration verification requires both indexes to be present, ready, and valid. + +One concurrent index per nontransactional migration gives each failure one auditable Flyway repair boundary. A cancelled build can leave an invalid index, so rollout must inspect PostgreSQL catalog state, remove only the failed artifact with `DROP INDEX CONCURRENTLY`, repair the exact migration record under the approved deployment procedure, and rerun without editing an applied migration. ## Replay-key authority @@ -118,7 +138,7 @@ One transaction performs the following sequence: 8. derive root and bounded generation; 9. require the inherited root to exist in the owner namespace and have null lineage fields; 10. insert one new `PENDING` row with the verified payload and lineage; -11. let PostgreSQL validate same-owner references, terminal source, first root, exact generation continuity, initial lifecycle, and the transition from mutable lifecycle state to referenced immutable evidence; +11. let PostgreSQL validate same-owner references, immediate-source digest equality, terminal source, first root, exact generation continuity, initial lifecycle, and the transition from mutable lifecycle state to referenced immutable evidence; 12. return only the new operator-safe job identity. The source is never updated. Read-then-write state resurrection is prohibited. @@ -173,9 +193,10 @@ The exact-head suite must prove: 10. concurrent creation produces one row and an in-progress or later replay outcome; 11. the new job can be claimed and follows normal lifecycle contracts; 12. service defense in depth rejects an inherited root that is itself a replay row; -13. PostgreSQL 18 rejects cross-owner references, nonterminal sources, generation-one root divergence, derived roots, skipped generations, lineage mutation, and mutation of referenced root or immediate-source evidence; -14. migration completeness, source and root constraints, trigger/function presence, self-reference prohibition, naming, deletion restriction, rollout, and rollback are documented and tested; -15. all added production statements and branches retain zero-missed configured coverage and no project test is skipped. +13. PostgreSQL 18 rejects cross-owner references, nonterminal sources, immediate-source digest mismatch, generation-one root divergence, derived roots, skipped generations, lineage mutation, and mutation of referenced root or immediate-source evidence; +14. PostgreSQL 18 applies V8 and V9 independently and requires both replay lookup indexes to be ready and valid; +15. migration completeness, source and root constraints, trigger/function presence, self-reference prohibition, descriptive naming, deletion restriction, concurrent-index recovery, rollout, and rollback are documented and tested; +16. all added production statements and branches retain zero-missed configured coverage and no project test is skipped. ## Operational limitation @@ -189,6 +210,8 @@ Nottingham, M., Wilde, E., & Dalal, S. (2023). *Problem details for HTTP APIs* ( PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: Constraints*. https://www.postgresql.org/docs/18/ddl-constraints.html +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: CREATE INDEX*. https://www.postgresql.org/docs/18/sql-createindex.html + PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: CREATE TRIGGER*. https://www.postgresql.org/docs/18/sql-createtrigger.html PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: INSERT*. https://www.postgresql.org/docs/18/sql-insert.html From bfd705eaefd228d7f274e4d4593da4508fd7e821 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:26:26 +0900 Subject: [PATCH 090/103] fix(ci): diagnose replay controller coverage --- .github/workflows/ci.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1111f30f..8b5a6866 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -62,6 +62,7 @@ jobs: "com/xtrmetl/etl/job/EtlJobService": "EtlJobService.java", "com/xtrmetl/etl/job/EtlJobReplayService": "EtlJobReplayService.java", "com/xtrmetl/etl/controller/EtlJobController": "EtlJobController.java", + "com/xtrmetl/etl/controller/EtlJobReplayController": "EtlJobReplayController.java", "com/xtrmetl/etl/service/Sha256Digest": "Sha256Digest.java", } reports = sorted(Path(".").glob("**/target/site/jacoco/jacoco.xml")) From 9b0a125ce81c28dc5ba80c07f13f1a4ada3cdf90 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:27:45 +0900 Subject: [PATCH 091/103] docs(adr): align replay authority and index rollout --- ...2026-08-07-immutable-durable-job-replay.md | 62 ++++++++++++++----- 1 file changed, 48 insertions(+), 14 deletions(-) diff --git a/docs/adr/2026-08-07-immutable-durable-job-replay.md b/docs/adr/2026-08-07-immutable-durable-job-replay.md index b13f4c7a..225c2ab6 100644 --- a/docs/adr/2026-08-07-immutable-durable-job-replay.md +++ b/docs/adr/2026-08-07-immutable-durable-job-replay.md @@ -3,7 +3,7 @@ - **Status:** Proposed while the replay pull request is stacked; Accepted only after direct-`develop` gates and merge - **Date:** 2026-08-07 - **Decision owners:** mightyETL maintainers -- **Scope:** `etl-service` durable-job admission, persistence, operator API, and provenance +- **Scope:** `etl-service` durable-job admission, persistence, operator API, provenance, and PostgreSQL rollout ## Context @@ -11,9 +11,13 @@ Durable jobs deliberately clear `request_payload` after success, failure, or can Rewinding a terminal row to `PENDING` would erase the original terminal fact, mix attempt histories, invalidate conditional status validators, and make concurrent cancellation or success reasoning substantially harder. Retaining terminal payloads solely for replay would expand sensitive-data retention. Treating semantically equivalent JSON as the same work would also allow hidden payload changes under a replay label. -A structural schema with nullable source, root, and generation fields is not sufficient on its own. Composite foreign keys can prove same-owner existence, and check constraints can prove field completeness and bounds, but they cannot prove that a source is terminal, a root is the first row in the chain, a generation increments exactly once, or an existing replay has never been reparented. Those properties must remain true even for maintenance scripts, data imports, and future writers that do not execute the Java service. +A structural schema with nullable source, root, and generation fields is not sufficient on its own. Composite foreign keys can prove same-owner existence, and check constraints can prove field completeness and bounds, but they cannot prove that a source is terminal, the replay row preserves the exact source request digest, a root is the first row in the chain, a generation increments exactly once, or an existing replay has never been reparented. Those properties must remain true even for maintenance scripts, data imports, and future writers that do not execute the Java service. -Cross-row validity also creates a temporal requirement. Once a descendant commits a reference to terminal source or root evidence, a later direct writer must not change the referenced status, digest, payload state, attempt/failure state, cancellation evidence, or lifecycle timestamps. Otherwise the descendant's historical meaning can change after admission even though its lineage identifiers remain untouched. Child insertion and parent mutation therefore require one database-owned serialization boundary. +Cross-row validity also creates a temporal requirement. Once a descendant commits a reference to terminal source or root evidence, a later direct writer must not change the referenced status, digest, payload state, attempt/failure state, cancellation evidence, or lifecycle timestamps. Otherwise the descendant's historical meaning can change after admission even though its lineage identifiers remain untouched. + +The trigger must preserve that temporal boundary without introducing a deadlock. Child insertion needs source/root row locks to bind exact evidence. A parent update already owns the parent row lock before a `BEFORE UPDATE` trigger runs, so locking child rows and then returning to ancestors would invert lock order. The parent mutation path therefore needs an indexed descendant existence lookup without child row locks. + +The source/root foreign keys and descendant lookup also need bounded access paths as history grows. Building those indexes inside the transactional lineage migration would block production writers, while grouping multiple `CREATE INDEX CONCURRENTLY` statements into one nontransactional Flyway migration would make partial failure recovery ambiguous. ## Decision @@ -29,20 +33,36 @@ A derived row stores: Source and root references are composite owner-scoped foreign keys to `(job_record_id, principal_scope_hash)` and use `ON DELETE RESTRICT`. A root row has all three lineage fields null; a replay row has all three non-null. Generation cannot exceed the supported bound. -V7 also creates the PL/pgSQL function `validate_etl_job_replay_lineage()` and the row-level `etl_job_replay_lineage_guard_trigger`. The trigger is the database authority for exact source/root/generation continuity and referenced evidence immutability: +V7 creates the PL/pgSQL function `validate_etl_job_replay_lineage()` and the row-level `etl_job_replay_lineage_guard_trigger`. The trigger is the database authority for exact request and lineage continuity: - replay rows are inserted only as `PENDING`, attempt zero, with a retained payload; - immediate source and root belong to the same principal namespace as the new row; - the immediate source is `FAILED` or `CANCELLED`; +- the replay row `request_digest` equals the immediate source `request_digest`; - the root is terminal and has no lineage fields; - generation one uses the same source and root; - later generations retain that root and equal the immediate source generation plus one; - lineage columns are immutable after insertion; and - after any descendant references a row as immediate source or root, the terminal status, request evidence, attempt/failure state, cancellation evidence, and lifecycle timestamps are immutable. -The trigger uses PostgreSQL `FOR UPDATE` row locks for both source/root validation during child insertion and descendant lookup during a parent-evidence update. If a parent update commits first, a later child validates and binds the resulting evidence. If child insertion locks and references the parent first, a later conflicting parent update waits, observes the committed descendant, and fails closed. Ordinary lifecycle updates remain available until the row becomes referenced historical evidence. +The trigger uses one directional lock protocol: + +- an INSERT locks source and root rows with `FOR UPDATE`, validates exact evidence, and then commits or rejects; +- an UPDATE already owns its parent row lock, performs an indexed descendant existence read without locking child rows, rejects protected evidence changes when a descendant exists, and returns before the INSERT-only source/root locking path. + +If a parent update commits first, a later child validates and binds the resulting evidence. If child insertion locks and references the parent first, a later conflicting parent update waits, observes the committed descendant, and fails closed. Ordinary lifecycle updates remain available until the row becomes referenced historical evidence. + +V8 and V9 provide the online lookup boundary: + +- `V8__add_etl_job_replay_source_lookup_index.sql` creates the partial `etl_job_replay_source_lookup_index` concurrently; +- `V9__add_etl_job_replay_root_lookup_index.sql` creates the partial `etl_job_replay_root_lookup_index` concurrently; +- each migration owns exactly one `CREATE INDEX CONCURRENTLY` statement; +- each companion `.sql.conf` sets `executeInTransaction=false`; +- migration verification requires both indexes to be ready and valid. -The Java service independently verifies the inherited root is an actual first root before insertion. This is defense in depth; it does not replace the trigger. The new row enters the ordinary `PENDING` lifecycle and uses the existing worker, PostgreSQL claim, exact lease, success, failure, polling, cancellation, and ETag contracts. +A failed concurrent build is repaired by inspecting catalog state, removing only the invalid artifact with `DROP INDEX CONCURRENTLY`, repairing the exact Flyway migration record through the approved deployment process, and rerunning without editing an applied migration. + +The Java service independently verifies source digest equality and the inherited root before insertion. This is defense in depth; it does not replace the trigger. The new row enters the ordinary `PENDING` lifecycle and uses the existing worker, PostgreSQL claim, exact lease, success, failure, polling, cancellation, and ETag contracts. ```mermaid sequenceDiagram @@ -64,7 +84,8 @@ sequenceDiagram A-->>O: 202 + Location + Idempotency-Replayed W->>P: ordinary lease-fenced claim P->>T: later mutation of referenced terminal evidence - T->>T: lock descendant rows and reject mutation + T->>T: indexed child-existence read without child lock + T-->>P: reject protected evidence mutation ``` ## Replay identity @@ -115,7 +136,15 @@ Rejected because maintenance scripts, migrations, import processes, or future se ### Use only foreign keys and check constraints -Rejected because those constraints cannot express exact generation succession, immutable cross-row root identity, or the transition from mutable terminal state to referenced immutable evidence. A row-level trigger is required for those cross-row and temporal invariants. +Rejected because those constraints cannot express exact digest equality, generation succession, immutable cross-row root identity, or the transition from mutable terminal state to referenced immutable evidence. A row-level trigger is required for those cross-row and temporal invariants. + +### Lock descendants during parent mutation + +Rejected because child insertion already locks ancestors. A parent trigger that locks child rows and then reaches ancestors creates child-to-ancestor lock inversion and can deadlock replay admission against lifecycle maintenance. + +### Build both indexes in one nontransactional migration + +Rejected because a second-index failure could leave a valid first index and a failed Flyway version with no one-artifact repair boundary. Separate V8/V9 migrations preserve auditable failure and rollback semantics. ### Leave referenced evidence mutable @@ -131,8 +160,10 @@ Rejected because a descendant would preserve the same source/root identifiers wh - ordinary worker and cancellation machinery is reused; - payload retention does not increase; - concurrent retries have one database-owned outcome; -- database maintenance and import paths cannot create discontinuous or reparented lineage; -- cross-owner, nonterminal, derived-root, skipped-generation, lineage-mutation, and referenced-evidence-mutation attempts fail closed. +- database maintenance and import paths cannot create digest-divergent, discontinuous, or reparented lineage; +- cross-owner, nonterminal, derived-root, skipped-generation, lineage-mutation, and referenced-evidence-mutation attempts fail closed; +- source/root lookups remain indexed as durable history grows; +- concurrent index failure has one migration and one artifact to repair. ### Costs @@ -140,15 +171,16 @@ Rejected because a descendant would preserve the same source/root identifiers wh - self-referencing lineage constrains retention and deletion order; - every connector needs an explicit replay-safety classification; - migrations and generation bounds require real PostgreSQL verification; -- the trigger adds same-transaction row locks and lookups to replay insertion and referenced-evidence updates; +- the trigger adds same-transaction ancestor locks to replay insertion and indexed descendant reads to referenced-evidence updates; - a terminal row cannot receive later maintenance edits after it becomes lineage evidence without a separately reviewed migration strategy; -- trigger and function lifecycle must be included in downgrade and disaster-recovery rehearsals. +- V8/V9 are nontransactional and require explicit ready/valid inspection and invalid-index recovery; +- trigger, function, and index lifecycle must be included in downgrade and disaster-recovery rehearsals. ## Verification -Acceptance requires exact-head tests for source immutability, owner isolation, exact payload matching, same-key replay, conflicting-key reuse, concurrent admission, lineage inheritance, generation exhaustion, ordinary worker behavior, cancellation compatibility, RFC 9457 responses, privacy exclusions, and zero-missed configured production coverage. +Acceptance requires exact-head tests for source immutability, owner isolation, exact payload matching, same-key replay, conflicting-key reuse, concurrent admission, lineage inheritance, generation exhaustion, ordinary worker behavior, cancellation compatibility, RFC 9457 responses, privacy exclusions, lock-order safety, and zero-missed configured production coverage. -A direct-`develop` GitHub Actions gate applies every versioned migration to PostgreSQL 18, verifies replay and cancellation columns, verifies both lineage foreign keys use `ON DELETE RESTRICT`, verifies trigger and function presence, executes valid generation-one and generation-two inserts, rejects nonterminal sources, different generation-one roots, derived roots, skipped generations, cross-owner references, lineage mutation, referenced root-digest mutation, and referenced immediate-source failure-evidence mutation, protects source/root deletion, rehearses transactional rollback, and creates a non-empty schema-only dump. SAST, security, dependency, SBOM, review-thread, and non-author exact-head approval gates remain mandatory. +A direct-`develop` GitHub Actions gate applies every versioned migration to PostgreSQL 18, verifies replay and cancellation columns, verifies both lineage foreign keys use `ON DELETE RESTRICT`, verifies trigger and function presence, verifies both replay lookup indexes are ready and valid, executes valid generation-one and generation-two inserts, rejects nonterminal sources, request-digest mismatch, different generation-one roots, derived roots, skipped generations, cross-owner references, lineage mutation, referenced root-digest mutation, and referenced immediate-source failure-evidence mutation, protects source/root deletion, rehearses ordered rollback, and creates a non-empty schema-only dump. SAST, security, dependency, SBOM, review-thread, and non-author exact-head approval gates remain mandatory. ## References — APA 7th edition @@ -160,6 +192,8 @@ Nottingham, M., Wilde, E., & Dalal, S. (2023). *Problem details for HTTP APIs* ( PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: Constraints*. https://www.postgresql.org/docs/18/ddl-constraints.html +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: CREATE INDEX*. https://www.postgresql.org/docs/18/sql-createindex.html + PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: CREATE TRIGGER*. https://www.postgresql.org/docs/18/sql-createtrigger.html PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: INSERT*. https://www.postgresql.org/docs/18/sql-insert.html From 025798b656575e8d6f1af05e812bc608d3086eb4 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:29:59 +0900 Subject: [PATCH 092/103] docs(doctoring): bind replay evidence and online indexes --- .../durable-job-replay-standards-evidence.md | 45 ++++++++++++++++--- 1 file changed, 38 insertions(+), 7 deletions(-) diff --git a/docs/doctoring/durable-job-replay-standards-evidence.md b/docs/doctoring/durable-job-replay-standards-evidence.md index 825c59c6..d64c3e1f 100644 --- a/docs/doctoring/durable-job-replay-standards-evidence.md +++ b/docs/doctoring/durable-job-replay-standards-evidence.md @@ -4,7 +4,7 @@ mightyETL models replay as creation of a new durable job derived from an immutable terminal source. The source is never returned to `PENDING`. Only owner-scoped `FAILED` and `CANCELLED` sources are eligible, and the operator must resupply byte-identical bounded JSON whose SHA-256 digest equals the source `request_digest`. -The accepted replay response uses `202 Accepted` with a monitor URI because durable admission does not assert completion. Deterministic failures use RFC 9457 problem details. PostgreSQL owns the replay transaction, uniqueness, owner-scoped foreign keys, exact source/root/generation continuity, lineage-column immutability, and source-row serialization. Replay lineage is compatible with PROV-O derivation semantics, while the relational database remains authoritative. +The accepted replay response uses `202 Accepted` with a monitor URI because durable admission does not assert completion. Deterministic failures use RFC 9457 problem details. PostgreSQL owns the replay transaction, uniqueness, owner-scoped foreign keys, immediate-source digest equality, exact source/root/generation continuity, lineage-column immutability, referenced-evidence immutability, and source-row serialization. Replay lineage is compatible with PROV-O derivation semantics, while the relational database remains authoritative. ## Normative mapping @@ -14,7 +14,8 @@ The accepted replay response uses `202 Accepted` with a monitor URI because dura | Stable machine-readable failures | RFC 9457 | Fixed problem type, title, status, detail, and `error_code` without exception text | | Atomic new-row creation and conflict handling | PostgreSQL 18 `INSERT` and transaction documentation | One transaction validates source ownership, digest, replay identity, and lineage before inserting one new job | | Same-owner source and root existence | PostgreSQL 18 constraints | Composite foreign keys bind source and root to the new row's `principal_scope_hash`, and `ON DELETE RESTRICT` protects retained history | -| Exact lineage transition | PostgreSQL 18 `CREATE TRIGGER` and PL/pgSQL trigger functions | A row-level `BEFORE INSERT OR UPDATE OF` trigger validates terminal source, first root, exact generation successor, initial pending lifecycle, and lineage-column immutability | +| Exact request and lineage transition | PostgreSQL 18 `CREATE TRIGGER` and PL/pgSQL trigger functions | A row-level `BEFORE INSERT OR UPDATE OF` trigger validates terminal source, immediate-source digest equality, first root, exact generation successor, initial pending lifecycle, and lineage-column immutability | +| Online descendant and foreign-key lookup | PostgreSQL 18 `CREATE INDEX` | Separate partial source/root indexes are built with one `CREATE INDEX CONCURRENTLY` per nontransactional Flyway migration and must be ready and valid | | Derivation lineage | W3C PROV-O | New job is derived from the immediate source and preserves an immutable first-root/generation chain | | Domain separation rationale | NIST SP 800-185 | Replay-key hashing uses a versioned replay-specific domain; the SHA-256 construction does not claim cSHAKE or TupleHash conformance | @@ -23,15 +24,41 @@ The accepted replay response uses `202 Accepted` with a monitor URI because dura The composite owner-scoped foreign keys prove that the named source and root exist in the same tenant namespace. The complete-lineage check proves that lineage fields are either all null or all present, bounds generation, and rejects direct self-reference. Those declarative rules cannot express all cross-row temporal invariants: - source must already be `FAILED` or `CANCELLED`; +- the replay row `request_digest` must equal the immediate source `request_digest`; - the root must be the first job, with every lineage field null; - generation one must use the same row as source and root; - every later generation must inherit that root and equal the immediate source generation plus one; -- an existing replay row must never be reparented. +- an existing replay row must never be reparented; and +- terminal evidence must become immutable after any descendant references the row. -PostgreSQL `CREATE TRIGGER` permits a row-level trigger to run before selected insert or update events, and PL/pgSQL trigger functions receive `NEW`, `OLD`, `TG_OP`, and related context. mightyETL uses that database mechanism to reject invalid writes before persistence. The service repeats the inherited-root identity check as defense in depth, but a maintenance script or import path cannot bypass the relational authority merely by omitting Java validation. +PostgreSQL `CREATE TRIGGER` permits a row-level trigger to run before selected insert or update events, and PL/pgSQL trigger functions receive `NEW`, `OLD`, `TG_OP`, and related context. mightyETL uses that database mechanism to reject invalid writes before persistence. The service repeats source-digest and inherited-root checks as defense in depth, but a maintenance script or import path cannot bypass the relational authority merely by omitting Java validation. The trigger raises the fixed SQLSTATE class `23514` without embedding principal values, job identifiers, payloads, hashes, SQL, or exception causes. Application and operator logs must classify the failure with a finite internal integrity code rather than copying raw database text. +## Concurrency and lock-order evidence + +Child insertion needs an exact source/root snapshot. The INSERT trigger therefore locks the immediate source and root with PostgreSQL `FOR UPDATE` before validating status, digest, root identity, and generation continuity. + +A parent UPDATE already owns the parent row lock before the `BEFORE UPDATE` trigger executes. The immutable-evidence path performs an indexed descendant existence lookup without taking child row locks, rejects the mutation when a descendant exists, and returns before the INSERT-only source/root lock path. This establishes one lock direction: + +```text +child insertion → source/root row locks +parent mutation → existing parent row lock + descendant existence read +``` + +A parent update that commits first defines the evidence a later child validates. A child that obtains the source/root lock first causes a later conflicting parent update to wait; after the child commits, the parent sees the descendant and fails closed. The parent never locks a child and then reaches back to an ancestor, avoiding child-to-ancestor lock inversion. + +## Why online indexes are separate migrations + +The self-referencing foreign keys and immutable-evidence lookup need indexes beginning with `replay_source_job_record_id` and `replay_root_job_record_id`. Building them with ordinary `CREATE INDEX` would block production writes. PostgreSQL `CREATE INDEX CONCURRENTLY` preserves table availability but cannot run inside a transaction block and can leave an invalid index after cancellation or failure. + +mightyETL therefore uses two migrations: + +- `V8__add_etl_job_replay_source_lookup_index.sql` owns only `etl_job_replay_source_lookup_index`; +- `V9__add_etl_job_replay_root_lookup_index.sql` owns only `etl_job_replay_root_lookup_index`. + +Each companion `.sql.conf` sets `executeInTransaction=false`. One index per nontransactional migration makes failure, Flyway repair, and rollback independently auditable. PostgreSQL migration verification requires both `pg_index.indisready` and `pg_index.indisvalid`. An interrupted build is not accepted as passing evidence; operators remove only the failed artifact with `DROP INDEX CONCURRENTLY`, repair the exact Flyway migration record through the approved process, and rerun without editing an applied migration. + ## Security and privacy boundary The HTTP response and ordinary telemetry exclude raw principals, replay keys, payloads, request digests, internal hashes, source/root identifiers, lease identifiers, SQL, target identities, and exception messages. Foreign-owned and absent source identifiers remain indistinguishable. `SUCCEEDED` is excluded because repeating a committed target effect is not safe merely because the original request bytes are known. @@ -41,15 +68,17 @@ Connector replay is enabled only when target effects participate in the mightyET ## Verification obligations - real PostgreSQL 18 migration rehearsal for trigger and function presence, composite self-referencing foreign keys, and `ON DELETE RESTRICT`; -- exact-payload acceptance and byte-different rejection; +- both replay lookup indexes present, ready, and valid after separate V8/V9 concurrent builds; +- exact-payload acceptance and immediate-source digest mismatch rejection at both service and database boundaries; - owner-safe missing/foreign behavior; - same-key replay, key reuse conflict, and concurrent admission tests; - source immutability and source/root/generation lineage tests; -- database rejection of nonterminal sources, generation-one root divergence, derived roots, skipped generations, cross-owner references, and lineage mutation; +- database rejection of nonterminal sources, generation-one root divergence, derived roots, skipped generations, cross-owner references, lineage mutation, and referenced-evidence mutation; +- lock-order regression proving the parent UPDATE path does not lock descendants before ancestor validation; - service defense-in-depth rejection when an inherited root is itself a replay row; - generation-bound rejection; - ordinary worker claim, lease fencing, cancellation, polling, and ETag compatibility; -- transactional rollback rehearsal that restores trigger, function, columns, and named constraints; +- transactional V7 rollback rehearsal plus operational V8/V9 concurrent-index rollback and invalid-index recovery evidence; - configured production instruction, line, method, and branch coverage with zero misses; - direct-base CI, dependency, SBOM, SAST, security, review-thread, and independent-approval gates before merge. @@ -63,6 +92,8 @@ Nottingham, M., Wilde, E., & Dalal, S. (2023). *Problem details for HTTP APIs* ( PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: Constraints*. https://www.postgresql.org/docs/18/ddl-constraints.html +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: CREATE INDEX*. https://www.postgresql.org/docs/18/sql-createindex.html + PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: CREATE TRIGGER*. https://www.postgresql.org/docs/18/sql-createtrigger.html PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: INSERT*. https://www.postgresql.org/docs/18/sql-insert.html From e249a8d7592aa25de795ba2b46e1953bce40ba0e Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:33:09 +0900 Subject: [PATCH 093/103] test(etl): require terminal-only replay descendant scans --- .../etl/job/EtlJobReplayMigrationTest.java | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java index b6f724ca..275cc44c 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java @@ -112,6 +112,22 @@ void updateGuardAvoidsChildToAncestorLockInversion() throws IOException { ); } + @Test + void referencedEvidenceLookupRunsOnlyForReplayEligibleTerminalRows() throws IOException { + String migration = normalizedMigration(); + Pattern terminalEvidenceGuard = Pattern.compile( + "IF OLD\\.job_status IN \\('FAILED', 'CANCELLED'\\) AND \\( " + + "OLD\\.job_status IS DISTINCT FROM NEW\\.job_status .*" + + "OLD\\.updated_at IS DISTINCT FROM NEW\\.updated_at \\) THEN " + + "PERFORM 1 FROM etl_job_records AS child_record" + ); + + assertTrue( + terminalEvidenceGuard.matcher(migration).find(), + "Only replay-eligible terminal rows should pay the descendant lookup cost" + ); + } + private static String normalizedMigration() throws IOException { return Files.readString( projectRoot().resolve( From d41f70490bcd4031cba3f5275ac7f8a06e9e940a Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:34:26 +0900 Subject: [PATCH 094/103] fix(etl): bound replay descendant scans --- .../V7__add_etl_job_replay_lineage.sql | 22 ++++++++++--------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql b/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql index 609a645c..db21a654 100644 --- a/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql +++ b/etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql @@ -61,16 +61,18 @@ BEGIN USING ERRCODE = '23514'; END IF; - IF OLD.job_status IS DISTINCT FROM NEW.job_status - OR OLD.request_digest IS DISTINCT FROM NEW.request_digest - OR OLD.request_payload IS DISTINCT FROM NEW.request_payload - OR OLD.attempt_count IS DISTINCT FROM NEW.attempt_count - OR OLD.failure_code IS DISTINCT FROM NEW.failure_code - OR OLD.cancellation_key_hash IS DISTINCT FROM NEW.cancellation_key_hash - OR OLD.cancellation_code IS DISTINCT FROM NEW.cancellation_code - OR OLD.job_cancelled_at IS DISTINCT FROM NEW.job_cancelled_at - OR OLD.created_at IS DISTINCT FROM NEW.created_at - OR OLD.updated_at IS DISTINCT FROM NEW.updated_at THEN + IF OLD.job_status IN ('FAILED', 'CANCELLED') AND ( + OLD.job_status IS DISTINCT FROM NEW.job_status + OR OLD.request_digest IS DISTINCT FROM NEW.request_digest + OR OLD.request_payload IS DISTINCT FROM NEW.request_payload + OR OLD.attempt_count IS DISTINCT FROM NEW.attempt_count + OR OLD.failure_code IS DISTINCT FROM NEW.failure_code + OR OLD.cancellation_key_hash IS DISTINCT FROM NEW.cancellation_key_hash + OR OLD.cancellation_code IS DISTINCT FROM NEW.cancellation_code + OR OLD.job_cancelled_at IS DISTINCT FROM NEW.job_cancelled_at + OR OLD.created_at IS DISTINCT FROM NEW.created_at + OR OLD.updated_at IS DISTINCT FROM NEW.updated_at + ) THEN -- The row being updated is already locked by PostgreSQL. Every child insertion -- locks its source and root before it can commit, so an existence lookup is enough -- to serialize this mutation without taking child locks in the reverse direction. From f33af0227926721d80397773050d2aa86fcc8c90 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:37:00 +0900 Subject: [PATCH 095/103] test(etl): permit documented replay guard comments --- .../java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java index 275cc44c..d8b4ea9e 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java @@ -118,7 +118,7 @@ void referencedEvidenceLookupRunsOnlyForReplayEligibleTerminalRows() throws IOEx Pattern terminalEvidenceGuard = Pattern.compile( "IF OLD\\.job_status IN \\('FAILED', 'CANCELLED'\\) AND \\( " + "OLD\\.job_status IS DISTINCT FROM NEW\\.job_status .*" - + "OLD\\.updated_at IS DISTINCT FROM NEW\\.updated_at \\) THEN " + + "OLD\\.updated_at IS DISTINCT FROM NEW\\.updated_at \\) THEN .*?" + "PERFORM 1 FROM etl_job_records AS child_record" ); From ba7fda84c1f0622231859e4b5cf3667288457186 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:42:56 +0900 Subject: [PATCH 096/103] test(etl): require exact replay index catalog verification --- .../EtlJobReplayPostgresWorkflowTest.java | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/documentation/EtlJobReplayPostgresWorkflowTest.java b/etl-service/src/test/java/com/xtrmetl/etl/documentation/EtlJobReplayPostgresWorkflowTest.java index 8c660702..0fe4e7e3 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/documentation/EtlJobReplayPostgresWorkflowTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/documentation/EtlJobReplayPostgresWorkflowTest.java @@ -41,6 +41,31 @@ void directDevelopWorkflowRunsCompleteReplayMigrationChainOnPostgres18() throws assertTrue(verifier.contains( "etl-service/src/test/postgresql/replay_lineage_migration.sql" )); + assertTrue(verifier.contains( + "pg_get_indexdef(index_record.indexrelid, 1, true)" + )); + assertTrue(verifier.contains( + "pg_get_expr(index_record.indpred, index_record.indrelid)" + )); + assertTrue(verifier.contains("index_record.indnkeyatts = 1")); + assertTrue(verifier.contains("index_record.indnatts = 1")); + assertTrue(verifier.contains("NOT index_record.indisunique")); + assertTrue(verifier.contains( + "pg_get_indexdef(index_record.indexrelid, 1, true) = " + + "'replay_source_job_record_id'" + )); + assertTrue(verifier.contains( + "pg_get_expr(index_record.indpred, index_record.indrelid) = " + + "'(replay_source_job_record_id IS NOT NULL)'" + )); + assertTrue(verifier.contains( + "pg_get_indexdef(index_record.indexrelid, 1, true) = " + + "'replay_root_job_record_id'" + )); + assertTrue(verifier.contains( + "pg_get_expr(index_record.indpred, index_record.indrelid) = " + + "'(replay_root_job_record_id IS NOT NULL)'" + )); } @Test From fd7c61f3a9167ec89bb8c30c4d56c6db3448a92c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:46:04 +0900 Subject: [PATCH 097/103] fix(etl): verify exact replay index definitions --- scripts/verify-postgresql-migrations.sh | 32 ++++++++++++++++++++----- 1 file changed, 26 insertions(+), 6 deletions(-) diff --git a/scripts/verify-postgresql-migrations.sh b/scripts/verify-postgresql-migrations.sh index 46946bb1..b099c077 100644 --- a/scripts/verify-postgresql-migrations.sh +++ b/scripts/verify-postgresql-migrations.sh @@ -98,6 +98,10 @@ BEGIN RAISE EXCEPTION 'replay source and root must each use a self-reference with ON DELETE RESTRICT'; END IF; + -- Names and ready/valid flags are insufficient: CREATE INDEX CONCURRENTLY IF NOT EXISTS + -- would retain a valid but incorrectly shaped index with the expected name. Reconstruct the + -- indexed column and stored predicate from PostgreSQL catalogs and require the exact bounded + -- one-column, nonunique partial-index contracts used by the lineage trigger. SELECT count(*) INTO replay_lookup_index_count FROM pg_class AS index_class @@ -106,15 +110,31 @@ BEGIN JOIN pg_class AS table_record ON table_record.oid = index_record.indrelid WHERE table_record.relname = 'etl_job_records' - AND index_class.relname IN ( - 'etl_job_replay_source_lookup_index', - 'etl_job_replay_root_lookup_index' - ) AND index_record.indisready - AND index_record.indisvalid; + AND index_record.indisvalid + AND index_record.indnkeyatts = 1 + AND index_record.indnatts = 1 + AND NOT index_record.indisunique + AND ( + ( + index_class.relname = 'etl_job_replay_source_lookup_index' + AND pg_get_indexdef(index_record.indexrelid, 1, true) + = 'replay_source_job_record_id' + AND pg_get_expr(index_record.indpred, index_record.indrelid) + = '(replay_source_job_record_id IS NOT NULL)' + ) + OR + ( + index_class.relname = 'etl_job_replay_root_lookup_index' + AND pg_get_indexdef(index_record.indexrelid, 1, true) + = 'replay_root_job_record_id' + AND pg_get_expr(index_record.indpred, index_record.indrelid) + = '(replay_root_job_record_id IS NOT NULL)' + ) + ); IF replay_lookup_index_count <> 2 THEN - RAISE EXCEPTION 'replay lookup indexes are missing or invalid'; + RAISE EXCEPTION 'replay lookup indexes are missing, invalid, or have unexpected definitions'; END IF; SELECT string_agg(pg_get_constraintdef(constraint_record.oid), ' ') From b4082961711a6bbc9a4160616433705313f38878 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:47:50 +0900 Subject: [PATCH 098/103] test(docs): require exact replay index verification evidence --- .../DurableJobReplayDocumentationTest.java | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java b/etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java index 2ede03eb..8cb34b05 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobReplayDocumentationTest.java @@ -112,6 +112,28 @@ void doctoringPinsReplayStandardsAndVersionedKeyDomains() throws IOException { assertTrue(standardsEvidence.contains("lineage-column immutability")); } + @Test + void verificationDocsRequireExactReplayIndexCatalogDefinitions() throws IOException { + String runbook = read("docs/operations/durable-job-replay.md") + .replaceAll("\\s+", " "); + String standardsEvidence = read( + "docs/doctoring/durable-job-replay-standards-evidence.md" + ).replaceAll("\\s+", " "); + String changelog = read("CHANGELOG.md").replaceAll("\\s+", " "); + + assertTrue(runbook.contains( + "exact indexed column, one-key/one-attribute nonunique shape, " + + "and `IS NOT NULL` partial predicate" + )); + assertTrue(standardsEvidence.contains( + "`pg_get_indexdef` reconstructs each indexed column and `pg_get_expr` " + + "reconstructs each stored partial predicate" + )); + assertTrue(changelog.contains( + "exact replay-index column, predicate, and one-column nonunique shape" + )); + } + private static String read(String relativePath) throws IOException { return Files.readString(projectRoot().resolve(relativePath), StandardCharsets.UTF_8); } From ba08f863fc2ede7779be0eca3a361b04e9291d9d Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:50:22 +0900 Subject: [PATCH 099/103] fix(etl): retain replay index verifier diagnostics contract --- scripts/verify-postgresql-migrations.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/verify-postgresql-migrations.sh b/scripts/verify-postgresql-migrations.sh index b099c077..c61c756c 100644 --- a/scripts/verify-postgresql-migrations.sh +++ b/scripts/verify-postgresql-migrations.sh @@ -134,7 +134,7 @@ BEGIN ); IF replay_lookup_index_count <> 2 THEN - RAISE EXCEPTION 'replay lookup indexes are missing, invalid, or have unexpected definitions'; + RAISE EXCEPTION 'replay lookup indexes are missing or invalid, or have unexpected definitions'; END IF; SELECT string_agg(pg_get_constraintdef(constraint_record.oid), ' ') From 48e9ed9256dc8d31c6a11ddbd59d7389ee32d534 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:52:38 +0900 Subject: [PATCH 100/103] docs(etl): document exact replay index verification --- docs/operations/durable-job-replay.md | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/docs/operations/durable-job-replay.md b/docs/operations/durable-job-replay.md index 3e6d3917..6bbfe716 100644 --- a/docs/operations/durable-job-replay.md +++ b/docs/operations/durable-job-replay.md @@ -156,8 +156,13 @@ inserts, lifecycle updates, and deletes remain available during the build. Postg concurrent index creation inside a transaction block, so the companion `.sql.conf` files set `executeInTransaction=false`; application configuration also sets `spring.flyway.postgresql.transactional-lock=false` so Flyway does not wrap these migrations in a -PostgreSQL transactional advisory lock. Migration verification requires both indexes to be -`indisready` and `indisvalid` before rollout is considered complete. +PostgreSQL transactional advisory lock. + +Migration verification requires both indexes to be `indisready` and `indisvalid`, then reconstructs +the exact indexed column, one-key/one-attribute nonunique shape, and `IS NOT NULL` partial predicate +from PostgreSQL catalogs. A same-named ready and valid index with a different column, included +attribute, uniqueness contract, or predicate fails closed instead of being mistaken for the required +lineage-support index. ## Worker behavior @@ -201,7 +206,8 @@ trigger enforcement, or replay-key idempotency. as operationally ready. 5. Verify that exactly one `validate_etl_job_replay_lineage` function and one `etl_job_replay_lineage_guard_trigger` exist on `etl_job_records`, and that both replay lookup - indexes are ready and valid. + indexes are ready and valid with the exact column, one-column nonunique shape, and partial + predicate recorded in the migration contract. 6. Smoke-test a disposable failed source, exact payload acceptance, same-key retry, and key conflict. 7. Confirm the source is unchanged and the new row has source/root/generation lineage. 8. In an isolated migration rehearsal, confirm PostgreSQL rejects: @@ -242,7 +248,8 @@ for a deliberately separate replay. A cancelled or interrupted `CREATE INDEX CONCURRENTLY` can leave an invalid index behind. Stop the migration rollout and preserve the exact application SHA, Flyway schema history, PostgreSQL logs, and sanitized index metadata. Do not mark the migration successful while either replay lookup index is -missing, not ready, or invalid. +missing, not ready, invalid, or differently defined from its exact indexed-column, attribute-count, +nonunique, and partial-predicate contract. Confirm no active migration process is still using the affected index, then remove only the invalid artifact outside an explicit transaction: @@ -256,7 +263,7 @@ Drop only the index that actually failed; the two commands are shown together as index inventory. After the invalid index is removed, repair the failed Flyway migration record using the approved deployment procedure and rerun that exact migration. Its companion configuration must still contain `executeInTransaction=false`. Do not edit an applied migration, create an untracked -replacement index, or bypass the ready/valid verification query. +replacement index, or bypass exact catalog-definition verification. ### Trigger, referenced evidence, or lineage integrity rejection @@ -325,8 +332,14 @@ https://www.postgresql.org/docs/18/sql-createtrigger.html PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: INSERT*. https://www.postgresql.org/docs/18/sql-insert.html +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: pg_index*. +https://www.postgresql.org/docs/18/catalog-pg-index.html + PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: PL/pgSQL trigger functions*. https://www.postgresql.org/docs/18/plpgsql-trigger.html +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: System information functions and operators*. +https://www.postgresql.org/docs/18/functions-info.html + World Wide Web Consortium. (2013). *PROV-O: The PROV ontology*. https://www.w3.org/TR/prov-o/ From 94ba0079fcd78806a7b5e77a0346aef0416931cd Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:53:59 +0900 Subject: [PATCH 101/103] docs(doctoring): record exact replay index evidence --- .../durable-job-replay-standards-evidence.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/doctoring/durable-job-replay-standards-evidence.md b/docs/doctoring/durable-job-replay-standards-evidence.md index d64c3e1f..47b3bc88 100644 --- a/docs/doctoring/durable-job-replay-standards-evidence.md +++ b/docs/doctoring/durable-job-replay-standards-evidence.md @@ -15,7 +15,7 @@ The accepted replay response uses `202 Accepted` with a monitor URI because dura | Atomic new-row creation and conflict handling | PostgreSQL 18 `INSERT` and transaction documentation | One transaction validates source ownership, digest, replay identity, and lineage before inserting one new job | | Same-owner source and root existence | PostgreSQL 18 constraints | Composite foreign keys bind source and root to the new row's `principal_scope_hash`, and `ON DELETE RESTRICT` protects retained history | | Exact request and lineage transition | PostgreSQL 18 `CREATE TRIGGER` and PL/pgSQL trigger functions | A row-level `BEFORE INSERT OR UPDATE OF` trigger validates terminal source, immediate-source digest equality, first root, exact generation successor, initial pending lifecycle, and lineage-column immutability | -| Online descendant and foreign-key lookup | PostgreSQL 18 `CREATE INDEX` | Separate partial source/root indexes are built with one `CREATE INDEX CONCURRENTLY` per nontransactional Flyway migration and must be ready and valid | +| Online descendant and foreign-key lookup | PostgreSQL 18 `CREATE INDEX`, `pg_index`, and system-information functions | Separate partial source/root indexes are built with one `CREATE INDEX CONCURRENTLY` per nontransactional Flyway migration and must match their exact ready, valid, nonunique, one-column partial-index definitions | | Derivation lineage | W3C PROV-O | New job is derived from the immediate source and preserves an immutable first-root/generation chain | | Domain separation rationale | NIST SP 800-185 | Replay-key hashing uses a versioned replay-specific domain; the SHA-256 construction does not claim cSHAKE or TupleHash conformance | @@ -57,7 +57,9 @@ mightyETL therefore uses two migrations: - `V8__add_etl_job_replay_source_lookup_index.sql` owns only `etl_job_replay_source_lookup_index`; - `V9__add_etl_job_replay_root_lookup_index.sql` owns only `etl_job_replay_root_lookup_index`. -Each companion `.sql.conf` sets `executeInTransaction=false`. One index per nontransactional migration makes failure, Flyway repair, and rollback independently auditable. PostgreSQL migration verification requires both `pg_index.indisready` and `pg_index.indisvalid`. An interrupted build is not accepted as passing evidence; operators remove only the failed artifact with `DROP INDEX CONCURRENTLY`, repair the exact Flyway migration record through the approved process, and rerun without editing an applied migration. +Each companion `.sql.conf` sets `executeInTransaction=false`. One index per nontransactional migration makes failure, Flyway repair, and rollback independently auditable. PostgreSQL migration verification requires both `pg_index.indisready` and `pg_index.indisvalid`. + +Ready and valid flags alone do not prove that a same-named index has the required definition. `pg_index.indnkeyatts` and `pg_index.indnatts` establish that the contract has exactly one key attribute and no included attributes, while `indisunique` proves the expected nonunique shape. `pg_get_indexdef` reconstructs each indexed column and `pg_get_expr` reconstructs each stored partial predicate. The verifier therefore rejects a same-named index unless it targets the exact source or root column and has the matching `IS NOT NULL` predicate. An interrupted or definition-mismatched build is not accepted as passing evidence; operators remove only the failed artifact with `DROP INDEX CONCURRENTLY`, repair the exact Flyway migration record through the approved process, and rerun without editing an applied migration. ## Security and privacy boundary @@ -68,7 +70,7 @@ Connector replay is enabled only when target effects participate in the mightyET ## Verification obligations - real PostgreSQL 18 migration rehearsal for trigger and function presence, composite self-referencing foreign keys, and `ON DELETE RESTRICT`; -- both replay lookup indexes present, ready, and valid after separate V8/V9 concurrent builds; +- both replay lookup indexes present, ready, valid, nonunique, one-column, and bound to the exact source/root columns and `IS NOT NULL` predicates after separate V8/V9 concurrent builds; - exact-payload acceptance and immediate-source digest mismatch rejection at both service and database boundaries; - owner-safe missing/foreign behavior; - same-key replay, key reuse conflict, and concurrent admission tests; @@ -98,6 +100,10 @@ PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: CREAT PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: INSERT*. https://www.postgresql.org/docs/18/sql-insert.html +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: pg_index*. https://www.postgresql.org/docs/18/catalog-pg-index.html + PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: PL/pgSQL trigger functions*. https://www.postgresql.org/docs/18/plpgsql-trigger.html +PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: System information functions and operators*. https://www.postgresql.org/docs/18/functions-info.html + World Wide Web Consortium. (2013). *PROV-O: The PROV ontology*. https://www.w3.org/TR/prov-o/ From f5fd3bab3d7e028901011d245c99509aa880dcb0 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 20:58:53 +0900 Subject: [PATCH 102/103] docs(changelog): record exact replay index verification --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f75d56e6..acee43b5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Authenticated operators can now create an ordinary pending durable job from an immutable failed or cancelled source only after resupplying a byte-identical bounded JSON payload; the terminal source remains unchanged and succeeded sources remain non-replayable. - Replay lineage now remains database-authoritative across every writer: PostgreSQL validates the terminal immediate source, first root, exact source/root/generation continuity, generation-one identity, and one-step generation succession, while immutable lineage fields prevent post-insert reparenting. - Once a terminal job is referenced as a replay source or root, PostgreSQL row-lock serialization now freezes its status, request evidence, attempt/failure state, cancellation evidence, and lifecycle timestamps so descendants cannot silently acquire different historical meaning. +- PostgreSQL migration verification now reconstructs and validates the exact replay-index column, predicate, and one-column nonunique shape in addition to readiness and validity, so a same-named but incorrectly defined index fails closed. - Authenticated operators can now perform owner-scoped durable-job cancellation for `PENDING` and `RUNNING` work through an idempotent action that commits terminal `CANCELLED`, clears payload and lease state, stores only `cancellation_key_hash` plus a fixed code and timestamp, and returns stable RFC 9457 conflicts when success or failure already won. - Cancellation-first races now make the former exact lease stale and roll back transactional target and response-ledger effects; success-first races remain `SUCCEEDED`, while same-key cancellation replays and different-key reuse fails closed. - Owner-scoped durable-job status responses now emit deterministic weak SHA-256 `ETag` validators; ordinary and wildcard `If-None-Match` requests return an empty RFC 9110 `304 Not Modified` response only after authenticated owner-safe lookup, while `Cache-Control: no-store` remains unchanged. @@ -29,7 +30,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - The managed Jackson component set now uses the patched 2.21.5 BOM, closing CVE-2026-54515, CVE-2026-59889, and GHSA-mhm7-754m-9p8w while keeping core, annotations, datatype, and module artifacts aligned. - Durable `POST /api/etl/jobs` submissions now return RFC 9110 `202 Accepted`, a stable job representation, `Location` status-monitor metadata, and explicit replay metadata without changing the synchronous `/api/etl/process` contract. - Concurrent requests using the same authenticated-principal-scoped semantic idempotency key now return immediate RFC 9457 `409 etl_idempotency_request_in_progress` responses through PostgreSQL `pg_try_advisory_xact_lock`; retries after completion still replay the committed response. -- `POST /api/etl/process` now supports optional authenticated-principal-scoped `Idempotency-Key` retries with atomic target writes, durable response replay, payload-conflict rejection, and explicit replay response metadata. +- `POST /api/etl/process` now supports optional authenticated-principal-scoped `Idempotency-Key` retries with atomic ETL and response ledger commits, PostgreSQL transaction advisory locks, RFC 9651 structured key interoperability, Flyway migration, RFC 9457 conflicts, and operator documentation. - `Idempotency-Key` now prefers the quoted RFC 9651 Structured Field String representation while retaining and normalizing the legacy raw representation to the same durable ledger key. - ETL request errors now use RFC 9457 `application/problem+json` responses with a stable `errorCode`, fixed type URI, explicit 400/401/404/409/413/422/503/500 taxonomy, and no internal exception text in client responses. - ETL requests now enforce bounded UTF-8 payload and record-count limits, prevalidate and transform the complete batch before the first JDBC call, and commit accepted records inside one Spring transaction. From 75339599b18062df9c38ef5f9ddd2f47e74b73b0 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 21:01:10 +0900 Subject: [PATCH 103/103] fix(changelog): preserve idempotent retry wording --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index acee43b5..2f0ac24c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,7 +30,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - The managed Jackson component set now uses the patched 2.21.5 BOM, closing CVE-2026-54515, CVE-2026-59889, and GHSA-mhm7-754m-9p8w while keeping core, annotations, datatype, and module artifacts aligned. - Durable `POST /api/etl/jobs` submissions now return RFC 9110 `202 Accepted`, a stable job representation, `Location` status-monitor metadata, and explicit replay metadata without changing the synchronous `/api/etl/process` contract. - Concurrent requests using the same authenticated-principal-scoped semantic idempotency key now return immediate RFC 9457 `409 etl_idempotency_request_in_progress` responses through PostgreSQL `pg_try_advisory_xact_lock`; retries after completion still replay the committed response. -- `POST /api/etl/process` now supports optional authenticated-principal-scoped `Idempotency-Key` retries with atomic ETL and response ledger commits, PostgreSQL transaction advisory locks, RFC 9651 structured key interoperability, Flyway migration, RFC 9457 conflicts, and operator documentation. +- `POST /api/etl/process` now supports optional authenticated-principal-scoped `Idempotency-Key` retries with atomic target writes, durable response replay, payload-conflict rejection, and explicit replay response metadata. - `Idempotency-Key` now prefers the quoted RFC 9651 Structured Field String representation while retaining and normalizing the legacy raw representation to the same durable ledger key. - ETL request errors now use RFC 9457 `application/problem+json` responses with a stable `errorCode`, fixed type URI, explicit 400/401/404/409/413/422/503/500 taxonomy, and no internal exception text in client responses. - ETL requests now enforce bounded UTF-8 payload and record-count limits, prevalidate and transform the complete batch before the first JDBC call, and commit accepted records inside one Spring transaction.