Skip to content

feat(etl): cancel durable jobs with lease-fenced state - #133

Closed
seonghobae wants to merge 35 commits into
feat/durable-job-conditional-status-v2from
feat/durable-job-cancellation
Closed

feat(etl): cancel durable jobs with lease-fenced state#133
seonghobae wants to merge 35 commits into
feat/durable-job-conditional-status-v2from
feat/durable-job-cancellation

Conversation

@seonghobae

@seonghobae seonghobae commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Superseded by repaired-stack PR #147

This pull request is closed as superseded by #147, feat(etl): replace cancellation on repaired conditional-status stack.

Reason: this branch is rooted on the obsolete old conditional-status chain through closed PR #131. Replacement #147 is rooted on the exact repaired predecessor #146, currently descends from 9e4d69e0bb33ab57627c697a5d028b5309eca2bb, and has fresh literal-head CI, Dependency Review, CycloneDX SBOM, CodeRabbit status, and zero unresolved review threads on its current head. Continuing this PR would preserve an obsolete ancestry boundary and make old checks/reviews easy to misread as transferable evidence.

This branch and its fail-first history are intentionally preserved. No checks, reviews, approvals, statuses, base snapshots, or synthetic-merge evidence from this PR transfer to #147.


Stack dependency and current repair state

This pull request is locally stacked on PR #131 at exact base addfe57e8948a8e274b297cf629582895fb760fc; exact current head is ee50859d654b89fdba5b14424fe7b8afc3b6c99d. The immediate local boundary remains exact and GitHub reports this PR mergeable.

The global stack is not eligible for progression. PR #122 is closed as superseded by draft replacement #138. The earliest invalid boundary remains exact current PR #121 head 75265fae9fcf5faac6f8b9cab63247ac86f23dd4 -> #138, which is diverged. PR #129 also remains non-mergeable against the closed former-worker branch. Do not deepen or repair this cancellation slice until the upstream #121 -> #138 -> repaired/replacement #129 -> #130 -> #131 chain is valid. Old checks, reviews, statuses, approvals, and base snapshots do not transfer across later repairs.

The former cancellation head 78bd456218e39e20cd3b335f989b0d2e78f946fc was reconciled non-destructively with PR #131's exact head at merge commit ee50859d654b89fdba5b14424fe7b8afc3b6c99d, preserving complete implementation and fail-first history without force push, ours/theirs shortcuts, discarded predecessor work, or rewritten test evidence.

Closes #132.

Buyer-visible product gap

Enterprise operators can submit, execute, list, poll, and conditionally validate durable ETL jobs, but previously could not stop unwanted pending or running work. This slice adds an authenticated owner-scoped, idempotent cancellation action backed by a database-owned terminal CANCELLED state.

Implemented bounded slice

  • POST /api/etl/jobs/{job_record_id}/cancellation with the existing authenticated principal and a bounded Idempotency-Key;
  • one conditional owner-matched database update as the cancellation authority;
  • CANCELLED as a fifth terminal lifecycle state;
  • atomic payload and lease clearing for both PENDING and RUNNING work;
  • domain-separated principal-and-job-scoped SHA-256 replay identity, fixed etl_job_cancelled_by_owner code, and database cancellation timestamp without retaining raw principals or raw keys;
  • Idempotency-Replayed: false for the first committed transition and true for the same semantic replay;
  • stable RFC 9457 errors for missing/reused cancellation keys, owner-safe not-found, already-succeeded, already-failed, and unresolved concurrent transition;
  • weak status ETag invalidation after cancellation;
  • no Retry-After on the terminal cancelled representation;
  • transactional V6 migration with descriptive multi-word snake_case columns and complete lifecycle checks;
  • rollout, monitoring, incident, rollback, domain-separation, and non-transactional connector limitation evidence.

Concurrency and atomicity boundary

If cancellation commits before exact-lease success, the former worker's terminal predicate updates zero rows and raises StaleEtlJobLeaseException; Spring rolls back transactional target and etl_idempotency_records writes. If success commits first, cancellation returns 409 etl_job_already_succeeded and does not rewrite the terminal state. Exactly one terminal outcome wins.

Two simultaneous requests using the same cancellation key converge to one first transition and one replay. Simultaneous requests using different keys converge to one transition and one stable etl_job_cancellation_key_reused conflict. The row remains one terminal, payload-free, lease-free resource.

Cancellation invalidates the database lease but does not forcibly terminate arbitrary computation. Remote warehouses, files, APIs, and brokers that cannot join the same transaction require connector-native idempotency, cancellation, or compensation before equivalent external-side-effect reversal can be claimed.

Reality-based test and review evidence

Exact-head tests cover pending/running cancellation, lease clearing and stale-lease behavior, same-key replay, conflicting-key rejection, concurrent convergence, domain-separated replay hashes, owner isolation, terminal-state conflicts, invalid key/principal rejection, cancellation-first rollback, cancelled polling/ETag compatibility, controller failure paths, migration, operations, standards, changelog, and rollback contracts.

CodeRabbit's former-head manual inspection is historical only and is not the required independent approval.

Current exact-head evidence

For exact head ee50859d654b89fdba5b14424fe7b8afc3b6c99d on its current stacked target:

Successful exact-head evidence:

  • CI run 31093886791: success;
  • Dependency Review run 31093886395: success;
  • CycloneDX SBOM run 31093886924: success;
  • CodeRabbit commit status: success;
  • unresolved review threads: zero;
  • formal reviews: zero.

Not passing / still blocking:

  • global predecessor stack integrity is not satisfied because #121 -> #138 is diverged and feat(etl): list durable jobs with owner-scoped cursors #129 is not a valid next boundary;
  • SAST Semgrep and Security Scan are absent on this non-default stacked base;
  • qualifying independent non-author APPROVED review is absent.

The successful evidence above is exact-head evidence only for this current target and does not authorize stack progression or transfer to a later head/base. Queued, pending, absent, skipped-required, neutral-required, cancelled, failed, stale-head, predecessor-head, old-base, and synthetic-merge-only evidence are not passing.

Privacy and naming boundaries

Raw principals, raw submission or cancellation keys, payloads, internal hashes, lease identifiers, SQL, exception messages, target identities, and unbounded labels remain absent from responses and ordinary telemetry. The versioned replay domain includes the principal hash and job identifier so reuse of one raw client key cannot be correlated across jobs or tenants from stored cancellation hashes. New database objects are cancellation_key_hash, cancellation_code, job_cancelled_at, and named lifecycle constraints, all using descriptive multi-word snake_case.

Standards and documentation

The design, implementation plan, intake contract, operations runbook, and doctoring evidence record APA 7th references to RFC 9110, RFC 9457, PostgreSQL 18 application consistency and UPDATE semantics, and NIST SP 800-185 domain-separation methodology. The SHA-256 implementation does not claim cSHAKE or TupleHash conformance. CHANGELOG.md records the user-visible behavior, security boundary, migration, and transactional limitation.

Merge policy

This historical PR is superseded and must not merge. Continue the cancellation slice only through #147 and regenerate affected evidence there after every head/base movement.

@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 55d2f330-5442-47d4-9db0-1a40aef4d78d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

Copy link
Copy Markdown
Collaborator Author

@opencode-agent @cwl-noema-review Please perform independent exact-head review of f38fc572e516d7605345432d16614c79fac21b37 only. Verify that cancellation success is bound to the owner-matched conditional database transition; pending cancellation prevents every later worker claim; a cancellation-first race makes the former exact lease stale and rolls back transactional target and response-ledger effects; success-first and failure-first outcomes remain unchanged with stable conflicts; replay is principal/job scoped without raw key retention; foreign and missing jobs remain indistinguishable; CANCELLED clears payload and lease fields, emits no Retry-After, and invalidates the prior status ETag; migration and rollback contracts are fail-closed. Exact-head Ubuntu, macOS, Windows, Dependency Review, CycloneDX SBOM, CodeRabbit status, configured zero-missed production coverage, mergeability, and zero unresolved threads are successful. Submit APPROVE only if this exact SHA has no blocker. Preserve manual-merge and stack order.

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

Review exact current head b1c4f26d68ffab7b0ba28fef318f576366062fc3 only. Exact-head Ubuntu, macOS, Windows, Dependency Review, CycloneDX SBOM, CodeRabbit status, 421 no-skip project tests, and configured zero-missed production coverage succeed on the repaired stack. Verify the owner-scoped conditional cancellation authority, pending cancellation preventing worker claims, running payload and lease clearing, cancellation-versus-success atomicity, stale former-lease rollback, same-job replay and conflicting-key rejection, domain-separated principal-and-job-scoped replay identity, owner-safe not-found behavior, RFC 9457 taxonomy, V6 lifecycle constraints, terminal polling/ETag behavior, privacy boundaries, and the limitation to transactional target effects. Treat any later head as requiring a new review.

Copy link
Copy Markdown
Collaborator Author

@opencode-agent @cwl-noema-review Please perform independent exact-head review of b1c4f26d68ffab7b0ba28fef318f576366062fc3 only. Verify that cancellation success is bound to the owner-matched conditional database transition; pending cancellation prevents every later worker claim; a cancellation-first race makes the former exact lease stale and rolls back transactional target and response-ledger effects; success-first and failure-first outcomes remain unchanged with stable conflicts; replay is principal/job scoped without raw key retention or cross-job/tenant hash correlation; foreign and missing jobs remain indistinguishable; CANCELLED clears payload and lease fields, emits no Retry-After, and invalidates the prior status ETag; migration, domain-version compatibility, and rollback contracts are fail-closed. Exact-head Ubuntu, macOS, Windows, Dependency Review, CycloneDX SBOM, CodeRabbit status, 421 no-skip tests, zero-missed configured production coverage, mergeability, and zero unresolved threads are successful. Submit APPROVE only if this exact SHA has no blocker. Preserve manual-merge and stack order.

@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor
🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

expected='b1c4f26d68ffab7b0ba28fef318f576366062fc3'
actual="$(git rev-parse HEAD)"
printf 'HEAD=%s\nEXPECTED=%s\n' "$actual" "$expected"
git status --short
git show --no-patch --format='commit=%H%nparents=%P%nsubject=%s' "$expected"
printf '\nChanged files from stated stack base:\n'
git diff --name-status aa7bb331faf8e7b3be439a17c36bafed9fc9638d "$expected"
printf '\nCandidate cancellation and migration files:\n'
fd -t f . | rg -i '(cancel|etl.*job|job.*etl|v6|migration|openapi|controller|service|repository|runbook|rollback|changelog)'

Length of output: 17639


🏁 Script executed:

#!/bin/bash
set -euo pipefail

for f in \
  etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobController.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobService.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobCancellation.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobStatus.java \
  etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobPollingAdvice.java \
  etl-service/src/main/java/com/xtrmetl/etl/service/EtlRequestError.java
do
  printf '\n===== OUTLINE: %s =====\n' "$f"
  ast-grep outline "$f" --items all
done

printf '\n===== V6 MIGRATION =====\n'
cat -n etl-service/src/main/resources/db/migration/V6__add_etl_job_cancellation.sql

printf '\n===== CANCELLATION AND LEASE SQL/CALL SITES =====\n'
rg -n -i -C 5 \
  'cancel|cancellation|job_cancelled_at|cancellation_key_hash|etl_job_cancelled_by_owner|StaleEtlJobLeaseException|lease_token|UPDATE\s+etl_job' \
  etl-service/src/main/java/com/xtrmetl/etl

printf '\n===== FOCUSED TEST OUTLINES =====\n'
for f in etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCancellation*Test.java \
         etl-service/src/test/java/com/xtrmetl/etl/controller/EtlJobPollingAdviceTest.java \
         etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobControllerTest.java \
         etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobControllerFailureTest.java
do
  [ -f "$f" ] || continue
  printf '\n===== OUTLINE: %s =====\n' "$f"
  ast-grep outline "$f" --items all
done

Length of output: 99527


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n===== CONTROLLER CANCELLATION AND ETAG =====\n'
sed -n '190,350p' etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobController.java

printf '\n===== CANCELLATION SERVICE =====\n'
sed -n '60,125p;272,345p;440,490p;630,715p' \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobService.java

printf '\n===== LEASE CLAIM AND TERMINAL PREDICATES =====\n'
sed -n '40,165p;245,325p' \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java
sed -n '45,75p' etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobExecutionService.java

printf '\n===== PRIOR MIGRATION CONSTRAINT DEFINITIONS =====\n'
rg -n -C 3 'CONSTRAINT (etl_job_status_value_check|etl_job_payload_lifecycle_check|etl_job_lease_lifecycle_check|etl_job_failure_lifecycle_check)' \
  etl-service/src/main/resources/db/migration/V2__create_etl_job_records.sql \
  etl-service/src/main/resources/db/migration/V3__add_etl_job_lease_fencing.sql

printf '\n===== ATOMICITY, LEASE FENCE, CLAIM, DOMAIN, AND CONDITIONAL-STATUS ASSERTIONS =====\n'
for f in \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCancellationAtomicityIntegrationTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCancellationLeaseFenceIntegrationTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCancellationClaimIntegrationTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCancellationKeyDomainIntegrationTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCancellationConditionalStatusTest.java
do
  printf '\n===== %s =====\n' "$f"
  cat -n "$f"
done

Length of output: 72935


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n===== CANCELLATION CONTROLLER AND FAILURE CONTRACT ASSERTIONS =====\n'
rg -n -A45 -B4 \
  'cancelsAnOwnedJobAndReturnsTheTerminalStatus|marksAnIdenticalCancellationAsReplayed|requiresAuthenticationAndAKeyBeforeCancellationServiceAccess|preservesTypedCancellationConflicts|treatsMalformedCancellationIdentifiersAsOwnerSafeNotFound|mapsCancellation(Database|Unexpected)FailuresWithoutLeakingMessages' \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobControllerTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobControllerFailureTest.java

printf '\n===== SERVICE REPLAY, OWNER-SAFETY, AND TERMINAL-CONFLICT ASSERTIONS =====\n'
rg -n -A50 -B4 \
  'cancel|Cancellation|replay|foreign|notFound|alreadySucceeded|alreadyFailed|keyReused' \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobServiceIntegrationTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobServiceBoundaryTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobMigrationDocumentationTest.java

printf '\n===== V6 MIGRATION CONTRACT ASSERTIONS =====\n'
rg -n -A50 -B4 \
  'V6|cancellation_key_hash|cancellation_code|job_cancelled_at|CANCELLED|DROP CONSTRAINT|rollback|fail.closed|fail-closed' \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobMigrationDocumentationTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobCancellationDocumentationTest.java

printf '\n===== DOCUMENTED TRANSACTIONAL LIMITATION AND ROLLBACK BOUNDARY =====\n'
rg -n -i -C 4 \
  'non-transactional|transactional target|connector-native|compensation|cannot.*cancel|rollback|external.*effect|limitation' \
  docs/operations/durable-job-cancellation.md \
  docs/etl/durable-job-intake.md \
  CHANGELOG.md

printf '\n===== PRIVACY-BOUNDARY SEARCH IN NEW CANCELLATION SOURCES =====\n'
rg -n -i -C 2 \
  'principal|idempotency|cancellation.key|payload|lease|hash|exception' \
  etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobController.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobCancellation.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobService.java

Length of output: 208821


@seonghobae I reviewed only commit b1c4f26d68ffab7b0ba28fef318f576366062fc3.

I found no blocking issue in the inspected cancellation implementation and contract tests.

Verified items:

  • EtlJobService.cancelOwned uses one transactional, owner-matched conditional UPDATE.
  • The transition accepts only PENDING and RUNNING.
  • Cancellation clears request_payload and every lease field atomically.
  • A same-key replay returns Idempotency-Replayed: true.
  • A different key for an already cancelled job returns etl_job_cancellation_key_reused.
  • The replay hash includes the versioned domain, principal-scope hash, job identifier, and normalized key.
  • Foreign and missing jobs return the same etl_job_not_found classification.
  • Success-first and failure-first outcomes remain terminal and return stable conflicts.
  • Cancellation-first invalidates the exact former lease. The later success transition raises StaleEtlJobLeaseException and rolls back same-transaction target and response-ledger writes.
  • CANCELLED has V6 lifecycle constraints for payload removal, lease removal, hash, code, and timestamp.
  • Cancelled polling removes Retry-After. A prior active weak ETag does not validate the cancelled representation.
  • Responses exclude raw principals, raw cancellation keys, payloads, lease identifiers, and internal hashes.
  • The runbook correctly limits the rollback guarantee to target and ledger effects in the same transaction. It does not claim reversal of arbitrary external effects.

I did not execute repository code in the sandbox. I relied on the reported exact-head CI, coverage, dependency-review, SBOM, and test evidence.

Keep manual-merge and the stack order. Do not treat this review as an approval or as evidence for a later commit.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

Review exact current head 78bd456218e39e20cd3b335f989b0d2e78f946fc only. Exact-head Ubuntu, macOS, Windows, Dependency Review, CycloneDX SBOM, CodeRabbit status, 422 no-skip ETL-service tests plus the remaining reactor-module suites, and configured zero-missed production coverage succeed on the repaired stack. Verify the owner-scoped conditional cancellation authority, pending cancellation preventing worker claims, concurrent identical-key first-plus-replay convergence, concurrent different-key success-plus-stable-conflict convergence, running payload and lease clearing, cancellation-versus-success atomicity, stale former-lease rollback, same-job replay and conflicting-key rejection, domain-separated principal-and-job-scoped replay identity, owner-safe not-found behavior, RFC 9457 taxonomy, V6 lifecycle constraints, terminal polling/ETag behavior, privacy boundaries, and the limitation to transactional target effects. Treat any later head as requiring a new review.

Copy link
Copy Markdown
Collaborator Author

@opencode-agent @cwl-noema-review Please perform independent exact-head review of 78bd456218e39e20cd3b335f989b0d2e78f946fc only. Verify cancellation success is bound to the owner-matched conditional database transition; pending cancellation prevents later claims; simultaneous same-key requests converge to one first transition plus one replay and simultaneous different-key requests to one transition plus one stable conflict; cancellation-first makes the former exact lease stale and rolls back transactional target and response-ledger effects; success-first and failure-first outcomes remain unchanged; replay is principal-and-job scoped without raw key retention or cross-job/tenant hash correlation; foreign and missing jobs remain indistinguishable; CANCELLED clears payload and lease fields, has no Retry-After, and invalidates the prior status ETag; migration, domain-version compatibility, and rollback fail closed. Exact-head Ubuntu, macOS, Windows, Dependency Review, CycloneDX SBOM, CodeRabbit status, configured zero-missed production coverage, mergeability, and zero unresolved threads are successful. Submit APPROVE only for this SHA. Preserve manual-merge and stack order.

@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor
🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

expected='78bd456218e39e20cd3b335f989b0d2e78f946fc'
actual="$(git rev-parse HEAD)"
printf 'HEAD=%s\nEXPECTED=%s\n' "$actual" "$expected"
git show --no-patch --format='commit=%H%nparents=%P%nsubject=%s' "$expected"
if [ "$actual" != "$expected" ]; then
  echo 'ERROR: checked-out HEAD does not match the requested commit.' >&2
  exit 2
fi

printf '\n=== Changed files from stack base ===\n'
git diff --name-status aa7bb331faf8e7b3be439a17c36bafed9fc9638d "$expected"

printf '\n=== Candidate source and test files ===\n'
fd -t f . | rg -i '(EtlJob(Cancellation|Service|LeaseRepository|ExecutionService|Controller|PollingAdvice)|V6__|Cancellation.*Test|MigrationDocumentationTest|durable-job-cancellation|durable-job-intake|CHANGELOG)'

printf '\n=== Source outlines ===\n'
for f in \
  etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobController.java \
  etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobPollingAdvice.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobService.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobExecutionService.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobCancellation.java
do
  if [ -f "$f" ]; then
    printf '\n--- %s ---\n' "$f"
    ast-grep outline "$f" --items all
  fi
done

printf '\n=== Cancellation, lease, polling, and error implementation ===\n'
rg -n -i -C 7 \
  'cancelOwned|cancellation|cancellation_key_hash|cancellation_code|job_cancelled_at|etl_job_cancelled_by_owner|StaleEtlJobLeaseException|lease_token|request_payload|Idempotency-Replayed|Retry-After|ETag|not_found|already_succeeded|already_failed|key_reused' \
  etl-service/src/main/java/com/xtrmetl/etl

printf '\n=== V6 migration ===\n'
cat -n etl-service/src/main/resources/db/migration/V6__add_etl_job_cancellation.sql

printf '\n=== Related lifecycle constraints in earlier migrations ===\n'
rg -n -C 5 \
  'etl_job_status_value_check|etl_job_payload_lifecycle_check|etl_job_lease_lifecycle_check|etl_job_failure_lifecycle_check' \
  etl-service/src/main/resources/db/migration

printf '\n=== Cancellation tests and documentation evidence ===\n'
rg -n -i -C 4 \
  'simultaneous|concurrent|replay|key.reused|foreign|not.found|already.succeeded|already.failed|stale|lease|claim|ETag|Retry-After|payload|transactional target|connector-native|compensation|non-transactional|raw.*key|raw.*principal|domain' \
  etl-service/src/test/java/com/xtrmetl/etl \
  docs/operations/durable-job-cancellation.md \
  docs/etl/durable-job-intake.md \
  CHANGELOG.md

Length of output: 203148


🏁 Script executed:

#!/bin/bash
set -euo pipefail

expected='78bd456218e39e20cd3b335f989b0d2e78f946fc'
test "$(git rev-parse HEAD)" = "$expected"

printf '\n=== EtlJobService cancellation methods and SQL ===\n'
rg -n -A120 -B20 \
  'EtlJobCancellation cancelOwned|int cancelOwned|cancelOwned\(|UPDATE etl_job_records|cancellationReplay|cancellationKeyHash|CANCELLED_BY_OWNER_CODE' \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobService.java

printf '\n=== Lease claim, success fence, and stale handling ===\n'
rg -n -A100 -B20 \
  'claimNext|markSucceeded|markFailed|StaleEtlJobLeaseException|UPDATE etl_job_records|lease_claim_id' \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobExecutionService.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorker.java

printf '\n=== Cancellation endpoint, status endpoint, and polling advice ===\n'
rg -n -A110 -B15 \
  'cancellation|cancelOwned|Idempotency-Replayed|Retry-After|ETag|If-None-Match|CANCELLED|ProblemDetail|EtlRequestException' \
  etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobController.java \
  etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobPollingAdvice.java \
  etl-service/src/main/java/com/xtrmetl/etl/service/EtlRequestError.java \
  etl-service/src/main/java/com/xtrmetl/etl/service/EtlRequestException.java

printf '\n=== V6 migration, exact text ===\n'
cat -n etl-service/src/main/resources/db/migration/V6__add_etl_job_cancellation.sql

printf '\n=== Targeted atomicity, former-lease, controller, and polling tests ===\n'
for f in \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCancellationAtomicityIntegrationTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCancellationLeaseFenceIntegrationTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCancellationConditionalStatusTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobControllerTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobControllerFailureTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/documentation/DurableJobCancellationDocumentationTest.java
do
  if [ -f "$f" ]; then
    printf '\n--- %s ---\n' "$f"
    cat -n "$f"
  fi
done

printf '\n=== Transactional-boundary documentation ===\n'
rg -n -i -C 5 \
  'transactional target|same transaction|response-ledger|external|non-transactional|connector-native|compensation|rollback|cancellation.*does not|cannot.*cancel' \
  docs/operations/durable-job-cancellation.md \
  docs/etl/durable-job-intake.md \
  CHANGELOG.md

Length of output: 243864


@seonghobae I reviewed only commit 78bd456218e39e20cd3b335f989b0d2e78f946fc.

I found no blocking issue in the inspected implementation and contract tests.

Verified items:

  • EtlJobService.cancelOwned uses one transactional, owner-matched conditional UPDATE as the cancellation authority.
  • The transition accepts only PENDING and RUNNING.
  • Cancellation atomically clears the retained payload, failure code, and all lease fields.
  • A cancelled pending job cannot be claimed by a later worker.
  • Same-key concurrent requests produce one committed cancellation and one replay.
  • Different-key concurrent requests produce one committed cancellation and one etl_job_cancellation_key_reused conflict.
  • The cancellation replay hash uses the versioned domain, principal-scope hash, job identifier, and normalized key. Raw keys are not retained.
  • Foreign and missing job identifiers have the same etl_job_not_found result.
  • Success-first and failure-first outcomes return stable conflicts without rewriting the terminal state.
  • Cancellation-first invalidates the former exact lease. The later success update affects zero rows, raises StaleEtlJobLeaseException, and rolls back same-transaction target and response-ledger writes.
  • V6__add_etl_job_cancellation.sql adds CANCELLED and fail-closed payload, lease, cancellation-hash, cancellation-code, and cancellation-timestamp constraints.
  • The cancellation endpoint returns Idempotency-Replayed: false for the first transition and true for a semantic replay.
  • The RFC 9457 error taxonomy has stable codes and does not expose diagnostic messages.
  • A cancelled status omits Retry-After. Cancellation changes the weak status ETag.
  • Response models and documented telemetry boundaries exclude raw principals, raw keys, payloads, leases, hashes, SQL, and exception text.
  • The runbook limits rollback guarantees to target and response-ledger writes in the same transaction. It does not claim external-side-effect reversal.

I did not execute repository code in the sandbox. I relied on the reported exact-head CI, coverage, dependency-review, SBOM, and test evidence.

Keep manual-merge and the required stack order. This review does not apply to a later commit.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Preserve the complete cancellation implementation and fail-first history while incorporating PR #131 exact head addfe57. Resolve CHANGELOG.md additively and retain exact-head publication-policy safeguards.
seonghobae added a commit that referenced this pull request Aug 6, 2026
Preserve the complete replay implementation and fail-first history while incorporating PR #133 exact head ee50859. Resolve CHANGELOG.md additively and retain exact-head publication-policy safeguards.

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review
@cwl-noema-review
@opencode-agent

Please review the exact live head against its declared #131 stacked base only. Verify cancellation uses one authenticated owner-matched conditional database transition, only PENDING/RUNNING can become terminal CANCELLED, payload and every lease field clear atomically, exact former leases cannot commit later, same-key/different-key concurrency converges deterministically, principal/job domain-separated replay identity stores no raw key, success/failure races preserve one terminal outcome, cancellation-first target/ledger effects roll back, database objects are descriptive multi-word snake_case, and non-transactional connector limitations remain explicit. Review only; do not retarget, update the branch, remove manual-merge, merge, release, or alter protections.

@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

repo="ContextualWisdomLab/mightyETL"
pr="133"

printf '=== Live PR metadata ===\n'
gh pr view "$pr" --repo "$repo" --json number,url,state,isDraft,headRefOid,baseRefOid,baseRefName,mergeable,title \
  --jq '{number,url,state,isDraft,headRefOid,baseRefOid,baseRefName,mergeable,title}'

printf '\n=== Declared predecessor PR `#131` metadata ===\n'
gh pr view 131 --repo "$repo" --json number,url,state,headRefOid,baseRefOid,baseRefName \
  --jq '{number,url,state,headRefOid,baseRefOid,baseRefName}'

printf '\n=== Checked-out commit and ancestry ===\n'
head="$(git rev-parse HEAD)"
printf 'checked_out_head=%s\n' "$head"
git show --no-patch --format='commit=%H%nparents=%P%nsubject=%s' "$head"

base="$(gh pr view 131 --repo "$repo" --json headRefOid --jq '.headRefOid')"
printf 'predecessor_head=%s\n' "$base"
if git merge-base --is-ancestor "$base" "$head"; then
  printf 'predecessor_is_ancestor=true\n'
else
  printf 'predecessor_is_ancestor=false\n'
fi

printf '\n=== Files changed from predecessor head ===\n'
git diff --name-status "$base" "$head"

printf '\n=== Candidate files ===\n'
fd -t f . | rg -i \
  '(EtlJob(Service|Cancellation|LeaseRepository|ExecutionService|Worker|Controller|PollingAdvice)|V6__|Cancellation.*(Test|sql)|durable-job-cancellation|durable-job-intake|CHANGELOG)'

Length of output: 7419


🏁 Script executed:

#!/bin/bash
set -euo pipefail

expected='ee50859d654b89fdba5b14424fe7b8afc3b6c99d'
test "$(git rev-parse HEAD)" = "$expected"

printf '=== Source outlines ===\n'
for f in \
  etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobController.java \
  etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobPollingAdvice.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobCancellation.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobService.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobExecutionService.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobStatus.java \
  etl-service/src/main/java/com/xtrmetl/etl/service/EtlRequestError.java
do
  printf '\n--- %s ---\n' "$f"
  ast-grep outline "$f" --items all
done

printf '\n=== Cancellation authority, replay identity, and response behavior ===\n'
rg -n -A105 -B20 \
  'cancelOwned|cancellationReplay|cancellationKeyHash|CANCELLED_BY_OWNER_CODE|UPDATE etl_job_records|cancellation_key_hash|cancellation_code|job_cancelled_at|Idempotency-Replayed|cancellation' \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobService.java \
  etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobController.java

printf '\n=== Claim, terminal lease fence, and transaction boundary ===\n'
rg -n -A105 -B20 \
  'claimNext|markSucceeded|markFailed|StaleEtlJobLeaseException|lease_claim_id|lease_token|UPDATE etl_job_records' \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobExecutionService.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobWorker.java

printf '\n=== Polling and error classification ===\n'
rg -n -A80 -B15 \
  'CANCELLED|Retry-After|ETag|If-None-Match|not_found|already_succeeded|already_failed|key_reused|ProblemDetail' \
  etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobPollingAdvice.java \
  etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobController.java \
  etl-service/src/main/java/com/xtrmetl/etl/service/EtlRequestError.java

printf '\n=== V6 migration and prior lifecycle constraints ===\n'
cat -n etl-service/src/main/resources/db/migration/V6__add_etl_job_cancellation.sql
rg -n -C 5 \
  'etl_job_status_value_check|etl_job_payload_lifecycle_check|etl_job_lease_lifecycle_check|etl_job_failure_lifecycle_check' \
  etl-service/src/main/resources/db/migration

printf '\n=== Focused contract tests ===\n'
for f in \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCancellationAtomicityIntegrationTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCancellationClaimIntegrationTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCancellationConcurrencyIntegrationTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCancellationKeyDomainIntegrationTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCancellationLeaseFenceIntegrationTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCancellationConditionalStatusTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCancellationServiceBoundaryTest.java
do
  printf '\n--- %s ---\n' "$f"
  cat -n "$f"
done

printf '\n=== Transactional limitation and privacy documentation ===\n'
rg -n -i -C 5 \
  'non-transactional|transactional target|same transaction|response-ledger|external.*effect|connector-native|compensation|raw.*key|raw.*principal|domain.separation|rollback' \
  docs/operations/durable-job-cancellation.md \
  docs/etl/durable-job-intake.md \
  CHANGELOG.md \
  docs/doctoring/durable-job-cancellation-key-domain-separation.md

Length of output: 200495


🏁 Script executed:

#!/bin/bash
set -euo pipefail

expected='ee50859d654b89fdba5b14424fe7b8afc3b6c99d'
test "$(git rev-parse HEAD)" = "$expected"

printf '=== EtlJobService cancellation implementation ===\n'
rg -n \
  'CANCEL|cancelOwned|cancellationKeyHash|findCancellation|findOwned|requireActive.*Transaction|principal.*Hash|sha256' \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobService.java

sed -n '1,150p;250,390p;590,760p;760,920p' \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobService.java

printf '\n=== Cancellation value object ===\n'
cat -n etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobCancellation.java

printf '\n=== Controller completion and status ETag construction ===\n'
sed -n '250,330p;430,530p' \
  etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobController.java

printf '\n=== Service and controller cancellation contract assertions ===\n'
rg -n -A55 -B5 \
  'cancel|Cancellation|foreign|missing|notFound|alreadySucceeded|alreadyFailed|replay|keyReused|Idempotency-Replayed|payload|lease' \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobServiceIntegrationTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCancellationTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobControllerTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobControllerFailureTest.java

printf '\n=== All cancellation SQL mutation sites ===\n'
rg -n -U -P \
  '(?s)UPDATE etl_job_records\s+SET\s+job_status = .{0,800}?CANCELLED.{0,800}?WHERE' \
  etl-service/src/main/java/com/xtrmetl/etl \
  etl-service/src/test/java/com/xtrmetl/etl/job

Length of output: 123329


@seonghobae I reviewed only live head ee50859d654b89fdba5b14424fe7b8afc3b6c99d against #131 head addfe57e8948a8e274b297cf629582895fb760fc.

The declared #131 head is an ancestor of the reviewed head.

I found no blocking issue in the requested scope.

Verified items:

  • EtlJobService.cancelOwned uses one transactional, authenticated owner-matched conditional database update as the cancellation authority.
  • The update permits only PENDING and RUNNING rows to transition to CANCELLED.
  • The same update clears request_payload, failure_code, lease_claim_id, lease_owner_id, and lease_expires_at.
  • The update stores only cancellation_key_hash, fixed etl_job_cancelled_by_owner, and job_cancelled_at.
  • The cancellation replay identity includes the versioned domain, principal-scope hash, job identifier, and normalized key. Raw cancellation keys are not stored.
  • The owner-scoped follow-up read returns the same etl_job_not_found result for foreign and missing jobs.
  • Same-key concurrent requests converge to one first cancellation and one replay.
  • Different-key concurrent requests converge to one first cancellation and one etl_job_cancellation_key_reused conflict.
  • markSucceeded requires the exact live lease. A cancellation clears the lease fields. The former worker then receives StaleEtlJobLeaseException.
  • The cancellation-first test verifies rollback of target and etl_idempotency_records writes in the worker transaction.
  • Success-first and failure-first outcomes remain terminal. Cancellation returns stable conflict results and does not rewrite either state.
  • V6 adds descriptive multi-word snake_case columns and named lifecycle constraints. The constraints require cleared payload and lease fields for CANCELLED.
  • A cancelled representation removes Retry-After. Its weak ETag differs from the prior active representation.
  • The controller emits Idempotency-Replayed: false for the first transition and true for an identical replay.
  • Response models and error handling exclude raw principals, raw keys, payloads, hashes, lease identifiers, SQL, and exception messages.
  • The operations documentation limits rollback to target and response-ledger writes in the same transaction. It does not claim reversal of non-transactional connector effects.

I did not execute repository code in the sandbox. I used static inspection and the exact-head CI and coverage evidence that you provided.

I did not retarget, update, merge, release, remove manual-merge, or change protections. This review does not apply to a later head.

You are interacting with an AI system.

@opencode-agent

opencode-agent Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Queued @cwl-noema-review ; Rejected @opencode-agent: repository is absent from OPENCODE_REPOSITORY_DISPATCH_TARGETS for PR #133 at head ee50859d654b89fdba5b14424fe7b8afc3b6c99d. Central exact-name Actions artifacts are the durable dispatch ledger; existing review workflows remain authoritative for the final verdict and failure evidence.

@opencode-agent

opencode-agent Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Queued @cwl-noema-review ; Rejected @opencode-agent: repository is absent from OPENCODE_REPOSITORY_DISPATCH_TARGETS for PR #133 at head ee50859d654b89fdba5b14424fe7b8afc3b6c99d. Central exact-name Actions artifacts are the durable dispatch ledger; existing review workflows remain authoritative for the final verdict and failure evidence.

@opencode-agent

opencode-agent Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Queued @cwl-noema-review ; Rejected @opencode-agent: repository is absent from OPENCODE_REPOSITORY_DISPATCH_TARGETS for PR #133 at head ee50859d654b89fdba5b14424fe7b8afc3b6c99d. Central exact-name Actions artifacts are the durable dispatch ledger; existing review workflows remain authoritative for the final verdict and failure evidence.

@opencode-agent

opencode-agent Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Queued @cwl-noema-review ; Rejected @opencode-agent: repository is absent from OPENCODE_REPOSITORY_DISPATCH_TARGETS for PR #133 at head ee50859d654b89fdba5b14424fe7b8afc3b6c99d. Central exact-name Actions artifacts are the durable dispatch ledger; existing review workflows remain authoritative for the final verdict and failure evidence.

@opencode-agent

opencode-agent Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Queued @cwl-noema-review ; Rejected @opencode-agent: repository is absent from OPENCODE_REPOSITORY_DISPATCH_TARGETS for PR #133 at head ee50859d654b89fdba5b14424fe7b8afc3b6c99d. Central exact-name Actions artifacts are the durable dispatch ledger; existing review workflows remain authoritative for the final verdict and failure evidence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant