feat(etl): replace cancellation on repaired conditional-status stack

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- Owner-scoped durable-job cancellation now commits an idempotent `CANCELLED` terminal state through one principal-bound conditional update, stores the domain-separated `cancellation_key_hash` without raw identity, clears payload and lease state, invalidates stale workers, removes cancelled-job polling guidance, and exposes a no-store authenticated cancellation resource. Cancellation-first races roll back transactional target and response-ledger effects; external non-transactional side effects remain outside this guarantee.
 - Owner-scoped durable-job status responses now emit deterministic weak SHA-256 `ETag` validators; ordinary and wildcard `If-None-Match` requests return an empty RFC 9110 `304 Not Modified` response only after authenticated owner-safe lookup, while `Cache-Control: no-store` remains unchanged.
 - Active durable-job status responses now emit an RFC 9110 `Retry-After` delay for `PENDING` and `RUNNING` states only when local worker execution is enabled, derived from the bounded worker fixed-delay configuration with upward whole-second rounding; terminal states and intake-only maintenance mode omit the advisory.
 - Durable job operators can now list only their own jobs through bounded newest-first keyset pagination with canonical opaque cursors, deterministic timestamp-plus-UUID ordering, `Cache-Control: no-store`, and RFC 8288 next-page links without offset drift or cross-tenant existence leakage.
@@ -153,7 +154,7 @@ Through code analysis, identified the platform as:
 - Microservices-based architecture using Spring Cloud
 - Real-time Change Data Capture using Debezium
 - Event streaming via Apache Kafka
-- Service discovery with Netflix Eureka
+- Service discovery and registration
 - Distributed tracing with Zipkin
 
 #### Key Components Documented

docs/doctoring/durable-job-cancellation-key-domain-separation.md

@@ -0,0 +1,88 @@
+# Durable-job cancellation replay identity domain separation
+
+## Decision
+
+mightyETL never stores a raw cancellation `Idempotency-Key`. It stores one lowercase SHA-256 replay identity computed from an explicit versioned domain, the authenticated-principal hash, the durable job identifier, and the normalized key:
+
+```text
+SHA-256(
+  "mightyetl:durable-job-cancellation:v1:"
+  || principal_scope_hash
+  || ":"
+  || job_record_id
+  || ":"
+  || normalized_cancellation_key
+)
+```
+
+This value is used only to prove that a later request addresses the same principal, the same job, and the same semantic cancellation key. It is not an authentication credential and grants no job authority; every transition and replay read independently binds the owner hash and job identifier in SQL.
+
+## Threat addressed
+
+Hashing the raw client key alone would hide its plaintext but preserve equality across every row. A database observer could correlate two jobs or tenants that reused the same cancellation key even though ordinary API representations never reveal the key or hash.
+
+The versioned contextual prefix and explicit principal/job components partition the replay identity. The same normalized raw key therefore produces:
+
+- the same stored hash for the same principal and job, preserving deterministic replay;
+- a different stored hash for another job in the same principal namespace;
+- a different stored hash for the same job-shaped identifier under another principal namespace;
+- a different stored hash after a deliberate future domain-version change.
+
+The implementation does not claim to use cSHAKE, TupleHash, or another NIST SP 800-185 primitive. It uses the existing SHA-256 utility with an unambiguous fixed-layout contextual input. NIST SP 800-185 is cited as primary methodological evidence for customization and tuple/domain separation concepts, not as an implementation-conformance claim. NIST announced in March 2025 that SP 800-185 will be revised; until a replacement is finalized, this document cites the current final publication and the revision decision separately.
+
+## Compatibility boundary
+
+The exact domain string is persisted protocol behavior:
+
+```text
+mightyetl:durable-job-cancellation:v1:
+```
+
+Changing it would make every existing cancelled row fail same-key replay comparison. A future `v2` requires an explicit migration and dual-read compatibility window or a documented replay-breaking release. Silent replacement of the prefix is prohibited.
+
+The current concatenation is unambiguous because:
+
+- `principal_scope_hash` is exactly 64 lowercase hexadecimal characters;
+- the separator is a literal colon;
+- `job_record_id` is the canonical UUID text form;
+- the second separator is a literal colon;
+- the normalized cancellation key follows the bounded safe-ASCII profile and is the final component.
+
+If a later version introduces variable-width or independently nested components, use explicit length prefixes or a tuple-hash construction rather than extending this layout informally.
+
+## Test-first evidence
+
+`EtlJobCancellationKeyDomainIntegrationTest` uses the same raw cancellation key for:
+
+1. two different jobs owned by one principal;
+2. one job owned by another principal.
+
+The test requires three distinct 64-character stored hashes. Existing service-integration tests separately prove that a quoted and legacy-raw representation of the same key on the same job replay one cancellation, while a genuinely different key fails with `etl_job_cancellation_key_reused`.
+
+## Privacy and logging
+
+The raw cancellation key and resulting hash remain absent from:
+
+- HTTP response bodies and headers;
+- RFC 9457 problem details;
+- ordinary logs;
+- metric labels;
+- status, list, polling, and ETag representations;
+- worker lease models.
+
+The hash is a replay identity stored in `cancellation_key_hash`; it is not safe to publish merely because it is one-way. Database access, backups, exports, and support tooling must treat it as internal pseudonymous security data.
+
+## Rollback
+
+Rolling application code back across this change can make same-key replay behavior inconsistent if an older binary derives a raw-key-only hash. Keep the domain-separated implementation deployed while rows created by it are active. A rollback requires either:
+
+- retaining the new comparison algorithm in the older release line; or
+- a reviewed data migration with explicit compatibility evidence.
+
+Never rewrite hashes from user-supplied guesses and never log candidate keys while diagnosing replay mismatches.
+
+## References — APA 7th
+
+Kelsey, J., Chang, S., & Perlner, R. (2016). *SHA-3 derived functions: cSHAKE, KMAC, TupleHash, and ParallelHash* (NIST Special Publication 800-185). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800-185
+
+National Institute of Standards and Technology. (2025, March 12). *Decision to update FIPS 202 and revise SP 800-185*. https://csrc.nist.gov/news/2025/decision-to-update-fips-202-and-revise-sp-800-185

docs/operations/durable-job-cancellation.md

@@ -0,0 +1,244 @@
+# Durable ETL job cancellation
+
+## Purpose
+
+Authenticated operators can stop a durable ETL job that is still `PENDING` or `RUNNING` through:
+
+```http
+POST /api/etl/jobs/{job_record_id}/cancellation HTTP/1.1
+Authorization: Basic <credentials>
+Idempotency-Key: "70dc8b50-e8b2-4e1a-8c5f-d84814708a77"
+```
+
+A successful response proves that the database transition committed or that the same principal,
+job identifier, and normalized cancellation key had already committed. HTTP request acceptance by
+itself is never treated as cancellation success.
+
+The first slice establishes a database-owned terminal state. It does not forcibly terminate a Java
+thread, interrupt arbitrary connector computation, or compensate a non-transactional external
+warehouse.
+
+## HTTP contract
+
+A first cancellation returns:
+
+```http
+HTTP/1.1 200 OK
+Cache-Control: no-store
+Idempotency-Replayed: false
+ETag: W/"<sha256>"
+Content-Type: application/json
+
+{
+  "jobRecordId": "cf4f083f-8c90-4f34-a8b6-b53761de44ef",
+  "jobStatus": "CANCELLED",
+  "attemptCount": 1,
+  "createdAt": "2026-08-05T01:00:00Z",
+  "updatedAt": "2026-08-06T03:00:00Z"
+}
+```
+
+Repeating the same semantic key returns `Idempotency-Replayed: true` and the same terminal resource.
+A different key for the already-cancelled job returns
+`422 etl_job_cancellation_key_reused`. Malformed, missing, and foreign-owned identifiers share the
+same `404 etl_job_not_found` response. Cancellation after committed success or failure returns
+`409 etl_job_already_succeeded` or `409 etl_job_already_failed` under RFC 9110 current-state conflict
+semantics.
+
+Every success and covered failure uses `Cache-Control: no-store`. The response exposes no payload,
+principal, cancellation key, key hash, lease identifier, SQL, exception message, or target identity.
+
+## State machine
+
+```mermaid
+stateDiagram-v2
+    [*] --> PENDING
+    PENDING --> RUNNING: lease-fenced claim
+    PENDING --> CANCELLED: owner cancellation
+    RUNNING --> PENDING: exact-lease retry
+    RUNNING --> SUCCEEDED: target + ledger + exact-lease commit
+    RUNNING --> FAILED: exact-lease terminal failure
+    RUNNING --> CANCELLED: owner cancellation wins
+    CANCELLED --> CANCELLED: same-key replay
+```
+
+`SUCCEEDED`, `FAILED`, and `CANCELLED` are terminal. Cancelled status never receives `Retry-After`.
+Its state and `updatedAt` value also invalidate every earlier weak status `ETag`.
+
+## Database authority
+
+`EtlJobService.cancelOwned` first validates the job identifier, cancellation key, and authenticated
+principal, then performs one conditional update inside a Spring transaction:
+
+```sql
+UPDATE etl_job_records
+SET job_status = 'CANCELLED',
+    request_payload = NULL,
+    failure_code = NULL,
+    lease_claim_id = NULL,
+    lease_owner_id = NULL,
+    lease_expires_at = NULL,
+    cancellation_key_hash = ?,
+    cancellation_code = 'etl_job_cancelled_by_owner',
+    job_cancelled_at = CURRENT_TIMESTAMP,
+    updated_at = CURRENT_TIMESTAMP
+WHERE job_record_id = ?
+  AND principal_scope_hash = ?
+  AND job_status IN ('PENDING', 'RUNNING');
+```
+
+The update count is the authority. A follow-up owner-scoped read classifies a zero-row result as:
+
+- identical `CANCELLED` replay;
+- conflicting cancellation key;
+- already `SUCCEEDED`;
+- already `FAILED`;
+- an active row whose concurrent transition is still unresolved; or
+- owner-safe not found.
+
+The raw principal and key are never stored. Their lowercase SHA-256 values are used only for owner
+selection and replay identity.
+
+## Cancellation-versus-success race
+
+### Cancellation commits first
+
+```mermaid
+sequenceDiagram
+    participant C as Cancellation request
+    participant D as PostgreSQL
+    participant W as Worker transaction
+    C->>D: conditional PENDING/RUNNING → CANCELLED
+    D-->>C: one row committed
+    W->>D: target + response ledger writes
+    W->>D: markSucceeded(exact former lease)
+    D-->>W: zero rows updated
+    W-->>D: rollback target + ledger
+```
+
+The cancelled row has no lease fields. The former worker's exact-live-lease success predicate updates
+zero rows and raises `StaleEtlJobLeaseException`; Spring rolls back its target and
+`etl_idempotency_records` writes.
+
+### Success commits first
+
+```mermaid
+sequenceDiagram
+    participant W as Worker transaction
+    participant D as PostgreSQL
+    participant C as Cancellation request
+    W->>D: target + response ledger + SUCCEEDED commit
+    C->>D: conditional PENDING/RUNNING → CANCELLED
+    D-->>C: zero rows updated
+    C->>D: owner-scoped terminal read
+    D-->>C: SUCCEEDED
+    C-->>C: 409 etl_job_already_succeeded
+```
+
+Exactly one terminal state wins. The endpoint never rewrites `SUCCEEDED` or `FAILED` as cancelled.
+
+## Migration
+
+`V6__add_etl_job_cancellation.sql` adds these descriptive multi-word `snake_case` columns:
+
+- `cancellation_key_hash`;
+- `cancellation_code`;
+- `job_cancelled_at`.
+
+It replaces lifecycle checks so that:
+
+- `PENDING` and `RUNNING` retain a payload;
+- `SUCCEEDED`, `FAILED`, and `CANCELLED` have no payload;
+- only `RUNNING` has lease fields;
+- only `FAILED` has `failure_code`;
+- only `CANCELLED` has the three cancellation fields;
+- hash and code values satisfy bounded fixed formats.
+
+The migration is transactional. Before production rollout, rehearse it against a representative
+PostgreSQL 18 copy and confirm that no out-of-contract legacy row violates the replacement checks.
+Monitor migration duration, lock wait, transaction age, replication lag, and application error rates.
+
+## Rollout
+
+1. Verify PR exact-head CI on Ubuntu, macOS, and Windows with no skipped project test.
+2. Verify dependency review, CycloneDX SBOM, SAST, security scan, unresolved-thread, and independent
+   current-head approval gates.
+3. Apply Flyway V6 before serving the new route.
+4. Keep `mightyetl.etl.jobs.intake-enabled=false` during a conservative schema-only rollout if the
+   deployment process cannot guarantee application/schema ordering.
+5. Enable the new application build and perform an owner-isolation smoke test with a disposable job.
+6. Verify a first cancellation, same-key replay, different-key rejection, and a status read.
+7. Confirm cancelled rows have null payload and lease fields and a fixed cancellation code.
+8. Observe worker `stale` outcomes during deliberate running-job cancellation; this is expected
+   fencing evidence, not a duplicate-execution success.
+
+## Monitoring
+
+The cancellation endpoint uses fixed observation name `etl.jobs.cancel`. Do not attach job IDs,
+principals, raw keys, key hashes, lease IDs, payloads, SQL, exception classes, messages, target
+identities, or queue depth as metric labels.
+
+Monitor at least:
+
+- request rate and HTTP outcome count;
+- cancellation latency;
+- database update latency and lock waits;
+- worker `stale` outcome changes;
+- cancellation replay and key-conflict rate;
+- cancelled rows retaining payload or lease fields, which must remain zero;
+- target or ledger effects associated with cancellation-first tests, which must remain zero.
+
+## Incident response
+
+### Cancellation returns `etl_job_cancellation_in_progress`
+
+Re-read the owner-scoped status. A concurrent claim, retry, success, failure, or cancellation may have
+won after the request's conditional update. Do not retry with a new idempotency key until the current
+terminal or active state is understood.
+
+### Worker reports stale after cancellation
+
+This is the expected safety outcome when cancellation invalidates a running lease. Confirm the target
+and response-ledger transaction rolled back. Repeated stale outcomes without operator cancellations
+may indicate lease expiry, another worker, or database clock/latency problems.
+
+### Cancelled row retains payload or lease data
+
+Treat this as a high-severity lifecycle integrity incident. Stop intake and workers, preserve the row
+and transaction evidence, verify the deployed schema constraints and application SHA, and do not
+manually rewrite the state until the root cause and rollback effects are understood.
+
+## Rollback
+
+Stop serving the cancellation endpoint before application rollback. Older binaries do not understand
+`CANCELLED`, so they must not read or process cancelled rows as if only four states existed.
+
+Do not drop V6 columns or restore the old status constraint while any cancelled row remains. A
+controlled database rollback must first archive cancelled resources and their audit evidence under an
+approved retention policy. Never silently map `CANCELLED` to `FAILED` or `SUCCEEDED`.
+
+After cancelled rows are safely removed and every older binary is deployed, an explicit reviewed
+migration may drop the V6 constraints and columns and restore the four-state lifecycle. Do not edit or
+repair the applied V6 migration file in place.
+
+## Connector limitation
+
+The cancellation-first rollback guarantee is valid for target and response-ledger writes that join the
+same transaction and database as the job state. A remote warehouse, file upload, external API, or
+message broker that cannot participate in that transaction requires connector-native cancellation,
+idempotency, or compensation before the same guarantee can be advertised. This release deliberately
+does not claim arbitrary external side-effect reversal.
+
+## References — APA 7th
+
+Fielding, R., Nottingham, M., & Reschke, J. (2022). *HTTP semantics* (RFC 9110). RFC Editor.
+https://www.rfc-editor.org/rfc/rfc9110
+
+Nottingham, M., Wilde, E., & Dalal, S. (2023). *Problem details for HTTP APIs* (RFC 9457). RFC Editor.
+https://www.rfc-editor.org/rfc/rfc9457
+
+PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: Data consistency checks at
+the application level*. https://www.postgresql.org/docs/18/applevel-consistency.html
+
+PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: UPDATE*.
+https://www.postgresql.org/docs/18/sql-update.html