Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
1e46160
docs(etl): define lease-fenced durable job cancellation
seonghobae Aug 6, 2026
32a51e9
docs(etl): plan durable job cancellation implementation
seonghobae Aug 6, 2026
79e6329
test(etl): require durable job cancellation migration
seonghobae Aug 6, 2026
1866db7
test(etl): define owner-scoped cancellation behavior
seonghobae Aug 6, 2026
bf65dfa
feat(etl): add durable job cancellation schema
seonghobae Aug 6, 2026
e56afe6
feat(etl): add durable job cancellation result
seonghobae Aug 6, 2026
ff62af9
feat(etl): add cancelled durable job state
seonghobae Aug 6, 2026
53072fd
feat(etl): classify durable job cancellation conflicts
seonghobae Aug 6, 2026
d26d971
feat(etl): cancel owner-scoped durable jobs
seonghobae Aug 6, 2026
9f3075e
test(etl): cover durable job cancellation result
seonghobae Aug 6, 2026
846e63e
test(etl): cover cancellation validation and race boundaries
seonghobae Aug 6, 2026
5848a6b
test(etl): define durable job cancellation HTTP contract
seonghobae Aug 6, 2026
cded856
test(etl): cover cancellation HTTP failure boundaries
seonghobae Aug 6, 2026
ce74ea0
feat(etl): expose durable job cancellation action
seonghobae Aug 6, 2026
0a8d517
test(etl): keep cancelled jobs terminal for polling
seonghobae Aug 6, 2026
488aa92
test(etl): invalidate active validators after cancellation
seonghobae Aug 6, 2026
9994e5e
test(etl): prove cancellation invalidates active leases
seonghobae Aug 6, 2026
288336f
test(etl): prove cancellation rolls back durable effects
seonghobae Aug 6, 2026
3d77d00
fix(etl): treat cancelled jobs as terminal polling state
seonghobae Aug 6, 2026
3d9eafc
docs(etl): add durable job cancellation runbook
seonghobae Aug 6, 2026
803f549
test(etl): require cancellation operations evidence
seonghobae Aug 6, 2026
9aa306c
docs(etl): record durable job cancellation
seonghobae Aug 6, 2026
2589fa5
docs(etl): align durable job lifecycle with cancellation
seonghobae Aug 6, 2026
f1d5c2a
test(etl): align cancellation rollback documentation assertion
seonghobae Aug 6, 2026
8b17ce3
test(etl): align cancellation terminal-outcome wording
seonghobae Aug 6, 2026
c8c7274
test(etl): match exact cancellation evidence wording
seonghobae Aug 6, 2026
ace17fe
docs(etl): clarify transactional cancellation scope
seonghobae Aug 6, 2026
f38fc57
test(etl): prove pending cancellation prevents worker claims
seonghobae Aug 6, 2026
75cece6
fix(etl): domain-separate cancellation replay identity
seonghobae Aug 6, 2026
e68ec0f
test(etl): prevent cross-job cancellation-key correlation
seonghobae Aug 6, 2026
6fed78e
docs(etl): record cancellation replay domain separation
seonghobae Aug 6, 2026
b725c33
test(etl): require cancellation replay domain separation evidence
seonghobae Aug 6, 2026
b1c4f26
test(etl): keep cancellation changelog contract buyer-focused
seonghobae Aug 6, 2026
78bd456
test(etl): prove concurrent cancellation convergence
seonghobae Aug 6, 2026
ee50859
merge: reconcile cancellation with conditional-status predecessor
seonghobae Aug 6, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Changed

- Authenticated operators can now perform owner-scoped durable-job cancellation for `PENDING` and `RUNNING` work through an idempotent action that commits terminal `CANCELLED`, clears payload and lease state, stores only `cancellation_key_hash` plus a fixed code and timestamp, and returns stable RFC 9457 conflicts when success or failure already won.
- Cancellation-first races now make the former exact lease stale and roll back transactional target and response-ledger effects; success-first races remain `SUCCEEDED`, while same-key cancellation replays and different-key reuse fails closed.
- Owner-scoped durable-job status responses now emit deterministic weak SHA-256 `ETag` validators; ordinary and wildcard `If-None-Match` requests return an empty RFC 9110 `304 Not Modified` response only after authenticated owner-safe lookup, while `Cache-Control: no-store` remains unchanged.
- Active durable-job status responses now emit an RFC 9110 `Retry-After` delay for `PENDING` and `RUNNING` states only when local worker execution is enabled, derived from the bounded worker fixed-delay configuration with upward whole-second rounding; terminal states and intake-only maintenance mode omit the advisory.
- The durable job pagination index now uses PostgreSQL `CREATE INDEX CONCURRENTLY` with a migration-local Flyway `executeInTransaction=false` companion configuration, preserving production writers and documenting invalid-index recovery and concurrent rollback.
Expand All @@ -35,6 +37,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- Transactional migration `V6__add_etl_job_cancellation.sql`, owner-safe cancellation API and replay model, exact lease-invalidation and cancellation-versus-success integration tests, plus rollout, incident, connector-limitation, and rollback evidence in `docs/operations/durable-job-cancellation.md`.
- Deterministic ordinary, wildcard, changed-state, changed-failure-code, null-versus-empty, and unrelated-response conditional polling tests, complete controller Javadoc, privacy and rollback guidance, and APA 7th standards evidence in `docs/etl/durable-job-polling.md`.
- Controller-scoped polling advice, deterministic active/terminal lifecycle tests, disabled-worker fail-closed behavior, sub-second rounding coverage, rollback guidance, and APA 7th standards evidence in `docs/etl/durable-job-polling.md`.
- Owner-scoped durable job list models and HTTP contract, strict cursor and page-limit validation, one-extra-row next-page detection, the descriptive `etl_job_owner_pagination_index`, deterministic tenant-isolation and equal-timestamp tests, migration rollback guidance, and APA 7th standards evidence in `docs/etl/durable-job-intake.md`.
Expand Down Expand Up @@ -74,6 +77,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Security

- Cancellation stores only a principal-scoped SHA-256 replay identity and a fixed machine code, exposes no raw principal, key, hash, payload, lease, SQL, exception, or target detail, and requires an owner-matched conditional database update before reporting success.
- Conditional status validators are SHA-256 digests of only the complete owner-authorized operator-safe representation; payloads, raw principals, idempotency keys, internal hashes, leases, SQL, and exception text remain excluded, and wildcard evaluation occurs only after owner-safe lookup.
- Polling advice exposes only a bounded delay integer and is omitted when local execution is disabled or terminal; it never contains job, lease, principal, key, hash, payload, SQL, exception, target, or queue-depth data.
- Job-list cursors contain only ordering keys, never authority or sensitive values; every page query independently binds the hashed authenticated principal, malformed and non-canonical cursors fail before database access, and list responses exclude payloads, principals, keys, hashes, lease identifiers, SQL, and exception text.
Expand Down
88 changes: 88 additions & 0 deletions docs/doctoring/durable-job-cancellation-key-domain-separation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Durable-job cancellation replay identity domain separation

## Decision

mightyETL never stores a raw cancellation `Idempotency-Key`. It stores one lowercase SHA-256 replay identity computed from an explicit versioned domain, the authenticated-principal hash, the durable job identifier, and the normalized key:

```text
SHA-256(
"mightyetl:durable-job-cancellation:v1:"
|| principal_scope_hash
|| ":"
|| job_record_id
|| ":"
|| normalized_cancellation_key
)
```

This value is used only to prove that a later request addresses the same principal, the same job, and the same semantic cancellation key. It is not an authentication credential and grants no job authority; every transition and replay read independently binds the owner hash and job identifier in SQL.

## Threat addressed

Hashing the raw client key alone would hide its plaintext but preserve equality across every row. A database observer could correlate two jobs or tenants that reused the same cancellation key even though ordinary API representations never reveal the key or hash.

The versioned contextual prefix and explicit principal/job components partition the replay identity. The same normalized raw key therefore produces:

- the same stored hash for the same principal and job, preserving deterministic replay;
- a different stored hash for another job in the same principal namespace;
- a different stored hash for the same job-shaped identifier under another principal namespace;
- a different stored hash after a deliberate future domain-version change.

The implementation does not claim to use cSHAKE, TupleHash, or another NIST SP 800-185 primitive. It uses the existing SHA-256 utility with an unambiguous fixed-layout contextual input. NIST SP 800-185 is cited as primary methodological evidence for customization and tuple/domain separation concepts, not as an implementation-conformance claim.

## Compatibility boundary

The exact domain string is persisted protocol behavior:

```text
mightyetl:durable-job-cancellation:v1:
```

Changing it would make every existing cancelled row fail same-key replay comparison. A future `v2` requires an explicit migration and dual-read compatibility window or a documented replay-breaking release. Silent replacement of the prefix is prohibited.

The current concatenation is unambiguous because:

- `principal_scope_hash` is exactly 64 lowercase hexadecimal characters;
- the separator is a literal colon;
- `job_record_id` is the canonical UUID text form;
- the second separator is a literal colon;
- the normalized cancellation key follows the bounded safe-ASCII profile and is the final component.

If a later version introduces variable-width or independently nested components, use explicit length prefixes or a tuple-hash construction rather than extending this layout informally.

## Test-first evidence

`EtlJobCancellationKeyDomainIntegrationTest` uses the same raw cancellation key for:

1. two different jobs owned by one principal;
2. one job owned by another principal.

The test requires three distinct 64-character stored hashes. Existing service-integration tests separately prove that a quoted and legacy-raw representation of the same key on the same job replay one cancellation, while a genuinely different key fails with `etl_job_cancellation_key_reused`.

## Privacy and logging

The raw cancellation key and resulting hash remain absent from:

- HTTP response bodies and headers;
- RFC 9457 problem details;
- ordinary logs;
- metric labels;
- status, list, polling, and ETag representations;
- worker lease models.

The hash is a replay identity stored in `cancellation_key_hash`; it is not safe to publish merely because it is one-way. Database access, backups, exports, and support tooling must treat it as internal pseudonymous security data.

## Rollback

Rolling application code back across this change can make same-key replay behavior inconsistent if an older binary derives a raw-key-only hash. Keep the domain-separated implementation deployed while rows created by it are active. A rollback requires either:

- retaining the new comparison algorithm in the older release line; or
- a reviewed data migration with explicit compatibility evidence.

Never rewrite hashes from user-supplied guesses and never log candidate keys while diagnosing replay mismatches.

## References — APA 7th

Kelsey, J., Chang, S., & Perlner, R. (2016). *SHA-3 derived functions: cSHAKE, KMAC, TupleHash, and ParallelHash* (NIST Special Publication 800-185). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800-185

National Institute of Standards and Technology. (2025, March 12). *Decision to update FIPS 202 and revise SP 800-185*. https://csrc.nist.gov/news/2025/decision-to-update-fips-202-and-revise-sp-800-185
Loading