[Product Gap] Renew active durable-job leases with exact fencing

[seonghobae]

Buyer-visible gap

mightyETL can durably accept, claim, execute, inspect, cancel, and replay ETL jobs, but the current worker lease is fixed at claim time. A valid job whose execution lasts longer than the initial lease can therefore appear abandoned and become eligible for another worker even though the original worker is still healthy. Exact terminal fencing prevents the stale worker from committing database-owned success, but duplicate computation, connector load, and non-transactional external effects remain commercially material risks.

This issue is implementation-ready only after PRs #121, #122, #129, #130, #131, #133, and #135 integrate in order. Do not create a parallel implementation PR while that stack is open. Existing checks, reviews, and approvals do not transfer to the later implementation head.

Proposed bounded vertical slice

Add database-authoritative renewal for one exact live claim without changing job ownership or introducing a second execution authority.

A renewal may extend lease_expires_at only when all predicates still match:

job_record_id       = expected job
job_status          = RUNNING
lease_claim_id      = exact private claim token
lease_owner_id      = exact bounded worker owner
lease_expires_at    > database actual current time

Use one conditional PostgreSQL UPDATE ... RETURNING in a short independent transaction. A zero-row result means the claim is stale, expired, cancelled, terminal, or superseded and must stop further renewal immediately. The database remains the authority; scheduler liveness and process-local state are advisory only.

Time and scheduling contract

• Derive renewal timestamps from PostgreSQL actual wall-clock time, not the application host clock.
• Do not use transaction-start CURRENT_TIMESTAMP for a long-lived renewal transaction. PostgreSQL documents that CURRENT_TIMESTAMP is fixed at transaction start, while clock_timestamp() returns actual current time.
• Keep each heartbeat transaction short and independent from target execution and the final target-plus-ledger-plus-terminal transaction.
• Validate configuration so the renewal interval is materially shorter than the lease duration and preserves a bounded safety margin after realistic scheduler jitter and database latency.
• Do not run heartbeat work on the same single fixed-delay scheduler thread that may be occupied by long-running execution. Spring documents that fixed delay is measured from completion and that some fixed-delay configurations operate on one scheduler thread.
• Stop renewal on terminal success, retry, failure, cancellation, stale-claim detection, interruption, and application shutdown.
• Renewal must never create or recover a claim. Only the ordinary database claim operation may establish ownership.

Exact concurrency invariants

healthy exact claimant renews first
→ lease_expires_at advances using database time
→ another worker cannot reclaim during the renewed interval

cancellation or terminal transition commits first
→ renewal predicate updates zero rows
→ heartbeat stops
→ stale worker cannot restore RUNNING or extend the terminal row

lease expires before renewal commits
→ renewal predicate updates zero rows
→ old worker is stale
→ later exact terminal write remains fenced and rolls back database-owned effects

new claimant supersedes an expired claim
→ old lease_claim_id cannot renew the new claim
→ only the new exact token may renew

Renewal and final success must never overwrite one another through broad job-ID-only updates. Every write remains anchored to the exact claim token and expected lifecycle state.

Lease and fencing research boundary

Gray and Cheriton formalized leases as time-bounded rights that trade bounded inconsistency risk against communication and failure-recovery costs. Chubby later demonstrated that a live lock or session lease alone is not sufficient to protect an external resource: clients need sequencers or equivalent fencing evidence so a stale holder cannot act after a successor acquires authority.

For mightyETL, the PostgreSQL row and exact lease_claim_id predicate are the authority for database-owned lifecycle, target, and response-ledger effects. A renewal extends only that exact authority window. It does not make a remote warehouse, object store, file system, API, or broker obey the database lease. A connector that cannot join the transaction must receive and enforce an exact operation identity, monotonic fencing value, idempotency contract, or compensation protocol before the product can claim duplicate-effect safety.

Modular MSA boundary

The implementation must preserve both standalone and imported-module operation:

• keep the renewal decision behind a narrow repository/port interface whose input is the exact private claim and bounded duration, and whose output is renewed or stale without returning payload or tenant data;
• keep scheduler ownership in an adapter so embedded, Spring-managed, and externally orchestrated deployments can supply their own bounded executor without changing lease semantics;
• keep PostgreSQL actual-time and conditional-update semantics in the production adapter rather than duplicating authority in process-local clocks;
• expose no public lease token and no cross-service endpoint in the first slice;
• allow a future central orchestrator to observe finite-cardinality health and lifecycle signals without becoming the claim authority;
• preserve compatibility with organization-level .github quality/security workflows and future naruon event transport without requiring either dependency for standalone execution.

CSAP, SOC 2, and privacy control evidence

The feature must produce operational evidence suitable for availability, processing-integrity, security, and change-management controls rather than merely adding a heartbeat loop.

• Use a least-privileged application database role that can update only the required durable-job columns through the normal application path; no scheduler credential may bypass owner or exact-claim predicates.
• Record configuration provenance, deployment version, migration version, enable/disable state, and bounded aggregate outcomes so operators can demonstrate approved change, monitoring, incident response, and rollback.
• Alert on renewal database failures, stale-claim rates, scheduler lag, exhaustion of the configured safety margin, and abnormal reclaim frequency using finite-cardinality dimensions.
• Keep audit and ordinary telemetry payload-blind. Never record request payloads, raw principals, private claim tokens, idempotency keys, SQL, exception text, or target identifiers merely to prove renewal activity.
• Do not solve privacy by masking business payloads. The renewal path does not need PII and therefore must not read, copy, transform, mask, or emit it. Use data minimization by architectural separation, encryption in transit/at rest, role-based access, purpose-bound access to the ordinary job path, and explicit retention/deletion controls instead.
• Preserve evidence that a zero-row renewal is treated as loss of authority, not silently retried as success. This supports processing-integrity and availability assertions.
• Include threat modeling for stolen claim tokens, scheduler starvation, clock skew, database failover, replayed renewal calls, and a compromised worker attempting to extend another claim.
• Include a control-to-evidence matrix in doctoring that maps tests, metrics, runbooks, migrations, and review gates to relevant CSAP SaaS and SOC 2 Trust Services Criteria concerns without claiming certification solely from code.

Reality-based test requirements

Use PostgreSQL behavior and production transaction boundaries, not only mocked time or H2 compatibility. Add deterministic tests for:

1. execution longer than the original lease remains singly owned when healthy renewals succeed;
2. two replicas cannot both obtain a live claim while renewal remains current;
3. a worker that stops renewing is reclaimed only after database-observed expiry;
4. an expired token cannot renew, retry, fail, or succeed;
5. cancellation before renewal produces a zero-row renewal and preserves CANCELLED;
6. success before renewal produces a zero-row renewal and preserves SUCCEEDED;
7. retry or failure before renewal cannot be reverted to RUNNING;
8. a superseded old token cannot extend the replacement claimant's lease;
9. database clock and an intentionally skewed application clock still produce correct expiry behavior;
10. configuration rejects zero, negative, equal-to-lease, and insufficient-safety-margin intervals;
11. scheduler starvation is prevented when execution exceeds one renewal interval;
12. transient renewal database failure has a bounded fail-closed policy and never assumes success;
13. application shutdown stops heartbeat tasks without extending leases after ownership is abandoned;
14. finite-cardinality metrics distinguish renewal success, stale claim, configuration rejection, scheduler lag, and database failure without job IDs, principals, claim tokens, SQL, or exception text;
15. a stolen, expired, or replayed claim token cannot extend another owner, another job, or a successor claim;
16. the renewal code path cannot access or emit the job payload and remains correct when the payload contains realistic regulated personal information;
17. configuration and enablement changes produce deterministic control evidence without logging secret or personal values;
18. database failover or connection interruption cannot be interpreted as a successful renewal;
19. every added production statement and branch is covered, no project test is skipped, and every public production API has beginner-readable Javadoc;
20. operations documentation includes rollout, alerting, incident handling, rollback, clock-skew evidence, and the CSAP/SOC 2 control-to-evidence matrix.

Safety and scope boundaries

Lease renewal reduces false expiry for healthy long-running work; it does not make arbitrary external connectors transactional. Warehouses, files, APIs, brokers, and other destinations that cannot participate in the same database transaction still require connector-native idempotency, exact operation keys, monotonic fencing, cancellation, or compensation before duplicate-effect safety can be claimed.

Do not add arbitrary process termination, unbounded heartbeats, owner-controlled lease durations, public lease identifiers, high-cardinality telemetry, priority scheduling, recurring jobs, bulk replay, or a graphical control plane in this slice.

Use existing descriptive multi-word snake_case columns where possible. Any new database object must contain at least two descriptive words and include migration and rollback evidence.

Standards and primary documentation — APA 7th

American Institute of Certified Public Accountants. (2023). 2017 Trust Services Criteria for security, availability, processing integrity, confidentiality, and privacy (with revised points of focus—2022). AICPA & CIMA. https://www.aicpa-cima.com/resources/download/2017-trust-services-criteria-with-revised-points-of-focus-2022

Burrows, M. (2006). The Chubby lock service for loosely-coupled distributed systems. In Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation (pp. 335–350). USENIX Association. https://www.usenix.org/conference/osdi-06/chubby-lock-service-loosely-coupled-distributed-systems

Gray, C. G., & Cheriton, D. R. (1989). Leases: An efficient fault-tolerant mechanism for distributed file cache consistency. In Proceedings of the Twelfth ACM Symposium on Operating Systems Principles (pp. 202–210). Association for Computing Machinery. https://doi.org/10.1145/74850.74870

Korea Internet & Security Agency. (2025). 2025 cloud service security assurance program guide. KISA. https://isms.kisa.or.kr/main/csap/notice

Korea Internet & Security Agency. (2026). Cloud security assurance program: Program overview. KISA. https://isms.kisa.or.kr/main/csap/intro/index.jsp

PostgreSQL Global Development Group. (2026). PostgreSQL 18 documentation: UPDATE. https://www.postgresql.org/docs/18/sql-update.html

PostgreSQL Global Development Group. (2026). PostgreSQL 18 documentation: Date/time functions and operators. https://www.postgresql.org/docs/18/functions-datetime.html

PostgreSQL Global Development Group. (2026). PostgreSQL 18 documentation: Concurrency control. https://www.postgresql.org/docs/18/mvcc.html

Spring Framework. (2026). Task execution and scheduling. https://docs.spring.io/spring-framework/reference/integration/scheduling.html

PostgreSQL documents that UPDATE ... RETURNING reports values only for rows actually updated, enabling a zero-row stale-claim decision without a separate authoritative read. Its date/time documentation distinguishes transaction-start timestamps from clock_timestamp() actual time. Spring's scheduling documentation establishes fixed-delay completion semantics, scheduler-thread constraints, and graceful-shutdown behavior that the heartbeat design must account for. The lease and Chubby research establishes why time-bounded ownership must remain exact and why external resources need fencing beyond a live session. KISA CSAP and AICPA Trust Services Criteria establish the need for documented scope, change control, monitoring, availability, processing integrity, confidentiality, privacy, and auditable operating evidence.