feat(etl): add lease-fenced durable job worker

[seonghobae]

Purpose

Replace superseded diverged durable-worker PR #138 with a worker slice that descends from the exact current PR #121 predecessor, preserve the original fail-first history without destructive rewriting, and complete a production-ready lease-fenced durable execution boundary with aligned operator documentation.

Exact current stack boundary

• immediate predecessor PR: ci: schedule NVIDIA OpenCode maintenance agent #121;
• base branch: ci/hourly-opencode-nvidia-nim;
• exact live predecessor/base tip: 75265fae9fcf5faac6f8b9cab63247ac86f23dd4;
• replacement branch: repair/durable-job-lease-worker-75265fa;
• exact current head: fda8ded9b07a6b0aebb3df8ef3009063655ebfc8;
• current synthetic merge revision: 626e5fef633c684ba28af0a6e6e62c7038945bfe;
• GitHub mergeability: true;
• Draft: true.

A fresh literal comparison from exact predecessor 75265fae9fcf5faac6f8b9cab63247ac86f23dd4 to this head is ahead: 32 commits ahead, zero behind, with the predecessor itself as the exact merge base. No force push, -X ours, -X theirs, stale synthetic merge, or rewritten fail-first evidence was used. Checks, reviews, approvals, comments, and base snapshots from #138 or predecessor heads do not transfer.

Stack-divergence RCA

PR #121 advanced after the former durable-worker branch was cut. The obsolete worker branch therefore stopped descending from the exact live predecessor. Preserving stack integrity required a replacement branch from the new exact predecessor rather than destructive history rewriting. #138 is closed as superseded and its branch remains historical evidence.

PR #129 still targets the obsolete feat/durable-job-lease-worker branch and is diverged from that branch's live tip. #130, #131, #133, and #135 remain later locally intact but globally ineligible stack items. Do not deepen that chain until this earliest replacement boundary is eligible; old downstream checks, reviews, approvals, and base snapshots will not transfer when repaired.

RCA → options → feasibility → execution → proof

RED 1 — missing production worker

Exact head 5a3f3dab938163c00592507a2174a268fff3f89e reached production compilation successfully and then failed test compilation because EtlJobWorkerTest referenced the still-absent production EtlJobWorker at six compile sites. CI 31290377356 checked out and verified that literal head on Ubuntu, macOS, and Windows. The failure was therefore at the intended production boundary, not an import, fixture, dependency, runner, or stale-checkout defect.

Material remediation options were evaluated against the live stack and available authority:

• execute_now — add the bounded worker implementation: the former durable-worker branch already contained a worker matching the current collaborators and tests; exact old tip 05f0ec8ea3cdc715747966812462516d7b85c94e had previously completed literal-head CI 31177538550, Dependency Review 31177538330, and SBOM 31177538585. Reapplying that production boundary to the replacement ancestry required no new credential, permission, external dependency, history rewrite, or test weakening.
• reject — remove or weaken the worker tests: this would erase the intended contract rather than fix the root cause.
• reject — move the replacement to the obsolete branch or a stale synthetic merge: this would break exact predecessor ancestry and invalidate evidence.
• reject — invent a broader worker architecture: the tested implementation already addressed the demonstrated gap; expanding scope would increase blast radius without improving root-cause fit.

Commit 3a70fdcc129e318d4cd6094c164e63994e585bde added the bounded EtlJobWorker production class. Exact-head CI 31290656673 then succeeded on Ubuntu, macOS, and Windows; Dependency Review 31290656763, CycloneDX SBOM 31290656685, and CodeRabbit exact-head status also succeeded.

RED 2 — authoritative documentation contradicted production

After the worker became source-local GREEN, acquisition-readiness inspection found that docs/etl/durable-job-intake.md still said the system “does not execute jobs yet,” and CHANGELOG.md still described worker execution and terminal payload clearing as unimplemented. Existing documentation tests encoded those stale statements, so a green build did not prove documentation truthfulness.

Commit 38f4af8cf4c48c7d814a8214419db27e24add992 changed only the documentation contract tests first. Exact-head CI 31290922060 reached the intended boundary and macOS ran 369 tests with exactly two failures, both newly added assertions:

• runbookDocumentsAcceptedSemanticsOwnershipAndActiveWorkerBoundary;
• changelogRecordsLeaseFencedDurableWorkerExecution.

That fail-first result disproved the stale documentation without touching production behavior.

GREEN 2 — truthful runbook and changelog

• bdc46c6f27a8e2f1b3600366a5bb0f68edb2641c updates the durable-job runbook to document the active opt-in worker, FOR UPDATE SKIP LOCKED claim arbitration, exact lease fencing, transactional success, terminal payload clearing, bounded retries, stable failure codes, finite telemetry, disabled-by-default intake/worker controls, rollback behavior, and the remaining later-stack boundary. It records RFC 9110, RFC 9457, RFC 9651, the expired Idempotency-Key draft only as work-in-progress evidence, and PostgreSQL 18 SELECT documentation in the references.
• exact current head fda8ded9b07a6b0aebb3df8ef3009063655ebfc8 updates CHANGELOG.md to remove the obsolete “worker not implemented” claim and record the lease-fenced worker behavior.

Exact-current-head proof

For literal head fda8ded9b07a6b0aebb3df8ef3009063655ebfc8 over unchanged exact predecessor 75265fae9fcf5faac6f8b9cab63247ac86f23dd4:

• CI 31291130025: success;
• Ubuntu job 93188386197: exact-source checkout, exact-SHA verification, and Unix tests succeeded;
• macOS job 93188386213: exact-source checkout and exact-SHA verification succeeded; all 369 etl-service tests passed with zero failures/errors/skips and all configured JaCoCo coverage checks were met; cdc-service ran 106 tests successfully and the gateway suite ran 3 tests successfully;
• Windows job 93188386190: exact-source checkout, exact-SHA verification, and Windows tests succeeded;
• conditional test_self_hosted: skipped and not counted as positive evidence;
• Dependency Review 31291130034: success;
• CycloneDX SBOM 31291130012: success;
• CodeRabbit exact-head commit status: success;
• unresolved review threads: zero;
• formal reviews: zero;
• comparison to exact predecessor: ahead 32, behind 0;
• GitHub mergeability: true.

SAST Semgrep and Security Scan are absent for this non-default stacked base and therefore remain not passing rather than being inferred from predecessor or synthetic evidence. No qualifying independent non-author formal APPROVED review exists on this exact head.

Implemented worker boundary

The replacement slice now provides:

• database-owned single-job claim with PostgreSQL FOR UPDATE SKIP LOCKED;
• bounded lease duration and maximum attempts;
• exact lease owner/token/expiry fencing for retry, success, and terminal failure;
• transaction-aware ETL execution and durable idempotency integration;
• worker activation disabled by default with bounded scheduling configuration;
• at-most-one claimed job per poll;
• transient retry only while attempts remain;
• stable non-sensitive failure codes for deterministic request, integrity, target, target-unavailable, and unexpected runtime failures;
• stale-lease outcomes that cannot overwrite a newer worker's state;
• finite-cardinality idle, succeeded, retried, failed, and stale metrics without payloads, principals, keys, job IDs, lease IDs, SQL, exception classes, or exception messages;
• terminal payload clearing enforced by lifecycle persistence constraints;
• descriptive multi-word snake_case database objects and migration/rollback guidance;
• beginner-readable public Javadocs and deterministic worker, property, repository, transaction, migration, concurrency, failure, and documentation-alignment tests.

Merge and downstream repair policy

Keep this PR Draft. The replacement is stack-valid and exact-head source-local GREEN, but it is not merge-eligible while #121 remains unmerged and while required direct-base security/review evidence is absent. Do not request or synthesize approval, reinterpret absent SAST/Security as passing, or deepen #129 and later stack items.

After #121 becomes merge-eligible and integrates, re-anchor or retarget this replacement against the exact protected predecessor and regenerate every affected exact-head/base check, security result, review, and approval. Only after this predecessor is eligible may the pagination, polling, conditional status, cancellation, and replay chain be repaired in dependency order.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6657e0a4-4683-41cf-b6ee-1aa17528e58d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}