feat(runtime): add atomic workflow scheduler state-store and recovery boundary

[seonghobae]

Problem

The pure selectRunnableWorkflowTasks selector returns scheduling candidates only. Production execution authority requires one Noema-owned durable state/repository boundary binding an admitted execution/plan revision to atomic task claims, checkpoint CAS, effect-aware recovery, cancellation, blocked-descendant handling and bounded provenance.

Current implementation state — 2026-09-07 KST

Protected Noema is main@0dec8d84b1e4744e7a9c6a77e2e2631a183ee2ab after normal integration of trust prerequisite #554. Protected central .github/main remains GitHub-verified c9052e607e5f3cc76e73207e7786b21500721b79.

Active implementation owner is PR #542 exact 4616b5e93e19d51973aea330aa4124b51725b795, open / Draft. Its ancestry already contains ordinary/non-force two-parent restack 7f743f4ee8d81c8d52511ef421d8e40a32edf2a8, combining predecessor #542 2bb6076d911a526570176294c99fd86421c152eb with protected main@0dec8d84.... The branch preserves durable workflow state/execution-plan authority plus protected ALLOWED_WORKFLOW_SHA = c9052e607e5f3cc76e73207e7786b21500721b79; provider/model routing, quarantine/security/outbound policy and foreign domain truth remain outside this boundary.

Hosted RED → minimum causal repair

Post-restack exact 7f743f4... CI 34063013448 checked out the exact head, resolved live base to 0dec8d84..., passed lockfile control, install and typecheck, then reported 577 passing test files / one failing test and 4,079 passing tests / one failing test. The only failure was the retained-provenance hostile fixture: it replaced receipt index 0 with null, simultaneously destroying the native sequence-one initialized causal root and the receipt record shape. Production correctly rejected the earlier root invariant, while the test demanded the later malformed-receipt diagnostic.

Commit 4616b5e93e19d51973aea330aa4124b51725b795 is the minimum causal repair. The fixture now preserves the valid sequence-one initialized receipt, advances the synthetic ledger to sequence 2, and appends null as the second receipt. The test therefore isolates the non-record-receipt invariant without changing production validation, synthetic-data scope or the repository's 100% coverage gate.

Fresh current-head evidence for 4616b5e...:

• ci 34064157863: terminal success.
• reviewer-ci 34064157866: terminal success.
• required Security Scan 34064157896: terminal success.
• patch-validator-image 34064157873: in progress/non-passing.

Fresh review-thread authority contains no unresolved finding. No predecessor result transfers as merge authority.

Required production semantics

1. Atomic workflow snapshot/task claim under exact execution, plan, dependency, cancellation and claim preconditions.
2. Checkpoint compare-and-swap; replay is explicit and does not manufacture a new transition.
3. Effect-specific recovery; side effects are never silently retried from transport failure.
4. Explicit blocked descendants and deterministic/versioned scheduling policy.
5. One execution-scoped durable authority and one admitted plan identity; no public/cross-service command surface.
6. Bounded, payload-minimized provenance for claim/effect/completion/checkpoint/cancellation/recovery. Native sequence-one provenance starts at initialized; supported pre-ledger state may begin observable provenance with its first valid task_claimed receipt rather than inventing older history.
7. Owned production branches remain executable under the repository's 100% statement/branch/line gate; hostile corruption rejection paths are not exempted as unreachable.

Completion evidence

• Production Durable Object composition exists behind Noema-owned ports/repositories.
• Execution-scoped plan authority and legacy-state conflict handling exist.
• Atomic claim, checkpoint replay/CAS, cancellation, recovery and provenance contracts exist with hostile/concurrency regressions.
• Retained ledger cardinality and explicit-empty-ledger corruptions fail closed.
• Forged sequence-one checkpoint_committed native root fails closed.
• Hosted compatibility/coverage/stale-base REDs were classified and repaired without weakening production controls.
• fix(trust): roll audited central workflow source to c9052e6 #554 integrated normally first at protected main@0dec8d84....
• feat(workflow): add atomic durable task claim and checkpoint CAS #542 was ordinary/non-force restacked without losing NOEMA_WORKFLOW_STATE or current trust-source authority.
• Post-restack fixture ambiguity was isolated and repaired at 4616b5e...; current CI, reviewer-ci and required Security Scan are terminal success.
• Current unchanged exact head receives terminal-success patch-validator-image plus fresh live-base evidence.
• Normal protected integration preserves every valid feat(workflow): add atomic durable task claim and checkpoint CAS #542 delta; no simple close or predecessor-evidence transfer.
• Applicable protected-runtime/release evidence is collected before deployed atomicity or immutable release is claimed.

Documentation / architecture

Keep PRD/TRD/ARCHITECTURE/Context Map/Ubiquitous Language/ADR/UML/TEST_STRATEGY/OPERABILITY/recovery/TRACEABILITY/CHANGELOG aligned with the executable boundary. The sole cross-lane docs/product-technical-gap-baseline.md writer remains #547. Source composition and candidate CI are not Cloudflare production/runtime proof until protected operational evidence exists.

Related: #528, #545, PR #542.

[seonghobae]

Current #542 repair note — exact branch head 16c896eea3c8fc7112637eb1a4c0da089218c5e4 now closes one cancellation/recovery authority defect found during fresh source review. Test-first e83d7f09afdec4a9e16b66a98db83f4a6c522dcb adds the case where an idempotent task has durably crossed effect start before execution cancellation. Previous production recovery would clear the active claim and manufacture terminal cancelled, even though the first external invocation may already have completed. Production e2e49d1c506ee29f46d857c6624ffd832bb78782 now fails closed for cancellation + started/legacy-unknown idempotent evidence and retains the exact running claim for explicit outcome/reconciliation; ADR-0013 is code-current at 16c896ee.... No exact hosted RED was observed before the causal repair because the test-first head did not acquire a runner; current exact-head CI/reviewer/image are queued and required Security had not materialized, so this remains Draft/non-passing and must be revalidated after #528 integrates.

[seonghobae]

2026-09-04 KST fresh #542 application-boundary review found another issue-owned fail-open path in the required “effect start before side effect” semantics. executeNextWorkflowTask awaited statePort.markEffectStarted(...) but discarded the returned snapshot, so a state adapter that resolved without actually proving the exact claim crossed the durable effect boundary could still be followed by effectPort.execute(...).

TDD/source lineage on feat/workflow-state-store-atomic-claim:

• RED 0c0c6811e7b0548e4223867023b0540f9c31696c: a real repository claim is created, while the injected state port's markEffectStarted only returns the current durable snapshot (effectStarted=false). The runner is required to reject and never invoke the effect.
• production 637da325d7ffa5eeb5c376d2522c81b473d3642e: the runner now requires the returned snapshot to prove the same execution/plan/task/claim/attempt remains running with effectStarted=true before effect invocation; otherwise WorkflowTaskEffectAuthorityError fails closed.

Current PR #542 head is 637da325d7ffa5eeb5c376d2522c81b473d3642e on protected main@bbee33270b496255d785c766fc009a5f9162a695. Hosted GREEN is not claimed: current ci 33815852157 job 100847724463 is still queued, steps=[], runner_id=null; reviewer-ci 33815851966, required Security Scan 33815852018, and patch-validator-image 33815852061 are also queued. The central runner-control-plane owner .github#712 has received this exact-head canary. Issue #541 therefore remains open; no predecessor evidence transfers and the branch stays Draft.

[seonghobae]

Fresh executable application-boundary finding on #542: RED 891fa741083e5dcfe5a88b8b6d94db56705a2b10 adds a state-port fixture whose effect executes successfully but whose completeTask(...) returns only the still-running effect-start snapshot. The predecessor runner accepted that stale snapshot as a successful WorkflowTaskRunResult, so application composition could report completion without durable proof that the observed outcome became terminal authority. Production 3685ce56b09f50d4d653570d596b29496cfacc55 now requires the returned completion snapshot to match the exact execution/plan/task attempt, equal the observed terminal outcome, clear the active claim, retain effect-start evidence, and advance the transition sequence beyond the verified effect-start snapshot. Mismatch fails closed with WorkflowTaskTerminalAuthorityError; it does not fabricate retry/compensation or foreign authority. Hosted GREEN remains pending on the new exact head.

[seonghobae]

#542의 application-service claim authority를 다시 검토해 새 fail-open 경로를 확인하고 source-level TDD로 수리했습니다.

executeNextWorkflowTask(...)가 WorkflowTaskExecutionStatePort.claimNextRunnableTask(...) 반환값을 admitted plan/caller request에 다시 결합하지 않은 채 effect-start 단계로 넘기고 있었습니다. 따라서 잘못된 adapter가 durable repository에서는 실제 side_effecting task를 claim한 뒤, 호출자에게는 같은 identity에 effect: "pure"를 붙인 claim을 반환할 수 있었습니다. 기존 effect-start/terminal snapshots에는 effect classification이 반복되지 않으므로 후속 snapshot checks만으로는 이 substitution을 검출할 수 있고, effect port가 잘못된 recovery semantics의 claim을 받을 수 있었습니다.

• RED 4f4625dfee0a293425da0ae182f6dd28c4783920: real repository에 side_effecting claim을 남긴 뒤 injected state port가 pure로 바꾼 claim을 반환하는 회귀를 추가했습니다. predecessor runner는 이를 effect-start/completion까지 진행할 수 있습니다.
• production da225d15fe8681f7f36a5490a0de2d88c9969f27: WorkflowTaskClaimAuthorityError와 claim/plan binding 검증을 추가했습니다. effect-start 전에 executionId/planId/requested claimId/positive safe-integer attempt/admitted task identity/exact admitted effect classification이 모두 일치해야 합니다.

이 변경은 Noema의 Workflow / Task Execution + State / Checkpoint application boundary에만 있습니다. provider/model routing, foreign domain truth, quarantine/security verdict, outbound authority, retry/compensation authority는 가져오지 않았습니다.

현재 exact #542 head는 da225d15fe8681f7f36a5490a0de2d88c9969f27이고 Security Scan 33824089404, ci 33824089340, patch-validator-image 33824089343, reviewer-ci 33824089400이 모두 queued/non-passing입니다. 따라서 source repair만 기록하고 exact-head GREEN/merge-ready로 승격하지 않습니다.

[seonghobae]

2026-09-04 KST fresh #542 application-service claim review found a second bounded-attempt fail-open at the state-port seam. The durable repository and private command adapter already reject attempts above MAX_AUTOMATIC_RECOVERY_ATTEMPTS, but executeNextWorkflowTask(...) revalidated only that a returned claim attempt was a positive safe integer. A faulty/incompatible state adapter could therefore return attempt 4+ for an otherwise valid admitted task and reach effect-start handling instead of being rejected as impossible Noema execution authority.

TDD/source lineage on feat/workflow-state-store-atomic-claim:

• RED c744f9dde9ffd30df1b99ce332b6440be6184905: creates a real retained claim, substitutes only attempt = MAX_AUTOMATIC_RECOVERY_ATTEMPTS + 1 at the WorkflowTaskExecutionStatePort return boundary, and requires WorkflowTaskClaimAuthorityError before markEffectStarted, effect execution, or completion.
• production f3a02262ba7b15fc287fbfe61d9f61c6d3b744bb: requireClaimAuthority(...) now binds the returned attempt to the same canonical 1..MAX_AUTOMATIC_RECOVERY_ATTEMPTS interval already enforced by durable state and command admission. No retry/compensation/provider/security/foreign-domain authority was added.

Current exact #542 head is f3a02262ba7b15fc287fbfe61d9f61c6d3b744bb on protected main@bbee33270b496255d785c766fc009a5f9162a695. New exact-head runs materialized as ci 33827829254, reviewer-ci 33827829267, required Security Scan 33827829256, and patch-validator-image 33827829271; all remain non-terminal. CI job 100884174260 is still queued with steps=[], runner_id=null, runner_name=null. Hosted GREEN is therefore not claimed and #541 remains open.

[seonghobae]

#542 advanced with another application-boundary authority repair. RED 3f42df7650f420c2adb8ae42555b7b814868417f proves that a storage-independent WorkflowTaskExecutionStatePort could echo a caller-supplied non-canonical claim identity and the predecessor runner would proceed to effect-start persistence. Production da3c53c97b23fab7706a475e09c23806aa74d147 now rejects claim IDs outside the same 1–128 printable non-whitespace ASCII contract before any effect-start/effect/completion call. Exact-head hosted evidence is still non-terminal (ci 33831824296, reviewer-ci 33831824261, Security Scan 33831824262, patch-validator-image 33831824266 all queued), so this is source-level RED→repair evidence only and does not complete #541 acceptance.

[seonghobae]

Fresh #542 application-boundary repair on exact head eca1e634803dc85287077b0b14351800fed2537b:

• RED/source regression fb54babc3072ea7ed59c17389cb505dc03054b31 changes the malformed caller-claim test to require rejection before WorkflowTaskExecutionStatePort.claimNextRunnableTask(...) is invoked. The predecessor runner only validated the state adapter's returned claim, so a non-canonical caller claim ID could cross the application→state authority boundary before failure.
• Production eca1e634803dc85287077b0b14351800fed2537b admits caller claim IDs against the canonical 1–128 printable non-whitespace ASCII contract before the state-port call, then independently revalidates the returned claim/effect/attempt authority as before.
• No retry, compensation, provider routing, foreign domain truth, quarantine/security verdict, or outbound authority was added.

This is source-level causal repair only until the unchanged exact head executes. Current ci 33835157976 / job 100906049938 is still queued with steps=[], runner_id=null; reviewer-ci, required Security Scan, and patch-validator-image are also non-terminal. Do not transfer predecessor GREEN.

[seonghobae]

Fresh review finding on #542 exact 9236775bad5476a70601c3dd0331211d42eaed12: the four hosted lanes are GREEN, but merge authority is now blocked by a newly verified durable-provenance integrity gap. validateTransitionLedger permits a positive transitionSequence with fewer than min(sequence, MAX_TRANSITION_RECEIPTS) retained receipts (including an empty vector), although every production appendTransition preserves exactly that bounded suffix. Corrupted Durable Object state can therefore erase retained causal receipts and still pass readState, contrary to ADR-0013's observable bounded-provenance invariant. Existing hostile tests cover partial ledger presence, sequence below receipt count, oversized vectors, malformed receipt fields, and the deliberate pre-ledger case where both fields are absent; they do not cover shortened receipt retention with a positive sequence. Required repair: hostile RED for shortened/empty retained suffix, then fail closed unless receipts.length === Math.min(sequence, MAX_TRANSITION_RECEIPTS), while preserving the both-fields-absent legacy compatibility case. The existing GREEN generation is no longer merge-authoritative even before the #554 restack.

[seonghobae]

Repair landed on the same #542 branch. Test-first 8e7e0bbf97ec3508445f1d522c923b5074d5f7db adds the hostile sequence-one/zero-retained-receipt RED. Production 35f80f2eb4a0e861eaa833ef2bb474c012deafbe adds only the five-line causal invariant requiring transitionReceipts.length === Math.min(transitionSequence, MAX_TRANSITION_RECEIPTS) whenever ledger fields are present. The explicit pre-ledger compatibility case still requires both fields to be absent. Compare from prior exact 9236775... is 2 commits / 2 paths: the new regression (+67) and the production validator (+5), no other semantic delta. Fresh exact-head CI 34038387937, reviewer 34038387942, Security 34038387882, image 34038387881 are newly queued; prior GREEN does not transfer. #542 remains Draft and still requires post-#554 non-force semantic restack/revalidation.

[seonghobae]

Follow-up hostile edge is now repaired too. 8afef5416a73bc2662f2e3fe10412dcc0798daeb proves an explicitly present transitionSequence=0 / empty receipt ledger must not be accepted as the legacy pre-ledger format; production never persists that shape because initialized is appended before the first put. 55194e1d75f1ca31e87a34dc78d1ef1216006825 adds the three-line fail-closed check while leaving the true compatibility path (both ledger fields absent) intact. Current exact #542 head is therefore 55194e1...; fresh CI 34038628405, reviewer 34038628417, Security 34038628437, image 34038628409 are non-terminal. Previous GREEN generations remain non-authoritative.

[seonghobae]

2026-09-06 KST fresh authority update: central .github/main has advanced again to ee5567f7b15f0441a61ec2435415603b9518f1c6. The dd0b96... -> ee5567... compare is exactly one central commit and changes only CodeQL-dispatch / required-workflow queue contract tests; .github/workflows/noema-review.yml is unchanged, but GitHub OIDC job_workflow_sha source identity still moves with the protected central commit. PR #554 was therefore repaired test-first from predecessor 12e9efe07f72acdf9f1eb0f7ef0fec54d2ad633c: RED contract 8ae02909c21ba0f791c12ee97d3f7294a2468ce4 requires ee5567...; production exact head 8501d25507d4ecd3024205a92a9a3de4157300da changes only wrangler.toml.ALLOWED_WORKFLOW_SHA to that audited source and was advanced non-force. Its fresh CI/reviewer/Security/image generation is currently non-terminal, so the predecessor four-GREEN generation does not transfer.

#542 remains exact 1909f232dec32cf5d5de40d927af9c22366d2a85 with all current inline review threads resolved. Its four hosted exact-head lanes are also still non-terminal. #554 remains the foundation prerequisite: after it integrates, #542 must ordinary/non-force restack onto the new protected main while preserving the private NOEMA_WORKFLOW_STATE binding and the then-current central trust pin, then regenerate all exact-head evidence. No provider/model, quarantine/security, outbound or foreign-domain authority moves into Noema.