fix(runtime): make local operation state transactional and durable

[doublewhy]

Parent design

• design(runtime): define the runtime control-plane architecture #1151
• Work package: CP-6
• Original requirement parent: API-404: Secure, Durable, And Idempotent Control-Plane Semantics #8

Gap claim

The local control-plane JSON store performs unlocked whole-file read/modify/write operations. Concurrent writers lose records; replacement is not file-and-directory fsynced; idempotency lookup is linear. API-404 cannot claim durable or idempotent operation state.

Existing surface audit

The audit covers local store serialization, audit append, state transition, idempotency lookup, control-plane execution, repository contracts, and issue #8. An in-process lock would not protect multiple workers or crash consistency. ADR-104 now owns the lifecycle, concurrency, recovery, identity, and profile decisions that were previously implicit here.

Scope

Re-land the transactional local-store work from PR #1136 under the CP-1, CP-2, CP-4, and CP-5 contracts. Provide atomic claim and terminal-commit capabilities, verified WAL/durability admission, path and permission hardening, integrity checks, strict lossless snapshot/operation codecs, and a one-time legacy JSON import with durable backup. Recovery classification belongs to CP-3 and actor-scoped receipt semantics belong to CP-7.

Dependencies

• CP-1
• CP-2
• CP-4
• CP-5

Acceptance criteria

• The P1 provider supplies atomic claim, snapshot-revision compare-and-swap, terminal snapshot/record/audit commit, and single-owner lease capabilities.
• Startup acquires ownership before schema inspection, migration, codec loading, reconciliation, or state reads.
• The store pins immutable target/run identity and refuses mismatched reopen.
• Published/domain snapshot and operation state round-trip through one closed, lossless codec; malformed or unknown state fails closed.
• WAL/synchronization configuration is verified from the provider's granted result.
• Legacy import verifies counts and digests, fsyncs data and directory changes, retains a durable backup, and switches only after validation.
• No multi-process, high-availability, multitenancy, automatic-replay, or exactly-once backend-effect claim is introduced.

Verification

• Store-contract and crash-consistency tests, including process and commit-boundary fault injection.
• Concurrent-owner, revision-race, path/symlink/permission, corruption, and interrupted-migration tests.
• Full snapshot/operation round-trip tests, including participant episode closure records and malformed-carrier rejection.
• Profile P1 conformance through CP-9.

Traceability

• Requirement: API-404
• Decision: ADR-104
• Formal model: specs/formal/runtime-control-plane/README.md

[doublewhy]

Integration review addendum: recovery and topology gaps

Gap Claim

Review of the completed SQLite remediation found that the generic operation path still performs four independently durable steps: claim RUNNING, invoke the backend, save the snapshot, and save terminal status. A process exit between those boundaries can leave an applied backend effect with stale state, or a durable new snapshot whose operation remains RUNNING. Restart currently loads that record unchanged and an idempotent retry returns it without reconciling or replaying. API-404 therefore cannot yet claim restart-safe idempotency.

The same review found that each RuntimeControlPlane permanently caches snapshot and operation maps while HTTP mutation serialization is application-local. A shared SQLite file does not make multiple app workers coherent; two workers can apply from stale snapshots, and status reads can remain stale. The supported topology must fail closed as one process until a revision-CAS/lease design exists.

Existing Surface Audit

Checked RuntimeControlPlane.__init__, _idempotent_receipt, execute_operation, participant execution, ControlPlaneStore, LocalControlPlaneStore, participant transition CAS methods, API call serialization, operation schemas, API-404, and the issue #1092/#1093 decisions. Participant transition commits already provide the correct atomic snapshot/record pattern, but generic terminal commits and startup reconciliation are absent. No durable backend-effect ledger exists, so blind automatic replay is unsafe.

Lineage and Precedent

This extends the existing API-404 operation lifecycle and SQLite store; it does not add another queue or state family. SQLite transactions remain the single-host durable boundary. The existing FAILED terminal state is used with a stable recovery diagnostic to represent an interrupted, explicitly non-successful operation without changing the published enum/schema.

Literature and Practice

• SQLite transaction semantics: https://www.sqlite.org/lang_transaction.html
• SQLite atomic commit model: https://www.sqlite.org/atomiccommit.html
• Uvicorn process-worker deployment boundary: https://www.uvicorn.org/deployment/

Alternatives

1. Leave orphan RUNNING records and rely on operators: rejected because retries remain stuck and the status falsely implies live work.
2. Automatically replay every orphan: rejected because a backend side effect may already have happened.
3. Introduce a new public indeterminate enum immediately: rejected for this patch because it changes the published schema and every consumer.
4. Chosen: atomically commit snapshot plus terminal status, reconcile startup orphans to FAILED with a stable operation-interrupted diagnostic, preserve the idempotency claim so retries return that explicit non-success result, and enforce an exclusive single-process local-store lease.

Chosen Architecture

Add a store-level atomic terminal commit. On startup, reconcile every persisted RUNNING record before exposing the control plane. Never invoke the backend during reconciliation. Preserve the operation id and idempotency key, change only status to terminal failure, and include stable recovery evidence. Acquire a nonblocking process lease for the local database and reject a second live owner; release it on explicit close/context exit and finalization. This is process-topology enforcement, not cross-process CAS.

Documentation Defense

Amend the #1092/#1093 decisions and runtime architecture to state crash boundaries, orphan semantics, lease lifecycle, single-process scope, and why recovery cannot prove whether an external effect occurred. No public schema migration is required.

Verification Plan

Inject failures after claim, backend return, snapshot serialization/write, and terminal commit; restart at every boundary; verify no terminal snapshot/status split; verify retries never re-invoke the backend; verify recovered status is explicit non-success; verify one local-store process owner and lease release/reacquisition; retain all legacy migration, concurrency, participant transition, API, Ruff, policy, governance, and full verification gates.

[Brad-Edwards]

Deferred pending the runtime control-plane architecture work in #1151. The design proposed here and in PR #1136 will be reconsidered once #1151 establishes the applicable control-plane architectures and determines whether this design fits them.