fix(runtime): make local operation state transactional and durable

[doublewhy]

Plain-language summary

• Context: OpenRAE keeps local control-plane snapshots, operation status, idempotency claims, and audit history so interrupted runs can be inspected and retried safely.
• Problem: The previous JSON store could lose concurrent writes, and a crash could leave backend effects, the snapshot, and operation status disagreeing. A second local process could also act from stale cached state.
• Fix: Store local state in a transactional SQLite WAL database, commit terminal snapshots and records together, recover interrupted operations without replaying unknown backend effects, and admit exactly one local runtime owner at a time.

Issue Tracking

Closes #1092

Part of #8.

What changes

• replaces whole-file JSON writes with indexed SQLite transactions and unique idempotency claims;
• requires WAL and durable synchronization, validates SQLite files without retaining application-owned database descriptors, and synchronizes legacy backups before admission;
• migrates legacy JSON under an exclusive lock, verifies the imported state, preserves the original files, and retains a timestamped backup;
• persists the operation claim before backend execution, atomically commits the resulting snapshot and terminal record, and marks orphaned non-terminal records failed on restart without replaying the backend;
• holds a process-bound lease for the lifetime of the runtime and drains active calls before release;
• preserves the exact legacy custom-store protocol through one explicitly non-crash-atomic, deprecated compatibility seam;
• publishes participant episode closure records in runtime-snapshot-v1 so the public contract matches the exhaustive durable snapshot codec.

Current-base integration

This branch merges dev at 5d2f738fb137760480a3ef9b2eae023f5df1c65b through merge commit 2b19c4ac0ce387a6b805ea5cdbe06ae782eebe8f. The durability changes remain limited to the local control-plane store; inherited HTTP admission, offload, scheduler, and realization behavior comes from the current base.

Scope boundary

This is a single-host, single-process local reference store. It does not claim distributed consensus, multi-worker cache coherence, durable job scheduling, backend-effect exactly-once guarantees, or request offload.

Verification

• RAES_REQUIREMENT_UID=API-404 implementations/python/.venv/bin/python tools/verify_all.py: passed — 7,449 passed, 1 skipped, and 4 xfailed;
• focused control-plane API tests: 54 passed;
• repository policy, schema publication, generated-schema drift, formal evidence replay, and contract checks: passed.

[doublewhy]

Fresh Sonar cleanup pushed in 8e95170d.

• Replaced the over-broad BaseException catch with Exception, so process interrupts are not swallowed while ordinary reconciliation failures still poison the runtime.
• Centralized the repeated database-file label without changing path validation.
• Verification: 115 crash-consistency tests passed; both path modules reached 99% combined focused coverage (all changed lines covered and control_plane_store_paths.py at 100% line/branch); Ruff, repository policy, and diff hygiene passed.

The plain-language summary and Closes #1092 mapping are unchanged.

[doublewhy]

Fresh Linux CI exposed and this update fixes one final WAL lifecycle race.

SQLite may unlink an ephemeral -wal/-shm path between pathname lookup and metadata return, yielding st_nlink == 0. The prior hard-link check rejected that benign disappearance and caused one of four concurrent writers to exit. Commit a6dcee0 now treats only zero-link sidecars as disappeared, still rejects multiple links, and keeps the main database pinned to exactly one link.

Verification: 115 crash-consistency tests pass; the affected path module is 100% line/branch covered; the original four-process write test passed 10 consecutive runs; Ruff, repository policy, and diff hygiene pass. Fresh CI has restarted on this exact SHA.

[doublewhy]

Fresh Sonar analyzed the WAL-race head and reported one maintainability issue: the path-admission helper crossed the configured complexity threshold by one. Commit 5981ebe extracts link-count admission into a focused helper without changing behavior or suppressing the rule.

The full 115-test crash-consistency suite still passes, the path module remains 100% line/branch covered, and Ruff plus repository policy pass. Fresh CI/Sonar has restarted on this exact head.

[Brad-Edwards]

@doublewhy Thank you for this, this raised some important architectural issues that haven't been grappled with. Issue moved to milestone 68 (Runtime Control-Plane) and is blocked on #1151

[Brad-Edwards]

Review result: changes required before merge.

• Align the implementation with the authoritative ADR-104 and CP-1/CP-2/CP-4/CP-5 provider contracts referenced by issue fix(runtime): make local operation state transactional and durable #1092. Those contracts are not currently present on dev, so they must be established before this implementation can claim conformance.
• Merge current dev and resolve the conflicts in runtime-architecture.md, API-404 traceability, control_plane.py, control_plane_store.py, and test_runtime_control_plane_api.py without dropping current authentication, bounded-offload, realization, or thread-safety behavior.
• Acquire local runtime ownership before database creation, schema inspection, migration, codec loading, reconciliation, or state reads. Enforce ownership on mutating store paths.
• Persist immutable target/run identity and reject reopening a store for a different identity.
• Add snapshot-revision compare-and-swap to terminal and transition commits so stale writers cannot replace current state.
• Commit the terminal snapshot, operation record, caller principal, and audit event as one atomic write set.
• Make snapshot and operation codecs closed and lossless: reject missing, unknown, malformed, or mismatched fields. The current completeness test exercises only the reconstructed field map and does not prove decoder rejection.
• Validate legacy inputs as descriptor-bound, no-follow regular files; reject symlinks, special files, unsafe ownership/link state, and replacement races. Verify backup bytes/digests as well as imported row counts.
• Add the required deprecation-ledger record for the custom-store version-4 removal commitment, or remove that removal claim.
• Re-run current-base CI, CodeQL/security checks, schema publication/drift checks, policy gates, and crash/concurrency fault-injection coverage after integration.

SQLite WAL, atomic idempotency claims, conservative interrupted-operation handling, and a profiled single-host durable provider remain a reasonable implementation direction.

[Brad-Edwards]

[core] Reloaded non-terminal operations are never sealed after an uncertain commit

If a backend effect completes but the terminal store call raises before committing, the durable claim can remain ACCEPTED or RUNNING. _resynchronize_after_store_error reloads that record and leaves the runtime healthy without applying the interrupted-operation reconciliation policy. A same-key retry then returns the existing non-terminal receipt, and polling cannot progress until the process happens to restart. Route the reloaded records through reconcile_interrupted_operations before publishing them; if reconciliation itself fails, poison the runtime as for a failed reload.

[Brad-Edwards]

gc_codex_review cycle 1 of 3 complete for PR #1136. Posted by the MCP server to enforce the hard-cap-3 contract (issues #794, #804). Do not edit or delete — used by the next gc_codex_review invocation to count cycles.