docs(runtime): define the runtime control-plane architecture

docs/decisions/adrs/README.md

@@ -252,3 +252,4 @@ adr-101-adversarial-participant-flow-control
 | [101](adr-101-adversarial-participant-flow-control.md) | Adversarial Participant Boundary Flow Control | accepted | 2026-07-30 |
 | [102](adr-102-mixed-cross-backend-participant-control.md) | Mixed Cross-Backend Participant Control | accepted | 2026-07-31 |
 | [103](adr-103-branch-aware-python-coverage-policy.md) | Branch-Aware Python Coverage Policy | accepted | 2026-08-13 |
+| [104](adr-104-runtime-control-plane-architecture.md) | Runtime Control-Plane Architecture | accepted | 2026-08-17 |

docs/decisions/adrs/adr-104-runtime-control-plane-architecture.md

@@ -0,0 +1,191 @@
+# ADR-104: Runtime Control-Plane Architecture
+
+## Status
+
+accepted
+
+## Date
+
+2026-08-17
+
+## Classification
+
+Classification: FM0
+
+Required artifacts: an evidence-backed current-state assessment, a profiled
+composition architecture with explicit state authority and failure semantics,
+a requirement and surface disposition, a dependency-ordered implementation
+program, and a structural test pinning the design set.
+
+Waivers: issue #1151 is design authority. It does not implement or select a
+storage engine by code change, publish new portable schemas, change runtime
+execution behavior, claim a distributed or highly available topology, or
+report any coordination, consistency, or recovery result as demonstrated;
+every profile guarantee named here becomes binding only when its
+implementation issue lands with tests.
+
+## Context
+
+`raes_runtime` exposes `RuntimeControlPlane` with an in-memory default store,
+an optional local JSON store, and a reference HTTP adapter. The repository
+has never decided what the control plane is: an in-process facade, an
+embedded component, a deployable service, or a family of implementations.
+Issue #1092 and PR #1136 exposed the gap by introducing SQLite durability,
+recovery behavior, single-process ownership, and store-compatibility rules
+inside what appeared to be a local storage change; both were deferred to this
+decision.
+
+The integration review of that work recorded two structural findings that
+bound this design. First, the generic operation path performs independently
+durable steps — claim `RUNNING`, invoke the backend, save the snapshot, save
+terminal status — with no lock of its own and the in-memory snapshot mutated
+before the durable write, so concurrent in-process submissions race and a
+process exit can strand an applied backend effect behind stale state, while
+an idempotent retry after restart returns the stale record without
+reconciling. Second, each `RuntimeControlPlane` permanently caches snapshot
+and operation maps while HTTP mutation serialization is application-local,
+so a shared database alone cannot make multiple application workers
+coherent.
+
+Expected RAES use spans hermetic tests, single-user local execution, embedded
+RAE/env-pack/ETV consumers, air-gapped deployments, and long-lived services.
+One deployment topology cannot serve all of these, and `RuntimeControlPlane`
+must not silently promise the strongest one.
+
+## Decision
+
+### 1. The control plane is a contract with profiled implementations
+
+`RuntimeControlPlane` names a portable contract — operation submission,
+idempotent receipts, snapshot access, participant transitions, audit — that
+conforming implementations provide under declared operating profiles. RAES
+ships reference implementations; it does not define one universal deployment
+model or a mandatory service.
+
+### 2. Operating profiles declare guarantees and nonclaims
+
+- **P0 ephemeral**: in-memory store, one process, no durability claims.
+  Intended for tests and single-scenario embedding. Loss of the process is
+  loss of the run.
+- **P1 local durable**: one owning process admitted by a store lease;
+  crash-consistent authoritative state through a transactional local store;
+  startup reconciliation of interrupted operations. Intended for local
+  tools, embedded consumers, and air-gapped single-host use.
+- **P2 served**: the reference HTTP adapter fronting a P1 core; many
+  clients, exactly one owning service process; mutations serialized by the
+  owner; reads carry the snapshot revision they observed.
+- **P3 coordinated**: multi-process or multi-host ownership. Explicitly a
+  nonclaim of this decision: the extension seams (lease provider, revision
+  compare-and-swap, coordination provider) are contracted now, and no P3
+  implementation or guarantee is asserted until a future ADR accepts one.
+
+### 3. State is classified by authority
+
+Authoritative state — runtime snapshots, operation records, idempotency
+claims, participant transition records — lives only in the profile's store
+and changes only through its transactions. Derived state — in-memory maps,
+indexes, receipt caches — must be rebuildable from authoritative state and
+must never answer a request in a way that contradicts it; any cache kept
+across a mutation boundary carries an explicit coherence rule. Audit events
+are append-only evidence with provenance and are never rewritten in place.
+External backend effects are not control-plane state: the control plane
+records intent and observed outcome, and where neither is available it
+records indeterminacy rather than inferring success or absence.
+
+### 4. Operations are durable work with atomic terminal commits
+
+An operation's lifecycle is recorded before its effects: the claim that an
+operation is running is durable before the backend is invoked, and the
+terminal transition commits the resulting snapshot, the terminal operation
+record, and the audit event in one store transaction, extending the pattern
+participant transitions already use to every operation. After process loss,
+startup reconciliation classifies each non-terminal operation as
+effect-absent (safe to fail closed), effect-applied (state is advanced from
+observation), or indeterminate — an explicit terminal outcome with a stable
+diagnostic that requires operator or embedder action. Interrupted work is
+never replayed automatically, and retained idempotency claims keep client
+retries from blindly re-invoking the backend.
+
+### 5. Concurrency control is ownership-first
+
+Exactly one writer owns a store at a time in P1 and P2, admitted through a
+store lease. Snapshot commits carry a revision and commit by
+compare-and-swap, so a stale writer fails closed instead of overwriting.
+Idempotency keys are unique claims in the authoritative store, not cache
+entries. Within a process, one operation lock serializes every mutation
+path — today the generic execution path is unlocked and the participant and
+manager paths hold two unordered locks; across processes, admission is the
+lease, not advisory locking.
+
+### 6. Boundaries against the rest of RAES
+
+SDL authoring, processor compile and plan behavior, `RuntimeTarget`
+attachment, backend contracts, and realization semantics are unchanged and
+outside control-plane authority. The control plane consumes compiled plans
+and backend interfaces; backends remain responsible for their own effect
+semantics. Embedding applications select a profile and own process
+lifecycle, configuration, and upgrade sequencing; persistence and
+coordination providers implement the store, lease, and clock contracts; the
+control plane owns operation bookkeeping, receipts, snapshots, transitions,
+and audit.
+
+### 7. Disposition of the incumbent surfaces
+
+`RuntimeControlPlane`, the store protocol, the in-memory store, and the
+reference HTTP adapter are retained and brought under this contract. The
+local JSON store is superseded by the P1 transactional store and retained
+only as a migration source. Issue #1092 is re-scoped into the P1
+implementation program; PR #1136 and its successor branches are its
+principal input — the atomic claim, single-transaction terminal commit,
+owner lease, WAL admission, path hardening, and migration work already
+implemented there converge with this decision and are re-landed as the P1
+store issues, while their recovery semantics are reworked to the
+reconciliation classification above instead of a blanket
+interrupted-to-failed conversion. The full disposition table, including
+every store module and test surface, lives in the design set's requirement
+disposition.
+
+## Alternatives Considered
+
+### One mandatory control-plane service
+
+Rejected: hermetic tests, embedded consumers, and air-gapped single-host use
+cannot depend on a deployable service, and a mandatory service would move
+RAES's default posture from library to infrastructure.
+
+### A single durable store with implicit multi-process sharing
+
+Rejected: the #1092 integration review showed that shared storage without
+revision CAS and lease admission leaves workers acting on stale caches;
+correctness would rest on deployment discipline the contract cannot see.
+
+### Desired-state reconciliation as the only operation model
+
+Rejected for now: RAES operations wrap backend calls whose effects are not
+uniformly observable or idempotent; a reconciler that assumes re-application
+is safe would replay indeterminate effects. The reconciliation classification
+in this decision leaves room for a future declarative profile without
+asserting one.
+
+### Extending the JSON store in place
+
+Rejected: whole-file read-modify-replace cannot express atomic multi-record
+commits, unique idempotency claims, or lease admission, and #1092 already
+demonstrated its lost-update and partial-state failures.
+
+## Consequences
+
+- Positive: embedders get explicit, testable guarantees per profile instead
+  of an implicit strongest-case promise; the #1092/#1136 work regains a
+  home with its architectural questions answered; indeterminate outcomes
+  become first-class instead of silent.
+- Negative: the operation lifecycle and store contracts change, which
+  touches the runtime execution path, the HTTP adapter, and every store
+  implementation; migration and compatibility work is unavoidable.
+- Risk: reconciliation classification depends on backend observability;
+  where backends cannot report effect state, operations will park as
+  indeterminate and require embedder policy, which is safe but may be
+  operationally noisy until backends improve observation surfaces.
+- The implementation program in the design set orders this work into
+  bounded issues in the Runtime Control-Plane milestone; no guarantee named
+  here is claimable before its issue lands.

docs/decisions/adrs/adr-index.yaml

@@ -509,3 +509,6 @@ adrs:
   - id: ADR-103
     path: docs/decisions/adrs/adr-103-branch-aware-python-coverage-policy.md
     pin: a44acbc2db1b5349ba316ba0d09bd9b46310db016cb87937b08bc6bc107755aa
+  - id: ADR-104
+    path: docs/decisions/adrs/adr-104-runtime-control-plane-architecture.md
+    pin: b27359f9d4bc21bbd3b13570202fc03a4107a0d0fa66b18937210fde18168c8e

docs/decisions/issue-1151-runtime-control-plane-architecture-preflight.md

@@ -0,0 +1,73 @@
+# Issue #1151 — Runtime Control-Plane Architecture Preflight
+
+Date: 2026-08-17
+
+Issue: #1151. Requirement: `API-404`.
+
+This note frames the architecture decision before the design lands. It is
+guidance only: it does not publish contracts, change runtime behavior, or
+select a storage engine by incidental code change. The binding record is
+ADR-104 together with the design set under
+`docs/research/runtime-control-plane/`.
+
+## Why the incumbent surfaces cannot be extended in place
+
+Issue #1092 and PR #1136 hardened the local JSON store into a transactional
+SQLite store and were deferred because the changes embed architectural
+choices — single-process ownership, recovery semantics, store compatibility
+rules — that the repository has never decided. The integration review of that
+work recorded two structural gaps that no store swap can close by itself:
+
+- The generic operation path performs independently durable steps (claim
+  `RUNNING`, invoke the backend, save the snapshot, save terminal status). A
+  process exit between steps leaves an applied backend effect with stale
+  state, and restart returns the stale record to an idempotent retry without
+  reconciling.
+- Every `RuntimeControlPlane` instance permanently caches snapshot and
+  operation maps, and HTTP mutation serialization is application-local. A
+  shared database therefore does not make two application workers coherent;
+  the supported topology must fail closed as one owning process until a
+  revision-CAS and lease design exists.
+
+## Decisive boundaries
+
+- The control plane is a contract with profiled implementations, not one
+  deployable service. Embedders select an operating profile; each profile
+  states its guarantees and nonclaims explicitly.
+- State is classified before it is stored: authoritative records (snapshots,
+  operation records, idempotency claims), derived caches (rebuildable,
+  never load-bearing), and append-only audit evidence. External backend
+  effects are never assumed from control-plane state; they are observed or
+  declared indeterminate.
+- An operation is durable work with a recorded lifecycle. Terminal effects
+  commit atomically (snapshot, terminal record, audit) and interrupted
+  operations surface as explicit indeterminate outcomes that require
+  reconciliation, never silent replay.
+- Concurrency control is ownership-first: one writer per store at profile
+  P1 (lease-admitted), optimistic revision checks on snapshot commits, and
+  unique idempotency claims in the authoritative store.
+
+## Non-goals for the design issue
+
+- No storage engine is implemented or selected by code change here; the
+  design constrains providers through the store contract and its required
+  admission checks.
+- No distributed or highly available topology is claimed. The design records
+  the extension seams (lease, revision CAS, coordination provider) that a
+  future profile would implement, and states the nonclaim plainly.
+- No change to SDL authoring, processor compile behavior, or backend
+  contracts. The control plane consumes those boundaries; it does not own
+  them.
+
+## Gotchas and anti-patterns
+
+- A durable claim without a returned admission result is not durable; store
+  providers must verify what the engine actually granted (journal mode,
+  sync level), following the WAL admission finding from issue #1092.
+- Idempotency receipts served from a permanent in-memory cache reintroduce
+  the stale-read hazard in every multi-worker topology; receipts must be
+  answered from the authoritative store or from a cache with an explicit
+  coherence rule.
+- Startup reconciliation must distinguish "the backend effect is known
+  absent", "known applied", and "indeterminate"; collapsing these into one
+  retryable failure state re-creates the replay hazard the audit found.

docs/requirements/API-404/requirement.md

@@ -22,6 +22,10 @@ Requirement inventory phase. Status audit deferred until the full canonical grap
 ## Traceability
 
 - IMPLEMENTS → GITHUB_ISSUE `8` (API-404: Secure, Durable, And Idempotent Control-Plane Semantics)
+- IMPLEMENTS → GITHUB_ISSUE `1151` (design(runtime): define the runtime control-plane architecture)
+- DOCUMENTS → ADR `docs/decisions/adrs/adr-104-runtime-control-plane-architecture.md` (ADR-104: Runtime Control-Plane Architecture)
+- DOCUMENTS → DOCUMENTATION `docs/research/runtime-control-plane/index.md` (Runtime control-plane architecture design set)
+- TESTS → TEST `implementations/python/tests/test_issue_1151_runtime_control_plane_design.py` (Structural acceptance gate for the design set)
 - IMPLEMENTS → GITHUB_ISSUE `1090` (Fail-closed bearer-token authentication and target binding)
 - IMPLEMENTS → GITHUB_ISSUE `1091` (Bounded pre-routing HTTP request admission)
 - DOCUMENTS → GITHUB_ISSUE `1093` (In-process HTTP offload and rejection-audit slice)