From e717432473a1b3fbd977c837d5bd76c5494ab170 Mon Sep 17 00:00:00 2001 From: Yernat Yestekov <2068106+doublewhy@users.noreply.github.com> Date: Mon, 17 Aug 2026 17:55:00 -0700 Subject: [PATCH 1/2] docs(runtime): define the runtime control-plane architecture Issue #1151 owns the architectural design that issue #1092 and PR #1136 exposed the need for. This landing delivers the decision and design set: - ADR-104 records the decision: the control plane is a portable contract with profiled implementations (P0 ephemeral, P1 local durable, P2 served, P3 coordinated-as-nonclaim), state classified by authority, write-ahead operation claims with one atomic terminal commit, startup reconciliation that classifies interrupted work as effect-absent, effect-applied, or indeterminate, and ownership-first concurrency (store lease plus snapshot revision compare-and-swap). - docs/research/runtime-control-plane/ carries the evidence-backed current-state assessment (verified against dev at 701858d1), the composition architecture, the requirement and surface disposition (including issue #1092 and PR #1136), and the dependency-ordered implementation program (CP-1 through CP-12) for the Runtime Control-Plane milestone; implementation-program.json is the machine-readable authority and a structural test pins the set. - The issue-1151 preflight note frames the decisive boundaries; API-404 gains traceability to the ADR, design set, and structural test; the requirement registry maps API-404 into a runtime-control-plane phase; and the ADR index, acceptance-content pin, and identity-cutover digests are updated together. No runtime behavior changes; no storage engine is implemented or selected by code change. Work packages are filed as milestone issues once this design is accepted. Part of #1151. Requirement: API-404. Co-Authored-By: Claude Fable 5 --- docs/decisions/adrs/README.md | 1 + ...-104-runtime-control-plane-architecture.md | 185 ++++++++++++++++++ docs/decisions/adrs/adr-index.yaml | 3 + ...me-control-plane-architecture-preflight.md | 73 +++++++ docs/requirements/API-404/requirement.md | 4 + .../composition-architecture.md | 144 ++++++++++++++ .../current-state-assessment.md | 111 +++++++++++ .../implementation-program.json | 133 +++++++++++++ .../implementation-program.md | 142 ++++++++++++++ docs/research/runtime-control-plane/index.md | 28 +++ .../requirement-disposition.md | 47 +++++ ...issue_1151_runtime_control_plane_design.py | 103 ++++++++++ tools/policy/historical_identity_records.json | 4 +- tools/policy/requirement_order.yaml | 3 + 14 files changed, 979 insertions(+), 2 deletions(-) create mode 100644 docs/decisions/adrs/adr-104-runtime-control-plane-architecture.md create mode 100644 docs/decisions/issue-1151-runtime-control-plane-architecture-preflight.md create mode 100644 docs/research/runtime-control-plane/composition-architecture.md create mode 100644 docs/research/runtime-control-plane/current-state-assessment.md create mode 100644 docs/research/runtime-control-plane/implementation-program.json create mode 100644 docs/research/runtime-control-plane/implementation-program.md create mode 100644 docs/research/runtime-control-plane/index.md create mode 100644 docs/research/runtime-control-plane/requirement-disposition.md create mode 100644 implementations/python/tests/test_issue_1151_runtime_control_plane_design.py diff --git a/docs/decisions/adrs/README.md b/docs/decisions/adrs/README.md index 405abefb1..b9fc55b8b 100644 --- a/docs/decisions/adrs/README.md +++ b/docs/decisions/adrs/README.md @@ -252,3 +252,4 @@ adr-101-adversarial-participant-flow-control | [101](adr-101-adversarial-participant-flow-control.md) | Adversarial Participant Boundary Flow Control | accepted | 2026-07-30 | | [102](adr-102-mixed-cross-backend-participant-control.md) | Mixed Cross-Backend Participant Control | accepted | 2026-07-31 | | [103](adr-103-branch-aware-python-coverage-policy.md) | Branch-Aware Python Coverage Policy | accepted | 2026-08-13 | +| [104](adr-104-runtime-control-plane-architecture.md) | Runtime Control-Plane Architecture | accepted | 2026-08-17 | diff --git a/docs/decisions/adrs/adr-104-runtime-control-plane-architecture.md b/docs/decisions/adrs/adr-104-runtime-control-plane-architecture.md new file mode 100644 index 000000000..3ef2c1d55 --- /dev/null +++ b/docs/decisions/adrs/adr-104-runtime-control-plane-architecture.md @@ -0,0 +1,185 @@ +# ADR-104: Runtime Control-Plane Architecture + +## Status + +accepted + +## Date + +2026-08-17 + +## Classification + +Classification: FM0 + +Required artifacts: an evidence-backed current-state assessment, a profiled +composition architecture with explicit state authority and failure semantics, +a requirement and surface disposition, a dependency-ordered implementation +program, and a structural test pinning the design set. + +Waivers: issue #1151 is design authority. It does not implement or select a +storage engine by code change, publish new portable schemas, change runtime +execution behavior, claim a distributed or highly available topology, or +report any coordination, consistency, or recovery result as demonstrated; +every profile guarantee named here becomes binding only when its +implementation issue lands with tests. + +## Context + +`raes_runtime` exposes `RuntimeControlPlane` with an in-memory default store, +an optional local JSON store, and a reference HTTP adapter. The repository +has never decided what the control plane is: an in-process facade, an +embedded component, a deployable service, or a family of implementations. +Issue #1092 and PR #1136 exposed the gap by introducing SQLite durability, +recovery behavior, single-process ownership, and store-compatibility rules +inside what appeared to be a local storage change; both were deferred to this +decision. + +The integration review of that work recorded two structural findings that +bound this design. First, the generic operation path performs independently +durable steps — claim `RUNNING`, invoke the backend, save the snapshot, save +terminal status — so a process exit can strand an applied backend effect +behind stale state, and an idempotent retry after restart returns the stale +record without reconciling. Second, each `RuntimeControlPlane` permanently +caches snapshot and operation maps while HTTP mutation serialization is +application-local, so a shared database alone cannot make multiple +application workers coherent. + +Expected RAES use spans hermetic tests, single-user local execution, embedded +RAE/env-pack/ETV consumers, air-gapped deployments, and long-lived services. +One deployment topology cannot serve all of these, and `RuntimeControlPlane` +must not silently promise the strongest one. + +## Decision + +### 1. The control plane is a contract with profiled implementations + +`RuntimeControlPlane` names a portable contract — operation submission, +idempotent receipts, snapshot access, participant transitions, audit — that +conforming implementations provide under declared operating profiles. RAES +ships reference implementations; it does not define one universal deployment +model or a mandatory service. + +### 2. Operating profiles declare guarantees and nonclaims + +- **P0 ephemeral**: in-memory store, one process, no durability claims. + Intended for tests and single-scenario embedding. Loss of the process is + loss of the run. +- **P1 local durable**: one owning process admitted by a store lease; + crash-consistent authoritative state through a transactional local store; + startup reconciliation of interrupted operations. Intended for local + tools, embedded consumers, and air-gapped single-host use. +- **P2 served**: the reference HTTP adapter fronting a P1 core; many + clients, exactly one owning service process; mutations serialized by the + owner; reads carry the snapshot revision they observed. +- **P3 coordinated**: multi-process or multi-host ownership. Explicitly a + nonclaim of this decision: the extension seams (lease provider, revision + compare-and-swap, coordination provider) are contracted now, and no P3 + implementation or guarantee is asserted until a future ADR accepts one. + +### 3. State is classified by authority + +Authoritative state — runtime snapshots, operation records, idempotency +claims, participant transition records — lives only in the profile's store +and changes only through its transactions. Derived state — in-memory maps, +indexes, receipt caches — must be rebuildable from authoritative state and +must never answer a request in a way that contradicts it; any cache kept +across a mutation boundary carries an explicit coherence rule. Audit events +are append-only evidence with provenance and are never rewritten in place. +External backend effects are not control-plane state: the control plane +records intent and observed outcome, and where neither is available it +records indeterminacy rather than inferring success or absence. + +### 4. Operations are durable work with atomic terminal commits + +An operation's lifecycle is recorded before its effects: the claim that an +operation is running is durable before the backend is invoked, and the +terminal transition commits the resulting snapshot, the terminal operation +record, and the audit event in one store transaction, extending the pattern +participant transitions already use to every operation. After process loss, +startup reconciliation classifies each non-terminal operation as +effect-absent (safe to fail closed), effect-applied (state is advanced from +observation), or indeterminate — an explicit terminal outcome with a stable +diagnostic that requires operator or embedder action. Interrupted work is +never replayed automatically, and retained idempotency claims keep client +retries from blindly re-invoking the backend. + +### 5. Concurrency control is ownership-first + +Exactly one writer owns a store at a time in P1 and P2, admitted through a +store lease. Snapshot commits carry a revision and commit by +compare-and-swap, so a stale writer fails closed instead of overwriting. +Idempotency keys are unique claims in the authoritative store, not cache +entries. Within a process, existing lock discipline continues to serialize +mutation; across processes, admission is the lease, not advisory locking. + +### 6. Boundaries against the rest of RAES + +SDL authoring, processor compile and plan behavior, `RuntimeTarget` +attachment, backend contracts, and realization semantics are unchanged and +outside control-plane authority. The control plane consumes compiled plans +and backend interfaces; backends remain responsible for their own effect +semantics. Embedding applications select a profile and own process +lifecycle, configuration, and upgrade sequencing; persistence and +coordination providers implement the store, lease, and clock contracts; the +control plane owns operation bookkeeping, receipts, snapshots, transitions, +and audit. + +### 7. Disposition of the incumbent surfaces + +`RuntimeControlPlane`, the store protocol, the in-memory store, and the +reference HTTP adapter are retained and brought under this contract. The +local JSON store is superseded by the P1 transactional store and retained +only as a migration source. Issue #1092 is re-scoped into the P1 +implementation program; PR #1136 is its principal input — the SQLite WAL +admission, atomic participant commits, path hardening, and migration work +align with this decision and are re-landed as the P1 store issues, while its +recovery semantics are reworked to the reconciliation classification above +instead of a blanket interrupted-to-failed conversion. The full disposition +table, including every store module and test surface, lives in the design +set's requirement disposition. + +## Alternatives Considered + +### One mandatory control-plane service + +Rejected: hermetic tests, embedded consumers, and air-gapped single-host use +cannot depend on a deployable service, and a mandatory service would move +RAES's default posture from library to infrastructure. + +### A single durable store with implicit multi-process sharing + +Rejected: the #1092 integration review showed that shared storage without +revision CAS and lease admission leaves workers acting on stale caches; +correctness would rest on deployment discipline the contract cannot see. + +### Desired-state reconciliation as the only operation model + +Rejected for now: RAES operations wrap backend calls whose effects are not +uniformly observable or idempotent; a reconciler that assumes re-application +is safe would replay indeterminate effects. The reconciliation classification +in this decision leaves room for a future declarative profile without +asserting one. + +### Extending the JSON store in place + +Rejected: whole-file read-modify-replace cannot express atomic multi-record +commits, unique idempotency claims, or lease admission, and #1092 already +demonstrated its lost-update and partial-state failures. + +## Consequences + +- Positive: embedders get explicit, testable guarantees per profile instead + of an implicit strongest-case promise; the #1092/#1136 work regains a + home with its architectural questions answered; indeterminate outcomes + become first-class instead of silent. +- Negative: the operation lifecycle and store contracts change, which + touches the runtime execution path, the HTTP adapter, and every store + implementation; migration and compatibility work is unavoidable. +- Risk: reconciliation classification depends on backend observability; + where backends cannot report effect state, operations will park as + indeterminate and require embedder policy, which is safe but may be + operationally noisy until backends improve observation surfaces. +- The implementation program in the design set orders this work into + bounded issues in the Runtime Control-Plane milestone; no guarantee named + here is claimable before its issue lands. diff --git a/docs/decisions/adrs/adr-index.yaml b/docs/decisions/adrs/adr-index.yaml index b57042518..ca027a2af 100644 --- a/docs/decisions/adrs/adr-index.yaml +++ b/docs/decisions/adrs/adr-index.yaml @@ -509,3 +509,6 @@ adrs: - id: ADR-103 path: docs/decisions/adrs/adr-103-branch-aware-python-coverage-policy.md pin: a44acbc2db1b5349ba316ba0d09bd9b46310db016cb87937b08bc6bc107755aa + - id: ADR-104 + path: docs/decisions/adrs/adr-104-runtime-control-plane-architecture.md + pin: bf15139353bc39af301261cb7799fab26ea354cf06a2cb64a59c50a663258cea diff --git a/docs/decisions/issue-1151-runtime-control-plane-architecture-preflight.md b/docs/decisions/issue-1151-runtime-control-plane-architecture-preflight.md new file mode 100644 index 000000000..e6053af34 --- /dev/null +++ b/docs/decisions/issue-1151-runtime-control-plane-architecture-preflight.md @@ -0,0 +1,73 @@ +# Issue #1151 — Runtime Control-Plane Architecture Preflight + +Date: 2026-08-17 + +Issue: #1151. Requirement: `API-404`. + +This note frames the architecture decision before the design lands. It is +guidance only: it does not publish contracts, change runtime behavior, or +select a storage engine by incidental code change. The binding record is +ADR-104 together with the design set under +`docs/research/runtime-control-plane/`. + +## Why the incumbent surfaces cannot be extended in place + +Issue #1092 and PR #1136 hardened the local JSON store into a transactional +SQLite store and were deferred because the changes embed architectural +choices — single-process ownership, recovery semantics, store compatibility +rules — that the repository has never decided. The integration review of that +work recorded two structural gaps that no store swap can close by itself: + +- The generic operation path performs independently durable steps (claim + `RUNNING`, invoke the backend, save the snapshot, save terminal status). A + process exit between steps leaves an applied backend effect with stale + state, and restart returns the stale record to an idempotent retry without + reconciling. +- Every `RuntimeControlPlane` instance permanently caches snapshot and + operation maps, and HTTP mutation serialization is application-local. A + shared database therefore does not make two application workers coherent; + the supported topology must fail closed as one owning process until a + revision-CAS and lease design exists. + +## Decisive boundaries + +- The control plane is a contract with profiled implementations, not one + deployable service. Embedders select an operating profile; each profile + states its guarantees and nonclaims explicitly. +- State is classified before it is stored: authoritative records (snapshots, + operation records, idempotency claims), derived caches (rebuildable, + never load-bearing), and append-only audit evidence. External backend + effects are never assumed from control-plane state; they are observed or + declared indeterminate. +- An operation is durable work with a recorded lifecycle. Terminal effects + commit atomically (snapshot, terminal record, audit) and interrupted + operations surface as explicit indeterminate outcomes that require + reconciliation, never silent replay. +- Concurrency control is ownership-first: one writer per store at profile + P1 (lease-admitted), optimistic revision checks on snapshot commits, and + unique idempotency claims in the authoritative store. + +## Non-goals for the design issue + +- No storage engine is implemented or selected by code change here; the + design constrains providers through the store contract and its required + admission checks. +- No distributed or highly available topology is claimed. The design records + the extension seams (lease, revision CAS, coordination provider) that a + future profile would implement, and states the nonclaim plainly. +- No change to SDL authoring, processor compile behavior, or backend + contracts. The control plane consumes those boundaries; it does not own + them. + +## Gotchas and anti-patterns + +- A durable claim without a returned admission result is not durable; store + providers must verify what the engine actually granted (journal mode, + sync level), following the WAL admission finding from issue #1092. +- Idempotency receipts served from a permanent in-memory cache reintroduce + the stale-read hazard in every multi-worker topology; receipts must be + answered from the authoritative store or from a cache with an explicit + coherence rule. +- Startup reconciliation must distinguish "the backend effect is known + absent", "known applied", and "indeterminate"; collapsing these into one + retryable failure state re-creates the replay hazard the audit found. diff --git a/docs/requirements/API-404/requirement.md b/docs/requirements/API-404/requirement.md index 038affaba..42de4ebcc 100644 --- a/docs/requirements/API-404/requirement.md +++ b/docs/requirements/API-404/requirement.md @@ -22,6 +22,10 @@ Requirement inventory phase. Status audit deferred until the full canonical grap ## Traceability - IMPLEMENTS → GITHUB_ISSUE `8` (API-404: Secure, Durable, And Idempotent Control-Plane Semantics) +- IMPLEMENTS → GITHUB_ISSUE `1151` (design(runtime): define the runtime control-plane architecture) +- DOCUMENTS → ADR `docs/decisions/adrs/adr-104-runtime-control-plane-architecture.md` (ADR-104: Runtime Control-Plane Architecture) +- DOCUMENTS → DOCUMENTATION `docs/research/runtime-control-plane/index.md` (Runtime control-plane architecture design set) +- TESTS → TEST `implementations/python/tests/test_issue_1151_runtime_control_plane_design.py` (Structural acceptance gate for the design set) - IMPLEMENTS → GITHUB_ISSUE `1090` (Fail-closed bearer-token authentication and target binding) - IMPLEMENTS → GITHUB_ISSUE `1091` (Bounded pre-routing HTTP request admission) - DOCUMENTS → GITHUB_ISSUE `1093` (In-process HTTP offload and rejection-audit slice) diff --git a/docs/research/runtime-control-plane/composition-architecture.md b/docs/research/runtime-control-plane/composition-architecture.md new file mode 100644 index 000000000..6283fe492 --- /dev/null +++ b/docs/research/runtime-control-plane/composition-architecture.md @@ -0,0 +1,144 @@ +# Runtime Control-Plane Composition Architecture + +Date: 2026-08-17 + +Parent issue: [#1151](https://github.com/OpenRAE/rae/issues/1151) + +This document expands ADR-104 into component boundaries, ownership, data and +control flow, lifecycle, persistence and coordination interactions, security +boundaries, and failure and recovery behavior. Nothing here is claimable +before its implementation work package (see the +[implementation program](implementation-program.md)) lands with tests. + +## 1. Position in the RAES stack + +SDL authoring and the processor produce compiled plans; backends realize +effects; `RuntimeTarget` binds a backend to a runtime. The control plane +sits between an embedding application and those boundaries: it accepts +operation submissions against a target, records their lifecycle, holds the +authoritative runtime snapshot, mediates participant transitions, and emits +audit evidence. It owns bookkeeping and admission — never backend effect +semantics, plan compilation, or SDL meaning. + +## 2. The contract and its implementations + +`RuntimeControlPlane` is the portable contract: submit an operation, obtain +an idempotent receipt, read snapshots, drive participant transitions, and +observe audit events. Conforming implementations differ only in the +guarantees their operating profile declares. The contract carries a +capability surface (work package CP-10) so an embedder can interrogate the +active profile's guarantees and nonclaims instead of assuming them. + +## 3. Operating profiles + +| Profile | Ownership | Durability | Intended embedders | +| --- | --- | --- | --- | +| P0 ephemeral | one process, no admission | none — loss of process is loss of run | hermetic tests, single-scenario embedding | +| P1 local durable | one process, lease-admitted | crash-consistent authoritative store | local tools, RAE/env-pack/ETV embedding, air-gapped hosts | +| P2 served | one owning service process, many clients | P1 core | shared local services | +| P3 coordinated | multiple processes or hosts | seam only | none — explicit nonclaim pending a future ADR | + +A profile is a declaration, not a mode switch: P1 and P2 are the same core +with the adapter in front; P0 is the in-memory store under the same +contract; P3 exists only as the lease, revision, and coordination seams the +lower profiles already exercise. + +## 4. State authority + +- **Authoritative**: runtime snapshots, operation records, idempotency + claims, participant transition records. They live in the profile's store + and change only inside its transactions. +- **Derived**: in-memory snapshot and operation maps, receipt caches, + indexes. Rebuildable from authoritative state on demand; each cache that + survives a mutation boundary carries an explicit coherence rule + (invalidate-on-commit under revision CAS). A derived structure never + answers admission or receipt queries on its own authority. +- **Evidentiary**: append-only audit events with provenance. Never + rewritten; terminal audit events commit in the same transaction as the + state they describe. +- **External**: backend effects. The control plane records intent and + observed outcome; where neither is observable it records indeterminacy. + No control-plane read infers an external effect. + +## 5. Operation lifecycle and control flow + +1. **Admission**: the submission is validated against the target and an + idempotency claim is taken — a unique insert in the authoritative store. + A duplicate claim returns the recorded receipt; it never re-invokes the + backend. +2. **Write-ahead claim**: the operation record becomes durably `RUNNING` + before the backend is invoked. +3. **Invocation**: the backend call executes outside any store transaction; + the store never holds locks across external effects. +4. **Terminal commit**: exactly one transaction writes the resulting + snapshot (revision-checked), the terminal operation record, and the + audit event. Participant transitions already follow this shape; the + generic path adopts it (CP-2). +5. **Receipt**: idempotent reads return the committed terminal state from + the store or a coherence-ruled cache (CP-7). + +## 6. Failure and recovery behavior + +After process loss, storage loss short of media failure, cancellation, or +timeout, startup reconciliation (CP-3) classifies every non-terminal +operation: + +- **effect-absent**: the claim exists but the backend verifiably did not + act; the operation fails closed with a stable diagnostic. +- **effect-applied**: backend observation confirms the effect; state is + advanced from observation and committed atomically. +- **indeterminate**: neither is establishable; the operation terminates in + the explicit indeterminate state, its idempotency claim is retained so + client retries cannot blindly re-invoke, and the embedder-visible surface + requires a deliberate resolution action. + +Nothing replays automatically. Host loss and network partition reduce to +process loss at P0–P2 because ownership is single-process; the partition +cases that require distributed reasoning are exactly the P3 nonclaim. + +## 7. Concurrency control + +- **In-process**: existing lock discipline serializes mutation within the + owner. +- **Cross-process**: the store lease (CP-5) admits exactly one owner; a + second process fails closed at admission rather than corrupting state. +- **Optimistic safety net**: snapshot commits carry a revision and commit + by compare-and-swap (CP-4), so even an ownership bug cannot silently + overwrite; the stale writer errors. +- **Idempotency**: unique claims in the store are the concurrency-safe + dedup primitive; caches are advisory. + +## 8. Persistence and coordination seams + +The store contract owns transactions, unique claims, revisions, and the +lease. Providers must verify granted admission results (journal mode, sync +level) rather than trusting requests — the WAL admission finding from issue +#1092, generalized. The clock, audit sink, lease provider, and a future +coordination provider are separate seams so `RuntimeControlPlane` never +encodes a deployment topology. A provider that cannot supply a required +capability fails construction; profiles never degrade silently. + +## 9. Security boundaries + +Authenticated and authorized access (API-404) terminates at the profile +boundary: P0/P1 inherit the embedding process's identity and the store's +filesystem protections (owned directories, tightened modes, identity-pinned +databases per the PR #1136 hardening); P2 adds the HTTP adapter's +authentication and target binding (#1090, #1133) in front of the same core. +Audit events carry actor provenance in every profile. + +## 10. Lifecycle, configuration, and operations + +The embedder owns process lifecycle, configuration, upgrade sequencing, and +backup scheduling; the control plane owns schema versioning, one-time +migration from superseded stores (with durable, fsynced backups), integrity +verification at startup, and refusing to open state it cannot verify. +Operational health and the indeterminate-operation runbook are CP-12. + +## 11. What this composition does not do + +It does not make two application workers coherent over a shared database +(P3 nonclaim), does not give P0 any durability, does not let the HTTP +adapter scale writes beyond its single owner, and does not decide a +specific storage engine here — CP-6 lands the P1 reference store under the +contract, and any conforming provider may replace it. diff --git a/docs/research/runtime-control-plane/current-state-assessment.md b/docs/research/runtime-control-plane/current-state-assessment.md new file mode 100644 index 000000000..bf873f039 --- /dev/null +++ b/docs/research/runtime-control-plane/current-state-assessment.md @@ -0,0 +1,111 @@ +# Runtime Control-Plane Current-State Assessment + +Date: 2026-08-17 + +Parent issue: [#1151](https://github.com/OpenRAE/rae/issues/1151) + +Evidence below cites the `dev` tree at the time of assessment +(`701858d1`). Line-level behavior was verified by direct reading, not +inferred from documentation. + +## 1. The contract surface today + +`raes_runtime/control_plane.py` defines `RuntimeControlPlane`. Its +constructor binds one `RuntimeTarget`, one `ControlPlaneStore` (default +`InMemoryControlPlaneStore`), then loads and permanently caches the +snapshot (`self._snapshot = … load_snapshot()`) and the full operation-record +map (`self._operations = self._store.load_records()`). Mutation is +serialized in-process by `RLock`s (`_operation_lock`, +`_participant_control_lock`, `_trusted_provisioning_plan_lock`). Nothing +invalidates or rebuilds these caches after construction: a second process +sharing the same store is invisible to the first. + +## 2. The store protocol + +`control_plane_store.py::ControlPlaneStore` (a `Protocol`) exposes +`load_snapshot`, `save_snapshot`, `load_records`, `save_record`, +`find_by_idempotency`, `append_audit`, `read_audit`, and two transition +commits: `commit_control_transition(expected_head=…)` and +`commit_participant_transition(expected_history_heads=…)`. The transition +commits are the one place the store contract already expresses atomic +multi-record commits guarded by expected-head compare-and-swap +(`_require_expected_control_head`, `_require_expected_history_heads`). +Snapshots and generic operation records have no revision or expectation +parameter: `save_snapshot` and `save_record` overwrite unconditionally. + +Two implementations exist on `dev`: + +- `InMemoryControlPlaneStore` — dictionaries, no durability, correct for + profile P0. +- `LocalControlPlaneStore` (`control_plane_store_local.py`) — four JSON + files (`snapshot.json`, `operations.json`, `audit.jsonl`, + `control-transition-state.json`). Each write is atomic per file via a + temporary file and `os.replace`, but a logical commit spanning snapshot + plus record plus audit is two or three separate file replacements, and + `operations.json` is a whole-file read-modify-replace: concurrent + writers lose updates, and a crash between file replacements leaves the + files mutually inconsistent. Issue #1092 documented these failures and + PR #1136 replaced this store with a transactional SQLite design; both + are deferred to this decision. + +## 3. The generic execution path + +`control_plane_execution.py::execute_operation` performs, in order: + +1. `_idempotent_receipt(...)` — `find_by_idempotency` against the store, + while `get_operation` and `get_snapshot` answer from the permanent + caches; in the JSON store the lookup and the later claim are separate + steps, not one atomic unique claim; +2. `_persist_record(...)` with `OperationState.RUNNING`; +3. `_call_backend_apply(...)` — the external effect; +4. `control_plane._store.save_snapshot(...)`; +5. `_persist_record(...)` with the terminal `SUCCEEDED`/`FAILED` status. + +Steps 2, 4, and 5 are independently durable. A process exit after step 3 +leaves an applied backend effect with a `RUNNING` record and a stale +snapshot; after step 4, a new snapshot with a non-terminal record. On +restart the record loads unchanged, and step 1 returns it to an idempotent +retry without reconciliation. This confirms the first finding of the +issue #1092 integration review and motivates ADR-104 §4 (write-ahead claim, +one atomic terminal commit, startup reconciliation). The participant +transition path does not share this defect: it commits through the store's +transition-commit methods as one guarded unit and is the template CP-2 +generalizes. + +## 4. The reference HTTP adapter + +`control_plane_api/` implements the served surface. `_offload.py` keeps +blocking work off the event loop and serializes target mutation through a +single `asyncio.Lock` per application instance — correct while exactly one +service process owns the store, and exactly the application-local +serialization the #1092 review flagged: two service processes over one +store would each hold their own lock and their own stale caches. Bearer +authentication and target binding come from #1090/#1133 +(`control_plane_security.py`, `control_plane_api_guards.py`), and workflow +timeout reconciliation fails closed per #1132 +(`control_plane_timeouts.py`). These surfaces are retained under P2. + +## 5. Test surfaces pinning today's behavior + +- `test_runtime_control_plane.py`, `test_runtime_conformance.py` — the + contract and in-memory behavior. +- `test_runtime_control_plane_api.py` — adapter admission, auth, offload + serialization (#1133). +- `test_dsl_437_snapshot_durability_conformance.py`, + `test_run_307_shared_operational_state.py` — snapshot durability and + shared-state reads (#1148). +- PR #1136 additionally carries a 115-test crash-consistency suite bound + to its SQLite store; the implementation program absorbs it into CP-9. + +## 6. Gap summary against ADR-104 + +| Gap | Evidence | Owning work package | +| --- | --- | --- | +| No explicit lifecycle contract; no indeterminate terminal state | `OperationState` in `raes_contracts.runtime_state` lacks it; interrupted work stays `RUNNING` | CP-1 | +| Non-atomic terminal effects on the generic path | `execute_operation` steps 2/4/5 above | CP-2 | +| No startup reconciliation | records load unchanged at construction | CP-3 | +| Unconditional snapshot/record writes | `save_snapshot`/`save_record` have no expected revision | CP-4 | +| No ownership admission | any process may open any store | CP-5 | +| Non-transactional durable store | `LocalControlPlaneStore` JSON files | CP-6 | +| Non-atomic idempotency claims; status and snapshot reads answered from permanent caches | `find_by_idempotency` + later `save_record`; `get_operation` over `self._operations`, `get_snapshot` over `self._snapshot` | CP-7 | +| Application-local mutation serialization | `_offload.py` `asyncio.Lock` | CP-8 | diff --git a/docs/research/runtime-control-plane/implementation-program.json b/docs/research/runtime-control-plane/implementation-program.json new file mode 100644 index 000000000..05e9d5e59 --- /dev/null +++ b/docs/research/runtime-control-plane/implementation-program.json @@ -0,0 +1,133 @@ +{ + "schema_version": "runtime-control-plane-program/v1", + "assessment_date": "2026-08-17", + "parent_issue": 1151, + "milestone": "Runtime Control-Plane", + "requirement": "API-404", + "deliverables": [ + "docs/decisions/issue-1151-runtime-control-plane-architecture-preflight.md", + "docs/decisions/adrs/adr-104-runtime-control-plane-architecture.md", + "docs/research/runtime-control-plane/index.md", + "docs/research/runtime-control-plane/current-state-assessment.md", + "docs/research/runtime-control-plane/composition-architecture.md", + "docs/research/runtime-control-plane/requirement-disposition.md", + "docs/research/runtime-control-plane/implementation-program.md", + "docs/research/runtime-control-plane/implementation-program.json" + ], + "profiles": [ + { + "id": "P0", + "name": "ephemeral", + "claims": ["in-process contract conformance"], + "nonclaims": ["durability", "multi-process ownership", "restart recovery"] + }, + { + "id": "P1", + "name": "local durable", + "claims": [ + "single lease-admitted owner", + "crash-consistent authoritative state", + "atomic terminal commits", + "startup reconciliation with explicit indeterminacy" + ], + "nonclaims": ["multi-process ownership", "high availability"] + }, + { + "id": "P2", + "name": "served", + "claims": [ + "P1 core behind the reference HTTP adapter", + "owner-serialized mutation", + "revision-carrying reads" + ], + "nonclaims": ["multi-owner writes", "high availability"] + }, + { + "id": "P3", + "name": "coordinated", + "claims": [], + "nonclaims": ["everything: seam only, requires a future ADR"] + } + ], + "work_packages": [ + { + "id": "CP-1", + "title": "Operation lifecycle contract", + "depends_on": [], + "surfaces": ["contracts/schemas/control-plane", "raes_runtime lifecycle states"] + }, + { + "id": "CP-2", + "title": "Atomic terminal commit in the generic execution path", + "depends_on": ["CP-1"], + "surfaces": ["raes_runtime control_plane_execution"] + }, + { + "id": "CP-3", + "title": "Startup reconciliation with effect classification", + "depends_on": ["CP-1", "CP-2"], + "surfaces": ["raes_runtime control_plane recovery"] + }, + { + "id": "CP-4", + "title": "Snapshot revision compare-and-swap and derived-cache demotion", + "depends_on": ["CP-1"], + "surfaces": ["raes_runtime control_plane", "store protocol"] + }, + { + "id": "CP-5", + "title": "Store lease admission", + "depends_on": [], + "surfaces": ["store protocol", "local lease implementation"] + }, + { + "id": "CP-6", + "title": "Transactional local store (re-lands PR #1136)", + "depends_on": ["CP-1", "CP-2", "CP-4", "CP-5"], + "surfaces": ["raes_runtime control_plane_store_local and sibling modules"] + }, + { + "id": "CP-7", + "title": "Atomic idempotency claims and cache demotion", + "depends_on": ["CP-6"], + "surfaces": ["raes_runtime idempotent receipt path", "get_operation/get_snapshot caches"] + }, + { + "id": "CP-8", + "title": "Served-profile alignment of the reference HTTP adapter", + "depends_on": ["CP-4", "CP-7"], + "surfaces": ["raes_runtime control_plane_api"] + }, + { + "id": "CP-9", + "title": "Crash and profile conformance suite", + "depends_on": ["CP-2", "CP-3", "CP-6"], + "surfaces": ["implementations/python/tests"] + }, + { + "id": "CP-10", + "title": "Profile declaration and capability discovery", + "depends_on": ["CP-1", "CP-2", "CP-3", "CP-4", "CP-5", "CP-6", "CP-7", "CP-8"], + "surfaces": ["raes_runtime profile surface", "docs/explain/sdl/runtime-architecture.md"] + }, + { + "id": "CP-11", + "title": "API-404 requirement update and traceability", + "depends_on": ["CP-10"], + "surfaces": ["docs/requirements/API-404"] + }, + { + "id": "CP-12", + "title": "Recovery runbook and operator tooling", + "depends_on": ["CP-3", "CP-6"], + "surfaces": ["docs", "operator tooling"] + } + ], + "dispositions": { + "issue_1092": "superseded by CP-1 through CP-7; close on package filing", + "pr_1136": "split: store work re-lands as CP-6, recovery reworked under CP-3, tests absorbed into CP-9", + "runtime_control_plane": "retain and change under the profiled contract", + "local_json_store": "supersede; migration source only", + "http_adapter": "retain as the P2 reference; change under CP-8" + } +} diff --git a/docs/research/runtime-control-plane/implementation-program.md b/docs/research/runtime-control-plane/implementation-program.md new file mode 100644 index 000000000..04a8da5ad --- /dev/null +++ b/docs/research/runtime-control-plane/implementation-program.md @@ -0,0 +1,142 @@ +# Runtime Control-Plane Implementation Program + +Date: 2026-08-17 + +Parent issue: [#1151](https://github.com/OpenRAE/rae/issues/1151) + +Milestone: `Runtime Control-Plane` + +The machine-readable authority is +[`implementation-program.json`](implementation-program.json). + +## Definition delivered by issue 1151 + +Issue #1151 delivers ADR-104, the current-state assessment, the profiled +composition architecture, the requirement and surface disposition, and this +dependency-ordered program. It does not implement any work package below; +each package is filed as its own issue in the Runtime Control-Plane +milestone when this design is accepted, and no guarantee is claimable before +its package lands with tests. + +## Dependency graph + +```text + CP-1 lifecycle contract + | \ + v v + CP-2 atomic CP-4 revision CAS CP-5 store lease + terminal | | + commit | | + | +----------+-----------+ + v | + CP-3 startup v + reconciliation CP-6 transactional local store (re-lands PR #1136) + | | + | v + | CP-7 idempotency claims in store + | | + +------------+ v + | CP-8 served profile (HTTP adapter) + v | + CP-9 crash/profile conformance suite + | + v + CP-10 profile declaration and capability discovery + | + +--> CP-11 API-404 requirement update + +--> CP-12 recovery runbook and operator tooling +``` + +## Work packages + +Each package below becomes one milestone issue with this scope, ordering, +and verification expectation. Requirement traceability is `API-404` +throughout; packages touching snapshot semantics also cite `SEM-222`. + +### CP-1 — Operation lifecycle contract + +Define the operation lifecycle in contracts: states including the explicit +indeterminate terminal outcome, the stable diagnostics that accompany each +transition, and the state-transition table. Update +`contracts/schemas/control-plane` accordingly. Verification: contract tests +enumerate every legal and illegal transition. Depends on: nothing. + +### CP-2 — Atomic terminal commit + +Rework the generic execution path so the running claim is durable before +backend invocation and the terminal transition commits snapshot, terminal +record, and audit event in one store transaction, matching the participant +transition path. Verification: kill-point tests at every step boundary show +no observable partial terminal state. Depends on: CP-1. + +### CP-3 — Startup reconciliation + +Classify every non-terminal operation at startup as effect-absent, +effect-applied, or indeterminate, using backend observation where the +backend offers it; never replay automatically; park indeterminate outcomes +behind the CP-1 diagnostic and an embedder-visible surface. Verification: +restart tests per classification, including the no-observation backend. +Depends on: CP-1, CP-2. + +### CP-4 — Snapshot revision compare-and-swap + +Version every snapshot commit; a writer holding a stale revision fails +closed. Demote in-memory maps to rebuildable derived state with an explicit +coherence rule. Verification: interleaved-writer tests and cache-rebuild +tests. Depends on: CP-1. + +### CP-5 — Store lease admission + +Contract a store ownership lease; exactly one owner per store at P1/P2. +Local implementation follows the PR #1136 lease module. Verification: +concurrent-process admission tests. Depends on: nothing (integrates with +CP-4). + +### CP-6 — Transactional local store + +Re-land the PR #1136 SQLite store under the CP-1/CP-2/CP-4/CP-5 contracts: +WAL admission by returned result, unique idempotency claims, path and +permission hardening, integrity digests, one-time migration with durable +backups. Recovery behavior conforms to CP-3 instead of blanket +interrupted-to-failed conversion. Verification: the absorbed +crash-consistency suite plus store-contract conformance. Depends on: CP-1, +CP-2, CP-4, CP-5. + +### CP-7 — Atomic idempotency claims and cache demotion + +Make the idempotency lookup-and-claim one atomic store operation (unique +claim), and demote the permanent status and snapshot caches behind +`get_operation`/`get_snapshot` to rebuildable derived state with an +explicit coherence rule. Verification: multi-restart retry tests; +stale-cache injection. Depends on: CP-6. + +### CP-8 — Served profile alignment + +Bring the reference HTTP adapter to profile P2: single owning service +process, owner-serialized mutations, reads carrying the snapshot revision +they observed, and explicit stale-read rules. Verification: extension of +the #1133 admission suite. Depends on: CP-4, CP-7. + +### CP-9 — Crash and profile conformance suite + +One suite that runs each profile's guarantee set: kill-point injection at +every commit boundary, restart reconciliation assertions, lease contention, +revision conflicts, and P0's explicit nonclaims. Absorbs the 115-test crash +suite from PR #1136. Depends on: CP-2, CP-3, CP-6. + +### CP-10 — Profile declaration and capability discovery + +Embedders select a profile and can interrogate its guarantees and nonclaims +programmatically; `docs/explain/sdl/runtime-architecture.md` gains the +profile model. Depends on: CP-1 through CP-8. + +### CP-11 — API-404 requirement update + +Rewrite API-404 traceability to the profiled contract, recording which +clauses each profile satisfies and P0's durability waiver; sync Ground +Control. Depends on: CP-10. + +### CP-12 — Recovery runbook and operator tooling + +Operator flow for indeterminate operations, backup/restore, upgrade and +migration sequencing, and health surfaces. Depends on: CP-3, CP-6. diff --git a/docs/research/runtime-control-plane/index.md b/docs/research/runtime-control-plane/index.md new file mode 100644 index 000000000..6d2139c56 --- /dev/null +++ b/docs/research/runtime-control-plane/index.md @@ -0,0 +1,28 @@ +# Runtime Control-Plane Architecture + +Issue [#1151](https://github.com/OpenRAE/rae/issues/1151) defines what the +RAES runtime control-plane is: a portable contract with profiled +implementations spanning hermetic tests, single-user local execution, +embedded RAE/env-pack/ETV consumers, air-gapped deployments, and served +topologies. It answers the state-authority, operation-lifecycle, +concurrency, and failure-recovery questions that issue #1092 and PR #1136 +exposed, and it disposes of those surfaces explicitly. + +- [Architecture preflight](../../decisions/issue-1151-runtime-control-plane-architecture-preflight.md) +- [ADR-104](../../decisions/adrs/adr-104-runtime-control-plane-architecture.md) +- [Current-state assessment](current-state-assessment.md) +- [Composition architecture](composition-architecture.md) +- [Requirement and surface disposition](requirement-disposition.md) +- [Implementation program](implementation-program.md) +- [Machine-readable program](implementation-program.json) + +The requirement authority is `API-404` (ACTIVE). The implementation program +is owned by the **Runtime Control-Plane** milestone; its work packages are +filed as issues once this design is accepted. + +Issue #1151 does not implement or select a storage engine by code change, +publish new portable schemas, change runtime execution behavior, or claim a +distributed or highly available topology. Profile P3 (coordinated +multi-process ownership) is recorded as a seam and an explicit nonclaim; no +consistency, coordination, or recovery guarantee named in this set is +claimable before its implementation issue lands with tests. diff --git a/docs/research/runtime-control-plane/requirement-disposition.md b/docs/research/runtime-control-plane/requirement-disposition.md new file mode 100644 index 000000000..7299810fb --- /dev/null +++ b/docs/research/runtime-control-plane/requirement-disposition.md @@ -0,0 +1,47 @@ +# Requirement and Surface Disposition + +Date: 2026-08-17 + +Parent issue: [#1151](https://github.com/OpenRAE/rae/issues/1151) + +This document records the evidence-backed disposition of every incumbent +control-plane surface, requirement, and in-flight change, as ADR-104 §7 +requires. Dispositions are one of: **retain**, **change**, **split**, +**supersede**, or **remove**. + +## Requirements + +| Requirement | Disposition | Detail | +| --- | --- | --- | +| `API-404` (ACTIVE, wave 1) | retain, change | Remains the control-plane authority. Work package CP-11 rewrites its traceability to the profiled contract and records which clauses (authenticated access, durable state, idempotent submission, auditable lifecycle) each profile satisfies; P0 explicitly waives durability. | +| `SEM-222` (touched by PR #1136) | retain | Snapshot semantic-integrity obligations are unchanged by this design; the P1 store must preserve them through the round-trip codec guard PR #1136 introduced. | + +## Code surfaces + +| Surface | Disposition | Detail | +| --- | --- | --- | +| `raes_runtime.control_plane.RuntimeControlPlane` | retain, change | Stays the portable contract entry point. CP-1/CP-2 change the operation lifecycle to write-ahead claims and atomic terminal commits; CP-7 makes idempotency claims atomic and demotes the status/snapshot caches. | +| Generic operation execution path (`control_plane_execution`) | change | Four independently durable steps become claim → invoke → one terminal transaction (snapshot, terminal record, audit). CP-2. | +| Participant transition CAS path | retain | Already commits snapshot, record, and audit as one unit; becomes the template the generic path adopts. | +| In-memory default store | retain | Profile P0's store. Gains the store-contract conformance surface (CP-1, CP-9) with no durability claims. | +| `LocalControlPlaneStore` (JSON, whole-file replace) | supersede | Migration source only. The P1 transactional store replaces it; a one-time import with durable backups (per PR #1136) carries state forward. Removal follows after the migration window, tracked in CP-6. | +| SQLite store modules from PR #1136 (`control_plane_store_local`, `_paths`, `_lease`, `_snapshots`, `_compatibility`, `_legacy`, `control_plane_durability`, `control_plane_lifecycle`, `control_plane_recovery`) | change, adopt | Principal input to CP-6. WAL admission by returned result, unique idempotency claims, atomic participant commits, POSIX path hardening, integrity digests, and migration/backup durability are adopted as designed. The blanket interrupted-to-`FAILED` startup conversion is reworked to CP-3's classification (effect-absent / effect-applied / indeterminate). | +| Reference HTTP adapter (`control_plane_api*`) | retain, change | Becomes the P2 reference. CP-8 pins single-owner service posture, owner-serialized mutation, and revision-carrying reads with explicit stale-read rules. | +| `RuntimeTarget` and backend contracts | retain | Outside control-plane authority (ADR-104 §6). Unchanged. | +| Permanent snapshot/operation caches in `RuntimeControlPlane` | change | Demoted to derived state: rebuildable, coherence-ruled, never authoritative for receipts or admission. CP-4/CP-7. | + +## In-flight work + +| Item | Disposition | Detail | +| --- | --- | --- | +| Issue #1092 | supersede | Its problem statement is absorbed by ADR-104 §§3–5 and its remedy is re-scoped into CP-1 through CP-7. Close #1092 when those work packages are filed, linking each residual gap from its integration-review addendum to the package that owns it. | +| PR #1136 | split, adopt | Deferred by the maintainer pending this design. Re-land as CP-6 (store), with its recovery behavior reworked under CP-3 and its crash-consistency suite absorbed into CP-9. The PR's preflight note (`issue-1092-local-control-plane-durability-preflight.md`) lands with CP-6 as the store-level design record, subordinate to ADR-104. | +| Issue #8 (API-404 tracking) | retain | Gains the profile-scoped acceptance framing through CP-11. | + +## Documentation and test surfaces + +| Surface | Disposition | Detail | +| --- | --- | --- | +| `docs/explain/sdl/runtime-architecture.md` | change | Gains the profile model and the operation-lifecycle description once CP-10 lands; until then it must not claim durability or multi-process support. | +| Crash-consistency tests from PR #1136 (115 tests) | adopt | Absorbed into CP-9's profile conformance suite with kill-point injection at every commit boundary of the new lifecycle. | +| `test_http_api_admission` / offload surfaces (#1133) | retain | P2 admission behavior is unchanged by this design; CP-8 extends the same suite. | diff --git a/implementations/python/tests/test_issue_1151_runtime_control_plane_design.py b/implementations/python/tests/test_issue_1151_runtime_control_plane_design.py new file mode 100644 index 000000000..1a6727888 --- /dev/null +++ b/implementations/python/tests/test_issue_1151_runtime_control_plane_design.py @@ -0,0 +1,103 @@ +"""Structural acceptance gate for issue #1151's control-plane design program.""" + +from __future__ import annotations + +import json +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[3] +PROGRAM_PATH = REPO_ROOT / "docs/research/runtime-control-plane/implementation-program.json" +MILESTONE = "Runtime Control-Plane" + +REQUIRED_DELIVERABLES = { + "docs/decisions/issue-1151-runtime-control-plane-architecture-preflight.md", + "docs/decisions/adrs/adr-104-runtime-control-plane-architecture.md", + "docs/research/runtime-control-plane/index.md", + "docs/research/runtime-control-plane/current-state-assessment.md", + "docs/research/runtime-control-plane/composition-architecture.md", + "docs/research/runtime-control-plane/requirement-disposition.md", + "docs/research/runtime-control-plane/implementation-program.md", + "docs/research/runtime-control-plane/implementation-program.json", +} +REQUIRED_PROFILES = {"P0", "P1", "P2", "P3"} +REQUIRED_WORK_PACKAGES = { + "CP-1", + "CP-2", + "CP-3", + "CP-4", + "CP-5", + "CP-6", + "CP-7", + "CP-8", + "CP-9", + "CP-10", + "CP-11", + "CP-12", +} +REQUIRED_DISPOSITIONS = { + "issue_1092", + "pr_1136", + "runtime_control_plane", + "local_json_store", + "http_adapter", +} + + +def _program() -> dict: + return json.loads(PROGRAM_PATH.read_text(encoding="utf-8")) + + +def test_design_set_is_complete_and_present() -> None: + program = _program() + assert set(program["deliverables"]) == REQUIRED_DELIVERABLES + for deliverable in REQUIRED_DELIVERABLES: + path = REPO_ROOT / deliverable + assert path.is_file(), f"missing deliverable {deliverable}" + assert path.stat().st_size > 0, f"empty deliverable {deliverable}" + + +def test_program_names_the_requirement_and_milestone() -> None: + program = _program() + assert program["requirement"] == "API-404" + assert program["milestone"] == MILESTONE + assert program["parent_issue"] == 1151 + + +def test_profiles_declare_claims_and_nonclaims() -> None: + program = _program() + profiles = {profile["id"]: profile for profile in program["profiles"]} + assert set(profiles) == REQUIRED_PROFILES + for profile in profiles.values(): + assert profile["nonclaims"], f"{profile['id']} must state explicit nonclaims" + assert profiles["P3"]["claims"] == [], "P3 is a seam-only nonclaim" + + +def test_work_packages_form_an_acyclic_dependency_program() -> None: + program = _program() + packages = {package["id"]: package for package in program["work_packages"]} + assert set(packages) == REQUIRED_WORK_PACKAGES + for package in packages.values(): + for dependency in package["depends_on"]: + assert dependency in packages, f"{package['id']} depends on unknown {dependency}" + + resolved: set[str] = set() + remaining = dict(packages) + while remaining: + ready = [pid for pid, package in remaining.items() if set(package["depends_on"]) <= resolved] + assert ready, f"dependency cycle among {sorted(remaining)}" + for pid in ready: + resolved.add(pid) + del remaining[pid] + + +def test_dispositions_cover_the_deferred_surfaces() -> None: + program = _program() + assert set(program["dispositions"]) == REQUIRED_DISPOSITIONS + + +def test_adr_and_design_docs_reference_each_other() -> None: + adr = (REPO_ROOT / "docs/decisions/adrs/adr-104-runtime-control-plane-architecture.md").read_text(encoding="utf-8") + index = (REPO_ROOT / "docs/research/runtime-control-plane/index.md").read_text(encoding="utf-8") + assert "#1151" in adr + assert "adr-104-runtime-control-plane-architecture.md" in index + assert "API-404" in index diff --git a/tools/policy/historical_identity_records.json b/tools/policy/historical_identity_records.json index 03b6296fa..911f87912 100644 --- a/tools/policy/historical_identity_records.json +++ b/tools/policy/historical_identity_records.json @@ -47,7 +47,7 @@ "record_class": "historical-index", "rationale": "Indexes immutable pre-cutover ADR titles, paths, pins, and amendment summaries without making them current identity surfaces.", "occurrences": 4, - "content_sha256": "fb313cf4bb3281dd64a9c763cf8e0fbc3819e6ea36a6b297045f7fdf45281565" + "content_sha256": "bc4f3aa84b1b62c139aff2308bbb6377a93431e9b0ea7b411dbef4b8debdc468" }, { "path": "docs/decisions/adrs/adr-000-use-adrs.md", @@ -495,7 +495,7 @@ "record_class": "historical-index", "rationale": "Indexes immutable pre-cutover ADR titles, paths, pins, and amendment summaries without making them current identity surfaces.", "occurrences": 4, - "content_sha256": "f800c15b5d7131c89dde6e40b69e9ced8d303f15a4a9709ba89067f1b45819f4" + "content_sha256": "426c29ffec3274a91e60053e83542f04091b41b863d62cf35b95035030b86d78" }, { "path": "docs/decisions/cage-2-replication-design.md", diff --git a/tools/policy/requirement_order.yaml b/tools/policy/requirement_order.yaml index 113a8719c..1013d1294 100644 --- a/tools/policy/requirement_order.yaml +++ b/tools/policy/requirement_order.yaml @@ -12,6 +12,9 @@ phases: - id: sdl-language patterns: - ^DSL- + - id: runtime-control-plane + requirements: + - API-404 - id: api400-core requirements: - API-412 From 3442f768f4d6bc0f8f916f3e3ae1d106231bfa3d Mon Sep 17 00:00:00 2001 From: Yernat Yestekov <2068106+doublewhy@users.noreply.github.com> Date: Mon, 17 Aug 2026 17:58:49 -0700 Subject: [PATCH 2/2] docs(runtime): fold the full surface audit into the design set MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The completed control-plane audit corrected and sharpened the evidence: - the generic execute_operation path holds no lock at all (only participant actions take _participant_control_lock), the in-memory snapshot is reassigned before the durable write, and the participant and manager paths hold two unordered locks — recorded in ADR-104's context and concurrency clauses and in CP-2's scope - OperationState.ACCEPTED exists but is never used; the idempotency check is check-then-act and an empty stored fingerprint disables the reuse-mismatch check - the JSON store never fsyncs, its idempotency lookup is a linear scan, and load_snapshot arbitrates files by a transition-count heuristic - the API-408 GET retrieval routes take the HTTP mutation lock; CP-8 now moves reads off it - nothing runs the adapter (factory only, uvicorn unused), matching the issue #1093 preflight's own topology nonclaim, cited directly - the durable-store implementation already exists on three successor branches with a shape that converges with ADR-104 §§4-5 (atomic claim_record, AtomicControlPlaneStore, RuntimeOwnerLease, unified operation lock); CP-6 re-lands that work rather than redesigning it, and the ~1,250-line crash-consistency suite is CP-9's acceptance bar The ADR acceptance pin and the identity-cutover digest for the ADR index are recomputed for the amended text. Part of #1151. Requirement: API-404. Co-Authored-By: Claude Fable 5 --- ...-104-runtime-control-plane-architecture.md | 36 ++++--- docs/decisions/adrs/adr-index.yaml | 2 +- .../current-state-assessment.md | 99 ++++++++++++++----- .../implementation-program.md | 26 +++-- .../requirement-disposition.md | 5 +- tools/policy/historical_identity_records.json | 2 +- 6 files changed, 116 insertions(+), 54 deletions(-) diff --git a/docs/decisions/adrs/adr-104-runtime-control-plane-architecture.md b/docs/decisions/adrs/adr-104-runtime-control-plane-architecture.md index 3ef2c1d55..b7d46c128 100644 --- a/docs/decisions/adrs/adr-104-runtime-control-plane-architecture.md +++ b/docs/decisions/adrs/adr-104-runtime-control-plane-architecture.md @@ -38,12 +38,14 @@ decision. The integration review of that work recorded two structural findings that bound this design. First, the generic operation path performs independently durable steps — claim `RUNNING`, invoke the backend, save the snapshot, save -terminal status — so a process exit can strand an applied backend effect -behind stale state, and an idempotent retry after restart returns the stale -record without reconciling. Second, each `RuntimeControlPlane` permanently -caches snapshot and operation maps while HTTP mutation serialization is -application-local, so a shared database alone cannot make multiple -application workers coherent. +terminal status — with no lock of its own and the in-memory snapshot mutated +before the durable write, so concurrent in-process submissions race and a +process exit can strand an applied backend effect behind stale state, while +an idempotent retry after restart returns the stale record without +reconciling. Second, each `RuntimeControlPlane` permanently caches snapshot +and operation maps while HTTP mutation serialization is application-local, +so a shared database alone cannot make multiple application workers +coherent. Expected RAES use spans hermetic tests, single-user local execution, embedded RAE/env-pack/ETV consumers, air-gapped deployments, and long-lived services. @@ -110,8 +112,10 @@ Exactly one writer owns a store at a time in P1 and P2, admitted through a store lease. Snapshot commits carry a revision and commit by compare-and-swap, so a stale writer fails closed instead of overwriting. Idempotency keys are unique claims in the authoritative store, not cache -entries. Within a process, existing lock discipline continues to serialize -mutation; across processes, admission is the lease, not advisory locking. +entries. Within a process, one operation lock serializes every mutation +path — today the generic execution path is unlocked and the participant and +manager paths hold two unordered locks; across processes, admission is the +lease, not advisory locking. ### 6. Boundaries against the rest of RAES @@ -131,13 +135,15 @@ and audit. reference HTTP adapter are retained and brought under this contract. The local JSON store is superseded by the P1 transactional store and retained only as a migration source. Issue #1092 is re-scoped into the P1 -implementation program; PR #1136 is its principal input — the SQLite WAL -admission, atomic participant commits, path hardening, and migration work -align with this decision and are re-landed as the P1 store issues, while its -recovery semantics are reworked to the reconciliation classification above -instead of a blanket interrupted-to-failed conversion. The full disposition -table, including every store module and test surface, lives in the design -set's requirement disposition. +implementation program; PR #1136 and its successor branches are its +principal input — the atomic claim, single-transaction terminal commit, +owner lease, WAL admission, path hardening, and migration work already +implemented there converge with this decision and are re-landed as the P1 +store issues, while their recovery semantics are reworked to the +reconciliation classification above instead of a blanket +interrupted-to-failed conversion. The full disposition table, including +every store module and test surface, lives in the design set's requirement +disposition. ## Alternatives Considered diff --git a/docs/decisions/adrs/adr-index.yaml b/docs/decisions/adrs/adr-index.yaml index ca027a2af..8ea24aec2 100644 --- a/docs/decisions/adrs/adr-index.yaml +++ b/docs/decisions/adrs/adr-index.yaml @@ -511,4 +511,4 @@ adrs: pin: a44acbc2db1b5349ba316ba0d09bd9b46310db016cb87937b08bc6bc107755aa - id: ADR-104 path: docs/decisions/adrs/adr-104-runtime-control-plane-architecture.md - pin: bf15139353bc39af301261cb7799fab26ea354cf06a2cb64a59c50a663258cea + pin: b27359f9d4bc21bbd3b13570202fc03a4107a0d0fa66b18937210fde18168c8e diff --git a/docs/research/runtime-control-plane/current-state-assessment.md b/docs/research/runtime-control-plane/current-state-assessment.md index bf873f039..2645a522e 100644 --- a/docs/research/runtime-control-plane/current-state-assessment.md +++ b/docs/research/runtime-control-plane/current-state-assessment.md @@ -14,11 +14,17 @@ inferred from documentation. constructor binds one `RuntimeTarget`, one `ControlPlaneStore` (default `InMemoryControlPlaneStore`), then loads and permanently caches the snapshot (`self._snapshot = … load_snapshot()`) and the full operation-record -map (`self._operations = self._store.load_records()`). Mutation is -serialized in-process by `RLock`s (`_operation_lock`, -`_participant_control_lock`, `_trusted_provisioning_plan_lock`). Nothing -invalidates or rebuilds these caches after construction: a second process -sharing the same store is invisible to the first. +map (`self._operations = self._store.load_records()`). Nothing invalidates +or rebuilds these caches after construction: a second process sharing the +same store is invisible to the first. + +In-process locking is partial and unordered. `_participant_control_lock` +covers participant actions and control mediation, and the separate +`RuntimeManager` mixin holds its own `_participant_execution_lock` with no +ordering discipline between the two; the generic operation path holds no +lock at all, so concurrent direct library submissions race `_snapshot`, +`_operations`, and the store writes even before any multi-process question +arises. The HTTP adapter's mutation lock protects only HTTP callers. ## 2. The store protocol @@ -40,11 +46,14 @@ Two implementations exist on `dev`: - `LocalControlPlaneStore` (`control_plane_store_local.py`) — four JSON files (`snapshot.json`, `operations.json`, `audit.jsonl`, `control-transition-state.json`). Each write is atomic per file via a - temporary file and `os.replace`, but a logical commit spanning snapshot - plus record plus audit is two or three separate file replacements, and - `operations.json` is a whole-file read-modify-replace: concurrent - writers lose updates, and a crash between file replacements leaves the - files mutually inconsistent. Issue #1092 documented these failures and + temporary file and `os.replace` with no fsync of the file or directory, + but a logical commit spanning snapshot plus record plus audit is two or + three separate file replacements; `operations.json` is a whole-file + read-modify-replace (concurrent writers lose updates, idempotency lookup + is a linear scan), `audit.jsonl` is an unlocked append, and + `load_snapshot` arbitrates between `snapshot.json` and the committed + transition blob by comparing a participant-transition count — a + heuristic, not a version. Issue #1092 documented these failures and PR #1136 replaced this store with a transactional SQLite design; both are deferred to this decision. @@ -61,30 +70,50 @@ Two implementations exist on `dev`: 4. `control_plane._store.save_snapshot(...)`; 5. `_persist_record(...)` with the terminal `SUCCEEDED`/`FAILED` status. -Steps 2, 4, and 5 are independently durable. A process exit after step 3 -leaves an applied backend effect with a `RUNNING` record and a stale +Steps 2, 4, and 5 are independently durable, the in-memory snapshot is +reassigned before the durable write (a failed `save_snapshot` diverges +memory from disk), and no lock guards the sequence. A process exit after +step 3 leaves an applied backend effect with a `RUNNING` record and a stale snapshot; after step 4, a new snapshot with a non-terminal record. On restart the record loads unchanged, and step 1 returns it to an idempotent -retry without reconciliation. This confirms the first finding of the -issue #1092 integration review and motivates ADR-104 §4 (write-ahead claim, -one atomic terminal commit, startup reconciliation). The participant -transition path does not share this defect: it commits through the store's -transition-commit methods as one guarded unit and is the template CP-2 -generalizes. +retry without reconciliation. `OperationState.ACCEPTED` exists in +`raes_contracts.runtime_state` but is never used — the path claims straight +to `RUNNING`. The idempotency check itself is check-then-act: two +concurrent submissions with one key can both miss the lookup and both mint +operations, and a record persisted with an empty request fingerprint +disables the reuse-mismatch check for every later retry. This confirms the +first finding of the issue #1092 integration review and motivates ADR-104 +§4 (write-ahead claim, one atomic terminal commit, startup reconciliation). +The participant transition path does not share the atomicity defect: it +commits through the store's transition-commit methods as one guarded unit +and is the template CP-2 generalizes. ## 4. The reference HTTP adapter -`control_plane_api/` implements the served surface. `_offload.py` keeps -blocking work off the event loop and serializes target mutation through a -single `asyncio.Lock` per application instance — correct while exactly one -service process owns the store, and exactly the application-local -serialization the #1092 review flagged: two service processes over one -store would each hold their own lock and their own stale caches. Bearer +`control_plane_api/` implements the served surface behind one composition +boundary, `create_control_plane_app(control_plane, *, security=None)`. +`_offload.py` keeps blocking work off the event loop and serializes target +mutation through a single `asyncio.Lock` per application instance — correct +while exactly one service process owns the store, and exactly the +application-local serialization the #1092 review flagged: two service +processes over one store would each hold their own lock and their own stale +caches. The three API-408 participant-retrieval `GET` routes also go +through the mutation lock because they append evidence, coupling read +throughput to backend mutation latency (a CP-8 concern). Bearer authentication and target binding come from #1090/#1133 (`control_plane_security.py`, `control_plane_api_guards.py`), and workflow timeout reconciliation fails closed per #1132 (`control_plane_timeouts.py`). These surfaces are retained under P2. +Nothing in the repository runs the adapter: `create_control_plane_app` is a +factory, `uvicorn` is a declared but unused dependency, and there is no +serve entrypoint or CLI command. Every consumer today is an embedded ASGI +test client, so the served topology is asserted by documentation — the +issue #1093 preflight states plainly that the in-process lock "does not +create a distributed queue" and that "a future multi-host service must use +a durable broker/worker design with explicit leases and recovery" — which +is exactly the boundary ADR-104 records as profile P2 and the P3 nonclaim. + ## 5. Test surfaces pinning today's behavior - `test_runtime_control_plane.py`, `test_runtime_conformance.py` — the @@ -94,14 +123,30 @@ timeout reconciliation fails closed per #1132 - `test_dsl_437_snapshot_durability_conformance.py`, `test_run_307_shared_operational_state.py` — snapshot durability and shared-state reads (#1148). -- PR #1136 additionally carries a 115-test crash-consistency suite bound - to its SQLite store; the implementation program absorbs it into CP-9. +- PR #1136 and its successor branches additionally carry + `test_issue_1092_control_plane_crash_consistency.py` (~1,250 lines): + terminal-commit idempotence, per-write-boundary rollback, restart + reconciliation, second-owner rejection with clean handoff, WAL and + integrity guards, and legacy-migration rollback. The implementation + program absorbs it into CP-9 as the acceptance bar the new lifecycle + must keep meeting. + +The durable-store implementation itself exists, unlanded, on three +branches (`API-404-durable-store-successor`, `API-404-fsync-wal`, and the +`integration-openrae-current-dev` integration branch): an atomic +`claim_record`, an `AtomicControlPlaneStore` capability protocol with +`commit_terminal_operation` and `reconcile_interrupted_records`, a +flock-based `RuntimeOwnerLease`, fsync-disciplined path helpers, a +compatibility adapter for legacy stores, and a unified `_operation_lock`. +Its shape converges with ADR-104 §§4–5; CP-6 reconciles and re-lands that +work rather than redesigning it. ## 6. Gap summary against ADR-104 | Gap | Evidence | Owning work package | | --- | --- | --- | -| No explicit lifecycle contract; no indeterminate terminal state | `OperationState` in `raes_contracts.runtime_state` lacks it; interrupted work stays `RUNNING` | CP-1 | +| No explicit lifecycle contract; no indeterminate terminal state | `OperationState` lacks it (`ACCEPTED` exists unused); interrupted work stays `RUNNING` | CP-1 | +| Generic path unlocked; two unordered lock domains | no lock in `execute_operation`; `_participant_control_lock` vs the manager's `_participant_execution_lock` | CP-2, CP-4 | | Non-atomic terminal effects on the generic path | `execute_operation` steps 2/4/5 above | CP-2 | | No startup reconciliation | records load unchanged at construction | CP-3 | | Unconditional snapshot/record writes | `save_snapshot`/`save_record` have no expected revision | CP-4 | diff --git a/docs/research/runtime-control-plane/implementation-program.md b/docs/research/runtime-control-plane/implementation-program.md index 04a8da5ad..d3fa42ba1 100644 --- a/docs/research/runtime-control-plane/implementation-program.md +++ b/docs/research/runtime-control-plane/implementation-program.md @@ -66,8 +66,11 @@ enumerate every legal and illegal transition. Depends on: nothing. Rework the generic execution path so the running claim is durable before backend invocation and the terminal transition commits snapshot, terminal record, and audit event in one store transaction, matching the participant -transition path. Verification: kill-point tests at every step boundary show -no observable partial terminal state. Depends on: CP-1. +transition path. Unify in-process locking behind one operation lock — the +generic path is unlocked today and the participant and manager paths hold +two unordered locks. Verification: kill-point tests at every step boundary +show no observable partial terminal state; concurrent in-process +submission tests. Depends on: CP-1. ### CP-3 — Startup reconciliation @@ -94,10 +97,15 @@ CP-4). ### CP-6 — Transactional local store -Re-land the PR #1136 SQLite store under the CP-1/CP-2/CP-4/CP-5 contracts: -WAL admission by returned result, unique idempotency claims, path and -permission hardening, integrity digests, one-time migration with durable -backups. Recovery behavior conforms to CP-3 instead of blanket +Re-land the PR #1136 store work — already implemented across +`API-404-durable-store-successor`, `API-404-fsync-wal`, and +`integration-openrae-current-dev` (atomic `claim_record`, +`AtomicControlPlaneStore` with `commit_terminal_operation` and +`reconcile_interrupted_records`, `RuntimeOwnerLease`, fsync path +discipline, compatibility adapter) — under the CP-1/CP-2/CP-4/CP-5 +contracts: WAL admission by returned result, unique idempotency claims, +path and permission hardening, integrity digests, one-time migration with +durable backups. Recovery behavior conforms to CP-3 instead of blanket interrupted-to-failed conversion. Verification: the absorbed crash-consistency suite plus store-contract conformance. Depends on: CP-1, CP-2, CP-4, CP-5. @@ -114,8 +122,10 @@ stale-cache injection. Depends on: CP-6. Bring the reference HTTP adapter to profile P2: single owning service process, owner-serialized mutations, reads carrying the snapshot revision -they observed, and explicit stale-read rules. Verification: extension of -the #1133 admission suite. Depends on: CP-4, CP-7. +they observed, and explicit stale-read rules. Move the API-408 retrieval +routes off the mutation lock by giving evidence appends their own ordered +path, so reads stop contending with backend mutation latency. Verification: +extension of the #1133 admission suite. Depends on: CP-4, CP-7. ### CP-9 — Crash and profile conformance suite diff --git a/docs/research/runtime-control-plane/requirement-disposition.md b/docs/research/runtime-control-plane/requirement-disposition.md index 7299810fb..200bc1be5 100644 --- a/docs/research/runtime-control-plane/requirement-disposition.md +++ b/docs/research/runtime-control-plane/requirement-disposition.md @@ -15,6 +15,7 @@ requires. Dispositions are one of: **retain**, **change**, **split**, | --- | --- | --- | | `API-404` (ACTIVE, wave 1) | retain, change | Remains the control-plane authority. Work package CP-11 rewrites its traceability to the profiled contract and records which clauses (authenticated access, durable state, idempotent submission, auditable lifecycle) each profile satisfies; P0 explicitly waives durability. | | `SEM-222` (touched by PR #1136) | retain | Snapshot semantic-integrity obligations are unchanged by this design; the P1 store must preserve them through the round-trip codec guard PR #1136 introduced. | +| Adjacent UIDs bound to these surfaces (`API-402`, `API-403`, `API-408`, `RUN-300`, `RUN-304`, `RUN-308`, `RUN-311`, `RUN-316`–`RUN-319`, `SEM-230`, `SEM-233`, `DSL-435`–`DSL-437`, `SEM-204`, `SEM-214`) | retain | Their statements are unchanged. `API-403` already pins the per-target contract that profile P2 keeps; `RUN-319`/`SEM-233` constrain the append-only transition commits the design preserves; `API-408`'s retrieval routes gain a read path off the mutation lock under CP-8 without a statement change. | ## Code surfaces @@ -35,7 +36,7 @@ requires. Dispositions are one of: **retain**, **change**, **split**, | Item | Disposition | Detail | | --- | --- | --- | | Issue #1092 | supersede | Its problem statement is absorbed by ADR-104 §§3–5 and its remedy is re-scoped into CP-1 through CP-7. Close #1092 when those work packages are filed, linking each residual gap from its integration-review addendum to the package that owns it. | -| PR #1136 | split, adopt | Deferred by the maintainer pending this design. Re-land as CP-6 (store), with its recovery behavior reworked under CP-3 and its crash-consistency suite absorbed into CP-9. The PR's preflight note (`issue-1092-local-control-plane-durability-preflight.md`) lands with CP-6 as the store-level design record, subordinate to ADR-104. | +| PR #1136 and successor branches (`API-404-durable-store-successor`, `API-404-fsync-wal`, `integration-openrae-current-dev`) | split, adopt | Deferred by the maintainer pending this design. The implemented atomic `claim_record`, `AtomicControlPlaneStore` (`commit_terminal_operation`, `reconcile_interrupted_records`), `RuntimeOwnerLease`, fsync path discipline, compatibility adapter, and unified `_operation_lock` re-land as CP-6 (with CP-2/CP-5 contracts), recovery reworked under CP-3's classification, and the crash-consistency suite absorbed into CP-9. The PR's preflight note (`issue-1092-local-control-plane-durability-preflight.md`) lands with CP-6 as the store-level design record, subordinate to ADR-104. | | Issue #8 (API-404 tracking) | retain | Gains the profile-scoped acceptance framing through CP-11. | ## Documentation and test surfaces @@ -43,5 +44,5 @@ requires. Dispositions are one of: **retain**, **change**, **split**, | Surface | Disposition | Detail | | --- | --- | --- | | `docs/explain/sdl/runtime-architecture.md` | change | Gains the profile model and the operation-lifecycle description once CP-10 lands; until then it must not claim durability or multi-process support. | -| Crash-consistency tests from PR #1136 (115 tests) | adopt | Absorbed into CP-9's profile conformance suite with kill-point injection at every commit boundary of the new lifecycle. | +| `test_issue_1092_control_plane_crash_consistency.py` (successor branches, ~1,250 lines) | adopt | Absorbed into CP-9's profile conformance suite with kill-point injection at every commit boundary of the new lifecycle; it is the acceptance bar the re-landed store must keep meeting. | | `test_http_api_admission` / offload surfaces (#1133) | retain | P2 admission behavior is unchanged by this design; CP-8 extends the same suite. | diff --git a/tools/policy/historical_identity_records.json b/tools/policy/historical_identity_records.json index 911f87912..b9ce1aab7 100644 --- a/tools/policy/historical_identity_records.json +++ b/tools/policy/historical_identity_records.json @@ -495,7 +495,7 @@ "record_class": "historical-index", "rationale": "Indexes immutable pre-cutover ADR titles, paths, pins, and amendment summaries without making them current identity surfaces.", "occurrences": 4, - "content_sha256": "426c29ffec3274a91e60053e83542f04091b41b863d62cf35b95035030b86d78" + "content_sha256": "63c39b328abbe9974b18e0c414c5650e9be8521bbf51926efae7650af8f96d22" }, { "path": "docs/decisions/cage-2-replication-design.md",