From bea0fbb8e3776ff603153a7746ec4efa2dedff09 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Tue, 21 Jul 2026 08:41:31 -0700 Subject: [PATCH 01/28] docs(rfc-0013): specify recovery lifecycle sidecars Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ad92a900-c04a-45b7-9276-3765ae97b1d3 --- rfcs/0013-cloud-serializable-sqlite-state.md | 47 ++++ rfcs/0013/portable-handoff-v1-spec.md | 181 +++++++++++++++ .../0013/recovery-point-components-v1-spec.md | 211 +++++++++++++++++ rfcs/0013/restored-admission-v1-spec.md | 215 ++++++++++++++++++ 4 files changed, 654 insertions(+) create mode 100644 rfcs/0013/portable-handoff-v1-spec.md create mode 100644 rfcs/0013/recovery-point-components-v1-spec.md create mode 100644 rfcs/0013/restored-admission-v1-spec.md diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index efbd97ec..8644f102 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -55,6 +55,7 @@ The first landed implementation uses SQLite `VACUUM INTO` to capture committed W - This RFC does not require OpenClaw to own upload, tenant routing, retention, or encryption policy. - This RFC does not require hot writes over a network filesystem. - This RFC does not define WAL bundles, leases, promotion, fencing, or standby orchestration. +- This RFC does not define another Gateway pause, drain, or suspension API. - This RFC does not change `openclaw backup create` archive behavior. ## Proposal @@ -172,6 +173,39 @@ flowchart LR The diagram is a responsibility split. The default local runtime can ignore the host box entirely. Hosted deployments can use the snapshot directory as the sync boundary without copying live SQLite sidecars. +### Optional Follow-On Composition + +RFC 0013 is the completed one-database artifact contract. It is also the +owner-authored substrate for optional recovery workflows, but those workflows +must compose the landed command rather than reinterpret live SQLite files or +duplicate snapshot creation, verification, repository, or restore behavior. + +The draft implementer-facing follow-on contracts are: + +- [Recovery Point Components v1](0013/recovery-point-components-v1-spec.md): + compose verified SQLite snapshots with explicit non-SQLite owner artifacts + and external or reconstruction obligations. +- [Portable Handoff v1](0013/portable-handoff-v1-spec.md): combine the existing + cooperative Gateway suspension fence with final owner capture, durable host + acceptance, and generation-bound source destruction authority. +- [Restored Admission v1](0013/restored-admission-v1-spec.md): restore exact + accepted components into fresh paths and keep admission closed until + scheduler and required owner readiness complete. + +These sidecars do not change `openclaw backup sqlite`. They do not make every +ordinary snapshot a portable recovery point, add a continuity-specific storage +provider, or make Lobster part of the core contract. + +OpenClaw `main` already provides the host-neutral +`gateway.suspend.prepare|status|resume` contract from +[openclaw/openclaw#103618](https://github.com/openclaw/openclaw/pull/103618), +with the validation and import-boundary repair from +[openclaw/openclaw#103925](https://github.com/openclaw/openclaw/pull/103925). +A follow-on handoff must reuse that cooperative tracked-work fence. It must not +introduce another Gateway pause API. The existing contract intentionally leaves +external ingress, third-party Channel transports, unregistered background work, +and full process/filesystem consistency to the host. + ### Snapshot Semantics The unit of snapshotting is one existing OpenClaw-owned SQLite database: @@ -267,6 +301,16 @@ The original contributor prototype was [openclaw/openclaw#94805](https://github. The stress harness remains tracked separately in [openclaw/openclaw#94967](https://github.com/openclaw/openclaw/pull/94967). Broader state ownership and continuity work remains related to [openclaw/openclaw#101290](https://github.com/openclaw/openclaw/issues/101290). +Adjacent shipped lifecycle foundations are: + +- [openclaw/openclaw#103618](https://github.com/openclaw/openclaw/pull/103618), + which added cooperative host suspension; and +- [openclaw/openclaw#103925](https://github.com/openclaw/openclaw/pull/103925), + which restored its architecture and validation gates. + +The optional follow-on sidecars consume those contracts as implemented on +current `main`. + ## Rationale This approach solves the reliability problem at the correct boundary. SQLite remains local and authoritative while OpenClaw is running. Durability is handled by verified artifacts, manifests, and explicit restore procedures. @@ -288,3 +332,6 @@ Deferring WAL bundles is intentional. Full snapshots provide the first correct r - restore-on-boot host integration - leases, promotion, fencing, and managed failover - dedicated snapshot targets for future owner stores + +The three RFC 0013 sidecars narrow the first portable follow-on without +promoting those future items into this completed SQLite contract. diff --git a/rfcs/0013/portable-handoff-v1-spec.md b/rfcs/0013/portable-handoff-v1-spec.md new file mode 100644 index 00000000..ee12721c --- /dev/null +++ b/rfcs/0013/portable-handoff-v1-spec.md @@ -0,0 +1,181 @@ +# Portable Handoff v1 Specification + +This document is the implementer-facing planned-handoff specification for RFC +0013 follow-on recovery points. It composes the shipped cooperative Gateway +suspension contract with final owner capture and durable host acceptance. + +Status: draft, tied to RFC 0013. + +## Scope + +This specification defines: + +- host ingress fencing before planned handoff; +- reuse of `gateway.suspend.prepare|status|resume`; +- final capture after the tracked Gateway work fence is ready; +- immutable host acceptance of one aggregate recovery point; +- exact replay after transport or coordinator uncertainty; +- generation-bound source destruction authority; +- hold and quarantine behavior. + +This specification does not define: + +- another Gateway pause, drain, or suspension API; +- storage-provider plugin registration; +- object-store, filesystem, sidecar, or queue implementation; +- retention, placement, encryption-key, or idle policy; +- forced-termination recovery beyond the last accepted recovery point; +- restored admission or Elastic wake. + +## Shipped Suspension Contract + +Current OpenClaw `main` provides: + +```text +gateway.suspend.prepare +gateway.suspend.status +gateway.suspend.resume +``` + +The contract was introduced by +[openclaw/openclaw#103618](https://github.com/openclaw/openclaw/pull/103618) +and its architecture and validation gates were restored by +[openclaw/openclaw#103925](https://github.com/openclaw/openclaw/pull/103925). + +`prepare`: + +- accepts a stable host operation ID; +- closes new tracked root, session, and command admission; +- pauses automatic cron scheduling; +- synchronously inspects tracked active work; +- returns `busy` and reopens normal operation when work remains; or +- returns a renewable two-minute suspension lease when ready. + +The lease is cooperative and refuse-only. It does not: + +- persist external ingress; +- stop third-party Channel transports; +- prove unregistered plugin or background work idle; +- stop the process tree; +- snapshot the filesystem; or +- authorize source destruction. + +The handoff binding must preserve those boundaries. + +## Planned Handoff + +The host executes: + +```text +fence host-controlled ingress + -> call gateway.suspend.prepare with stable handoff identity + -> retry busy without stopping the runtime + -> receive ready suspension lease + -> stop the Gateway and remaining authoritative writers cleanly + -> create and verify final owner components + -> assemble and verify one aggregate recovery point + -> durably accept exact recovery-point bytes and identity + -> finalize the source generation + -> derive safeToDestroy +``` + +If the handoff is abandoned before process stop, the controller calls +`gateway.suspend.resume` with the exact suspension ID. A lost control path does +not authorize progress; the host waits for lease expiry or proves the process +is already stopped through its existing supervisor contract. + +The host must complete clean Gateway stop before the ready lease expires or +renew the lease by repeating `gateway.suspend.prepare` with the same stable +request ID while the Gateway remains responsive. `gateway.suspend.status` does +not renew the lease. If the host cannot complete or renew the clean-stop window, +it abandons the planned handoff, resumes when possible, and leaves source +compute intact. After the process stops, the host must keep every supervisor, +health-recovery, and autoscaling start path fenced until handoff completes or +is explicitly abandoned. + +The host must not extend the suspension lease semantics by treating a stale +lease, health probe, process absence, or local snapshot path as handoff success. + +## Final Capture + +Final capture reuses owner primitives: + +- RFC 0013 `backup sqlite` for global and per-agent databases; +- the Recovery Point Components v1 contract for aggregate composition; and +- existing owner-specific capture for non-SQLite components. + +Final capture must not run while an authoritative writer can still mutate the +captured source. A force kill, uncertain process termination, conventional +shutdown warning, capture blocker, verification failure, or unknown writer +state blocks the planned handoff. Recovery after such a failure uses the last +previously accepted recovery point and does not claim a new clean final point. + +The final recovery point is immutable. Retrying the same handoff identity must +return the same recovery-point identity or a typed conflict. + +## Host Acceptance + +The host durably records: + +- logical runtime and continuity lineage; +- source runtime generation; +- handoff identity; +- aggregate recovery-point identity and manifest digest; +- exact accepted byte count and digest; +- durability class; +- accepted-at time; and +- storage-owned opaque reference. + +The opaque reference may identify a file, blob, object, or service record. It +must not expose credentials in OpenClaw metadata. + +Local artifact creation and host acceptance are separate facts. A host may +claim acceptance only after its selected durability boundary confirms the +exact immutable bytes. + +## Replay And Uncertainty + +Every mutating step is idempotent under the handoff identity. + +- Lost response after capture: replay returns the same local recovery point. +- Lost response after host acceptance: replay queries the same acceptance + operation and returns the same receipt. +- Different bytes under the same identity: quarantine. +- Unknown storage outcome: hold; do not capture a replacement and infer the + first attempt failed. +- Source process or adapter replacement before local state is durable: + quarantine unless the host can prove the exact accepted recovery point. + +## Destruction Authority + +`safeToDestroy` is a host-owned durable conclusion bound to: + +- runtime lineage; +- source generation; +- handoff identity; +- accepted recovery point; and +- the current lifecycle revision. + +It authorizes removal of only the source compute generation. It never +authorizes deletion of recovery points, persistent tenant data, external +credentials, registry records, or another generation. + +New retained work or an operator cancellation must revoke or race with +destruction through the host's durable lifecycle authority. OpenClaw does not +poll for that race after it has stopped. + +## Conformance + +V1 conformance must prove: + +- current-main `gateway.suspend.*` is called rather than reimplemented; +- busy leaves the runtime running; +- ready closes tracked admission and pauses scheduling; +- lease renewal repeats `prepare` with the same request ID; +- host ingress is fenced separately; +- third-party and unregistered work limitations remain explicit; +- final capture occurs only after authoritative writers stop; +- one exact recovery point is accepted durably; +- response loss replays the same acceptance; +- digest conflict and unknown outcome block destruction; and +- `safeToDestroy` is generation-scoped and cannot purge persistent data. diff --git a/rfcs/0013/recovery-point-components-v1-spec.md b/rfcs/0013/recovery-point-components-v1-spec.md new file mode 100644 index 00000000..a1b03b4e --- /dev/null +++ b/rfcs/0013/recovery-point-components-v1-spec.md @@ -0,0 +1,211 @@ +# Recovery Point Components v1 Specification + +This document is the implementer-facing component-composition specification for +RFC 0013, SQLite Snapshot Backup Artifacts. RFC 0013 defines one verified +SQLite snapshot directory. This sidecar defines how a recovery workflow may +compose those directories with other owner-authored artifacts without changing +the SQLite snapshot contract. + +Status: draft, tied to RFC 0013. + +## Scope + +This specification defines: + +- one aggregate recovery point made from immutable owner components; +- use of RFC 0013 global and per-agent snapshot directories as SQLite + components; +- explicit non-SQLite owner components; +- captured, external, reconstructed, and ephemeral obligations; +- exact component identity, ordering, compatibility, and verification; +- fail-closed aggregate manifest parsing and conformance. + +This specification does not define: + +- another SQLite snapshot command, repository, or manifest; +- mutation of an RFC 0013 snapshot after publication; +- upload, storage transport, retention, or encryption implementation; +- Gateway suspension, final handoff, or source destruction; +- restore-on-boot, restored admission, hibernation, or wake; +- arbitrary restore hooks or a generic capture-provider registry. + +## Shipped Foundation + +An RFC 0013 component is a completed directory produced and verified by: + +```text +openclaw backup sqlite create +openclaw backup sqlite verify +openclaw backup sqlite restore +``` + +The aggregate layer must treat the directory and its strict `manifest.json` and +`database.sqlite` bytes as immutable. It must not: + +- copy a live `.sqlite`, `-wal`, `-shm`, or `-journal` file; +- reimplement `VACUUM INTO`; +- weaken role, owner, schema, index, ACL, DACL, link, or race validation; +- claim that a locally present path is durable host acceptance; or +- treat a successful SQLite snapshot as complete OpenClaw recovery state. + +## Ownership + +| Owner | Authority | +| --- | --- | +| SQLite owner | RFC 0013 capture, manifest, verification, and fresh restore. | +| Non-SQLite owner | Artifact contents, compatibility, and restore semantics for its state. | +| Recovery-point composer | Required-component inventory, exact identities, dependency order, and obligation closure. | +| Host or operator | Durable destination, encryption, access control, retention, and recovery-point selection. | + +The composer does not inspect private SQLite schema to infer application state. +It consumes the verified owner result. + +## Component Model + +A recovery point contains a closed, ordered component list. V1 component kinds +are: + +```text +sqlite-global +sqlite-agent +config +workspace +plugin-source +``` + +The SQLite kinds reference RFC 0013 snapshot directories. The other kinds must +be produced by their existing semantic owner from bounded regular-file +inventories. V1 does not allow third-party component registration. + +| Kind | Semantic owner | Eligibility | +| --- | --- | --- | +| `sqlite-global` | OpenClaw shared-state database owner | Eligible through RFC 0013. | +| `sqlite-agent` | OpenClaw per-agent database owner | Eligible through RFC 0013. | +| `config` | OpenClaw config loader and include resolver | Reserved until the owner exposes bounded capture and restore verification. | +| `workspace` | OpenClaw workspace owner | Reserved until the owner exposes bounded capture and restore verification. | +| `plugin-source` | OpenClaw plugin installation owner | Reserved until the owner exposes bounded capture and restore verification. | + +Reserved kinds are specified so the aggregate contract has stable ownership. +They are not eligible components until their named owner contract exists and +passes this specification's verification requirements. A host must not invent +path-copy behavior to make a reserved kind appear implemented. + +Every component records: + +- stable component ID; +- kind and owner; +- immutable artifact digest and size; +- owner manifest digest when the component has its own manifest; +- compatibility identity; +- dependency IDs; +- capture time; +- required or optional status. + +Paths are materialization details, not component identity. + +## Aggregate Manifest + +Illustrative V1 shape: + +```json +{ + "version": "openclaw-recovery-point/v1", + "recoveryPointId": "recovery-point-42", + "createdAt": "2026-07-21T15:00:00.000Z", + "components": [ + { + "id": "sqlite/global", + "kind": "sqlite-global", + "owner": "openclaw-state", + "artifactSha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "artifactSizeBytes": 1048576, + "ownerManifestSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "compatibility": "openclaw-state-schema/7", + "dependsOn": [], + "required": true + } + ], + "obligations": { + "external": [], + "reconstructed": [], + "ephemeral": [] + } +} +``` + +The exact serialization must be deterministic. `recoveryPointId` is the +lowercase hexadecimal SHA-256 digest of the canonical UTF-8 JSON serialization +of the aggregate manifest with the `recoveryPointId` field omitted. Canonical +JSON sorts object keys recursively, preserves array order, and emits no +insignificant whitespace. + +Unknown required fields, +unknown component kinds, duplicate IDs, missing dependencies, dependency +cycles, digest mismatches, and unsupported major versions fail closed. + +The aggregate manifest does not copy the complete RFC 0013 manifest. It binds +that owner manifest by digest and preserves it beside the component artifact. + +## Portability And Sensitive State + +An ordinary RFC 0013 snapshot can contain auth profiles, session state, plugin +state, and credentials-adjacent records. This sidecar does not silently remove +or rewrite those rows. + +A deployment may call a recovery point portable only when it contains no +reissuable credential bytes and every sensitive surface has one explicit +treatment: + +- **captured**: the owner approves non-reissuable runtime-owned bytes for the + selected host protection domain; +- **external**: the destination re-resolves the value from its existing owner; +- **reconstructed**: the owner establishes a safe default or rebuilds from + declared inputs; or +- **ephemeral**: normal startup recreates it and readiness does not depend on + its captured bytes. + +Host-managed or reissuable credentials, OAuth tokens, provider tokens, and +short-lived session credentials must be `external` or omitted. They cannot use +the `captured` treatment in V1. + +Manifest obligations carry stable owner and treatment identifiers, counts, and +readiness requirements. They must not carry secret values, credential bytes, +raw prompts, message payloads, or arbitrary commands. + +If a credential-free portable projection is required, its state owner must +define and verify that projection as a separate operation. It must not change +ordinary `backup sqlite` behavior or mutate the live database. + +## Verification + +Aggregate verification must: + +1. parse the closed manifest; +2. verify deterministic identity; +3. verify every component digest and size from content-pinned reads; +4. invoke the owner verifier for every required component; +5. validate dependency ordering and compatibility; +6. validate every obligation against the supported owner/treatment pairs; and +7. return one exact verified recovery-point identity. + +Verification failure must not create restore targets or return a +success-shaped partial result. + +## Conformance + +V1 conformance must prove: + +- one global and one per-agent RFC 0013 component; +- deterministic aggregate identity; +- exact owner-manifest binding; +- canonical recovery-point identity; +- rejection of missing, duplicate, unknown, corrupt, or reordered components; +- rejection of dependency cycles and unsupported compatibility; +- explicit sensitive-state treatment and rejection of reissuable credential + bytes; +- no secret values in aggregate metadata; and +- no duplicate SQLite capture or verification implementation. + +The first reserved non-SQLite kind gains its own conformance cases only after +its named semantic owner exposes the bounded capture and restore-verification +contract required by this specification. diff --git a/rfcs/0013/restored-admission-v1-spec.md b/rfcs/0013/restored-admission-v1-spec.md new file mode 100644 index 00000000..51073f5d --- /dev/null +++ b/rfcs/0013/restored-admission-v1-spec.md @@ -0,0 +1,215 @@ +# Restored Admission v1 Specification + +This document is the implementer-facing restore and admission specification for +accepted RFC 0013 follow-on recovery points. It defines the ordering and +evidence required before a restored Gateway can receive work. + +Status: draft, tied to RFC 0013. + +## Scope + +This specification defines: + +- a durable restore hold before target mutation or process start; +- exact accepted recovery-point selection; +- fresh-path component restore and owner verification; +- external and reconstructed obligation handling; +- scheduler reconciliation; +- required owner and generic Gateway readiness; +- one-time restored admission; +- same-child replay after coordinator failure. + +This specification does not define: + +- another SQLite restore implementation; +- a public generic restore-hook or readiness-provider registry; +- capture, host storage, retention, or source destruction; +- host placement, proxy transport, or Channel delivery; +- Elastic wake policy. + +This specification applies to any recovery point previously accepted through a +conforming durability boundary. That includes restore of the latest accepted +point after forced source loss. It does not create an accepted recovery point, +define the forced-loss RPO, or permit restore from an unaccepted local path. + +## Shipped Foundation + +SQLite components restore through RFC 0013: + +```text +openclaw backup sqlite verify +openclaw backup sqlite restore --target +``` + +The restore path must preserve RFC 0013's content-pinned verification, +fresh-target requirement, stale-sidecar rejection, and owner validation. It +must not copy snapshot bytes directly into an existing live database. + +The host may use current-main `gateway.suspend.*` for planned source handoff. +Restored admission does not add or modify that suspension API. + +## Restore Hold + +Before creating any original target or starting a Gateway, the host acquires a +durable hold bound to: + +- logical runtime and continuity lineage; +- lifecycle owner generation; +- destination runtime generation; +- accepted recovery-point identity; +- aggregate manifest digest; +- restore operation identity; and +- destination owner. + +Every launcher, health recovery, restart, warm-up, and autoscaling path must +reject ordinary startup while the hold is active. + +The hold is released only by: + +- exact restored-admission completion; +- explicit operator rollback before any restored process becomes runnable; or +- quarantine and a separately authorized recovery action. + +Timeout does not silently release a partially applied restore. + +Quarantine has no automatic exit in V1. A lifecycle owner may exit only through +an explicit durable revocation record that binds the quarantined hold and +operation identity. A later attempt uses a new restore-operation identity and a +newly selected accepted recovery point. A newer attempt must not implicitly +supersede, delete, or reuse the quarantined hold or its evidence. + +## Restore Ordering + +The binding executes: + +```text +verify accepted aggregate manifest + -> verify every required component and compatibility identity + -> create fresh owner-private destination roots + -> restore components in declared dependency order + -> resolve external obligations through existing owners + -> perform declared reconstruction through existing owners + -> start one restored Gateway with private restore evidence + -> reconcile scheduler state + -> satisfy required owner readiness + -> satisfy generic Gateway readiness + -> durably record restored-ready evidence + -> consume that exact evidence to open admission +``` + +The exact private transport into the restored process is an implementation +binding, not a new public RFC 0013 command. It must be bounded, owner-private, +versioned, and absent during ordinary startup. + +## Required Evidence + +The restored-start binding must preserve: + +- accepted recovery-point and aggregate manifest identity; +- destination runtime generation; +- lifecycle owner generation; +- component restore receipt identities; +- scheduler reconciliation identity; +- required owner-readiness identities; +- Gateway incarnation identity; +- one admission identity; and +- one canonical readiness generation. + +These identities are independent. A process ID, worker name, health probe, +container generation, or local path cannot substitute for them. + +The durable completion record contains identities and bounded disposition +metadata only. It must not contain credential values, raw artifact locations, +message payloads, prompts, or arbitrary commands. + +## Scheduler And Owner Readiness + +OpenClaw remains authoritative for: + +- restored scheduler definitions; +- due and missed-run reconciliation; +- duplicate suppression; +- catch-up policy; +- earliest semantic deadline; and +- readiness to admit ordinary work. + +The host must not parse cron expressions or synthesize due work. + +External or reconstructed obligations are evaluated by their existing owners. +Examples include credential availability and plugin dependency reconstruction. +Missing required evidence holds readiness closed. An obligation classified as +not required must be decided by its owner from restored effective +configuration, not by the host. + +## Admission + +Restore completion, process startup, `/healthz`, container readiness, and a +successful SQLite open do not independently authorize work. + +Admission opens only after one canonical restored-ready record binds: + +- the accepted recovery point; +- destination and lifecycle generations; +- component restore receipts; +- scheduler reconciliation; +- required owner readiness; +- Gateway readiness; and +- the one-time admission identity. + +Ordinary startup remains unchanged because it has no restored-start evidence. +Ordinary startup cannot consume a committed restore hold, and restored startup +cannot use an ordinary admission path. + +## Crash Replay + +The required crash boundary is: + +```text +restored-ready durably recorded + -> coordinator exits before completing its host operation +``` + +A fresh coordinator must: + +- resolve the same destination child; +- establish a fresh process-local control or proxy connection; +- read the retained restored-ready evidence; +- verify the same readiness generation; and +- complete without repeating destination allocation, preparation, component + restore, scheduler reconciliation, or admission. + +If the child, generation, recovery point, or retained evidence differs, the +operation conflicts or quarantines. It must not allocate a second winner while +the first can still be authoritative. + +All persistent journals and paths are scoped by stable operation identity. +Fixed shared paths across independent attempts are invalid. + +## Failure Dispositions + +- **retry same incarnation**: transient response loss with the same child and + exact retained evidence; +- **hold**: missing dependency, temporary owner unavailability, or a timeout + before any destination target mutation; +- **quarantine**: corrupt artifact, identity conflict, stale generation, + contradictory replay, or a timeout or partially applied state after target + mutation when the winner is uncertain. + +No failure opens admission or emits a success-shaped ready result. + +## Conformance + +V1 conformance must prove: + +- RFC 0013 verify and fresh restore are reused; +- ordinary startup is unchanged; +- the restore hold fences every start path; +- no target mutation occurs before aggregate verification; +- component dependency order is preserved; +- external and reconstructed obligations remain owner-evaluated; +- scheduler reconciliation precedes readiness; +- admission opens exactly once from exact durable evidence; +- process health alone cannot open admission; +- coordinator crash replay reuses the same child and readiness generation; +- preparation and restore execute exactly once; and +- stale, contradictory, corrupt, and fixed-path collision cases fail closed. From 0c129ae68b7b4624a2b31a664bffaf3479ddface Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Wed, 22 Jul 2026 06:49:59 -0700 Subject: [PATCH 02/28] docs(rfc-0013): close recovery-point boundaries --- rfcs/0013/portable-handoff-v1-spec.md | 22 +++++- .../0013/recovery-point-components-v1-spec.md | 71 +++++++++++++++---- rfcs/0013/restored-admission-v1-spec.md | 3 + 3 files changed, 81 insertions(+), 15 deletions(-) diff --git a/rfcs/0013/portable-handoff-v1-spec.md b/rfcs/0013/portable-handoff-v1-spec.md index ee12721c..7c081e47 100644 --- a/rfcs/0013/portable-handoff-v1-spec.md +++ b/rfcs/0013/portable-handoff-v1-spec.md @@ -104,6 +104,13 @@ Final capture reuses owner primitives: - the Recovery Point Components v1 contract for aggregate composition; and - existing owner-specific capture for non-SQLite components. +An aggregate made directly from ordinary RFC 0013 snapshots remains +`host-protected`. It may move only into a host protection domain whose +encryption, access control, and credential exposure policy accepts those exact +bytes. A deployment must not call the point credential-free portable unless it +also verifies owner-authored portability receipts bound to every component and +artifact digest. An obligation list alone is not portability evidence. + Final capture must not run while an authoritative writer can still mutate the captured source. A force kill, uncertain process termination, conventional shutdown warning, capture blocker, verification failure, or unknown writer @@ -121,7 +128,10 @@ The host durably records: - source runtime generation; - handoff identity; - aggregate recovery-point identity and manifest digest; -- exact accepted byte count and digest; +- acceptance-set identity; +- exact aggregate-manifest digest and size; +- the canonical component acceptance inventory, including each owner-manifest + and artifact digest and size; - durability class; - accepted-at time; and - storage-owned opaque reference. @@ -130,8 +140,11 @@ The opaque reference may identify a file, blob, object, or service record. It must not expose credentials in OpenClaw metadata. Local artifact creation and host acceptance are separate facts. A host may -claim acceptance only after its selected durability boundary confirms the -exact immutable bytes. +claim acceptance only after its selected durability boundary confirms every +byte sequence in the closed acceptance inventory. The storage implementation +may package those bytes as files, blobs, or service records, but must return +the same canonical acceptance-set identity. No digest of an unspecified +directory, archive, or concatenation is valid evidence. ## Replay And Uncertainty @@ -176,6 +189,9 @@ V1 conformance must prove: - third-party and unregistered work limitations remain explicit; - final capture occurs only after authoritative writers stop; - one exact recovery point is accepted durably; +- the accepted logical byte set is closed and deterministically identified; +- host-protected snapshots cannot be reported as credential-free portable + without exact owner portability receipts; - response loss replays the same acceptance; - digest conflict and unknown outcome block destruction; and - `safeToDestroy` is generation-scoped and cannot purge persistent data. diff --git a/rfcs/0013/recovery-point-components-v1-spec.md b/rfcs/0013/recovery-point-components-v1-spec.md index a1b03b4e..92707bca 100644 --- a/rfcs/0013/recovery-point-components-v1-spec.md +++ b/rfcs/0013/recovery-point-components-v1-spec.md @@ -16,7 +16,10 @@ This specification defines: - use of RFC 0013 global and per-agent snapshot directories as SQLite components; - explicit non-SQLite owner components; +- a closed required-component inventory supplied by the runtime state owner; - captured, external, reconstructed, and ephemeral obligations; +- an explicit protection classification that cannot imply credential-free + portability; - exact component identity, ordering, compatibility, and verification; - fail-closed aggregate manifest parsing and conformance. @@ -95,7 +98,7 @@ Every component records: - stable component ID; - kind and owner; - immutable artifact digest and size; -- owner manifest digest when the component has its own manifest; +- owner manifest digest and size when the component has its own manifest; - compatibility identity; - dependency IDs; - capture time; @@ -112,6 +115,13 @@ Illustrative V1 shape: "version": "openclaw-recovery-point/v1", "recoveryPointId": "recovery-point-42", "createdAt": "2026-07-21T15:00:00.000Z", + "inventory": { + "version": "openclaw-runtime-sqlite-inventory/v1", + "requiredComponentIds": ["sqlite/global", "sqlite/agent/main"] + }, + "protection": { + "mode": "host-protected" + }, "components": [ { "id": "sqlite/global", @@ -120,6 +130,7 @@ Illustrative V1 shape: "artifactSha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "artifactSizeBytes": 1048576, "ownerManifestSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "ownerManifestSizeBytes": 512, "compatibility": "openclaw-state-schema/7", "dependsOn": [], "required": true @@ -143,6 +154,13 @@ Unknown required fields, unknown component kinds, duplicate IDs, missing dependencies, dependency cycles, digest mismatches, and unsupported major versions fail closed. +The state owner supplies the complete required component IDs for the selected +runtime before composition. V1 requires exactly one global component and the +exact selected set of per-agent components. The inventory is canonicalized, +stored in the manifest, and covered by `recoveryPointId`. Composition fails on +a missing or extra component; successfully composing one agent cannot imply +that every agent owned by the selected runtime was captured. + The aggregate manifest does not copy the complete RFC 0013 manifest. It binds that owner manifest by digest and preserves it beside the component artifact. @@ -150,11 +168,15 @@ that owner manifest by digest and preserves it beside the component artifact. An ordinary RFC 0013 snapshot can contain auth profiles, session state, plugin state, and credentials-adjacent records. This sidecar does not silently remove -or rewrite those rows. +or rewrite those rows. A recovery point composed directly from ordinary RFC +0013 snapshots therefore has `protection.mode = host-protected`. The aggregate +manifest cannot label itself credential-free or portable. -A deployment may call a recovery point portable only when it contains no -reissuable credential bytes and every sensitive surface has one explicit -treatment: +A deployment may call a recovery point credential-free portable only when a +separate owner-authored portability receipt proves that every component is an +approved portable projection, binds the exact component and artifact digest, +names the owner contract and version, and proves every sensitive surface has +one explicit treatment: - **captured**: the owner approves non-reissuable runtime-owned bytes for the selected host protection domain; @@ -168,14 +190,34 @@ Host-managed or reissuable credentials, OAuth tokens, provider tokens, and short-lived session credentials must be `external` or omitted. They cannot use the `captured` treatment in V1. -Manifest obligations carry stable owner and treatment identifiers, counts, and -readiness requirements. They must not carry secret values, credential bytes, -raw prompts, message payloads, or arbitrary commands. +Manifest obligation arrays imply the `external`, `reconstructed`, or +`ephemeral` treatment. Entries carry a closed obligation kind, stable owner, +identifier, and readiness requirement. V1 initially supports only named +SecretRef, plugin-dependency, and runtime-cache owner pairs; unknown owner, +kind, or treatment combinations fail closed. Obligations must not carry secret +values, credential bytes, raw prompts, message payloads, or arbitrary commands. +Captured state is represented by its owner component, not by an obligation +entry. If a credential-free portable projection is required, its state owner must define and verify that projection as a separate operation. It must not change ordinary `backup sqlite` behavior or mutate the live database. +## Acceptance Byte Set + +The recovery point defines a closed logical acceptance byte set without +standardizing a tarball, directory tree, object-store key, or transport: + +1. the canonical aggregate-manifest bytes; +2. each exact owner-manifest byte sequence; and +3. each exact component-artifact byte sequence. + +An acceptance inventory records the aggregate-manifest digest and size plus, +for every canonically ordered component, its component ID, owner-manifest +digest and size, and artifact digest and size. Its identity is the SHA-256 of +the canonical inventory with that identity field omitted. Storage packaging +and paths remain host-owned and are not part of component identity. + ## Verification Aggregate verification must: @@ -184,9 +226,10 @@ Aggregate verification must: 2. verify deterministic identity; 3. verify every component digest and size from content-pinned reads; 4. invoke the owner verifier for every required component; -5. validate dependency ordering and compatibility; -6. validate every obligation against the supported owner/treatment pairs; and -7. return one exact verified recovery-point identity. +5. require the component IDs to equal the closed owner inventory; +6. validate dependency ordering and compatibility; +7. validate every obligation against the supported owner/treatment pairs; and +8. return one exact verified recovery-point identity and acceptance inventory. Verification failure must not create restore targets or return a success-shaped partial result. @@ -195,7 +238,8 @@ success-shaped partial result. V1 conformance must prove: -- one global and one per-agent RFC 0013 component; +- one global and the exact owner-selected per-agent RFC 0013 components; +- rejection of missing and extra components against the bound inventory; - deterministic aggregate identity; - exact owner-manifest binding; - canonical recovery-point identity; @@ -203,6 +247,9 @@ V1 conformance must prove: - rejection of dependency cycles and unsupported compatibility; - explicit sensitive-state treatment and rejection of reissuable credential bytes; +- ordinary snapshot composition remaining `host-protected` until an exact + owner portability receipt exists; +- deterministic acceptance inventory over the closed logical byte set; - no secret values in aggregate metadata; and - no duplicate SQLite capture or verification implementation. diff --git a/rfcs/0013/restored-admission-v1-spec.md b/rfcs/0013/restored-admission-v1-spec.md index 51073f5d..f1f7d96c 100644 --- a/rfcs/0013/restored-admission-v1-spec.md +++ b/rfcs/0013/restored-admission-v1-spec.md @@ -58,6 +58,7 @@ durable hold bound to: - destination runtime generation; - accepted recovery-point identity; - aggregate manifest digest; +- acceptance-set identity; - restore operation identity; and - destination owner. @@ -106,6 +107,7 @@ versioned, and absent during ordinary startup. The restored-start binding must preserve: - accepted recovery-point and aggregate manifest identity; +- exact acceptance-set identity; - destination runtime generation; - lifecycle owner generation; - component restore receipt identities; @@ -149,6 +151,7 @@ successful SQLite open do not independently authorize work. Admission opens only after one canonical restored-ready record binds: - the accepted recovery point; +- the accepted logical byte set; - destination and lifecycle generations; - component restore receipts; - scheduler reconciliation; From 99bd3c29556df1620e201a0ffea74a02d51cc28f Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Wed, 22 Jul 2026 20:10:23 -0700 Subject: [PATCH 03/28] docs(rfc-0013): link implementation evidence --- rfcs/0013-cloud-serializable-sqlite-state.md | 19 ++++++++++++++++++- rfcs/0013/portable-handoff-v1-spec.md | 15 +++++++++++++++ .../0013/recovery-point-components-v1-spec.md | 12 ++++++++++++ 3 files changed, 45 insertions(+), 1 deletion(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 8644f102..d531bfbd 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -3,7 +3,7 @@ title: SQLite Snapshot Backup Artifacts authors: - giodl created: 2026-06-18 -last_updated: 2026-07-21 +last_updated: 2026-07-22 status: completed issue: https://github.com/openclaw/openclaw/pull/105718 rfc_pr: https://github.com/openclaw/rfcs/pull/20 @@ -196,6 +196,23 @@ These sidecars do not change `openclaw backup sqlite`. They do not make every ordinary snapshot a portable recovery point, add a continuity-specific storage provider, or make Lobster part of the core contract. +Draft OpenClaw implementation evidence is available for the two owner-side +slices exercised by these sidecars: + +- [openclaw/openclaw#112385](https://github.com/openclaw/openclaw/pull/112385) + composes verified global and owner-selected per-agent RFC 0013 snapshots into + one deterministic `host-protected` recovery point and exact acceptance byte + inventory. +- [giodl73-repo/openclaw#152](https://github.com/giodl73-repo/openclaw/pull/152) + is a fork-only draft stacked on #112385. It adds one hidden offline final + capture operation with durable intent, exact committed-result replay, and + fail-closed quarantine for conflicting or incomplete attempts. + +These drafts are evidence for owner review, not normative dependencies. They do +not move Gateway suspension, external ingress fencing, clean process shutdown, +durable host acceptance, publication, restore admission, or source destruction +into OpenClaw. + OpenClaw `main` already provides the host-neutral `gateway.suspend.prepare|status|resume` contract from [openclaw/openclaw#103618](https://github.com/openclaw/openclaw/pull/103618), diff --git a/rfcs/0013/portable-handoff-v1-spec.md b/rfcs/0013/portable-handoff-v1-spec.md index 7c081e47..6f586d01 100644 --- a/rfcs/0013/portable-handoff-v1-spec.md +++ b/rfcs/0013/portable-handoff-v1-spec.md @@ -6,6 +6,21 @@ suspension contract with final owner capture and durable host acceptance. Status: draft, tied to RFC 0013. +## Draft Implementation Evidence + +[giodl73-repo/openclaw#152](https://github.com/giodl73-repo/openclaw/pull/152) +is the fork-only OpenClaw evidence slice for final owner capture, stacked on +[openclaw/openclaw#112385](https://github.com/openclaw/openclaw/pull/112385). +It uses the existing RFC 0013 SQLite provider, binds the exact runtime lineage, +source generation, handoff identity, selected-agent inventory, and +host-supplied closure evidence, and durably replays one committed recovery +point after response loss. + +The operation is deliberately offline and host-invoked. It does not suspend or +stop Gateway, fence external ingress, accept or publish bytes, or authorize +source destruction. The pull request is implementation evidence; this +specification remains the normative responsibility boundary. + ## Scope This specification defines: diff --git a/rfcs/0013/recovery-point-components-v1-spec.md b/rfcs/0013/recovery-point-components-v1-spec.md index 92707bca..b8ade2b9 100644 --- a/rfcs/0013/recovery-point-components-v1-spec.md +++ b/rfcs/0013/recovery-point-components-v1-spec.md @@ -8,6 +8,18 @@ the SQLite snapshot contract. Status: draft, tied to RFC 0013. +## Draft Implementation Evidence + +[openclaw/openclaw#112385](https://github.com/openclaw/openclaw/pull/112385) +is the paired draft evidence slice. It implements the V1 global and +owner-selected per-agent SQLite composition path, deterministic recovery-point +and acceptance-set identities, exact owner-manifest and artifact binding, +closed obligation validation, and the `host-protected` classification. + +The pull request is evidence for review of this contract. Its types and file +layout are not normative, and the specification does not require that pull +request to merge before the ownership decisions can be accepted. + ## Scope This specification defines: From 163c569700e569d076f1330a2e138d901c5710ad Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Wed, 22 Jul 2026 20:34:20 -0700 Subject: [PATCH 04/28] docs(rfc-0013): motivate safe scale-to-zero --- rfcs/0013-cloud-serializable-sqlite-state.md | 52 ++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index d531bfbd..0e6aebc3 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -180,6 +180,44 @@ owner-authored substrate for optional recovery workflows, but those workflows must compose the landed command rather than reinterpret live SQLite files or duplicate snapshot creation, verification, repository, or restore behavior. +#### Why a recovery lifecycle is needed + +Per-user and event-driven hosts can stop paying for resident compute only when +they can retire one Gateway generation and later admit work on a replacement +without losing accepted state. A SQLite snapshot alone cannot prove that +transaction. It does not identify the complete set of state owners, prove that +the host durably accepted every required byte, authorize retirement of the +source generation, or prove that the replacement restored the same recovery +point before accepting work. + +Without those semantics, hosts must choose between keeping idle Gateways warm +or maintaining private shutdown, copying, restore-ordering, and readiness +inference paths. Managed-host experience has exposed the resulting failure +classes: accepted ingress can outlive the compute that should process it, +scheduled wake can race idle retirement, replacement can lose scheduler +continuity, and a cold runtime can appear ready before its required owner state +is restored. + +Infrastructure platforms already provide the compute lifecycle primitives: + +- [Fly Machines](https://fly.io/docs/launch/autostop-autostart/) can stop all + Machines and autostart them for traffic. +- [E2B](https://e2b.dev/docs/sandbox/auto-resume) pauses and automatically + resumes persistent sandboxes. +- [Daytona](https://www.daytona.io/docs/en/persistence/) preserves sandbox + files across stop/start and offers archive or VM pause/resume paths. +- [Azure Container Apps](https://learn.microsoft.com/azure/container-apps/scale-app) + scales to zero and wakes from configured event sources. +- [Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/best-practices/websockets/) + hibernate while retaining wakeable WebSocket delivery. + +Those platforms do not know OpenClaw's owner inventory, SQLite invariants, +scheduler state, or readiness boundary. The follow-on contracts define that +application-consistent layer. A host remains responsible for retained ingress, +wake registration, compute placement, and external durability; OpenClaw and +its state owners provide the exact recovery point and restored-admission proof +that make those host primitives safe to use. + The draft implementer-facing follow-on contracts are: - [Recovery Point Components v1](0013/recovery-point-components-v1-spec.md): @@ -213,6 +251,20 @@ not move Gateway suspension, external ingress fencing, clean process shutdown, durable host acceptance, publication, restore admission, or source destruction into OpenClaw. +The intended observable outcome of the complete host composition is: + +- idle compute may reach zero without treating a raw live filesystem copy as a + recovery point; +- accepted ingress remains retained until restored admission succeeds; +- autonomous scheduled work does not require an unrelated user message to + recover from absent compute; +- the source generation is not destroyed before exact durable acceptance; and +- replacement readiness names the accepted recovery point it restored. + +The first two OpenClaw drafts prove only the owner-side recovery-point and final +capture slices. Host acceptance, retained ingress, wake, destruction, and +restored-start integration remain separate review and implementation work. + OpenClaw `main` already provides the host-neutral `gateway.suspend.prepare|status|resume` contract from [openclaw/openclaw#103618](https://github.com/openclaw/openclaw/pull/103618), From b7e9703a31e1800e7f534918763b4a2e4f1ccb3f Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Wed, 22 Jul 2026 20:40:35 -0700 Subject: [PATCH 05/28] docs(rfc-0013): link upstream final capture draft --- rfcs/0013-cloud-serializable-sqlite-state.md | 4 ++-- rfcs/0013/portable-handoff-v1-spec.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 0e6aebc3..c5dd53b8 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -241,8 +241,8 @@ slices exercised by these sidecars: composes verified global and owner-selected per-agent RFC 0013 snapshots into one deterministic `host-protected` recovery point and exact acceptance byte inventory. -- [giodl73-repo/openclaw#152](https://github.com/giodl73-repo/openclaw/pull/152) - is a fork-only draft stacked on #112385. It adds one hidden offline final +- [openclaw/openclaw#112865](https://github.com/openclaw/openclaw/pull/112865) + is a draft stacked on #112385. It adds one hidden offline final capture operation with durable intent, exact committed-result replay, and fail-closed quarantine for conflicting or incomplete attempts. diff --git a/rfcs/0013/portable-handoff-v1-spec.md b/rfcs/0013/portable-handoff-v1-spec.md index 6f586d01..773632d9 100644 --- a/rfcs/0013/portable-handoff-v1-spec.md +++ b/rfcs/0013/portable-handoff-v1-spec.md @@ -8,8 +8,8 @@ Status: draft, tied to RFC 0013. ## Draft Implementation Evidence -[giodl73-repo/openclaw#152](https://github.com/giodl73-repo/openclaw/pull/152) -is the fork-only OpenClaw evidence slice for final owner capture, stacked on +[openclaw/openclaw#112865](https://github.com/openclaw/openclaw/pull/112865) +is the OpenClaw draft evidence slice for final owner capture, stacked on [openclaw/openclaw#112385](https://github.com/openclaw/openclaw/pull/112385). It uses the existing RFC 0013 SQLite provider, binds the exact runtime lineage, source generation, handoff identity, selected-agent inventory, and From d5f62c76b1a8d3e227a70167aa7d490451b84459 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Wed, 22 Jul 2026 21:34:59 -0700 Subject: [PATCH 06/28] docs(rfc-0013): link restored admission evidence --- rfcs/0013-cloud-serializable-sqlite-state.md | 17 +++++++++++------ rfcs/0013/restored-admission-v1-spec.md | 16 ++++++++++++++++ 2 files changed, 27 insertions(+), 6 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index c5dd53b8..92aecbe9 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -234,7 +234,7 @@ These sidecars do not change `openclaw backup sqlite`. They do not make every ordinary snapshot a portable recovery point, add a continuity-specific storage provider, or make Lobster part of the core contract. -Draft OpenClaw implementation evidence is available for the two owner-side +Draft OpenClaw implementation evidence is available for the three owner-side slices exercised by these sidecars: - [openclaw/openclaw#112385](https://github.com/openclaw/openclaw/pull/112385) @@ -245,11 +245,16 @@ slices exercised by these sidecars: is a draft stacked on #112385. It adds one hidden offline final capture operation with durable intent, exact committed-result replay, and fail-closed quarantine for conflicting or incomplete attempts. +- [openclaw/openclaw#112896](https://github.com/openclaw/openclaw/pull/112896) + is a draft stacked on #112865. It restores one exact accepted aggregate to + fresh canonical paths, holds when required owner evidence is absent, and + keeps Gateway work admission closed through scheduler reconciliation and + owner readiness. These drafts are evidence for owner review, not normative dependencies. They do not move Gateway suspension, external ingress fencing, clean process shutdown, -durable host acceptance, publication, restore admission, or source destruction -into OpenClaw. +durable host acceptance, publication, host wake, or source destruction into +OpenClaw. The intended observable outcome of the complete host composition is: @@ -261,9 +266,9 @@ The intended observable outcome of the complete host composition is: - the source generation is not destroyed before exact durable acceptance; and - replacement readiness names the accepted recovery point it restored. -The first two OpenClaw drafts prove only the owner-side recovery-point and final -capture slices. Host acceptance, retained ingress, wake, destruction, and -restored-start integration remain separate review and implementation work. +The three OpenClaw drafts prove only the owner-side recovery-point, final +capture, and restored-admission slices. Host acceptance, retained ingress, +wake, and destruction remain separate review and implementation work. OpenClaw `main` already provides the host-neutral `gateway.suspend.prepare|status|resume` contract from diff --git a/rfcs/0013/restored-admission-v1-spec.md b/rfcs/0013/restored-admission-v1-spec.md index f1f7d96c..b4678892 100644 --- a/rfcs/0013/restored-admission-v1-spec.md +++ b/rfcs/0013/restored-admission-v1-spec.md @@ -6,6 +6,22 @@ evidence required before a restored Gateway can receive work. Status: draft, tied to RFC 0013. +## Draft Implementation Evidence + +[openclaw/openclaw#112896](https://github.com/openclaw/openclaw/pull/112896) +is the OpenClaw draft evidence slice for exact fresh-target restore and +readiness-gated admission, stacked on final-capture draft +[openclaw/openclaw#112865](https://github.com/openclaw/openclaw/pull/112865). +It reuses the RFC 0013 SQLite provider, durably binds the restore receipt and +private startup descriptor, reconciles the canonical scheduler, and opens work +admission only after owner readiness. + +The draft deliberately holds when required external or reconstructed +obligations lack accepted owner evidence. It does not add wake routing, +placement, idle policy, host acceptance, or a public restore-hook registry. +This pull request is implementation evidence; this specification remains the +normative responsibility boundary. + ## Scope This specification defines: From 022ad4e1588320ed436fdf37e28318b3170c8764 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Sun, 26 Jul 2026 07:36:45 -0700 Subject: [PATCH 07/28] docs(rfc-0013): narrow restored admission evidence --- rfcs/0013/restored-admission-v1-spec.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/rfcs/0013/restored-admission-v1-spec.md b/rfcs/0013/restored-admission-v1-spec.md index b4678892..9732f0e2 100644 --- a/rfcs/0013/restored-admission-v1-spec.md +++ b/rfcs/0013/restored-admission-v1-spec.md @@ -127,8 +127,8 @@ The restored-start binding must preserve: - destination runtime generation; - lifecycle owner generation; - component restore receipt identities; -- scheduler reconciliation identity; -- required owner-readiness identities; +- stable, generation-bound scheduler-reconciliation evidence identity; +- normalized required-owner readiness evidence identity; - Gateway incarnation identity; - one admission identity; and - one canonical readiness generation. @@ -140,6 +140,14 @@ The durable completion record contains identities and bounded disposition metadata only. It must not contain credential values, raw artifact locations, message payloads, prompts, or arbitrary commands. +V1 may derive the scheduler and readiness evidence identities only after the +semantic owner operation succeeds. The derivation binds a versioned evidence +label, the exact restore receipt, recovery-point and acceptance-set identities, +the destination runtime generation, and the normalized outcome. It is not a +digest of mutable scheduler status, does not claim to identify the complete +scheduler definition set, and does not substitute for a richer owner-authored +receipt when an owner later exposes one. + ## Scheduler And Owner Readiness OpenClaw remains authoritative for: @@ -170,8 +178,8 @@ Admission opens only after one canonical restored-ready record binds: - the accepted logical byte set; - destination and lifecycle generations; - component restore receipts; -- scheduler reconciliation; -- required owner readiness; +- generation-bound scheduler-reconciliation evidence; +- normalized required-owner readiness evidence; - Gateway readiness; and - the one-time admission identity. From 06812c585a6e6ad1afc62adf3915c61f17d0ce01 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Sun, 26 Jul 2026 09:22:26 -0700 Subject: [PATCH 08/28] docs(rfc-0013): require sqlite recovery journals --- rfcs/0013-cloud-serializable-sqlite-state.md | 9 ++++++--- rfcs/0013/restored-admission-v1-spec.md | 6 ++++++ 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 92aecbe9..3b785d91 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -243,13 +243,16 @@ slices exercised by these sidecars: inventory. - [openclaw/openclaw#112865](https://github.com/openclaw/openclaw/pull/112865) is a draft stacked on #112385. It adds one hidden offline final - capture operation with durable intent, exact committed-result replay, and - fail-closed quarantine for conflicting or incomplete attempts. + capture operation with operation-scoped SQLite intent, exact + committed-result replay, and fail-closed quarantine for conflicting or + incomplete attempts. - [openclaw/openclaw#112896](https://github.com/openclaw/openclaw/pull/112896) is a draft stacked on #112865. It restores one exact accepted aggregate to fresh canonical paths, holds when required owner evidence is absent, and keeps Gateway work admission closed through scheduler reconciliation and - owner readiness. + owner readiness. Restore intent, results, startup descriptors, and ready + evidence share the dedicated SQLite recovery journal rather than JSON + runtime sidecars. These drafts are evidence for owner review, not normative dependencies. They do not move Gateway suspension, external ingress fencing, clean process shutdown, diff --git a/rfcs/0013/restored-admission-v1-spec.md b/rfcs/0013/restored-admission-v1-spec.md index 9732f0e2..33a9cc13 100644 --- a/rfcs/0013/restored-admission-v1-spec.md +++ b/rfcs/0013/restored-admission-v1-spec.md @@ -212,6 +212,12 @@ the first can still be authoritative. All persistent journals and paths are scoped by stable operation identity. Fixed shared paths across independent attempts are invalid. +OpenClaw-owned recovery intent, committed restore results, startup descriptors, +and restored-ready evidence use an operation-scoped SQLite journal. They are +not steady-state JSON sidecars. A dedicated journal is required because final +capture must commit intent before snapshotting the shared state database, and +restore must commit intent before the fresh shared state database exists. + ## Failure Dispositions - **retry same incarnation**: transient response loss with the same child and From 60e9020828a74bb96ac6f49c9d23b2b61059c459 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Sun, 26 Jul 2026 11:02:04 -0700 Subject: [PATCH 09/28] docs: ground scale-to-zero follow-on evidence --- rfcs/0013-cloud-serializable-sqlite-state.md | 43 +++++++++++++++++++- 1 file changed, 42 insertions(+), 1 deletion(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 3b785d91..0201586e 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -3,7 +3,7 @@ title: SQLite Snapshot Backup Artifacts authors: - giodl created: 2026-06-18 -last_updated: 2026-07-22 +last_updated: 2026-07-26 status: completed issue: https://github.com/openclaw/openclaw/pull/105718 rfc_pr: https://github.com/openclaw/rfcs/pull/20 @@ -180,6 +180,25 @@ owner-authored substrate for optional recovery workflows, but those workflows must compose the landed command rather than reinterpret live SQLite files or duplicate snapshot creation, verification, repository, or restore behavior. +The scale-to-zero outcome and user evidence are tracked in +[openclaw/openclaw#114145](https://github.com/openclaw/openclaw/issues/114145). +This follow-on deliberately builds on, rather than replaces, work Vincent and +Peter already added: Vincent's verified SQLite snapshots in +[openclaw/openclaw#105718](https://github.com/openclaw/openclaw/pull/105718), +Peter's cooperative host suspension in +[openclaw/openclaw#103618](https://github.com/openclaw/openclaw/pull/103618), +and Vincent's suspension validation repair in +[openclaw/openclaw#103925](https://github.com/openclaw/openclaw/pull/103925). +It also composes with Josh Lehman's SQLite-backed session and transcript runtime +in [openclaw/openclaw#98236](https://github.com/openclaw/openclaw/pull/98236) +and restart-recovery admission behavior in +[openclaw/openclaw#111869](https://github.com/openclaw/openclaw/pull/111869). + +The three implementation PRs linked below demonstrate one bounded composition; +they are not a required decomposition. If maintainers prefer a smaller or +different core seam that satisfies the same recovery and restored-admission +outcomes, that is a valid resolution of the umbrella issue. + #### Why a recovery lifecycle is needed Per-user and event-driven hosts can stop paying for resident compute only when @@ -198,6 +217,25 @@ scheduled wake can race idle retirement, replacement can lose scheduler continuity, and a cold runtime can appear ready before its required owner state is restored. +Existing reports ground those outcomes in OpenClaw operator needs: + +- [openclaw/openclaw#13616](https://github.com/openclaw/openclaw/issues/13616) + requests unified backup and restore for config, cron, and session state. +- [openclaw/openclaw#63392](https://github.com/openclaw/openclaw/issues/63392) + requests per-agent backup and restore rather than whole-instance rollback. +- [openclaw/openclaw#104412](https://github.com/openclaw/openclaw/issues/104412) + shows recurring cron work can be skipped silently while Gateway is stopped. +- [openclaw/openclaw#101290](https://github.com/openclaw/openclaw/issues/101290) + shows why hosts need owner-safe snapshot boundaries instead of touching live + SQLite state from another process. +- [openclaw/openclaw#107433](https://github.com/openclaw/openclaw/issues/107433) + requests explicit protected and reconstructable-state obligations. + +[openclaw/openclaw#113306](https://github.com/openclaw/openclaw/issues/113306) +tracks adjacent crash-durability and identity hardening in the underlying +SQLite snapshot implementation; the follow-on lifecycle does not claim to fix +that separate issue. + Infrastructure platforms already provide the compute lifecycle primitives: - [Fly Machines](https://fly.io/docs/launch/autostop-autostart/) can stop all @@ -210,6 +248,9 @@ Infrastructure platforms already provide the compute lifecycle primitives: scales to zero and wakes from configured event sources. - [Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/best-practices/websockets/) hibernate while retaining wakeable WebSocket delivery. +- [Modal](https://modal.com/docs/guide/scale) scales Functions to zero by + default and offers + [Sandbox snapshots](https://modal.com/docs/guide/sandbox-snapshots). Those platforms do not know OpenClaw's owner inventory, SQLite invariants, scheduler state, or readiness boundary. The follow-on contracts define that From b6769f73e8378599fbb3f1df7f7203d2299e3cf3 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Sun, 26 Jul 2026 11:17:29 -0700 Subject: [PATCH 10/28] docs: define deferred wake composition --- rfcs/0013-cloud-serializable-sqlite-state.md | 61 ++++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 0201586e..6b4b2276 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -314,6 +314,67 @@ The three OpenClaw drafts prove only the owner-side recovery-point, final capture, and restored-admission slices. Host acceptance, retained ingress, wake, and destruction remain separate review and implementation work. +#### Deferred cron and retained-ingress wake composition + +The current three-PR evidence stack intentionally stops at restored admission. +A complete scale-to-zero host also needs to wake without relying on an +unrelated user request and to retain accepted work while no Gateway process +exists. + +Before source retirement, OpenClaw should produce one owner-authored semantic +wake registration bound to the accepted final recovery point and scheduler +evidence. It contains a nullable `nextRequiredAt` and a bounded reason class, +not cron expressions, job names, prompts, or payloads. The host may subtract a +configured cold-start lead from that timestamp, but it must not decide whether +a cron job is due, invent catch-up work, or suppress duplicates. +The authoritative receipt is produced at final capture and again after restored +reconciliation; it does not require a host callback for every live cron +mutation. + +A retained-ingress owner such as Teams must durably store its payload or opaque +reference and deduplication identity before acknowledging upstream delivery. +That accepted cause revokes an in-progress sleep authorization or joins an +idempotent wake request. Teams messages, API work, semantic deadlines, and +operator requests may coalesce one compute-provisioning attempt, but each owner +retains its independent delivery, retry, and acknowledgement state. + +The host then selects an accepted recovery point, grants one destination +generation, restores through the existing contract, and waits for the exact +restored-admission readiness identity. Only then may retained-work owners use +their existing generation-fenced delivery paths. After wake, OpenClaw remains +authoritative for due and missed-run reconciliation, catch-up policy, duplicate +suppression, and the next semantic deadline; a host alarm is only a provisioning +trigger and must not invoke cron jobs directly. + +Wake callers cannot select a recovery point or destination generation and +cannot supply readiness. Central lifecycle records contain bounded cause and +generation identities only; retained payloads and credentials remain with +their existing owners. + +Unknown provisioning, restore, readiness, or delivery outcomes retain the wake +causes and hold or quarantine. A timeout is not permission to acknowledge work, +start a second authoritative generation, or open admission. + +The bounded follow-on PR plan is: + +1. **OpenClaw semantic wake registration:** add an owner-authored receipt bound + to the final recovery point and scheduler evidence, with nullable + `nextRequiredAt`; reuse the existing cron owner's `nextWakeAtMs` and startup + reconciliation rather than adding another scheduler. +2. **Host wake authority and coalescing:** atomically accept the recovery point, + wake registration, and revocable sleep authority; retain Teams/API causes; + schedule semantic deadlines; and coalesce causes into one idempotent + ensure-runtime-ready operation. +3. **Readiness-gated delivery and replay:** consume the exact restored-admission + result before delivering retained work, preserve per-owner acknowledgement, + and prove replay across a host-process failure with one Teams cause and one + cron deadline sharing one cold start. + +These are review and evidence slices, not a required repository decomposition. +The host slices may be combined if the same ownership, race, replay, and +conformance boundaries remain independently reviewable. OpenClaw does not gain +a Teams transport, compute scheduler, retained-payload store, or placement API. + OpenClaw `main` already provides the host-neutral `gateway.suspend.prepare|status|resume` contract from [openclaw/openclaw#103618](https://github.com/openclaw/openclaw/pull/103618), From 9d2ffb523b521ae418479907e99c1f888269c078 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Sun, 26 Jul 2026 11:25:06 -0700 Subject: [PATCH 11/28] docs: reuse shipped cron projection hooks --- rfcs/0013-cloud-serializable-sqlite-state.md | 40 +++++++++++++------- 1 file changed, 27 insertions(+), 13 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 6b4b2276..4e77adce 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -321,15 +321,25 @@ A complete scale-to-zero host also needs to wake without relying on an unrelated user request and to retain accepted work while no Gateway process exists. -Before source retirement, OpenClaw should produce one owner-authored semantic -wake registration bound to the accepted final recovery point and scheduler -evidence. It contains a nullable `nextRequiredAt` and a bounded reason class, -not cron expressions, job names, prompts, or payloads. The host may subtract a -configured cold-start lead from that timestamp, but it must not decide whether -a cron job is due, invent catch-up work, or suppress duplicates. -The authoritative receipt is produced at final capture and again after restored -reconciliation; it does not require a host callback for every live cron -mutation. +OpenClaw already exposes the owner mechanism Peter added for this job: +post-commit `cron_changed: scheduled` signals in +[openclaw/openclaw#103647](https://github.com/openclaw/openclaw/pull/103647) +and lifecycle-owned `cron_reconciled` snapshots in +[openclaw/openclaw#104368](https://github.com/openclaw/openclaw/pull/104368). +The documented safe external projection pattern uses `cron_reconciled` to adopt +the exact scheduler, treats `cron_changed` only as a coalescible reread hint, +and completes its `replaceAll` callback only after the host durably accepts the +projection. Those callbacks remain tracked root work, so the existing +`gateway.suspend.*` fence cannot report idle while a projection write is still +in flight. + +The host should bind the durably accepted projection revision, accepted final +recovery point, and revocable sleep authority in one transaction. That host +wake registration contains a nullable `nextRequiredAt` and a bounded reason +class, not cron expressions, job names, prompts, or payloads. The host may +subtract a configured cold-start lead from that timestamp, but it must not +decide whether a cron job is due, invent catch-up work, or suppress duplicates. +No continuous callback beyond the shipped reconciliation hooks is required. A retained-ingress owner such as Teams must durably store its payload or opaque reference and deduplication identity before acknowledging upstream delivery. @@ -357,10 +367,11 @@ start a second authoritative generation, or open admission. The bounded follow-on PR plan is: -1. **OpenClaw semantic wake registration:** add an owner-authored receipt bound - to the final recovery point and scheduler evidence, with nullable - `nextRequiredAt`; reuse the existing cron owner's `nextWakeAtMs` and startup - reconciliation rather than adding another scheduler. +1. **Host cron-projection adapter:** implement the documented + `cron_reconciled` plus `cron_changed` projection pattern, durably replace the + complete external wake set, and bind its accepted revision to the final + recovery point and sleep authority. No new OpenClaw core surface is expected + by default. 2. **Host wake authority and coalescing:** atomically accept the recovery point, wake registration, and revocable sleep authority; retain Teams/API causes; schedule semantic deadlines; and coalesce causes into one idempotent @@ -374,6 +385,9 @@ These are review and evidence slices, not a required repository decomposition. The host slices may be combined if the same ownership, race, replay, and conformance boundaries remain independently reviewable. OpenClaw does not gain a Teams transport, compute scheduler, retained-payload store, or placement API. +If end-to-end proof exposes an invariant the shipped hooks and suspension fence +cannot express, the follow-up should be the smallest extension of those owner +seams rather than a parallel continuity scheduler or wake API. OpenClaw `main` already provides the host-neutral `gateway.suspend.prepare|status|resume` contract from From efea5da99577c24651c77b2383a8e88f039046aa Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Sun, 26 Jul 2026 11:51:00 -0700 Subject: [PATCH 12/28] docs: link host wake authority evidence --- rfcs/0013-cloud-serializable-sqlite-state.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 4e77adce..1b2f1c95 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -371,7 +371,12 @@ The bounded follow-on PR plan is: `cron_reconciled` plus `cron_changed` projection pattern, durably replace the complete external wake set, and bind its accepted revision to the final recovery point and sleep authority. No new OpenClaw core surface is expected - by default. + by default. The first host-side authority evidence is the fork-only draft + [giodl/lobster#38](https://microsoft.ghe.com/giodl/lobster/pull/38) + (Microsoft GHE access required); it binds a content-addressed projection to + final acceptance and destruction authority. Permanent `replaceAll` adoption + and temporary scheduler rehydration remain follow-on proof, so that draft + does not yet claim the complete adapter. 2. **Host wake authority and coalescing:** atomically accept the recovery point, wake registration, and revocable sleep authority; retain Teams/API causes; schedule semantic deadlines; and coalesce causes into one idempotent From 62877d983ac1fde8d1c8e195c59bd2f8ef1a7616 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Sun, 26 Jul 2026 14:27:38 -0700 Subject: [PATCH 13/28] docs(rfc-0013): link durable wake projection store --- rfcs/0013-cloud-serializable-sqlite-state.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 1b2f1c95..6e1e4cc0 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -374,9 +374,12 @@ The bounded follow-on PR plan is: by default. The first host-side authority evidence is the fork-only draft [giodl/lobster#38](https://microsoft.ghe.com/giodl/lobster/pull/38) (Microsoft GHE access required); it binds a content-addressed projection to - final acceptance and destruction authority. Permanent `replaceAll` adoption - and temporary scheduler rehydration remain follow-on proof, so that draft - does not yet claim the complete adapter. + final acceptance and destruction authority. Stacked fork-only draft + [giodl/lobster#39](https://microsoft.ghe.com/giodl/lobster/pull/39) + (Microsoft GHE access required) adds permanent atomic `replaceAll`, stale + writer supersession, and recovery into Lobster's existing Redis scheduler. + Connecting the shipped OpenClaw hooks to that host journal remains the final + live adapter proof; neither draft claims that wiring yet. 2. **Host wake authority and coalescing:** atomically accept the recovery point, wake registration, and revocable sleep authority; retain Teams/API causes; schedule semantic deadlines; and coalesce causes into one idempotent From 5b40b4812c2ca953956480755f29525d20c3f7e3 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Sun, 26 Jul 2026 15:56:16 -0700 Subject: [PATCH 14/28] docs(rfc-0013): link cancellable wake acceptance --- rfcs/0013-cloud-serializable-sqlite-state.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 6e1e4cc0..7621fc2a 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -378,8 +378,14 @@ The bounded follow-on PR plan is: [giodl/lobster#39](https://microsoft.ghe.com/giodl/lobster/pull/39) (Microsoft GHE access required) adds permanent atomic `replaceAll`, stale writer supersession, and recovery into Lobster's existing Redis scheduler. - Connecting the shipped OpenClaw hooks to that host journal remains the final - live adapter proof; neither draft claims that wiring yet. + Stacked fork-only draft + [giodl/lobster#41](https://microsoft.ghe.com/giodl/lobster/pull/41) + (Microsoft GHE access required) makes prepare, commit, abort, and newer- + attempt supersession contend in the same permanent CAS state, so transport + disconnect is never treated as cancellation proof across fungible host + pods. Connecting the shipped OpenClaw hooks to that host protocol remains + the final live adapter proof; these drafts do not claim production endpoint, + authentication, storage-bootstrap, or rollout wiring yet. 2. **Host wake authority and coalescing:** atomically accept the recovery point, wake registration, and revocable sleep authority; retain Teams/API causes; schedule semantic deadlines; and coalesce causes into one idempotent From e349b694d555bf7bc46dba104e38b16b060db607 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Sun, 26 Jul 2026 16:17:34 -0700 Subject: [PATCH 15/28] docs(rfc-0013): link cron projection plugin evidence --- rfcs/0013-cloud-serializable-sqlite-state.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 7621fc2a..1c162bf5 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -383,9 +383,15 @@ The bounded follow-on PR plan is: (Microsoft GHE access required) makes prepare, commit, abort, and newer- attempt supersession contend in the same permanent CAS state, so transport disconnect is never treated as cancellation proof across fungible host - pods. Connecting the shipped OpenClaw hooks to that host protocol remains - the final live adapter proof; these drafts do not claim production endpoint, - authentication, storage-bootstrap, or rollout wiring yet. + pods. Stacked fork-only draft + [giodl/lobster#42](https://microsoft.ghe.com/giodl/lobster/pull/42) + (Microsoft GHE access required) connects the shipped OpenClaw hooks to the + bounded prepare/commit/abort protocol as an image-present, baked-disabled + plugin. It rereads complete scheduler state, projects no prompts, payloads, + cron expressions, credentials, or execution authority, and makes lifecycle + cancellation explicit. These drafts still do not claim the production + endpoint, authentication, permanent-store bootstrap, ECS activation, or + rollout wiring. 2. **Host wake authority and coalescing:** atomically accept the recovery point, wake registration, and revocable sleep authority; retain Teams/API causes; schedule semantic deadlines; and coalesce causes into one idempotent From 637b81132aa8a7493d73eee96b69d35d332999fc Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Sun, 26 Jul 2026 17:20:32 -0700 Subject: [PATCH 16/28] docs(rfc-0013): link trusted cron projection wiring --- rfcs/0013-cloud-serializable-sqlite-state.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 1c162bf5..15d6003f 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -389,9 +389,18 @@ The bounded follow-on PR plan is: bounded prepare/commit/abort protocol as an image-present, baked-disabled plugin. It rereads complete scheduler state, projects no prompts, payloads, cron expressions, credentials, or execution authority, and makes lifecycle - cancellation explicit. These drafts still do not claim the production - endpoint, authentication, permanent-store bootstrap, ECS activation, or - rollout wiring. + cancellation explicit. Stacked fork-only draft + [giodl/lobster#43](https://microsoft.ghe.com/giodl/lobster/pull/43) + (Microsoft GHE access required) completes the inactive production path + through Lobster's existing loopback proxy-pipe bridge: trusted runtime + context supplies tenant, user, provider, and Gateway-role authority; Blob + CAS stores the winner; and every accepted-receipt path rehydrates the + existing Redis wake scheduler before reporting success. This removes the + need for a new public callback or reusable bearer-token design. The plugin + remains baked-disabled; permanent Blob-container provisioning and per-user + ECS activation through the shared Gateway start overlay, plus routing an + already-due recovered deadline into retained-cause runtime readiness, remain + explicit owner-reviewed rollout work. 2. **Host wake authority and coalescing:** atomically accept the recovery point, wake registration, and revocable sleep authority; retain Teams/API causes; schedule semantic deadlines; and coalesce causes into one idempotent From 0ae7347cd0dd6af8598408a28c4336408823920e Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Sun, 26 Jul 2026 17:51:10 -0700 Subject: [PATCH 17/28] docs(rfc-0013): link cron activation evidence --- rfcs/0013-cloud-serializable-sqlite-state.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 15d6003f..52bed68e 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -396,11 +396,16 @@ The bounded follow-on PR plan is: context supplies tenant, user, provider, and Gateway-role authority; Blob CAS stores the winner; and every accepted-receipt path rehydrates the existing Redis wake scheduler before reporting success. This removes the - need for a new public callback or reusable bearer-token design. The plugin - remains baked-disabled; permanent Blob-container provisioning and per-user - ECS activation through the shared Gateway start overlay, plus routing an - already-due recovered deadline into retained-cause runtime readiness, remain - explicit owner-reviewed rollout work. + need for a new public callback or reusable bearer-token design. Stacked + fork-only draft + [giodl/lobster#45](https://microsoft.ghe.com/giodl/lobster/pull/45) + (Microsoft GHE access required) completes the default-off activation + boundary: it provisions the private permanent Blob container, adds an + explicit per-user ECS opt-in through the shared Gateway start overlay, + resets that opt-in across pooled-user reuse, and treats an already-due + recovered projection as tracked retry work so ordinary cooperative + suspension cannot settle before OpenClaw reconciles it. No user is enabled + by the draft; the actual ECS cohort remains an owner rollout decision. 2. **Host wake authority and coalescing:** atomically accept the recovery point, wake registration, and revocable sleep authority; retain Teams/API causes; schedule semantic deadlines; and coalesce causes into one idempotent From ba41e62470a94fa9f978af3ac54b652db21bfc4c Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Mon, 27 Jul 2026 00:44:56 -0700 Subject: [PATCH 18/28] docs(rfc): link retained wake authority proof --- rfcs/0013-cloud-serializable-sqlite-state.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 52bed68e..784c54de 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -409,7 +409,14 @@ The bounded follow-on PR plan is: 2. **Host wake authority and coalescing:** atomically accept the recovery point, wake registration, and revocable sleep authority; retain Teams/API causes; schedule semantic deadlines; and coalesce causes into one idempotent - ensure-runtime-ready operation. + ensure-runtime-ready operation. Fork-only draft + [giodl/lobster#46](https://microsoft.ghe.com/giodl/lobster/pull/46) + (Microsoft GHE access required) is the bounded authority/core evidence: it + composes the existing final-handoff revocation race and managed-start seam, + keeps payloads and credentials with their owners, and grants one destination + generation for concurrent retained causes. Production Teams/cron owner + adapters and delivery remain follow-up slices rather than new central + lifecycle responsibilities. 3. **Readiness-gated delivery and replay:** consume the exact restored-admission result before delivering retained work, preserve per-owner acknowledgement, and prove replay across a host-process failure with one Teams cause and one From bc1ddd0089c919b5fd5f2b2645e95d54e63ce09b Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Mon, 27 Jul 2026 07:15:32 -0700 Subject: [PATCH 19/28] docs(rfc): link retained wake owner adapters --- rfcs/0013-cloud-serializable-sqlite-state.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 784c54de..999dda28 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -416,7 +416,12 @@ The bounded follow-on PR plan is: keeps payloads and credentials with their owners, and grants one destination generation for concurrent retained causes. Production Teams/cron owner adapters and delivery remain follow-up slices rather than new central - lifecycle responsibilities. + lifecycle responsibilities. Stacked fork-only draft + [giodl/lobster#47](https://microsoft.ghe.com/giodl/lobster/pull/47) + (Microsoft GHE access required) adds metadata-only native Teams and exact + permanent cron projection/provider/deadline validation into that authority; + route/cohort activation and readiness-gated delivery remain owner-controlled + follow-ups. 3. **Readiness-gated delivery and replay:** consume the exact restored-admission result before delivering retained work, preserve per-owner acknowledgement, and prove replay across a host-process failure with one Teams cause and one From e2d8324430bb1b4849e9c1902e73d4d5e371100f Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Mon, 27 Jul 2026 10:09:22 -0700 Subject: [PATCH 20/28] docs(rfc-0013): link retained wake delivery proof --- rfcs/0013-cloud-serializable-sqlite-state.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 999dda28..60f0fade 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -420,12 +420,20 @@ The bounded follow-on PR plan is: [giodl/lobster#47](https://microsoft.ghe.com/giodl/lobster/pull/47) (Microsoft GHE access required) adds metadata-only native Teams and exact permanent cron projection/provider/deadline validation into that authority; - route/cohort activation and readiness-gated delivery remain owner-controlled - follow-ups. + route/cohort activation remains an owner-controlled follow-up. 3. **Readiness-gated delivery and replay:** consume the exact restored-admission result before delivering retained work, preserve per-owner acknowledgement, and prove replay across a host-process failure with one Teams cause and one - cron deadline sharing one cold start. + cron deadline sharing one cold start. Stacked fork-only draft + [giodl/lobster#51](https://microsoft.ghe.com/giodl/lobster/pull/51) + (Microsoft GHE access required) provides that core evidence with a separate + payload-free permanent dispatch journal, exact recovery/destination/readiness + fencing, independent per-cause progress, and deterministic crash replay. + It does not activate production Teams or cron delivery. Live wiring remains + fail-closed until Lobster owners choose an accepted-final-handoff lookup and + canonical mapping from opaque logical runtime identity to the Teams/cron + semantic owners; each owner must enforce the destination/readiness fence in + the operation that controls its side effect. These are review and evidence slices, not a required repository decomposition. The host slices may be combined if the same ownership, race, replay, and From b632e2909a3bcc287c4f60f79c5bf4ed7ba96b37 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Mon, 27 Jul 2026 14:09:46 -0700 Subject: [PATCH 21/28] docs(rfc-0013): frame host runtime lifecycle contract --- rfcs/0013-cloud-serializable-sqlite-state.md | 94 ++++++++++++++++++++ 1 file changed, 94 insertions(+) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 60f0fade..3e8f5dfb 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -314,6 +314,100 @@ The three OpenClaw drafts prove only the owner-side recovery-point, final capture, and restored-admission slices. Host acceptance, retained ingress, wake, and destruction remain separate review and implementation work. +#### One bidirectional host/runtime contract + +The lifecycle can be presented as one contract without moving host authority +into OpenClaw or introducing a generic lifecycle coordinator. The useful seam +has two directions: + +- OpenClaw calls a host implementation only while an OpenClaw process exists, + for example to durably accept a closed recovery point or replace the host's + wake projection. +- The host calls OpenClaw's running Gateway and offline recovery operations to + suspend a source, capture its closed state, restore an accepted point, and + observe whether the restored destination is admissible. + +In TypeScript-like pseudocode, the smallest useful shape is: + +```ts +interface ContinuityHost { + acceptRecoveryPoint( + request: AcceptRecoveryPointRequest, + ): Promise; + replaceWakeProjection( + request: ReplaceWakeProjectionRequest, + ): Promise; +} + +interface HostedRuntime { + prepareSuspension( + request: PrepareSuspensionRequest, + ): Promise; + getSuspensionStatus( + request: SuspensionStatusRequest, + ): Promise; + getRestoreStatus( + request: RestoreStatusRequest, + ): Promise; +} + +interface OfflineRecovery { + captureFinal( + request: FinalRecoveryPointRequest, + ): Promise; + restoreAccepted( + request: RestoredRecoveryPointRequest, + ): Promise; +} +``` + +These interfaces describe protocol ownership, not an abstract base class that +a host must subclass. A scale-to-zero host cannot rely on in-process virtual +methods because no OpenClaw process exists to receive wake or provisioning +work. A conforming host may implement `ContinuityHost` through a plugin, +loopback service, CLI adapter, or another authenticated transport. Stable +operation identities, schemas, replay behavior, and typed conflicts are the +portable contract; transport and placement remain host-owned. + +The names above are illustrative. `gateway.suspend.prepare|status|resume` +already supplies the running-source portion. Maintainers may choose to expose +the destination portion as a narrow `gateway.restore.status` method, project +the same closed phases through existing Status/readiness surfaces, or use a +smaller equivalent seam. In every version, OpenClaw computes readiness and +opens its own admission; the host cannot assert readiness or invoke an +`admit` method. + +One host-owned coordinator may compose those contracts as +`prepareHibernate`, `ensureRuntimeReady`, and `inspectLifecycle`. Those are +host operations, not new OpenClaw Gateway methods. The resulting evidence +chain is: + +```text +SuspensionReady + -> FinalRecoveryPointResult + -> HostAcceptanceReceipt + -> generation-scoped safeToDestroy + -> retained wake cause + -> RestoredAdmissionReadyRecord + -> owner delivery and acknowledgement +``` + +Each authority contributes only its own fact. `safeToDestroy` is derived from +durable acceptance rather than supplied by a caller. Wake callers cannot +select a recovery point, destination generation, or readiness result. Teams, +cron, and API owners retain their own payload, retry, deduplication, and +acknowledgement state. Unknown outcomes retain work and hold or quarantine; +timeouts do not imply success. + +This framing is intended to simplify the evidence implementation: reuse the +existing suspension methods, scheduler reconciliation hooks, recovery +operations, and readiness records; keep one host lifecycle state machine; and +avoid a second pause API, a generic host callback registry, duplicate Gateway +admission state, or a central retained-payload store. A shared SDK package is +not required for V1. The concrete request and result types should remain next +to their owner surfaces until another host demonstrates a stable extraction +boundary. + #### Deferred cron and retained-ingress wake composition The current three-PR evidence stack intentionally stops at restored admission. From 69c79535a0e88b98acac574f40ce2ad684971a7f Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Mon, 27 Jul 2026 14:19:50 -0700 Subject: [PATCH 22/28] docs(rfc-0013): propose gateway restore status --- rfcs/0013-cloud-serializable-sqlite-state.md | 109 +++++++--------- rfcs/0013/restored-admission-v1-spec.md | 123 +++++++++++++++++++ 2 files changed, 164 insertions(+), 68 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 3e8f5dfb..33f6d57e 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -316,66 +316,40 @@ wake, and destruction remain separate review and implementation work. #### One bidirectional host/runtime contract -The lifecycle can be presented as one contract without moving host authority -into OpenClaw or introducing a generic lifecycle coordinator. The useful seam -has two directions: - -- OpenClaw calls a host implementation only while an OpenClaw process exists, - for example to durably accept a closed recovery point or replace the host's - wake projection. -- The host calls OpenClaw's running Gateway and offline recovery operations to - suspend a source, capture its closed state, restore an accepted point, and - observe whether the restored destination is admissible. - -In TypeScript-like pseudocode, the smallest useful shape is: - -```ts -interface ContinuityHost { - acceptRecoveryPoint( - request: AcceptRecoveryPointRequest, - ): Promise; - replaceWakeProjection( - request: ReplaceWakeProjectionRequest, - ): Promise; -} - -interface HostedRuntime { - prepareSuspension( - request: PrepareSuspensionRequest, - ): Promise; - getSuspensionStatus( - request: SuspensionStatusRequest, - ): Promise; - getRestoreStatus( - request: RestoreStatusRequest, - ): Promise; -} - -interface OfflineRecovery { - captureFinal( - request: FinalRecoveryPointRequest, - ): Promise; - restoreAccepted( - request: RestoredRecoveryPointRequest, - ): Promise; -} -``` - -These interfaces describe protocol ownership, not an abstract base class that -a host must subclass. A scale-to-zero host cannot rely on in-process virtual -methods because no OpenClaw process exists to receive wake or provisioning -work. A conforming host may implement `ContinuityHost` through a plugin, -loopback service, CLI adapter, or another authenticated transport. Stable -operation identities, schemas, replay behavior, and typed conflicts are the -portable contract; transport and placement remain host-owned. - -The names above are illustrative. `gateway.suspend.prepare|status|resume` -already supplies the running-source portion. Maintainers may choose to expose -the destination portion as a narrow `gateway.restore.status` method, project -the same closed phases through existing Status/readiness surfaces, or use a -smaller equivalent seam. In every version, OpenClaw computes readiness and -opens its own admission; the host cannot assert readiness or invoke an -`admit` method. +The recommended V1 surface is one bidirectional protocol contract without an +OpenClaw-owned lifecycle coordinator or abstract `Host` base class: + +- current-main `gateway.suspend.prepare|status|resume` supplies the live-source + fence; +- the final-capture and accepted-restore operations supply the offline owner + work; and +- a new read-only `gateway.restore.status` supplies the exact live-destination + admission fact. + +`gateway.restore.status` follows the design Peter established for +`gateway.suspend.status`: a closed TypeBox request and result, one stable +operation identity, `operator.read` scope, no mutation, and typed conflict +behavior. The request names the expected `restoreOperationId`. An ordinary +Gateway returns `not-restored`; the matching restored Gateway returns `held` +or `ready`; a different active restore operation fails rather than returning a +success-shaped result. The `ready` result names the destination generation, +accepted recovery point and byte set, restore receipt, scheduler and owner +readiness evidence, admission identity, and final readiness identity. + +The method projects the durable restored-admission record already required by +this RFC. It does not add another journal or state machine. Offline restore and +preflight failures remain offline failures; they are not represented as a +live `restoring` phase. A quarantined start does not expose a live Gateway and +therefore is not a success result. The detailed request, result, availability, +and error contract is normative in +[Restored Admission v1](0013/restored-admission-v1-spec.md#gateway-restore-status). + +There is deliberately no `gateway.restore.admit` method. OpenClaw computes +readiness, durably records it, and opens its own admission. The host may only +observe the result and compare it with the operation and generation it owns. +Normal user-work methods remain unavailable while restored admission is held; +the status method is allowed only through an authenticated pre-admission +control path, following the existing suspension-control pattern. One host-owned coordinator may compose those contracts as `prepareHibernate`, `ensureRuntimeReady`, and `inspectLifecycle`. Those are @@ -399,14 +373,13 @@ cron, and API owners retain their own payload, retry, deduplication, and acknowledgement state. Unknown outcomes retain work and hold or quarantine; timeouts do not imply success. -This framing is intended to simplify the evidence implementation: reuse the -existing suspension methods, scheduler reconciliation hooks, recovery -operations, and readiness records; keep one host lifecycle state machine; and -avoid a second pause API, a generic host callback registry, duplicate Gateway -admission state, or a central retained-payload store. A shared SDK package is -not required for V1. The concrete request and result types should remain next -to their owner surfaces until another host demonstrates a stable extraction -boundary. +This contract simplifies the evidence implementation: reuse the existing +suspension methods, scheduler reconciliation hooks, recovery operations, +admission fence, and readiness journal; keep one host lifecycle state machine; +and avoid a second pause API, generic host callback registry, duplicate +Gateway admission state, or central retained-payload store. The Gateway +Protocol package should own the new wire schemas beside `gateway-suspend`; a +separate lifecycle SDK package is not required for V1. #### Deferred cron and retained-ingress wake composition diff --git a/rfcs/0013/restored-admission-v1-spec.md b/rfcs/0013/restored-admission-v1-spec.md index 33a9cc13..7e641c0e 100644 --- a/rfcs/0013/restored-admission-v1-spec.md +++ b/rfcs/0013/restored-admission-v1-spec.md @@ -187,6 +187,123 @@ Ordinary startup remains unchanged because it has no restored-start evidence. Ordinary startup cannot consume a committed restore hold, and restored startup cannot use an ordinary admission path. +## Gateway Restore Status + +V1 adds one read-only Gateway Protocol method: + +```text +gateway.restore.status +``` + +The method is the live-destination observation seam for a host. It follows the +existing `gateway.suspend.status` conventions: + +- core-owned and host-neutral; +- closed TypeBox request and result schemas in the Gateway Protocol package; +- `operator.read` scope; +- no control-plane-write classification; +- no config key or environment variable; +- idempotent and side-effect free; and +- exposed by authenticated Gateway control transports that opt into core RPC + methods, including the existing Admin HTTP RPC path. + +The request is: + +```ts +type GatewayRestoreStatusParams = { + restoreOperationId: string; +}; +``` + +`restoreOperationId` must satisfy the same bounded non-empty token rules as +the restored-admission operation. It is required even though only one restored +operation can own a Gateway incarnation. Requiring it prevents an observation +for one host attempt from being mistaken for another generation's readiness. + +The successful result is a closed union: + +```ts +type GatewayRestoreStatusResult = + | { + status: "not-restored"; + } + | { + status: "held"; + reason: "scheduler-reconciliation" | "owner-readiness" | "ready-commit"; + retryAfterMs: number; + runtimeLineage: string; + lifecycleOwnerGeneration: string; + destinationRuntimeGeneration: string; + restoreOperationId: string; + destinationOwner: string; + admissionIdentity: string; + recoveryPointId: string; + acceptanceSetId: string; + restoreReceiptIdentity: string; + } + | { + status: "ready"; + runtimeLineage: string; + lifecycleOwnerGeneration: string; + destinationRuntimeGeneration: string; + restoreOperationId: string; + destinationOwner: string; + admissionIdentity: string; + recoveryPointId: string; + acceptanceSetId: string; + restoreReceiptIdentity: string; + schedulerIdentity: string; + ownerReadinessIdentity: string; + readinessIdentity: string; + }; +``` + +`not-restored` means the running Gateway has no restored-start evidence. It is +not permission for a host restore operation to deliver retained work. `held` +means the expected restored incarnation is live but ordinary work admission is +still closed. `ready` is returned only after the exact ready record is durable +and the same admission identity has opened Gateway work admission. + +All strings and arrays use explicit protocol bounds. `retryAfterMs` is a +non-negative bounded integer and only a polling hint. Results contain no local +paths, artifact locations, credentials, retained payloads, prompts, or owner +diagnostic text. + +The handler must derive `held` and `ready` from the same in-memory startup +binding and operation-scoped SQLite journal that enforce admission. It must not +create a second restore-status file, infer readiness from `/healthz`, or hash a +mutable runtime status object. A replayed call returns the same identity fields +for the same durable record. The Gateway may retain the already validated +status projection in memory; polling must not synchronously reopen or rehash +the recovery journal on every request. Transition to `ready` occurs only after +the durable record commit succeeds. + +If a different restore operation owns the running Gateway, the method returns +`UNAVAILABLE` with bounded details reason `restored-admission-conflict`; it +must not return that operation's identity as a success result. Invalid or +oversized tokens return `INVALID_REQUEST`. Journal corruption, contradictory +evidence, or failure to read a required committed record returns `UNAVAILABLE` +and keeps work admission closed. No error is success-shaped. + +The method must remain callable while restored admission is held through an +authenticated pre-admission control path. It is exempt only from the restored- +admission work fence, like suspension control methods are exempt from the +suspension work fence; it does not make other RPC methods available. A host +without such a control path may use `/readyz` only as a backoff hint and call +`gateway.restore.status` after readiness, but it must not deliver retained +work until the exact `ready` result matches its operation and destination +generation. + +For a fresh destination, the recommended held-state transport is the existing +authenticated Admin HTTP RPC route bound to the host-controlled loopback path. +The status method does not justify a new unauthenticated probe, public listener, +bearer-token scheme, or sidecar process. A mismatched-operation error must not +disclose the active operation's identities. + +There is no `gateway.restore.admit`, `resume`, or caller-supplied readiness +method. Admission remains an OpenClaw-owned consequence of durable owner +evidence, not a host command. + ## Crash Replay The required crash boundary is: @@ -243,6 +360,12 @@ V1 conformance must prove: - scheduler reconciliation precedes readiness; - admission opens exactly once from exact durable evidence; - process health alone cannot open admission; +- `gateway.restore.status` is read-only, operation-fenced, and returns exact + durable identities for `ready`; +- ordinary and mismatched restored Gateways cannot return success-shaped + readiness for the requested operation; +- the status method remains available through an authenticated control path + while ordinary restored work admission is held; - coordinator crash replay reuses the same child and readiness generation; - preparation and restore execute exactly once; and - stale, contradictory, corrupt, and fixed-path collision cases fail closed. From 9757de042f218f733dea6e1ccdfeaa41f58cc185 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Mon, 27 Jul 2026 14:20:49 -0700 Subject: [PATCH 23/28] docs(rfc-0013): separate status proposal from evidence --- rfcs/0013-cloud-serializable-sqlite-state.md | 4 +++- rfcs/0013/restored-admission-v1-spec.md | 5 +++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 33f6d57e..1b03e2c3 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -293,7 +293,9 @@ slices exercised by these sidecars: keeps Gateway work admission closed through scheduler reconciliation and owner readiness. Restore intent, results, startup descriptors, and ready evidence share the dedicated SQLite recovery journal rather than JSON - runtime sidecars. + runtime sidecars. It does not yet expose the proposed + `gateway.restore.status` method; that public Gateway Protocol surface remains + an explicit owner-review decision. These drafts are evidence for owner review, not normative dependencies. They do not move Gateway suspension, external ingress fencing, clean process shutdown, diff --git a/rfcs/0013/restored-admission-v1-spec.md b/rfcs/0013/restored-admission-v1-spec.md index 7e641c0e..ec9ca8fc 100644 --- a/rfcs/0013/restored-admission-v1-spec.md +++ b/rfcs/0013/restored-admission-v1-spec.md @@ -16,6 +16,11 @@ It reuses the RFC 0013 SQLite provider, durably binds the restore receipt and private startup descriptor, reconciles the canonical scheduler, and opens work admission only after owner readiness. +The draft already produces the durable record that the proposed +`gateway.restore.status` method would project, but it does not yet add that +public Gateway Protocol surface. The method below is the recommended V1 design +for owner review, not a claim about the current implementation head. + The draft deliberately holds when required external or reconstructed obligations lack accepted owner evidence. It does not add wake routing, placement, idle policy, host acceptance, or a public restore-hook registry. From 96c1062ae695c0df58cf8f5ead2e684f52f57ee0 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Mon, 27 Jul 2026 15:11:23 -0700 Subject: [PATCH 24/28] docs(rfc-0013): simplify lifecycle contract --- rfcs/0013-cloud-serializable-sqlite-state.md | 116 ++++++------------ rfcs/0013/portable-handoff-v1-spec.md | 4 +- .../0013/recovery-point-components-v1-spec.md | 4 +- rfcs/0013/restored-admission-v1-spec.md | 16 +-- 4 files changed, 49 insertions(+), 91 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 1b03e2c3..a1d96952 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -259,7 +259,7 @@ wake registration, compute placement, and external durability; OpenClaw and its state owners provide the exact recovery point and restored-admission proof that make those host primitives safe to use. -The draft implementer-facing follow-on contracts are: +The implementer-facing follow-on contracts are deliberately split by owner: - [Recovery Point Components v1](0013/recovery-point-components-v1-spec.md): compose verified SQLite snapshots with explicit non-SQLite owner artifacts @@ -271,33 +271,42 @@ The draft implementer-facing follow-on contracts are: accepted components into fresh paths and keep admission closed until scheduler and required owner readiness complete. +The split is mechanical rather than architectural: + +| Contract | Owner-side input | Durable output | Stops before | +| --- | --- | --- | --- | +| Recovery Point Components | Verified owner artifacts plus the closed selected-owner inventory | `recoveryPointId` and `acceptanceSetId` | Suspension, storage, or wake | +| Portable Handoff | Suspension-ready source generation plus final owner capture | Host acceptance receipt and generation-scoped `safeToDestroy` | Restore or destination admission | +| Restored Admission | Exact accepted recovery point plus destination generation | Restore receipt and `gateway.restore.status = ready` | Retained-work delivery | + +The IDs join the contracts; no sidecar owns another sidecar's work. + These sidecars do not change `openclaw backup sqlite`. They do not make every ordinary snapshot a portable recovery point, add a continuity-specific storage provider, or make Lobster part of the core contract. -Draft OpenClaw implementation evidence is available for the three owner-side -slices exercised by these sidecars: +OpenClaw implementation evidence is available for the three owner-side slices: - [openclaw/openclaw#112385](https://github.com/openclaw/openclaw/pull/112385) composes verified global and owner-selected per-agent RFC 0013 snapshots into one deterministic `host-protected` recovery point and exact acceptance byte inventory. - [openclaw/openclaw#112865](https://github.com/openclaw/openclaw/pull/112865) - is a draft stacked on #112385. It adds one hidden offline final + is stacked on #112385. It adds one hidden offline final capture operation with operation-scoped SQLite intent, exact committed-result replay, and fail-closed quarantine for conflicting or incomplete attempts. - [openclaw/openclaw#112896](https://github.com/openclaw/openclaw/pull/112896) - is a draft stacked on #112865. It restores one exact accepted aggregate to + is stacked on #112865. It restores one exact accepted aggregate to fresh canonical paths, holds when required owner evidence is absent, and keeps Gateway work admission closed through scheduler reconciliation and owner readiness. Restore intent, results, startup descriptors, and ready evidence share the dedicated SQLite recovery journal rather than JSON - runtime sidecars. It does not yet expose the proposed - `gateway.restore.status` method; that public Gateway Protocol surface remains - an explicit owner-review decision. + runtime sidecars. It also implements the proposed read-only + `gateway.restore.status` projection as review evidence. The public Gateway + Protocol surface remains an explicit owner decision before merge or ship. -These drafts are evidence for owner review, not normative dependencies. They do +These PRs are evidence for owner review, not normative dependencies. They do not move Gateway suspension, external ingress fencing, clean process shutdown, durable host acceptance, publication, host wake, or source destruction into OpenClaw. @@ -312,7 +321,7 @@ The intended observable outcome of the complete host composition is: - the source generation is not destroyed before exact durable acceptance; and - replacement readiness names the accepted recovery point it restored. -The three OpenClaw drafts prove only the owner-side recovery-point, final +The three OpenClaw PRs prove only the owner-side recovery-point, final capture, and restored-admission slices. Host acceptance, retained ingress, wake, and destruction remain separate review and implementation work. @@ -434,75 +443,24 @@ Unknown provisioning, restore, readiness, or delivery outcomes retain the wake causes and hold or quarantine. A timeout is not permission to acknowledge work, start a second authoritative generation, or open admission. -The bounded follow-on PR plan is: - -1. **Host cron-projection adapter:** implement the documented - `cron_reconciled` plus `cron_changed` projection pattern, durably replace the - complete external wake set, and bind its accepted revision to the final - recovery point and sleep authority. No new OpenClaw core surface is expected - by default. The first host-side authority evidence is the fork-only draft - [giodl/lobster#38](https://microsoft.ghe.com/giodl/lobster/pull/38) - (Microsoft GHE access required); it binds a content-addressed projection to - final acceptance and destruction authority. Stacked fork-only draft - [giodl/lobster#39](https://microsoft.ghe.com/giodl/lobster/pull/39) - (Microsoft GHE access required) adds permanent atomic `replaceAll`, stale - writer supersession, and recovery into Lobster's existing Redis scheduler. - Stacked fork-only draft - [giodl/lobster#41](https://microsoft.ghe.com/giodl/lobster/pull/41) - (Microsoft GHE access required) makes prepare, commit, abort, and newer- - attempt supersession contend in the same permanent CAS state, so transport - disconnect is never treated as cancellation proof across fungible host - pods. Stacked fork-only draft - [giodl/lobster#42](https://microsoft.ghe.com/giodl/lobster/pull/42) - (Microsoft GHE access required) connects the shipped OpenClaw hooks to the - bounded prepare/commit/abort protocol as an image-present, baked-disabled - plugin. It rereads complete scheduler state, projects no prompts, payloads, - cron expressions, credentials, or execution authority, and makes lifecycle - cancellation explicit. Stacked fork-only draft - [giodl/lobster#43](https://microsoft.ghe.com/giodl/lobster/pull/43) - (Microsoft GHE access required) completes the inactive production path - through Lobster's existing loopback proxy-pipe bridge: trusted runtime - context supplies tenant, user, provider, and Gateway-role authority; Blob - CAS stores the winner; and every accepted-receipt path rehydrates the - existing Redis wake scheduler before reporting success. This removes the - need for a new public callback or reusable bearer-token design. Stacked - fork-only draft - [giodl/lobster#45](https://microsoft.ghe.com/giodl/lobster/pull/45) - (Microsoft GHE access required) completes the default-off activation - boundary: it provisions the private permanent Blob container, adds an - explicit per-user ECS opt-in through the shared Gateway start overlay, - resets that opt-in across pooled-user reuse, and treats an already-due - recovered projection as tracked retry work so ordinary cooperative - suspension cannot settle before OpenClaw reconciles it. No user is enabled - by the draft; the actual ECS cohort remains an owner rollout decision. -2. **Host wake authority and coalescing:** atomically accept the recovery point, - wake registration, and revocable sleep authority; retain Teams/API causes; - schedule semantic deadlines; and coalesce causes into one idempotent - ensure-runtime-ready operation. Fork-only draft - [giodl/lobster#46](https://microsoft.ghe.com/giodl/lobster/pull/46) - (Microsoft GHE access required) is the bounded authority/core evidence: it - composes the existing final-handoff revocation race and managed-start seam, - keeps payloads and credentials with their owners, and grants one destination - generation for concurrent retained causes. Production Teams/cron owner - adapters and delivery remain follow-up slices rather than new central - lifecycle responsibilities. Stacked fork-only draft - [giodl/lobster#47](https://microsoft.ghe.com/giodl/lobster/pull/47) - (Microsoft GHE access required) adds metadata-only native Teams and exact - permanent cron projection/provider/deadline validation into that authority; - route/cohort activation remains an owner-controlled follow-up. -3. **Readiness-gated delivery and replay:** consume the exact restored-admission - result before delivering retained work, preserve per-owner acknowledgement, - and prove replay across a host-process failure with one Teams cause and one - cron deadline sharing one cold start. Stacked fork-only draft - [giodl/lobster#51](https://microsoft.ghe.com/giodl/lobster/pull/51) - (Microsoft GHE access required) provides that core evidence with a separate - payload-free permanent dispatch journal, exact recovery/destination/readiness - fencing, independent per-cause progress, and deterministic crash replay. - It does not activate production Teams or cron delivery. Live wiring remains - fail-closed until Lobster owners choose an accepted-final-handoff lookup and - canonical mapping from opaque logical runtime identity to the Teams/cron - semantic owners; each owner must enforce the destination/readiness fence in - the operation that controls its side effect. +The bounded host follow-on has three independently reviewable responsibilities: + +1. **Project scheduler wake metadata:** consume `cron_reconciled`, use + `cron_changed` only as a reread hint, and durably bind the complete accepted + projection to the final recovery point and sleep authority. Fork-only + evidence is in Lobster PRs + [#38](https://microsoft.ghe.com/giodl/lobster/pull/38) through + [#45](https://microsoft.ghe.com/giodl/lobster/pull/45) (Microsoft access + required). +2. **Retain and coalesce wake causes:** keep Teams/API payloads with their + owners, revoke sleep atomically, and grant one destination generation for + concurrent causes. Fork-only evidence is in Lobster PRs + [#46](https://microsoft.ghe.com/giodl/lobster/pull/46) and + [#47](https://microsoft.ghe.com/giodl/lobster/pull/47). +3. **Deliver only after exact readiness:** match `gateway.restore.status` + against the host-owned restore operation and destination generation before + invoking each owner's existing delivery path. Fork-only replay evidence is + in Lobster PR [#51](https://microsoft.ghe.com/giodl/lobster/pull/51). These are review and evidence slices, not a required repository decomposition. The host slices may be combined if the same ownership, race, replay, and diff --git a/rfcs/0013/portable-handoff-v1-spec.md b/rfcs/0013/portable-handoff-v1-spec.md index 773632d9..9bd64e4e 100644 --- a/rfcs/0013/portable-handoff-v1-spec.md +++ b/rfcs/0013/portable-handoff-v1-spec.md @@ -6,10 +6,10 @@ suspension contract with final owner capture and durable host acceptance. Status: draft, tied to RFC 0013. -## Draft Implementation Evidence +## Implementation Evidence [openclaw/openclaw#112865](https://github.com/openclaw/openclaw/pull/112865) -is the OpenClaw draft evidence slice for final owner capture, stacked on +is the OpenClaw evidence slice for final owner capture, stacked on [openclaw/openclaw#112385](https://github.com/openclaw/openclaw/pull/112385). It uses the existing RFC 0013 SQLite provider, binds the exact runtime lineage, source generation, handoff identity, selected-agent inventory, and diff --git a/rfcs/0013/recovery-point-components-v1-spec.md b/rfcs/0013/recovery-point-components-v1-spec.md index b8ade2b9..2c6d0f6c 100644 --- a/rfcs/0013/recovery-point-components-v1-spec.md +++ b/rfcs/0013/recovery-point-components-v1-spec.md @@ -8,10 +8,10 @@ the SQLite snapshot contract. Status: draft, tied to RFC 0013. -## Draft Implementation Evidence +## Implementation Evidence [openclaw/openclaw#112385](https://github.com/openclaw/openclaw/pull/112385) -is the paired draft evidence slice. It implements the V1 global and +is the paired evidence slice. It implements the V1 global and owner-selected per-agent SQLite composition path, deterministic recovery-point and acceptance-set identities, exact owner-manifest and artifact binding, closed obligation validation, and the `host-protected` classification. diff --git a/rfcs/0013/restored-admission-v1-spec.md b/rfcs/0013/restored-admission-v1-spec.md index ec9ca8fc..a6f8f590 100644 --- a/rfcs/0013/restored-admission-v1-spec.md +++ b/rfcs/0013/restored-admission-v1-spec.md @@ -6,20 +6,20 @@ evidence required before a restored Gateway can receive work. Status: draft, tied to RFC 0013. -## Draft Implementation Evidence +## Implementation Evidence [openclaw/openclaw#112896](https://github.com/openclaw/openclaw/pull/112896) -is the OpenClaw draft evidence slice for exact fresh-target restore and -readiness-gated admission, stacked on final-capture draft +is the OpenClaw evidence slice for exact fresh-target restore and +readiness-gated admission, stacked on final-capture PR [openclaw/openclaw#112865](https://github.com/openclaw/openclaw/pull/112865). It reuses the RFC 0013 SQLite provider, durably binds the restore receipt and private startup descriptor, reconciles the canonical scheduler, and opens work admission only after owner readiness. -The draft already produces the durable record that the proposed -`gateway.restore.status` method would project, but it does not yet add that -public Gateway Protocol surface. The method below is the recommended V1 design -for owner review, not a claim about the current implementation head. +The PR also implements the proposed `gateway.restore.status` projection over +that same durable record. The method below is the recommended V1 design and +representative proof for owner review; it is not an approved public contract +until the Gateway protocol owner accepts it. The draft deliberately holds when required external or reconstructed obligations lack accepted owner evidence. It does not add wake routing, @@ -269,7 +269,7 @@ means the expected restored incarnation is live but ordinary work admission is still closed. `ready` is returned only after the exact ready record is durable and the same admission identity has opened Gateway work admission. -All strings and arrays use explicit protocol bounds. `retryAfterMs` is a +All strings use explicit protocol bounds. `retryAfterMs` is a non-negative bounded integer and only a polling hint. Results contain no local paths, artifact locations, credentials, retained payloads, prompts, or owner diagnostic text. From 1037578ce6e35ec98a5fb2ef596453d4c0080095 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Mon, 27 Jul 2026 21:04:40 -0700 Subject: [PATCH 25/28] docs(rfc-0013): clarify scale-to-zero lifecycle facts --- rfcs/0013-cloud-serializable-sqlite-state.md | 82 ++++++++++++++++--- rfcs/0013/portable-handoff-v1-spec.md | 31 ++++--- .../0013/recovery-point-components-v1-spec.md | 29 ++++++- rfcs/0013/restored-admission-v1-spec.md | 78 +++++++++++++++++- 4 files changed, 191 insertions(+), 29 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index a1d96952..2b40e37d 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -266,7 +266,7 @@ The implementer-facing follow-on contracts are deliberately split by owner: and external or reconstruction obligations. - [Portable Handoff v1](0013/portable-handoff-v1-spec.md): combine the existing cooperative Gateway suspension fence with final owner capture, durable host - acceptance, and generation-bound source destruction authority. + acceptance, and generation-bound source-compute retirement authority. - [Restored Admission v1](0013/restored-admission-v1-spec.md): restore exact accepted components into fresh paths and keep admission closed until scheduler and required owner readiness complete. @@ -276,7 +276,7 @@ The split is mechanical rather than architectural: | Contract | Owner-side input | Durable output | Stops before | | --- | --- | --- | --- | | Recovery Point Components | Verified owner artifacts plus the closed selected-owner inventory | `recoveryPointId` and `acceptanceSetId` | Suspension, storage, or wake | -| Portable Handoff | Suspension-ready source generation plus final owner capture | Host acceptance receipt and generation-scoped `safeToDestroy` | Restore or destination admission | +| Portable Handoff | Suspension-ready source generation plus final owner capture | Host acceptance receipt and generation-scoped `sourceComputeRetirementAuthorized` | Restore or destination admission | | Restored Admission | Exact accepted recovery point plus destination generation | Restore receipt and `gateway.restore.status = ready` | Retained-work delivery | The IDs join the contracts; no sidecar owns another sidecar's work. @@ -308,7 +308,8 @@ OpenClaw implementation evidence is available for the three owner-side slices: These PRs are evidence for owner review, not normative dependencies. They do not move Gateway suspension, external ingress fencing, clean process shutdown, -durable host acceptance, publication, host wake, or source destruction into +durable host acceptance, publication, host wake, or source-compute retirement +into OpenClaw. The intended observable outcome of the complete host composition is: @@ -318,12 +319,13 @@ The intended observable outcome of the complete host composition is: - accepted ingress remains retained until restored admission succeeds; - autonomous scheduled work does not require an unrelated user message to recover from absent compute; -- the source generation is not destroyed before exact durable acceptance; and +- source compute is not retired before exact durable acceptance; and - replacement readiness names the accepted recovery point it restored. The three OpenClaw PRs prove only the owner-side recovery-point, final capture, and restored-admission slices. Host acceptance, retained ingress, -wake, and destruction remain separate review and implementation work. +wake, and source-compute retirement remain separate review and implementation +work. #### One bidirectional host/runtime contract @@ -362,28 +364,84 @@ Normal user-work methods remain unavailable while restored admission is held; the status method is allowed only through an authenticated pre-admission control path, following the existing suspension-control pattern. +#### RFC 0018 readiness composition + +When the opt-in canonical readiness facility from +[RFC 0018](https://github.com/openclaw/rfcs/pull/33) is available, restored +admission composes with it rather than creating a second evaluator. A restored +start publishes one required `RecoveryPointRestored` condition whose primary +subject is the stable `openclaw/gateway` role. Related stable subject references +identify the restore operation, accepted recovery point, destination runtime +generation, scheduler reconciliation, and required state-owner roles; their +current IDs and generations live in RFC 0018's identity package rather than in +the references. The condition remains `Unknown` or `False` while the restore +hold is active and becomes `True` only from the same durable record that opens +admission. + +`gateway.restore.status` remains the stronger operation-fenced query: its +caller supplies the expected restore operation and receives typed conflict +behavior. `/readyz`, `openclaw ready`, Status, and Gateway readiness RPC expose +the ordinary RFC 0018 projection for the active runtime. Both surfaces must +project the same readiness generation and subject identities. Neither may +infer restore completion from process health, database-open success, or a +container probe. This composition is contingent on RFC 0018 acceptance and +activation; `gateway.restore.status` remains independently useful and does not +depend on RFC 0018. + One host-owned coordinator may compose those contracts as `prepareHibernate`, `ensureRuntimeReady`, and `inspectLifecycle`. Those are host operations, not new OpenClaw Gateway methods. The resulting evidence chain is: ```text -SuspensionReady - -> FinalRecoveryPointResult - -> HostAcceptanceReceipt - -> generation-scoped safeToDestroy +GatewaySuspensionReady + -> SourceWritersClosed + -> RecoveryPointCaptured + -> RecoveryPointAccepted + -> generation-scoped SourceComputeRetirementAuthorized -> retained wake cause - -> RestoredAdmissionReadyRecord + -> RecoveryPointRestored + -> AdmissionReady -> owner delivery and acknowledgement ``` -Each authority contributes only its own fact. `safeToDestroy` is derived from -durable acceptance rather than supplied by a caller. Wake callers cannot +These are distinct facts, not aliases for one global `synced` state: + +- `GatewaySuspensionReady` proves OpenClaw's cooperative tracked-work fence for + one source generation. +- `SourceWritersClosed` proves the host supervisor closed the Gateway process + and every other authoritative writer it owns for that generation. +- `RecoveryPointCaptured` proves immutable owner artifacts and one closed + recovery-point identity. +- `RecoveryPointAccepted` proves the host durability boundary accepted the + exact logical byte set. +- `SourceComputeRetirementAuthorized` permits removal of only the named source + compute generation. It never authorizes persistent-state deletion. +- `RecoveryPointRestored` proves exact fresh-target restore and owner + reconciliation for one destination generation. +- `AdmissionReady` proves that same destination may accept ordinary work. + +Each authority contributes only its own fact. +`SourceComputeRetirementAuthorized` is derived from durable acceptance rather +than supplied by a caller. Wake callers cannot select a recovery point, destination generation, or readiness result. Teams, cron, and API owners retain their own payload, retry, deduplication, and acknowledgement state. Unknown outcomes retain work and hold or quarantine; timeouts do not imply success. +The diagram shows the success path, not an irreversible linear workflow. A new +retained-work cause may race any pre-retirement step. The host must atomically +revoke or defeat `SourceComputeRetirementAuthorized` while source compute still +exists; after retirement, the same cause joins the single destination wake and +restore operation. + +Planned and forced transitions deliberately have different guarantees: + +| Transition | New recovery point | Source retirement | Recovery guarantee | +| --- | --- | --- | --- | +| Planned handoff | Requires Gateway suspension, closed source writers, final capture, and exact durable acceptance | Only after generation-scoped authorization | The newly accepted point | +| Forced source loss | None may be inferred from process absence or partial files | Host policy handles already-lost compute; no clean-retirement claim is created | The last previously accepted point and its declared RPO | + This contract simplifies the evidence implementation: reuse the existing suspension methods, scheduler reconciliation hooks, recovery operations, admission fence, and readiness journal; keep one host lifecycle state machine; diff --git a/rfcs/0013/portable-handoff-v1-spec.md b/rfcs/0013/portable-handoff-v1-spec.md index 9bd64e4e..c96e50ee 100644 --- a/rfcs/0013/portable-handoff-v1-spec.md +++ b/rfcs/0013/portable-handoff-v1-spec.md @@ -18,7 +18,7 @@ point after response loss. The operation is deliberately offline and host-invoked. It does not suspend or stop Gateway, fence external ingress, accept or publish bytes, or authorize -source destruction. The pull request is implementation evidence; this +source-compute retirement. The pull request is implementation evidence; this specification remains the normative responsibility boundary. ## Scope @@ -30,7 +30,7 @@ This specification defines: - final capture after the tracked Gateway work fence is ready; - immutable host acceptance of one aggregate recovery point; - exact replay after transport or coordinator uncertainty; -- generation-bound source destruction authority; +- generation-bound source-compute retirement authority; - hold and quarantine behavior. This specification does not define: @@ -73,7 +73,7 @@ The lease is cooperative and refuse-only. It does not: - prove unregistered plugin or background work idle; - stop the process tree; - snapshot the filesystem; or -- authorize source destruction. +- authorize source-compute retirement. The handoff binding must preserve those boundaries. @@ -91,7 +91,7 @@ fence host-controlled ingress -> assemble and verify one aggregate recovery point -> durably accept exact recovery-point bytes and identity -> finalize the source generation - -> derive safeToDestroy + -> derive sourceComputeRetirementAuthorized ``` If the handoff is abandoned before process stop, the controller calls @@ -174,9 +174,10 @@ Every mutating step is idempotent under the handoff identity. - Source process or adapter replacement before local state is durable: quarantine unless the host can prove the exact accepted recovery point. -## Destruction Authority +## Source Compute Retirement Authority -`safeToDestroy` is a host-owned durable conclusion bound to: +`sourceComputeRetirementAuthorized` is a host-owned durable conclusion bound +to: - runtime lineage; - source generation; @@ -184,13 +185,18 @@ Every mutating step is idempotent under the handoff identity. - accepted recovery point; and - the current lifecycle revision. -It authorizes removal of only the source compute generation. It never +It authorizes retirement of only the source compute generation. It never authorizes deletion of recovery points, persistent tenant data, external credentials, registry records, or another generation. -New retained work or an operator cancellation must revoke or race with -destruction through the host's durable lifecycle authority. OpenClaw does not -poll for that race after it has stopped. +New retained work or an operator cancellation must revoke or race with source +compute retirement through the host's durable lifecycle authority. OpenClaw +does not poll for that race after it has stopped. + +This fact is not a general `synced` or clean-filesystem state. It is valid only +for the exact runtime lineage, source generation, handoff, recovery point, and +lifecycle revision named by the record. Process absence, an expired suspension +lease, a local snapshot path, or a successful health probe cannot create it. ## Conformance @@ -208,5 +214,6 @@ V1 conformance must prove: - host-protected snapshots cannot be reported as credential-free portable without exact owner portability receipts; - response loss replays the same acceptance; -- digest conflict and unknown outcome block destruction; and -- `safeToDestroy` is generation-scoped and cannot purge persistent data. +- digest conflict and unknown outcome block source compute retirement; and +- `sourceComputeRetirementAuthorized` is generation-scoped and cannot purge + persistent data. diff --git a/rfcs/0013/recovery-point-components-v1-spec.md b/rfcs/0013/recovery-point-components-v1-spec.md index 2c6d0f6c..d8479886 100644 --- a/rfcs/0013/recovery-point-components-v1-spec.md +++ b/rfcs/0013/recovery-point-components-v1-spec.md @@ -40,7 +40,7 @@ This specification does not define: - another SQLite snapshot command, repository, or manifest; - mutation of an RFC 0013 snapshot after publication; - upload, storage transport, retention, or encryption implementation; -- Gateway suspension, final handoff, or source destruction; +- Gateway suspension, final handoff, or source-compute retirement; - restore-on-boot, restored admission, hibernation, or wake; - arbitrary restore hooks or a generic capture-provider registry. @@ -75,6 +75,21 @@ The aggregate layer must treat the directory and its strict `manifest.json` and The composer does not inspect private SQLite schema to infer application state. It consumes the verified owner result. +### Required-component inventory authority + +The runtime state owner, not the host and not the aggregate composer, selects +the complete required-component inventory. The owner must derive it from the +same activation-pinned runtime and selected-agent state that governs the source +generation, then bind that inventory revision and source generation into the +recovery-point input. A caller-supplied list of agent IDs, a filesystem scan, +or whatever databases happen to exist is not authoritative. + +The composer validates and records the owner result but cannot add, remove, or +reinterpret components. Retrying the same capture operation with a different +owner inventory conflicts. A changed selected-agent set requires a new owner +inventory revision and a new recovery point. Missing owner evidence fails +closed rather than treating an empty or partial inventory as complete. + ## Component Model A recovery point contains a closed, ordered component list. V1 component kinds @@ -129,6 +144,9 @@ Illustrative V1 shape: "createdAt": "2026-07-21T15:00:00.000Z", "inventory": { "version": "openclaw-runtime-sqlite-inventory/v1", + "owner": "openclaw-state", + "sourceRuntimeGeneration": "runtime-generation-17", + "revision": "inventory-revision-9", "requiredComponentIds": ["sqlite/global", "sqlite/agent/main"] }, "protection": { @@ -169,9 +187,10 @@ cycles, digest mismatches, and unsupported major versions fail closed. The state owner supplies the complete required component IDs for the selected runtime before composition. V1 requires exactly one global component and the exact selected set of per-agent components. The inventory is canonicalized, -stored in the manifest, and covered by `recoveryPointId`. Composition fails on -a missing or extra component; successfully composing one agent cannot imply -that every agent owned by the selected runtime was captured. +stored in the manifest with its owner, source runtime generation, and revision, +and covered by `recoveryPointId`. Composition fails on a missing or extra +component; successfully composing one agent cannot imply that every agent +owned by the selected runtime was captured. The aggregate manifest does not copy the complete RFC 0013 manifest. It binds that owner manifest by digest and preserves it beside the component artifact. @@ -251,6 +270,8 @@ success-shaped partial result. V1 conformance must prove: - one global and the exact owner-selected per-agent RFC 0013 components; +- the inventory is bound to the source runtime generation and owner revision; +- caller-supplied IDs and filesystem discovery cannot replace owner selection; - rejection of missing and extra components against the bound inventory; - deterministic aggregate identity; - exact owner-manifest binding; diff --git a/rfcs/0013/restored-admission-v1-spec.md b/rfcs/0013/restored-admission-v1-spec.md index a6f8f590..6021d67e 100644 --- a/rfcs/0013/restored-admission-v1-spec.md +++ b/rfcs/0013/restored-admission-v1-spec.md @@ -37,6 +37,7 @@ This specification defines: - external and reconstructed obligation handling; - scheduler reconciliation; - required owner and generic Gateway readiness; +- RFC 0018 readiness projection for the restored recovery point; - one-time restored admission; - same-child replay after coordinator failure. @@ -44,7 +45,7 @@ This specification does not define: - another SQLite restore implementation; - a public generic restore-hook or readiness-provider registry; -- capture, host storage, retention, or source destruction; +- capture, host storage, retention, or source-compute retirement; - host placement, proxy transport, or Channel delivery; - Elastic wake policy. @@ -192,6 +193,53 @@ Ordinary startup remains unchanged because it has no restored-start evidence. Ordinary startup cannot consume a committed restore hold, and restored startup cannot use an ordinary admission path. +## Canonical Readiness Projection + +When enabled, restored admission uses the opt-in readiness result defined by +[RFC 0018](https://github.com/openclaw/rfcs/pull/33). It does not define a +parallel health evaluator or provider registry. This composition is contingent +on RFC 0018 acceptance and activation; the restore hold, admission fence, and +`gateway.restore.status` remain independently complete without it. + +A restored start publishes one required condition and adds the referenced +objects to RFC 0018's identity package. Stable refs name roles; current IDs and +generations name their occupants: + +```text +type: RecoveryPointRestored +subjectRef: openclaw/gateway +relatedSubjectRefs: + - openclaw/restore-operation + - openclaw/recovery-point + - openclaw/runtime-generation + - openclaw/scheduler-reconciliation + - openclaw/state-owner/... + +identity.subjects: + - { ref: openclaw/gateway, kind: gateway, id: } + - { ref: openclaw/restore-operation, kind: restore-operation, + id: } + - { ref: openclaw/recovery-point, kind: recovery-point, + id: } + - { ref: openclaw/runtime-generation, kind: runtime-generation, + generation: } +``` + +The related subject list is bounded by RFC 0018 and uses its canonical subject +deduplication and lifetime rules. `RecoveryPointRestored` is `Unknown` while +required durable evidence is unavailable, `False` when current evidence proves +the selected restore is not admissible, and `True` only after the same durable +record that authorizes admission commits. A required non-`True` result keeps +aggregate readiness false. + +`gateway.restore.status` and RFC 0018 readiness are two projections of one +owner record. The status method adds caller-supplied operation fencing and +typed conflict behavior; `/readyz`, `openclaw ready`, Status, and Gateway +readiness RPC report the active runtime's canonical result. They must expose +the same readiness generation and subject identities. No projection may infer +restore completion from `/healthz`, process existence, database-open success, +or container readiness. + ## Gateway Restore Status V1 adds one read-only Gateway Protocol method: @@ -340,6 +388,28 @@ not steady-state JSON sidecars. A dedicated journal is required because final capture must commit intent before snapshotting the shared state database, and restore must commit intent before the fresh shared state database exists. +### Recovery journal ownership and lifecycle + +OpenClaw core owns the recovery journal schema, versioning, atomic transitions, +migrations, integrity checks, and repair classification. Hosts may retain, +protect, and back up the journal with the runtime protection domain, but must +not write rows, advance transitions, or infer success from file presence. + +The journal must be opened and integrity-checked before a restored start can +publish readiness. Unsupported schema versions, failed migrations, malformed +terminal records, and contradictory operation identities hold or quarantine; +they never fall back to ordinary startup. Repair is an explicit offline owner +operation and cannot fabricate committed acceptance, restore, or readiness +evidence. + +An operation record remains available while its restore hold, destination +generation, status query, crash replay, or retained-delivery reference can +still be active. Cleanup may remove a terminal record only after the owner can +prove that no supported replay or status consumer references it and the host's +retention obligation has expired. Cleanup is idempotent and cannot remove the +currently active restored-admission record. Orphaned or partially committed +records are quarantined rather than age-deleted. + ## Failure Dispositions - **retry same incarnation**: transient response loss with the same child and @@ -363,6 +433,10 @@ V1 conformance must prove: - component dependency order is preserved; - external and reconstructed obligations remain owner-evaluated; - scheduler reconciliation precedes readiness; +- `RecoveryPointRestored` uses RFC 0018 subjects, lifetimes, aggregation, and + bounded projection rules; +- canonical readiness and `gateway.restore.status` project the same durable + readiness generation and identities; - admission opens exactly once from exact durable evidence; - process health alone cannot open admission; - `gateway.restore.status` is read-only, operation-fenced, and returns exact @@ -373,4 +447,6 @@ V1 conformance must prove: while ordinary restored work admission is held; - coordinator crash replay reuses the same child and readiness generation; - preparation and restore execute exactly once; and +- journal schema, migration, corruption, repair, retention, and cleanup remain + OpenClaw-owner operations and fail closed; and - stale, contradictory, corrupt, and fixed-path collision cases fail closed. From 9ef221db86ff66c99f1baeb1ac2067f98af00fb1 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Tue, 28 Jul 2026 06:58:31 -0700 Subject: [PATCH 26/28] docs(rfc-0013): bound restored status admission --- rfcs/0013-cloud-serializable-sqlite-state.md | 7 ++++++- rfcs/0013/restored-admission-v1-spec.md | 17 ++++++++++------- 2 files changed, 16 insertions(+), 8 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 2b40e37d..3f750119 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -362,7 +362,12 @@ readiness, durably records it, and opens its own admission. The host may only observe the result and compare it with the operation and generation it owns. Normal user-work methods remain unavailable while restored admission is held; the status method is allowed only through an authenticated pre-admission -control path, following the existing suspension-control pattern. +control path, following the existing suspension-control pattern. That path +must admit only a bounded, non-enrolling control handshake, track it as active +root work until authentication and connection completion, and lose to any +restart fence. It must not admit nodes, device pairing or enrollment, remote +clients, presence-bearing sessions, or any RPC other than the read-only status +query while work admission is held. #### RFC 0018 readiness composition diff --git a/rfcs/0013/restored-admission-v1-spec.md b/rfcs/0013/restored-admission-v1-spec.md index 6021d67e..053e83bf 100644 --- a/rfcs/0013/restored-admission-v1-spec.md +++ b/rfcs/0013/restored-admission-v1-spec.md @@ -339,13 +339,16 @@ evidence, or failure to read a required committed record returns `UNAVAILABLE` and keeps work admission closed. No error is success-shaped. The method must remain callable while restored admission is held through an -authenticated pre-admission control path. It is exempt only from the restored- -admission work fence, like suspension control methods are exempt from the -suspension work fence; it does not make other RPC methods available. A host -without such a control path may use `/readyz` only as a backoff hint and call -`gateway.restore.status` after readiness, but it must not deliver retained -work until the exact `ready` result matches its operation and destination -generation. +authenticated pre-admission control path. The path may reserve tracked work to +complete authentication, but the pre-auth request shape must be limited to a +non-enrolling host probe and the resulting connection must remain scoped to +read-only status inspection. The lease lasts through handshake completion so +a concurrent restart drains it rather than racing it. Restart always wins; +nodes, pairing or enrollment, remote clients, presence-bearing sessions, and +ordinary RPC methods remain fenced. A host without such a control path may use +`/readyz` only as a backoff hint and call `gateway.restore.status` after +readiness, but it must not deliver retained work until the exact `ready` result +matches its operation and destination generation. For a fresh destination, the recommended held-state transport is the existing authenticated Admin HTTP RPC route bound to the host-controlled loopback path. From 81ad4c8c9759c846576f0e3c60c3cb97ee4f5a4a Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Wed, 12 Aug 2026 19:38:50 -0700 Subject: [PATCH 27/28] docs: refresh scale-to-zero recovery state Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f6da23b8-7231-471e-8fb9-effac01a3e73 --- rfcs/0013-cloud-serializable-sqlite-state.md | 46 +++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 3f750119..04ca1f0f 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -3,7 +3,7 @@ title: SQLite Snapshot Backup Artifacts authors: - giodl created: 2026-06-18 -last_updated: 2026-07-26 +last_updated: 2026-08-13 status: completed issue: https://github.com/openclaw/openclaw/pull/105718 rfc_pr: https://github.com/openclaw/rfcs/pull/20 @@ -312,6 +312,50 @@ durable host acceptance, publication, host wake, or source-compute retirement into OpenClaw. +#### Current main leverage and remaining gaps + +OpenClaw has added useful owner-level foundations since this follow-on was +first drafted: + +- [openclaw/openclaw#118393](https://github.com/openclaw/openclaw/pull/118393) + now binds Cron cancellation settlement to each active run and keeps + unresolved work visible to Gateway suspension through bounded shutdown. + This strengthens the source-work fence; it does not register or deliver a + wake after compute reaches zero. +- [openclaw/openclaw#117705](https://github.com/openclaw/openclaw/pull/117705) + keeps Gateway-backed agent turns on a cold CLI path. This reduces replacement + startup overhead; it does not provide retained ingress, compute wake, or + restored-readiness gating. +- Vincent's snapshot durability campaign in + [openclaw/openclaw#113306](https://github.com/openclaw/openclaw/issues/113306) + has landed durable parent publication and pending-snapshot recovery. The + generic helper's staging-cleanup ownership contract remains an explicit + maintainer decision. +- Vincent's open + [openclaw/openclaw#117258](https://github.com/openclaw/openclaw/pull/117258) + isolates post-commit auth snapshot publication per runtime owner. It is + relevant evidence for owner-local reconciliation, but it is not a portable + recovery receipt or a restored-admission signal. + +The remaining end-to-end gaps are therefore narrower but still host/runtime +integration work: + +1. resolve the authoritative selected-agent inventory, recovery-journal + lifecycle, and public `gateway.restore.status` ownership; +2. bind one durable accepted-final-handoff lookup to the destination runtime + generation; +3. retain and coalesce Teams, API, and Cron wake causes while compute is absent; +4. place or wake replacement compute without granting callers recovery-point + or admission authority; and +5. replay each retained cause only after exact restored readiness, with durable + acknowledgement and crash recovery. + +There is also an RFC-ownership decision before merge: repository review has +asked whether this lifecycle belongs in a standalone draft RFC rather than as +an optional follow-on to completed RFC 0013. Until maintainers settle that +placement, RFC 0013 remains authoritative only for the completed SQLite +artifact contract and these lifecycle sections remain review material. + The intended observable outcome of the complete host composition is: - idle compute may reach zero without treating a raw live filesystem copy as a From 65d74e137db53e517b21fc17b63696fa082717e9 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Thu, 13 Aug 2026 11:03:53 -0700 Subject: [PATCH 28/28] docs: clarify scale-to-zero follow-on Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f6da23b8-7231-471e-8fb9-effac01a3e73 --- rfcs/0013-cloud-serializable-sqlite-state.md | 47 +++++++++++++------- 1 file changed, 32 insertions(+), 15 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 04ca1f0f..c4efca53 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -180,6 +180,25 @@ owner-authored substrate for optional recovery workflows, but those workflows must compose the landed command rather than reinterpret live SQLite files or duplicate snapshot creation, verification, repository, or restore behavior. +For scale-to-zero, the goal is simple: let an idle OpenClaw Gateway stop using +compute, then wake a replacement when work arrives without losing state or +sending work to it before it is ready. + +Most of the OpenClaw-side foundation already exists. Vincent's snapshot work +provides durable state capture and restore. Peter's Gateway suspension and Cron +work provides a cooperative way to stop and reconcile tracked work. The +remaining need is a small handoff between OpenClaw and its host: + +1. the host remembers that work is waiting and wakes the replacement; +2. OpenClaw restores the accepted state and reports when the Gateway is ready; +3. the host waits for that readiness before sending the queued work. + +This follow-on does not add a host scheduler, Teams transport, compute +placement service, or retained-payload store to OpenClaw. It defines the +application-consistency facts a host needs to use its existing infrastructure +safely. The detailed contracts below preserve generation fencing, replay, and +failure handling for implementations. + The scale-to-zero outcome and user evidence are tracked in [openclaw/openclaw#114145](https://github.com/openclaw/openclaw/issues/114145). This follow-on deliberately builds on, rather than replaces, work Vincent and @@ -199,7 +218,7 @@ they are not a required decomposition. If maintainers prefer a smaller or different core seam that satisfies the same recovery and restored-admission outcomes, that is a valid resolution of the umbrella issue. -#### Why a recovery lifecycle is needed +#### What snapshots and suspension do not finish Per-user and event-driven hosts can stop paying for resident compute only when they can retire one Gateway generation and later admit work on a replacement @@ -312,7 +331,7 @@ durable host acceptance, publication, host wake, or source-compute retirement into OpenClaw. -#### Current main leverage and remaining gaps +#### What already exists and what remains OpenClaw has added useful owner-level foundations since this follow-on was first drafted: @@ -337,18 +356,16 @@ first drafted: relevant evidence for owner-local reconciliation, but it is not a portable recovery receipt or a restored-admission signal. -The remaining end-to-end gaps are therefore narrower but still host/runtime -integration work: +The remaining gaps are narrower and sit at the host/OpenClaw boundary: -1. resolve the authoritative selected-agent inventory, recovery-journal - lifecycle, and public `gateway.restore.status` ownership; -2. bind one durable accepted-final-handoff lookup to the destination runtime - generation; -3. retain and coalesce Teams, API, and Cron wake causes while compute is absent; -4. place or wake replacement compute without granting callers recovery-point - or admission authority; and -5. replay each retained cause only after exact restored readiness, with durable - acknowledgement and crash recovery. +1. agree who owns the selected-agent inventory, recovery journal, and public + `gateway.restore.status` method; +2. let the host find the accepted recovery point and choose one fenced + replacement generation; +3. remember and combine Teams, API, and Cron reasons to wake while compute is + absent; and +4. send each queued item only after OpenClaw reports exact restored readiness, + with durable acknowledgement and crash replay. There is also an RFC-ownership decision before merge: repository review has asked whether this lifecycle belongs in a standalone draft RFC rather than as @@ -371,7 +388,7 @@ capture, and restored-admission slices. Host acceptance, retained ingress, wake, and source-compute retirement remain separate review and implementation work. -#### One bidirectional host/runtime contract +#### Keep the host/OpenClaw handoff small The recommended V1 surface is one bidirectional protocol contract without an OpenClaw-owned lifecycle coordinator or abstract `Host` base class: @@ -499,7 +516,7 @@ Gateway admission state, or central retained-payload store. The Gateway Protocol package should own the new wire schemas beside `gateway-suspend`; a separate lifecycle SDK package is not required for V1. -#### Deferred cron and retained-ingress wake composition +#### Wake and deliver work after compute reaches zero The current three-PR evidence stack intentionally stops at restored admission. A complete scale-to-zero host also needs to wake without relying on an