From b5b35e4bb58187d60a542cc79f94caeb4ac4149a Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Thu, 18 Jun 2026 14:25:10 -0700 Subject: [PATCH 01/24] Add cloud-serializable SQLite state RFC --- rfcs/0013-cloud-serializable-sqlite-state.md | 181 +++++++++++++++++++ 1 file changed, 181 insertions(+) create mode 100644 rfcs/0013-cloud-serializable-sqlite-state.md diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md new file mode 100644 index 00000000..c9c23998 --- /dev/null +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -0,0 +1,181 @@ +--- +title: Cloud-Serializable SQLite State +authors: + - giodl +created: 2026-06-18 +last_updated: 2026-06-18 +status: draft +issue: +rfc_pr: TBD +--- + +# Proposal: Cloud-Serializable SQLite State + +## Summary + +Define durability and recovery requirements for OpenClaw-owned SQLite state in managed deployments. SQLite remains the hot local runtime database; cloud storage holds verified snapshots and deltas, not live database files. + +## Motivation + +OpenClaw is moving runtime state into SQLite-backed stores. That is a good local runtime shape, but managed deployments need more than local files: state must survive process restarts, container replacement, node loss, and service redeploys without relying on shared network filesystems or whole-file copy as the only durability mechanism. + +The operational problem is not primarily database selection. It is that SQLite state needs a defined way to become portable, streamable, restorable, and schedulable as an owned artifact. A managed service needs clear semantics for checkpointing, incremental durability, ownership transfer, restore verification, and failover. + +## Goals + +- Keep SQLite as the hot local runtime database for this proposal. +- Define what it means for OpenClaw-owned SQLite databases to be cloud-serializable. +- Require consistent checkpoints that handle SQLite WAL state correctly. +- Define verified restore as a first-class behavior, including booting OpenClaw from restored state. +- Avoid hot writes over network filesystems as a durability or concurrency strategy. +- Define incremental durability requirements beyond repeated whole-file copies. +- Define writer ownership and lease expectations for each durable SQLite database. +- Define lifecycle metadata needed to validate, order, restore, and audit snapshots and deltas. +- Keep the design compatible with existing global state, per-agent state, and dedicated store boundaries. + +## Non-Goals + +- This RFC does not choose PostgreSQL, libSQL, remote SQLite, object storage, or any other backend product. +- This RFC does not define a general database abstraction layer. +- This RFC does not replace the session/transcript migration plan tracked by openclaw/openclaw#88838. +- This RFC does not define tenant isolation, row-level authorization, or a multi-tenant schema model. +- This RFC does not define FTS/vector search portability. +- This RFC does not require real-time multi-writer SQLite over shared storage. +- This RFC does not define the final backup UI, CLI, or managed-service control plane. + +## Proposal + +### Cloud-serializable SQLite state + +An OpenClaw-owned SQLite database is cloud-serializable when it can be safely captured, uploaded, restored, verified, and resumed on a different host or container without relying on a live shared filesystem. + +The unit of durability is an existing OpenClaw-owned SQLite database, such as shared state, per-agent state, or a dedicated owner store. This RFC does not rename or redesign those logical units; it defines durability behavior that can apply to each unit. + +### Safe checkpointing + +Each durable SQLite database must have a checkpoint operation that produces a consistent restore point. + +A checkpoint must: + +- handle `.sqlite`, `-wal`, and `-shm` state correctly +- avoid half-copied database state +- record the schema version and database identity +- record the checkpoint cursor or equivalent replay position +- produce enough metadata to verify restore integrity + +The implementation may use SQLite online backup APIs, WAL checkpoints, page-level capture, or another implementation-specific mechanism, but the observable contract must be a consistent restore point. + +### Snapshot and delta artifacts + +Cloud storage should store durable artifacts, not a live database file used directly by the runtime. + +The artifact model should support: + +- periodic compact snapshots +- incremental deltas between snapshots +- ordered manifests for snapshots and deltas +- content hashes or equivalent integrity checks +- resumable upload and download +- restore from the latest valid snapshot plus ordered deltas + +The delta mechanism can be WAL-frame based, page based, logical-change based, or backend-native. The RFC requires the contract, not one specific encoding. + +### Writer ownership and leases + +OpenClaw must not treat a network filesystem as the concurrency model for hot SQLite writes. + +Each durable SQLite database should have explicit writer ownership. A managed deployment can move ownership, but only through a controlled sequence: + +1. acquire ownership or a writer lease for the database +2. hydrate local disk from a verified restore point when needed +3. open and write SQLite locally +4. periodically checkpoint and upload durable artifacts +5. release ownership with a final durable checkpoint +6. allow failover to restore from the latest verified durable point + +Concurrent readers and replicas can be designed later, but the write path must have one clear owner at a time unless a future RFC defines a stronger multi-writer mechanism. + +### Restore verification + +Restore is a required behavior, not an incidental backup side effect. + +A restore operation must: + +- download the selected snapshot and required deltas +- verify artifact ordering and integrity +- hydrate local database files before runtime opens them +- run SQLite integrity checks or equivalent validation +- confirm the restored schema version is supported +- record the restore point OpenClaw is resuming from + +The first implementation milestone should prove that OpenClaw can boot from restored state on a fresh host or container. + +### Lifecycle metadata + +Each durable database needs metadata sufficient to reason about ownership, replay, and integrity. + +At minimum, durability metadata should include: + +- database id +- database kind or owner +- schema version +- current writer owner or lease holder +- snapshot generation +- checkpoint or WAL cursor +- artifact manifest id +- integrity hash or verification record +- last durable upload time +- restore source and restore point when hydrated + +The exact storage location for this metadata is implementation-defined, but it must be available before opening a database for managed-runtime writes. + +### Durability provider shape + +The implementation can start as a SQLite-specific durability provider rather than a database abstraction layer. + +A minimal shape is: + +```ts +type SqliteDurabilityProvider = { + checkpoint(dbRef): Promise; + uploadSnapshot(dbRef): Promise; + uploadDelta(dbRef, sinceCursor): Promise; + restore(targetPath, restorePoint): Promise; + verify(targetPath): Promise; +}; +``` + +This keeps SQLite runtime access local while making persistence cloud-aware. + +### First milestone + +The first implementation milestone should be intentionally small: + +1. choose one existing OpenClaw-owned SQLite database +2. produce a consistent checkpoint artifact +3. restore it into a fresh directory or host +4. verify integrity +5. boot OpenClaw from restored state +6. document that hot writes over network filesystems remain unsupported + +Incremental deltas, leases, and failover should follow after snapshot/restore is proven. + +## Rationale + +This approach targets the reliability problem directly. It does not require OpenClaw to choose a second database backend before it has defined durability and restore semantics for the SQLite state it already owns. + +Treating cloud storage as artifact storage avoids the common failure mode where object storage or network filesystems are used as if they were local disk. SQLite remains local and authoritative while running. Managed durability comes from verified snapshots, deltas, manifests, and restore procedures. + +Explicit writer ownership keeps horizontal service orchestration honest. A service can move work between hosts, but it must move ownership and restore state deliberately rather than letting several instances write the same SQLite database through shared storage. + +The proposal also keeps logical storage boundaries out of scope. OpenClaw already has shared state, per-agent state, and owner-specific stores; this RFC defines how any of those databases can become durable cloud artifacts. + +## Unresolved questions + +- Which existing SQLite database should be used for the first snapshot/restore proof? +- Should the first checkpoint implementation use SQLite online backup, WAL checkpointing, page capture, or a higher-level export format? +- What object/blob storage provider should be used for the first managed-service proof? +- What is the acceptable data-loss window for managed deployments before deltas are implemented? +- Where should writer lease metadata live before a database is opened? +- Should restore verification run during startup, doctor, a managed-control-plane action, or all three? +- Which artifacts should be included with database restore for support/debug exports versus canonical runtime recovery? From 1c0bbe1b4b2090a74f83e056bc95f3a181660f51 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Thu, 18 Jun 2026 14:26:06 -0700 Subject: [PATCH 02/24] Update RFC PR metadata --- rfcs/0013-cloud-serializable-sqlite-state.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index c9c23998..0a05083e 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -6,7 +6,7 @@ created: 2026-06-18 last_updated: 2026-06-18 status: draft issue: -rfc_pr: TBD +rfc_pr: https://github.com/openclaw/rfcs/pull/20 --- # Proposal: Cloud-Serializable SQLite State From 56abc0adbac2c8bdb2f15f49d4486ea8d8ac29ac Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Thu, 18 Jun 2026 14:49:20 -0700 Subject: [PATCH 03/24] Clarify opt-in durability provider model --- rfcs/0013-cloud-serializable-sqlite-state.md | 108 ++++++++++++++++--- 1 file changed, 94 insertions(+), 14 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 0a05083e..a79087e2 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -13,7 +13,7 @@ rfc_pr: https://github.com/openclaw/rfcs/pull/20 ## Summary -Define durability and recovery requirements for OpenClaw-owned SQLite state in managed deployments. SQLite remains the hot local runtime database; cloud storage holds verified snapshots and deltas, not live database files. +Define opt-in durability and recovery requirements for OpenClaw-owned SQLite state in managed deployments. SQLite remains the hot local runtime database; optional durability providers store verified snapshots and deltas as artifacts, not live database files. ## Motivation @@ -21,9 +21,12 @@ OpenClaw is moving runtime state into SQLite-backed stores. That is a good local The operational problem is not primarily database selection. It is that SQLite state needs a defined way to become portable, streamable, restorable, and schedulable as an owned artifact. A managed service needs clear semantics for checkpointing, incremental durability, ownership transfer, restore verification, and failover. +This proposal is intentionally opt-in. Default local OpenClaw should keep its existing SQLite behavior and backup commands unless an operator enables a durability provider or managed deployment mode. + ## Goals - Keep SQLite as the hot local runtime database for this proposal. +- Keep cloud-serializable durability opt-in, with no default cloud dependency. - Define what it means for OpenClaw-owned SQLite databases to be cloud-serializable. - Require consistent checkpoints that handle SQLite WAL state correctly. - Define verified restore as a first-class behavior, including booting OpenClaw from restored state. @@ -32,11 +35,14 @@ The operational problem is not primarily database selection. It is that SQLite s - Define writer ownership and lease expectations for each durable SQLite database. - Define lifecycle metadata needed to validate, order, restore, and audit snapshots and deltas. - Keep the design compatible with existing global state, per-agent state, and dedicated store boundaries. +- Let artifact storage, retention, scheduling, and managed failover be provider-owned or deployment-owned. ## Non-Goals - This RFC does not choose PostgreSQL, libSQL, remote SQLite, object storage, or any other backend product. - This RFC does not define a general database abstraction layer. +- This RFC does not make cloud durability mandatory for local, self-hosted, or development OpenClaw installs. +- This RFC does not require object-store credentials, a lease service, or a managed-service control plane in the default runtime. - This RFC does not replace the session/transcript migration plan tracked by openclaw/openclaw#88838. - This RFC does not define tenant isolation, row-level authorization, or a multi-tenant schema model. - This RFC does not define FTS/vector search portability. @@ -45,6 +51,75 @@ The operational problem is not primarily database selection. It is that SQLite s ## Proposal +### Opt-in durability mode + +Cloud-serializable durability is an optional mode for managed and production operators. It should not change the default local SQLite runtime. + +Default OpenClaw keeps: + +- local SQLite state +- existing backup create and verify behavior +- no cloud dependency +- no object storage configuration +- no writer lease service +- no additional managed-service scheduler + +Opt-in durability mode adds a provider-backed durability path. Core owns the SQLite-safe primitives and contract. Providers or deployments own where artifacts go, how retention is scheduled, and how managed failover is orchestrated. + +Core should own: + +- consistent SQLite checkpoint creation +- restore or hydrate before opening runtime state +- restored database verification +- lifecycle metadata shape +- safety rules such as no hot writes over network filesystems + +Providers can own: + +- local snapshot artifact storage +- S3-compatible artifact storage +- Azure Blob or other cloud artifact storage +- retention policy and upload scheduling +- writer lease coordination +- managed failover orchestration +- integration with external tools such as Litestream or LiteFS, if later accepted + +### Architecture + +```mermaid +flowchart LR + subgraph Runtime[Default OpenClaw runtime] + App[OpenClaw process] + DB[(Local SQLite database)] + App -->|sync local reads/writes| DB + end + + subgraph Core[Core durability primitives] + Checkpoint[checkpoint] + Verify[verify] + Restore[restore / hydrate] + Metadata[lifecycle metadata] + end + + subgraph Providers[Optional durability providers] + Local[local snapshot provider] + Object[object/blob artifact provider] + Lease[writer lease / failover provider] + end + + DB --> Checkpoint + Checkpoint --> Metadata + Checkpoint --> Local + Checkpoint --> Object + Object --> Restore + Local --> Restore + Restore --> Verify + Verify --> DB + Lease -. optional ownership .-> App +``` + +The diagram is a responsibility split, not a runtime requirement. Default OpenClaw can run with only the runtime box. Managed deployments opt into the provider side. + ### Cloud-serializable SQLite state An OpenClaw-owned SQLite database is cloud-serializable when it can be safely captured, uploaded, restored, verified, and resumed on a different host or container without relying on a live shared filesystem. @@ -63,7 +138,7 @@ A checkpoint must: - record the checkpoint cursor or equivalent replay position - produce enough metadata to verify restore integrity -The implementation may use SQLite online backup APIs, WAL checkpoints, page-level capture, or another implementation-specific mechanism, but the observable contract must be a consistent restore point. +The implementation may use SQLite online backup APIs, `VACUUM INTO`, WAL checkpoints, page-level capture, or another implementation-specific mechanism, but the observable contract must be a consistent restore point. ### Snapshot and delta artifacts @@ -78,13 +153,13 @@ The artifact model should support: - resumable upload and download - restore from the latest valid snapshot plus ordered deltas -The delta mechanism can be WAL-frame based, page based, logical-change based, or backend-native. The RFC requires the contract, not one specific encoding. +The delta mechanism can be WAL-frame based, page based, logical-change based, external-tool based, or backend-native. The RFC requires the contract, not one specific encoding. ### Writer ownership and leases OpenClaw must not treat a network filesystem as the concurrency model for hot SQLite writes. -Each durable SQLite database should have explicit writer ownership. A managed deployment can move ownership, but only through a controlled sequence: +Each durable SQLite database should have explicit writer ownership when the deployment allows failover or multiple possible hosts. A managed deployment can move ownership, but only through a controlled sequence: 1. acquire ownership or a writer lease for the database 2. hydrate local disk from a verified restore point when needed @@ -97,11 +172,11 @@ Concurrent readers and replicas can be designed later, but the write path must h ### Restore verification -Restore is a required behavior, not an incidental backup side effect. +Restore is a required behavior for opt-in durability mode, not an incidental backup side effect. A restore operation must: -- download the selected snapshot and required deltas +- download or locate the selected snapshot and required deltas - verify artifact ordering and integrity - hydrate local database files before runtime opens them - run SQLite integrity checks or equivalent validation @@ -119,7 +194,7 @@ At minimum, durability metadata should include: - database id - database kind or owner - schema version -- current writer owner or lease holder +- current writer owner or lease holder, when leases are enabled - snapshot generation - checkpoint or WAL cursor - artifact manifest id @@ -127,7 +202,7 @@ At minimum, durability metadata should include: - last durable upload time - restore source and restore point when hydrated -The exact storage location for this metadata is implementation-defined, but it must be available before opening a database for managed-runtime writes. +The exact storage location for this metadata is implementation-defined, but opt-in managed deployments must be able to access it before opening a database for managed-runtime writes. ### Durability provider shape @@ -145,20 +220,20 @@ type SqliteDurabilityProvider = { }; ``` -This keeps SQLite runtime access local while making persistence cloud-aware. +This keeps SQLite runtime access local while making persistence cloud-aware. A local snapshot provider can be the reference implementation. Cloud/object-store providers can come later without changing the default local runtime. ### First milestone -The first implementation milestone should be intentionally small: +The first implementation milestone should be intentionally small and opt-in: 1. choose one existing OpenClaw-owned SQLite database -2. produce a consistent checkpoint artifact +2. produce a consistent local snapshot artifact 3. restore it into a fresh directory or host 4. verify integrity 5. boot OpenClaw from restored state 6. document that hot writes over network filesystems remain unsupported -Incremental deltas, leases, and failover should follow after snapshot/restore is proven. +Incremental deltas, object storage, leases, and failover should follow after snapshot/restore is proven. ## Rationale @@ -166,6 +241,10 @@ This approach targets the reliability problem directly. It does not require Open Treating cloud storage as artifact storage avoids the common failure mode where object storage or network filesystems are used as if they were local disk. SQLite remains local and authoritative while running. Managed durability comes from verified snapshots, deltas, manifests, and restore procedures. +Making the feature opt-in keeps the default OpenClaw runtime simple. Local and development users should not need object storage, a lease service, or a managed scheduler to keep using SQLite. + +Keeping core responsible for SQLite-safe primitives is important because safe checkpoints and restores need access to database paths, WAL behavior, schema versions, and integrity checks. Provider-owned artifact storage keeps cloud credentials, retention policy, and managed failover out of the default core runtime. + Explicit writer ownership keeps horizontal service orchestration honest. A service can move work between hosts, but it must move ownership and restore state deliberately rather than letting several instances write the same SQLite database through shared storage. The proposal also keeps logical storage boundaries out of scope. OpenClaw already has shared state, per-agent state, and owner-specific stores; this RFC defines how any of those databases can become durable cloud artifacts. @@ -173,9 +252,10 @@ The proposal also keeps logical storage boundaries out of scope. OpenClaw alread ## Unresolved questions - Which existing SQLite database should be used for the first snapshot/restore proof? -- Should the first checkpoint implementation use SQLite online backup, WAL checkpointing, page capture, or a higher-level export format? -- What object/blob storage provider should be used for the first managed-service proof? +- Should the first checkpoint implementation use SQLite online backup, `VACUUM INTO`, WAL checkpointing, page capture, or a higher-level export format? +- Should the reference provider be a local snapshot provider only, or should it include one object/blob storage provider? - What is the acceptable data-loss window for managed deployments before deltas are implemented? - Where should writer lease metadata live before a database is opened? - Should restore verification run during startup, doctor, a managed-control-plane action, or all three? - Which artifacts should be included with database restore for support/debug exports versus canonical runtime recovery? +- Should external tools such as Litestream or LiteFS be provider integrations, deployment recommendations, or out of scope for OpenClaw-owned code? From 39963098a31896cb705b9d3dbd54477d2457ca6b Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Thu, 18 Jun 2026 14:58:59 -0700 Subject: [PATCH 04/24] Refocus RFC on snapshot plugin --- rfcs/0013-cloud-serializable-sqlite-state.md | 284 ++++++++++++------- 1 file changed, 180 insertions(+), 104 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index a79087e2..6b21b39b 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -1,5 +1,5 @@ --- -title: Cloud-Serializable SQLite State +title: SQLite State Snapshot Plugin authors: - giodl created: 2026-06-18 @@ -9,62 +9,89 @@ issue: rfc_pr: https://github.com/openclaw/rfcs/pull/20 --- -# Proposal: Cloud-Serializable SQLite State +# Proposal: SQLite State Snapshot Plugin ## Summary -Define opt-in durability and recovery requirements for OpenClaw-owned SQLite state in managed deployments. SQLite remains the hot local runtime database; optional durability providers store verified snapshots and deltas as artifacts, not live database files. +Define an opt-in `snapshot` plugin for OpenClaw-owned SQLite state. The plugin produces SQLite-safe state snapshots that can be verified, restored, and used as the foundation for future failover. + +SQLite remains the hot local runtime database. The plugin does not replace SQLite, introduce a second required database backend, or make cloud storage mandatory. It gives managed and production operators a clearer answer to a narrower problem: how OpenClaw state becomes a portable, restorable artifact when a process, container, or host needs to be replaced. + +The plugin can expose its own commands and also extend the existing `openclaw backup` command surface. ## Motivation -OpenClaw is moving runtime state into SQLite-backed stores. That is a good local runtime shape, but managed deployments need more than local files: state must survive process restarts, container replacement, node loss, and service redeploys without relying on shared network filesystems or whole-file copy as the only durability mechanism. +OpenClaw is moving runtime state into SQLite-backed stores. That is a good local runtime shape, but operational reliability depends on more than having files on disk. + +The pain points are concrete: + +- copying a live SQLite database file can miss WAL state or capture a half-consistent database +- whole-file copying gets expensive as state grows +- network filesystems are not a safe concurrency strategy for hot SQLite writes +- backup archives are only useful if restore is verified and repeatable +- container or host replacement needs state hydration before OpenClaw opens the database +- failover cannot be credible until OpenClaw has a known-good restore point -The operational problem is not primarily database selection. It is that SQLite state needs a defined way to become portable, streamable, restorable, and schedulable as an owned artifact. A managed service needs clear semantics for checkpointing, incremental durability, ownership transfer, restore verification, and failover. +The operational question is therefore not primarily "which database should OpenClaw use?" Greater minds can make the long-term database choice separately. This RFC focuses on the SQLite state OpenClaw already owns and asks for a practical extension point around it. -This proposal is intentionally opt-in. Default local OpenClaw should keep its existing SQLite behavior and backup commands unless an operator enables a durability provider or managed deployment mode. +The proposed answer is a `snapshot` plugin: an opt-in extension that turns local SQLite state into verified snapshot artifacts. Those artifacts can later support cloud uploads, retention policies, warm standby, and failover, but the first value is simpler: make state capture and restore correct. ## Goals - Keep SQLite as the hot local runtime database for this proposal. -- Keep cloud-serializable durability opt-in, with no default cloud dependency. -- Define what it means for OpenClaw-owned SQLite databases to be cloud-serializable. -- Require consistent checkpoints that handle SQLite WAL state correctly. -- Define verified restore as a first-class behavior, including booting OpenClaw from restored state. +- Make snapshot behavior opt-in through a plugin extension. +- Produce consistent SQLite snapshots that handle WAL state correctly. +- Make restore and verification first-class behaviors, not incidental backup side effects. +- Allow the plugin to add commands under `openclaw snapshot` and extend `openclaw backup`. +- Keep default local OpenClaw behavior unchanged when the plugin is not installed or enabled. - Avoid hot writes over network filesystems as a durability or concurrency strategy. -- Define incremental durability requirements beyond repeated whole-file copies. -- Define writer ownership and lease expectations for each durable SQLite database. -- Define lifecycle metadata needed to validate, order, restore, and audit snapshots and deltas. +- Define lifecycle metadata needed to validate, order, restore, and audit snapshots. +- Leave cloud artifact storage, retention, scheduling, and failover orchestration to optional providers or later RFCs. - Keep the design compatible with existing global state, per-agent state, and dedicated store boundaries. -- Let artifact storage, retention, scheduling, and managed failover be provider-owned or deployment-owned. ## Non-Goals - This RFC does not choose PostgreSQL, libSQL, remote SQLite, object storage, or any other backend product. - This RFC does not define a general database abstraction layer. -- This RFC does not make cloud durability mandatory for local, self-hosted, or development OpenClaw installs. +- This RFC does not make snapshots, cloud storage, or managed failover mandatory for local, self-hosted, or development OpenClaw installs. - This RFC does not require object-store credentials, a lease service, or a managed-service control plane in the default runtime. - This RFC does not replace the session/transcript migration plan tracked by openclaw/openclaw#88838. - This RFC does not define tenant isolation, row-level authorization, or a multi-tenant schema model. - This RFC does not define FTS/vector search portability. - This RFC does not require real-time multi-writer SQLite over shared storage. -- This RFC does not define the final backup UI, CLI, or managed-service control plane. +- This RFC does not define the final managed failover control plane. ## Proposal -### Opt-in durability mode +### Plugin shape -Cloud-serializable durability is an optional mode for managed and production operators. It should not change the default local SQLite runtime. +Add an opt-in `snapshot` plugin that owns SQLite-safe snapshot and restore workflows for OpenClaw state. -Default OpenClaw keeps: +The plugin should be installable and removable like other OpenClaw plugins. When it is absent, default OpenClaw keeps its current local SQLite behavior and existing backup commands. -- local SQLite state -- existing backup create and verify behavior -- no cloud dependency -- no object storage configuration -- no writer lease service -- no additional managed-service scheduler +The plugin can add a direct command surface: -Opt-in durability mode adds a provider-backed durability path. Core owns the SQLite-safe primitives and contract. Providers or deployments own where artifacts go, how retention is scheduled, and how managed failover is orchestrated. +```text +openclaw snapshot create +openclaw snapshot verify +openclaw snapshot restore +openclaw snapshot list +openclaw snapshot status +``` + +It can also extend the existing backup surface so operators can use the name OpenClaw already has for this class of work: + +```text +openclaw backup snapshot +openclaw backup restore +openclaw backup status +``` + +The intent is not to split the product vocabulary. `snapshot` is the plugin and capability name; `backup` remains the user-facing home for backup and restore workflows where that is more natural. + +### Responsibility split + +Core should expose or own the SQLite-safe primitives that require knowledge of OpenClaw state paths, WAL behavior, schema versions, and integrity checks. Core should own: @@ -74,14 +101,23 @@ Core should own: - lifecycle metadata shape - safety rules such as no hot writes over network filesystems -Providers can own: +The `snapshot` plugin should own: + +- snapshot command UX +- local snapshot artifact creation +- snapshot manifest creation and verification +- restore workflow orchestration +- optional integration with `openclaw backup` +- provider hooks for storage backends + +Optional providers can own: -- local snapshot artifact storage +- local snapshot repositories - S3-compatible artifact storage - Azure Blob or other cloud artifact storage - retention policy and upload scheduling - writer lease coordination -- managed failover orchestration +- warm standby or managed failover orchestration - integration with external tools such as Litestream or LiteFS, if later accepted ### Architecture @@ -91,171 +127,211 @@ flowchart LR subgraph Runtime[Default OpenClaw runtime] App[OpenClaw process] DB[(Local SQLite database)] - App -->|sync local reads/writes| DB + App -->|local reads/writes| DB end - subgraph Core[Core durability primitives] + subgraph Core[SQLite-safe core primitives] Checkpoint[checkpoint] Verify[verify] Restore[restore / hydrate] Metadata[lifecycle metadata] end - subgraph Providers[Optional durability providers] - Local[local snapshot provider] - Object[object/blob artifact provider] - Lease[writer lease / failover provider] + subgraph Plugin[snapshot plugin] + Cmd[openclaw snapshot] + Backup[openclaw backup extensions] + Manifest[snapshot manifest] + LocalRepo[(local snapshot repo)] + end + + subgraph Providers[Optional providers] + Object[(object/blob storage)] + Retention[retention schedule] + Failover[standby / failover] end DB --> Checkpoint Checkpoint --> Metadata - Checkpoint --> Local - Checkpoint --> Object + Checkpoint --> Manifest + Cmd --> Checkpoint + Cmd --> Restore + Backup --> Cmd + Manifest --> LocalRepo + LocalRepo --> Restore + LocalRepo --> Object Object --> Restore - Local --> Restore Restore --> Verify Verify --> DB - Lease -. optional ownership .-> App + Retention -. optional .-> LocalRepo + Failover -. later .-> Restore ``` -The diagram is a responsibility split, not a runtime requirement. Default OpenClaw can run with only the runtime box. Managed deployments opt into the provider side. - -### Cloud-serializable SQLite state +The diagram is a responsibility split, not a default runtime requirement. Default OpenClaw can run with only the runtime box. Operators opt into the plugin when they need verified snapshot and restore workflows. -An OpenClaw-owned SQLite database is cloud-serializable when it can be safely captured, uploaded, restored, verified, and resumed on a different host or container without relying on a live shared filesystem. +### Snapshot semantics -The unit of durability is an existing OpenClaw-owned SQLite database, such as shared state, per-agent state, or a dedicated owner store. This RFC does not rename or redesign those logical units; it defines durability behavior that can apply to each unit. +An OpenClaw-owned SQLite database is snapshot-safe when it can be captured, verified, restored, and resumed on another host or directory without relying on a live shared filesystem. -### Safe checkpointing +The unit of snapshotting is an existing OpenClaw-owned SQLite database, such as shared state, per-agent state, or a dedicated owner store. This RFC does not rename or redesign those logical units; it defines snapshot behavior that can apply to each unit. -Each durable SQLite database must have a checkpoint operation that produces a consistent restore point. - -A checkpoint must: +A snapshot must: - handle `.sqlite`, `-wal`, and `-shm` state correctly - avoid half-copied database state - record the schema version and database identity -- record the checkpoint cursor or equivalent replay position +- record the checkpoint cursor or equivalent replay position when available - produce enough metadata to verify restore integrity +- be restorable before OpenClaw opens the database for runtime writes -The implementation may use SQLite online backup APIs, `VACUUM INTO`, WAL checkpoints, page-level capture, or another implementation-specific mechanism, but the observable contract must be a consistent restore point. +The implementation may use SQLite online backup APIs, `VACUUM INTO`, WAL checkpoints, page-level capture, or another implementation-specific mechanism. The observable contract is a consistent restore point. -### Snapshot and delta artifacts +### Snapshot artifacts -Cloud storage should store durable artifacts, not a live database file used directly by the runtime. +Snapshot storage should store durable artifacts, not a live database file used directly by the runtime. The artifact model should support: -- periodic compact snapshots -- incremental deltas between snapshots -- ordered manifests for snapshots and deltas +- compact snapshot artifacts +- ordered manifests - content hashes or equivalent integrity checks -- resumable upload and download -- restore from the latest valid snapshot plus ordered deltas - -The delta mechanism can be WAL-frame based, page based, logical-change based, external-tool based, or backend-native. The RFC requires the contract, not one specific encoding. - -### Writer ownership and leases +- optional incremental deltas after the first milestone +- resumable upload and download when a remote provider is configured +- restore from the latest valid snapshot plus any required ordered deltas -OpenClaw must not treat a network filesystem as the concurrency model for hot SQLite writes. - -Each durable SQLite database should have explicit writer ownership when the deployment allows failover or multiple possible hosts. A managed deployment can move ownership, but only through a controlled sequence: - -1. acquire ownership or a writer lease for the database -2. hydrate local disk from a verified restore point when needed -3. open and write SQLite locally -4. periodically checkpoint and upload durable artifacts -5. release ownership with a final durable checkpoint -6. allow failover to restore from the latest verified durable point - -Concurrent readers and replicas can be designed later, but the write path must have one clear owner at a time unless a future RFC defines a stronger multi-writer mechanism. +The delta mechanism can be WAL-frame based, page based, logical-change based, external-tool based, or backend-native. This RFC requires the contract, not one specific encoding. ### Restore verification -Restore is a required behavior for opt-in durability mode, not an incidental backup side effect. +Restore is a required behavior for the `snapshot` plugin, not an incidental backup side effect. A restore operation must: -- download or locate the selected snapshot and required deltas +- locate the selected snapshot and required artifacts - verify artifact ordering and integrity - hydrate local database files before runtime opens them - run SQLite integrity checks or equivalent validation - confirm the restored schema version is supported - record the restore point OpenClaw is resuming from -The first implementation milestone should prove that OpenClaw can boot from restored state on a fresh host or container. +The first implementation milestone should prove that OpenClaw can boot from restored state on a fresh directory, host, or container. + +### Failover path + +The plugin is not required to implement automatic failover in the first milestone, but it should be designed as the foundation for failover. + +Failover becomes possible when OpenClaw has: + +1. a recent verified snapshot or restore point +2. a way to hydrate local disk before startup +3. a way to confirm schema and integrity before runtime writes +4. a clear owner for the database after restore +5. optional deltas or upload scheduling to reduce the data-loss window + +A later RFC can define leases, promotion, fencing, standby replicas, and managed orchestration. This RFC provides the snapshot and restore substrate those systems need. + +### Writer ownership + +OpenClaw must not treat a network filesystem as the concurrency model for hot SQLite writes. + +Each snapshot-managed SQLite database should have explicit writer ownership when the deployment allows failover or multiple possible hosts. A managed deployment can move ownership, but only through a controlled sequence: + +1. acquire ownership or a writer lease for the database, if leases are enabled +2. hydrate local disk from a verified restore point when needed +3. open and write SQLite locally +4. periodically create and publish snapshot artifacts +5. release ownership with a final verified snapshot when supported +6. allow another host to restore from the latest verified durable point + +Concurrent readers and replicas can be designed later, but the write path must have one clear owner at a time unless a future RFC defines a stronger multi-writer mechanism. ### Lifecycle metadata -Each durable database needs metadata sufficient to reason about ownership, replay, and integrity. +Each snapshot needs metadata sufficient to reason about restore, replay, and integrity. -At minimum, durability metadata should include: +At minimum, snapshot metadata should include: - database id - database kind or owner - schema version -- current writer owner or lease holder, when leases are enabled - snapshot generation -- checkpoint or WAL cursor +- checkpoint or WAL cursor when available - artifact manifest id - integrity hash or verification record -- last durable upload time +- snapshot creation time - restore source and restore point when hydrated +- current writer owner or lease holder, when leases are enabled -The exact storage location for this metadata is implementation-defined, but opt-in managed deployments must be able to access it before opening a database for managed-runtime writes. +The exact storage location for this metadata is implementation-defined, but the `snapshot` plugin must be able to read enough metadata to verify and restore a snapshot without opening a possibly unsafe runtime database first. -### Durability provider shape +### Provider shape -The implementation can start as a SQLite-specific durability provider rather than a database abstraction layer. +The implementation can start as a SQLite-specific snapshot provider rather than a database abstraction layer. A minimal shape is: ```ts -type SqliteDurabilityProvider = { - checkpoint(dbRef): Promise; - uploadSnapshot(dbRef): Promise; - uploadDelta(dbRef, sinceCursor): Promise; - restore(targetPath, restorePoint): Promise; - verify(targetPath): Promise; +type SqliteSnapshotProvider = { + create(dbRef): Promise; + verify(snapshotRef): Promise; + restore(snapshotRef, targetPath): Promise; + list?(): Promise; + status?(): Promise; +}; +``` + +A later provider can add remote persistence: + +```ts +type RemoteSnapshotProvider = SqliteSnapshotProvider & { + upload(snapshotRef): Promise; + download(snapshotRef, targetPath): Promise; + prune?(policy): Promise; }; ``` -This keeps SQLite runtime access local while making persistence cloud-aware. A local snapshot provider can be the reference implementation. Cloud/object-store providers can come later without changing the default local runtime. +This keeps SQLite runtime access local while making state artifacts portable. A local snapshot provider can be the reference implementation. Cloud/object-store providers can come later without changing the default local runtime. ### First milestone The first implementation milestone should be intentionally small and opt-in: -1. choose one existing OpenClaw-owned SQLite database -2. produce a consistent local snapshot artifact -3. restore it into a fresh directory or host -4. verify integrity -5. boot OpenClaw from restored state -6. document that hot writes over network filesystems remain unsupported +1. implement the `snapshot` plugin with a local snapshot repository +2. choose one existing OpenClaw-owned SQLite database +3. produce a consistent local snapshot artifact +4. verify the snapshot manifest and SQLite integrity +5. restore it into a fresh directory or host +6. boot OpenClaw from restored state +7. expose the workflow through `openclaw snapshot` and optionally `openclaw backup snapshot` / `openclaw backup restore` +8. document that hot writes over network filesystems remain unsupported -Incremental deltas, object storage, leases, and failover should follow after snapshot/restore is proven. +Incremental deltas, object storage, leases, standby replicas, and automatic failover should follow after snapshot and restore are proven. ## Rationale -This approach targets the reliability problem directly. It does not require OpenClaw to choose a second database backend before it has defined durability and restore semantics for the SQLite state it already owns. +This approach targets the reliability problem directly. It does not require OpenClaw to choose a second database backend before it has defined capture and restore semantics for the SQLite state it already owns. + +Calling the extension `snapshot` keeps the first deliverable concrete. It describes the artifact OpenClaw needs before higher-level reliability features can exist. It also avoids overpromising automatic failover before leases, promotion, and orchestration are designed. + +Keeping `backup` as an extended command surface respects the CLI that already exists. Users who think in backup and restore terms can stay under `openclaw backup`; operators who install the plugin can use `openclaw snapshot` when they need the more specific state-artifact workflow. -Treating cloud storage as artifact storage avoids the common failure mode where object storage or network filesystems are used as if they were local disk. SQLite remains local and authoritative while running. Managed durability comes from verified snapshots, deltas, manifests, and restore procedures. +Treating remote storage as artifact storage avoids the common failure mode where object storage or network filesystems are used as if they were local disk. SQLite remains local and authoritative while running. Reliability comes from verified snapshots, manifests, restore procedures, and later deltas. Making the feature opt-in keeps the default OpenClaw runtime simple. Local and development users should not need object storage, a lease service, or a managed scheduler to keep using SQLite. -Keeping core responsible for SQLite-safe primitives is important because safe checkpoints and restores need access to database paths, WAL behavior, schema versions, and integrity checks. Provider-owned artifact storage keeps cloud credentials, retention policy, and managed failover out of the default core runtime. +Keeping core responsible for SQLite-safe primitives is important because safe snapshots and restores need access to database paths, WAL behavior, schema versions, and integrity checks. Provider-owned artifact storage keeps cloud credentials, retention policy, and managed failover out of the default core runtime. Explicit writer ownership keeps horizontal service orchestration honest. A service can move work between hosts, but it must move ownership and restore state deliberately rather than letting several instances write the same SQLite database through shared storage. -The proposal also keeps logical storage boundaries out of scope. OpenClaw already has shared state, per-agent state, and owner-specific stores; this RFC defines how any of those databases can become durable cloud artifacts. +The proposal also keeps logical storage boundaries out of scope. OpenClaw already has shared state, per-agent state, and owner-specific stores; this RFC defines how any of those databases can become restorable snapshot artifacts. ## Unresolved questions +- Should the plugin command be exactly `snapshot`, or should `snapshot` exist only as a backup subcommand? - Which existing SQLite database should be used for the first snapshot/restore proof? - Should the first checkpoint implementation use SQLite online backup, `VACUUM INTO`, WAL checkpointing, page capture, or a higher-level export format? -- Should the reference provider be a local snapshot provider only, or should it include one object/blob storage provider? +- Should the reference provider be a local snapshot repository only, or should it include one object/blob storage provider? - What is the acceptable data-loss window for managed deployments before deltas are implemented? -- Where should writer lease metadata live before a database is opened? +- Where should writer ownership metadata live before a database is opened? - Should restore verification run during startup, doctor, a managed-control-plane action, or all three? - Which artifacts should be included with database restore for support/debug exports versus canonical runtime recovery? - Should external tools such as Litestream or LiteFS be provider integrations, deployment recommendations, or out of scope for OpenClaw-owned code? From 94416227ce48a2f368788d3043acd4e2191aba3f Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Thu, 18 Jun 2026 15:01:52 -0700 Subject: [PATCH 05/24] Clarify reusable snapshot provider contract --- rfcs/0013-cloud-serializable-sqlite-state.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 6b21b39b..92604733 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -17,7 +17,7 @@ Define an opt-in `snapshot` plugin for OpenClaw-owned SQLite state. The plugin p SQLite remains the hot local runtime database. The plugin does not replace SQLite, introduce a second required database backend, or make cloud storage mandatory. It gives managed and production operators a clearer answer to a narrower problem: how OpenClaw state becomes a portable, restorable artifact when a process, container, or host needs to be replaced. -The plugin can expose its own commands and also extend the existing `openclaw backup` command surface. +The plugin can expose its own commands and also extend the existing `openclaw backup` command surface. The snapshot provider contract should apply to any OpenClaw-owned SQLite database, and the plugin can be the first consumer of primitives that core may later use directly if snapshot and restore become baseline behavior. ## Motivation @@ -42,12 +42,14 @@ The proposed answer is a `snapshot` plugin: an opt-in extension that turns local - Make snapshot behavior opt-in through a plugin extension. - Produce consistent SQLite snapshots that handle WAL state correctly. - Make restore and verification first-class behaviors, not incidental backup side effects. +- Define a reusable SQLite snapshot provider contract for OpenClaw-owned SQLite databases. - Allow the plugin to add commands under `openclaw snapshot` and extend `openclaw backup`. - Keep default local OpenClaw behavior unchanged when the plugin is not installed or enabled. - Avoid hot writes over network filesystems as a durability or concurrency strategy. - Define lifecycle metadata needed to validate, order, restore, and audit snapshots. - Leave cloud artifact storage, retention, scheduling, and failover orchestration to optional providers or later RFCs. - Keep the design compatible with existing global state, per-agent state, and dedicated store boundaries. +- Leave room for core to adopt the same primitives later if snapshot and restore become required OpenClaw behavior. ## Non-Goals @@ -69,6 +71,8 @@ Add an opt-in `snapshot` plugin that owns SQLite-safe snapshot and restore workf The plugin should be installable and removable like other OpenClaw plugins. When it is absent, default OpenClaw keeps its current local SQLite behavior and existing backup commands. +The plugin is the first proposed consumer of the snapshot provider contract, not the only possible consumer. Over time, the same contract can be used by core backup/restore commands, doctor checks, managed startup hydration, or other OpenClaw features that need a consistent SQLite restore point. + The plugin can add a direct command surface: ```text @@ -174,6 +178,8 @@ An OpenClaw-owned SQLite database is snapshot-safe when it can be captured, veri The unit of snapshotting is an existing OpenClaw-owned SQLite database, such as shared state, per-agent state, or a dedicated owner store. This RFC does not rename or redesign those logical units; it defines snapshot behavior that can apply to each unit. +The provider contract should therefore take a database reference rather than assume one hard-coded database path. Core can decide which SQLite databases are eligible, and the plugin or provider can apply the same snapshot semantics to each eligible database. + A snapshot must: - handle `.sqlite`, `-wal`, and `-shm` state correctly @@ -267,6 +273,8 @@ The exact storage location for this metadata is implementation-defined, but the The implementation can start as a SQLite-specific snapshot provider rather than a database abstraction layer. +The provider contract should be reusable by any OpenClaw feature that needs to capture or restore an OpenClaw-owned SQLite database. The `snapshot` plugin is the first proposed packaging and CLI surface, but the contract should not depend on plugin-only state. + A minimal shape is: ```ts @@ -291,6 +299,8 @@ type RemoteSnapshotProvider = SqliteSnapshotProvider & { This keeps SQLite runtime access local while making state artifacts portable. A local snapshot provider can be the reference implementation. Cloud/object-store providers can come later without changing the default local runtime. +If the design proves broadly useful, core can adopt the same contract for built-in backup restore, startup hydration, or state migration workflows without requiring the `snapshot` plugin command surface to become mandatory. + ### First milestone The first implementation milestone should be intentionally small and opt-in: @@ -320,6 +330,8 @@ Making the feature opt-in keeps the default OpenClaw runtime simple. Local and d Keeping core responsible for SQLite-safe primitives is important because safe snapshots and restores need access to database paths, WAL behavior, schema versions, and integrity checks. Provider-owned artifact storage keeps cloud credentials, retention policy, and managed failover out of the default core runtime. +The provider contract gives OpenClaw a path from plugin experimentation to core adoption. The first implementation can live as an opt-in plugin while the contract stays general enough for future core backup, restore, startup hydration, or migration workflows. + Explicit writer ownership keeps horizontal service orchestration honest. A service can move work between hosts, but it must move ownership and restore state deliberately rather than letting several instances write the same SQLite database through shared storage. The proposal also keeps logical storage boundaries out of scope. OpenClaw already has shared state, per-agent state, and owner-specific stores; this RFC defines how any of those databases can become restorable snapshot artifacts. From 798f80c292c8d0136a39c10b2bd528af20735fec Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Thu, 18 Jun 2026 15:12:00 -0700 Subject: [PATCH 06/24] Add snapshot implementation roadmap --- rfcs/0013-cloud-serializable-sqlite-state.md | 91 +++++++++++++++----- 1 file changed, 69 insertions(+), 22 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 92604733..0fc1851a 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -17,7 +17,7 @@ Define an opt-in `snapshot` plugin for OpenClaw-owned SQLite state. The plugin p SQLite remains the hot local runtime database. The plugin does not replace SQLite, introduce a second required database backend, or make cloud storage mandatory. It gives managed and production operators a clearer answer to a narrower problem: how OpenClaw state becomes a portable, restorable artifact when a process, container, or host needs to be replaced. -The plugin can expose its own commands and also extend the existing `openclaw backup` command surface. The snapshot provider contract should apply to any OpenClaw-owned SQLite database, and the plugin can be the first consumer of primitives that core may later use directly if snapshot and restore become baseline behavior. +The plugin exposes its own `openclaw snapshot` command surface. It can leave room for later integration with `openclaw backup`, but the first implementation stack should stay plugin-scoped. The snapshot provider contract should apply to any OpenClaw-owned SQLite database, and the plugin can be the first consumer of primitives that core may later use directly if snapshot and restore become baseline behavior. ## Motivation @@ -43,7 +43,8 @@ The proposed answer is a `snapshot` plugin: an opt-in extension that turns local - Produce consistent SQLite snapshots that handle WAL state correctly. - Make restore and verification first-class behaviors, not incidental backup side effects. - Define a reusable SQLite snapshot provider contract for OpenClaw-owned SQLite databases. -- Allow the plugin to add commands under `openclaw snapshot` and extend `openclaw backup`. +- Allow the plugin to add commands under `openclaw snapshot`. +- Leave `openclaw backup` integration as a possible later follow-up, not part of the initial proof stack. - Keep default local OpenClaw behavior unchanged when the plugin is not installed or enabled. - Avoid hot writes over network filesystems as a durability or concurrency strategy. - Define lifecycle metadata needed to validate, order, restore, and audit snapshots. @@ -62,6 +63,7 @@ The proposed answer is a `snapshot` plugin: an opt-in extension that turns local - This RFC does not define FTS/vector search portability. - This RFC does not require real-time multi-writer SQLite over shared storage. - This RFC does not define the final managed failover control plane. +- This RFC does not change the existing `openclaw backup create` or `openclaw backup verify` behavior. ## Proposal @@ -73,7 +75,21 @@ The plugin should be installable and removable like other OpenClaw plugins. When The plugin is the first proposed consumer of the snapshot provider contract, not the only possible consumer. Over time, the same contract can be used by core backup/restore commands, doctor checks, managed startup hydration, or other OpenClaw features that need a consistent SQLite restore point. -The plugin can add a direct command surface: +The plugin should follow the bundled extension pattern used by plugins such as `policy` and `oc-path`: + +```text +extensions/snapshot/ + package.json + openclaw.plugin.json + index.ts + src/ + snapshot-provider.ts + sqlite-snapshot.ts + manifest.ts + local-repository.ts +``` + +The plugin adds a direct command surface: ```text openclaw snapshot create @@ -83,7 +99,7 @@ openclaw snapshot list openclaw snapshot status ``` -It can also extend the existing backup surface so operators can use the name OpenClaw already has for this class of work: +Later, if maintainers want one user-facing home for backup and restore workflows, the same provider contract can be wired under the existing backup command surface: ```text openclaw backup snapshot @@ -91,7 +107,7 @@ openclaw backup restore openclaw backup status ``` -The intent is not to split the product vocabulary. `snapshot` is the plugin and capability name; `backup` remains the user-facing home for backup and restore workflows where that is more natural. +That integration is intentionally not part of the initial implementation roadmap. The first proof should stay scoped to the `snapshot` plugin so it can demonstrate correctness without changing the existing backup command behavior. ### Responsibility split @@ -111,11 +127,11 @@ The `snapshot` plugin should own: - local snapshot artifact creation - snapshot manifest creation and verification - restore workflow orchestration -- optional integration with `openclaw backup` - provider hooks for storage backends -Optional providers can own: +Optional future integrations and providers can own: +- integration with `openclaw backup` - local snapshot repositories - S3-compatible artifact storage - Azure Blob or other cloud artifact storage @@ -143,7 +159,6 @@ flowchart LR subgraph Plugin[snapshot plugin] Cmd[openclaw snapshot] - Backup[openclaw backup extensions] Manifest[snapshot manifest] LocalRepo[(local snapshot repo)] end @@ -159,7 +174,6 @@ flowchart LR Checkpoint --> Manifest Cmd --> Checkpoint Cmd --> Restore - Backup --> Cmd Manifest --> LocalRepo LocalRepo --> Restore LocalRepo --> Object @@ -301,20 +315,54 @@ This keeps SQLite runtime access local while making state artifacts portable. A If the design proves broadly useful, core can adopt the same contract for built-in backup restore, startup hydration, or state migration workflows without requiring the `snapshot` plugin command surface to become mandatory. -### First milestone +### Implementation roadmap + +The initial implementation should be a short PR stack that proves correctness before expanding product surface. + +#### PR 1: provider proof + +Add the bundled `snapshot` plugin scaffold and a local SQLite snapshot provider. + +This PR should include: + +- `extensions/snapshot` plugin manifest, package metadata, and entrypoint +- `SqliteSnapshotProvider` contract +- local snapshot repository +- snapshot manifest and content hash verification +- SQLite-safe snapshot creation for one database reference +- tests against a WAL-mode SQLite database +- an internal restore in tests to prove the artifact is usable + +This PR does not need the full public restore CLI. It should prove that the provider can create and verify a correct SQLite snapshot artifact. + +#### PR 2: public snapshot CLI + +Expose the user-facing plugin commands: + +```text +openclaw snapshot create +openclaw snapshot verify +openclaw snapshot restore +``` + +This PR should add target-directory safety checks, restore manifest validation, SQLite integrity checks after restore, and docs for the `snapshot` plugin command surface. + +#### PR 3: fresh-state boot proof + +Prove that a restored snapshot can hydrate a fresh OpenClaw state directory before runtime opens SQLite. + +This PR should demonstrate that OpenClaw can start from restored state in a fresh directory, host, or container-style environment. That proof is what makes the plugin a credible failover substrate rather than only an archive utility. -The first implementation milestone should be intentionally small and opt-in: +#### Later work -1. implement the `snapshot` plugin with a local snapshot repository -2. choose one existing OpenClaw-owned SQLite database -3. produce a consistent local snapshot artifact -4. verify the snapshot manifest and SQLite integrity -5. restore it into a fresh directory or host -6. boot OpenClaw from restored state -7. expose the workflow through `openclaw snapshot` and optionally `openclaw backup snapshot` / `openclaw backup restore` -8. document that hot writes over network filesystems remain unsupported +After the three initial PRs, follow-up RFCs or implementation PRs can consider: -Incremental deltas, object storage, leases, standby replicas, and automatic failover should follow after snapshot and restore are proven. +- `openclaw backup` integration +- incremental deltas +- object/blob storage providers +- retention and scheduling +- leases, promotion, fencing, and managed failover +- external tool integrations such as Litestream or LiteFS ## Rationale @@ -322,7 +370,7 @@ This approach targets the reliability problem directly. It does not require Open Calling the extension `snapshot` keeps the first deliverable concrete. It describes the artifact OpenClaw needs before higher-level reliability features can exist. It also avoids overpromising automatic failover before leases, promotion, and orchestration are designed. -Keeping `backup` as an extended command surface respects the CLI that already exists. Users who think in backup and restore terms can stay under `openclaw backup`; operators who install the plugin can use `openclaw snapshot` when they need the more specific state-artifact workflow. +Keeping the first implementation stack under `openclaw snapshot` keeps the proof small and plugin-scoped. Existing `openclaw backup create` and `openclaw backup verify` behavior can remain unchanged while the snapshot provider proves the harder SQLite correctness and restore semantics. Treating remote storage as artifact storage avoids the common failure mode where object storage or network filesystems are used as if they were local disk. SQLite remains local and authoritative while running. Reliability comes from verified snapshots, manifests, restore procedures, and later deltas. @@ -338,7 +386,6 @@ The proposal also keeps logical storage boundaries out of scope. OpenClaw alread ## Unresolved questions -- Should the plugin command be exactly `snapshot`, or should `snapshot` exist only as a backup subcommand? - Which existing SQLite database should be used for the first snapshot/restore proof? - Should the first checkpoint implementation use SQLite online backup, `VACUUM INTO`, WAL checkpointing, page capture, or a higher-level export format? - Should the reference provider be a local snapshot repository only, or should it include one object/blob storage provider? From 4d0320b72de45ce79dfa0b03084482de659c0165 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Thu, 18 Jun 2026 19:06:14 -0700 Subject: [PATCH 07/24] clarify snapshot database-first targets --- rfcs/0013-cloud-serializable-sqlite-state.md | 31 +++++++++++++------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 0fc1851a..b0390cd7 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -3,7 +3,7 @@ title: SQLite State Snapshot Plugin authors: - giodl created: 2026-06-18 -last_updated: 2026-06-18 +last_updated: 2026-06-19 status: draft issue: rfc_pr: https://github.com/openclaw/rfcs/pull/20 @@ -17,11 +17,11 @@ Define an opt-in `snapshot` plugin for OpenClaw-owned SQLite state. The plugin p SQLite remains the hot local runtime database. The plugin does not replace SQLite, introduce a second required database backend, or make cloud storage mandatory. It gives managed and production operators a clearer answer to a narrower problem: how OpenClaw state becomes a portable, restorable artifact when a process, container, or host needs to be replaced. -The plugin exposes its own `openclaw snapshot` command surface. It can leave room for later integration with `openclaw backup`, but the first implementation stack should stay plugin-scoped. The snapshot provider contract should apply to any OpenClaw-owned SQLite database, and the plugin can be the first consumer of primitives that core may later use directly if snapshot and restore become baseline behavior. +The plugin exposes its own `openclaw snapshot` command surface. It can leave room for later integration with `openclaw backup`, but the first implementation stack should stay plugin-scoped. The snapshot provider contract should apply to the database-first layout OpenClaw has already landed: a global control-plane database plus per-agent data-plane databases, and any future dedicated owner store that follows the same ownership model. The plugin can be the first consumer of primitives that core may later use directly if snapshot and restore become baseline behavior. ## Motivation -OpenClaw is moving runtime state into SQLite-backed stores. That is a good local runtime shape, but operational reliability depends on more than having files on disk. +OpenClaw is moving runtime state into SQLite-backed stores. The database-first SQLite alignment landed in openclaw/openclaw#94646 makes that shape more explicit: `state/openclaw.sqlite` is the global control-plane database, while `agents//agent/openclaw-agent.sqlite` is the per-agent data-plane database for agent-owned state such as memory indexes and auth/profile state. That is a good local runtime shape, but operational reliability depends on more than having files on disk. The pain points are concrete: @@ -49,7 +49,7 @@ The proposed answer is a `snapshot` plugin: an opt-in extension that turns local - Avoid hot writes over network filesystems as a durability or concurrency strategy. - Define lifecycle metadata needed to validate, order, restore, and audit snapshots. - Leave cloud artifact storage, retention, scheduling, and failover orchestration to optional providers or later RFCs. -- Keep the design compatible with existing global state, per-agent state, and dedicated store boundaries. +- Build on the existing database-first units: global control-plane SQLite, per-agent data-plane SQLite, and any dedicated owner store. - Leave room for core to adopt the same primitives later if snapshot and restore become required OpenClaw behavior. ## Non-Goals @@ -84,7 +84,6 @@ extensions/snapshot/ index.ts src/ snapshot-provider.ts - sqlite-snapshot.ts manifest.ts local-repository.ts ``` @@ -190,7 +189,13 @@ The diagram is a responsibility split, not a default runtime requirement. Defaul An OpenClaw-owned SQLite database is snapshot-safe when it can be captured, verified, restored, and resumed on another host or directory without relying on a live shared filesystem. -The unit of snapshotting is an existing OpenClaw-owned SQLite database, such as shared state, per-agent state, or a dedicated owner store. This RFC does not rename or redesign those logical units; it defines snapshot behavior that can apply to each unit. +The unit of snapshotting is an existing OpenClaw-owned SQLite database. The primary units are: + +- the global control-plane database at `state/openclaw.sqlite` +- one per-agent data-plane database at `agents//agent/openclaw-agent.sqlite` +- any future dedicated owner store that has explicit ownership, schema, and lifecycle metadata + +This RFC does not rename or redesign those logical units; openclaw/openclaw#94646 makes them concrete enough for snapshot to target. The RFC defines snapshot behavior that can apply to each eligible database. The provider contract should therefore take a database reference rather than assume one hard-coded database path. Core can decide which SQLite databases are eligible, and the plugin or provider can apply the same snapshot semantics to each eligible database. @@ -272,6 +277,8 @@ At minimum, snapshot metadata should include: - database id - database kind or owner +- database role, such as global control-plane or per-agent data-plane +- owning agent id when the snapshot is for a per-agent database - schema version - snapshot generation - checkpoint or WAL cursor when available @@ -329,11 +336,11 @@ This PR should include: - `SqliteSnapshotProvider` contract - local snapshot repository - snapshot manifest and content hash verification -- SQLite-safe snapshot creation for one database reference +- SQLite-safe snapshot creation for one database reference using shared core/package primitives where OpenClaw already owns the SQLite invariants - tests against a WAL-mode SQLite database - an internal restore in tests to prove the artifact is usable -This PR does not need the full public restore CLI. It should prove that the provider can create and verify a correct SQLite snapshot artifact. +This PR does not need the full public restore CLI. It should prove that the provider can create and verify a correct SQLite snapshot artifact without publishing a premature provider API surface. #### PR 2: public snapshot CLI @@ -347,6 +354,8 @@ openclaw snapshot restore This PR should add target-directory safety checks, restore manifest validation, SQLite integrity checks after restore, and docs for the `snapshot` plugin command surface. +The CLI should accept an explicit database path for the proof path, but the intended product model is not "any random SQLite file forever." The command should be able to grow toward named OpenClaw database targets such as global state or a specific agent database once core exposes the eligible database registry cleanly. + #### PR 3: fresh-state boot proof Prove that a restored snapshot can hydrate a fresh OpenClaw state directory before runtime opens SQLite. @@ -368,6 +377,8 @@ After the three initial PRs, follow-up RFCs or implementation PRs can consider: This approach targets the reliability problem directly. It does not require OpenClaw to choose a second database backend before it has defined capture and restore semantics for the SQLite state it already owns. +The database-first work in openclaw/openclaw#94646 improves this RFC because it gives snapshot a concrete target model. Snapshot does not have to invent logical database units. It can operate over the already-established global control-plane database and per-agent data-plane databases, then extend to dedicated owner stores only when those stores have comparable ownership and lifecycle metadata. + Calling the extension `snapshot` keeps the first deliverable concrete. It describes the artifact OpenClaw needs before higher-level reliability features can exist. It also avoids overpromising automatic failover before leases, promotion, and orchestration are designed. Keeping the first implementation stack under `openclaw snapshot` keeps the proof small and plugin-scoped. Existing `openclaw backup create` and `openclaw backup verify` behavior can remain unchanged while the snapshot provider proves the harder SQLite correctness and restore semantics. @@ -382,11 +393,11 @@ The provider contract gives OpenClaw a path from plugin experimentation to core Explicit writer ownership keeps horizontal service orchestration honest. A service can move work between hosts, but it must move ownership and restore state deliberately rather than letting several instances write the same SQLite database through shared storage. -The proposal also keeps logical storage boundaries out of scope. OpenClaw already has shared state, per-agent state, and owner-specific stores; this RFC defines how any of those databases can become restorable snapshot artifacts. +The proposal also keeps storage ownership decisions out of scope. OpenClaw already has global control-plane state, per-agent data-plane state, and owner-specific stores; this RFC defines how those databases become restorable snapshot artifacts. ## Unresolved questions -- Which existing SQLite database should be used for the first snapshot/restore proof? +- Which database-first unit should be used for the first named snapshot/restore proof: global control-plane state, one per-agent data-plane database, or both? - Should the first checkpoint implementation use SQLite online backup, `VACUUM INTO`, WAL checkpointing, page capture, or a higher-level export format? - Should the reference provider be a local snapshot repository only, or should it include one object/blob storage provider? - What is the acceptable data-loss window for managed deployments before deltas are implemented? From b5749b6588471dbfa4efec3b413db12d2dc83f26 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Thu, 18 Jun 2026 19:29:02 -0700 Subject: [PATCH 08/24] clarify hosted snapshot artifact boundary --- rfcs/0013-cloud-serializable-sqlite-state.md | 36 ++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index b0390cd7..b965531f 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -17,6 +17,8 @@ Define an opt-in `snapshot` plugin for OpenClaw-owned SQLite state. The plugin p SQLite remains the hot local runtime database. The plugin does not replace SQLite, introduce a second required database backend, or make cloud storage mandatory. It gives managed and production operators a clearer answer to a narrower problem: how OpenClaw state becomes a portable, restorable artifact when a process, container, or host needs to be replaced. +The central contract is a state-artifact boundary: OpenClaw turns live local SQLite state into verified artifacts, while a host such as Lobster/Aether decides where those artifacts live and when they are uploaded, retained, downloaded, or replayed. + The plugin exposes its own `openclaw snapshot` command surface. It can leave room for later integration with `openclaw backup`, but the first implementation stack should stay plugin-scoped. The snapshot provider contract should apply to the database-first layout OpenClaw has already landed: a global control-plane database plus per-agent data-plane databases, and any future dedicated owner store that follows the same ownership model. The plugin can be the first consumer of primitives that core may later use directly if snapshot and restore become baseline behavior. ## Motivation @@ -31,11 +33,14 @@ The pain points are concrete: - backup archives are only useful if restore is verified and repeatable - container or host replacement needs state hydration before OpenClaw opens the database - failover cannot be credible until OpenClaw has a known-good restore point +- hosted OpenClaw platforms need a stable way to persist OpenClaw-owned state without reverse-engineering which files are authoritative, which SQLite sidecars are live-only, and which database units are safe to rehydrate The operational question is therefore not primarily "which database should OpenClaw use?" Greater minds can make the long-term database choice separately. This RFC focuses on the SQLite state OpenClaw already owns and asks for a practical extension point around it. The proposed answer is a `snapshot` plugin: an opt-in extension that turns local SQLite state into verified snapshot artifacts. Those artifacts can later support cloud uploads, retention policies, warm standby, and failover, but the first value is simpler: make state capture and restore correct. +This also gives hosted OpenClaw deployments a cleaner integration point. The host should not need to copy `*.sqlite`, `*.sqlite-wal`, and `*.sqlite-shm` files directly or infer durability rules from ignore files. OpenClaw should provide the SQLite-aware translation into clean artifacts; the host should provide destination storage, schedule, retention, encryption policy, and container lifecycle integration. + ## Goals - Keep SQLite as the hot local runtime database for this proposal. @@ -43,12 +48,14 @@ The proposed answer is a `snapshot` plugin: an opt-in extension that turns local - Produce consistent SQLite snapshots that handle WAL state correctly. - Make restore and verification first-class behaviors, not incidental backup side effects. - Define a reusable SQLite snapshot provider contract for OpenClaw-owned SQLite databases. +- Define the state-artifact boundary that lets hosted platforms persist OpenClaw state without understanding OpenClaw's internal SQLite file layout. - Allow the plugin to add commands under `openclaw snapshot`. - Leave `openclaw backup` integration as a possible later follow-up, not part of the initial proof stack. - Keep default local OpenClaw behavior unchanged when the plugin is not installed or enabled. - Avoid hot writes over network filesystems as a durability or concurrency strategy. - Define lifecycle metadata needed to validate, order, restore, and audit snapshots. - Leave cloud artifact storage, retention, scheduling, and failover orchestration to optional providers or later RFCs. +- Let hosts such as Lobster/Aether own durable destination policy while OpenClaw owns the correctness of the local SQLite artifact operation. - Build on the existing database-first units: global control-plane SQLite, per-agent data-plane SQLite, and any dedicated owner store. - Leave room for core to adopt the same primitives later if snapshot and restore become required OpenClaw behavior. @@ -56,6 +63,7 @@ The proposed answer is a `snapshot` plugin: an opt-in extension that turns local - This RFC does not choose PostgreSQL, libSQL, remote SQLite, object storage, or any other backend product. - This RFC does not define a general database abstraction layer. +- This RFC does not require OpenClaw to own the hosting platform's upload, retention, tenant routing, encryption, or object-storage policy. - This RFC does not make snapshots, cloud storage, or managed failover mandatory for local, self-hosted, or development OpenClaw installs. - This RFC does not require object-store credentials, a lease service, or a managed-service control plane in the default runtime. - This RFC does not replace the session/transcript migration plan tracked by openclaw/openclaw#88838. @@ -114,6 +122,7 @@ Core should expose or own the SQLite-safe primitives that require knowledge of O Core should own: +- eligible database discovery or registry for OpenClaw-owned SQLite databases - consistent SQLite checkpoint creation - restore or hydrate before opening runtime state - restored database verification @@ -134,11 +143,26 @@ Optional future integrations and providers can own: - local snapshot repositories - S3-compatible artifact storage - Azure Blob or other cloud artifact storage +- ODSP, blob, git, durable-volume, or other host-specific persistence adapters - retention policy and upload scheduling - writer lease coordination - warm standby or managed failover orchestration - integration with external tools such as Litestream or LiteFS, if later accepted +In other words, OpenClaw should own this translation: + +```text +live OpenClaw SQLite database -> verified snapshot artifact + manifest +``` + +The hosting platform should own this policy: + +```text +verified snapshot artifact + manifest -> durable destination and restore timing +``` + +That split keeps SQLite correctness in the codebase that owns the schema and file layout, while keeping cloud credentials, tenant routing, retention, and platform lifecycle outside the default OpenClaw runtime. + ### Architecture ```mermaid @@ -164,6 +188,7 @@ flowchart LR subgraph Providers[Optional providers] Object[(object/blob storage)] + Host[(host persistence layer)] Retention[retention schedule] Failover[standby / failover] end @@ -176,7 +201,9 @@ flowchart LR Manifest --> LocalRepo LocalRepo --> Restore LocalRepo --> Object + LocalRepo --> Host Object --> Restore + Host --> Restore Restore --> Verify Verify --> DB Retention -. optional .-> LocalRepo @@ -185,6 +212,8 @@ flowchart LR The diagram is a responsibility split, not a default runtime requirement. Default OpenClaw can run with only the runtime box. Operators opt into the plugin when they need verified snapshot and restore workflows. +For hosted OpenClaw, the same split becomes the host integration contract. The host can ask OpenClaw to materialize a clean artifact before upload and can hydrate local disk from a verified artifact before OpenClaw opens SQLite. The host does not need to treat live SQLite sidecars as durable sync inputs. + ### Snapshot semantics An OpenClaw-owned SQLite database is snapshot-safe when it can be captured, verified, restored, and resumed on another host or directory without relying on a live shared filesystem. @@ -225,6 +254,8 @@ The artifact model should support: The delta mechanism can be WAL-frame based, page based, logical-change based, external-tool based, or backend-native. This RFC requires the contract, not one specific encoding. +Artifacts should be suitable for host persistence. A hosting platform should be able to upload, retain, copy, and later download the artifact set without preserving process-local SQLite sidecars or relying on a mounted shared filesystem. + ### Restore verification Restore is a required behavior for the `snapshot` plugin, not an incidental backup side effect. @@ -362,6 +393,8 @@ Prove that a restored snapshot can hydrate a fresh OpenClaw state directory befo This PR should demonstrate that OpenClaw can start from restored state in a fresh directory, host, or container-style environment. That proof is what makes the plugin a credible failover substrate rather than only an archive utility. +This proof should also document the host contract: which OpenClaw command or API materializes the artifact, which manifest fields the host can store without interpreting SQLite internals, and what must be restored before OpenClaw opens the database. + #### Later work After the three initial PRs, follow-up RFCs or implementation PRs can consider: @@ -389,6 +422,8 @@ Making the feature opt-in keeps the default OpenClaw runtime simple. Local and d Keeping core responsible for SQLite-safe primitives is important because safe snapshots and restores need access to database paths, WAL behavior, schema versions, and integrity checks. Provider-owned artifact storage keeps cloud credentials, retention policy, and managed failover out of the default core runtime. +Hosted deployments make that core responsibility more important, not less. A host can persist a directory, but OpenClaw should define which database artifacts are safe to persist. Without that boundary, every host integration has to rediscover SQLite sidecar rules and OpenClaw database ownership independently. + The provider contract gives OpenClaw a path from plugin experimentation to core adoption. The first implementation can live as an opt-in plugin while the contract stays general enough for future core backup, restore, startup hydration, or migration workflows. Explicit writer ownership keeps horizontal service orchestration honest. A service can move work between hosts, but it must move ownership and restore state deliberately rather than letting several instances write the same SQLite database through shared storage. @@ -403,5 +438,6 @@ The proposal also keeps storage ownership decisions out of scope. OpenClaw alrea - What is the acceptable data-loss window for managed deployments before deltas are implemented? - Where should writer ownership metadata live before a database is opened? - Should restore verification run during startup, doctor, a managed-control-plane action, or all three? +- What is the minimum host-facing API or command shape needed for Lobster/Aether-style platforms to request artifact materialization and pre-start hydration without importing plugin-specific policy? - Which artifacts should be included with database restore for support/debug exports versus canonical runtime recovery? - Should external tools such as Litestream or LiteFS be provider integrations, deployment recommendations, or out of scope for OpenClaw-owned code? From b36b20a42b861f64ef802f8f3ad97a5c91f3baff Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Thu, 18 Jun 2026 19:32:14 -0700 Subject: [PATCH 09/24] clarify syncable sqlite artifact creation --- rfcs/0013-cloud-serializable-sqlite-state.md | 41 ++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index b965531f..fd28c4fc 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -41,6 +41,17 @@ The proposed answer is a `snapshot` plugin: an opt-in extension that turns local This also gives hosted OpenClaw deployments a cleaner integration point. The host should not need to copy `*.sqlite`, `*.sqlite-wal`, and `*.sqlite-shm` files directly or infer durability rules from ignore files. OpenClaw should provide the SQLite-aware translation into clean artifacts; the host should provide destination storage, schedule, retention, encryption policy, and container lifecycle integration. +The unsafe sync inputs are specific: + +- `state/openclaw.sqlite` can be stale by itself when committed writes still live in `state/openclaw.sqlite-wal`. +- `agents//agent/openclaw-agent.sqlite` can be stale by itself when committed writes still live in `agents//agent/openclaw-agent.sqlite-wal`. +- `*.sqlite-wal`, `*.sqlite-shm`, and `*.sqlite-journal` are process-local SQLite sidecars, not durable host-sync artifacts. +- Copying a directory while OpenClaw is writing can split one logical database state across files captured at different moments. + +The syncable file is created deliberately. OpenClaw should open the source database through SQLite, wait for any required busy timeout or writer barrier, materialize a clean database copy using SQLite online backup, `VACUUM INTO`, or an equivalent SQLite-aware checkpoint/copy mechanism, run integrity verification, and write a manifest that records the database identity, schema version, source path, hash, and restore metadata. The host syncs that artifact set, not the live SQLite files. + +For hosts that sync on filesystem changes, the sync trigger should be the completed artifact write, not arbitrary writes inside the live OpenClaw state directory. The artifact directory should contain completed database artifacts and manifests only. Live `*.sqlite-wal`, `*.sqlite-shm`, and `*.sqlite-journal` files should not be created there, because a sync agent can observe and upload them as soon as they appear. + ## Goals - Keep SQLite as the hot local runtime database for this proposal. @@ -124,6 +135,7 @@ Core should own: - eligible database discovery or registry for OpenClaw-owned SQLite databases - consistent SQLite checkpoint creation +- materialization of a host-syncable database artifact from a live local SQLite database - restore or hydrate before opening runtime state - restored database verification - lifecycle metadata shape @@ -163,6 +175,21 @@ verified snapshot artifact + manifest -> durable destination and restore timing That split keeps SQLite correctness in the codebase that owns the schema and file layout, while keeping cloud credentials, tenant routing, retention, and platform lifecycle outside the default OpenClaw runtime. +The host-facing flow should be explicit: + +```text +before upload/sync: + 1. host or operator asks OpenClaw to snapshot an eligible database + 2. OpenClaw creates a clean database artifact in a staging location + 3. OpenClaw verifies the artifact and writes the final artifact + manifest into the sync-owned artifact directory + 4. host sync is triggered by the completed artifact/manifest write + +before startup after replacement: + 1. host downloads/selects a verified artifact set + 2. OpenClaw verifies and hydrates local database files + 3. OpenClaw opens SQLite for runtime writes only after hydration succeeds +``` + ### Architecture ```mermaid @@ -239,6 +266,8 @@ A snapshot must: The implementation may use SQLite online backup APIs, `VACUUM INTO`, WAL checkpoints, page-level capture, or another implementation-specific mechanism. The observable contract is a consistent restore point. +For the initial proof, `VACUUM INTO` is acceptable because it asks SQLite to produce a compact, consistent destination database. That is different from asking the host to copy a hot `.sqlite` file. It is also different from routinely vacuuming OpenClaw's runtime databases; the operation happens only when creating a snapshot artifact. + ### Snapshot artifacts Snapshot storage should store durable artifacts, not a live database file used directly by the runtime. @@ -256,6 +285,18 @@ The delta mechanism can be WAL-frame based, page based, logical-change based, ex Artifacts should be suitable for host persistence. A hosting platform should be able to upload, retain, copy, and later download the artifact set without preserving process-local SQLite sidecars or relying on a mounted shared filesystem. +The sync-owned artifact directory should not be the live SQLite runtime directory. If the host syncs on file save, OpenClaw should write completed artifacts into a separate artifact location after verification. That keeps the host from observing transient SQLite sidecars or partially materialized runtime state. + +Artifacts should be created at explicit lifecycle moments: + +- on demand, when an operator or plugin command requests a snapshot +- before a managed host uploads/syncs OpenClaw state +- before container shutdown or ownership release when the platform can coordinate that moment +- before migration or other state-changing maintenance when a rollback point is required +- periodically, when a provider adds scheduling or retention policy + +Restore artifacts should be consumed before OpenClaw opens the target database for runtime writes. A host that downloads artifacts after OpenClaw has already opened SQLite risks racing the runtime and should be treated as outside this contract. + ### Restore verification Restore is a required behavior for the `snapshot` plugin, not an incidental backup side effect. From 72f145a4654467257d4b0b9b95bd2e60135ae5b0 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Thu, 18 Jun 2026 19:35:41 -0700 Subject: [PATCH 10/24] clarify core sqlite sync primitive ownership --- rfcs/0013-cloud-serializable-sqlite-state.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index fd28c4fc..1a7d0e1c 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -19,7 +19,7 @@ SQLite remains the hot local runtime database. The plugin does not replace SQLit The central contract is a state-artifact boundary: OpenClaw turns live local SQLite state into verified artifacts, while a host such as Lobster/Aether decides where those artifacts live and when they are uploaded, retained, downloaded, or replayed. -The plugin exposes its own `openclaw snapshot` command surface. It can leave room for later integration with `openclaw backup`, but the first implementation stack should stay plugin-scoped. The snapshot provider contract should apply to the database-first layout OpenClaw has already landed: a global control-plane database plus per-agent data-plane databases, and any future dedicated owner store that follows the same ownership model. The plugin can be the first consumer of primitives that core may later use directly if snapshot and restore become baseline behavior. +The plugin exposes its own `openclaw snapshot` command surface. It can leave room for later integration with `openclaw backup`, but the first implementation stack should stay plugin-scoped. The core OpenClaw responsibility is narrower and more fundamental: provide the SQLite-aware primitive that materializes a host-syncable database artifact and document that live SQLite sidecars must be ignored by host sync. The extension proves and packages the opt-in workflow around that primitive. ## Motivation @@ -60,6 +60,8 @@ For hosts that sync on filesystem changes, the sync trigger should be the comple - Make restore and verification first-class behaviors, not incidental backup side effects. - Define a reusable SQLite snapshot provider contract for OpenClaw-owned SQLite databases. - Define the state-artifact boundary that lets hosted platforms persist OpenClaw state without understanding OpenClaw's internal SQLite file layout. +- Make host-syncable SQLite artifact creation a core OpenClaw capability, even if the first user-facing commands live in the opt-in `snapshot` plugin. +- Document host sync guidance: ignore live SQLite sidecars and sync completed snapshot artifacts/manifests instead. - Allow the plugin to add commands under `openclaw snapshot`. - Leave `openclaw backup` integration as a possible later follow-up, not part of the initial proof stack. - Keep default local OpenClaw behavior unchanged when the plugin is not installed or enabled. @@ -76,6 +78,7 @@ For hosts that sync on filesystem changes, the sync trigger should be the comple - This RFC does not define a general database abstraction layer. - This RFC does not require OpenClaw to own the hosting platform's upload, retention, tenant routing, encryption, or object-storage policy. - This RFC does not make snapshots, cloud storage, or managed failover mandatory for local, self-hosted, or development OpenClaw installs. +- This RFC does not require the `snapshot` plugin to be the only long-term consumer of the core SQLite artifact primitive. - This RFC does not require object-store credentials, a lease service, or a managed-service control plane in the default runtime. - This RFC does not replace the session/transcript migration plan tracked by openclaw/openclaw#88838. - This RFC does not define tenant isolation, row-level authorization, or a multi-tenant schema model. @@ -129,19 +132,20 @@ That integration is intentionally not part of the initial implementation roadmap ### Responsibility split -Core should expose or own the SQLite-safe primitives that require knowledge of OpenClaw state paths, WAL behavior, schema versions, and integrity checks. +Core should own the SQLite-safe primitives and sync guidance that require knowledge of OpenClaw state paths, WAL behavior, schema versions, and integrity checks. This is core functionality because the host cannot safely infer those rules from the filesystem alone. Core should own: - eligible database discovery or registry for OpenClaw-owned SQLite databases - consistent SQLite checkpoint creation - materialization of a host-syncable database artifact from a live local SQLite database +- guidance or generated ignore rules for host sync: ignore `*.sqlite-wal`, `*.sqlite-shm`, and `*.sqlite-journal`; do not use the live SQLite runtime directory as the sync-owned artifact directory - restore or hydrate before opening runtime state - restored database verification - lifecycle metadata shape - safety rules such as no hot writes over network filesystems -The `snapshot` plugin should own: +The `snapshot` plugin should own the opt-in workflow around the core primitive: - snapshot command UX - local snapshot artifact creation From 2cfb414ffd3366cdd1f8944a2add0ede02e5bd0d Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Thu, 18 Jun 2026 19:40:27 -0700 Subject: [PATCH 11/24] tighten snapshot sync artifact framing --- rfcs/0013-cloud-serializable-sqlite-state.md | 48 ++++++++------------ 1 file changed, 19 insertions(+), 29 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 1a7d0e1c..2de111a2 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -13,33 +13,19 @@ rfc_pr: https://github.com/openclaw/rfcs/pull/20 ## Summary -Define an opt-in `snapshot` plugin for OpenClaw-owned SQLite state. The plugin produces SQLite-safe state snapshots that can be verified, restored, and used as the foundation for future failover. +Define an opt-in `snapshot` plugin that gives file-syncing hosts a safe file to sync for OpenClaw-owned SQLite state. -SQLite remains the hot local runtime database. The plugin does not replace SQLite, introduce a second required database backend, or make cloud storage mandatory. It gives managed and production operators a clearer answer to a narrower problem: how OpenClaw state becomes a portable, restorable artifact when a process, container, or host needs to be replaced. +Hosted OpenClaw environments such as Scout/Lobster persist OpenClaw state by syncing files when they are saved. Live SQLite files are the wrong sync boundary: `state/openclaw.sqlite` or `agents//agent/openclaw-agent.sqlite` can be incomplete without their WAL, and `*.sqlite-wal`, `*.sqlite-shm`, and `*.sqlite-journal` are process-local sidecars rather than durable artifacts. -The central contract is a state-artifact boundary: OpenClaw turns live local SQLite state into verified artifacts, while a host such as Lobster/Aether decides where those artifacts live and when they are uploaded, retained, downloaded, or replayed. +The `snapshot` extension provides the missing translation step. It asks SQLite to materialize a clean database artifact, verifies it, writes a manifest, and publishes the completed artifact set into a sync-owned location. That completed artifact write, not arbitrary live database churn, is what the host should sync. -The plugin exposes its own `openclaw snapshot` command surface. It can leave room for later integration with `openclaw backup`, but the first implementation stack should stay plugin-scoped. The core OpenClaw responsibility is narrower and more fundamental: provide the SQLite-aware primitive that materializes a host-syncable database artifact and document that live SQLite sidecars must be ignored by host sync. The extension proves and packages the opt-in workflow around that primitive. +SQLite remains the hot local runtime database. This RFC does not choose a replacement database, make cloud storage mandatory, or require managed failover. It defines a state-artifact boundary: OpenClaw owns the SQLite-aware artifact operation, while the host owns upload, retention, routing, encryption, and restore timing. Whether that boundary eventually belongs in core or remains extension-owned is an explicit design question; this RFC proves it first as a `snapshot` extension. ## Motivation -OpenClaw is moving runtime state into SQLite-backed stores. The database-first SQLite alignment landed in openclaw/openclaw#94646 makes that shape more explicit: `state/openclaw.sqlite` is the global control-plane database, while `agents//agent/openclaw-agent.sqlite` is the per-agent data-plane database for agent-owned state such as memory indexes and auth/profile state. That is a good local runtime shape, but operational reliability depends on more than having files on disk. +OpenClaw is moving runtime state into SQLite-backed stores. The database-first SQLite alignment landed in openclaw/openclaw#94646 makes that shape more explicit: `state/openclaw.sqlite` is the global control-plane database, while `agents//agent/openclaw-agent.sqlite` is the per-agent data-plane database. That is a good local runtime shape, but hosted reliability depends on more than having files on disk. -The pain points are concrete: - -- copying a live SQLite database file can miss WAL state or capture a half-consistent database -- whole-file copying gets expensive as state grows -- network filesystems are not a safe concurrency strategy for hot SQLite writes -- backup archives are only useful if restore is verified and repeatable -- container or host replacement needs state hydration before OpenClaw opens the database -- failover cannot be credible until OpenClaw has a known-good restore point -- hosted OpenClaw platforms need a stable way to persist OpenClaw-owned state without reverse-engineering which files are authoritative, which SQLite sidecars are live-only, and which database units are safe to rehydrate - -The operational question is therefore not primarily "which database should OpenClaw use?" Greater minds can make the long-term database choice separately. This RFC focuses on the SQLite state OpenClaw already owns and asks for a practical extension point around it. - -The proposed answer is a `snapshot` plugin: an opt-in extension that turns local SQLite state into verified snapshot artifacts. Those artifacts can later support cloud uploads, retention policies, warm standby, and failover, but the first value is simpler: make state capture and restore correct. - -This also gives hosted OpenClaw deployments a cleaner integration point. The host should not need to copy `*.sqlite`, `*.sqlite-wal`, and `*.sqlite-shm` files directly or infer durability rules from ignore files. OpenClaw should provide the SQLite-aware translation into clean artifacts; the host should provide destination storage, schedule, retention, encryption policy, and container lifecycle integration. +The host-sync problem is concrete. A host can watch files and sync them as they are saved, but it should not sync OpenClaw's live SQLite working set as the durability boundary. The unsafe sync inputs are specific: @@ -52,23 +38,27 @@ The syncable file is created deliberately. OpenClaw should open the source datab For hosts that sync on filesystem changes, the sync trigger should be the completed artifact write, not arbitrary writes inside the live OpenClaw state directory. The artifact directory should contain completed database artifacts and manifests only. Live `*.sqlite-wal`, `*.sqlite-shm`, and `*.sqlite-journal` files should not be created there, because a sync agent can observe and upload them as soon as they appear. +Deltas do not remove this requirement. Ryan's underlying concern is that whole-file copies do not scale, but unmanaged file deltas over live SQLite make the correctness problem worse: a file-sync tool can observe DB pages, WAL frames, and sidecars at different moments without knowing SQLite ordering or checkpoint state. Deltas should be a later optimization from a verified snapshot cursor, not a replacement for the initial clean artifact boundary. + ## Goals - Keep SQLite as the hot local runtime database for this proposal. - Make snapshot behavior opt-in through a plugin extension. -- Produce consistent SQLite snapshots that handle WAL state correctly. +- Produce a host-syncable SQLite artifact and manifest from live OpenClaw SQLite state. +- Handle WAL state correctly without syncing live SQLite sidecars as durable artifacts. - Make restore and verification first-class behaviors, not incidental backup side effects. - Define a reusable SQLite snapshot provider contract for OpenClaw-owned SQLite databases. - Define the state-artifact boundary that lets hosted platforms persist OpenClaw state without understanding OpenClaw's internal SQLite file layout. -- Make host-syncable SQLite artifact creation a core OpenClaw capability, even if the first user-facing commands live in the opt-in `snapshot` plugin. +- Leave core-vs-extension ownership open while proving the mechanism first in the `snapshot` extension. - Document host sync guidance: ignore live SQLite sidecars and sync completed snapshot artifacts/manifests instead. - Allow the plugin to add commands under `openclaw snapshot`. - Leave `openclaw backup` integration as a possible later follow-up, not part of the initial proof stack. - Keep default local OpenClaw behavior unchanged when the plugin is not installed or enabled. - Avoid hot writes over network filesystems as a durability or concurrency strategy. +- Treat deltas as a future SQLite-aware optimization after full snapshot artifacts are correct. - Define lifecycle metadata needed to validate, order, restore, and audit snapshots. - Leave cloud artifact storage, retention, scheduling, and failover orchestration to optional providers or later RFCs. -- Let hosts such as Lobster/Aether own durable destination policy while OpenClaw owns the correctness of the local SQLite artifact operation. +- Let hosts such as Scout/Lobster own durable destination policy while OpenClaw owns the correctness of the local SQLite artifact operation. - Build on the existing database-first units: global control-plane SQLite, per-agent data-plane SQLite, and any dedicated owner store. - Leave room for core to adopt the same primitives later if snapshot and restore become required OpenClaw behavior. @@ -132,9 +122,9 @@ That integration is intentionally not part of the initial implementation roadmap ### Responsibility split -Core should own the SQLite-safe primitives and sync guidance that require knowledge of OpenClaw state paths, WAL behavior, schema versions, and integrity checks. This is core functionality because the host cannot safely infer those rules from the filesystem alone. +OpenClaw must provide the SQLite-safe artifact contract because the host cannot safely infer it from the filesystem alone. The open design decision is packaging: the primitive may become core functionality, or the `snapshot` extension may remain the owner of the command and provider workflow while relying on a smaller core SQLite helper. -Core should own: +OpenClaw should provide: - eligible database discovery or registry for OpenClaw-owned SQLite databases - consistent SQLite checkpoint creation @@ -148,7 +138,7 @@ Core should own: The `snapshot` plugin should own the opt-in workflow around the core primitive: - snapshot command UX -- local snapshot artifact creation +- local snapshot artifact creation and publication into a sync-owned artifact directory - snapshot manifest creation and verification - restore workflow orchestration - provider hooks for storage backends @@ -243,7 +233,7 @@ flowchart LR The diagram is a responsibility split, not a default runtime requirement. Default OpenClaw can run with only the runtime box. Operators opt into the plugin when they need verified snapshot and restore workflows. -For hosted OpenClaw, the same split becomes the host integration contract. The host can ask OpenClaw to materialize a clean artifact before upload and can hydrate local disk from a verified artifact before OpenClaw opens SQLite. The host does not need to treat live SQLite sidecars as durable sync inputs. +For hosted OpenClaw, the same split becomes the host integration contract. The host can ask OpenClaw or the `snapshot` extension to materialize a clean artifact before upload and can hydrate local disk from a verified artifact before OpenClaw opens SQLite. The host does not need to treat live SQLite sidecars as durable sync inputs. ### Snapshot semantics @@ -465,9 +455,9 @@ Treating remote storage as artifact storage avoids the common failure mode where Making the feature opt-in keeps the default OpenClaw runtime simple. Local and development users should not need object storage, a lease service, or a managed scheduler to keep using SQLite. -Keeping core responsible for SQLite-safe primitives is important because safe snapshots and restores need access to database paths, WAL behavior, schema versions, and integrity checks. Provider-owned artifact storage keeps cloud credentials, retention policy, and managed failover out of the default core runtime. +Keeping the SQLite-safe artifact contract inside OpenClaw is important because safe snapshots and restores need access to database paths, WAL behavior, schema versions, and integrity checks. Provider-owned artifact storage keeps cloud credentials, retention policy, and managed failover out of the default runtime. -Hosted deployments make that core responsibility more important, not less. A host can persist a directory, but OpenClaw should define which database artifacts are safe to persist. Without that boundary, every host integration has to rediscover SQLite sidecar rules and OpenClaw database ownership independently. +Hosted deployments make that responsibility more important, not less. A host can persist a directory, but OpenClaw should define which database artifacts are safe to persist. Without that boundary, every host integration has to rediscover SQLite sidecar rules and OpenClaw database ownership independently. The provider contract gives OpenClaw a path from plugin experimentation to core adoption. The first implementation can live as an opt-in plugin while the contract stays general enough for future core backup, restore, startup hydration, or migration workflows. From d8d6b1aea62f7d006f44cbe4c267a71824c9e8a6 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Thu, 18 Jun 2026 20:08:41 -0700 Subject: [PATCH 12/24] make snapshot RFC a core command proposal --- rfcs/0013-cloud-serializable-sqlite-state.md | 105 +++++++++---------- 1 file changed, 50 insertions(+), 55 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 2de111a2..1c4aab03 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -1,5 +1,5 @@ --- -title: SQLite State Snapshot Plugin +title: SQLite State Snapshot Command authors: - giodl created: 2026-06-18 @@ -9,17 +9,17 @@ issue: rfc_pr: https://github.com/openclaw/rfcs/pull/20 --- -# Proposal: SQLite State Snapshot Plugin +# Proposal: SQLite State Snapshot Command ## Summary -Define an opt-in `snapshot` plugin that gives file-syncing hosts a safe file to sync for OpenClaw-owned SQLite state. +Define a narrow core `openclaw snapshot` command that gives file-syncing hosts a safe file to sync for OpenClaw-owned SQLite state. Hosted OpenClaw environments such as Scout/Lobster persist OpenClaw state by syncing files when they are saved. Live SQLite files are the wrong sync boundary: `state/openclaw.sqlite` or `agents//agent/openclaw-agent.sqlite` can be incomplete without their WAL, and `*.sqlite-wal`, `*.sqlite-shm`, and `*.sqlite-journal` are process-local sidecars rather than durable artifacts. -The `snapshot` extension provides the missing translation step. It asks SQLite to materialize a clean database artifact, verifies it, writes a manifest, and publishes the completed artifact set into a sync-owned location. That completed artifact write, not arbitrary live database churn, is what the host should sync. +The `snapshot` command provides the missing translation step. It asks SQLite to materialize a clean database artifact, verifies it, writes a manifest, and publishes the completed artifact set into a sync-owned location. That completed artifact write, not arbitrary live database churn, is what the host should sync. -SQLite remains the hot local runtime database. This RFC does not choose a replacement database, make cloud storage mandatory, or require managed failover. It defines a state-artifact boundary: OpenClaw owns the SQLite-aware artifact operation, while the host owns upload, retention, routing, encryption, and restore timing. Whether that boundary eventually belongs in core or remains extension-owned is an explicit design question; this RFC proves it first as a `snapshot` extension. +SQLite remains the hot local runtime database. This RFC does not choose a replacement database, make cloud storage mandatory, or require managed failover. It defines a state-artifact boundary: OpenClaw core owns the SQLite-aware artifact operation, while the host owns upload, retention, routing, encryption, and restore timing. ## Motivation @@ -43,17 +43,17 @@ Deltas do not remove this requirement. Ryan's underlying concern is that whole-f ## Goals - Keep SQLite as the hot local runtime database for this proposal. -- Make snapshot behavior opt-in through a plugin extension. +- Make snapshot behavior explicit through a narrow core `openclaw snapshot` command. - Produce a host-syncable SQLite artifact and manifest from live OpenClaw SQLite state. - Handle WAL state correctly without syncing live SQLite sidecars as durable artifacts. - Make restore and verification first-class behaviors, not incidental backup side effects. - Define a reusable SQLite snapshot provider contract for OpenClaw-owned SQLite databases. - Define the state-artifact boundary that lets hosted platforms persist OpenClaw state without understanding OpenClaw's internal SQLite file layout. -- Leave core-vs-extension ownership open while proving the mechanism first in the `snapshot` extension. +- Keep the command narrow: create, verify, and restore syncable SQLite artifacts. - Document host sync guidance: ignore live SQLite sidecars and sync completed snapshot artifacts/manifests instead. -- Allow the plugin to add commands under `openclaw snapshot`. +- Add commands under `openclaw snapshot`. - Leave `openclaw backup` integration as a possible later follow-up, not part of the initial proof stack. -- Keep default local OpenClaw behavior unchanged when the plugin is not installed or enabled. +- Keep default local OpenClaw runtime behavior unchanged unless the snapshot command is invoked. - Avoid hot writes over network filesystems as a durability or concurrency strategy. - Treat deltas as a future SQLite-aware optimization after full snapshot artifacts are correct. - Define lifecycle metadata needed to validate, order, restore, and audit snapshots. @@ -68,7 +68,7 @@ Deltas do not remove this requirement. Ryan's underlying concern is that whole-f - This RFC does not define a general database abstraction layer. - This RFC does not require OpenClaw to own the hosting platform's upload, retention, tenant routing, encryption, or object-storage policy. - This RFC does not make snapshots, cloud storage, or managed failover mandatory for local, self-hosted, or development OpenClaw installs. -- This RFC does not require the `snapshot` plugin to be the only long-term consumer of the core SQLite artifact primitive. +- This RFC does not require cloud providers or host integrations to live in core. - This RFC does not require object-store credentials, a lease service, or a managed-service control plane in the default runtime. - This RFC does not replace the session/transcript migration plan tracked by openclaw/openclaw#88838. - This RFC does not define tenant isolation, row-level authorization, or a multi-tenant schema model. @@ -79,28 +79,11 @@ Deltas do not remove this requirement. Ryan's underlying concern is that whole-f ## Proposal -### Plugin shape +### Command shape -Add an opt-in `snapshot` plugin that owns SQLite-safe snapshot and restore workflows for OpenClaw state. +Add a core `openclaw snapshot` command that owns SQLite-safe snapshot and restore workflows for OpenClaw state. -The plugin should be installable and removable like other OpenClaw plugins. When it is absent, default OpenClaw keeps its current local SQLite behavior and existing backup commands. - -The plugin is the first proposed consumer of the snapshot provider contract, not the only possible consumer. Over time, the same contract can be used by core backup/restore commands, doctor checks, managed startup hydration, or other OpenClaw features that need a consistent SQLite restore point. - -The plugin should follow the bundled extension pattern used by plugins such as `policy` and `oc-path`: - -```text -extensions/snapshot/ - package.json - openclaw.plugin.json - index.ts - src/ - snapshot-provider.ts - manifest.ts - local-repository.ts -``` - -The plugin adds a direct command surface: +The command surface should stay direct: ```text openclaw snapshot create @@ -110,6 +93,17 @@ openclaw snapshot list openclaw snapshot status ``` +The first named target shape can be: + +```text +openclaw snapshot create --target global +openclaw snapshot create --agent main +openclaw snapshot verify +openclaw snapshot restore --target +``` + +This does not imply automatic scheduling, cloud storage, failover, or a new database abstraction. The core command produces and verifies the syncable artifact; the host decides where that artifact is stored and when it is restored. + Later, if maintainers want one user-facing home for backup and restore workflows, the same provider contract can be wired under the existing backup command surface: ```text @@ -118,11 +112,11 @@ openclaw backup restore openclaw backup status ``` -That integration is intentionally not part of the initial implementation roadmap. The first proof should stay scoped to the `snapshot` plugin so it can demonstrate correctness without changing the existing backup command behavior. +That integration is intentionally not part of the initial implementation roadmap. The first proof should stay scoped to the `snapshot` command so it can demonstrate correctness without changing the existing backup command behavior. ### Responsibility split -OpenClaw must provide the SQLite-safe artifact contract because the host cannot safely infer it from the filesystem alone. The open design decision is packaging: the primitive may become core functionality, or the `snapshot` extension may remain the owner of the command and provider workflow while relying on a smaller core SQLite helper. +OpenClaw core must provide the SQLite-safe artifact contract because the host cannot safely infer it from the filesystem alone. OpenClaw should provide: @@ -135,13 +129,13 @@ OpenClaw should provide: - lifecycle metadata shape - safety rules such as no hot writes over network filesystems -The `snapshot` plugin should own the opt-in workflow around the core primitive: +The `snapshot` command should own the operator workflow around the core primitive: - snapshot command UX - local snapshot artifact creation and publication into a sync-owned artifact directory - snapshot manifest creation and verification - restore workflow orchestration -- provider hooks for storage backends +- provider hooks for future storage backends, if accepted Optional future integrations and providers can own: @@ -201,7 +195,7 @@ flowchart LR Metadata[lifecycle metadata] end - subgraph Plugin[snapshot plugin] + subgraph Command[Core snapshot command] Cmd[openclaw snapshot] Manifest[snapshot manifest] LocalRepo[(local snapshot repo)] @@ -231,9 +225,9 @@ flowchart LR Failover -. later .-> Restore ``` -The diagram is a responsibility split, not a default runtime requirement. Default OpenClaw can run with only the runtime box. Operators opt into the plugin when they need verified snapshot and restore workflows. +The diagram is a responsibility split, not a default managed-hosting requirement. Default local OpenClaw can run with only the runtime box. Operators and hosts use the snapshot command when they need verified snapshot and restore workflows. -For hosted OpenClaw, the same split becomes the host integration contract. The host can ask OpenClaw or the `snapshot` extension to materialize a clean artifact before upload and can hydrate local disk from a verified artifact before OpenClaw opens SQLite. The host does not need to treat live SQLite sidecars as durable sync inputs. +For hosted OpenClaw, the same split becomes the host integration contract. The host can ask OpenClaw to materialize a clean artifact before upload and can hydrate local disk from a verified artifact before OpenClaw opens SQLite. The host does not need to treat live SQLite sidecars as durable sync inputs. ### Snapshot semantics @@ -247,7 +241,7 @@ The unit of snapshotting is an existing OpenClaw-owned SQLite database. The prim This RFC does not rename or redesign those logical units; openclaw/openclaw#94646 makes them concrete enough for snapshot to target. The RFC defines snapshot behavior that can apply to each eligible database. -The provider contract should therefore take a database reference rather than assume one hard-coded database path. Core can decide which SQLite databases are eligible, and the plugin or provider can apply the same snapshot semantics to each eligible database. +The provider contract should therefore take a database reference rather than assume one hard-coded database path. Core can decide which SQLite databases are eligible and apply the same snapshot semantics to each eligible database. A snapshot must: @@ -277,13 +271,15 @@ The artifact model should support: The delta mechanism can be WAL-frame based, page based, logical-change based, external-tool based, or backend-native. This RFC requires the contract, not one specific encoding. +Delta support must be anchored to a verified snapshot generation. The system should not treat arbitrary file-sync deltas from the live runtime directory as a restore stream. A valid delta design needs ordering, a base snapshot cursor, integrity checks, and replay rules that SQLite/OpenClaw can verify before opening the restored database. + Artifacts should be suitable for host persistence. A hosting platform should be able to upload, retain, copy, and later download the artifact set without preserving process-local SQLite sidecars or relying on a mounted shared filesystem. The sync-owned artifact directory should not be the live SQLite runtime directory. If the host syncs on file save, OpenClaw should write completed artifacts into a separate artifact location after verification. That keeps the host from observing transient SQLite sidecars or partially materialized runtime state. Artifacts should be created at explicit lifecycle moments: -- on demand, when an operator or plugin command requests a snapshot +- on demand, when an operator or host requests a snapshot - before a managed host uploads/syncs OpenClaw state - before container shutdown or ownership release when the platform can coordinate that moment - before migration or other state-changing maintenance when a rollback point is required @@ -293,7 +289,7 @@ Restore artifacts should be consumed before OpenClaw opens the target database f ### Restore verification -Restore is a required behavior for the `snapshot` plugin, not an incidental backup side effect. +Restore is a required behavior for `openclaw snapshot`, not an incidental backup side effect. A restore operation must: @@ -308,7 +304,7 @@ The first implementation milestone should prove that OpenClaw can boot from rest ### Failover path -The plugin is not required to implement automatic failover in the first milestone, but it should be designed as the foundation for failover. +The command is not required to implement automatic failover in the first milestone, but it should be designed as the foundation for failover. Failover becomes possible when OpenClaw has: @@ -354,13 +350,13 @@ At minimum, snapshot metadata should include: - restore source and restore point when hydrated - current writer owner or lease holder, when leases are enabled -The exact storage location for this metadata is implementation-defined, but the `snapshot` plugin must be able to read enough metadata to verify and restore a snapshot without opening a possibly unsafe runtime database first. +The exact storage location for this metadata is implementation-defined, but `openclaw snapshot` must be able to read enough metadata to verify and restore a snapshot without opening a possibly unsafe runtime database first. ### Provider shape The implementation can start as a SQLite-specific snapshot provider rather than a database abstraction layer. -The provider contract should be reusable by any OpenClaw feature that needs to capture or restore an OpenClaw-owned SQLite database. The `snapshot` plugin is the first proposed packaging and CLI surface, but the contract should not depend on plugin-only state. +The provider contract should be reusable by any OpenClaw feature that needs to capture or restore an OpenClaw-owned SQLite database. `openclaw snapshot` is the first proposed CLI surface, but the contract should not depend on command-only state. A minimal shape is: @@ -386,19 +382,18 @@ type RemoteSnapshotProvider = SqliteSnapshotProvider & { This keeps SQLite runtime access local while making state artifacts portable. A local snapshot provider can be the reference implementation. Cloud/object-store providers can come later without changing the default local runtime. -If the design proves broadly useful, core can adopt the same contract for built-in backup restore, startup hydration, or state migration workflows without requiring the `snapshot` plugin command surface to become mandatory. +If the design proves broadly useful, the same contract can support backup restore, startup hydration, or state migration workflows without changing the narrow `openclaw snapshot` command. ### Implementation roadmap The initial implementation should be a short PR stack that proves correctness before expanding product surface. -#### PR 1: provider proof +#### PR 1: core snapshot provider proof -Add the bundled `snapshot` plugin scaffold and a local SQLite snapshot provider. +Add the shared SQLite snapshot provider and local artifact repository. This PR should include: -- `extensions/snapshot` plugin manifest, package metadata, and entrypoint - `SqliteSnapshotProvider` contract - local snapshot repository - snapshot manifest and content hash verification @@ -408,9 +403,9 @@ This PR should include: This PR does not need the full public restore CLI. It should prove that the provider can create and verify a correct SQLite snapshot artifact without publishing a premature provider API surface. -#### PR 2: public snapshot CLI +#### PR 2: core snapshot CLI -Expose the user-facing plugin commands: +Expose the user-facing core commands: ```text openclaw snapshot create @@ -418,7 +413,7 @@ openclaw snapshot verify openclaw snapshot restore ``` -This PR should add target-directory safety checks, restore manifest validation, SQLite integrity checks after restore, and docs for the `snapshot` plugin command surface. +This PR should add target-directory safety checks, restore manifest validation, SQLite integrity checks after restore, and docs for the `openclaw snapshot` command surface. The CLI should accept an explicit database path for the proof path, but the intended product model is not "any random SQLite file forever." The command should be able to grow toward named OpenClaw database targets such as global state or a specific agent database once core exposes the eligible database registry cleanly. @@ -426,7 +421,7 @@ The CLI should accept an explicit database path for the proof path, but the inte Prove that a restored snapshot can hydrate a fresh OpenClaw state directory before runtime opens SQLite. -This PR should demonstrate that OpenClaw can start from restored state in a fresh directory, host, or container-style environment. That proof is what makes the plugin a credible failover substrate rather than only an archive utility. +This PR should demonstrate that OpenClaw can start from restored state in a fresh directory, host, or container-style environment. That proof is what makes the command a credible failover substrate rather than only an archive utility. This proof should also document the host contract: which OpenClaw command or API materializes the artifact, which manifest fields the host can store without interpreting SQLite internals, and what must be restored before OpenClaw opens the database. @@ -447,9 +442,9 @@ This approach targets the reliability problem directly. It does not require Open The database-first work in openclaw/openclaw#94646 improves this RFC because it gives snapshot a concrete target model. Snapshot does not have to invent logical database units. It can operate over the already-established global control-plane database and per-agent data-plane databases, then extend to dedicated owner stores only when those stores have comparable ownership and lifecycle metadata. -Calling the extension `snapshot` keeps the first deliverable concrete. It describes the artifact OpenClaw needs before higher-level reliability features can exist. It also avoids overpromising automatic failover before leases, promotion, and orchestration are designed. +Calling the command `snapshot` keeps the first deliverable concrete. It describes the artifact OpenClaw needs before higher-level reliability features can exist. It also avoids overpromising automatic failover before leases, promotion, and orchestration are designed. -Keeping the first implementation stack under `openclaw snapshot` keeps the proof small and plugin-scoped. Existing `openclaw backup create` and `openclaw backup verify` behavior can remain unchanged while the snapshot provider proves the harder SQLite correctness and restore semantics. +Keeping the first implementation stack under `openclaw snapshot` keeps the proof small and command-scoped. Existing `openclaw backup create` and `openclaw backup verify` behavior can remain unchanged while the snapshot provider proves the harder SQLite correctness and restore semantics. Treating remote storage as artifact storage avoids the common failure mode where object storage or network filesystems are used as if they were local disk. SQLite remains local and authoritative while running. Reliability comes from verified snapshots, manifests, restore procedures, and later deltas. @@ -459,7 +454,7 @@ Keeping the SQLite-safe artifact contract inside OpenClaw is important because s Hosted deployments make that responsibility more important, not less. A host can persist a directory, but OpenClaw should define which database artifacts are safe to persist. Without that boundary, every host integration has to rediscover SQLite sidecar rules and OpenClaw database ownership independently. -The provider contract gives OpenClaw a path from plugin experimentation to core adoption. The first implementation can live as an opt-in plugin while the contract stays general enough for future core backup, restore, startup hydration, or migration workflows. +The provider contract gives OpenClaw a path from a narrow command to future backup, restore, startup hydration, or migration workflows. Explicit writer ownership keeps horizontal service orchestration honest. A service can move work between hosts, but it must move ownership and restore state deliberately rather than letting several instances write the same SQLite database through shared storage. @@ -473,6 +468,6 @@ The proposal also keeps storage ownership decisions out of scope. OpenClaw alrea - What is the acceptable data-loss window for managed deployments before deltas are implemented? - Where should writer ownership metadata live before a database is opened? - Should restore verification run during startup, doctor, a managed-control-plane action, or all three? -- What is the minimum host-facing API or command shape needed for Lobster/Aether-style platforms to request artifact materialization and pre-start hydration without importing plugin-specific policy? +- What is the minimum host-facing API or command shape needed for Scout/Lobster-style platforms to request artifact materialization and pre-start hydration? - Which artifacts should be included with database restore for support/debug exports versus canonical runtime recovery? - Should external tools such as Litestream or LiteFS be provider integrations, deployment recommendations, or out of scope for OpenClaw-owned code? From 9a8d088497f712ced3eca3bb32e1617b5d3d3eba Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Thu, 18 Jun 2026 20:16:57 -0700 Subject: [PATCH 13/24] add snapshot delta roadmap --- rfcs/0013-cloud-serializable-sqlite-state.md | 79 +++++++++++++++++++- 1 file changed, 76 insertions(+), 3 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 1c4aab03..545dc326 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -273,6 +273,40 @@ The delta mechanism can be WAL-frame based, page based, logical-change based, ex Delta support must be anchored to a verified snapshot generation. The system should not treat arbitrary file-sync deltas from the live runtime directory as a restore stream. A valid delta design needs ordering, a base snapshot cursor, integrity checks, and replay rules that SQLite/OpenClaw can verify before opening the restored database. +### Delta model + +Deltas are the answer to Ryan's scaling concern, but only after OpenClaw has a +verified full snapshot contract. A delta is not "whatever changed on disk since +the sync tool last ran." A valid delta is an OpenClaw-authored artifact that is +created from a known base snapshot generation and replayed in manifest order. + +The first delta design should keep these invariants: + +- every delta names its base snapshot generation or previous delta cursor +- every delta has a monotonically ordered sequence number +- every delta records the source database id, schema version, and page size or + equivalent compatibility data +- every delta records content hashes before upload and after download +- restore applies deltas only after verifying the base snapshot +- restore rejects gaps, forks, duplicate sequence numbers, incompatible schema + versions, and failed integrity checks +- OpenClaw verifies the final restored database before runtime opens SQLite + +The encoding can be decided later. Plausible encodings include: + +- WAL-frame deltas, if OpenClaw can safely capture frame order and checkpoint + boundaries +- page-level deltas, if OpenClaw can identify changed pages from a verified + base snapshot +- logical deltas, if a future schema layer exposes stable logical changes +- external-tool deltas, if an accepted provider such as Litestream or LiteFS can + satisfy the same manifest, ordering, and restore verification contract + +The milestone should be: full snapshot first, verified restore second, then +ordered deltas from a snapshot cursor. File-sync deltas over the live runtime +directory remain out of scope because they do not carry SQLite ordering, +checkpoint, or replay semantics. + Artifacts should be suitable for host persistence. A hosting platform should be able to upload, retain, copy, and later download the artifact set without preserving process-local SQLite sidecars or relying on a mounted shared filesystem. The sync-owned artifact directory should not be the live SQLite runtime directory. If the host syncs on file save, OpenClaw should write completed artifacts into a separate artifact location after verification. That keeps the host from observing transient SQLite sidecars or partially materialized runtime state. @@ -417,7 +451,20 @@ This PR should add target-directory safety checks, restore manifest validation, The CLI should accept an explicit database path for the proof path, but the intended product model is not "any random SQLite file forever." The command should be able to grow toward named OpenClaw database targets such as global state or a specific agent database once core exposes the eligible database registry cleanly. -#### PR 3: fresh-state boot proof +#### PR 3: named OpenClaw database targets + +Teach `openclaw snapshot` to address OpenClaw-owned databases by stable names +instead of only accepting arbitrary paths. + +This PR should include: + +- `--target global` for `state/openclaw.sqlite` +- `--agent ` for `agents//agent/openclaw-agent.sqlite` +- manifest fields for database role, agent id, schema version, and source path +- host-sync guidance that says live SQLite sidecars are ignored and completed + artifacts are the sync input + +#### PR 4: fresh-state boot proof Prove that a restored snapshot can hydrate a fresh OpenClaw state directory before runtime opens SQLite. @@ -425,12 +472,38 @@ This PR should demonstrate that OpenClaw can start from restored state in a fres This proof should also document the host contract: which OpenClaw command or API materializes the artifact, which manifest fields the host can store without interpreting SQLite internals, and what must be restored before OpenClaw opens the database. +#### PR 5: delta manifest contract + +Add the manifest shape and verification rules for ordered deltas without +requiring a production delta encoder yet. + +This PR should include: + +- base snapshot generation and delta sequence fields +- hash and byte-size fields for delta artifacts +- restore validation for gaps, forks, duplicate sequence numbers, and wrong base + snapshot generation +- tests that prove invalid delta chains are rejected before database restore + +#### PR 6: reference delta encoder proof + +Add one narrow SQLite-aware delta encoder proof. The RFC does not require one +encoding, but the first implementation should choose a small reference path and +prove that restore works from: + +```text +full snapshot + ordered deltas -> verified local SQLite database +``` + +The proof can use WAL-frame, page-level, logical, or external-tool-backed +encoding, but it must satisfy the delta manifest contract and run SQLite +integrity verification after replay. + #### Later work -After the three initial PRs, follow-up RFCs or implementation PRs can consider: +After the initial PRs, follow-up RFCs or implementation PRs can consider: - `openclaw backup` integration -- incremental deltas - object/blob storage providers - retention and scheduling - leases, promotion, fencing, and managed failover From ccbfc2285b7972e5b840a5e39fd6cbba6c108229 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Thu, 18 Jun 2026 20:36:00 -0700 Subject: [PATCH 14/24] narrow delta plan to wal bundles --- rfcs/0013-cloud-serializable-sqlite-state.md | 83 +++++++++----------- 1 file changed, 36 insertions(+), 47 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 545dc326..e8c31f46 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -38,7 +38,7 @@ The syncable file is created deliberately. OpenClaw should open the source datab For hosts that sync on filesystem changes, the sync trigger should be the completed artifact write, not arbitrary writes inside the live OpenClaw state directory. The artifact directory should contain completed database artifacts and manifests only. Live `*.sqlite-wal`, `*.sqlite-shm`, and `*.sqlite-journal` files should not be created there, because a sync agent can observe and upload them as soon as they appear. -Deltas do not remove this requirement. Ryan's underlying concern is that whole-file copies do not scale, but unmanaged file deltas over live SQLite make the correctness problem worse: a file-sync tool can observe DB pages, WAL frames, and sidecars at different moments without knowing SQLite ordering or checkpoint state. Deltas should be a later optimization from a verified snapshot cursor, not a replacement for the initial clean artifact boundary. +Deltas do not remove this requirement. Ryan's underlying concern is that whole-file copies do not scale, but unmanaged file deltas over live SQLite make the correctness problem worse: a file-sync tool can observe DB pages, WAL frames, and sidecars at different moments without knowing SQLite ordering or checkpoint state. If high-frequency sync is needed, the simple follow-up should be ordered WAL-bundle artifacts anchored to a verified full snapshot. ## Goals @@ -55,7 +55,7 @@ Deltas do not remove this requirement. Ryan's underlying concern is that whole-f - Leave `openclaw backup` integration as a possible later follow-up, not part of the initial proof stack. - Keep default local OpenClaw runtime behavior unchanged unless the snapshot command is invoked. - Avoid hot writes over network filesystems as a durability or concurrency strategy. -- Treat deltas as a future SQLite-aware optimization after full snapshot artifacts are correct. +- Treat WAL bundles as the first high-frequency optimization after full snapshot artifacts are correct. - Define lifecycle metadata needed to validate, order, restore, and audit snapshots. - Leave cloud artifact storage, retention, scheduling, and failover orchestration to optional providers or later RFCs. - Let hosts such as Scout/Lobster own durable destination policy while OpenClaw owns the correctness of the local SQLite artifact operation. @@ -265,45 +265,36 @@ The artifact model should support: - compact snapshot artifacts - ordered manifests - content hashes or equivalent integrity checks -- optional incremental deltas after the first milestone +- optional WAL bundles after the first milestones - resumable upload and download when a remote provider is configured -- restore from the latest valid snapshot plus any required ordered deltas +- restore from the latest valid snapshot plus any required ordered WAL bundles -The delta mechanism can be WAL-frame based, page based, logical-change based, external-tool based, or backend-native. This RFC requires the contract, not one specific encoding. +The follow-up high-frequency mechanism should start with WAL bundles, not a broad delta abstraction. -Delta support must be anchored to a verified snapshot generation. The system should not treat arbitrary file-sync deltas from the live runtime directory as a restore stream. A valid delta design needs ordering, a base snapshot cursor, integrity checks, and replay rules that SQLite/OpenClaw can verify before opening the restored database. +WAL bundles must be anchored to a verified snapshot generation. The system should not treat arbitrary file-sync deltas from the live runtime directory as a restore stream. A valid bundle design needs ordering, a base snapshot cursor, integrity checks, and replay rules that SQLite/OpenClaw can verify before opening the restored database. -### Delta model +### WAL bundle model -Deltas are the answer to Ryan's scaling concern, but only after OpenClaw has a -verified full snapshot contract. A delta is not "whatever changed on disk since -the sync tool last ran." A valid delta is an OpenClaw-authored artifact that is -created from a known base snapshot generation and replayed in manifest order. +WAL bundles are the first proposed answer to Ryan's scaling concern, but only +after OpenClaw has a verified full snapshot contract. A WAL bundle is not the +live `*.sqlite-wal` file. It is an OpenClaw-authored artifact cut from a known +SQLite state range, written to staging, verified, and then published into the +sync-owned artifact directory. -The first delta design should keep these invariants: +The first WAL bundle design should keep these invariants: -- every delta names its base snapshot generation or previous delta cursor -- every delta has a monotonically ordered sequence number -- every delta records the source database id, schema version, and page size or +- every bundle names its base snapshot generation or previous bundle cursor +- every bundle has a monotonically ordered sequence number +- every bundle records the source database id, schema version, and page size or equivalent compatibility data -- every delta records content hashes before upload and after download -- restore applies deltas only after verifying the base snapshot +- every bundle records content hashes before upload and after download +- restore applies bundles only after verifying the base snapshot - restore rejects gaps, forks, duplicate sequence numbers, incompatible schema versions, and failed integrity checks - OpenClaw verifies the final restored database before runtime opens SQLite -The encoding can be decided later. Plausible encodings include: - -- WAL-frame deltas, if OpenClaw can safely capture frame order and checkpoint - boundaries -- page-level deltas, if OpenClaw can identify changed pages from a verified - base snapshot -- logical deltas, if a future schema layer exposes stable logical changes -- external-tool deltas, if an accepted provider such as Litestream or LiteFS can - satisfy the same manifest, ordering, and restore verification contract - The milestone should be: full snapshot first, verified restore second, then -ordered deltas from a snapshot cursor. File-sync deltas over the live runtime +simple ordered WAL bundles from a snapshot cursor. File-sync deltas over the live runtime directory remain out of scope because they do not carry SQLite ordering, checkpoint, or replay semantics. @@ -346,7 +337,7 @@ Failover becomes possible when OpenClaw has: 2. a way to hydrate local disk before startup 3. a way to confirm schema and integrity before runtime writes 4. a clear owner for the database after restore -5. optional deltas or upload scheduling to reduce the data-loss window +5. optional WAL bundles or upload scheduling to reduce the data-loss window A later RFC can define leases, promotion, fencing, standby replicas, and managed orchestration. This RFC provides the snapshot and restore substrate those systems need. @@ -472,32 +463,29 @@ This PR should demonstrate that OpenClaw can start from restored state in a fres This proof should also document the host contract: which OpenClaw command or API materializes the artifact, which manifest fields the host can store without interpreting SQLite internals, and what must be restored before OpenClaw opens the database. -#### PR 5: delta manifest contract +#### PR 5: simple WAL bundle proof -Add the manifest shape and verification rules for ordered deltas without -requiring a production delta encoder yet. +Add the smallest WAL-bundle proof that reduces full-snapshot frequency without +blessing live-file deltas. This PR should include: -- base snapshot generation and delta sequence fields -- hash and byte-size fields for delta artifacts +- base snapshot generation and bundle sequence fields +- hash and byte-size fields for WAL bundle artifacts +- staging then publish into the sync-owned artifact directory - restore validation for gaps, forks, duplicate sequence numbers, and wrong base snapshot generation -- tests that prove invalid delta chains are rejected before database restore - -#### PR 6: reference delta encoder proof - -Add one narrow SQLite-aware delta encoder proof. The RFC does not require one -encoding, but the first implementation should choose a small reference path and -prove that restore works from: +- tests that prove invalid bundle chains are rejected before database restore +- proof that restore works from: ```text -full snapshot + ordered deltas -> verified local SQLite database +full snapshot + ordered WAL bundles -> verified local SQLite database ``` -The proof can use WAL-frame, page-level, logical, or external-tool-backed -encoding, but it must satisfy the delta manifest contract and run SQLite -integrity verification after replay. +This PR should not add retention policy, object storage, failover, or multiple +delta encodings. Compaction can be simple: replay the latest full snapshot plus +bundles into a temp database, verify it, publish a new full snapshot, then leave +pruning policy to later work. #### Later work @@ -506,6 +494,7 @@ After the initial PRs, follow-up RFCs or implementation PRs can consider: - `openclaw backup` integration - object/blob storage providers - retention and scheduling +- WAL bundle compaction and pruning policy - leases, promotion, fencing, and managed failover - external tool integrations such as Litestream or LiteFS @@ -519,7 +508,7 @@ Calling the command `snapshot` keeps the first deliverable concrete. It describe Keeping the first implementation stack under `openclaw snapshot` keeps the proof small and command-scoped. Existing `openclaw backup create` and `openclaw backup verify` behavior can remain unchanged while the snapshot provider proves the harder SQLite correctness and restore semantics. -Treating remote storage as artifact storage avoids the common failure mode where object storage or network filesystems are used as if they were local disk. SQLite remains local and authoritative while running. Reliability comes from verified snapshots, manifests, restore procedures, and later deltas. +Treating remote storage as artifact storage avoids the common failure mode where object storage or network filesystems are used as if they were local disk. SQLite remains local and authoritative while running. Reliability comes from verified snapshots, manifests, restore procedures, and later WAL bundles. Making the feature opt-in keeps the default OpenClaw runtime simple. Local and development users should not need object storage, a lease service, or a managed scheduler to keep using SQLite. @@ -538,7 +527,7 @@ The proposal also keeps storage ownership decisions out of scope. OpenClaw alrea - Which database-first unit should be used for the first named snapshot/restore proof: global control-plane state, one per-agent data-plane database, or both? - Should the first checkpoint implementation use SQLite online backup, `VACUUM INTO`, WAL checkpointing, page capture, or a higher-level export format? - Should the reference provider be a local snapshot repository only, or should it include one object/blob storage provider? -- What is the acceptable data-loss window for managed deployments before deltas are implemented? +- What is the acceptable data-loss window for managed deployments before WAL bundles are implemented? - Where should writer ownership metadata live before a database is opened? - Should restore verification run during startup, doctor, a managed-control-plane action, or all three? - What is the minimum host-facing API or command shape needed for Scout/Lobster-style platforms to request artifact materialization and pre-start hydration? From 99a56e291133fe1aa4cde2f1baec4fb52c1d8492 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Thu, 18 Jun 2026 20:40:29 -0700 Subject: [PATCH 15/24] Split snapshot roadmap into phases --- rfcs/0013-cloud-serializable-sqlite-state.md | 54 ++++++++++++++++---- 1 file changed, 43 insertions(+), 11 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index e8c31f46..ba416e14 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -411,9 +411,26 @@ If the design proves broadly useful, the same contract can support backup restor ### Implementation roadmap -The initial implementation should be a short PR stack that proves correctness before expanding product surface. +The implementation should be split into two phases. -#### PR 1: core snapshot provider proof +Phase 1 is the committed snapshot command work. It proves full verified +snapshots, named OpenClaw database targets, and fresh-state restore. It should +also collect the metrics that decide whether Phase 2 is worth doing: + +- snapshot artifact size by database role +- snapshot duration +- upload or sync bytes and time, when the host can report them +- restore duration +- snapshot frequency required to meet the desired recovery point objective +- observed WAL growth between snapshots +- runtime impact while snapshotting + +Phase 2 is gated. It should start only if Phase 1 data shows full snapshots are +too large, too slow, or too infrequent for hosted deployments. Phase 2's planned +shape is WAL bundles plus the minimum compaction proof needed to keep bundle +chains bounded. + +#### Phase 1 / PR 1: core snapshot provider proof Add the shared SQLite snapshot provider and local artifact repository. @@ -428,7 +445,7 @@ This PR should include: This PR does not need the full public restore CLI. It should prove that the provider can create and verify a correct SQLite snapshot artifact without publishing a premature provider API surface. -#### PR 2: core snapshot CLI +#### Phase 1 / PR 2: core snapshot CLI Expose the user-facing core commands: @@ -442,7 +459,7 @@ This PR should add target-directory safety checks, restore manifest validation, The CLI should accept an explicit database path for the proof path, but the intended product model is not "any random SQLite file forever." The command should be able to grow toward named OpenClaw database targets such as global state or a specific agent database once core exposes the eligible database registry cleanly. -#### PR 3: named OpenClaw database targets +#### Phase 1 / PR 3: named OpenClaw database targets Teach `openclaw snapshot` to address OpenClaw-owned databases by stable names instead of only accepting arbitrary paths. @@ -455,7 +472,7 @@ This PR should include: - host-sync guidance that says live SQLite sidecars are ignored and completed artifacts are the sync input -#### PR 4: fresh-state boot proof +#### Phase 1 / PR 4: fresh-state boot proof and metrics Prove that a restored snapshot can hydrate a fresh OpenClaw state directory before runtime opens SQLite. @@ -463,7 +480,15 @@ This PR should demonstrate that OpenClaw can start from restored state in a fres This proof should also document the host contract: which OpenClaw command or API materializes the artifact, which manifest fields the host can store without interpreting SQLite internals, and what must be restored before OpenClaw opens the database. -#### PR 5: simple WAL bundle proof +This PR should record or report the Phase 1 metrics when available: snapshot +size, snapshot duration, restore duration, WAL growth, and upload/sync bytes and +time when the host supplies them. It should also document the greenlight +criteria maintainers would use before starting Phase 2. + +#### Phase 2 / PR 5: simple WAL bundle proof + +Phase 2 is not automatically required by Phase 1. It needs a maintainer +greenlight based on Phase 1 metrics. Add the smallest WAL-bundle proof that reduces full-snapshot frequency without blessing live-file deltas. @@ -483,9 +508,9 @@ full snapshot + ordered WAL bundles -> verified local SQLite database ``` This PR should not add retention policy, object storage, failover, or multiple -delta encodings. Compaction can be simple: replay the latest full snapshot plus -bundles into a temp database, verify it, publish a new full snapshot, then leave -pruning policy to later work. +delta encodings. It should include a simple compaction proof: replay the latest +full snapshot plus bundles into a temp database, verify it, and publish a new +full snapshot generation. Conservative pruning policy can remain later work. #### Later work @@ -494,7 +519,7 @@ After the initial PRs, follow-up RFCs or implementation PRs can consider: - `openclaw backup` integration - object/blob storage providers - retention and scheduling -- WAL bundle compaction and pruning policy +- WAL bundle retention and pruning policy - leases, promotion, fencing, and managed failover - external tool integrations such as Litestream or LiteFS @@ -510,6 +535,12 @@ Keeping the first implementation stack under `openclaw snapshot` keeps the proof Treating remote storage as artifact storage avoids the common failure mode where object storage or network filesystems are used as if they were local disk. SQLite remains local and authoritative while running. Reliability comes from verified snapshots, manifests, restore procedures, and later WAL bundles. +Phase 2 is deliberately gated. WAL bundles reduce sync cost and restore-point +gaps only if Phase 1 metrics show full snapshots are insufficient. Keeping WAL +bundles behind a greenlight avoids committing to replay and compaction +complexity before OpenClaw knows database size, snapshot time, host sync cost, +restore time, WAL growth, and recovery point needs. + Making the feature opt-in keeps the default OpenClaw runtime simple. Local and development users should not need object storage, a lease service, or a managed scheduler to keep using SQLite. Keeping the SQLite-safe artifact contract inside OpenClaw is important because safe snapshots and restores need access to database paths, WAL behavior, schema versions, and integrity checks. Provider-owned artifact storage keeps cloud credentials, retention policy, and managed failover out of the default runtime. @@ -527,7 +558,8 @@ The proposal also keeps storage ownership decisions out of scope. OpenClaw alrea - Which database-first unit should be used for the first named snapshot/restore proof: global control-plane state, one per-agent data-plane database, or both? - Should the first checkpoint implementation use SQLite online backup, `VACUUM INTO`, WAL checkpointing, page capture, or a higher-level export format? - Should the reference provider be a local snapshot repository only, or should it include one object/blob storage provider? -- What is the acceptable data-loss window for managed deployments before WAL bundles are implemented? +- Which Phase 1 metric thresholds should greenlight Phase 2 WAL bundles: artifact size, snapshot duration, sync bytes/time, restore duration, WAL growth, or required recovery point objective? +- Should simple compaction land with the first WAL bundle proof or as the next Phase 2 PR? - Where should writer ownership metadata live before a database is opened? - Should restore verification run during startup, doctor, a managed-control-plane action, or all three? - What is the minimum host-facing API or command shape needed for Scout/Lobster-style platforms to request artifact materialization and pre-start hydration? From df107b512b3ab60dc9fdbf0f631052c126e3686a Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Thu, 18 Jun 2026 22:21:41 -0700 Subject: [PATCH 16/24] Link snapshot implementation PR stack --- rfcs/0013-cloud-serializable-sqlite-state.md | 30 +++++++++++++++++--- 1 file changed, 26 insertions(+), 4 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index ba416e14..61bd2d75 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -430,10 +430,21 @@ too large, too slow, or too infrequent for hosted deployments. Phase 2's planned shape is WAL bundles plus the minimum compaction proof needed to keep bundle chains bounded. +The current Phase 1 implementation stack is: + +| PR | Purpose | +| -------------------------------------------------------------------------- | ------------------------------------ | +| [openclaw/openclaw#94694](https://github.com/openclaw/openclaw/pull/94694) | Core snapshot provider proof | +| [openclaw/openclaw#94717](https://github.com/openclaw/openclaw/pull/94717) | Core `openclaw snapshot` CLI | +| [openclaw/openclaw#94799](https://github.com/openclaw/openclaw/pull/94799) | Named OpenClaw database targets | +| [openclaw/openclaw#94805](https://github.com/openclaw/openclaw/pull/94805) | Safe-sync artifact and restore proof | + #### Phase 1 / PR 1: core snapshot provider proof Add the shared SQLite snapshot provider and local artifact repository. +Implementation: [openclaw/openclaw#94694](https://github.com/openclaw/openclaw/pull/94694). + This PR should include: - `SqliteSnapshotProvider` contract @@ -459,6 +470,8 @@ This PR should add target-directory safety checks, restore manifest validation, The CLI should accept an explicit database path for the proof path, but the intended product model is not "any random SQLite file forever." The command should be able to grow toward named OpenClaw database targets such as global state or a specific agent database once core exposes the eligible database registry cleanly. +Implementation: [openclaw/openclaw#94717](https://github.com/openclaw/openclaw/pull/94717). + #### Phase 1 / PR 3: named OpenClaw database targets Teach `openclaw snapshot` to address OpenClaw-owned databases by stable names @@ -472,19 +485,28 @@ This PR should include: - host-sync guidance that says live SQLite sidecars are ignored and completed artifacts are the sync input -#### Phase 1 / PR 4: fresh-state boot proof and metrics +Implementation: [openclaw/openclaw#94799](https://github.com/openclaw/openclaw/pull/94799). + +#### Phase 1 / PR 4: safe-sync artifact and restore proof -Prove that a restored snapshot can hydrate a fresh OpenClaw state directory before runtime opens SQLite. +Prove that a completed snapshot artifact, not the live SQLite file family, is +the host-sync boundary. -This PR should demonstrate that OpenClaw can start from restored state in a fresh directory, host, or container-style environment. That proof is what makes the command a credible failover substrate rather than only an archive utility. +This PR should demonstrate that OpenClaw can create a snapshot from a named +database target, copy only the completed snapshot directory, restore from that +copied artifact into a fresh local SQLite path, and verify the restored +database. That proof is what makes the command a credible failover substrate +rather than only an archive utility. -This proof should also document the host contract: which OpenClaw command or API materializes the artifact, which manifest fields the host can store without interpreting SQLite internals, and what must be restored before OpenClaw opens the database. +This proof should also document the host contract: which OpenClaw command or API materializes the artifact, which files are safe to sync, which live SQLite sidecars should be ignored, which manifest fields the host can store without interpreting SQLite internals, and what must be restored before OpenClaw opens the database. This PR should record or report the Phase 1 metrics when available: snapshot size, snapshot duration, restore duration, WAL growth, and upload/sync bytes and time when the host supplies them. It should also document the greenlight criteria maintainers would use before starting Phase 2. +Implementation: [openclaw/openclaw#94805](https://github.com/openclaw/openclaw/pull/94805). + #### Phase 2 / PR 5: simple WAL bundle proof Phase 2 is not automatically required by Phase 1. It needs a maintainer From 0e368ad612770e60eb0f923cca5784253cae07e7 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Fri, 19 Jun 2026 06:39:19 -0700 Subject: [PATCH 17/24] Add snapshot stress PR to RFC roadmap --- rfcs/0013-cloud-serializable-sqlite-state.md | 32 +++++++++++++++----- 1 file changed, 25 insertions(+), 7 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 61bd2d75..258a54ef 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -432,12 +432,13 @@ chains bounded. The current Phase 1 implementation stack is: -| PR | Purpose | -| -------------------------------------------------------------------------- | ------------------------------------ | -| [openclaw/openclaw#94694](https://github.com/openclaw/openclaw/pull/94694) | Core snapshot provider proof | -| [openclaw/openclaw#94717](https://github.com/openclaw/openclaw/pull/94717) | Core `openclaw snapshot` CLI | -| [openclaw/openclaw#94799](https://github.com/openclaw/openclaw/pull/94799) | Named OpenClaw database targets | -| [openclaw/openclaw#94805](https://github.com/openclaw/openclaw/pull/94805) | Safe-sync artifact and restore proof | +| PR | Purpose | +| -------------------------------------------------------------------------- | --------------------------------------------- | +| [openclaw/openclaw#94694](https://github.com/openclaw/openclaw/pull/94694) | Core snapshot provider proof | +| [openclaw/openclaw#94717](https://github.com/openclaw/openclaw/pull/94717) | Core `openclaw snapshot` CLI | +| [openclaw/openclaw#94799](https://github.com/openclaw/openclaw/pull/94799) | Named OpenClaw database targets | +| [openclaw/openclaw#94805](https://github.com/openclaw/openclaw/pull/94805) | Safe-sync artifact and restore proof | +| [openclaw/openclaw#94967](https://github.com/openclaw/openclaw/pull/94967) | Snapshot stress harness for Phase 1 greenlight | #### Phase 1 / PR 1: core snapshot provider proof @@ -507,7 +508,24 @@ criteria maintainers would use before starting Phase 2. Implementation: [openclaw/openclaw#94805](https://github.com/openclaw/openclaw/pull/94805). -#### Phase 2 / PR 5: simple WAL bundle proof +#### Phase 1 / PR 5: snapshot stress harness + +Add an opt-in harness that measures the Phase 1 behavior before maintainers +decide whether Phase 2 is worth building. + +This PR should prove that snapshot artifacts still restore cleanly while a +writer is committing transaction batches against the source database. It should +cover both `--target global` and `--agent `, report snapshot/restore p50/p95 +timings, snapshot bytes, WAL bytes after the run, writer rows, and restore +verification counts. + +This PR should not make stress mandatory in normal CI. It is a local/release +validation tool for maintainers and operators who need evidence before tuning +snapshot frequency or greenlighting WAL bundles. + +Implementation: [openclaw/openclaw#94967](https://github.com/openclaw/openclaw/pull/94967). + +#### Phase 2 / PR 6: simple WAL bundle proof Phase 2 is not automatically required by Phase 1. It needs a maintainer greenlight based on Phase 1 metrics. From 48efa66375ee6d967c5c01f03221279338c31116 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Fri, 19 Jun 2026 07:33:36 -0700 Subject: [PATCH 18/24] Collapse snapshot implementation roadmap --- rfcs/0013-cloud-serializable-sqlite-state.md | 57 +++++--------------- 1 file changed, 13 insertions(+), 44 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 258a54ef..1f391f87 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -434,19 +434,18 @@ The current Phase 1 implementation stack is: | PR | Purpose | | -------------------------------------------------------------------------- | --------------------------------------------- | -| [openclaw/openclaw#94694](https://github.com/openclaw/openclaw/pull/94694) | Core snapshot provider proof | -| [openclaw/openclaw#94717](https://github.com/openclaw/openclaw/pull/94717) | Core `openclaw snapshot` CLI | -| [openclaw/openclaw#94799](https://github.com/openclaw/openclaw/pull/94799) | Named OpenClaw database targets | -| [openclaw/openclaw#94805](https://github.com/openclaw/openclaw/pull/94805) | Safe-sync artifact and restore proof | +| [openclaw/openclaw#94805](https://github.com/openclaw/openclaw/pull/94805) | Core snapshot command and safe-sync artifact | | [openclaw/openclaw#94967](https://github.com/openclaw/openclaw/pull/94967) | Snapshot stress harness for Phase 1 greenlight | -#### Phase 1 / PR 1: core snapshot provider proof +Earlier provider, CLI, and named-target slices were collapsed into +[openclaw/openclaw#94805](https://github.com/openclaw/openclaw/pull/94805) so +the core feature can be reviewed as one product boundary. -Add the shared SQLite snapshot provider and local artifact repository. +#### Phase 1 / PR A: core snapshot command and safe-sync artifact -Implementation: [openclaw/openclaw#94694](https://github.com/openclaw/openclaw/pull/94694). - -This PR should include: +Add the shared SQLite snapshot provider, local artifact repository, public +`openclaw snapshot` command surface, named database targets, and safe-sync +artifact proof. - `SqliteSnapshotProvider` contract - local snapshot repository @@ -454,50 +453,20 @@ This PR should include: - SQLite-safe snapshot creation for one database reference using shared core/package primitives where OpenClaw already owns the SQLite invariants - tests against a WAL-mode SQLite database - an internal restore in tests to prove the artifact is usable - -This PR does not need the full public restore CLI. It should prove that the provider can create and verify a correct SQLite snapshot artifact without publishing a premature provider API surface. - -#### Phase 1 / PR 2: core snapshot CLI - -Expose the user-facing core commands: +- user-facing core commands: ```text openclaw snapshot create openclaw snapshot verify openclaw snapshot restore ``` - -This PR should add target-directory safety checks, restore manifest validation, SQLite integrity checks after restore, and docs for the `openclaw snapshot` command surface. - -The CLI should accept an explicit database path for the proof path, but the intended product model is not "any random SQLite file forever." The command should be able to grow toward named OpenClaw database targets such as global state or a specific agent database once core exposes the eligible database registry cleanly. - -Implementation: [openclaw/openclaw#94717](https://github.com/openclaw/openclaw/pull/94717). - -#### Phase 1 / PR 3: named OpenClaw database targets - -Teach `openclaw snapshot` to address OpenClaw-owned databases by stable names -instead of only accepting arbitrary paths. - -This PR should include: - - `--target global` for `state/openclaw.sqlite` - `--agent ` for `agents//agent/openclaw-agent.sqlite` - manifest fields for database role, agent id, schema version, and source path - host-sync guidance that says live SQLite sidecars are ignored and completed artifacts are the sync input - -Implementation: [openclaw/openclaw#94799](https://github.com/openclaw/openclaw/pull/94799). - -#### Phase 1 / PR 4: safe-sync artifact and restore proof - -Prove that a completed snapshot artifact, not the live SQLite file family, is -the host-sync boundary. - -This PR should demonstrate that OpenClaw can create a snapshot from a named -database target, copy only the completed snapshot directory, restore from that -copied artifact into a fresh local SQLite path, and verify the restored -database. That proof is what makes the command a credible failover substrate -rather than only an archive utility. +- target-directory safety checks, restore manifest validation, SQLite integrity checks after restore, and docs for the `openclaw snapshot` command surface +- proof that OpenClaw can create a snapshot from a named database target, copy only the completed snapshot directory, restore from that copied artifact into a fresh local SQLite path, and verify the restored database This proof should also document the host contract: which OpenClaw command or API materializes the artifact, which files are safe to sync, which live SQLite sidecars should be ignored, which manifest fields the host can store without interpreting SQLite internals, and what must be restored before OpenClaw opens the database. @@ -508,7 +477,7 @@ criteria maintainers would use before starting Phase 2. Implementation: [openclaw/openclaw#94805](https://github.com/openclaw/openclaw/pull/94805). -#### Phase 1 / PR 5: snapshot stress harness +#### Phase 1 / PR B: snapshot stress harness Add an opt-in harness that measures the Phase 1 behavior before maintainers decide whether Phase 2 is worth building. @@ -525,7 +494,7 @@ snapshot frequency or greenlighting WAL bundles. Implementation: [openclaw/openclaw#94967](https://github.com/openclaw/openclaw/pull/94967). -#### Phase 2 / PR 6: simple WAL bundle proof +#### Phase 2 / PR C: simple WAL bundle proof Phase 2 is not automatically required by Phase 1. It needs a maintainer greenlight based on Phase 1 metrics. From 459fb59386958ba8b458ffd7c85a3d6c76574d2a Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Fri, 19 Jun 2026 12:01:16 -0700 Subject: [PATCH 19/24] Align snapshot RFC with memory search target --- rfcs/0013-cloud-serializable-sqlite-state.md | 28 +++++++++++++++++--- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 1f391f87..f5311c31 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -98,6 +98,7 @@ The first named target shape can be: ```text openclaw snapshot create --target global openclaw snapshot create --agent main +openclaw snapshot create --target memory-search --agent main openclaw snapshot verify openclaw snapshot restore --target ``` @@ -237,10 +238,23 @@ The unit of snapshotting is an existing OpenClaw-owned SQLite database. The prim - the global control-plane database at `state/openclaw.sqlite` - one per-agent data-plane database at `agents//agent/openclaw-agent.sqlite` +- a configured per-agent memory-search SQLite store, for example + `agents.defaults.memorySearch.store.path` with `{agentId}` substitution - any future dedicated owner store that has explicit ownership, schema, and lifecycle metadata This RFC does not rename or redesign those logical units; openclaw/openclaw#94646 makes them concrete enough for snapshot to target. The RFC defines snapshot behavior that can apply to each eligible database. +The memory-search target matters for hosted deployments such as Lobster because +the host may place that SQLite database outside the default OpenClaw state tree, +for example on local ephemeral disk at `/tmp/memory/{agentId}.sqlite`. The host +should not need to hard-code that path or copy the live SQLite file family. +Instead, OpenClaw should resolve the configured store path and materialize a +safe snapshot artifact through the same core command: + +```text +openclaw snapshot create --target memory-search --agent main +``` + The provider contract should therefore take a database reference rather than assume one hard-coded database path. Core can decide which SQLite databases are eligible and apply the same snapshot semantics to each eligible database. A snapshot must: @@ -462,6 +476,8 @@ openclaw snapshot restore ``` - `--target global` for `state/openclaw.sqlite` - `--agent ` for `agents//agent/openclaw-agent.sqlite` +- `--target memory-search --agent ` for a configured per-agent + memory-search SQLite store - manifest fields for database role, agent id, schema version, and source path - host-sync guidance that says live SQLite sidecars are ignored and completed artifacts are the sync input @@ -484,9 +500,11 @@ decide whether Phase 2 is worth building. This PR should prove that snapshot artifacts still restore cleanly while a writer is committing transaction batches against the source database. It should -cover both `--target global` and `--agent `, report snapshot/restore p50/p95 -timings, snapshot bytes, WAL bytes after the run, writer rows, and restore -verification counts. +cover `--target global` and `--agent ` at minimum, and should be able to add +dedicated targets such as `--target memory-search --agent ` as the core +target registry grows. It should report snapshot/restore p50/p95 timings, +snapshot bytes, WAL bytes after the run, writer rows, and restore verification +counts. This PR should not make stress mandatory in normal CI. It is a local/release validation tool for maintainers and operators who need evidence before tuning @@ -536,7 +554,7 @@ After the initial PRs, follow-up RFCs or implementation PRs can consider: This approach targets the reliability problem directly. It does not require OpenClaw to choose a second database backend before it has defined capture and restore semantics for the SQLite state it already owns. -The database-first work in openclaw/openclaw#94646 improves this RFC because it gives snapshot a concrete target model. Snapshot does not have to invent logical database units. It can operate over the already-established global control-plane database and per-agent data-plane databases, then extend to dedicated owner stores only when those stores have comparable ownership and lifecycle metadata. +The database-first work in openclaw/openclaw#94646 improves this RFC because it gives snapshot a concrete target model. Snapshot does not have to invent logical database units. It can operate over the already-established global control-plane database and per-agent data-plane databases, then extend to dedicated owner stores such as memory-search when those stores have comparable ownership and lifecycle metadata. Calling the command `snapshot` keeps the first deliverable concrete. It describes the artifact OpenClaw needs before higher-level reliability features can exist. It also avoids overpromising automatic failover before leases, promotion, and orchestration are designed. @@ -572,5 +590,7 @@ The proposal also keeps storage ownership decisions out of scope. OpenClaw alrea - Where should writer ownership metadata live before a database is opened? - Should restore verification run during startup, doctor, a managed-control-plane action, or all three? - What is the minimum host-facing API or command shape needed for Scout/Lobster-style platforms to request artifact materialization and pre-start hydration? +- Should every dedicated SQLite store use a named `--target`, or should some + stores only be reachable through a lower-level database registry/API? - Which artifacts should be included with database restore for support/debug exports versus canonical runtime recovery? - Should external tools such as Litestream or LiteFS be provider integrations, deployment recommendations, or out of scope for OpenClaw-owned code? From 2586c7aadf26c7bc5a17ff503889cf756baf0254 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Fri, 19 Jun 2026 13:26:17 -0700 Subject: [PATCH 20/24] Align snapshot RFC with database-first memory search --- rfcs/0013-cloud-serializable-sqlite-state.md | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index f5311c31..468456ec 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -238,18 +238,15 @@ The unit of snapshotting is an existing OpenClaw-owned SQLite database. The prim - the global control-plane database at `state/openclaw.sqlite` - one per-agent data-plane database at `agents//agent/openclaw-agent.sqlite` -- a configured per-agent memory-search SQLite store, for example - `agents.defaults.memorySearch.store.path` with `{agentId}` substitution +- memory-search tables owned by the per-agent database-first store - any future dedicated owner store that has explicit ownership, schema, and lifecycle metadata This RFC does not rename or redesign those logical units; openclaw/openclaw#94646 makes them concrete enough for snapshot to target. The RFC defines snapshot behavior that can apply to each eligible database. The memory-search target matters for hosted deployments such as Lobster because -the host may place that SQLite database outside the default OpenClaw state tree, -for example on local ephemeral disk at `/tmp/memory/{agentId}.sqlite`. The host -should not need to hard-code that path or copy the live SQLite file family. -Instead, OpenClaw should resolve the configured store path and materialize a -safe snapshot artifact through the same core command: +the host should not need to hard-code private SQLite paths or copy the live +SQLite file family. OpenClaw should resolve the database-first owner path and +materialize a safe snapshot artifact through the same core command: ```text openclaw snapshot create --target memory-search --agent main @@ -476,8 +473,8 @@ openclaw snapshot restore ``` - `--target global` for `state/openclaw.sqlite` - `--agent ` for `agents//agent/openclaw-agent.sqlite` -- `--target memory-search --agent ` for a configured per-agent - memory-search SQLite store +- `--target memory-search --agent ` for memory-search state in the + per-agent database-first store - manifest fields for database role, agent id, schema version, and source path - host-sync guidance that says live SQLite sidecars are ignored and completed artifacts are the sync input From 7f09e0a22adc9e18caf7f0b1a16113eae703f0f5 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Fri, 19 Jun 2026 13:46:05 -0700 Subject: [PATCH 21/24] Move memory search snapshot target to future design --- rfcs/0013-cloud-serializable-sqlite-state.md | 24 ++++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 468456ec..2453fa31 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -98,7 +98,6 @@ The first named target shape can be: ```text openclaw snapshot create --target global openclaw snapshot create --agent main -openclaw snapshot create --target memory-search --agent main openclaw snapshot verify openclaw snapshot restore --target ``` @@ -238,20 +237,23 @@ The unit of snapshotting is an existing OpenClaw-owned SQLite database. The prim - the global control-plane database at `state/openclaw.sqlite` - one per-agent data-plane database at `agents//agent/openclaw-agent.sqlite` -- memory-search tables owned by the per-agent database-first store - any future dedicated owner store that has explicit ownership, schema, and lifecycle metadata This RFC does not rename or redesign those logical units; openclaw/openclaw#94646 makes them concrete enough for snapshot to target. The RFC defines snapshot behavior that can apply to each eligible database. -The memory-search target matters for hosted deployments such as Lobster because -the host should not need to hard-code private SQLite paths or copy the live -SQLite file family. OpenClaw should resolve the database-first owner path and -materialize a safe snapshot artifact through the same core command: +Memory-search matters for hosted deployments such as Lobster because hosts +should not need to hard-code private SQLite paths or copy the live SQLite file +family. Today, memory-search state is owned by the per-agent database-first +store, so Phase 1 can protect it through the per-agent database target: ```text -openclaw snapshot create --target memory-search --agent main +openclaw snapshot create --agent main ``` +A future memory-search-only target would need a separate design because it +would either need a dedicated owner database or a true logical export. It should +not be presented as memory-only while snapshotting the full per-agent database. + The provider contract should therefore take a database reference rather than assume one hard-coded database path. Core can decide which SQLite databases are eligible and apply the same snapshot semantics to each eligible database. A snapshot must: @@ -473,8 +475,6 @@ openclaw snapshot restore ``` - `--target global` for `state/openclaw.sqlite` - `--agent ` for `agents//agent/openclaw-agent.sqlite` -- `--target memory-search --agent ` for memory-search state in the - per-agent database-first store - manifest fields for database role, agent id, schema version, and source path - host-sync guidance that says live SQLite sidecars are ignored and completed artifacts are the sync input @@ -498,8 +498,8 @@ decide whether Phase 2 is worth building. This PR should prove that snapshot artifacts still restore cleanly while a writer is committing transaction batches against the source database. It should cover `--target global` and `--agent ` at minimum, and should be able to add -dedicated targets such as `--target memory-search --agent ` as the core -target registry grows. It should report snapshot/restore p50/p95 timings, +future dedicated targets only after their owner database or logical export +contract is explicit. It should report snapshot/restore p50/p95 timings, snapshot bytes, WAL bytes after the run, writer rows, and restore verification counts. @@ -551,7 +551,7 @@ After the initial PRs, follow-up RFCs or implementation PRs can consider: This approach targets the reliability problem directly. It does not require OpenClaw to choose a second database backend before it has defined capture and restore semantics for the SQLite state it already owns. -The database-first work in openclaw/openclaw#94646 improves this RFC because it gives snapshot a concrete target model. Snapshot does not have to invent logical database units. It can operate over the already-established global control-plane database and per-agent data-plane databases, then extend to dedicated owner stores such as memory-search when those stores have comparable ownership and lifecycle metadata. +The database-first work in openclaw/openclaw#94646 improves this RFC because it gives snapshot a concrete target model. Snapshot does not have to invent logical database units. It can operate over the already-established global control-plane database and per-agent data-plane databases, then extend to dedicated owner stores only when those stores have comparable ownership and lifecycle metadata. Calling the command `snapshot` keeps the first deliverable concrete. It describes the artifact OpenClaw needs before higher-level reliability features can exist. It also avoids overpromising automatic failover before leases, promotion, and orchestration are designed. From 6e4615948d376581cbf0e74343b08e0a5f4c61b1 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Mon, 22 Jun 2026 10:42:42 -0700 Subject: [PATCH 22/24] Add Scout snapshot pilot checkpoint --- rfcs/0013-cloud-serializable-sqlite-state.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 2453fa31..1412eb4d 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -3,7 +3,7 @@ title: SQLite State Snapshot Command authors: - giodl created: 2026-06-18 -last_updated: 2026-06-19 +last_updated: 2026-06-22 status: draft issue: rfc_pr: https://github.com/openclaw/rfcs/pull/20 @@ -450,6 +450,22 @@ The current Phase 1 implementation stack is: | [openclaw/openclaw#94805](https://github.com/openclaw/openclaw/pull/94805) | Core snapshot command and safe-sync artifact | | [openclaw/openclaw#94967](https://github.com/openclaw/openclaw/pull/94967) | Snapshot stress harness for Phase 1 greenlight | +Before maintainers accept this RFC or land the core command, the same +artifact-boundary idea should be exercised in Microsoft Scout/Lobster for about +one week through the Lobster-owned snapshot plugin path. That pilot is not a +replacement for the core command proposal; it is the operational evidence +checkpoint for it. The pilot should answer: + +- whether hosts can reliably sync only completed `manifest.json` and + `database.sqlite` snapshot artifacts while ignoring live SQLite files, + temporary files, and sidecars +- how often Scout needs snapshots to meet its recovery expectations +- observed snapshot artifact size and creation time for real hosted use +- whether users or operators need a core `openclaw snapshot` command, a + host-owned plugin command, or both +- whether Phase 2 WAL bundles are justified by measured size, timing, or + frequency data + Earlier provider, CLI, and named-target slices were collapsed into [openclaw/openclaw#94805](https://github.com/openclaw/openclaw/pull/94805) so the core feature can be reviewed as one product boundary. From e26ef5c73814ab1fc327f878a4ae7e10af2d9a68 Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Mon, 6 Jul 2026 17:58:00 -0700 Subject: [PATCH 23/24] Frame snapshots as SQLite backup artifacts --- rfcs/0013-cloud-serializable-sqlite-state.md | 91 +++++++++----------- 1 file changed, 41 insertions(+), 50 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 1412eb4d..897922a8 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -1,23 +1,23 @@ --- -title: SQLite State Snapshot Command +title: SQLite Snapshot Backup Artifacts authors: - giodl created: 2026-06-18 -last_updated: 2026-06-22 +last_updated: 2026-07-06 status: draft issue: rfc_pr: https://github.com/openclaw/rfcs/pull/20 --- -# Proposal: SQLite State Snapshot Command +# Proposal: SQLite Snapshot Backup Artifacts ## Summary -Define a narrow core `openclaw snapshot` command that gives file-syncing hosts a safe file to sync for OpenClaw-owned SQLite state. +Define a narrow core `openclaw backup sqlite snapshot` command that gives file-syncing hosts a safe file to sync for OpenClaw-owned SQLite state. Hosted OpenClaw environments such as Scout/Lobster persist OpenClaw state by syncing files when they are saved. Live SQLite files are the wrong sync boundary: `state/openclaw.sqlite` or `agents//agent/openclaw-agent.sqlite` can be incomplete without their WAL, and `*.sqlite-wal`, `*.sqlite-shm`, and `*.sqlite-journal` are process-local sidecars rather than durable artifacts. -The `snapshot` command provides the missing translation step. It asks SQLite to materialize a clean database artifact, verifies it, writes a manifest, and publishes the completed artifact set into a sync-owned location. That completed artifact write, not arbitrary live database churn, is what the host should sync. +The SQLite snapshot backup command provides the missing translation step. It asks SQLite to materialize a clean database artifact, verifies it, writes a manifest, and publishes the completed artifact set into a sync-owned location. That completed artifact write, not arbitrary live database churn, is what the host should sync. SQLite remains the hot local runtime database. This RFC does not choose a replacement database, make cloud storage mandatory, or require managed failover. It defines a state-artifact boundary: OpenClaw core owns the SQLite-aware artifact operation, while the host owns upload, retention, routing, encryption, and restore timing. @@ -43,7 +43,7 @@ Deltas do not remove this requirement. Ryan's underlying concern is that whole-f ## Goals - Keep SQLite as the hot local runtime database for this proposal. -- Make snapshot behavior explicit through a narrow core `openclaw snapshot` command. +- Make snapshot behavior explicit through a narrow core `openclaw backup sqlite snapshot` command. - Produce a host-syncable SQLite artifact and manifest from live OpenClaw SQLite state. - Handle WAL state correctly without syncing live SQLite sidecars as durable artifacts. - Make restore and verification first-class behaviors, not incidental backup side effects. @@ -51,9 +51,9 @@ Deltas do not remove this requirement. Ryan's underlying concern is that whole-f - Define the state-artifact boundary that lets hosted platforms persist OpenClaw state without understanding OpenClaw's internal SQLite file layout. - Keep the command narrow: create, verify, and restore syncable SQLite artifacts. - Document host sync guidance: ignore live SQLite sidecars and sync completed snapshot artifacts/manifests instead. -- Add commands under `openclaw snapshot`. +- Add commands under `openclaw backup sqlite snapshot`. - Leave `openclaw backup` integration as a possible later follow-up, not part of the initial proof stack. -- Keep default local OpenClaw runtime behavior unchanged unless the snapshot command is invoked. +- Keep default local OpenClaw runtime behavior unchanged unless the SQLite snapshot backup command is invoked. - Avoid hot writes over network filesystems as a durability or concurrency strategy. - Treat WAL bundles as the first high-frequency optimization after full snapshot artifacts are correct. - Define lifecycle metadata needed to validate, order, restore, and audit snapshots. @@ -81,38 +81,29 @@ Deltas do not remove this requirement. Ryan's underlying concern is that whole-f ### Command shape -Add a core `openclaw snapshot` command that owns SQLite-safe snapshot and restore workflows for OpenClaw state. +Add a core `openclaw backup sqlite snapshot` command that owns SQLite-safe snapshot and restore workflows for OpenClaw state. -The command surface should stay direct: +The command surface should stay narrow inside the existing backup area: ```text -openclaw snapshot create -openclaw snapshot verify -openclaw snapshot restore -openclaw snapshot list -openclaw snapshot status +openclaw backup sqlite snapshot create +openclaw backup sqlite snapshot verify +openclaw backup sqlite snapshot restore +openclaw backup sqlite snapshot list ``` The first named target shape can be: ```text -openclaw snapshot create --target global -openclaw snapshot create --agent main -openclaw snapshot verify -openclaw snapshot restore --target +openclaw backup sqlite snapshot create --target global +openclaw backup sqlite snapshot create --agent main +openclaw backup sqlite snapshot verify +openclaw backup sqlite snapshot restore --target ``` This does not imply automatic scheduling, cloud storage, failover, or a new database abstraction. The core command produces and verifies the syncable artifact; the host decides where that artifact is stored and when it is restored. -Later, if maintainers want one user-facing home for backup and restore workflows, the same provider contract can be wired under the existing backup command surface: - -```text -openclaw backup snapshot -openclaw backup restore -openclaw backup status -``` - -That integration is intentionally not part of the initial implementation roadmap. The first proof should stay scoped to the `snapshot` command so it can demonstrate correctness without changing the existing backup command behavior. +This keeps the feature out of the top-level command namespace while still making the SQLite implication explicit. `openclaw backup create` and `openclaw backup verify` keep their existing archive behavior; `backup sqlite snapshot` is the per-database SQLite artifact path. ### Responsibility split @@ -129,9 +120,9 @@ OpenClaw should provide: - lifecycle metadata shape - safety rules such as no hot writes over network filesystems -The `snapshot` command should own the operator workflow around the core primitive: +The `backup sqlite snapshot` command should own the operator workflow around the core primitive: -- snapshot command UX +- SQLite snapshot backup command UX - local snapshot artifact creation and publication into a sync-owned artifact directory - snapshot manifest creation and verification - restore workflow orchestration @@ -195,8 +186,8 @@ flowchart LR Metadata[lifecycle metadata] end - subgraph Command[Core snapshot command] - Cmd[openclaw snapshot] + subgraph Command[Core SQLite snapshot backup command] + Cmd[openclaw backup sqlite snapshot] Manifest[snapshot manifest] LocalRepo[(local snapshot repo)] end @@ -225,7 +216,7 @@ flowchart LR Failover -. later .-> Restore ``` -The diagram is a responsibility split, not a default managed-hosting requirement. Default local OpenClaw can run with only the runtime box. Operators and hosts use the snapshot command when they need verified snapshot and restore workflows. +The diagram is a responsibility split, not a default managed-hosting requirement. Default local OpenClaw can run with only the runtime box. Operators and hosts use the SQLite snapshot backup command when they need verified snapshot and restore workflows. For hosted OpenClaw, the same split becomes the host integration contract. The host can ask OpenClaw to materialize a clean artifact before upload and can hydrate local disk from a verified artifact before OpenClaw opens SQLite. The host does not need to treat live SQLite sidecars as durable sync inputs. @@ -247,7 +238,7 @@ family. Today, memory-search state is owned by the per-agent database-first store, so Phase 1 can protect it through the per-agent database target: ```text -openclaw snapshot create --agent main +openclaw backup sqlite snapshot create --agent main ``` A future memory-search-only target would need a separate design because it @@ -327,7 +318,7 @@ Restore artifacts should be consumed before OpenClaw opens the target database f ### Restore verification -Restore is a required behavior for `openclaw snapshot`, not an incidental backup side effect. +Restore is a required behavior for `openclaw backup sqlite snapshot`, not an incidental backup side effect. A restore operation must: @@ -388,13 +379,13 @@ At minimum, snapshot metadata should include: - restore source and restore point when hydrated - current writer owner or lease holder, when leases are enabled -The exact storage location for this metadata is implementation-defined, but `openclaw snapshot` must be able to read enough metadata to verify and restore a snapshot without opening a possibly unsafe runtime database first. +The exact storage location for this metadata is implementation-defined, but `openclaw backup sqlite snapshot` must be able to read enough metadata to verify and restore a snapshot without opening a possibly unsafe runtime database first. ### Provider shape The implementation can start as a SQLite-specific snapshot provider rather than a database abstraction layer. -The provider contract should be reusable by any OpenClaw feature that needs to capture or restore an OpenClaw-owned SQLite database. `openclaw snapshot` is the first proposed CLI surface, but the contract should not depend on command-only state. +The provider contract should be reusable by any OpenClaw feature that needs to capture or restore an OpenClaw-owned SQLite database. `openclaw backup sqlite snapshot` is the first proposed CLI surface, but the contract should not depend on command-only state. A minimal shape is: @@ -420,13 +411,13 @@ type RemoteSnapshotProvider = SqliteSnapshotProvider & { This keeps SQLite runtime access local while making state artifacts portable. A local snapshot provider can be the reference implementation. Cloud/object-store providers can come later without changing the default local runtime. -If the design proves broadly useful, the same contract can support backup restore, startup hydration, or state migration workflows without changing the narrow `openclaw snapshot` command. +If the design proves broadly useful, the same contract can support backup restore, startup hydration, or state migration workflows without changing the narrow `openclaw backup sqlite snapshot` command. ### Implementation roadmap The implementation should be split into two phases. -Phase 1 is the committed snapshot command work. It proves full verified +Phase 1 is the committed SQLite snapshot backup command work. It proves full verified snapshots, named OpenClaw database targets, and fresh-state restore. It should also collect the metrics that decide whether Phase 2 is worth doing: @@ -447,7 +438,7 @@ The current Phase 1 implementation stack is: | PR | Purpose | | -------------------------------------------------------------------------- | --------------------------------------------- | -| [openclaw/openclaw#94805](https://github.com/openclaw/openclaw/pull/94805) | Core snapshot command and safe-sync artifact | +| [openclaw/openclaw#94805](https://github.com/openclaw/openclaw/pull/94805) | Core SQLite snapshot backup command and safe-sync artifact | | [openclaw/openclaw#94967](https://github.com/openclaw/openclaw/pull/94967) | Snapshot stress harness for Phase 1 greenlight | Before maintainers accept this RFC or land the core command, the same @@ -461,7 +452,7 @@ checkpoint for it. The pilot should answer: temporary files, and sidecars - how often Scout needs snapshots to meet its recovery expectations - observed snapshot artifact size and creation time for real hosted use -- whether users or operators need a core `openclaw snapshot` command, a +- whether users or operators need a core `openclaw backup sqlite snapshot` command, a host-owned plugin command, or both - whether Phase 2 WAL bundles are justified by measured size, timing, or frequency data @@ -470,10 +461,10 @@ Earlier provider, CLI, and named-target slices were collapsed into [openclaw/openclaw#94805](https://github.com/openclaw/openclaw/pull/94805) so the core feature can be reviewed as one product boundary. -#### Phase 1 / PR A: core snapshot command and safe-sync artifact +#### Phase 1 / PR A: core SQLite snapshot backup command and safe-sync artifact Add the shared SQLite snapshot provider, local artifact repository, public -`openclaw snapshot` command surface, named database targets, and safe-sync +`openclaw backup sqlite snapshot` command surface, named database targets, and safe-sync artifact proof. - `SqliteSnapshotProvider` contract @@ -485,16 +476,16 @@ artifact proof. - user-facing core commands: ```text -openclaw snapshot create -openclaw snapshot verify -openclaw snapshot restore +openclaw backup sqlite snapshot create +openclaw backup sqlite snapshot verify +openclaw backup sqlite snapshot restore ``` - `--target global` for `state/openclaw.sqlite` - `--agent ` for `agents//agent/openclaw-agent.sqlite` - manifest fields for database role, agent id, schema version, and source path - host-sync guidance that says live SQLite sidecars are ignored and completed artifacts are the sync input -- target-directory safety checks, restore manifest validation, SQLite integrity checks after restore, and docs for the `openclaw snapshot` command surface +- target-directory safety checks, restore manifest validation, SQLite integrity checks after restore, and docs for the `openclaw backup sqlite snapshot` command surface - proof that OpenClaw can create a snapshot from a named database target, copy only the completed snapshot directory, restore from that copied artifact into a fresh local SQLite path, and verify the restored database This proof should also document the host contract: which OpenClaw command or API materializes the artifact, which files are safe to sync, which live SQLite sidecars should be ignored, which manifest fields the host can store without interpreting SQLite internals, and what must be restored before OpenClaw opens the database. @@ -556,7 +547,7 @@ full snapshot generation. Conservative pruning policy can remain later work. After the initial PRs, follow-up RFCs or implementation PRs can consider: -- `openclaw backup` integration +- broader `openclaw backup` archive integration using the same provider contract - object/blob storage providers - retention and scheduling - WAL bundle retention and pruning policy @@ -569,9 +560,9 @@ This approach targets the reliability problem directly. It does not require Open The database-first work in openclaw/openclaw#94646 improves this RFC because it gives snapshot a concrete target model. Snapshot does not have to invent logical database units. It can operate over the already-established global control-plane database and per-agent data-plane databases, then extend to dedicated owner stores only when those stores have comparable ownership and lifecycle metadata. -Calling the command `snapshot` keeps the first deliverable concrete. It describes the artifact OpenClaw needs before higher-level reliability features can exist. It also avoids overpromising automatic failover before leases, promotion, and orchestration are designed. +Calling the subcommand `snapshot` under `backup sqlite` keeps the first deliverable concrete without giving it top-level command real estate. The path says what it is: a backup-area SQLite artifact, not a broad backup archive and not automatic failover. -Keeping the first implementation stack under `openclaw snapshot` keeps the proof small and command-scoped. Existing `openclaw backup create` and `openclaw backup verify` behavior can remain unchanged while the snapshot provider proves the harder SQLite correctness and restore semantics. +Keeping the first implementation stack under `openclaw backup sqlite snapshot` keeps the proof small and command-scoped. Existing `openclaw backup create` and `openclaw backup verify` behavior can remain unchanged while the snapshot provider proves the harder SQLite correctness and restore semantics. Treating remote storage as artifact storage avoids the common failure mode where object storage or network filesystems are used as if they were local disk. SQLite remains local and authoritative while running. Reliability comes from verified snapshots, manifests, restore procedures, and later WAL bundles. From 864d7dee6ba6b9cf9e2f902f174408e5fbc570db Mon Sep 17 00:00:00 2001 From: Gio Della-Libera Date: Tue, 21 Jul 2026 07:24:35 -0700 Subject: [PATCH 24/24] Update SQLite snapshot RFC to landed implementation --- rfcs/0013-cloud-serializable-sqlite-state.md | 652 +++++-------------- 1 file changed, 171 insertions(+), 481 deletions(-) diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index 897922a8..efbd97ec 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -3,9 +3,9 @@ title: SQLite Snapshot Backup Artifacts authors: - giodl created: 2026-06-18 -last_updated: 2026-07-06 -status: draft -issue: +last_updated: 2026-07-21 +status: completed +issue: https://github.com/openclaw/openclaw/pull/105718 rfc_pr: https://github.com/openclaw/rfcs/pull/20 --- @@ -13,160 +13,119 @@ rfc_pr: https://github.com/openclaw/rfcs/pull/20 ## Summary -Define a narrow core `openclaw backup sqlite snapshot` command that gives file-syncing hosts a safe file to sync for OpenClaw-owned SQLite state. +Define a core `openclaw backup sqlite` command that gives operators and file-syncing hosts a SQLite-safe artifact for one OpenClaw-owned database. -Hosted OpenClaw environments such as Scout/Lobster persist OpenClaw state by syncing files when they are saved. Live SQLite files are the wrong sync boundary: `state/openclaw.sqlite` or `agents//agent/openclaw-agent.sqlite` can be incomplete without their WAL, and `*.sqlite-wal`, `*.sqlite-shm`, and `*.sqlite-journal` are process-local sidecars rather than durable artifacts. +OpenClaw keeps SQLite as the hot local runtime database. The snapshot command is the translation layer between live SQLite state and durable host storage: it materializes a compact `database.sqlite`, writes a strict `manifest.json`, verifies both, and publishes only the completed snapshot directory. -The SQLite snapshot backup command provides the missing translation step. It asks SQLite to materialize a clean database artifact, verifies it, writes a manifest, and publishes the completed artifact set into a sync-owned location. That completed artifact write, not arbitrary live database churn, is what the host should sync. +This RFC does not choose a replacement database, require cloud storage, or add managed failover. It defines the local artifact contract that hosts can safely sync, retain, verify, and restore before OpenClaw opens SQLite again. -SQLite remains the hot local runtime database. This RFC does not choose a replacement database, make cloud storage mandatory, or require managed failover. It defines a state-artifact boundary: OpenClaw core owns the SQLite-aware artifact operation, while the host owns upload, retention, routing, encryption, and restore timing. +Implementation landed in [openclaw/openclaw#105718](https://github.com/openclaw/openclaw/pull/105718), commit [2f25e9cba384acfc06cdf83640f236fdb7c1af33](https://github.com/openclaw/openclaw/commit/2f25e9cba384acfc06cdf83640f236fdb7c1af33). It superseded the original prototype in [openclaw/openclaw#94805](https://github.com/openclaw/openclaw/pull/94805) while preserving the SQLite-safe artifact direction. ## Motivation -OpenClaw is moving runtime state into SQLite-backed stores. The database-first SQLite alignment landed in openclaw/openclaw#94646 makes that shape more explicit: `state/openclaw.sqlite` is the global control-plane database, while `agents//agent/openclaw-agent.sqlite` is the per-agent data-plane database. That is a good local runtime shape, but hosted reliability depends on more than having files on disk. +OpenClaw stores important runtime state in SQLite-backed files. That is the right local runtime shape, but raw file syncing is the wrong durability boundary. -The host-sync problem is concrete. A host can watch files and sync them as they are saved, but it should not sync OpenClaw's live SQLite working set as the durability boundary. - -The unsafe sync inputs are specific: +The unsafe sync inputs are concrete: - `state/openclaw.sqlite` can be stale by itself when committed writes still live in `state/openclaw.sqlite-wal`. - `agents//agent/openclaw-agent.sqlite` can be stale by itself when committed writes still live in `agents//agent/openclaw-agent.sqlite-wal`. - `*.sqlite-wal`, `*.sqlite-shm`, and `*.sqlite-journal` are process-local SQLite sidecars, not durable host-sync artifacts. - Copying a directory while OpenClaw is writing can split one logical database state across files captured at different moments. -The syncable file is created deliberately. OpenClaw should open the source database through SQLite, wait for any required busy timeout or writer barrier, materialize a clean database copy using SQLite online backup, `VACUUM INTO`, or an equivalent SQLite-aware checkpoint/copy mechanism, run integrity verification, and write a manifest that records the database identity, schema version, source path, hash, and restore metadata. The host syncs that artifact set, not the live SQLite files. - -For hosts that sync on filesystem changes, the sync trigger should be the completed artifact write, not arbitrary writes inside the live OpenClaw state directory. The artifact directory should contain completed database artifacts and manifests only. Live `*.sqlite-wal`, `*.sqlite-shm`, and `*.sqlite-journal` files should not be created there, because a sync agent can observe and upload them as soon as they appear. +The syncable file must be created deliberately. OpenClaw should open the source database through SQLite, materialize a clean copy using a SQLite-aware mechanism, verify it, and publish a completed artifact set. A host should sync that artifact set, not arbitrary live database churn. -Deltas do not remove this requirement. Ryan's underlying concern is that whole-file copies do not scale, but unmanaged file deltas over live SQLite make the correctness problem worse: a file-sync tool can observe DB pages, WAL frames, and sidecars at different moments without knowing SQLite ordering or checkpoint state. If high-frequency sync is needed, the simple follow-up should be ordered WAL-bundle artifacts anchored to a verified full snapshot. +The first landed implementation uses SQLite `VACUUM INTO` to capture committed WAL state into a compact database. That is different from copying a hot `.sqlite` file, and it is different from routinely vacuuming the runtime database. The operation happens only when creating a snapshot artifact. ## Goals -- Keep SQLite as the hot local runtime database for this proposal. -- Make snapshot behavior explicit through a narrow core `openclaw backup sqlite snapshot` command. -- Produce a host-syncable SQLite artifact and manifest from live OpenClaw SQLite state. -- Handle WAL state correctly without syncing live SQLite sidecars as durable artifacts. -- Make restore and verification first-class behaviors, not incidental backup side effects. -- Define a reusable SQLite snapshot provider contract for OpenClaw-owned SQLite databases. -- Define the state-artifact boundary that lets hosted platforms persist OpenClaw state without understanding OpenClaw's internal SQLite file layout. -- Keep the command narrow: create, verify, and restore syncable SQLite artifacts. -- Document host sync guidance: ignore live SQLite sidecars and sync completed snapshot artifacts/manifests instead. -- Add commands under `openclaw backup sqlite snapshot`. -- Leave `openclaw backup` integration as a possible later follow-up, not part of the initial proof stack. -- Keep default local OpenClaw runtime behavior unchanged unless the SQLite snapshot backup command is invoked. -- Avoid hot writes over network filesystems as a durability or concurrency strategy. -- Treat WAL bundles as the first high-frequency optimization after full snapshot artifacts are correct. -- Define lifecycle metadata needed to validate, order, restore, and audit snapshots. -- Leave cloud artifact storage, retention, scheduling, and failover orchestration to optional providers or later RFCs. -- Let hosts such as Scout/Lobster own durable destination policy while OpenClaw owns the correctness of the local SQLite artifact operation. -- Build on the existing database-first units: global control-plane SQLite, per-agent data-plane SQLite, and any dedicated owner store. -- Leave room for core to adopt the same primitives later if snapshot and restore become required OpenClaw behavior. +- Keep SQLite as the hot local runtime database. +- Provide a narrow core command for one-database SQLite snapshot artifacts. +- Support the shared OpenClaw state database and per-agent databases. +- Produce a host-syncable `database.sqlite` plus `manifest.json`. +- Handle WAL state correctly without syncing live SQLite sidecars. +- Verify snapshot shape, content hash, SQLite integrity, schema, role, owner, and canonical indexes. +- Restore only to a fresh target path, never by replacing a live database in place. +- Keep upload, scheduling, retention, failover, and restore-on-boot outside the Phase 1 command. ## Non-Goals -- This RFC does not choose PostgreSQL, libSQL, remote SQLite, object storage, or any other backend product. +- This RFC does not choose PostgreSQL, libSQL, remote SQLite, object storage, or another backend. - This RFC does not define a general database abstraction layer. -- This RFC does not require OpenClaw to own the hosting platform's upload, retention, tenant routing, encryption, or object-storage policy. -- This RFC does not make snapshots, cloud storage, or managed failover mandatory for local, self-hosted, or development OpenClaw installs. -- This RFC does not require cloud providers or host integrations to live in core. -- This RFC does not require object-store credentials, a lease service, or a managed-service control plane in the default runtime. -- This RFC does not replace the session/transcript migration plan tracked by openclaw/openclaw#88838. -- This RFC does not define tenant isolation, row-level authorization, or a multi-tenant schema model. -- This RFC does not define FTS/vector search portability. -- This RFC does not require real-time multi-writer SQLite over shared storage. -- This RFC does not define the final managed failover control plane. -- This RFC does not change the existing `openclaw backup create` or `openclaw backup verify` behavior. +- This RFC does not make cloud storage or managed failover mandatory. +- This RFC does not require OpenClaw to own upload, tenant routing, retention, or encryption policy. +- This RFC does not require hot writes over a network filesystem. +- This RFC does not define WAL bundles, leases, promotion, fencing, or standby orchestration. +- This RFC does not change `openclaw backup create` archive behavior. ## Proposal -### Command shape - -Add a core `openclaw backup sqlite snapshot` command that owns SQLite-safe snapshot and restore workflows for OpenClaw state. - -The command surface should stay narrow inside the existing backup area: - -```text -openclaw backup sqlite snapshot create -openclaw backup sqlite snapshot verify -openclaw backup sqlite snapshot restore -openclaw backup sqlite snapshot list -``` +### Command Shape -The first named target shape can be: +Add SQLite snapshot operations under the existing backup command: ```text -openclaw backup sqlite snapshot create --target global -openclaw backup sqlite snapshot create --agent main -openclaw backup sqlite snapshot verify -openclaw backup sqlite snapshot restore --target +openclaw backup sqlite create --global --repository +openclaw backup sqlite create --agent --repository +openclaw backup sqlite list --repository +openclaw backup sqlite verify +openclaw backup sqlite verify --scratch +openclaw backup sqlite restore --target ``` -This does not imply automatic scheduling, cloud storage, failover, or a new database abstraction. The core command produces and verifies the syncable artifact; the host decides where that artifact is stored and when it is restored. - -This keeps the feature out of the top-level command namespace while still making the SQLite implication explicit. `openclaw backup create` and `openclaw backup verify` keep their existing archive behavior; `backup sqlite snapshot` is the per-database SQLite artifact path. - -### Responsibility split +The command is deliberately under `backup sqlite`, not a top-level `snapshot` command. `openclaw backup create` and `openclaw backup verify` remain broad archive commands. `openclaw backup sqlite` is the per-database SQLite artifact path. -OpenClaw core must provide the SQLite-safe artifact contract because the host cannot safely infer it from the filesystem alone. +### Responsibility Split -OpenClaw should provide: +OpenClaw core owns the SQLite-aware artifact contract because the host cannot infer it safely from filesystem events. -- eligible database discovery or registry for OpenClaw-owned SQLite databases -- consistent SQLite checkpoint creation -- materialization of a host-syncable database artifact from a live local SQLite database -- guidance or generated ignore rules for host sync: ignore `*.sqlite-wal`, `*.sqlite-shm`, and `*.sqlite-journal`; do not use the live SQLite runtime directory as the sync-owned artifact directory -- restore or hydrate before opening runtime state -- restored database verification -- lifecycle metadata shape -- safety rules such as no hot writes over network filesystems +OpenClaw owns: -The `backup sqlite snapshot` command should own the operator workflow around the core primitive: +- eligible database roles +- SQLite-safe snapshot creation +- snapshot repository validation +- staging and atomic publication +- manifest creation and strict parsing +- content hashing +- verification against OpenClaw database role and owner invariants +- restore to a fresh local SQLite file +- rejection of stale sidecars, unsafe paths, symlinks, hardlinks, and race-prone staging paths -- SQLite snapshot backup command UX -- local snapshot artifact creation and publication into a sync-owned artifact directory -- snapshot manifest creation and verification -- restore workflow orchestration -- provider hooks for future storage backends, if accepted +The host or operator owns: -Optional future integrations and providers can own: +- upload and sync destination +- retention policy +- encryption and access control outside the local repository +- restore timing +- startup orchestration +- failover policy -- integration with `openclaw backup` -- local snapshot repositories -- S3-compatible artifact storage -- Azure Blob or other cloud artifact storage -- ODSP, blob, git, durable-volume, or other host-specific persistence adapters -- retention policy and upload scheduling -- writer lease coordination -- warm standby or managed failover orchestration -- integration with external tools such as Litestream or LiteFS, if later accepted - -In other words, OpenClaw should own this translation: +The contract is: ```text -live OpenClaw SQLite database -> verified snapshot artifact + manifest +live OpenClaw SQLite database -> verified snapshot directory +verified snapshot directory -> durable destination and pre-start restore ``` -The hosting platform should own this policy: +### Host Flow + +Before upload or sync: ```text -verified snapshot artifact + manifest -> durable destination and restore timing +1. host or operator runs openclaw backup sqlite create +2. OpenClaw validates the source database role and repository +3. OpenClaw creates and verifies a compact database artifact in private staging +4. OpenClaw writes manifest.json and database.sqlite +5. OpenClaw publishes the completed snapshot directory +6. host syncs only completed snapshot directories ``` -That split keeps SQLite correctness in the codebase that owns the schema and file layout, while keeping cloud credentials, tenant routing, retention, and platform lifecycle outside the default OpenClaw runtime. - -The host-facing flow should be explicit: +Before startup after replacement: ```text -before upload/sync: - 1. host or operator asks OpenClaw to snapshot an eligible database - 2. OpenClaw creates a clean database artifact in a staging location - 3. OpenClaw verifies the artifact and writes the final artifact + manifest into the sync-owned artifact directory - 4. host sync is triggered by the completed artifact/manifest write - -before startup after replacement: - 1. host downloads/selects a verified artifact set - 2. OpenClaw verifies and hydrates local database files - 3. OpenClaw opens SQLite for runtime writes only after hydration succeeds +1. host selects or downloads a completed snapshot directory +2. OpenClaw verifies the snapshot from a content-pinned private copy +3. OpenClaw restores to a fresh local SQLite target +4. OpenClaw opens SQLite for runtime writes only after restore succeeds ``` ### Architecture @@ -179,422 +138,153 @@ flowchart LR App -->|local reads/writes| DB end - subgraph Core[SQLite-safe core primitives] - Checkpoint[checkpoint] + subgraph Core[OpenClaw backup sqlite] + Create[create] Verify[verify] - Restore[restore / hydrate] - Metadata[lifecycle metadata] + Restore[restore] + Manifest[manifest.json] + Artifact[database.sqlite] end - subgraph Command[Core SQLite snapshot backup command] - Cmd[openclaw backup sqlite snapshot] - Manifest[snapshot manifest] - LocalRepo[(local snapshot repo)] + subgraph Repo[Local snapshot repository] + SnapshotDir[completed snapshot directory] end - subgraph Providers[Optional providers] - Object[(object/blob storage)] - Host[(host persistence layer)] - Retention[retention schedule] - Failover[standby / failover] + subgraph Host[Host/operator policy] + Sync[file/object sync] + Retain[retention] + Startup[pre-start hydration] end - DB --> Checkpoint - Checkpoint --> Metadata - Checkpoint --> Manifest - Cmd --> Checkpoint - Cmd --> Restore - Manifest --> LocalRepo - LocalRepo --> Restore - LocalRepo --> Object - LocalRepo --> Host - Object --> Restore - Host --> Restore - Restore --> Verify - Verify --> DB - Retention -. optional .-> LocalRepo - Failover -. later .-> Restore + DB --> Create + Create --> Verify + Create --> Manifest + Create --> Artifact + Manifest --> SnapshotDir + Artifact --> SnapshotDir + SnapshotDir --> Sync + Sync --> Retain + Sync --> Startup + Startup --> Verify + Verify --> Restore + Restore --> DB ``` -The diagram is a responsibility split, not a default managed-hosting requirement. Default local OpenClaw can run with only the runtime box. Operators and hosts use the SQLite snapshot backup command when they need verified snapshot and restore workflows. +The diagram is a responsibility split. The default local runtime can ignore the host box entirely. Hosted deployments can use the snapshot directory as the sync boundary without copying live SQLite sidecars. -For hosted OpenClaw, the same split becomes the host integration contract. The host can ask OpenClaw to materialize a clean artifact before upload and can hydrate local disk from a verified artifact before OpenClaw opens SQLite. The host does not need to treat live SQLite sidecars as durable sync inputs. +### Snapshot Semantics -### Snapshot semantics +The unit of snapshotting is one existing OpenClaw-owned SQLite database: -An OpenClaw-owned SQLite database is snapshot-safe when it can be captured, verified, restored, and resumed on another host or directory without relying on a live shared filesystem. +- the shared OpenClaw state database +- one per-agent database -The unit of snapshotting is an existing OpenClaw-owned SQLite database. The primary units are: +The landed implementation also has a generic role in the manifest format, but the public named sources are intentionally strict. A future dedicated owner store can become eligible only after it has explicit role, owner, schema, and lifecycle invariants. -- the global control-plane database at `state/openclaw.sqlite` -- one per-agent data-plane database at `agents//agent/openclaw-agent.sqlite` -- any future dedicated owner store that has explicit ownership, schema, and lifecycle metadata - -This RFC does not rename or redesign those logical units; openclaw/openclaw#94646 makes them concrete enough for snapshot to target. The RFC defines snapshot behavior that can apply to each eligible database. - -Memory-search matters for hosted deployments such as Lobster because hosts -should not need to hard-code private SQLite paths or copy the live SQLite file -family. Today, memory-search state is owned by the per-agent database-first -store, so Phase 1 can protect it through the per-agent database target: +A snapshot directory contains exactly: ```text -openclaw backup sqlite snapshot create --agent main +manifest.json +database.sqlite ``` -A future memory-search-only target would need a separate design because it -would either need a dedicated owner database or a true logical export. It should -not be presented as memory-only while snapshotting the full per-agent database. - -The provider contract should therefore take a database reference rather than assume one hard-coded database path. Core can decide which SQLite databases are eligible and apply the same snapshot semantics to each eligible database. - -A snapshot must: - -- handle `.sqlite`, `-wal`, and `-shm` state correctly -- avoid half-copied database state -- record the schema version and database identity -- record the checkpoint cursor or equivalent replay position when available -- produce enough metadata to verify restore integrity -- be restorable before OpenClaw opens the database for runtime writes - -The implementation may use SQLite online backup APIs, `VACUUM INTO`, WAL checkpoints, page-level capture, or another implementation-specific mechanism. The observable contract is a consistent restore point. - -For the initial proof, `VACUUM INTO` is acceptable because it asks SQLite to produce a compact, consistent destination database. That is different from asking the host to copy a hot `.sqlite` file. It is also different from routinely vacuuming OpenClaw's runtime databases; the operation happens only when creating a snapshot artifact. - -### Snapshot artifacts - -Snapshot storage should store durable artifacts, not a live database file used directly by the runtime. - -The artifact model should support: - -- compact snapshot artifacts -- ordered manifests -- content hashes or equivalent integrity checks -- optional WAL bundles after the first milestones -- resumable upload and download when a remote provider is configured -- restore from the latest valid snapshot plus any required ordered WAL bundles - -The follow-up high-frequency mechanism should start with WAL bundles, not a broad delta abstraction. - -WAL bundles must be anchored to a verified snapshot generation. The system should not treat arbitrary file-sync deltas from the live runtime directory as a restore stream. A valid bundle design needs ordering, a base snapshot cursor, integrity checks, and replay rules that SQLite/OpenClaw can verify before opening the restored database. - -### WAL bundle model - -WAL bundles are the first proposed answer to Ryan's scaling concern, but only -after OpenClaw has a verified full snapshot contract. A WAL bundle is not the -live `*.sqlite-wal` file. It is an OpenClaw-authored artifact cut from a known -SQLite state range, written to staging, verified, and then published into the -sync-owned artifact directory. - -The first WAL bundle design should keep these invariants: - -- every bundle names its base snapshot generation or previous bundle cursor -- every bundle has a monotonically ordered sequence number -- every bundle records the source database id, schema version, and page size or - equivalent compatibility data -- every bundle records content hashes before upload and after download -- restore applies bundles only after verifying the base snapshot -- restore rejects gaps, forks, duplicate sequence numbers, incompatible schema - versions, and failed integrity checks -- OpenClaw verifies the final restored database before runtime opens SQLite - -The milestone should be: full snapshot first, verified restore second, then -simple ordered WAL bundles from a snapshot cursor. File-sync deltas over the live runtime -directory remain out of scope because they do not carry SQLite ordering, -checkpoint, or replay semantics. - -Artifacts should be suitable for host persistence. A hosting platform should be able to upload, retain, copy, and later download the artifact set without preserving process-local SQLite sidecars or relying on a mounted shared filesystem. - -The sync-owned artifact directory should not be the live SQLite runtime directory. If the host syncs on file save, OpenClaw should write completed artifacts into a separate artifact location after verification. That keeps the host from observing transient SQLite sidecars or partially materialized runtime state. - -Artifacts should be created at explicit lifecycle moments: - -- on demand, when an operator or host requests a snapshot -- before a managed host uploads/syncs OpenClaw state -- before container shutdown or ownership release when the platform can coordinate that moment -- before migration or other state-changing maintenance when a rollback point is required -- periodically, when a provider adds scheduling or retention policy - -Restore artifacts should be consumed before OpenClaw opens the target database for runtime writes. A host that downloads artifacts after OpenClaw has already opened SQLite risks racing the runtime and should be treated as outside this contract. - -### Restore verification - -Restore is a required behavior for `openclaw backup sqlite snapshot`, not an incidental backup side effect. - -A restore operation must: - -- locate the selected snapshot and required artifacts -- verify artifact ordering and integrity -- hydrate local database files before runtime opens them -- run SQLite integrity checks or equivalent validation -- confirm the restored schema version is supported -- record the restore point OpenClaw is resuming from - -The first implementation milestone should prove that OpenClaw can boot from restored state on a fresh directory, host, or container. - -### Failover path - -The command is not required to implement automatic failover in the first milestone, but it should be designed as the foundation for failover. - -Failover becomes possible when OpenClaw has: - -1. a recent verified snapshot or restore point -2. a way to hydrate local disk before startup -3. a way to confirm schema and integrity before runtime writes -4. a clear owner for the database after restore -5. optional WAL bundles or upload scheduling to reduce the data-loss window - -A later RFC can define leases, promotion, fencing, standby replicas, and managed orchestration. This RFC provides the snapshot and restore substrate those systems need. +The manifest records schema version, snapshot id, creation time, database role, database owner fields when applicable, database basename, SQLite `user_version`, artifact path, SHA-256, and size. -### Writer ownership +Snapshot creation must: -OpenClaw must not treat a network filesystem as the concurrency model for hot SQLite writes. +- validate the live source database before reading it +- use SQLite to produce a compact artifact that includes committed WAL state +- verify the generated database +- hash the generated artifact +- publish only a completed directory +- refuse unsafe repositories and publication races +- avoid producing live SQLite sidecars in the sync-owned artifact directory -Each snapshot-managed SQLite database should have explicit writer ownership when the deployment allows failover or multiple possible hosts. A managed deployment can move ownership, but only through a controlled sequence: +Global snapshots may sanitize transient runtime rows before publication when those rows are not durable state and should not be retained in deleted pages. -1. acquire ownership or a writer lease for the database, if leases are enabled -2. hydrate local disk from a verified restore point when needed -3. open and write SQLite locally -4. periodically create and publish snapshot artifacts -5. release ownership with a final verified snapshot when supported -6. allow another host to restore from the latest verified durable point +### Verification And Restore -Concurrent readers and replicas can be designed later, but the write path must have one clear owner at a time unless a future RFC defines a stronger multi-writer mechanism. +Verification is first-class. It must reject malformed, tampered, incomplete, or unsafe snapshots before SQLite opens untrusted bytes. -### Lifecycle metadata +The landed command verifies: -Each snapshot needs metadata sufficient to reason about restore, replay, and integrity. - -At minimum, snapshot metadata should include: - -- database id -- database kind or owner -- database role, such as global control-plane or per-agent data-plane -- owning agent id when the snapshot is for a per-agent database +- strict manifest shape +- artifact size and SHA-256 +- SQLite integrity +- foreign keys - schema version -- snapshot generation -- checkpoint or WAL cursor when available -- artifact manifest id -- integrity hash or verification record -- snapshot creation time -- restore source and restore point when hydrated -- current writer owner or lease holder, when leases are enabled - -The exact storage location for this metadata is implementation-defined, but `openclaw backup sqlite snapshot` must be able to read enough metadata to verify and restore a snapshot without opening a possibly unsafe runtime database first. - -### Provider shape - -The implementation can start as a SQLite-specific snapshot provider rather than a database abstraction layer. - -The provider contract should be reusable by any OpenClaw feature that needs to capture or restore an OpenClaw-owned SQLite database. `openclaw backup sqlite snapshot` is the first proposed CLI surface, but the contract should not depend on command-only state. - -A minimal shape is: - -```ts -type SqliteSnapshotProvider = { - create(dbRef): Promise; - verify(snapshotRef): Promise; - restore(snapshotRef, targetPath): Promise; - list?(): Promise; - status?(): Promise; -}; -``` - -A later provider can add remote persistence: - -```ts -type RemoteSnapshotProvider = SqliteSnapshotProvider & { - upload(snapshotRef): Promise; - download(snapshotRef, targetPath): Promise; - prune?(policy): Promise; -}; -``` +- database role and owner +- OpenClaw-owned index definitions +- unexpected entries +- symlinks and hardlinks +- path identity and publication-race hazards +- private staging path ownership and ACL safety -This keeps SQLite runtime access local while making state artifacts portable. A local snapshot provider can be the reference implementation. Cloud/object-store providers can come later without changing the default local runtime. +Restore repeats verification and writes only to a fresh target path. It refuses an existing target and refuses stale `-wal`, `-shm`, or `-journal` sidecars. Activating a restored database remains an explicit offline operator step. -If the design proves broadly useful, the same contract can support backup restore, startup hydration, or state migration workflows without changing the narrow `openclaw backup sqlite snapshot` command. +### Security And Sensitivity -### Implementation roadmap +SQLite snapshot artifacts can contain auth profiles, session state, plugin state, per-agent state, and credentials-adjacent records. Snapshot repositories must be protected with the same access controls, encryption, retention policy, and destination restrictions as live OpenClaw state. -The implementation should be split into two phases. +The implementation fails closed rather than falling back to raw file copies when it cannot prove the repository, staging root, manifest, artifact, path identity, role, owner, ACL, or schema invariants. -Phase 1 is the committed SQLite snapshot backup command work. It proves full verified -snapshots, named OpenClaw database targets, and fresh-state restore. It should -also collect the metrics that decide whether Phase 2 is worth doing: +### WAL Bundles -- snapshot artifact size by database role -- snapshot duration -- upload or sync bytes and time, when the host can report them -- restore duration -- snapshot frequency required to meet the desired recovery point objective -- observed WAL growth between snapshots -- runtime impact while snapshotting +Ryan's scaling concern about whole-file copies is real, but file-sync deltas over live SQLite are not a safe answer. They can observe database pages, WAL frames, and sidecars at different moments without SQLite ordering or checkpoint semantics. -Phase 2 is gated. It should start only if Phase 1 data shows full snapshots are -too large, too slow, or too infrequent for hosted deployments. Phase 2's planned -shape is WAL bundles plus the minimum compaction proof needed to keep bundle -chains bounded. - -The current Phase 1 implementation stack is: - -| PR | Purpose | -| -------------------------------------------------------------------------- | --------------------------------------------- | -| [openclaw/openclaw#94805](https://github.com/openclaw/openclaw/pull/94805) | Core SQLite snapshot backup command and safe-sync artifact | -| [openclaw/openclaw#94967](https://github.com/openclaw/openclaw/pull/94967) | Snapshot stress harness for Phase 1 greenlight | - -Before maintainers accept this RFC or land the core command, the same -artifact-boundary idea should be exercised in Microsoft Scout/Lobster for about -one week through the Lobster-owned snapshot plugin path. That pilot is not a -replacement for the core command proposal; it is the operational evidence -checkpoint for it. The pilot should answer: - -- whether hosts can reliably sync only completed `manifest.json` and - `database.sqlite` snapshot artifacts while ignoring live SQLite files, - temporary files, and sidecars -- how often Scout needs snapshots to meet its recovery expectations -- observed snapshot artifact size and creation time for real hosted use -- whether users or operators need a core `openclaw backup sqlite snapshot` command, a - host-owned plugin command, or both -- whether Phase 2 WAL bundles are justified by measured size, timing, or - frequency data - -Earlier provider, CLI, and named-target slices were collapsed into -[openclaw/openclaw#94805](https://github.com/openclaw/openclaw/pull/94805) so -the core feature can be reviewed as one product boundary. - -#### Phase 1 / PR A: core SQLite snapshot backup command and safe-sync artifact - -Add the shared SQLite snapshot provider, local artifact repository, public -`openclaw backup sqlite snapshot` command surface, named database targets, and safe-sync -artifact proof. - -- `SqliteSnapshotProvider` contract -- local snapshot repository -- snapshot manifest and content hash verification -- SQLite-safe snapshot creation for one database reference using shared core/package primitives where OpenClaw already owns the SQLite invariants -- tests against a WAL-mode SQLite database -- an internal restore in tests to prove the artifact is usable -- user-facing core commands: - -```text -openclaw backup sqlite snapshot create -openclaw backup sqlite snapshot verify -openclaw backup sqlite snapshot restore -``` -- `--target global` for `state/openclaw.sqlite` -- `--agent ` for `agents//agent/openclaw-agent.sqlite` -- manifest fields for database role, agent id, schema version, and source path -- host-sync guidance that says live SQLite sidecars are ignored and completed - artifacts are the sync input -- target-directory safety checks, restore manifest validation, SQLite integrity checks after restore, and docs for the `openclaw backup sqlite snapshot` command surface -- proof that OpenClaw can create a snapshot from a named database target, copy only the completed snapshot directory, restore from that copied artifact into a fresh local SQLite path, and verify the restored database +If Phase 1 metrics show full snapshots are too large, too slow, or too infrequent, the next design should be ordered WAL-bundle artifacts anchored to a verified full snapshot. A WAL bundle would be an OpenClaw-authored artifact, not the live `*.sqlite-wal` file. -This proof should also document the host contract: which OpenClaw command or API materializes the artifact, which files are safe to sync, which live SQLite sidecars should be ignored, which manifest fields the host can store without interpreting SQLite internals, and what must be restored before OpenClaw opens the database. +The minimum WAL-bundle design should include: -This PR should record or report the Phase 1 metrics when available: snapshot -size, snapshot duration, restore duration, WAL growth, and upload/sync bytes and -time when the host supplies them. It should also document the greenlight -criteria maintainers would use before starting Phase 2. +- base snapshot generation +- monotonic bundle sequence number +- source database role and owner +- schema and page-size compatibility data +- content hash and byte size +- staging then publish into the artifact repository +- restore rejection for gaps, forks, duplicates, incompatible schema, and failed integrity checks +- final SQLite verification before runtime opens the restored database -Implementation: [openclaw/openclaw#94805](https://github.com/openclaw/openclaw/pull/94805). +WAL bundles, compaction, pruning, upload scheduling, and restore-on-boot remain follow-up work. -#### Phase 1 / PR B: snapshot stress harness +## Implementation -Add an opt-in harness that measures the Phase 1 behavior before maintainers -decide whether Phase 2 is worth building. +Phase 1 landed in [openclaw/openclaw#105718](https://github.com/openclaw/openclaw/pull/105718): -This PR should prove that snapshot artifacts still restore cleanly while a -writer is committing transaction batches against the source database. It should -cover `--target global` and `--agent ` at minimum, and should be able to add -future dedicated targets only after their owner database or logical export -contract is explicit. It should report snapshot/restore p50/p95 timings, -snapshot bytes, WAL bytes after the run, writer rows, and restore verification -counts. +- `openclaw backup sqlite create|list|verify|restore` +- strict local snapshot repository +- `VACUUM INTO` artifact creation +- committed WAL-state capture +- manifest and SHA-256 verification +- fresh-target restore +- global and per-agent database roles +- protected Windows DACL creation +- POSIX and macOS ownership/ACL checks +- symlink, hardlink, path-race, and publication-race defenses +- native Linux, Windows, and macOS proof -This PR should not make stress mandatory in normal CI. It is a local/release -validation tool for maintainers and operators who need evidence before tuning -snapshot frequency or greenlighting WAL bundles. +The original contributor prototype was [openclaw/openclaw#94805](https://github.com/openclaw/openclaw/pull/94805). It was closed as superseded because the landed implementation needed stricter database roles, schema/index/owner validation, fresh-only restore, content-pinned verification, durable publication, and cross-platform path-security hardening. -Implementation: [openclaw/openclaw#94967](https://github.com/openclaw/openclaw/pull/94967). - -#### Phase 2 / PR C: simple WAL bundle proof - -Phase 2 is not automatically required by Phase 1. It needs a maintainer -greenlight based on Phase 1 metrics. - -Add the smallest WAL-bundle proof that reduces full-snapshot frequency without -blessing live-file deltas. - -This PR should include: - -- base snapshot generation and bundle sequence fields -- hash and byte-size fields for WAL bundle artifacts -- staging then publish into the sync-owned artifact directory -- restore validation for gaps, forks, duplicate sequence numbers, and wrong base - snapshot generation -- tests that prove invalid bundle chains are rejected before database restore -- proof that restore works from: - -```text -full snapshot + ordered WAL bundles -> verified local SQLite database -``` - -This PR should not add retention policy, object storage, failover, or multiple -delta encodings. It should include a simple compaction proof: replay the latest -full snapshot plus bundles into a temp database, verify it, and publish a new -full snapshot generation. Conservative pruning policy can remain later work. - -#### Later work - -After the initial PRs, follow-up RFCs or implementation PRs can consider: - -- broader `openclaw backup` archive integration using the same provider contract -- object/blob storage providers -- retention and scheduling -- WAL bundle retention and pruning policy -- leases, promotion, fencing, and managed failover -- external tool integrations such as Litestream or LiteFS +The stress harness remains tracked separately in [openclaw/openclaw#94967](https://github.com/openclaw/openclaw/pull/94967). Broader state ownership and continuity work remains related to [openclaw/openclaw#101290](https://github.com/openclaw/openclaw/issues/101290). ## Rationale -This approach targets the reliability problem directly. It does not require OpenClaw to choose a second database backend before it has defined capture and restore semantics for the SQLite state it already owns. - -The database-first work in openclaw/openclaw#94646 improves this RFC because it gives snapshot a concrete target model. Snapshot does not have to invent logical database units. It can operate over the already-established global control-plane database and per-agent data-plane databases, then extend to dedicated owner stores only when those stores have comparable ownership and lifecycle metadata. - -Calling the subcommand `snapshot` under `backup sqlite` keeps the first deliverable concrete without giving it top-level command real estate. The path says what it is: a backup-area SQLite artifact, not a broad backup archive and not automatic failover. - -Keeping the first implementation stack under `openclaw backup sqlite snapshot` keeps the proof small and command-scoped. Existing `openclaw backup create` and `openclaw backup verify` behavior can remain unchanged while the snapshot provider proves the harder SQLite correctness and restore semantics. - -Treating remote storage as artifact storage avoids the common failure mode where object storage or network filesystems are used as if they were local disk. SQLite remains local and authoritative while running. Reliability comes from verified snapshots, manifests, restore procedures, and later WAL bundles. +This approach solves the reliability problem at the correct boundary. SQLite remains local and authoritative while OpenClaw is running. Durability is handled by verified artifacts, manifests, and explicit restore procedures. -Phase 2 is deliberately gated. WAL bundles reduce sync cost and restore-point -gaps only if Phase 1 metrics show full snapshots are insufficient. Keeping WAL -bundles behind a greenlight avoids committing to replay and compaction -complexity before OpenClaw knows database size, snapshot time, host sync cost, -restore time, WAL growth, and recovery point needs. +The command belongs in core because core owns the database roles, schema invariants, SQLite capabilities, and restore safety checks. Hosts can persist files, but they should not need to rediscover which live SQLite files are safe or unsafe to copy. -Making the feature opt-in keeps the default OpenClaw runtime simple. Local and development users should not need object storage, a lease service, or a managed scheduler to keep using SQLite. +Putting the feature under `openclaw backup sqlite` keeps it narrow and operator-facing without implying a broad backup archive, a new database backend, or automatic failover. -Keeping the SQLite-safe artifact contract inside OpenClaw is important because safe snapshots and restores need access to database paths, WAL behavior, schema versions, and integrity checks. Provider-owned artifact storage keeps cloud credentials, retention policy, and managed failover out of the default runtime. +Deferring WAL bundles is intentional. Full snapshots provide the first correct restore point and produce the metrics needed to decide whether incremental artifacts are worth the complexity. -Hosted deployments make that responsibility more important, not less. A host can persist a directory, but OpenClaw should define which database artifacts are safe to persist. Without that boundary, every host integration has to rediscover SQLite sidecar rules and OpenClaw database ownership independently. +## Future Work -The provider contract gives OpenClaw a path from a narrow command to future backup, restore, startup hydration, or migration workflows. - -Explicit writer ownership keeps horizontal service orchestration honest. A service can move work between hosts, but it must move ownership and restore state deliberately rather than letting several instances write the same SQLite database through shared storage. - -The proposal also keeps storage ownership decisions out of scope. OpenClaw already has global control-plane state, per-agent data-plane state, and owner-specific stores; this RFC defines how those databases become restorable snapshot artifacts. - -## Unresolved questions - -- Which database-first unit should be used for the first named snapshot/restore proof: global control-plane state, one per-agent data-plane database, or both? -- Should the first checkpoint implementation use SQLite online backup, `VACUUM INTO`, WAL checkpointing, page capture, or a higher-level export format? -- Should the reference provider be a local snapshot repository only, or should it include one object/blob storage provider? -- Which Phase 1 metric thresholds should greenlight Phase 2 WAL bundles: artifact size, snapshot duration, sync bytes/time, restore duration, WAL growth, or required recovery point objective? -- Should simple compaction land with the first WAL bundle proof or as the next Phase 2 PR? -- Where should writer ownership metadata live before a database is opened? -- Should restore verification run during startup, doctor, a managed-control-plane action, or all three? -- What is the minimum host-facing API or command shape needed for Scout/Lobster-style platforms to request artifact materialization and pre-start hydration? -- Should every dedicated SQLite store use a named `--target`, or should some - stores only be reachable through a lower-level database registry/API? -- Which artifacts should be included with database restore for support/debug exports versus canonical runtime recovery? -- Should external tools such as Litestream or LiteFS be provider integrations, deployment recommendations, or out of scope for OpenClaw-owned code? +- stress and crash-injection validation for concurrent writes +- high-frequency snapshot metrics across hosted deployments +- ordered WAL-bundle artifacts +- compaction and pruning for bounded bundle chains +- upload/download providers +- scheduling and retention policy +- restore-on-boot host integration +- leases, promotion, fencing, and managed failover +- dedicated snapshot targets for future owner stores