Define and implement the Ceph-backed volume lifecycle

docs/manifest.md

@@ -32,6 +32,7 @@ Odorobo should provision. It contains:
   Cloud Hypervisor has no per-disk boot index, so providers must preserve this order when
   translating the manifest. The control plane owns attachment identity and ordering; the
   provider driver/storage transform resolves the URI or volume ID to a node-local device path.
+  Current URI support and attachment lifecycle are documented in [the storage contract](storage.md).
 - `networks`: stable network IDs and optional guest MAC addresses. The control plane owns
   the attachment identity and requested MAC; the provider networking transform resolves
   the network to a host interface or tap device.

docs/provisioning.md

@@ -52,8 +52,25 @@ reservations from failed startup attempts are rolled back only after confirmed
 process exit. An active lease cannot be reused—even for the same VM ID—until
 exit is confirmed. Every reservation has a distinct persisted lease token;
 startup rollback, exit recording, and deletion retries use that token so one
-incarnation cannot release a newer incarnation's lease. After an unclean agent/host failure, stale active leases may
-need manual recovery rather than assuming the old VMM is gone. A requested CID is
+incarnation cannot release a newer incarnation's lease.
+
+On SIGTERM/SIGINT the agent stops its VMs before exiting (waiting up to 60
+seconds), so each VMM is reaped and its lease is marked stopped. A second
+signal exits immediately. The shipped systemd unit uses `KillMode=mixed` so
+only the agent gets the SIGTERM (the VMMs run in its cgroup), and a 90 second
+`TimeoutStopSec` so systemd doesn't SIGKILL it mid-shutdown. If you write your
+own unit, keep both. The single-instance lockfile (`/var/lock/odorobo.lock`) is
+an flock, so an agent that was killed outright doesn't block the next start.
+
+The registry also records each lease's VMM process by PID, kernel start time,
+and boot ID, so a process from before a reboot is never mistaken for a new one.
+After an unclean agent exit, a restarted agent treats a lease as stopped once
+that exact process is gone, meaning every one of its threads has exited. An explicit delete
+of a VM whose VMM is still running with no actor left (for example after the
+agent was killed) SIGKILLs that exact process before releasing its lease and
+runtime files. A lease that never had its process recorded (the agent died
+between reserving the CID and spawning the VMM, or recording failed) still
+needs manual recovery. A requested CID is
 reserved if free; a conflict fails VM creation rather than silently assigning
 a different CID. `GetVMInfo` returns the effective VM manifest, including the
 allocated CID in `observed.vsock_guest_cid` and the desired socket path;
@@ -69,9 +86,27 @@ a host need distinct key paths.
 
 Deletion is acknowledged only after the known VM-owning agents confirm process
 exit and node-local cleanup. Missing owners keep deletion pending; retry after
-those agents become reachable. Cleanup ownership survives discovery-cache
-eviction in scheduler memory, but is not a durable cluster inventory across
-scheduler restarts. Keep node identities stable when recovering failed agents.
+those agents become reachable, or delete with `?force=true` to drop owners whose
+agent can't be reached at all (an agent that answers with an error, or that
+might have gotten the request but timed out, still fails the delete). Every
+create and delete the scheduler sends carries a time-ordered stamp, so a delete
+that arrives after its VM ID was recreated is acked without touching the new
+VM. Cleanup ownership survives discovery-cache eviction in scheduler
+memory, and after a scheduler restart it is rebuilt from agent status: each
+agent reports running VMs and `retained_vms` (IDs that still hold node-local
+state, like a stopped CID lease). An agent that was unreachable since the
+restart can't be known as an owner, so keep node identities stable when
+recovering failed agents.
+
+If an agent reports a VM the scheduler already finished deleting (like a node
+skipped by a forced delete coming back), the scheduler deletes it there in the
+background. Stopped VMs are never cleaned up this way. Until that node acks a
+delete or reports a full status without the VM, the ID can't be recreated, since
+the old copy could still be running on the same storage. Use a new VM ID if the
+node isn't coming back. Stop/delete tombstones are kept in scheduler memory
+only: stopped ones for an hour, deleted ones for a day (longer while a node
+might still hold the VM), and pending deletes until they finish. A scheduler
+restart forgets all of this.
 
 Storage teardown releases only resources acquired by Odorobo. Shared mappings
 are reference-counted within the agent process; pre-existing external mappings

docs/scheduler_routes.md

@@ -8,6 +8,14 @@ we need CRUD. we keep track of if its active.
 
 MACs for VMs need to be stored somewhere. it can be either in the DHT or in the Dashboard. doesnt matter.
 
+### deleting VMs
+
+`POST /vms` refuses an ID that some agent still reports running when the scheduler has no record of it, like right after a scheduler restart before discovery catches up, or a VM whose exit couldn't be confirmed. starting it somewhere else would run two copies on the same storage. retry once discovery finds it (the retry then returns the existing VM), or delete it first. an agent that only holds a stopped VM's lease doesn't block this, since recreating is how a stopped VM gets started again.
+
+`DELETE /vms/{vmid}` only succeeds once every node that might still hold something for the VM (a running process, a stopped vsock CID lease, storage mappings) confirms cleanup. if one of those nodes is unreachable, the delete fails and the ID can't be recreated until a retry works.
+
+`DELETE /vms/{vmid}?force=true` is for when a node is gone for good. owners whose agent can't be reached at all (no route, the dial fails, or no agent is running there) are dropped with a warning instead of failing the delete. an agent that answers and says cleanup failed still fails the delete, force or not, since it's alive and something is actually wrong. so does one that might have gotten the request but didn't answer (a timeout or a connection dropped mid-request), since a slow teardown looks exactly like that. retry, and an agent that's really gone shows up as unreachable. whatever a dropped node still holds is leaked, but if it ever comes back and reports the VM, the scheduler deletes it there in the background. until then `POST /vms` with the same ID is refused, since the old copy might still be running on the same storage. use a new VM ID if the node is gone for good.
+
 ## volumes
 
 we again need CRUD, and we keep track of the volumes cause we want to just check what is in the SAN

docs/storage.md

@@ -1,52 +1,174 @@
-# Storage Integration
+# VM storage contract
 
-Orodobo adds a storage integration layer on top of Cloud Hypervisor's block device support, using config transformers to convert custom URI schemes into valid block device paths for Cloud Hypervisor.
-This allows users to specify storage devices in a more flexible way, and enables simpler integration with networked storage backends, by having Odorobo abstract and attach/detach storage devices on demand based on the VM's lifecycle and provisioning hooks.
+Odorobo accepts storage attachments through the VM manifest's ordered
+`desired.storage` list. The Cloud Hypervisor driver resolves each supported URI
+to a host-local file or block-device path when creating a VM. It does not copy
+an RBD image onto each compute node.
 
-Currently, Odorobo's Storage transformer supports the following backends:
+The initial storage contract supports local files and kernel-mapped Ceph RBD
+images. iSCSI remains available through the existing transform, but is outside
+the Ceph lifecycle contract described here. iSCSI URIs must be exactly
+`iscsi://<portal>[:<port>]/<iqn>/<lun>` (port defaults to 3260), where `<lun>`
+is plain decimal digits that fit in 32 bits, optionally written `lun<n>`. The
+LUN is required. URIs with a malformed or out-of-range LUN, extra path segments,
+a query or fragment, or a percent-encoded portal or IQN are rejected before any
+login rather than falling back to LUN 0. This defines behavior for new
+manifest attachments; it does not migrate existing Odorobo deployments or
+rewrite existing local images. A manifest attachment has a stable
+attachment ID, one source (`uri` or `volume_id`), and an optional `read_only`
+flag. `volume_id` resolution is not yet defined; use a supported URI.
 
-- **Local Disk Images**: Specify a local disk image with the `file://` URI scheme, e.g. `file:///var/lib/odorobo/images/my-disk.qcow2`. This is simply passed through to Cloud Hypervisor as if the `file://` prefix was not there, implemented as a dummy transformer for validating the storage transformer API.
-- **Ceph RBD**: Specify a Ceph RBD image with the `rbd://` URI scheme, e.g. `rbd://my-pool/my-image`. Odorobo will use the `rbd` command line tool to map the RBD image to a local block device (e.g. `/dev/rbd0`), and then pass this block device path to Cloud Hypervisor.
-- **iSCSI**: Specify an iSCSI target with the `iscsi://` URI scheme, e.g. `iscsi://<target-ip>:<port>/<iqn>/<lun>`. Odorobo will use the `iscsiadm` command line tool to log in to the iSCSI target and map the specified LUN to a local block device (e.g. `/dev/disk/by-path/ip-<target-ip>:<port>-iscsi-<iqn>-lun-<lun>`), and then pass this block device path to Cloud Hypervisor.
+## Local file images
 
-To use the storage transformers, simply specify the desired URI in place of the `path` field for a block device in your VM config. For example:
+Use an absolute `file://` URI, for example:
 
 ```json
-{
-  ...
-  "disks": [
-    {
-      "path": "rdb://my-pool/my-image",
-      "id": "disk0",
-    }
-  ],
-}
+{ "id": "root", "uri": "file:///var/lib/odorobo/images/root.raw" }
 ```
 
-Odorobo will automatically detect the `rbd://` scheme, map the RBD image to a local block device, and pass the block device path to Cloud Hypervisor when creating the VM, transforming them to:
+The path refers to a file already present on the node running the VM. Odorobo
+passes the path to Cloud Hypervisor; it does not copy, create, or delete the
+image. The file must be readable (and writable unless the attachment is
+read-only) by the service account and accessible inside the Cloud Hypervisor
+process's mount namespace. Local-file attachments have no external resource
+to unmap during teardown.
+
+## Ceph RBD images
+
+Specify an existing RBD image as `rbd://<pool>/<image>`:
 
 ```json
 {
-  ...
-  "disks": [
-    {
-      "path": "/dev/rbd/my-pool/my-image",
-      "id": "rdb://my-pool/my-image?id=disk0",
-    }
-  ],
+  "id": "root",
+  "uri": "rbd://odorobo-vms/root-01",
+  "read_only": false
 }
 ```
 
-The `id` field is transformed to include the original URI for reference, and can be used in provisioning hooks to identify which disk is which when attaching/detaching storage devices on demand.
+The pool and image are part of the VM attachment URI, not a local cache path.
+For the initial contract, pool and image names use ASCII letters, digits, `.`,
+`_`, or `-`; names cannot be empty or start with `-`. Pool names must be
+lowercase; the transformer validates the original URI and rejects case variants
+rather than normalizing them. The URI must have exactly one image path segment.
+Credentials, ports, query parameters, fragments, namespaces, and snapshot
+selectors are not accepted. In particular,
+`rbd://pool/ns/image` and `rbd://pool/image@snapshot` are not supported by this
+contract. These restrictions prevent an ambiguous URI from selecting a
+resource different from the one Odorobo tracks.
+
+The referenced pool and image must already exist and be reachable by the
+configured Ceph client. Odorobo maps the image with the `rbd` CLI and passes the
+resulting kernel block device to Cloud Hypervisor. It does not create pools or
+images, clone/import images, resize them, or delete them. Deleting a VM removes
+its attachment and may unmap a mapping owned by Odorobo; it never deletes the
+RBD image or pool. Provisioning, snapshotting, importing, and image deletion
+remain operator-managed operations.
+
+`read_only: true` makes the Cloud Hypervisor disk read-only. Odorobo's host-side
+attachment reference counting prevents a mapping from being unmapped while
+another VM still references it; it does not make concurrent writable use of a
+filesystem safe. Share writable images only when the guest workload and
+filesystem are explicitly designed for multi-writer access. Read-only sharing
+is the usual safe shared-image pattern.
+
+### Node prerequisites and Ceph client configuration
+
+Each VM node needs the `rbd` executable (typically from `ceph-common`), the
+kernel RBD module, permission to map block devices, and access to the Ceph
+cluster. Odorobo passes these optional environment variables through to `rbd`:
+
+| Environment variable | `rbd` option | Purpose |
+| --- | --- | --- |
+| `CEPH_CONFIG` | `--conf` | Ceph configuration file path |
+| `CEPH_ID` | `--id` | Ceph client identity, such as `odorobo` |
+| `CEPH_KEYFILE` | `--keyfile` | Path to the client key file; do not put key contents in a manifest |
+| `CEPH_CLUSTER` | `--cluster` | Ceph cluster name |
+
+When omitted, the Ceph CLI uses its normal defaults. These values are node
+configuration: all RBD references on a node use the same configured client
+context. Keep key files readable only by the Odorobo service account and grant
+only the pool capabilities needed by the node. Pool and image selection is
+configured per attachment in its `rbd://` URI.
+
+Odorobo asks `rbd device map` not to wait on a udev event in its own namespace.
+When the host's Ceph udev rule is present, the driver prefers the stable path
+`/dev/rbd/<pool>/<image>`. Otherwise, it falls back to the device reported by
+`rbd device list` (for example `/dev/rbd0`). The udev rule commonly provided by
+Ceph packages is:
+
+```udev
+KERNEL=="rbd[0-9]*", ENV{DEVTYPE}=="disk", PROGRAM="/usr/bin/ceph-rbdnamer %k", SYMLINK+="rbd/%c"
+KERNEL=="rbd[0-9]*", ENV{DEVTYPE}=="partition", PROGRAM="/usr/bin/ceph-rbdnamer %k", SYMLINK+="rbd/%c-part%n"
+```
+
+Install it as `/etc/udev/rules.d/50-rbd.rules` only if the distribution's Ceph
+packages do not already provide an equivalent rule.
+
+## Attachment and mapping lifecycle
+
+The control plane owns the desired attachment ID, source, read-only setting,
+and disk order. On a node, the storage provider owns mapping and cleanup
+bookkeeping; Cloud Hypervisor only receives the resolved path. The lifecycle
+is:
+
+1. **Requested** — the manifest names a URI-backed attachment.
+2. **Acquiring** — the storage provider records a VM lease before starting map
+   or path-resolution work. A failed or cancelled create therefore cannot
+   silently lose track of a partial mapping.
+3. **Attached** — the resolved path is passed to Cloud Hypervisor. Multiple
+   Odorobo VMs using the same canonical RBD resource share one provider
+   acquisition and increment its reference count.
+4. **Releasing** — after the VM process has exited, teardown releases that
+   VM's lease. An Odorobo-owned mapping is unmapped only when its final active
+   lease is released. A mapping that existed before Odorobo acquired it is
+   borrowed and is never automatically unmapped by Odorobo.
+5. **Released** — the lease is removed. Repeated cleanup of an already-released
+   lease is a no-op.
+
+A confirmed failure to map can be retried. If a command's result is ambiguous
+(for example, the command timed out but the mapping may have been created),
+Odorobo quarantines the resource rather than guessing who owns it: it blocks
+reuse and automatic unmapping and reports that operator reconciliation is
+required. Do not manually unmap a device while an affected VM may still be
+running. The in-memory lease ledger is not persisted across agent restarts, so
+inventory and reconcile outstanding RBD mappings before restarting an agent
+that has active or uncertain storage operations.
+
+### Operation deadlines
+
+Storage work runs in background tasks; callers (VM create and delete) only
+wait for it up to a deadline. The deadline is derived from the storage command
+limits rather than picked by hand: each `rbd`/`iscsiadm` invocation gets 30
+seconds plus a 2-second reap grace, and one disk resolution can run up to four
+of them (RBD `device list`, `device map`, `device list`, then another
+`device list` after up to 10 seconds of waiting for the udev path), plus up to
+four 5-second iSCSI DNS lookups. That gives a per-operation budget of about
+158 seconds. VM create applies it to each disk separately, and Cloud
+Hypervisor's create timeout is 60 seconds on top of every disk's budget, so a
+slow but working map isn't torn down as a failed create. The scheduler keeps a
+pending placement for the VM's whole worst-case startup plus two minutes
+before rescheduling it, since the create request itself times out after 120
+seconds while the agent is still starting the VM. VM delete releases
+disks one after another, so its deadline is one budget per lease (and two per
+still-pending resolution, which must finish and may add a lease).
+
+A "Storage operation deadline exceeded" error does not mean the operation
+failed or was cancelled. The work is still running and the agent keeps the VM's
+leases and any quarantine while it does. Retrying the delete (or the teardown
+after a failed create) joins the in-flight cleanup instead of starting a second
+one, and a new create for the same VM is refused until the earlier work
+finishes. Do not reuse or manually unmap/log out the storage while it is still
+in flight.
+
+VM shutdown/deletion releases VM attachments according to the VM lifecycle,
+but does not delete backing images. A VM's configured disk order is preserved;
+the first manifest storage entry is the preferred boot disk.
 
+## Mocking and tests
 
-> [!NOTE]
-> Ceph RBD integration requires the following udev rule to allow Odorobo to easily find the mapped block device for a given RBD image:
->
-> ```
-> # /etc/udev/rules.d/50-rbd.rules
-> KERNEL=="rbd[0-9]*", ENV{DEVTYPE}=="disk", PROGRAM="/usr/bin/ceph-rbdnamer %k", SYMLINK+="rbd/%c"
-> KERNEL=="rbd[0-9]*", ENV{DEVTYPE}=="partition", PROGRAM="/usr/bin/ceph-rbdnamer %k", SYMLINK+="rbd/%c-part%n"
-> ```
-> 
-> This rule is commonly included in Ceph packages (`ceph-common` in Fedora), but if your distribution does not include it by default, you will need to add it manually for RBD support to work properly. This allows Odorobo to find the mapped block device for a given RBD image under `/dev/rbd/`, which is necessary for passing the correct block device path to Cloud Hypervisor.
+Storage backends implement a mockable provider interface for acquisition,
+resolution, and release. Lifecycle tests use fake providers to cover shared
+leases, borrowed mappings, partial failures, ambiguous command outcomes,
+retries, cancellation, and idempotent cleanup; they do not require a live Ceph
+cluster. The local Ceph development environment is documented in
+[`.local/dev/README.md`](../.local/dev/README.md).

odorobo/benches/agent_status.rs

@@ -18,6 +18,7 @@ fn status(vm_count: usize) -> AgentStatus {
             vms.sort_unstable();
             vms
         },
+        retained_vms: Vec::new(),
         metadata: ObjectMetadata::default(),
     }
 }
@@ -44,6 +45,7 @@ fn main() {
             revision: 1,
             added,
             removed,
+            retained_vms: Vec::new(),
             used_vcpus: base.used_vcpus,
             used_ram: base.used_ram,
         };