diff --git a/README.md b/README.md index f3f110a6dd..7759e6b339 100644 --- a/README.md +++ b/README.md @@ -205,10 +205,10 @@ If you need to delete the local `kind` cluster and its registry (if it was creat We provide several sample applications demonstrating Agent Substrate's capabilities: -1. **[Counter Demo](demos/counter/README.md)**: A stateful Go HTTP server demonstrating state preservation across suspends/resumes, and dynamic CRD routing. +1. **[Counter Demo](demos/counter/README.md)**: A stateful Go HTTP server demonstrating state preservation across suspends/resumes, and on-demand actor resumption and routing via the Substrate router. 2. **[Sandbox Demo (Antigravity)](demos/sandbox/README.md)**: A secure, sandboxed execution environment (running Alpine Linux) that allows arbitrary shell execution while preserving filesystem state across sessions. 3. **[Claude Code Multiplex](demos/claude-code-multiplex/README.md)**: Demonstrates oversubscribing physical hardware by multiplexing multiple Claude Code agents onto a limited pool of workers. -4. **[Multi-Template](demos/multi-template/README.md)**: Two `ActorTemplate`s running different binaries share one `WorkerPool`, across three namespaces. +4. **[Multi-Template](demos/multi-template/README.md)**: Two `ActorTemplate`s running different binaries share one `WorkerPool`, even though the templates live in different atespaces. 5. **[Request Parking](demos/parking/README.md)**: An oversubscribed pool where the router holds inbound requests until a worker frees up, instead of returning `503`. 6. **[Autoscaled WorkerPool](demos/autoscaled-workerpool/README.md)**: Scales a `WorkerPool` on its assigned-worker count with an HPA fed by prometheus-adapter. diff --git a/benchmarking/workloads/deploy.sh b/benchmarking/workloads/deploy.sh index 3638ec8178..ac95056edf 100755 --- a/benchmarking/workloads/deploy.sh +++ b/benchmarking/workloads/deploy.sh @@ -44,7 +44,7 @@ fi WORKER_COUNT=1 SANDBOX_CLASS="gvisor" -# Actor memory limit (ActorTemplate spec.resources.limits.memory). The default +# Actor memory limit (ActorTemplate resources.limits.memory). The default # is the smallest size microvm admits (128Mi VMM reserve + 128Mi guest floor), # so benchmark actors do not inherit the 2 GiB kata default and drag its page # cache into every memory snapshot. Raise it for RAM-consuming suites. diff --git a/cmd/ateapi/internal/controlapi/template_reconciler.go b/cmd/ateapi/internal/controlapi/template_reconciler.go index 1842bbe944..b99336ce6f 100644 --- a/cmd/ateapi/internal/controlapi/template_reconciler.go +++ b/cmd/ateapi/internal/controlapi/template_reconciler.go @@ -351,8 +351,7 @@ func goldenSnapshotDone(snapshotStatus *ateapipb.GoldenSnapshotStatus) bool { // goldenSnapshotWarmupFor returns 0 when every container has a readyz probe // (ResumeActor already blocked until the workload reported 200), and the -// default warmup otherwise. Mirrors the CRD controller's function of the -// same name; keep both in sync. +// default warmup otherwise. func goldenSnapshotWarmupFor(containers []*ateapipb.Container) time.Duration { if len(containers) == 0 { return goldenSnapshotWarmup diff --git a/demos/claude-code-multiplex/ui/server.go b/demos/claude-code-multiplex/ui/server.go index c9a28cbffb..a2a00d7e96 100644 --- a/demos/claude-code-multiplex/ui/server.go +++ b/demos/claude-code-multiplex/ui/server.go @@ -397,7 +397,7 @@ func handlePods(w http.ResponseWriter, r *http.Request) { } // handleActors returns actor-shaped JSON sourced from ateapi.ListActors. -// ActorTemplates / WorkerPools (k8s CRDs) are no longer surfaced — +// ActorTemplate / WorkerPool resources are no longer surfaced — // substrate-native Actors are the canonical demo entity. func handleActors(w http.ResponseWriter, r *http.Request) { if ateClient == nil { diff --git a/docs/api-guide.md b/docs/api-guide.md index d2b3b58fb3..10438094f8 100644 --- a/docs/api-guide.md +++ b/docs/api-guide.md @@ -62,7 +62,7 @@ moves. A pinned pool cannot put those pods back on a moved node, so the old version drains away node by node. An unpinned pool breaks this constraints. #### Worker Capacity (`spec.template.resources`) -Setting `resources.limits` (CPU and Memory) on a `WorkerPool` establishes each worker pod's **capacity** — the envelope available to host an actor sandbox, taken from the `ateom` container's limits. The scheduler only places an actor on a worker whose capacity is `>=` the actor's declared resource limits (see [Sandbox Right-Sizing](#sandbox-right-sizing-specresources) on the `ActorTemplate`). +Setting `resources.limits` (CPU and Memory) on a `WorkerPool` establishes each worker pod's **capacity** — the envelope available to host an actor sandbox, taken from the `ateom` container's limits. The scheduler only places an actor on a worker whose capacity is `>=` the actor's declared resource limits (see [Sandbox Right-Sizing](#sandbox-right-sizing-resources) on the `ActorTemplate`). - Size a pool's `limits` to the largest actor it should host. An actor occupies its whole worker, so worker capacity is the per-actor ceiling, not a shared budget. - Capacity is advisory for placement only: a worker that declares no CPU/memory limit reports zero capacity for that dimension, which the scheduler treats as **unconstrained** (placement is never blocked by missing data). The actual sandbox size still comes from the `ActorTemplate`. @@ -113,24 +113,24 @@ worker pod on a GPU node and reserves the device, and nothing else reads it. The `ActorTemplate` defines the code, environment, and state-management policies for a specific type of agent. It is used to generate the "Golden Snapshot" from which all actors of this type are derived. -### Specification (`ActorTemplateSpec`) +### Specification (`ActorTemplate`) | Field | Type | Description | | :--- | :--- | :--- | | `containers` | `[]Container` | **Required.** The workload definition — see [Container Fields](#container-fields) below. Each container may also declare an optional `readyz` HTTP probe — see [Container Readiness Probe](#container-readiness-probe-readyz). | | `sandboxConfig` | `SandboxConfig` | **Required.** The sandbox runtime selection: `sandboxClass` (**required**, `SANDBOX_CLASS_GVISOR` or `SANDBOX_CLASS_MICROVM`) picks the runtime family this template's actors require — only `WorkerPool`s whose `sandboxClass` matches are eligible — and `configName` (**required**) names the cluster-scoped [`SandboxConfig`](#3-sandboxconfig-the-sandbox-itself) object supplying the sandbox binaries. | -| `workerSelector` | `*LabelSelector` | Optional. Gates which `WorkerPool`s actors from this template may use, by matching against each pool's labels. If unset, all pools are eligible (subject to the actor's own `worker_selector`). | +| `workerSelector` | `*Selector` | Optional. Gates which `WorkerPool`s actors from this template may use, by matching against each pool's labels (`matchLabels`). If unset, all pools are eligible (subject to the actor's own `worker_selector`). | | `snapshotsConfig` | `SnapshotsConfig` | **Required.** The base object-storage location snapshots are written under, plus the pause/commit/resume scopes. See [Snapshot Storage Layout](#snapshot-storage-layout). | | `volumes` | `[]Volume` | Optional. Volumes the containers may mount, each a `durableDir`, an `externalVolumeTemplate` (see [CSI Volumes Guide](csi-volumes.md)), or a `systemInfo` volume (see [SystemInfo Volumes](#systeminfo-volumes)). Every declared volume must be mounted by at least one container. A `microvm` template may declare several `durableDir` volumes; a `gvisor` template is limited to one. | -| `resources` | `*ResourceRequirements` | Optional. Declares each actor's compute size via `limits` — see [Sandbox Right-Sizing](#sandbox-right-sizing-specresources). Immutable, like the rest of the spec. | +| `resources` | `*ResourceRequirements` | Optional. Declares each actor's compute size via `limits` — see [Sandbox Right-Sizing](#sandbox-right-sizing-resources). Immutable, like the rest of the template. | The sandbox itself — the binaries (e.g. the gVisor `runsc` binary) and the `pauseImage` holding the sandbox's namespaces — comes from the cluster-scoped [`SandboxConfig`](#3-sandboxconfig-the-sandbox-itself) object named by `sandboxConfig.configName`. At runtime the workers resolve it from the `WorkerPool` side — by name (`workerPool.spec.sandboxConfigName`) or, by default, the cluster default `SandboxConfig` for the pool's `sandboxClass`. -Because a snapshot is not restorable across sandbox runtimes, `sandboxClass` is a **hard scheduling gate**: an actor is only ever placed on a `WorkerPool` of the matching class. It is AND'd with `workerSelector` (and the actor's `worker_selector`), which can only narrow the eligible pools further. It has no default — `sandboxConfig` is required — and, like the rest of the spec, is immutable, so each template's class is fixed at creation. +Because a snapshot is not restorable across sandbox runtimes, `sandboxClass` is a **hard scheduling gate**: an actor is only ever placed on a `WorkerPool` of the matching class. It is AND'd with `workerSelector` (and the actor's `worker_selector`), which can only narrow the eligible pools further. It has no default — `sandboxConfig` is required — and, like the rest of the template, is immutable, so each template's class is fixed at creation. -### Sandbox Right-Sizing (`spec.resources`) +### Sandbox Right-Sizing (`resources`) -Unlike a Pod, an actor is sized by its **`limits`** (CPU and Memory): the size is a property of the template, baked into snapshots, so it lives on the immutable `ActorTemplate` spec. Declared limits do three things: +Unlike a Pod, an actor is sized by its **`limits`** (CPU and Memory): the size is a property of the template, baked into snapshots, so it lives on the immutable `ActorTemplate`. Declared limits do three things: 1. **Size the sandbox.** The limits are supplied to the sandbox over the actor RPCs (control plane → atelet → ateom): - **gVisor (`ateom-gvisor`)** — applied to the container OCI spec: `limits.cpu` sets the cgroup v2 CPU quota (`cpu.max`) and the Sentry vCPU count (`--cpu-num-from-quota`); `limits.memory` sets the cgroup v2 memory limit (`memory.max`) and bounds the virtual total memory the sandbox reports (so JVM/Go do not over-allocate from host RAM). @@ -149,7 +149,7 @@ Substrate uses a **Uniform DNS Mesh**: every actor created from a template is au ### SystemInfo Volumes -To deliver identity information, including credentials, to a running actor, you can use a SystemInfo volume. Define it in `spec.volumes`, and mount it into each container that needs it. +To deliver identity information, including credentials, to a running actor, you can use a SystemInfo volume. Define it in `volumes`, and mount it into each container that needs it. Available information sources: @@ -216,7 +216,7 @@ Each entry in `containers` describes one process to run in the actor's sandbox. | `args` | `[]string` | Optional. Arguments to the entrypoint. If unset, the image's `CMD` is used (unless `command` is set, which discards the image's `CMD`). If set, it replaces the image's `CMD`. | | `env` | `[]EnvVar` | Optional. Literal `value` entries. | | `readyz` | `ContainerReadyz` | Optional. HTTP readiness probe — see [Container Readiness Probe](#container-readiness-probe-readyz). | -| `volumeMounts` | `[]VolumeMount` | Optional. Mounts a `spec.volumes` entry (e.g. `durableDir`) into this container. | +| `volumeMounts` | `[]VolumeMount` | Optional. Mounts a `volumes` entry (e.g. `durableDir`) into this container. | | `securityContext` | `SecurityContext` | Optional. Security settings for the container process — see [Container Capabilities](#container-capabilities-securitycontextcapabilities). | | `resources` | `ContainerResources` | Optional. Compute limits for this container, enforced inside the actor's sandbox. Only `limits` is supported, and only `cpu` and `memory`. See [Per-container limits](#per-container-limits). | @@ -266,7 +266,7 @@ A container that exceeds its memory limit is OOM-killed on its own; the actor's Per-container limits are micro-VM only today. gVisor applies cgroup limits at the sandbox level: one sentry backs every container in the actor, so a per-container cgroup is created and then stays empty ([google/gvisor#190](https://github.com/google/gvisor/issues/190)). A template that sets `resources` with `sandboxClass: gvisor` is rejected. -These limits subdivide the sandbox that [`spec.resources`](#sandbox-right-sizing-specresources) already sized; a container that declares none is bounded by the guest as a whole, not by a copy of the actor's total. A micro-VM guest is sized from `spec.resources.limits.memory` minus the VMM reserve, or from the pool's [`SandboxConfig`](#3-sandboxconfig-the-sandbox-itself) when the template declares no actor-level limit. The CPU ceiling is the guest's vCPU count, which falls back to the pool's `default_vcpus` (1 unless the `SandboxConfig` raises it), so a template that declares no `spec.resources.limits.cpu` caps each container, and their sum, at `1000m`. A limit above either ceiling can never bind, so the actor fails to start with an error naming both the limit and the ceiling. +These limits subdivide the sandbox that [`resources`](#sandbox-right-sizing-resources) already sized; a container that declares none is bounded by the guest as a whole, not by a copy of the actor's total. A micro-VM guest is sized from `resources.limits.memory` minus the VMM reserve, or from the pool's [`SandboxConfig`](#3-sandboxconfig-the-sandbox-itself) when the template declares no actor-level limit. The CPU ceiling is the guest's vCPU count, which falls back to the pool's `default_vcpus` (1 unless the `SandboxConfig` raises it), so a template that declares no `resources.limits.cpu` caps each container, and their sum, at `1000m`. A limit above either ceiling can never bind, so the actor fails to start with an error naming both the limit and the ceiling. Each limit is validated on its own at apply, but the sum across the actor's containers is only checked when the actor first runs, against the real guest size. A template whose limits do not fit is accepted by the API server and fails on its first actor. @@ -411,13 +411,13 @@ When an `ActorTemplate` is created: 4. The template enters the `Ready` phase. ### Resumption Lifecycle -Once a template is `Ready`, creating an actor logically (via `kubectl-ate create actor`) allows it to be resumed instantly on any free worker in the referenced `WorkerPool`. Substrate bypasses the standard container boot and restores the process directly from its last saved state. +Once a template is `Ready`, creating an actor logically (via `kubectl ate create actor`) allows it to be resumed instantly on any free worker in the referenced `WorkerPool`. Substrate bypasses the standard container boot and restores the process directly from its last saved state. --- ## 5. Best Practices * **Startup Logic:** Place expensive initialization (loading large models, establishing baseline connections) in your application's entry point. These will be captured in the Golden Snapshot and won't need to be repeated on every resumption. -* **Symmetry:** Ensure your `ActorTemplate` and `WorkerPool` are in the same namespace or have appropriate RBAC permissions to reference each other. +* **Placement:** Ensure your `ActorTemplate`'s `sandboxClass` matches your `WorkerPool`'s, and use the template's `workerSelector` to target specific pools — pool selection is by label match, not by namespace or RBAC. * **Version Management:** When updating code, create a new `ActorTemplate` (e.g. `v2`). Substrate treats each template as an immutable state root. --- diff --git a/docs/api-style-guide.md b/docs/api-style-guide.md index 60709b0deb..822ab57805 100644 --- a/docs/api-style-guide.md +++ b/docs/api-style-guide.md @@ -240,7 +240,7 @@ Rules: - RPC name **must** begin with `Update` followed by the singular resource name. - Response **must** be the resource itself — not an `UpdateActorResponse` wrapper. - **Output-only** fields — server-managed, never set by the client (`uid`, `version`, `create_time`, `update_time`, and the whole `status` submessage). Whatever the request carries in them is ignored and the server's own values are kept. - - **Immutable** fields — caller-set at creation but fixed thereafter (`atespace`, `name`, and resource-specific ones such as an actor's `actor_template_name`). A request that changes one - including by omitting it, which would clear it - **must** return `INVALID_ARGUMENT` naming the field. + - **Immutable** fields — caller-set at creation but fixed thereafter (`atespace`, `name`, and resource-specific ones such as an actor's `source_snapshot_tag`). A request that changes one - including by omitting it, which would clear it - **must** return `INVALID_ARGUMENT` naming the field. - The embedded resource field **must** be named after the resource's own snake_case type name (e.g. `actor` for `Actor`), not a generic name like `resource` or `body`. - Unknown fields in the request are preserved, so a client built against a newer schema does not lose data by round-tripping through an older one. - The resource's `atespace` and `name` identify the resource to update; they are not themselves updatable. diff --git a/docs/architecture.md b/docs/architecture.md index a6dcbbde73..bd9da1c7cb 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -136,8 +136,9 @@ There are a few different personas that interact with the system: 3) **Agent developers**: These are the people who deploy agents into a substrate for users or higher-level systems to consume. They might have to be aware that they are using Kubernetes (some concepts are represented as CRDs, - such as ActorTemplates), or they might be using a higher level API which - itself uses Agent Substrate under the hood. + such as WorkerPools), or they might be using a higher level API which + itself uses Agent Substrate under the hood. ActorTemplates are managed + through the substrate API (`kubectl-ate`), not through Kubernetes. 4) **Agent users**: These are the people who interact with agents running in the substrate. They might be end-users of an application that is built on top @@ -207,20 +208,22 @@ worker. Agent Substrate categorizes resources into two groups based on their persistence requirements and the frequency of state transitions. -### System Configuration (Declarative/CRD-based) +### System Configuration (Declarative) -These resources define the intended state of the system and are managed via -Kubernetes CRD APIs. They are used for administrative operations and actor -environment definitions. +These resources define the intended state of the system. They are used for +administrative operations and actor environment definitions. - * **WorkerPool**: Defines a pool of "warm" compute capacity. It manages a - fleet of standby worker pods initialized and ready to receive resumed actor - states. Optional `spec.template` fields configure worker pod node - selection, tolerations, priority class, and node affinity. + * **WorkerPool** (Kubernetes CRD): Defines a pool of "warm" compute + capacity. It manages a fleet of standby worker pods initialized and ready + to receive resumed actor states. Optional `spec.template` fields configure + worker pod node selection, tolerations, priority class, and node affinity. - * **ActorTemplate**: An immutable definition of an actor-version. It - encapsulates the container image, configuration, and environment required - to generate a "golden" snapshot. + * **ActorTemplate** (ate API resource): An immutable definition of an + actor-version. It encapsulates the container image, configuration, and + environment required to generate a "golden" snapshot. ActorTemplates are + created and managed through the substrate gRPC API (e.g. `kubectl ate + create actor-template`) and stored in the control-plane state store; they + are not Kubernetes objects. ### Dynamic Instance State (Database-based) @@ -249,21 +252,22 @@ and Performance: lookups and atomic worker assignments that bypass the eventual consistency and variable latency of standard Kubernetes API servers. - 3. **Governance**: Using Kubernetes objects for the environment (WorkerPools - and Templates) allows platform teams to apply familiar RBAC, auditing, - and policy enforcement to the underlying infrastructure. + 3. **Governance**: Keeping infrastructure resources (WorkerPools, + SandboxConfigs) as Kubernetes objects allows platform teams to apply + familiar RBAC, auditing, and policy enforcement to the underlying + infrastructure, while workload definitions (ActorTemplates) are managed + through the substrate API instead, which authenticates callers itself + but does not yet implement authorization (see + [authentication.md](authentication.md)). ### Resource Model -The CRDs and control-plane records described above, with their relationships and -multiplicities (UML class diagram): +The Kubernetes resources and control-plane records described above, with their +relationships and multiplicities (UML class diagram): ```mermaid classDiagram namespace kube-apiserver { - class ActorTemplate { - <> - } class WorkerPool { <> } @@ -275,6 +279,9 @@ classDiagram } namespace ate-api-server { + class ActorTemplate { + <> + } class Actor { <> status @@ -287,10 +294,10 @@ classDiagram } } - ActorTemplate "1" --> "1" WorkerPool : workerPoolRef + ActorTemplate ..> WorkerPool : worker_selector (labels) WorkerPool ..> Deployment : reconciled by atecontroller Deployment "1" *-- "*" WorkerPod : manages - Actor ..> ActorTemplate : derived from + Actor "*" --> "1" ActorTemplate : actor_template Actor "0..1" --> "0..1" Worker : runs on Worker "1" --> "1" WorkerPod : maps to ``` diff --git a/docs/csi-volumes.md b/docs/csi-volumes.md index 1d41bf62c0..253bd9a704 100644 --- a/docs/csi-volumes.md +++ b/docs/csi-volumes.md @@ -66,10 +66,10 @@ External volumes are declared on the `ActorTemplate` resource. For complete deta To attach a CSI volume to an actor: -1. Define the volume under `spec.volumes` with an `externalVolumeTemplate`. -2. Mount the volume inside one or more containers under `spec.containers[].volumeMounts`. +1. Define the volume under `volumes` with an `externalVolumeTemplate`. +2. Mount the volume inside one or more containers under `containers[].volumeMounts`. -#### `spec.volumes[]` +#### `volumes[]` ```yaml volumes: @@ -83,7 +83,7 @@ volumes: * `externalVolumeTemplate.capacity`: Quantity string representing the requested volume size (e.g. `1Gi`, `50Gi`). * `externalVolumeTemplate.storageClassName`: Name of a Kubernetes `StorageClass` present in the cluster whose `provisioner` matches a registered `CSIDriverConfig`. -#### `spec.containers[].volumeMounts[]` +#### `containers[].volumeMounts[]` ```yaml volumeMounts: @@ -91,11 +91,11 @@ volumeMounts: mountPath: /var/data ``` -* `name`: Must match the declared `spec.volumes[].name`. +* `name`: Must match the declared `volumes[].name`. * `mountPath`: Unix path inside the container sandbox where the volume will be mounted. > [!NOTE] -> All declared volumes in `spec.volumes` must be mounted by at least one container. +> All declared volumes in `volumes` must be mounted by at least one container. --- diff --git a/docs/dev/code-layout.md b/docs/dev/code-layout.md index 1a6c7b8dea..3310f15684 100644 --- a/docs/dev/code-layout.md +++ b/docs/dev/code-layout.md @@ -48,7 +48,7 @@ Each subdirectory of `cmd/` corresponds to one compiled binary: | Directory | Binary / Purpose | |----------------------|-------------------------------------------------------| | `cmd/ateapi` | Control-plane API server (gRPC) | -| `cmd/atecontroller` | Kubernetes controller for WorkerPool/ActorTemplate | +| `cmd/atecontroller` | Kubernetes controller for WorkerPools (and network/egress resources) | | `cmd/atelet` | Node supervisor (DaemonSet) | | `cmd/atenet` | Network proxy / Envoy external-processing server | | `cmd/ateom-gvisor` | In-pod gVisor container image entry point | diff --git a/docs/glossary.md b/docs/glossary.md index 202f4f2ab7..dd3eeb8d0c 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -5,36 +5,39 @@ This document defines the core terms used across Agent Substrate. For how the pieces fit together, see the [Architecture](architecture.md) and [API Guide](api-guide.md). -## Resources (declarative, Kubernetes CRDs) - -- **ActorTemplate**: the definition of an actor "class": the container image(s) - and snapshot configuration. Creating an `ActorTemplate` triggers creation of - a [Golden Snapshot](#snapshots). It is treated as immutable: you create a new - template for a new version rather than editing an existing one. It is - analogous to a Pod template, but for a checkpointable workload. - -- **WorkerPool**: declares warm compute capacity, a fleet of pre-started worker - pods. It is reconciled into a Kubernetes `Deployment` by the - [atecontroller](#components). - -- **SandboxConfig**: a cluster-scoped resource holding the sandbox binaries for - one runtime family (the gVisor `runsc` binary, or a micro-VM - kernel/firmware/config), plus the pause image for the sandbox's root - container. A `WorkerPool` resolves its sandbox from the config it names, or - from the cluster default for its class, so one config pins the runtime version - for many templates. +## Resources (declarative) + +- **ActorTemplate** (ate API resource): the definition of an actor "class": + the container image(s) and snapshot configuration. Creating an + `ActorTemplate` triggers creation of a [Golden Snapshot](#snapshots). It is + treated as immutable: you create a new template for a new version rather + than editing an existing one. It is analogous to a Pod template, but for a + checkpointable workload. ActorTemplates are created and managed through the + substrate gRPC API (e.g. `kubectl ate create actor-template`) and stored in + the control-plane database; they are not Kubernetes objects. + +- **WorkerPool** (Kubernetes CRD): declares warm compute capacity, a fleet of + pre-started worker pods. It is reconciled into a Kubernetes `Deployment` by + the [atecontroller](#components). + +- **SandboxConfig** (Kubernetes CRD): a cluster-scoped resource holding the + sandbox binaries for one runtime family (the gVisor `runsc` binary, or a + micro-VM kernel/firmware/config), plus the pause image for the sandbox's + root container. A `WorkerPool` resolves its sandbox from the config it + names, or from the cluster default for its class, so one config pins the + runtime version for many templates. ## Records (dynamic state, in the control-plane store) -These are not Kubernetes objects; they live in the control-plane database -because they change too frequently for etcd. +These live in the control-plane database because they change too frequently +for etcd. - **Atespace**: the isolation boundary an Actor belongs to, and the first half of its identity: an Actor is addressed by `(atespace, name)`, so the same name can exist in two atespaces. Atespaces are global-scoped, not Kubernetes - namespaces, and are distinct from the namespace an `ActorTemplate` lives in. - One must exist before any Actor can be created in it, and it can only be - deleted once empty. + namespaces — an `ActorTemplate` also lives in an atespace, while + `WorkerPool`s live in Kubernetes namespaces. One must exist before any + Actor can be created in it, and it can only be deleted once empty. - **Actor**: a single instance derived from an `ActorTemplate`, identified by a DNS-1123 name. It is the unit that is suspended and resumed, and it moves