ci(internal-workflows): relocate the watchdog fixture

.github/workflows/test-watchdog.yaml

@@ -14,7 +14,7 @@ on:
   pull_request:
     paths:
     - .github/workflows/test-watchdog.yaml
-    - templates/kubernetes/homelab-workspace/script-memory-watchdog*.sh
+    - templates/kubernetes/homelab-workspace/scripts/script-memory-watchdog*.sh
   workflow_dispatch:
 
 concurrency:
@@ -40,4 +40,4 @@ jobs:
       # Exits non-zero when any assertion fails, which is what fails the build.
       # Needs nothing but bash and a writable TMPDIR: builds its fixtures under
       # mktemp -d, touches no network, finishes in a couple of seconds.
-      run: ./templates/kubernetes/homelab-workspace/script-memory-watchdog-test.sh
+      run: ./templates/kubernetes/homelab-workspace/scripts/script-memory-watchdog-test.sh

CLAUDE.md

@@ -35,7 +35,7 @@ CI (`.github/workflows/lint.yaml`) runs the same checks per file-type via reusab
 A separate workflow, `.github/workflows/test-watchdog.yaml`, runs the one thing here that's a test rather than a linter: its `watchdog` job runs `script-memory-watchdog-test.sh` and fails the build on a failed assertion. It's repo-local rather than a reusable workflow because `ppat/github-workflows` has nothing for "execute a test script", and the suite needs only bash and a writable `TMPDIR`:
 
 ```bash
-./templates/kubernetes/homelab-workspace/script-memory-watchdog-test.sh
+./templates/kubernetes/homelab-workspace/scripts/script-memory-watchdog-test.sh
 ```
 
 There is no local way to build/publish the image against the private registry cache (`test-image.yaml` needs the Tailscale-routed credentials CI has), but the template test path needs neither secrets nor CI: `test-template.yaml`'s Kind/Compose/`coder` sequence runs the same on a laptop — see [TESTING.md](TESTING.md) for the local recipe and for how a change actually gets exercised, and the **Release flow** section below for how it ships for real.
@@ -77,11 +77,12 @@ Quick orientation map — for what each piece is *for* and the decisions behind
 | `env.tf` | `coder_env` resources exposed to the agent |
 | `scripts.tf` | `coder_script` resources — the memory watchdog daemon and the weekly `vscode-server` GC schedule |
 | `variables.tf` | Publish-time image, storage-class, and dotfiles-script controls |
-| `script-agent-startup.sh` / `script-prepare-workspace.sh` | Scripts run on agent/workspace startup |
-| `script-container-entrypoint.sh` | The workspace container's `command`. Wipes `/tmp` and `exec`s Coder's generated `/scripts/workspace-init.sh` — the wipe must precede the agent, see the gotcha below |
-| `script-memory-watchdog.sh` | Userspace memory watchdog — see [DESIGN.md](DESIGN.md#design-tensions-and-decisions). It bounds the **standing population of restartable helpers** against a fixed **2048 MiB envelope** for the VS Code tree (per-role shares of it, PSS, ten-minute dwell) and records every per-process sweep. It does **not** try to prevent an acute OOM. `memory_watchdog_mode` selects `observe` / `enforce` (the default; arms every role). `enforce-all` is retired and honoured as `enforce` |
-| `script-memory-watchdog-test.sh` | Fixture tests for the watchdog's envelope and shares, process selection, dwell, the oversize rule and the kill-rate report. Run by hand (`./script-memory-watchdog-test.sh`) and by the `watchdog` job in `.github/workflows/test-watchdog.yaml`. `kill` is shadowed by a function throughout — the fixture pids are real pids in whatever container runs the suite |
-| `script-vscode-server-gc.sh` | Weekly GC of `~/.vscode-server` (interrupted downloads, superseded server versions/extensions, orphaned CLI binaries — see the script's own header for the exact signal per class, and the `coder_script.vscode_server_gc` comment in `scripts.tf` for why it's template-owned rather than dotfiles-owned) |
+| `scripts/script-agent-startup.sh` / `scripts/script-prepare-workspace.sh` | Scripts run on agent/workspace startup |
+| `scripts/script-container-entrypoint.sh` | The workspace container's `command`. Wipes `/tmp` and `exec`s Coder's generated `/scripts/workspace-init.sh` — the wipe must precede the agent, see the gotcha below |
+| `scripts/script-memory-watchdog.sh` | Userspace memory watchdog — see [DESIGN.md](DESIGN.md#design-tensions-and-decisions). It bounds the **standing population of restartable helpers** against a fixed **2048 MiB envelope** for the VS Code tree (per-role shares of it, PSS, ten-minute dwell) and records every per-process sweep. It does **not** try to prevent an acute OOM. `memory_watchdog_mode` selects `observe` / `enforce` (the default; arms every role). `enforce-all` is retired and honoured as `enforce` |
+| `scripts/script-memory-watchdog-test.sh` | Fixture tests for the watchdog's envelope and shares, process selection, dwell, the oversize rule and the kill-rate report. Run by hand (`./scripts/script-memory-watchdog-test.sh`) and by the `watchdog` job in `.github/workflows/test-watchdog.yaml`. `kill` is shadowed by a function throughout — the fixture pids are real pids in whatever container runs the suite |
+| `scripts/script-vscode-server-gc.sh` | Weekly GC of `~/.vscode-server` (interrupted downloads, superseded server versions/extensions, orphaned CLI binaries — see the script's own header for the exact signal per class, and the `coder_script.vscode_server_gc` comment in `scripts.tf` for why it's template-owned rather than dotfiles-owned) |
+| `config/supervisord.conf` | Native Supervisor configuration mounted alongside the scripts |
 
 **Image** (`images/homelab-workspace/Dockerfile`): three build stages — `base` (minimal bootstrap deps) → `system-base` (`unminimize` + full interactive toolset) → final stage (env vars into `/etc/environment`, fixed-UID/GID `coder` user, `USER coder`). All `apt`-touching `RUN` steps use BuildKit cache mounts — match that pattern when adding packages.
 

DESIGN.md

@@ -48,7 +48,7 @@ flowchart TB
 
 - **Coder platform** — the control plane that runs these templates is itself deployed to the cluster elsewhere: the Helm release in [`homelab-ops-kubernetes-apps`](../homelab-ops-kubernetes-apps/apps/subsystems/coder/helm-release-coder.yaml) and the Flux Kustomization that wires it into the homelab cluster in [`homelab-ops-kubernetes-clusters`](../homelab-ops-kubernetes-clusters/clusters/homelab/kustomizations/apps-coder.yaml). This repo produces what runs *on* that platform, not the platform itself.
 - **Image** ([`Dockerfile`](images/homelab-workspace/Dockerfile)) — the baseline every workspace needs and nothing more. Baked in so startup is fast and the result reproducible; the cost of a rebuild-and-release is only worth paying for things wanted everywhere.
-- **Template** ([`deployment.tf`](templates/kubernetes/homelab-workspace/deployment.tf), [`script-prepare-workspace.sh`](templates/kubernetes/homelab-workspace/script-prepare-workspace.sh)) — builds the pod and closes the gap between the baseline image and a usable workspace: a small set of extra apt packages (build toolchains and the like — left out of the image because they're large and only occasionally needed, yet only sensibly obtained as apt packages) and Homebrew installed and prepared so the layer above has something to build on.
+- **Template** ([`deployment.tf`](templates/kubernetes/homelab-workspace/deployment.tf), [`script-prepare-workspace.sh`](templates/kubernetes/homelab-workspace/scripts/script-prepare-workspace.sh)) — builds the pod and closes the gap between the baseline image and a usable workspace: a small set of extra apt packages (build toolchains and the like — left out of the image because they're large and only occasionally needed, yet only sensibly obtained as apt packages) and Homebrew installed and prepared so the layer above has something to build on.
 - **Dotfiles** ([`dotfiles`](../dotfiles)) — at provision time the operator's dotfiles configure the shell/environment and install the actual day-to-day tooling via Homebrew, mise, and aqua. This is per-operator and changes constantly, so it lives with the operator rather than in the template.
 
 The rule that ties the layers together: a package or tool belongs in the *lowest* layer that still makes sense for it. Universal and stable → image. Occasionally-needed, apt-only, and too heavy to bake in → the template's `system_packages` parameter. Personal, fast-moving, or not an apt package → dotfiles. This is what keeps the image small and reproducible while still letting one-off needs be self-served — the same trade examined from the package angle in [Reproducible image vs. self-service packages](#design-tensions-and-decisions) below.
@@ -79,7 +79,7 @@ The rule that ties the layers together: a package or tool belongs in the *lowest
 
 **A userspace memory watchdog that bounds the standing population of helper processes.** The workspace pod has a hard memory limit and no way to enforce anything below it: `/sys/fs/cgroup` is mounted read-only, `cgroup.subtree_control` is empty, the cgroup namespace is private and the workspace user has no capabilities, so `memory.high` and a child cgroup both need `privileged: true` or a read-write host mount — exactly what *Unprivileged by default* above exists to prevent. Until recently `memory.oom.group` was `1`, so a cgroup OOM killed every process in the container together and `oom_score_adj` could not influence a victim selection that no longer happened. The kubelet now sets `singleProcessOOMKill`, so a container created since that change gets `memory.oom.group = 0` and an OOM kills only the offending process — verified from inside a pod created after the rollout, while a container predating it still reads `1` because the value is fixed when the container is created. That setting lives on the nodes and in no repository (`ppat/homelab-ops-kubernetes-clusters#948`), so it is a property of node state rather than a guarantee this template can rely on.
 
-[`script-memory-watchdog.sh`](templates/kubernetes/homelab-workspace/script-memory-watchdog.sh) does **not** try to prevent that OOM, and the earlier version of this section, which said it did, was wrong on the evidence. The recorded kills are spikes: 70–220 MB/s, idle to dead inside a minute, with an agent session or `node` named as the victim in the kernel log and never a VS Code process. A poll loop cannot win that race — a generic biggest-RSS killer beats the kernel only at a 0.3 s interval, loses at 0.5 s, and under the `oom.group = 1` that then applied was killed by the very event it lost. The graded shedding ladder that used to live here was exercised against a real spike, climbed correctly, and logged `no-candidates`: the runaway was not in the tree it managed, and the whole editor tree it *could* have shed is ~0.7 GiB, or five seconds of that growth rate. The ladder has been removed rather than tuned, and the measurement it did well — unreclaimable memory, pressure, refault and reclaim rates — has been kept.
+[`script-memory-watchdog.sh`](templates/kubernetes/homelab-workspace/scripts/script-memory-watchdog.sh) does **not** try to prevent that OOM, and the earlier version of this section, which said it did, was wrong on the evidence. The recorded kills are spikes: 70–220 MB/s, idle to dead inside a minute, with an agent session or `node` named as the victim in the kernel log and never a VS Code process. A poll loop cannot win that race — a generic biggest-RSS killer beats the kernel only at a 0.3 s interval, loses at 0.5 s, and under the `oom.group = 1` that then applied was killed by the very event it lost. The graded shedding ladder that used to live here was exercised against a real spike, climbed correctly, and logged `no-candidates`: the runaway was not in the tree it managed, and the whole editor tree it *could* have shed is ~0.7 GiB, or five seconds of that growth rate. The ladder has been removed rather than tuned, and the measurement it did well — unreclaimable memory, pressure, refault and reclaim rates — has been kept.
 
 What a poll loop is good at is growth measured in MB per *minute*, and that drift is the real, daily problem: the operator was policing it by hand for months, repeatedly killing VS Code to save agent sessions, with a python MCP server holding 1.66 GB at one of the kills. So the watchdog now keeps the resting population of **restartable helpers** inside per-role budgets. The goal is runway rather than rescue: when a spike does arrive, it starts from as much free memory as the pod can offer. Every process it may signal has a supervisor — VS Code respawns its own forks and language servers, an agent session respawns its MCP servers — so a wrong kill costs a reload, not a session, and that asymmetry is what licenses being aggressive.
 

TESTING.md

@@ -105,7 +105,7 @@ no `-f` needed.
 
 Everything else here is declarative and is covered by the stages above. `script-memory-watchdog.sh` decides at runtime whether to kill a process, so it gets two things neither lint nor a template push can provide:
 
-- **Fixtures** — `./templates/kubernetes/homelab-workspace/script-memory-watchdog-test.sh`, also run by the `watchdog` job in `.github/workflows/test-watchdog.yaml`. The suite is built around negative assertions paired with the mutation that must flip them: "it did not kill the agent session" proves nothing unless removing one rule makes it kill the agent session. Two of its own safety properties matter as much as its assertions: `kill` is shadowed by a function throughout, because the fixture pids name real processes in whatever container runs the suite; and every load of the watchdog redirects its stdout emission to a temporary file, with the suite exiting outright if that seam did not take — the real default is the container's own stdout, and a run without the seam has already put fixture kill lines into a live workspace's log stream.
+- **Fixtures** — `./templates/kubernetes/homelab-workspace/scripts/script-memory-watchdog-test.sh`, also run by the `watchdog` job in `.github/workflows/test-watchdog.yaml`. The suite is built around negative assertions paired with the mutation that must flip them: "it did not kill the agent session" proves nothing unless removing one rule makes it kill the agent session. Two of its own safety properties matter as much as its assertions: `kill` is shadowed by a function throughout, because the fixture pids name real processes in whatever container runs the suite; and every load of the watchdog redirects its stdout emission to a temporary file, with the suite exiting outright if that seam did not take — the real default is the container's own stdout, and a run without the seam has already put fixture kill lines into a live workspace's log stream.
 - **A live drill on a disposable workspace with workspace-owned storage**, which can be wrecked freely. Fixtures cannot answer whether a real process tree classifies correctly, whether a supervisor really does respawn what was killed, or whether the respawned process comes back inside its share (a drift the watchdog should keep policing) or above it (an `oversize` it should report and leave alone). The drill that has been run: a stand-in session root (`exec -a claude bash`, to exercise the shebang branch of the claude-root guard rather than the trivial compiled-binary one) with an over-budget helper that a supervisor respawns, a second helper inside its budget, and a process detached into its own session with `setsid` — the same mechanism Claude Code uses to detach a Bash tool call — larger than both.
 
 Neither replaces the other, and the live one is where every defect that mattered in this component has been found.