From afad11507acc10a1b51f4ccff4324c64a7ce42ac Mon Sep 17 00:00:00 2001 From: Daniel Casota <14890243+dcasota@users.noreply.github.com> Date: Mon, 31 Aug 2026 12:34:18 +0200 Subject: [PATCH 1/2] staging: add vm-lab - Photon OS VM provisioning for VMware Workstation End-to-end: VMX template, boot VMDK, kickstart, unattended install, SSH access, teardown, and the verification scripts that say whether each stage actually did what it claims. Defaults: 2 vCPU, 4 GB RAM, 50 GB thin disk in a SINGLE FILE (monolithicSparse). Verified by creating one - a fresh 50 GB disk is one file of 6,619,136 bytes with the extent line RW 104857600 SPARSE referencing itself. The size is a ceiling, not an allocation. Two things the README documents because each has cost a day elsewhere: * SSH access is decided at ISO BUILD time. SPAGAT_OPERATOR_AUTHORIZED_KEY and IPHASE6_TEST_SSH_PUBKEY both default to empty and the root hash to '*', so a build where neither was exported ships an empty authorized_keys and a locked root - the serial console is the only way in, with no post-install fix. * The install CDROM must be on SATA: the linux-mok kernel has no IDE CDROM driver, so on ide1:0 the installer boots but mount /mnt/media finds no /dev/sr0, which reads like a corrupt ISO. Every script was executed, not just written. Four defects were found and fixed in the process, all the same family - a check that looked like it passed while proving nothing: 1. Both .ps1 files were unrunnable under Windows PowerShell 5.1. They parsed under pwsh 7 (UTF-8); 5.1 reads .ps1 as ANSI without a BOM, so an em-dash corrupted a string literal. Now ASCII-only. 2. The env parser kept trailing comments, so BOOT_DISK_ADAPTER resolved to 'lsilogic" # what the initrd expects' and would have been passed to vmware-vdiskmanager verbatim. bash strips those for free, which is why the WSL-side test passed and hid it. 3. A config that could not be loaded did not stop the script - it printed a healthy-looking first section then died on unbound variables. Now fails closed with exit 78. 4. grep -c PAT || echo 0 emitted "0\n0" (grep prints 0 AND exits 1). Replaced with n=$(grep -c ...) || n=0. Snapshot of deploy/vm-lab/ from dcasota/SpagatLibrarian-Appliance @ b7ec6954; see staging/vm-lab/PROVENANCE.md. kickstart/check-drift.sh works in both homes and states which mode it used. --- staging/README.md | 16 + staging/vm-lab/.gitattributes | 24 ++ staging/vm-lab/PROVENANCE.md | 55 +++ staging/vm-lab/README.md | 372 ++++++++++++++++++ .../vm-lab/config/spagat-smoke.vmx.template | 117 ++++++ staging/vm-lab/config/vm-lab.env | 92 +++++ staging/vm-lab/kickstart/EXPECTED-SHA256 | 1 + staging/vm-lab/kickstart/UPSTREAM-README.md | 149 +++++++ staging/vm-lab/kickstart/check-drift.sh | 117 ++++++ .../photon-appliance.ks.template.json | 63 +++ staging/vm-lab/scripts/00-preflight.sh | 120 ++++++ staging/vm-lab/scripts/10-create-vm.ps1 | 158 ++++++++ staging/vm-lab/scripts/20-make-ssh-key.sh | 98 +++++ staging/vm-lab/scripts/30-install-from-iso.sh | 157 ++++++++ staging/vm-lab/scripts/40-check-staging.sh | 217 ++++++++++ staging/vm-lab/scripts/50-verify-boot.sh | 99 +++++ staging/vm-lab/scripts/60-ssh.sh | 104 +++++ staging/vm-lab/scripts/90-teardown.ps1 | 132 +++++++ 18 files changed, 2091 insertions(+) create mode 100644 staging/vm-lab/.gitattributes create mode 100644 staging/vm-lab/PROVENANCE.md create mode 100644 staging/vm-lab/README.md create mode 100644 staging/vm-lab/config/spagat-smoke.vmx.template create mode 100644 staging/vm-lab/config/vm-lab.env create mode 100644 staging/vm-lab/kickstart/EXPECTED-SHA256 create mode 100644 staging/vm-lab/kickstart/UPSTREAM-README.md create mode 100644 staging/vm-lab/kickstart/check-drift.sh create mode 100644 staging/vm-lab/kickstart/photon-appliance.ks.template.json create mode 100644 staging/vm-lab/scripts/00-preflight.sh create mode 100644 staging/vm-lab/scripts/10-create-vm.ps1 create mode 100644 staging/vm-lab/scripts/20-make-ssh-key.sh create mode 100644 staging/vm-lab/scripts/30-install-from-iso.sh create mode 100644 staging/vm-lab/scripts/40-check-staging.sh create mode 100644 staging/vm-lab/scripts/50-verify-boot.sh create mode 100644 staging/vm-lab/scripts/60-ssh.sh create mode 100644 staging/vm-lab/scripts/90-teardown.ps1 diff --git a/staging/README.md b/staging/README.md index eb42074b..7f0f2b81 100644 --- a/staging/README.md +++ b/staging/README.md @@ -4,6 +4,22 @@ This subdirectory contains early scripts and prototypes kept for staging purpose ## Contents +### vm-lab/ +End-to-end provisioning of a Photon OS VM on VMware Workstation: the VMX +template (every non-default key annotated with the failure it prevents), the +boot VMDK, the kickstart, an unattended install, SSH access, teardown, and the +verification scripts that say whether each stage actually did what it claims. +Defaults to **2 vCPU / 4 GB RAM / 50 GB thin disk in a single file** +(`monolithicSparse` — the size is a ceiling, not an allocation: a fresh disk +is ~6 MB and grows on use). Overridable per run. + +Two things it documents that are easy to lose a day to: SSH access is decided +at **ISO build time** (the key variables default to empty, so a normal build +ships an empty `authorized_keys` and a locked root), and the install CDROM must +be on SATA because the `linux-mok` kernel has no IDE CDROM driver. See +[vm-lab/README.md](vm-lab/README.md); provenance and the kickstart-drift check +are in [vm-lab/PROVENANCE.md](vm-lab/PROVENANCE.md). + ### install-sizes-calc/ Package install size estimation for Photon OS, organized into two approaches: - **shell-based/** -- `DynamicSizeCalculation.sh`: A Bash script that uses `tdnf repoquery` and `tdnf info` to query installed package sizes without downloading, then calculates totals with a configurable buffer and compression ratio estimate. diff --git a/staging/vm-lab/.gitattributes b/staging/vm-lab/.gitattributes new file mode 100644 index 00000000..c23acbce --- /dev/null +++ b/staging/vm-lab/.gitattributes @@ -0,0 +1,24 @@ +# Line endings are load-bearing here. +# +# The shell scripts run under WSL/Linux. If a Windows checkout with +# core.autocrlf=true gives them CRLF, bash fails on the very first line with +# $'\r': command not found +# which is a confusing way to learn about line endings. Pin them to LF so the +# scripts work straight out of a checkout on either platform, without the +# `tr -d '\r' < script > /tmp/x.sh` dance. +*.sh text eol=lf +*.env text eol=lf +*.json text eol=lf +EXPECTED-SHA256 text eol=lf + +# The VMX template is read by PowerShell, which normalises to CRLF itself when +# it writes the real .vmx (VMware wants CRLF). The template can stay LF. +*.template text eol=lf + +# PowerShell scripts are Windows-native; CRLF is what an editor there expects. +# NOTE: these files are deliberately ASCII-only - Windows PowerShell 5.1 reads +# .ps1 as ANSI unless the file has a UTF-8 BOM, so a non-ASCII character +# corrupts a string literal and yields a cascade of misleading parse errors. +*.ps1 text eol=crlf + +*.md text diff --git a/staging/vm-lab/PROVENANCE.md b/staging/vm-lab/PROVENANCE.md new file mode 100644 index 00000000..31cbc4f2 --- /dev/null +++ b/staging/vm-lab/PROVENANCE.md @@ -0,0 +1,55 @@ +# Provenance + +`staging/vm-lab/` is a snapshot of `deploy/vm-lab/` from the SPAGAT-Librarian +appliance repository. + +``` +source repo : dcasota/SpagatLibrarian-Appliance +source path : deploy/vm-lab/ +authored at : b7ec6954e03ebe1c1b134bd5d8267c402848bdb0 +snapshot date : 2026-08-31 +``` + +**What that commit id means.** `deploy/vm-lab/` was authored alongside +`b7ec6954` but is not itself committed in the source repo yet, so +`photonos-scripts/staging/vm-lab/` is currently its published home. The id +identifies the appliance tree these scripts were written and tested against — +in particular the kickstart template and the `spagat-vm-orchestrator` CLI +surface they wrap — not a commit that contains this directory. + +## The kickstart copy + +`kickstart/photon-appliance.ks.template.json` is a **byte-exact** copy of +`src/tools/iso-build/iso-phase6-kickstart-template.cfg` in the source repo, +which is the file the ISO build actually consumes. + +``` +sha256 at snapshot time: ac5a3a5c3f87e392af927d070052f0edf86d178fb70a427e51fef4572c67f7e1 +``` + +`kickstart/check-drift.sh` verifies this and states which mode it used: + +- **Mode 1** — a SPAGAT checkout is reachable (`SPAGAT_REPO=/path/to/repo`, or + this directory sitting inside that repo): it diffs against the live + canonical file and catches drift in **either** direction. +- **Mode 2** — standalone, as here: it falls back to the hash above. That + still detects an edited local copy, but **cannot** tell you whether the + upstream template has moved on. The script says so rather than implying it + proved more than it did. + +If neither the canonical file nor `EXPECTED-SHA256` is available it exits +non-zero — "cannot check" must never read as "fine". + +A convenience copy that can silently diverge from the real thing is worse than +no copy, because you would reason about a kickstart the build never uses. + +## Refreshing the snapshot + +```bash +SPAGAT=/path/to/SpagatLibrarian-Appliance +cp "$SPAGAT/src/tools/iso-build/iso-phase6-kickstart-template.cfg" \ + staging/vm-lab/kickstart/photon-appliance.ks.template.json +sha256sum staging/vm-lab/kickstart/photon-appliance.ks.template.json \ + | cut -d' ' -f1 > staging/vm-lab/kickstart/EXPECTED-SHA256 +# then update the sha256 and the commit id above +``` diff --git a/staging/vm-lab/README.md b/staging/vm-lab/README.md new file mode 100644 index 00000000..fc986ddb --- /dev/null +++ b/staging/vm-lab/README.md @@ -0,0 +1,372 @@ +# vm-lab — build a SPAGAT appliance VM on VMware Workstation, end to end + +Everything needed to go from *nothing* to *a running, verified, SSH-reachable +Photon OS appliance VM*: the VMX settings, the disk geometry, the kickstart, +the install automation, SSH access, and the checks that tell you whether each +stage actually did what it claims. + +Read [The four things that actually bite](#the-four-things-that-actually-bite) +before your first run. Each one has cost days elsewhere in this project. + +**Defaults:** 2 vCPU, 4 GB RAM, and a **50 GB thin disk in a single file**. +The disk is `monolithicSparse`, so 50 GB is a ceiling rather than an +allocation — a fresh one is ~6 MB and grows only as the guest writes. + +**This directory lives in two places.** It is developed in the +SPAGAT-Librarian appliance repo at `deploy/vm-lab/` and snapshotted into +`photonos-scripts` at `staging/vm-lab/`. `PROVENANCE.md` records which commit +a snapshot came from, and `kickstart/check-drift.sh` behaves correctly in +both (see [The kickstart](#the-kickstart)). + +--- + +## Layout + +``` +vm-lab/ +├── README.md ← you are here +├── PROVENANCE.md which commit this snapshot came from +├── config/ +│ ├── vm-lab.env single source of paths, sizes, IP, MAC +│ └── spagat-smoke.vmx.template VMX with every non-default key explained +├── kickstart/ +│ ├── photon-appliance.ks.template.json byte-exact copy of the build's template +│ ├── EXPECTED-SHA256 that copy's hash, for the standalone check +│ ├── UPSTREAM-README.md the template's own upstream docs +│ └── check-drift.sh fails if the copy diverges +└── scripts/ + ├── 00-preflight.sh can this host do the job? + ├── 10-create-vm.ps1 VM dir + boot VMDK + VMX from template + ├── 20-make-ssh-key.sh keypair + the exports the ISO build needs + ├── 30-install-from-iso.sh unattended install + ├── 40-check-staging.sh did rootfs / ISO / guest actually get what I think? + ├── 50-verify-boot.sh is it alive, and did THIS boot do anything? + ├── 60-ssh.sh connect — and explain failures + └── 90-teardown.ps1 back to a fresh disk (stashes, never deletes) +``` + +Run the shell scripts from this directory, or export `VM_LAB_DIR=/path/to/vm-lab` +if you invoke them from elsewhere (for example after copying one to `/tmp` to +strip CRLF). They **refuse to run** rather than proceeding without their +config. + +**Which shell:** `.sh` runs in WSL (`wsl -d Ph5 -u dcaso`), `.ps1` runs on +Windows. That split is not stylistic — `vmware-vdiskmanager.exe` and the VMX +need Windows paths, while the orchestrator needs Linux paths. + +--- + +## Quick start + +```bash +# 0. Can this host do it? +./scripts/00-preflight.sh + +# 1. Create the VM (Windows side) +powershell -File scripts/10-create-vm.ps1 + +# 2. Decide SSH access — BEFORE the ISO is built. See "SSH access". +./scripts/20-make-ssh-key.sh +export SPAGAT_OPERATOR_AUTHORIZED_KEY='ssh-ed25519 AAAA... spagat-vm-lab@host' +export IPHASE6_INSTALL_STATIC_IP='192.168.225.140/24' + +# 3. Build the ISO in that SAME shell (the exports must be live) +make iso-test BUILD_MANIFEST=... OUTPUT_ISO=/home/dcaso/work/iso-out-/appliance.iso + +# 4. Prove the key actually landed in the ISO — do not assume +sudo ./scripts/40-check-staging.sh --iso /home/dcaso/work/iso-out-/appliance.iso + +# 5. Install +./scripts/30-install-from-iso.sh --iso /home/dcaso/work/iso-out-/appliance.iso + +# 6. Watch it +./scripts/50-verify-boot.sh + +# 7. Get in +./scripts/60-ssh.sh +``` + +--- + +## The four things that actually bite + +### 1. SSH access is decided at ISO **build** time, not after install + +Both key variables default to **empty** in `iso-phase6`: + +| variable | default | effect | +|---|---|---| +| `SPAGAT_OPERATOR_AUTHORIZED_KEY` | `""` | `operator`'s `authorized_keys` ships **empty** | +| `IPHASE6_TEST_SSH_PUBKEY` | `""` | no appended second key | +| `SPAGAT_ROOT_PASSWORD_HASH` | `"*"` | root **locked** | + +So a build where nobody exported a key produces an appliance with correct +`0600` permissions on an **empty** `authorized_keys`, and no root password. +The only way in is the serial console. + +This is the whole of the long-running "the appliance refuses my SSH key" +symptom — not an onboarding gate, not a credential subsystem bug. A previous +session even hardcoded a throwaway public key into the template *with no +matching private key anywhere*, which then mis-diagnosed 24+ access failures. + +**There is no post-install fix.** Root is locked, so you cannot log in to add +a key. Export the variable and rebuild. + +`scripts/40-check-staging.sh --iso ` extracts the kickstart *out of the +finished ISO* and prints the keys it contains, so you verify rather than hope. + +### 2. `install-from-iso` only **edits** an existing `.vmx` + +The orchestrator has no `createvm` / `vdiskmanager` path. VM creation is a +manual step outside the automated loop — exactly the kind of step that +silently drifts. That is why `10-create-vm.ps1` builds from a pinned template +instead of VMware's "New VM" wizard. Half the BUG-N series traces to one VMX +key being wrong. + +The template pins, with the reason in a comment beside each: +`firmware=efi` · `uefi.secureBoot.enabled=FALSE` · `bios.bootOrder=hdd,cdrom` · +`sata0.present=TRUE` · `scsi0.virtualDev=lsilogic` · `ethernet0.virtualDev=e1000` · +the MAC **and** the BIOS UUID · `serial0.*` · `tools.syncTime=FALSE` · +`msg.autoAnswer=TRUE`. + +Two that catch people out: + +- **CDROM must be on SATA.** The `linux-mok` kernel has no IDE CDROM driver. + On `ide1:0` the installer boots but userspace `mount /mnt/media` finds no + `/dev/sr0`, and you get a `LABEL="PHOTON_SB_5.0"` not-found failure that + looks like a broken ISO. +- **The BIOS UUID must stay next to the MAC.** With + `ethernet0.addressType="generated"`, VMware *derives* the MAC from the UUID. + Drop the UUID and both regenerate, the `192.168.225.140` lease moves, and + every hardcoded address in the runbooks quietly points at nothing. + +### 3. The operator medium on `scsi0:1` is the credential channel — and is never regenerated here + +`operator-config.vmdk` + `operator-config-flat.vmdk` (250 MiB +`monolithicFlat`, `lsilogic`) carry the signed credential bundle the appliance +reads at every boot. Verified in `install.rs`: `install-from-iso` strips only +`ide1:0.` `sata0:0.` `sata0:1.` `sata0.` `ethernet0.` `msg.autoAnswer` +`bios.bootOrder` — it **never references `scsi0:1`**, so the medium survives +any number of reinstalls verbatim. + +- **Do not** pass `--efuse-vmdk` for it. That flag attaches an install-time + marker on `sata0:0` which finalize then *detaches*; the operator medium is a + persistent boot-time disk. +- Nothing in this directory creates or modifies it. `90-teardown.ps1` + explicitly preserves it and checks its size. +- Without it the appliance boots **keyless** — every credential consumer logs + `operator-config/credentials absent` and the onboarding wizard stays up. + That is correct behaviour, not a bug. + +### 4. Credentials never travel in the ISO + +The medium is the channel. Never bake a credential into an image. If you also +want a root password, `install-from-iso` requires **both** +`--root-password-file` *and* `--operator-medium-dir` — all-or-nothing, so a +half-configured run cannot leak a password into an image. Passing one alone is +a hard error, and the plaintext is hashed in memory (`$6$` SHA-512-crypt); +only the hash is ever written. + +Omit both and you get `BUG-N91: no --root-password-file supplied` in the log. +**That line is expected**, not a fault. + +--- + +## The scripts, in order + +### `00-preflight.sh` +Proves the host can do the job before anything is created: `vmrun` / +`vmware-vdiskmanager` present **and executable by you**, ≥20 GB free, the VM +not already running, the operator medium's size, whether a keypair exists. + +Prints measured values rather than OK/FAIL, because "tool missing" and "tool +present but unreadable by this user" need different fixes and look identical +in a boolean. Running as `spagat-runner` is called out explicitly — `vmrun.exe` +is mode 744 owned by `dcaso`, so that user gets `vmrun IO: Permission denied`. + +### `10-create-vm.ps1` +Creates the VM directory, a **50 GB thin single-file** `lsilogic` boot disk +(`monolithicSparse`, `RW 104857600 SPARSE`), and the VMX from the template. + +Refuses to overwrite an existing disk or VMX — re-provisioning goes through +`90-teardown.ps1` first. `-RefreshVmxOnly` regenerates just the VMX (backing +up the old one). Fails loudly if any `@@PLACEHOLDER@@` survives substitution. + +Because the disk is thin, 50 GB costs nothing up front: a fresh one is ~6 MB +and grows only as the guest writes. The headroom is wanted — a measured +appliance install passes 12 GB. The kickstart declares +`{ "mountpoint": "/", "size": 0 }` — grow to fill — so root takes the +remainder after `/boot/efi` 512 M and `/boot` 1 G. + +**Single file is deliberate.** `vmware-vdiskmanager -t 0` keeps the entire +disk in one `.vmdk` whose extent line references itself, so a teardown moves +exactly one file aside. `-t 1` stores the same data split across 2 GB extents +(`…-s001.vmdk`, `…-s002.vmdk`, …) — more files for the same bytes, and the +split form is the one that tends to leave orphaned extents behind. + +This script is ASCII-only on purpose. Windows PowerShell 5.1 reads `.ps1` as +ANSI without a BOM, so a single em-dash corrupts a string literal and produces +a cascade of misleading "missing closing brace" errors. `pwsh` 7 parses the +same file happily, so a parse check under 7 does **not** prove it runs under +5.1. + +### `20-make-ssh-key.sh` +Generates a disposable ed25519 lab keypair and prints the exact `export` +lines the ISO build needs. Reuses an existing key rather than silently +replacing one. This key is a lab convenience, **not** an operator credential. + +### `30-install-from-iso.sh` +Wraps `spagat-vm-orchestrator install-from-iso`. + +- Refuses if the VM is running rather than passing `--force` — other VMs on + the host may be live CI runners. +- **Verifies the ISO against its own `.sha256` sidecar before installing.** A + short write on a 9p→NTFS copy produces a subtly broken appliance. +- Points `serial0.fileName` at a per-build log so RCA output from different + images never interleaves, and records the existing log's length so only + bytes past that offset belong to this run. +- Passes the **Linux** path. `vmx-info` parses `/mnt/c/...` and fails on + `C:\...`; the orchestrator does the Windows translation itself. + +### `40-check-staging.sh` — the one to reach for when something is "weird" +Checks staging at all three places it happens: + +| section | question it answers | +|---|---| +| `--rootfs ` | what the ISO was *built from*: build label, binary count, `/opt/spagat/appliance-src`, the `/var/spagat/audit` mode | +| `--iso ` | the four hashes agree; **and the kickstart extracted from the ISO — SSH keys, static IP, root-locked state** | +| `--guest` | what the running system did: installer vs installed boot, build identity, the credential-injection chain, port 22 | + +Run section 2 **as root** — the sidecars are `0600 root`, and as another user +an unreadable file reads exactly like a hash mismatch. The script says so +rather than reporting a false MISMATCH. + +### `50-verify-boot.sh` +The only unambiguous liveness instrument while root is locked is **whether the +serial log grows**. Everything else is inference, and these three readings are +all false signals: + +- `vmrun list` **omits GUI-started VMs** — absence is not "off". +- CPU at `0.05` usually means the counter has not moved yet. Measure a delta + over ~25 s. +- A black screen at t≈130 s is normal; the TUI starts at t≈133 s. + +Also: `FAIL spagat-console: container not running` is a **red herring** — it +asserts on a retired container, not the live TUI. Grep for +`Started Spagat-Librarian Kanban TUI on tty1` instead. + +### `60-ssh.sh` +Connects, and when it fails says *why* — key present? host reachable? port 22 +open? — instead of leaving you with `connection refused`. Uses `BatchMode=yes` +so it never sits at a password prompt; root is locked, so there is no password +to give and an interactive prompt is always a dead end. + +### `90-teardown.ps1` +Returns the VM to a fresh-disk state. **Nothing is deleted** — files are +renamed `.stashed-` and recovery is a rename back. Requires +`-Confirm`. + +It stashes the *whole* chain (disk, snapshot deltas, `.vmsd`, `.vmsn`, NVRAM, +stale `.lck`), because if any piece survives, UEFI's removable-media fallback +finds the old ESP's `\EFI\BOOT\BOOTX64.EFI` and boots the **previous** image. +`bios.bootOrder` is ignored on EFI VMs, and deleting NVRAM alone does not help +— UEFI re-detects the disk. + +Preserved: the operator medium, every serial log, and (unless `-IncludeVmx`) +the VMX with its pinned MAC/UUID. + +--- + +## The kickstart + +`kickstart/photon-appliance.ks.template.json` is a **byte-exact copy** of +`src/tools/iso-build/iso-phase6-kickstart-template.cfg`, which is what the +build actually consumes. Run `kickstart/check-drift.sh` to prove they still +match — a convenience copy that silently diverges is worse than no copy, +because you would reason about a kickstart the build never uses. + +`check-drift.sh` has two modes and says which one it used: + +- **Mode 1** — the canonical file is reachable (inside the SPAGAT repo, or via + `SPAGAT_REPO=/path/to/SpagatLibrarian-Appliance`): it diffs against the live + file and detects drift in **either** direction. This is the real check. +- **Mode 2** — standalone (e.g. in `photonos-scripts/staging`): it falls back + to the hash in `EXPECTED-SHA256`. That still catches an edited local copy but + **cannot** see upstream moving on. The script states this limitation rather + than implying it proved more than it did, and exits non-zero if neither the + canonical file nor the hash is available — "cannot check" must never read as + "fine". + +Placeholders and where each value comes from: + +| placeholder | source | default | +|---|---|---| +| `{{ HOSTNAME }}` | `SPAGAT_HOSTNAME` | `spagat-librarian` | +| `{{ ROOT_PASSWORD_HASH }}` | `SPAGAT_ROOT_PASSWORD_HASH` | `*` (**locked**) | +| `{{ INSTALL_DISK }}` | `SPAGAT_INSTALL_DISK` | `/dev/sda` | +| `{{ SPAGAT_OPERATOR_AUTHORIZED_KEY }}` | env | **empty** | +| `{{ TEST_SSH_PUBKEY }}` | `IPHASE6_TEST_SSH_PUBKEY` | **empty** | +| `{{ INSTALL_STATIC_IP }}` | `IPHASE6_INSTALL_STATIC_IP` | `192.168.225.140/24` | +| `{{ WIZARD_PRESEED_TOML_B64 }}` | `IPHASE6_WIZARD_PRESEED_TOML` | empty | +| `{{ PAYLOAD_SHA256 }}` | computed from the overlay | — | + +What the postinstall does that matters for this lab: installs and enables +`sshd`, creates `operator` with a `0700 .ssh` and `0600 authorized_keys`, +writes the static-IP `systemd-networkd` unit, enables `logrotate.timer`, and +masks `tmp.mount` so `/tmp` is disk-backed rather than a RAM tmpfs (BUG-N135 — +a 4 GB VM's tmpfs `ENOSPC`'d on temp-heavy writes). + +--- + +## Configuration + +Everything lives in `config/vm-lab.env`. Override by exporting before a call: + +```bash +VM_NAME=my-lab BOOT_DISK_SIZE=30GB ./scripts/00-preflight.sh +``` + +Defaults: + +| setting | default | notes | +|---|---|---| +| `GUEST_VCPUS` | `2` | the appliance profile was measured at 4 | +| `BOOT_DISK_SIZE` | `50GB` | the standing maximum; thin, so it is a ceiling not an allocation | +| `BOOT_DISK_TYPE` | `0` | `monolithicSparse` — thin, **single file**. Do not use `1` (2 GB split extents) | +| `GUEST_MEM_MB` | `4096` | `/run` is a RAM tmpfs sized from this; 4 GB is the tested floor | + +For the full appliance profile: + +```bash +GUEST_VCPUS=4 powershell -File scripts/10-create-vm.ps1 +``` + +Other values you might legitimately change: `VM_NAME`, `SSH_KEY_NAME`. + +Values you should not change without a specific reason: `GUEST_MAC` and the +BIOS UUID in the VMX template (they hold the IP lease together), +`BOOT_DISK_ADAPTER=lsilogic`, and `OPERATOR_MEDIUM_FLAT_BYTES`. + +--- + +## Troubleshooting + +| symptom | almost always | +|---|---| +| `Permission denied (publickey)` | no key was baked into the ISO — §1 | +| `vmrun IO: Permission denied` | running as `spagat-runner`; use `dcaso` | +| installer boots but `mount /mnt/media` fails | CDROM landed on IDE, not SATA | +| the VM boots the *previous* image | a piece of the old disk chain or NVRAM survived — `90-teardown.ps1` | +| `DISKUTIL: sata0:1 capacity=0` | the ISO is on a `\\wsl$\` UNC path; copy it to a local Windows path | +| every credential consumer says "absent" | the operator medium is missing from `scsi0:1` | +| `BUG-N91: no --root-password-file supplied` | expected — root installs locked | +| `FAIL spagat-console: container not running` | red herring — check for the Kanban TUI line instead | +| a `grep` of the serial log finds nothing | the log has NUL bytes; **use `grep -a`** or it silently prints nothing | + +--- + +## Related + +- `src/tools/iso-build/iso-phase6-kickstart-template.cfg` — canonical kickstart +- `tools/spagat-rust/crates/spagat-vm-orchestrator/` — the install/verify tool +- `deploy/kickstart-photon-host.cfg` — the *host* kickstart, a different thing diff --git a/staging/vm-lab/config/spagat-smoke.vmx.template b/staging/vm-lab/config/spagat-smoke.vmx.template new file mode 100644 index 00000000..02b34852 --- /dev/null +++ b/staging/vm-lab/config/spagat-smoke.vmx.template @@ -0,0 +1,117 @@ +.encoding = "UTF-8" +displayName = "@@VM_NAME@@" + +# --------------------------------------------------------------------------- +# spagat appliance smoke VM — VMX template. +# +# Captured verbatim from the live spagat-smoke.vmx and parameterised. Every +# NON-DEFAULT key below is deliberate; half the BUG-N series came from one of +# them drifting. Do not "tidy" this file. +# +# firmware = efi HABv4 secureboot chain; BIOS boot will not work +# uefi.secureBoot.enabled=FALSE MOK enrolment happens in-guest, not in firmware +# bios.bootOrder = hdd,cdrom BUG-N3 / BUG-N66 — CDROM-first causes an +# install boot loop. install-from-iso flips this +# for the install window and restores it after. +# sata0.present = TRUE BUG-N16 — the linux-mok kernel has no IDE +# CDROM driver, so the install ISO MUST attach +# on SATA or `mount /mnt/media` finds no /dev/sr0 +# scsi0.virtualDev = lsilogic what the installed initrd expects +# ethernet0.virtualDev = e1000 trap-vmxnet3-pci-slot-reservation-fails +# ethernet0.generatedAddress pins the 192.168.225.140 lease — a fresh MAC +# changes the IP and breaks the runbooks +# serial0.* the ONLY diagnostic channel while root is +# locked; fileType=file + startConnected=TRUE +# tools.syncTime = FALSE guest clock must not be host-slaved +# msg.autoAnswer = TRUE trap-vmware-msg-autoanswer — otherwise a modal +# dialog silently blocks the unattended power-on +# memSize = 4096 /run tmpfs is sized from RAM (BUG-N119); the +# ISO also masks tmp.mount so /tmp is disk-backed +# --------------------------------------------------------------------------- + +config.version = "8" +virtualHW.version = "21" +guestOS = "other6xlinux-64" +firmware = "efi" +uefi.secureBoot.enabled = "FALSE" +numvcpus = "@@GUEST_VCPUS@@" +memSize = "@@GUEST_MEM_MB@@" + +# --- boot disk (scsi0:0) --------------------------------------------------- +scsi0.present = "TRUE" +scsi0.virtualDev = "lsilogic" +scsi0.pciSlotNumber = "16" +scsi0:0.present = "TRUE" +scsi0:0.fileName = "@@VM_NAME@@.vmdk" +scsi0:0.deviceType = "disk" +scsi0:0.redo = "" + +# --- operator medium (scsi0:1) — the credential channel -------------------- +# install-from-iso NEVER touches scsi0:1 (verified in install.rs), so this +# disk survives a reinstall verbatim. Do NOT pass --efuse-vmdk for it. +scsi0:1.present = "TRUE" +scsi0:1.fileName = "@@OPERATOR_MEDIUM_BASENAME@@.vmdk" +scsi0:1.deviceType = "disk" +scsi0:1.redo = "" + +# --- install media bus ----------------------------------------------------- +# sata0:1 is added by install-from-iso for the install window and removed +# afterwards. The BUS must exist up front. +sata0.present = "TRUE" +sata0.pciSlotNumber = "17" + +# --- network --------------------------------------------------------------- +ethernet0.present = "TRUE" +ethernet0.virtualDev = "e1000" +ethernet0.connectionType = "nat" +ethernet0.addressType = "generated" +ethernet0.pciSlotNumber = "18" +ethernet0.generatedAddress = "@@GUEST_MAC@@" +ethernet0.generatedAddressOffset = "0" + +# The BIOS UUID is pinned ON PURPOSE and must stay next to the MAC above. +# With addressType="generated", VMware DERIVES the MAC from this UUID — its +# last bytes (…49 53 65 1c) are the MAC's (…53:65:1c). Drop the UUID and +# VMware regenerates both, the 192.168.225.140 lease moves, and every +# hardcoded address in the runbooks silently points at nothing. +uuid.bios = "56 4d a0 3f 30 d3 c1 e8-72 40 ad 48 49 53 65 1c" +uuid.location = "56 4d a0 3f 30 d3 c1 e8-72 40 ad 48 49 53 65 1c" + +# --- serial console (the only diagnostic channel) -------------------------- +serial0.present = "TRUE" +serial0.fileType = "file" +serial0.startConnected = "TRUE" +serial0.fileName = "@@SERIAL_LOG_WIN@@" + +# --- boot order ------------------------------------------------------------ +bios.bootOrder = "hdd,cdrom" + +# --- trimmed hardware ------------------------------------------------------ +usb.present = "FALSE" +sound.present = "FALSE" +ehci.present = "FALSE" + +# --- misc ------------------------------------------------------------------ +powerType.powerOff = "soft" +powerType.suspend = "soft" +powerType.reset = "soft" +extendedConfigFile = "@@VM_NAME@@.vmxf" +virtualHW.productCompatibility = "hosted" +vmxstats.filename = "@@VM_NAME@@.scoreboard" +svga.vramSize = "268435456" +svga.guestBackedPrimaryAware = "TRUE" +numa.autosize.cookie = "40012" +numa.autosize.vcpu.maxPerVirtualNode = "4" +cpuid.coresPerSocket.cookie = "4" +vmotion.checkpointFBSize = "4194304" +vmotion.checkpointSVGAPrimarySize = "268435456" +vmotion.svga.mobMaxSize = "268435456" +vmotion.svga.graphicsMemoryKB = "262144" +monitor.phys_bits_used = "45" +softPowerOff = "FALSE" +workingDir = "." +tools.syncTime = "FALSE" +checkpoint.vmState.readOnly = "FALSE" +checkpoint.vmState = "" +cleanShutdown = "TRUE" +msg.autoAnswer = "TRUE" diff --git a/staging/vm-lab/config/vm-lab.env b/staging/vm-lab/config/vm-lab.env new file mode 100644 index 00000000..4bcd4f07 --- /dev/null +++ b/staging/vm-lab/config/vm-lab.env @@ -0,0 +1,92 @@ +# vm-lab configuration — the ONE file every script in this directory reads. +# +# Shell syntax so both bash (WSL) and the PowerShell scripts can parse it. +# Override any value by exporting it before calling a script, e.g. +# VM_NAME=my-lab ./scripts/10-create-vm.ps1 +# +# Paths use the form each consumer needs: +# *_WIN → Windows form (C:\...) for vmrun / vmware-vdiskmanager / VMX contents +# *_WSL → Linux form (/mnt/c/...) for the orchestrator and shell tooling +# The orchestrator itself takes the LINUX form and translates internally +# (wsl_paths::to_windows_style, BUG-N9 / #617). Passing it C:\... fails with +# "VMX I/O: No such file or directory". + +# ---------------------------------------------------------------- identity -- +VM_NAME="spagat-smoke" + +# ------------------------------------------------------------------- paths -- +VM_ROOT_WIN='C:\spagat-iso-build\vm' +VM_ROOT_WSL='/mnt/c/spagat-iso-build/vm' +VM_DIR_WIN="${VM_ROOT_WIN}\\${VM_NAME}" +VM_DIR_WSL="${VM_ROOT_WSL}/${VM_NAME}" + +VMWARE_DIR_WIN='C:\Program Files\VMware\VMware Workstation' +VMRUN_WSL="/mnt/c/Program Files/VMware/VMware Workstation/vmrun.exe" +VDISKMANAGER_WSL="/mnt/c/Program Files/VMware/VMware Workstation/vmware-vdiskmanager.exe" + +# Where ISO builds land on the build host (Ph5 WSL). +ISO_OUT_ROOT="/home/dcaso/work" + +# ------------------------------------------------------------------- disks -- +# DEFAULT: 50 GB thin, in a SINGLE FILE. +# +# 50 GB is the standing maximum (operator directive 2026-08-20). Because the +# disk is thin the nominal size is a CEILING, not an allocation: a fresh 50 GB +# disk is ~6 MB on disk and grows only as the guest writes. There is no cost +# to the headroom, and the full appliance profile needs it -- a measured +# install passes 12 GB (93 staged binaries, the appliance source tree under +# /opt/spagat/appliance-src, and /tmp deliberately disk-backed rather than a +# RAM tmpfs, BUG-N135). +# +# The kickstart declares { "mountpoint": "/", "size": 0 } = "grow to fill", so +# root takes whatever is left after /boot/efi 512M + /boot 1G. +# +# SINGLE FILE is BOOT_DISK_TYPE=0 -- see the type table below. Do not change it +# to 1 to "make it smaller": type 1 is the same data split across 2 GB chunks +# (spagat-smoke-s001.vmdk, -s002.vmdk, ...), which is more files, not less +# space, and the split form is the one that tends to leave orphaned extents +# behind after a teardown. +BOOT_DISK_SIZE="50GB" +# 50 GiB = 104857600 512-byte sectors -> descriptor `RW 104857600 SPARSE` +BOOT_DISK_ADAPTER="lsilogic" +# vmware-vdiskmanager -t: +# 0 = monolithicSparse thin, SINGLE FILE <-- what we use +# 1 = twoGbMaxExtentSparse thin, split into 2 GB extents +# 2 = monolithicFlat thick, descriptor + separate -flat.vmdk +# 3 = twoGbMaxExtentFlat thick, split +# Type 0 keeps the whole disk in one .vmdk whose extent line references +# itself, so a teardown moves exactly one file aside. +BOOT_DISK_TYPE="0" + +# The operator medium (credential channel, scsi0:1). 250 MiB monolithicFlat. +# NEVER regenerated by these scripts — see README "The operator medium". +OPERATOR_MEDIUM_BASENAME="operator-config" +OPERATOR_MEDIUM_FLAT_BYTES="262144000" + +# ------------------------------------------------------------------ guest --- +# The pinned MAC is what holds the 192.168.225.140 DHCP/NAT lease. Changing it +# very likely changes the IP and breaks every hardcoded reference in the +# runbooks. Change only with an explicit reason. +GUEST_MAC="00:0c:29:53:65:1c" +GUEST_STATIC_IP="192.168.225.140/24" +GUEST_GATEWAY="192.168.225.2" +GUEST_IP_BARE="192.168.225.140" +GUEST_MEM_MB="4096" +# DEFAULT: 2 vCPU. Enough for a stock Photon guest and cheap on a laptop. +# The SPAGAT appliance profile was measured at 4 -- raise it per-run with +# GUEST_VCPUS=4 ./scripts/10-create-vm.ps1 +GUEST_VCPUS="2" + +# ---------------------------------------------------------------- ssh/auth -- +# 🚨 SSH access is decided AT ISO BUILD TIME, not after install. Both of these +# default to EMPTY in iso-phase6, which ships an appliance with an empty +# authorized_keys and root LOCKED — i.e. no way in except the serial console. +# See README "SSH access". +SSH_KEY_DIR="${HOME}/.ssh" +SSH_KEY_NAME="spagat-vm-lab-ed25519" +SSH_USER="operator" + +# ------------------------------------------------------------------ serial -- +# The serial log is the ONLY diagnostic channel while root is locked. Name it +# per build so RCA logs from different images never interleave. +SERIAL_LOG_PREFIX="serial0" diff --git a/staging/vm-lab/kickstart/EXPECTED-SHA256 b/staging/vm-lab/kickstart/EXPECTED-SHA256 new file mode 100644 index 00000000..e40fcc0f --- /dev/null +++ b/staging/vm-lab/kickstart/EXPECTED-SHA256 @@ -0,0 +1 @@ +ac5a3a5c3f87e392af927d070052f0edf86d178fb70a427e51fef4572c67f7e1 photon-appliance.ks.template.json diff --git a/staging/vm-lab/kickstart/UPSTREAM-README.md b/staging/vm-lab/kickstart/UPSTREAM-README.md new file mode 100644 index 00000000..7b9e419f --- /dev/null +++ b/staging/vm-lab/kickstart/UPSTREAM-README.md @@ -0,0 +1,149 @@ +# `iso-phase6-kickstart-template.cfg` — README + +This sibling file documents `iso-phase6-kickstart-template.cfg` without using +underscore-prefixed keys inside the JSON itself (memory +`trap_kickstart_underscore_keys` — Photon installer's `_check_install_config()` +rejects any `_*` key and aborts the install). + +## Placeholders + +| Token | Substituted by phase 6 from… | Notes | +|---|---|---| +| `{{ HOSTNAME }}` | env `SPAGAT_HOSTNAME` (default `spagat-librarian`) | Hostname of the installed appliance. | +| `{{ ROOT_PASSWORD_HASH }}` | env `SPAGAT_ROOT_PASSWORD_HASH` (default `*` = locked) | Crypted hash. Default locks root; operator can override at build time. | +| `{{ SPAGAT_OPERATOR_AUTHORIZED_KEY }}` | env `SPAGAT_OPERATOR_AUTHORIZED_KEY` (default empty) | Verbatim SSH pubkey line written to `operator`'s `authorized_keys`. Empty = no coordinator key. Character-set validated by phase 6 (single quote, double quote, backslash, newline rejected). | +| `{{ TEST_SSH_PUBKEY }}` | env `IPHASE6_TEST_SSH_PUBKEY` (default empty) | Task #706 test-preseed path. Second SSH pubkey APPENDED to `authorized_keys` after `SPAGAT_OPERATOR_AUTHORIZED_KEY`. `make iso-test` sets it from `$(TEST_SSH_PUBKEY)`. Same charset validator as above. | +| `{{ WIZARD_PRESEED_TOML_B64 }}` | env `IPHASE6_WIZARD_PRESEED_TOML_B64` (default empty) | Task #706 test-preseed path. Standard-alphabet, no-line-wrap base64 of an `spagat_appliance_config::ApplianceConfig` TOML. Postinstall decodes to `/etc/spagat/appliance-config.toml` (0640, root:spagat per BUG-N65; was root:operator per feedback #837, superset via `m operator spagat`). Rust wrapper validates the raw TOML (`IPHASE6_WIZARD_PRESEED_TOML`) parses + carries `[operator]` table BEFORE base64-encoding — see `iso_phase6::preseed`. | +| `{{ WIZARD_COMPLETE_MARKER }}` | env `IPHASE6_WIZARD_COMPLETE_MARKER` (default empty) | Literal `"1"` or empty. When `"1"`, postinstall touches `/var/spagat/state/wizard-complete` as an operator-visible breadcrumb. NOTE: this marker file is NOT consulted by the ADR-0060 onboarding gate; the ACTUAL wizard-skip mechanism is a non-sentinel `[operator]` identity in the preseed above (see `spagat_appliance_config::first_boot_pending`). | +| `{{ INSTALL_STATIC_IP }}` | env `IPHASE6_INSTALL_STATIC_IP` (default `192.168.225.140/24`) | CIDR baked into the first-boot systemd-networkd `10-eth0-static.network` unit's `Address=` line. Gateway + DNS are still hardcoded to `192.168.225.2` — separate follow-up. | +| `{{ INSTALL_DISK }}` | env `SPAGAT_INSTALL_DISK` (default `/dev/sda`) | Target install disk. | +| `{{ LINUX_FLAVOR }}` | static `linux-mok` | The HABv4 MOK kernel variant phase 5 built. | +| `{{ PAYLOAD_SHA256 }}` | computed sha256 of `overlay.tar.zst` | Installer refuses install if on-disk payload sha mismatches. | +| `{{ SERVICES_ENABLE_JSON }}` | JSON array, walked from `OUTPUT_ROOTFS_DIR/etc/systemd/system/*.{service,timer,target}` | All units the installer should `systemctl enable` in the target rootfs. | +| `{{ SERVICES_TARGETS_JSON }}` | JSON object, walked from `OUTPUT_ROOTFS_DIR/etc/systemd/system/*.target.wants/` and `multi-user.target.wants/`, `timers.target.wants/` | Target → unit symlinks the installer should create. | + +## How substitution works + +Phase 6 does literal token replacement of `{{ TOKEN }}` strings (with leading + +trailing whitespace tolerated). It does NOT use a templating engine; the +template stays valid-shape JSON after substitution because: + +- String tokens (`HOSTNAME`, `ROOT_PASSWORD_HASH`, `INSTALL_DISK`, + `LINUX_FLAVOR`, `PAYLOAD_SHA256`) sit inside `"..."` already in the template. +- Composite tokens (`SERVICES_ENABLE_JSON`, `SERVICES_TARGETS_JSON`) are bare + JSON values that phase 6 substitutes with valid JSON arrays/objects. + +After substitution phase 6 runs `python3 -c 'import json,sys; json.load(open(sys.argv[1]))'` +on the result to fail-fast on any malformed substitution. + +## What the installer does with it + +The Photon installer (HABv4-patched, plus our M21.6.f1 monkey-patch from §7 of +spec 040, revised at M21.6.f1.f3) consumes this file at install-time: + +1. `spagat_kickstart.py` (appended to the installer's initrd at + `/usr/lib/python3.X/site-packages/photon_installer/installer_patches/`) is + imported at startup via a hook line appended to + `photon_installer/__init__.py`. It mutates the class-level + `Installer.known_keys` set to allow the three `spagat_*` keys past + `_check_install_config()`. (M21.6.f1.f3 simplified from wrapping the check + method — the upstream class is `Installer`, not `InstallerConfig`, and + `known_keys` is a public class attribute purpose-built for whitelisting.) +2. The standard installer install phase runs: partitions, package install, + kernel install (`linux-mok`), grub install. +3. The post-install phase runs our `_spagat_post`: sha-verifies + extracts + `spagat/overlay.tar.zst` into `/mnt/photon-root`, `systemctl enable`s every + unit in `spagat_services_enable`, creates the symlinks in + `spagat_services_targets`. +4. Reboot → MOK Quickstart UI (HABv4) → first boot → multi-user.target → kanban + on tty1. + +### Build-time receipt: `/.spagat-installer-patch-applied` + +Phase 6 drops a stamp file at the initrd root containing: + +``` +patch_version: M21.6.f1.f3 +applied_at: +installer_module: /usr/lib/python3.X/site-packages/photon_installer/installer.py +installer_init: /usr/lib/python3.X/site-packages/photon_installer/__init__.py +patches_dir: /usr/lib/python3.X/site-packages/photon_installer/installer_patches +spagat_keys: spagat_overlay,spagat_services_enable,spagat_services_targets +``` + +The phase 6 fixture smoke step 16 unpacks the patched initrd and asserts this +file exists. Bastion-side smoke should do the same after the next install +campaign. Absence means the monkey-patch was not applied and the install will +crash at `_check_install_config()` — the M21.6.f1.f2 → f3 regression. + +## `postinstall` — coreutils swap (MVP.B.f7, #518) + +The `postinstall` shell block runs in the target rootfs chroot after all +packages have been installed but before reboot. Alongside the +`/etc/issue.spagat` marker, it force-swaps `coreutils-minimal` with the full +`coreutils` RPM shipped on the ISO. + +**Why:** Photon 5's `minimal` package group resolves to `coreutils-minimal`, +which ships only `/usr/sbin/chroot`. The explicit `coreutils` entry in the +`packages` list is treated as a no-op by tdnf because `coreutils-minimal` +declares `Provides: coreutils` and there is no upgrade signal on a plain +`install` request. Without the swap, `/usr/bin/install` is absent from the +installed rootfs and every spagat-*.service unit with an +`ExecStartPre=/usr/bin/install -d …` line dies at first boot with: + +``` +Failed at step NAMESPACE spawning /usr/bin/install: No such file or directory +``` + +**Why not %post-level:** the linux-mok RPM `%post` runs inside the installer's +RPM transaction — the RPM DB is locked, so `tdnf install` / `rpm -Uvh` from +inside the `%post` is unsupported. `postinstall` runs after the transaction +completes and has a clean DB. + +**Why rpm-direct, not tdnf:** the target rootfs's `/etc/yum.repos.d/` points +at online Photon repos. The appliance is designed for air-gapped install, so +`tdnf install coreutils` would fail without network. The ISO exposes every +package RPM at `/mnt/media/RPMS/x86_64/` throughout install, so `rpm -Uvh` +against the on-media RPM works offline. + +The `[0-9]` glob + `grep -Ev 'minimal|lang|selinux'` filter deliberately +picks the un-suffixed `coreutils-.ph5.x86_64.rpm` (not the -lang +or -selinux siblings, and never the -minimal we are replacing). Version +number is not hardcoded — same shape survives a coreutils bump. + +## What does NOT belong here + +- Comments inside the JSON itself (Photon installer rejects standard JSON + comments; the template is pure JSON with NO `_*` underscore-prefixed keys — + any doc / comment text lives here in this sibling README file). +- Operator secrets in plaintext (use the `crypted: true` hash form). +- Container image digests (those live in the overlay tarball's + `var/cache/spagat-images/*.tar`). + + +## Disk sizing & /tmp placement (BUG-N135, task #948) + +The appliance VM disk **must be <= 50 GB total**. For the VMware smoke VM that +is a 50 GB thin VMDK -- the descriptor's extent line reads +`RW 104857600 SECTORS ...` (104857600 x 512 B = 50 GiB). The kickstart +partition plan sizes `/boot/efi` (512 MiB) and `/boot` (1024 MiB) fixed and +gives **all remaining space to `/`** (`"size": 0`), so it adapts to any disk +<= 50 GB with no hardcoded root size. + +**/tmp is disk-backed, not tmpfs.** Photon 5 mounts a RAM-backed `tmpfs` on +`/tmp` by default (~50% of RAM, ~2 GB on the 4 GB smoke VM). Anything that +writes growing or persistent data there fills RAM and is wiped on reboot, +which surfaced as recurring **"No space left on device"**. Two layers fix it: + +1. **Real fix** -- `spagat-console` (the C UI) now writes its SQLite kanban DB + to `/var/lib/spagat/console-state` and its subagent/CLI scratch files to + `/var/tmp/spagat` (both disk-backed, created by `tmpfiles.d/spagat.conf`). + `spagat-console.service` pins `SPAGAT_DB` + `TMPDIR` at those paths and + lists them in `ReadWritePaths`. The C fallback in `get_db_path()` no longer + points at `/tmp`. +2. **Backstop** -- the `postinstall` step above masks `tmp.mount` + (`ln -sf /dev/null /etc/systemd/system/tmp.mount`), so even a stray `/tmp` + writer lands on the root disk instead of RAM. + +`systemd-tmpfiles` still age-cleans `/tmp` and `/var/tmp/spagat` (1d), so +neither grows without bound. diff --git a/staging/vm-lab/kickstart/check-drift.sh b/staging/vm-lab/kickstart/check-drift.sh new file mode 100644 index 00000000..be6114d8 --- /dev/null +++ b/staging/vm-lab/kickstart/check-drift.sh @@ -0,0 +1,117 @@ +#!/bin/bash +# check-drift - the kickstart copy in this directory must stay identical to +# the canonical template the ISO build actually consumes. +# +# WHY THIS EXISTS: a convenience copy that can silently diverge from the real +# thing is worse than no copy at all - you would read this directory, reason +# about a kickstart the build never uses, and be confidently wrong. +# +# TWO MODES, because this directory ships in two places: +# +# 1. Next to the SPAGAT repo -> diff against the LIVE canonical file: +# src/tools/iso-build/iso-phase6-kickstart-template.cfg +# This is the strong check: it detects drift in EITHER direction. +# +# 2. Standalone (e.g. inside photonos-scripts/staging) -> the canonical +# file is not reachable, so fall back to the recorded snapshot hash in +# EXPECTED-SHA256. Weaker - it cannot see upstream moving - but it still +# catches an edited local copy, and it says plainly which mode it ran in +# rather than quietly proving less than you think. +# +# Point it at a SPAGAT checkout to force mode 1: +# SPAGAT_REPO=/path/to/SpagatLibrarian-Appliance ./check-drift.sh +set -u +# ${BASH_SOURCE[0]} may point at a COPY in /tmp (the CRLF-stripping workflow), +# so locate the kickstart directory rather than assuming it is alongside. +_here="$(cd "$(dirname "${BASH_SOURCE[0]}")" 2>/dev/null && pwd)" +HERE="" +for _d in "${VM_LAB_DIR:-}/kickstart" "$_here" "$PWD/kickstart" "$PWD"; do + if [ -n "$_d" ] && [ -f "$_d/photon-appliance.ks.template.json" ]; then HERE="$_d"; break; fi +done +if [ -z "$HERE" ]; then + echo "FATAL: cannot locate photon-appliance.ks.template.json" >&2 + echo " Run from the vm-lab/kickstart directory, or export VM_LAB_DIR=/path/to/vm-lab" >&2 + exit 2 +fi +COPY="$HERE/photon-appliance.ks.template.json" +HASHFILE="$HERE/EXPECTED-SHA256" +REL="src/tools/iso-build/iso-phase6-kickstart-template.cfg" + +echo "=== local copy ===" +echo " $COPY" +actual=$(sha256sum "$COPY" | cut -d' ' -f1) +echo " sha256: $actual" +echo " bytes : $(stat -c%s "$COPY")" + +# Find a canonical file if one is reachable. +CANON="" +for c in "${SPAGAT_REPO:-}/$REL" \ + "$HERE/../../../$REL" \ + "$HERE/../../../../SpagatLibrarian-Appliance/$REL"; do + if [ -n "$c" ] && [ -f "$c" ]; then CANON="$c"; break; fi +done + +rc=0 +echo +if [ -n "$CANON" ]; then + echo "=== MODE 1: diffing against the live canonical file ===" + echo " $CANON" + echo " sha256: $(sha256sum "$CANON" | cut -d' ' -f1)" + echo + if cmp -s "$CANON" "$COPY"; then + echo "IN SYNC - byte-identical to what the build consumes." + else + echo "*** DRIFT - the copy no longer matches the canonical template ***" + echo + diff -u "$CANON" "$COPY" | head -n 60 + echo + echo "The canonical file is the source of truth. Refresh with:" + echo " cp '$CANON' '$COPY'" + echo " sha256sum '$COPY' | cut -d' ' -f1 > '$HASHFILE'" + rc=1 + fi + + echo + echo "=== control: the comparison can actually fail ===" + tmp=$(mktemp); cp "$CANON" "$tmp"; printf '\n#drift-canary\n' >> "$tmp" + if cmp -s "$CANON" "$tmp"; then + echo " *** a modified file compared EQUAL - this check proves nothing ***"; rc=1 + else + echo " a modified copy is correctly detected as different" + fi + rm -f "$tmp" +else + echo "=== MODE 2: canonical file not reachable - hash fallback ===" + echo " (looked for $REL under \$SPAGAT_REPO and two relative guesses)" + if [ ! -f "$HASHFILE" ]; then + echo " *** no EXPECTED-SHA256 either - drift CANNOT be checked. ***" + echo " This is the one outcome that must not be read as 'fine'." + exit 3 + fi + expected=$(grep -oE '[0-9a-f]{64}' "$HASHFILE" | head -n1) + echo " expected (recorded at snapshot time): $expected" + echo " actual : $actual" + if [ "$actual" = "$expected" ]; then + echo + echo "UNCHANGED since the snapshot." + echo "NOTE: this proves the LOCAL copy was not edited. It canNOT tell" + echo "you whether the upstream template has moved on - for that, run" + echo "with SPAGAT_REPO pointing at a SpagatLibrarian-Appliance checkout." + else + echo + echo "*** the local copy has been EDITED since the snapshot ***" + rc=1 + fi + + echo + echo "=== control: the hash comparison can actually fail ===" + if [ "$expected" = "0000000000000000000000000000000000000000000000000000000000000000" ]; then + echo " *** placeholder hash - meaningless ***"; rc=1 + elif [ "$actual" = "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff" ]; then + echo " *** impossible actual hash ***"; rc=1 + else + echo " a wrong hash would not match -> the comparison discriminates" + fi +fi + +exit $rc diff --git a/staging/vm-lab/kickstart/photon-appliance.ks.template.json b/staging/vm-lab/kickstart/photon-appliance.ks.template.json new file mode 100644 index 00000000..1723bab5 --- /dev/null +++ b/staging/vm-lab/kickstart/photon-appliance.ks.template.json @@ -0,0 +1,63 @@ +{ + "hostname": "{{ HOSTNAME }}", + "password": { "crypted": true, "text": "{{ ROOT_PASSWORD_HASH }}" }, + "disk": "{{ INSTALL_DISK }}", + "linux_flavor": "{{ LINUX_FLAVOR }}", + "packages": ["minimal", "coreutils", "kbd", "linux-mok", "initramfs", "shim-signed-mok", "openssh", "logrotate"], + + "partitions": [ + { "mountpoint": "/boot/efi", "size": 512, "filesystem": "vfat" }, + { "mountpoint": "/boot", "size": 1024, "filesystem": "ext4" }, + { "mountpoint": "/", "size": 0, "filesystem": "ext4" } + ], + + "postinstall": [ + "#!/bin/sh", + "if [ ! -x /usr/bin/install ]; then", + " CU=$(ls /mnt/media/RPMS/x86_64/coreutils-[0-9]*.ph5.x86_64.rpm 2>/dev/null | grep -Ev 'minimal|lang|selinux' | head -1)", + " if [ -n \"$CU\" ]; then", + " rpm -e --nodeps coreutils-minimal 2>/dev/null || true", + " rpm -Uvh --replacefiles --replacepkgs --nodeps \"$CU\" 2>/dev/null || true", + " fi", + "fi", + "echo 'spagat-librarian first-boot stub (M21.6.f1)' > /etc/issue.spagat", + "# BUG-N135 (task #948): mask tmp.mount so /tmp stays on the root disk instead", + "# of a RAM tmpfs (~2GB on a 4GB VM). Backstop to the spagat-console DB + scratch", + "# moving to /var; prevents any /tmp writer from exhausting RAM (No space left).", + "ln -sf /dev/null /etc/systemd/system/tmp.mount", + "systemctl enable sshd.service 2>/dev/null || true", + "# BUG-N147.f1 (task #978): enable logrotate.timer so the spagat-bridges rule", + "# (shipped via the overlay at /etc/logrotate.d/spagat-bridges) actually runs.", + "# Photon's logrotate RPM ships the timer but %systemd_post does NOT enable it,", + "# and the appliance has no cronie, so bridge-aa.log on /var/spagat/bridges would", + "# never rotate without this. Enabled explicitly rather than relying on a preset.", + "systemctl enable logrotate.timer 2>/dev/null || true", + "install -d -m 0755 /etc/systemd/network", + "printf '[Match]\nName=eth0 en*\n\n[Network]\nAddress={{ INSTALL_STATIC_IP }}\nGateway=192.168.225.2\nDNS=192.168.225.2\n' > /etc/systemd/network/10-eth0-static.network", + "systemctl enable systemd-networkd.service 2>/dev/null || true", + "systemctl enable systemd-resolved.service 2>/dev/null || true", + "systemctl disable systemd-networkd-wait-online.service 2>/dev/null || true", + "getent group operator >/dev/null 2>&1 || groupadd operator", + "id -u operator >/dev/null 2>&1 || useradd -m -s /bin/bash -g operator operator", + "install -d -m 0700 -o operator -g operator /home/operator/.ssh", + "_SPAGAT_OPERATOR_AUTHORIZED_KEY='{{ SPAGAT_OPERATOR_AUTHORIZED_KEY }}'; if [ -n \"$_SPAGAT_OPERATOR_AUTHORIZED_KEY\" ]; then printf '%s\n' \"$_SPAGAT_OPERATOR_AUTHORIZED_KEY\" > /home/operator/.ssh/authorized_keys; else : > /home/operator/.ssh/authorized_keys; fi", + "_TEST_SSH_PUBKEY='{{ TEST_SSH_PUBKEY }}'; if [ -n \"$_TEST_SSH_PUBKEY\" ]; then printf '%s\n' \"$_TEST_SSH_PUBKEY\" >> /home/operator/.ssh/authorized_keys; fi", + "chmod 0600 /home/operator/.ssh/authorized_keys", + "chown operator:operator /home/operator/.ssh/authorized_keys", + "install -d -m 0755 -o root -g root /etc/spagat", + "getent group spagat >/dev/null 2>&1 || groupadd --system spagat", + "_WIZARD_PRESEED_TOML_B64='{{ WIZARD_PRESEED_TOML_B64 }}'; if [ -n \"$_WIZARD_PRESEED_TOML_B64\" ]; then printf '%s' \"$_WIZARD_PRESEED_TOML_B64\" | base64 -d > /etc/spagat/appliance-config.toml && chmod 0640 /etc/spagat/appliance-config.toml && chown root:spagat /etc/spagat/appliance-config.toml; fi", + "install -d -m 0755 -o root -g root /var/spagat /var/spagat/state", + "_WIZARD_COMPLETE_MARKER='{{ WIZARD_COMPLETE_MARKER }}'; if [ \"$_WIZARD_COMPLETE_MARKER\" = \"1\" ]; then : > /var/spagat/state/wizard-complete && chmod 0644 /var/spagat/state/wizard-complete && chown root:root /var/spagat/state/wizard-complete; fi", + "true" + ], + + "spagat_overlay": { + "payload_path": "/mnt/media/spagat/overlay.tar.zst", + "payload_sha256": "{{ PAYLOAD_SHA256 }}", + "install_root": "/mnt/photon-root/photon-chroot" + }, + + "spagat_services_enable": {{ SERVICES_ENABLE_JSON }}, + "spagat_services_targets": {{ SERVICES_TARGETS_JSON }} +} diff --git a/staging/vm-lab/scripts/00-preflight.sh b/staging/vm-lab/scripts/00-preflight.sh new file mode 100644 index 00000000..a4f80f49 --- /dev/null +++ b/staging/vm-lab/scripts/00-preflight.sh @@ -0,0 +1,120 @@ +#!/bin/bash +# 00-preflight — prove the host can do the job BEFORE anything is created. +# +# Every check prints what it measured, not just OK/FAIL, because "the tool is +# missing" and "the tool is there but unreadable by this user" need different +# fixes and look identical in a boolean. +# +# Exit 0 = safe to proceed. Non-zero = stop and read the output. +set -u +# --- locate + load the config, FAIL CLOSED -------------------------------- +# ${BASH_SOURCE[0]} may point at a COPY in /tmp: the standard WSL workflow +# here is: tr -d CR < script > /tmp/x.sh ; bash /tmp/x.sh (strips CRLF), +# which breaks any path computed relative to the script. Try the plausible +# locations and REFUSE to continue if none has the config - a previous +# version merely failed to source it and carried on, printing a healthy +# looking first section before dying on unbound variables further down. +_here="$(cd "$(dirname "${BASH_SOURCE[0]}")" 2>/dev/null && pwd)" +CFG="" +for _c in "${VM_LAB_DIR:-}/config/vm-lab.env" \ + "$_here/../config/vm-lab.env" \ + "$_here/config/vm-lab.env" \ + "$PWD/config/vm-lab.env" \ + "$PWD/../config/vm-lab.env"; do + if [ -n "$_c" ] && [ -f "$_c" ]; then CFG="$_c"; break; fi +done +if [ -z "$CFG" ]; then + echo "FATAL: cannot locate config/vm-lab.env" >&2 + echo " Run from deploy/vm-lab/, or: export VM_LAB_DIR=/path/to/deploy/vm-lab" >&2 + exit 78 +fi +# shellcheck source=../config/vm-lab.env +. "$CFG" +HERE="$_here" + +fail=0 +ok() { printf ' OK %s\n' "$1"; } +bad() { printf ' FAIL %s\n' "$1"; fail=1; } +note() { printf ' %s\n' "$1"; } + +echo "=== identity ===" +note "user: $(id -un) (uid $(id -u))" +if [ "$(id -un)" = "spagat-runner" ]; then + bad "running as spagat-runner — vmrun.exe is mode 744 owned by dcaso; you will get 'vmrun IO: Permission denied'" +else + ok "not spagat-runner" +fi + +echo +echo "=== VMware Workstation tooling ===" +for t in "$VMRUN_WSL" "$VDISKMANAGER_WSL"; do + if [ -e "$t" ]; then + if [ -x "$t" ]; then ok "$(basename "$t")"; else bad "$(basename "$t") present but NOT executable by $(id -un)"; fi + else + bad "missing: $t" + fi +done +if [ -x "$VMRUN_WSL" ]; then + note "vmrun reports: $("$VMRUN_WSL" -T ws list 2>&1 | head -n 1)" +fi + +echo +echo "=== VM directory ===" +if [ -d "$VM_DIR_WSL" ]; then + ok "$VM_DIR_WSL" + note "contents: $(ls -1 "$VM_DIR_WSL" 2>/dev/null | wc -l) entries" +else + note "$VM_DIR_WSL does not exist yet — 10-create-vm.ps1 will create it" +fi + +echo +echo "=== free space on the VM volume ===" +avail_k=$(df -k "$(dirname "$VM_DIR_WSL")" 2>/dev/null | tail -n 1 | awk '{print $4}') +if [ -n "${avail_k:-}" ]; then + note "available: $((avail_k/1024/1024)) GB" + # A 50 GB thin disk starts ~6 MB but the ISO alone is ~5.4 GB, and the + # installed system grows to ~12 GB. 20 GB is the honest floor. + if [ "$avail_k" -gt $((20*1024*1024)) ]; then ok "≥20 GB free"; else bad "under 20 GB free — the ISO copy (5.4 GB) plus a grown disk will not fit"; fi +else + bad "could not measure free space on $(dirname "$VM_DIR_WSL")" +fi + +echo +echo "=== operator medium (credential channel) ===" +med="$VM_DIR_WSL/${OPERATOR_MEDIUM_BASENAME}.vmdk" +flat="$VM_DIR_WSL/${OPERATOR_MEDIUM_BASENAME}-flat.vmdk" +if [ -f "$med" ] && [ -f "$flat" ]; then + sz=$(stat -c%s "$flat") + ok "descriptor + flat present" + note "flat bytes: $sz (expected $OPERATOR_MEDIUM_FLAT_BYTES)" + if [ "$sz" = "$OPERATOR_MEDIUM_FLAT_BYTES" ]; then ok "flat size matches"; else bad "flat size differs — this is NOT the verified medium"; fi + grep -E 'createType|ddb.adapterType' "$med" | sed 's/^/ /' +else + note "absent — the appliance will boot KEYLESS (no credentials)." + note "This directory never generates it; see README 'The operator medium'." +fi + +echo +echo "=== running VMs (these must not be disturbed) ===" +if [ -x "$VMRUN_WSL" ]; then + "$VMRUN_WSL" -T ws list 2>/dev/null | sed 's/^/ /' + if "$VMRUN_WSL" -T ws list 2>/dev/null | grep -q "$VM_NAME"; then + bad "$VM_NAME is RUNNING — stop it before creating or reinstalling" + else + ok "$VM_NAME is not running" + fi +fi + +echo +echo "=== ssh keypair ===" +if [ -f "$SSH_KEY_DIR/$SSH_KEY_NAME" ]; then + ok "private key $SSH_KEY_DIR/$SSH_KEY_NAME" + note "pub: $(cut -d' ' -f1,3 "$SSH_KEY_DIR/$SSH_KEY_NAME.pub" 2>/dev/null)" +else + note "no keypair yet — 20-make-ssh-key.sh creates one." + note "🚨 The pubkey must be baked at ISO BUILD time; it cannot be added later." +fi + +echo +if [ "$fail" -eq 0 ]; then echo "PREFLIGHT: PASS"; else echo "PREFLIGHT: FAIL — fix the above before continuing"; fi +exit $fail diff --git a/staging/vm-lab/scripts/10-create-vm.ps1 b/staging/vm-lab/scripts/10-create-vm.ps1 new file mode 100644 index 00000000..214fc5b0 --- /dev/null +++ b/staging/vm-lab/scripts/10-create-vm.ps1 @@ -0,0 +1,158 @@ +# 10-create-vm - create the VM directory, the boot VMDK, and the VMX. +# +# ASCII-ONLY BY POLICY. Windows PowerShell 5.1 (`powershell -File ...`) reads +# .ps1 as ANSI unless the file carries a UTF-8 BOM, so a stray em-dash or +# arrow silently corrupts a string literal and the parser then reports a +# cascade of bogus "missing closing brace" errors far from the real line. +# pwsh 7 decodes the same file as UTF-8 and parses it fine, so a parse check +# run under 7 does NOT prove the file runs under 5.1. Keep this file 7-bit. +# +# WHY POWERSHELL: vmware-vdiskmanager.exe and the VMX both want Windows paths, +# and this is the one step that must run on the Windows side. +# +# WHY A TEMPLATE AND NOT THE VMware "New VM" WIZARD: `spagat-vm-orchestrator +# install-from-iso` only EDITS an existing .vmx - it has no createvm / +# vdiskmanager path. So VM creation sits outside the automated loop, which is +# exactly the kind of step that silently drifts. Half the BUG-N series came +# from a VMX key being wrong. The template pins every one of them. +# +# IDEMPOTENCE / SAFETY: this script REFUSES to overwrite an existing boot disk +# or VMX. Re-provisioning is 90-teardown.ps1 (which moves aside, never deletes) +# followed by this script. +[CmdletBinding()] +param( + # Recreate the VMX from the template even if one exists. Never touches the + # boot disk or the operator medium. + [switch]$RefreshVmxOnly +) +$ErrorActionPreference = 'Stop' + +$here = Split-Path -Parent $MyInvocation.MyCommand.Path +$cfgPath = Join-Path $here '..\config\vm-lab.env' + +# Parse the shell env file: KEY="value" / KEY='value' / KEY=value, ignoring +# comments and blanks. Expands ${VAR} against values already parsed. +# +# TRAILING COMMENTS MATTER. `KEY="value" # why` must yield `value`, not +# `"value" # why`. bash strips that for free, so a WSL-side test of the same +# file passes while this parser silently produced a value with a comment glued +# on - which would have been handed straight to vmware-vdiskmanager as an +# adapter name. Match the quoted form first, and for bare values cut at the +# first `#`. +$cfg = @{} +foreach ($line in Get-Content $cfgPath) { + if ($line -match '^\s*#' -or $line -match '^\s*$') { continue } + if ($line -notmatch '^\s*([A-Za-z_][A-Za-z0-9_]*)=\s*(.*)$') { continue } + $k = $Matches[1] + $raw = $Matches[2] + if ($raw -match '^"([^"]*)"') { $v = $Matches[1] } + elseif ($raw -match "^'([^']*)'") { $v = $Matches[1] } + else { $v = (($raw -split '#', 2)[0]).Trim() } + $v = [regex]::Replace($v, '\$\{([A-Za-z_][A-Za-z0-9_]*)\}', { param($m) if ($cfg.ContainsKey($m.Groups[1].Value)) { $cfg[$m.Groups[1].Value] } else { '' } }) + $v = $v -replace '\\\\', '\' + $cfg[$k] = $v +} + +$vmName = $cfg['VM_NAME'] +$vmDir = $cfg['VM_DIR_WIN'] +$vdm = Join-Path $cfg['VMWARE_DIR_WIN'] 'vmware-vdiskmanager.exe' +$vmrun = Join-Path $cfg['VMWARE_DIR_WIN'] 'vmrun.exe' +$vmxPath = Join-Path $vmDir "$vmName.vmx" +$vmdkPath = Join-Path $vmDir "$vmName.vmdk" +$template = Join-Path $here '..\config\spagat-smoke.vmx.template' + +Write-Output "=== target ===" +Write-Output " VM : $vmName" +Write-Output " dir : $vmDir" +Write-Output " disk : $($cfg['BOOT_DISK_SIZE']) $($cfg['BOOT_DISK_ADAPTER']) type=$($cfg['BOOT_DISK_TYPE'])" +Write-Output " template: $template" + +if (-not (Test-Path $vdm)) { throw "vmware-vdiskmanager not found: $vdm" } + +Write-Output "" +Write-Output "=== refuse if the VM is running ===" +if (Test-Path $vmrun) { + $running = & $vmrun -T ws list + if ($running -match [regex]::Escape($vmName)) { + throw "$vmName is RUNNING. Stop it first (vmrun stop). Refusing to touch its files." + } + Write-Output " not running" + Write-Output " other VMs (left alone):" + $running | Where-Object { $_ -match '\.vmx$' } | ForEach-Object { " $_" } +} + +if (-not (Test-Path $vmDir)) { + New-Item -ItemType Directory -Path $vmDir -Force | Out-Null + Write-Output " created $vmDir" +} + +# --------------------------------------------------------------- boot disk -- +Write-Output "" +Write-Output "=== boot disk ===" +if ($RefreshVmxOnly) { + Write-Output " -RefreshVmxOnly: leaving the disk alone" +} elseif (Test-Path $vmdkPath) { + Write-Output " ALREADY EXISTS - refusing to overwrite: $vmdkPath" + Write-Output " To re-provision: run 90-teardown.ps1 first (moves aside, never deletes)." +} else { + & $vdm -c -s $cfg['BOOT_DISK_SIZE'] -a $cfg['BOOT_DISK_ADAPTER'] -t $cfg['BOOT_DISK_TYPE'] $vmdkPath 2>&1 | + ForEach-Object { " $_" } + if (-not (Test-Path $vmdkPath)) { throw "vdiskmanager did not create $vmdkPath" } + Write-Output " created, descriptor says:" + Get-Content $vmdkPath | Select-String -Pattern '^RW |createType|ddb.adapterType' | ForEach-Object { " $_" } +} + +# --------------------------------------------------------------------- VMX -- +Write-Output "" +Write-Output "=== VMX ===" +if ((Test-Path $vmxPath) -and -not $RefreshVmxOnly) { + Write-Output " ALREADY EXISTS - refusing to overwrite: $vmxPath" + Write-Output " Use -RefreshVmxOnly to regenerate it from the template." +} else { + if (Test-Path $vmxPath) { + $bak = "$vmxPath.pre-refresh-$(Get-Date -Format 'yyyyMMddTHHmmssZ')" + Move-Item -LiteralPath $vmxPath -Destination $bak + Write-Output " existing VMX moved aside -> $(Split-Path $bak -Leaf)" + } + $serialWin = Join-Path $vmDir "$($cfg['SERIAL_LOG_PREFIX'])-$vmName.log" + $body = Get-Content $template -Raw + $body = $body. + Replace('@@VM_NAME@@', $vmName). + Replace('@@GUEST_VCPUS@@', $cfg['GUEST_VCPUS']). + Replace('@@GUEST_MEM_MB@@', $cfg['GUEST_MEM_MB']). + Replace('@@GUEST_MAC@@', $cfg['GUEST_MAC']). + Replace('@@OPERATOR_MEDIUM_BASENAME@@', $cfg['OPERATOR_MEDIUM_BASENAME']). + Replace('@@SERIAL_LOG_WIN@@', $serialWin) + # VMX files are CRLF + UTF-8 without BOM. + $utf8NoBom = New-Object System.Text.UTF8Encoding($false) + [System.IO.File]::WriteAllText($vmxPath, ($body -replace "`r?`n", "`r`n"), $utf8NoBom) + Write-Output " wrote $vmxPath" + + $left = Select-String -Path $vmxPath -Pattern '@@[A-Z_]+@@' + if ($left) { throw "unsubstituted placeholders remain: $($left -join ', ')" } + Write-Output " no placeholders left" +} + +# ------------------------------------------------------------------ verify -- +Write-Output "" +Write-Output "=== the keys that must be right (drift here is how BUG-Ns start) ===" +$mustHave = 'firmware|secureBoot|bootOrder|sata0.present|scsi0.virtualDev|' + + 'scsi0:0.fileName|scsi0:1.fileName|ethernet0.virtualDev|' + + 'generatedAddress|uuid.bios|serial0.fileName|msg.autoAnswer|' + + 'tools.syncTime|memSize|numvcpus' +Get-Content $vmxPath | Select-String -Pattern $mustHave | ForEach-Object { " $_" } + +Write-Output "" +Write-Output "=== operator medium (NOT created here, see README) ===" +$medNames = @("$($cfg['OPERATOR_MEDIUM_BASENAME']).vmdk", "$($cfg['OPERATOR_MEDIUM_BASENAME'])-flat.vmdk") +foreach ($n in $medNames) { + $p = Join-Path $vmDir $n + if (Test-Path $p) { + " present {0,-30} {1,14:n0} bytes" -f $n, (Get-Item $p).Length + } else { + " ABSENT {0,-30} -> the appliance will boot KEYLESS" -f $n + } +} + +Write-Output "" +Write-Output "NEXT: scripts/20-make-ssh-key.sh (decide SSH access BEFORE building the ISO)" diff --git a/staging/vm-lab/scripts/20-make-ssh-key.sh b/staging/vm-lab/scripts/20-make-ssh-key.sh new file mode 100644 index 00000000..abc18741 --- /dev/null +++ b/staging/vm-lab/scripts/20-make-ssh-key.sh @@ -0,0 +1,98 @@ +#!/bin/bash +# 20-make-ssh-key — create the lab keypair and print the exact exports the ISO +# build needs. +# +# 🚨 READ THIS BEFORE ANYTHING ELSE +# +# SSH access to the appliance is decided AT ISO BUILD TIME. There is no +# post-install path: root installs LOCKED by default +# (DEFAULT_SPAGAT_ROOT_PASSWORD_HASH = "*"), and both SSH key variables +# default to EMPTY: +# +# SPAGAT_OPERATOR_AUTHORIZED_KEY = "" (iso-phase6 config.rs) +# IPHASE6_TEST_SSH_PUBKEY = "" +# +# so a normal build ships `operator`'s authorized_keys with correct 0600 +# perms and NO CONTENT. That is the whole of the long-running "no accepted +# SSH key" symptom — not an onboarding gate, not a credential bug: nobody +# exported the variable. +# +# If you build an ISO without exporting one of these, your only way into the +# guest is the serial console. +set -u +# --- locate + load the config, FAIL CLOSED -------------------------------- +# ${BASH_SOURCE[0]} may point at a COPY in /tmp: the standard WSL workflow +# here is: tr -d CR < script > /tmp/x.sh ; bash /tmp/x.sh (strips CRLF), +# which breaks any path computed relative to the script. Try the plausible +# locations and REFUSE to continue if none has the config - a previous +# version merely failed to source it and carried on, printing a healthy +# looking first section before dying on unbound variables further down. +_here="$(cd "$(dirname "${BASH_SOURCE[0]}")" 2>/dev/null && pwd)" +CFG="" +for _c in "${VM_LAB_DIR:-}/config/vm-lab.env" \ + "$_here/../config/vm-lab.env" \ + "$_here/config/vm-lab.env" \ + "$PWD/config/vm-lab.env" \ + "$PWD/../config/vm-lab.env"; do + if [ -n "$_c" ] && [ -f "$_c" ]; then CFG="$_c"; break; fi +done +if [ -z "$CFG" ]; then + echo "FATAL: cannot locate config/vm-lab.env" >&2 + echo " Run from deploy/vm-lab/, or: export VM_LAB_DIR=/path/to/deploy/vm-lab" >&2 + exit 78 +fi +# shellcheck source=../config/vm-lab.env +. "$CFG" +HERE="$_here" + +KEY="$SSH_KEY_DIR/$SSH_KEY_NAME" + +mkdir -p "$SSH_KEY_DIR" +chmod 0700 "$SSH_KEY_DIR" + +if [ -f "$KEY" ]; then + echo "=== keypair already exists — reusing (not regenerating) ===" +else + echo "=== generating ed25519 keypair ===" + # No passphrase: this is a disposable lab key for an unattended install. + # It is NOT an operator credential and must never be reused elsewhere. + ssh-keygen -t ed25519 -N '' -C "spagat-vm-lab@$(hostname)" -f "$KEY" +fi +chmod 0600 "$KEY" +chmod 0644 "$KEY.pub" + +echo +echo " private: $KEY" +echo " public : $KEY.pub" +echo " finger : $(ssh-keygen -lf "$KEY.pub")" + +PUB="$(cat "$KEY.pub")" + +echo +echo "==============================================================" +echo " EXPORT THESE BEFORE BUILDING THE ISO" +echo "==============================================================" +cat <" +echo +echo "Root stays LOCKED regardless. To also set a root password you must pass" +echo "BOTH --root-password-file AND --operator-medium-dir to install-from-iso" +echo "(all-or-nothing, BUG-N91); passing one alone is a hard error." diff --git a/staging/vm-lab/scripts/30-install-from-iso.sh b/staging/vm-lab/scripts/30-install-from-iso.sh new file mode 100644 index 00000000..1bf3331d --- /dev/null +++ b/staging/vm-lab/scripts/30-install-from-iso.sh @@ -0,0 +1,157 @@ +#!/bin/bash +# 30-install-from-iso — unattended install onto the lab VM. +# +# Usage: +# ./30-install-from-iso.sh --iso /home/dcaso/work/iso-out-/.iso +# ./30-install-from-iso.sh --iso --orchestrator +# +# WHAT THIS WRAPS +# spagat-vm-orchestrator install-from-iso — the tested path. It edits the +# existing VMX in place: attaches the ISO on sata0:1, flips bios.bootOrder +# to cdrom-first for the install window, powers on, waits for install +# completion, then restores hdd-first, detaches the CDROM and moves NVRAM +# aside. It NEVER touches scsi0:1, so the operator medium survives verbatim. +# +# DELIBERATE OMISSIONS +# --force auto-stops a running instance. Other VMs on this host +# may be live CI runners; this script refuses instead. +# --efuse-vmdk attaches an install-time marker on sata0:0 that finalize +# then DETACHES. The operator medium is a PERSISTENT +# boot-time disk on scsi0:1 — passing this flag for it is +# wrong. +# --root-password-file / --operator-medium-dir +# all-or-nothing (BUG-N91). Omitted here, so root installs +# LOCKED. `BUG-N91: no --root-password-file supplied` in +# the log is EXPECTED, not a fault. +# +# PATH FORM: pass the LINUX path. vmx-info parses /mnt/c/... and fails on +# C:\... with "VMX I/O: No such file or directory"; the orchestrator does the +# Windows translation itself (BUG-N9 / #617). +set -u +# --- locate + load the config, FAIL CLOSED -------------------------------- +# ${BASH_SOURCE[0]} may point at a COPY in /tmp: the standard WSL workflow +# here is: tr -d CR < script > /tmp/x.sh ; bash /tmp/x.sh (strips CRLF), +# which breaks any path computed relative to the script. Try the plausible +# locations and REFUSE to continue if none has the config - a previous +# version merely failed to source it and carried on, printing a healthy +# looking first section before dying on unbound variables further down. +_here="$(cd "$(dirname "${BASH_SOURCE[0]}")" 2>/dev/null && pwd)" +CFG="" +for _c in "${VM_LAB_DIR:-}/config/vm-lab.env" \ + "$_here/../config/vm-lab.env" \ + "$_here/config/vm-lab.env" \ + "$PWD/config/vm-lab.env" \ + "$PWD/../config/vm-lab.env"; do + if [ -n "$_c" ] && [ -f "$_c" ]; then CFG="$_c"; break; fi +done +if [ -z "$CFG" ]; then + echo "FATAL: cannot locate config/vm-lab.env" >&2 + echo " Run from deploy/vm-lab/, or: export VM_LAB_DIR=/path/to/deploy/vm-lab" >&2 + exit 78 +fi +# shellcheck source=../config/vm-lab.env +. "$CFG" +HERE="$_here" + +ISO="" +ORCH="" +INSTALL_WINDOW=1200 +BOOT_WINDOW=600 +while [ $# -gt 0 ]; do + case "$1" in + --iso) ISO="$2"; shift 2 ;; + --orchestrator) ORCH="$2"; shift 2 ;; + --install-window-sec) INSTALL_WINDOW="$2"; shift 2 ;; + --boot-window-sec) BOOT_WINDOW="$2"; shift 2 ;; + *) echo "unknown arg: $1"; exit 64 ;; + esac +done +[ -n "$ISO" ] || { echo "usage: $0 --iso [--orchestrator ]"; exit 64; } + +VMX="$VM_DIR_WSL/$VM_NAME.vmx" +STAMP="$(basename "$ISO" .iso)" +SERIAL="$VM_DIR_WSL/${SERIAL_LOG_PREFIX}-${STAMP}.log" +LOG="$ISO_OUT_ROOT/install-${STAMP}.log" +OFFSET_FILE="$ISO_OUT_ROOT/serial-offset-${STAMP}" + +# Locate the orchestrator: prefer an explicit path, else the newest build. +if [ -z "$ORCH" ]; then + ORCH=$(ls -1t "$ISO_OUT_ROOT"/iso-build-*/tools/spagat-rust/target/x86_64-unknown-linux-musl/release/spagat-vm-orchestrator 2>/dev/null | head -n 1) +fi + +echo "===== pre-flight =====" +echo " orchestrator : ${ORCH:-}" +[ -n "$ORCH" ] && [ -x "$ORCH" ] || { echo " *** orchestrator not found or not executable ***"; exit 2; } +echo " vmx : $VMX" +echo " iso : $ISO" +[ -f "$VMX" ] || { echo " *** VMX missing — run 10-create-vm.ps1 first ***"; exit 3; } +[ -f "$ISO" ] || { echo " *** ISO missing ***"; exit 3; } + +echo -n " $VM_NAME running? : " +if "$ORCH" list 2>/dev/null | grep -q "$VM_NAME"; then + echo "YES — stop it first. NOT passing --force (other VMs here may be live CI runners)." + exit 4 +fi +echo "no" +echo " other VMs (left alone):" +"$ORCH" list 2>/dev/null | sed 's/^/ /' + +# Integrity: the ISO must match its own sidecar. 9p -> NTFS is where a short +# write hides, and a truncated ISO installs a subtly broken appliance. +echo -n " iso sha256 : " +actual=$(sha256sum "$ISO" | cut -d' ' -f1); echo "$actual" +if [ -f "$ISO.sha256" ]; then + expect=$(grep -oE '[0-9a-f]{64}' "$ISO.sha256" | head -n1) + echo " sidecar : $expect" + [ "$actual" = "$expect" ] || { echo " *** HASH MISMATCH — refusing to install ***"; exit 5; } + echo " hash OK" +else + echo " (no .sha256 sidecar next to the ISO — cannot cross-check)" +fi + +echo " operator medium at scsi0:1:" +grep -E 'scsi0:1' "$VMX" | sed 's/^/ /' +if [ -f "$VM_DIR_WSL/${OPERATOR_MEDIUM_BASENAME}-flat.vmdk" ]; then + echo " flat bytes: $(stat -c%s "$VM_DIR_WSL/${OPERATOR_MEDIUM_BASENAME}-flat.vmdk")" +else + echo " *** ABSENT — this install will boot KEYLESS ***" +fi + +# Point the serial log at this build so RCA logs never interleave, and record +# where the existing log ends: only bytes past that offset belong to this run. +if ! grep -q "${SERIAL_LOG_PREFIX}-${STAMP}.log" "$VMX"; then + cp "$VMX" "$VMX.pre-${STAMP}-$(date -u +%Y%m%dT%H%M%SZ)" + win_serial="${VM_DIR_WIN}\\${SERIAL_LOG_PREFIX}-${STAMP}.log" + esc=$(printf '%s' "$win_serial" | sed 's/\\/\\\\/g') + sed -i "s|^serial0.fileName = .*|serial0.fileName = \"$esc\"|" "$VMX" +fi +grep -E '^serial0.fileName' "$VMX" | sed 's/^/ /' + +OFF=$(stat -c%s "$SERIAL" 2>/dev/null || echo 0) +echo "$OFF" > "$OFFSET_FILE" +echo " serial offset: $OFF (read ONLY past this — previous boots stay as evidence)" + +echo +echo "===== launching install-from-iso =====" +: > "$LOG" +nohup "$ORCH" install-from-iso \ + --vmx "$VMX" \ + --iso "$ISO" \ + --install-window-sec "$INSTALL_WINDOW" \ + --boot-window-sec "$BOOT_WINDOW" \ + >> "$LOG" 2>&1 & +PID=$! +echo " pid $PID at $(date -u +%H:%M:%SZ)" +echo " log $LOG" +echo " serial $SERIAL" +echo " offset $OFFSET_FILE" + +sleep 90 +echo +echo "===== first 25 log lines =====" +sed -E 's/\x1b\[[0-9;]*m//g' "$LOG" | head -n 25 | cut -c1-190 +echo +kill -0 "$PID" 2>/dev/null && echo " RUNNING" || echo " EXITED — read $LOG" +echo -n " serial bytes: "; stat -c%s "$SERIAL" 2>/dev/null || echo "not created yet" +echo +echo "NEXT: scripts/50-verify-boot.sh --stamp $STAMP" diff --git a/staging/vm-lab/scripts/40-check-staging.sh b/staging/vm-lab/scripts/40-check-staging.sh new file mode 100644 index 00000000..465f64bf --- /dev/null +++ b/staging/vm-lab/scripts/40-check-staging.sh @@ -0,0 +1,217 @@ +#!/bin/bash +# 40-check-staging — answer "did the thing I think is staged actually get +# staged?" at each of the three places staging happens. +# +# --rootfs the BUILT ROOTFS the ISO was made from (iso-rootfs-) +# --iso the finished ISO + its sidecars +# --guest the RUNNING guest, via its serial log +# +# Pass any combination. With no flag it checks whatever it can find. +# +# DESIGN RULE: every check prints the measured value, and the ones that can +# be vacuous carry a control. A check that prints nothing is indistinguishable +# from a check that passed — that failure mode is the whole reason this script +# exists. +set -u +# --- locate + load the config, FAIL CLOSED -------------------------------- +# ${BASH_SOURCE[0]} may point at a COPY in /tmp: the standard WSL workflow +# here is: tr -d CR < script > /tmp/x.sh ; bash /tmp/x.sh (strips CRLF), +# which breaks any path computed relative to the script. Try the plausible +# locations and REFUSE to continue if none has the config - a previous +# version merely failed to source it and carried on, printing a healthy +# looking first section before dying on unbound variables further down. +_here="$(cd "$(dirname "${BASH_SOURCE[0]}")" 2>/dev/null && pwd)" +CFG="" +for _c in "${VM_LAB_DIR:-}/config/vm-lab.env" \ + "$_here/../config/vm-lab.env" \ + "$_here/config/vm-lab.env" \ + "$PWD/config/vm-lab.env" \ + "$PWD/../config/vm-lab.env"; do + if [ -n "$_c" ] && [ -f "$_c" ]; then CFG="$_c"; break; fi +done +if [ -z "$CFG" ]; then + echo "FATAL: cannot locate config/vm-lab.env" >&2 + echo " Run from deploy/vm-lab/, or: export VM_LAB_DIR=/path/to/deploy/vm-lab" >&2 + exit 78 +fi +# shellcheck source=../config/vm-lab.env +. "$CFG" +HERE="$_here" + +ROOTFS=""; ISO=""; GUEST=0 +while [ $# -gt 0 ]; do + case "$1" in + --rootfs) ROOTFS="$2"; shift 2 ;; + --iso) ISO="$2"; shift 2 ;; + --guest) GUEST=1; shift ;; + *) echo "unknown arg: $1"; exit 64 ;; + esac +done +if [ -z "$ROOTFS" ] && [ -z "$ISO" ] && [ "$GUEST" -eq 0 ]; then + ROOTFS=$(ls -1td "$ISO_OUT_ROOT"/iso-rootfs-* 2>/dev/null | head -n 1) + ISO=$(ls -1t "$ISO_OUT_ROOT"/iso-out-*/*.iso 2>/dev/null | head -n 1) + GUEST=1 + echo "(no flags — auto-selected the newest of each)" +fi + +# ---------------------------------------------------------------- 1. ROOTFS -- +if [ -n "$ROOTFS" ] && [ -d "$ROOTFS" ]; then +echo "==============================================================" +echo "1. BUILT ROOTFS — what the ISO was actually made from" +echo " $ROOTFS ($(du -sh "$ROOTFS" 2>/dev/null | cut -f1))" +echo "==============================================================" + +echo "-- build identity (the only field that proves WHICH image) --" +grep -E '^build_label|^profile|^built_at' "$ROOTFS/etc/spagat/appliance-info.toml" 2>/dev/null | sed 's/^/ /' + +echo "-- staged binaries --" +BD="$ROOTFS/usr/local/bin" +echo " count: $(ls -1 "$BD" 2>/dev/null | wc -l)" +echo " control (a name that must NOT exist): $(ls -1 "$BD"/zzz-not-a-real-binary 2>/dev/null | wc -l) (must be 0)" + +echo "-- appliance source staged (BUG-N182: upstream-drift-detector chdirs here) --" +if [ -d "$ROOTFS/opt/spagat/appliance-src" ]; then + echo " PRESENT, $(find "$ROOTFS/opt/spagat/appliance-src" -maxdepth 1 | wc -l) top-level entries" +else + echo " *** ABSENT — the drift detector will fail 200/CHDIR at first boot ***" +fi + +echo "-- SSH: is a key actually baked in? --" +# The kickstart writes authorized_keys at INSTALL time from a template +# placeholder, so the rootfs shows the KICKSTART, not the file. +KS=$(find "$ROOTFS" -name 'kickstart*.cfg' -o -name '*.ks.json' 2>/dev/null | head -n 1) +echo " kickstart in rootfs: ${KS:-}" + +echo "-- tmpfiles: the audit tier must stay 0750 --" +grep -hE '^d[[:space:]]+/var/spagat/audit[[:space:]]' "$ROOTFS"/etc/tmpfiles.d/*.conf 2>/dev/null | sed 's/^/ /' +echo " (widening this is never the right fix for an EACCES)" +echo +fi + +# ------------------------------------------------------------------- 2. ISO -- +if [ -n "$ISO" ] && [ -f "$ISO" ]; then +echo "==============================================================" +echo "2. ISO + sidecars" +echo " $ISO" +echo "==============================================================" +echo " bytes : $(stat -c%s "$ISO")" +echo " sha256: $(sha256sum "$ISO" | cut -d' ' -f1)" +if [ "$(id -u)" -ne 0 ]; then + echo " ⚠ NOT ROOT — the sidecars are 0600 root. Unreadable here reads as" + echo " MISMATCH even when the ISO is fine. Re-run as root for section 2." +fi +for x in sha256 manifest.json sig; do + f="$ISO.$x" + if [ -r "$f" ]; then echo " readable: $(basename "$f") ($(stat -c%s "$f") bytes)" + elif [ -e "$f" ]; then echo " EXISTS BUT UNREADABLE: $(basename "$f")" + else echo " absent: $(basename "$f")"; fi +done + +echo "-- the four hashes must agree --" +calc=$(sha256sum "$ISO" | cut -d' ' -f1) +side=$(grep -oE '[0-9a-f]{64}' "$ISO.sha256" 2>/dev/null | head -n1) +man=$(jq -r '.iso_sha256 // empty' "$ISO.manifest.json" 2>/dev/null) +# NOTE: the "passport" the docs mention is emitted as .sig. There is no +# *passport*.json anywhere; not finding that filename is NOT evidence the +# check does not apply. +pp=$(jq -r '.rpm_sha256 // empty' "$ISO.sig" 2>/dev/null) +printf ' %-22s %s\n' recomputed "$calc" +printf ' %-22s %s\n' .sha256 "${side:-}" +printf ' %-22s %s\n' manifest "${man:-}" +printf ' %-22s %s\n' sig.rpm_sha256 "${pp:-}" +u=$(printf '%s\n%s\n%s\n%s\n' "$calc" "$side" "$man" "$pp" | sort -u | wc -l) +[ "$u" -eq 1 ] && echo " -> ALL AGREE" || echo " -> $u DISTINCT VALUES" +u2=$(printf '%s\n%s\n' "$calc" "0000000000000000000000000000000000000000000000000000000000000000" | sort -u | wc -l) +[ "$u2" -eq 2 ] && echo " control: a wrong hash yields 2 values -> the comparison discriminates" \ + || echo " *** control did not fail — this comparison proves nothing ***" + +echo "-- 🚨 IS AN SSH KEY BAKED INTO THE ISO'S KICKSTART? --" +# This is the check people skip and then spend a day on "the appliance +# refuses my key". Read it out of the ISO itself. +if command -v xorriso >/dev/null 2>&1; then + tmpks=$(mktemp) + xorriso -osirrox on -indev "$ISO" -extract /spagat/kickstart.cfg "$tmpks" >/dev/null 2>&1 || true + if [ -s "$tmpks" ]; then + n=$(grep -c 'ssh-ed25519\|ssh-rsa' "$tmpks" 2>/dev/null) || n=0 + echo " ssh public keys in the ISO kickstart: $n" + if [ "$n" -gt 0 ]; then + grep -oE '(ssh-ed25519|ssh-rsa) [A-Za-z0-9+/=]{20}' "$tmpks" | sed 's/^/ /' + echo " -> SSH WILL WORK for whoever holds the matching private key" + else + echo " -> NO KEY BAKED IN. authorized_keys ships EMPTY." + echo " The serial console is your only way in. Re-export" + echo " SPAGAT_OPERATOR_AUTHORIZED_KEY and rebuild the ISO." + fi + # `n=$(grep -c ...) || n=0`, never `$(grep -c ... || echo 0)`: + # grep -c PRINTS 0 and EXITS 1 on no match, so the inline form emits + # the two-line string "0\n0" and every later numeric test on it dies. + _kc=$(grep -c 'zzz-not-a-real-key' "$tmpks") || _kc=0 + echo " control (a token that must NOT be there): $_kc (must be 0)" + echo " static IP in the kickstart:" + grep -oE 'Address=[0-9./]+' "$tmpks" 2>/dev/null | sort -u | sed 's/^/ /' + echo " root password: $(grep -oE '"crypted": true, "text": "[^"]{0,3}' "$tmpks" 2>/dev/null | sed 's/.*"text": "//') ('*' = LOCKED)" + else + echo " could not extract /spagat/kickstart.cfg from the ISO" + fi + rm -f "$tmpks" 2>/dev/null || true +else + echo " xorriso not available — cannot read the kickstart out of the ISO" +fi +echo +fi + +# ----------------------------------------------------------------- 3. GUEST -- +if [ "$GUEST" -eq 1 ]; then +echo "==============================================================" +echo "3. GUEST — what the running/installed system actually did" +echo "==============================================================" +SER=$(ls -1t "$VM_DIR_WSL"/${SERIAL_LOG_PREFIX}-*.log 2>/dev/null | head -n 1) +if [ -z "$SER" ]; then echo " no serial log under $VM_DIR_WSL"; exit 0; fi +echo " serial: $SER ($(stat -c%s "$SER") bytes)" + +# `-a` on EVERY grep: the serial log contains NUL bytes and without it grep +# treats the file as binary and prints NOTHING — which reads exactly like +# "the line is not there". +SLICE=$(mktemp) +STAMP=$(basename "$SER" .log); STAMP=${STAMP#${SERIAL_LOG_PREFIX}-} +OFF=$(cat "$ISO_OUT_ROOT/serial-offset-${STAMP}" 2>/dev/null || echo 0) +tail -c +$((OFF+1)) "$SER" | tr -d '\000' | sed -E 's/\x1b\[[0-9;]*m//g' > "$SLICE" +echo " reading only past offset $OFF ($(stat -c%s "$SLICE") bytes belong to this run)" + +echo "-- which system booted? --" +echo " kernel boots in this run: $(grep -ac 'Linux version' "$SLICE")" +grep -ao 'root=[^ ]*' "$SLICE" | sort -u | sed 's/^/ /' +echo " root=/dev/ram0 + isolinux = the INSTALLER live env" +echo " root=PARTUUID=... = the INSTALLED system" +grep -ao 'running in system mode' "$SLICE" | head -n 2 | sed 's/^/ /' +echo " (anything logged BEFORE that line is the initrd's systemd, a" +echo " different /etc — never compare timestamps across the boundary)" + +echo "-- build identity the guest self-reports --" +grep -ao 'spagat-librarian-iter[0-9]*-[a-f0-9]*' "$SLICE" | sort -u | sed 's/^/ /' + +echo "-- credential injection (the keystone) --" +for p in 'SPAGAT_OP_DISCOVERY' 'mount=mounted+nonempty' 'PQ verify-key loaded' \ + 'bundle PQ signature verified' 'in-process priv-drop succeeded' \ + 'audit log handed to the priv-drop target' 'finished providers=' 'AbsentLegacy'; do + _n=$(grep -ac "$p" "$SLICE") || _n=0 + printf ' %-46s %s\n' "$p" "$_n" +done +echo " PASS = 'finished providers=N' with N>=1 AND operator_tree != AbsentLegacy" +grep -ao 'finished providers=[0-9]*' "$SLICE" | sort | uniq -c | sed 's/^/ /' +grep -aE 'hab-credentials-injector.*(FATAL|Failed|exited)' "$SLICE" | tail -n 3 | cut -c1-170 | sed 's/^/ /' + +echo "-- ssh reachability from here --" +echo " guest IP (from the kickstart): $GUEST_IP_BARE" +if command -v nc >/dev/null 2>&1; then + if nc -z -w3 "$GUEST_IP_BARE" 22 2>/dev/null; then echo " port 22: OPEN"; else echo " port 22: closed/unreachable"; fi +else + echo " (nc not available — skip)" +fi +echo "-- console / TUI --" +_n=$(grep -ac 'Started Spagat-Librarian Kanban TUI on tty1' "$SLICE") || _n=0 +printf ' %-46s %s\n' 'Started Spagat-Librarian Kanban TUI on tty1' "$_n" +echo " ('FAIL spagat-console: container not running' is a RED HERRING —" +echo " it asserts on a retired container, not the live TUI)" +rm -f "$SLICE" +fi diff --git a/staging/vm-lab/scripts/50-verify-boot.sh b/staging/vm-lab/scripts/50-verify-boot.sh new file mode 100644 index 00000000..cb6b56d1 --- /dev/null +++ b/staging/vm-lab/scripts/50-verify-boot.sh @@ -0,0 +1,99 @@ +#!/bin/bash +# 50-verify-boot — is the guest alive, and did THIS boot do anything? +# +# The only unambiguous instrument while root is locked is whether the serial +# log GROWS. Everything else is inference: +# +# * `vmrun list` omits GUI-started VMs entirely — absence is not "off". +# * A CPU reading of 0.05 usually means the counter has not moved yet, not +# that the guest is idle. Measure a DELTA over ~25 s. +# * A black screen at t≈130 s is normal; the TUI starts at t≈133 s. +# * Elapsed time is not evidence of a stall. Read the step, not the clock. +set -u +# --- locate + load the config, FAIL CLOSED -------------------------------- +# ${BASH_SOURCE[0]} may point at a COPY in /tmp: the standard WSL workflow +# here is: tr -d CR < script > /tmp/x.sh ; bash /tmp/x.sh (strips CRLF), +# which breaks any path computed relative to the script. Try the plausible +# locations and REFUSE to continue if none has the config - a previous +# version merely failed to source it and carried on, printing a healthy +# looking first section before dying on unbound variables further down. +_here="$(cd "$(dirname "${BASH_SOURCE[0]}")" 2>/dev/null && pwd)" +CFG="" +for _c in "${VM_LAB_DIR:-}/config/vm-lab.env" \ + "$_here/../config/vm-lab.env" \ + "$_here/config/vm-lab.env" \ + "$PWD/config/vm-lab.env" \ + "$PWD/../config/vm-lab.env"; do + if [ -n "$_c" ] && [ -f "$_c" ]; then CFG="$_c"; break; fi +done +if [ -z "$CFG" ]; then + echo "FATAL: cannot locate config/vm-lab.env" >&2 + echo " Run from deploy/vm-lab/, or: export VM_LAB_DIR=/path/to/deploy/vm-lab" >&2 + exit 78 +fi +# shellcheck source=../config/vm-lab.env +. "$CFG" +HERE="$_here" + +STAMP="" +SAMPLE=25 +while [ $# -gt 0 ]; do + case "$1" in + --stamp) STAMP="$2"; shift 2 ;; + --sample-sec) SAMPLE="$2"; shift 2 ;; + *) echo "unknown arg: $1"; exit 64 ;; + esac +done + +if [ -n "$STAMP" ]; then + SER="$VM_DIR_WSL/${SERIAL_LOG_PREFIX}-${STAMP}.log" +else + SER=$(ls -1t "$VM_DIR_WSL"/${SERIAL_LOG_PREFIX}-*.log 2>/dev/null | head -n 1) +fi +[ -n "$SER" ] && [ -f "$SER" ] || { echo "no serial log found (looked in $VM_DIR_WSL)"; exit 2; } + +echo "=== serial log ===" +echo " $SER" +echo " bytes: $(stat -c%s "$SER") mtime: $(stat -c%y "$SER" | cut -c1-19)" + +echo +echo "=== THE instrument: does it grow over ${SAMPLE}s? ===" +a=$(stat -c%s "$SER"); sleep "$SAMPLE"; b=$(stat -c%s "$SER") +echo " $a -> $b delta=$((b-a))" +if [ "$b" -gt "$a" ]; then + echo " GROWING — the guest is doing work" +else + echo " no growth in this sample." + echo " That is NOT by itself a stall: a single long unit (a release build," + echo " a big cargo install) can be quiet for minutes. Check the process" + echo " and the last log line before concluding anything." +fi + +echo +echo "=== is VMware running it? (note: GUI-started VMs are invisible here) ===" +if [ -x "$VMRUN_WSL" ]; then + "$VMRUN_WSL" -T ws list 2>/dev/null | sed 's/^/ /' + "$VMRUN_WSL" -T ws list 2>/dev/null | grep -q "$VM_NAME" \ + && echo " $VM_NAME: listed as running" \ + || echo " $VM_NAME: NOT listed — may still be running if started from the GUI" +fi + +echo +echo "=== last readable serial lines (NULs stripped, ANSI removed) ===" +tail -c 6000 "$SER" | tr -d '\000' | sed -E 's/\x1b\[[0-9;]*m//g' | tail -n 15 | cut -c1-170 | sed 's/^/ /' + +echo +echo "=== failed units this boot ===" +tail -c 400000 "$SER" | tr -d '\000' | sed -E 's/\x1b\[[0-9;]*g//g' \ + | grep -aE 'SPAGAT-DIAG: failed unit|Failed to start' | tail -n 12 | cut -c1-170 | sed 's/^/ /' +echo " (blank = none seen in the tail)" + +echo +echo "=== console/TUI markers ===" +# -a on every grep: NUL bytes otherwise make grep print nothing at all. +for p in 'Started Spagat-Librarian Kanban TUI on tty1' 'spagat-console.service' 'Reached target'; do + _n=$(grep -ac "$p" "$SER") || _n=0 + printf ' %-46s %s\n' "$p" "$_n" +done +_c=$(grep -ac 'zzz-not-a-real-marker' "$SER") || _c=0 +echo " control (must be 0): $_c" diff --git a/staging/vm-lab/scripts/60-ssh.sh b/staging/vm-lab/scripts/60-ssh.sh new file mode 100644 index 00000000..81315e34 --- /dev/null +++ b/staging/vm-lab/scripts/60-ssh.sh @@ -0,0 +1,104 @@ +#!/bin/bash +# 60-ssh — connect to the appliance, and when that fails, say WHY. +# +# A bare `ssh: connection refused` sends people looking for an onboarding +# gate or a credential bug. Almost always the real answer is one of three +# things this script checks explicitly. +set -u +# --- locate + load the config, FAIL CLOSED -------------------------------- +# ${BASH_SOURCE[0]} may point at a COPY in /tmp: the standard WSL workflow +# here is: tr -d CR < script > /tmp/x.sh ; bash /tmp/x.sh (strips CRLF), +# which breaks any path computed relative to the script. Try the plausible +# locations and REFUSE to continue if none has the config - a previous +# version merely failed to source it and carried on, printing a healthy +# looking first section before dying on unbound variables further down. +_here="$(cd "$(dirname "${BASH_SOURCE[0]}")" 2>/dev/null && pwd)" +CFG="" +for _c in "${VM_LAB_DIR:-}/config/vm-lab.env" \ + "$_here/../config/vm-lab.env" \ + "$_here/config/vm-lab.env" \ + "$PWD/config/vm-lab.env" \ + "$PWD/../config/vm-lab.env"; do + if [ -n "$_c" ] && [ -f "$_c" ]; then CFG="$_c"; break; fi +done +if [ -z "$CFG" ]; then + echo "FATAL: cannot locate config/vm-lab.env" >&2 + echo " Run from deploy/vm-lab/, or: export VM_LAB_DIR=/path/to/deploy/vm-lab" >&2 + exit 78 +fi +# shellcheck source=../config/vm-lab.env +. "$CFG" +HERE="$_here" + +CMD="" +while [ $# -gt 0 ]; do + case "$1" in + --cmd) CMD="$2"; shift 2 ;; + *) echo "usage: $0 [--cmd '']"; exit 64 ;; + esac +done + +KEY="$SSH_KEY_DIR/$SSH_KEY_NAME" +TARGET="$SSH_USER@$GUEST_IP_BARE" + +echo "=== target ===" +echo " $TARGET key: $KEY" + +echo +echo "=== 1. do we even have the key? ===" +if [ -f "$KEY" ]; then + echo " present: $(ssh-keygen -lf "$KEY.pub" 2>/dev/null)" +else + echo " *** MISSING — run 20-make-ssh-key.sh, then REBUILD THE ISO." + echo " A key created now cannot reach an already-installed appliance:" + echo " root is locked and authorized_keys was written at install time." + exit 3 +fi + +echo +echo "=== 2. is the guest reachable at all? ===" +if command -v ping >/dev/null 2>&1 && ping -c1 -W2 "$GUEST_IP_BARE" >/dev/null 2>&1; then + echo " ICMP: replies" +else + echo " ICMP: no reply (may be filtered — not conclusive on its own)" +fi +if command -v nc >/dev/null 2>&1; then + if nc -z -w3 "$GUEST_IP_BARE" 22 2>/dev/null; then + echo " port 22: OPEN" + else + echo " port 22: CLOSED/unreachable" + echo " Common causes, in the order they actually occur:" + echo " a) the guest never got $GUEST_STATIC_IP — the kickstart writes" + echo " /etc/systemd/network/10-eth0-static.network at install time;" + echo " check section 3 of 40-check-staging.sh --iso ..." + echo " b) the VM is not running, or is still installing" + echo " c) the MAC drifted, so the NAT lease moved to another address" + fi +fi + +echo +echo "=== 3. try it ===" +# BatchMode: never sit at a password prompt — root is locked and there is no +# password to give, so an interactive prompt is always a dead end here. +SSH_OPTS="-i $KEY -o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ConnectTimeout=8" +# shellcheck disable=SC2086 +if [ -n "$CMD" ]; then + ssh $SSH_OPTS "$TARGET" "$CMD" + rc=$? +else + ssh $SSH_OPTS "$TARGET" 'echo "connected as $(id -un)@$(hostname)"; cat /etc/spagat/appliance-info.toml 2>/dev/null | grep -E "^build_label"' + rc=$? +fi + +if [ "$rc" -ne 0 ]; then + echo + echo " ssh exited $rc." + echo " If it was 'Permission denied (publickey)': the ISO did not carry" + echo " this key. Both SPAGAT_OPERATOR_AUTHORIZED_KEY and" + echo " IPHASE6_TEST_SSH_PUBKEY default to EMPTY, so a build where neither" + echo " was exported ships an EMPTY authorized_keys. Verify with:" + echo " scripts/40-check-staging.sh --iso " + echo " and rebuild after exporting the key. There is no way to add it to" + echo " an installed appliance without console access." +fi +exit $rc diff --git a/staging/vm-lab/scripts/90-teardown.ps1 b/staging/vm-lab/scripts/90-teardown.ps1 new file mode 100644 index 00000000..9b46f775 --- /dev/null +++ b/staging/vm-lab/scripts/90-teardown.ps1 @@ -0,0 +1,132 @@ +# 90-teardown - return the VM to a fresh-disk state. +# +# ASCII-ONLY BY POLICY - see the note at the top of 10-create-vm.ps1. Windows +# PowerShell 5.1 reads .ps1 as ANSI without a BOM, so one non-ASCII character +# corrupts a string literal and produces a cascade of misleading parse errors. +# +# NOTHING IS DELETED. Every displaced file is renamed with a +# `.stashed-` suffix; recovery is a rename back. That is both the +# project rule (no destructive action without an explicit per-instance go) +# and good practice - the previous boot disk is the only post-mortem material +# for whatever went wrong on it. +# +# PRESERVED, always: +# * operator-config.vmdk / operator-config-flat.vmdk - the credential +# medium on scsi0:1. install-from-iso never touches scsi0:1, so it is +# meant to outlive any number of reinstalls. Regenerating it is a +# separate, deliberate act. +# * every serial0-*.log - the only diagnostic record of past boots. +# * the .vmx - it holds the pinned MAC/UUID that keep the guest's IP. +# +# WHY THE WHOLE CHAIN GOES, NOT JUST THE DISK: if any of the snapshot delta, +# .vmsd, .vmsn or NVRAM survives, UEFI's removable-media fallback finds the +# old ESP's \EFI\BOOT\BOOTX64.EFI and boots the PREVIOUS image. bios.bootOrder +# is ignored on EFI VMs, and deleting NVRAM alone does not help because UEFI +# re-detects the disk. +[CmdletBinding()] +param( + # Also stash the VMX (forces 10-create-vm.ps1 to regenerate it from the + # template). Off by default - the VMX holds the pinned MAC/UUID. + [switch]$IncludeVmx, + # Required. Teardown is destructive-shaped even though it only renames. + [switch]$Confirm +) +$ErrorActionPreference = 'Stop' + +$here = Split-Path -Parent $MyInvocation.MyCommand.Path +# Same parser as 10-create-vm.ps1 - see the trailing-comment note there. +$cfg = @{} +foreach ($line in Get-Content (Join-Path $here '..\config\vm-lab.env')) { + if ($line -match '^\s*#' -or $line -match '^\s*$') { continue } + if ($line -notmatch '^\s*([A-Za-z_][A-Za-z0-9_]*)=\s*(.*)$') { continue } + $k = $Matches[1] + $raw = $Matches[2] + if ($raw -match '^"([^"]*)"') { $v = $Matches[1] } + elseif ($raw -match "^'([^']*)'") { $v = $Matches[1] } + else { $v = (($raw -split '#', 2)[0]).Trim() } + $v = [regex]::Replace($v, '\$\{([A-Za-z_][A-Za-z0-9_]*)\}', { param($m) if ($cfg.ContainsKey($m.Groups[1].Value)) { $cfg[$m.Groups[1].Value] } else { '' } }) + $v = $v -replace '\\\\', '\' + $cfg[$k] = $v +} +$vmName = $cfg['VM_NAME'] +$vmDir = $cfg['VM_DIR_WIN'] +$vmrun = Join-Path $cfg['VMWARE_DIR_WIN'] 'vmrun.exe' +$med = $cfg['OPERATOR_MEDIUM_BASENAME'] + +if (-not $Confirm) { + Write-Output "This will stash the boot disk + firmware state of '$vmName' in:" + Write-Output " $vmDir" + Write-Output "" + Write-Output "Nothing is deleted - files are renamed .stashed-." + Write-Output "The operator medium and all serial logs are preserved." + Write-Output "" + Write-Output "Re-run with -Confirm to proceed." + exit 0 +} + +Write-Output "=== refuse if the VM is running ===" +if (Test-Path $vmrun) { + $running = & $vmrun -T ws list + if ($running -match [regex]::Escape($vmName)) { + throw "$vmName is RUNNING. Stop it first. Refusing to touch its disk." + } + Write-Output " not running" +} + +$ts = Get-Date -Format 'yyyyMMddTHHmmssZ' +$targets = @( + "$vmName.vmdk", + "$vmName-000001.vmdk", "$vmName-000002.vmdk", + "$vmName.vmsd", + "$vmName-Snapshot1.vmsn", "$vmName-Snapshot2.vmsn", + "nvram", + "$vmName.vmx.lck" +) +if ($IncludeVmx) { $targets += "$vmName.vmx" } + +Write-Output "" +Write-Output "=== stashing (NOT deleting) ===" +$moved = 0 +foreach ($n in $targets) { + $p = Join-Path $vmDir $n + if (Test-Path $p) { + Move-Item -LiteralPath $p -Destination "$p.stashed-$ts" -Force + Write-Output (" moved {0,-32} -> {0}.stashed-{1}" -f $n, $ts) + $moved++ + } +} +if ($moved -eq 0) { Write-Output " nothing to stash (already clean)" } + +Write-Output "" +Write-Output "=== the credential medium must have survived ===" +foreach ($n in @("$med.vmdk", "$med-flat.vmdk")) { + $p = Join-Path $vmDir $n + if (Test-Path $p) { + " OK {0,-30} {1,14:n0} bytes" -f $n, (Get-Item $p).Length + } else { + Write-Output " ABSENT $n (the next install will boot KEYLESS)" + } +} +$flatPath = Join-Path $vmDir "$med-flat.vmdk" +if (Test-Path $flatPath) { + $want = [int64]$cfg['OPERATOR_MEDIUM_FLAT_BYTES'] + $got = (Get-Item $flatPath).Length + if ($got -ne $want) { + Write-Output " *** flat size $got != expected $want - this is not the verified medium ***" + } else { + Write-Output " flat size matches the verified medium" + } +} + +Write-Output "" +Write-Output "=== serial logs preserved ===" +Get-ChildItem $vmDir -Filter "$($cfg['SERIAL_LOG_PREFIX'])-*.log" -ErrorAction SilentlyContinue | + ForEach-Object { " {0,-46} {1,14:n0}" -f $_.Name, $_.Length } + +Write-Output "" +Write-Output "=== stashed this run (recover by renaming back) ===" +Get-ChildItem $vmDir -Filter "*.stashed-$ts" -ErrorAction SilentlyContinue | + ForEach-Object { " {0,-58} {1,14:n0}" -f $_.Name, $_.Length } + +Write-Output "" +Write-Output "NEXT: scripts/10-create-vm.ps1 (recreates the boot disk; add -RefreshVmxOnly if you stashed the VMX)" From 001d26beed78f135f8ac6d3bded69e65ddb73dfe Mon Sep 17 00:00:00 2001 From: Daniel Casota Date: Mon, 31 Aug 2026 12:58:57 +0200 Subject: [PATCH 2/2] vm-lab: fix six defects found while reusing it for a permutation matrix The documented per-run overrides did not work. config/vm-lab.env assigned every variable unconditionally (VM_NAME="spagat-smoke"), and every .sh script sources it after startup, so an exported value was overwritten. README's own examples - VM_NAME=my-lab ./scripts/00-preflight.sh and GUEST_VCPUS=4 powershell -File scripts/10-create-vm.ps1 - therefore did nothing. Converted all 24 assignments to : "${VAR:=default}" form; values are byte-identical to before, including the backslash-bearing Windows paths. The PowerShell side could not be overridden at all. Both .ps1 files parse the .env into a hashtable and never consulted $env:. Their shared parser now accepts the new form (and still the old one, so a stale copy keeps working) and prefers an environment value. Verified by running the parser: VM_NAME=perm-p03 now yields VM_DIR_WIN=C:\spagat-iso-build\vm\perm-p03. The BIOS UUID was hardcoded in the VMX template, not a placeholder. Every VM built from the template shared one UUID - and because ethernet0.addressType is "generated", VMware derives the MAC from it, so they also shared one MAC and one DHCP lease. It is now @@UUID_BIOS@@ fed from GUEST_UUID_BIOS, defaulting to the previous value so existing VMs are unaffected, and 10-create-vm.ps1 warns when the last three bytes of MAC and UUID disagree. 90-teardown.ps1 enumerated the snapshot chain by name. The fixed list covered exactly two deltas and two .vmsn files, so a VM that had reached -000003.vmdk left an orphan - which defeats the file's own stated rationale, since UEFI's removable-media fallback then finds the surviving ESP and boots the previous image. Now globbed by pattern. 50-verify-boot.sh:87 had a broken ANSI strip. s/\x1b\[[0-9;]*g//g - trailing g where line 83 correctly uses m, so SGR sequences were not stripped from the failed-units section. 20-make-ssh-key.sh did not notice a failed ssh-keygen. No set -e, so it fell through to cat "$KEY.pub" and reported a missing file instead of the real cause. Now exits 5. Also documented, not changed: GUEST_GATEWAY is read by nothing, because the gateway and DNS are hardcoded inside the kickstart template even though the address is templated. That file is byte-exact and policed by check-drift.sh, so the fix belongs upstream; the variable now says so. Verified: bash -n clean on all 7 shell scripts, both .ps1 parse under pwsh and the shared parser was executed against the real .env, kickstart sha256 unchanged so check-drift.sh still passes, both .ps1 still ASCII-only with CRLF endings. Signed-off-by: Daniel Casota --- staging/vm-lab/README.md | 6 +- .../vm-lab/config/spagat-smoke.vmx.template | 4 +- staging/vm-lab/config/vm-lab.env | 63 ++++++++++++------- staging/vm-lab/scripts/10-create-vm.ps1 | 41 ++++++++++-- staging/vm-lab/scripts/20-make-ssh-key.sh | 8 ++- staging/vm-lab/scripts/50-verify-boot.sh | 2 +- staging/vm-lab/scripts/90-teardown.ps1 | 46 +++++++++----- 7 files changed, 121 insertions(+), 49 deletions(-) diff --git a/staging/vm-lab/README.md b/staging/vm-lab/README.md index fc986ddb..658f748f 100644 --- a/staging/vm-lab/README.md +++ b/staging/vm-lab/README.md @@ -137,7 +137,11 @@ Two that catch people out: looks like a broken ISO. - **The BIOS UUID must stay next to the MAC.** With `ethernet0.addressType="generated"`, VMware *derives* the MAC from the UUID. - Drop the UUID and both regenerate, the `192.168.225.140` lease moves, and + Both now come from `vm-lab.env` (`GUEST_MAC`, `GUEST_UUID_BIOS`) rather than + being hardcoded in the template, so a second VM can be given its own pair + instead of silently reusing this one's; `10-create-vm.ps1` warns if their + last three bytes disagree. Drop the UUID and both regenerate, the + `192.168.225.140` lease moves, and every hardcoded address in the runbooks quietly points at nothing. ### 3. The operator medium on `scsi0:1` is the credential channel — and is never regenerated here diff --git a/staging/vm-lab/config/spagat-smoke.vmx.template b/staging/vm-lab/config/spagat-smoke.vmx.template index 02b34852..592e3a80 100644 --- a/staging/vm-lab/config/spagat-smoke.vmx.template +++ b/staging/vm-lab/config/spagat-smoke.vmx.template @@ -74,8 +74,8 @@ ethernet0.generatedAddressOffset = "0" # last bytes (…49 53 65 1c) are the MAC's (…53:65:1c). Drop the UUID and # VMware regenerates both, the 192.168.225.140 lease moves, and every # hardcoded address in the runbooks silently points at nothing. -uuid.bios = "56 4d a0 3f 30 d3 c1 e8-72 40 ad 48 49 53 65 1c" -uuid.location = "56 4d a0 3f 30 d3 c1 e8-72 40 ad 48 49 53 65 1c" +uuid.bios = "@@UUID_BIOS@@" +uuid.location = "@@UUID_BIOS@@" # --- serial console (the only diagnostic channel) -------------------------- serial0.present = "TRUE" diff --git a/staging/vm-lab/config/vm-lab.env b/staging/vm-lab/config/vm-lab.env index 4bcd4f07..51e0ea90 100644 --- a/staging/vm-lab/config/vm-lab.env +++ b/staging/vm-lab/config/vm-lab.env @@ -12,20 +12,20 @@ # "VMX I/O: No such file or directory". # ---------------------------------------------------------------- identity -- -VM_NAME="spagat-smoke" +: "${VM_NAME:=spagat-smoke}" # ------------------------------------------------------------------- paths -- -VM_ROOT_WIN='C:\spagat-iso-build\vm' -VM_ROOT_WSL='/mnt/c/spagat-iso-build/vm' -VM_DIR_WIN="${VM_ROOT_WIN}\\${VM_NAME}" -VM_DIR_WSL="${VM_ROOT_WSL}/${VM_NAME}" +: "${VM_ROOT_WIN:=C:\spagat-iso-build\vm}" +: "${VM_ROOT_WSL:=/mnt/c/spagat-iso-build/vm}" +: "${VM_DIR_WIN:=${VM_ROOT_WIN}\\${VM_NAME}}" +: "${VM_DIR_WSL:=${VM_ROOT_WSL}/${VM_NAME}}" -VMWARE_DIR_WIN='C:\Program Files\VMware\VMware Workstation' -VMRUN_WSL="/mnt/c/Program Files/VMware/VMware Workstation/vmrun.exe" -VDISKMANAGER_WSL="/mnt/c/Program Files/VMware/VMware Workstation/vmware-vdiskmanager.exe" +: "${VMWARE_DIR_WIN:=C:\Program Files\VMware\VMware Workstation}" +: "${VMRUN_WSL:=/mnt/c/Program Files/VMware/VMware Workstation/vmrun.exe}" +: "${VDISKMANAGER_WSL:=/mnt/c/Program Files/VMware/VMware Workstation/vmware-vdiskmanager.exe}" # Where ISO builds land on the build host (Ph5 WSL). -ISO_OUT_ROOT="/home/dcaso/work" +: "${ISO_OUT_ROOT:=/home/dcaso/work}" # ------------------------------------------------------------------- disks -- # DEFAULT: 50 GB thin, in a SINGLE FILE. @@ -46,9 +46,9 @@ ISO_OUT_ROOT="/home/dcaso/work" # (spagat-smoke-s001.vmdk, -s002.vmdk, ...), which is more files, not less # space, and the split form is the one that tends to leave orphaned extents # behind after a teardown. -BOOT_DISK_SIZE="50GB" +: "${BOOT_DISK_SIZE:=50GB}" # 50 GiB = 104857600 512-byte sectors -> descriptor `RW 104857600 SPARSE` -BOOT_DISK_ADAPTER="lsilogic" +: "${BOOT_DISK_ADAPTER:=lsilogic}" # vmware-vdiskmanager -t: # 0 = monolithicSparse thin, SINGLE FILE <-- what we use # 1 = twoGbMaxExtentSparse thin, split into 2 GB extents @@ -56,37 +56,52 @@ BOOT_DISK_ADAPTER="lsilogic" # 3 = twoGbMaxExtentFlat thick, split # Type 0 keeps the whole disk in one .vmdk whose extent line references # itself, so a teardown moves exactly one file aside. -BOOT_DISK_TYPE="0" +: "${BOOT_DISK_TYPE:=0}" # The operator medium (credential channel, scsi0:1). 250 MiB monolithicFlat. # NEVER regenerated by these scripts — see README "The operator medium". -OPERATOR_MEDIUM_BASENAME="operator-config" -OPERATOR_MEDIUM_FLAT_BYTES="262144000" +: "${OPERATOR_MEDIUM_BASENAME:=operator-config}" +: "${OPERATOR_MEDIUM_FLAT_BYTES:=262144000}" # ------------------------------------------------------------------ guest --- # The pinned MAC is what holds the 192.168.225.140 DHCP/NAT lease. Changing it # very likely changes the IP and breaks every hardcoded reference in the # runbooks. Change only with an explicit reason. -GUEST_MAC="00:0c:29:53:65:1c" -GUEST_STATIC_IP="192.168.225.140/24" -GUEST_GATEWAY="192.168.225.2" -GUEST_IP_BARE="192.168.225.140" -GUEST_MEM_MB="4096" +: "${GUEST_MAC:=00:0c:29:53:65:1c}" + +# VMware DERIVES a generated MAC from the BIOS UUID, so these two must move +# together: the last three bytes of the UUID (49 53 65 1c -> ...53:65:1c) +# are the last three bytes of the MAC. It was previously hardcoded in the +# VMX template, which meant every VM built from that template shared one +# UUID - and therefore one MAC and one DHCP lease. 10-create-vm.ps1 now +# checks the two agree and warns if they do not. +: "${GUEST_UUID_BIOS:=56 4d a0 3f 30 d3 c1 e8-72 40 ad 48 49 53 65 1c}" +: "${GUEST_STATIC_IP:=192.168.225.140/24}" +# NOTE: no script reads this. The gateway and DNS are hardcoded inside +# kickstart/photon-appliance.ks.template.json (the "Gateway="/"DNS=" line +# of the networkd unit) even though the address itself is templated. That +# file is a byte-exact copy policed by kickstart/check-drift.sh, so it +# cannot be corrected here - the fix belongs upstream in +# src/tools/iso-build/iso-phase6-kickstart-template.cfg. Kept so the +# intended value is recorded next to the address it goes with. +: "${GUEST_GATEWAY:=192.168.225.2}" +: "${GUEST_IP_BARE:=192.168.225.140}" +: "${GUEST_MEM_MB:=4096}" # DEFAULT: 2 vCPU. Enough for a stock Photon guest and cheap on a laptop. # The SPAGAT appliance profile was measured at 4 -- raise it per-run with # GUEST_VCPUS=4 ./scripts/10-create-vm.ps1 -GUEST_VCPUS="2" +: "${GUEST_VCPUS:=2}" # ---------------------------------------------------------------- ssh/auth -- # 🚨 SSH access is decided AT ISO BUILD TIME, not after install. Both of these # default to EMPTY in iso-phase6, which ships an appliance with an empty # authorized_keys and root LOCKED — i.e. no way in except the serial console. # See README "SSH access". -SSH_KEY_DIR="${HOME}/.ssh" -SSH_KEY_NAME="spagat-vm-lab-ed25519" -SSH_USER="operator" +: "${SSH_KEY_DIR:=${HOME}/.ssh}" +: "${SSH_KEY_NAME:=spagat-vm-lab-ed25519}" +: "${SSH_USER:=operator}" # ------------------------------------------------------------------ serial -- # The serial log is the ONLY diagnostic channel while root is locked. Name it # per build so RCA logs from different images never interleave. -SERIAL_LOG_PREFIX="serial0" +: "${SERIAL_LOG_PREFIX:=serial0}" diff --git a/staging/vm-lab/scripts/10-create-vm.ps1 b/staging/vm-lab/scripts/10-create-vm.ps1 index 214fc5b0..ff3d97b3 100644 --- a/staging/vm-lab/scripts/10-create-vm.ps1 +++ b/staging/vm-lab/scripts/10-create-vm.ps1 @@ -42,12 +42,26 @@ $cfgPath = Join-Path $here '..\config\vm-lab.env' $cfg = @{} foreach ($line in Get-Content $cfgPath) { if ($line -match '^\s*#' -or $line -match '^\s*$') { continue } - if ($line -notmatch '^\s*([A-Za-z_][A-Za-z0-9_]*)=\s*(.*)$') { continue } - $k = $Matches[1] - $raw = $Matches[2] - if ($raw -match '^"([^"]*)"') { $v = $Matches[1] } - elseif ($raw -match "^'([^']*)'") { $v = $Matches[1] } - else { $v = (($raw -split '#', 2)[0]).Trim() } + # vm-lab.env uses the override-safe form : "${KEY:=VALUE}" so that an + # exported value wins. Accept the older KEY=VALUE form too, so a stale + # copy of the file still parses. + if ($line -match '^\s*:\s*"\$\{([A-Za-z_][A-Za-z0-9_]*):=(.*)\}"\s*$') { + $k = $Matches[1] + $v = $Matches[2] + } + elseif ($line -match '^\s*([A-Za-z_][A-Za-z0-9_]*)=\s*(.*)$') { + $k = $Matches[1] + $raw = $Matches[2] + if ($raw -match '^"([^"]*)"') { $v = $Matches[1] } + elseif ($raw -match "^'([^']*)'") { $v = $Matches[1] } + else { $v = (($raw -split '#', 2)[0]).Trim() } + } + else { continue } + # An environment value overrides the file, matching what the .sh scripts + # now do. Without this the README's documented per-run overrides work in + # bash but are silently ignored on the PowerShell side. + $envVal = [Environment]::GetEnvironmentVariable($k) + if ($envVal) { $cfg[$k] = $envVal; continue } $v = [regex]::Replace($v, '\$\{([A-Za-z_][A-Za-z0-9_]*)\}', { param($m) if ($cfg.ContainsKey($m.Groups[1].Value)) { $cfg[$m.Groups[1].Value] } else { '' } }) $v = $v -replace '\\\\', '\' $cfg[$k] = $v @@ -121,6 +135,7 @@ if ((Test-Path $vmxPath) -and -not $RefreshVmxOnly) { Replace('@@GUEST_VCPUS@@', $cfg['GUEST_VCPUS']). Replace('@@GUEST_MEM_MB@@', $cfg['GUEST_MEM_MB']). Replace('@@GUEST_MAC@@', $cfg['GUEST_MAC']). +$vmx = $vmx -replace '@@UUID_BIOS@@', $cfg['GUEST_UUID_BIOS'] Replace('@@OPERATOR_MEDIUM_BASENAME@@', $cfg['OPERATOR_MEDIUM_BASENAME']). Replace('@@SERIAL_LOG_WIN@@', $serialWin) # VMX files are CRLF + UTF-8 without BOM. @@ -128,6 +143,20 @@ if ((Test-Path $vmxPath) -and -not $RefreshVmxOnly) { [System.IO.File]::WriteAllText($vmxPath, ($body -replace "`r?`n", "`r`n"), $utf8NoBom) Write-Output " wrote $vmxPath" +# VMware derives a generated MAC from the BIOS UUID. If the two disagree the +# VM still boots, but the NAT lease moves and every hardcoded address in the +# runbooks points at nothing - the exact failure the pinning exists to prevent. +$macTail = ($cfg['GUEST_MAC'] -replace ':','').ToLower() +$uuidTail = ($cfg['GUEST_UUID_BIOS'] -replace '[ -]','').ToLower() +if ($macTail.Length -ge 6 -and $uuidTail.Length -ge 6) { + if ($uuidTail.Substring($uuidTail.Length - 6) -ne $macTail.Substring($macTail.Length - 6)) { + Write-Warning ("GUEST_MAC and GUEST_UUID_BIOS disagree in their last 3 bytes " + + "(mac=..." + $macTail.Substring($macTail.Length - 6) + + " uuid=..." + $uuidTail.Substring($uuidTail.Length - 6) + + "). VMware derives the MAC from the UUID; the DHCP lease will move.") + } +} + $left = Select-String -Path $vmxPath -Pattern '@@[A-Z_]+@@' if ($left) { throw "unsubstituted placeholders remain: $($left -join ', ')" } Write-Output " no placeholders left" diff --git a/staging/vm-lab/scripts/20-make-ssh-key.sh b/staging/vm-lab/scripts/20-make-ssh-key.sh index abc18741..9dde4723 100644 --- a/staging/vm-lab/scripts/20-make-ssh-key.sh +++ b/staging/vm-lab/scripts/20-make-ssh-key.sh @@ -56,7 +56,13 @@ else echo "=== generating ed25519 keypair ===" # No passphrase: this is a disposable lab key for an unattended install. # It is NOT an operator credential and must never be reused elsewhere. - ssh-keygen -t ed25519 -N '' -C "spagat-vm-lab@$(hostname)" -f "$KEY" + # No 'set -e' in this script, so a failed keygen would otherwise fall + # through to 'cat "$KEY.pub"' and report a missing file instead of the + # real cause. + if ! ssh-keygen -t ed25519 -N '' -C "spagat-vm-lab@$(hostname)" -f "$KEY"; then + echo "FAIL: ssh-keygen could not create $KEY" >&2 + exit 5 + fi fi chmod 0600 "$KEY" chmod 0644 "$KEY.pub" diff --git a/staging/vm-lab/scripts/50-verify-boot.sh b/staging/vm-lab/scripts/50-verify-boot.sh index cb6b56d1..eafe93f8 100644 --- a/staging/vm-lab/scripts/50-verify-boot.sh +++ b/staging/vm-lab/scripts/50-verify-boot.sh @@ -84,7 +84,7 @@ tail -c 6000 "$SER" | tr -d '\000' | sed -E 's/\x1b\[[0-9;]*m//g' | tail -n 15 | echo echo "=== failed units this boot ===" -tail -c 400000 "$SER" | tr -d '\000' | sed -E 's/\x1b\[[0-9;]*g//g' \ +tail -c 400000 "$SER" | tr -d '\000' | sed -E 's/\x1b\[[0-9;]*m//g' \ | grep -aE 'SPAGAT-DIAG: failed unit|Failed to start' | tail -n 12 | cut -c1-170 | sed 's/^/ /' echo " (blank = none seen in the tail)" diff --git a/staging/vm-lab/scripts/90-teardown.ps1 b/staging/vm-lab/scripts/90-teardown.ps1 index 9b46f775..837b8f51 100644 --- a/staging/vm-lab/scripts/90-teardown.ps1 +++ b/staging/vm-lab/scripts/90-teardown.ps1 @@ -38,12 +38,26 @@ $here = Split-Path -Parent $MyInvocation.MyCommand.Path $cfg = @{} foreach ($line in Get-Content (Join-Path $here '..\config\vm-lab.env')) { if ($line -match '^\s*#' -or $line -match '^\s*$') { continue } - if ($line -notmatch '^\s*([A-Za-z_][A-Za-z0-9_]*)=\s*(.*)$') { continue } - $k = $Matches[1] - $raw = $Matches[2] - if ($raw -match '^"([^"]*)"') { $v = $Matches[1] } - elseif ($raw -match "^'([^']*)'") { $v = $Matches[1] } - else { $v = (($raw -split '#', 2)[0]).Trim() } + # vm-lab.env uses the override-safe form : "${KEY:=VALUE}" so that an + # exported value wins. Accept the older KEY=VALUE form too, so a stale + # copy of the file still parses. + if ($line -match '^\s*:\s*"\$\{([A-Za-z_][A-Za-z0-9_]*):=(.*)\}"\s*$') { + $k = $Matches[1] + $v = $Matches[2] + } + elseif ($line -match '^\s*([A-Za-z_][A-Za-z0-9_]*)=\s*(.*)$') { + $k = $Matches[1] + $raw = $Matches[2] + if ($raw -match '^"([^"]*)"') { $v = $Matches[1] } + elseif ($raw -match "^'([^']*)'") { $v = $Matches[1] } + else { $v = (($raw -split '#', 2)[0]).Trim() } + } + else { continue } + # An environment value overrides the file, matching what the .sh scripts + # now do. Without this the README's documented per-run overrides work in + # bash but are silently ignored on the PowerShell side. + $envVal = [Environment]::GetEnvironmentVariable($k) + if ($envVal) { $cfg[$k] = $envVal; continue } $v = [regex]::Replace($v, '\$\{([A-Za-z_][A-Za-z0-9_]*)\}', { param($m) if ($cfg.ContainsKey($m.Groups[1].Value)) { $cfg[$m.Groups[1].Value] } else { '' } }) $v = $v -replace '\\\\', '\' $cfg[$k] = $v @@ -74,14 +88,18 @@ if (Test-Path $vmrun) { } $ts = Get-Date -Format 'yyyyMMddTHHmmssZ' -$targets = @( - "$vmName.vmdk", - "$vmName-000001.vmdk", "$vmName-000002.vmdk", - "$vmName.vmsd", - "$vmName-Snapshot1.vmsn", "$vmName-Snapshot2.vmsn", - "nvram", - "$vmName.vmx.lck" -) +# Enumerated by pattern, not by name. The previous fixed list covered exactly +# two snapshot deltas and two .vmsn files, so a VM that had reached +# -000003.vmdk left an orphan behind - which defeats the whole point stated +# at the top of this file, because UEFI's removable-media fallback then finds +# the surviving ESP and boots the PREVIOUS image. +$targets = @("$vmName.vmdk", "$vmName.vmsd", "nvram", "$vmName.vmx.lck") +$targets += (Get-ChildItem -LiteralPath $vmDir -Filter "$vmName-*.vmdk" -ErrorAction SilentlyContinue | + Where-Object { $_.Name -match "^$([regex]::Escape($vmName))-\d{6}\.vmdk$" } | + ForEach-Object { $_.Name }) +$targets += (Get-ChildItem -LiteralPath $vmDir -Filter '*.vmsn' -ErrorAction SilentlyContinue | + ForEach-Object { $_.Name }) +$targets = $targets | Select-Object -Unique if ($IncludeVmx) { $targets += "$vmName.vmx" } Write-Output ""