staging: add mission-control - execute the ISO permutation matrix

staging/mission-control/.gitattributes

@@ -0,0 +1,4 @@
+*.sh   text eol=lf
+*.env  text eol=lf
+*.tsv  text eol=lf
+*.json text eol=lf

staging/mission-control/README.md

@@ -0,0 +1,153 @@
+# mission-control — execute the ISO permutation matrix
+
+`ISO-PERMUTATION-MATRIX.md` says of itself: **"No builds were run."** Fourteen
+of its sixteen rows are resolution predictions or code reading; only two were
+ever installed. This directory is the part that was missing — it builds the
+ISOs, stages the VMs, drives the installs, and verifies the result against an
+oracle that names the PR behind every failure.
+
+It reuses the mechanics of `../vm-lab` (VMX discipline, thin disks,
+stash-never-delete teardown, the serial-log liveness instrument) and drops
+everything specific to the SPAGAT-Librarian appliance, which is out of scope.
+
+---
+
+## The one thing that makes this tractable
+
+POI's `isoInstaller` does not only read `ks=` from the kernel command line. It
+reads **`guestinfo.kickstart.data`** (base64) and `guestinfo.kickstart.url`
+through `vmtoolsd`, and `/usr/bin/vmtoolsd` is present in the installer initrd
+(`open-vm-tools` is in `packages_installer_initrd.json`).
+
+So a per-permutation kickstart is **one line in the VMX**. No ISO remaster, no
+HTTP server, no typing at a boot menu. That splits the matrix cleanly:
+
+| Layer | Axes | Cost |
+|---|---|---|
+| **Build time** | ISO type × installer version | 4 ISOs, cached and reused |
+| **Install time** | STIG × filesystem × kickstart-vs-UI | free |
+
+34 permutations, 4 ISOs.
+
+## Why both kickstart and UI
+
+Not thoroughness — they exercise different code, and each has a failure mode
+the other cannot reach.
+
+- The **STIG menu is UI-only.** `stigenable.py` is reached solely from the
+  curses configurator, so a kickstart can never "answer yes"; it has to list
+  `KS_STIG_PACKAGES` by hand. That is what `variant=stigpkgs` does.
+- The **`security:` key is kickstart-only** on POI 2.8. Rows `s01`/`s02` cover
+  it, and `s02` (`security: {fips: …}`) is reachable *exclusively* from a
+  kickstart on either installer version.
+- The **same failure looks different** on the two paths. In UI mode a missing
+  package reduces to `InstallerError("Installer failed")` on screen with the
+  real cause only in `/var/log/installer`; in kickstart mode the tdnf error
+  surfaces directly. An oracle calibrated on one path misreads the other.
+
+---
+
+## Layout
+
+```
+mission-control/
+├── config/
+│   ├── mission-control.env        every value overridable: ${VAR:=default}
+│   ├── permutations.tsv           the matrix as data, with doc vs expected verdict
+│   └── photon-matrix.vmx.template 10 placeholders, incl. per-permutation UUID
+├── lib/
+│   ├── common.sh                  config locator, structured results, identity
+│   └── oracle.sh                  the assertions - each names the PR it proves
+└── bin/
+    ├── mc-preflight.sh            can this host do the job?
+    ├── mc-build-iso.sh            build-axis tuple -> cached ISO
+    ├── mc-gen-kickstart.sh        permutation -> kickstart JSON
+    ├── mc-create-vm.sh            thin disk + VMX + guestinfo injection
+    ├── mc-install.sh              autonomous or operator-driven install
+    ├── mc-verify.sh               run the oracle, harvest the logs
+    ├── mc-teardown.sh             stash the whole chain, keep the evidence
+    └── mc-run.sh                  drive it all, report at the end
+```
+
+## Use
+
+```bash
+export MC_DIR=$PWD                 # or run from this directory
+./bin/mc-preflight.sh              # measured host readiness
+./bin/mc-run.sh --all --plan       # what would run, builds nothing
+./bin/mc-run.sh --only k01,k03     # two autonomous permutations
+./bin/mc-run.sh --only p03         # interactive: prompts, then waits
+./bin/mc-run.sh --report           # re-print from stored results
+```
+
+Runs are **sequential**: every ISO build shares `$PHOTON_TREE/stage`, and C:
+has ~138 GB free, so VMs are torn down after verification rather than kept.
+
+## How a PR regression shows up
+
+Every assertion carries the PR it proves, so the report names the culprit:
+
+```
+ID     ISO      POI     STIG  FS     MODE  DOC       RESULT    PRs implicated
+p03    minimal  2.8     yes   ext4   ui    fails     FAIL(2)   POI#11
+```
+
+`DOC` is the verdict the matrix recorded *before* the PRs. A row whose result
+reproduces `DOC`'s `fails` is a regression, and the PRs column says which one.
+The oracle can currently implicate **PR#9, PR#21, PR#22, PR#24, POI#9, POI#10,
+POI#11**.
+
+Results land in `$MC_RESULTS_DIR/<perm>/`: `checks.jsonl` (one JSON object per
+assertion), `kickstart.json`, and `logs/` with dmesg, `journalctl -b`,
+`journalctl -p err`, failed units, `rpm -qa`, `/proc/cmdline`, mounts,
+`/var/log/{installer,ansible-stig,messages}`, the mkinitrd log and the POI
+manifest.
+
+The matrix supplies a *dependency-resolution* oracle only — `Error(1011)`,
+media RPM presence, NEVRs. It gives no dmesg/journalctl/`/var/log` criteria at
+all, so that layer is new here.
+
+---
+
+## Things that will bite
+
+**`Error(1032)` is not a success signal.** It is the `--assumeno` dry-run
+abort. A real install never emits it. Only `Error(1011)` means "a package the
+installer asked for is not on the media".
+
+**Never match a single package name.** `list(set(packages))` in `installer.py`
+randomises which of the six missing names tdnf reports first. The matrix
+reproduced `rsyslog` where the user saw `libselinux-utils`; both were right.
+Match on `Error(1011)` and set membership.
+
+**`grep -a` does not work here.** In a non-interactive shell `/usr/bin/grep` is
+toybox, which has no `-a` and returns *zero matches* on a NUL-bearing serial
+log rather than erroring. `mc_grep_count` strips NULs first. Interactively
+`grep` is `ugrep`, which behaves differently again — so anything that works at
+your prompt may still be wrong inside a script.
+
+**`sed \U` and `grep -P` are GNU extensions** and are absent here for the same
+reason. `mc_win_path` uses `tr`.
+
+**UEFI ignores `bios.bootOrder`.** The NVRAM decides. Both install and teardown
+stash `.nvram`, or the firmware's removable-media fallback finds the old ESP
+and boots the *previous* image — which looks exactly like an install that
+silently did nothing.
+
+**Purge stale installer RPMs before every build.** tdnf takes the highest
+release it can see, so a months-old `photon-os-installer` left in
+`stage/RPMS/` silently wins and ships on the ISO. `mc-build-iso.sh` deletes
+them and records the NEVR that actually shipped. A run that exercises a stale
+installer is worse than no run: it reports a verdict for code nobody ships.
+
+**Identity is positional, not hashed.** A `cksum`-based index collided on this
+very matrix (`k04`/`k16` and `k09`/`s02` shared a MAC, UUID and IP) and could
+reach `.240`, inside VMnet8's DHCP range. The ordinal in `permutations.tsv` is
+unique by construction and keeps addresses at `.41–.74`.
+
+**Do not blanket-stop VMs.** This host runs other VMs, including live CI
+runners. Every operation targets `mc-<perm>` by name.
+
+**Keep VMs off OneDrive.** The hand-made test VM lived under
+`OneDrive/Dokumente/Virtual Machines` and disappeared mid-session. Mission
+control uses `C:\photon-mc\vm`.

staging/mission-control/bin/mc-build-iso.sh

@@ -0,0 +1,94 @@
+#!/bin/bash
+# mc-build-iso.sh - resolve one build-axis tuple to an ISO, building if needed.
+#
+# The build-time axes are ISO type and installer version, and nothing else.
+# Everything the matrix varies at install time (STIG, filesystem, kickstart vs
+# UI) is injected per VM, so 34 permutations need only 4 ISOs.
+#
+# usage: mc-build-iso.sh --iso-type minimal|full --poi 2.8|latest [--canister prebuilt|build|acvp|kat] [--force]
+set -u
+_here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+. "$_here/../lib/common.sh"
+. "$(mc_find_config "$_here")"
+
+ISO_TYPE=minimal POI=2.8 CANISTER=prebuilt FORCE=0
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --iso-type) ISO_TYPE="$2"; shift 2 ;;
+        --poi) POI="$2"; shift 2 ;;
+        --canister) CANISTER="$2"; shift 2 ;;
+        --force) FORCE=1; shift ;;
+        *) mc_die "unknown arg: $1" 64 ;;
+    esac
+done
+
+case "$ISO_TYPE" in minimal) IMG=minimal-iso ;; full) IMG=iso ;; *) mc_die "bad --iso-type" 64 ;; esac
+KEY="${ISO_TYPE}-poi${POI}-${CANISTER}"
+DEST="$MC_ISO_CACHE/$KEY"
+mkdir -p "$DEST" "$MC_BUILD_LOG_DIR"
+
+if [ "$FORCE" -eq 0 ] && [ -f "$DEST/photon.iso" ]; then
+    mc_log "cache hit: $KEY -> $DEST/photon.iso"
+    printf '%s\n' "$DEST/photon.iso"; exit 0
+fi
+
+# --- the stale-RPM landmine ----------------------------------------------
+# tdnf picks the highest release it can see, so a months-old
+# photon-os-installer left in stage/RPMS silently wins and lands on the ISO.
+# A test run that exercises a stale installer is worse than no test run: it
+# reports a verdict for code nobody is shipping.
+STAGE_RPMS="$PHOTON_TREE/stage/RPMS"
+if [ -d "$STAGE_RPMS" ]; then
+    n=$(find "$STAGE_RPMS" -name 'photon-os-installer-*.rpm' 2>/dev/null | wc -l)
+    if [ "$n" -gt 0 ]; then
+        mc_log "purging $n cached photon-os-installer RPM(s) so the build cannot pick a stale one"
+        find "$STAGE_RPMS" -name 'photon-os-installer-*.rpm' -delete
+    fi
+fi
+
+# --- installer version, without merging anything -------------------------
+# The point of this harness is to test PRs BEFORE they merge, so requiring a
+# merge to reach the poi=latest rows would invert that. Instead each variant
+# gets its own patch: poi=2.8 uses the downstream set as-is, poi=latest uses
+# the same set with dcasota/photon#26 (installer v2.9) substituted for the
+# 2.8-only installer PR. Both are generated from the PR branches and both are
+# verified to apply to a pristine 5.0 before use.
+#
+# runPh5_normal.sh resolves its patch relative to its OWN directory, so the
+# variant is selected by staging a script directory rather than by editing the
+# build script. SCRIPT_DIR is used for nothing else (runPh5_normal.sh:75,163).
+VARIANT_PATCH="$MC_VARIANT_PATCH_DIR/poi-${POI}.patch"
+[ -f "$VARIANT_PATCH" ] || mc_die "no variant patch at $VARIANT_PATCH - run mc-make-variant-patches.sh" 3
+
+STAGE_DIR="$MC_WORK/scriptdir/$KEY"
+mkdir -p "$STAGE_DIR/photonos-patches"
+cp "$PHOTON_SCRIPTS/runPh5_normal.sh" "$STAGE_DIR/runPh5_normal.sh"
+cp "$VARIANT_PATCH" "$STAGE_DIR/photonos-patches/downstream-fixes.patch"
+mc_log "staged build dir $STAGE_DIR with poi-${POI}.patch ($(grep -c '^+++ ' "$VARIANT_PATCH") files)"
+
+SPEC="$PHOTON_TREE/SPECS/photon-os-installer/photon-os-installer.spec"
+have=$(awk '/^Version:/{print $2; exit}' "$SPEC" 2>/dev/null); [ -n "$have" ] || have='?'
+mc_log "installer version in the pristine tree: $have (the variant patch sets the one under test)"
+
+BUILD_LOG="$MC_BUILD_LOG_DIR/${KEY}-$(date -u +%Y%m%dT%H%M%SZ).log"
+mc_log "building $IMG (canister=$CANISTER) -> $BUILD_LOG"
+mc_log "this takes hours; the run script polls rather than blocking"
+
+sh "$STAGE_DIR/runPh5_normal.sh" /root common 5.0 "$DEST" "$IMG" "$CANISTER" \
+    > "$BUILD_LOG" 2>&1
+rc=$?
+[ $rc -eq 0 ] || mc_die "build failed (rc=$rc), see $BUILD_LOG" "$rc"
+
+iso=$(find "$DEST" -maxdepth 1 -name '*.iso' -newer "$BUILD_LOG" 2>/dev/null | head -1)
+[ -n "$iso" ] || iso=$(find "$DEST" -maxdepth 1 -name '*.iso' | head -1)
+[ -n "$iso" ] || mc_die "build reported success but produced no ISO in $DEST" 4
+[ "$iso" = "$DEST/photon.iso" ] || ln -sf "$(basename "$iso")" "$DEST/photon.iso"
+
+# --- assert what actually shipped ----------------------------------------
+poi_on_media=$(xorriso -osirrox on -indev "$iso" -find /RPMS -name 'photon-os-installer-*.rpm' 2>/dev/null \
+               | sed 's|.*/||' | tr -d "'" | head -1)
+mc_log "installer on the produced media: ${poi_on_media:-ABSENT}"
+printf '%s\n' "$poi_on_media" > "$DEST/poi-nevr.txt"
+sha256sum "$iso" | awk '{print $1}' > "$DEST/photon.iso.sha256"
+mc_log "cached: $DEST/photon.iso"
+printf '%s\n' "$DEST/photon.iso"

staging/mission-control/bin/mc-create-vm.sh

@@ -0,0 +1,92 @@
+#!/bin/bash
+# mc-create-vm.sh - VM directory, thin boot disk, and VMX for one permutation.
+#
+# vm-lab splits this into a .ps1 because vmware-vdiskmanager wants Windows
+# paths. That split costs a whole second language with its own CRLF and
+# ASCII-only-for-PowerShell-5.1 constraints, and cannot be tested from here.
+# The .exe runs fine from WSL, so this stays in bash and converts the two
+# paths it needs by hand.
+#
+# usage: mc-create-vm.sh --id <perm> --iso <path> [--kickstart <file>] [--recreate]
+set -u
+_here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+. "$_here/../lib/common.sh"
+. "$(mc_find_config "$_here")"
+
+PERM="" ISO="" KS="" RECREATE=0
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --id) PERM="$2"; shift 2 ;;
+        --iso) ISO="$2"; shift 2 ;;
+        --kickstart) KS="$2"; shift 2 ;;
+        --recreate) RECREATE=1; shift ;;
+        *) mc_die "unknown arg: $1" 64 ;;
+    esac
+done
+[ -n "$PERM" ] || mc_die "--id is required" 64
+[ -n "$ISO" ] && [ -f "$ISO" ] || mc_die "--iso must name an existing file" 3
+
+VM="mc-$PERM"
+DIR_WSL="$MC_VM_ROOT_WSL/$VM"
+DIR_WIN="$(mc_win_path "$DIR_WSL")"
+IDX=$(mc_perm_index "$PERM")
+MAC=$(mc_mac_for "$IDX")
+UUID=$(mc_uuid_for "$IDX")
+SERIAL_WSL="$DIR_WSL/${SERIAL_LOG_PREFIX}-${VM}.log"
+
+if [ -d "$DIR_WSL" ] && [ "$RECREATE" -eq 1 ]; then
+    mc_log "recreate: stashing $DIR_WSL"
+    mv "$DIR_WSL" "${DIR_WSL}.stashed-$(date -u +%Y%m%dT%H%M%SZ)"
+fi
+mkdir -p "$DIR_WSL"
+
+# --- boot disk ------------------------------------------------------------
+# -t 0 is monolithicSparse: one file, thin. A fresh 32 GB disk is a few MB and
+# grows only as the guest writes. The hand-made test VM on this host is
+# monolithicFlat and commits its full size up front; 34 of those would not fit
+# in the free space on C:.
+if [ ! -f "$DIR_WSL/$VM.vmdk" ]; then
+    "$VDISKMANAGER" -c -s "$BOOT_DISK_SIZE" -a "$BOOT_DISK_ADAPTER" -t "$BOOT_DISK_TYPE" \
+        "$DIR_WIN\\$VM.vmdk" >/dev/null 2>&1 || mc_die "vmware-vdiskmanager failed for $VM" 5
+    mc_log "created thin disk: $(du -h "$DIR_WSL/$VM.vmdk" 2>/dev/null | cut -f1) of $BOOT_DISK_SIZE"
+else
+    mc_log "disk already present, keeping it"
+fi
+
+# --- kickstart injection --------------------------------------------------
+# POI's isoInstaller reads guestinfo.kickstart.data via vmtoolsd, and
+# /usr/bin/vmtoolsd is present in the installer initrd. So an autonomous
+# permutation needs no ISO remaster and no typing at the boot menu.
+# Omitting the line entirely is what selects the interactive path: with no
+# kickstart the installer falls through to the curses configurator, which is
+# the only place the STIG menu exists.
+if [ -n "$KS" ] && [ -f "$KS" ]; then
+    GUESTINFO="guestinfo.kickstart.data = \"$(base64 -w0 < "$KS")\""
+    mc_log "kickstart injected via guestinfo ($(wc -c < "$KS") bytes)"
+else
+    GUESTINFO="# no kickstart: interactive install, operator drives the curses configurator"
+    mc_log "no kickstart - interactive permutation"
+fi
+
+TPL="$_here/../config/photon-matrix.vmx.template"
+python3 - "$TPL" "$DIR_WSL/$VM.vmx" "$VM" "$GUEST_VCPUS" "$GUEST_MEM_MB" "$MAC" "$UUID" \
+         "$(mc_win_path "$ISO")" "$(mc_win_path "$SERIAL_WSL")" "$GUESTINFO" <<'PY'
+import sys
+tpl, out, vm, vcpu, mem, mac, uuid, iso, serial, guestinfo = sys.argv[1:11]
+s = open(tpl).read()
+for k, v in (("VM_NAME", vm), ("GUEST_VCPUS", vcpu), ("GUEST_MEM_MB", mem),
+             ("GUEST_MAC", mac), ("UUID_BIOS", uuid), ("ISO_PATH_WIN", iso),
+             ("SERIAL_LOG_WIN", serial), ("GUESTINFO_KICKSTART", guestinfo),
+             ("NIC_DEV", "vmxnet3"), ("SECUREBOOT", "FALSE")):
+    s = s.replace("@@%s@@" % k, v)
+import re
+left = re.findall(r"@@[A-Z_]+@@", s)
+if left:
+    sys.exit("FAIL: unsubstituted placeholders survived: %s" % sorted(set(left)))
+open(out, "w").write(s)
+PY
+[ $? -eq 0 ] || mc_die "VMX generation failed" 6
+
+mc_log "vm=$VM ip=$(mc_ip_for "$IDX") mac=$MAC"
+mc_log "vmx=$DIR_WSL/$VM.vmx"
+printf '%s\n' "$DIR_WSL/$VM.vmx"