staging: add mission-control - execute the ISO permutation matrix

[dcasota]

ISO-PERMUTATION-MATRIX.md says of itself: "No builds were run." Fourteen of its sixteen rows are dependency-resolution predictions or code reading; only two were ever installed. This is the execution layer it lacks.

It reuses vm-lab's mechanics — VMX discipline, thin disks, stash-never-delete teardown, the serial-log liveness instrument — and drops everything specific to the SPAGAT-Librarian appliance, which is out of scope. In particular it does not need spagat-vm-orchestrator, which is a cargo artifact of a repo this project does not have; vmrun plus VMX edits cover everything it did except install-completion detection, which is done here by watching the boot source change in the serial log.

What makes 34 permutations cost 4 builds

POI's isoInstaller does not only read ks= from the kernel cmdline — _load_ks_config_vmware() reads guestinfo.kickstart.data through vmtoolsd. I verified the mechanism is actually present, not merely coded for:

usr/bin/vmtoolsd        78024 bytes   <- in the installer initrd
usr/bin/vmware-rpctool  32872 bytes
"open-vm-tools"                       <- in packages_installer_initrd.json

So a per-permutation kickstart is one line in the VMX. No ISO remaster, no HTTP server, no typing at a boot menu. That separates the axes cleanly:

Layer	Axes	Cost
Build time	ISO type × installer version	4 ISOs, cached
Install time	STIG × filesystem × kickstart-vs-UI	free

Why both install paths are driven

Not thoroughness — each has a failure mode the other cannot reach.

• The STIG menu is UI-only (stigenable.py is reached solely from the curses configurator), so a kickstart can never answer it and must list KS_STIG_PACKAGES by hand.
• The security: key failure is kickstart-only on POI 2.8, and security: {fips: …} is reachable exclusively from a kickstart on either version. Rows s01/s02 cover it.
• The same failure surfaces differently: UI mode reduces a missing package to InstallerError("Installer failed") on screen with the cause only in /var/log/installer. An oracle calibrated on one path misreads the other.

How a PR regression is identified

Every assertion names the PR it proves, so the report names the culprit rather than saying something broke:

ID     ISO      POI   STIG  FS    MODE  DOC     RESULT   PRs implicated
p03    minimal  2.8   yes   ext4  ui    fails   FAIL(2)  POI#11

DOC is the verdict the matrix recorded before the PRs. A row reproducing DOC's fails is the regression signal. Covers PR#9, PR#21, PR#22, PR#24, POI#9, POI#10, POI#11.

Results land in $MC_RESULTS_DIR/<perm>/: checks.jsonl (one JSON object per assertion), the generated kickstart.json, and logs/ with dmesg, journalctl -b, journalctl -p err, failed units, rpm -qa, /proc/cmdline, mounts, /var/log/{installer,ansible-stig,messages}, the mkinitrd log and the POI manifest. The matrix supplies a resolution oracle only — it gives no dmesg/journalctl//var/log criteria at all, so that layer is new.

Verified on this host, not assumed

• vmrun and vmware-vdiskmanager execute from WSL; mc-preflight.sh passes
• A real VM was created end to end: 4.1 MB thin disk for a 32 GB volume, unique MAC/UUID, correct Windows paths, kickstart injected as guestinfo
• The interactive variant emits no active guestinfo line at all — which is what selects the curses configurator
• Teardown stashed the chain and preserved the VMX and serial logs
• Generated kickstarts validate against POI's own known_keys for all four variants, so none can abort the install on an unknown key
• The oracle ran against a live Photon guest and reported correctly

Four portability traps found while building it

grep -a does not work here. In a non-interactive shell /usr/bin/grep is toybox, which has no -a and returns zero matches on a NUL-bearing serial log instead of erroring — the same silent-zero trap vm-lab's README documents, reached by a different route. Interactively grep is ugrep, which supports both -a and -P, so anything verified at a prompt can still be wrong inside a script. mc_grep_count strips NULs first and trusts neither.

grep -P and sed \U are GNU extensions and absent for the same reason; mc_win_path uses tr.

vmrun output is CRLF — every parse strips \r. My own preflight reported "0 VMs running" against vmrun's "Total running VMs: 2" until I did.

A hashed permutation index collides. A cksum-based index collided on this very matrix — k04/k16 and k09/s02 would have shared a MAC, a UUID and an IP — and could reach .240, inside VMnet8's DHCP range of .128–.254. The index is now the ordinal in permutations.tsv: unique by construction, addresses confined to .41–.74.

Operational notes

Runs are sequential: every ISO build shares $PHOTON_TREE/stage, and C: has ~138 GB free, so VMs are torn down after verification rather than kept. Nothing ever blanket-stops VMs — this host runs other VMs including live CI runners, so every operation targets mc-<perm> by name. VMs live at C:\photon-mc\vm, deliberately not under OneDrive: the hand-made test VM lived under OneDrive\Dokumente\Virtual Machines and vanished mid-session.

Two ISOs the matrix needs (minimal/latest, full/latest) cannot be built until the installer version bump lands — mc-build-iso.sh refuses with a clear message rather than silently building the wrong thing.