Skip to content

test: add a preflight check to the factory-edge staging script #5504

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
auser merged 1 commit into from
Nov 15, 2026
+53 −9
Merged

test: add a preflight check to the factory-edge staging script #5504

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion scripts/factory-edge-demo-live.test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ envrun() { # <mode> <script> <env assignments...>: the script runs while a ticke
( while :; do env MVMCTL="${work}/mvmctl" FAKE_STATE="${work}/state" FAKE_MODE="${mode}" MVM_HOME="${work}/home" "${work}/mvmctl" __tick >/dev/null 2>&1 || true; sleep 0.1; done ) &
tick=$!
env MVMCTL="${work}/mvmctl" FAKE_STATE="${work}/state" FAKE_MODE="${mode}" MVM_HOME="${work}/home" \
WAIT_SECS=3 ATTEMPT_AFTER=0 "$@" bash "${repo}/scripts/${script}" 2>&1; rc=$?
WAIT_SECS=3 ATTEMPT_AFTER=0 DNS_CHECK=0 "$@" bash "${repo}/scripts/${script}" 2>&1; rc=$?
kill "${tick}" 2>/dev/null; wait "${tick}" 2>/dev/null || true
return "${rc}"
}
Expand All @@ -120,6 +120,14 @@ out2="$(stage ok HYPERVISOR=hvf)" || fail "staging twice failed: ${out2}"
out3="$(env MVMCTL="${work}/mvmctl" FAKE_STATE="${work}/state" FAKE_MODE=ok MVM_HOME="${work}/home" HYPERVISOR=hvf bash "${repo}/scripts/factory-edge-stage.sh" --down 2>&1)"
[ "$(ls "${work}"/state/m.* 2>/dev/null | wc -l | tr -d ' ')" = 0 ] || fail "--down did not remove the staged machines: ${out3}"

# --check verifies every prerequisite and starts nothing; a granted host that does not resolve is caught.
fresh
out="$(env MVMCTL="${work}/mvmctl" FAKE_STATE="${work}/state" FAKE_MODE=ok MVM_HOME="${work}/home" DNS_CHECK=0 HYPERVISOR=hvf bash "${repo}/scripts/factory-edge-stage.sh" --check 2>&1)" || fail "--check failed on a prepared home: ${out}"
grep -q '^READY:' <<<"${out}" || fail "--check did not report READY: ${out}"
[ "$(ls "${work}"/state/m.* 2>/dev/null | wc -l | tr -d ' ')" = 0 ] || fail "--check started a machine"
out="$(env MVMCTL="${work}/mvmctl" FAKE_STATE="${work}/state" FAKE_MODE=ok MVM_HOME="${work}/home" DNS_CHECK=1 GRANT_A=no-such-host.invalid:443 HYPERVISOR=hvf bash "${repo}/scripts/factory-edge-stage.sh" --check 2>&1)" && fail "--check passed with a granted host that cannot resolve: ${out}"
grep -q 'does not resolve' <<<"${out}" || fail "an unresolvable grant was not reported: ${out}"

bad_stage() { # <mode> <what must be caught> <text the failure must name>
fresh; local o; o="$(stage "$1" HYPERVISOR=hvf)" && fail "$2: staging passed: ${o}"
grep -qF "$3" <<<"${o}" || fail "$2: failed, but not for the expected reason ($3): ${o}"
Expand Down
37 changes: 32 additions & 5 deletions scripts/factory-edge-stage.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
#
# scripts/factory-edge-stage.sh stage the workloads (replaces any earlier staging)
# scripts/factory-edge-stage.sh --down remove the staged workloads
# scripts/factory-edge-stage.sh --check check every prerequisite and start nothing
#
# Three named machines are started in the background, each holding its own
# scoped network grant:
Expand Down Expand Up @@ -34,6 +35,7 @@
# (default 30)
# ATTEMPT_EVERY seconds between its requests (default 30)
# INSPECT_EVERY seconds between an inspection workload's requests (default 5)
# DNS_CHECK 0 skips the check that each granted host resolves (default 1)
set -uo pipefail

MVMCTL="${MVMCTL:-mvmctl}"
Expand All @@ -46,6 +48,7 @@ WAIT_SECS="${WAIT_SECS:-120}"
ATTEMPT_AFTER="${ATTEMPT_AFTER:-30}"
ATTEMPT_EVERY="${ATTEMPT_EVERY:-30}"
INSPECT_EVERY="${INSPECT_EVERY:-5}"
DNS_CHECK="${DNS_CHECK:-1}"
if [ -z "${HYPERVISOR:-}" ]; then
if [ "$(uname -s)" = Darwin ]; then HYPERVISOR=hvf; else HYPERVISOR=firecracker; fi
fi
Expand Down Expand Up @@ -108,12 +111,36 @@ down() { # remove every machine an earlier staging left behind

if [ "${1:-}" = --down ]; then down; echo "staged workloads removed"; exit 0; fi

# Everything the demo needs, checked without starting anything.
preflight() {
mvm --version >/dev/null 2>&1 || fail "no usable mvmctl ($MVMCTL): the demo assumes it is installed"
[ -d "$MVM_HOME" ] || fail "demo home $MVM_HOME does not exist: prepare it first (mvmctl image pull $IMAGE, then one throwaway run)"
mvm image ls </dev/null | grep -qF "${IMAGE#docker.io/}" \
|| fail "$IMAGE is not cached in $MVM_HOME: prepare it first (mvmctl image pull $IMAGE); staging does not install"
say "mvmctl $(mvm --version | awk '{print $2}') on $HYPERVISOR, home $MVM_HOME, $IMAGE cached"
# Every grant is resolved to addresses when the workload is admitted, so each granted host must resolve now.
if [ "$DNS_CHECK" != 0 ]; then
local g h
for g in "$GRANT_A" "$GRANT_B" "$APPROVED"; do
h="${g%%:*}"
python3 -I -c "
import socket, sys
socket.setdefaulttimeout(8)
try: socket.getaddrinfo(sys.argv[1], 443)
except OSError as e: sys.exit(str(e))" "$h" 2>/dev/null \
|| fail "$h does not resolve: every granted destination is resolved when its workload is admitted, so the demo needs working DNS"
done
say "every granted destination resolves (${GRANT_A%%:*}, ${GRANT_B%%:*}, ${APPROVED%%:*})"
fi
local free_kb
free_kb="$(df -k "$MVM_HOME" | awk 'NR == 2 { print $4 }')"
[ "${free_kb:-0}" -ge 1048576 ] || fail "less than 1 GB free where $MVM_HOME lives"
say "enough free disk space"
}

echo "== 0. preconditions (nothing is installed or prepared here)"
mvm --version >/dev/null 2>&1 || fail "no usable mvmctl ($MVMCTL): the demo assumes it is installed"
[ -d "$MVM_HOME" ] || fail "demo home $MVM_HOME does not exist: prepare it first (mvmctl image pull $IMAGE, then one throwaway run)"
mvm image ls </dev/null | grep -qF "${IMAGE#docker.io/}" \
|| fail "$IMAGE is not cached in $MVM_HOME: prepare it first (mvmctl image pull $IMAGE); staging does not install"
say "mvmctl $(mvm --version | awk '{print $2}') on $HYPERVISOR, home $MVM_HOME, $IMAGE cached"
preflight
if [ "${1:-}" = --check ]; then echo "READY: staging will find everything it needs. Run scripts/factory-edge-stage.sh."; exit 0; fi

echo "== 1. staging the three workloads"
down
Expand Down
15 changes: 12 additions & 3 deletions specs/contracts/host-interface-distributed-execution.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -610,10 +610,19 @@ before it starts, and any first-use preparation during a run is reported as a
3. With that home set, run `mvmctl image pull curlimages/curl:8.21.0` and one
throwaway `mvmctl run` of that image, which builds the workload runtime.

**Running.** `scripts/factory-edge-stage.sh`, then
**Running.** Before presenting, `scripts/factory-edge-stage.sh --check` verifies
every prerequisite and starts nothing: a working `mvmctl`, the demo home, the
cached image, enough disk, and that every granted destination resolves, because
a grant is resolved to addresses when its workload is admitted and the demo
therefore needs working DNS. Then `scripts/factory-edge-stage.sh`, then
`scripts/factory-edge-demo-live.sh`, then `scripts/factory-edge-stage.sh --down`.
Staging replaces any earlier staging, and each staging uses its own machine
names because `mvmctl explain <name>` is ambiguous when a name has been reused.
Stage shortly before presenting, ten minutes or so: every lookup a staged
workload makes is a signed audit-chain entry, so the chain, and with it the cost
of `explain` and `trust audit verify`, grows for as long as the workloads run.
After a reboot the staged machines are gone, so stage again; the demo home, the
installed release and the cached image persist. Staging replaces any earlier
staging, and each staging uses its own machine names because `mvmctl explain
<name>` is ambiguous when a name has been reused.
On a live run against `v0.28.0-rc.2` on HVF staging took about 15 s and the demo
about 30 s while other builds held the host at a load average near 40. The
maintenance workload makes its first request `ATTEMPT_AFTER` seconds (default
Expand Down