hardware in the loop testing - #376
Merged
Merged
Conversation
Design for a self-hosted Actions runner driving real Vector boards from this public repo, covering boot smoke tests, API contract tests, the full game-config matrix, and the previous-5-versions upgrade matrix. The security model is the load-bearing part: PR code is built on GitHub-hosted runners and only artifacts cross onto the bench, with the harness itself always executing from the default branch via workflow_run. Design only -- no implementation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
Contributor
|
Developer build links: Sys11 (Tiny) WPC EM WhiteStar DataEast Classic |
Pin every GitHub Action to a commit SHA with the version in a trailing comment. deploy_docs, docs-on-pr, specialfeatures-on-pr, validate-json-configs and version-bump-guard were on floating @v4/@v5 tags; build_release was pinned but to older revisions. All six now pin to the latest patch of the major version already in use, so this is a supply-chain change with no behaviour change. Pin every dependency in dev/requirements.txt. mpy-cross was the notable gap: it emits the .mpy bytecode the boards must import, and leaving it unpinned meant a future release could silently change the bytecode version and produce builds no deployed board can load. Pinned to 1.28.0.post2, the version CI was already resolving. Verified a full sys11 build succeeds and emits 41 .mpy files at bytecode version 6, matching what 1.23.0 produces. Design doc updates: the upgrade matrix now runs on every PR rather than nightly, with the combined per-PR wall-clock cost quantified and the levers to reduce it listed; runner host section reflects that the Pi is a Zero 2 W; new open question about which mpy-cross version the build should actually target, since the shipped UF2s carry MicroPython v1.24.1 for System 11/9 and v1.26.0-preview for WPC and Data East. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
mullinmax
changed the title
Step-by-step setup for the Pi Zero 2 W bench runner: memory headroom, unprivileged runner user, pinned Python environment, stable board device names via udev, network isolation verification, bench manifest, Actions runner install with verified checksums, job hooks, and an end-to-end smoke workflow. Each phase ends with a verification step. Hook scripts and YAML blocks are syntax-checked and the pre-job health check is behaviour-tested for both the healthy and missing-board paths. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
Board detection and flashing already work through the repo's dev pipeline (detect_boards.py identifies boards by querying systemConfig.vectorSystem, so port ordering does not matter), which removes the need for udev rules, a bench manifest, and job hooks at setup time. Drops the swap tuning, dedicated user, udev rules, dnsmasq, bench manifest, secrets file, job hooks, and logrotate config down to a short list of commands. The most likely failure modes are kept as one-line fixes at the bottom rather than pre-emptive configuration. The removed detail remains in git history if any of it turns out to be needed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
Drop the download checksum step and the arch detection (arm64 confirmed in both kernel and userland). Remove the dialout group step from the main flow since the default login user already has it; keep it in the troubleshooting list. Boards share the VLAN with the Pi, so there is no network configuration in this doc. Bench wifi credentials go in the runner's .env, which is read at service start and exported to every job. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
Converts the manual command list into a POSIX sh script artefact plus an otamaker manifest, so the bench Pi can be provisioned remotely instead of by hand. The artefact contract shapes the design: the script runs as root, so it drops to the runner user via runuser for everything that should not be root (the Actions runner refuses to configure as root); output goes to the journal, so each phase logs a line; and exit codes follow the 0/1/2 success/failure/reboot convention. Made re-runnable, since a redeploy is the natural way to refresh credentials or pull a newer harness: an existing clone is fetched rather than re-cloned, an existing runner download and registration are left alone, and .env is rewritten in place touching only the VECTOR_HIL_ lines it owns. Board detection is non-fatal so a transient USB state does not fail the whole deployment. Placeholder credentials are rejected up front. Verified against a stubbed root environment: fresh install, re-run idempotency, .env non-duplication and preservation of unmanaged vars, and failure propagation for apt, registration, and a service that starts but does not stay active. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
Adds the built .tar.zst plus its SHA-256 so Raspberry Pi Connect can be pointed at a URL in this repo rather than at a locally built file. Committing the artefact required moving credentials out of it first. Baking the registration token and wifi password into the script before packaging would have published the bench wifi password to a public repo. The script now sources /etc/vector-hil.conf on the device instead, which never enters the repo, and refuses to run if that file is group- or world-writable or not owned by root. That also removes the rebuild-per-deploy step: the artefact carries no per-deployment state, so redeploying is pointing Connect at the same URL again. The registration token is only needed for the first deploy, since the script skips registration once the runner is registered. build.sh produces the archive reproducibly - fixed mtimes and ownership, sorted entries - so an unchanged source rebuilds byte-identically and a changed checksum in a diff means the contents really changed. It also syntax-checks the script and refuses to package one containing literal credentials. Marks *.tar.zst as binary in .gitattributes; the repo sets `* text=auto`, and any normalization of the artefact would invalidate the checksum the device verifies against. Verified: reproducible rebuild, credential guard, and the config-file paths (missing, bad permissions, missing token when unregistered, and tokenless redeploy when already registered). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
The documented raw.githubusercontent URL points at main, which does not resolve until this merges. Adds the branch URL so the artefact can be deployed and tested first, plus a note that GitHub does not render binary blobs in the Files changed view, so the committed artefact is easy to mistake for missing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
Raspberry Pi Connect's remote update requires A/B image support, which only Pi 4 and 5 have, so the Zero 2 W cannot be provisioned that way. Removes the artefact, its manifest, the reproducible build script, the committed tarball and checksum, and the device-side config file. Replaces them with dev/hil/setup-runner.sh, run directly on the Pi as the login user with sudo where needed. Credentials come from environment variables rather than a file, so nothing sensitive lands in the repo or on disk. Keeps the properties that were worth having: idempotent re-runs, non-fatal board detection, and a check that the service actually stayed up rather than reporting success for one that immediately died. Verified against stubs: missing credentials, missing token while unregistered, full install, and a tokenless re-run preserving unmanaged .env entries. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
Nothing in the repo targeted the self-hosted runner yet, so there was no way to validate the bench. This checks that the runner picks up jobs, the bench environment reaches them, the serial devices are present and accessible, and every detected board answers over mpremote. It fails if no boards are found or if a detected board does not respond. Triggers on push to this branch as well as workflow_dispatch, because a workflow_dispatch workflow is not dispatchable until it exists on the default branch - without the push trigger the bench could not be validated before merging. Drop the push trigger once this is on main. No pull_request trigger: this targets a self-hosted runner on a private network, and pull_request would let any fork PR run against it. Fork gating is designed in DESIGN.md but not built. The job does no checkout, matching the trust model - the bench runs code from the clone it already has, not from a PR. Parses only the last line of detect_boards.py output, since it prints progress text before the JSON. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
First bench run failed at board detection with an empty result while /dev/ttyACM0-2 were present and the runner user was in dialout. dev/detect_boards.py shells out to a bare `mpremote` resolved from PATH, and catches OSError - which covers FileNotFoundError - returning an empty list. Invoking the venv's python by absolute path does not put the venv's bin on PATH, so mpremote was missing and "tool not installed" surfaced as "no boards attached". Adds the venv to GITHUB_PATH and invokes python and mpremote by name so subprocesses resolve the same interpreter and tools. Also splits out an explicit mpremote check that prints `mpremote devs` directly, because detect_boards.py collapses "mpremote missing", "mpremote failed" and "no boards attached" into the same empty result - which is what made the first failure ambiguous. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
Contributor
There was a problem hiding this comment.
Pull request overview
This PR strengthens CI supply-chain hygiene by pinning GitHub Actions and Python dev-tooling dependencies, and it introduces an initial hardware-in-the-loop (HIL) bench plan (plus bench runner setup/smoke workflow) under dev/hil/ to validate the self-hosted runner wiring before the full harness exists.
Changes:
- Pin GitHub Actions
uses:references to specific commit SHAs across the touched workflows. - Pin previously-floating Python tooling dependencies in
dev/requirements.txt(notablympy-cross) to fixed versions. - Add HIL bench documentation and bootstrap artifacts (
dev/hil/DESIGN.md, runner setup script/docs) plus a self-hosted runner smoke workflow.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| dev/requirements.txt | Pins dev pipeline Python dependencies (incl. mpy-cross) and documents the bytecode-compat rationale. |
| dev/hil/setup-runner.sh | Adds a Pi bootstrap script to provision a self-hosted runner, venv, and bench env vars. |
| dev/hil/RUNNER_SETUP.md | Documents how to run the runner setup and how to validate the bench. |
| dev/hil/DESIGN.md | Design proposal for HIL architecture, threat model, gating, and test matrix strategy. |
| .github/workflows/hil-smoke.yml | Adds a self-hosted runner smoke workflow to validate bench wiring and board responsiveness. |
| .github/workflows/build_release.yml | Updates pinned SHAs for key actions used in the build/release workflow. |
| .github/workflows/deploy_docs.yml | Pins checkout/setup-python actions by SHA. |
| .github/workflows/docs-on-pr.yml | Pins checkout/setup-python actions by SHA. |
| .github/workflows/specialfeatures-on-pr.yml | Pins checkout/setup-python actions by SHA. |
| .github/workflows/validate-json-configs.yml | Pins checkout/setup-python actions by SHA. |
| .github/workflows/version-bump-guard.yml | Pins checkout/setup-python actions by SHA (plus EOF newline normalization). |
| .gitattributes | Forces LF for dev/hil/*.sh to avoid CRLF shebang breakage on the Pi. |
Suppressed comments (2)
dev/hil/setup-runner.sh:145
- The
.envfile is written with unescaped SSID/password values. If either contains spaces, quotes, or backslashes (common for SSIDs), systemd’s EnvironmentFile parsing can break and the runner won’t export the intended values. Quote and escape these values when writing.env.
{
echo "VECTOR_HIL_WIFI_SSID=$VECTOR_HIL_WIFI_SSID"
echo "VECTOR_HIL_WIFI_PASSWORD=$VECTOR_HIL_WIFI_PASSWORD"
echo "VECTOR_HIL_VENV=$VENV_DIR"
echo "VECTOR_HIL_REPO=$REPO_DIR"
dev/hil/setup-runner.sh:127
- The design doc’s hardening checklist calls out using an ephemeral runner (
--ephemeral) to reduce persistence between jobs, but the setup script registers a persistent runner service (config.shwithout--ephemeral). This leaves a larger post-compromise surface (state in_work/, caches, etc.) than the threat model intends. Recommend reconciling the setup script with the checklist (either implement an ephemeral registration flow + cleanup hooks, or document why persistence is acceptable for now).
( cd "$RUNNER_DIR" && ./config.sh \
--url "$REPO_URL" \
--token "$RUNNER_TOKEN" \
--labels "$RUNNER_LABELS" \
--unattended --replace >/dev/null ) \
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Review feedback on #376. The setup script fetched but never checked anything out, so an existing clone stayed on whatever was last checked out. That contradicted both the trusted-checkout model and this repo's own docs, which claim re-running is how you pull a newer harness - it was not. Now checks out origin/$REPO_BRANCH (default main) detached after fetch, and refuses to run against a clone with uncommitted changes rather than discarding them. Rejects newlines in the wifi credentials. The runner parses .env by splitting each line on the first '=' and taking the rest verbatim (Runner.Listener/Program.cs) - there is no EnvironmentFile= in the generated unit and runsvc.sh does not source it - so spaces, quotes and backslashes are already safe and must not be escaped. A newline is the only value that cannot be represented. Documents why the runner is persistent rather than ephemeral, in both the design checklist and the setup guide, instead of leaving the two in silent disagreement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
mullinmax
changed the title
Contributor
Author
|
Thanks — three of the four are addressed in cad0383, and one is incorrect and worth recording so nobody "fixes" it later. Stale checkout after PR description said "design only" — correct. That was stale from an earlier revision. Description rewritten to describe what's actually here, including the smoke workflow and the bench run. Ephemeral runner — correct that the docs disagreed with the script. The deviation is deliberate but was undocumented after a rewrite dropped the explanation. Now recorded in both DESIGN.md §9 and RUNNER_SETUP.md: ephemeral runners deregister after every job, so something must mint a fresh registration token each time, which means a PAT with
var envContents = File.ReadAllLines(envFile);
var separatorIndex = env.IndexOf('=');
envValue = env.Substring(separatorIndex + 1); // rest of the line, verbatim
Environment.SetEnvironmentVariable(envKey, envValue);Split on the first There is one real failure mode nearby: a newline can't be represented in that format at all. The script now rejects it up front instead of writing a corrupt file. Generated by Claude Code |
Adds dev/hil/flash_and_check.py plus a workflow to drive it: inventory, resolve target per board, build, flash with bench wifi config, then health-check the API over USB and HTTP once the board boots. The identification problem is the interesting part. Nothing on a board reports what hardware it is - systemConfig.vectorSystem is a build-time constant baked into whatever was last flashed, and machine.unique_id() is the RP2040 chip id, stable per board but silent about which system it is wired for. The last smoke run had all three boards reporting sys11, so flashing from self-report would have perpetuated a mis-flash. So an explicit chip-id map (VECTOR_HIL_BOARD_MAP) is authoritative when set, and without one the harness falls back to self-report but refuses to flash when two boards claim the same system, printing the chip ids needed to build the map. --inventory-only prints them without touching anything. Health checks: version matches the built source, faults are only the HDWR02 expected on a bare bench, game status and config list are sane, the flashed config is the active one, leaders and the auth challenge route answer, then the same board over HTTP for version, index page and faults. HDWR01 warns rather than fails and suppresses the active-config assertion, since it means the board took the safe_mode path. The workflow uses actions/checkout, which is safe only because every trigger is repo-internal; noted inline that a fork-reachable trigger would have to move back to the runner's pinned clone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
The first flash run correctly refused to proceed: all three bench boards report sys11, so autodetection would have flashed sys11 firmware onto the WPC and Data East hardware. Resolving that needs a chip-id map, which needs a way to tell the physical boards apart. --identify blinks each board in turn, using the Pico W onboard LED (which works from the REPL regardless of what firmware is loaded) plus the Vector board's WS2812 in blue when the flashed firmware provides the driver. One pass over the bench identifies every board, rather than unplugging them one at a time and diffing the inventory. Documents the whole flow in RUNNER_SETUP.md, including why detection cannot identify hardware and where the map lives. Skipping CI on this commit: without the map set, a triggered run would fail at the same resolve step and add nothing. Identification is a hands-on task to run directly on the Pi while watching the bench. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
Running the harness from a login shell failed with FileNotFoundError on mpremote. VECTOR_HIL_VENV is exported by the runner service into Actions jobs, not by an interactive shell, so the documented command prefixed PATH with an empty path and fell through to the system interpreter. The harness now locates the bench venv itself - VECTOR_HIL_VENV, then the directory of the running interpreter, then <repo>/.venv - prepends its bin to PATH, and uses its python for the build and flash subprocesses. That matters beyond our own calls: dev/build.py shells out to a bare mpy-cross and dev/flash.py to a bare mpremote, and the system interpreter would not have the build dependencies either. If mpremote still cannot be found it now says so and gives a working command, rather than raising a traceback from subprocess. Corrects the same bad command in RUNNER_SETUP.md and the module docstring. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
Build cost on the Zero 2 W is unmeasured - three targets of mpy-cross plus web minification, against a 45 minute job timeout. Each stage now reports its duration and the run ends with a breakdown, so if the timeout gets tight we know which stage to attack rather than guessing. Also triggers the first flash run now that VECTOR_HIL_BOARD_MAP is set on the bench. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
The first full flash run flashed all three boards, booted them, joined wifi and passed every USB API check - then failed on my own HTTP assertions. The board serves web assets pre-gzipped with Content-Encoding: gzip (backend.py:183) and urllib does not decompress automatically, so the index-page check was looking for "<html" inside gzip bytes. The board's own log confirmed it had served GET / 200 correctly. Now decompresses, and accepts <!doctype as well as <html. The wpc board separately failed with IncompleteRead(0 bytes read, 2359 more expected). phew is single-threaded on a microcontroller that is also fielding discovery broadcasts, so an occasional dropped body is not worth failing a bench run over; HTTP gets three attempts before giving up, and reports the underlying error when it does. Also logs the whole active-config object rather than a "name" key it does not have, which made a passing check print "None". Verified both paths against a stub server that serves gzip and simulates a truncated body. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
…[skip ci] The design predicted HDWR02 would always fire on a bare bench and that HDWR01 might fire nondeterministically and silently invalidate the config coverage. Both were wrong, and the first full flash run measured it: three boards freshly flashed for their real targets all booted with no faults at all. HDWR01 simply did not appear, so the floating-pin concern does not need a resistor pack and the config assertions are not vacuous. HDWR02 cannot appear: adr_activity_ok() is defined but never called, in either src/common/main.py or src/data_east/main.py. That is a firmware finding rather than a bench one - "No Bus Activity" is precisely the diagnostic a customer with a dead bus needs, and nothing can currently raise it. Worth its own issue: reconnect the check or drop the code, but a dead diagnostic in faults.py is the worst of both. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
The flash/health-check harness now reports per-stage timings, so the design can carry real figures instead of guesses: builds are 14-22s per target, flashing 22-28s per board, and boot to a fully health-checked API 10-15s. A full flash-and-verify of all three boards is about 3.5 minutes. Build cost was called out as a risk against the 45 minute job timeout and a candidate for caching. It is not - that concern is retired. The per-config reboot loop that drives the config matrix is still an estimate and is now labelled as one, since only flash-and-verify has been measured. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
Boot is slow and variable, so the harness now watches the serial console
for the firmware announcing readiness instead of polling an API that is
not listening yet. Polling told us nothing about why a board was late and
burned the whole timeout when one failed to boot; reading the console
gives an exact ready signal and, on failure, the boot log that explains
it. The transcript is now printed whenever a health check fails.
The marker is "Server: Loop Forever", printed immediately before
loop.run_forever() (phew/server.py:381). Deliberately not the earlier
"> starting web server on port 80" line - that is printed before
start_server is even scheduled, so matching it returns while the socket is
still closed. A pty-based test covers exactly that distinction. Since the
marker still precedes run_forever(), a short settle follows it, with
http_get's retries covering the remainder. backend.go() runs
connect_to_wifi() before either line, so the marker covers both transports.
HTTP coverage grows from three requests to the index page plus twelve
read-only routes, all required to return JSON, and adds checks only this
transport can make:
- authentication is enforced (password_check must 401 without credentials)
- USB bypasses auth by design (backend.py:280), so HTTP is the only place
the gate can be proven
- challenges are single-use, not a repeated nonce
- version and config count agree between USB and HTTP, since the two
bridges share a route table but not their plumbing
Boot timeout raised to 150s. Verified against a stub server reproducing
the board's gzip, 401 and nonce behaviour, including the mismatch and
unenforced-auth failure paths.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
The console watch timed out on all three boards while they were in fact healthy - the boot log it dumped showed steady-state runtime chatter (RESOURCE, DISCOVERY PING/PONG between the boards' own IPs, FRAM cycles), not boot output. The marker prints exactly once per boot. dev/flash.py resets at the end of flashing, but flashing loops over every board before any health check starts, so the first board had booted 50-75 seconds earlier and its marker was long past. The previous polling approach was immune to this because it did not care when the board became ready; watching a one-shot line is not. So the harness now issues the reset itself at the start of each health check, immediately before opening the console. The wait becomes deterministic and the reported boot time actually measures a boot. A failed reset is now its own clear error rather than a 150s timeout. Worth noting the boot-log dump added in the same series is what made this diagnosable at a glance - without it this looked like three boards failing to boot. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
The reset fix worked - all three boards reported their ready marker (12.5s, 13.9s, 23.6s) - which exposed two further problems. sys11 failed on my own over-strict assertion. /api/game/name is documented as "Plain-text game name" (backend.py:451) and returns a bare string, so requiring JSON from every route was wrong. Routes now carry their expected body kind; text routes are checked for a non-empty body instead. Worth noting separately that route_wrapper still labels that response application/json, which is a small inconsistency in the firmware rather than in the test. wpc and data_east answered 404 to the very first USB request for /api/version. The route table cannot be incomplete - the server only starts if backend.py imports fully - so the board must have parsed a different path. usb_comms accumulates stdin into a module-level buffer until it sees a newline (usb_comms.py:132), so any untermined bytes left over from the raw-REPL session that issued the reset prefix the next request and break the _routes lookup. The harness now sends a lone newline to flush that buffer, waits one scheduler turn for it to be consumed, and clears both ends of the line first. If a USB request still fails, the error now includes the board's own narration. usb_comms prints "USB REQ: route not found: <path>" but UsbApiClient discards every line that is not a response, which is why the last run could not say which path the board actually saw. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPbnmZNF6DFnLQBtpQpcW4
mullinmax
changed the title
paulmullin
approved these changes
Aug 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Stands up hardware-in-the-loop testing on a self-hosted runner, and pins the supply chain that feeds it.
dev/hil/DESIGN.mddev/hil/setup-runner.shdev/hil/flash_and_check.py.github/workflows/hil-smoke.yml.github/workflows/hil-flash-check.ymldev/requirements.txtdeps to versionsCovers design goals G1 (boots and runs) and G2 (API reachable) from DESIGN.md, on real hardware, both transports. G3 (config matrix) and G4 (upgrade matrix) are designed but not built.
Related Issues
None yet. Two firmware findings below deserve their own.
Motivation and Context
Host-side CI is good, but nothing proved the firmware boots and runs on hardware. The gaps that matter: a config that passes schema validation but exhausts RAM on load, an update that bricks a board upgrading from an older release, an API route that regresses on one transport but not the other.
The repo is public, so the design has to make running untrusted code on hardware safe rather than avoid it. The central property: PR code is built on GitHub-hosted runners, and only build artifacts cross onto the bench. The harness executes from a trusted ref via
workflow_run, which GitHub runs from the default branch regardless of what the triggering PR contains — so a PR cannot modify the workflow, the harness,conftest.py, or the dependency set the Pi executes.That answers "separate repo?" too: a separate repo would not have been safer on its own. One that still checked out the PR and ran pytest on the Pi would be equally compromised. Trusted-ref execution is the property that matters, not the repo boundary.
Board identification
Nothing on a board reports what hardware it is.
systemConfig.vectorSystemis a build-time constant baked into whatever was last flashed, andmachine.unique_id()is the RP2040 chip id — stable per board, silent about the system. Autodetection can only report what a board is running, which is exactly wrong after a mis-flash.This was not hypothetical: all three bench boards were running
sys11firmware, including the Data East board. Flashing from self-report would have re-flashed it as sys11 and reported success. SoVECTOR_HIL_BOARD_MAP(chip id → target) is authoritative when set, and without it the harness refuses to flash when two boards claim the same system.--identifyblinks each board so the map can be built once.mpy-crosspinningIt emits the
.mpybytecode the boards must import, and the header carries a version the board's MicroPython will refuse if it doesn't recognise it. Unpinned, CI resolved 1.28.0.post2 while the shipped UF2s carry MicroPython v1.24.1 (System 11/9) and v1.26.0-preview (WPC, Data East), with nothing asserting the pairing is valid. Pinned to what CI already resolved, so the pin freezes current behaviour. What it should be is a hardware decision — open question 1 in the design.Testing
Runs on the actual bench. Latest flash + health check is green across all three boards:
Per board: built, wiped, flashed with bench config, reset, waited on the console for
Server: Loop Forever, then checked over USB (version matches the built source, faults, game status, config list, active config is the one flashed, leaders, auth challenge) and over HTTP (index page plus 12 read-only routes, auth enforced, challenges single-use, version and config count agreeing between transports).Measured timings, which retire the build-caching concern the design raised:
Off-hardware verification: full
dev/build.pyunder the pinned dependency set produces 41.mpyfiles at bytecode version 6, identical to whatmpy-cross1.23.0 emits, confirming the pin is behaviour-preserving.setup-runner.shexercised against a stubbed environment (missing credentials, missing token, full install, tokenless re-run, dirty-clone refusal, newline-in-credential refusal). HTTP checks tested against a stub reproducing the board's gzip, 401 and nonce behaviour. Boot-marker detection tested over a pseudo-terminal, including that the earlier> starting web server on port 80line is correctly not treated as ready.No firmware or
src/changes, so no version bump required.Types of Changes
Checklist
Additional Notes
Firmware findings, each worth its own issue — none block this PR:
HDWR02: No Bus Activityis unreachable.adr_activity_ok()is defined and never called, in bothsrc/common/main.py:69andsrc/data_east/main.py:82. That fault is exactly the diagnostic a customer with a dead bus needs, and nothing can currently raise it. Found because the design predicted it would always fire on a bare bench and it never did.usb_commsline buffering is fragile. It accumulates stdin into a module-level buffer until a newline (usb_comms.py:132), so any unterminated bytes silently prefix the next request and the board answers 404 for a route that exists. The harness now flushes with a newline first, but a firmware-side guard would make the USB transport robust for every client./api/game/namereturns a bare string labelledContent-Type: application/json. The route is documented as plain text, so the body is right and the header is wrong.detect_boards.pyswallowsOSErrorinto an empty result, so "mpremote is missing" is indistinguishable from "no boards attached". It cost a debug cycle here.dev/sync.pyshells out to a barepythonthe same way.Before merging: the
push:triggers on both HIL workflows are branch-scoped for pre-merge testing and go dead on merge, at which pointworkflow_dispatchstarts working from the Actions UI. Removing them is tidiness, not correctness — happy either way.trench-coathas the same floating-tag exposure — 5 unpinned actions across its 2 workflows, plus unpinnedpre-commitandpytest. Left alone to keep this scoped; happy to do a companion PR.Suggested follow-up: a CI check that greps for
uses:.*@v[0-9]so floating tags can't creep back in.