Skip to content

ci(firmware): derive the PlatformIO impact map from the firmware tree - #341

Merged
mottihoresh merged 2 commits into
mainfrom
mh/elastic-williams-a157a6
Oct 15, 2026
Merged

mottihoresh merged 2 commits into
mainfrom
mh/elastic-williams-a157a6

Conversation

@mottihoresh

@mottihoresh mottihoresh commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

library_dependents in .github/firmware-platformio-impact.json is hand-written and had drifted to roughly a fifth of the real dependency graph:

  • it listed a Firmware/Prototype/ tree that does not exist and has no history in this repo;
  • it had no NodeStatus key at all;
  • it listed 2 of A4EC's 9 dependents.

The first two fail loud — a missing project and an unknown library both hit the selector's escape hatch, so a CANProtocol or NodeStatus change fell back to a full build. A4EC's failure is silent: the selector returned 2 projects and skipped the other 7.

Why hand-maintenance can't fix it

The graph is not in the platformio.ini files alone. Each Firmware/Libraries/*/library.json declares its own dependencies:

CANProtocol -> HIDControls, NodeStatus, STM32Board
PanelBridge -> A4EC, CANProtocol, NodeStatus, STM32Board
PanelGroup  -> CANProtocol, STM32Board
DrumDisplay -> NodeStatus, PanelGroup

A project listing only PanelGroup still compiles four libraries. Reading lib_deps alone under-counts NodeStatus by 9 projects and HIDControls by 2. The ini files also spell dependencies three ways — bare names resolved via lib_extra_dirs, file:// paths, and registry packages — and put lib_deps on a shared [env_base] that each env extends.

Approach

Generate the map instead. .github/scripts/gen-platformio-impact-map.py walks both sources, takes the transitive closure, and either rewrites the map (--write) or fails with a diff when the committed copy disagrees (--check).

The generated file stays committed, so the graph is reviewable in a diff. The comparison is exact, not a superset check, so over-listing can't survive either — it drops CANProtocol -> Templates/SimGateway, which nothing supports.

The check runs in the existing node-ids job: pure Python, already on PRs, so drift is caught on every PR at no added cost. full_build_paths gains the generator, because a change to the derivation invalidates every selection.

Verification

Run with the selector (--mode affected --changed-file …), not by reading the JSON:

changed before after
Libraries/A4EC/… 2 (silently wrong) 9
Libraries/NodeStatus/… full 33 (unknown library) 29
Libraries/CANProtocol/… full 33 (missing project) 28
Libraries/HIDControls/… full 33 31
Libraries/STM32Board/… full 33 29
Libraries/PanelGroup/… full 33 23
Libraries/DrumDisplay/… full 33 (no key) 2
Libraries/PanelBridge/… 2 3
Libraries/SimGateway/… 1 3
Panels/Center_Armament/src/main.cpp 1 1
Libraries/PanelGroup/README.md 0 0
the map or either script full 33 full 33

Drift detection tested both ways — a hand-edited entry and a newly added project each make --check exit 1 with a diff; the generator is idempotent on re-run; --mode full still enumerates all 33 projects.

Interaction with #339 — tested, not assumed

#339 hand-edits this same file (SoakRig + DrumDisplay entries), so the two conflict. I merged it into this branch in a throwaway worktree to check:

  • the conflict is real but confined to .github/firmware-platformio-impact.json — nothing else;
  • resolving it is git checkout --ours on that file, then --write. The generator now prints exactly that when it meets an unresolved conflict, instead of a json.loads traceback;
  • both SoakRig boards are then picked up automatically from their own inis — no hand entries. Post-merge counts over 35 projects: A4EC 11, CANProtocol 30, DrumDisplay 3, HIDControls 33, NodeStatus 31, PanelBridge 3, PanelGroup 25, STM32Board 31, SimGateway 3.

That exercise also turned up a gap worth noting: #339 adds SoakRig under A4EC, CANProtocol, HIDControls, PanelGroup and STM32Board, but not NodeStatus, which both boards also pull in via CANProtocol. Nothing breaks if #339 merges alone — it just keeps under-listing until this regenerates on top. Merge #339 first.

Notes for review

  • Two entries that looked wrong are correct and are kept: HIDControls -> Tests/CANProtocol and Tests/STM32Board. Both declare CANProtocol, which depends on HIDControls. Reading only their platformio.ini makes them look spurious.
  • The firmware build job still does not run on PRs (if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'), and --mode affected is invoked nowhere in the repo. So this PR makes the map correct and drift-proof but does not by itself change what CI builds. Re-enabling affected-mode PR builds is a separate call and deliberately not made here.
  • The node-ids job keeps its id and display name so any required-status-check config stays valid, even though it now does two checks.

library_dependents in .github/firmware-platformio-impact.json was
hand-written and had drifted to roughly a fifth of the real graph. It
listed a Firmware/Prototype/ tree that does not exist and has no history
here, had no NodeStatus key at all, and listed 2 of A4EC's 9 dependents.

Two of those failures are loud: a missing project and an unknown library
both hit the selector's escape hatch, so a CANProtocol or NodeStatus
change fell back to a full build. A4EC's is silent — the selector happily
returned 2 projects and skipped the other 7.

Hand-maintenance cannot fix this, because the graph is not in the
platformio.ini files alone. Each Firmware/Libraries/*/library.json
declares its own dependencies (CANProtocol pulls HIDControls, NodeStatus
and STM32Board; PanelGroup pulls CANProtocol and STM32Board), so a
project listing only PanelGroup still compiles four libraries. Reading
lib_deps alone under-counts NodeStatus by 9 projects and HIDControls by
2. The ini files also spell dependencies three ways — bare names resolved
via lib_extra_dirs, file:// paths, and registry packages — and put
lib_deps on a shared [env_base] that each env extends.

So generate the map instead. gen-platformio-impact-map.py walks both
sources, takes the transitive closure, and either rewrites the map
(--write) or fails with a diff when the committed copy disagrees
(--check). The generated file stays committed so the graph is reviewable
in a diff, and the comparison is exact rather than a superset check, so
over-listing cannot survive either: it drops CANProtocol ->
Templates/SimGateway, which nothing supports.

The check runs in the existing node-ids job. That job is pure Python and
already runs on pull requests, so drift is caught on every PR at no added
cost even though the build job itself stays schedule- and
dispatch-only. The pull_request path filter now covers .github/scripts/**
and the map, and the generator joins full_build_paths because a change to
the derivation invalidates every selection.

Resulting dependent counts, verified by running the selector with
--mode affected rather than by reading the JSON: A4EC 9, CANProtocol 28,
DrumDisplay 2, HIDControls 31, NodeStatus 29, PanelBridge 3,
PanelGroup 23, STM32Board 29, SimGateway 3. A panel sketch change still
selects 1 project, a Firmware doc change 0, and full mode still
enumerates all 33.

Two entries that looked wrong are correct and are kept: HIDControls ->
Tests/CANProtocol and Tests/STM32Board. Both projects declare
CANProtocol, which depends on HIDControls.
The generator reads the committed map to carry full_build_paths over, which
is hand-maintained and cannot be derived. On an unresolved merge conflict that
read hit json.loads and failed with a raw traceback.

Two branches touching the map at once is the expected case, not a rare one --
#339 conflicts with this branch right now -- so turn that traceback into the
resolution: take either side, then regenerate over the top of it.
@mottihoresh
mottihoresh merged commit a9892d4 into main Oct 15, 2026
10 checks passed
@mottihoresh
mottihoresh deleted the mh/elastic-williams-a157a6 branch October 15, 2026 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant