Repository navigation
ci(firmware): derive the PlatformIO impact map from the firmware tree - #341
Merged
Merged
Conversation
library_dependents in .github/firmware-platformio-impact.json was hand-written and had drifted to roughly a fifth of the real graph. It listed a Firmware/Prototype/ tree that does not exist and has no history here, had no NodeStatus key at all, and listed 2 of A4EC's 9 dependents. Two of those failures are loud: a missing project and an unknown library both hit the selector's escape hatch, so a CANProtocol or NodeStatus change fell back to a full build. A4EC's is silent — the selector happily returned 2 projects and skipped the other 7. Hand-maintenance cannot fix this, because the graph is not in the platformio.ini files alone. Each Firmware/Libraries/*/library.json declares its own dependencies (CANProtocol pulls HIDControls, NodeStatus and STM32Board; PanelGroup pulls CANProtocol and STM32Board), so a project listing only PanelGroup still compiles four libraries. Reading lib_deps alone under-counts NodeStatus by 9 projects and HIDControls by 2. The ini files also spell dependencies three ways — bare names resolved via lib_extra_dirs, file:// paths, and registry packages — and put lib_deps on a shared [env_base] that each env extends. So generate the map instead. gen-platformio-impact-map.py walks both sources, takes the transitive closure, and either rewrites the map (--write) or fails with a diff when the committed copy disagrees (--check). The generated file stays committed so the graph is reviewable in a diff, and the comparison is exact rather than a superset check, so over-listing cannot survive either: it drops CANProtocol -> Templates/SimGateway, which nothing supports. The check runs in the existing node-ids job. That job is pure Python and already runs on pull requests, so drift is caught on every PR at no added cost even though the build job itself stays schedule- and dispatch-only. The pull_request path filter now covers .github/scripts/** and the map, and the generator joins full_build_paths because a change to the derivation invalidates every selection. Resulting dependent counts, verified by running the selector with --mode affected rather than by reading the JSON: A4EC 9, CANProtocol 28, DrumDisplay 2, HIDControls 31, NodeStatus 29, PanelBridge 3, PanelGroup 23, STM32Board 29, SimGateway 3. A panel sketch change still selects 1 project, a Firmware doc change 0, and full mode still enumerates all 33. Two entries that looked wrong are correct and are kept: HIDControls -> Tests/CANProtocol and Tests/STM32Board. Both projects declare CANProtocol, which depends on HIDControls.
The generator reads the committed map to carry full_build_paths over, which is hand-maintained and cannot be derived. On an unresolved merge conflict that read hit json.loads and failed with a raw traceback. Two branches touching the map at once is the expected case, not a rare one -- #339 conflicts with this branch right now -- so turn that traceback into the resolution: take either side, then regenerate over the top of it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
library_dependentsin.github/firmware-platformio-impact.jsonis hand-written and had drifted to roughly a fifth of the real dependency graph:Firmware/Prototype/tree that does not exist and has no history in this repo;NodeStatuskey at all;The first two fail loud — a missing project and an unknown library both hit the selector's escape hatch, so a
CANProtocolorNodeStatuschange fell back to a full build. A4EC's failure is silent: the selector returned 2 projects and skipped the other 7.Why hand-maintenance can't fix it
The graph is not in the
platformio.inifiles alone. EachFirmware/Libraries/*/library.jsondeclares its own dependencies:A project listing only
PanelGroupstill compiles four libraries. Readinglib_depsalone under-counts NodeStatus by 9 projects and HIDControls by 2. The ini files also spell dependencies three ways — bare names resolved vialib_extra_dirs,file://paths, and registry packages — and putlib_depson a shared[env_base]that each envextends.Approach
Generate the map instead.
.github/scripts/gen-platformio-impact-map.pywalks both sources, takes the transitive closure, and either rewrites the map (--write) or fails with a diff when the committed copy disagrees (--check).The generated file stays committed, so the graph is reviewable in a diff. The comparison is exact, not a superset check, so over-listing can't survive either — it drops
CANProtocol -> Templates/SimGateway, which nothing supports.The check runs in the existing
node-idsjob: pure Python, already on PRs, so drift is caught on every PR at no added cost.full_build_pathsgains the generator, because a change to the derivation invalidates every selection.Verification
Run with the selector (
--mode affected --changed-file …), not by reading the JSON:Libraries/A4EC/…Libraries/NodeStatus/…unknown library)Libraries/CANProtocol/…missing project)Libraries/HIDControls/…Libraries/STM32Board/…Libraries/PanelGroup/…Libraries/DrumDisplay/…Libraries/PanelBridge/…Libraries/SimGateway/…Panels/Center_Armament/src/main.cppLibraries/PanelGroup/README.mdDrift detection tested both ways — a hand-edited entry and a newly added project each make
--checkexit 1 with a diff; the generator is idempotent on re-run;--mode fullstill enumerates all 33 projects.Interaction with #339 — tested, not assumed
#339 hand-edits this same file (SoakRig + DrumDisplay entries), so the two conflict. I merged it into this branch in a throwaway worktree to check:
.github/firmware-platformio-impact.json— nothing else;git checkout --ourson that file, then--write. The generator now prints exactly that when it meets an unresolved conflict, instead of ajson.loadstraceback;That exercise also turned up a gap worth noting: #339 adds SoakRig under A4EC, CANProtocol, HIDControls, PanelGroup and STM32Board, but not NodeStatus, which both boards also pull in via CANProtocol. Nothing breaks if #339 merges alone — it just keeps under-listing until this regenerates on top. Merge #339 first.
Notes for review
HIDControls -> Tests/CANProtocolandTests/STM32Board. Both declareCANProtocol, which depends onHIDControls. Reading only theirplatformio.inimakes them look spurious.if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'), and--mode affectedis invoked nowhere in the repo. So this PR makes the map correct and drift-proof but does not by itself change what CI builds. Re-enabling affected-mode PR builds is a separate call and deliberately not made here.node-idsjob keeps its id and display name so any required-status-check config stays valid, even though it now does two checks.