feat: read and show the Gateway's link-health counters (GET_DIAG) #169

Consolidated 2026-10-27 against firmware 0.3.0 (firmware-v0.3.0).

Firmware refs are on OpenSkyhawk f9fed409, the 0.3.0 release commit. AxisCal.h is identical at the client's sync pin e82ac6e; SimGateway.cpp/.h differ only by the #389 USB-IRQ fix and the #402 calibration/flash guard, neither of which touches DIAG. FP = Firmware/ScratchPad/FirmwarePlan/. Client refs are on SkyHawkClient main aa7b090.

Goal

DIAG is how the Gateway's USB side reports its health. The cockpit side reports its own health on its own, through HEALTH_0 in the node-0 row (#94). The USB side sends nothing unless asked: GET_DIAG → DIAG is its only health (FP07:389-400).

What it says that matters to you is whether the USB side recovered itself: a software re-plug, a recovery reboot or a watchdog reset (OpenSkyHawk/OpenSkyhawk#355, PR #358). When it has, the client shows it on the Gateway's existing fault surfaces and writes one Log › Events row, so the evidence survives until you replug the Gateway and wipe it.

The other counters (link errors, dropped data) are developer diagnostics. They go to the opt-in debug log only, and never on screen.

Design

Client design: https://claude.ai/artifact/R8DXWWC4uyAby54tBQLo8v. Self-recovery goes on the surfaces the cockpit side's faults already use (#94):

• the Gateway tag (pill, devind--warn);
• the Device popover's Health line, opened from the tag and from the Gateway's Attention cog (CockpitMap.dc.html:712-725);
• the Gateway's Cockpit Attention item (gwItems);
• Log › Events (client rows, Log.dc.html EV).

Drawn in the final boards (2026-10-28): Overview.dc.html (tag, popover line, bus-off + recovered), CockpitMap.dc.html (popover, Attention flag RECOVERED ITSELF), Log.dc.html (the gold Events row). This ticket adds no new surface, control or state of its own.

Background: the record (firmware 0.3.0)

• Wire: request GET_DIAG 0x09 (no payload), reply DIAG 0x87, 64 bytes, in the usual calibration envelope (AxisCal.h:205, :219, :368). Bytes 0–46 are used. 47–63 are RESERVED: sent as 0, ignored on receive. A new counter would fill them (AxisCal.h:339-342, :364; FP03:385).
  • Built field by field in CalDiagField order (SimGateway.cpp:499-521). A static_assert breaks the firmware build if a field is added without filling it (:523-524).
  • CAL_PROTO_VERSION stays 1 (FP03:391-395).
• When it's answered: at any time, in or out of a calibration session (SimGateway.cpp:677-702; FP03:445, :468). The uploader only takes its own UPL_* types (SimGateway.cpp:665-673), so GET_DIAG is answered during an upload or a flash too. Nothing refuses or delays it.
• Lifetime: everything lives in RAM from boot (SimGateway.cpp:314), except the three u8 fields at 44–46. Those are copied from watchdog scratch registers that survive a warm reboot (:1146-1167).
  • So the u16 counters and all the times start again from 0 on every boot, including a watchdog or recovery reboot.
  • All of it resets on power-on, and on the Gateway that means a USB replug (FP03:388-389).
  • There's no way to reset it from the client, and there will be none (no new firmware requests).
• Widths:
  • Times are u32 millis() since this boot, and 0 means never (:1051, :1128 write 1 for a real event at 0 ms). They wrap after about 49.7 days of uptime.
  • u16 counters saturate at 0xFFFF, never wrap (_bump, SimGateway.cpp:316). The u8 counts saturate at 0xFF (:1162, :1182).
• The UART counts are passes, not bytes. The four UART error counters, and UART_RX_OVERFLOW, count loop() passes on which that error bit (or the queue overflow) was set: statusTick() reads the PL011 RSR once per loop() (SimGateway.cpp:1332, :1360-1370; AxisCal.h:334-336). A count says that errors happened, not how many bytes were bad.

• The recovery ladder (FP07:366-372; LinkRecovery.h:29-32):
  1. Unhealthy for 5 s: re-plug.
  2. Still unhealthy 5 s later: reboot.
  3. After a recovery reboot: re-plug every 60 s, and never reboot again until the link has been healthy once.
• Firmware versions:
  • GET_DIAG first shipped in 0.2.0 (9e17fa28, #358, is an ancestor of firmware-v0.2.0), with a 47-byte DIAG (AxisCal.h:342 at firmware-v0.2.0).
  • The 64-byte layout is 0.3.0: 5771d0d2 (#359, #295 A27) is in firmware-v0.3.0 but not in firmware-v0.2.0.
  • firmware-v0.1.0 has no GET_DIAG.
  • The client accepts only CAL_DIAG_BYTES (64) (codec.ts:416-424; generated payloadLen: 64, axis-cal.generated.ts:208). So a 0.2.0 Gateway's 47-byte reply isn't a frame to it, and the read ends not supported after the timeout, the same as 0.1.0. The client's minimum firmware is 0.3.0 anyway (client CLAUDE.md § Minimum firmware).
• Bench check (2026-10-27): the firmware side answers on real hardware. tools/simgw_bench/simgw_bench.py diag <port> (OpenSkyhawk/OpenSkyhawk main 88039eb2) against a Gateway with the 64-byte layout, VID 0x2E8A / PID 0x4134, client closed:
  • DIAG came back with 64 bytes. UPTIME_MS read about 65.7 s, RESET_REASON 0 (power-on), and every counter and time was 0.
  • That matches a fresh power-on with no faults. It's also the baseline for Done when step 1.
  • This proves the firmware side only. The client's read path (feat(gateway): read the Gateway's link-health counters (GET_DIAG) #181) still hasn't run against a Gateway.
• Two sides, two health paths:
  • The cockpit side (PanelBridge, NODE_ID 0) reports its own health without being asked: HEALTH_0 every 1 s rolls into the node-0 _NODE_STATUS row (PanelBridge.cpp:868-873). The client shows it on the Gateway's fault surfaces (feat: show the PanelBridge's own health (node 0) outside the cockpit plan #94, gateway-health.ts:58-133).
  • The USB side (SimGateway) sends nothing on its own. DIAG, on request, is its only health. Its LEDs show FAULT live, and nothing else (FP07:281-283).
  • NodeFaultCode::HOST_LINK_LOST (NodeStatus.h:158, "PanelBridge lost the host serial link") is defined but never raised. No library sets it, and PanelBridge.cpp:872 names it "a future host-link watchdog". The client already has note text for it (gateway-health.ts:43-44), which never shows.
• Not related:
  • fix(firmware): DiagSerial never blocks the loop — drop whole lines when the TX buffer is full OpenSkyhawk#398 (DiagSerial, the STM32 debug UART) counts its drops on the board only.
  • CAN DIAG_ERR (CANProtocol.h:220) is defined but never sent.

State of the client today

What's on main: #181 (merged 76f6a2f) and the sync from #178 (#170). The renderer has nothing for it.

• Read:
  • GET_DIAG on every connect, right after a successful HELLO, then GET_CAL (hid-calibration-service.ts:193-205).
  • Again on GatewayConnection.refreshLinkHealth(), while the link is up (gateway-connection.ts:416-419). That's exposed to the renderer as the intent device:diag-refresh / refreshGatewayDiag (gateway/ipc.ts:18; preload/index.ts:78; shared/app/ipc.ts:260-264), and nothing calls it.
  • Never periodic, never retried.
• Timeout: 2 s (cal-link.ts:38), read as { supported: false }, logged once (hid-calibration-service.ts:250-273). The DCS export forward is held while the request is pending (cal-link.ts:111-113).
• Decode: decodeDiag, 64 bytes only (codec.ts:412-447).
• Model: gateway-diag.ts:10-51:
  • resetReason by name;
  • firstFault;
  • selfRecovered = reset reason ≠ power-on, or USB_REATTACHES > 0.
• Carried: the whole record on device:status as linkHealth, only while the link is up (gateway-connection.ts:406-414; shared/gateway/ipc.ts:38-41, :91-141). No surface reads it.
• Debug log: a cal.diag row with every field, on every read, opt-in (hid-calibration-service.ts:262-272; app-services.ts:70; debug-log.ts:27-28).
• Constants: all from npm run sync (axis-cal.generated.ts:136-138, :208, :376-409). The length table is cross-checked against CAL_DIAG_BYTES (sync-a4ec.ts:921-926). No re-pin is needed.
• The cockpit side's health (feat: show the PanelBridge's own health (node 0) outside the cockpit plan #94):
  • gatewayStatus(row, notResponding) is the one owner of what the Gateway shows (gateway-health.ts:1-12, :65-133). It shows only the worst problem.
  • GatewayWatch holds it for one open port (gateway-health.ts:224-320), driven by NodeHub (node-hub.ts:155, :230-247, :338-341).
  • Every surface reads it:
    • the tag and the popover line (gateway-view.ts:22-38; DevicePopover.tsx:87-88);
    • the Attention item (cockpit-snapshot/attention.ts:142-149);
    • the Events rows (gatewayEvents, gateway-health.ts:196-218).

What's wrong with it:

• No evidence by default. The record only reaches the opt-in debug log and a device:status field nothing reads. So a Gateway that recovered itself today leaves no trace, and a replug wipes the record (FP03:388-389).
• Never run against a Gateway. feat(gateway): read the Gateway's link-health counters (GET_DIAG) #181: "Not run against the Gateway: the bench is down". Its mock Gateway isn't in the repo.
• selfRecovered is framed as fault evidence ("the evidence #355 needs", shared/gateway/ipc.ts:138; gateway-diag.ts:45), and treats an unknown reset reason as recovered.
• Field widths are written by hand in the decoder (codec.ts:413, :427-446).
• The layout pin's comment is wrong: the whole-object toEqual fails on a reserved-byte counter (reference.test.ts:326-357).
• Missing tests: a 47-byte wire DIAG, an unknown reset reason, "no HELLO → no GET_DIAG".

What this ticket changes:

• the fix items below;
• it removes the full record from device:status, and the refresh: the renderer intent and the main-side API it exists for.

Scope

Each item is marked keep (on main, checked against the code), fix (on main, changed or removed here) or build (not there yet).

Reading the record

• keep: sync. CAL_TYPE.GET_DIAG/DIAG, CAL_DIAG_FIELD, CAL_DIAG_BYTES, CAL_DIAG_RESET_REASONS via npm run sync (feat: node status v4 — tolerant row parsing and the 0xFF00 roster request (firmware #359 re-sync) #170 / fix(nodes): node status v4 — re-sync to firmware main, tolerant rows, roster request at 0xFF00 #178). Pin e82ac6e, and AxisCal.h there is the same as 0.3.0.
• keep: decoder. decodeDiag reads every field at its synced offset, ignores the reserved bytes and accepts 64 bytes only (codec.ts:412-447). Fixtures in codec.test.ts:393-497.
• fix: field widths from the synced offsets. Each field runs to the next offset, the last to RESERVED. Derive them in the sync, or in one helper over CAL_DIAG_FIELD, and read each field by its derived width (client CLAUDE.md § No magic numbers).
• fix: the layout pin (reference.test.ts:326-357). Pin only the fields the decoder reads, plus RESERVED ≥ 47, so a new counter in the reserved bytes doesn't break CI: it's a firmware minor (FP03:405-407; AxisCal.h:339-342). Make the comment say so. (Decided 2026-10-27.)
• keep: read on connect only, after HELLO. Timeout → not supported, never retried. The export forward is held while pending.
  • No periodic reads. The counters accumulate from boot, and any self-recovery (re-plug or reboot) drops the port, so the read on reconnect catches it.
  • The firmware answers GET_DIAG during calibration and uploads (SimGateway.cpp:665-702), so the connect read never has to wait for either.
• fix: remove the refresh. Decided 2026-10-27: there's no on-demand read and no read after updates. A recovery always drops the port, so a later read in the same connection can't find a recovery the connect read missed; it would only refresh developer counters. Remove:
  • the renderer intent device:diag-refresh / refreshGatewayDiag: gateway/ipc.ts:18, preload/index.ts:78, shared/app/ipc.ts:260-264, app/ipc.ts:72-74, shared/gateway/ipc.ts:220, the harness test-harness/services.ts:258, the renderer mock skyhawk-mock.ts:340, and ipc.compat.test-d.ts:128, :290, :334;
  • the main-side refresh that exists only for it: GatewayConnection.refreshLinkHealth() and the readLinkHealth dependency (gateway-connection.ts:105-109, :416-419; app-services.ts:79);
  • its seam tests: gateway-diag.seam.test.ts:81-93, and the refresh half of :109-121.
• keep: the cal.diag debug row with the full record, opt-in, on every connect. It's the developer view of every counter, including the two that survive a reboot. (Was ticked as user evidence; it isn't one, since it's opt-in. The Events row below replaces it.)

What "recovered itself" means: one owner, gateway-diag.ts

• fix: the rule, this boot only. Rename selfRecovered to e.g. recoveredThisBoot. It's true when:

  • RESET_REASON is WATCHDOG or RECOVERY, or
  • USB_REATTACHES > 0 (it counts from this boot).

  Hand back how, for the copy: watchdog / recovery reboot / re-plug, the reset reason first.

  • RECOVERY_REBOOTS and WATCHDOG_RESETS survive warm reboots, so they describe earlier boots. They go to the debug row only, and never light the tag, Attention or the Events row.
  • An unknown reset reason doesn't count.
  • Reword the docs to "it re-plugged or restarted itself this boot", never an error.

• fix: the record stops riding on device:status. No surface shows the counters, so linkHealth leaves DeviceStatus (shared/gateway/ipc.ts:38-41). Main hands only the conclusion on (next items), and the full record goes to the debug row. pushLinkHealth becomes that hand-off.

• won't fix (accepted): firstFault across a millis() wrap (gateway-diag.ts:28-35). It's wrong only after ~49.7 days of uptime, and sessions are hours long. It stays in the debug row only.

Showing it: one owner, gateway-health.ts, as #94 does

• build: the recovery joins the Gateway's health. gateway-diag.ts decides whether and how it recovered this boot. gateway-health.ts decides how the Gateway shows it, alongside the cockpit side's HEALTH_0:
  • Input: gatewayStatus(row, notResponding, recovered). NodeHub feeds GatewayWatch the conclusion from the connect read: app-services.ts:71 routes it to the hub, next to report() / answered().
    • GatewayWatch.reset() on port open and close drops it with the link, as it drops the row (node-hub.ts:230-235).
    • A read that lands after the link went down is still discarded.
  • Both stay visible. The cockpit side's worst problem stays kind. The recovery is a second fact on GatewayStatus (e.g. recovered: { how, note }), so neither hides the other.
    • Severity is the worse of the two, and the recovery alone is warn. It never makes err, never the Overview alert, never "a problem" wording.
    • A recovery is enough to give the Gateway a status before its node-0 row arrives.
  • Surfaces, all reading GatewayStatus only:
    • tag devind--warn (gateway-view.ts:22-38);
    • the popover gets a line saying what happened, e.g. "Restarted itself to recover its link to this computer" (DevicePopover.tsx:87-88);
    • the Attention item (attention.ts:142-149) shows it when nothing worse does.
  • Shown on every connection to that boot. It clears only with the link, or at a power-on, which zeroes the count and resets the reason.
• build: one Log › Events row per new recovery.
  • gatewayEvents (gateway-health.ts:196-218), the one owner of the Gateway's rows, writes one client · Gateway row (warn, event e.g. "recovered itself", detail = how).
  • Only when the recovery is new since the last row for that Gateway. Whether it's new is a pure function in gateway-diag.ts. The memory is held app-session only (not on disk), keyed by the Gateway's USB serial (gateway-connection.ts:345-351, read at hid-calibration-service.ts:191; no serial → one shared key). It survives GatewayWatch.reset().
    • A recovery is new if its boot differs from the last row's, or, in the same boot, USB_REATTACHES has gone up.
    • The boot is the read time minus UPTIME_MS, matched within a named client constant (e.g. BOOT_MATCH_MS). The counts alone can't tell boots apart: a power-on zeroes USB_REATTACHES, so a re-plug in a new boot can show the same count as the last row.
  • A reconnect to the same boot with nothing new (Disconnect / Connect, app restart within the session, PC wake) writes nothing. There's no "cleared" row.
• build: copy as the design writes it. Never name the transport: say "its link to this computer", not "USB link" / "UART" / "CDC". The user's example, "restarted itself to recover the USB link", is reworded so.

Tests

• gateway-diag.recovered: reset reason watchdog → true (how = watchdog); recovery → true; USB_REATTACHES > 0 with power-on → true (how = re-plug); all clear → false.
• diag.old-counts-dont-flag: power-on reason, USB_REATTACHES 0, RECOVERY_REBOOTS / WATCHDOG_RESETS > 0 → not recovered, and the debug row still carries both counts.
• gateway-diag.unknown-reason: an unknown reset reason with no re-plugs → not recovered, and unknown in the debug row.
• gateway-health.recovered-alone: a recovery with a healthy or absent node-0 row → warn, tag, popover line and Attention item. Never err, no alert.
• gateway-health.recovered-with-fault: a cockpit-side bus-off plus a recovery → err from bus-off, and the recovery still on the status.
• events.first-recovery: the first connect that reads a recovery writes one Events row.
• events.no-duplicate-on-reconnect: close and reopen the port to the same boot (same serial, same boot time, same counts) → the tag shows again, and no second row.
• events.new-reattach-same-boot: same boot, USB_REATTACHES up by one on reconnect → one new row.
• events.new-boot: a later boot (different boot time) with a recovery → a new row, even when USB_REATTACHES equals the last row's.
• events.per-gateway: two serials keep separate memory.
• gateway-diag.seam.recovered (via makeServices()): a simulated Gateway answering a recovered DIAG on connect → the Gateway's status carries it, the Events row is written, and device:status carries no linkHealth.
• gateway-diag.seam.no-hello: no HELLO answer → no GET_DIAG, and nothing recovered.
• gateway-diag.seam.connect-only: after connect, no further GET_DIAG is ever sent while the link stays up.
• codec.diag-widths: the widths derived from CAL_DIAG_FIELD match what the decoder reads.
• codec.diag-047: a 47-byte DIAG frame on the wire (0.2.0 layout) isn't taken as a frame, and the read ends not supported.
• reference.diag-pin: the pin passes with an extra field added in the reserved bytes (RESERVED moved past 47), and fails if a field the decoder reads moves.

Design (resolved)

The boards above draw the recovery. Decided 2026-10-28 and built in PR #195, where the code differs from the boards (a board touch-up is pending):

• The wording is "since it was plugged in", not "this session". The recovery lasts until a power-on (= replug), across app restarts.
• The re-plug and watchdog lines: "Recovered itself since it was plugged in — re-plugged itself to recover its link to this computer." / "… — its watchdog restarted it."
• Popover order: Health → recovery line → health note → Firmware (Overview's order).
• A recovery before the cockpit side reports health: no Health row, the line alone.
• The gold line is the .gw__note.gw--warn class.
• After a real app restart, the Events row is written again (the Log is cleared on restart). A clock step or the uptime wrap may write one extra row; that's accepted.

Out of scope

• Showing the counters (link errors, dropped data, stalls, and the two counts that survive a reboot) anywhere but the debug log. They're developer diagnostics, and never a fault.
• Reading DIAG after updates or on demand: removed (Scope).
• Resetting the counters from the client: there's no firmware request, and none will be added.
• Any new firmware reporting or new DIAG fields, including raising HOST_LINK_LOST (no demonstrated need). Any change to _NODE_STATUS / _NODE_TELEM.
• The unused HOST_LINK_LOST note (gateway-health.ts:43-44): feat: show the PanelBridge's own health (node 0) outside the cockpit plan #94's, left as is.
• The cockpit side's health and LED: feat: show the PanelBridge's own health (node 0) outside the cockpit plan #94, feat(panelbridge): a dead node's warning LED clears itself; the client judges missing boards OpenSkyhawk#393, feat: Cockpit shows your cockpit — seen/adopted boards only, offline-only Hide/Forget for boards and PDUs #93. Input throttling: feat(firmware): input throttling — a noisy node can't flood the Gateway link or starve the sim OpenSkyhawk#391. DiagSerial: fix(firmware): DiagSerial never blocks the loop — drop whole lines when the TX buffer is full OpenSkyhawk#398. None adds to DIAG.
• Pointing docs/firmware/debugging.md:82-92 at the client: a firmware-repo docs change once this ships.

Blocked by

• Firmware: none (0.3.0). The sync is done (feat: node status v4 — tolerant row parsing and the 0xFF00 roster request (firmware #359 re-sync) #170, closed).

Blocks

None.

Done when

• Every test above passes in CI, with npm run lint, npm run lint:deps and npm run typecheck. No literal offset, length, width or reset-reason value in src/ outside @shared/firmware. No device:diag-refresh left in src/.
• The surfaces match the approved boards.
• On the bench, with a 0.3.0 Gateway (2026-10-28, PR feat(gateway): show when the Gateway recovered itself (GET_DIAG) #195). This was the first hardware run of feat(gateway): read the Gateway's link-health counters (GET_DIAG) #181's read path.
  1. Pass. The client's cal.diag row agrees with simgw_bench.py diag. A power-on gives POWER_ON with every count at 0, and nothing is flagged. Disconnect / Connect gives exactly one read per connect, and both reads are the same boot.
  2. Pass. After a cockpit-side power cycle (~5 s), the client shows Not responding (feat: show the PanelBridge's own health (node 0) outside the cockpit plan #94) while it's off. After the reconnect, uartFraming / uartBreak / uartRestarts rise. It's still POWER_ON, and nothing is flagged.
  3. Removed. "PC shutdown with USB power kept" can't happen: the Gateway's RP2040 is powered only by the computer's USB.
  4. Pass, via UF2 flashes. Every flash (0.3.0 ↔ 0.2.0 / 0.1.0) restarts as POWER_ON, with WATCHDOG_RESETS and RECOVERY_REBOOTS at 0. So an update never reads as a recovery. feat: update the Gateway from a firmware release #173's bench check repeats this with the client's own update path.
  • Older firmware: a 0.2.0 Gateway (47-byte DIAG) and a 0.1.0 Gateway (no GET_DIAG) both end not supported after 2.0 s. Calibration still loads, and nothing is flagged.
• A real self-recovery isn't provoked on the bench. It's carried by test(firmware): v1.0 stability gate — multi-node live-DCS soak + API/wire freeze review OpenSkyhawk#290 (§ Gateway link health; don't replug if the Gateway drops off). When it happens there, the tag and one Events row should show it, and a reconnect shouldn't repeat the row.

Landed in PR #195 (squash), plus review fixes: only the counters it reads are checked, so a new firmware counter can't break CI; main decides problem / noHealth and the screens only word them.

Open questions

None.