Flaky int test: test_safe_09_command_loss_triggers_safe_mode_and_reboot fails two different ways

[nateinaction]

test_safe_09_command_loss_triggers_safe_mode_and_reboot is flaky in the UART integration job. It failed twice in a row on the night of 2026-07-28 (runs 30408763128 and 30410305836, both on PR #472) while passing in the four runs before them on the same branch and the same firmware path. The failures are unrelated to the PR they landed on: ModeManager, ProvesRouter.packetRouted and the command-loss path are untouched by it.

The two runs failed at two different assertions.

1. EnteringSafeMode history snapshot races the event (run 30408763128)

fprime_test_api.assert_event(f"{component}.CommandLossDetected", timeout=5)

events = fprime_test_api.get_event_test_history()
entering_events = [
    e for e in events if "EnteringSafeMode" in str(e.get_template().get_name())
]
assert len(entering_events) > 0, (
    "EnteringSafeMode event should be emitted on command loss"
)

CommandLossDetected arrived on time (FSW 93.552s, "Command loss detected after 1 seconds"), but FSW emits EnteringSafeMode strictly afterwards: commandLossCheck() logs CommandLossDetected, then calls runSafeModeSequence(), and only then enterSafeMode() logs EnteringSafeMode (ModeManager.cpp:547 and ModeManager.cpp:428). runSafeModeSequence() turns off the non-critical load switches over I2C in between, so there is a variable gap. Snapshotting the history the instant CommandLossDetected lands reads it before EnteringSafeMode has been downlinked.

Fix: wait for the event instead of snapshotting, e.g.

entering = fprime_test_api.assert_event(f"{component}.EnteringSafeMode", timeout=10)
assert "contact" in entering.get_display_text().lower()

2. CommandLossDetected never arrives within 5 s (run 30410305836)

19:17:15.623 [Test API] Sending Command: modeManager.COMM_LOSS_TIME_PRM_SET ['{"seconds": 1, "useconds": 0}']
19:17:15.815 [GDS] EVR ... (92.148s) Opcode 0x1000a00c dispatched to port 38
19:17:15.817 [GDS] EVR ... (92.149s) Opcode 0x1000a00c completed
19:17:20.837 [Test API] Item search timed out and ended unsuccessfully.   # CommandLossDetected (268476430)

The parameter set reached the board and completed, the board stayed responsive (teardown GET_CURRENT_MODE answers at FSW 97.448s), and mode was still NORMAL — so the 1 Hz commandLossCheck() did not trip within ~5.3 s of the period being set to 1 s. No dropped uplink: the PRM_SET itself dispatched and completed. Root cause not yet identified; the check needs m_commandLossCounter >= 1 after a single 1 Hz tick, so a 5 s timeout is only ~4 ticks of margin.

Mitigation: raise the CommandLossDetected timeout well above 5 s so a few missed/late rate-group ticks don't fail the run, and keep the failure message specific enough to tell "never triggered" apart from "triggered late".

Notes

• The test is marked requires_watchdog_jumper but CI runs it unfiltered, so both failure modes are live in the UART job.
• HLTH_PING_WARN/HLTH_PING_LATE on CdhCore_cmdDisp and BootCountCorrupted show up in the GDS artifacts of green runs too (e.g. 30362949189), so they are bench background, not a signal for this failure.