ComQueue permanently wedges in WAITING when the one-shot arming comStatus is dropped at boot (silent total downlink loss; exposed by PR #472's littlefs Kconfig)

[Mikefly123]

Summary

The ComCcsds downlink stack has a boot-time race that permanently kills all downlink (telemetry, events, command acks) with zero diagnostics. The one-shot arming comStatus=SUCCESS that moves Svc::ComQueue out of its initial WAITING state can be silently dropped if ComQueue's IPC message queue is full at the moment it arrives — and at boot it frequently is, because every component's startup events land on the async comPacketQueueIn before the comQueue thread first drains. Once the arming message is lost there is no retry path in the protocol: ComQueue sits in WAITING forever, and the board looks like it "ignores all commands" (uplink actually works; the acks/events just never come back).

This was found while bench-validating PR #472 merged with feat/ota-bench: adding CONFIG_FILE_SYSTEM_LITTLEFS=y (the /keys key-store mount) shifted boot timing just enough that the boot event burst reliably won the race. littlefs is not at fault — any change to startup workload (new components, more startup events, slower init) can flip a passing build into a permanently wedged one.

Mechanism (GDB-traced on bench hardware)

1. During setupTopology, startup events flood ComQueue.comPacketQueueIn (async, drop semantics), filling the 20-deep IPC queue before the comQueue thread runs.
2. ComAggregator::preamble() emits its single arming comStatus=SUCCESS (the ComStub::drvConnected SUCCESS is separately absorbed by the aggregator state machine, so the preamble is the only arming source that reaches ComQueue).
3. The generated comStatusIn_handlerBase does a NONBLOCKING queue send, gets FULL, and hits FW_ASSERT(qStatus == OP_OK). On our deployment the assert hook (Fw::defaultReportAssert) reports to a console that is disabled (CONFIG_LOG=n) and execution continues — the message is simply gone, invisibly.
4. ComQueue::m_state stays WAITING forever. Captured trace of the fatal sequence: [ENQ comStatus] → [PriorityQueue FULL, size=7] → [assert hook] → permanent WAITING.

Both ComCcsdsUart.comQueue and ComCcsdsLora.comQueue wedge identically.

Reproducibility

Deterministic per-binary, hardware-agnostic — verified on two bench V5e units:

Build	Board 1	Board 2
feat/ota-bench-pr472 pre-fix (littlefs present)	FAIL every boot (m_state=WAITING)	FAIL every boot (identical signature)
Same + fix below	PASS 3× (incl. cold reset)	PASS 2×

Fix (implemented, on branch feat/ota-bench-pr472, pin bump pending fork push)

Declare comStatusIn with block queue-full semantics in Svc/ComQueue/ComQueue.fpp (fprime fork commit de5b46601, branch fix/comqueue-blocking-comstatus; pinned by c3e21d2eb9b):

async input port comStatusIn: Fw.SuccessCondition block

The status sender (aggregator thread, low-rate) blocks until a slot frees — delivery guaranteed, zero RAM cost, no deadlock exposure (the comQueue thread never blocks back toward the aggregator; it sends at most one in-flight frame per status).

Rejected alternative — deepening the queue: QueueSizes.comQueue at 48/64/100 all exhausted the malloc arena at boot (Svc::BufferManager needs a ~27KB contiguous allocation late in configComponents and faults on the NULL; heap slack on this build is <8KB). Documented in ComCcsdsConfig.fpp. Depth also only re-wins the race probabilistically; block removes it categorically.

Follow-ups

• Push fix/comqueue-blocking-comstatus to the OSSF fprime fork and bump the pin on the mainline branches (currently local-only on the bench).
• Consider upstreaming to nasa/fprime — this can wedge any ComCcsds deployment whose assert hook doesn't halt. Related fragility: Svc::ComAggregator: 10 Hz timeout signal FW_ASSERTs (queue FULL) whenever the com path pauses — board-killing on ground test #432 (aggregator timeout FW_ASSERT on full queue) is the same "com stack message-drop asserts are load-bearing" family.
• Separate hazard worth its own issue: with CONFIG_LOG=n, Fw::defaultReportAssert makes every FW_ASSERT failure invisible while execution continues. An assert-to-FATAL (or assert-to-event) hook would have surfaced this bug immediately.

Bench evidence: probe = headless GDS + GET_SEQ_NUM round-trip; broken builds show 0 events and ~12 stray downlink bytes; fixed builds show full SequenceNumberGet/OpCodeCompleted round-trips and flowing telemetry.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WpBURCutAx8281i59nj6fo