feat: PROVES USP radio integration (collapses PRs #1–#3) - #5
Closed
Mikefly123 wants to merge 7 commits into
Closed
Conversation
timeout_handler forwards a periodic (10 Hz in our configuration) timeout signal into the component's internal state-machine queue. PR nasa#4402 added an m_allow_timeout guard that suppresses this signal while the state machine is in WAIT_STATUS, preventing queue overflow when a com transmission is slow. That guard does not cover the FILL state (m_allow_timeout == true): if the component's own dispatch thread stalls (e.g. downstream backpressure on a serial/radio link, or thread starvation under system load) for longer than queue_depth / timeout_rate, ticks keep arriving and filling the depth-bounded queue while nothing drains it. Once full, the autocoded signal-send in the state machine (aggregationMachine_sendSignal timeout) hits its FW_ASSERT on QUEUE_FULL, which is fatal in flight configurations. fill and status are each flow-controlled to at most one in-flight message by the com protocol, so timeout is the only unbounded producer feeding this queue. Since timeout ticks are periodic and idempotent (a missed tick is retried on the next cycle), skip forwarding the signal whenever the queue does not have headroom for it plus the (at most one each) in-flight fill/status signals, rather than only checking m_allow_timeout. Observed on hardware (PROVES CubeSat, RP2350/Zephyr, F' v3.1.1) during HWIL soak testing: a stalled dispatch thread (CDC-ACM host-side USB stall) let 10 Hz ticks fill a depth-15 queue in ~1.5 s, triggering the queue-full assert and a hard reboot. Generative AI (Claude, Anthropic) was used to help root-cause this defect during HWIL debugging and to draft this change; disclosed per AI_POLICY.md in the upstream pull request description.
The existing test_timeout_overflow_prevention only exercises the WAIT_STATUS-adjacent path: it drives a comStatusIn failure first, so m_allow_timeout is already false during the flood. It does not put the state machine in FILL and stall dispatch, so it does not cover the FILL-state gap closed by the previous commit. Add test_timeout_overflow_prevention_fill_state(), the complement of the existing test: run test_initial() to land in FILL (m_allow_timeout == true) with an empty aggregation buffer and empty queue, then invoke timeout 2 * TEST_INSTANCE_QUEUE_DEPTH times back-to-back with no intervening drain -- simulating a stalled dispatch thread receiving periodic sched ticks. After every invocation, assert the queue never reaches full depth. Then confirm the flood had no side effects (dataOut never fired), confirm recovery via dispatchCurrentMessages once the simulated stall clears, and confirm normal fill/timeout operation resumes cleanly afterward. Note: dispatchOne's underlying doDispatch() issues a blocking Os::Queue::receive with no "queue empty" return path of its own (MSG_DISPATCH_EMPTY is only produced by dispatchCurrentMessages, which snapshots the message count up front). Draining must use dispatchCurrentMessages rather than looping dispatchOne until an empty status, which would hang. Verified: with the bounded-timeout fix applied, all 11 ComAggregator UTs pass (including the new test). With the fix reverted, the new test fails deterministically at the 20th flood iteration (getMessagesAvailable() == TEST_INSTANCE_QUEUE_DEPTH == 20), confirming it exercises the regression this fix addresses. Generative AI (Claude, Anthropic) was used to help draft this test during HWIL follow-up work; disclosed per AI_POLICY.md in the upstream pull request description.
Adds the drop queue-full annotation to two classes of periodic async input ports across Svc, where none currently exists: 1. The Svc.Sched async input on active components that receive a rate- group tick but did not have drop set: CmdSequencer.schedIn, CmdDispatcher.run, TlmChan.Run, TlmPacketizer.Run, FileDownlink.Run, BufferLogger.schedIn, DpManager.schedIn, DpWriter.schedIn. 2. PingIn/pingIn async ports across Svc, including ActiveRateGroup.PingIn and FpySequencer.pingIn (the latter previously priority 10 assert, with an open TODO questioning priority/behavior), plus two ports added upstream since this defect class was first scoped: BufferAccumulator.pingIn and FileDispatcher.pingIn. This is the same defect class fixed for ComAggregator (see companion PR): any active component fed periodic ticks by a rate group can, if its own dispatch thread stalls longer than queue_depth / tick_rate, accumulate enough queued ticks to trip the autocoded FW_ASSERT on queue-full -- turning a transient stall into an unrecoverable FATAL/reboot. Captured on hardware (PROVES CubeSat, RP2350/Zephyr) during HWIL soak testing, in two independently reproduced instances: - CmdSequencer::schedIn_handlerBase (10 Hz sched tick) hit the identical QUEUE_FULL assert as the ComAggregator case, while a sequencer's dispatch thread was stalled. - ActiveRateGroup::PingIn_handlerBase hit the same assert via Svc::Health's 1 Hz ping -- the health-check mechanism itself killed the board. Health's ping-timeout policy exists specifically to detect an unresponsive component and react gracefully; asserting on the ping enqueue short-circuits that design. Both classes of tick (rate-group sched, and ping) are periodic and idempotent by construction -- a dropped tick is simply retried on the next cycle -- so drop is behaviorally safe. Upstream already uses drop for exactly this reason on ComQueue.run and ActiveRateGroup.CycleIn; this extends the same reasoning to the remaining periodic producers that were missed. Related to issue nasa#4195 ("Add a 'Drop But Warn' on Queue Full"), which is in the same design space but currently leans assert-by-default; this change is narrower in scope (periodic/idempotent producers only, per-port opt-in via the existing drop annotation, no new mechanism). This is an fpp-only annotation change consumed by the autocoder; no new C++ logic. A reviewer sweep of existing per-component UTs that assert on QUEUE_FULL behavior for these ports is recommended before merge. Generative AI (Claude, Anthropic) was used to help root-cause this defect class during HWIL debugging (two independently captured hardware instances of the same assert) and to enumerate affected ports; disclosed per AI_POLICY.md in the upstream pull request description.
…gnedSizeType to I16 TlmEntry stores FwSignedSizeType packetOffset[MAX_PACKETIZER_PACKETS] per channel. The value is a byte offset into a ComBuffer-sized packet (bounded by FW_COM_BUFFER_MAX_SIZE, typically a few hundred bytes) or the -1 'not in this packet' sentinel -- a 64-bit signed type per slot is 4-8x oversized. Shrink to I16 (keeps the -1 sentinel; supports packet offsets to 32 KB) with a config-time FW_ASSERT range guard at the single assignment site in setPacketList. Validated on a real CubeSat deployment (MAX_PACKETIZER_CHANNELS=202, MAX_PACKETIZER_PACKETS=22, RP2350/Zephyr, v5e flight board): - -27,472 B BSS (60,480 -> 33,008 B, -45%) in CdhCore::tlmSend - Svc/TlmPacketizer UTs: 12/12 passed (1 pre-existing skip) - HWIL-verified: clean boot, NO_OP round-trip, packetized telemetry decodes bit-correct, SEND_PKT works; live heap walk matches nm prediction exactly (free-at-boot 11,504 -> 38,976 B) Tracked by Open-Source-Space-Foundation/proves-core-reference#469.
…age from FwSignedSizeType to I16
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This was referenced Jul 26, 2026
Author
|
Superseded by native stacked PR chain #1 → #2 → #3 (stack nasa#6, linked via |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Collapses the three open fix PRs against
pcr-usp-baseinto a single integration branch so the superproject's.gitmoduleslib/fprimesubmodule pin has one commit to point at, instead of juggling three separate branches.This branch is a straight
--no-ffmerge of the three PRs in order, with no additional changes. The constituent PRs stay open for traceability / individual review history; this PR is not intended to replace them, only to provide a mergeable integration point for the submodule pin.Constituent PRs (stay open — do not close)
fix/com-aggregator-bounded-timeout— Svc::ComAggregator: bound timeout-signal enqueue in FILL state (includes unit tests)fix/sched-tick-drop-annotations— Svc: drop periodic Sched/Ping ticks on queue-full instead of assertingfix/tlmpacketizer-i16-offset— Svc::TlmPacketizer: shrink per-channel packetOffset storage from FwSignedSizeType to I16Related
Verification
Diff of this branch vs base
8a62e455matches the union of the reference patches (fprime-com-aggregator-bounded-timeout.patch,fprime-sched-tick-drop.patch,fprime-tlmpacketizer-i16-offset.patch) plus the expected UT-file additions from PR #1 (Svc/ComAggregator/test/ut/ComAggregatorTestMain.cpp,ComAggregatorTester.cpp,ComAggregatorTester.hpp). No merge conflicts; no unaccounted-for deltas.🤖 Generated with Claude Code