Svc::ComAggregator: bound timeout-signal enqueue in FILL state - #1
Open
Mikefly123 wants to merge 2 commits into
Open
Svc::ComAggregator: bound timeout-signal enqueue in FILL state#1Mikefly123 wants to merge 2 commits into
Mikefly123 wants to merge 2 commits into
Conversation
timeout_handler forwards a periodic (10 Hz in our configuration) timeout signal into the component's internal state-machine queue. PR nasa#4402 added an m_allow_timeout guard that suppresses this signal while the state machine is in WAIT_STATUS, preventing queue overflow when a com transmission is slow. That guard does not cover the FILL state (m_allow_timeout == true): if the component's own dispatch thread stalls (e.g. downstream backpressure on a serial/radio link, or thread starvation under system load) for longer than queue_depth / timeout_rate, ticks keep arriving and filling the depth-bounded queue while nothing drains it. Once full, the autocoded signal-send in the state machine (aggregationMachine_sendSignal timeout) hits its FW_ASSERT on QUEUE_FULL, which is fatal in flight configurations. fill and status are each flow-controlled to at most one in-flight message by the com protocol, so timeout is the only unbounded producer feeding this queue. Since timeout ticks are periodic and idempotent (a missed tick is retried on the next cycle), skip forwarding the signal whenever the queue does not have headroom for it plus the (at most one each) in-flight fill/status signals, rather than only checking m_allow_timeout. Observed on hardware (PROVES CubeSat, RP2350/Zephyr, F' v3.1.1) during HWIL soak testing: a stalled dispatch thread (CDC-ACM host-side USB stall) let 10 Hz ticks fill a depth-15 queue in ~1.5 s, triggering the queue-full assert and a hard reboot. Generative AI (Claude, Anthropic) was used to help root-cause this defect during HWIL debugging and to draft this change; disclosed per AI_POLICY.md in the upstream pull request description.
The existing test_timeout_overflow_prevention only exercises the WAIT_STATUS-adjacent path: it drives a comStatusIn failure first, so m_allow_timeout is already false during the flood. It does not put the state machine in FILL and stall dispatch, so it does not cover the FILL-state gap closed by the previous commit. Add test_timeout_overflow_prevention_fill_state(), the complement of the existing test: run test_initial() to land in FILL (m_allow_timeout == true) with an empty aggregation buffer and empty queue, then invoke timeout 2 * TEST_INSTANCE_QUEUE_DEPTH times back-to-back with no intervening drain -- simulating a stalled dispatch thread receiving periodic sched ticks. After every invocation, assert the queue never reaches full depth. Then confirm the flood had no side effects (dataOut never fired), confirm recovery via dispatchCurrentMessages once the simulated stall clears, and confirm normal fill/timeout operation resumes cleanly afterward. Note: dispatchOne's underlying doDispatch() issues a blocking Os::Queue::receive with no "queue empty" return path of its own (MSG_DISPATCH_EMPTY is only produced by dispatchCurrentMessages, which snapshots the message count up front). Draining must use dispatchCurrentMessages rather than looping dispatchOne until an empty status, which would hang. Verified: with the bounded-timeout fix applied, all 11 ComAggregator UTs pass (including the new test). With the fix reverted, the new test fails deterministically at the 20th flood iteration (getMessagesAvailable() == TEST_INSTANCE_QUEUE_DEPTH == 20), confirming it exercises the regression this fix addresses. Generative AI (Claude, Anthropic) was used to help draft this test during HWIL follow-up work; disclosed per AI_POLICY.md in the upstream pull request description.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Author
This was referenced Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes fprime issue nasa#432:
ComAggregator's 10 Hz state-machine timeout signal wasunconditionally re-enqueued from the FILL state even when the component's message
queue had no headroom, which could trip the autocoded queue-full assertion if the
dispatch thread stalled. This bounds the enqueue to only happen when there is queue
headroom.
Base branch note
This PR targets
pcr-usp-base, a branch pinned at8a62e455a90b6d4f498c332d45d65a2a819988d8(the v4.2.2-era commit that
lib/fprimeis pinned to inproves-core-reference#439),
not
devel. This fork'sdeveltracks nasa/fprime upstream and has divergedsignificantly from the pinned commit, so basing directly on
develwould pull inhundreds of unrelated upstream commits.
pcr-usp-baseexists purely as a stableintegration point for PRs that need to land against the pinned commit.
Provenance
Cherry-picked from
Mikefly123/fprimebranchfix/com-aggregator-bounded-timeout(commits
67e8b32c4fix +ea6452497UT, upstream-shaped for eventual nasa/fprimesubmission) onto
pcr-usp-base. Both cherry-picks applied cleanly.Verified byte-equivalent: the source diff (
Svc/ComAggregator/ComAggregator.cpp)produced by this branch vs.
pcr-usp-baseis identical to applying the carriedpatch
fprime-com-aggregator-bounded-timeout.patchto a pristine checkout of8a62e455. The only difference is the added unit test files (bonus, not presentin the carried patch):
Svc/ComAggregator/test/ut/ComAggregatorTestMain.cppSvc/ComAggregator/test/ut/ComAggregatorTester.cppSvc/ComAggregator/test/ut/ComAggregatorTester.hppCanonical home
Fork branch
ground-radio-controller-usp-carry(c08cc19) carries a nasa#432-class fixfor GRC PR nasa#25. This PR is intended to become the canonical home for the fix so that
GRC and proves-core-reference consume the same branch/commit rather than maintaining
parallel copies.
Related