fix(Svc): never FATAL on comms overload — drop instead of assert - #4
fix(Svc): never FATAL on comms overload — drop instead of assert#4Mikefly123 wants to merge 7 commits into
Conversation
Works around the fact that `external parameters` don't respect default values.
* Add website local rendering and improve navigability * update docstring * Change file permissions to appease CI ??
* Fix FileWorker compatibility with 32-bit baremetal platforms * Improve non-Posix registration of FileTester Co-authored-by: Copilot <copilot@github.com> * use STATUS instead of INFO --------- Co-authored-by: thomas-bc <thomas.boyerchammard@gmail.com> Co-authored-by: Copilot <copilot@github.com>
Three FW_ASSERT sites turned transient overload (e.g. an SD-stalled FileUplink backing buffers up until the com buffer pool exhausted) into a fatal cascade that left the flight processor unrecoverable: - Ccsds::SpacePacketFramer: bufferAllocate failure fed an invalid buffer into serializeFrom -> FW_ASSERT. Now drops the frame with a throttled FrameDropped WARNING_HI event, returns the data buffer upstream, and emits a substitute comStatus SUCCESS so the ComQueue handshake keeps flowing (comStatusIn is async on ComQueue, so no re-entrancy; the dropped frame never reaches the driver, preserving 1:1 status tokens). - CmdSequencer schedIn/pingIn: queue-full on the 10Hz tick or a health ping FW_ASSERTed. Both are droppable -- a missed tick resumes timer bookkeeping next cycle, and a missed ping correctly surfaces as a late-ping health warning instead of a crash. - ActiveRateGroup PingIn: same queue-full assert, same drop rationale. Observed on PROVES flight hardware (RP2350/Zephyr): a single stalled SD write during a 204KB file uplink produced 660 CmdSequencer schedIn asserts in one run, stopped the watchdog, and required a power cycle. With these changes the same overload degrades to dropped frames and health warnings with the board fully commandable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WpBURCutAx8281i59nj6fo
|
Warning Review limit reached
Next review available in: 26 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (12)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…never-fatal-comms
5 participants
Problem
Three
FW_ASSERTsites turn transient overload into an unrecoverable fatal cascade:Svc::Ccsds::SpacePacketFramer::dataIn_handler— a failedbufferAllocate(pool exhausted) feeds an invalid buffer intoserializeFrom, which FW_ASSERTs.Svc::CmdSequencerschedIn/pingIn— queue-full on a 10Hz sched tick or health ping FW_ASSERTs in the generated handler base.Svc::ActiveRateGroupPingIn— same queue-full assert (CycleInalready hasdrop).Observed on PROVES flight hardware (RP2350/Zephyr): one stalled SD write during a 204KB file uplink backed buffers up until the comms pool exhausted, producing 660 CmdSequencer schedIn asserts in a single run, stopping the watchdog, and leaving the processor byte-silent until power cycle (Open-Source-Space-Foundation/proves-core-reference#471).
Fix
FrameDroppedWARNING_HI event, return the data buffer viadataReturnOut, and emit a substitutecomStatusOutSUCCESS so the ComQueue handshake keeps flowing. Safe:ComQueue.comStatusInis async (no re-entrancy), and the dropped frame never reaches the driver so status tokens stay 1:1.schedIn/pingIn, ActiveRateGroupPingIn:dropsemantics. A missed sched tick resumes bookkeeping next cycle; a missed ping surfaces as a late-ping health warning rather than a crash.Verification
On the hardware that reproduced the FATAL 2/2: full UART file-transfer integration suite (9 tests incl. two 204KB round trips and three consecutive 204KB uplinks) green from one boot; induced pool exhaustion now degrades to
FrameDropped/NoBuffsAvailable/HLTH_PING_LATEwarnings with the board fully commandable.Companion changes: OSSF/fprime-extras fan-out map capacity PR + proves-core-reference config/topology PR (pin bump lands there). Intended for upstream (nasa/fprime) submission after review here.
🤖 Generated with Claude Code
https://claude.ai/code/session_01WpBURCutAx8281i59nj6fo