Skip to content

failure-modes.md

Latest commit

 

History

History
41 lines (30 loc) · 5.26 KB

failure-modes.md

File metadata and controls

41 lines (30 loc) · 5.26 KB
Raw

Failure Modes

Catalog of how the platform can fail, how each failure is detected, and how it is reported to the user.

Failure Mode Catalog

ID Failure Mode Detection User Signal Recovery
F-001 Pellet Feeder Jam Pellet presence sensor fails to detect a pellet within 30s (kDefaultFeedTimeoutMs) / 4096 steps (kDefaultFeedMaxSteps) during feeding, or sensor remains continuously obstructed. Status LED blinks red; reports ServiceStatus::Jam or ServiceStatus::Timeout in CAN heartbeat/event; base station dashboard alert. Clear hopper/dispenser obstruction. Send CAN Abort command to clear the sticky fault state and return to Idle.
F-002 Actuator Lowering Failure Actuator motor (M2) lowers to seek home, but PG2 (home sensor) fails to trigger within 8s (kDefaultLowerTimeoutMs) or 2048 steps (kDefaultLowerSteps). Status LED blinks red; reports ServiceStatus::Timeout in CAN heartbeat/event; base station dashboard alert. Inspect actuator path for mechanical blocks, verify PG2 alignment, and send CAN Abort command.
F-003 Actuator Raising / Home Clearance Failure Actuator motor (M2) raises pellet, but PG2 fails to clear within 5s (kPg2ClearOnRaiseMs) of starting or the raise motion fails to complete within 8s (kDefaultRaiseTimeoutMs). Status LED blinks red; reports ServiceStatus::Jam or ServiceStatus::Timeout in CAN heartbeat/event; base station alert. Clear obstruction, check sensor wiring, and send CAN Abort command.
F-004 Access Port Open Warning (Extended Access) Spring access port sensor remains open continuously for >30s (kAccessPortOpenWarnMs). Module status LED flashes yellow (warning); reports open access port state in CAN heartbeat; base station UI alert. Inspect spring mechanism or remove any physical obstructions. Clears automatically when spring access port returns to resting state (non-sticky).
F-005 Node Offline (CAN Communication Failure) Base station fails to receive CAN heartbeats (0x200 + nodeId) or general CAN messages from an active node for longer than the timeout period (typically 2-3x the heartbeat interval, e.g., 10-15s). Base station UI marks node as "Offline" (red indicator); logs network event. Verify RJ45 bus cabling, check the 12 V power supply distribution, and confirm that the 120 Ω CAN termination resistor is active on the last node.
F-006 Node Initialization Failure Identity/NVS loading fails (identity.begin() != Ok) or CAN transceiver hardware/TWAI driver fails to start (can.begin() != Ok). Status LED blinks fast during boot; node fails to participate in CAN discovery; remains unassigned. Check MCU board, re-flash firmware, or format NVS storage to force fresh ID discovery.

Detection Categories

  • Automatically detected by firmware: The SFM node checks local sensors (pellet presence sensor, home position sensor, catch attempt sensor) and stepper motor position step-counters and timeouts locally, updating its internal FSM and immediately broadcasting events.
  • Detectable from base station / host software: Base station registers node presence, tracks response times, and monitors missing heartbeats on the CAN bus.
  • Requires manual inspection: Physical damage, loose RJ45 connectors, or stepper driver heating issues.

Reporting Paths

  1. Local Indication: When a fault is declared, the SFM node switches the local status LED.
  2. Network Event Broadcast: The module sends a CanEvent::Fault message (ID 0x300 + nodeId) over the CAN bus containing the exact ServiceStatus code.
  3. Heartbeat Updates: The module packs its current FSM state (byte 0) and the sticky fault/warning code (byte 5) into its periodic 0x200 + nodeId heartbeat packet.
  4. Base Station Host UI: The Raspberry Pi base station monitors the CAN bus, logs incoming event frames, updates the node state registry, and notifies the supervisor software/user interface.

Base Station Presence & Offline Detection

The base station uses a watchdog mechanism based on CAN communication to determine if a node is online:

  • Periodic Heartbeats: Active nodes transmit a status heartbeat frame (0x200 + nodeId) periodically. The default interval is 5 seconds (kDefaultHeartbeatIntervalMs), which can be configured at runtime via CanCmd::SetConfig.
  • Offline Classification: The base station tracks the timestamp of the last message received from each registered Node ID. If no packet is received for 15 seconds (3x the default heartbeat interval), the base station classifies the node as offline.
  • Rejoin Processing: If a node goes offline but then broadcasts a message (e.g., standard heartbeat or rejoin discovery frame), the base station processes the message, checks if the MAC address matches the registered configuration, and marks the node back as online.

Cross-references