rns: stop the path sweep outlasting the watchdog - #120
Merged
Conversation
The dead-path sweep dereferenced every entry in the path table in one pass, and a dereference reads the record back through microStore — one file opened on LittleFS per entry, each open a directory lookup and several 512-byte flash reads that disable the cache and stall the other core. The cost grew with the table. On a V4 with 200 stored paths the pass took about 32 s against a 30 s watchdog, and the node rebooted every ninety seconds for two days. The sweep now spends at most kWalkBudgetMs per pass and resumes from a cursor, so every entry is still reached and no pass is long; the walk feeds the watchdog as it goes, which is what makes a slow filesystem survivable rather than fatal. Stepping the iterator is metadata-only, so positions a pass has no use for are skipped without touching flash. tables: gains snap <ms>, the worst walk since boot — the figure that says whether a node is near the same edge.
There was a problem hiding this comment.
🟢 Approval recommended
Only minor documentation/comment wording issues were identified; the functional changes align with the watchdog-timeout mitigation described in the PR.
Pull request overview
This PR prevents the dead-path sweep over the microStore-backed path table from running long enough to trip the watchdog by time-slicing the sweep across snapshot passes and feeding the watchdog during the walk. It also surfaces the worst snapshot walk duration since boot so operators can see whether a node is approaching filesystem/path-table limits.
Changes:
- Adds a per-pass time budget and cursor-based resumption for the dead-path sweep to avoid unbounded filesystem reads in one pass.
- Feeds the watchdog periodically during path-table record reads to keep long walks survivable.
- Introduces and logs a new
Tables::snapWalkMaxMsmetric (worst snapshot walk time since boot).
File summaries
| File | Description |
|---|---|
| src/rns/RnsTransport.h | Adds Tables::snapWalkMaxMs to report worst snapshot-walk duration since boot. |
| src/rns/RnsTransport.cpp | Implements budgeted, cursor-resumed sweeping and watchdog feeding during path-table reads; publishes the new metric. |
| src/main.cpp | Extends periodic tables log line to include snapWalkMaxMs. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+1715
to
+1718
| // Records read between watchdog feeds. The budget above ends the walk, but it | ||
| // is only checked between records, and one record on a sick filesystem can | ||
| // take a long time on its own — so the walk keeps reporting while it runs | ||
| // rather than relying on finishing. |
Comment on lines
+99
to
+100
| // pass long. A number here in the seconds means the RNS task is spending | ||
| // that long not forwarding. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The dead-path sweep dereferenced every entry in the path table in one pass, and a dereference reads the record back through microStore — one file opened on LittleFS per entry, each open a directory lookup and several 512-byte flash reads that disable the cache and stall the other core. The cost grew with the table. On a V4 with 200 stored paths the pass took about 32 s against a 30 s watchdog, and the node rebooted every ninety seconds for two days.
The sweep now spends at most kWalkBudgetMs per pass and resumes from a cursor, so every entry is still reached and no pass is long; the walk feeds the watchdog as it goes, which is what makes a slow filesystem survivable rather than fatal. Stepping the iterator is metadata-only, so positions a pass has no use for are skipped without touching flash.
tables: gains snap , the worst walk since boot — the figure that says whether a node is near the same edge.