Skip to content

[Nexthop][bugfix] Drain in-flight event sinks before warm and cold boot exit to fix FDB UAF - #1668

Open
maisey-nexthop wants to merge 2 commits into
meta:mainfrom
nexthop-ai:maisey.thrift-shutdown-fix
Open

maisey-nexthop wants to merge 2 commits into
meta:mainfrom
nexthop-ai:maisey.thrift-shutdown-fix

Conversation

@maisey-nexthop

@maisey-nexthop maisey-nexthop commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Pre-submission checklist

  • I've run the linters locally and fixed lint errors related to the files I modified in this PR. You can install the linters by running pip install -r requirements-dev.txt && pre-commit install
  • pre-commit run

Summary

Drain active hw-agent event sinks before SwSwitch teardown. Cancellation alone does not wait for callbacks already processing events. Track active sinks, reject new ones during shutdown, and wait before graceful or cold-boot teardown.

The drain waits up to 5 seconds. On timeout, it logs the active sink count and continues shutdown.

Test Plan

Benchmark tests pass consistently.

Ran the memory_scale_benchmarks suite on a TH6 unit with the cold-boot drain applied: SystemScaleChurnMemoryBenchmark and SystemScaleMemoryBenchmark both complete, with no rc -11 and no event syncer drain timeouts. Without the change, the churn benchmark exited with rc -11 in MacTableManager::handleL2LearningUpdate.

@meta-cla meta-cla Bot added the CLA Signed label Sep 23, 2026
@maisey-nexthop
maisey-nexthop force-pushed the maisey.thrift-shutdown-fix branch from 0756864 to 5f4f443 Compare October 13, 2026 23:01
@maisey-nexthop maisey-nexthop changed the title [Nexthop] Drain in-flight event sink coroutines before graceful exit to fix FDB UAF [Nexthop] Drain in-flight event sinks before warm and cold boot exit to fix FDB UAF Oct 13, 2026
@github-actions github-actions Bot removed the nexthop label Oct 13, 2026
@maisey-nexthop
maisey-nexthop marked this pull request as ready for review October 13, 2026 23:44
@maisey-nexthop
maisey-nexthop requested a review from a team as a code owner October 13, 2026 23:44
@maisey-nexthop maisey-nexthop changed the title [Nexthop] Drain in-flight event sinks before warm and cold boot exit to fix FDB UAF [Nexthop][bugfix] Drain in-flight event sinks before warm and cold boot exit to fix FDB UAF Nov 2, 2026
@github-actions github-actions Bot added nexthop project:bugfix FBOSS project: bugfix labels Nov 2, 2026
@maisey-nexthop
maisey-nexthop force-pushed the maisey.thrift-shutdown-fix branch 2 times, most recently from b239831 to 5096189 Compare November 4, 2026 17:49
…to fix FDB UAF

Signed-off-by: maisey-nexthop <maisey@nexthop.ai>
exitForColdBoot() only requested cancellation of the hw agent event
sinks before SwSwitch::stop(), which resets macTableManager_. An FDB
event still being delivered by co_notifyFdbEvent could then call
l2LearningUpdateReceived() on a null MacTableManager and SIGSEGV.

Wait for the sinks to drain after cancelling, as handleExitSignal()
now does.

Signed-off-by: maisey-nexthop <maisey@nexthop.ai>
@maisey-nexthop
maisey-nexthop force-pushed the maisey.thrift-shutdown-fix branch from 5096189 to 25620fd Compare November 4, 2026 18:52

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant