Fix event handler leak in ExecuteLocal by unregistering handlers on cleanup#951
Open
kshinjo-pfr wants to merge 1 commit intoros2:rollingfrom
Open
Fix event handler leak in ExecuteLocal by unregistering handlers on cleanup#951kshinjo-pfr wants to merge 1 commit intoros2:rollingfrom
kshinjo-pfr wants to merge 1 commit intoros2:rollingfrom
Conversation
b4fe2ad to
b3ecefd
Compare
…leanup ExecuteLocal registers 6 event handlers in execute() but never unregisters them when the process exits. This causes a memory leak proportional to the number of processes launched over the lifetime of a LaunchService, as the handlers and all objects they reference remain in context._event_handlers indefinitely. This change: - Stores registered event handlers and context as instance variables - Adds __unregister_event_handlers() helper that safely removes all handlers registered by this action - Unregisters handlers in __flush_buffers/__flush_cached_buffers after all ProcessExited handlers (including on_exit) have fired - Unregisters handlers when process fails to start (no ProcessExited) - Fixes _shutdown_process() deferred handler accumulation by tracking and replacing the OnProcessStart handler instead of creating new ones - Adds tests verifying handler cleanup and stability during respawn Fixes ros2#565 Signed-off-by: Koki Shinjo <kshinjo@pfrobotics.jp>
b3ecefd to
54a8377
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #565.
ExecuteLocalregisters 6 event handlers inexecute()but never unregisters them when the process exits normally. This causes a memory leak proportional to the number of processes launched over the lifetime of aLaunchService, as the handlers (and all objects they reference, including the action itself) remain incontext._event_handlersindefinitely.Additionally,
_shutdown_process()registers a newOnProcessStarthandler each time it is called before the process has started, which can also accumulate.Measured impact: On a robotics system running ROS 2 Jazzy with 12
ros2 launchprocesses (each managing composable node containers), we observed ~56 MB/h memory growth from Python launch processes alone, leading to OOM kills every 17-29 hours.Problem demonstration
The new tests confirm the leak exists on unpatched code:
test_event_handlers_cleaned_up_after_process_exittest_multiple_processes_handler_cleanuptest_event_handlers_stable_during_respawnThe first two tests fail without this fix, proving that event handlers accumulate in
context._event_handlersand are never cleaned up. EachExecuteLocalinstance leaks exactly 6 handlers.Changes
__unregister_event_handlers()helper method__flush_buffers/__flush_cached_buffersafter allProcessExitedhandlers (includingon_exit) have had a chance to fireProcessExitedevent), unregister handlers directly_shutdown_process()deferred handler accumulation: track and replace theOnProcessStarthandler instead of creating new onesDesign note
Handlers cannot be unregistered in
__cleanup()because__cleanup()runs afteremit_event(ProcessExited)enqueues the event but before the event is actually processed. Unregistering there would preventon_exitcallbacks from firing. Instead, unregistration is deferred to__flush_buffers/__flush_cached_buffers, which are themselvesOnProcessExithandlers and run after all otherProcessExitedhandlers (includingon_exit). They check__completed_future.done()to determine if the action is truly complete (i.e., not respawning).Testing
Is this user-facing behavior change?
No. This is an internal cleanup that prevents memory leaks. All existing behavior (respawn, on_exit callbacks, shutdown) is preserved.
Did you use Generative AI?
Yes. Claude (Anthropic) was used to assist with code analysis, implementation, and test writing. All changes were reviewed and verified by a human.
Additional Information
The 6 event handlers registered per
ExecuteProcessinstance are:EventHandlerforShutdownProcessEventHandlerforSignalProcessOnProcessIOfor stdin/stdout/stderrOnShutdownfor graceful shutdownOnProcessExitfor user'son_exitcallbackOnProcessExitfor buffer flushingThese were never unregistered, keeping the
ExecuteLocalinstance and its I/O buffers alive for the entire lifetime of theLaunchService.