Limit path depth and add cache cleanup#523
Merged
Conversation
Prevent excessive memory usage and reduce GC pressure by limiting traversal path depth in data-filter (cap depth logic at 20). In grouper worker, add a periodic cache cleanup interval (every 5 minutes) started on worker start and cleared on finish to avoid unbounded cache growth. Free large references after delta computation by nulling event payloads to allow garbage collection. Also tighten memoization for findSimilarEvent (max reduced from 200 to 50 and ttl set to 600s) to further limit memory retained by caches.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR aims to reduce memory usage/GC pressure in the Grouper worker pipeline by limiting deep traversal work in the data filter, bounding cache growth, and tightening memoization settings.
Changes:
- Added a periodic cache cleanup interval in
GrouperWorker.start()and cleared it infinish(). - Adjusted
findSimilarEventmemoization limits (max) and TTL. - Modified
DataFiltertraversal to (intended) cap path depth during recursion.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| workers/grouper/src/index.ts | Adds periodic cache clearing, changes memoization settings, and attempts to free large references after delta computation. |
| workers/grouper/src/data-filter.ts | Attempts to cap traversal path depth to reduce memory allocations for deeply nested objects. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
neSpecc
reviewed
Feb 10, 2026
Limit path growth in data filter to avoid creating new arrays past 20 levels (reduces excessive allocations for deeply nested objects). Import TimeMs and replace magic numbers: set MEMOIZATION_TTL to 600_000, use TimeMs.MINUTE for cache cleanup interval, and apply MEMOIZATION_TTL to the memoize decorator. Clear large delta references by setting them to undefined to aid GC. Add a test that verifies filtering works on objects nested >20 levels without causing excessive memory allocations.
Introduce MAX_TRAVERSAL_DEPTH in data-filter.ts and replace the hardcoded depth check (20) to prevent excessive memory allocations from deep object nesting. Add CACHE_CLEANUP_INTERVAL_MINUTES in index.ts and use it for the cache cleanup setInterval instead of the literal 5, improving readability and making these tuning values easier to adjust.
Remove the unnecessary eslint-disable-next-line on the memoize import and apply the no-unused-vars ignore to the MEMOIZATION_TTL constant instead. This ensures the linter suppression targets the unused constant (decorators not counted) rather than the import, improving clarity.
Add an explanatory comment and an `/* eslint-disable-next-line no-unused-vars */` directive before the `memoize` import in workers/grouper/src/index.ts. This prevents ESLint from flagging the import as unused since decorators (which rely on the import) are not recognized as usages by the linter.
neSpecc
approved these changes
Feb 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Review required!
Prevent excessive memory usage and reduce GC pressure by limiting traversal path depth in data-filter (cap depth logic at 20). In grouper worker, add a periodic cache cleanup interval (every 5 minutes) started on worker start and cleared on finish to avoid unbounded cache growth. Free large references after delta computation by nulling event payloads to allow garbage collection. Also tighten memoization for findSimilarEvent (max reduced from 200 to 50 and ttl set to 600s) to further limit memory retained by caches.