Skip to content

Memory grows unbounded during long pagination #7

Description

@sujito00

Bug

The paginated methods (searchAll, getUserTweetsAll, getBookmarksAll) use a seen Set to deduplicate tweets. This Set accumulates all tweet IDs indefinitely.

File: src/api/client.ts (lines ~514-623)

For getBookmarksAll() with maxPages = Infinity, this could grow to millions of entries on accounts with large bookmark collections.

Expected behavior

Either:

  • Use a bounded data structure (e.g., last N IDs) since duplicates typically only appear in adjacent pages
  • Stream results instead of accumulating them (yield tweets as they're fetched)
  • Add a hard cap with a warning

Impact

Low for typical use. High for power users exporting large datasets (bookmarks --all on accounts with 10k+ bookmarks).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions