Skip to content

Remove timed-out block requests from the replication queue - #942

Closed
hakarrr wants to merge 8 commits into
holepunchto:mainfrom
hakarrr:fix-893-unref-queued-requests
Closed

hakarrr wants to merge 8 commits into
holepunchto:mainfrom
hakarrr:fix-893-unref-queued-requests

Conversation

@hakarrr

@hakarrr hakarrr commented Nov 8, 2026 •

Copy link
Copy Markdown

Fixes #893

BlockRequest._unref() set queued = false but left the request in replicator._queued, and the only place that array is drained is the peer update loop, which needs a connected peer. On a core with no peers, every block request that timed out or was cancelled stayed in _queued until the core closed - one retained BlockRequest per timed-out get, which the issue measures at ~3 GB of heap growth in a week for a process polling mostly-offline cores.

The current revision marks each cancelled request before compacting the queue from BlockRequest._unref(). Compaction is amortized rather than splicing once per cancellation: once more than 16 stale entries dominate the queue, a single pass removes them. This bounds retained timed-out requests to 16 in the regression scenario; it does not promise an empty queue after every timeout.

The current regression performs 100 sequential timed-out gets with no connected peer, then a concurrent batch of 40. It checks REQUEST_TIMEOUT and the maximum observed queue length between compaction passes, asserting that it stays at most 16. This supersedes the original five-request/empty-queue test description.

Original validation (before the subsequent compaction revisions): the new test against both the unfixed and fixed code, test/basic.js, plus prettier --check and lunte on the changed files. One caveat: the full test/replicate.js file OOMs on my 1 GB test box at the pre-existing "one missing block, big core (slow)" test, so the new test also ran standalone; the change itself is confined to BlockRequest._unref().

hakarrr and others added 2 commits November 9, 2026 01:26
BlockRequest._unref() set queued = false but left the request in replicator._queued, which is only drained while a peer is connected. On a core with no peers, every block request that timed out or was cancelled stayed in the queue until the core closed, leaking one BlockRequest per timed-out get.

Co-authored-by: generaltony007 <144252170+GeneralTony007@users.noreply.github.com>
On a core with no peers, run gets that time out and assert that the replication queue is empty afterwards. Without the fix, each timed-out request stays in replicator._queued until the core closes.

Co-authored-by: generaltony007 <144252170+GeneralTony007@users.noreply.github.com>
@hakarrr
hakarrr requested a review from a team November 8, 2026 19:57
@greptile-apps

greptile-apps Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] The PR appears safe to merge.

Summary

Removes cancelled block requests from the replication queue in shared cleanup passes, even when no peer is connected.

  • Timed-out block requests stop piling up in queues without peers.

Reviews (5) · Last reviewed commit: "Assert the queue bound between compactio..." · Reviewed by Greptile

Comment thread lib/replicator.js Resolved
Comment thread test/replicate.js Resolved
hakarrr and others added 2 commits November 9, 2026 02:14
Review feedback: indexOf + splice made each unreference O(n) in the queue length, so mass cancellation became quadratic. Stale entries are now counted and the queue is compacted in a single amortized pass from _queueBlock once they dominate, keeping each unreference O(1). A connected peer's sweep still drops stale entries as before.

Co-authored-by: generaltony007 <144252170+GeneralTony007@users.noreply.github.com>
Review feedback: the test accepted any error, so it could pass without exercising the timeout path. Now matches REQUEST_TIMEOUT, runs 100 timed-out requests, and asserts the queue retains at most the amortization bound (17) instead of all of them.

Co-authored-by: generaltony007 <144252170+GeneralTony007@users.noreply.github.com>
Comment thread lib/replicator.js Resolved
hakarrr and others added 2 commits November 9, 2026 02:48
… linger

Review feedback: compaction waited for a later _queueBlock call, so a batch of requests that all timed out together (or a closing session) stayed in _queued until some future read. The amortized single-pass compaction now runs from _unref itself: still O(1) per unreference (one pass per 16+ stale entries), but cleanup no longer depends on later reads or a connected peer.

Co-authored-by: generaltony007 <144252170+GeneralTony007@users.noreply.github.com>
Adds a 40-request batch that all times out at once, so the test also covers cleanup when no later read triggers it. Fails against the previous revision (whole batch retained), passes now. The bound tightens to 16, matching the compaction threshold.

Co-authored-by: generaltony007 <144252170+GeneralTony007@users.noreply.github.com>
Comment thread lib/replicator.js Resolved
hakarrr and others added 2 commits November 9, 2026 02:57
The compaction pass filtered before the request being cancelled was marked, so that entry survived every pass and went uncounted; sequential timeouts could push the queue one past the bound. Mark it first so the pass drops it too.

Co-authored-by: generaltony007 <144252170+GeneralTony007@users.noreply.github.com>
Tracks the largest the queue grows across the whole run instead of only checking at the end. Sequential timeouts used to peak one past the bound between passes; now covered.

Co-authored-by: generaltony007 <144252170+GeneralTony007@users.noreply.github.com>
@hakarrr

hakarrr commented Nov 13, 2026

Copy link
Copy Markdown
Author

Could a maintainer review this change to remove timed-out block requests from the replication queue? Please let me know if there are concerns with the approach or specific changes needed to move it forward. Thanks.

@hakarrr hakarrr closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Timed-out block requests are never removed from replicator._queued when no peer is connected

1 participant