Repository navigation
SWF(perf): walk import sizes in parallel batches - #1954
Merged
Merged
Conversation
Avoid serial filesystem round trips for large workspaces while retaining link sizes and regenerable byte totals. Settle every started operation before reporting budget expiry so no disk work outlives the call.
Contributor
|
I'll fix CI failures and address comments from users with write access that start with 'Devin'.
|
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Fei2411
commented
Nov 21, 2026
Fei2411
left a comment
Owner
Author
There was a problem hiding this comment.
Code Reaper review — verdict: pass (head 6539ca9)
- Walk is equivalent: same
lstat(links not followed), unreadable entries count as 0,REGENERABLEinherited per entry, tallies only summed, so batch order does not matter. OnlyOVERcan reject a batch, sincelstat/readdirerrors are caught. - Budget: every task checks the clock before
lstatand beforereaddir,allSettleddrains the batch, and the post-batch check stops new work. That keeps #1950's{ sessions }over-budget result (#1950 is already indev; this PR's diff againstdevis onlyimport.mjsand its test). node --test test/import.test.mjsinapp/: 33 pass, 0 fail, including the serial-equivalence and in-flight-drain tests.- Reproduced the speedup: dev vs this branch, 6,000 × 1 KB files under
bot/workspace/node_modules, median of 11 runs: 288 ms → 74 ms (3.9×), output identical. - Non-blocking: the frozen
serialTeamSizesin the test copiesREGENERABLE/LEFT_BEHINDby hand, so a future change to those sets has to update the test too. Windows is untested locally; the Windows package CI job was still pending when I reviewed. - Overlaps: no other open PR touches
app/src/import.mjsor its test. PR body has theAgent: Devin (Cognition)line.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Import's size estimate (
teamSizes) is ~3.5–4× faster: it stats up to 16 files at a time instead of one by one, with identical results and the same time budget from #1950.Agent: Devin (Cognition)
Issue
No issue; perf finding from a code scout. Builds on #1950 (teamSizes budget).
What changed / 改了什么
app/src/import.mjsmeasure: the recursive serial walk becomes a pending-entry queue processed in batches of 16. Stilllstat(links counted as themselves, no cycle following), unreadable entries count as 0, regenerable-folder inheritance, per-call counters.Promise.allSettled, so when the budget runs out mid-batch every already-startedlstat/readdirsettles beforeteamSizesreturns Start teamSizes time budget before Session discovery #1950's over-budget result ({ sessions }, sizes omitted); no new work starts after expiry.app/test/import.test.mjs: deep-equal against a frozen copy of the old serial walk (nested dirs, broken and self-referencing symlinks, unreadable file/dir, empty and missing roots, concurrent calls), plus a regression test that expires the budget with stats in flight and asserts the call waits for them. Swapping in a naivePromise.allmakes it fail:must wait for in-flight stats after another task expires.Benchmark — full
teamSizescall,origin/devmodule vs this branch, workspacebot/workspace/node_modules/pkgN/fileM.js(20 × 1 KB files per package), budget 120 s so nothing truncates; 3 warmups, median of 21 alternating runs; outputs deep-equal on every run. Node v24.22.3, macOS 26.6.2, Apple M4 Pro (VM), warm cache.At the default 2 s budget this also means large workspaces get a size instead of timing out.
Try it:
cd app && node --test test/import.test.mjsUntested: Windows and network drives (paths use
join; the self-link fixture uses a junction on Windows, but it was only run on macOS). Cold-cache disks not benchmarked.Changelog
Importing a team from DSH measures large workspaces faster.
How I checked it / 怎么验证的
cd app && npm test— 532 pass, 0 fail, 3 skipped;node --test test/import.test.mjs— 33/33 passnode scripts/build-client.mjs --check— n/a, nods-bot/changenpm run lint— n/a,app/has no lint scriptindex.js/src/host/changeChecklist / 检查
src/client/changed → n/aCo-authored-by. / 没有把 AI 工具写成作者或Co-authored-by。Implemented in session Parallelize import size walk in teamSizes
Review with Devin · Open in Devin Desktop