Skip to content

SWF(perf): walk import sizes in parallel batches - #1954

Merged
Fei2411 merged 1 commit into
devfrom
chore/import-size-parallel-walk
Nov 21, 2026
Merged

Fei2411 merged 1 commit into
devfrom
chore/import-size-parallel-walk

Conversation

@Fei2411

@Fei2411 Fei2411 commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

Import's size estimate (teamSizes) is ~3.5–4× faster: it stats up to 16 files at a time instead of one by one, with identical results and the same time budget from #1950.

Agent: Devin (Cognition)

Issue

No issue; perf finding from a code scout. Builds on #1950 (teamSizes budget).

What changed / 改了什么

  • app/src/import.mjs measure: the recursive serial walk becomes a pending-entry queue processed in batches of 16. Still lstat (links counted as themselves, no cycle following), unreadable entries count as 0, regenerable-folder inheritance, per-call counters.
  • Budget expiry: each batch uses Promise.allSettled, so when the budget runs out mid-batch every already-started lstat/readdir settles before teamSizes returns Start teamSizes time budget before Session discovery #1950's over-budget result ({ sessions }, sizes omitted); no new work starts after expiry.
  • app/test/import.test.mjs: deep-equal against a frozen copy of the old serial walk (nested dirs, broken and self-referencing symlinks, unreadable file/dir, empty and missing roots, concurrent calls), plus a regression test that expires the budget with stats in flight and asserts the call waits for them. Swapping in a naive Promise.all makes it fail: must wait for in-flight stats after another task expires.

Benchmark — full teamSizes call, origin/dev module vs this branch, workspace bot/workspace/node_modules/pkgN/fileM.js (20 × 1 KB files per package), budget 120 s so nothing truncates; 3 warmups, median of 21 alternating runs; outputs deep-equal on every run. Node v24.22.3, macOS 26.6.2, Apple M4 Pro (VM), warm cache.

Files Before After Speedup
1,000 25.93 ms 6.49 ms 3.99×
6,000 158.47 ms 43.70 ms 3.63×
30,000 816.41 ms 231.24 ms 3.53×

At the default 2 s budget this also means large workspaces get a size instead of timing out.

Try it: cd app && node --test test/import.test.mjs

Untested: Windows and network drives (paths use join; the self-link fixture uses a junction on Windows, but it was only run on macOS). Cold-cache disks not benchmarked.

Changelog

Importing a team from DSH measures large workspaces faster.

How I checked it / 怎么验证的

  • cd app && npm test — 532 pass, 0 fail, 3 skipped; node --test test/import.test.mjs — 33/33 pass
  • node scripts/build-client.mjs --check — n/a, no ds-bot/ change
  • npm run lint — n/a, app/ has no lint script
  • Smoke test — n/a, no index.js / src/host/ change
  • Screenshot — n/a, no interface change

Checklist / 检查

  • One issue, one subject. / 只做一件事。
  • src/client/ changed → n/a
  • No keys, tokens or personal data, also in screenshots. / 没有密钥、token 或个人信息,截图里也没有。
  • No AI tool as author or Co-authored-by. / 没有把 AI 工具写成作者或 Co-authored-by。

Software Factory opened this PR for Lulu
Implemented in session Parallelize import size walk in teamSizes
Review with Devin · Open in Devin Desktop


Avoid serial filesystem round trips for large workspaces while retaining link sizes and regenerable byte totals. Settle every started operation before reporting budget expiry so no disk work outlives the call.
@devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access that start with 'Devin'.

  • Disable automatic comment, CI, and merge conflict monitoring

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-21T09:46:47.116003Z 6539ca9 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@Fei2411 Fei2411 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Reaper review — verdict: pass (head 6539ca9)

  • Walk is equivalent: same lstat (links not followed), unreadable entries count as 0, REGENERABLE inherited per entry, tallies only summed, so batch order does not matter. Only OVER can reject a batch, since lstat/readdir errors are caught.
  • Budget: every task checks the clock before lstat and before readdir, allSettled drains the batch, and the post-batch check stops new work. That keeps #1950's { sessions } over-budget result (#1950 is already in dev; this PR's diff against dev is only import.mjs and its test).
  • node --test test/import.test.mjs in app/: 33 pass, 0 fail, including the serial-equivalence and in-flight-drain tests.
  • Reproduced the speedup: dev vs this branch, 6,000 × 1 KB files under bot/workspace/node_modules, median of 11 runs: 288 ms → 74 ms (3.9×), output identical.
  • Non-blocking: the frozen serialTeamSizes in the test copies REGENERABLE/LEFT_BEHIND by hand, so a future change to those sets has to update the test too. Windows is untested locally; the Windows package CI job was still pending when I reviewed.
  • Overlaps: no other open PR touches app/src/import.mjs or its test. PR body has the Agent: Devin (Cognition) line.

Written by Devin

@Fei2411
Fei2411 merged commit 10f658a into dev Nov 21, 2026
15 checks passed
@Fei2411
Fei2411 deleted the chore/import-size-parallel-walk branch November 21, 2026 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant