Skip to content

feat(install): create node_modules for workspace members#34970

Open
bartlomieju wants to merge 7 commits into
mainfrom
feat/workspace-member-node-modules
Open

feat(install): create node_modules for workspace members#34970
bartlomieju wants to merge 7 commits into
mainfrom
feat/workspace-member-node-modules

Conversation

@bartlomieju

@bartlomieju bartlomieju commented Jun 6, 2026

Copy link
Copy Markdown
Member

When installing a workspace, deno install previously created only a
single node_modules directory at the workspace root. Native Node.js
tooling run from within a workspace member (for example svelte-check,
astro, or eslint plugins) expects a local node_modules containing
the member's dependencies, the way npm and pnpm lay workspaces out, and
fails without it. Sharing only the root node_modules also hides missing
dependencies, since a member can resolve a package that only a sibling
declares.

This makes both linker modes create a node_modules directory inside
each workspace member and symlink that member's direct dependencies into
it: npm dependencies point at their resolved location (the shared
node_modules/.deno store for the isolated linker, or the package's
top-level or nested directory for the hoisted linker) and sibling
workspace members are linked to their directories. The workspace root is
left untouched and members without dependencies get no directory.

Stale links are pruned on reinstall the same way the root node_modules
prunes them: when a member drops a dependency (or a sibling member is
removed), its link is removed from the member's node_modules before
re-linking so the dropped dependency stops being resolvable. Only
symlinks and junctions are touched, so real files a user placed in the
member's node_modules are left alone.

Known limitations (left for follow-ups):

  • A member's node_modules/.bin is not populated yet, so tools invoked
    as node_modules/.bin/<tool> from within a member still resolve via
    the root .bin rather than a local one.
  • In the hoisted linker, a direct dependency whose resolved version is
    not placed anywhere in the hoisted layout (a version-conflict corner
    case) is silently skipped rather than linked.

Closes #26743
Closes #27550

When installing a workspace, `deno install` previously created only a
single `node_modules` directory at the workspace root. Native Node.js
tooling run from within a workspace member (for example `svelte-check`,
`astro`, or `eslint` plugins) expects a local `node_modules` containing
the member's dependencies, and npm/pnpm provide exactly that. The shared
root `node_modules` also hides missing dependencies, since a member can
resolve a package that only a sibling declares (shadow dependencies).

This makes the local (isolated) installer create a `node_modules`
directory inside each workspace member and symlink that member's direct
dependencies into it: npm dependencies point into the shared
`node_modules/.deno` store, and sibling workspace members are symlinked
to their directories. The workspace root is left untouched and members
without dependencies get no directory.

Closes #26743
Closes #27550
Extend the previous change to the hoisted linker mode. Each workspace
member gets a `node_modules` directory with its direct dependencies
symlinked to their actual locations in the hoisted layout (top-level
packages, or nested ones in case of version conflicts) and sibling
workspace members linked to their directories.

The spec test now exercises both the isolated and hoisted linkers.
The per-member node_modules linking skipped the workspace root by
comparing member_node_modules to root_node_modules_path. That comparison
is unreliable: root_node_modules_path is canonicalized while target_dir
is not, so on Windows they differ by 8.3 short names or casing for the
same directory. The root member was then re-processed, and in hoisted
mode its top-level dependency got symlinked onto itself, producing a
self-referential link (FilesystemLoop, os error 1921) that broke the
hoisted linker.

Mark the root member when building the workspace package list by
comparing config folder urls, which are representation-stable, and skip
it in both linker modes.
@bartlomieju bartlomieju added this to the 2.9.0 milestone Jun 7, 2026
@bartlomieju

Copy link
Copy Markdown
Member Author

Review

Overview

Both linker modes now create a node_modules inside each non-root workspace member and symlink the member's direct deps into it (npm deps point at their resolved store location, sibling members link to their source dirs), matching the npm/pnpm workspace layout. Closes #26743 and #27550.

Correctness (verified)

  • hoisted_package_path matches the real on-disk placement: top-level resolves to join_package_name(root_node_modules, name) (same as hoisted.rs:445) and nested to root_node_modules.join(parent_path).join("node_modules")/name (same as hoisted.rs:484-489), so the symlinks point at directories that actually exist.
  • Using the is_root flag (folder_url == root_dir_url) instead of path-comparing the canonicalized root_node_modules_path against the non-canonicalized target_dir is the right approach. Good catch on the Windows 8.3/casing rationale.
  • deps is direct-only, so members get the intended flat member layout rather than a full hoisted tree.
  • local.rs cache usage matches the root pattern (with_dep + insert returning true on new/changed, gated before created_dir), and symlink_package_dir is idempotent so re-runs are safe.

Issues / Risks

  • No pruning of stale member symlinks (medium). The root linker prunes removed deps via setup_cache.remove_dep plus a previous/current diff (local.rs:359). The new per-member block only inserts/creates, never removes. If a member drops a dependency from package.json (or a member is deleted), the stale symlink lingers in member/node_modules across installs, which re-creates exactly the "shadow dependency stays resolvable" problem this PR is meant to eliminate. Worth wiring member dirs into the same stale-removal diff, or clearing managed links before re-linking.
  • No node_modules/.bin for members (minor). npm/pnpm populate <member>/node_modules/.bin with each dep's bin entries. The motivation names eslint/svelte-check/astro, which are commonly invoked as node_modules/.bin/<tool> from within the member, so those still will not be found locally. Probably fine for now, but worth noting as a known limitation in the PR body.
  • hoisted_package_path returns None silently on conflicting versions (minor). The nested branch returns the first match and the caller continues on None. Every resolved version should be placed somewhere, so this is low risk, but in a version-conflict corner case a member's direct dep could silently not get linked. This nested branch is also the only path with no test coverage.

Test coverage

Covers both linker modes, a member's own npm dep (#26743), and a sibling workspace link including verifying the link target's package.json name (#27550). Gaps: no coverage for the nested/conflicting-version branch of hoisted_package_path, devDependencies, or re-install after removing a dep (which would catch the stale-symlink issue above).

Style

Comments are excellent and explain the why (npm/pnpm parity, shadow deps, the Windows path-comparison rationale) rather than restating the code. The member-linking loop is duplicated across both linkers with mode-specific target resolution; given how divergent the target computation is (store path vs hoisted path, cache vs no cache) the duplication is defensible, but a shared helper taking a resolve-target closure would reduce drift risk later.

Verdict

Solid, well-motivated feature with correct path computation and reasonable tests. The main thing I would want addressed before merge is stale member-symlink pruning, since without it removed deps stay resolvable from members and partially undermine the goal. The missing .bin and the untested nested-conflict path are worth at least documenting.

When a workspace member dropped a dependency (or a sibling member was
removed), its symlink lingered in the member's node_modules across
installs, leaving the dropped dependency resolvable from the member and
recreating the shadow-dependency problem the per-member layout is meant
to eliminate. Both linkers now prune links whose alias is no longer one
of the member's current direct dependencies before re-linking, mirroring
the root node_modules pruning. Only symlinks and junctions are removed,
so real files a user placed there are left untouched.

Also covers a member's devDependencies and re-install-after-removal in
the spec test.
@bartlomieju

Copy link
Copy Markdown
Member Author

A couple of optional test-coverage follow-ups worth noting (neither blocking):

1. Assert the workspace root node_modules is left untouched

The whole per-member path hinges on the is_root skip. If that flag ever
computes wrong (e.g. root_dir_url() vs folder_url normalization drift, a
trailing-slash mismatch, or a future config_folders() refactor), the loop
would run remove_stale_member_symlinks against the root's fully-populated
node_modules. That wouldn't crash — it would prune the root down to the root
package's direct deps and silently delete legitimately-resolvable transitive
packages. Nothing in the current suite would catch that.

Cheapest guard: in the existing fixture, assert a known transitive package is
still present at the root after the reinstall-that-prunes step.

2. Hoisted version-conflict "silently skipped" case

hoisted_package_path returns None when a member's resolved dependency
version isn't placed anywhere in the hoisted layout, and the link is skipped
with no error. Deferring the fix is fine, but two cheap things now:

  • Replace the silent continue at the None branch with a debug log (or a
    one-line warning), so "my member resolves the wrong version" is diagnosable
    instead of looking like the feature silently didn't run.
  • Add a regression test that pins the current behavior (link absent / wrong
    version) with a // known limitation note, so the eventual fix has an
    existing test that flips to "link present / correct version" rather than the
    corner case being entirely uncovered. The fixture needs a genuine
    cross-member version conflict, which is the annoying part — and exactly why
    deferring the fix is reasonable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create symbolic links to workspaces in node_modules directory for better Node.js compatibility NODE_MODULES not created in sub projects

1 participant