[Packi1992]

Problem

Pass 2 of mergeExactSameType (src/pipeline/normalize.ts) compares every pair of keepers. Each comparison rebuilds both nodes' slug sets with slugifyLabel (NFKC plus two Unicode regex replaces). Each merge also rescans idRewrite. So normalize() does O(n²) regex work, and a sync stops being practical well before 50k nodes.

Fix

The semantics stay those of the pairwise scan. For each keeper a in order, it is compared with every later keeper b that has not been dropped. a includes the aliases it absorbed earlier in the same scan. The change is how candidates are found:

• Memoized slug sets. Each keeper's slug set is computed once. Only the current keeper can change while the scan runs, and its set is extended with the aliases it absorbs. If its label changes, the set is recomputed.
• Indexes instead of all pairs.
  • slug → keeper indices covers exact same-type merges and the cross-type cross_type_label_clash review.
  • The alias_suggestion pass uses plural/singular lookups in the same index plus a type + acronym → keeper indices index.
  • Candidates go into a min-heap and are visited in ascending keeper index. So merges, keepers (first seen wins) and review items come out in the same order as before.
  • After a merge, only later keepers that match the newly absorbed slugs are added.
  • Dropped keepers are pruned from index lists as the lists are scanned, and each list is pushed at most once per keeper. This keeps common slugs from being rescanned.
• Chained rewrites. They are resolved once at the end, with find plus path compression, instead of rescanning idRewrite on every merge.
• Hub keepers stay cheap. A keeper that absorbs many nodes appends to a cached alias set instead of rebuilding its alias list on every merge. The match predicates walk the smaller slug set. suggestsAliasMatch checks plural/singular/acronym forms by set lookup instead of comparing every slug pair. It is the same symmetric relation with the same length ≥ 2 rule.

mergeExactSameType is now exported from pipeline/normalize for tests (not from the package index). It also returns a candidatePairs work counter.

Output equivalence

tests/normalize-merge-scale.test.ts contains a verbatim copy of the old pairwise pass. It runs both versions on seeded synthetic corpora and asserts deep equality of:

• kept nodes
• idRewrite entries, in insertion order
• review items, in order
• progress messages

Corpora:

• 3000 nodes, vocabulary 400
• 2000 nodes, vocabulary 60, with Unicode (ß, İ, fi)
• 1500 nodes, vocabulary 15

The data has colliding labels within and across types, reused and explicit ids, acronym and plural aliases, and empty and punctuation-only labels. The test asserts that each corpus actually produces merges and all three review reasons.

Checks done outside the committed test:

• Full normalize() output (nodes, triples, review items, progress) from main's build and this branch is identical on 60 seeded corpora (200 to 2,400 nodes), and at 5k, 10k and 20k nodes.
• Two deliberate breakages of the new code are caught by the test: not queueing candidates for absorbed aliases, and skipping the rewrite chaining.

A second test bounds the work at 4k and 16k nodes:

• slugifyLabel calls ≤ 4·(nodes + aliases)
• visited candidate pairs ≤ 4·(input + output) and < 1 % of all keeper pairs

There is no wall-clock timing in the tests.

Numbers

Full normalize() on seeded synthetic nodes (with a triple per node), Node 24, one core. Two generator settings:

nodes	main	this branch
5,000	13.9 s / 8.0 s	70 ms / 46 ms
10,000	53.8 s / 32.9 s	98 ms / 92 ms
20,000	212.3 s / 130.1 s	185 ms / 205 ms

With this branch only, the first setting takes 0.42 s at 40k nodes, 0.95 s at 80k and 2.0 s at 160k.

Not changed

Progress output in non-TTY / --compact mode is unchanged. Plain progress lines are already available with GSD_GRAPH_PROGRESS=1. Showing them by default would change CLI output for piped use, so it seemed out of scope here. The pass is now short enough that this matters much less.

npm run build and npm test (435/435) pass on Node 22 and 24; npm run test:coverage passes on Node 24.

Fixes #4

🤖 Generated with Claude Code

[Packi1992]

Real-world check with this branch (plus #7) installed locally, on the same corpus as in #4: ~2.7k Markdown sources (1,639 re-extracted), resulting in ~47.5k nodes and ~7.5k triples. The incremental sync now finishes in about 2 s; before, it had not finished after 47 min.