cms-api: Replace lodash.uniqWith with custom content scope deduplication

[fraxachun]

Problem

UserPermissionsService deduplicates content scopes in several places using lodash.uniqWith with a deep-equal comparator. uniqWith compares every candidate against every element already kept, so it runs in O(n²) and does a full deep-equal per comparison. With large content-scope lists this becomes a serious bottleneck (the original observation was ~15 s for 2000 scopes when resolving users).

Solution

Content scopes are flat objects of primitive values, so deep equality isn't actually required — a stable string key built from the scope's sorted entries uniquely identifies a scope. This lets us deduplicate in a single O(n) pass with a Map, extracted into a pure, unit-tested util (deduplicate-by-content-scope.ts):

export function deduplicateByContentScope<T>(items: T[], getScope: (item: T) => ContentScope): T[] {
    const seen = new Map<string, T>();
    for (const item of items) {
        const key = JSON.stringify(Object.entries(getScope(item)).sort(([a], [b]) => a.localeCompare(b)));
        if (!seen.has(key)) {
            seen.set(key, item);
        }
    }
    return [...seen.values()];
}

All three uniqWith call sites in user-permissions.service.ts now use this helper. The lodash.uniqwith dependency (and its @types package) is removed. lodash.isequal stays — it's still used for scope membership checks elsewhere.

Like uniqWith, the helper keeps the first occurrence of each scope and preserves input order.

This PR extracts only the deduplication improvement from #5999. The per-request availableContentScopes cache from that PR is intentionally left out and can be handled separately.

Tests

deduplicate-by-content-scope.spec.ts pins the behavior that must be preserved from lodash.uniqWith(scopes, isEqual). Each case asserts both the exact expected result and parity against a local uniqWith(isEqual) mirror, covering:

• empty input, distinct scopes, and exact-duplicate removal (first occurrence kept, order preserved)
• key-order-insensitive equality ({domain, language} == {language, domain})
• keeping scopes that differ in a single value or key count
• distinguishing primitive value types (1 vs "1" vs true)
• deduplicating by the extracted scope while ignoring the surrounding item (e.g. differing label)
• not mutating the input

Note on scope of equivalence: the helper is equivalent to uniqWith(isEqual) for the actual shape of content scopes — flat objects of primitive values. It intentionally does not replicate isEqual's deep comparison of nested objects/arrays, since content scopes never contain those. This assumption is documented at the call site.

Speed comparison

Benchmark deduplicating a list where half the entries are duplicates, best of 3 runs. Both algorithms return the identical set of deduplicated scopes.

Total scopes	Unique scopes	lodash.uniqWith	deduplicateByContentScope	Speed-up
500	250	23.8 ms	0.57 ms	~42×
1000	500	91.9 ms	0.76 ms	~120×
2000	1000	375.1 ms	1.36 ms	~275×
4000	2000	1350.4 ms	2.76 ms	~490×

The uniqWith timings roughly quadruple as the input doubles (O(n²)), while the Map-based approach scales linearly — so the gap widens with list size.

Changeset

Included (@comet/cms-api patch).

[VPS-Obi]

Should we add unit tests for this?

[fraxachun]

be6c2c1

[Copilot]

Pull request overview

This PR improves performance in @comet/cms-api by replacing lodash.uniqWith-based deep-equality deduplication of content scopes (O(n²)) with a custom Map-based deduplication helper keyed by a stable scope representation (O(n)).

Changes:

• Replaced three lodash.uniqWith(..., isEqual) call sites in UserPermissionsService with a new deduplicateByContentScope(...) helper.
• Added a dedicated util (deduplicate-by-content-scope.ts) plus unit tests to pin behavior and parity with the prior implementation.
• Removed lodash.uniqwith and @types/lodash.uniqwith dependencies and updated the lockfile; included a patch changeset for @comet/cms-api.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pnpm-lock.yaml	Removes lodash.uniqwith and its types from the workspace lockfile.
packages/api/cms-api/src/user-permissions/user-permissions.service.ts	Switches deduplication of content scopes/content-scope-with-label lists to the new Map-based helper.
packages/api/cms-api/src/user-permissions/deduplicate-by-content-scope.ts	Introduces the O(n) deduplication utility used by UserPermissionsService.
packages/api/cms-api/src/user-permissions/deduplicate-by-content-scope.spec.ts	Adds Vitest coverage to preserve previous uniqWith(isEqual) behavior.
packages/api/cms-api/package.json	Drops lodash.uniqwith and @types/lodash.uniqwith from dependencies/devDependencies.
.changeset/replace-lodash-uniqwith-content-scope-dedupe.md	Adds a patch changeset describing the perf improvement and dependency removal.

Files not reviewed (1)

• pnpm-lock.yaml: Generated file

[nsams]

Adding comments that reference a previous implementation is odd and useless.

Suggested change

	// Replaces lodash's deep-equal uniqWith, which compares every candidate against every kept element (O(n^2)) and is too
	// slow for large scope lists. Content scopes are flat objects of primitive values, so their sorted entries form a stable
	// string key that stands in for deep equality, allowing deduplication in a single O(n) pass.
	// Like uniqWith, the first occurrence of each scope is kept and the input order is preserved.

[fraxachun]

Good point. Reworded in 70ac8db — dropped the references to the previous implementation and kept only the note on why the sorted-entries string key is a valid stand-in for equality.

---

Generated by Claude Code

[nsams]

Taking one step back: what is the use case where this de-duplication is needed?

[fraxachun]

Taking one step back: what is the use case where this de-duplication is needed?

For example if a user has content scopes by rule and additionally added content scopes manually. I don't think we can remove this mechanism.

[nsams]

For example if a user has content scopes by rule and additionally added content scopes manually. I don't think we can remove this mechanism.

does it hurt if there are duplicate scopes? If it is for presentation to the user, can we deduplicate on the client side?