perf(presets): keep large preset libraries light - #606
Merged
Merged
Conversation
Importing a ZIP or a library found on the machine means thousands of presets, and the store could not carry them. Measured with the corpus (10,347 MilkDrop presets, 116 MB) in an isolated copy: presets:list took 3.8 s, a single save held the main process for about 2.4 s, the panel used 631 MB, and every web client received 116 MB on every change. The store now reads the files once, in the background at start-up, and keeps them; saves and deletes update the cache, and list() only compares the folder's file names, so files added or removed by hand still show up. Bulk saves are written in batches with pauses, report progress and keep the pack's own order. A save or delete sends only what changed (presets-delta); the full list is sent only when a page opens. Pages apply changes in the store's order, newest first with ties broken by id, so presets saved in the same millisecond no longer land in a machine-dependent order. Web clients get MilkDrop presets without their sources. The overlay engine fetches a source by id (/milkdrop/preset, behind the token) only when it is about to draw that preset, keeps the current one on screen until it arrives, and drops the pick if it cannot be fetched. Found on the way: every preset change rebuilt the visualizer window's whole layer stack, so importing or deleting a MilkDrop preset restarted the MilkDrop picture. The stack is now rebuilt only when a Studio preset changes. The MilkDrop panel no longer keeps a second copy of the list. Same corpus after the change: list 0.3 s, a save 29 ms (91 ms with a window open), panel 273 MB. With 2,000 presets and a web overlay in a real browser, the connect message went from about 30 MB to 271 KB. Refs #574, #560
The background load builds its cache from a list of names taken when it starts. A preset deleted after the load had read its file came back in the cache: list() dropped it again on its next folder check, but get() did not, so the web overlay could still be served a deleted preset's source. Deletes made while the load runs are now removed from what it installs, and get() falls back to the file for a preset saved during the load. The race is tested deterministically, with the load's reads held until the delete has happened. The engine also says so once in the console if a page ever receives a sourceless preset with no bridge to fetch the source; today only the web overlay gets sourceless presets. Refs #574, #560
This was referenced Sep 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 1 of 2 for #574. Importing a ZIP or a library found on the machine means thousands of presets, and the store could not carry them. Measured in an isolated copy with the corpus (10,347 MilkDrop presets, 116 MB):
presets:listtook 3.8 s;The user chose to scale the store before adding the importer (part 2).
Store (
src/main/presets-store.js)list()compares only the folder's file names, so a file added or removed by hand still shows up. A file edited by hand shows up after a restart.Change broadcast. A save or delete sends only what changed (
presets-delta); the full list is sent only when a page opens. Pages apply it in the store's order: newest first, ties broken by id, so presets saved in the same millisecond no longer land in a machine-dependent order.Web clients get MilkDrop presets without their sources.
/milkdrop/preset, behind the token) only when it is about to draw that preset.Found on the way. Every preset change rebuilt the visualizer window's whole layer stack, so importing or deleting a MilkDrop preset restarted the MilkDrop picture on screen. The stack is now rebuilt only when a Studio preset changes. The MilkDrop panel no longer keeps a second copy of the list.
Measured after the change, same corpus:
presets:list0.3 s, a save 29 ms (91 ms with a window open), the panel 273 MB. With 2,000 presets and a web overlay open in a real (headless) browser:Not done here. Visualizer windows still hold every source (141 MB with 10,347 presets), since they draw from them directly. The importer itself is part 2.
Tests. 2052 unit tests pass, 20 of them new. They cover:
26 of 26 mutations are caught. A review found that a delete made while the background load runs could come back in the cache (
get()would still find it); that is fixed and tested deterministically.npm run smokepasses, and the user's settings and 635 preset files are unchanged.Refs #574, #560