MilkDrop: import whole libraries from ZIP packs, folders and the machine (#574, part 2)

[CaYatur]

Part 2 of 2 for #574: the importer. Part 1 (#606) made the store carry large libraries.

The MilkDrop panel gains 🗜 Import from ZIP Pack, 📁 Import from Folder and 🔎 Search This Computer. Every import has two steps. First, a scan shows the user what will be added: presets, textures, MB, and what will be skipped. Nothing is copied until the user confirms. The plan stays in the main process, and the page gets only a summary and a token.

What it does

• What is taken:

  • .milk files in nested folders.
  • Images in textures or sprites folders.
  • Images next to the presets, but only when a preset asks for them with sampler_<name>. A preview picture beside each preset is not a texture. The summary lists these images separately instead of counting them into the size.

• Skipped and reported:

  • .milk2 files (MilkDrop 3 compatibility: an Automatic / MilkDrop 2 / MilkDrop 3 mode #567).
  • Presets over 450 KB and textures over 8 MB.
  • Encrypted or unsupported ZIP entries.

  macOS __MACOSX and AppleDouble leftovers, hidden folders and node_modules are not looked at.

• Duplicates: a preset with the same name and the same content is skipped. One with the same name but different content is imported.

• ZIP reader: our own code.

  • The scan reads only the central directory.
  • Supports ZIP64 and stored and deflate entries.
  • Checks the CRC and caps the output of each entry.
  • Refuses a suspicious compression ratio and reads at most 200,000 entries.
  • Reads CP437 and UTF-8 names.
  • Uses only an entry's file name, so an entry can't write outside the destination (zip-slip).

• Textures: copied into the app's own folder (userData/milkdrop-textures).

  • That folder is searched after the texture folder the user chose, so when a name exists in both, the user's copy wins.
  • A texture already in the app's folder with different content is kept, and the clash is reported.
  • milkdropLibrary.textureRev makes every engine reload its texture list after an import: windows, Spout/Syphon, the preview and the web overlay.

• Tags: the first folder below the pack's common prefix becomes a tag (MilkDrop: favourites, tags and search by author #576), unless it has a generic name. The user can turn this off.

• Search This Computer:

  • Where: known install folders (Winamp, foobar2000, projectM), then the user's Downloads, Desktop, Music and Documents, using the paths the system reports (app.getPath; OneDrive can move them, and Linux localizes their names).
  • Candidates: every subfolder and ZIP in those places, plus loose presets sitting directly in one of them.
  • Finding is breadth first. A depth-first walk spent the whole budget in the first huge folder and never reached a small pack beside it; the first end-to-end run showed exactly that.
  • Budget: finding gets at most 60% of 8 seconds. Counting gets the rest, shared among the libraries found.
  • Unfinished counts: a library whose count did not finish is listed with "+" or "not counted". It is scanned in full before the confirmation, so the confirmation always shows real numbers and nothing is imported from a partial scan.
  • If finding ran out of time, the panel says so and points to Import from Folder.

Measured

• Isolated store, real packs:
  • The Cream of the Crop folder was scanned in 1.1 s (9,795 presets, 11 category tags) and imported in 6.2 s.
  • Importing it again took 1.2 s and skipped every preset as a duplicate.
  • The original pack's ZIP was scanned in 6 ms and imported in 287 ms.
• Isolated app copy, panel driven through CDP:
  • The search listed a test pack and the libraries in my own Downloads in 5–7 s, never over the budget.
  • Importing the test pack added 79 presets, tagged Dancer (74) and Fractal (5), and copied worms.jpg into the app's folder.
  • The visualizer window loaded worms.jpg with no texture folder chosen.
• Web overlay in headless Edge: it was opened before the import. After the import it asked for its texture list again, and it loaded worms.jpg from the server when a preset needing it was chosen.
• Unfinished count: an 80,000-preset folder that the search counted only to 10,013 was listed as "10,013+". Import scanned it again in 10.8 s, and the confirmation showed 80,000 presets and 80 folder tags. Cancelling imported nothing.
• English UI: checked the same way.
• Test gates:
  • npm test: 2081/2081.
  • npm run smoke: PASS, and the user-data checksums were unchanged.
  • The new tests also pass with process.platform set to Linux.

Not done

• .milk2 (MilkDrop 3 compatibility: an Automatic / MilkDrop 2 / MilkDrop 3 mode #567).
• RAR and 7z packs.
• A ZIP inside a ZIP, or inside a chosen folder, is not opened.
• The search does not look at other drives or anywhere outside the places above. Import from Folder reaches those.

Tests

29 new tests:

• 7 for the ZIP reader, using a hand-written ZIP writer that includes ZIP64.
• 22 for the importer:
  • real temporary folders and archives;
  • zip-slip, duplicates, textures and tags;
  • the search order and breadth-first finding, made deterministic with a file budget instead of time;
  • a library found but not counted, and the rescan;
  • the main-process handlers and the engine's texture counter;
  • the panel drawn with a fake DOM, including the confirmation and the rescan.

57 of 57 mutations are caught.

Refs #574, #560