From cd03b41c415128dbc798d624b5ad034a01dd3a6a Mon Sep 17 00:00:00 2001 From: InstaZDLL Date: Sat, 24 Oct 2026 20:16:33 +0200 Subject: [PATCH 1/3] docs: RFC-018 proposes an incremental scan that reads only what changed A file whose size, modification time and sidecar signature are unchanged is skipped before it is hashed or its tags read; results stream to the database in path order instead of being collected whole; one unreadable folder is logged and counted, and the tracks under it are left as they are rather than marked missing. Raised by the 2026-10-24 audit against Navidrome. Signed-off-by: InstaZDLL --- docs/rfcs/RFC-018-incremental-scan.md | 130 ++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 docs/rfcs/RFC-018-incremental-scan.md diff --git a/docs/rfcs/RFC-018-incremental-scan.md b/docs/rfcs/RFC-018-incremental-scan.md new file mode 100644 index 00000000..7f3604b1 --- /dev/null +++ b/docs/rfcs/RFC-018-incremental-scan.md @@ -0,0 +1,130 @@ +# RFC-018 — An incremental scan that reads only what changed, and survives one bad folder + +- **Status**: Proposed +- **Implemented by**: nothing yet. The *Status* field above never flips in this + project; this line is the one that says what is running. +- **Date**: 2026-10-24 +- **Authors**: the WaveFlow project +- **Depends on**: [RFC-002](RFC-002-waveflow-server-v2.md) (the scanner and + track identity) + +## Problem + +The audits of 2026-10-24, run against Navidrome, found three problems in +`src/scanner.rs`. Each was confirmed in the code. + +1. **Every scan reads every byte of the library.** `extract_file` computes the + quick hash, then the full BLAKE3 hash of the whole file, then reads its tags + and extracts its cover. All of that happens *before* the scan compares size + and modification time to decide that the file is unchanged and can be + skipped. An "incremental" scan therefore costs as much I/O as a full one. + For 2 TB of FLAC on a disk reading 150 MB/s, that is about four hours of + reading, every six hours by default (`WAVEFLOW_SCAN_INTERVAL_SECS`). + Navidrome skips unchanged folders, then compares file modification times, + before it reads anything. +2. **Every result is held in memory before anything is written.** The + extractions are collected into one `Vec` + (`.buffer_unordered(..).collect()`), sorted, and only then applied in + batches of 25. Memory grows with the library, and a scan that dies late + loses all of its work. +3. **One unreadable folder fails the whole scan.** `discover_audio` walks the + tree and propagates the first error (`let entry = entry?;`). A single + folder the server cannot read, such as a Synology `@eaDir` with the wrong + owner or a mount that has gone away, stops discovery for the entire + library. + +## Decision 1 — look before reading + +For each discovered file, the scan first asks the database what it already +knows (size, modification time, availability), and the file system for the +same two values plus the sidecars' signature (below). This costs one `stat` +per file and one per possible sidecar, and no reads. + +A file is **unchanged**, and skipped without being opened, when all of these +hold: + +- its size and modification time equal the stored ones; +- the track is available; +- its sidecar signature equals the stored one; +- this is not a full scan (`database rescan --full`, or the identity-spec + change that forces one); +- the track already has the identity hint the skip path writes today, so a row + from before that column still gets its one extraction. + +**Anything else reads the file as today**: the quick and full hashes, the tags, +the cover and the lyrics. The match by path, then by content hash, then by +relocation hint, is unchanged. + +**The sidecar signature.** The scan already reads lyrics from `.lrc` and +`.txt` beside the audio file. A sidecar added, edited or removed beside +an unchanged audio file must still be picked up. So a new nullable column, +`track.sidecar_signature`, records the size and modification time of each of +the two possible sidecars, absent ones included. It is computed from `stat` +alone. `NULL` (every row, right after the migration) never matches, so the +first scan after an upgrade reads every file once and fills the column, and the +scans after it skip. + +**What the trade costs.** A file rewritten in place with its size and +modification time both preserved is missed until the next full scan. Some tag +editors preserve the modification time; none can preserve both while changing +the tags, except by padding to the same size. This is the trade Navidrome and +every incremental scanner make. The full scan stays available for anyone who +suspects it, and `docs/operations.md` says so. + +## Decision 2 — stream, in path order + +Extraction keeps its parallelism, but results are applied **as they arrive**, in +batches of 25. They are still applied in path order, so the outcome stays +deterministic: paths are sorted once, after discovery, and extracted through an +ordered buffer (`buffered`, not `buffer_unordered`). Memory therefore holds at +most a few batches, not the library. Each batch commits on its own, so a scan +that dies keeps what it wrote, and the next run resumes on a library mostly +already up to date. + +Progress (`processed`, `skipped`, …) moves as batches commit, which is also what +the scan's progress stream shows. + +## Decision 3 — one bad folder costs that folder + +`discover_audio` no longer stops at the first error. An entry it cannot read is +**logged** with its path relative to the library root, never the absolute +path. It is **counted** in the scan's `errors`, and the scan **goes on**. + +**What sat under an unreadable folder is not marked missing.** The tracks +whose path lies under a directory that could not be read are left exactly as +they are. They are neither seen nor marked unavailable, so a folder that is +briefly unreadable (a mount coming back, a permission being fixed) does not +make its albums vanish and come back. Only the library root failing to open +still fails the scan, since then nothing can be told apart. + +## What does not change + +- Track identity, relocation and the derived album and artist ids. +- The full scan, the boot and scheduled scans, and their triggers. +- Symbolic links are still not followed (see below). + +## The order of the work + +1. **This RFC.** +2. **One PR**: + - the pre-check and its sidecar signature, with their migration; + - ordered streaming of the results; + - tolerant discovery; + - tests for each rule: a skipped file is never opened, a sidecar change is + caught, a file under an unreadable folder is not marked missing, memory + stays bounded, the scan resumes after a failure; + - `docs/operations.md`. + +## What is still open + +- **Folder-level skipping**, as Navidrome does: a directory whose modification + time has not moved need not even be listed. It is a further gain on large + libraries, but directory modification times are not reliable on every file + system and network share, so it waits for a measurement. +- **Watching the file system** (`notify`), so that a change is seen in seconds + rather than at the next interval. The periodic scan would stay as the safety + net. +- **Ignore rules**: `@eaDir`, `.Trash-*`, `$RECYCLE.BIN`, hidden folders, and + an `.ndignore`-style file. +- **Following symbolic links**, confined to the library root after + canonicalisation, for libraries built as link farms (beets). From aada69dbacd467d9b843875db69c0be3bb134151 Mon Sep 17 00:00:00 2001 From: InstaZDLL Date: Sat, 24 Oct 2026 20:16:42 +0200 Subject: [PATCH 2/3] docs: RFC-018 names #322 in its Implemented by line Signed-off-by: InstaZDLL --- docs/rfcs/RFC-018-incremental-scan.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/rfcs/RFC-018-incremental-scan.md b/docs/rfcs/RFC-018-incremental-scan.md index 7f3604b1..4e9e233c 100644 --- a/docs/rfcs/RFC-018-incremental-scan.md +++ b/docs/rfcs/RFC-018-incremental-scan.md @@ -1,7 +1,8 @@ # RFC-018 — An incremental scan that reads only what changed, and survives one bad folder - **Status**: Proposed -- **Implemented by**: nothing yet. The *Status* field above never flips in this +- **Implemented by**: in progress: [#322](https://github.com/InstaZDLL/waveflow-server/pull/322) + (this RFC). The *Status* field above never flips in this project; this line is the one that says what is running. - **Date**: 2026-10-24 - **Authors**: the WaveFlow project From f85b744c504fb3406c3b9ef563af29aba08b95c4 Mon Sep 17 00:00:00 2001 From: InstaZDLL Date: Sat, 24 Oct 2026 20:27:40 +0200 Subject: [PATCH 3/3] docs: RFC-018 encodes an absent sidecar, so NULL means only never computed Signed-off-by: InstaZDLL --- docs/rfcs/RFC-018-incremental-scan.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/docs/rfcs/RFC-018-incremental-scan.md b/docs/rfcs/RFC-018-incremental-scan.md index 4e9e233c..eaad18dc 100644 --- a/docs/rfcs/RFC-018-incremental-scan.md +++ b/docs/rfcs/RFC-018-incremental-scan.md @@ -61,9 +61,19 @@ relocation hint, is unchanged. an unchanged audio file must still be picked up. So a new nullable column, `track.sidecar_signature`, records the size and modification time of each of the two possible sidecars, absent ones included. It is computed from `stat` -alone. `NULL` (every row, right after the migration) never matches, so the -first scan after an upgrade reads every file once and fills the column, and the -scans after it skip. +alone, and it is a string that is never `NULL` once written: + +```text +lrc::|txt:: +``` + +An absent sidecar writes `-` in place of its size and modification time: +`lrc:-|txt:-` is a file with no sidecar at all. Absence is therefore a value +like any other, and adding, editing or deleting either sidecar changes the +string. `NULL` keeps one meaning only: never computed. Every row right after +the migration is `NULL`, and `NULL` never matches, so the first scan after an +upgrade reads every file once and fills the column; the scans after it skip, +sidecar or not. **What the trade costs.** A file rewritten in place with its size and modification time both preserved is missed until the next full scan. Some tag