feat: multi-invocation incremental builds — --sync-key, changes/verify subcommands, snapshot hardening

[ww2283]

Motivation

Callers that split large index builds into checkpointed multi-leg invocations (e.g. automation driving leann build with varying --docs lists) hit two limitations:

1. Snapshot identity is a hash of the entire --docs list — any variation between invocations creates a fresh snapshot namespace, losing all Merkle incrementality: unchanged files re-embed and duplicate.
2. No way to inspect pending changes or index integrity — a build interrupted mid-write (SIGKILL) can leave cross-artifact inconsistency that a subsequent build completes with exit 0.

Changes (all opt-in / additive; no default behavior changes)

• leann build --sync-key <key> — caller-supplied stable snapshot identity. Two builds with different --docs lists but the same key share one snapshot; files common to both are embedded once. Mismatched keys are rejected unless --force rekeys.
• leann changes <index> [--docs ...] [--sync-key K] — non-mutating Merkle diff; prints one JSON doc {added, modified, removed} (sorted) on stdout, exit 0. Errors (exit 1) on missing index, empty sync scope, wrong sync key, or corrupt snapshot rather than reporting a false clean delta.
• leann verify <index> — cross-artifact integrity check: passages.jsonl vs offsets pickle, IVF id-map exact inversion, FAISS ntotal vs id-map cardinality, mapped ids present in the inverted lists, next_id monotonicity, sync snapshots unpickle. Exit 0 healthy / 1 with findings.
• Zero-chunk deltas commit the snapshot — a delta producing no chunks previously skipped the snapshot write, so the same no-op delta was rediscovered forever. Guarded: if any document failed to load, the build aborts without committing so failures stay visible.
• LEANN_NO_REGISTER=1 — skip global ~/.leann/projects.json registration, for automation running builds in disposable directories.
• Snapshot robustness: corrupt snapshots and unreadable sync_roots.json fail loud on build (--force resets); a transiently unreadable file keeps its previous hash instead of being classified as removed (which deleted its chunks); unreadable subtrees abort change detection instead of reading as mass removal; leann watch skips a tick on these errors instead of dying.

Testing

40+ new tests across tests/test_sync_key.py, test_cli_changes.py, test_cli_verify.py, test_zero_chunk_commit.py, test_no_register.py, plus additions to test_sync.py/test_watch_sync_scope.py. The verify fixtures build a real IndexIVFFlat + DirectMap.Hashtable. The changes went through three review rounds (two independent reviewer passes + a re-review) on our fork before this PR; 17 findings fixed.

Rebased onto current main (dc85934); import leann + the touched suites pass and ruff is clean on this branch.

[ww2283]

Note on the Linux build failures: they are unrelated to this PR's code — current auditwheel (installed unpinned in build-reusable.yml) raised its floor to patchelf >= 0.14.5, while ubuntu-22.04's apt ships 0.14.3, so every auditwheel repair step fails with ValueError: patchelf 0.14.3 found. This will affect all PRs and release builds, not just this one.

I've folded a one-line fix into this branch (ec83163): pip-install patchelf>=0.14.5 into the same build env as auditwheel, so the repair step resolves the newer binary from .uv-build/bin. Happy to split it into a separate PR if you'd prefer to keep this one feature-only.

(The earlier macos failure was a real bug in this PR's verify tests — fixed in 1c2f04b: with faiss-cpu and the hnsw-bundled faiss both loaded, SWIG type-table clashes made downcast_index(...).invlists return None, silently skipping the mapped-id check; it now runs in a single-faiss subprocess.)

[ww2283]

Appended 076ea38 — fix for a verify false-positive found in live use after this series: with --id-scheme=content-hash, byte-identical chunks legitimately share a passage id, so verify's bijection invariants flagged healthy indexes. Invariants are now checked against the actual many-to-one contract (duplicate jsonl ids allowed only with identical text; idx cardinality vs unique ids; passage_to_id validated as a last-wins partial inverse). Includes a duplicate-content test fixture; 18 verify tests pass. Details: ww2283#6.

[ASuresh0524]

@ww2283 Can we fix CI?

[ww2283]

@ASuresh0524 of course, the ci is too comprehensive:)

[andylizf]

I read the whole diff. The production surface is small (506 of the 2,181 added lines, in cli.py, sync.py and registry.py; the rest is tests) and the comment on each guard saying what it defends against made it easy to follow. Four things before I approve.

1. test_env_unset_registers_project fails on Windows, and it is new in this PR. All four windows-2022 jobs (3.11 through 3.14) report 1 failed, 389 passed, 71 skipped on it; macOS and Linux are green, so nothing else in the suite is unhappy. The failure is AssertionError: assert False, where the false term is WindowsPath('C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/test_env_unset_registers_proje0/home/.leann/projects.json').exists().

The fake_home fixture sets only HOME, but register_project_directory resolves the registry with Path.home(), and on Windows that reads USERPROFILE and ignores HOME. So the call writes into the runner's real home and the assertion looks in the temp dir. Setting USERPROFILE alongside HOME in the fixture fixes it.

Same fixture, second effect worth fixing at the same time: test_no_register_env_set_skips_registration passes vacuously on Windows, because it asserts the absence of a path that nothing was going to write to either way. As it stands it cannot catch a regression in LEANN_NO_REGISTER there.

2. The --sync-key contract in the description is not the one in the code. The description says two builds with different --docs lists and the same key share one snapshot, with the files common to both embedded once. --help says the snapshot is "independent of the exact --docs invocation". Both read as: pass whatever subset this leg covers, the key holds on to the rest.

What the code does: _create_synchronizers builds the keyed manifest from the current invocation's --docs alone, then hands it to FileSynchronizer(explicit_files=...) against the shared keyed snapshot. detect_changes classifies everything in the snapshot but absent from that manifest as removed, and the build deletes those chunks. test_directory_dropped_from_docs_triggers_rebuild_with_remaining_files_only asserts exactly that, so the behaviour is deliberate.

The real contract is then "every invocation passes the cumulative list", and a caller who reads the motivation and passes leg-specific lists silently loses the earlier legs. Could you spell that out in --help and the changelog, and reword the sentence about differing --docs lists? test_same_key_different_docs_lists_share_snapshot_and_load_only_new_files covers {a,b} then {a,b,c}, which stays inside the contract, so no test currently contradicts the docs.

3. The _load_errors guard only covers the incremental branch. The check sits right after load_documents on the incremental path. Two routes get past it: a first build or --force, which never enters that path; and the rebuild fallback, whose else branch calls load_documents a second time, resetting _load_errors to 0 before the full rebuild. Both end at fs.create_snapshot() with no check in between, so a file that failed to load is hashed into the snapshot as if indexed, and the next incremental run sees no change and never retries it. That is the state the guard exists to prevent. test_loader_failure_does_not_advance_snapshot, test_swallowed_loader_failure_blocks_snapshot_commit and test_partial_loader_failure_aborts_before_mutating_index all build once and then add files, so all three sit on the incremental branch.

4. Dropping self.indexes_dir.mkdir(...) from LeannCLI.init is a default behaviour change. Minor, and I think an improvement, since leann no longer creates .leann/indexes/ in whatever directory you happen to run any subcommand from. But the description says there are no default behaviour changes and the changelog does not mention it, while test_cli_construction_does_not_create_indexes_dir_in_cwd shows it is deliberate. It deserves a changelog line. I checked the callers: build_index does its own mkdir(parents=True, exist_ok=True) and the read paths guard with .exists(), so nothing breaks.

The patchelf>=0.14.5 pin that came along is unrelated to this PR but correct, and Linux is green again.

One process note: the branch conflicts with main now. cli.py has moved twice since you last pushed (#395 on 08-21, #269 on 09-05), so this needs a rebase and a fresh CI run before it can go in.