Skip to content

fix(cli): copy chunk metadata before id assignment to break dict aliasing - #8

Merged
ww2283 merged 1 commit into
mainfrom
bugfix/issue-7-chunk-id-aliasing
Aug 19, 2026
Merged

ww2283 merged 1 commit into
mainfrom
bugfix/issue-7-chunk-id-aliasing

Conversation

@ww2283

@ww2283 ww2283 commented Aug 19, 2026

Copy link
Copy Markdown
Owner

Fixes #7.

Problem

Chunkers can return chunks for one file that alias the same metadata dict object. Both LeannCLI._assign_chunk_ids and LeannCLI._assign_unique_chunk_ids wrote each per-chunk id into that shared dict via setdefault, so the last chunk's id overwrote all previous ones. LeannBuilder.add_text resolves the passage id from metadata["id"], so every chunk of a multi-chunk file got the last chunk's id — corrupting passages.idx and the faiss id map, and failing leann verify with "id appears with different text" on every multi-file incremental build.

Fix

Copy the metadata dict at id-assignment time in both assigners: c["metadata"] = {**c.get("metadata", {}), "id": sid}.

Verification

  • New unit tests reproducing the shared-dict aliasing (tests/test_incremental_build.py::test_assign_unique_chunk_ids_with_shared_metadata_dict, ::test_assign_chunk_ids_with_shared_metadata_dict) — red on base, green with fix.
  • E2E: two-leg incremental IVF build (--sync-key, 2 files then +3 multi-chunk files, 59 chunks each) — leann verify exits 0, all 295 passage ids distinct, zero duplicate-id lines.
  • ruff check / ruff format --check clean; TDD exit gate PASS (0 self-inflicted test/lint findings).

@ww2283
ww2283 merged commit 9149417 into main Aug 19, 2026
4 of 5 checks passed
@ww2283
ww2283 deleted the bugfix/issue-7-chunk-id-aliasing branch August 19, 2026 23:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

incremental add: all chunks of a file share one passage id (shared metadata dict aliasing)

1 participant