fix(verify): allow duplicate content-hash passage ids (#5) - #6
Merged
Merged
Conversation
Content-hash ids are legitimately many-to-one for byte-identical chunks; verify's bijection invariants flagged healthy indexes. - jsonl dup ids allowed only when text is identical per id - idx cardinality compared against unique jsonl ids - IVF passage_to_id checked as last-wins partial inverse over deduped id_to_passage values
Codex review of PR #6: type-prefixed text hash so 1 vs "1" counts as different text, and type-independent sort of conflicting ids so mixed int/str ids report findings instead of raising TypeError.
ww2283
added a commit
that referenced
this pull request
Aug 19, 2026
* fix(verify): allow duplicate content-hash passage ids (#5) Content-hash ids are legitimately many-to-one for byte-identical chunks; verify's bijection invariants flagged healthy indexes. - jsonl dup ids allowed only when text is identical per id - idx cardinality compared against unique jsonl ids - IVF passage_to_id checked as last-wins partial inverse over deduped id_to_passage values * docs: changelog entry for verify duplicate-id fix (#5) * fix(verify): harden duplicate-id conflict check against mixed types Codex review of PR #6: type-prefixed text hash so 1 vs "1" counts as different text, and type-independent sort of conflicting ids so mixed int/str ids report findings instead of raising TypeError.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5.
Problem
leann verifyassumed passage ids are unique (a bijection acrosspassages.jsonl,passages.idx, and the IVF id maps). With--id-scheme=content-hash, ids aresha256(text)[:16], so byte-identical chunks (empty__init__.py, repeated headers) legitimately share one id — the builder writes every chunk to jsonl, the offset map is last-wins per id, and the IVFpassage_to_idis the last-wins inverse of a many-to-oneid_to_passage. verify flagged all of this as corruption on healthy indexes while search worked fine.Change (
packages/leann-core/src/leann/cli.py)len(passage_to_id)vslen(set(id_to_passage.values())).passage_to_id[pid]=fidmust map back viaid_to_passage[fid] == pid, andset(passage_to_id) == set(id_to_passage.values()); the offsets set check is unchanged.No build-side changes; the storage contract is correct as-is.
Tests
tests/test_cli_verify.py: new_make_ivf_index_with_entriesfixture helper taking explicit(pid, text)pairs (last-wins idx/passage_to_id, oneid_to_passageentry per faiss label) + 4 tests: healthy duplicate-content index passes; duplicate id with different text, wrongpassage_to_idinversion, and pid missing frompassage_to_idall still fail. TDD: reproduction test written first and confirmed red on the old code.E2E / smoke checklist (all verified)
uv run pytest tests/test_cli_verify.py— 16 passed (4 new + 12 existing).--id-scheme=content-hashover a 129-chunk corpus containing 7 byte-identical files →leann verifyexits 0 (previously false-positived); jsonl confirmed to contain 129 lines / 123 unique ids with one id ×7.leann searchreturns correct results, including the deduplicated passage.ivf_id_map.json(passage_to_id[pid]=999999) → verify exits 1 withdoes not map back in id_to_passage; restored map → exits 0 again.ruff format --check/ruff checkclean on both files.