Re-ingesting the same source duplicates chunks in InMemoryRetriever and FAISSRetriever

[adaumsilva]

Description

RAGPipeline.ingest(path) called twice on the same file doubles the index:

pipe.ingest("doc.txt")   # 12 chunks
pipe.ingest("doc.txt")   # 12 chunks
len(pipe.retriever)      # 24  — same 12 chunk ids stored twice

ChromaRetriever.add() uses upsert, so it is idempotent on chunk id. InMemoryRetriever and FAISSRetriever blindly append. The Retriever.add docstring in base.py does not say which behaviour is correct.

Motivation

Re-running an ingestion script (the normal workflow while iterating) silently pollutes results: the same chunk appears multiple times in top_k, crowding out genuinely different context. Chunk ids are already deterministic (<doc_id>:<index>, where doc_id is a hash of the source path), so idempotency is achievable.

Acceptance criteria

• Retriever.add docstring in ragframework/base.py defines the contract: adding a chunk whose id already exists replaces it (upsert semantics)
• InMemoryRetriever.add() replaces existing rows by id (maintain an id -> row index dict)
• FAISSRetriever.add() handles duplicates. IndexHNSWFlat does not support removal, so pick and document one approach: (a) keep an id -> label map and tombstone superseded labels (filtered out in retrieve, over-fetching to compensate), or (b) wrap the index in faiss.IndexIDMap2 if it supports the chosen index type. Explain the trade-off in the docstring.
• Tests for all three retrievers: add the same chunk twice → len() unchanged, retrieval returns it once, and the new content/embedding wins
• CHANGELOG.md updated under [Unreleased] → ### Fixed

Files to touch

• ragframework/base.py — Retriever.add docstring
• ragframework/retriever/in_memory.py
• ragframework/retriever/faiss.py
• tests/test_retriever/test_in_memory.py, tests/test_retriever/test_faiss.py, tests/test_retriever/test_chroma.py

Resources

• FAISS: removing vectors
• ChromaRetriever.add for the upsert reference behaviour

Estimated effort: Medium (half day)

[UdayKumar9381]

Hi @adaumsilva,

I'd like to work on Issue #28.

I'll update the retriever contract to define upsert semantics, so adding an existing chunk ID replaces the previous chunk rather than creating a duplicate.

I'll also update InMemoryRetriever and FAISSRetriever to handle duplicate IDs correctly, add regression tests covering duplicate ingestion and replacement behavior across the retrievers, and update the changelog.

I'll verify the changes with the existing test suite and code-quality checks before opening a PR.

Thanks!

[adaumsilva]

Assigned to @UdayKumar9381