Skip to content

Separate retrieval and embeddings and deduplicate ingestion - #1

Draft
ianlyoo wants to merge 1 commit into
mainfrom
refactor/20261002
Draft

ianlyoo wants to merge 1 commit into
mainfrom
refactor/20261002

Conversation

@ianlyoo

@ianlyoo ianlyoo commented Oct 2, 2026

Copy link
Copy Markdown
Owner

Embedding loading/cache/fallback and result conversion/reranking are separated from vector-store persistence. One ingest request keeps the first nonempty document for each ID, skips persisted IDs and counts new unique documents, avoiding duplicate-ID failures across batches.

Validation: Baseline 32 → final 36 tests; 54 exact retrieval parity comparisons; lint and outgoing-history scan passed. Independent review passed.

Compatibility and limits: Tests exercise temporary Chroma stores with deterministic hash embeddings. No real Sheets calls, embedding downloads or provider requests; public signatures and result schemas stay stable.

This is a draft branch proposal; no merge, release, version bump or deployment is included.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant