Problem Statement
smem reindex can re-embed the neurons that have no vector (--missing-only, the default) or every neuron in the brain (--all). There is nothing in between, and the gap matters because the failure mode that most needs re-embedding leaves the vector field populated, not empty.
When a neuron's content is replaced but its embedding is not recomputed, the row ends up holding a vector that describes the previous text. --missing-only cannot see it, because the field is not missing. The only remedy on offer is --all, which re-embeds the entire brain to fix a handful of rows - on a large brain that is a long, costly job, and on a metered embedding provider it is a bill.
Proposed Solution
A third mode that identifies rows whose stored vector no longer matches their current content: re-embed the content, compare the result against the stored vector, and only write when they diverge beyond a threshold.
smem reindex --stale # report and fix drifted vectors
smem reindex --stale --dry-run # report only: how many, which
smem reindex --stale --threshold 0.98 # cosine below this counts as stale
Use Case
The practical one: after any episode where content was updated through a path that did not refresh the derived fields, an operator wants to know whether their brain is affected and repair only what is. Today the honest answer is "re-embed everything and hope", which is expensive enough that people will skip it and keep a quietly wrong index.
A dry run is useful on its own, even without the repair - it answers "is my recall index actually describing my current content?", which is not answerable today.
Alternatives Considered
--all. Correct but disproportionate; cost scales with brain size rather than with damage.- Reactive detection at recall time. Would spread the embedding cost into the query path, which seems worse than an explicit maintenance command.
- Nothing. Defensible if you consider the write paths authoritative, though it leaves no way to verify that.
Additional Context
This came out of the follow-up discussion on #166, where the same shape of problem was fixed at one write site. A --stale mode is the general safety net for the class rather than a fix for any one call site: it does not care how the vector came to disagree with the content.
Cost note worth weighing: --stale re-embeds everything it inspects in order to compare, so a full-brain --stale sweep costs roughly what --all costs in provider calls, and saves the writes rather than the embeddings. The saving is real for a targeted run or a dry run, and honest to state up front rather than have someone discover it.
No urgency on this one - filing it while it is fresh, measured on v3.8.0 (ae8e8743).
Thanks for the recent run of releases; it makes it easy to check ideas against a current tree.
Problem Statement
smem reindexcan re-embed the neurons that have no vector (--missing-only, the default) or every neuron in the brain (--all). There is nothing in between, and the gap matters because the failure mode that most needs re-embedding leaves the vector field populated, not empty.When a neuron's content is replaced but its embedding is not recomputed, the row ends up holding a vector that describes the previous text.
--missing-onlycannot see it, because the field is not missing. The only remedy on offer is--all, which re-embeds the entire brain to fix a handful of rows - on a large brain that is a long, costly job, and on a metered embedding provider it is a bill.Proposed Solution
A third mode that identifies rows whose stored vector no longer matches their current content: re-embed the content, compare the result against the stored vector, and only write when they diverge beyond a threshold.
Use Case
The practical one: after any episode where content was updated through a path that did not refresh the derived fields, an operator wants to know whether their brain is affected and repair only what is. Today the honest answer is "re-embed everything and hope", which is expensive enough that people will skip it and keep a quietly wrong index.
A dry run is useful on its own, even without the repair - it answers "is my recall index actually describing my current content?", which is not answerable today.
Alternatives Considered
--all. Correct but disproportionate; cost scales with brain size rather than with damage.Additional Context
This came out of the follow-up discussion on #166, where the same shape of problem was fixed at one write site. A
--stalemode is the general safety net for the class rather than a fix for any one call site: it does not care how the vector came to disagree with the content.Cost note worth weighing:
--stalere-embeds everything it inspects in order to compare, so a full-brain--stalesweep costs roughly what--allcosts in provider calls, and saves the writes rather than the embeddings. The saving is real for a targeted run or a dry run, and honest to state up front rather than have someone discover it.No urgency on this one - filing it while it is fresh, measured on v3.8.0 (
ae8e8743).Thanks for the recent run of releases; it makes it easy to check ideas against a current tree.