docs: add Tier 7 to the roadmap after an external review - #27
Open
tonytonycoder11 wants to merge 1 commit into
Open
docs: add Tier 7 to the roadmap after an external review#27tonytonycoder11 wants to merge 1 commit into
tonytonycoder11 wants to merge 1 commit into
Conversation
An external review raised a set of items against Kmemo. This records the ones worth building as M19 to M23, and writes down why the rest were declined, so the same suggestions do not have to be re-litigated later. The two that come first are the exact-match fast path and shadow mode. Every getOrPut currently embeds, a network call, even when the prompt is byte for byte one already cached, and an identical prompt in the same scope needs no guards at all, so the fast path adds no false-hit risk by construction rather than by measurement (M19). Shadow mode runs the full lookup and serves nothing, reporting what would have matched across a range of thresholds, which removes the step that actually blocks adoption: ThresholdCalibrator needs a labelled set the team does not have yet (M20). M19 also makes conversation keying explicit, since caching a mid-dialogue turn on its text alone produces the exact failure the guards exist to prevent, arrived at through the key where no guard can see it. Invalidation beyond TTL documents the scope-versioning pattern that already works before growing the CacheStore seam, since the seam change touches the TCK and all four adapters (M21). A CachePolicy veto covers data that must never be persisted, and records that per-tenant isolation is already scope and already tested (M22). The comparative benchmark finally measures the central claim against GPTCache and a threshold-only baseline instead of only against this project's own corpus (M23). Declined, with the reason recorded next to each: provider prompt caching as a third layer, which has no seam here to own; reversible PII redaction, which relocates the sensitive data rather than removing it; distributed propagation of the negative cache, which would defend a local safety property with a coherence problem; a shipped event-bus module, since CacheListener is already that seam; and distributed warming locks. In M18, a cross-encoder reranker is declined as duplicating the Verifier, while quantization is accepted only with exact rescoring of the survivors, so the precision loss never reaches the accept or reject decision. PCA is declined for the same reason. M23 deliberately excludes cross-runtime latency against GPTCache: a JVM versus Python wall-clock figure compares runtimes while appearing to compare caches, and publishing it would undercut the honesty the corpus work rests on. M17 gains the on-device rationale that makes the KMP migration worth its cost.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An external review raised a set of items against Kmemo. This triages them into a new Tier 7 (M19 to M23) and records, next to each declined item, the reason it was declined, so the same suggestions do not have to be re-litigated later.
The filter is deliberately narrow: an item earns a milestone only if it answers something a team actually hits once the cache is on a request path. Nothing here is added for completeness.
Accepted
getOrPutembeds, a network call, even when the prompt is byte for byte one already cached. An identical prompt in the same scope is the same question, so the fast path needs no guards at all and adds no false-hit risk by construction rather than by measurement. Embedding, not search, dominates what a lookup costs.ThresholdCalibratorneeds a labelled set, so the honest answer to "what threshold should I use?" is currently "measure it, on data you first have to build". That step, not the cache, is what stops teams putting a semantic cache in front of production traffic.CachePolicykmemo-slf4jredacts prompts in logs, but the cache stores prompts and responses verbatim. For a regulated adopter the first question is whether they can prove a class of data never got persisted.Three judgement calls worth surfacing, because they changed the suggestion rather than accepting it:
kmemo-langchain4jalready keys on the whole conversation, but coregetOrPuthas no notion of a turn. A caller who caches turn five of a dialogue on its text alone serves a confident wrong answer, which is precisely the failure this library exists to prevent, arrived at through the key rather than the threshold, where no guard can see it.scopeis already the primitive; what is missing is the documented pattern and a helper.pricing-v3.2topricing-v3.3) and clear the old one: no new API, atomic cutover, and for many teams the entire answer. Tags and predicate deletes come only where that is too coarse, because that is a seam change touchingCacheStore, the TCK and all four adapters.kmemo-benchmarks, across Kmemo's own configurations.Declined, with reasons recorded in the roadmap
CacheListeneralready is that seam; bridging it to a broker is a few lines in the application, where the broker dependency belongs. Ships as a recipe.warm()race wastes embedding calls at startup; it corrupts nothing.In M18, a cross-encoder reranker is declined as duplicating the
Verifierseam while putting an ML dependency next to a core with one dependency. Quantization is accepted, but only with exact rescoring of the survivors, so the precision loss lands on which candidates are considered and never on the accept or reject decision itself: the same discipline the HNSW store already follows. PCA is declined for the reason rescoring cannot fix.Scope
Documentation only:
ROADMAP.mdand the README Roadmap section. No code, noCHANGELOG.mdentry, since nothing shipped. Checked againstROADMAP-CONVENTIONS.md: no✅without a version, tier and milestone headings conform, and the README carries theShipped/Next/Laterlabels the standard calls for.M17 gains the on-device rationale that makes the KMP migration worth its cost. It is the same migration as Kdrant's M25 on the same toolchain, and both now say so. The companion change is NaCode-Studios/Kdrant.