Skip to content

docs: add Tier 7 to the roadmap after an external review - #27

Open
tonytonycoder11 wants to merge 1 commit into
mainfrom
docs/roadmap-tier-7-production-depth
Open

docs: add Tier 7 to the roadmap after an external review#27
tonytonycoder11 wants to merge 1 commit into
mainfrom
docs/roadmap-tier-7-production-depth

Conversation

@tonytonycoder11

Copy link
Copy Markdown
Contributor

An external review raised a set of items against Kmemo. This triages them into a new Tier 7 (M19 to M23) and records, next to each declined item, the reason it was declined, so the same suggestions do not have to be re-litigated later.

The filter is deliberately narrow: an item earns a milestone only if it answers something a team actually hits once the cache is on a request path. Nothing here is added for completeness.

Accepted

Milestone Why it earns a place
M19 · Exact-match fast path and conversation-aware keys Every getOrPut embeds, a network call, even when the prompt is byte for byte one already cached. An identical prompt in the same scope is the same question, so the fast path needs no guards at all and adds no false-hit risk by construction rather than by measurement. Embedding, not search, dominates what a lookup costs.
M20 · Shadow mode ThresholdCalibrator needs a labelled set, so the honest answer to "what threshold should I use?" is currently "measure it, on data you first have to build". That step, not the cache, is what stops teams putting a semantic cache in front of production traffic.
M21 · Invalidation beyond TTL A TTL is a guess about when knowledge might go stale; it is not a way to act on knowing that it just did.
M22 · CachePolicy kmemo-slf4j redacts prompts in logs, but the cache stores prompts and responses verbatim. For a regulated adopter the first question is whether they can prove a class of data never got persisted.
M23 · The comparative benchmark The central claim, that Kmemo rejects near misses other caches serve, has never been measured against another cache, only against this project's own corpus.

Three judgement calls worth surfacing, because they changed the suggestion rather than accepting it:

  • M19 also makes conversation keying explicit. kmemo-langchain4j already keys on the whole conversation, but core getOrPut has no notion of a turn. A caller who caches turn five of a dialogue on its text alone serves a confident wrong answer, which is precisely the failure this library exists to prevent, arrived at through the key rather than the threshold, where no guard can see it. scope is already the primitive; what is missing is the documented pattern and a helper.
  • M21 documents the pattern that already works first. Version the scope (pricing-v3.2 to pricing-v3.3) and clear the old one: no new API, atomic cutover, and for many teams the entire answer. Tags and predicate deletes come only where that is too coarse, because that is a seam change touching CacheStore, the TCK and all four adapters.
  • M23 deliberately excludes cross-runtime latency against GPTCache. A JVM versus Python wall-clock figure compares runtimes while appearing to compare caches, and publishing it would undercut the honesty the rest of the corpus work rests on. Only precision, recall, F1 and false-hit rate on identical inputs. Latency and throughput stay in kmemo-benchmarks, across Kmemo's own configurations.

Declined, with reasons recorded in the roadmap

  • Provider prompt caching as a third layer. It happens inside the provider's API. There is no seam for Kmemo to own, and presenting it as a Kmemo layer would take credit for a saving the library does not produce.
  • Reversible PII redaction. Reversible means the mapping is stored, which relocates the sensitive data rather than removing it: a weaker posture than not caching the entry (M22), sold as a stronger one.
  • Distributed propagation of the negative cache. It is deliberately local and only ever reuses a vector, never suppressing a search, which is exactly why it cannot manufacture a false hit. Making it coherent and distributed would add a coherence problem to defend that property, in exchange for saving embedding calls.
  • A shipped event-bus module. CacheListener already is that seam; bridging it to a broker is a few lines in the application, where the broker dependency belongs. Ships as a recipe.
  • Distributed cache-warming locks. A warm() race wastes embedding calls at startup; it corrupts nothing.

In M18, a cross-encoder reranker is declined as duplicating the Verifier seam while putting an ML dependency next to a core with one dependency. Quantization is accepted, but only with exact rescoring of the survivors, so the precision loss lands on which candidates are considered and never on the accept or reject decision itself: the same discipline the HNSW store already follows. PCA is declined for the reason rescoring cannot fix.

Scope

Documentation only: ROADMAP.md and the README Roadmap section. No code, no CHANGELOG.md entry, since nothing shipped. Checked against ROADMAP-CONVENTIONS.md: no without a version, tier and milestone headings conform, and the README carries the Shipped / Next / Later labels the standard calls for.

M17 gains the on-device rationale that makes the KMP migration worth its cost. It is the same migration as Kdrant's M25 on the same toolchain, and both now say so. The companion change is NaCode-Studios/Kdrant.

An external review raised a set of items against Kmemo. This records the ones
worth building as M19 to M23, and writes down why the rest were declined, so
the same suggestions do not have to be re-litigated later.

The two that come first are the exact-match fast path and shadow mode. Every
getOrPut currently embeds, a network call, even when the prompt is byte for
byte one already cached, and an identical prompt in the same scope needs no
guards at all, so the fast path adds no false-hit risk by construction rather
than by measurement (M19). Shadow mode runs the full lookup and serves
nothing, reporting what would have matched across a range of thresholds, which
removes the step that actually blocks adoption: ThresholdCalibrator needs a
labelled set the team does not have yet (M20). M19 also makes conversation
keying explicit, since caching a mid-dialogue turn on its text alone produces
the exact failure the guards exist to prevent, arrived at through the key
where no guard can see it.

Invalidation beyond TTL documents the scope-versioning pattern that already
works before growing the CacheStore seam, since the seam change touches the
TCK and all four adapters (M21). A CachePolicy veto covers data that must
never be persisted, and records that per-tenant isolation is already scope and
already tested (M22). The comparative benchmark finally measures the central
claim against GPTCache and a threshold-only baseline instead of only against
this project's own corpus (M23).

Declined, with the reason recorded next to each: provider prompt caching as a
third layer, which has no seam here to own; reversible PII redaction, which
relocates the sensitive data rather than removing it; distributed propagation
of the negative cache, which would defend a local safety property with a
coherence problem; a shipped event-bus module, since CacheListener is already
that seam; and distributed warming locks. In M18, a cross-encoder reranker is
declined as duplicating the Verifier, while quantization is accepted only with
exact rescoring of the survivors, so the precision loss never reaches the
accept or reject decision. PCA is declined for the same reason.

M23 deliberately excludes cross-runtime latency against GPTCache: a JVM
versus Python wall-clock figure compares runtimes while appearing to compare
caches, and publishing it would undercut the honesty the corpus work rests on.

M17 gains the on-device rationale that makes the KMP migration worth its cost.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

1 participant