You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
His framing, which is the right one: a rejected candidate produces a MissReason and gets counted somewhere. A fallback to compute produces nothing. It is a cache miss that never registers as a miss, so the failure mode with the least drama is the one with no telemetry pointing at it.
Checked, and it is not covered
CacheEvent has four variants: Hit, Miss, Write, Eviction. None concerns the embedder.
CacheStats carries lookups, hits, misses, belowThreshold, guardRejections, verifierRejections, writes. Nothing for a degraded lookup.
MissReason has EMPTY_SCOPE, BELOW_THRESHOLD, REJECTED_BY_GUARD, REJECTED_BY_VERIFIER. A fallback is not a miss, so it correctly does not belong here, which is exactly why it falls through the floor.
M8 shipped EmbedFailurePolicy and M9 shipped observability, both in 0.4.0, but M9 delivered per-MissReason and per-guard counters plus embed / search / verify timers. A timer says how long the call took, not that it failed and the cache stepped aside.
So the policy that exists to keep the cache "never worse than no cache" is also the one path that leaves no trace. A team running with FALL_BACK_TO_COMPUTE and a flapping embedder sees a hit rate quietly collapse with no signal naming the cause, and the natural first suspicion is the thing that is instrumented, the guards.
Shape of a fix
Smallest useful version, roughly in order:
A CacheEvent.Degraded (name to bikeshed) carrying the policy that fired, the failure cause and whether the call was retried, emitted on the same listener seam so it costs nothing when unused.
A counter in CacheStats so stats() alone answers "how often did the cache step aside".
Bind it in kmemo-micrometer next to the existing counters, which is where an alert would actually hang.
Worth deciding whether RetryingEmbedder exhausting its attempts is the same event or a distinct one, since one is a degraded lookup and the other is a reliability signal about the provider.
Mike's argument for doing it before there is production traffic rather than after is the part worth keeping: nobody notices this counter is missing until a postmortem needs it, and then it is not there.
Raised by Mike Czerwinski in the thread on the dev.to write-up:
https://dev.to/jugeni/comment/3c10k
Post: https://dev.to/tonytonycoder11/kmemo-a-semantic-cache-for-llm-calls-that-refuses-to-serve-you-the-wrong-answer-54h7
His framing, which is the right one: a rejected candidate produces a
MissReasonand gets counted somewhere. A fallback to compute produces nothing. It is a cache miss that never registers as a miss, so the failure mode with the least drama is the one with no telemetry pointing at it.Checked, and it is not covered
CacheEventhas four variants:Hit,Miss,Write,Eviction. None concerns the embedder.CacheStatscarrieslookups,hits,misses,belowThreshold,guardRejections,verifierRejections,writes. Nothing for a degraded lookup.MissReasonhasEMPTY_SCOPE,BELOW_THRESHOLD,REJECTED_BY_GUARD,REJECTED_BY_VERIFIER. A fallback is not a miss, so it correctly does not belong here, which is exactly why it falls through the floor.EmbedFailurePolicyand M9 shipped observability, both in0.4.0, but M9 delivered per-MissReasonand per-guard counters plus embed / search / verify timers. A timer says how long the call took, not that it failed and the cache stepped aside.So the policy that exists to keep the cache "never worse than no cache" is also the one path that leaves no trace. A team running with
FALL_BACK_TO_COMPUTEand a flapping embedder sees a hit rate quietly collapse with no signal naming the cause, and the natural first suspicion is the thing that is instrumented, the guards.Shape of a fix
Smallest useful version, roughly in order:
CacheEvent.Degraded(name to bikeshed) carrying the policy that fired, the failure cause and whether the call was retried, emitted on the same listener seam so it costs nothing when unused.CacheStatssostats()alone answers "how often did the cache step aside".kmemo-micrometernext to the existing counters, which is where an alert would actually hang.Worth deciding whether
RetryingEmbedderexhausting its attempts is the same event or a distinct one, since one is a degraded lookup and the other is a reliability signal about the provider.Mike's argument for doing it before there is production traffic rather than after is the part worth keeping: nobody notices this counter is missing until a postmortem needs it, and then it is not there.