Repository navigation
feat: v0.6 — shadow mode, cache.stats(), server tools in Anthropic streams - #9
Merged
Merged
Conversation
The numbers the CLI's stats command prints are now available from code: exact counts, expired entries, hits, sizes and the top namespaces and models. It checks the table first so a missing one says to run migrate(), and the CLI now goes through it instead of a parallel internal call.
With shadow: true every request is looked up but never served: the model is always called, lookup events are flagged shadow, misses are still stored, and onShadow receives each hit that would have been served next to the fresh answer. It lives in find(), the single place a hit becomes a served answer, so wrap, streams and the stream() helpers all get it. Shadow lookups have their own Prometheus counter and never count as tokens saved.
…arch
Server tool calls and their results are stored and replayed like the SDK accumulates them: tool input is assembled from input_json_delta and parsed when the block stops (an empty buffer is {}), result blocks arrive whole, and stop_details and container are carried over. Answers that ran code or carry a container are no longer stored on either path, since the container they point at expires.
…a stream The assembler merged only numeric usage fields from message_delta, so server_tool_use and output_tokens_details kept their message_start values: a replayed web-search answer reported 0 web search requests instead of 1. It now merges every non-null field, as MessageStream does.
get() returns null in shadow mode, set() ignores shadow, and onShadow only fires when the fresh answer could itself be stored. Also fixes a shadow test that asserted the wrong cache's call count.
Adds the would-have-hit ratio and shadow lookups by result, and has the metrics demo send 10% of its traffic in shadow mode so both have data (checked by examples/metrics/verify.ts, now 19 checks). Also mounts the dashboard folder instead of the file, since a single-file bind mount kept serving the old copy after an edit, and gives the similarity bars a fixed color instead of the default red threshold.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
shadow: trueon the cache or per call: every request is looked up, but the model is always called and its answer returned. Lookup events are flaggedshadow, misses are still stored, andonShadowreceives each hit that would have been served next to the fresh answer. It lives infind(), the single place a hit becomes served, sowrap,createwith and without streaming and the.stream()helpers all get it. Metrics:llm_cache_shadow_lookups_total, never counted as tokens saved.cache.stats()statsuses it.input_json_deltalike the SDK does;stop_detailsandcontainerare carried over.containerare no longer stored, streamed or not: the container expires, so a replay would return a dead id.onShadowis the only hook that receives response content, and the README says so next to the options.Testing
find()breaks the shadow tests on every path;onShadowreported the exact hit and the semantic hit (0.961) with the stored answer next to the fresh one.Not validated
MessageStreamand typed events.containerwhen code execution is only offered. The rule refuses anycontainer, so the answer doesn't change the behaviour.