Skip to content

fix: key collisions and logprobs lost on replayed streams - #11

Merged
mateusands merged 4 commits into
mainfrom
fix/v0.6.1
Sep 29, 2026
Merged

mateusands merged 4 commits into
mainfrom
fix/v0.6.1

Conversation

@mateusands

@mateusands mateusands commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

Two defects found by an external review of c726aa4. I reproduced both before fixing them, and the plan review found a third.

Defect Cause Fix
Different requests could share a cache key (false hit) The canonical copy was a plain {}: an own __proto__ key replaced the prototype and vanished, and toJSON was ignored, so every Date hashed as {}. Null-prototype copy; toJSON followed like JSON.stringify (only when it is a function); other non-plain objects (Map, Set, RegExp, class instances) are unkeyable and bypass the cache.
A BigInt or a cycle in params made wrap() reject instead of calling the model deriveKey threw and no caller caught it, breaking fail-open. deriveKey turns any failure into a bypass.
An OpenAI stream replayed from the cache lost logprobs, system_fingerprint and service_tier The assembler stored logprobs: null and kept only id / created / model. Accumulate like ChatCompletionStream (first logprobs copied, later content appended; last value wins for the top-level fields) and replay them.

KEY_VERSION stays 1. Plain JSON, which is what the SDK wrappers send, hashes exactly as before. A test pins the hash computed on c726aa4 for a typical request with system prompt, prior turns, tools and nested params. Only inputs that used to collide get new hashes; their old rows simply miss.

Testing

  • 249 tests from a clean install. Lint, typecheck, build and the consumer check pass. The integration suite is also green on pgvector 0.7.4 / PG17 and 0.8.0 / PG13.
  • Every new test failed before its fix.
  • Guards checked by breaking them on purpose:
    • going back to a plain {} copy fails the __proto__ test;
    • not appending later logprobs fails the replay test.
  • The replay test compares against the SDK's own ChatCompletionStream.finalChatCompletion().

Code review follow-up

An independent code review found six more points, fixed in 8831524 and 65e89c8:

  • the catch now covers only serialization: a malformed input throws again, and a bypass reports why to onError with a new key stage;
  • plain objects from another realm (vm, jsdom) are hashed again;
  • toJSON is applied once per value, as JSON.stringify does;
  • a null service_tier survives replay;
  • comments and the changelog are corrected.

Each fix has a test, and the review re-checked both commits. The golden hash still holds. CI green on 65e89c8.

The canonical copy was a plain object, so an own __proto__ key replaced the prototype and vanished, and it ignored toJSON, so every Date hashed as {}. Keys now use a null-prototype copy, follow toJSON like JSON.stringify, and treat other non-plain objects (Map, Set, class instances) as unkeyable. deriveKey turns any failure, including BigInt and cycles, into a bypass so the call still reaches the model. Plain JSON hashes are unchanged (pinned by a test), so KEY_VERSION stays 1.
The stream assembler stored logprobs as null and kept only id, created and model from the chunks, so a cached stream answered a logprobs request without them. It now accumulates logprobs per choice like ChatCompletionStream (first copied, later content appended) and keeps system_fingerprint and service_tier, and the replay emits them.
From code review. The catch now covers only serialization, so a malformed input still throws instead of turning into a silent bypass, and the reason for a bypass goes to onError with a new key stage. Plain objects from another realm (vm, jsdom, some edge runtimes) are hashed again, toJSON is applied once per value as JSON.stringify does, and the comments say why class instances are rejected.
The replay checked service_tier by truthiness, so a null the stream carried came back absent. It now checks for undefined, and a comment records why refusal logprobs are never appended.
@mateusands
mateusands merged commit 0c347e4 into main Sep 29, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant