Skip to content

fix(router): commit K3 distribution seeds after streaming completion #65

Description

@DavidBellamy

Context

The bounded K3 distribution-seed path currently rejects every streaming request because its terminal-success hook only runs for non-streaming responses. Live M2 traffic is about 95 percent streaming, so the deployed escape hatch has selected only one seed while hot cache owners still queue work and idle peers remain available.

Goals

  • Extend distribution seeding to scalar OpenAI Chat Completions streaming.
  • Commit a provisional cache owner only after backend completion and clean client-visible body EOF.
  • Preserve failure quarantine on backend errors, parse errors, send failures, and client disconnects.
  • Keep the partition seed cap at one for the first release.

Non-goals

  • No first-token commit.
  • No fanout or n greater than one.
  • No streaming support for other endpoints in the first release.
  • No increase to the seed cap.

Proposed design

A private producer witness records that the backend emitted Complete, chunk processing succeeded, and terminal DONE was queued. A response-body wrapper retains the existing non-cloneable LoadGuards. It commits the distribution seed only when the body reaches clean EOF and the producer witness is true. Any error or Drop before clean EOF leaves the guard uncommitted, causing the existing failure quarantine and exact lease release.

Safety and tests

  • Producer success alone never commits before body EOF.
  • Clean EOF commits once and only once.
  • Disconnect, body error, backend error, missing Complete, and terminal-send failure quarantine.
  • Cap one blocks a second seed while the first stream lives.
  • Existing target, revision, prefix, scheduler proof, and fair-share bindings remain unchanged.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions