Skip to content

feat(admission): wire scheduler capacity credits #58

Description

@DavidBellamy

Problem

Comet owns per-user waiting queues and hierarchical output-token WFQ, but only SMG knows whether a model partition can accept another request. The merged registry in #54 provides replay-safe tokens but is intentionally unwired.

Wire the registry to authoritative priority-scheduler slots so Comet can reserve capacity without putting request bodies into SMG's shared queue.

Design context: LLM360/comet#576 and #53.

Acceptance criteria

  • The feature is disabled by default and inert when disabled.
  • A service-key-authenticated internal issue endpoint selects an exact configured admission partition and atomically acquires one scheduler slot without queueing.
  • Issued credits bind allocator generation, policy epoch, partition, canonical model, tenant, request attempt, and positive output-token estimate.
  • The inference path redeems a matching credit at most once and reuses its held SchedulerPermit instead of acquiring a second slot or entering the shared scheduler queue.
  • Missing, malformed, expired, replayed, cross-generation, cross-model, cross-partition, cross-tenant, or cross-request credits cannot bypass admission.
  • Cancellation and TTL expiry release the held scheduler permit, and permit payloads are dropped outside registry locks.
  • Credit issue is immediate: no available slot returns retryable 429 rather than queueing inside SMG.
  • When explicitly enabled with partitioned engine-feedback admission, the scheduler's total ceiling follows telemetry revisions; SlotPool inflight remains the authoritative local usage count.
  • Handler-side adaptive admission remains a safety backstop for telemetry races. A resulting pre-execution 429 releases the redeemed permit normally and is not charged by Comet.
  • Credit issue, redemption, cancellation, expiry, binding mismatch, replay, capacity shrink, and concurrent single-winner behavior have focused tests.
  • Metrics use bounded partition/outcome labels and never expose bearer tokens.

Scope boundaries

  • No Comet user-facing admission API or client wrapper in this issue.
  • No SMG per-user queue or fair-share ordering.
  • No engine, pool, worker-selection, or cache-routing changes.
  • The first production consumer will use explicit admission partitions only.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions