Skip to content

Page large admin results without dropping data - #2

Merged
tin-berri merged 4 commits into
BerriAI:mainfrom
moe-berri:feat/progressive-results
Sep 24, 2026
Merged

tin-berri merged 4 commits into
BerriAI:mainfrom
moe-berri:feat/progressive-results

Conversation

@moe-berri

@moe-berri moe-berri commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator

Large user, team, model and request-log reads currently embed entire related collections or histories. This makes compact results the default without discarding fields: small responses remain complete, while large responses provide complete small values, record previews and exact paths to unread data. Agents can read the original result or selected sections without repeating the gateway operation.

All 65 administrative actions retain their names, arguments and server-side validation. Standard MCP discovery returns complete schemas by default, so clients with native tool search can defer loading them directly. The server does not enable a client setting. An optional lightweight discovery mode supports clients that eagerly load every schema.

Changes

  • Add read_admin_result for exact field, collection and Unicode string reads from immutable, caller-bound snapshots. Indexes distinguish unloaded data from null, false, zero and empty values. Keep records whole at page boundaries when they fit.
  • Add describe_admin_tool for complete schemas, including advanced parameters and referenced definitions. Inline top-level body references so clients can display their fields. Discovery mode retains individual tool names and safety annotations; there is no execution dispatcher.
  • Deliver key creation/rotation results inline, without caching secrets. Other cached write results can be inspected without repeating a mutation. Snapshots expire after five minutes; bounded capacity falls back to the complete inline response instead of evicting an unexpired result.
  • Recognize data[i].model_info.key as a catalog identifier only in the two reviewed model-info reads. Ordinary model identifiers no longer disable paging. The field's value is still checked for recognizable credentials, and other fields/endpoints retain conservative secret detection.
  • Retain fresh authorization before execution and disclosure, including snapshot reads. Stream administrative responses with a 16 MB limit instead of downloading an unbounded body before applying the old 2 MB guard.
  • Correct the two reviewed POST reads in read-only mode and preserve JSON-null request bodies.
  • Document Claude Code, Cursor, OpenCode and VS Code/Copilot discovery behavior, with concise instructions explaining when to search for these tools.
  • Make the default uvx launcher track main and refresh the connector on each MCP startup, including when its package version is unchanged. Document the one-time migration for existing pinned installs, restart behavior, fixed-version alternatives and Docker redeployment.

response.view=full remains available for complete inline results. LITELLM_ADMIN_RESPONSE_VIEW defaults to compact; LITELLM_ADMIN_SCHEMA_MODE defaults to full. Agents remain free to request full results.

Validation

  • 87 tests pass. The two new model-info paging regressions failed before the fix and now verify complete reconstruction, full overrides and no repeated gateway fetches. Additional cases check that secrets in the identifier value, nested metadata, other paths and other endpoints still prevent caching.

  • Wheel and source distribution build successfully. Live verification uses a separately installed wheel, not an editable source import.

  • Verify automatic updates with a moving Git branch and unchanged package version: the same launcher/cache loaded the first commit, the next commit, and then the unchanged latest commit. The exact README launcher also initialized against the live gateway and completed a read-only key-list call; installed Git provenance matched upstream main.

  • Post-fix live list_models: initial text decreased from 194,169 bytes to 840 bytes (approximately 52,244 to 267 o200k_base comparison tokens). Every field and catalog identifier reconstructed exactly; full responses match the previous PR build. These are response measurements, not billing or end-to-end task-cost claims.

  • Post-fix live get_model: the unfiltered 8.36 MB catalog returned a compact index, and all 1,290 records reconstructed exactly across 629 detail reads. Snapshot contents, full overrides, previous-build data and gateway catalog identifiers matched in the recheck.

  • Independent live audit of the preceding head b11adc1 verified all 65 argument schemas against the installed base; exact user/team/key/model/budget output parity; lossless Unicode/edge-value paging; temporary team/key/budget lifecycles and cleanup; rotation, HTTP authentication, read-only POST reads, caller isolation, real credential revocation, cache capacity and real five-minute expiry.

  • Four fresh Codex/OpenCode client tasks in that audit returned correct results, including the full 67-tool catalog and optional discovery mode. Claude Code connected to the MCP, but an upstream Anthropic insufficient-credit error blocked inference. Native tool-search deferral remains client-controlled and was not established by those runs. Not every administrative write endpoint was exercised live.

  • Full reads and reading every page can still be large. The benchmark verifies exact reconstruction and reports first-page and all-field retrieval costs; a smaller first response does not establish equivalent reasoning quality or lower total task cost.

  • GitHub Actions passed for 8b41440, including tests, distribution builds, Docker build and container smoke test.

@tin-berri
tin-berri merged commit e374120 into BerriAI:main Sep 24, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants