Add Status tab with auto-revival, latency-based smart routing, cost tracking, and bug fixes - #11
Open
MIsmail3018 wants to merge 2 commits into
Open
Conversation
… admin dashboard
Three new features integrated into the Status tab and exposed via REST endpoints:
1. AutoRevival (~/rotato/src/keyRotator.js)
- Re-tests quarantined keys every 1 hour against their provider base URL
- Auto-unquarantines keys that are alive again (HTTP <400)
- Manual trigger via POST /admin/api/revival/run
- Prevents 'permanent' dead-key loss when a provider comes back online
2. LatencyTracker + smart routing (~/rotato/src/openaiClient.js)
- Rolling 20-sample latency window per key (avg/p50/p95)
- State persisted to ~/rotato/latency_state.json across restarts
- getNextKey() now sorts candidates by average latency, fastest first
- Dashboard shows live latency rank with gold/silver/bronze medals
3. CostTracker (~/rotato/src/keyRotator.js)
- Records tokens_in/tokens_out/cost per request from upstream usage blocks
- 24h summary by provider AND by model
- Input vs output token split on dashboard
- Last 10,000 records kept (ring buffer)
New REST endpoints (all require admin auth):
- GET /admin/api/status - named keys + dead keys + 24h usage
- GET /admin/api/latency - latency stats per key
- GET /admin/api/cost - cost/token summary 24h + by provider/model
- GET /admin/api/quarantine - full quarantine state
- POST /admin/api/quarantine/clear - body {key?} clears one or all
- POST /admin/api/revival/run - triggers AutoRevival immediately
Dashboard (~/rotato/public/admin.html) adds a new 'Status' tab with:
- 4 stat cards (providers, named keys, dead keys, requests/24h)
- Dead keys panel with human names + quarantine expiry
- Per-provider quota bars (red >=90%, yellow >=70%, green <70%)
- Per-key usage rows with 429 count
- Smart Routing card (blue left-border, latency rank)
- Cost Summary card (green left-border, 24h totals)
- Auto-refresh every 10s when tab active
Tested 2026-06-07 with 14 real requests: 6 keys tracked, latency range
1053ms-4601ms, 13497 tokens, 0 errors. AutoRevival scheduler fires at
30s after startup and every 1h thereafter.
1. Providers stat showed 0 — `Object.keys()` on a Map returns []. Fix: use `Array.from(this.config.getProviders().keys())`. 2. omni-route-key marked Dead (HTTP 403) — 403 is 'endpoint forbidden' (OmniRoute doesn't expose GET /v1/models), not 'key invalid'. Fix: remove 403 from deadKeyStatusCodes in openaiClient.js (was: Set([401, 403]); now: Set([401])). Also: AutoRevival.testKey now returns null on 403/404 to mean 'endpoint not available, don't mark dead' (distinct from false=key invalid, true=key alive). 3. Provider quota bars included rotato's own auth-gate rejects. Fix: filter out entries where status=401 and keyUsed is null (these are rotato rejecting missing/invalid ACCESS_KEY) before building per-provider usage. Count them as auth_rejected_24h instead, exposed as separate stat in the response. 4. '(no key)' rows polluting per-provider display — same root cause as p32929#3, plus filter in dashboard: hide rows where name === '(no key)' from the per-provider cards. Bonus: filter out providers from the response that aren't in the configured list (cleaned up 'openai' and 'unknown' with 0 requests showing as fake provider rows). Manually cleared the stale omni-route-key entry from quarantine_state.json since the trigger condition no longer exists.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Three new features for the rotato admin dashboard and key rotation pipeline:
getNextKey()now sorts candidates by average latency, fastest first. State persists across restarts.tokens_in/tokens_out/costper request from upstreamusageblocks. 24h summary by provider AND by model. 10k-record ring buffer.New REST endpoints (admin auth required)
GET /admin/api/status- named keys + dead keys + 24h usageGET /admin/api/latency- latency stats per keyGET /admin/api/cost- cost/token summary 24h + by provider/modelGET /admin/api/quarantine- full quarantine statePOST /admin/api/quarantine/clear- body{key?}clears one or allPOST /admin/api/revival/run- triggers AutoRevival immediatelyDashboard
New "Status" tab in
public/admin.htmlwith:Files changed
public/admin.html- +279 lines (Status tab + 3 new render functions)src/keyRotator.js- +366 lines (LatencyTracker, CostTracker, AutoRevival classes)src/openaiClient.js- +42 lines (latency recording on success/error, cost recording)src/server.js- +205 lines (6 new endpoints, AutoRevival wiring)Total: 4 files changed, 858 insertions(+), 34 deletions(-)
Tested
2026-06-07 with 14 real requests: 6 keys tracked, latency range 1053ms-4601ms, 13497 tokens, 0 errors. AutoRevival scheduler fires at 30s after startup and every 1h thereafter.
Notes for reviewer
BASE_URLin.env, it will be picked up fromconfig.providers.costfield is recorded as 0 because rotato does not currently know per-model pricing; a follow-up could parsemodel_pricing.jsonif you maintain one upstream.Update: Bug fixes from user review
After the first review surfaced 4 data bugs in the Status tab, this branch now also includes:
Also filtered out zero-count ghost providers (e.g. 'openai'/'unknown' with 0 requests) from the response.
Total: 4 files, +40/-11 on top of the original +858/-34.