Measuring interest: API/Persistent-Usage-Monitor

[cleyesode]

I have a few iterations for per response context tracking n_tokens via API. Also, for persistent context monitor in chat. I'd like to measure you interest ahead of PR.

How do you feel about this in light of the existing /usage metrics estimation (and compression) system?

For API usage, I don't presume you are open to a complete replacement of existing integrations, or would you prefer parallell development (or perhaps coexistence as a fallback system when API metrics are unavailable)?

For persistent usage monitor in chat, rendering has some issues for scrolling and window resizing - these would be most easily overcome via sidebar - something you are comfortable with?

A perfect compliment to compression, accurate context would open the doors to enhanced user awareness on the near side, and agent self-awareness on the far side. Autonomous looping would be an excellent place to go from there.

It would also be fairly easy to create var for user setting max context (per config server) in env.

Thoughts? Thanks

[will-lamerton]

Thanks for the proposal! There's a strong foundation to build what already exists in the codebase!

What already exists:

• 4 tokenizer backends (OpenAI/tiktoken, Anthropic, Llama, fallback) in source/tokenization/
• A 6-category token breakdown (system, user, assistant, tool defs, tool results, total) in source/usage/calculator.ts
• Session tracking with persistent storage (last 100 sessions, 30-day aggregates) in source/usage/
• Real-time context % via useContextPercentage hook
• Auto-compact triggered at configurable thresholds
• /usage command with progress bars and category breakdowns

The opportunity is:

1. API-reported token counts - Right now tokens are estimated client-side (4 chars/token heuristic or tiktoken). The Vercel AI SDK already exposes usage fields in streaming responses, so wiring actual API-reported n_tokens into the existing SessionTracker would be the highest-value change. The current estimation system stays as a fallback for providers that don't report usage.

2. Env var overrides for context limits - Context limits currently resolve via models.dev API -> hardcoded fallback. Adding something like NANOCODER_MAX_CONTEXT would be a quick, useful win for users who want to cap usage below the model's actual limit. Fits cleanly into source/config/. I'm doing this today to close [Feature] Allow to specify context length manually #379 :)

3. Persistent monitor - You're right that rendering is tricky. Rather than a sidebar (Ink.js doesn't handle that well, and terminal real estate is tight), I'd suggest enhancing the existing DevelopmentModeIndicator in source/components/development-mode-indicator.tsx - it already shows ctx: NN% inline and could be extended to show more detail without the scroll/resize headaches.

Suggestion on approach: Parallel over replacing - layer API-reported numbers on top of the existing estimation system rather than ripping it out. A scoped first PR focused on API token tracking would be a great starting point.

Let me know what you think!

Would welcome your thoughts too @Avtrkrb :)

[cleyesode]

@Avtrkrb very good, thanks for the excellent guidance. I have a good version for inline reporting that clearly indicates when using API vs fallback, so I'll start there.

The issue that needs a proper sorting is allocations for % n_tokens to be attributed dynamically to each of the existing context categories (system prompt, user messages, assistant messages, tool messages, tool definitions). With n_tokens being the lump sum, your existing estimation system could be used to define ratios and assign appropriately.

While not strictly necessary for inline reporting of only n_tokens, this would allow for /usage menu to use API with fallback without any other aesthetic considerations - I'm thinking this may be the cleanest approach to avoid confusion that might arise from displaying both estimated and actual token counts within the same menu.

And awesome on the #379 !

I'll PR sometime this weekend. Thanks

[cleyesode]

@will-lamerton **Apologies for the misnomer - let me know if any issues with this direction @Avtrkrb !