refactor(server): move token counting out of algorithms

[nachiketb-nvidia]

What

Move Anthropic token counting out of libsy algorithms and into the server/client boundary.

• Remove count_tokens and count_tokens_client from Algorithm.
• Remove token-count capability methods from RoutedLlmClient and LlmTargetSet.
• Expose target-specific token counting directly on TranslatingLlmClient.
• Resolve and retain one Anthropic-format completion target per server route at startup.

Why

Token counting is provider HTTP behavior, not routing-algorithm behavior. The previous design forced every algorithm wrapper to expose a client and let a shared client select an arbitrary model from an internal HashMap.

How

The server considers only a route's completion targets, excluding judge-only targets. It prefers case-insensitive target names or model IDs containing opus, then sonnet, then haiku; ties and unrecognized names preserve route order. The selected model ID is passed directly to the HTTP client. Routes without an Anthropic-format target continue returning 400 count_tokens_unsupported.

What to review

• Token counting no longer appears in libsy or the neutral client protocol.
• Route-target ordering for each supported algorithm configuration.
• Exact-model selection when one client serves multiple Anthropic models.

Validation

• cargo test -p switchyard-server
• cargo clippy --workspace --all-targets -- -D warnings
• cargo fmt --all -- --check

Live differential validation

Compared main at 702383e8 with this MR at 7acb9227 using the same authenticated upstream and the same structured Anthropic request containing system text, content blocks, and tools.

Configuration	Before	After	Result
Single Opus target	200 {"input_tokens":578}	200 {"input_tokens":578}	Byte-identical
Haiku, Sonnet, Opus route	200 {"input_tokens":578}	200 {"input_tokens":578}	Byte-identical

All four server logs were clean. Single-target behavior is semantically and externally identical. For multiple targets, the endpoint contract remains identical while target selection intentionally changes from arbitrary client-map iteration to deterministic Opus preference.

Summary by CodeRabbit

• New Features

  • Token counting now uses the requested model and selects a suitable Anthropic-compatible route target.
  • Added model capability detection for token-count support.
  • Routes can configure optional token-count targets with preference for Opus, Sonnet, and Haiku models.

• Bug Fixes

  • Improved errors for missing Anthropic targets, unsupported models, and streaming token-count requests.

• Documentation

  • Updated token-counting endpoint documentation with target-selection rules.

[coderabbitai]

Walkthrough

Token counting now uses model-specific TranslatingLlmClient methods. Route configuration selects optional Anthropic targets. Route handlers access this metadata directly, while algorithm-level token-counting APIs are removed.

Changes

Token counting API and routing cleanup

Layer / File(s)	Summary
Client API and routing cleanup crates/libsy-llm-client/src/client.rs, crates/libsy/src/core/algorithm.rs, crates/libsy/src/algorithms/*	TranslatingLlmClient adds model-specific capability detection and token counting. Legacy token-counting methods are removed from algorithms and target sets.
Route target selection crates/switchyard-server/src/config.rs, crates/switchyard-server/src/lib.rs	Routes retain optional count-token targets. Selection filters supported completion targets and applies Claude model-name priority.
Route resolution and handler integration crates/switchyard-server/src/lib.rs, crates/switchyard-server/README.md, crates/switchyard-server/tests/server.rs	Token-count requests use route metadata and return a client error when no Anthropic target exists. Documentation and tests reflect the target selection behavior.

Estimated code review effort: 4 (Complex) | ~45 minutes

Poem

A rabbit hops through routes of hay,
Finds Claude targets on the way.
Opus first, then Sonnet bright,
Haiku waits in ordered sight.
Tokens count, and errors stay in flight.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 28.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: moving token counting from routing algorithms into the server and client boundary.

✨ Finishing Touches 💡 1

⚔️ Resolve merge conflicts 💡

• Resolve merge conflict in branch nachiketb/remove-count-tokens-from-libsy

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/libsy-llm-client/src/client.rs`:
- Around line 140-167: In count_tokens, validate llm_request.stream before
calling send_encoded and reject true with the existing InvalidRequest error.
Ensure the Anthropic count-token request cannot contain a stream=true field from
preserved or configured body data, while leaving non-streaming requests
unchanged; the later EncodedResponse::Streaming arm can then be removed as
unreachable.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5fbcec6a-455d-4d74-8f8a-df08ce786d5e

📥 Commits

Reviewing files that changed from the base of the PR and between dcce01a and 7acb922.

📒 Files selected for processing (12)

• crates/libsy-llm-client/src/client.rs
• crates/libsy/src/algorithms/fall_through.rs
• crates/libsy/src/algorithms/llm_class.rs
• crates/libsy/src/algorithms/passthrough.rs
• crates/libsy/src/algorithms/rand.rs
• crates/libsy/src/algorithms/stage.rs
• crates/libsy/src/core/algorithm.rs
• crates/protocol/src/client.rs
• crates/switchyard-server/README.md
• crates/switchyard-server/src/config.rs
• crates/switchyard-server/src/lib.rs
• crates/switchyard-server/tests/server.rs

💤 Files with no reviewable changes (2)

• crates/protocol/src/client.rs
• crates/libsy/src/core/algorithm.rs

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://NVIDIA-NeMo.github.io/Switchyard/pr-preview/pr-316/
Built to branch gh-pages at 2026-08-05 21:53 UTC. Preview will be ready when the GitHub Pages deployment is complete.