Skip to content

test(pixel): pin the per-call provider eligibility matrix - #5709

Closed
tang-vu wants to merge 1 commit into
Osmantic:public-betafrom
tang-vu:test/beta-runtime-policy-20260917
Closed

tang-vu wants to merge 1 commit into
Osmantic:public-betafrom
tang-vu:test/beta-runtime-policy-20260917

Conversation

@tang-vu

@tang-vu tang-vu commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds tests/pixel_inference/test_runtime_policy.py — broader coverage for bin/pixel_provider/runtime_policy.py, the per-call provider eligibility matrix.

Why this matters

select_candidates runs on every routed generation call before runtime_gateway forwards it: it picks the leader-then-backups order, enforces the output-token budget, detects tools/vision/reasoning requirements from the payload, blocks cloud providers unless policy.allowCloud, and prevents route cycles through the ODS/Pixel pseudo-models. A silent wrong answer here either calls a provider that cannot serve the request (generation fails) or routes a request to a cloud provider the owner never authorized (privacy leak).

Relationship to existing coverage

tests/pixel_inference/test_provider_policy.py already pins the provider-side selection semantics: leader-first ordering, incompatible-backup skipping (contextTokens, maxOutputTokens, supportsTools, enabled), maxAttempts, cloud-not-authorized, route cycles for all three pseudo-models, reasoning requirement, leader-incompatible, provider-routing-disabled, and deep-copy non-mutation.

This suite is a superset that adds the payload-detection half of the contract, which the existing file does not exercise:

  • Budget — max_tokens/max_completion_tokens bounds (0, negative, non-int, bool, float, over leader limit), the min(1024, maxOutputTokens) default, backups skipped below the requested budget.
  • Payload capability detection — tools required via the tools field, a role='tool' message, or tool_calls on any message; vision required only via image_url content parts (a type='text' list does not trigger it).
  • Reasoning asymmetry — reasoning leader drops non-reasoning backups; non-reasoning leader admits reasoning backups.
  • Cloud policy — cloud allowed when policy.allowCloud is true; cloud backup (not just leader) rejected without it.
  • Ordering detail — explicit backup order preservation, maxAttempts=1 yields leader only, empty backups, skipped list does not record truncation.

Overlapping cases (ordering, route cycles, disabled routing, incompatible skips) intentionally re-pin the same invariants under the independent fixture, matching the superset precedent used in earlier contract PRs.

Overlap check

Searched runtime_policy, provider routing, candidate — no existing PR touches this module. The related runtime_gateway policy work (#5690) covers the HTTP gateway layer; this is the separate pure selection function called before the gateway builds candidates.

Validation

$ pytest tests/pixel_inference/test_runtime_policy.py -x -q
36 passed in 3.71s

Test-only change; revert is a single file deletion.

@tang-vu tang-vu closed this Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant