-
Notifications
You must be signed in to change notification settings - Fork 0
fix(sidecar): discover Bytez free models and suppress expected 413 #1651
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
503c1c8
fix(sidecar): admit Bytez price evidence and hide expected 413
seonghobae 9b2bccb
test(sidecar): pin Bytez discovery and quiet 413 contract
seonghobae d91c777
docs(adr): record Bytez discovery pin and isolated 413 probe
seonghobae a598f50
test(sidecar): expose Bytez non-token free evidence gap
seonghobae 90dee49
fix(sidecar): preserve Bytez non-token free evidence
seonghobae f20ab84
fix(sidecar): preserve existing cost policy contract
seonghobae a67a84e
docs(sidecar): trace Bytez non-token price evidence
seonghobae b42a0ef
merge: integrate evidence-driven OpenCode review baseline
seonghobae 9481922
merge: integrate queue-pressure cleanup baseline
seonghobae File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
44 changes: 44 additions & 0 deletions
44
docs/doctoring/bytez-provider-meter-free-evidence-20260902.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| # Bytez provider-meter free-evidence repair — 2026-09-02 | ||
|
|
||
| ## Incident and owner boundary | ||
|
|
||
| `ContextualWisdomLab/.github` consumes the exact vendored `ContextualWisdomLab/contextual-orchestrator` discovery runtime when it constructs the central review sidecar. The review control plane owns admission of discovered routes into `orchestrator/free`; the reusable provider parser and its source-price semantics remain owned by `contextual-orchestrator`. | ||
|
|
||
| PR #1651 pins contextual-orchestrator commit `045d17da5e2aea56a97e241ee158ab1628d78660`. At that immutable source, the Bytez parser treats `meterPrice` as provider-native GPU/time-meter evidence rather than fabricating prompt/completion token prices. Its regression contract proves that `"0 / sec"` yields `DiscoveredModel.is_free == True` while both per-1k token price fields remain `None`; missing, malformed, boolean, and nonzero meter rates remain non-free. This is the upstream authority used here. | ||
|
|
||
| ## Root cause | ||
|
|
||
| The central launcher preserved the upstream `is_free` route identity but the central policy required both `prompt_price_per_1k` and `completion_price_per_1k`. Consequently, an exact-zero Bytez meter price was reclassified from upstream free evidence to `COST_UNKNOWN`, so Bytez could never enter the authorized free review pool even when discovery succeeded. | ||
|
|
||
| The defect was not a Bytez pricing problem and was not repaired by inventing token prices. It was an Anti-Corruption Layer loss: a provider-native price dimension was collapsed into a token-only central contract. | ||
|
|
||
| ## RED → GREEN evidence | ||
|
|
||
| The RED integration regression is `tests/test_contextual_orchestrator_bytez_catalog_integration.py` at commit `a598f500f6c278b44c40ea093954eb1de508a595`. It passes a pinned-runtime-shaped Bytez row through the real launcher `_report_rows`, then `parse_discovery_report`, then `build_zdr_prioritized_catalog`. Before the production repair, the route is `COST_UNKNOWN` and cannot be selected. | ||
|
|
||
| Production repair commits `90dee49e4d357b655480b86a4201291f9be02cc3` and `f20ab8469e5875732e587f69c3ba950b4169ef80` preserve the upstream exact-zero Bytez attestation as a separate `non_token_price_evidence` object: | ||
|
|
||
| ```json | ||
| { | ||
| "source": "bytez.meterPrice", | ||
| "price": 0.0, | ||
| "unit": "provider_meter_unit" | ||
| } | ||
| ``` | ||
|
|
||
| The existing `_normalize_cost_evidence` token-vector compatibility contract remains unchanged: a generic free marker without a complete token vector is still unknown. Only Bytez rows whose pinned upstream parser already attested exact-zero provider-meter price receive the non-token evidence object. Bytez rows without that attestation remain unknown and fail closed. | ||
|
|
||
| Selected-route audit evidence carries the same non-token object so the central review record does not erase why the route qualified as free. | ||
|
|
||
| ## Invariants | ||
|
|
||
| - Never fabricate Bytez prompt/completion per-token prices. | ||
| - Never infer free status from model name, provider name alone, missing price, or a nonzero/malformed meter rate. | ||
| - `OPENAI_API_KEY` remains excluded from `orchestrator/free` admission by the independent source-credential policy. | ||
| - ZDR/private-target admission remains independent from cost evidence and still fails closed. | ||
| - Provider discovery failure remains failure/absence evidence; this repair does not relabel an HTTP 500 or unavailable Bytez catalog as success. | ||
| - The central policy consumes the pinned upstream parser contract; mutable open-PR bytes are not runtime authority. | ||
|
|
||
| ## Follow-up boundary | ||
|
|
||
| A future provider-native pricing model with a different billing dimension requires its own explicit upstream evidence contract and central adapter decision. This Bytez repair is not a generic rule that `is_free=True` can replace missing price evidence for arbitrary providers. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
70 changes: 70 additions & 0 deletions
70
tests/test_contextual_orchestrator_bytez_catalog_integration.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| """End-to-end contract for Bytez free-price discovery and catalog admission.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| from pathlib import Path | ||
| import runpy | ||
| from types import SimpleNamespace | ||
|
|
||
| import pytest | ||
|
|
||
| from scripts.ci import contextual_orchestrator_review_policy as policy | ||
|
|
||
| _REPO_ROOT = Path(__file__).resolve().parents[1] | ||
| _LAUNCHER = _REPO_ROOT / "scripts/ci/contextual_orchestrator_review_launcher.py" | ||
| _BYTEZ_MODEL = "0-hero/Matter-0.1-Slim-7B-C" | ||
|
|
||
|
|
||
| def _report_bytez(*, free: bool) -> list[dict[str, object]]: | ||
| """Pass one pinned-runtime-shaped Bytez row through the real launcher adapter.""" | ||
| report_rows = runpy.run_path(str(_LAUNCHER))["_report_rows"] | ||
| discovered = SimpleNamespace( | ||
| provider_name="bytez", | ||
| model_id=_BYTEZ_MODEL, | ||
| agent_id="bytez_matter_01_slim_7b_c", | ||
| chat_base_url="https://api.bytez.com/models/v2/openai/v1", | ||
| credential_name="BYTEZ_API_KEY", | ||
| auth_scheme="raw-token", | ||
| output_modalities=("text",), | ||
| prompt_price_per_1k=None, | ||
| completion_price_per_1k=None, | ||
| currency_code="USD", | ||
| ) | ||
| free_routes = frozenset({("bytez", _BYTEZ_MODEL)}) if free else frozenset() | ||
| return report_rows([discovered], free_routes) | ||
|
|
||
|
|
||
| def test_zero_meter_price_survives_launcher_policy_and_catalog() -> None: | ||
| """Exact-zero Bytez meter pricing must enter free without fake token prices.""" | ||
| report_rows = _report_bytez(free=True) | ||
| assert report_rows[0]["is_free"] is True | ||
| assert report_rows[0]["prompt_price_per_1k"] is None | ||
| assert report_rows[0]["completion_price_per_1k"] is None | ||
|
|
||
| parsed = policy.parse_discovery_report({"models": report_rows}) | ||
| assert parsed[0]["cost_evidence"] == policy.COST_FREE | ||
| assert parsed[0]["non_token_price_evidence"] == { | ||
| "source": "bytez.meterPrice", | ||
| "price": 0.0, | ||
| "unit": "provider_meter_unit", | ||
| } | ||
| assert parsed[0]["prompt_price_per_1k"] is None | ||
| assert parsed[0]["completion_price_per_1k"] is None | ||
|
|
||
| result = policy.build_zdr_prioritized_catalog(parsed, pool="free") | ||
| assert [agent["model"] for agent in result["agents"]] == [_BYTEZ_MODEL] | ||
| assert result["agents"][0]["credential_key"] == "BYTEZ_API_KEY" | ||
| assert "cost:free" in result["agents"][0]["tags"] | ||
| assert result["report"]["selected"][0]["non_token_price_evidence"] == ( | ||
| parsed[0]["non_token_price_evidence"] | ||
| ) | ||
|
|
||
|
|
||
| def test_unattested_bytez_meter_price_remains_unknown() -> None: | ||
| """No free identity from the pinned parser means no Bytez free admission.""" | ||
| parsed = policy.parse_discovery_report({"models": _report_bytez(free=False)}) | ||
| assert parsed[0]["cost_evidence"] == policy.COST_UNKNOWN | ||
| assert parsed[0]["non_token_price_evidence"] is None | ||
|
|
||
| with pytest.raises(policy.PolicyError, match="would fail closed"): | ||
| policy.build_zdr_prioritized_catalog(parsed, pool="free") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.