diff --git a/docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md b/docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md index 9677f4ddba..217b3cc0b1 100644 --- a/docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md +++ b/docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md @@ -1,6 +1,6 @@ # ADR-0003: Vendored contextual-orchestrator review sidecar with governed gateway pools -- Status: accepted, amended 2026-08-30 (see "2026-08-30 amendment" below — Strix +- Status: accepted, amended 2026-09-02 (see amendment history below — Strix now uses `orchestrator/free`, not the `orchestrator/auto` this header originally recorded) - Date: 2026-08-27 @@ -24,7 +24,7 @@ all five, and auto-optimize routing by cost. 1. **Vendoring, pinned**: `scripts/ci/contextual_orchestrator_review_sidecar.sh` clones `ContextualWisdomLab/contextual-orchestrator` at an exact SHA - (`8cd99f139915131ba0239bce12a5d6a5fd85394e` today) into `RUNNER_TEMP`. The + (`045d17da5e2aea56a97e241ee158ab1628d78660` today) into `RUNNER_TEMP`. The source's `requirements.lock` is installed with `--require-hashes` and `--no-deps`, so dependency resolution cannot silently move the reviewed runtime. @@ -104,7 +104,12 @@ all five, and auto-optimize routing by cost. OpenAI image-input limit of 512 MB total payload per request; it is not treated as a universal JSON default or as the Files API's separate 512 MB per-file limit. The sidecar startup probe verifies the configured HTTP - boundary before any review model runs. + boundary before any review model runs. The over-limit request must still + return HTTP 413, but its expected server diagnostic is captured and asserted + instead of being shown as an operational failure. Accepted-size and tool + schema probes use the pinned client's deterministic mock response explicitly, + so this startup contract has no provider-egress or provider-availability + dependency. ## Consequences @@ -232,3 +237,16 @@ all five, and auto-optimize routing by cost. runner capable of completing the work. This amendment supersedes all fixed readiness and inference-attempt budgets in ADR 0005. +- **2026-09-02 amendment: Bytez price discovery and body-limit probe isolation.** + The vendored pin advances from `8cd99f139915131ba0239bce12a5d6a5fd85394e` + to `045d17da5e2aea56a97e241ee158ab1628d78660`, the first reviewed revision + that maps Bytez catalog `meterPrice` evidence into the discovery model's + `is_free` classification. Only an exact zero price is eligible for + `orchestrator/free`; missing, malformed, or nonzero price evidence remains + fail-closed. A Bytez catalog HTTP failure remains a bounded, non-fatal + provider-discovery error and is never reclassified as successful discovery. + The startup over-limit request still has to return HTTP 413, but its expected + server diagnostic is captured and asserted rather than exposed as a runtime + fault. Accepted-size and tool-schema probes call the pinned client's + deterministic mock response explicitly and therefore perform no provider + call. diff --git a/docs/doctoring/bytez-provider-meter-free-evidence-20260902.md b/docs/doctoring/bytez-provider-meter-free-evidence-20260902.md new file mode 100644 index 0000000000..cc820d0025 --- /dev/null +++ b/docs/doctoring/bytez-provider-meter-free-evidence-20260902.md @@ -0,0 +1,44 @@ +# Bytez provider-meter free-evidence repair — 2026-09-02 + +## Incident and owner boundary + +`ContextualWisdomLab/.github` consumes the exact vendored `ContextualWisdomLab/contextual-orchestrator` discovery runtime when it constructs the central review sidecar. The review control plane owns admission of discovered routes into `orchestrator/free`; the reusable provider parser and its source-price semantics remain owned by `contextual-orchestrator`. + +PR #1651 pins contextual-orchestrator commit `045d17da5e2aea56a97e241ee158ab1628d78660`. At that immutable source, the Bytez parser treats `meterPrice` as provider-native GPU/time-meter evidence rather than fabricating prompt/completion token prices. Its regression contract proves that `"0 / sec"` yields `DiscoveredModel.is_free == True` while both per-1k token price fields remain `None`; missing, malformed, boolean, and nonzero meter rates remain non-free. This is the upstream authority used here. + +## Root cause + +The central launcher preserved the upstream `is_free` route identity but the central policy required both `prompt_price_per_1k` and `completion_price_per_1k`. Consequently, an exact-zero Bytez meter price was reclassified from upstream free evidence to `COST_UNKNOWN`, so Bytez could never enter the authorized free review pool even when discovery succeeded. + +The defect was not a Bytez pricing problem and was not repaired by inventing token prices. It was an Anti-Corruption Layer loss: a provider-native price dimension was collapsed into a token-only central contract. + +## RED → GREEN evidence + +The RED integration regression is `tests/test_contextual_orchestrator_bytez_catalog_integration.py` at commit `a598f500f6c278b44c40ea093954eb1de508a595`. It passes a pinned-runtime-shaped Bytez row through the real launcher `_report_rows`, then `parse_discovery_report`, then `build_zdr_prioritized_catalog`. Before the production repair, the route is `COST_UNKNOWN` and cannot be selected. + +Production repair commits `90dee49e4d357b655480b86a4201291f9be02cc3` and `f20ab8469e5875732e587f69c3ba950b4169ef80` preserve the upstream exact-zero Bytez attestation as a separate `non_token_price_evidence` object: + +```json +{ + "source": "bytez.meterPrice", + "price": 0.0, + "unit": "provider_meter_unit" +} +``` + +The existing `_normalize_cost_evidence` token-vector compatibility contract remains unchanged: a generic free marker without a complete token vector is still unknown. Only Bytez rows whose pinned upstream parser already attested exact-zero provider-meter price receive the non-token evidence object. Bytez rows without that attestation remain unknown and fail closed. + +Selected-route audit evidence carries the same non-token object so the central review record does not erase why the route qualified as free. + +## Invariants + +- Never fabricate Bytez prompt/completion per-token prices. +- Never infer free status from model name, provider name alone, missing price, or a nonzero/malformed meter rate. +- `OPENAI_API_KEY` remains excluded from `orchestrator/free` admission by the independent source-credential policy. +- ZDR/private-target admission remains independent from cost evidence and still fails closed. +- Provider discovery failure remains failure/absence evidence; this repair does not relabel an HTTP 500 or unavailable Bytez catalog as success. +- The central policy consumes the pinned upstream parser contract; mutable open-PR bytes are not runtime authority. + +## Follow-up boundary + +A future provider-native pricing model with a different billing dimension requires its own explicit upstream evidence contract and central adapter decision. This Bytez repair is not a generic rule that `is_free=True` can replace missing price evidence for arbitrary providers. diff --git a/scripts/ci/contextual_orchestrator_review_policy.py b/scripts/ci/contextual_orchestrator_review_policy.py index 53e66cfa36..910b8da3a9 100644 --- a/scripts/ci/contextual_orchestrator_review_policy.py +++ b/scripts/ci/contextual_orchestrator_review_policy.py @@ -3,9 +3,11 @@ ``orchestrator/free`` remains strictly zero-priced and admits only provider accounts explicitly authorized for that pool. ``orchestrator/auto`` may retain other globally discovered providers, including OpenAI, when their independent -policy permits them. Models without a complete price vector remain visible in -audit counts but are never admitted to CI review. Partial, malformed, or -contradictory price vectors fail closed. +policy permits them. Models without complete price evidence remain visible in +audit counts but are never admitted to CI review. Token-priced routes require a +complete prompt/completion vector; Bytez may instead carry the exact-zero +provider-meter attestation represented by contextual-orchestrator's ``is_free`` +result. Partial, malformed, or contradictory price evidence fails closed. """ from __future__ import annotations @@ -110,12 +112,14 @@ def _normalize_cost_evidence( completion_price: object, currency_code: object, ) -> tuple[str, float | None, float | None, str | None]: - """Classify complete free, priced, or wholly unavailable price evidence. - - A provider that publishes neither price component is retained for audit but - is not eligible for review routing. A partial vector is ambiguous and - rejected. Free markers remain authoritative only when any accompanying - published vector is complete, valid, and zero-priced. + """Classify complete free, priced, or wholly unavailable token evidence. + + A provider that publishes neither token-price component is retained for + audit but is not eligible on this evidence path. A partial vector is + ambiguous and rejected. Free markers remain authoritative only when any + accompanying published token vector is complete, valid, and zero-priced. + Provider-native non-token evidence is normalized separately so this + compatibility contract does not fabricate or reinterpret token prices. """ if prompt_price is None and completion_price is None: return (COST_UNKNOWN, None, None, None) @@ -138,6 +142,29 @@ def _normalize_cost_evidence( ) +def _bytez_non_token_price_evidence( + *, + is_free: bool, + prompt_price: object, + completion_price: object, +) -> dict[str, object] | None: + """Preserve Bytez exact-zero provider-meter evidence without token prices. + + The pinned contextual-orchestrator Bytez parser sets ``is_free`` only when + the provider's structured ``meterPrice`` rate parses as exactly zero, while + deliberately leaving prompt/completion per-token prices unset because Bytez + bills by provider meter time. A missing or nonzero meter price therefore + arrives as ``is_free=False`` and remains unknown here. + """ + if is_free and prompt_price is None and completion_price is None: + return { + "source": "bytez.meterPrice", + "price": 0.0, + "unit": "provider_meter_unit", + } + return None + + def parse_discovery_report(report: Mapping[str, Any]) -> list[dict[str, Any]]: """Validate and normalize a contextual-orchestrator discovery report.""" rows = report.get("models") @@ -177,15 +204,35 @@ def parse_discovery_report(report: Mapping[str, Any]) -> list[dict[str, Any]]: is_free = is_free_route(row.get("is_free")) route = f"{provider}/{model}" - cost_evidence, prompt_price, completion_price, currency_code = ( - _normalize_cost_evidence( + prompt_price_input = row.get("prompt_price_per_1k") + completion_price_input = row.get("completion_price_per_1k") + non_token_price_evidence = ( + _bytez_non_token_price_evidence( + is_free=is_free, + prompt_price=prompt_price_input, + completion_price=completion_price_input, + ) + if provider == "bytez" + else None + ) + if non_token_price_evidence is not None: + cost_evidence = COST_FREE + prompt_price = None + completion_price = None + currency_code = None + else: + ( + cost_evidence, + prompt_price, + completion_price, + currency_code, + ) = _normalize_cost_evidence( route=route, is_free=is_free, - prompt_price=row.get("prompt_price_per_1k"), - completion_price=row.get("completion_price_per_1k"), + prompt_price=prompt_price_input, + completion_price=completion_price_input, currency_code=row.get("currency_code"), ) - ) candidate_id = row.get("agent_id") or f"{provider}_{model}" normalized.append( { @@ -197,6 +244,7 @@ def parse_discovery_report(report: Mapping[str, Any]) -> list[dict[str, Any]]: "prompt_price_per_1k": prompt_price, "completion_price_per_1k": completion_price, "currency_code": currency_code, + "non_token_price_evidence": non_token_price_evidence, "base_url": row.get("base_url") or PROVIDER_BASE_URLS[provider], "credential_key": credential_key, "auth_scheme": row.get("auth_scheme") @@ -379,6 +427,7 @@ def build_zdr_prioritized_catalog( "model": row["model"], "agent_id": entry["id"], "cost_evidence": _cost_evidence(row), + "non_token_price_evidence": row.get("non_token_price_evidence"), "zdr": is_zdr_model( str(row["provider"]), model=str(row["model"]), diff --git a/scripts/ci/contextual_orchestrator_review_sidecar.sh b/scripts/ci/contextual_orchestrator_review_sidecar.sh index 0ab2ae66d2..4205e4c5ab 100755 --- a/scripts/ci/contextual_orchestrator_review_sidecar.sh +++ b/scripts/ci/contextual_orchestrator_review_sidecar.sh @@ -14,7 +14,7 @@ # (fail-closed zero-cost) pool. set -euo pipefail -ORCHESTRATOR_PIN_SHA="${ORCHESTRATOR_PIN_SHA:-8cd99f139915131ba0239bce12a5d6a5fd85394e}" +ORCHESTRATOR_PIN_SHA="${ORCHESTRATOR_PIN_SHA:-045d17da5e2aea56a97e241ee158ab1628d78660}" ORCHESTRATOR_GIT_URL="${ORCHESTRATOR_GIT_URL:-https://github.com/ContextualWisdomLab/contextual-orchestrator.git}" # The Strix gate and Noema SSRF guard accept this one process-local origin. # Keep it fixed so an environment override cannot create an unvalidated sidecar. @@ -108,7 +108,9 @@ log "installing hash-pinned orchestrator dependencies at ${checked_out}" PYTHONPATH="$ORCHESTRATOR_SOURCE:$ORG_REPO_ROOT" "$sidecar_python" -c \ 'from contextual_orchestrator.credentials import get_credential; from contextual_orchestrator.model_discovery import discover_all_models, free_discovered_models; from contextual_orchestrator.orchestrator import ModelClient, TaskOrchestrator, load_agents; from contextual_orchestrator.review_gateway import register_review_credentials; from contextual_orchestrator.server import SecurityConfig, serve' PYTHONPATH="$ORCHESTRATOR_SOURCE:$ORG_REPO_ROOT" "$sidecar_python" - <<'PY' +import contextlib import http.client +import io import json import threading @@ -127,7 +129,9 @@ class CaptureClient(ModelClient): def proxy_send(self, agent, endpoint, payload): self.proxy_payloads.append(json.loads(json.dumps(payload, ensure_ascii=False))) - return super().proxy_send(agent, endpoint, payload) + # This contract exercises the loopback gateway only; provider egress + # would turn an offline startup check into an availability dependency. + return self._mock_raw(agent, endpoint, payload) client = CaptureClient() @@ -145,19 +149,25 @@ thread = threading.Thread(target=server.serve_forever, daemon=True) thread.start() try: connection = http.client.HTTPConnection("127.0.0.1", server.server_address[1], timeout=5) - connection.request( - "POST", - "/v1/chat/completions", - body=b"", - headers={ - "Authorization": "Bearer contract", - "Content-Type": "application/json", - "Content-Length": str(REVIEW_MAX_BODY_BYTES + 1), - }, + expected_rejection_log = io.StringIO() + with contextlib.redirect_stderr(expected_rejection_log): + connection.request( + "POST", + "/v1/chat/completions", + body=b"", + headers={ + "Authorization": "Bearer contract", + "Content-Type": "application/json", + "Content-Length": str(REVIEW_MAX_BODY_BYTES + 1), + }, + ) + response = connection.getresponse() + assert response.status == 413, response.status + response.read() + assert ( + "request_failed status=413 code=request_too_large" + in expected_rejection_log.getvalue() ) - response = connection.getresponse() - assert response.status == 413, response.status - response.read() connection.close() def post_payload(payload): diff --git a/tests/test_contextual_orchestrator_bytez_catalog_integration.py b/tests/test_contextual_orchestrator_bytez_catalog_integration.py new file mode 100644 index 0000000000..c2fd8be188 --- /dev/null +++ b/tests/test_contextual_orchestrator_bytez_catalog_integration.py @@ -0,0 +1,70 @@ +"""End-to-end contract for Bytez free-price discovery and catalog admission.""" + +from __future__ import annotations + +from pathlib import Path +import runpy +from types import SimpleNamespace + +import pytest + +from scripts.ci import contextual_orchestrator_review_policy as policy + +_REPO_ROOT = Path(__file__).resolve().parents[1] +_LAUNCHER = _REPO_ROOT / "scripts/ci/contextual_orchestrator_review_launcher.py" +_BYTEZ_MODEL = "0-hero/Matter-0.1-Slim-7B-C" + + +def _report_bytez(*, free: bool) -> list[dict[str, object]]: + """Pass one pinned-runtime-shaped Bytez row through the real launcher adapter.""" + report_rows = runpy.run_path(str(_LAUNCHER))["_report_rows"] + discovered = SimpleNamespace( + provider_name="bytez", + model_id=_BYTEZ_MODEL, + agent_id="bytez_matter_01_slim_7b_c", + chat_base_url="https://api.bytez.com/models/v2/openai/v1", + credential_name="BYTEZ_API_KEY", + auth_scheme="raw-token", + output_modalities=("text",), + prompt_price_per_1k=None, + completion_price_per_1k=None, + currency_code="USD", + ) + free_routes = frozenset({("bytez", _BYTEZ_MODEL)}) if free else frozenset() + return report_rows([discovered], free_routes) + + +def test_zero_meter_price_survives_launcher_policy_and_catalog() -> None: + """Exact-zero Bytez meter pricing must enter free without fake token prices.""" + report_rows = _report_bytez(free=True) + assert report_rows[0]["is_free"] is True + assert report_rows[0]["prompt_price_per_1k"] is None + assert report_rows[0]["completion_price_per_1k"] is None + + parsed = policy.parse_discovery_report({"models": report_rows}) + assert parsed[0]["cost_evidence"] == policy.COST_FREE + assert parsed[0]["non_token_price_evidence"] == { + "source": "bytez.meterPrice", + "price": 0.0, + "unit": "provider_meter_unit", + } + assert parsed[0]["prompt_price_per_1k"] is None + assert parsed[0]["completion_price_per_1k"] is None + + result = policy.build_zdr_prioritized_catalog(parsed, pool="free") + assert [agent["model"] for agent in result["agents"]] == [_BYTEZ_MODEL] + assert result["agents"][0]["credential_key"] == "BYTEZ_API_KEY" + assert "cost:free" in result["agents"][0]["tags"] + assert result["report"]["selected"][0]["non_token_price_evidence"] == ( + parsed[0]["non_token_price_evidence"] + ) + + +def test_unattested_bytez_meter_price_remains_unknown() -> None: + """No free identity from the pinned parser means no Bytez free admission.""" + parsed = policy.parse_discovery_report({"models": _report_bytez(free=False)}) + assert parsed[0]["cost_evidence"] == policy.COST_UNKNOWN + assert parsed[0]["non_token_price_evidence"] is None + + with pytest.raises(policy.PolicyError, match="would fail closed"): + policy.build_zdr_prioritized_catalog(parsed, pool="free") diff --git a/tests/test_contextual_orchestrator_review_sidecar_contract.py b/tests/test_contextual_orchestrator_review_sidecar_contract.py index 0a63356dad..8fee875c6d 100644 --- a/tests/test_contextual_orchestrator_review_sidecar_contract.py +++ b/tests/test_contextual_orchestrator_review_sidecar_contract.py @@ -40,7 +40,7 @@ ) GATEWAY_MODEL = "contextual-orchestrator/orchestrator/free" -ORCH_PIN_SHA = "8cd99f139915131ba0239bce12a5d6a5fd85394e" +ORCH_PIN_SHA = "045d17da5e2aea56a97e241ee158ab1628d78660" def _read(path: Path) -> str: @@ -360,6 +360,12 @@ def test_sidecar_probes_the_pinned_server_body_limit_at_http_boundary() -> None: assert "accepted_size = 64 * 1024 + 1" in text assert "REVIEW_MAX_BODY_BYTES + 1" in text assert "assert response.status == 413" in text + assert "expected_rejection_log = io.StringIO()" in text + assert "with contextlib.redirect_stderr(expected_rejection_log):" in text + assert '"request_failed status=413 code=request_too_large"' in text + assert "in expected_rejection_log.getvalue()" in text + assert "return self._mock_raw(agent, endpoint, payload)" in text + assert "return super().proxy_send(agent, endpoint, payload)" not in text assert "_request_body_size" not in text assert "class CaptureClient(ModelClient):" in text assert '"description": description' in text