Skip to content
Merged
24 changes: 21 additions & 3 deletions docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# ADR-0003: Vendored contextual-orchestrator review sidecar with governed gateway pools

- Status: accepted, amended 2026-08-30 (see "2026-08-30 amendment" below — Strix
- Status: accepted, amended 2026-09-02 (see amendment history below — Strix
now uses `orchestrator/free`, not the `orchestrator/auto` this header
originally recorded)
- Date: 2026-08-27
Expand All @@ -24,7 +24,7 @@ all five, and auto-optimize routing by cost.

1. **Vendoring, pinned**: `scripts/ci/contextual_orchestrator_review_sidecar.sh`
clones `ContextualWisdomLab/contextual-orchestrator` at an exact SHA
(`8cd99f139915131ba0239bce12a5d6a5fd85394e` today) into `RUNNER_TEMP`. The
(`045d17da5e2aea56a97e241ee158ab1628d78660` today) into `RUNNER_TEMP`. The
source's `requirements.lock` is installed with `--require-hashes` and
`--no-deps`, so dependency resolution cannot silently move the reviewed
runtime.
Expand Down Expand Up @@ -104,7 +104,12 @@ all five, and auto-optimize routing by cost.
OpenAI image-input limit of 512 MB total payload per request; it is not
treated as a universal JSON default or as the Files API's separate 512 MB
per-file limit. The sidecar startup probe verifies the configured HTTP
boundary before any review model runs.
boundary before any review model runs. The over-limit request must still
return HTTP 413, but its expected server diagnostic is captured and asserted
instead of being shown as an operational failure. Accepted-size and tool
schema probes use the pinned client's deterministic mock response explicitly,
so this startup contract has no provider-egress or provider-availability
dependency.

## Consequences

Expand Down Expand Up @@ -232,3 +237,16 @@ all five, and auto-optimize routing by cost.
runner capable of completing the work.
This amendment supersedes all fixed readiness and inference-attempt budgets
in ADR 0005.
- **2026-09-02 amendment: Bytez price discovery and body-limit probe isolation.**
The vendored pin advances from `8cd99f139915131ba0239bce12a5d6a5fd85394e`
to `045d17da5e2aea56a97e241ee158ab1628d78660`, the first reviewed revision
that maps Bytez catalog `meterPrice` evidence into the discovery model's
`is_free` classification. Only an exact zero price is eligible for
`orchestrator/free`; missing, malformed, or nonzero price evidence remains
fail-closed. A Bytez catalog HTTP failure remains a bounded, non-fatal
provider-discovery error and is never reclassified as successful discovery.
The startup over-limit request still has to return HTTP 413, but its expected
server diagnostic is captured and asserted rather than exposed as a runtime
fault. Accepted-size and tool-schema probes call the pinned client's
deterministic mock response explicitly and therefore perform no provider
call.
44 changes: 44 additions & 0 deletions docs/doctoring/bytez-provider-meter-free-evidence-20260902.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Bytez provider-meter free-evidence repair — 2026-09-02

## Incident and owner boundary

`ContextualWisdomLab/.github` consumes the exact vendored `ContextualWisdomLab/contextual-orchestrator` discovery runtime when it constructs the central review sidecar. The review control plane owns admission of discovered routes into `orchestrator/free`; the reusable provider parser and its source-price semantics remain owned by `contextual-orchestrator`.

PR #1651 pins contextual-orchestrator commit `045d17da5e2aea56a97e241ee158ab1628d78660`. At that immutable source, the Bytez parser treats `meterPrice` as provider-native GPU/time-meter evidence rather than fabricating prompt/completion token prices. Its regression contract proves that `"0 / sec"` yields `DiscoveredModel.is_free == True` while both per-1k token price fields remain `None`; missing, malformed, boolean, and nonzero meter rates remain non-free. This is the upstream authority used here.

## Root cause

The central launcher preserved the upstream `is_free` route identity but the central policy required both `prompt_price_per_1k` and `completion_price_per_1k`. Consequently, an exact-zero Bytez meter price was reclassified from upstream free evidence to `COST_UNKNOWN`, so Bytez could never enter the authorized free review pool even when discovery succeeded.

The defect was not a Bytez pricing problem and was not repaired by inventing token prices. It was an Anti-Corruption Layer loss: a provider-native price dimension was collapsed into a token-only central contract.

## RED → GREEN evidence

The RED integration regression is `tests/test_contextual_orchestrator_bytez_catalog_integration.py` at commit `a598f500f6c278b44c40ea093954eb1de508a595`. It passes a pinned-runtime-shaped Bytez row through the real launcher `_report_rows`, then `parse_discovery_report`, then `build_zdr_prioritized_catalog`. Before the production repair, the route is `COST_UNKNOWN` and cannot be selected.

Production repair commits `90dee49e4d357b655480b86a4201291f9be02cc3` and `f20ab8469e5875732e587f69c3ba950b4169ef80` preserve the upstream exact-zero Bytez attestation as a separate `non_token_price_evidence` object:

```json
{
"source": "bytez.meterPrice",
"price": 0.0,
"unit": "provider_meter_unit"
}
```

The existing `_normalize_cost_evidence` token-vector compatibility contract remains unchanged: a generic free marker without a complete token vector is still unknown. Only Bytez rows whose pinned upstream parser already attested exact-zero provider-meter price receive the non-token evidence object. Bytez rows without that attestation remain unknown and fail closed.

Selected-route audit evidence carries the same non-token object so the central review record does not erase why the route qualified as free.

## Invariants

- Never fabricate Bytez prompt/completion per-token prices.
- Never infer free status from model name, provider name alone, missing price, or a nonzero/malformed meter rate.
- `OPENAI_API_KEY` remains excluded from `orchestrator/free` admission by the independent source-credential policy.
- ZDR/private-target admission remains independent from cost evidence and still fails closed.
- Provider discovery failure remains failure/absence evidence; this repair does not relabel an HTTP 500 or unavailable Bytez catalog as success.
- The central policy consumes the pinned upstream parser contract; mutable open-PR bytes are not runtime authority.

## Follow-up boundary

A future provider-native pricing model with a different billing dimension requires its own explicit upstream evidence contract and central adapter decision. This Bytez repair is not a generic rule that `is_free=True` can replace missing price evidence for arbitrary providers.
77 changes: 63 additions & 14 deletions scripts/ci/contextual_orchestrator_review_policy.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,11 @@
``orchestrator/free`` remains strictly zero-priced and admits only provider
accounts explicitly authorized for that pool. ``orchestrator/auto`` may retain
other globally discovered providers, including OpenAI, when their independent
policy permits them. Models without a complete price vector remain visible in
audit counts but are never admitted to CI review. Partial, malformed, or
contradictory price vectors fail closed.
policy permits them. Models without complete price evidence remain visible in
audit counts but are never admitted to CI review. Token-priced routes require a
complete prompt/completion vector; Bytez may instead carry the exact-zero
provider-meter attestation represented by contextual-orchestrator's ``is_free``
result. Partial, malformed, or contradictory price evidence fails closed.
"""

from __future__ import annotations
Expand Down Expand Up @@ -110,12 +112,14 @@ def _normalize_cost_evidence(
completion_price: object,
currency_code: object,
) -> tuple[str, float | None, float | None, str | None]:
"""Classify complete free, priced, or wholly unavailable price evidence.

A provider that publishes neither price component is retained for audit but
is not eligible for review routing. A partial vector is ambiguous and
rejected. Free markers remain authoritative only when any accompanying
published vector is complete, valid, and zero-priced.
"""Classify complete free, priced, or wholly unavailable token evidence.

A provider that publishes neither token-price component is retained for
audit but is not eligible on this evidence path. A partial vector is
ambiguous and rejected. Free markers remain authoritative only when any
accompanying published token vector is complete, valid, and zero-priced.
Provider-native non-token evidence is normalized separately so this
compatibility contract does not fabricate or reinterpret token prices.
"""
if prompt_price is None and completion_price is None:
return (COST_UNKNOWN, None, None, None)
Expand All @@ -138,6 +142,29 @@ def _normalize_cost_evidence(
)


def _bytez_non_token_price_evidence(
*,
is_free: bool,
prompt_price: object,
completion_price: object,
) -> dict[str, object] | None:
"""Preserve Bytez exact-zero provider-meter evidence without token prices.

The pinned contextual-orchestrator Bytez parser sets ``is_free`` only when
the provider's structured ``meterPrice`` rate parses as exactly zero, while
deliberately leaving prompt/completion per-token prices unset because Bytez
bills by provider meter time. A missing or nonzero meter price therefore
arrives as ``is_free=False`` and remains unknown here.
"""
if is_free and prompt_price is None and completion_price is None:
return {
"source": "bytez.meterPrice",
"price": 0.0,
"unit": "provider_meter_unit",
}
return None


def parse_discovery_report(report: Mapping[str, Any]) -> list[dict[str, Any]]:
"""Validate and normalize a contextual-orchestrator discovery report."""
rows = report.get("models")
Expand Down Expand Up @@ -177,15 +204,35 @@ def parse_discovery_report(report: Mapping[str, Any]) -> list[dict[str, Any]]:

is_free = is_free_route(row.get("is_free"))
route = f"{provider}/{model}"
cost_evidence, prompt_price, completion_price, currency_code = (
_normalize_cost_evidence(
prompt_price_input = row.get("prompt_price_per_1k")
completion_price_input = row.get("completion_price_per_1k")
non_token_price_evidence = (
_bytez_non_token_price_evidence(
is_free=is_free,
prompt_price=prompt_price_input,
completion_price=completion_price_input,
)
if provider == "bytez"
else None
)
if non_token_price_evidence is not None:
cost_evidence = COST_FREE
prompt_price = None
completion_price = None
currency_code = None
else:
(
cost_evidence,
prompt_price,
completion_price,
currency_code,
) = _normalize_cost_evidence(
route=route,
is_free=is_free,
prompt_price=row.get("prompt_price_per_1k"),
completion_price=row.get("completion_price_per_1k"),
prompt_price=prompt_price_input,
completion_price=completion_price_input,
currency_code=row.get("currency_code"),
)
Comment thread
seonghobae marked this conversation as resolved.
)
candidate_id = row.get("agent_id") or f"{provider}_{model}"
normalized.append(
{
Expand All @@ -197,6 +244,7 @@ def parse_discovery_report(report: Mapping[str, Any]) -> list[dict[str, Any]]:
"prompt_price_per_1k": prompt_price,
"completion_price_per_1k": completion_price,
"currency_code": currency_code,
"non_token_price_evidence": non_token_price_evidence,
"base_url": row.get("base_url") or PROVIDER_BASE_URLS[provider],
"credential_key": credential_key,
"auth_scheme": row.get("auth_scheme")
Expand Down Expand Up @@ -379,6 +427,7 @@ def build_zdr_prioritized_catalog(
"model": row["model"],
"agent_id": entry["id"],
"cost_evidence": _cost_evidence(row),
"non_token_price_evidence": row.get("non_token_price_evidence"),
"zdr": is_zdr_model(
str(row["provider"]),
model=str(row["model"]),
Expand Down
38 changes: 24 additions & 14 deletions scripts/ci/contextual_orchestrator_review_sidecar.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# (fail-closed zero-cost) pool.
set -euo pipefail

ORCHESTRATOR_PIN_SHA="${ORCHESTRATOR_PIN_SHA:-8cd99f139915131ba0239bce12a5d6a5fd85394e}"
ORCHESTRATOR_PIN_SHA="${ORCHESTRATOR_PIN_SHA:-045d17da5e2aea56a97e241ee158ab1628d78660}"
Comment thread
seonghobae marked this conversation as resolved.
ORCHESTRATOR_GIT_URL="${ORCHESTRATOR_GIT_URL:-https://github.com/ContextualWisdomLab/contextual-orchestrator.git}"
# The Strix gate and Noema SSRF guard accept this one process-local origin.
# Keep it fixed so an environment override cannot create an unvalidated sidecar.
Expand Down Expand Up @@ -108,7 +108,9 @@ log "installing hash-pinned orchestrator dependencies at ${checked_out}"
PYTHONPATH="$ORCHESTRATOR_SOURCE:$ORG_REPO_ROOT" "$sidecar_python" -c \
'from contextual_orchestrator.credentials import get_credential; from contextual_orchestrator.model_discovery import discover_all_models, free_discovered_models; from contextual_orchestrator.orchestrator import ModelClient, TaskOrchestrator, load_agents; from contextual_orchestrator.review_gateway import register_review_credentials; from contextual_orchestrator.server import SecurityConfig, serve'
PYTHONPATH="$ORCHESTRATOR_SOURCE:$ORG_REPO_ROOT" "$sidecar_python" - <<'PY'
import contextlib
import http.client
import io
import json
import threading

Expand All @@ -127,7 +129,9 @@ class CaptureClient(ModelClient):

def proxy_send(self, agent, endpoint, payload):
self.proxy_payloads.append(json.loads(json.dumps(payload, ensure_ascii=False)))
return super().proxy_send(agent, endpoint, payload)
# This contract exercises the loopback gateway only; provider egress
# would turn an offline startup check into an availability dependency.
return self._mock_raw(agent, endpoint, payload)
Comment thread
seonghobae marked this conversation as resolved.


client = CaptureClient()
Expand All @@ -145,19 +149,25 @@ thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
try:
connection = http.client.HTTPConnection("127.0.0.1", server.server_address[1], timeout=5)
connection.request(
"POST",
"/v1/chat/completions",
body=b"",
headers={
"Authorization": "Bearer contract",
"Content-Type": "application/json",
"Content-Length": str(REVIEW_MAX_BODY_BYTES + 1),
},
expected_rejection_log = io.StringIO()
with contextlib.redirect_stderr(expected_rejection_log):
connection.request(
"POST",
"/v1/chat/completions",
body=b"",
headers={
"Authorization": "Bearer contract",
"Content-Type": "application/json",
"Content-Length": str(REVIEW_MAX_BODY_BYTES + 1),
},
)
response = connection.getresponse()
assert response.status == 413, response.status
response.read()
assert (
"request_failed status=413 code=request_too_large"
in expected_rejection_log.getvalue()
)
Comment thread
seonghobae marked this conversation as resolved.
response = connection.getresponse()
assert response.status == 413, response.status
response.read()
connection.close()

def post_payload(payload):
Expand Down
70 changes: 70 additions & 0 deletions tests/test_contextual_orchestrator_bytez_catalog_integration.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
"""End-to-end contract for Bytez free-price discovery and catalog admission."""

from __future__ import annotations

from pathlib import Path
import runpy
from types import SimpleNamespace

import pytest

from scripts.ci import contextual_orchestrator_review_policy as policy

_REPO_ROOT = Path(__file__).resolve().parents[1]
_LAUNCHER = _REPO_ROOT / "scripts/ci/contextual_orchestrator_review_launcher.py"
_BYTEZ_MODEL = "0-hero/Matter-0.1-Slim-7B-C"


def _report_bytez(*, free: bool) -> list[dict[str, object]]:
"""Pass one pinned-runtime-shaped Bytez row through the real launcher adapter."""
report_rows = runpy.run_path(str(_LAUNCHER))["_report_rows"]
discovered = SimpleNamespace(
provider_name="bytez",
model_id=_BYTEZ_MODEL,
agent_id="bytez_matter_01_slim_7b_c",
chat_base_url="https://api.bytez.com/models/v2/openai/v1",
credential_name="BYTEZ_API_KEY",
auth_scheme="raw-token",
output_modalities=("text",),
prompt_price_per_1k=None,
completion_price_per_1k=None,
currency_code="USD",
)
free_routes = frozenset({("bytez", _BYTEZ_MODEL)}) if free else frozenset()
return report_rows([discovered], free_routes)


def test_zero_meter_price_survives_launcher_policy_and_catalog() -> None:
"""Exact-zero Bytez meter pricing must enter free without fake token prices."""
report_rows = _report_bytez(free=True)
assert report_rows[0]["is_free"] is True
assert report_rows[0]["prompt_price_per_1k"] is None
assert report_rows[0]["completion_price_per_1k"] is None

parsed = policy.parse_discovery_report({"models": report_rows})
assert parsed[0]["cost_evidence"] == policy.COST_FREE
assert parsed[0]["non_token_price_evidence"] == {
"source": "bytez.meterPrice",
"price": 0.0,
"unit": "provider_meter_unit",
}
assert parsed[0]["prompt_price_per_1k"] is None
assert parsed[0]["completion_price_per_1k"] is None

result = policy.build_zdr_prioritized_catalog(parsed, pool="free")
assert [agent["model"] for agent in result["agents"]] == [_BYTEZ_MODEL]
assert result["agents"][0]["credential_key"] == "BYTEZ_API_KEY"
assert "cost:free" in result["agents"][0]["tags"]
assert result["report"]["selected"][0]["non_token_price_evidence"] == (
parsed[0]["non_token_price_evidence"]
)


def test_unattested_bytez_meter_price_remains_unknown() -> None:
"""No free identity from the pinned parser means no Bytez free admission."""
parsed = policy.parse_discovery_report({"models": _report_bytez(free=False)})
assert parsed[0]["cost_evidence"] == policy.COST_UNKNOWN
assert parsed[0]["non_token_price_evidence"] is None

with pytest.raises(policy.PolicyError, match="would fail closed"):
policy.build_zdr_prioritized_catalog(parsed, pool="free")
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
)

GATEWAY_MODEL = "contextual-orchestrator/orchestrator/free"
ORCH_PIN_SHA = "8cd99f139915131ba0239bce12a5d6a5fd85394e"
ORCH_PIN_SHA = "045d17da5e2aea56a97e241ee158ab1628d78660"


def _read(path: Path) -> str:
Expand Down Expand Up @@ -360,6 +360,12 @@ def test_sidecar_probes_the_pinned_server_body_limit_at_http_boundary() -> None:
assert "accepted_size = 64 * 1024 + 1" in text
assert "REVIEW_MAX_BODY_BYTES + 1" in text
assert "assert response.status == 413" in text
assert "expected_rejection_log = io.StringIO()" in text
assert "with contextlib.redirect_stderr(expected_rejection_log):" in text
assert '"request_failed status=413 code=request_too_large"' in text
assert "in expected_rejection_log.getvalue()" in text
assert "return self._mock_raw(agent, endpoint, payload)" in text
assert "return super().proxy_send(agent, endpoint, payload)" not in text
assert "_request_body_size" not in text
assert "class CaptureClient(ModelClient):" in text
assert '"description": description' in text
Expand Down
Loading