Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions docs/doctoring/hourly-free-pool-policy-fixture-rca.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Hourly free-pool policy fixture RCA — 2026-09-01

## Scope

Protected `main` at `960b08456de4c87a5a833938220d6d83f68d61c1` failed `Hourly NVIDIA NIM Review Repair` run `33498263904`, job `99825357734`, in step `Verify hourly scheduler and NVIDIA NIM autofix contracts`.

## Exact failure evidence

The hosted pytest run produced two deterministic failures in `tests/test_contextual_orchestrator_review_policy.py` and then missed the 100% policy coverage gate:

- `test_build_catalog_applies_account_cap` still expected two `openai` rows to be admitted to the default `orchestrator/free` pool and raised `KeyError: 'openai'` after the rows were correctly excluded.
- `test_build_catalog_respects_limit` constructed its twenty free candidates entirely from `openai`; the post-#1587 policy correctly rejected that free-pool source set and raised `PolicyError: no free model route is available ... orchestrator/free would fail closed`.

The failure is therefore stale test-fixture evidence, not a product regression, provider/network transient, permission failure, or expected governance failure. The triggering protected-main commit is merge commit `960b08456de4c87a5a833938220d6d83f68d61c1` from PR #1587, whose intended contract keeps all provider credentials globally discoverable while admitting only `BYTEZ_API_KEY`, `NVIDIA_NIM_API_KEY`, `NVIDIA_NIM_API_KEY_SUB`, and `OPENROUTER_API_KEY` sources to `orchestrator/free`. PR #1587 added dedicated credential-boundary regressions but did not update these two older generic policy fixtures.

## Smallest repair

Change only the two generic fixtures so the behaviors they are intended to test—per-account limiting and total catalog limiting—use an authorized free-pool provider (`openrouter`) instead of the deliberately excluded `openai` source. No production policy, warning, security gate, review gate, coverage threshold, or fail-closed behavior is changed.

RED evidence is the protected-main run above. The repair commit is `7190562128067983c864fd56a3c4c13ea345a351`; compare against protected main is exactly three additions and three deletions in one test file.

## Related but separate blocker

Open PR #1591 is not a safe substitute for this fixture repair. Although its exact-head hourly self-test currently succeeds, unresolved independent review evidence shows that its admission-only catalog can return more than twelve agents while `contextual_orchestrator_review_launcher._bounded_fallback_catalog_limit()` still rejects `primary_count > 12`, aborting sidecar startup. That production-path issue must be repaired and re-reviewed separately rather than bypassed to make protected main green.

## Verification

Hosted exact-head Checks on the repair PR are authoritative. After merge, rerun the failed protected-main workflow and re-fetch the exact protected-main Checks. Pending, queued, skipped, or stale predecessor evidence is not treated as success.
6 changes: 3 additions & 3 deletions tests/test_contextual_orchestrator_review_policy.py
Original file line number Diff line number Diff line change
Expand Up @@ -354,7 +354,7 @@ def test_build_catalog_applies_account_cap() -> None:
for i in range(6)
]
+ [
{"provider": "openai", "model": f"o{i}", "agent_id": f"oa_{i}", "is_free": True, **FREE_PRICE}
{"provider": "openrouter", "model": f"o{i}", "agent_id": f"or_{i}", "is_free": True, **FREE_PRICE}
for i in range(3)
]
}
Expand All @@ -367,14 +367,14 @@ def test_build_catalog_applies_account_cap() -> None:
account_counts[account] = account_counts.get(account, 0) + 1
assert account_counts["nvidia_nim"] == 2
assert account_counts["nvidia_nim_sub"] == 2
assert account_counts["openai"] == 2
assert account_counts["openrouter"] == 2


def test_build_catalog_respects_limit() -> None:
"""The catalog never exceeds the configured agent limit."""
report = {
"models": [
{"provider": "openai", "model": f"m{i}", "agent_id": f"oa_{i}", "is_free": True, **FREE_PRICE}
{"provider": "openrouter", "model": f"m{i}", "agent_id": f"or_{i}", "is_free": True, **FREE_PRICE}
Comment thread
seonghobae marked this conversation as resolved.
for i in range(20)
]
}
Expand Down
Loading