Strix run ended "cancelled" at 6h with no verdict: 909 provider attempts, 92% on one route, 0 exhaustion events, request identity absent from evidence

[claude]

What happened

strix on .github#1187@541cadd1 ran from 15:19:48Z to 21:20:05Z and was cancelled at exactly 6 h 00 m 17 s — GitHub Actions' hard per-job ceiling, not any policy timeout. The required check is now cancelled, which is not a verdict at all: no findings, no gate decision, and a required context that branch protection cannot interpret.

Job 101503665803; every step succeeded through Prepare Strix model input file, and Run Strix (quick) is the cancelled one. Evidence artifact strix-reports (id 9997372950) uploaded successfully, so the whole run is inspectable.

The run never had a termination condition

From contextual-orchestrator-sidecar.stderr.log (2,691 lines, 15:25:29Z → 21:19:46Z):

event	count
provider_attempt (starts)	909
provider_attempt_failed	603
error_type=TimeoutError	541
error_type=HTTPError	62
request_failed status=500 code=internal_error	536
provider_exhausted	0
circuit_failure	55
circuit_opened	3
circuit_cleared	43

The final log line is a new provider_attempt at 21:19:46, 19 seconds before the platform killed the job. Nothing in the run was converging; it was stopped from outside.

92% of all attempts went to one route. Attempts per agent_id:

agent_id	attempts
nvidia_nim_deepseek_ai_deepseek_v4_flash_0731	837
nvidia_nim_sub_deepseek_ai_deepseek_v4_pro_0813	55
nvidia_nim_deepseek_ai_deepseek_v4_pro_0813	4
every other agent (13 of them)	1 each

The 13 single-attempt rows are the preflight probes. After preflight, the request path re-selected one already-failing route 837 times over six hours.

This is not in-call retry: attempt= is 1/1 on 904 of 909 attempts (the other five are one 1/3→2/3 sequence). The per-call retry budget is 1. The repetition comes from the caller re-selecting the same route after each failure, and provider_exhausted never firing once in 837 consecutive failures on it.

Sustained, not a tail-end artifact — attempts per hour: 15:00 51 (partial hour), 16:00 258, 17:00 196, 18:00 204, 19:00 146, 20:00 41, 21:00 13.

The circuit breaker did not keep the dead route out

55 circuit_failure produced only 3 circuit_opened, against 43 circuit_cleared. An earlier run on this same pool logged circuit_opened … threshold=3 reset_seconds=30.0. A 30-second reset against a failure cycle whose dominant mode is a ~90-second socket idle timeout means the breaker re-admits the same route roughly every cycle, so it records failures without ever excluding the route for a meaningful interval. I am reporting the observed counts and that outcome; I have not read the selection code closely enough to assert which specific branch decides re-admission.

Why this is not the same issue as #1915

#1915 tracks the free pool having no provider-family diversity, and this run reproduces that again — third independent confirmation today:

"candidate_count": 24, "ready_count": 5, "target_ready": 8

with all five ready routes on nvidia_nim / nvidia_nim_sub, provider_discovery_failed provider=bytez code=http_status_500, and both OpenRouter free routes deferred.

But the two are separable. Restoring provider diversity would change which routes get hammered, not the fact that a degraded pool produces an unbounded retry loop. An upstream-wide outage with a perfectly diverse pool yields the same six-hour burn. Conversely, a bounded exhaustion condition would turn this run into a fail-closed verdict in minutes regardless of pool composition.

Consequences already observable

1. cancelled required checks. Sweep pre-#1669 PRs stuck with a wrongly-cancelled current-head Strix/OpenCode/Noema check #1756 is already sweeping PRs stuck behind wrongly-cancelled Strix/OpenCode/Noema checks. This is a live production path that creates them.
2. Runner occupancy. Measured 2026-09-06T18:1x on this repository: status=in_progress = 18 runs, 14 of them Strix Security Scan, aged 97–381 minutes, against status=queued = 150 with the oldest waiting 39 minutes. If each stuck scan runs to the 6-hour ceiling, that is a large fraction of the runner pool held by runs that will produce no verdict. Related but distinct from ops: three required workflows each boot a runner for the same "Detect changed scope" job #1976.

What this issue is not asking for

Not a wall-clock timeout on the model path. docs/product-goal-directive.md §8 accepts that central OpenCode/Strix/Noema may take more than two hours per model, and #1889/#1890/#1892 each added a 900-second cap on genuine multi-hour-hang evidence and were all reverted (#1891, #1895). Elapsed inference time must not become a model-failure verdict, and nothing here argues otherwise: a single 90-second timeout is a normal event, and 837 of them is not a slow model.

The lever is attempt/route exhaustion, which is orthogonal to elapsed time:

• provider_exhausted firing after a bounded number of consecutive failures on the same agent_id, so the selector stops re-picking it.
• A breaker reset window that is not shorter than the failure cycle it is meant to damp.
• A terminal condition when every ready route has been exhausted, so the gate fails closed with evidence instead of being killed with none.

All three bound attempts, not duration. A route that is genuinely slow but progressing is untouched by any of them.

Reproduction

• Artifact strix-reports id 9997372950 on run 34036172117 (contextual-orchestrator-preflight.json, contextual-orchestrator-sidecar.stderr.log, gate-console.log).
• Same-day second sample with the same pool shape: noema-review on docs: confirm review pipeline already routes through orchestrator/free, not NIM directly #1884@396b4dee, artifact noema-sidecar-evidence id 9994541963 — ready_count: 6, all nvidia_nim/nvidia_nim_sub, ends request_failed status=502 code=provider_connection_error.

---

Generated by Claude Code

[claude]

Sharpening my own "runner occupancy" point before anyone acts on it

The consequences section above pairs "18 in_progress, 14 of them Strix" with "150 queued, oldest waiting 39 minutes." Those numbers are what the API returned at 18:1x, but putting them side by side invites the inference that the long Strix runs are what makes the queue deep. A second sample says they are not, and I would rather correct that here than have it steer the fix.

Measured 2026-09-07T00:07Z on this repository:

	2026-09-06 18:1x	2026-09-07 00:07
in_progress	18 (14 Strix)	6 (5 Strix + 1 Noema)
queued	150	161
oldest queued	39 min	32 min

The queue is being served normally throughout: median queued age 14.1 min, newest 0.4 min, last success concluded 6.6 min ago and last failure 0.6 min ago. Queue depth is dominated by arrival rate, and it grew while concurrent Strix runs fell from 14 to 5 — so it is not a backlog behind the long jobs.

What the long jobs actually cost is unchanged and is what this issue is about: each of the six currently running has held a slot for 81–205 minutes, and on the #1187 evidence a scan on a degraded pool holds one until the 6-hour ceiling kills it with no verdict at all. The waste is the slot-hours and the missing gate decision, not queue latency for everything else.

This does not change the defect, the reproduction, or the requested remedy — attempt/route exhaustion rather than an elapsed-time cap. It only removes a claim the evidence does not support.

---

Generated by Claude Code

[seonghobae]

독립 집계에 따른 주장 범위 정정

CO 담당 작업이 기존 artifact 9997372950을 다운로드해 집계했고, 이 작업은 그 감사 receipt를 직접 읽었습니다. 설치 CO는 414f22973658c4ddc3d4320fcf7acd9b4e8ba991, stderr SHA256은 ce5ed2846ecf284674cf5c6d0d80e674e79bfc4e58c27f40c878fde3bc25d70a입니다.

909 starts / 603 recorded failures / 541 TimeoutError / 62 HTTPError / 536 internal_error / provider_exhausted 0 및 circuit 55·3·43 집계는 일치합니다. 다만 주된 route는 837 starts와 564 recorded failures입니다. 따라서 본문의 '837 consecutive failures'는 입증되지 않았습니다. 나머지 273 starts를 성공이나 실패로 임의 분류해서는 안 됩니다.

stderr request_id 필드는 0개이고 gate-console은 2줄, stdout은 비어 있습니다. 이 자료만으로 caller retry, 독립·동시 logical request, 요청 내부 fallback을 구분할 수 없습니다. 소진 이벤트 부재만으로 무한 caller loop를 확정할 수도 없습니다. tail의 30초 circuit reset 뒤 재허용은 관측되지만 전체 반복의 원인이나 올바른 reset 정책을 증명하지는 않습니다.

취소 시각 6h00m17s 또한 플랫폼 취소 주체의 직접 증거와 구분해야 합니다. 다음 조사는 exact-source의 caller/route exhaustion 계약과 요청 정체성을 보존한 runtime evidence입니다. 경과 시간 상한 추가, 공급자 재호출, 재실행 또는 기존 실행 취소는 하지 않았습니다.

[claude]

Independently reproduced, and three of this issue's claims retracted

Re-derived from the same artifact on this side. The audit boundary matches exactly:

stderr SHA256 = ce5ed2846ecf284674cf5c6d0d80e674e79bfc4e58c27f40c878fde3bc25d70a

quantity	this issue's body	your count	re-derived now
total provider_attempt starts	909	909	909
total provider_attempt_failed	603	603	603
main-route starts	837	837	837
main-route recorded failures	—	564	564
main-route starts − failures	—	273	273
request_id occurrences	—	0	0

You are right and the issue body is wrong. Retracting three claims:

1. "837 consecutive failures" is not supported. 837 is the start count; 564 are recorded failures on that route, and 273 starts have no recorded outcome. The log emits no success-shaped event at all (provider_attempt, provider_attempt_failed, request_failed, circuit_*, provider_backoff, provider_discovery_failed, discovery_diagnostics_complete are the only event kinds present), so those 273 cannot be classified in either direction. I should not have written a failure count I had not counted.

2. "the repetition is the caller re-selecting a dead route rather than in-call retry" is not established. attempt=1/1 bounds the per-call retry budget and nothing more. With zero request_id fields there is no way to group 909 starts into logical requests, so caller retry, independent concurrent requests, and in-request fallback are indistinguishable from this evidence. The title's "has no exhaustion condition" inherits the same defect: provider_exhausted: 0 is an absence, and an absence of exhaustion events is not proof of an unbounded loop.

3. "killed by the 6-hour Actions ceiling" overstates attribution. 15:19:48Z → 21:20:05Z = 6 h 00 m 17 s and conclusion: cancelled are consistent with the platform ceiling, but I have no direct evidence naming the canceller and should have said so.

I have retitled the issue accordingly.

What the evidence still supports, unchanged

• One job produced 909 provider attempt starts and 603 recorded failures, of which 541 are TimeoutError and 62 HTTPError, alongside 536 request_failed status=500 code=internal_error.
• 92% of starts (837/909) went to one agent_id. That is a distribution fact about starts and needs no outcome classification.
• provider_exhausted appears 0 times; circuit_failure 55, circuit_opened 3, circuit_cleared 43, circuit_reset 3.
• The preflight for this run: candidate_count: 24, ready_count: 5, target_ready: 8, all five ready routes on nvidia_nim/nvidia_nim_sub ([CO owner] eliminate orchestrator/free provider-family SPOF before gateway release #1915).
• The outcome, which holds regardless of who cancelled it: a required check ended cancelled after six hours with no findings and no gate decision. cancelled is not a verdict, and Sweep pre-#1669 PRs stuck with a wrongly-cancelled current-head Strix/OpenCode/Noema check #1756 is already sweeping PRs stuck in that state.

On the next investigation

Agreed on exact-source caller/route exhaustion contracts plus runtime evidence that preserves request identity. The concrete blocker is the request_id row above: zero occurrences is exactly what makes this undiagnosable from artifacts. Until a correlation id is emitted per logical request and carried onto every provider_attempt / provider_attempt_failed line, any future run produces the same ambiguity and the same argument. That seems worth doing before, not after, the contract work — it is what would let the next run answer the question this one cannot.

Noting for the record that neither side has added an elapsed-time cap, re-called providers, re-run, or cancelled anything. This issue does not ask for a time cap; if what remains after these retractions is judged too thin to act on, closing it in favour of a narrower evidence-instrumentation issue would be a reasonable outcome.

---

Generated by Claude Code