ops(llm): #2582 has recurred — hp-llm-worker unhealthy for ~11h, and the #1947 phase-2 sweep is running contended, not cold

[Xore]

Found while checking on the #1947 phase-2 sweep. Two coupled problems, one live incident and one that affects how a benchmark gets published.

1. #2582 has recurred

hp-llm-worker is unhealthy with FailingStreak=3 and has been unable to get
qwen3:14b resident for about eleven hours:

69 x  msg="Load failed" model=<qwen3:14b blob> error="timed out waiting for llama-server to start: context canceled"
first 2026-09-05T04:04:50Z   last 2026-09-05T14:47:49Z

ghidra-ollama-1 runs OLLAMA_MAX_LOADED_MODELS=1, and the #1947 sweep has the
slot — currently GLM-4.6-REAP-218B-A32B:i1-IQ1_S at 57 GB served, 65%/35%
CPU/GPU. Every ~4 minutes the worker asks for a different model, cannot get it,
and gives up.

This is #2582 (llm-worker permanently unhealthy: shared ollama is saturated by the ghidra/revdeck eval zoo) reproducing exactly. It closed 2026-08-28.

Mitigated now by stopping hp-llm-worker for the duration of the sweep. It
was producing nothing — eleven hours of consecutive load failures — so stopping
it loses no capability and removes the contention. RestartPolicy=unless-stopped
respects the manual stop, and the autoheal: true label only acts on running
unhealthy containers, so it will stay down until deliberately restarted.
ghidra-revdeck-1 was checked and is idle (healthz only); it was left running.

2. The harder half: phase 2 is not running the cold-slot protocol

sweep_extra.sh does ollama stop "$TAG" before each run, and nothing else. The
cold protocol that #2641 established — and that the whole part-4 re-measurement
rests on — additionally requires hp-llm-worker and ghidra-revdeck-1 stopped
so nothing else touches the GPU. Phase 2 has been running without that.

The contention is not theoretical. From ghidra-ollama-1's own access log:

14:47:46 | 200 | 3m57s | 172.16.38.1 | POST /v1/chat/completions   <- the benchmark
14:47:49 | 499 | 3m46s | 172.16.3.4  | POST /api/chat              <- hp-llm-worker, cancelled
14:47:59 | 500 | 1m39s | 172.16.3.4  | POST /api/chat              <- hp-llm-worker, failed

Benchmark requests and worker requests are interleaved on the same single slot.

Why this matters for the matrix

#2642/#2644/#2646 established that a warm or contended slot does not
reproduce while a cold one does: qwen3:14b measured 60/61 warm with 1 of 14
answers byte-identical, and 60/60 cold with 14 of 14. The #1805-c survey had to
be demoted to a survey for exactly this reason, and its uniform ±0 turned out
to be lane contention masquerading as determinism.

Phase 2 has 12 escalations (cells where run1 != run2 and the sweep went to
N=3), against phase 4's cold cells which were ±0 across three runs on four
architectures. That is consistent with contention — though for the
GLM-4.6-REAP-218B rows specifically, 65% CPU offload is an alternative
explanation (float reduction order across threads), so this should not be stated
as proven.

The decision this forces

36 of 52 phase-2 models are already measured under contention. Stopping the
worker now makes the remainder cold, which means phase 2 would span two
regimes. Options, none free:

• A — annotate. Publish phase 2 as a contended-regime survey, explicitly not
  comparable to phase 4's cold cells. Cheapest, and honest, but it reproduces the
  Benchmark fidelity: run real Ghidra over the #159 corpus so models are scored on production-shaped input #1805-c outcome the epic exists to avoid.
• B — split. Mark the 36 contended rows and the remainder cold, in two
  clearly separated blocks. Accurate but awkward to read and to compare.
• C — re-run phase 2 cold. Correct, and roughly two days of GPU.

Recommendation: decide before the sweep finishes, because C gets cheaper the
earlier it is chosen and the 12 escalations are the only direct evidence either
way. Whatever is chosen, sweep_extra.sh should gain the worker-stop step so
this cannot recur silently — it is the one part of the cold protocol the script
does not implement.

Scope

• Decide A / B / C for phase 2's regime.
• Add the live-worker stop/restore to sweep_extra.sh (committed copy at
  analysis/ghidra/benchmarks/corpus/sweep_extra.sh), so the cold protocol is
  enforced by the driver rather than by an operator remembering.
• Restart hp-llm-worker when the sweep ends, and give llm-worker permanently unhealthy: shared ollama is saturated by the ghidra/revdeck eval zoo (qwen3:14b can't stay resident) #2582 a regression
  guard so an eleven-hour unhealthy streak surfaces on its own rather than
  being found by someone looking at something else.
• Consider whether OLLAMA_MAX_LOADED_MODELS=1 plus a shared slot is the
  right shape at all while a multi-day benchmark is scheduled.

Refs #1947, #2582, #2641, #2642, #2646, #2644

[Xore]

Claiming for round-7 batch A live-ops sweep. Will re-verify against the live rebuilt host before making any change; will post progress here.

[Xore]

Parked rather than worked: confirmed live (18:01, during this batch) that a benchmark process (record_bench...) is actively running against ghidra-ollama-1's single loaded-model slot right now, from a concurrent session — not a stale snapshot from when this issue was filed. Every remaining item in this issue's scope (restarting the worker, the A/B/C regime decision, editing sweep_extra.sh) touches that same contested surface, so none of it was safe to act on without risking the in-flight run.

hp-llm-worker is confirmed fully absent (not just stopped) and Arcane agrees (status: stopped, running_count: 0) — consistent with the issue's own mitigation, just never reverted. Split out the one non-benchmark-adjacent piece of scope (a regression guard so this doesn't need another accidental discovery next time) as #3030.

Leaving this open until whoever owns the #1947 sweep thread confirms it's done, restarts the worker, and makes the A/B/C call — not closing this from this batch. Full detail in the batch handover doc.

[Xore]

Measured: contention did not corrupt phase 2's scores. No re-run needed.

Rather than argue from the 12 escalations, I ran the two-model cold probe in the
gap that opened when the sweep stopped itself (#3031). Same harness, same pin
a99e765, same corpus, hp-llm-worker down, sequential, ollama stop before
every run. Artefacts in /mnt-1/benchmarks/coldprobe/, driver committed as
coldprobe.sh.

model	tier	contended	cold
DeepHat-V1-7B:Q4_K_M — 4.7 GB, fully GPU-resident	A	[63, 63]	[63, 63]	identical
ravenx-cyberagent-35b:Q4_K_M — 21 GB, spills to CPU	B	[64, 63, 64]	[64, 64, 64]	escalation resolved to the same majority

The control is the load-bearing row. A fully-resident model has no
CPU-offload nondeterminism, so if contention moved scores it would show there.
It reproduced exactly.

The escalated cell resolved to the value the protocol had already chosen.
Contended [64, 63, 64] → majority 64; cold [64, 64, 64]. The 63 was a
single-run artifact, and the N=2→N=3 escalation caught and corrected it. So the
escalations cost extra GPU, not accuracy.

Every escalated cell with three runs, checked

A  codellama_7b-instruct-q4_K_M          [57, 58, 58] -> 58  (2/3)
A  qwen3_8b                              [64, 65, 65] -> 65  (2/3)
B  gemma-4-31B-it-qat-q4_0-heretic       [62, 63, 63] -> 63  (2/3)
A  pki/nova-24b-cybersec                 [58, 59, 58] -> 58  (2/3)
A  Foundation-Sec-8B-Instruct-Q8_0       [61, 62, 61] -> 61  (2/3)
A  Foundation-Sec-1.1-8B-Instruct-Q8_0   [62, 64, 62] -> 62  (2/3)
B  ravenx-cyberagent-35b                 [64, 63, 64] -> 64  (2/3)   <- probed, cold agrees
A  GLM-4.6-REAP-218B:i1-IQ1_S            [55, 54, 55] -> 55  (2/3)
B  GLM-4.6-REAP-218B:i1-IQ1_S            [55, 57, 54] -> NO MAJORITY

Eight of nine resolve 2-of-3. The ninth does not resolve at all — three
distinct values, so N=3 cannot produce a majority and that cell currently has
no defensible score. It is also the most CPU-offloaded row in the roster (57 GB
served, 65%/35% CPU/GPU), which points at spill nondeterminism rather than
contention. It needs N>=5 or an explicit UNRESOLVED mark; it must not be
published as 55, 57 or 54.

That is a gap in the escalation protocol itself, independent of this issue:
sweep_extra.sh escalates once and then stops, with no handling for "still no
majority".

Recommendation

Option A — keep phase 2 and annotate. Not B, not C. The two-day re-run buys
nothing the probe did not just establish for free:

• resident-model scores are unaffected by contention (exact reproduction);
• the one escalated cell probed cold agrees with its contended majority;
• the escalation protocol demonstrably did its job on 8 of 9 cells.

Annotate the matrix with: phase 2 ran with hp-llm-worker live; escalated cells
were resolved by N=3; wall-clock columns from phase 2 are not comparable to
phase 4's and should not be used as a speed measure.

Honest limits of this probe

Two models, not 52. The control is strong evidence for resident models and the
raven is one spilling model — a third architecture would strengthen it, and a
model whose contended majority was itself split (the GLM Tier B row) is exactly
the case the probe does not cover. If anything later depends on a margin of 1
point on a spilling row, re-probe that row rather than trusting this.

Also done

hp-llm-worker stays down until phase 2 finishes — it was failing anyway, and
keeping it down removes the escalation cost even though it does not change
scores. It must be restarted afterwards; that is still on this issue's checklist.

[Xore]

Review verdict: PARK (confirmed) — re-verified independently at 20:27, not from the handover's 18:01 snapshot: sweep_extra.sh is still running, a record_bench run started 19:56, VRAM samples are seconds old and the single slot holds a 44 GB model, so the park premise still holds and hp-llm-worker is genuinely absent rather than merely stopped. Note the A/B/C decision was made on this issue 15 minutes after the park (Option A), leaving two open items: restart the worker once phase 2 finishes, and annotate the matrix. Filed #3041 for the escalation-protocol gap that comment surfaced.

[Xore]

Park re-confirmed at 23:13 CEST, and the A/B/C decision folded in

Re-checked the park's premise rather than carrying forward the earlier snapshot:

$ ps aux | grep -iE 'record_bench|sweep_extra'
xore 2304417  18:14  bash sweep_extra.sh                       <- still running

$ docker exec ghidra-ollama-1 ollama ps
hf.co/mradermacher/XORTRON.CriminalComputing.LARGE.2026.3-i1-GGUF:i1-IQ2_XXS  44 GB  56%/44% CPU/GPU

$ ls -lat /mnt-1/benchmarks/ | head -3
-rw-r--r--. 1 xore xore  42759 Sep  5 23:12 vram_samples.tsv
-rw-rw-r--. 1 xore xore  86391 Sep  5 23:09 extra.log

$ docker inspect hp-llm-worker
error: no such object: hp-llm-worker

VRAM samples are a minute old and the single loaded-model slot is occupied by a 44 GB model.
The surface every item in this issue's scope reaches into is contended right now, so the park
stands unchanged.

The A/B/C question is no longer open. It was answered on this issue at 16:17Z — Option A,
keep phase 2 and annotate — on the evidence that resident-model scores reproduce exactly under
contention and that the one probed escalated cell agrees cold. Recording it here so the park
note does not still read as if that decision were pending.

That leaves exactly two residual items, both still blocked on the same thing:

1. Restart hp-llm-worker once phase 2 finishes. It is absent, not merely stopped
   (Arcane's own record: status: stopped, running_count: 0), which is why nothing has
   restarted it and rules out an autoheal loop.
2. Annotate the matrix per Option A: phase 2 ran with hp-llm-worker live; escalated cells
   resolved by N=3; phase-2 wall-clock columns are not comparable to phase 4's.

Before acting on either, check ps aux | grep -iE 'ghidra|benchmark' and
ls -lat /mnt-1/benchmarks/ for sub-minute-old files — do not restart speculatively.

Related splits already filed: #3030 (surfacing a long unhealthy streak on its own — a
monitoring gap, not a benchmark decision) and #3041 (sweep_extra.sh escalates once with no
handling for "still no majority"; the GLM-4.6-REAP-218B:i1-IQ1_S Tier B cell is [55, 57, 54]
and must not publish as any of the three).