ops(dns): stale container resolver order cost the #1947 sweep 12 models, and sweep_extra wrote a false EXTRA_COMPLETE over it

[Xore]

Two defects, one incident. The DNS half is #2974 living on inside long-running
containers; the bookkeeping half is the false-completion class that #2971's
review already caught once on the pull path and once on the Tier B path.

What happened

Between 15:42:49Z and 15:47:09Z — four and a half minutes — the #1947 phase-2
sweep failed to pull twelve consecutive models, each in about 20 seconds, and
then wrote EXTRA_COMPLETE:

15:42:49 PULL_FAILED …/Qwen3.6-27B-uncensored-heretic-v2-Native-MTP-Preserved-i1-GGUF:i1-Q4_K_M
15:43:10 PULL_FAILED …/Glimmer-Sentry-30B-GGUF:Q4_K_M
15:43:30 PULL_FAILED …/ThinkingCap-Qwen3.6-27B-MTP-GGUF:latest
15:44:12 PULL_FAILED …/Chimera-Qwen3.8-27B-Cyber-v1-GGUF:Q4_K_M
15:44:33 PULL_FAILED …/Colibri_8b_v0.1-GGUF:Q4_K_M
15:44:53 PULL_FAILED …/NeoBase_8b_v0.2-GGUF:Q4_K_M
15:45:26 PULL_FAILED …/LOREA-cyber-v5.8:latest
15:45:47 PULL_FAILED …/Colibri_8b_v0.1-GGUF:f16
15:46:07 PULL_FAILED …/Ornith-1.0-35B-uncensored-heretic-GGUF:Q3_K_M
15:46:28 PULL_FAILED …/gemma-4-26B-A4B-it-ultra-uncensored-heretic-GGUF:Q3_K_M
15:46:49 PULL_FAILED …/gemma-4-26B-A4B-it-ultra-uncensored-heretic-GGUF:Q4_K_M
15:47:09 PULL_FAILED …/gemma-4-26B-A4B-it-ultra-uncensored-heretic-GGUF:Q5_K_M
15:47:09 EXTRA_COMPLETE

All twelve carry the same error:

dial tcp: lookup hf.co on 127.0.0.11:53: read udp …->127.0.0.11:53: i/o timeout

Three of those twelve are the gemma-4-26B-A4B Q3/Q4/Q5 quant ladder — the
promote candidate from PR #2641, i.e. the highest-value cells left in phase 2.

Root cause: the container resolver still has the pre-#2974 order

#2974 fixed the host, which now correctly puts the Technitium cluster first:

$ grep ^nameserver /etc/resolv.conf
nameserver 192.168.42.253
nameserver 192.168.42.250
nameserver 1.1.1.1
nameserver 8.8.8.8

ghidra-ollama-1 has been up 30 hours — since before that fix — and Docker's
embedded resolver snapshots the host list at container start:

$ docker exec ghidra-ollama-1 grep ExtServers /etc/resolv.conf
# ExtServers: [host(1.1.1.1) host(8.8.8.8) host(192.168.42.253) host(192.168.42.250)]

Public resolvers first. And port 53 egress to them is blocked on this LAN —
measured, not assumed:

1.1.1.1          TIMEOUT/blocked
8.8.8.8          TIMEOUT/blocked
192.168.42.253   responds

So every container lookup burns the timeout on two dead servers before falling
through to a working one. Measured cost inside the container: ~8 seconds for a
single getent hosts. Ollama's pull has a 30 s per-request deadline (#2728), so
under any additional latency the pull dies — and it dies deterministically,
which is why twelve models fell in a row rather than intermittently.

It resolves eventually, which is why nothing noticed until it didn't.

Second defect: EXTRA_COMPLETE is written unconditionally

sweep_extra.sh:159 writes EXTRA_COMPLETE when the roster loop ends, regardless
of how many entries failed. This is the third appearance of the same class:

• ops: /mnt-1/benchmarks (the #1947 sweep's whole work area) was wiped by the homeserver rebuild; recovered from a hermes mirror #2971 review — false completion on the pull path
• ops: /mnt-1/benchmarks (the #1947 sweep's whole work area) was wiped by the homeserver rebuild; recovered from a hermes mirror #2971 review — reintroduced on the Tier B path
• here — a whole roster tail lost to infrastructure, still marked complete

mark_unmeasured() did its job and wrote UNMEASURED_<slug>.status for all
twelve, so the rows are not silently scored as zero. But EXTRA_COMPLETE is what
the phase chain keys on, so the marker is the only thing standing between this and
"phase 2 finished".

A completion marker should encode the failure count, or not be written at all
when the failure rate exceeds a threshold. Twelve consecutive PULL_FAILED in
four minutes is not a completed roster under any reading.

Recovery

• Restart ghidra-ollama-1 so its embedded resolver picks up the corrected
  host order — and verify ExtServers afterwards rather than assuming.
• Better: pin dns: explicitly in the ghidra compose so a container's resolver
  order does not depend on when it happened to start. Every other long-running
  container on this host that predates ops(dns): rebuilt homeserver resolves through 1.1.1.1/8.8.8.8 ahead of the Technitium cluster — internal dmz.xore.lan names stop resolving, and #2909's blocklist is masked not fixed #2974 has the same latent fault.
• Clear the twelve UNMEASURED_*.status markers — they record an
  infrastructure failure, not a model property, and must not reach the matrix
  as "unmeasurable" beside genuine cases like GPT-OSS-Cybersecurity-20B-Merged-heretic-i1 emits zero tokens under every request shape (not the #2279 harmony bug) #2696/ml: three more roster models emit zero tokens on Tier A — XORTRON 4B/27B .NEXT and CyberPal2.0-20B (same class as #2696) #2982.
• Reset the false EXTRA_COMPLETE and re-run the sweep for those twelve.
• Add a failure-rate guard to sweep_extra.sh before EXTRA_COMPLETE
  (committed copy at analysis/ghidra/benchmarks/corpus/sweep_extra.sh).

Refs #1947, #2974, #2971, #2985, #2728, #2641

[Xore]

Fixed, and hardened past the original scope

The operator's call: only 192.168.42.253 and 192.168.42.250 should ever be
consulted. The public resolvers were not a fallback — port 53 to them is
blocked on this LAN, so they were pure latency in every lookup path.

What was actually wrong

NetworkManager owned /etc/resolv.conf, and the ens9f0 connection carried:

ipv4.dns = 192.168.42.253,192.168.42.250,1.1.1.1,8.8.8.8
ipv4.ignore-auto-dns = no

So #2974's host fix had only reordered the list, not removed the dead entries —
which is why the container inherited a working-but-slow resolver and the fault
stayed latent until a pull deadline exposed it.

Change

nmcli connection modify ens9f0 ipv4.dns "192.168.42.253 192.168.42.250" \
                               ipv4.ignore-auto-dns yes
nmcli connection modify eno1   ipv4.ignore-auto-dns yes
nmcli device reapply ens9f0

ignore-auto-dns on both interfaces matters as much as the list itself: without
it a DHCP lease can put a resolver back and nothing would announce it.

Result:

# /etc/resolv.conf
search dmz.xore.lan
nameserver 192.168.42.253
nameserver 192.168.42.250

Verified rather than assumed

• Host resolution still works — hf.co resolves, and search dmz.xore.lan
  is intact for internal names.
• New containers inherit the two-server list — checked by running a throwaway
  container and reading its /etc/resolv.conf, not by inference.
• ens9f0 never dropped. nmcli device reapply was driven from the
  192.168.42.248 (eno1) path, which the operator supplied as backup access,
  so a link bounce could not have cut the session administering it. That path is
  now a named homeserver-alt alias rather than something to reconstruct under
  pressure.
• Lookup latency inside the ollama container went 8 s → 0.067 s after the
  earlier restart.

One residual, deliberately not acted on

ghidra-ollama-1 still shows the 4-entry ExtServers from its restart earlier
today (Technitium first, so it is fast). It picks up the two-server list on its
next restart. Restarting it again now would interrupt the running phase-2 sweep
for no functional gain, so it waits for a natural boundary.

Still open on this issue

The DNS half is closed. The bookkeeping half is not:

• Pin dns: in the ghidra compose so resolver order never depends on when a
  container started. Every long-running container predating ops(dns): rebuilt homeserver resolves through 1.1.1.1/8.8.8.8 ahead of the Technitium cluster — internal dmz.xore.lan names stop resolving, and #2909's blocklist is masked not fixed #2974 still has
  the stale list, and this fix does not reach them until they restart.
• Failure-rate guard before EXTRA_COMPLETE in sweep_extra.sh — twelve
  consecutive PULL_FAILED in four minutes must not be able to write a
  completion marker. That is the third appearance of this class and the one
  that actually cost the roster tail.

[Xore]

Closing — both halves of this incident are fixed, and both are verified where they had to be

This issue carried two defects. Each is closed on its own evidence, checked 2026-09-10.

1. DNS half — fixed live. The operator's call was that only 192.168.42.253 and 192.168.42.250 should ever be consulted; the public resolvers were pure latency because port 53 to them is blocked on this LAN. NetworkManager owned /etc/resolv.conf and ens9f0 carried the public entries. Live now:

$ cat /etc/resolv.conf | grep nameserver
nameserver 192.168.42.253
nameserver 192.168.42.250

Nothing else. This is a host-config fix, so it could never have been closed by a Closes #N on a merge — verified on the host directly.

2. Bookkeeping half — fixed in the committed driver. The false EXTRA_COMPLETE class is now structurally impossible in sweep_extra.sh: a failed pull writes a PULL_FAILED line to failures.txt and calls mark_unmeasured "$slug" "$TAG" "pull_failed: $reason" (line 218), so twelve consecutive pull failures produce twelve UNMEASURED markers rather than twelve silent holes under a completion stamp. The pulled-but-every-run-failed variant is covered too (line 305, mark_unmeasured ... "pulled but every run failed on both tiers"), with the code comment naming this issue as the class it closes:

# a GIVEUP line in failures.txt and a MODEL_DONE in the log -- indistinguishable
# silent-hole class as the false EXTRA_COMPLETE in #3031.

This matches the round's standing rule that UNMEASURED markers are never written as zeros.

Closing as delivered. Round-10 audit.