ops: RAID VD backing /mnt-1 went OFFLINE — two-drive fault in one enclosure, XFS shut down

[Xore]

What happened

/mnt-1 (XFS on /dev/sdb1, the RAID virtual disk the controller logs as VD 00/0) went offline on homeserver tonight. Discovered while diagnosing why PR #3157's backend-service CI check failed — that failure is a symptom of this, not a regression of #3105.

Timeline from dmesg -T / journalctl -k, all today 2026-09-09:

• 21:16:22 — write I/O error on sdb1; controller logs VD 00/0 is now OFFLINE (FATAL); XFS: log I/O error -19 → Filesystem has been shut down due to log error (0x2). Please unmount the filesystem and rectify the problem(s).
• 21:17:42 — Enclosure PD 44(c None/p1) phy bad for slot 3 (CRIT)
• 21:21:21 — Unable to access device PD 0c(e0x44/s2) (FATAL) — a second, different physical drive

megacli -PDList -aAll right now confirms two drives down in the same enclosure:

• Slot 2 (Device Id 12): Unconfigured(bad), Raw Size 0 KB
• Slot 3: absent from the PDList output entirely (not enumerating — consistent with the "phy bad" message)
• Slots 0, 1, 4, 5, 6, 7: Online, Spun Up, 0 media errors — healthy

Two drives failing ~1 minute apart in the same enclosure is enough to take a RAID5 VD offline (single-parity tolerates one failure). megacli -LDInfo -Lall -aAll no longer enumerates VD 0 at all — only Virtual Drive: 1 (Target Id 1, RAID5, 7TB, backing sdc1//var) shows, and that one is mid-Reconstruction (35% complete, 153 min elapsed) despite reporting State: Optimal. Flagging the concurrent reconstruction as possibly related to the same enclosure/backplane/power event — not confirmed, /var itself is not currently erroring.

Current filesystem state is inconsistent depending on syscall path: mount still lists /dev/sdb1 on /mnt-1, and df -h /mnt-1 sometimes returns a (presumably stale/cached) size line, but ls, getfacl, and lsof +D /mnt-1 all fail with Input/output error. Treat /mnt-1 as dead, not degraded-but-usable.

Impact

• /mnt-1 hosts buildx-cache/ (→ CI backend-service build failures, e.g. PR fix(benchmarks): snapshot sentinel + earlier lineage assert in T0 merge driver #3157), benchmarks/ (round-7 Unsloth work, epic Epic: round 7 — Unsloth training and dynamic requant, served via Ollama, scored on a fresh three-slot benchmark #3079), and is bind-mount source for /mnt-1/training (round7_t0_merge_export.sh and friends).
• None of this is covered by backup-essentials.timer — that backup is deliberately scoped to secrets/certs/small config, not ES data/payloads/benchmark or training artifacts. If the two down drives don't come back, whatever was on /mnt-1 beyond the last manual mirror (see ops: /mnt-1/benchmarks (the #1947 sweep's whole work area) was wiped by the homeserver rebuild; recovered from a hermes mirror #2971 — /mnt-1/benchmarks was already rebuilt once from a hermes mirror after a prior wipe) is at risk.
• /var (sdc1, VD1) is NOT currently erroring, just mid-reconstruction. Root/boot/home (nvme0n1, LVM) are unaffected.

Not done yet

No remediation attempted. This needs physical-hardware judgment (reseat drives in enclosure slots 2/3, check backplane/cabling to that enclosure, possibly engage the RAID controller's rebuild/foreign-config-clear flow) that shouldn't be done unilaterally over SSH — clear foreign configuration / force-online on a double-drive-fault RAID5 VD is a real data-loss risk if the wrong recovery path is taken. Left /mnt-1 mounted (XFS already self-shut-down, not actively getting worse) rather than force-unmounting blind.

Immediate consequence

PR #3157 (issue #3139 fix) is correctly held on red CI — backend-service fails on open /mnt-1/buildx-cache/backend-service/oci-layout: permission denied → Input/output error under the hood, not a real build problem. Will re-run that check once /mnt-1 is back, not before.

[Xore]

Decision: complete rerun of the old-pin 378 results

The 378 old-pin (a99e7655) result JSONs on /mnt-1/benchmarks/1947full/ will not be recovered from the dead RAID. Instead, they will be completely rerun.

Rationale:

• The /mnt-1 VD0 is dead (slots 2+3 both faulted, no foreign config, RAID5 single-parity exhausted). Physical recovery requires hands-on reseat/backplane check and is not guaranteed to recover data — a clear foreign on a double-drive-fault RAID5 is a real data-loss path.
• All round-7 work has already migrated to /var (7T, surviving). The old-pin results were measured on a stale 14-case rubric with the record_baseline.py: no retry/incremental save (one ollama restart cost 10 cells), and context_tokens/keep_alive are pinned in reports but never sent #2644 defect (no num_ctx/keep_alive pin, warm-slot contamination). The round-7 17-case pin at 32dbdeb1 supersedes them.
• Rerun the ~38 roster tags from the old pin through the current infrastructure on /var if cross-round comparison is needed — same scripts, same card, clean cold protocol.
• Round-7 results (96 tags, 17 cases, 367 JSONs) are fully restored from the hermes mirror and are not affected by this decision.

Impact: the old-pin matrix becomes reproducible on-demand rather than a frozen snapshot. No code changes needed — the sweep scripts already work from /var.\n\n/close

[Xore]

Closed: decision made — complete rerun on /var instead of RAID recovery. See comment above for rationale.