test: cover flagged edge cases + fix snapshot persistence-health bug

[VijitSingh97]

What

A source-vs-tests audit across Pithead's four testing tiers. It adds coverage — at the lowest honest tier (all tier 1) — for branches, guards, and render paths that had none, and fixes one real bug found along the way. No production behaviour changes except the bug fix.

The bug (found while auditing)

storage_service.save_snapshot's TypeError branch (a snapshot json.dumps can't serialize) called logger.error directly instead of _db_error like every other write path — so db_healthy stayed True and the #131 "DB write failing" badge never fired while snapshots silently stopped persisting. One-line root-cause fix + a regression test asserting is_db_healthy() flips to False.

Tests added (tier 1)

Area	Behaviour now covered
pytest	Unserializable snapshot flips db_healthy (guards the fix above)
shell	Firewall iptables -I install-failure → warn + roll back, not half-open (#270)
shell	Wallet subaddress / integrated payout → abort apply (#250, silent reward-loss)
shell	monero.mode=remote with no host → abort; non-/24 network.subnet → abort (#180)
shell	ensure_owner conditional chown + whole-tree recursive scan, no -maxdepth (#255)
frontend	ProxyTotals reddens rejected only on reject_level=high
frontend	Per-worker api_ok=false + reject_flag badges; XvB/Unknown PoolBadge variants
frontend	#278/#313 invariant: Tari ✔ gated on a live gRPC channel, never on active-but-dead
frontend	Gauge done→✔ vs syncing→percent

Each test fails if the logic is inverted (e.g. the Tari test asserts check-inline is absent for active-but-dead; the snapshot test asserts db_healthy is False).

Docs

• docs/test-inventory.md regenerated: 764 → 770 enumerated cases (drift check green).
• docs/testing-strategy.md Known-gaps ledger gains a Coverage-audit follow-ups section (below), so the gaps stay tracked.

Additional gaps found — not covered here (need Docker / real box → tier 3/4)

These are the real completeness risks the audit surfaced. None touch the code changed in this PR, so per "test once at the lowest honest tier" they're follow-ups, not part of this change. The first two are divergence backstops for paths this PR proves only with stubs:

• Firewall rollback, real kernel. Enforce Tor-only egress at the network layer (fail-closed) — stop relying on per-app config #270 rollback is proven with stubbed iptables (control flow only). Whether a real kernel strips a partially inserted DOCKER-USER rule and ends fail-closed is never exercised with an induced mid-insert failure → needs a --fault-injection case.
• ensure_owner real mixed-ownership tree. The Run containers as a non-root user (drop root inside the container) #255 scenario (user-owned dir, root-owned contents, one chown -R) can only be built as root → needs a --lifecycle step.
• Real-container monerod failover in PR CI — only runs on the manual tier-4 box; the mini-stack breaks Tari, not monerod.
• Non-blocking-Tari "ignore" path with real containers — unit-only; the path that silently kills yield if it regresses to a reject.
• monerod busy / mid-reorg failover — contract test proves the client reads busy as unreachable; no scenario asserts the dashboard rejects workers on a busy-but-alive node.
• Double outage, both-must-recover — unit-only; recovery ordering unproven end-to-end.
• Partial-start / stop-failure idempotency — control-loop retry is unit-only; no tier-3/4 injects a docker start/stop error.
• pithead doctor on a real box — only its exit code is unit-tested; the NTP/clock-drift check is never fault-injected.
• Disk-full / ENOSPC verdict — only a headroom warning is checked; a real unhealthy-on-ENOSPC verdict is never forced.
• Tor-container-down partial start — no Caddy/Tor in the mini-stack, so SOCKS-unreachable is exercised at no tier below the manual box.
• Insecure + main matrix row — dashboard.secure=false only pairs with pool=nano, so insecure+main has no row.

Verification (all local)

• make lint → all checks passed
• make test → dashboard 641 passed / 94.73% cov (gate 80%), pithead shell 432/0, selftest 97/0, compose ✓, fakes 12 ✓
• make test-patch-coverage → 100% on the changed line (gate 90%)
• frontend (node --test) → 66 pass / 0 fail
• make test-inventory-check → up to date

Fixtures use synthetic values only (4AAA…/8AAA… wallets, RFC-1918 IPs) — respects .gitleaks.toml.

🤖 Generated with Claude Code

[VijitSingh97]

Follow-up commit eb3b336: the first two "additional gaps" listed above are now implemented as tier-4 live backstops, folded into the existing --fault-injection / --lifecycle phases (no new flags), following the --auth-fail-closed "live counterpart to a tier-1 assertion" pattern.

• --fault-injection → fault_firewall_rollback: shadows iptables with a wrapper that fails every -I, re-runs apply_tor_egress_firewall, asserts the box ends fail-closed (no pithead-tor-egress rule left half-installed), then reinstates the real firewall. Belt-and-braces reinstate is unconditional, so a mid-phase abort can't leave clearnet egress open (Enforce Tor-only egress at the network layer (fail-closed) — stop relying on per-app config #270).
• --lifecycle: plants a root-owned file under the dashboard data dir and asserts the pool-flip apply (ensure_directories → ensure_owner) chowns it to uid 1000 — the Run containers as a non-root user (drop root inside the container) #255 "scan contents, not just the dir" regression.

Verified: shellcheck + shfmt clean, bash -n parses, integration self-test 97/0, make lint green, inventory drift-check green. Not executed against a real box — both are destructive-then-restored, local-box only, and run at the release gate. The remaining 9 gaps stay tracked in docs/testing-strategy.md.

[VijitSingh97]

Live-validated on a real box (commit 72fae38)

Ran the two tier-4 backstops against a real synced box. Validation caught a real bug in the firewall test — exactly the kind of divergence tier-4 exists to find:

• ensure_owner (Run containers as a non-root user (drop root inside the container) #255) — passed as written: a root-owned inode planted deep in a tree is migrated to uid 1000; a clean tree is a graceful no-op.
• Firewall rollback (Enforce Tor-only egress at the network layer (fail-closed) — stop relying on per-app config #270) — first run failed: apply_tor_egress_firewall calls sudo iptables -I, and sudo's secure_path ignores a PATH-shadowed iptables, so the real insert succeeded and the rollback branch never ran (the rc-0 assertion was a false green). Fixed by shadowing sudo instead (still resolved via PATH; fails only the iptables … -I insert, execs real sudo otherwise). Re-validated: with the real -I failing, the partial ruleset rolls back to 0 pithead-tor-egress rules, then reinstates to 7.

The box was restored byte-identical to its pre-test iptables-save snapshot; deployed pithead untouched (sha unchanged), stack healthy, all artifacts removed.

Note: the box runs release v1.0.3 (which predates these functions), so validation used the develop-branch pithead at a temp path against throwaway targets — it exercises the real functions on the real kernel, not the full run.sh harness end-to-end. The full harness run happens at the release gate once develop is deployed there.