feat(tailscale): opt-in tailnet monitoring per Spark

CHANGELOG.md

@@ -7,6 +7,17 @@ Format: version sections are listed newest first.
 
 ---
 
+## [1.7.0] — 2026-08-10
+
+### Added
+- **Tailnet monitoring** — opt-in per Spark (`tailscaleMonitoring`, default **off**); runs `tailscale status --json` on the host and shows a **Tailnet** card beside Network. Closes a blind spot shared by every LAN-based check: when `tailscaled` loses its coordination-server session, SSH/GPU/LLM all keep reporting healthy while the Spark is unreachable from off-LAN.
+- **Off-tailnet verdict from the node itself** — reads `Self.Online` on each host rather than a peer's (possibly stale) view of it.
+- **Failure reason, not just state** — surfaces Tailscale's own `Health` messages (e.g. *"hasn't received a network map from the coordination server in 2m7s"*), `BackendState`, tailnet IP, DERP relay region, version, and an expired-node-key warning.
+- **Edit Spark** — `Tailnet monitoring` checkbox.
+- Env: `POLL_INTERVAL_TAILSCALE` (default `30000`), `TAILSCALE_PROBE_TIMEOUT_MS` (default `8000`).
+
+---
+
 ## [1.6.0] — 2026-08-07
 
 ### Added

README.md

@@ -30,6 +30,7 @@ sparkDash is a real-time web dashboard for one or more **NVIDIA DGX Spark (GB10)
 - [Latest version changelog](#latest-version-changelog)
 - [Features](#features)
 - [ComfyUI monitoring](#comfyui-monitoring)
+- [Tailnet monitoring](#tailnet-monitoring)
 - [Full changelog](./CHANGELOG.md)
 - [Quick start](#quick-start)
 - [Architecture](#architecture)
@@ -47,6 +48,10 @@ sparkDash is a real-time web dashboard for one or more **NVIDIA DGX Spark (GB10)
 
 ## Latest version changelog
 
+### Version 1.7.0 — Tailnet monitoring
+- **Tailnet** — opt-in per Spark (`tailscaleMonitoring`): flags a Spark that is healthy on the LAN but has fallen off its tailnet, which no LAN-based check can see
+- **Reason shown, not just state** — surfaces Tailscale's own `Health` messages plus backend state, tailnet IP, DERP relay, version, and expired-key warning
+
 ### Version 1.6.0 — ComfyUI monitoring & compact default
 - **ComfyUI** — opt-in per Spark (port 8188): live jobs, progress, last run, cancel, queue ETA, Open on LAN IP, model inventory
 - **Overview** — Comfy status chip (`idle` / `run` / `Nq`)
@@ -66,6 +71,7 @@ Full history: [CHANGELOG.md](./CHANGELOG.md)
 | **Local + remote** | Host metrics via sysfs/proc/`nvidia-smi`; remotes over SSH (key or password) |
 | **LLM probe** | Auto-detects llama.cpp, vLLM, sglang, or ds4-server; live tok/s per server |
 | **ComfyUI** | Opt-in probe: queue/jobs, progress, cancel, Open link, inventory, overview chip |
+| **Tailnet** | Opt-in probe: flags a Spark that is healthy on the LAN but off its tailnet |
 | **Decode benchmark** | Multi-concurrency streaming decode tok/s (server + per-stream), persisted last run |
 | **Prompt Showcase** | Full-page multi-terminal LLM streaming demo (up to 32 prompts) with live tok/s and copy-out |
 | **vLLM health** | KV cache %, run/wait queue, TTFT/E2E/ITL p95, preemptions, prefix cache, MTP accept from Prometheus `/metrics` |
@@ -140,6 +146,61 @@ Env (optional): `COMFY_PORT` (default `8188`), `COMFY_PROBE_TIMEOUT_MS`, `POLL_I
 
 ---
 
+## Tailnet monitoring
+
+Opt-in per Spark. Runs `tailscale status --json` on the host and shows a **Tailnet** card
+beside Network.
+
+This closes a blind spot that every LAN-based check shares, including sparkDash's own SSH
+liveness. When `tailscaled` loses its session with the coordination server, short connections
+keep retrying and succeeding — SSH answers, the GPU reports, the LLM serves, and the dashboard
+says **up**, correctly. But the Spark is unreachable from anywhere *off* the LAN: your phone on
+cellular, the admin console, another tailnet node. The box looks perfectly healthy and is
+invisible. This card is the difference.
+
+### What is supported
+
+| Capability | Details |
+|------------|---------|
+| **Opt-in per Spark** | `tailscaleMonitoring` (default **off**) |
+| **Off-tailnet detection** | `Self.Online` — the node's *own* view of the coordination server |
+| **Reason, not just state** | Tailscale's `Health` messages, e.g. *"hasn't received a network map from the coordination server in 2m7s"* |
+| **Backend state** | `Running` / `Stopped` / `NeedsLogin` / `NoState` |
+| **Identity** | Tailnet IP, hostname, DERP relay region, tailscale version |
+| **Key expiry** | Flags an expired node key (which a restart will not fix — it needs re-auth) |
+
+Asked of **each node about itself**, deliberately: one node's view of a *peer* can be stale, so
+peer state is never used as the verdict.
+
+Not claimed: this does not manage Tailscale. It never runs `tailscale up`, `down`, or `login` —
+the probe is read-only.
+
+### How to enable (per Spark)
+
+1. Open **Edit Spark**.
+2. Tick **Tailnet monitoring**.
+3. Save. The Tailnet card appears under Resources, next to Network.
+
+### Host requirements
+
+- The `tailscale` CLI on the monitored host, and `tailscaled` running.
+- Remote Sparks: reached over the existing SSH connection — no extra setup.
+- Local Spark in Docker: tailscaled's socket lives on the *host*, so the probe enters the host
+  mount namespace via `nsenter` (the same approach used for `nvidia-smi`). This needs
+  `/host/proc` bind-mounted, which the shipped compose file already does.
+- A node with key expiry disabled reports no expiry — that is normal, not an error.
+
+### Config fields (persisted on the Spark)
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `tailscaleMonitoring` | `false` | Run `tailscale status --json` and show the Tailnet card |
+
+Env (optional): `POLL_INTERVAL_TAILSCALE` (default `30000` — tailnet state changes slowly and
+each poll is an SSH round-trip), `TAILSCALE_PROBE_TIMEOUT_MS` (default `8000`).
+
+---
+
 ## Quick start
 
 ```bash

package.json

@@ -1,6 +1,6 @@
 {
   "name": "sparkdash",
-  "version": "1.6.0",
+  "version": "1.7.0",
   "description": "sparkDash — Multi-DGX Spark Monitoring Dashboard",
   "type": "module",
   "scripts": {

server/collectors/TailscaleProbe.js

@@ -0,0 +1,185 @@
+/**
+ * TailscaleProbe — reports whether a Spark is actually present on its tailnet.
+ *
+ * Why this exists: a Spark can be perfectly healthy — SSH answering, GPU idle,
+ * LLM serving — while `tailscaled` has lost its session with the coordination
+ * server. Short connections retry and survive, so every LAN-based check
+ * (including sparkDash's own SSH liveness) still reports "up", but the box is
+ * unreachable from anywhere off the LAN: phone on cellular, the admin console,
+ * another tailnet node. The dashboard and the tailnet disagree and the dashboard
+ * looks right.
+ *
+ * `tailscale status --json` is the authoritative answer, and it must be asked of
+ * the node itself — one node's view of a *peer* can be stale, so peer state is
+ * deliberately not used here.
+ *
+ * Command: `tailscale status --json` (read-only; no state is changed)
+ */
+import { TAILSCALE_PROBE_TIMEOUT_MS, HOST_PATHS } from "../config.js";
+import { sshExec } from "./ssh.js";
+import fs from "fs";
+import path from "path";
+
+/**
+ * @param {unknown} v
+ * @returns {string | null}
+ */
+function str(v) {
+  if (typeof v !== "string") return null;
+  const s = v.trim();
+  return s.length > 0 ? s : null;
+}
+
+/**
+ * Normalize `tailscale status --json` into the fields the UI needs.
+ *
+ * Exported for unit testing — keep it pure (no I/O, no clock).
+ *
+ * @param {object} raw - parsed output of `tailscale status --json`
+ * @returns {object} normalized fields (without `error`)
+ */
+export function parseTailscaleStatus(raw) {
+  const self = raw && typeof raw === "object" && raw.Self && typeof raw.Self === "object"
+    ? raw.Self
+    : null;
+
+  // Health is the payload that actually explains a failure, e.g.
+  // "Tailscale hasn't received a network map from the coordination server in 2m7s."
+  const health = Array.isArray(raw?.Health)
+    ? raw.Health.filter((m) => typeof m === "string" && m.trim().length > 0)
+    : [];
+
+  const ips = Array.isArray(self?.TailscaleIPs)
+    ? self.TailscaleIPs.filter((ip) => typeof ip === "string")
+    : [];
+
+  return {
+    available: self != null,
+    /**
+     * Self.Online is the node's OWN view of whether it is talking to the
+     * coordination server. This is the signal a LAN-only check cannot see.
+     * null when tailscale did not report it.
+     */
+    online: typeof self?.Online === "boolean" ? self.Online : null,
+    /** "Running" | "Stopped" | "NeedsLogin" | "NoState" — tailscaled's own state. */
+    backendState: str(raw?.BackendState),
+    hostName: str(self?.HostName),
+    dnsName: str(self?.DNSName),
+    tailscaleIp: ips[0] ?? null,
+    /** DERP relay region, or null when the node has a direct path. */
+    relay: str(self?.Relay),
+    /** ISO timestamp; null when key expiry is disabled for this node. */
+    keyExpiry: str(self?.KeyExpiry),
+    keyExpired: self?.Expired === true,
+    version: str(raw?.Version),
+    /** Human-readable reasons tailscale itself considers itself unhealthy. */
+    health,
+  };
+}
+
+export class TailscaleProbe {
+  /**
+   * @param {object} spark
+   */
+  constructor(spark) {
+    this.spark = spark;
+    this.error = null;
+  }
+
+  /** @param {object} spark */
+  setTarget(spark) {
+    this.spark = spark ?? this.spark;
+    this.error = null;
+  }
+
+  /** Symmetry with the other probes; nothing persistent to release. */
+  dispose() {}
+
+  /**
+   * Host PID 1 mount namespace, present when running as the bind-mounted
+   * container. Mirrors SystemCollector._hasHostProc.
+   */
+  _hasHostProc() {
+    return fs.existsSync(path.join(HOST_PATHS.PROC, "1", "ns", "mnt"));
+  }
+
+  /**
+   * Run the status command for an `isLocal` Spark.
+   *
+   * In the container the `tailscale` CLI and, more importantly, tailscaled's
+   * unix socket live on the *host*, so a plain `sh -c` finds neither. Enter the
+   * host mount namespace when it is available (same approach SystemCollector
+   * uses for nvidia-smi).
+   *
+   * @param {string} cmd
+   * @returns {Promise<string>}
+   */
+  async _execLocal(cmd) {
+    const { execFile } = await import("child_process");
+    const useHostNs = this._hasHostProc();
+    const file = useHostNs ? "nsenter" : "sh";
+    const args = useHostNs
+      ? ["--mount=" + path.join(HOST_PATHS.PROC, "1", "ns", "mnt"), "--", "sh", "-c", cmd]
+      : ["-c", cmd];
+    return new Promise((resolve, reject) => {
+      execFile(file, args, { timeout: TAILSCALE_PROBE_TIMEOUT_MS }, (err, stdout, stderr) => {
+        if (err) return reject(new Error(stderr?.trim() || err.message));
+        resolve(String(stdout).trim());
+      });
+    });
+  }
+
+  /**
+   * Never throws — on failure returns `_default()` with `error` set, matching
+   * the other probes.
+   *
+   * @returns {Promise<object>}
+   */
+  async probe() {
+    const cmd = "tailscale status --json";
+    let out;
+    try {
+      out = this.spark?.isLocal
+        ? await this._execLocal(cmd)
+        : await sshExec(this.spark, cmd, { timeoutMs: TAILSCALE_PROBE_TIMEOUT_MS });
+    } catch (err) {
+      // Most common causes: tailscale not installed, or tailscaled not running
+      // (the CLI exits non-zero). Both are legitimately "not on the tailnet".
+      this.error = err.message || "tailscale status failed";
+      return this._default();
+    }
+
+    let raw;
+    try {
+      raw = JSON.parse(out);
+    } catch {
+      this.error = "Unparseable `tailscale status --json` output";
+      return this._default();
+    }
+
+    const parsed = parseTailscaleStatus(raw);
+    if (!parsed.available) {
+      this.error = "No Self in `tailscale status --json`";
+      return this._default();
+    }
+    this.error = null;
+    return { ...parsed, error: null };
+  }
+
+  _default() {
+    return {
+      available: false,
+      online: null,
+      backendState: null,
+      hostName: null,
+      dnsName: null,
+      tailscaleIp: null,
+      relay: null,
+      keyExpiry: null,
+      keyExpired: false,
+      version: null,
+      health: [],
+      error: this.error,
+    };
+  }
+}