hpc-observer

A read-only observability agent for HPC clusters: it senses cluster state from multiple operational data sources, detects problems, and reports them — and is architecturally incapable of changing anything it observes.

It runs locally on a GPU node, feeds a small tool-calling LLM (served locally, e.g. vLLM) a curated view of cluster health through a whitelisted read-only tool interface, and emits a deduplicated report of new/ongoing/resolved issues. The reasoning model never touches raw data or a shell — only the fixed, audited tools.

Built for a real HPE/Cray-based HPC cluster (HPCM-managed, Weka storage, PBS scheduler, OneView hardware management). This is a sanitized, standalone extract — all site-specific identifiers (hostnames, IPs, domains, serial numbers, org names) have been replaced with placeholders for a fictional "democluster". The engineering is unchanged.

283 tests, standard-library only, fully offline (no GPU, no live cluster, no third-party packages required to run the suite).

python3 -m unittest discover -s hpc_observer/tests

Why it's built this way

An LLM is just a function: text in, text out — it has no hands. To make one useful and safe against a production cluster, the engineering is in the body around the model, not the model itself. Two principles drive the whole design:

1. Read-only in depth. The model can only call a fixed menu of typed, audited, read-only tools — there is no shell and no "run anything" escape hatch. Data is digested and served from a local store; the report sink is the single outbound write in the system.
2. Do the hard work deterministically, save the model for judgment. Boring, testable Python (the collectors) turns each raw firehose into a tidy summary. The model reasons over summaries and deltas — not millions of raw log lines — which is what makes a modest local model effective.

Architecture

data sources ──► collectors ──► state store ──► read-only tools ──► harness ──► report
(HPCM, Weka,     (deterministic  (timestamped     (audited          (LLM loop,   (dedup:
 consoles,        digesters,      snapshots,        whitelist,        model +      new/ongoing/
 Kafka, OneView)  no AI)          latest/previous)  no shell)         tools)       resolved)
                                        │                                │
                                   change detector                  local model
                                   (what changed?)                  (vLLM, OpenAI API)

• Collectors (collectors/) — one per source, each pure raw -> summary dict:
  • hpcm_inventory — node inventory, image/kernel drift, dirty images (separates real nodes from BMC/DPU/PDU/switch entries)
  • weka_state — storage cluster health + capacity
  • console_logs — per-node serial-console fault classification (kernel panic, rescue shell, NIC flaps, filesystem errors …)
  • kafka_telemetry — windowed roll-up of streaming metrics (tolerates both JSON and Prometheus-text wire formats)
  • oneview_inventory / oneview_alerts — hardware inventory + active hardware alerts
• State store (store/) — atomic timestamped JSON snapshots with latest/previous.
• Change detector (diff/) — curated deltas so only new or resolved conditions are surfaced, not standing ones.
• Tool registry (tools/) — the read-only menu exposed to the model; every call validated and audit-logged.
• Report sink (report/) — findings with stable identity (dedup across ticks), each citing its evidence; deterministic provenance + remediation hints.
• Harness (harness/) — model-agnostic loop over an OpenAI-compatible endpoint; a MockModelClient drives the entire pipeline in tests with no GPU.
• Ground truth (ground_truth.py) — a compact "expected fleet + known-down nodes" block injected into the prompt so the model doesn't invent problems or cry wolf.
• Eval harness (eval/) — seeds synthetic faults and scores whether the agent reports them.

Try it

Run the whole sense → reason → report loop with a built-in mock model (no model server needed):

python3 -m hpc_observer.agent --self-test --no-notify \
    --state-root /tmp/obs/state --report-root /tmp/obs/reports

Run just the deterministic "sense" half (collectors → snapshots):

python3 -m hpc_observer.runner --state-root /tmp/obs/state

Point at a real local model server:

python3 -m hpc_observer.agent \
    --model-url http://localhost:8000/v1 --model-name <served-model> \
    --state-root /var/lib/observer/state --report-root /var/lib/observer/reports

The read-only guarantee (defense in depth)

1. Data is consumed from read-only mounts / snapshots — never written back.
2. The model can only call whitelisted read-only tools; no shell, no mutating verb.
3. In deployment it runs as an unprivileged user in a hardened systemd unit that can write only its own data directory.
4. On the real cluster, the server side also enforces read-only (read-only NFS exports, a read-only storage token, consumer-only telemetry ACLs), so the boundary holds even if the node were compromised.

"No issues" always means the collectors ran and found nothing — never the agent crashed: the runner reports per-source ok/skipped/failed and exits non-zero on failure, and the report file is always written as proof-of-run.

Status & scope

The full software pipeline is implemented and tested offline. This public extract is the agent itself; the site-specific deployment automation (image recipe, systemd units, server-side export provisioning) and the real cluster data captures are intentionally not included.

License

MIT — see LICENSE.