diff --git a/CLAUDE.md b/CLAUDE.md index f9198029..c255ff16 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,27 +8,19 @@ Ansible-driven dotfiles + infra for a personal fleet (desktops, servers, VPS). E Three distinct concerns live here: 1. **Workstation config** (`roles/`, `site.yml`, `host_vars/`, `group_vars/`) — shells, packages, browser, SSH, Tailscale, kubeconfig, etc. -2. **Talos K8s cluster IaC** (`talos-k8s/`) — manifests for the Bihar + Karnataka cluster. Detailed handbook in [`docs/src/servers/talos-k8s/cluster.md`](docs/src/servers/talos-k8s/cluster.md). +2. **Home servers** (`home-k3s/`, `hives/`) — bihar and karnataka run Bluefin with a separate single-node k3s each, linked by KubeStellar; the hives run on bihar with karnataka as cold standby. `talos-k8s/` is the **archived** Talos-era IaC (kept for tests/history; do not apply). Current state of every host: [`docs/src/fleet.md`](docs/src/fleet.md). 3. **TunaOS AWS account IaC** (`aws/`, OpenTofu) — the account is **TunaOS infrastructure**, not the personal fleet: punjab's instance, backups (S3), IAM, budgets, and (terminated, still in the code) the AWS Talos cluster's VPC/nodes/EIPs. Credits are exhausted: everything left here costs real money. Handbook: [`docs/src/servers/aws/README.md`](docs/src/servers/aws/README.md). -### Two Talos clusters, not one +### Where things run (2026-11-18 — see [`docs/src/fleet.md`](docs/src/fleet.md)) -- **Home** (`bihar` + `karnataka`) — on the LAN. Configs: `~/.kube/config`, `~/.talos/config`. - **Currently powered down and in storage; expected back.** Treat it as - temporarily offline, not decommissioned — don't delete its configs, Bitwarden - notes, or `talos-k8s/` manifests, and expect `just doctor`/`kubectl` against it - to fail until it returns. -- **AWS** (`eu-north-1`, built 2026-08-27) — **terminated 2026-10-25** after credits were exhausted. It previously ran Matrix, the Hives and the CFP dashboard. The S3 bucket `hanthor-fleet-backups-*` and the himachal-1 mirror (`~/backups/tunaos-aws-2026-10-25`) were retained; verify individual AWS resources rather than assuming they are gone. OpenTofu still describes the old cluster and can propose recreating resources. - Configs retained as recovery material: `~/.kube/config-aws-migration`, `~/.talos/config-aws-migration`. - Handbook: [`docs/src/servers/aws-k8s/cluster.md`](docs/src/servers/aws-k8s/cluster.md). -- **Spindle VPS** — Matrix, standalone LiveKit, School, Reef and Hub run on an Oracle Always Free A1 VM in `eu-frankfurt-1`, 2 OCPU / 12 GB, joined to the tailnet and locked down. The kit and runbook are in [`docs/src/servers/spindle-vps/`](docs/src/servers/spindle-vps/README.md); Oracle API key/config are in Bitwarden. Cloudflare DNS failover is manual, not automatic. School and Reef (Compose under `/srv/hives`) are documented, with roster, identity and the safe edit procedure, in [`docs/src/cluster/hive.md`](docs/src/cluster/hive.md); more than one operator edits that directory. +- **bihar** — Bluefin (`utah:testing`), WiFi, `100.103.44.48`, `ssh james@bihar`. Single-node k3s + **KubeStellar hub** (ITS `its1`, WDS `wds1`). **Active hive host**: hub, hub-front, school, reef as rootful-podman compose in `/var/srv/hives` (wrapper `/var/srv/hives-host/hives`). In `inventory.yml` (`desktop`). Handbook: [`docs/src/servers/bihar.md`](docs/src/servers/bihar.md). +- **karnataka** — Bluefin, `100.66.158.127`. Single-node k3s with KubeVirt (VMs incl. **punjab**), ARC runners (`tuna-ci`), Frappe LMS, Corral, front-door; KubeStellar WEC; **hive cold standby**. Configured from the native-units branch (#348), not in `inventory.yml`. Handbook: [`docs/src/servers/karnataka.md`](docs/src/servers/karnataka.md). +- **spindle** — Oracle A1 VM in `eu-frankfurt-1` (2 OCPU, ARM): Matrix (Spindle + built-in OIDC), LiveKit, Caddy (public `92.5.82.232`, the front door for `school`/`reef`/`hub.tunaos.org`), the OMP auth broker, and the **hive failover coordinator** (lease `/var/lib/hive-failover/active`, `sudo hive-failover status|move|hold`). Kit/runbook: [`docs/src/servers/spindle-vps/`](docs/src/servers/spindle-vps/README.md); Oracle API key/config in Bitwarden. Not in `inventory.yml`. +- **punjab** — KubeVirt VM on karnataka (`corral-vms/punjab`), Ubuntu, `100.103.187.46`, user `ubuntu`. Agent box and this repo's Ansible host. The EC2 instance it replaced is `punjab-aws-retired`. +- **Hives** — hosting, failover lease and runbook: [`docs/src/servers/hives.md`](docs/src/servers/hives.md) (source in `hives/`); the hives themselves: [`docs/src/cluster/hive.md`](docs/src/cluster/hive.md). Exactly one host may run them; never start them by hand on the host the lease does not name. +- **KubeStellar / k3s** — [`docs/src/servers/home-k3s.md`](docs/src/servers/home-k3s.md). Fresh k3s on these hosts needs the firewalld rules there (cni0 → internal, pod→kubelet 10250, `k3s-pod-egress` policy). -Neither cluster's nodes are Ansible-managed — Talos has no SSH. They are not in -`inventory.yml`; access is `talosctl`/`kubectl` only. - -The two Hetzner VPSes it replaced (`matrix`, `telengana`) are retired but still -powered on for burn-in. **Nothing on them may be restarted** — see the -[cutover runbook](docs/matrix-cutover-runbook.md). +**Retired** (history only; see the [archive](docs/src/archive/talos-k8s.md)): the home Talos cluster (same hardware, replaced by k3s), the AWS Talos cluster (`eu-north-1`, terminated 2026-10-25; configs `~/.kube/config-aws-migration` / `~/.talos/config-aws-migration` kept as recovery material; handbook [`docs/src/servers/aws-k8s/cluster.md`](docs/src/servers/aws-k8s/cluster.md)), and the Hetzner VPSes `matrix` / `telengana` (**nothing on them may be restarted** — see the [cutover runbook](docs/matrix-cutover-runbook.md)). ## Core commands @@ -58,30 +50,26 @@ Roles are conditional on inventory group (`desktop`, `server`, `vps`, `test_flee ## Kubeconfig + Talos secrets -`~/.kube/config` and `~/.talos/config` contain cluster PKI — they live in **Bitwarden as secure notes** named `kubeconfig` and `talosconfig` (home cluster). The AWS cluster's are `kubeconfig-aws-migration` / `talosconfig-aws-migration`, written to `~/.kube/config-aws-migration` / `~/.talos/config-aws-migration`. Pull all four via `just apply-tags kube`; `just seed-kube` only seeds the two home-cluster notes. +`~/.kube/config` and `~/.talos/config` contain cluster PKI — they live in **Bitwarden as secure notes** named `kubeconfig` and `talosconfig`. **Those notes hold the retired home Talos cluster's configs**; the k3s clusters' kubeconfigs are `/etc/rancher/k3s/k3s.yaml` on each host (punjab's `~/.kube/config` targets karnataka; bihar's has contexts `bihar`, `its1`, `wds1`) and have not been re-seeded — don't run `just apply-tags kube` on a host whose `~/.kube/config` you need to keep without checking first. The AWS cluster's are `kubeconfig-aws-migration` / `talosconfig-aws-migration`, written to `~/.kube/config-aws-migration` / `~/.talos/config-aws-migration`. Pull all four via `just apply-tags kube`; `just seed-kube` only seeds the two home-cluster notes. `talos-k8s/.gitignore` excludes `controlplane.yaml`, `worker.yaml`, and `talosconfig` from git. -## Talos cluster - -Two nodes: **bihar** (control plane, Intel) and **karnataka** (worker, AMD Strix Halo APU). Talos `v1.13.2`, K8s `v1.36.1`, flannel CNI. AMD GPU exposed to Kubernetes via the Image Factory schematic. - -Production workloads: Lemonade (AMD-optimized local AI), KubeVirt v1.8.2 + KubeVirt Manager, Tailscale Operator (Ingress to `*.manatee-basking.ts.net`). +## Home k3s (replaces the Talos cluster) -**The cluster handbook is [`docs/src/servers/talos-k8s/cluster.md`](docs/src/servers/talos-k8s/cluster.md)** — hardware, network, reinstall, troubleshooting. +No Talos anywhere now. Each of bihar/karnataka has its own k3s v1.36.5 bound to its tailnet IP (flannel on `tailscale0`, traefik/servicelb disabled); kits in `home-k3s//` (`install-host.sh`, `config.yaml`, `k3s.service`). bihar uses service CIDR `10.96.0.0/12` (KubeFlex requirement). karnataka's AMD GPU is exposed through `amdgpu-device-plugin.yaml`; Lemonade is scaled to 0. The [k3s handbook](docs/src/servers/home-k3s.md) covers design, firewalld and KubeStellar. ## TunaOS AWS account (`aws/`) -- **punjab** (this repo's AWS-hosted Ansible host) is an EC2 `t3.large` in `us-east-1`; the former Talos cluster in `eu-north-1` is terminated. `runs-on` (us-east-2) is its own CloudFormation stack — don't import it. +- **punjab** is no longer in AWS (it is a VM on karnataka); `aws/punjab.tf` describes the retired EC2 instance. The former Talos cluster in `eu-north-1` is terminated. `runs-on` (us-east-2) is its own CloudFormation stack — don't import it. - **Do not blindly apply `aws/`:** its configuration still describes the terminated cluster and may propose recreating paid infrastructure. Inspect live resources and review plans before changes; credits are exhausted. - State is in S3 (`hanthor-fleet-backups-*/tofu/aws/`) and contains Talos PKI. `aws/terraform.tfvars` is gitignored and lives in Bitwarden note `aws-tofu-tfvars`. -- punjab uses Tailscale for access and SSM Session Manager for break-glass. Preserve its data and the S3 backups until a verified replacement and an explicit cleanup decision exist. +- Preserve the retired punjab EC2 instance's data and the S3 backups until an explicit cleanup decision exists. - The former cluster uploaded Postgres dumps to `s3://hanthor-fleet-backups-*/postgres/`; the `postgres-backup-writer` key is in Bitwarden, never in tofu state. - `prevent_destroy` in configuration does not prevent creation of resources missing from the account. Never put Talos `user_data` in the config. ## Automation (bots + Hive) -Renovate and the Hive open PRs; the live Hives are school (`hanthor-hive-agent`) and reef (`tuna-reef-hive`) on the Spindle VPS and cover only `tuna-os/*` repos, not this repo (see [`docs/src/cluster/hive.md`](docs/src/cluster/hive.md)). master is protected (CI + code-owner review, admin bypass). `.github/CODEOWNERS` is deny-by-default with carve-outs for paths that never run on the fleet (docs, tests, talos-k8s, pi skills, dep manifests) — only those auto-merge. See [`docs/src/automation.md`](docs/src/automation.md). Don't widen the carve-outs to anything the fleet timer executes. +Renovate and the Hive open PRs; the live Hives are school (`hanthor-hive-agent`) and reef (`tuna-reef-hive`) on bihar (karnataka standby) and cover only `tuna-os/*` repos, not this repo (see [`docs/src/cluster/hive.md`](docs/src/cluster/hive.md)). master is protected (CI + code-owner review, admin bypass). `.github/CODEOWNERS` is deny-by-default with carve-outs for paths that never run on the fleet (docs, tests, talos-k8s, pi skills, dep manifests) — only those auto-merge. See [`docs/src/automation.md`](docs/src/automation.md). Don't widen the carve-outs to anything the fleet timer executes. ## Don'ts @@ -94,7 +82,10 @@ Renovate and the Hive open PRs; the live Hives are school (`hanthor-hive-agent`) ## References -- [`docs/src/servers/talos-k8s/cluster.md`](docs/src/servers/talos-k8s/cluster.md) — Talos cluster handbook +- [`docs/src/fleet.md`](docs/src/fleet.md) — every host: live, intermittent, retired +- [`docs/src/servers/hives.md`](docs/src/servers/hives.md) — hive hosting + failover runbook +- [`docs/src/servers/home-k3s.md`](docs/src/servers/home-k3s.md) — k3s + KubeStellar +- [`docs/src/archive/`](docs/src/archive/talos-k8s.md) — retired Talos cluster handbooks - [`docs/src/roles/`](docs/src/roles/) — every Ansible role explained - [`docs/src/bitwarden.md`](docs/src/bitwarden.md) — BW vault structure - `Justfile` — every task recipe; `just --list` for a menu diff --git a/CONTEXT.md b/CONTEXT.md index 361ae75a..8a01ef41 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -2,7 +2,7 @@ > Created 2026-06-14 during architecture review. > Add terms as they are sharpened during grilling sessions or refactors. -> This is a glossary, not a spec. Implementation details live in `docs/src/roles/` and the cluster handbooks (`docs/src/servers/talos-k8s/cluster.md`, `docs/src/servers/aws-k8s/cluster.md`). +> This is a glossary, not a spec. Implementation details live in `docs/src/roles/` and the host pages (`docs/src/fleet.md`, `docs/src/servers/{bihar,karnataka,home-k3s,hives}.md`); the Talos handbooks are archived (`docs/src/archive/talos-home-cluster/cluster.md`, `docs/src/servers/aws-k8s/cluster.md`). ## Fleet @@ -26,9 +26,10 @@ ## Cluster -- **Node** — a Talos Linux machine in a K8s cluster. Home: `bihar` = control plane, `karnataka` = worker (powered down, expected back). AWS (`eu-north-1`): one control-plane + one worker. Neither is in `inventory.yml`. -- **Workload** — a Kubernetes deployment on a cluster (home: Lemonade, KubeVirt, Tailscale Operator; AWS: Matrix/ESS, the Hive, CFP dashboard). -- **Manifest** — a YAML file in `talos-k8s/` that defines a workload. +- **Node** — a k3s node. Home: `bihar` and `karnataka` are each a **separate single-node k3s** on Bluefin, linked by KubeStellar (hub on bihar: ITS `its1`, WDS `wds1`). The former Talos clusters (home, and AWS `eu-north-1`) are retired. +- **Workload** — a Kubernetes deployment on a cluster (karnataka: KubeVirt VMs incl. punjab, ARC runners, Frappe LMS, Corral, front-door; bihar: KubeStellar control plane). The hives are **not** a k8s workload: they run as podman compose on bihar (karnataka standby). +- **Lease** — the one-line file on spindle (`/var/lib/hive-failover/active`) naming the single host allowed to run the hives. +- **Manifest** — a YAML file that defines a workload: `home-k3s//` (current) or `talos-k8s/` (archived Talos-era). ## Architecture (from improve-codebase-architecture) diff --git a/README.md b/README.md index dcf11e4e..6fed6fae 100644 --- a/README.md +++ b/README.md @@ -59,18 +59,22 @@ From `inventory.yml`: | dilli | desktop | Secondary workstation (Bluefin) | | kerala | desktop | postmarketOS ARM device (musl, `apk`) | | mumbai | desktop | Debian VM on the phone (Android Virtualization Framework), `cli-only` | -| punjab | server | EC2 agent box (us-east-1) | +| punjab | server | Agent box, KubeVirt VM on karnataka | +| bihar | desktop | Home server (Bluefin): active hive host, KubeStellar hub | | goa | server | Raspberry Pi 5 (applies locally) | | vm | server | Local dev VM | | termux | termux_hosts | Raw Termux layer on the phone — `termux_packages` only | | test-fleet-fedora, test-fleet-node2 | test_fleet | KubeVirt test VMs (`machine_profile: vm-test`) | | matrix, telengana | vps | **Retired** Hetzner VPSes, powered on for burn-in — restart nothing on them | -The home Talos cluster, **bihar** + **karnataka**, is not Ansible-managed: it is -powered down and in storage, expected back. The AWS Talos cluster was terminated -on 2026-10-25. [Spindle](docs/src/servers/spindle-vps/README.md), an OCI VPS in -Frankfurt running Matrix, standalone LiveKit and the School, Reef and Hub Hives, -is not listed in this inventory. +**bihar** and **karnataka** are home servers on Bluefin with separate k3s +clusters linked by KubeStellar (the Talos cluster they ran before is archived). +bihar runs the School, Reef and Hub hives, with karnataka as a cold standby; +karnataka hosts KubeVirt (including punjab), CI runners and the LMS. +[Spindle](docs/src/servers/spindle-vps/README.md), an OCI VPS in Frankfurt, runs +Matrix and LiveKit, is the public front door (Caddy) for the hives and coordinates +their failover; it is not in this inventory. The AWS Talos cluster was terminated +on 2026-10-25. Current state of every host: [Fleet status](docs/src/fleet.md). The `llm` group is dormant (commented out in `inventory.yml`). --- @@ -85,8 +89,11 @@ Repo is public — no secrets in git, ever. All secrets are fetched from Bitward Full handbook lives in [`docs/`](docs/) and is published at [reilly.asia/infra/handbook](https://reilly.asia/infra/handbook/) — run `just docs` to serve it locally. -- [Talos cluster handbook](docs/src/servers/talos-k8s/cluster.md) — Bihar + Karnataka (home, offline), Lemonade, KubeVirt -- [Spindle VPS](docs/src/servers/spindle-vps/README.md) — Matrix, LiveKit and Hives on OCI +- [Fleet status](docs/src/fleet.md) — every host, live / intermittent / retired +- [Bihar](docs/src/servers/bihar.md), [Karnataka](docs/src/servers/karnataka.md), [Home k3s + KubeStellar](docs/src/servers/home-k3s.md) +- [Hives hosting and failover](docs/src/servers/hives.md) — bihar primary, karnataka standby, spindle front door +- [Spindle VPS](docs/src/servers/spindle-vps/README.md) — Matrix, LiveKit, Caddy front door on OCI +- [Archive](docs/src/archive/talos-k8s.md) — the retired home Talos cluster - [TunaOS AWS cluster handbook](docs/src/servers/aws-k8s/cluster.md) — archived cluster history - [Adding a new machine](docs/src/onboarding.md) - [Roles reference](docs/src/roles/) diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index 38589b85..62e6edc4 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md @@ -15,6 +15,7 @@ # Fleet Reference +- [Fleet status](fleet.md) - [Network](network.md) - [Gateway](network/gateway/README.md) - [KVM](network/kvm/README.md) @@ -24,25 +25,21 @@ - [Kanpur](desktop/kanpur/README.md) - [Kerala](desktop/kerala/README.md) - [Servers]() - - [Home Kubernetes Images & ISOs](servers/home-k8s-images.md) - - [Talos K8s Cluster (offline)](servers/talos-k8s/cluster.md) - - [Bihar](servers/talos-k8s/bihar/README.md) - - [Karnataka](servers/talos-k8s/karnataka/README.md) - - [Goa](servers/goa/README.md) + - [Bihar](servers/bihar.md) + - [Karnataka](servers/karnataka.md) + - [Home k3s & KubeStellar](servers/home-k3s.md) + - [Hives hosting & failover](servers/hives.md) - [Spindle VPS (matrix.reilly.asia)](servers/spindle-vps/README.md) - [Upgrade to main (draft)](servers/spindle-vps/upgrade-to-main.md) - [2026-11 media and sync incident](servers/spindle-vps/incident-2026-11-17-media.md) -- [VPS]() - - [Matrix (retired)](vps/matrix/README.md) - - [Telengana (retired)](vps/telengana/README.md) + - [Punjab (agent box)](servers/punjab/README.md) + - [Goa](servers/goa/README.md) + - [Home Kubernetes Images (not deployed)](servers/home-k8s-images.md) # TunaOS Infrastructure - [TunaOS AWS Account & IaC](servers/aws/README.md) - - [Talos Cluster (terminated; archive)](servers/aws-k8s/cluster.md) - - [Shutdown, 2026-10-25](servers/aws-k8s/shutdown-2026-10-25.md) - [Hive Agent Fleet](cluster/hive.md) - - [Punjab (agent box)](servers/punjab/README.md) # Roles Reference @@ -105,6 +102,13 @@ - [authentik](roles/authentik.md) - [n8n](roles/n8n.md) -# Cluster +# Archive -- [Talos Kubernetes](talos-k8s.md) +- [Home Talos cluster (replaced by k3s)](archive/talos-k8s.md) + - [Cluster handbook](archive/talos-home-cluster/cluster.md) + - [Bihar on Talos](archive/talos-home-cluster/bihar/README.md) + - [Karnataka on Talos](archive/talos-home-cluster/karnataka/README.md) +- [AWS Talos cluster (terminated 2026-10-25)](servers/aws-k8s/cluster.md) + - [Shutdown, 2026-10-25](servers/aws-k8s/shutdown-2026-10-25.md) +- [Matrix VPS (retired)](archive/vps/matrix/README.md) +- [Telengana VPS (retired)](archive/vps/telengana/README.md) diff --git a/docs/src/servers/talos-k8s/bihar/README.md b/docs/src/archive/talos-home-cluster/bihar/README.md similarity index 89% rename from docs/src/servers/talos-k8s/bihar/README.md rename to docs/src/archive/talos-home-cluster/bihar/README.md index 374c95b9..364c0c58 100644 --- a/docs/src/servers/talos-k8s/bihar/README.md +++ b/docs/src/archive/talos-home-cluster/bihar/README.md @@ -1,5 +1,7 @@ # Bihar +> **ARCHIVED (2026-11-18).** This page describes bihar on Talos, which was replaced by Bluefin + k3s. See [Fleet status](../../../fleet.md). + Kubernetes control plane. [Talos Linux](https://www.talos.dev/) node — not in `inventory.yml`, managed with `talosctl`/`kubectl` only. > **Powered down and in storage, expected back.** See the [cluster handbook](../cluster.md). diff --git a/docs/src/servers/talos-k8s/cluster.md b/docs/src/archive/talos-home-cluster/cluster.md similarity index 99% rename from docs/src/servers/talos-k8s/cluster.md rename to docs/src/archive/talos-home-cluster/cluster.md index 5ed88c70..6e222ca6 100644 --- a/docs/src/servers/talos-k8s/cluster.md +++ b/docs/src/archive/talos-home-cluster/cluster.md @@ -1,11 +1,13 @@ # Cluster Handbook — Talos Linux on Bihar + Karnataka +> **ARCHIVED (2026-11-18).** This page describes the home Talos cluster, which was replaced by Bluefin + k3s. See [Fleet status](../../fleet.md). + > **Status: powered down and in storage (as of 2026-08-27), expected back.** > Temporarily offline, *not* decommissioned — keep its configs, Bitwarden notes > (`kubeconfig` / `talosconfig`), and `talos-k8s/` manifests. `kubectl` and > `just doctor` will fail against it until it returns. Matrix and the School, -> Reef and Hub Hives now run on the [OCI Spindle VPS](../spindle-vps/README.md). -> The [AWS Talos cluster](../aws-k8s/cluster.md) was terminated on 2026-10-25. +> Reef and Hub Hives now run on the [OCI Spindle VPS](../../servers/spindle-vps/README.md). +> The [AWS Talos cluster](../../servers/aws-k8s/cluster.md) was terminated on 2026-10-25. > > **When it returns**, before applying anything under `talos-k8s/ai/`: > run [`talos-k8s/ai/migrate-pv-path.sh`](https://github.com/hanthor/dotfiles/blob/master/talos-k8s/ai/migrate-pv-path.sh) @@ -554,7 +556,7 @@ This is a replacement OS and **new K3s cluster** for the two home nodes, not a Talos upgrade. Bihar is the Intel server/control plane; Karnataka is the AMD worker. **No physical migration has been performed.** The home cluster is still powered down/in storage. Nothing here changes the terminated -[AWS Talos cluster's](../aws-k8s/cluster.md) recovery material; Corral +[AWS Talos cluster's](../../servers/aws-k8s/cluster.md) recovery material; Corral acceptance uses separate disposable KubeVirt VMs, not Talos node changes. Sections 1–11 remain the Talos recovery handbook, not instructions for the new OS. diff --git a/docs/src/servers/talos-k8s/karnataka/README.md b/docs/src/archive/talos-home-cluster/karnataka/README.md similarity index 90% rename from docs/src/servers/talos-k8s/karnataka/README.md rename to docs/src/archive/talos-home-cluster/karnataka/README.md index f98939dd..2cbcc796 100644 --- a/docs/src/servers/talos-k8s/karnataka/README.md +++ b/docs/src/archive/talos-home-cluster/karnataka/README.md @@ -1,5 +1,7 @@ # Karnataka +> **ARCHIVED (2026-11-18).** This page describes karnataka on Talos, which was replaced by Bluefin + k3s. See [Fleet status](../../../fleet.md). + Kubernetes worker node with AMD GPU. [Talos Linux](https://www.talos.dev/) node — not in `inventory.yml`, managed with `talosctl`/`kubectl` only. > **Powered down and in storage, expected back.** See the [cluster handbook](../cluster.md). diff --git a/docs/src/talos-k8s.md b/docs/src/archive/talos-k8s.md similarity index 81% rename from docs/src/talos-k8s.md rename to docs/src/archive/talos-k8s.md index 46a339e0..fd37244a 100644 --- a/docs/src/talos-k8s.md +++ b/docs/src/archive/talos-k8s.md @@ -1,8 +1,10 @@ # Talos Kubernetes Cluster +> **ARCHIVED (2026-11-18).** This page describes the home Talos cluster, which was replaced by Bluefin + k3s. See [Fleet status](../fleet.md). + Two-node Kubernetes cluster running on Talos Linux, managed from the `talos-k8s/` directory. -> **Powered down and in storage, expected back** — commands below fail until it returns. Matrix and School, Reef and Hub run on the [OCI Spindle VPS](servers/spindle-vps/README.md); the [AWS Talos cluster](servers/aws-k8s/cluster.md) was terminated on 2026-10-25. +> **Powered down and in storage, expected back** — commands below fail until it returns. Matrix and School, Reef and Hub run on the [OCI Spindle VPS](../servers/spindle-vps/README.md); the [AWS Talos cluster](../servers/aws-k8s/cluster.md) was terminated on 2026-10-25. ## Hardware @@ -49,7 +51,7 @@ Manifest: `talos-k8s/ai/lemonade.yaml` ### Additional services -See the [cluster handbook](servers/talos-k8s/cluster.md) for the full list — Forgejo, Hive, Authentik, n8n, AppFlowy Cloud, Grafana, Corral, and more. +See the [cluster handbook](talos-home-cluster/cluster.md) for the full list — Forgejo, Hive, Authentik, n8n, AppFlowy Cloud, Grafana, Corral, and more. ## Cluster Access @@ -68,4 +70,4 @@ Karnataka uses a custom Talos image from Image Factory that includes the `sidero ## Handbook -Full cluster documentation at [`docs/src/servers/talos-k8s/cluster.md`](servers/talos-k8s/cluster.md) (hardware details, reinstall procedure, troubleshooting). +Full cluster documentation at [`docs/src/servers/talos-k8s/cluster.md`](talos-home-cluster/cluster.md) (hardware details, reinstall procedure, troubleshooting). diff --git a/docs/src/vps/matrix/README.md b/docs/src/archive/vps/matrix/README.md similarity index 84% rename from docs/src/vps/matrix/README.md rename to docs/src/archive/vps/matrix/README.md index 622afe36..0fe15ac0 100644 --- a/docs/src/vps/matrix/README.md +++ b/docs/src/archive/vps/matrix/README.md @@ -1,8 +1,8 @@ # Matrix > **Retired 2026-08-27.** The Matrix homeserver moved first to the -> [AWS Talos cluster](../../servers/aws-k8s/cluster.md), terminated 2026-10-25, -> and now runs on the [OCI VPS](../../servers/spindle-vps/README.md). Restart +> [AWS Talos cluster](../../../servers/aws-k8s/cluster.md), terminated 2026-10-25, +> and now runs on the [OCI VPS](../../../servers/spindle-vps/README.md). Restart > nothing on this retired host; the page below is historical. See the > [Matrix cutover runbook](https://github.com/hanthor/dotfiles/blob/master/docs/matrix-cutover-runbook.md). @@ -27,6 +27,6 @@ upgrades on the box were disabled so nothing resurrects them. - A single VPS with no swap and unmanaged journald growth was fragile for K8s; the former AWS cluster needed more headroom (see the OOM incident in its - [archived handbook](../../servers/aws-k8s/cluster.md)). + [archived handbook](../../../servers/aws-k8s/cluster.md)). - `reilly.asia` DNS is on [Cloudflare](https://www.cloudflare.com/), which made the cutover a DNS change rather than a server move. diff --git a/docs/src/vps/telengana/README.md b/docs/src/archive/vps/telengana/README.md similarity index 74% rename from docs/src/vps/telengana/README.md rename to docs/src/archive/vps/telengana/README.md index 3e4dea0d..d0dba505 100644 --- a/docs/src/vps/telengana/README.md +++ b/docs/src/archive/vps/telengana/README.md @@ -1,8 +1,8 @@ # Telengana > **Retired 2026-08-27.** The tuna-os Hive moved from this host to the former -> [AWS Talos cluster](../../servers/aws-k8s/cluster.md), terminated 2026-10-25. -> School, Reef and Hub now run on the [OCI VPS](../../servers/spindle-vps/README.md). +> [AWS Talos cluster](../../../servers/aws-k8s/cluster.md), terminated 2026-10-25. +> School, Reef and Hub now run on the [OCI VPS](../../../servers/spindle-vps/README.md). > Restart nothing on this retired host; the page below is historical. Former VPS node in the fleet (`vps` group in `inventory.yml`). @@ -19,4 +19,4 @@ The box drifted from the fleet's hardening baseline (host firewall and fail2ban were never enabled, and Tailscale had been logged out, leaving admin access on the public interface). The former AWS cluster kept its admin APIs behind an allowlisted security group, and fleet servers use the -[`server_hardening`](../../roles/server_hardening.md) role. +[`server_hardening`](../../../roles/server_hardening.md) role. diff --git a/docs/src/automation.md b/docs/src/automation.md index 76686d00..d6e6f621 100644 --- a/docs/src/automation.md +++ b/docs/src/automation.md @@ -11,7 +11,7 @@ you, and anything that executes on the fleet still waits for you. |---|---|---| | **Renovate** | minor/patch/digest bumps (pi npm deps, Go, GitHub Actions, OpenTofu providers), weekly lockfile maintenance | yes, once CI is green | | **Renovate** (cluster) | image/chart bumps in `talos-k8s/`, grouped weekly, label `deploy-needed` | **no** — manifests are applied by hand, so a merge would only make git lie about the cluster | -| **Hive** (school and reef spokes on the [Spindle VPS](cluster/hive.md#current-deployment-spindle-vps); Apps `hanthor-hive-agent` and `tuna-reef-hive`) | its agents open PRs labelled `hold`; the Hive itself only merges PRs labelled `lgtm` | only if the PR stays inside the carve-outs below. **Neither hive's roster includes this repo today** (both cover only `tuna-os/*`), so no Hive PRs are expected here unless a hanthor spoke is rebuilt | +| **Hive** (school and reef spokes on [bihar, with karnataka as standby](cluster/hive.md#current-deployment-bihar-karnataka-standby); Apps `hanthor-hive-agent` and `tuna-reef-hive`) | its agents open PRs labelled `hold`; the Hive itself only merges PRs labelled `lgtm` | only if the PR stays inside the carve-outs below. **Neither hive's roster includes this repo today** (both cover only `tuna-os/*`), so no Hive PRs are expected here unless a hanthor spoke is rebuilt | | **You** | direct pushes, `just onboard`, etc. | admin bypass — unaffected | ## The gate @@ -56,7 +56,7 @@ was the seed of the former `hive.reilly.asia` spoke on the terminated AWS cluster (exported 2026-10-01); it does not describe a running Hive. For the current school and reef files, read-only inspection and the safe edit procedure, see the -[Hive operations guide](cluster/hive.md#current-deployment-spindle-vps): +[Hive operations guide](cluster/hive.md#current-deployment-bihar-karnataka-standby): ```bash ssh ubuntu@100.113.75.88 'docker ps --filter name=hives-; ls /srv/hives/*/data/hive.yaml.runtime' diff --git a/docs/src/bitwarden.md b/docs/src/bitwarden.md index 5fa9b1dc..a3b77cc4 100644 --- a/docs/src/bitwarden.md +++ b/docs/src/bitwarden.md @@ -52,6 +52,16 @@ Every item a role or script reads (grep `bw get` / `bw list` under `roles/` and | `bw-api-key` | Login | `scripts/bw-resolve.sh` — `bw login --apikey` on a remote | | `James Bitwarden` | Login | `scripts/bw-resolve.sh` — non-interactive unlock on a remote | +### Recovery items (not read by any role) + +These items are kept so that you can rebuild a lost host by hand. No role reads them. + +| Bitwarden Item | Type | Holds | +|---------------|------|-------| +| `hive-github-app-hanthor-hive-agent` | Secure Note | School's GitHub App: App ID, installation ID, PEM private key (the hub mounts the same key). See [Hive](cluster/hive.md#github-apps) | +| `hive-github-app-tuna-reef-hive` | Secure Note | Reef's GitHub App: App ID, installation ID, PEM private key | +| `hive-spindle-env` | Secure Note | The hives' `.env` (hub secret, dashboard token, omp broker token, provider keys). It was `/srv/hives/.env` on spindle and is now `/var/srv/hives/.env` on bihar and karnataka. The item name has not changed | + ## Running Without Secrets For fast iteration that doesn't need the vault — packages, dotfiles, desktop config — use the explicit no-secrets path: diff --git a/docs/src/cluster/hive.md b/docs/src/cluster/hive.md index 6642d2e8..b6902efd 100644 --- a/docs/src/cluster/hive.md +++ b/docs/src/cluster/hive.md @@ -1,15 +1,24 @@ # Hive — AI Agent Supervisor -## Current deployment: Spindle VPS +## Current deployment: bihar (karnataka standby) -Two Hive spokes, **school** and **reef**, and their **hub** run as Docker -Compose services on the [Spindle VPS](../servers/spindle-vps/README.md) under -`/srv/hives` (`ubuntu@100.113.75.88` over Tailscale). Caddy routes +Two Hive spokes, **school** and **reef**, and their **hub** run as compose +services under rootful podman on [bihar](../servers/bihar.md), in +`/var/srv/hives` (`ssh james@bihar`, commands need `sudo`). Until 2026-11-18 +they ran on the [Spindle VPS](../servers/spindle-vps/README.md) under +`/srv/hives`. Spindle's Caddy still routes [school.tunaos.org](https://school.tunaos.org), [reef.tunaos.org](https://reef.tunaos.org) and -[hub.tunaos.org](https://hub.tunaos.org) to them. Everything below was read from -the host on 2026-11-13 unless marked *unverified*. Container state alone does -not prove agents are healthy; see [Checking health](#checking-health). +[hub.tunaos.org](https://hub.tunaos.org), over the tailnet to the active host. +[karnataka](../servers/karnataka.md) is a cold standby. A lease on spindle +selects the active host. Run every command below on **the host the lease +names** (`ssh spindle sudo hive-failover status`). Hosting, failover and the +host-specific compose layer are described in +[Hives hosting and failover](../servers/hives.md). + +The layout and roster below were read on spindle on 2026-11-13. The move +changed the paths and the commands, but not the hive configuration. Container +state alone does not prove that agents are healthy. See [Checking health](#checking-health). This section is the current operations guide. The [archived cluster guide](#archived-cluster-guide) after it describes the retired @@ -18,7 +27,7 @@ Kubernetes deployment and must not be applied here. ### Layout ```text -/srv/hives/ +/var/srv/hives/ # on the active host; the host layer is /var/srv/hives-host/compose..yaml ├── compose.yaml # services: hub, hub-front, school, reef (network spindle_default) ├── compose.override.yaml # pins the running binary/dashboard, memory limits, session pool ├── .env # secrets and NTFY topic (mode 600; never print or commit) @@ -99,27 +108,68 @@ auto-merge Renovate major bumps, which the org forbids. 3. Re-parse both files and check the list lengths and that the new repo shows up on the hive's dashboard; no App installation step is needed. +### GitHub Apps + +Since 2026-10-30 School and Reef use separate GitHub Apps, so each has its own +installation rate limit (6,150 requests/hour across the 43 org repositories). Before +that change they shared one installation and used all of its limit every hour. + +| Hive | App (bot login) | App ID | Installation | Key in Bitwarden | +|---|---|---|---|---| +| school (the hub mounts the same key) | `hanthor-hive-agent` (`hanthor-hive-agent[bot]`) | 3942065 | 137498420 | `hive-github-app-hanthor-hive-agent` | +| reef | `tuna-reef-hive` (`tuna-reef-hive[bot]`) | 5546574 | 177007234 | `hive-github-app-tuna-reef-hive` | + +The IDs occur in three places for each hive: +- `GH_APP_ID` and `GH_APP_INSTALLATION_ID` in `compose.yaml`; +- `github.app_id` and `installation_id` in the seed; +- the same keys in `hive.yaml.runtime`. + +Both hives manage `.github`, `Tavern`, `bluefin-cli`, `corral`, `docs` and +`flatpak-index`. Each hive lists the bot of the other hive in +`github.additional_bot_logins`, so that the issues and PRs of the other hive count as +hive-authored. + +To restore a key: +1. Write the key block of the Bitwarden note to + `/var/srv/hives//secrets/gh-app-key.pem` on the active host (mode 600). +2. Run `chcon -t container_file_t` on the file. +3. Recreate the service with `sudo /var/srv/hives-host/hives up -d --no-deps `. + +The next `hive-sync` copies the key to the standby host. + +To make sure that each hive has its own rate limit, run: + +```bash +ssh james@bihar 'for c in hives-school-1 hives-reef-1; do sudo podman exec $c sh -c \ + "curl -s -H \"Authorization: Bearer \$(cat /var/run/hive-metrics/gh-app-token.cache)\" https://api.github.com/rate_limit" \ + | jq -c .resources.core; done' +``` + +Different `reset` times show separate installations. + ### Safe edit procedure -More than one operator edits `/srv/hives` (including agents). Re-read the files +More than one operator edits `/var/srv/hives` (including agents). Re-read the files immediately before editing and never overwrite a file you read earlier. -Do **not** `docker compose up` a spoke that is running to apply config: it +Do **not** `hives up` a spoke that is running to apply config: it triggered a container-rename bug. Remove the container first: ```bash -cd /srv/hives -docker compose rm -f -s reef # stop and remove the spoke container only +sudo -i # the files are root-owned on bihar/karnataka +cd /var/srv/hives +H=/var/srv/hives-host/hives # docker-compose wrapper: podman + host layer +$H rm -f -s reef # stop and remove the spoke container only python3 hive-roster.py add NAME reef/config/hive.yaml reef/data/hive.yaml.runtime -docker compose up -d --no-deps reef # recreate only this spoke -docker ps --filter name=hives- # verify: Up, and the right name +$H up -d --no-deps reef # recreate only this spoke +podman ps --filter name=hives- # verify: Up, and the right name ``` The edit step must (a) write a timestamped copy of each file into -`/srv/hives/backups/` first, (b) parse the YAML before and after, and (c) abort, +`/var/srv/hives/backups/` first, (b) parse the YAML before and after, and (c) abort, writing nothing, if anything other than the intended change would differ. This -helper does that for roster edits (save it as `/srv/hives/hive-roster.py` and -run it from `/srv/hives`; it needs PyYAML): +helper does that for roster edits (save it as `/var/srv/hives/hive-roster.py` and +run it from `/var/srv/hives`; it needs PyYAML): ```python #!/usr/bin/env python3 @@ -127,7 +177,7 @@ run it from `/srv/hives`; it needs PyYAML): import datetime, shutil, sys, yaml ORG = "tuna-os" -BACKUPS = "/srv/hives/backups" +BACKUPS = "/var/srv/hives/backups" PATHS = (("project", "repos"), ("dashboard", "matrix", "repos")) def seq_node(root, path): @@ -209,14 +259,14 @@ the maintainer; their exact semantics were not exercised for this page.) curl -s -o /dev/null -w '%{http_code}\n' https://school.tunaos.org/api/health # 200 curl -s -o /dev/null -w '%{http_code}\n' https://reef.tunaos.org/api/health # 200 curl -s https://hub.tunaos.org/api/fleet-stats # hub: /api/health is 404 -ssh ubuntu@100.113.75.88 'docker ps --filter name=hives-' +ssh james@bihar 'sudo podman ps --filter name=hives-' # or karnataka after a failover ``` For authenticated status from the host, call the API inside the container with the container's own token, so it is never printed: ```bash -ssh ubuntu@100.113.75.88 'docker exec hives-school-1 sh -c \ +ssh james@bihar 'sudo podman exec hives-school-1 sh -c \ "curl -sS -H \"X-Hive-Internal: \$HIVE_DASHBOARD_TOKEN\" http://127.0.0.1:3002/api/status"' \ | jq '.agents[] | {name, paused, state}' ``` @@ -227,11 +277,12 @@ PRs and issues (`gh search prs --author "hanthor-hive-agent[bot]"` and ### Known limits -- **Memory.** Spokes are capped at 2560 MB each (raised from 2 GB on 2026-11-13 - after hive was OOM-killed during rustc/govulncheck builds). The hub allows 2 - concurrent shared-pool agent sessions across both spokes (about 400 MB per - Claude session; the host had about 5 GB free and no swap). Do not raise either - without revisiting the host's headroom. +- **Memory.** On bihar and karnataka, each spoke can use 8 GB and the hub allows 4 + concurrent shared-pool sessions (`compose..yaml`). On spindle the limits were + 2560 MB per spoke and 2 sessions, and spokes were OOM-killed during + rustc/govulncheck builds. +- **Pinned images.** The hive and hive-hub images are pinned by digest with the + amd64 fork binary `hive-2ef7dc7c0-amd64`. Do not change to `:v6-latest`. - **Hive's own CI** for `tuna-os/hive` is blocked by GitHub ("Actions has been disabled"; a support ticket is pending), so fork builds are not CI-verified. - **Reply spam**: upstream @@ -240,9 +291,10 @@ PRs and issues (`gh search prs --author "hanthor-hive-agent[bot]"` and recognise its own earlier reply when posting as an App login. - **Concurrent edits and drift**: see the layout notes above; verify after every restart that the parsed seed and runtime agree on the roster. -- Not covered here: the hub's own config, the `omp-auth-broker` user service - the spokes call (`OMP_AUTH_BROKER_URL`), and Caddy routing, which were not - re-inspected for this page. The `hive.reilly.asia` hanthor spoke of the old AWS +- The spokes call the `omp-auth-broker` on spindle (`OMP_AUTH_BROKER_URL`, + `http://100.113.75.88:8765`). Caddy routing and failover are described in + [Hives hosting and failover](../servers/hives.md). This page does not describe + the hub's own configuration. The `hive.reilly.asia` hanthor spoke of the old AWS cluster has no container on this host; its current status is unverified. The [hive_ops role](../roles/hive_ops.md) targets the retired cluster, not this deployment. diff --git a/docs/src/desktop/himachal/README.md b/docs/src/desktop/himachal/README.md index a67782a7..2849e311 100644 --- a/docs/src/desktop/himachal/README.md +++ b/docs/src/desktop/himachal/README.md @@ -25,7 +25,7 @@ Laptop workstation in the hanthor fleet. Kubernetes, which was terminated on 2026-10-25; the host-vars comment saying those jobs now run in-cluster is stale. See [hive_ops](../../roles/hive_ops.md) and the - [current Hive deployment](../../cluster/hive.md#current-deployment-spindle-vps). + [current Hive deployment](../../cluster/hive.md#current-deployment-bihar-karnataka-standby). The fleet's GNOME role includes [PaperWM](https://github.com/paperwm/PaperWM), PaperShell and Copyous; their enabled state on this host was not checked. diff --git a/docs/src/fleet.md b/docs/src/fleet.md new file mode 100644 index 00000000..d550eaa6 --- /dev/null +++ b/docs/src/fleet.md @@ -0,0 +1,67 @@ +# Fleet status + +Snapshot of **2026-11-18**, made from `tailscale status` and the live hosts. +Update this page when a host changes role. To refresh it, run +`tailscale status --json | jq -r '.Peer[] | [.HostName,.Online,.LastSeen] | @tsv'`. + +## How it fits together + +``` +Internet ─▶ Cloudflare ─▶ spindle (Oracle, Frankfurt): Caddy, Matrix, LiveKit, OMP auth broker, hive failover coordinator + │ tailnet + ┌───────────────────┴────────────────────┐ + ▼ ▼ + bihar (home, WiFi) karnataka (home) + k3s + KubeStellar hub k3s + KubeVirt + ARC runners + LMS + hives: ACTIVE ───── hive-sync 15 min ───▶ hives: cold standby + └─ VM: punjab (agent box, Ansible host) +``` + +## Live hosts + +| Host | Kind | OS | Role | Docs | +|---|---|---|---|---| +| **bihar** | Home server (Intel, 62 GB) | Bluefin utah:testing | Active hive host; k3s + KubeStellar hub | [Bihar](servers/bihar.md) | +| **karnataka** | Home server (AMD Strix Halo, 125 GB) | Bluefin utah:testing | k3s: KubeVirt VMs, CI runners, LMS, Corral; hive standby | [Karnataka](servers/karnataka.md) | +| **spindle** | Oracle A1 VM, eu-frankfurt-1 (2 ARM cores, 12 GB) | Ubuntu 24.04 | Public front door (Caddy, `92.5.82.232`), Matrix (`reilly.asia`), LiveKit, OMP auth broker, hive failover coordinator | [Spindle VPS](servers/spindle-vps/README.md) | +| **punjab** | KubeVirt VM on karnataka (12 vCPU, 23 GB, 500 GB) | Ubuntu 24.04 | Agent box and fleet Ansible host. **Disk 96% full.** | [Punjab](servers/punjab/README.md) | +| **corral** | Tailscale sidecar pod on karnataka | n/a | Tailnet name for `corral-web` | [Karnataka](servers/karnataka.md) | +| **pixel-9a** (`termux`) | Phone | Android + Termux | Ansible `termux` host | [Inventory](inventory.md) | + +Cross-host services: +- The hives (school, reef, hub) are on bihar. If bihar fails, they move to karnataka. See [Hives hosting](servers/hives.md). +- KubeStellar is described in [Home k3s and KubeStellar](servers/home-k3s.md). + +## Intermittent (desktops and laptops) + +| Host | Last seen | Notes | Docs | +|---|---|---|---| +| kerala (`kerala-1`) | 2026-11-17 | Desktop. Earlier tailnet node `kerala` last seen 2026-08-29 | [Kerala](desktop/kerala/README.md) | +| himachal / himachal-1 | 2026-11-09 / 2026-11-10 | Desktop. Two tailnet nodes; remove the earlier one | [Himachal](desktop/himachal/README.md) | +| dilli | 2026-10-21 | Desktop | [Dilli](desktop/dilli/README.md) | +| kanpur | 2026-07-21 | Desktop or Windows test machine | [Kanpur](desktop/kanpur/README.md) | +| mumbai | 2026-07-20 | Debian VM on the phone | [Inventory](inventory.md) | +| goa | 2026-07-08 | Server | [Goa](servers/goa/README.md) | +| MacBook Air (2 nodes), CMF Phone 1 | 2026-10 | Personal devices, not managed | none | + +## Retired + +| Host | Retired | Notes | Docs | +|---|---|---|---| +| Home Talos cluster (bihar and karnataka on Talos) | 2026-10/11 | Replaced by Bluefin + k3s on the same hardware | [Archive](archive/talos-k8s.md) | +| AWS Talos cluster (eu-north-1) | 2026-10-25 | Terminated. Configurations kept as recovery material | [Archive](servers/aws-k8s/cluster.md) | +| punjab EC2 (`punjab-aws-retired`, 100.78.73.8) | 2026-10-28 | Replaced by the punjab VM on karnataka | [Punjab](servers/punjab/README.md) | +| matrix (`reilly-asia-matrix`), telengana | 2026-09 | Hetzner VPSes | [Archive](archive/vps/matrix/README.md) | +| Hives on spindle (`/srv/hives`) | 2026-11-18 | Moved to bihar. The copy is stopped and kept for rollback | [Hives hosting](servers/hives.md) | + +### Tailnet nodes to remove + +The tagged devices from the retired clusters are offline and can be deleted from +the Tailscale admin console. On 2026-11-18 these were: +- since 2026-07: `appflowy*`, `argocd`, `auth`, `forgejo`, `grafana`, `loki`, `prometheus`, `n8n`, `lemonade`, `home`, `kubevirt-manager*`, `default-deepseek-14b`, `repro-ctr-vm-vm`, `gate-*-vm` +- since 2026-10-25: `hive`, `hive-auth-egress-0/1`, `review`, `search`, `spindle-grafana`, `aws-migration-operator`, `tuna-v1-preflight-*` +- since 2026-09: `tailscale-operator*`, `secret-broker` +- since 2026-04: `uisp`, `chuppvm` +- since 2026-07 and 2026-08: `vm`, `WIN-G9NAVN79GN0` + +[hanthor/dotfiles#333](https://github.com/hanthor/dotfiles/pull/333) (`scripts/tailnet.py`) automates this cleanup. diff --git a/docs/src/index.md b/docs/src/index.md index de66b1cc..2c88ec20 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -3,8 +3,10 @@ Ansible-driven dotfiles and infrastructure for a personal fleet. Three distinct concerns live here: 1. **Workstation config** — shells, packages, browser, SSH, Tailscale, kubeconfig, GNOME, and more -2. **Home Talos K8s cluster** — manifests for the Bihar + Karnataka cluster -3. **Hosted infrastructure** — OCI spindle and AWS account configuration; the AWS Talos cluster is terminated +2. **Home servers** — bihar and karnataka (Bluefin + k3s, linked by KubeStellar); the hives run on bihar with karnataka as standby. The [Talos manifests](archive/talos-k8s.md) are archived +3. **Hosted infrastructure** — OCI spindle (Matrix, public front door, hive failover coordinator) and the AWS account; the AWS Talos cluster is terminated + +Current state of every host: **[Fleet status](fleet.md)**. > **This handbook is public.** It covers architecture, decisions and runbooks. > Never put secrets in it. Fleet addresses live in inventory or come from @@ -40,7 +42,8 @@ Ansible-managed hosts, from `inventory.yml`: | dilli | desktop | Desktop | Secondary workstation | | kerala | desktop | Mobile | postmarketOS ARM device | | mumbai | desktop | VM | Debian VM on the phone, `machine_profile: cli-only` | -| punjab | server | Server | EC2 agent box in us-east-1 | +| punjab | server | VM | Agent box, KubeVirt VM on karnataka | +| bihar | desktop | Server | Home server: active hive host, KubeStellar hub | | goa | server | Server | [Raspberry Pi 5](https://www.raspberrypi.com/products/raspberry-pi-5/) ARM server | | vm | server | VM | Local dev VM | | termux | termux_hosts | Phone | Raw Termux/Android layer | @@ -48,14 +51,11 @@ Ansible-managed hosts, from `inventory.yml`: | matrix | vps | VPS | **Retired** — see the cutover runbook | | telengana | vps | VPS | **Retired** — see the cutover runbook | -Not Ansible-managed (Talos — `talosctl`/`kubectl` only): - -- **bihar** (control plane) + **karnataka** (worker, AMD GPU) — the home cluster. - Powered down and in storage, expected back. See the [cluster handbook](servers/talos-k8s/cluster.md). -- The [AWS Talos cluster](servers/aws-k8s/cluster.md) was terminated on 2026-10-25; its handbook is historical. +Outside this inventory: -[Spindle](servers/spindle-vps/README.md) is an OCI Frankfurt VPS running Matrix, -standalone LiveKit, School, Reef and Hub. It is not currently listed in `inventory.yml`. +- **karnataka** — home k3s/KubeVirt server, configured from the native-units branch. See [Karnataka](servers/karnataka.md). +- [Spindle](servers/spindle-vps/README.md) — OCI Frankfurt VPS: Matrix, LiveKit, the Caddy front door for the hives, and the [hive failover](servers/hives.md) coordinator. +- The [AWS Talos cluster](servers/aws-k8s/cluster.md) was terminated on 2026-10-25, and the home Talos cluster was replaced by k3s ([archive](archive/talos-k8s.md)). The `llm` group is dormant (commented out in `inventory.yml`). diff --git a/docs/src/inventory.md b/docs/src/inventory.md index 26866a22..4adbc011 100644 --- a/docs/src/inventory.md +++ b/docs/src/inventory.md @@ -25,7 +25,7 @@ Headless hosts. ```yaml server: hosts: - punjab: # EC2 agent box (us-east-1; current instance status unverified) + punjab: # agent box, KubeVirt VM on karnataka (100.103.187.46) vm: # Local dev VM goa: # Raspberry Pi 5 (ARM) ``` @@ -71,14 +71,15 @@ termux_hosts: ``` ### `llm` (dormant) -Commented out in `inventory.yml` — its only member, karnataka, is now a Talos -worker. +Commented out in `inventory.yml` — karnataka, its former member, now runs k3s workloads. ### Not in the inventory -**bihar** and **karnataka** are Talos nodes (no SSH) and are managed with -`talosctl`/`kubectl` only. The home cluster is powered down and in storage, -expected back. The terminated AWS cluster's nodes were not in the inventory either. +**bihar** is in the `desktop` group (Bluefin, bootstrapped 2026-11-17). +**karnataka** is configured from the native-units branch +([#348](https://github.com/hanthor/dotfiles/pull/348)) and is not listed here. +Neither is a Talos node any more; see [Fleet status](fleet.md). **spindle** is +not in the inventory. The terminated AWS cluster's nodes were not in the inventory either. ## Per-Machine Configuration diff --git a/docs/src/network.md b/docs/src/network.md index a1e137c0..b357bcba 100644 --- a/docs/src/network.md +++ b/docs/src/network.md @@ -10,8 +10,7 @@ crosswalk. | Name | Role | |------|------| | gateway | Router (DNS, DHCP) — see [Gateway](network/gateway/README.md) | -| bihar | Talos K8s control plane (powered down, expected back) — reserved address | -| karnataka | Talos K8s worker, AMD GPU (powered down, expected back) — reserved address | +| karnataka | Home server: k3s, KubeVirt, AMD GPU — reserved address | | goa | Fleet control, Debian ARM (Raspberry Pi) — reserved address | | phone | Android phone (reserved address) | | printer | Network printer (DHCP) | @@ -28,7 +27,7 @@ tailnet address where one is needed. | Hostname | LAN | Group | |----------|-----|-------| | goa | reserved | server (applies locally) | -| punjab | — (AWS) | server | +| punjab | — (VM on karnataka) | server | | himachal | DHCP | desktop | | dilli | DHCP | desktop | | kanpur | DHCP | desktop | @@ -37,5 +36,5 @@ tailnet address where one is needed. | termux (ssh `:8022`) | phone | termux_hosts | | matrix | remote | vps — retired | | telengana | remote | vps — retired | -| bihar | reserved | Talos, not in inventory — offline | -| karnataka | reserved | Talos, not in inventory — offline | +| bihar | — (separate site, WiFi) | desktop — home server, hive host | +| karnataka | reserved | not in inventory (native units) — k3s host | diff --git a/docs/src/roles/hive_ops.md b/docs/src/roles/hive_ops.md index 8951697f..9364d40c 100644 --- a/docs/src/roles/hive_ops.md +++ b/docs/src/roles/hive_ops.md @@ -7,7 +7,7 @@ > **Retired AWS-cluster integration, not the current Hive operations guide.** > The AWS Talos cluster was terminated on 2026-10-25. The current school and > reef spokes and hub run as Docker Compose services on the -> [Spindle VPS](../cluster/hive.md#current-deployment-spindle-vps). Do not +> [bihar (karnataka standby)](../cluster/hive.md#current-deployment-bihar-karnataka-standby). Do not > enable this role or run its old Kubernetes helpers against that deployment. The role still exists in the repository. When enabled it installs host user diff --git a/docs/src/servers/bihar.md b/docs/src/servers/bihar.md new file mode 100644 index 00000000..0694058c --- /dev/null +++ b/docs/src/servers/bihar.md @@ -0,0 +1,57 @@ +# Bihar + +Home server, the primary host for the hives, and the KubeStellar hub. Since +2026-11-17 it runs Bluefin, not Talos. The [Talos page](../archive/talos-home-cluster/bihar/README.md) +is archived. + +| | | +|---|---| +| Hardware | ASRock B660M-STX, Intel, 24 threads, 62 GB of RAM | +| Disk | 1.9 TB NVMe (`/var`) | +| OS | Bluefin `ghcr.io/projectbluefin/utah:testing` (bootc: read-only `/usr`, writable `/etc` and `/var`) | +| Network | **WiFi** (`wlp3s0`). Not on the same LAN as karnataka. | +| Tailnet | `bihar` / `100.103.44.48` (`bihar.manatee-basking.ts.net`) | +| Access | `ssh james@bihar` (passwordless sudo through `/etc/sudoers.d/zz-james`) | +| Ansible | Yes: [`host_vars/bihar.yml`](https://github.com/hanthor/dotfiles/blob/master/host_vars/bihar.yml), `skip_pipewire_audio: true` | + +## Roles + +1. **Active hive host.** The hive containers (hub, hub-front, school and reef) run + under rootful podman compose, and spindle's Caddy sends public traffic to them. + See [Hives hosting and failover](hives.md). +2. **k3s and KubeStellar hub.** A single-node k3s runs KubeStellar 0.30.2, with + ITS `its1` and WDS `wds1`. bihar and karnataka are both workload clusters. + See [Home k3s and KubeStellar](home-k3s.md). + +## Host settings (applied 2026-11-17) + +- **No sleep:** the sleep, suspend and hibernate targets are masked, the + `90-nosleep.conf` drop-ins are in `/etc/systemd/{sleep,logind}.conf.d/`, and + the GDM dconf `sleep-inactive-ac-type='nothing'` is set. Before this change, + GNOME suspended the host after 15 min idle, and the host went offline. +- **Default target is `multi-user.target`:** no graphical login. Use + `systemctl isolate graphical.target` to start one. +- **Wake-on-LAN** is set to `magic` on `enp0s31f6`. The host uses WiFi, so + wake-on-LAN works only if you connect a cable. +- **To do:** in the firmware, set "Restore on AC/Power Loss" to **Power On**. + This setting is necessary because the hives depend on this host. +- Dotfiles bootstrap fixes for a new bootc host are in + [hanthor/dotfiles#385](https://github.com/hanthor/dotfiles/pull/385). +- Bitwarden phase 2 is not done yet. Run `just apply -e skip_pipewire_audio=true` from an + interactive terminal. + +## Timers + +| Unit | Purpose | +|---|---| +| `hive-reconcile.timer` | Follows the hive lease on spindle (every 1 min) | +| `hive-sync.timer` | Pushes the hive data to karnataka while bihar is active (every 15 min) | +| `fleet-osquery-report.timer` | Fleet facts | +| `tailscale-cert-renew.timer` | Tailscale TLS certificate | + +## Gotchas + +- Do not use `pkill -f ` over SSH if the pattern also occurs in the SSH + command line. The command stops your own session. +- `gsettings` over SSH needs `DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus`. +- `semanage` cannot write on bootc. Use `chcon` for SELinux labels. diff --git a/docs/src/servers/hives.md b/docs/src/servers/hives.md new file mode 100644 index 00000000..90cf771b --- /dev/null +++ b/docs/src/servers/hives.md @@ -0,0 +1,198 @@ +# Hives hosting and failover + +> **TunaOS infrastructure.** This page describes *where and how* the hives run. +> For the hives themselves (agents, configuration, health), see the +> [Hive operations guide](../cluster/hive.md). + +Last updated: 2026-11-18. The source for the scripts, compose layers and systemd +units is [`hives/`](https://github.com/hanthor/dotfiles/tree/master/hives) in this +repository. Each host also has a copy of this runbook at +`/var/srv/hives-host/RUNBOOK.md` (on spindle: `/var/lib/hive-failover/RUNBOOK.md`). + +## Overview + +The tuna-os hives run as one compose project named `hives`, with four containers: + +| Service | Purpose | Public URL | +|---|---|---| +| `hub` | Hub API that all spokes report to | `https://hub.tunaos.org/api/*`, `/login/*` | +| `hub-front` | Static hub page (nginx) | `https://hub.tunaos.org/` | +| `school` | Spoke for the tuna-os org (GitHub App 3942065) | `https://school.tunaos.org` | +| `reef` | Spoke (GitHub App 5546574) | `https://reef.tunaos.org` | + +Until 2026-11-18 the hives ran on spindle (Oracle ARM VM, Frankfurt, 2 cores and 11 GB of RAM). They now run on **bihar** (x86, 24 cores, 62 GB of RAM). **karnataka** is a cold standby. **spindle** keeps the public IP and Caddy, and it coordinates failover. + +``` +Internet ─▶ Cloudflare ─▶ spindle Caddy (92.5.82.232) + │ tailnet, ports 3100-3103 + ┌──────────────┴──────────────┐ + ▼ (active, holds the lease) ▼ (standby, stopped) + bihar 100.103.44.48 karnataka 100.66.158.127 + └── hive-sync every 15 min ──▶ +``` + +Outbound traffic from the hives (to GitHub and the model APIs) leaves through the active host's own internet connection. It does not go through spindle. + +The OMP auth broker still runs on spindle at `http://100.113.75.88:8765`, reached over the tailnet. + +## Host layout (bihar and karnataka are identical) + +| Path | Contents | +|---|---| +| `/var/srv/hives/` | Hive data and configuration, copied from spindle's `/srv/hives`: `compose.yaml`, `compose.override.yaml`, `.env`, `{hub,school,reef}/{config,data,secrets}`, `hub-front/`, `proxy/`, `bin/dashboard-*.html` | +| `/var/srv/hives-host/compose..yaml` | Host layer: ports, memory limits, broker URL, image digests, and the amd64 binary mount | +| `/var/srv/hives-host/bin/hive-2ef7dc7c0-amd64` | `tuna-os/hive` at `2ef7dc7c0` (branch `deploy/upstream-v6-20261113`) built with `CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath ./cmd/hive` | +| `/var/srv/hives-host/hives` | Wrapper that runs docker-compose against rootful podman with the three compose files | +| `/var/srv/hives-host/hive-reconcile` | Makes the host follow the lease (timer, every 1 min) | +| `/var/srv/hives-host/hive-sync` | Pushes the data to the peer host (timer, every 15 min) | +| `/var/srv/hives-host/keys/` | `lease` (read-only access to the lease on spindle) and `sync` (rsync to the peer host) | +| `/var/srv/hives-host/maintenance` | If this file exists, reconcile does not start the hives | + +Each host publishes its ports on its own tailnet IP only: + +| Port | Service | +|---|---| +| 3100 | hub-front | +| 3101 | hub | +| 3102 | school | +| 3103 | reef | + +A firewalld rich rule in the `public` zone accepts these ports only from spindle (`100.113.75.88`). Inside the compose network the services reach each other by name, for example `http://hub:3001`. The podman network is named `spindle_default` so that `compose.yaml` works unchanged. + +Images are pinned by digest, and the pin must match the binary: +- `ghcr.io/hivecommons/hive@sha256:d56425fb…` (school and reef) +- `ghcr.io/hivecommons/hive-hub@sha256:3c0162eb…` (hub) + +Do not change to `:v6-latest`. On 2026-11-18 a newer image made the hub log "failed to write token cache". **Upgrade the binary and the images together.** + +## Daily operations (on the active host) + +```bash +sudo /var/srv/hives-host/hives ps +sudo /var/srv/hives-host/hives logs -f --tail 50 school +sudo /var/srv/hives-host/hives restart reef +sudo podman stats --no-stream +``` + +To find the active host, run this on spindle: + +```bash +sudo hive-failover status +``` + +The command shows the lease, the health and container count of each host, and the targets in Caddy. + +### Change the hive configuration +1. Edit the files under `/var/srv/hives/` on the active host only. +2. Run `sudo /var/srv/hives-host/hives up -d`. + +The next `hive-sync` copies the change to the standby. + +### Change a host-specific setting +1. Edit `/var/srv/hives-host/compose..yaml` on **both** hosts. This covers ports, memory and image pins. +2. Run `sudo chcon -t container_file_t` on the file. + +### Upgrade the hive binary +1. Build the amd64 binary from the fork commit. +2. Copy it to `/var/srv/hives-host/bin/` on both hosts. +3. Change the binary mount and the image digests in both host files. +4. Run `hives up -d` on the active host. + +## Failover + +### The lease +The file `spindle:/var/lib/hive-failover/active` contains `bihar`, `karnataka` or `none`. Only the host named in the lease may run the hives. This rule prevents two hosts from running the same GitHub Apps at once. + +| Component | Where | Schedule | What it does | +|---|---|---|---| +| `hive-failover check` | spindle, `hive-failover.timer` | every 1 min | Checks the active host on ports 3101-3103. After 5 min of failures it sets the lease to `none`, stops the old host if it can, and waits. When the old host has 0 containers, or after 10 min, it starts the standby, sets the lease, changes Caddy and sends a ntfy notification on `$NTFY_TOPIC`. | +| `hive-reconcile` | both hosts, `hive-reconcile.timer` | every 1 min | If the lease names this host and fewer than 4 containers run, it starts them (unless `maintenance` exists). If the lease names a different host, it stops them. If it cannot read the lease for 8 min, it stops them (self-fence). | +| `hive-sync` | both hosts, `hive-sync.timer` | every 15 min | If the lease names this host, it uses rsync to push `/var/srv/hives` to the peer, without caches and toolchains. | +| `hive-cleanup` | both hosts, `hive-cleanup.timer` | every 1 h | On running spokes only: removes idle Claude session scratch and stale work dirs in `/tmp` inside the containers, removes dangling images and old backups, and warns when `/var` has less than 10 GB free. Ported from spindle, where the writable layers of the hive containers had reached 16.8 GB. | + +Logs: + +```bash +journalctl -t hive-failover # on spindle +journalctl -u hive-reconcile -u hive-sync # on the hosts +``` + +The podman containers have restart policy `unless-stopped` and `podman-restart.service` is disabled. As a result, a host that boots does not start the hives itself. Only reconcile starts them, and only when the lease names that host. + +### Commands (on spindle) + +```bash +sudo hive-failover status +sudo hive-failover move bihar # planned move or failback: stop, sync, start, Caddy +sudo hive-failover move karnataka +sudo hive-failover hold on # pause automatic failover (for example during maintenance) +sudo hive-failover hold off +``` + +`move` sets `hold on` while it runs, and clears it when it succeeds. If `move` fails, the lease stays at `none` and hold stays on. Correct the problem and run `move` again. + +### After an automatic failover +The hives run on karnataka. Failback is manual, so that an unstable WiFi connection cannot move the hives back and forth. +1. Make sure bihar is up: `ssh james@bihar uptime`. +2. Run `sudo hive-failover move bihar` on spindle. It takes about 1 min. + +### Planned maintenance on bihar +Use one of these methods: +- Run `sudo hive-failover move karnataka` first. +- Or run `sudo hive-failover hold on` on spindle, then `sudo touch /var/srv/hives-host/maintenance` and `hives stop` on bihar. When you finish, remove the file and run `hold off`. + +### Limits +- The data on the standby can be up to 15 min old. Rsync copies files while they are in use, so a file that changes during the copy can arrive in an inconsistent state. +- The self-fence after 8 min without spindle was tested by hand only. A real network partition has not been tested. +- If spindle is down, there is no public access and no failover. The hosts stop the hives after 8 min, because they cannot read the lease. + +### Drill on 2026-11-18 +1. Hives stopped on bihar at 13:55:44Z. +2. Spindle revoked the lease at 14:01:29Z. +3. Karnataka was healthy and Caddy changed at 14:01:56Z. +4. `move bihar` completed in 50 s, including the sync back. + +## Gotchas +- **SELinux (bihar and karnataka are bootc and enforcing):** + - Files mounted into containers must have the type `container_file_t`. `semanage` cannot write on bootc, so `hive-sync` runs `chcon -R` after each copy. Do not copy SELinux attributes with rsync (`-X`). + - systemd cannot run scripts that have the label `var_t` (the result is `203/EXEC Permission denied`). For this reason the units start the scripts as `/usr/bin/bash /var/srv/hives-host/