From bcd85b9441e35c22b5e809ffe1154e019779da06 Mon Sep 17 00:00:00 2001 From: James Reilly Date: Wed, 18 Nov 2026 18:20:27 +0000 Subject: [PATCH 1/3] docs: document the live fleet, add hive failover and home k3s kits, archive Talos Bring the handbook and repo in line with the fleet as of 2026-11-18: - Add docs/src/fleet.md: every tailnet host as live, intermittent or retired, plus a list of offline tagged nodes to remove. - Add host pages for bihar and karnataka (Bluefin + k3s, no longer Talos), a home k3s + KubeStellar page, and a hives hosting/failover runbook. - Add hives/: the failover lease scripts (hive-failover on spindle; hive-reconcile and hive-sync on the hosts), compose host layers and systemd units, byte-identical to what is deployed. - Add home-k3s/{bihar,karnataka}: the k3s install kits and karnataka's workload manifests (secrets are out of band; only secretKeyRefs here). - Update punjab (now a KubeVirt VM on karnataka), spindle (hives moved off; MAS replaced by Spindle OIDC; v0.1.6), the hive guide's current deployment section, README, index, inventory, network, CONTEXT and CLAUDE.md. - inventory.yml: point punjab at its VM (the old address was the retired EC2 instance) and add bihar as bootstrap.sh registered it. - Archive: move the home Talos handbooks and the retired Hetzner VPS pages under docs/src/archive/, banner talos-k8s/ (kept for tests), and mark the home-k8s server images as not deployed. Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 47 ++- CONTEXT.md | 9 +- README.md | 23 +- docs/src/SUMMARY.md | 30 +- .../talos-home-cluster}/bihar/README.md | 2 + .../talos-home-cluster}/cluster.md | 8 +- .../talos-home-cluster}/karnataka/README.md | 2 + docs/src/{ => archive}/talos-k8s.md | 8 +- docs/src/{ => archive}/vps/matrix/README.md | 6 +- .../src/{ => archive}/vps/telengana/README.md | 6 +- docs/src/automation.md | 4 +- docs/src/cluster/hive.md | 69 +++-- docs/src/desktop/himachal/README.md | 2 +- docs/src/fleet.md | 67 +++++ docs/src/index.md | 20 +- docs/src/inventory.md | 13 +- docs/src/network.md | 9 +- docs/src/roles/hive_ops.md | 2 +- docs/src/servers/bihar.md | 57 ++++ docs/src/servers/hives.md | 194 ++++++++++++ docs/src/servers/home-k3s.md | 79 +++++ docs/src/servers/home-k8s-images.md | 7 +- docs/src/servers/karnataka.md | 59 ++++ docs/src/servers/punjab/README.md | 90 +++--- docs/src/servers/spindle-vps/README.md | 35 ++- hives/README.md | 19 ++ hives/compose.bihar.yaml | 37 +++ hives/compose.karnataka.yaml | 37 +++ hives/hive-failover | 137 +++++++++ hives/hive-reconcile | 41 +++ hives/hive-sync | 31 ++ hives/hives | 8 + hives/systemd/hive-failover.service | 7 + hives/systemd/hive-failover.timer | 8 + hives/systemd/hive-reconcile.service | 7 + hives/systemd/hive-reconcile.timer | 8 + hives/systemd/hive-sync.service | 7 + hives/systemd/hive-sync.timer | 8 + home-k3s/README.md | 13 + home-k3s/bihar/KUBESTELLAR.md | 19 ++ home-k3s/bihar/config.yaml | 19 ++ home-k3s/bihar/install-host.sh | 33 +++ home-k3s/bihar/k3s.service | 25 ++ home-k3s/bihar/reset-k3s.sh | 13 + home-k3s/bihar/resolv.conf | 4 + home-k3s/bihar/wait-tailnet.sh | 11 + home-k3s/karnataka/amdgpu-device-plugin.yaml | 46 +++ home-k3s/karnataka/apply-stack.sh | 30 ++ home-k3s/karnataka/config.yaml | 12 + home-k3s/karnataka/corral-web.yaml | 222 ++++++++++++++ home-k3s/karnataka/frappe-lms.yaml | 279 ++++++++++++++++++ home-k3s/karnataka/front-door.yaml | 98 ++++++ home-k3s/karnataka/install-host.sh | 20 ++ home-k3s/karnataka/k3s.service | 25 ++ home-k3s/karnataka/lemonade.yaml | 103 +++++++ home-k3s/karnataka/local-path-profile.yaml | 7 + home-k3s/karnataka/resolv.conf | 4 + home-k3s/karnataka/wait-tailnet.sh | 11 + inventory.yml | 21 +- talos-k8s/README.md | 6 + 60 files changed, 2033 insertions(+), 191 deletions(-) rename docs/src/{servers/talos-k8s => archive/talos-home-cluster}/bihar/README.md (89%) rename docs/src/{servers/talos-k8s => archive/talos-home-cluster}/cluster.md (99%) rename docs/src/{servers/talos-k8s => archive/talos-home-cluster}/karnataka/README.md (90%) rename docs/src/{ => archive}/talos-k8s.md (81%) rename docs/src/{ => archive}/vps/matrix/README.md (84%) rename docs/src/{ => archive}/vps/telengana/README.md (74%) create mode 100644 docs/src/fleet.md create mode 100644 docs/src/servers/bihar.md create mode 100644 docs/src/servers/hives.md create mode 100644 docs/src/servers/home-k3s.md create mode 100644 docs/src/servers/karnataka.md create mode 100644 hives/README.md create mode 100644 hives/compose.bihar.yaml create mode 100644 hives/compose.karnataka.yaml create mode 100755 hives/hive-failover create mode 100755 hives/hive-reconcile create mode 100755 hives/hive-sync create mode 100755 hives/hives create mode 100644 hives/systemd/hive-failover.service create mode 100644 hives/systemd/hive-failover.timer create mode 100644 hives/systemd/hive-reconcile.service create mode 100644 hives/systemd/hive-reconcile.timer create mode 100644 hives/systemd/hive-sync.service create mode 100644 hives/systemd/hive-sync.timer create mode 100644 home-k3s/README.md create mode 100644 home-k3s/bihar/KUBESTELLAR.md create mode 100644 home-k3s/bihar/config.yaml create mode 100755 home-k3s/bihar/install-host.sh create mode 100644 home-k3s/bihar/k3s.service create mode 100644 home-k3s/bihar/reset-k3s.sh create mode 100644 home-k3s/bihar/resolv.conf create mode 100755 home-k3s/bihar/wait-tailnet.sh create mode 100644 home-k3s/karnataka/amdgpu-device-plugin.yaml create mode 100755 home-k3s/karnataka/apply-stack.sh create mode 100644 home-k3s/karnataka/config.yaml create mode 100644 home-k3s/karnataka/corral-web.yaml create mode 100644 home-k3s/karnataka/frappe-lms.yaml create mode 100644 home-k3s/karnataka/front-door.yaml create mode 100755 home-k3s/karnataka/install-host.sh create mode 100644 home-k3s/karnataka/k3s.service create mode 100644 home-k3s/karnataka/lemonade.yaml create mode 100644 home-k3s/karnataka/local-path-profile.yaml create mode 100644 home-k3s/karnataka/resolv.conf create mode 100755 home-k3s/karnataka/wait-tailnet.sh diff --git a/CLAUDE.md b/CLAUDE.md index f9198029..c255ff16 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,27 +8,19 @@ Ansible-driven dotfiles + infra for a personal fleet (desktops, servers, VPS). E Three distinct concerns live here: 1. **Workstation config** (`roles/`, `site.yml`, `host_vars/`, `group_vars/`) — shells, packages, browser, SSH, Tailscale, kubeconfig, etc. -2. **Talos K8s cluster IaC** (`talos-k8s/`) — manifests for the Bihar + Karnataka cluster. Detailed handbook in [`docs/src/servers/talos-k8s/cluster.md`](docs/src/servers/talos-k8s/cluster.md). +2. **Home servers** (`home-k3s/`, `hives/`) — bihar and karnataka run Bluefin with a separate single-node k3s each, linked by KubeStellar; the hives run on bihar with karnataka as cold standby. `talos-k8s/` is the **archived** Talos-era IaC (kept for tests/history; do not apply). Current state of every host: [`docs/src/fleet.md`](docs/src/fleet.md). 3. **TunaOS AWS account IaC** (`aws/`, OpenTofu) — the account is **TunaOS infrastructure**, not the personal fleet: punjab's instance, backups (S3), IAM, budgets, and (terminated, still in the code) the AWS Talos cluster's VPC/nodes/EIPs. Credits are exhausted: everything left here costs real money. Handbook: [`docs/src/servers/aws/README.md`](docs/src/servers/aws/README.md). -### Two Talos clusters, not one +### Where things run (2026-11-18 — see [`docs/src/fleet.md`](docs/src/fleet.md)) -- **Home** (`bihar` + `karnataka`) — on the LAN. Configs: `~/.kube/config`, `~/.talos/config`. - **Currently powered down and in storage; expected back.** Treat it as - temporarily offline, not decommissioned — don't delete its configs, Bitwarden - notes, or `talos-k8s/` manifests, and expect `just doctor`/`kubectl` against it - to fail until it returns. -- **AWS** (`eu-north-1`, built 2026-08-27) — **terminated 2026-10-25** after credits were exhausted. It previously ran Matrix, the Hives and the CFP dashboard. The S3 bucket `hanthor-fleet-backups-*` and the himachal-1 mirror (`~/backups/tunaos-aws-2026-10-25`) were retained; verify individual AWS resources rather than assuming they are gone. OpenTofu still describes the old cluster and can propose recreating resources. - Configs retained as recovery material: `~/.kube/config-aws-migration`, `~/.talos/config-aws-migration`. - Handbook: [`docs/src/servers/aws-k8s/cluster.md`](docs/src/servers/aws-k8s/cluster.md). -- **Spindle VPS** — Matrix, standalone LiveKit, School, Reef and Hub run on an Oracle Always Free A1 VM in `eu-frankfurt-1`, 2 OCPU / 12 GB, joined to the tailnet and locked down. The kit and runbook are in [`docs/src/servers/spindle-vps/`](docs/src/servers/spindle-vps/README.md); Oracle API key/config are in Bitwarden. Cloudflare DNS failover is manual, not automatic. School and Reef (Compose under `/srv/hives`) are documented, with roster, identity and the safe edit procedure, in [`docs/src/cluster/hive.md`](docs/src/cluster/hive.md); more than one operator edits that directory. +- **bihar** — Bluefin (`utah:testing`), WiFi, `100.103.44.48`, `ssh james@bihar`. Single-node k3s + **KubeStellar hub** (ITS `its1`, WDS `wds1`). **Active hive host**: hub, hub-front, school, reef as rootful-podman compose in `/var/srv/hives` (wrapper `/var/srv/hives-host/hives`). In `inventory.yml` (`desktop`). Handbook: [`docs/src/servers/bihar.md`](docs/src/servers/bihar.md). +- **karnataka** — Bluefin, `100.66.158.127`. Single-node k3s with KubeVirt (VMs incl. **punjab**), ARC runners (`tuna-ci`), Frappe LMS, Corral, front-door; KubeStellar WEC; **hive cold standby**. Configured from the native-units branch (#348), not in `inventory.yml`. Handbook: [`docs/src/servers/karnataka.md`](docs/src/servers/karnataka.md). +- **spindle** — Oracle A1 VM in `eu-frankfurt-1` (2 OCPU, ARM): Matrix (Spindle + built-in OIDC), LiveKit, Caddy (public `92.5.82.232`, the front door for `school`/`reef`/`hub.tunaos.org`), the OMP auth broker, and the **hive failover coordinator** (lease `/var/lib/hive-failover/active`, `sudo hive-failover status|move|hold`). Kit/runbook: [`docs/src/servers/spindle-vps/`](docs/src/servers/spindle-vps/README.md); Oracle API key/config in Bitwarden. Not in `inventory.yml`. +- **punjab** — KubeVirt VM on karnataka (`corral-vms/punjab`), Ubuntu, `100.103.187.46`, user `ubuntu`. Agent box and this repo's Ansible host. The EC2 instance it replaced is `punjab-aws-retired`. +- **Hives** — hosting, failover lease and runbook: [`docs/src/servers/hives.md`](docs/src/servers/hives.md) (source in `hives/`); the hives themselves: [`docs/src/cluster/hive.md`](docs/src/cluster/hive.md). Exactly one host may run them; never start them by hand on the host the lease does not name. +- **KubeStellar / k3s** — [`docs/src/servers/home-k3s.md`](docs/src/servers/home-k3s.md). Fresh k3s on these hosts needs the firewalld rules there (cni0 → internal, pod→kubelet 10250, `k3s-pod-egress` policy). -Neither cluster's nodes are Ansible-managed — Talos has no SSH. They are not in -`inventory.yml`; access is `talosctl`/`kubectl` only. - -The two Hetzner VPSes it replaced (`matrix`, `telengana`) are retired but still -powered on for burn-in. **Nothing on them may be restarted** — see the -[cutover runbook](docs/matrix-cutover-runbook.md). +**Retired** (history only; see the [archive](docs/src/archive/talos-k8s.md)): the home Talos cluster (same hardware, replaced by k3s), the AWS Talos cluster (`eu-north-1`, terminated 2026-10-25; configs `~/.kube/config-aws-migration` / `~/.talos/config-aws-migration` kept as recovery material; handbook [`docs/src/servers/aws-k8s/cluster.md`](docs/src/servers/aws-k8s/cluster.md)), and the Hetzner VPSes `matrix` / `telengana` (**nothing on them may be restarted** — see the [cutover runbook](docs/matrix-cutover-runbook.md)). ## Core commands @@ -58,30 +50,26 @@ Roles are conditional on inventory group (`desktop`, `server`, `vps`, `test_flee ## Kubeconfig + Talos secrets -`~/.kube/config` and `~/.talos/config` contain cluster PKI — they live in **Bitwarden as secure notes** named `kubeconfig` and `talosconfig` (home cluster). The AWS cluster's are `kubeconfig-aws-migration` / `talosconfig-aws-migration`, written to `~/.kube/config-aws-migration` / `~/.talos/config-aws-migration`. Pull all four via `just apply-tags kube`; `just seed-kube` only seeds the two home-cluster notes. +`~/.kube/config` and `~/.talos/config` contain cluster PKI — they live in **Bitwarden as secure notes** named `kubeconfig` and `talosconfig`. **Those notes hold the retired home Talos cluster's configs**; the k3s clusters' kubeconfigs are `/etc/rancher/k3s/k3s.yaml` on each host (punjab's `~/.kube/config` targets karnataka; bihar's has contexts `bihar`, `its1`, `wds1`) and have not been re-seeded — don't run `just apply-tags kube` on a host whose `~/.kube/config` you need to keep without checking first. The AWS cluster's are `kubeconfig-aws-migration` / `talosconfig-aws-migration`, written to `~/.kube/config-aws-migration` / `~/.talos/config-aws-migration`. Pull all four via `just apply-tags kube`; `just seed-kube` only seeds the two home-cluster notes. `talos-k8s/.gitignore` excludes `controlplane.yaml`, `worker.yaml`, and `talosconfig` from git. -## Talos cluster - -Two nodes: **bihar** (control plane, Intel) and **karnataka** (worker, AMD Strix Halo APU). Talos `v1.13.2`, K8s `v1.36.1`, flannel CNI. AMD GPU exposed to Kubernetes via the Image Factory schematic. - -Production workloads: Lemonade (AMD-optimized local AI), KubeVirt v1.8.2 + KubeVirt Manager, Tailscale Operator (Ingress to `*.manatee-basking.ts.net`). +## Home k3s (replaces the Talos cluster) -**The cluster handbook is [`docs/src/servers/talos-k8s/cluster.md`](docs/src/servers/talos-k8s/cluster.md)** — hardware, network, reinstall, troubleshooting. +No Talos anywhere now. Each of bihar/karnataka has its own k3s v1.36.5 bound to its tailnet IP (flannel on `tailscale0`, traefik/servicelb disabled); kits in `home-k3s//` (`install-host.sh`, `config.yaml`, `k3s.service`). bihar uses service CIDR `10.96.0.0/12` (KubeFlex requirement). karnataka's AMD GPU is exposed through `amdgpu-device-plugin.yaml`; Lemonade is scaled to 0. The [k3s handbook](docs/src/servers/home-k3s.md) covers design, firewalld and KubeStellar. ## TunaOS AWS account (`aws/`) -- **punjab** (this repo's AWS-hosted Ansible host) is an EC2 `t3.large` in `us-east-1`; the former Talos cluster in `eu-north-1` is terminated. `runs-on` (us-east-2) is its own CloudFormation stack — don't import it. +- **punjab** is no longer in AWS (it is a VM on karnataka); `aws/punjab.tf` describes the retired EC2 instance. The former Talos cluster in `eu-north-1` is terminated. `runs-on` (us-east-2) is its own CloudFormation stack — don't import it. - **Do not blindly apply `aws/`:** its configuration still describes the terminated cluster and may propose recreating paid infrastructure. Inspect live resources and review plans before changes; credits are exhausted. - State is in S3 (`hanthor-fleet-backups-*/tofu/aws/`) and contains Talos PKI. `aws/terraform.tfvars` is gitignored and lives in Bitwarden note `aws-tofu-tfvars`. -- punjab uses Tailscale for access and SSM Session Manager for break-glass. Preserve its data and the S3 backups until a verified replacement and an explicit cleanup decision exist. +- Preserve the retired punjab EC2 instance's data and the S3 backups until an explicit cleanup decision exists. - The former cluster uploaded Postgres dumps to `s3://hanthor-fleet-backups-*/postgres/`; the `postgres-backup-writer` key is in Bitwarden, never in tofu state. - `prevent_destroy` in configuration does not prevent creation of resources missing from the account. Never put Talos `user_data` in the config. ## Automation (bots + Hive) -Renovate and the Hive open PRs; the live Hives are school (`hanthor-hive-agent`) and reef (`tuna-reef-hive`) on the Spindle VPS and cover only `tuna-os/*` repos, not this repo (see [`docs/src/cluster/hive.md`](docs/src/cluster/hive.md)). master is protected (CI + code-owner review, admin bypass). `.github/CODEOWNERS` is deny-by-default with carve-outs for paths that never run on the fleet (docs, tests, talos-k8s, pi skills, dep manifests) — only those auto-merge. See [`docs/src/automation.md`](docs/src/automation.md). Don't widen the carve-outs to anything the fleet timer executes. +Renovate and the Hive open PRs; the live Hives are school (`hanthor-hive-agent`) and reef (`tuna-reef-hive`) on bihar (karnataka standby) and cover only `tuna-os/*` repos, not this repo (see [`docs/src/cluster/hive.md`](docs/src/cluster/hive.md)). master is protected (CI + code-owner review, admin bypass). `.github/CODEOWNERS` is deny-by-default with carve-outs for paths that never run on the fleet (docs, tests, talos-k8s, pi skills, dep manifests) — only those auto-merge. See [`docs/src/automation.md`](docs/src/automation.md). Don't widen the carve-outs to anything the fleet timer executes. ## Don'ts @@ -94,7 +82,10 @@ Renovate and the Hive open PRs; the live Hives are school (`hanthor-hive-agent`) ## References -- [`docs/src/servers/talos-k8s/cluster.md`](docs/src/servers/talos-k8s/cluster.md) — Talos cluster handbook +- [`docs/src/fleet.md`](docs/src/fleet.md) — every host: live, intermittent, retired +- [`docs/src/servers/hives.md`](docs/src/servers/hives.md) — hive hosting + failover runbook +- [`docs/src/servers/home-k3s.md`](docs/src/servers/home-k3s.md) — k3s + KubeStellar +- [`docs/src/archive/`](docs/src/archive/talos-k8s.md) — retired Talos cluster handbooks - [`docs/src/roles/`](docs/src/roles/) — every Ansible role explained - [`docs/src/bitwarden.md`](docs/src/bitwarden.md) — BW vault structure - `Justfile` — every task recipe; `just --list` for a menu diff --git a/CONTEXT.md b/CONTEXT.md index 361ae75a..8a01ef41 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -2,7 +2,7 @@ > Created 2026-06-14 during architecture review. > Add terms as they are sharpened during grilling sessions or refactors. -> This is a glossary, not a spec. Implementation details live in `docs/src/roles/` and the cluster handbooks (`docs/src/servers/talos-k8s/cluster.md`, `docs/src/servers/aws-k8s/cluster.md`). +> This is a glossary, not a spec. Implementation details live in `docs/src/roles/` and the host pages (`docs/src/fleet.md`, `docs/src/servers/{bihar,karnataka,home-k3s,hives}.md`); the Talos handbooks are archived (`docs/src/archive/talos-home-cluster/cluster.md`, `docs/src/servers/aws-k8s/cluster.md`). ## Fleet @@ -26,9 +26,10 @@ ## Cluster -- **Node** — a Talos Linux machine in a K8s cluster. Home: `bihar` = control plane, `karnataka` = worker (powered down, expected back). AWS (`eu-north-1`): one control-plane + one worker. Neither is in `inventory.yml`. -- **Workload** — a Kubernetes deployment on a cluster (home: Lemonade, KubeVirt, Tailscale Operator; AWS: Matrix/ESS, the Hive, CFP dashboard). -- **Manifest** — a YAML file in `talos-k8s/` that defines a workload. +- **Node** — a k3s node. Home: `bihar` and `karnataka` are each a **separate single-node k3s** on Bluefin, linked by KubeStellar (hub on bihar: ITS `its1`, WDS `wds1`). The former Talos clusters (home, and AWS `eu-north-1`) are retired. +- **Workload** — a Kubernetes deployment on a cluster (karnataka: KubeVirt VMs incl. punjab, ARC runners, Frappe LMS, Corral, front-door; bihar: KubeStellar control plane). The hives are **not** a k8s workload: they run as podman compose on bihar (karnataka standby). +- **Lease** — the one-line file on spindle (`/var/lib/hive-failover/active`) naming the single host allowed to run the hives. +- **Manifest** — a YAML file that defines a workload: `home-k3s//` (current) or `talos-k8s/` (archived Talos-era). ## Architecture (from improve-codebase-architecture) diff --git a/README.md b/README.md index dcf11e4e..6fed6fae 100644 --- a/README.md +++ b/README.md @@ -59,18 +59,22 @@ From `inventory.yml`: | dilli | desktop | Secondary workstation (Bluefin) | | kerala | desktop | postmarketOS ARM device (musl, `apk`) | | mumbai | desktop | Debian VM on the phone (Android Virtualization Framework), `cli-only` | -| punjab | server | EC2 agent box (us-east-1) | +| punjab | server | Agent box, KubeVirt VM on karnataka | +| bihar | desktop | Home server (Bluefin): active hive host, KubeStellar hub | | goa | server | Raspberry Pi 5 (applies locally) | | vm | server | Local dev VM | | termux | termux_hosts | Raw Termux layer on the phone — `termux_packages` only | | test-fleet-fedora, test-fleet-node2 | test_fleet | KubeVirt test VMs (`machine_profile: vm-test`) | | matrix, telengana | vps | **Retired** Hetzner VPSes, powered on for burn-in — restart nothing on them | -The home Talos cluster, **bihar** + **karnataka**, is not Ansible-managed: it is -powered down and in storage, expected back. The AWS Talos cluster was terminated -on 2026-10-25. [Spindle](docs/src/servers/spindle-vps/README.md), an OCI VPS in -Frankfurt running Matrix, standalone LiveKit and the School, Reef and Hub Hives, -is not listed in this inventory. +**bihar** and **karnataka** are home servers on Bluefin with separate k3s +clusters linked by KubeStellar (the Talos cluster they ran before is archived). +bihar runs the School, Reef and Hub hives, with karnataka as a cold standby; +karnataka hosts KubeVirt (including punjab), CI runners and the LMS. +[Spindle](docs/src/servers/spindle-vps/README.md), an OCI VPS in Frankfurt, runs +Matrix and LiveKit, is the public front door (Caddy) for the hives and coordinates +their failover; it is not in this inventory. The AWS Talos cluster was terminated +on 2026-10-25. Current state of every host: [Fleet status](docs/src/fleet.md). The `llm` group is dormant (commented out in `inventory.yml`). --- @@ -85,8 +89,11 @@ Repo is public — no secrets in git, ever. All secrets are fetched from Bitward Full handbook lives in [`docs/`](docs/) and is published at [reilly.asia/infra/handbook](https://reilly.asia/infra/handbook/) — run `just docs` to serve it locally. -- [Talos cluster handbook](docs/src/servers/talos-k8s/cluster.md) — Bihar + Karnataka (home, offline), Lemonade, KubeVirt -- [Spindle VPS](docs/src/servers/spindle-vps/README.md) — Matrix, LiveKit and Hives on OCI +- [Fleet status](docs/src/fleet.md) — every host, live / intermittent / retired +- [Bihar](docs/src/servers/bihar.md), [Karnataka](docs/src/servers/karnataka.md), [Home k3s + KubeStellar](docs/src/servers/home-k3s.md) +- [Hives hosting and failover](docs/src/servers/hives.md) — bihar primary, karnataka standby, spindle front door +- [Spindle VPS](docs/src/servers/spindle-vps/README.md) — Matrix, LiveKit, Caddy front door on OCI +- [Archive](docs/src/archive/talos-k8s.md) — the retired home Talos cluster - [TunaOS AWS cluster handbook](docs/src/servers/aws-k8s/cluster.md) — archived cluster history - [Adding a new machine](docs/src/onboarding.md) - [Roles reference](docs/src/roles/) diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index 38589b85..62e6edc4 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md @@ -15,6 +15,7 @@ # Fleet Reference +- [Fleet status](fleet.md) - [Network](network.md) - [Gateway](network/gateway/README.md) - [KVM](network/kvm/README.md) @@ -24,25 +25,21 @@ - [Kanpur](desktop/kanpur/README.md) - [Kerala](desktop/kerala/README.md) - [Servers]() - - [Home Kubernetes Images & ISOs](servers/home-k8s-images.md) - - [Talos K8s Cluster (offline)](servers/talos-k8s/cluster.md) - - [Bihar](servers/talos-k8s/bihar/README.md) - - [Karnataka](servers/talos-k8s/karnataka/README.md) - - [Goa](servers/goa/README.md) + - [Bihar](servers/bihar.md) + - [Karnataka](servers/karnataka.md) + - [Home k3s & KubeStellar](servers/home-k3s.md) + - [Hives hosting & failover](servers/hives.md) - [Spindle VPS (matrix.reilly.asia)](servers/spindle-vps/README.md) - [Upgrade to main (draft)](servers/spindle-vps/upgrade-to-main.md) - [2026-11 media and sync incident](servers/spindle-vps/incident-2026-11-17-media.md) -- [VPS]() - - [Matrix (retired)](vps/matrix/README.md) - - [Telengana (retired)](vps/telengana/README.md) + - [Punjab (agent box)](servers/punjab/README.md) + - [Goa](servers/goa/README.md) + - [Home Kubernetes Images (not deployed)](servers/home-k8s-images.md) # TunaOS Infrastructure - [TunaOS AWS Account & IaC](servers/aws/README.md) - - [Talos Cluster (terminated; archive)](servers/aws-k8s/cluster.md) - - [Shutdown, 2026-10-25](servers/aws-k8s/shutdown-2026-10-25.md) - [Hive Agent Fleet](cluster/hive.md) - - [Punjab (agent box)](servers/punjab/README.md) # Roles Reference @@ -105,6 +102,13 @@ - [authentik](roles/authentik.md) - [n8n](roles/n8n.md) -# Cluster +# Archive -- [Talos Kubernetes](talos-k8s.md) +- [Home Talos cluster (replaced by k3s)](archive/talos-k8s.md) + - [Cluster handbook](archive/talos-home-cluster/cluster.md) + - [Bihar on Talos](archive/talos-home-cluster/bihar/README.md) + - [Karnataka on Talos](archive/talos-home-cluster/karnataka/README.md) +- [AWS Talos cluster (terminated 2026-10-25)](servers/aws-k8s/cluster.md) + - [Shutdown, 2026-10-25](servers/aws-k8s/shutdown-2026-10-25.md) +- [Matrix VPS (retired)](archive/vps/matrix/README.md) +- [Telengana VPS (retired)](archive/vps/telengana/README.md) diff --git a/docs/src/servers/talos-k8s/bihar/README.md b/docs/src/archive/talos-home-cluster/bihar/README.md similarity index 89% rename from docs/src/servers/talos-k8s/bihar/README.md rename to docs/src/archive/talos-home-cluster/bihar/README.md index 374c95b9..364c0c58 100644 --- a/docs/src/servers/talos-k8s/bihar/README.md +++ b/docs/src/archive/talos-home-cluster/bihar/README.md @@ -1,5 +1,7 @@ # Bihar +> **ARCHIVED (2026-11-18).** This page describes bihar on Talos, which was replaced by Bluefin + k3s. See [Fleet status](../../../fleet.md). + Kubernetes control plane. [Talos Linux](https://www.talos.dev/) node — not in `inventory.yml`, managed with `talosctl`/`kubectl` only. > **Powered down and in storage, expected back.** See the [cluster handbook](../cluster.md). diff --git a/docs/src/servers/talos-k8s/cluster.md b/docs/src/archive/talos-home-cluster/cluster.md similarity index 99% rename from docs/src/servers/talos-k8s/cluster.md rename to docs/src/archive/talos-home-cluster/cluster.md index 5ed88c70..6e222ca6 100644 --- a/docs/src/servers/talos-k8s/cluster.md +++ b/docs/src/archive/talos-home-cluster/cluster.md @@ -1,11 +1,13 @@ # Cluster Handbook — Talos Linux on Bihar + Karnataka +> **ARCHIVED (2026-11-18).** This page describes the home Talos cluster, which was replaced by Bluefin + k3s. See [Fleet status](../../fleet.md). + > **Status: powered down and in storage (as of 2026-08-27), expected back.** > Temporarily offline, *not* decommissioned — keep its configs, Bitwarden notes > (`kubeconfig` / `talosconfig`), and `talos-k8s/` manifests. `kubectl` and > `just doctor` will fail against it until it returns. Matrix and the School, -> Reef and Hub Hives now run on the [OCI Spindle VPS](../spindle-vps/README.md). -> The [AWS Talos cluster](../aws-k8s/cluster.md) was terminated on 2026-10-25. +> Reef and Hub Hives now run on the [OCI Spindle VPS](../../servers/spindle-vps/README.md). +> The [AWS Talos cluster](../../servers/aws-k8s/cluster.md) was terminated on 2026-10-25. > > **When it returns**, before applying anything under `talos-k8s/ai/`: > run [`talos-k8s/ai/migrate-pv-path.sh`](https://github.com/hanthor/dotfiles/blob/master/talos-k8s/ai/migrate-pv-path.sh) @@ -554,7 +556,7 @@ This is a replacement OS and **new K3s cluster** for the two home nodes, not a Talos upgrade. Bihar is the Intel server/control plane; Karnataka is the AMD worker. **No physical migration has been performed.** The home cluster is still powered down/in storage. Nothing here changes the terminated -[AWS Talos cluster's](../aws-k8s/cluster.md) recovery material; Corral +[AWS Talos cluster's](../../servers/aws-k8s/cluster.md) recovery material; Corral acceptance uses separate disposable KubeVirt VMs, not Talos node changes. Sections 1–11 remain the Talos recovery handbook, not instructions for the new OS. diff --git a/docs/src/servers/talos-k8s/karnataka/README.md b/docs/src/archive/talos-home-cluster/karnataka/README.md similarity index 90% rename from docs/src/servers/talos-k8s/karnataka/README.md rename to docs/src/archive/talos-home-cluster/karnataka/README.md index f98939dd..2cbcc796 100644 --- a/docs/src/servers/talos-k8s/karnataka/README.md +++ b/docs/src/archive/talos-home-cluster/karnataka/README.md @@ -1,5 +1,7 @@ # Karnataka +> **ARCHIVED (2026-11-18).** This page describes karnataka on Talos, which was replaced by Bluefin + k3s. See [Fleet status](../../../fleet.md). + Kubernetes worker node with AMD GPU. [Talos Linux](https://www.talos.dev/) node — not in `inventory.yml`, managed with `talosctl`/`kubectl` only. > **Powered down and in storage, expected back.** See the [cluster handbook](../cluster.md). diff --git a/docs/src/talos-k8s.md b/docs/src/archive/talos-k8s.md similarity index 81% rename from docs/src/talos-k8s.md rename to docs/src/archive/talos-k8s.md index 46a339e0..fd37244a 100644 --- a/docs/src/talos-k8s.md +++ b/docs/src/archive/talos-k8s.md @@ -1,8 +1,10 @@ # Talos Kubernetes Cluster +> **ARCHIVED (2026-11-18).** This page describes the home Talos cluster, which was replaced by Bluefin + k3s. See [Fleet status](../fleet.md). + Two-node Kubernetes cluster running on Talos Linux, managed from the `talos-k8s/` directory. -> **Powered down and in storage, expected back** — commands below fail until it returns. Matrix and School, Reef and Hub run on the [OCI Spindle VPS](servers/spindle-vps/README.md); the [AWS Talos cluster](servers/aws-k8s/cluster.md) was terminated on 2026-10-25. +> **Powered down and in storage, expected back** — commands below fail until it returns. Matrix and School, Reef and Hub run on the [OCI Spindle VPS](../servers/spindle-vps/README.md); the [AWS Talos cluster](../servers/aws-k8s/cluster.md) was terminated on 2026-10-25. ## Hardware @@ -49,7 +51,7 @@ Manifest: `talos-k8s/ai/lemonade.yaml` ### Additional services -See the [cluster handbook](servers/talos-k8s/cluster.md) for the full list — Forgejo, Hive, Authentik, n8n, AppFlowy Cloud, Grafana, Corral, and more. +See the [cluster handbook](talos-home-cluster/cluster.md) for the full list — Forgejo, Hive, Authentik, n8n, AppFlowy Cloud, Grafana, Corral, and more. ## Cluster Access @@ -68,4 +70,4 @@ Karnataka uses a custom Talos image from Image Factory that includes the `sidero ## Handbook -Full cluster documentation at [`docs/src/servers/talos-k8s/cluster.md`](servers/talos-k8s/cluster.md) (hardware details, reinstall procedure, troubleshooting). +Full cluster documentation at [`docs/src/servers/talos-k8s/cluster.md`](talos-home-cluster/cluster.md) (hardware details, reinstall procedure, troubleshooting). diff --git a/docs/src/vps/matrix/README.md b/docs/src/archive/vps/matrix/README.md similarity index 84% rename from docs/src/vps/matrix/README.md rename to docs/src/archive/vps/matrix/README.md index 622afe36..0fe15ac0 100644 --- a/docs/src/vps/matrix/README.md +++ b/docs/src/archive/vps/matrix/README.md @@ -1,8 +1,8 @@ # Matrix > **Retired 2026-08-27.** The Matrix homeserver moved first to the -> [AWS Talos cluster](../../servers/aws-k8s/cluster.md), terminated 2026-10-25, -> and now runs on the [OCI VPS](../../servers/spindle-vps/README.md). Restart +> [AWS Talos cluster](../../../servers/aws-k8s/cluster.md), terminated 2026-10-25, +> and now runs on the [OCI VPS](../../../servers/spindle-vps/README.md). Restart > nothing on this retired host; the page below is historical. See the > [Matrix cutover runbook](https://github.com/hanthor/dotfiles/blob/master/docs/matrix-cutover-runbook.md). @@ -27,6 +27,6 @@ upgrades on the box were disabled so nothing resurrects them. - A single VPS with no swap and unmanaged journald growth was fragile for K8s; the former AWS cluster needed more headroom (see the OOM incident in its - [archived handbook](../../servers/aws-k8s/cluster.md)). + [archived handbook](../../../servers/aws-k8s/cluster.md)). - `reilly.asia` DNS is on [Cloudflare](https://www.cloudflare.com/), which made the cutover a DNS change rather than a server move. diff --git a/docs/src/vps/telengana/README.md b/docs/src/archive/vps/telengana/README.md similarity index 74% rename from docs/src/vps/telengana/README.md rename to docs/src/archive/vps/telengana/README.md index 3e4dea0d..d0dba505 100644 --- a/docs/src/vps/telengana/README.md +++ b/docs/src/archive/vps/telengana/README.md @@ -1,8 +1,8 @@ # Telengana > **Retired 2026-08-27.** The tuna-os Hive moved from this host to the former -> [AWS Talos cluster](../../servers/aws-k8s/cluster.md), terminated 2026-10-25. -> School, Reef and Hub now run on the [OCI VPS](../../servers/spindle-vps/README.md). +> [AWS Talos cluster](../../../servers/aws-k8s/cluster.md), terminated 2026-10-25. +> School, Reef and Hub now run on the [OCI VPS](../../../servers/spindle-vps/README.md). > Restart nothing on this retired host; the page below is historical. Former VPS node in the fleet (`vps` group in `inventory.yml`). @@ -19,4 +19,4 @@ The box drifted from the fleet's hardening baseline (host firewall and fail2ban were never enabled, and Tailscale had been logged out, leaving admin access on the public interface). The former AWS cluster kept its admin APIs behind an allowlisted security group, and fleet servers use the -[`server_hardening`](../../roles/server_hardening.md) role. +[`server_hardening`](../../../roles/server_hardening.md) role. diff --git a/docs/src/automation.md b/docs/src/automation.md index 76686d00..d6e6f621 100644 --- a/docs/src/automation.md +++ b/docs/src/automation.md @@ -11,7 +11,7 @@ you, and anything that executes on the fleet still waits for you. |---|---|---| | **Renovate** | minor/patch/digest bumps (pi npm deps, Go, GitHub Actions, OpenTofu providers), weekly lockfile maintenance | yes, once CI is green | | **Renovate** (cluster) | image/chart bumps in `talos-k8s/`, grouped weekly, label `deploy-needed` | **no** — manifests are applied by hand, so a merge would only make git lie about the cluster | -| **Hive** (school and reef spokes on the [Spindle VPS](cluster/hive.md#current-deployment-spindle-vps); Apps `hanthor-hive-agent` and `tuna-reef-hive`) | its agents open PRs labelled `hold`; the Hive itself only merges PRs labelled `lgtm` | only if the PR stays inside the carve-outs below. **Neither hive's roster includes this repo today** (both cover only `tuna-os/*`), so no Hive PRs are expected here unless a hanthor spoke is rebuilt | +| **Hive** (school and reef spokes on [bihar, with karnataka as standby](cluster/hive.md#current-deployment-bihar-karnataka-standby); Apps `hanthor-hive-agent` and `tuna-reef-hive`) | its agents open PRs labelled `hold`; the Hive itself only merges PRs labelled `lgtm` | only if the PR stays inside the carve-outs below. **Neither hive's roster includes this repo today** (both cover only `tuna-os/*`), so no Hive PRs are expected here unless a hanthor spoke is rebuilt | | **You** | direct pushes, `just onboard`, etc. | admin bypass — unaffected | ## The gate @@ -56,7 +56,7 @@ was the seed of the former `hive.reilly.asia` spoke on the terminated AWS cluster (exported 2026-10-01); it does not describe a running Hive. For the current school and reef files, read-only inspection and the safe edit procedure, see the -[Hive operations guide](cluster/hive.md#current-deployment-spindle-vps): +[Hive operations guide](cluster/hive.md#current-deployment-bihar-karnataka-standby): ```bash ssh ubuntu@100.113.75.88 'docker ps --filter name=hives-; ls /srv/hives/*/data/hive.yaml.runtime' diff --git a/docs/src/cluster/hive.md b/docs/src/cluster/hive.md index 6642d2e8..40398cdc 100644 --- a/docs/src/cluster/hive.md +++ b/docs/src/cluster/hive.md @@ -1,15 +1,24 @@ # Hive — AI Agent Supervisor -## Current deployment: Spindle VPS +## Current deployment: bihar (karnataka standby) -Two Hive spokes, **school** and **reef**, and their **hub** run as Docker -Compose services on the [Spindle VPS](../servers/spindle-vps/README.md) under -`/srv/hives` (`ubuntu@100.113.75.88` over Tailscale). Caddy routes +Two Hive spokes, **school** and **reef**, and their **hub** run as compose +services under rootful podman on [bihar](../servers/bihar.md), in +`/var/srv/hives` (`ssh james@bihar`, commands need `sudo`). Until 2026-11-18 +they ran on the [Spindle VPS](../servers/spindle-vps/README.md) under +`/srv/hives`. Spindle's Caddy still routes [school.tunaos.org](https://school.tunaos.org), [reef.tunaos.org](https://reef.tunaos.org) and -[hub.tunaos.org](https://hub.tunaos.org) to them. Everything below was read from -the host on 2026-11-13 unless marked *unverified*. Container state alone does -not prove agents are healthy; see [Checking health](#checking-health). +[hub.tunaos.org](https://hub.tunaos.org), over the tailnet to the active host. +[karnataka](../servers/karnataka.md) is a cold standby. A lease on spindle +selects the active host. Run every command below on **the host the lease +names** (`ssh spindle sudo hive-failover status`). Hosting, failover and the +host-specific compose layer are described in +[Hives hosting and failover](../servers/hives.md). + +The layout and roster below were read on spindle on 2026-11-13. The move +changed the paths and the commands, but not the hive configuration. Container +state alone does not prove that agents are healthy. See [Checking health](#checking-health). This section is the current operations guide. The [archived cluster guide](#archived-cluster-guide) after it describes the retired @@ -18,7 +27,7 @@ Kubernetes deployment and must not be applied here. ### Layout ```text -/srv/hives/ +/var/srv/hives/ # on the active host; the host layer is /var/srv/hives-host/compose..yaml ├── compose.yaml # services: hub, hub-front, school, reef (network spindle_default) ├── compose.override.yaml # pins the running binary/dashboard, memory limits, session pool ├── .env # secrets and NTFY topic (mode 600; never print or commit) @@ -101,25 +110,27 @@ auto-merge Renovate major bumps, which the org forbids. ### Safe edit procedure -More than one operator edits `/srv/hives` (including agents). Re-read the files +More than one operator edits `/var/srv/hives` (including agents). Re-read the files immediately before editing and never overwrite a file you read earlier. -Do **not** `docker compose up` a spoke that is running to apply config: it +Do **not** `hives up` a spoke that is running to apply config: it triggered a container-rename bug. Remove the container first: ```bash -cd /srv/hives -docker compose rm -f -s reef # stop and remove the spoke container only +sudo -i # the files are root-owned on bihar/karnataka +cd /var/srv/hives +H=/var/srv/hives-host/hives # docker-compose wrapper: podman + host layer +$H rm -f -s reef # stop and remove the spoke container only python3 hive-roster.py add NAME reef/config/hive.yaml reef/data/hive.yaml.runtime -docker compose up -d --no-deps reef # recreate only this spoke -docker ps --filter name=hives- # verify: Up, and the right name +$H up -d --no-deps reef # recreate only this spoke +podman ps --filter name=hives- # verify: Up, and the right name ``` The edit step must (a) write a timestamped copy of each file into -`/srv/hives/backups/` first, (b) parse the YAML before and after, and (c) abort, +`/var/srv/hives/backups/` first, (b) parse the YAML before and after, and (c) abort, writing nothing, if anything other than the intended change would differ. This -helper does that for roster edits (save it as `/srv/hives/hive-roster.py` and -run it from `/srv/hives`; it needs PyYAML): +helper does that for roster edits (save it as `/var/srv/hives/hive-roster.py` and +run it from `/var/srv/hives`; it needs PyYAML): ```python #!/usr/bin/env python3 @@ -127,7 +138,7 @@ run it from `/srv/hives`; it needs PyYAML): import datetime, shutil, sys, yaml ORG = "tuna-os" -BACKUPS = "/srv/hives/backups" +BACKUPS = "/var/srv/hives/backups" PATHS = (("project", "repos"), ("dashboard", "matrix", "repos")) def seq_node(root, path): @@ -209,14 +220,14 @@ the maintainer; their exact semantics were not exercised for this page.) curl -s -o /dev/null -w '%{http_code}\n' https://school.tunaos.org/api/health # 200 curl -s -o /dev/null -w '%{http_code}\n' https://reef.tunaos.org/api/health # 200 curl -s https://hub.tunaos.org/api/fleet-stats # hub: /api/health is 404 -ssh ubuntu@100.113.75.88 'docker ps --filter name=hives-' +ssh james@bihar 'sudo podman ps --filter name=hives-' # or karnataka after a failover ``` For authenticated status from the host, call the API inside the container with the container's own token, so it is never printed: ```bash -ssh ubuntu@100.113.75.88 'docker exec hives-school-1 sh -c \ +ssh james@bihar 'sudo podman exec hives-school-1 sh -c \ "curl -sS -H \"X-Hive-Internal: \$HIVE_DASHBOARD_TOKEN\" http://127.0.0.1:3002/api/status"' \ | jq '.agents[] | {name, paused, state}' ``` @@ -227,11 +238,12 @@ PRs and issues (`gh search prs --author "hanthor-hive-agent[bot]"` and ### Known limits -- **Memory.** Spokes are capped at 2560 MB each (raised from 2 GB on 2026-11-13 - after hive was OOM-killed during rustc/govulncheck builds). The hub allows 2 - concurrent shared-pool agent sessions across both spokes (about 400 MB per - Claude session; the host had about 5 GB free and no swap). Do not raise either - without revisiting the host's headroom. +- **Memory.** On bihar and karnataka, each spoke can use 8 GB and the hub allows 4 + concurrent shared-pool sessions (`compose..yaml`). On spindle the limits were + 2560 MB per spoke and 2 sessions, and spokes were OOM-killed during + rustc/govulncheck builds. +- **Pinned images.** The hive and hive-hub images are pinned by digest with the + amd64 fork binary `hive-2ef7dc7c0-amd64`. Do not change to `:v6-latest`. - **Hive's own CI** for `tuna-os/hive` is blocked by GitHub ("Actions has been disabled"; a support ticket is pending), so fork builds are not CI-verified. - **Reply spam**: upstream @@ -240,9 +252,10 @@ PRs and issues (`gh search prs --author "hanthor-hive-agent[bot]"` and recognise its own earlier reply when posting as an App login. - **Concurrent edits and drift**: see the layout notes above; verify after every restart that the parsed seed and runtime agree on the roster. -- Not covered here: the hub's own config, the `omp-auth-broker` user service - the spokes call (`OMP_AUTH_BROKER_URL`), and Caddy routing, which were not - re-inspected for this page. The `hive.reilly.asia` hanthor spoke of the old AWS +- The spokes call the `omp-auth-broker` on spindle (`OMP_AUTH_BROKER_URL`, + `http://100.113.75.88:8765`). Caddy routing and failover are described in + [Hives hosting and failover](../servers/hives.md). This page does not describe + the hub's own configuration. The `hive.reilly.asia` hanthor spoke of the old AWS cluster has no container on this host; its current status is unverified. The [hive_ops role](../roles/hive_ops.md) targets the retired cluster, not this deployment. diff --git a/docs/src/desktop/himachal/README.md b/docs/src/desktop/himachal/README.md index a67782a7..2849e311 100644 --- a/docs/src/desktop/himachal/README.md +++ b/docs/src/desktop/himachal/README.md @@ -25,7 +25,7 @@ Laptop workstation in the hanthor fleet. Kubernetes, which was terminated on 2026-10-25; the host-vars comment saying those jobs now run in-cluster is stale. See [hive_ops](../../roles/hive_ops.md) and the - [current Hive deployment](../../cluster/hive.md#current-deployment-spindle-vps). + [current Hive deployment](../../cluster/hive.md#current-deployment-bihar-karnataka-standby). The fleet's GNOME role includes [PaperWM](https://github.com/paperwm/PaperWM), PaperShell and Copyous; their enabled state on this host was not checked. diff --git a/docs/src/fleet.md b/docs/src/fleet.md new file mode 100644 index 00000000..d550eaa6 --- /dev/null +++ b/docs/src/fleet.md @@ -0,0 +1,67 @@ +# Fleet status + +Snapshot of **2026-11-18**, made from `tailscale status` and the live hosts. +Update this page when a host changes role. To refresh it, run +`tailscale status --json | jq -r '.Peer[] | [.HostName,.Online,.LastSeen] | @tsv'`. + +## How it fits together + +``` +Internet ─▶ Cloudflare ─▶ spindle (Oracle, Frankfurt): Caddy, Matrix, LiveKit, OMP auth broker, hive failover coordinator + │ tailnet + ┌───────────────────┴────────────────────┐ + ▼ ▼ + bihar (home, WiFi) karnataka (home) + k3s + KubeStellar hub k3s + KubeVirt + ARC runners + LMS + hives: ACTIVE ───── hive-sync 15 min ───▶ hives: cold standby + └─ VM: punjab (agent box, Ansible host) +``` + +## Live hosts + +| Host | Kind | OS | Role | Docs | +|---|---|---|---|---| +| **bihar** | Home server (Intel, 62 GB) | Bluefin utah:testing | Active hive host; k3s + KubeStellar hub | [Bihar](servers/bihar.md) | +| **karnataka** | Home server (AMD Strix Halo, 125 GB) | Bluefin utah:testing | k3s: KubeVirt VMs, CI runners, LMS, Corral; hive standby | [Karnataka](servers/karnataka.md) | +| **spindle** | Oracle A1 VM, eu-frankfurt-1 (2 ARM cores, 12 GB) | Ubuntu 24.04 | Public front door (Caddy, `92.5.82.232`), Matrix (`reilly.asia`), LiveKit, OMP auth broker, hive failover coordinator | [Spindle VPS](servers/spindle-vps/README.md) | +| **punjab** | KubeVirt VM on karnataka (12 vCPU, 23 GB, 500 GB) | Ubuntu 24.04 | Agent box and fleet Ansible host. **Disk 96% full.** | [Punjab](servers/punjab/README.md) | +| **corral** | Tailscale sidecar pod on karnataka | n/a | Tailnet name for `corral-web` | [Karnataka](servers/karnataka.md) | +| **pixel-9a** (`termux`) | Phone | Android + Termux | Ansible `termux` host | [Inventory](inventory.md) | + +Cross-host services: +- The hives (school, reef, hub) are on bihar. If bihar fails, they move to karnataka. See [Hives hosting](servers/hives.md). +- KubeStellar is described in [Home k3s and KubeStellar](servers/home-k3s.md). + +## Intermittent (desktops and laptops) + +| Host | Last seen | Notes | Docs | +|---|---|---|---| +| kerala (`kerala-1`) | 2026-11-17 | Desktop. Earlier tailnet node `kerala` last seen 2026-08-29 | [Kerala](desktop/kerala/README.md) | +| himachal / himachal-1 | 2026-11-09 / 2026-11-10 | Desktop. Two tailnet nodes; remove the earlier one | [Himachal](desktop/himachal/README.md) | +| dilli | 2026-10-21 | Desktop | [Dilli](desktop/dilli/README.md) | +| kanpur | 2026-07-21 | Desktop or Windows test machine | [Kanpur](desktop/kanpur/README.md) | +| mumbai | 2026-07-20 | Debian VM on the phone | [Inventory](inventory.md) | +| goa | 2026-07-08 | Server | [Goa](servers/goa/README.md) | +| MacBook Air (2 nodes), CMF Phone 1 | 2026-10 | Personal devices, not managed | none | + +## Retired + +| Host | Retired | Notes | Docs | +|---|---|---|---| +| Home Talos cluster (bihar and karnataka on Talos) | 2026-10/11 | Replaced by Bluefin + k3s on the same hardware | [Archive](archive/talos-k8s.md) | +| AWS Talos cluster (eu-north-1) | 2026-10-25 | Terminated. Configurations kept as recovery material | [Archive](servers/aws-k8s/cluster.md) | +| punjab EC2 (`punjab-aws-retired`, 100.78.73.8) | 2026-10-28 | Replaced by the punjab VM on karnataka | [Punjab](servers/punjab/README.md) | +| matrix (`reilly-asia-matrix`), telengana | 2026-09 | Hetzner VPSes | [Archive](archive/vps/matrix/README.md) | +| Hives on spindle (`/srv/hives`) | 2026-11-18 | Moved to bihar. The copy is stopped and kept for rollback | [Hives hosting](servers/hives.md) | + +### Tailnet nodes to remove + +The tagged devices from the retired clusters are offline and can be deleted from +the Tailscale admin console. On 2026-11-18 these were: +- since 2026-07: `appflowy*`, `argocd`, `auth`, `forgejo`, `grafana`, `loki`, `prometheus`, `n8n`, `lemonade`, `home`, `kubevirt-manager*`, `default-deepseek-14b`, `repro-ctr-vm-vm`, `gate-*-vm` +- since 2026-10-25: `hive`, `hive-auth-egress-0/1`, `review`, `search`, `spindle-grafana`, `aws-migration-operator`, `tuna-v1-preflight-*` +- since 2026-09: `tailscale-operator*`, `secret-broker` +- since 2026-04: `uisp`, `chuppvm` +- since 2026-07 and 2026-08: `vm`, `WIN-G9NAVN79GN0` + +[hanthor/dotfiles#333](https://github.com/hanthor/dotfiles/pull/333) (`scripts/tailnet.py`) automates this cleanup. diff --git a/docs/src/index.md b/docs/src/index.md index de66b1cc..2c88ec20 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -3,8 +3,10 @@ Ansible-driven dotfiles and infrastructure for a personal fleet. Three distinct concerns live here: 1. **Workstation config** — shells, packages, browser, SSH, Tailscale, kubeconfig, GNOME, and more -2. **Home Talos K8s cluster** — manifests for the Bihar + Karnataka cluster -3. **Hosted infrastructure** — OCI spindle and AWS account configuration; the AWS Talos cluster is terminated +2. **Home servers** — bihar and karnataka (Bluefin + k3s, linked by KubeStellar); the hives run on bihar with karnataka as standby. The [Talos manifests](archive/talos-k8s.md) are archived +3. **Hosted infrastructure** — OCI spindle (Matrix, public front door, hive failover coordinator) and the AWS account; the AWS Talos cluster is terminated + +Current state of every host: **[Fleet status](fleet.md)**. > **This handbook is public.** It covers architecture, decisions and runbooks. > Never put secrets in it. Fleet addresses live in inventory or come from @@ -40,7 +42,8 @@ Ansible-managed hosts, from `inventory.yml`: | dilli | desktop | Desktop | Secondary workstation | | kerala | desktop | Mobile | postmarketOS ARM device | | mumbai | desktop | VM | Debian VM on the phone, `machine_profile: cli-only` | -| punjab | server | Server | EC2 agent box in us-east-1 | +| punjab | server | VM | Agent box, KubeVirt VM on karnataka | +| bihar | desktop | Server | Home server: active hive host, KubeStellar hub | | goa | server | Server | [Raspberry Pi 5](https://www.raspberrypi.com/products/raspberry-pi-5/) ARM server | | vm | server | VM | Local dev VM | | termux | termux_hosts | Phone | Raw Termux/Android layer | @@ -48,14 +51,11 @@ Ansible-managed hosts, from `inventory.yml`: | matrix | vps | VPS | **Retired** — see the cutover runbook | | telengana | vps | VPS | **Retired** — see the cutover runbook | -Not Ansible-managed (Talos — `talosctl`/`kubectl` only): - -- **bihar** (control plane) + **karnataka** (worker, AMD GPU) — the home cluster. - Powered down and in storage, expected back. See the [cluster handbook](servers/talos-k8s/cluster.md). -- The [AWS Talos cluster](servers/aws-k8s/cluster.md) was terminated on 2026-10-25; its handbook is historical. +Outside this inventory: -[Spindle](servers/spindle-vps/README.md) is an OCI Frankfurt VPS running Matrix, -standalone LiveKit, School, Reef and Hub. It is not currently listed in `inventory.yml`. +- **karnataka** — home k3s/KubeVirt server, configured from the native-units branch. See [Karnataka](servers/karnataka.md). +- [Spindle](servers/spindle-vps/README.md) — OCI Frankfurt VPS: Matrix, LiveKit, the Caddy front door for the hives, and the [hive failover](servers/hives.md) coordinator. +- The [AWS Talos cluster](servers/aws-k8s/cluster.md) was terminated on 2026-10-25, and the home Talos cluster was replaced by k3s ([archive](archive/talos-k8s.md)). The `llm` group is dormant (commented out in `inventory.yml`). diff --git a/docs/src/inventory.md b/docs/src/inventory.md index 26866a22..4adbc011 100644 --- a/docs/src/inventory.md +++ b/docs/src/inventory.md @@ -25,7 +25,7 @@ Headless hosts. ```yaml server: hosts: - punjab: # EC2 agent box (us-east-1; current instance status unverified) + punjab: # agent box, KubeVirt VM on karnataka (100.103.187.46) vm: # Local dev VM goa: # Raspberry Pi 5 (ARM) ``` @@ -71,14 +71,15 @@ termux_hosts: ``` ### `llm` (dormant) -Commented out in `inventory.yml` — its only member, karnataka, is now a Talos -worker. +Commented out in `inventory.yml` — karnataka, its former member, now runs k3s workloads. ### Not in the inventory -**bihar** and **karnataka** are Talos nodes (no SSH) and are managed with -`talosctl`/`kubectl` only. The home cluster is powered down and in storage, -expected back. The terminated AWS cluster's nodes were not in the inventory either. +**bihar** is in the `desktop` group (Bluefin, bootstrapped 2026-11-17). +**karnataka** is configured from the native-units branch +([#348](https://github.com/hanthor/dotfiles/pull/348)) and is not listed here. +Neither is a Talos node any more; see [Fleet status](fleet.md). **spindle** is +not in the inventory. The terminated AWS cluster's nodes were not in the inventory either. ## Per-Machine Configuration diff --git a/docs/src/network.md b/docs/src/network.md index a1e137c0..b357bcba 100644 --- a/docs/src/network.md +++ b/docs/src/network.md @@ -10,8 +10,7 @@ crosswalk. | Name | Role | |------|------| | gateway | Router (DNS, DHCP) — see [Gateway](network/gateway/README.md) | -| bihar | Talos K8s control plane (powered down, expected back) — reserved address | -| karnataka | Talos K8s worker, AMD GPU (powered down, expected back) — reserved address | +| karnataka | Home server: k3s, KubeVirt, AMD GPU — reserved address | | goa | Fleet control, Debian ARM (Raspberry Pi) — reserved address | | phone | Android phone (reserved address) | | printer | Network printer (DHCP) | @@ -28,7 +27,7 @@ tailnet address where one is needed. | Hostname | LAN | Group | |----------|-----|-------| | goa | reserved | server (applies locally) | -| punjab | — (AWS) | server | +| punjab | — (VM on karnataka) | server | | himachal | DHCP | desktop | | dilli | DHCP | desktop | | kanpur | DHCP | desktop | @@ -37,5 +36,5 @@ tailnet address where one is needed. | termux (ssh `:8022`) | phone | termux_hosts | | matrix | remote | vps — retired | | telengana | remote | vps — retired | -| bihar | reserved | Talos, not in inventory — offline | -| karnataka | reserved | Talos, not in inventory — offline | +| bihar | — (separate site, WiFi) | desktop — home server, hive host | +| karnataka | reserved | not in inventory (native units) — k3s host | diff --git a/docs/src/roles/hive_ops.md b/docs/src/roles/hive_ops.md index 8951697f..9364d40c 100644 --- a/docs/src/roles/hive_ops.md +++ b/docs/src/roles/hive_ops.md @@ -7,7 +7,7 @@ > **Retired AWS-cluster integration, not the current Hive operations guide.** > The AWS Talos cluster was terminated on 2026-10-25. The current school and > reef spokes and hub run as Docker Compose services on the -> [Spindle VPS](../cluster/hive.md#current-deployment-spindle-vps). Do not +> [bihar (karnataka standby)](../cluster/hive.md#current-deployment-bihar-karnataka-standby). Do not > enable this role or run its old Kubernetes helpers against that deployment. The role still exists in the repository. When enabled it installs host user diff --git a/docs/src/servers/bihar.md b/docs/src/servers/bihar.md new file mode 100644 index 00000000..0694058c --- /dev/null +++ b/docs/src/servers/bihar.md @@ -0,0 +1,57 @@ +# Bihar + +Home server, the primary host for the hives, and the KubeStellar hub. Since +2026-11-17 it runs Bluefin, not Talos. The [Talos page](../archive/talos-home-cluster/bihar/README.md) +is archived. + +| | | +|---|---| +| Hardware | ASRock B660M-STX, Intel, 24 threads, 62 GB of RAM | +| Disk | 1.9 TB NVMe (`/var`) | +| OS | Bluefin `ghcr.io/projectbluefin/utah:testing` (bootc: read-only `/usr`, writable `/etc` and `/var`) | +| Network | **WiFi** (`wlp3s0`). Not on the same LAN as karnataka. | +| Tailnet | `bihar` / `100.103.44.48` (`bihar.manatee-basking.ts.net`) | +| Access | `ssh james@bihar` (passwordless sudo through `/etc/sudoers.d/zz-james`) | +| Ansible | Yes: [`host_vars/bihar.yml`](https://github.com/hanthor/dotfiles/blob/master/host_vars/bihar.yml), `skip_pipewire_audio: true` | + +## Roles + +1. **Active hive host.** The hive containers (hub, hub-front, school and reef) run + under rootful podman compose, and spindle's Caddy sends public traffic to them. + See [Hives hosting and failover](hives.md). +2. **k3s and KubeStellar hub.** A single-node k3s runs KubeStellar 0.30.2, with + ITS `its1` and WDS `wds1`. bihar and karnataka are both workload clusters. + See [Home k3s and KubeStellar](home-k3s.md). + +## Host settings (applied 2026-11-17) + +- **No sleep:** the sleep, suspend and hibernate targets are masked, the + `90-nosleep.conf` drop-ins are in `/etc/systemd/{sleep,logind}.conf.d/`, and + the GDM dconf `sleep-inactive-ac-type='nothing'` is set. Before this change, + GNOME suspended the host after 15 min idle, and the host went offline. +- **Default target is `multi-user.target`:** no graphical login. Use + `systemctl isolate graphical.target` to start one. +- **Wake-on-LAN** is set to `magic` on `enp0s31f6`. The host uses WiFi, so + wake-on-LAN works only if you connect a cable. +- **To do:** in the firmware, set "Restore on AC/Power Loss" to **Power On**. + This setting is necessary because the hives depend on this host. +- Dotfiles bootstrap fixes for a new bootc host are in + [hanthor/dotfiles#385](https://github.com/hanthor/dotfiles/pull/385). +- Bitwarden phase 2 is not done yet. Run `just apply -e skip_pipewire_audio=true` from an + interactive terminal. + +## Timers + +| Unit | Purpose | +|---|---| +| `hive-reconcile.timer` | Follows the hive lease on spindle (every 1 min) | +| `hive-sync.timer` | Pushes the hive data to karnataka while bihar is active (every 15 min) | +| `fleet-osquery-report.timer` | Fleet facts | +| `tailscale-cert-renew.timer` | Tailscale TLS certificate | + +## Gotchas + +- Do not use `pkill -f ` over SSH if the pattern also occurs in the SSH + command line. The command stops your own session. +- `gsettings` over SSH needs `DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus`. +- `semanage` cannot write on bootc. Use `chcon` for SELinux labels. diff --git a/docs/src/servers/hives.md b/docs/src/servers/hives.md new file mode 100644 index 00000000..8237c642 --- /dev/null +++ b/docs/src/servers/hives.md @@ -0,0 +1,194 @@ +# Hives hosting and failover + +> **TunaOS infrastructure.** This page describes *where and how* the hives run. +> For the hives themselves (agents, configuration, health), see the +> [Hive operations guide](../cluster/hive.md). + +Last updated: 2026-11-18. The source for the scripts, compose layers and systemd +units is [`hives/`](https://github.com/hanthor/dotfiles/tree/master/hives) in this +repository. Each host also has a copy of this runbook at +`/var/srv/hives-host/RUNBOOK.md` (on spindle: `/var/lib/hive-failover/RUNBOOK.md`). + +## Overview + +The tuna-os hives run as one compose project named `hives`, with four containers: + +| Service | Purpose | Public URL | +|---|---|---| +| `hub` | Hub API that all spokes report to | `https://hub.tunaos.org/api/*`, `/login/*` | +| `hub-front` | Static hub page (nginx) | `https://hub.tunaos.org/` | +| `school` | Spoke for the tuna-os org (GitHub App 3942065) | `https://school.tunaos.org` | +| `reef` | Spoke (GitHub App 5546574) | `https://reef.tunaos.org` | + +Until 2026-11-18 the hives ran on spindle (Oracle ARM VM, Frankfurt, 2 cores and 11 GB of RAM). They now run on **bihar** (x86, 24 cores, 62 GB of RAM). **karnataka** is a cold standby. **spindle** keeps the public IP and Caddy, and it coordinates failover. + +``` +Internet ─▶ Cloudflare ─▶ spindle Caddy (92.5.82.232) + │ tailnet, ports 3100-3103 + ┌──────────────┴──────────────┐ + ▼ (active, holds the lease) ▼ (standby, stopped) + bihar 100.103.44.48 karnataka 100.66.158.127 + └── hive-sync every 15 min ──▶ +``` + +Outbound traffic from the hives (to GitHub and the model APIs) leaves through the active host's own internet connection. It does not go through spindle. + +The OMP auth broker still runs on spindle at `http://100.113.75.88:8765`, reached over the tailnet. + +## Host layout (bihar and karnataka are identical) + +| Path | Contents | +|---|---| +| `/var/srv/hives/` | Hive data and configuration, copied from spindle's `/srv/hives`: `compose.yaml`, `compose.override.yaml`, `.env`, `{hub,school,reef}/{config,data,secrets}`, `hub-front/`, `proxy/`, `bin/dashboard-*.html` | +| `/var/srv/hives-host/compose..yaml` | Host layer: ports, memory limits, broker URL, image digests, and the amd64 binary mount | +| `/var/srv/hives-host/bin/hive-2ef7dc7c0-amd64` | `tuna-os/hive` at `2ef7dc7c0` (branch `deploy/upstream-v6-20261113`) built with `CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath ./cmd/hive` | +| `/var/srv/hives-host/hives` | Wrapper that runs docker-compose against rootful podman with the three compose files | +| `/var/srv/hives-host/hive-reconcile` | Makes the host follow the lease (timer, every 1 min) | +| `/var/srv/hives-host/hive-sync` | Pushes the data to the peer host (timer, every 15 min) | +| `/var/srv/hives-host/keys/` | `lease` (read-only access to the lease on spindle) and `sync` (rsync to the peer host) | +| `/var/srv/hives-host/maintenance` | If this file exists, reconcile does not start the hives | + +Each host publishes its ports on its own tailnet IP only: + +| Port | Service | +|---|---| +| 3100 | hub-front | +| 3101 | hub | +| 3102 | school | +| 3103 | reef | + +A firewalld rich rule in the `public` zone accepts these ports only from spindle (`100.113.75.88`). Inside the compose network the services reach each other by name, for example `http://hub:3001`. The podman network is named `spindle_default` so that `compose.yaml` works unchanged. + +Images are pinned by digest, and the pin must match the binary: +- `ghcr.io/hivecommons/hive@sha256:d56425fb…` (school and reef) +- `ghcr.io/hivecommons/hive-hub@sha256:3c0162eb…` (hub) + +Do not change to `:v6-latest`. On 2026-11-18 a newer image made the hub log "failed to write token cache". **Upgrade the binary and the images together.** + +## Daily operations (on the active host) + +```bash +sudo /var/srv/hives-host/hives ps +sudo /var/srv/hives-host/hives logs -f --tail 50 school +sudo /var/srv/hives-host/hives restart reef +sudo podman stats --no-stream +``` + +To find the active host, run this on spindle: + +```bash +sudo hive-failover status +``` + +The command shows the lease, the health and container count of each host, and the targets in Caddy. + +### Change the hive configuration +1. Edit the files under `/var/srv/hives/` on the active host only. +2. Run `sudo /var/srv/hives-host/hives up -d`. + +The next `hive-sync` copies the change to the standby. + +### Change a host-specific setting +1. Edit `/var/srv/hives-host/compose..yaml` on **both** hosts. This covers ports, memory and image pins. +2. Run `sudo chcon -t container_file_t` on the file. + +### Upgrade the hive binary +1. Build the amd64 binary from the fork commit. +2. Copy it to `/var/srv/hives-host/bin/` on both hosts. +3. Change the binary mount and the image digests in both host files. +4. Run `hives up -d` on the active host. + +## Failover + +### The lease +The file `spindle:/var/lib/hive-failover/active` contains `bihar`, `karnataka` or `none`. Only the host named in the lease may run the hives. This rule prevents two hosts from running the same GitHub Apps at once. + +| Component | Where | Schedule | What it does | +|---|---|---|---| +| `hive-failover check` | spindle, `hive-failover.timer` | every 1 min | Checks the active host on ports 3101-3103. After 5 min of failures it sets the lease to `none`, stops the old host if it can, and waits. When the old host has 0 containers, or after 10 min, it starts the standby, sets the lease, changes Caddy and sends a ntfy notification on `$NTFY_TOPIC`. | +| `hive-reconcile` | both hosts, `hive-reconcile.timer` | every 1 min | If the lease names this host and fewer than 4 containers run, it starts them (unless `maintenance` exists). If the lease names a different host, it stops them. If it cannot read the lease for 8 min, it stops them (self-fence). | +| `hive-sync` | both hosts, `hive-sync.timer` | every 15 min | If the lease names this host, it uses rsync to push `/var/srv/hives` to the peer, without caches and toolchains. | + +Logs: + +```bash +journalctl -t hive-failover # on spindle +journalctl -u hive-reconcile -u hive-sync # on the hosts +``` + +The podman containers have restart policy `unless-stopped` and `podman-restart.service` is disabled. As a result, a host that boots does not start the hives itself. Only reconcile starts them, and only when the lease names that host. + +### Commands (on spindle) + +```bash +sudo hive-failover status +sudo hive-failover move bihar # planned move or failback: stop, sync, start, Caddy +sudo hive-failover move karnataka +sudo hive-failover hold on # pause automatic failover (for example during maintenance) +sudo hive-failover hold off +``` + +`move` sets `hold on` while it runs, and clears it when it succeeds. If `move` fails, the lease stays at `none` and hold stays on. Correct the problem and run `move` again. + +### After an automatic failover +The hives run on karnataka. Failback is manual, so that an unstable WiFi connection cannot move the hives back and forth. +1. Make sure bihar is up: `ssh james@bihar uptime`. +2. Run `sudo hive-failover move bihar` on spindle. It takes about 1 min. + +### Planned maintenance on bihar +Use one of these methods: +- Run `sudo hive-failover move karnataka` first. +- Or run `sudo hive-failover hold on` on spindle, then `sudo touch /var/srv/hives-host/maintenance` and `hives stop` on bihar. When you finish, remove the file and run `hold off`. + +### Limits +- The data on the standby can be up to 15 min old. Rsync copies files while they are in use, so a file that changes during the copy can arrive in an inconsistent state. +- The self-fence after 8 min without spindle was tested by hand only. A real network partition has not been tested. +- If spindle is down, there is no public access and no failover. The hosts stop the hives after 8 min, because they cannot read the lease. + +### Drill on 2026-11-18 +1. Hives stopped on bihar at 13:55:44Z. +2. Spindle revoked the lease at 14:01:29Z. +3. Karnataka was healthy and Caddy changed at 14:01:56Z. +4. `move bihar` completed in 50 s, including the sync back. + +## Gotchas +- **SELinux (bihar and karnataka are bootc and enforcing):** files mounted into containers must have the type `container_file_t`. `semanage` cannot write on bootc, so `hive-sync` runs `chcon -R` after each copy. Do not copy SELinux attributes with rsync (`-X`). +- **The Caddyfile on spindle is a single-file bind mount:** `sed -i` makes a new inode, and Caddy continues to see the old file. Write in place (`cat new > Caddyfile`), or run `docker cp Caddyfile spindle-caddy-1:/tmp/Caddyfile` then `docker exec spindle-caddy-1 caddy reload --adapter caddyfile --config /tmp/Caddyfile`. `hive-failover` does both. +- **The spindle-to-host path uses a relay:** it goes through Tailscale DERP (Frankfurt), with approximately 170 ms of latency and 1 MB/s. This is sufficient for dashboard and API traffic. +- **Disk:** the caches (Go build, rustup and others) are not synced. After a move, the first builds on the new host are slower. +- **Use separate SSH keys:** + - `spindle:/root/.ssh/hive_failover` gives access to `james@` on both hosts. + - The `lease` key of each host gives `ubuntu@spindle` with a forced command: `cat /var/lib/hive-failover/active`. + - The `sync` key of each host gives `james@` on the peer host. + +## Install or rebuild a host + +The files in [`hives/`](https://github.com/hanthor/dotfiles/tree/master/hives) go to these locations: + +| Repository file | Destination | +|---|---| +| `hives`, `hive-reconcile`, `hive-sync` | `/var/srv/hives-host/` on bihar and karnataka (mode 0755) | +| `compose..yaml` | `/var/srv/hives-host/` on that host | +| `systemd/hive-reconcile.*`, `systemd/hive-sync.*` | `/etc/systemd/system/` on bihar and karnataka | +| `hive-failover` | `/usr/local/sbin/` on spindle | +| `systemd/hive-failover.*` | `/etc/systemd/system/` on spindle | + +On a new host, also do these steps: +1. Install `docker-compose` with brew. +2. Run `systemctl enable --now podman.socket`. +3. Create the podman network `spindle_default`. +4. Pull the pinned images. +5. Copy `bin/hive-2ef7dc7c0-amd64` from the other host. If neither host has it, build it from `tuna-os/hive` at `2ef7dc7c0`. +6. Create the SSH keys in `/var/srv/hives-host/keys/`, and add the firewalld rule for ports 3100-3103 from spindle. +7. Label the files: `chcon -R -t container_file_t /var/srv/hives /var/srv/hives-host`. +8. Enable `hive-reconcile.timer` and `hive-sync.timer`. + +## Rollback to spindle (emergency only) +Spindle's old copy is stopped in `/srv/hives` and is not up to date (from 2026-11-18 13:31Z). It uses the arm64 binary `bin/hive-2ef7dc7c0`. +1. On spindle, run `sudo hive-failover hold on`. +2. Stop the hives on the active host. +3. Copy `/var/srv/hives` back to spindle without `.cache`. The arm64 binaries must stay. +4. Change the hive blocks in Caddy back to `school:3001`, `reef:3001`, `hub:3001` and `hub-front:80`. +5. Run `docker compose start` in `/srv/hives`. + +After the move is stable, delete `/srv/hives` on spindle to recover approximately 30 GB. diff --git a/docs/src/servers/home-k3s.md b/docs/src/servers/home-k3s.md new file mode 100644 index 00000000..cf9bf420 --- /dev/null +++ b/docs/src/servers/home-k3s.md @@ -0,0 +1,79 @@ +# Home k3s and KubeStellar + +bihar and karnataka each run a **separate single-node k3s** on Bluefin. They +replace the two-node Talos cluster ([archive](../archive/talos-k8s.md)). +**KubeStellar** on bihar connects the two clusters. The hosts are not on the same +LAN, so all cluster traffic uses the tailnet. + +The kits are in [`home-k3s/`](https://github.com/hanthor/dotfiles/tree/master/home-k3s): +`bihar/` and `karnataka/` each contain `install-host.sh`, `config.yaml`, +`k3s.service`, `wait-tailnet.sh` and `resolv.conf`. + +## Shared design + +- The k3s binary is in `/var/usrlocal/bin/k3s`. `/usr` is read-only on bootc. + `k3s.service` waits for the tailnet (`k3s-wait-tailnet`). +- The API server, kubelet and node IP bind to the **tailnet IP only**. + Flannel uses `tailscale0`. +- traefik and servicelb are disabled. Services are exposed with `tailscale serve` + (karnataka) or ingress-nginx (bihar, KubeStellar only). +- Pod DNS uses `resolv.conf` (1.1.1.1 and 1.0.0.1). +- SELinux on the host stays enforcing. The k3s SELinux module cannot be installed + on bootc. + +### firewalld (required: without these rules metrics-server fails and pods cannot resolve DNS) + +```bash +# cni0 goes in the internal zone; pods can reach the kubelet +firewall-cmd --permanent --zone=internal --change-interface=cni0 +firewall-cmd --permanent --zone=internal --add-rich-rule='rule family=ipv4 source address=10.42.0.0/16 destination address=/32 port port=10250 protocol=tcp accept' +# pods can reach the internet +firewall-cmd --permanent --new-policy k3s-pod-egress +firewall-cmd --permanent --policy k3s-pod-egress --set-target CONTINUE +firewall-cmd --permanent --policy k3s-pod-egress --add-ingress-zone internal +firewall-cmd --permanent --policy k3s-pod-egress --add-egress-zone public +firewall-cmd --permanent --policy k3s-pod-egress --add-rich-rule='rule family=ipv4 source address=10.42.0.0/16 accept' +# API (and KubeStellar on bihar) from the tailnet only +firewall-cmd --permanent --zone=public --add-rich-rule='rule family=ipv4 source address=100.64.0.0/10 destination address=/32 port port=6443 protocol=tcp accept' +``` + +bihar's `install-host.sh` applies these rules. On karnataka the rules were applied +by hand. + +### Differences + +| | bihar | karnataka | +|---|---|---| +| service CIDR | `10.96.0.0/12` (KubeFlex's vcluster hard-codes the kind range) | k3s default `10.43.0.0/16` | +| Extras | ingress-nginx (SSL passthrough), KubeFlex, KubeStellar | KubeVirt, CDI, ARC, workloads (see [Karnataka](karnataka.md)) | +| Reset | `reset-k3s.sh` | none | + +## KubeStellar (hub on bihar) + +KubeStellar 0.30.2 core chart (`oci://ghcr.io/kubestellar/kubestellar/core-chart`): + +- **ITS `its1`:** vcluster with the OCM cluster-manager. +- **WDS `wds1`:** where you apply workloads. +- Both are served at `https://.100.103.44.48.sslip.io:9443`. Port 9443 is open + to the tailnet only. +- kubectl contexts on bihar: `bihar`, `its1`, `wds1` (in `~/.kube/config`). +- The join command is in `bihar:~/.kube/its1-join.cmd`. **It is a secret.** + +Workload execution clusters (joined with `clusteradm` 1.0.0 `--singleton`): + +| Cluster | Labels | +|---|---| +| bihar | `location-group=home name=bihar cpu=intel role=hub` | +| karnataka | `location-group=home name=karnataka cpu=amd gpu=amd kubevirt=true` | + +Use it like this: apply manifests with `kubectl --context wds1`, and add a +`BindingPolicy` with a `clusterSelectors` label match. Set +`wantSingletonReportedState: true` to get status back from one cluster. Notes are +in [`home-k3s/bihar/KUBESTELLAR.md`](https://github.com/hanthor/dotfiles/blob/master/home-k3s/bihar/KUBESTELLAR.md). + +### Problems found during installation (2026-11-17) +- If metrics-server is not ready, the KubeFlex hooks fail with "stale GroupVersion + discovery". Correct the firewall, then restart `kubeflex-controller-manager`. +- A failed `ks-core` release stopped on the uninstall-postgresql hook. To recover, + run `helm uninstall ks-core --no-hooks`, delete the `kubeflex-system`, + `its1-system` and `wds1-system` namespaces, then install again. diff --git a/docs/src/servers/home-k8s-images.md b/docs/src/servers/home-k8s-images.md index d04a7f6d..a076e2c6 100644 --- a/docs/src/servers/home-k8s-images.md +++ b/docs/src/servers/home-k8s-images.md @@ -1,5 +1,10 @@ # Home Kubernetes Server Images +> **Not deployed (2026-11-18).** bihar and karnataka run stock Bluefin +> (`ghcr.io/projectbluefin/utah:testing`), not these images; k3s is installed on +> the host from [`home-k3s/`](https://github.com/hanthor/dotfiles/tree/master/home-k3s). +> The `server-images` CI still builds them. Treat this page as a design for a future rebuild. + Bootable container images for the home K8s cluster (bihar, karnataka, starter). Built from **Fedora Hummingbird + GNOME desktop + K3s**, with graphical target disabled by default. ## Image Variants @@ -264,7 +269,7 @@ sudo tailscale up ## See Also -- [Talos K8s Cluster Handbook](./talos-k8s/cluster.md) — Bihar & karnataka hardware, networking, advanced operations +- [Talos K8s Cluster Handbook](../archive/talos-home-cluster/cluster.md) — Bihar & karnataka hardware, networking, advanced operations - [AWS K8s Cluster Handbook](./aws-k8s/cluster.md) — Production cluster migration info - [Roles Documentation](../roles/) — Individual role configurations - [GitHub Releases](https://github.com/hanthor/dotfiles/releases) — Download ISOs and archives diff --git a/docs/src/servers/karnataka.md b/docs/src/servers/karnataka.md new file mode 100644 index 00000000..fcb28a16 --- /dev/null +++ b/docs/src/servers/karnataka.md @@ -0,0 +1,59 @@ +# Karnataka + +The main home workload server: k3s with KubeVirt (punjab and the test VMs run +here), GitHub Actions runners, Frappe LMS, Corral, and the cold standby for the +hives. Since late 2026-10 it runs Bluefin, not Talos. The +[Talos page](../archive/talos-home-cluster/karnataka/README.md) is archived. + +| | | +|---|---| +| Hardware | AMD Strix Halo APU (32 threads, integrated GPU), 125 GB of RAM | +| Disk | 912 GB NVMe (`/var`), **88% used on 2026-11-18** | +| OS | Bluefin `ghcr.io/projectbluefin/utah:testing` | +| Tailnet | `karnataka` / `100.66.158.127` | +| Access | `ssh james@karnataka` (passwordless sudo). `kubectl` from punjab uses `~/.kube/config` | +| Ansible | Yes: [`host_vars/karnataka.yml`](https://github.com/hanthor/dotfiles/blob/master/host_vars/karnataka.yml) | + +## k3s + +k3s v1.36.5+k3s1 runs as a binary in `/var/usrlocal/bin`. It binds to the tailnet IP +only, uses flannel on `tailscale0`, and has traefik and servicelb disabled. The kit is in +[`home-k3s/karnataka/`](https://github.com/hanthor/dotfiles/tree/master/home-k3s/karnataka): +- `install-host.sh` installs k3s. +- `apply-stack.sh` installs KubeVirt v1.9.1, CDI v1.66.1, corral-web, the AMD GPU + device plugin and Lemonade. + +See [Home k3s and KubeStellar](home-k3s.md) for the shared design and the firewalld rules. + +### Workloads (2026-11-18) + +| Namespace | What | Exposed as | +|---|---|---| +| `corral-vms` | KubeVirt VMs: **punjab** (agent box, 500 Gi root), `messages` and the `macos-beeper*` disks (macOS for Beeper/iMessage), tuna-q test VMs, `mandelbrot-visual` (stopped) | punjab is on the tailnet | +| `corral` | `corral-web` (VM manager UI) and its Tailscale sidecar | `https://karnataka.manatee-basking.ts.net/`, tailnet node `corral` | +| `front-door` | Caddy link page and redirects (`/course`, `/learn`, `/frappe`, `/books`, `/corral`) | `http://karnataka/` | +| `lms` | Frappe Learning v16.1.0 (`frappe`, `db`, `redis`), local image from `frappe-lms-image/` | `https://karnataka.manatee-basking.ts.net:8443` | +| `beale` | `beale-course` (course site) | `beale.karnataka.ts.net` | +| `beeper-bridges` | `rustpush` (iMessage bridge) | none | +| `arc-systems`, `tuna-ci`, `tuna-ci-storage` | GitHub Actions runner scale sets `tuna-ci-24.04` (1-6), `tuna-ci-26.04` (0-2), `tuna-iso-gpu` (0-1), with scratch storage | none | +| `lemonade` | Lemonade AMD LLM server, **scaled to 0** | `:13305` (tailscale serve) | +| `open-cluster-management-agent*` | KubeStellar WEC agent for the hub on bihar | none | + +`tailscale serve` (tailnet only): +- `:80` goes to front-door. +- `:443` goes to corral-web. +- `:8443` goes to LMS. +- `:8444` goes to `127.0.0.1:8080` (RaveBSD books). +- `:13305` goes to Lemonade. + +## Hive standby + +`/var/srv/hives` is a copy of the hive data. When bihar is active, bihar pushes it +here every 15 min. The containers stay stopped until the lease on spindle names +karnataka. See [Hives hosting and failover](hives.md). + +## Notes + +- **Disk:** `/var` is 88% full. The ARC runner PVCs and the macOS VM disks use most of + the space. Check this before you add VMs. +- punjab is a VM on this host. If karnataka restarts, punjab also restarts. diff --git a/docs/src/servers/punjab/README.md b/docs/src/servers/punjab/README.md index ba8b22e6..092df898 100644 --- a/docs/src/servers/punjab/README.md +++ b/docs/src/servers/punjab/README.md @@ -1,65 +1,49 @@ # Punjab -> **TunaOS infrastructure.** This runs in the TunaOS AWS account and serves -> the TunaOS project (Hive, Matrix, CI), not James's personal fleet. See -> [TunaOS AWS Account & IaC](../aws/README.md). - -Headless agent/dev box — the fleet's AWS-hosted Ansible host. It runs in the -same AWS account as the former [AWS Talos cluster](../aws-k8s/cluster.md), -but is an ordinary fleet member, not a cluster node. The Talos cluster was -terminated; Matrix and the Hives now run on the [OCI VPS](../spindle-vps/README.md). - -## Hardware - -- EC2 `t3.large` (2 vCPU, 8 GiB, unlimited CPU credits), `us-east-1d` -- 250 GB gp3 root volume (unencrypted — predates EBS encryption-by-default; - snapshotted daily ×7 / weekly ×4 by DLM) -- Default VPC, stable Elastic IP -- IMDSv2 required, hop limit 1; termination protection on -- Instance role: Session Manager only -- Tailscale: `punjab` (MagicDNS) - -The instance, its security group and key pair are codified in -[`aws/punjab.tf`](https://github.com/hanthor/dotfiles/blob/master/aws/punjab.tf) — -see [AWS Account](../aws/README.md). - -## OS - -Ubuntu 24.04 LTS (stock `ubuntu-noble-24.04-amd64-server` AMI), AWS kernel. +Headless agent and development box, and the fleet's Ansible host. Since +2026-10-27 it is a **KubeVirt VM on [karnataka](../karnataka.md)**. The EC2 +instance it replaced (tailnet node `punjab-aws-retired`, `100.78.73.8`) was last seen on 2026-10-28. +See [EC2 history](#ec2-history-retired). + +| | | +|---|---| +| Runs on | karnataka k3s, `corral-vms/punjab` (VM created 2026-10-27) | +| Resources | 12 cores, 24 Gi of RAM, 500 Gi root PVC (`punjab-root`, local-path on karnataka) | +| OS | Ubuntu 24.04 LTS | +| Tailnet | `punjab` / `100.103.187.46` | +| Access | `ssh ubuntu@punjab` (there is no `james` user). For the VM console, run `virtctl console -n corral-vms punjab` from a host that has karnataka's kubeconfig | ## Role -- Agent host: Claude Code, Kiro, `pi`, run in tmux -- **Kiro Crew gateway** — `kirocrew.service` (system unit, runs as `ubuntu`, - binary `~/.local/bin/kirocrew`, env `/etc/kirocrew/kirocrew.env`). Installed - with Kiro's own installer; the [`kirocrew`](../../roles/kirocrew.md) role keeps - it running, its env file root-only, and its dashboard on localhost. -- AWS admin box: `awscli` + `opentofu` (via `extra_brews`) for the - [AWS IaC](../aws/README.md) +- Agent host: Claude Code, OMP and Codex sessions in tmux. Most git worktrees + are in `~/dev`. +- `kubectl` against karnataka's k3s: `~/.kube/config`. +- Hive fork builds: `~/dev/hive-v6-port` (`tuna-os/hive`, deploy branches). The + amd64 binary that bihar and karnataka run was built here. See [Hives hosting](../hives.md). +- `sccache.service` (user) is a shared compiler cache for cargo builds. +- The **Kiro Crew gateway is retired**: `kirocrew.service` is masked. -## Access +## Disk -- **Normal**: `ssh punjab` over Tailscale. -- **Break-glass** (Tailscale down): SSM Session Manager, no open port needed: - ```bash - aws ssm start-session --region us-east-1 --target - ``` - (needs `session-manager-plugin`; the instance ID is in the OpenTofu state). -- **Cluster**: `kubectl`/`talosctl` work from here with - `~/.kube/config-aws-migration` / `~/.talos/config-aws-migration`. -- **AWS CLI**: prefer the scoped admin user; it can plan the whole config - (read-only IAM). Log in as root (`aws login`) only for IAM changes, and don't - leave a root session lying around on a shared box. +On 2026-11-18 `/` was **96% full (19 GB free)**. `~/dev` used 156 GB. Delete +stale worktrees under `~/dev` and `~/tuna-wt` before you make the PVC larger. -## Host hardening +## Dependencies -- sshd drop-in (`sshd_harden: true`): no root login, no passwords/kbd-interactive, - no X11, `MaxAuthTries 3`. -- `unattended-upgrades` on (Ubuntu default). -- Kiro Crew dashboard binds localhost only — the `kirocrew` role asserts it. +punjab is a VM on karnataka. When karnataka restarts or k3s restarts its VMs, +punjab also restarts, and the agent tmux sessions stop. ## Ansible notes (`host_vars/punjab.yml`) -- `skip_proxy: true` — no podman on stock Ubuntu, so no Caddy quadlet. -- Not a desktop: no GNOME/flatpak/browser roles. -- `syncthing` only runs if its brew binary exists (it currently doesn't). +- `skip_proxy: true`: there is no podman on stock Ubuntu, so there is no Caddy quadlet. +- Not a desktop: no GNOME, flatpak or browser roles. +- `syncthing` runs only if its brew binary exists. + +## EC2 history (retired) + +From 2026-08 to 2026-10, punjab was an EC2 `t3.large` in `us-east-1`, the +Ansible host of the TunaOS AWS account ([AWS account](../aws/README.md)). It had +SSM break-glass access, daily DLM snapshots and Kiro Crew. It is described in +[`aws/punjab.tf`](https://github.com/hanthor/dotfiles/blob/master/aws/punjab.tf). +Its tailnet node is now `punjab-aws-retired`. Before you delete the instance and +its snapshots, confirm that the VM has all the data. diff --git a/docs/src/servers/spindle-vps/README.md b/docs/src/servers/spindle-vps/README.md index d59ea992..d911959e 100644 --- a/docs/src/servers/spindle-vps/README.md +++ b/docs/src/servers/spindle-vps/README.md @@ -1,22 +1,31 @@ -# Spindle VPS (Matrix and Hive) +# Spindle VPS (Matrix and the public front door) The Oracle Cloud VPS in Frankfurt runs the Matrix homeserver that was on the AWS Talos cluster, -terminated 2026-10-25. **Spindle**, **MAS** (delegated login), **Postgres** (MAS database only), -standalone **LiveKit** and **lk-jwt-service** (MatrixRTC; no ESS chart/runtime), and **Caddy** run -as Docker Compose services under `/srv/spindle`. - -The School and Reef Hives, Hub and branded hub frontend run separately under `/srv/hives`, sharing -the `spindle_default` network and Caddy ingress. Their layout, roster, identity and the safe way to -edit them are in the [Hive operations guide](../../cluster/hive.md#current-deployment-spindle-vps). -The `omp-auth-broker` user service also runs on this host. The Hive deployment's runtime -configuration and data are not restored by `deploy.sh`; preserve and back them up separately. Each -spoke is capped at 2560 MB (raised from 2 GB on 2026-11-13 after OOM kills during builds), so -budget that alongside Spindle and LiveKit when sizing the host. +terminated 2026-10-25. **Spindle**, **Postgres**, standalone **LiveKit** and **lk-jwt-service** +(MatrixRTC; no ESS chart/runtime), **synctap** and **Caddy** run as Docker Compose services under +`/srv/spindle` (state on 2026-11-18). On 2026-11-16, Spindle's built-in OIDC replaced **MAS** +(`auth.reilly.asia` now goes to Spindle). The MAS sections below are history. + +**The hives moved off this host on 2026-11-18.** School, Reef, Hub and the hub +frontend now run on [bihar](../bihar.md), with [karnataka](../karnataka.md) as a cold +standby. Spindle keeps three hive roles: + +- **Caddy front door:** `school`, `reef` and `hub.tunaos.org` go to the active host's + tailnet IP on ports 3100-3103. `hive-failover` rewrites these upstreams. +- **Failover coordinator:** the lease in `/var/lib/hive-failover/active`, + `/usr/local/sbin/hive-failover`, and `hive-failover.timer`. +- **OMP auth broker:** the `omp-auth-broker` user service on `100.113.75.88:8765`. The hives + connect to it over the tailnet. The `omp-auth-broker-docker-proxy` socket + (`172.18.0.1:18765`) was used only by the hives when they ran on this host. + +See [Hives hosting and failover](../hives.md). The stopped copy in `/srv/hives` is +kept for rollback only (`MOVED-TO-BIHAR.txt`). This removes the 2560 MB per-spoke +memory budget from this host. | Piece | Source | |---|---| | Host | Oracle Cloud Always Free, `VM.Standard.A1.Flex`, Ubuntu 24.04 aarch64, `eu-frankfurt-1` AD-1; public IPv4 `92.5.82.232`, Tailscale `100.113.75.88` | -| Spindle | `ghcr.io/tuna-os/spindle:v0.1.1` (deployed 2026-11-07; main `1d8dfd99`, arm64 release image). The upgrade from the `outbox-hotfix-arm64` build and its rehearsal are in [upgrade-to-main.md](upgrade-to-main.md) | +| Spindle | `ghcr.io/tuna-os/spindle:v0.1.6` running on 2026-11-18 (v0.1.1 deployed 2026-11-07; arm64 release images). The upgrade from the `outbox-hotfix-arm64` build and its rehearsal are in [upgrade-to-main.md](upgrade-to-main.md) | | MatrixRTC | `livekit-server:v1.13.5` + `lk-jwt-service:0.6.0` as standalone containers | | Data | `ess_ess-spindle-data.tar.gz` (the store, taken 15:33 UTC) and `postgres/2026-10-25/mas.dump` (15:22 UTC), from `~/backups/tunaos-aws-2026-10-25` on himachal-1 | | Config | production runtime + delegated-auth block, MAS under/overrides and LiveKit template; deploy.sh renders credentials from cluster-state | diff --git a/hives/README.md b/hives/README.md new file mode 100644 index 00000000..b659b89d --- /dev/null +++ b/hives/README.md @@ -0,0 +1,19 @@ +# hives/ — hosting and failover for the tuna-os hives + +Source for the scripts, compose host layers and systemd units that run the +School, Reef and Hub hives on **bihar** (active) with **karnataka** as a cold +standby, coordinated by a lease on **spindle**. + +The runbook is [docs/src/servers/hives.md](../docs/src/servers/hives.md). It covers +the design, the install locations, daily operations, failover and the gotchas. + +| File | Installed at | +|---|---| +| `hives`, `hive-reconcile`, `hive-sync` | `/var/srv/hives-host/` on bihar and karnataka | +| `compose..yaml` | `/var/srv/hives-host/` on that host | +| `systemd/hive-{reconcile,sync}.{service,timer}` | `/etc/systemd/system/` on bihar and karnataka | +| `hive-failover` | `/usr/local/sbin/` on spindle | +| `systemd/hive-failover.{service,timer}` | `/etc/systemd/system/` on spindle | + +There is no automated deploy. These files are copies of the deployed versions, +taken on 2026-11-18. Change both the repository and the hosts at the same time. diff --git a/hives/compose.bihar.yaml b/hives/compose.bihar.yaml new file mode 100644 index 00000000..f6d7be67 --- /dev/null +++ b/hives/compose.bihar.yaml @@ -0,0 +1,37 @@ +# Host-specific layer for running the hives on bihar (x86_64) instead of +# spindle (aarch64). Kept outside /var/srv/hives so the spindle rsync +# (--delete) never removes it. Use with: +# cd /var/srv/hives && docker-compose -f compose.yaml -f compose.override.yaml \ +# -f /var/srv/hives-host/compose.bihar.yaml up -d +# Caddy on spindle reaches these ports over the tailnet only. +# Images pinned to the digests spindle ran with hive-2ef7dc7c0 (2026-11-13 +# builds); v6-latest has moved on. Bump binary and images together. +services: + hub: + image: ghcr.io/hivecommons/hive-hub@sha256:3c0162ebb5db7a49ec31b162d303af36dcffd96770288e910dff964090eb8fa4 + ports: ["100.103.44.48:3101:3001"] + mem_limit: 1g + environment: + HIVE_SHARED_AGENT_POOL_MAX_SESSIONS: "4" + volumes: + - /var/srv/hives-host/bin/hive-2ef7dc7c0-amd64:/usr/local/bin/hive:ro + hub-front: + ports: ["100.103.44.48:3100:80"] + school: + image: ghcr.io/hivecommons/hive@sha256:d56425fbaa1b5854a96356d56ebc2f371f337038339c56c94dcf3f2ec29ead51 + ports: ["100.103.44.48:3102:3001"] + mem_limit: 8g + environment: + # The broker listens on spindle's tailnet IP; the old 172.18.0.1 socket + # proxy existed only for spindle's Docker bridge. + OMP_AUTH_BROKER_URL: "http://100.113.75.88:8765" + volumes: + - /var/srv/hives-host/bin/hive-2ef7dc7c0-amd64:/usr/local/bin/hive:ro + reef: + image: ghcr.io/hivecommons/hive@sha256:d56425fbaa1b5854a96356d56ebc2f371f337038339c56c94dcf3f2ec29ead51 + ports: ["100.103.44.48:3103:3001"] + mem_limit: 8g + environment: + OMP_AUTH_BROKER_URL: "http://100.113.75.88:8765" + volumes: + - /var/srv/hives-host/bin/hive-2ef7dc7c0-amd64:/usr/local/bin/hive:ro diff --git a/hives/compose.karnataka.yaml b/hives/compose.karnataka.yaml new file mode 100644 index 00000000..1d928b59 --- /dev/null +++ b/hives/compose.karnataka.yaml @@ -0,0 +1,37 @@ +# Host-specific layer for the cold-standby hives on karnataka (x86_64); same layout as bihar. Formerly +# spindle (aarch64). Kept outside /var/srv/hives so the spindle rsync +# (--delete) never removes it. Use with: +# cd /var/srv/hives && docker-compose -f compose.yaml -f compose.override.yaml \ +# -f /var/srv/hives-host/compose.karnataka.yaml up -d +# Caddy on spindle reaches these ports over the tailnet only. +# Images pinned to the digests spindle ran with hive-2ef7dc7c0 (2026-11-13 +# builds); v6-latest has moved on. Bump binary and images together. +services: + hub: + image: ghcr.io/hivecommons/hive-hub@sha256:3c0162ebb5db7a49ec31b162d303af36dcffd96770288e910dff964090eb8fa4 + ports: ["100.66.158.127:3101:3001"] + mem_limit: 1g + environment: + HIVE_SHARED_AGENT_POOL_MAX_SESSIONS: "4" + volumes: + - /var/srv/hives-host/bin/hive-2ef7dc7c0-amd64:/usr/local/bin/hive:ro + hub-front: + ports: ["100.66.158.127:3100:80"] + school: + image: ghcr.io/hivecommons/hive@sha256:d56425fbaa1b5854a96356d56ebc2f371f337038339c56c94dcf3f2ec29ead51 + ports: ["100.66.158.127:3102:3001"] + mem_limit: 8g + environment: + # The broker listens on spindle's tailnet IP; the old 172.18.0.1 socket + # proxy existed only for spindle's Docker bridge. + OMP_AUTH_BROKER_URL: "http://100.113.75.88:8765" + volumes: + - /var/srv/hives-host/bin/hive-2ef7dc7c0-amd64:/usr/local/bin/hive:ro + reef: + image: ghcr.io/hivecommons/hive@sha256:d56425fbaa1b5854a96356d56ebc2f371f337038339c56c94dcf3f2ec29ead51 + ports: ["100.66.158.127:3103:3001"] + mem_limit: 8g + environment: + OMP_AUTH_BROKER_URL: "http://100.113.75.88:8765" + volumes: + - /var/srv/hives-host/bin/hive-2ef7dc7c0-amd64:/usr/local/bin/hive:ro diff --git a/hives/hive-failover b/hives/hive-failover new file mode 100755 index 00000000..38368bd0 --- /dev/null +++ b/hives/hive-failover @@ -0,0 +1,137 @@ +#!/bin/bash +# hive-failover: spindle coordinates which host runs the hives (cold standby). +# +# The lease file names the one host that may run the hives: +# /var/lib/hive-failover/active bihar | karnataka | none +# Each host runs hive-reconcile every minute: it starts the hives when the +# lease names it, stops them when it does not, and stops them when it cannot +# read the lease for HOST_FENCE_MIN minutes. That keeps a single writer. +# +# hive-failover check run by hive-failover.timer every minute +# hive-failover status +# hive-failover move planned move / failback: stop, sync, start +# hive-failover hold on|off pause / resume automatic failover +set -uo pipefail + +STATE=/var/lib/hive-failover +LEASE=$STATE/active +declare -A IP=([bihar]=100.103.44.48 [karnataka]=100.66.158.127) +KEY=/root/.ssh/hive_failover +CADDYFILE=/srv/spindle/Caddyfile +CADDY_CTR=spindle-caddy-1 +FAIL_MIN=5 # unhealthy minutes before the lease is revoked +HOST_FENCE_MIN=8 # must match hive-reconcile on the hosts +START_WAIT_MIN=10 # > HOST_FENCE_MIN: start standby blind only after this +NTFY_TOPIC=$(sed -n 's/^NTFY_TOPIC=//p' /srv/hives/.env 2>/dev/null) + +log() { logger -t hive-failover -- "$*"; echo "$*"; } +notify() { log "$*"; [ -n "$NTFY_TOPIC" ] && curl -s -m10 -H "Title: hive failover" -d "$*" "https://ntfy.sh/$NTFY_TOPIC" >/dev/null; true; } +peer() { [ "$1" = bihar ] && echo karnataka || echo bihar; } +now() { date +%s; } +rsh() { + local h=$1; shift + ssh -i "$KEY" -o BatchMode=yes -o ConnectTimeout=15 -o ServerAliveInterval=10 -o ServerAliveCountMax=3 \ + -o UserKnownHostsFile=$STATE/known_hosts -o StrictHostKeyChecking=accept-new \ + "james@${IP[$h]}" "$@" +} +healthy() { + local ip=${IP[$1]} p + for p in 3101 3102 3103; do curl -sf -m15 -o /dev/null "http://$ip:$p/" || return 1; done +} +running_count() { rsh "$1" 'sudo podman ps -q --filter label=com.docker.compose.project=hives | wc -l'; } +set_lease() { echo "$1" > "$LEASE.tmp" && chmod 644 "$LEASE.tmp" && mv "$LEASE.tmp" "$LEASE"; } + +set_caddy() { + local ip=${IP[$1]} tmp + tmp=$(mktemp) + sed -E "s#reverse_proxy 100\.(103\.44\.48|66\.158\.127):(310[0-3])#reverse_proxy $ip:\2#" "$CADDYFILE" > "$tmp" + cmp -s "$tmp" "$CADDYFILE" || cat "$tmp" > "$CADDYFILE" # in place: keep the bind-mounted inode + rm -f "$tmp" + docker cp "$CADDYFILE" "$CADDY_CTR:/tmp/Caddyfile" >/dev/null && + docker exec "$CADDY_CTR" caddy reload --adapter caddyfile --config /tmp/Caddyfile >/dev/null 2>&1 +} + +wait_healthy() { # host minutes + local i + for ((i = 0; i < $2 * 6; i++)); do healthy "$1" && return 0; sleep 10; done + return 1 +} + +activate() { # host -> start, wait, switch Caddy, take lease + local h=$1 + # Lease first: hive-reconcile on $h would stop hives it sees running + # while the lease does not name it. The old host is already stopped. + set_lease "$h" + rsh "$h" 'sudo /var/srv/hives-host/hives up -d' >/dev/null 2>&1 || log "start on $h over ssh failed; hive-reconcile there starts it from the lease" + if wait_healthy "$h" 5; then + set_caddy "$h" && notify "hives now active on $h" + rm -f $STATE/down_since $STATE/pending + return 0 + fi + notify "hives on $h did not become healthy within 5 min (lease is $h, Caddy unchanged)" + return 1 +} + +cmd_check() { + [ -f $STATE/hold ] && exit 0 + local a t down n + a=$(cat "$LEASE") + if [ "$a" != none ]; then + if healthy "$a"; then rm -f $STATE/down_since; exit 0; fi + [ -f $STATE/down_since ] || now > $STATE/down_since + n=$(( ($(now) - $(cat $STATE/down_since)) / 60 )) + log "hives on $a unhealthy for ${n} min" + [ "$n" -lt "$FAIL_MIN" ] && exit 0 + t=$(peer "$a") + notify "hives on $a unhealthy for ${n} min: revoking lease, failing over to $t" + echo "$t" > $STATE/pending + set_lease none + rsh "$a" 'sudo /var/srv/hives-host/hives stop' >/dev/null 2>&1 || log "could not stop hives on $a; waiting for it to self-fence" + a=none + fi + [ -f $STATE/pending ] || exit 0 + t=$(cat $STATE/pending); [ "$t" ] || exit 0 + local old; old=$(peer "$t") + down=$(cat $STATE/down_since 2>/dev/null || now); n=$(( ($(now) - down) / 60 )) + local c; c=$(running_count "$old" 2>/dev/null) + if [ "$c" = 0 ] || [ "$n" -ge "$START_WAIT_MIN" ]; then + log "starting $t (old host $old running=${c:-unknown}, down ${n} min)" + activate "$t" + else + log "waiting for $old to stop or self-fence (running=${c:-unreachable}, down ${n} min)" + fi +} + +cmd_move() { + local h=${1:?usage: hive-failover move bihar|karnataka} cur + [ -n "${IP[$h]:-}" ] || { echo "unknown host $h"; exit 2; } + cur=$(cat "$LEASE") + [ "$cur" = "$h" ] && { echo "already active on $h"; exit 0; } + touch $STATE/hold + rm -f $STATE/pending + set_lease none + if [ "$cur" != none ]; then + log "stopping hives on $cur" + rsh "$cur" 'sudo /var/srv/hives-host/hives stop' || { echo "could not stop $cur; fix it or stop it by hand, then rerun"; exit 1; } + log "syncing $cur -> $h" + rsh "$cur" "sudo /var/srv/hives-host/hive-sync --force $h" || { echo "sync failed; lease left at none, hold on"; exit 1; } + fi + activate "$h" && rm -f $STATE/hold +} + +cmd_status() { + local h + echo "lease: $(cat "$LEASE") hold: $([ -f $STATE/hold ] && echo on || echo off) pending: $(cat $STATE/pending 2>/dev/null || echo -)" + for h in bihar karnataka; do + printf '%-10s healthy=%s running=%s\n' "$h" "$(healthy $h && echo yes || echo no)" "$(running_count $h 2>/dev/null || echo ?)" + done + grep -oE 'reverse_proxy 100\.[0-9.]+:310[0-3]' "$CADDYFILE" | sort -u +} + +case "${1:-}" in + check) exec 9>$STATE/lock; flock -n 9 || exit 0; cmd_check ;; + move) exec 9>$STATE/lock; flock 9; cmd_move "${2:-}" ;; + status) cmd_status ;; + hold) case "${2:-}" in on) touch $STATE/hold ;; off) rm -f $STATE/hold ;; *) echo "hold on|off"; exit 2 ;; esac ;; + *) sed -n '2,15p' "$0"; exit 2 ;; +esac diff --git a/hives/hive-reconcile b/hives/hive-reconcile new file mode 100755 index 00000000..4d438ba5 --- /dev/null +++ b/hives/hive-reconcile @@ -0,0 +1,41 @@ +#!/bin/bash +# hive-reconcile: make this host follow the lease on spindle (run each minute). +# lease == this host, hives not all running -> start them +# lease != this host, hives running -> stop them +# lease unreadable for FENCE_MIN minutes -> stop them (self-fence) +# Touch /var/srv/hives-host/maintenance to skip starts (stops still happen). +set -uo pipefail +H=/var/srv/hives-host +ME=$(hostname -s) +FENCE_MIN=8 # spindle waits longer than this before a blind failover +LAST_OK=/run/hive-reconcile.last-ok +log() { logger -t hive-reconcile -- "$*"; echo "$*"; } + +running=$(podman ps -q --filter label=com.docker.compose.project=hives | wc -l) +lease=$(ssh -i $H/keys/lease -o BatchMode=yes -o ConnectTimeout=15 \ + -o UserKnownHostsFile=$H/keys/known_hosts -o StrictHostKeyChecking=accept-new \ + ubuntu@100.113.75.88 2>/dev/null | tr -d '[:space:]') + +if [[ "$lease" =~ ^(bihar|karnataka|none)$ ]]; then + date +%s > $LAST_OK + if [ "$lease" = "$ME" ]; then + if [ "$running" -lt 4 ] && [ ! -e $H/maintenance ]; then + log "lease is $ME and $running/4 hive containers run: starting" + $H/hives up -d + fi + elif [ "$running" -gt 0 ]; then + log "lease is '$lease', not $ME: stopping hives" + $H/hives stop + fi + exit 0 +fi + +# Lease unreadable (spindle or the path to it is down). +[ -f $LAST_OK ] || date +%s > $LAST_OK # first run after boot: start the clock +mins=$(( ($(date +%s) - $(cat $LAST_OK)) / 60 )) +if [ "$running" -gt 0 ] && [ "$mins" -ge "$FENCE_MIN" ]; then + log "lease unreadable for ${mins} min: self-fencing (stopping hives)" + $H/hives stop +elif [ "$running" -gt 0 ]; then + log "lease unreadable for ${mins} min (fence at $FENCE_MIN)" +fi diff --git a/hives/hive-sync b/hives/hive-sync new file mode 100755 index 00000000..ae739608 --- /dev/null +++ b/hives/hive-sync @@ -0,0 +1,31 @@ +#!/bin/bash +# hive-sync: push this host's hive data to the cold-standby peer. +# hive-sync timer mode: push to the peer only while the lease names this host +# hive-sync --force P push to P now (hive-failover move uses this after a stop) +# Leaves out caches and per-agent toolchains; agents rebuild them on demand. +set -euo pipefail +H=/var/srv/hives-host +ME=$(hostname -s) +declare -A IP=([bihar]=100.103.44.48 [karnataka]=100.66.158.127) +SSHOPT="-i $H/keys/sync -o BatchMode=yes -o ConnectTimeout=15 -o UserKnownHostsFile=$H/keys/known_hosts -o StrictHostKeyChecking=accept-new" + +if [ "${1:-}" = --force ]; then + P=${2:?peer} +else + [ "$ME" = bihar ] && P=karnataka || P=bihar + lease=$(ssh -i $H/keys/lease -o BatchMode=yes -o ConnectTimeout=15 \ + -o UserKnownHostsFile=$H/keys/known_hosts -o StrictHostKeyChecking=accept-new \ + ubuntu@100.113.75.88 2>/dev/null | tr -d '[:space:]') + [ "$lease" = "$ME" ] || { echo "lease is '$lease', not $ME: no sync"; exit 0; } +fi +[ -n "${IP[$P]:-}" ] && [ "$P" != "$ME" ] || { echo "bad peer $P"; exit 2; } + +rsync -aHA --numeric-ids --delete --info=stats1 -e "ssh $SSHOPT" --rsync-path="sudo rsync" \ + --exclude='data/home/.cache' --exclude='data/home/go' \ + --exclude='data/home/agents/*/go' --exclude='data/home/agents/*/.rustup' \ + --exclude='data/home/agents/*/.cargo' --exclude='data/home/agents/*/.npm' \ + --exclude='data/home/agents/*/.cache' --exclude='data/home/agents/*/.omp/natives' \ + /var/srv/hives/ "james@${IP[$P]}:/var/srv/hives/" +ssh $SSHOPT "james@${IP[$P]}" 'sudo chcon -R -t container_file_t /var/srv/hives' +date -u +%FT%TZ > $H/last-sync-to-$P +echo "synced to $P" diff --git a/hives/hives b/hives/hives new file mode 100755 index 00000000..3e9e4d48 --- /dev/null +++ b/hives/hives @@ -0,0 +1,8 @@ +#!/bin/bash +# Run docker-compose for the hives against rootful podman with the host layer. +# Usage: sudo hives ps | up -d | logs -f school | stop ... +H=/var/srv/hives-host +exec env DOCKER_HOST=unix:///run/podman/podman.sock /home/linuxbrew/.linuxbrew/bin/docker-compose \ + --project-directory /var/srv/hives -p hives \ + -f /var/srv/hives/compose.yaml -f /var/srv/hives/compose.override.yaml \ + -f "$H/compose.$(hostname -s).yaml" "$@" diff --git a/hives/systemd/hive-failover.service b/hives/systemd/hive-failover.service new file mode 100644 index 00000000..b1d13738 --- /dev/null +++ b/hives/systemd/hive-failover.service @@ -0,0 +1,7 @@ +[Unit] +Description=Hive failover check (spindle coordinates bihar/karnataka) +After=network-online.target tailscaled.service docker.service +[Service] +Type=oneshot +ExecStart=/usr/local/sbin/hive-failover check +TimeoutStartSec=20min diff --git a/hives/systemd/hive-failover.timer b/hives/systemd/hive-failover.timer new file mode 100644 index 00000000..1e138025 --- /dev/null +++ b/hives/systemd/hive-failover.timer @@ -0,0 +1,8 @@ +[Unit] +Description=Run the hive failover check every minute +[Timer] +OnBootSec=2min +OnUnitInactiveSec=1min +AccuracySec=10s +[Install] +WantedBy=timers.target diff --git a/hives/systemd/hive-reconcile.service b/hives/systemd/hive-reconcile.service new file mode 100644 index 00000000..69d1c2f6 --- /dev/null +++ b/hives/systemd/hive-reconcile.service @@ -0,0 +1,7 @@ +[Unit] +Description=Follow the hive lease on spindle +After=network-online.target tailscaled.service podman.socket +[Service] +Type=oneshot +ExecStart=/var/srv/hives-host/hive-reconcile +TimeoutStartSec=10min diff --git a/hives/systemd/hive-reconcile.timer b/hives/systemd/hive-reconcile.timer new file mode 100644 index 00000000..c5d27377 --- /dev/null +++ b/hives/systemd/hive-reconcile.timer @@ -0,0 +1,8 @@ +[Unit] +Description=Follow the hive lease on spindle (timer) +[Timer] +OnBootSec=1min +OnUnitInactiveSec=1min +AccuracySec=10s +[Install] +WantedBy=timers.target diff --git a/hives/systemd/hive-sync.service b/hives/systemd/hive-sync.service new file mode 100644 index 00000000..c30076a6 --- /dev/null +++ b/hives/systemd/hive-sync.service @@ -0,0 +1,7 @@ +[Unit] +Description=Push hive data to the cold-standby peer +After=network-online.target tailscaled.service podman.socket +[Service] +Type=oneshot +ExecStart=/var/srv/hives-host/hive-sync +TimeoutStartSec=2h diff --git a/hives/systemd/hive-sync.timer b/hives/systemd/hive-sync.timer new file mode 100644 index 00000000..d3406b25 --- /dev/null +++ b/hives/systemd/hive-sync.timer @@ -0,0 +1,8 @@ +[Unit] +Description=Push hive data to the cold-standby peer (timer) +[Timer] +OnBootSec=10min +OnUnitInactiveSec=15min +AccuracySec=10s +[Install] +WantedBy=timers.target diff --git a/home-k3s/README.md b/home-k3s/README.md new file mode 100644 index 00000000..e752e089 --- /dev/null +++ b/home-k3s/README.md @@ -0,0 +1,13 @@ +# home-k3s/ — single-node k3s on bihar and karnataka + +Each host runs its own k3s on Bluefin, bound to its tailnet IP. These kits were +copied from `~/home-k3s` on each host on 2026-11-18. They replace the archived +`talos-k8s/`. + +- `bihar/`: k3s, firewall, `reset-k3s.sh`, and KubeStellar notes (hub: ITS `its1`, WDS `wds1`). +- `karnataka/`: k3s, `apply-stack.sh` (KubeVirt, CDI, corral-web, AMD GPU plugin, + Lemonade), front-door, and Frappe LMS. Secrets (for example `Secret/lms-secrets`) + are created out of band and are never stored here. + +Handbook: [docs/src/servers/home-k3s.md](../docs/src/servers/home-k3s.md), +[bihar](../docs/src/servers/bihar.md), [karnataka](../docs/src/servers/karnataka.md). diff --git a/home-k3s/bihar/KUBESTELLAR.md b/home-k3s/bihar/KUBESTELLAR.md new file mode 100644 index 00000000..8b219750 --- /dev/null +++ b/home-k3s/bihar/KUBESTELLAR.md @@ -0,0 +1,19 @@ +# Bihar k3s + KubeStellar hub + +k3s v1.36.5 single-node server on bihar tailnet IP 100.103.44.48 (install-host.sh, config.yaml). +- service-cidr 10.96.0.0/12: KubeFlex vcluster ITS hard-codes the kind range; k3s default 10.43/16 breaks it. +- firewalld: cni0 in zone internal, pod->kubelet 10250 rule, policy k3s-pod-egress (internal->public). + Without these, metrics-server fails (discovery breaks KubeFlex hooks) and pod DNS/egress fails. + +KubeStellar core chart 0.30.2 (helm release ks-core, ns default): + domain 100.103.44.48.sslip.io, externalPort 9443 (ingress-nginx hostPort, SSL passthrough, tailnet-only) + ITS its1 (vcluster) https://its1.100.103.44.48.sslip.io:9443 + WDS wds1 (k8s) https://wds1.100.103.44.48.sslip.io:9443 +kubectl contexts: bihar, its1, wds1 (~/.kube/config). + +WECs (clusteradm 1.0.0, --singleton): bihar, karnataka. Labels: location-group=home, name=, + bihar: cpu=intel role=hub; karnataka: cpu=amd gpu=amd kubevirt=true +Join token cmd: ~/.kube/its1-join.cmd (secret). Karnataka joined via its k3s.yaml pointed at 100.66.158.127. + +Deploy: apply objects to --context wds1 plus a BindingPolicy (clusterSelectors on labels); +add wantSingletonReportedState: true to get status back when bound to one cluster. diff --git a/home-k3s/bihar/config.yaml b/home-k3s/bihar/config.yaml new file mode 100644 index 00000000..9b162be1 --- /dev/null +++ b/home-k3s/bihar/config.yaml @@ -0,0 +1,19 @@ +# Bihar's tailnet address. K3s API and kubelet do not bind the LAN NIC. +bind-address: 100.103.44.48 +advertise-address: 100.103.44.48 +node-ip: 100.103.44.48 +flannel-iface: tailscale0 +# KubeFlex vcluster control planes hard-code the kind service range. +service-cidr: 10.96.0.0/12 +cluster-dns: 10.96.0.10 +resolv-conf: /etc/rancher/k3s/resolv.conf +tls-san: + - 100.103.44.48 + - bihar + - bihar.manatee-basking.ts.net +kubelet-arg: + - address=100.103.44.48 +disable: + - traefik + - servicelb +write-kubeconfig-mode: "0600" diff --git a/home-k3s/bihar/install-host.sh b/home-k3s/bihar/install-host.sh new file mode 100755 index 00000000..0cf970c3 --- /dev/null +++ b/home-k3s/bihar/install-host.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +# Run as root on Bihar. Leaves Utah, host firewall and Tailscale identity intact. +# K3s state stays in its default writable /var/lib/rancher/k3s. The upstream +# SELinux module cannot be added to Utah's read-only policy store at runtime; +# K3s SELinux container labeling is not enabled. Host SELinux stays enforcing. +set -euo pipefail +cd "$(dirname "$0")" +[ "$(id -u)" = 0 ] || { echo 'Run as root' >&2; exit 1; } +[ "$(hostname -s)" = bihar ] || { echo 'Wrong host' >&2; exit 1; } +ip -4 addr show dev tailscale0 | grep -q '100\.103\.44\.48/32' || { echo 'Expected Tailscale address is absent' >&2; exit 1; } +tmp=$(mktemp) +trap 'rm -f "$tmp"' EXIT +curl -fL --retry 3 'https://github.com/k3s-io/k3s/releases/download/v1.36.5%2Bk3s1/k3s' -o "$tmp" +printf '%s %s\n' d73847bcd3c5fccef0115b372e2f9a91f3032dc84bbf71518a4617565294d313 "$tmp" | sha256sum -c - +install -D -m 0755 "$tmp" /var/usrlocal/bin/k3s +install -D -m 0600 config.yaml /etc/rancher/k3s/config.yaml +install -D -m 0644 k3s.service /etc/systemd/system/k3s.service +install -D -m 0644 resolv.conf /etc/rancher/k3s/resolv.conf +install -D -m 0755 wait-tailnet.sh /var/usrlocal/libexec/k3s-wait-tailnet +# API reachable from the tailnet only (mirrors Karnataka). 9443 is ingress-nginx hostPort (KubeFlex control planes). +for p in 6443 9443; do firewall-cmd --permanent --zone=public --add-rich-rule="rule family=ipv4 source address=100.64.0.0/10 destination address=100.103.44.48/32 port port=$p protocol=tcp accept"; done +# Pod network -> kubelet (metrics-server); mirrors Karnataka. +firewall-cmd --permanent --zone=internal --change-interface=cni0 +firewall-cmd --permanent --zone=internal --add-rich-rule="rule family=ipv4 source address=10.42.0.0/16 destination address=100.103.44.48/32 port port=10250 protocol=tcp accept" +# Pod egress to the internet (internal -> public); mirrors Karnataka. +firewall-cmd --permanent --new-policy=k3s-pod-egress || true +firewall-cmd --permanent --policy=k3s-pod-egress --set-target=CONTINUE +firewall-cmd --permanent --policy=k3s-pod-egress --add-ingress-zone=internal +firewall-cmd --permanent --policy=k3s-pod-egress --add-egress-zone=public +firewall-cmd --permanent --policy=k3s-pod-egress --add-rich-rule="rule family=ipv4 source address=10.42.0.0/16 accept" +firewall-cmd --reload +systemctl daemon-reload +systemctl enable --now k3s.service diff --git a/home-k3s/bihar/k3s.service b/home-k3s/bihar/k3s.service new file mode 100644 index 00000000..4ce9f342 --- /dev/null +++ b/home-k3s/bihar/k3s.service @@ -0,0 +1,25 @@ +[Unit] +Description=K3s single-node server on Bihar +Documentation=https://docs.k3s.io/ +Wants=network-online.target tailscaled.service +After=network-online.target tailscaled.service + +[Service] +Type=notify +Environment=PATH=/var/usrlocal/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin +ExecStartPre=/var/usrlocal/libexec/k3s-wait-tailnet +ExecStartPre=-/usr/sbin/modprobe br_netfilter +ExecStartPre=-/usr/sbin/modprobe overlay +ExecStart=/var/usrlocal/bin/k3s server +KillMode=process +Delegate=yes +LimitNOFILE=1048576 +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s + +[Install] +WantedBy=multi-user.target diff --git a/home-k3s/bihar/reset-k3s.sh b/home-k3s/bihar/reset-k3s.sh new file mode 100644 index 00000000..958674ea --- /dev/null +++ b/home-k3s/bihar/reset-k3s.sh @@ -0,0 +1,13 @@ +#!/bin/bash +set -x +systemctl stop k3s +pgrep -f '^/var/lib/rancher/k3s/data/.*/containerd-shim' | xargs -r kill -9 +sleep 2 +awk '$2 ~ "^/(run/k3s|var/lib/kubelet|var/lib/rancher/k3s)" {print $2}' /proc/self/mounts | sort -r | while read -r m; do umount -l "$m"; done +ip link del cni0 2>/dev/null; ip link del flannel.1 2>/dev/null +iptables-save | grep -vE "KUBE-|CNI-|flannel" | iptables-restore +ip6tables-save | grep -vE "KUBE-|CNI-|flannel" | ip6tables-restore +rm -rf /var/lib/rancher/k3s /var/lib/kubelet /run/k3s /run/flannel /var/lib/cni /etc/rancher/k3s/k3s.yaml +install -m 0600 /home/james/home-k3s/config.yaml /etc/rancher/k3s/config.yaml +systemctl start k3s +systemctl is-active k3s diff --git a/home-k3s/bihar/resolv.conf b/home-k3s/bihar/resolv.conf new file mode 100644 index 00000000..2c38d459 --- /dev/null +++ b/home-k3s/bihar/resolv.conf @@ -0,0 +1,4 @@ +# Public DNS for Kubernetes pods; host Tailscale MagicDNS is not reachable +# from the pod network. Host DNS configuration remains unchanged. +nameserver 1.1.1.1 +nameserver 1.0.0.1 diff --git a/home-k3s/bihar/wait-tailnet.sh b/home-k3s/bihar/wait-tailnet.sh new file mode 100755 index 00000000..1343ac12 --- /dev/null +++ b/home-k3s/bihar/wait-tailnet.sh @@ -0,0 +1,11 @@ +#!/bin/sh +# The K3s API binds to this address; tailscaled can start before it exists. +set -u +for _ in $(seq 1 60); do + if ip -4 -o addr show dev tailscale0 2>/dev/null | grep -Fq 'inet 100.103.44.48/32 '; then + exit 0 + fi + sleep 1 +done +echo 'K3s cannot start: expected Tailscale address is not present' >&2 +exit 1 diff --git a/home-k3s/karnataka/amdgpu-device-plugin.yaml b/home-k3s/karnataka/amdgpu-device-plugin.yaml new file mode 100644 index 00000000..d9483da4 --- /dev/null +++ b/home-k3s/karnataka/amdgpu-device-plugin.yaml @@ -0,0 +1,46 @@ +# ROCm k8s-device-plugin v1.31.0.11 (upstream k8s-ds-amdgpu-dp.yaml), image pinned by digest. +# Advertises the Strix Halo iGPU as amd.com/gpu so GPU pods are scheduled, not hand-wired to /dev. +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: amdgpu-device-plugin-daemonset + namespace: kube-system +spec: + selector: + matchLabels: + name: amdgpu-dp-ds + template: + metadata: + labels: + name: amdgpu-dp-ds + spec: + restartPolicy: Always + nodeSelector: + kubernetes.io/arch: amd64 + priorityClassName: system-node-critical + tolerations: + - key: CriticalAddonsOnly + operator: Exists + containers: + - image: docker.io/rocm/k8s-device-plugin@sha256:e4df5dc9a7fa34e2344852256dcc5762171a6d68f1f9a34026ce26786ae335e2 + name: amdgpu-dp-cntr + securityContext: + privileged: true + capabilities: + drop: ["ALL"] + resources: + requests: + cpu: 10m + memory: 32Mi + volumeMounts: + - name: dp + mountPath: /var/lib/kubelet/device-plugins + - name: sys + mountPath: /sys + volumes: + - name: dp + hostPath: + path: /var/lib/kubelet/device-plugins + - name: sys + hostPath: + path: /sys diff --git a/home-k3s/karnataka/apply-stack.sh b/home-k3s/karnataka/apply-stack.sh new file mode 100755 index 00000000..a02a4f2e --- /dev/null +++ b/home-k3s/karnataka/apply-stack.sh @@ -0,0 +1,30 @@ +#!/usr/bin/env bash +# Run as root on Karnataka, against this cluster only. Upstream asset digests +# are from the GitHub release metadata; the Corral image is pinned by digest. +set -euo pipefail +cd "$(dirname "$0")" +[ "$(id -u)" = 0 ] || { echo 'Run as root' >&2; exit 1; } +[ "$(hostname -s)" = karnataka ] || { echo 'Wrong host' >&2; exit 1; } +k=(/var/usrlocal/bin/k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml --server https://100.66.158.127:6443) +tmp=$(mktemp -d) +trap 'rm -rf "$tmp"' EXIT +fetch() { + local name=$1 hash=$2 url=$3 + curl -fsSL --retry 3 "$url" -o "$tmp/$name" + printf '%s %s\n' "$hash" "$tmp/$name" | sha256sum -c - +} +fetch kubevirt-operator.yaml 9ac45a436c49d32e6fc21d542bcb7600b970705297b9e6752f3ef9b0a1d292da https://github.com/kubevirt/kubevirt/releases/download/v1.9.1/kubevirt-operator.yaml +fetch kubevirt-cr.yaml 43106136dbce3312bdbfdeae612aacafc6c12da518d233f90645b4685d84a2af https://github.com/kubevirt/kubevirt/releases/download/v1.9.1/kubevirt-cr.yaml +fetch cdi-operator.yaml c7d92bd1b2ee1a3952a5990000437c08d58109f31c0bd48ab6188ec27a13e224 https://github.com/kubevirt/containerized-data-importer/releases/download/v1.66.1/cdi-operator.yaml +fetch cdi-cr.yaml a497f90de608c1df26f9ee4095289373f17132d74a2042ca03e00c17c964f8a7 https://github.com/kubevirt/containerized-data-importer/releases/download/v1.66.1/cdi-cr.yaml +"${k[@]}" apply -f "$tmp/kubevirt-operator.yaml" +"${k[@]}" wait --for=condition=Established --timeout=120s crd/kubevirts.kubevirt.io +"${k[@]}" apply -f "$tmp/kubevirt-cr.yaml" +"${k[@]}" apply -f "$tmp/cdi-operator.yaml" +"${k[@]}" wait --for=condition=Established --timeout=120s crd/cdis.cdi.kubevirt.io +"${k[@]}" apply -f "$tmp/cdi-cr.yaml" +"${k[@]}" wait cdi/cdi --for=condition=Available --timeout=10m +"${k[@]}" patch storageprofile/local-path --type=merge --patch-file local-path-profile.yaml +"${k[@]}" apply -f corral-web.yaml +"${k[@]}" apply -f amdgpu-device-plugin.yaml +"${k[@]}" apply -f lemonade.yaml diff --git a/home-k3s/karnataka/config.yaml b/home-k3s/karnataka/config.yaml new file mode 100644 index 00000000..6801b3ce --- /dev/null +++ b/home-k3s/karnataka/config.yaml @@ -0,0 +1,12 @@ +# Karnataka's existing tailnet address. K3s API and kubelet do not bind the public NIC. +bind-address: 100.66.158.127 +advertise-address: 100.66.158.127 +node-ip: 100.66.158.127 +flannel-iface: tailscale0 +resolv-conf: /etc/rancher/k3s/resolv.conf +kubelet-arg: + - address=100.66.158.127 +disable: + - traefik + - servicelb +write-kubeconfig-mode: "0600" diff --git a/home-k3s/karnataka/corral-web.yaml b/home-k3s/karnataka/corral-web.yaml new file mode 100644 index 00000000..8ba80331 --- /dev/null +++ b/home-k3s/karnataka/corral-web.yaml @@ -0,0 +1,222 @@ +# Based on tuna-os/corral commit 8cff2423a2ba6704348ef225e2ab66438112411b deploy/corral-web.yaml. +# Ingress intentionally omitted; access through an authenticated SSH port-forward only. +# Corral web UI — Proxmox-style dashboard for KubeVirt, served in-cluster. +# Image: built from the corral repo Containerfile, pushed to +# ghcr.io/tuna-os/corral (public — no pull secret needed). +--- +apiVersion: v1 +kind: Namespace +metadata: + name: corral + labels: + pod-security.kubernetes.io/enforce: privileged +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: corral-web + namespace: corral +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: corral-web +rules: + - apiGroups: ["kubevirt.io"] + resources: ["virtualmachines", "virtualmachineinstances"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] + - apiGroups: ["subresources.kubevirt.io"] + resources: + - virtualmachines/start + - virtualmachines/stop + - virtualmachines/restart + - virtualmachines/migrate + - virtualmachines/addvolume + - virtualmachines/removevolume + - virtualmachineinstances/vnc + - virtualmachineinstances/console + - virtualmachineinstances/pause + - virtualmachineinstances/unpause + - virtualmachineinstances/guestosinfo + - virtualmachineinstances/filesystemlist + - virtualmachineinstances/userlist + verbs: ["get", "update"] + # Snapshot / restore / clone / export (Proxmox-parity ops) + - apiGroups: ["snapshot.kubevirt.io"] + resources: ["virtualmachinesnapshots", "virtualmachinerestores"] + verbs: ["get", "list", "create", "delete"] + - apiGroups: ["clone.kubevirt.io"] + resources: ["virtualmachineclones"] + verbs: ["get", "list", "create", "delete"] + - apiGroups: ["export.kubevirt.io"] + resources: ["virtualmachineexports"] + verbs: ["get", "list", "create", "delete"] + - apiGroups: ["cdi.kubevirt.io"] + resources: ["datavolumes"] + verbs: ["get", "list", "create", "delete"] + # PVC: create (new disks), patch (online expand), delete (cleanup) + - apiGroups: [""] + resources: ["persistentvolumeclaims"] + verbs: ["get", "list", "create", "patch", "update", "delete"] + # Capability detection (storage classes + snapshot classes) + - apiGroups: ["storage.k8s.io"] + resources: ["storageclasses"] + verbs: ["get", "list"] + - apiGroups: ["snapshot.storage.k8s.io"] + resources: ["volumesnapshotclasses", "volumesnapshots"] + verbs: ["get", "list"] + # Export download: virtctl vmexport --port-forward needs the token secret + + # a port-forward to the exporter pod. + - apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "create", "delete"] + # User-defined custom image/ISO sources persist in a ConfigMap. + - apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "list", "create", "update", "patch"] + - apiGroups: [""] + resources: ["pods"] + verbs: ["get", "list"] + # Bootc builder reads pod logs to extract build variables. + - apiGroups: [""] + resources: ["pods/log"] + verbs: ["get"] + - apiGroups: [""] + resources: ["pods/portforward"] + verbs: ["create"] + - apiGroups: [""] + resources: ["namespaces"] + verbs: ["get", "list", "create", "patch"] + - apiGroups: [""] + resources: ["nodes"] + verbs: ["get", "list"] + # VDI claims (#226, #227): one Lease per desktop pool member. The opt-in + # broker (CORRAL_VDI_BROKER) claims, renews, and releases them. + - apiGroups: ["coordination.k8s.io"] + resources: ["leases"] + verbs: ["get", "list", "create", "update", "delete"] + # Events tab (Proxmox-style task log) + live metrics on the summary + - apiGroups: [""] + resources: ["events"] + verbs: ["get", "list"] + - apiGroups: ["metrics.k8s.io"] + resources: ["pods", "nodes"] + verbs: ["get", "list"] + # Instancetype/preference pickers in the create wizard + - apiGroups: ["instancetype.kubevirt.io"] + resources: ["virtualmachineclusterinstancetypes", "virtualmachineclusterpreferences"] + verbs: ["get", "list"] + # Bootc builder creates jobs to build VM disks on-cluster. + - apiGroups: ["batch"] + resources: ["jobs"] + verbs: ["get", "list", "create", "delete"] + # Tailscale operator detection (ApplyProxy): tailscaleOperatorPresent() checks + # for the `tailscale` IngressClass the operator registers. + - apiGroups: ["networking.k8s.io"] + resources: ["ingressclasses"] + verbs: ["get", "list"] + # Per-VM Tailscale operator proxy (ApplyProxy): create the SA/Role/RoleBinding/ + # Service/Deployment that expose a VM on the tailnet; VM deletion cleans them up. + - apiGroups: ["apps"] + resources: ["deployments"] + verbs: ["get", "list", "create", "update", "patch", "delete"] + - apiGroups: [""] + resources: ["services", "serviceaccounts"] + verbs: ["get", "list", "create", "update", "patch", "delete"] + # Per-VM proxy + schedule RBAC (ApplyProxy/DeleteProxy, cronops): the only + # Role/RoleBinding objects corral-web ever creates are namespaced ones named + # corral--proxy or corral-sched — never ClusterRoles or ClusterRoleBindings, + # and the proxy Role it grants holds only vmi get + vnc get (see + # GenerateProxyRBAC). Cluster scope (rather than a Role in the corral + # namespace) is required because VM namespaces are dynamic per create (-n); + # there is no static namespace list to bind to. `list` is deliberately + # absent: no code path lists roles/rolebindings (apply + get + delete only). + - apiGroups: ["rbac.authorization.k8s.io"] + resources: ["roles", "rolebindings"] + verbs: ["get", "create", "update", "patch", "delete"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: corral-web +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: corral-web +subjects: + - kind: ServiceAccount + name: corral-web + namespace: corral +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: corral-web + namespace: corral + labels: + app: corral-web +spec: + replicas: 1 + selector: + matchLabels: + app: corral-web + template: + metadata: + labels: + app: corral-web + spec: + serviceAccountName: corral-web + containers: + - name: corral-web + image: ghcr.io/tuna-os/corral@sha256:b0dcfc6972a050b2ac48f00c192844f1c59bb061591839cf4c330e8136c3880f + imagePullPolicy: IfNotPresent + # --metrics starts the background fleet collector and serves + # Prometheus text exposition on /metrics (ADR-0011, pkg/web/prometheus.go). + # Without it the endpoint exists but always renders corral_collection_success 0: + # the collector that would populate it never runs. + args: ["--metrics"] + env: + # This deployment manages its Kubernetes cluster by default. Other + # configured contexts remain visible in the aggregated inventory. + - name: CORRAL_DEFAULT_BACKEND + value: kubevirt + # Default VM namespace (code default is corral-vms). + # Override with CORRAL_NAMESPACE=custom-ns if needed. + - name: CORRAL_NAMESPACE + value: corral-vms + # The UI is open inside the cluster. Only trusted cluster workloads + # and SSH administrators may access it; do not expose this service. + ports: + - containerPort: 8006 + name: http + livenessProbe: + httpGet: + path: /healthz + port: 8006 + initialDelaySeconds: 2 + periodSeconds: 10 + readinessProbe: + httpGet: + path: /readyz + port: 8006 + initialDelaySeconds: 2 + periodSeconds: 5 + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + memory: 256Mi +--- +apiVersion: v1 +kind: Service +metadata: + name: corral-web + namespace: corral +spec: + selector: + app: corral-web + ports: + - name: http + port: 80 + targetPort: 8006 diff --git a/home-k3s/karnataka/frappe-lms.yaml b/home-k3s/karnataka/frappe-lms.yaml new file mode 100644 index 00000000..61c01067 --- /dev/null +++ b/home-k3s/karnataka/frappe-lms.yaml @@ -0,0 +1,279 @@ +# Frappe Learning (LMS) for the courses in ~/courses. +# Image: localhost/frappe-lms:v16.1.0-, built by frappe-lms-image/build.sh: frappe_docker +# images/layered with payments version-16 and lms v16.1.0 plus frappe-lms-image/patches/*.patch +# (prominent audio player), imported into k3s containerd. ghcr.io/frappe/lms:v16.1.0 is unusable: upstream +# passes apps.json as a build-arg, but frappe_docker now reads it from a build secret, so it ships frappe alone. +# Secrets come from Secret/lms-secrets (db-root-password, admin-password), created out of band. +# Exposed on the tailnet by the host: tailscale serve --bg --https=8443 http://:8080 +--- +apiVersion: v1 +kind: Namespace +metadata: + name: lms +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: lms-sites + namespace: lms +spec: + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 20Gi +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: lms-db + namespace: lms +spec: + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 5Gi +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: db + namespace: lms +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app: lms-db + template: + metadata: + labels: + app: lms-db + spec: + containers: + - name: mariadb + image: docker.io/library/mariadb:11.8 + args: + - --character-set-server=utf8mb4 + - --collation-server=utf8mb4_unicode_ci + - --skip-character-set-client-handshake + - --innodb-snapshot-isolation=OFF + env: + - name: MARIADB_ROOT_PASSWORD + valueFrom: + secretKeyRef: + name: lms-secrets + key: db-root-password + ports: + - containerPort: 3306 + readinessProbe: + exec: + command: [healthcheck.sh, --connect, --innodb_initialized] + periodSeconds: 5 + volumeMounts: + - name: data + mountPath: /var/lib/mysql + resources: + requests: + memory: 256Mi + limits: + memory: 1Gi + volumes: + - name: data + persistentVolumeClaim: + claimName: lms-db +--- +apiVersion: v1 +kind: Service +metadata: + name: db + namespace: lms +spec: + selector: + app: lms-db + ports: + - port: 3306 +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: redis + namespace: lms +spec: + replicas: 1 + selector: + matchLabels: + app: lms-redis + template: + metadata: + labels: + app: lms-redis + spec: + containers: + - name: redis + image: docker.io/library/redis:6.2-alpine + ports: + - containerPort: 6379 + resources: + limits: + memory: 256Mi +--- +apiVersion: v1 +kind: Service +metadata: + name: redis + namespace: lms +spec: + selector: + app: lms-redis + ports: + - port: 6379 +--- +# All bench processes share the sites volume, so they run as one pod and talk over localhost. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: frappe + namespace: lms +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app: lms-frappe + template: + metadata: + labels: + app: lms-frappe + spec: + securityContext: + fsGroup: 1000 + initContainers: + - name: configure + image: localhost/frappe-lms:v16.1.0-b358f31a + imagePullPolicy: Never + # args, not command: the image ENTRYPOINT links baked assets into the sites volume. + args: + - bash + - -c + - | + set -e + ls -1 apps > sites/apps.txt + [ -f sites/common_site_config.json ] || echo '{}' > sites/common_site_config.json + bench set-config -g db_host db + bench set-config -gp db_port 3306 + bench set-config -g redis_cache redis://redis:6379/0 + bench set-config -g redis_queue redis://redis:6379/1 + bench set-config -g redis_socketio redis://redis:6379/1 + bench set-config -gp socketio_port 9000 + wait-for-it -t 300 db:3306 + wait-for-it -t 120 redis:6379 + if [ ! -d sites/learn ]; then + bench new-site learn --mariadb-user-host-login-scope='%' \ + --db-root-username root --db-root-password "$DB_ROOT_PASSWORD" \ + --admin-password "$ADMIN_PASSWORD" \ + --install-app payments --install-app lms --set-default + fi + bench --site learn set-config host_name "$HOST_NAME" + bench --site learn list-apps | grep -q '^lms ' || bench --site learn install-app payments lms + bench --site learn set-config -p max_file_size 209715200 + bench --site learn migrate + env: + - name: HOST_NAME + value: https://karnataka.manatee-basking.ts.net:8443 + - name: DB_ROOT_PASSWORD + valueFrom: + secretKeyRef: + name: lms-secrets + key: db-root-password + - name: ADMIN_PASSWORD + valueFrom: + secretKeyRef: + name: lms-secrets + key: admin-password + volumeMounts: + - name: sites + mountPath: /home/frappe/frappe-bench/sites + containers: + - name: backend + image: localhost/frappe-lms:v16.1.0-b358f31a + imagePullPolicy: Never + ports: + - containerPort: 8000 + readinessProbe: + tcpSocket: + port: 8000 + periodSeconds: 10 + volumeMounts: + - name: sites + mountPath: /home/frappe/frappe-bench/sites + resources: + requests: + memory: 512Mi + limits: + memory: 2Gi + - name: frontend + image: localhost/frappe-lms:v16.1.0-b358f31a + imagePullPolicy: Never + args: [nginx-entrypoint.sh] + env: + - name: BACKEND + value: 127.0.0.1:8000 + - name: SOCKETIO + value: 127.0.0.1:9000 + - name: FRAPPE_SITE_NAME_HEADER + value: learn + - name: UPSTREAM_REAL_IP_ADDRESS + value: 127.0.0.1 + - name: UPSTREAM_REAL_IP_HEADER + value: X-Forwarded-For + - name: UPSTREAM_REAL_IP_RECURSIVE + value: "off" + - name: PROXY_READ_TIMEOUT + value: "300" + - name: CLIENT_MAX_BODY_SIZE + value: 200m + ports: + - containerPort: 8080 + volumeMounts: + - name: sites + mountPath: /home/frappe/frappe-bench/sites + - name: websocket + image: localhost/frappe-lms:v16.1.0-b358f31a + imagePullPolicy: Never + args: [node, /home/frappe/frappe-bench/apps/frappe/socketio.js] + volumeMounts: + - name: sites + mountPath: /home/frappe/frappe-bench/sites + - name: worker + image: localhost/frappe-lms:v16.1.0-b358f31a + imagePullPolicy: Never + args: [bench, worker, --queue, "long,default,short"] + volumeMounts: + - name: sites + mountPath: /home/frappe/frappe-bench/sites + - name: scheduler + image: localhost/frappe-lms:v16.1.0-b358f31a + imagePullPolicy: Never + args: [bench, schedule] + volumeMounts: + - name: sites + mountPath: /home/frappe/frappe-bench/sites + volumes: + - name: sites + persistentVolumeClaim: + claimName: lms-sites +--- +apiVersion: v1 +kind: Service +metadata: + name: lms + namespace: lms +spec: + selector: + app: lms-frappe + ports: + - name: http + port: 8080 + targetPort: 8080 diff --git a/home-k3s/karnataka/front-door.yaml b/home-k3s/karnataka/front-door.yaml new file mode 100644 index 00000000..aa0774be --- /dev/null +++ b/home-k3s/karnataka/front-door.yaml @@ -0,0 +1,98 @@ +# Front door for short tailnet URLs: http://karnataka/ redirects to each app's own HTTPS origin. +# Apps keep their own origins because Frappe and RaveBSD both assume they own "/" (/api, /assets, ...), +# so neither can be proxied under a path prefix. +# Exposed by the host: tailscale serve --bg --http=80 http://:80 +--- +apiVersion: v1 +kind: Namespace +metadata: + name: front-door +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: caddyfile + namespace: front-door +data: + Caddyfile: | + { + admin off + auto_https off + } + :80 { + @course path /course /course/* + redir @course https://karnataka.manatee-basking.ts.net:8443/lms/courses/a-new-testament-biblical-theology-g-k-beale-nt575 302 + @learn path /learn /learn/* /lms /lms/* /frappe/learning /frappe/learning/* + redir @learn https://karnataka.manatee-basking.ts.net:8443/lms/courses 302 + @frappe path /frappe /frappe/* + redir @frappe https://karnataka.manatee-basking.ts.net:8443/app 302 + @books path /books /books/* /ravebsd /ravebsd/* + redir @books https://karnataka.manatee-basking.ts.net:8444/ 302 + @corral path /corral /corral/* + redir @corral https://karnataka.manatee-basking.ts.net/ 302 + handle { + header Content-Type "text/html; charset=utf-8" + respond < + karnataka + +

karnataka

+ /course: Beale NT Biblical Theology + /frappe/learning: all courses (also /learn) + /frappe: Frappe admin desk + /books: RaveBSD + /corral: Corral + HTML 200 + } + } +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: caddy + namespace: front-door +spec: + replicas: 1 + selector: + matchLabels: + app: front-door + template: + metadata: + labels: + app: front-door + annotations: + # Bump to roll the pod after editing the Caddyfile. + front-door/config-rev: "1" + spec: + containers: + - name: caddy + image: docker.io/library/caddy:2.11.9-alpine@sha256:78bd35025c7ba85961dd149ae670315129954e77abe2383ac47c89106b35ade9 + ports: + - containerPort: 80 + readinessProbe: + tcpSocket: + port: 80 + volumeMounts: + - name: config + mountPath: /etc/caddy + resources: + requests: + memory: 16Mi + limits: + memory: 64Mi + volumes: + - name: config + configMap: + name: caddyfile +--- +apiVersion: v1 +kind: Service +metadata: + name: front-door + namespace: front-door +spec: + selector: + app: front-door + ports: + - port: 80 + targetPort: 80 diff --git a/home-k3s/karnataka/install-host.sh b/home-k3s/karnataka/install-host.sh new file mode 100755 index 00000000..9c99d1ba --- /dev/null +++ b/home-k3s/karnataka/install-host.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +# Run as root on Karnataka. Leaves Utah, host firewall and Tailscale identity intact. +# K3s state stays in its default writable /var/lib/rancher/k3s. The upstream +# SELinux module cannot be added to Utah's read-only policy store at runtime; +# K3s SELinux container labeling is not enabled. Host SELinux stays enforcing. +set -euo pipefail +cd "$(dirname "$0")" +[ "$(id -u)" = 0 ] || { echo 'Run as root' >&2; exit 1; } +[ "$(hostname -s)" = karnataka ] || { echo 'Wrong host' >&2; exit 1; } +ip -4 addr show dev tailscale0 | grep -q '100\.66\.158\.127/32' || { echo 'Expected Tailscale address is absent' >&2; exit 1; } +tmp=$(mktemp) +trap 'rm -f "$tmp"' EXIT +curl -fL --retry 3 'https://github.com/k3s-io/k3s/releases/download/v1.36.5%2Bk3s1/k3s' -o "$tmp" +printf '%s %s\n' d73847bcd3c5fccef0115b372e2f9a91f3032dc84bbf71518a4617565294d313 "$tmp" | sha256sum -c - +install -D -m 0755 "$tmp" /var/usrlocal/bin/k3s +install -D -m 0600 config.yaml /etc/rancher/k3s/config.yaml +install -D -m 0644 k3s.service /etc/systemd/system/k3s.service +install -D -m 0644 resolv.conf /etc/rancher/k3s/resolv.conf +systemctl daemon-reload +systemctl enable --now k3s.service diff --git a/home-k3s/karnataka/k3s.service b/home-k3s/karnataka/k3s.service new file mode 100644 index 00000000..eef18552 --- /dev/null +++ b/home-k3s/karnataka/k3s.service @@ -0,0 +1,25 @@ +[Unit] +Description=K3s single-node server on Karnataka +Documentation=https://docs.k3s.io/ +Wants=network-online.target tailscaled.service +After=network-online.target tailscaled.service + +[Service] +Type=notify +Environment=PATH=/var/usrlocal/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin +ExecStartPre=/var/usrlocal/libexec/k3s-wait-tailnet +ExecStartPre=-/usr/sbin/modprobe br_netfilter +ExecStartPre=-/usr/sbin/modprobe overlay +ExecStart=/var/usrlocal/bin/k3s server +KillMode=process +Delegate=yes +LimitNOFILE=1048576 +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s + +[Install] +WantedBy=multi-user.target diff --git a/home-k3s/karnataka/lemonade.yaml b/home-k3s/karnataka/lemonade.yaml new file mode 100644 index 00000000..4e3d031f --- /dev/null +++ b/home-k3s/karnataka/lemonade.yaml @@ -0,0 +1,103 @@ +# Lemonade coding model server (llama.cpp on ROCm), formerly the lemonade.container Quadlet. +# Scaled to 0 while unused. To run it: set replicas: 1 here and re-apply. +# +# Memory: on this APU, model weights + KV cache live in amdgpu GTT, which is carved from system +# RAM but NOT charged to the pod's memory cgroup. Observed with Qwen3.8-Flash-Next IQ2_XS at +# ctx 32768: ~38Gi GTT + ~30Gi process RSS. The request covers both so the scheduler refuses to +# start lemonade unless that RAM is actually free next to the VMs; the limit only bounds the +# cgroup-charged part (same 80G cap the Quadlet had). +# +# Exposed on the tailnet by the host's `tailscale serve` (karnataka:13305 -> 10.43.133.5:13305). +apiVersion: v1 +kind: Namespace +metadata: + name: lemonade +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: lemonade + namespace: lemonade + labels: + app: lemonade +spec: + replicas: 0 + strategy: + type: Recreate + selector: + matchLabels: + app: lemonade + template: + metadata: + labels: + app: lemonade + spec: + nodeSelector: + kubernetes.io/hostname: karnataka + securityContext: + runAsUser: 10001 + runAsGroup: 999 + # render, video: matches the host's /dev/kfd and /dev/dri group ownership. + supplementalGroups: [105, 39] + terminationGracePeriodSeconds: 60 + containers: + - name: lemonade + image: ghcr.io/lemonade-sdk/lemonade-server@sha256:8ae593e309c5f8f348ea7404f20e27db83d0d247cc9ba3b2bf7fb6310e13e73c + workingDir: /opt/lemonade + command: ["./lemond", "--host", "0.0.0.0", "--no-broadcast"] + ports: + - name: http + containerPort: 13305 + resources: + requests: + cpu: "4" + memory: 72Gi + limits: + memory: 80Gi + amd.com/gpu: 1 + readinessProbe: + httpGet: + path: /api/v1/health + port: http + periodSeconds: 10 + volumeMounts: + - name: hf-cache + mountPath: /opt/lemonade/.cache/huggingface + - name: data + mountPath: /opt/lemonade/.cache/lemonade + - name: llama + mountPath: /opt/lemonade/llama + - name: config + mountPath: /opt/lemonade/.config/lemonade + volumes: + - name: hf-cache + hostPath: + path: /var/lib/tuna-ci-scratch/lemonade-cache + type: Directory + - name: data + hostPath: + path: /var/lib/tuna-ci-scratch/lemonade-data + type: Directory + - name: llama + hostPath: + path: /var/lib/tuna-ci-scratch/lemonade-llama + type: Directory + - name: config + hostPath: + path: /var/lib/tuna-ci-scratch/lemonade-config + type: Directory +--- +apiVersion: v1 +kind: Service +metadata: + name: lemonade + namespace: lemonade +spec: + # Pinned: the host's `tailscale serve` TCP forward targets this address. + clusterIP: 10.43.133.5 + selector: + app: lemonade + ports: + - name: http + port: 13305 + targetPort: http diff --git a/home-k3s/karnataka/local-path-profile.yaml b/home-k3s/karnataka/local-path-profile.yaml new file mode 100644 index 00000000..c60d6410 --- /dev/null +++ b/home-k3s/karnataka/local-path-profile.yaml @@ -0,0 +1,7 @@ +# CDI has no built-in profile for rancher.io/local-path. This provisioner +# supports filesystem RWO, not block volumes, RWX, expansion, or snapshots. +spec: + claimPropertySets: + - accessModes: [ReadWriteOnce] + volumeMode: Filesystem + cloneStrategy: copy diff --git a/home-k3s/karnataka/resolv.conf b/home-k3s/karnataka/resolv.conf new file mode 100644 index 00000000..2c38d459 --- /dev/null +++ b/home-k3s/karnataka/resolv.conf @@ -0,0 +1,4 @@ +# Public DNS for Kubernetes pods; host Tailscale MagicDNS is not reachable +# from the pod network. Host DNS configuration remains unchanged. +nameserver 1.1.1.1 +nameserver 1.0.0.1 diff --git a/home-k3s/karnataka/wait-tailnet.sh b/home-k3s/karnataka/wait-tailnet.sh new file mode 100755 index 00000000..5067e188 --- /dev/null +++ b/home-k3s/karnataka/wait-tailnet.sh @@ -0,0 +1,11 @@ +#!/bin/sh +# The K3s API binds to this address; tailscaled can start before it exists. +set -u +for _ in $(seq 1 60); do + if ip -4 -o addr show dev tailscale0 2>/dev/null | grep -Fq 'inet 100.66.158.127/32 '; then + exit 0 + fi + sleep 1 +done +echo 'K3s cannot start: expected Tailscale address is not present' >&2 +exit 1 diff --git a/inventory.yml b/inventory.yml index ab322567..f7784915 100644 --- a/inventory.yml +++ b/inventory.yml @@ -3,9 +3,11 @@ all: hosts: # `tailnet_name`: the host's MagicDNS name when it isn't the inventory # name; scripts/online_hosts.py matches online peers by it. + # punjab is a KubeVirt VM on karnataka since 2026-10-27 (corral-vms/punjab). + # The old EC2 instance is tailnet node punjab-aws-retired (100.78.73.8). punjab: - ansible_host: 100.78.73.8 - ansible_user: ubuntu # stock Ubuntu AMI login; there is no `james` here + ansible_host: 100.103.187.46 + ansible_user: ubuntu # Ubuntu 24.04 image; there is no `james` here ansible_connection: ssh # This targets the Debian VM running via Android's Virtualization # Framework on the phone (tailnet node name "mumbai"), not the raw @@ -35,9 +37,15 @@ all: vm: ansible_host: localhost ansible_connection: local - # ── Talos K8s cluster (not ansible-managed — talosctl / kubectl only) ── - # karnataka: Talos worker — AMD Strix Halo APU, 192.168.0.6 - # bihar: Talos control-plane — Intel, 192.168.0.5 + # ── Home servers (Bluefin + k3s since 2026-10/11; the Talos cluster is archived) ── + # bihar: active hive host + KubeStellar hub. Bootstrapped with `just apply` + # on 2026-11-17 (desktop group, as bootstrap.sh registered it). + bihar: + ansible_host: 100.103.44.48 + ansible_user: james + ansible_connection: ssh + # karnataka: k3s/KubeVirt host (punjab runs on it). Configured from the + # feat/native-laptop-units branch (hanthor/dotfiles#348), not this inventory. matrix: ansible_host: 100.73.19.81 ansible_connection: ssh @@ -68,12 +76,13 @@ all: children: desktop: hosts: + bihar: mumbai: kerala: kanpur: himachal: dilli: - # llm group is dormant — karnataka (former LLM host) is now a Talos worker + # llm group is dormant — karnataka (former LLM host) now runs k3s workloads # llm: # hosts: # karnataka: diff --git a/talos-k8s/README.md b/talos-k8s/README.md index c9461248..280eaff2 100644 --- a/talos-k8s/README.md +++ b/talos-k8s/README.md @@ -1,3 +1,9 @@ +> **ARCHIVED (2026-11-18).** This Talos cluster no longer exists: bihar and +> karnataka now run Bluefin with separate k3s clusters, configured from +> [`home-k3s/`](../home-k3s/). See [Fleet status](../docs/src/fleet.md) and +> [Home k3s and KubeStellar](../docs/src/servers/home-k3s.md). The manifests are kept +> because tests and history refer to them (`talos-k8s/hive/**`); do not apply them. + # Talos Linux 2-Node Kubernetes Cluster with AMD GPU Acceleration This directory contains the complete infrastructure-as-code configurations, manifests, and documentation for a highly custom, high-performance 2-node Kubernetes cluster utilizing **Talos Linux (v1.13.2)**, designed specifically for running hardware-accelerated LLM workloads (via **Lemonade**) and virtualized infrastructure (via **KubeVirt**). From bd8f291930cfb974581bb8ee15ab74118b5a0291 Mon Sep 17 00:00:00 2001 From: James Reilly Date: Wed, 18 Nov 2026 18:39:44 +0000 Subject: [PATCH 2/3] docs(hive): fold in GitHub Apps and Bitwarden recovery items from #359, updated for bihar Co-Authored-By: Claude Opus 5.5 --- docs/src/bitwarden.md | 10 ++++++++++ docs/src/cluster/hive.md | 39 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/docs/src/bitwarden.md b/docs/src/bitwarden.md index 5fa9b1dc..a3b77cc4 100644 --- a/docs/src/bitwarden.md +++ b/docs/src/bitwarden.md @@ -52,6 +52,16 @@ Every item a role or script reads (grep `bw get` / `bw list` under `roles/` and | `bw-api-key` | Login | `scripts/bw-resolve.sh` — `bw login --apikey` on a remote | | `James Bitwarden` | Login | `scripts/bw-resolve.sh` — non-interactive unlock on a remote | +### Recovery items (not read by any role) + +These items are kept so that you can rebuild a lost host by hand. No role reads them. + +| Bitwarden Item | Type | Holds | +|---------------|------|-------| +| `hive-github-app-hanthor-hive-agent` | Secure Note | School's GitHub App: App ID, installation ID, PEM private key (the hub mounts the same key). See [Hive](cluster/hive.md#github-apps) | +| `hive-github-app-tuna-reef-hive` | Secure Note | Reef's GitHub App: App ID, installation ID, PEM private key | +| `hive-spindle-env` | Secure Note | The hives' `.env` (hub secret, dashboard token, omp broker token, provider keys). It was `/srv/hives/.env` on spindle and is now `/var/srv/hives/.env` on bihar and karnataka. The item name has not changed | + ## Running Without Secrets For fast iteration that doesn't need the vault — packages, dotfiles, desktop config — use the explicit no-secrets path: diff --git a/docs/src/cluster/hive.md b/docs/src/cluster/hive.md index 40398cdc..b6902efd 100644 --- a/docs/src/cluster/hive.md +++ b/docs/src/cluster/hive.md @@ -108,6 +108,45 @@ auto-merge Renovate major bumps, which the org forbids. 3. Re-parse both files and check the list lengths and that the new repo shows up on the hive's dashboard; no App installation step is needed. +### GitHub Apps + +Since 2026-10-30 School and Reef use separate GitHub Apps, so each has its own +installation rate limit (6,150 requests/hour across the 43 org repositories). Before +that change they shared one installation and used all of its limit every hour. + +| Hive | App (bot login) | App ID | Installation | Key in Bitwarden | +|---|---|---|---|---| +| school (the hub mounts the same key) | `hanthor-hive-agent` (`hanthor-hive-agent[bot]`) | 3942065 | 137498420 | `hive-github-app-hanthor-hive-agent` | +| reef | `tuna-reef-hive` (`tuna-reef-hive[bot]`) | 5546574 | 177007234 | `hive-github-app-tuna-reef-hive` | + +The IDs occur in three places for each hive: +- `GH_APP_ID` and `GH_APP_INSTALLATION_ID` in `compose.yaml`; +- `github.app_id` and `installation_id` in the seed; +- the same keys in `hive.yaml.runtime`. + +Both hives manage `.github`, `Tavern`, `bluefin-cli`, `corral`, `docs` and +`flatpak-index`. Each hive lists the bot of the other hive in +`github.additional_bot_logins`, so that the issues and PRs of the other hive count as +hive-authored. + +To restore a key: +1. Write the key block of the Bitwarden note to + `/var/srv/hives//secrets/gh-app-key.pem` on the active host (mode 600). +2. Run `chcon -t container_file_t` on the file. +3. Recreate the service with `sudo /var/srv/hives-host/hives up -d --no-deps `. + +The next `hive-sync` copies the key to the standby host. + +To make sure that each hive has its own rate limit, run: + +```bash +ssh james@bihar 'for c in hives-school-1 hives-reef-1; do sudo podman exec $c sh -c \ + "curl -s -H \"Authorization: Bearer \$(cat /var/run/hive-metrics/gh-app-token.cache)\" https://api.github.com/rate_limit" \ + | jq -c .resources.core; done' +``` + +Different `reset` times show separate installations. + ### Safe edit procedure More than one operator edits `/var/srv/hives` (including agents). Re-read the files From df8b9edfed209daa804d80054f874cc35d64236a Mon Sep 17 00:00:00 2001 From: James Reilly Date: Wed, 18 Nov 2026 18:53:47 +0000 Subject: [PATCH 3/3] fix(hives): make the host timers actually run; add hive-cleanup hive-reconcile and hive-sync never ran from their timers: SELinux denies systemd exec of var_t scripts (203/EXEC). Start them through /usr/bin/bash. Under systemd, rsync also moves into rsync_t, where its ssh cannot read the key or ssh_config, so hive-sync.service runs as unconfined_t (setsebool -P does not persist on bootc). Port spindle's hive-cleanup to the podman hosts (hourly): idle Claude scratch and stale /tmp work dirs in running spokes, dangling images, old backups; measures /var. Document all of this and the reduced spindle rollback path (its stale data, arm64 binaries and images were removed). Co-Authored-By: Claude Opus 5.5 --- docs/src/servers/hives.md | 24 +++++---- hives/README.md | 3 +- hives/hive-cleanup | 77 ++++++++++++++++++++++++++++ hives/hive-reconcile | 6 +-- hives/systemd/hive-cleanup.service | 8 +++ hives/systemd/hive-cleanup.timer | 10 ++++ hives/systemd/hive-reconcile.service | 2 +- hives/systemd/hive-sync.service | 5 +- 8 files changed, 119 insertions(+), 16 deletions(-) create mode 100755 hives/hive-cleanup create mode 100644 hives/systemd/hive-cleanup.service create mode 100644 hives/systemd/hive-cleanup.timer diff --git a/docs/src/servers/hives.md b/docs/src/servers/hives.md index 8237c642..90cf771b 100644 --- a/docs/src/servers/hives.md +++ b/docs/src/servers/hives.md @@ -108,6 +108,7 @@ The file `spindle:/var/lib/hive-failover/active` contains `bihar`, `karnataka` o | `hive-failover check` | spindle, `hive-failover.timer` | every 1 min | Checks the active host on ports 3101-3103. After 5 min of failures it sets the lease to `none`, stops the old host if it can, and waits. When the old host has 0 containers, or after 10 min, it starts the standby, sets the lease, changes Caddy and sends a ntfy notification on `$NTFY_TOPIC`. | | `hive-reconcile` | both hosts, `hive-reconcile.timer` | every 1 min | If the lease names this host and fewer than 4 containers run, it starts them (unless `maintenance` exists). If the lease names a different host, it stops them. If it cannot read the lease for 8 min, it stops them (self-fence). | | `hive-sync` | both hosts, `hive-sync.timer` | every 15 min | If the lease names this host, it uses rsync to push `/var/srv/hives` to the peer, without caches and toolchains. | +| `hive-cleanup` | both hosts, `hive-cleanup.timer` | every 1 h | On running spokes only: removes idle Claude session scratch and stale work dirs in `/tmp` inside the containers, removes dangling images and old backups, and warns when `/var` has less than 10 GB free. Ported from spindle, where the writable layers of the hive containers had reached 16.8 GB. | Logs: @@ -152,7 +153,11 @@ Use one of these methods: 4. `move bihar` completed in 50 s, including the sync back. ## Gotchas -- **SELinux (bihar and karnataka are bootc and enforcing):** files mounted into containers must have the type `container_file_t`. `semanage` cannot write on bootc, so `hive-sync` runs `chcon -R` after each copy. Do not copy SELinux attributes with rsync (`-X`). +- **SELinux (bihar and karnataka are bootc and enforcing):** + - Files mounted into containers must have the type `container_file_t`. `semanage` cannot write on bootc, so `hive-sync` runs `chcon -R` after each copy. Do not copy SELinux attributes with rsync (`-X`). + - systemd cannot run scripts that have the label `var_t` (the result is `203/EXEC Permission denied`). For this reason the units start the scripts as `/usr/bin/bash /var/srv/hives-host/