diff --git a/docs/src/automation.md b/docs/src/automation.md index d6e6f621..4bfae220 100644 --- a/docs/src/automation.md +++ b/docs/src/automation.md @@ -50,14 +50,14 @@ manifest on fleet machines. Each live spoke persists its configuration to `data/hive.yaml.runtime` next to the seed `config/hive.yaml`, and the runtime file wins on restart, so roster or -level changes made in the dashboard stick. The `hive-config` ConfigMap in -[`talos-k8s/hive/spokes/hanthor/manifest.yaml`](https://github.com/hanthor/dotfiles/blob/master/talos-k8s/hive/spokes/hanthor/manifest.yaml) -was the seed of the former `hive.reilly.asia` spoke on the terminated AWS -cluster (exported 2026-10-01); it does not describe a running Hive. For the -current school and reef files, read-only inspection and the safe edit -procedure, see the +level changes made in the dashboard stick. hive.reilly.asia is the `hanthor` +spoke on bihar (rebuilt 2026-11-19); its seed is +[`hives/hanthor/hive.yaml`](https://github.com/hanthor/dotfiles/blob/master/hives/hanthor/hive.yaml). +The AWS-era ConfigMap in `talos-k8s/hive/spokes/hanthor/manifest.yaml` is +history. For the live files, read-only inspection and the safe edit procedure, +see the [Hive operations guide](cluster/hive.md#current-deployment-bihar-karnataka-standby): ```bash -ssh ubuntu@100.113.75.88 'docker ps --filter name=hives-; ls /srv/hives/*/data/hive.yaml.runtime' +ssh james@bihar 'sudo podman ps --filter name=hives-; sudo cat /var/srv/hives/hanthor/data/hive.yaml.runtime' ``` diff --git a/docs/src/cluster/hive.md b/docs/src/cluster/hive.md index b6902efd..065bf5c3 100644 --- a/docs/src/cluster/hive.md +++ b/docs/src/cluster/hive.md @@ -2,13 +2,14 @@ ## Current deployment: bihar (karnataka standby) -Two Hive spokes, **school** and **reef**, and their **hub** run as compose +Three Hive spokes, **school**, **reef** and **hanthor**, and their **hub** run as compose services under rootful podman on [bihar](../servers/bihar.md), in `/var/srv/hives` (`ssh james@bihar`, commands need `sudo`). Until 2026-11-18 they ran on the [Spindle VPS](../servers/spindle-vps/README.md) under `/srv/hives`. Spindle's Caddy still routes [school.tunaos.org](https://school.tunaos.org), -[reef.tunaos.org](https://reef.tunaos.org) and +[reef.tunaos.org](https://reef.tunaos.org), +[hive.reilly.asia](https://hive.reilly.asia) and [hub.tunaos.org](https://hub.tunaos.org), over the tailnet to the active host. [karnataka](../servers/karnataka.md) is a cold standby. A lease on spindle selects the active host. Run every command below on **the host the lease @@ -16,6 +17,21 @@ names** (`ssh spindle sudo hive-failover status`). Hosting, failover and the host-specific compose layer are described in [Hives hosting and failover](../servers/hives.md). +### hanthor (hive.reilly.asia) + +The personal-account spoke from the old AWS cluster was rebuilt on bihar on +2026-11-19. It is the `hanthor` service in `compose.override.yaml` (port 3104), +with its data in `/var/srv/hives/hanthor`. It uses the same GitHub App as +school (`hanthor-hive-agent`, App ID 3942065) through the App's installation on +the hanthor account (158987150), and the same key file. Its seed is +[`hives/hanthor/hive.yaml`](https://github.com/hanthor/dotfiles/blob/master/hives/hanthor/hive.yaml): +ACMM level 5, four agents (supervisor, scanner, quality, guide) on the `claude` +backend, and the repositories `dotfiles`, `reilly.asia`, `homebrew-tap`, +`shrimply` and `indiafoss-chat-android`. Compared with the AWS seed it drops +`indiafoss-companion` (archived), `hummingbird-github` (deleted) and +`rust-wayland-desktop` (now `tuna-os/tuna-desktop`, which school covers). The +AWS data volume was not kept, so the hive started with empty state. + The layout and roster below were read on spindle on 2026-11-13. The move changed the paths and the commands, but not the hive configuration. Container state alone does not prove that agents are healthy. See [Checking health](#checking-health). @@ -29,7 +45,8 @@ Kubernetes deployment and must not be applied here. ```text /var/srv/hives/ # on the active host; the host layer is /var/srv/hives-host/compose..yaml ├── compose.yaml # services: hub, hub-front, school, reef (network spindle_default) -├── compose.override.yaml # pins the running binary/dashboard, memory limits, session pool +├── compose.override.yaml # pins the running binary/dashboard, memory limits, session pool; +│ # also defines the hanthor spoke (added 2026-11-19) ├── .env # secrets and NTFY topic (mode 600; never print or commit) ├── bin/hive- # fork builds; compose.yaml mounts bin/hive, the override re-pins one ├── /config/hive.yaml # seed, mounted read-only at /etc/hive @@ -294,8 +311,7 @@ PRs and issues (`gh search prs --author "hanthor-hive-agent[bot]"` and - The spokes call the `omp-auth-broker` on spindle (`OMP_AUTH_BROKER_URL`, `http://100.113.75.88:8765`). Caddy routing and failover are described in [Hives hosting and failover](../servers/hives.md). This page does not describe - the hub's own configuration. The `hive.reilly.asia` hanthor spoke of the old AWS - cluster has no container on this host; its current status is unverified. The + the hub's own configuration. The [hive_ops role](../roles/hive_ops.md) targets the retired cluster, not this deployment. diff --git a/docs/src/fleet.md b/docs/src/fleet.md index d550eaa6..e319f3be 100644 --- a/docs/src/fleet.md +++ b/docs/src/fleet.md @@ -29,7 +29,7 @@ Internet ─▶ Cloudflare ─▶ spindle (Oracle, Frankfurt): Caddy, Matrix, Li | **pixel-9a** (`termux`) | Phone | Android + Termux | Ansible `termux` host | [Inventory](inventory.md) | Cross-host services: -- The hives (school, reef, hub) are on bihar. If bihar fails, they move to karnataka. See [Hives hosting](servers/hives.md). +- The hives (school, reef, hanthor, hub) are on bihar. If bihar fails, they move to karnataka. See [Hives hosting](servers/hives.md). - KubeStellar is described in [Home k3s and KubeStellar](servers/home-k3s.md). ## Intermittent (desktops and laptops) diff --git a/docs/src/servers/hives.md b/docs/src/servers/hives.md index 90cf771b..b1e533a1 100644 --- a/docs/src/servers/hives.md +++ b/docs/src/servers/hives.md @@ -4,14 +4,14 @@ > For the hives themselves (agents, configuration, health), see the > [Hive operations guide](../cluster/hive.md). -Last updated: 2026-11-18. The source for the scripts, compose layers and systemd +Last updated: 2026-11-19 (hanthor added). The source for the scripts, compose layers and systemd units is [`hives/`](https://github.com/hanthor/dotfiles/tree/master/hives) in this repository. Each host also has a copy of this runbook at `/var/srv/hives-host/RUNBOOK.md` (on spindle: `/var/lib/hive-failover/RUNBOOK.md`). ## Overview -The tuna-os hives run as one compose project named `hives`, with four containers: +The hives run as one compose project named `hives`, with five containers: | Service | Purpose | Public URL | |---|---|---| @@ -19,12 +19,13 @@ The tuna-os hives run as one compose project named `hives`, with four containers | `hub-front` | Static hub page (nginx) | `https://hub.tunaos.org/` | | `school` | Spoke for the tuna-os org (GitHub App 3942065) | `https://school.tunaos.org` | | `reef` | Spoke (GitHub App 5546574) | `https://reef.tunaos.org` | +| `hanthor` | Spoke for the `hanthor/*` personal repositories (App 3942065, installation 158987150). Added 2026-11-19 | `https://hive.reilly.asia` | Until 2026-11-18 the hives ran on spindle (Oracle ARM VM, Frankfurt, 2 cores and 11 GB of RAM). They now run on **bihar** (x86, 24 cores, 62 GB of RAM). **karnataka** is a cold standby. **spindle** keeps the public IP and Caddy, and it coordinates failover. ``` Internet ─▶ Cloudflare ─▶ spindle Caddy (92.5.82.232) - │ tailnet, ports 3100-3103 + │ tailnet, ports 3100-3104 ┌──────────────┴──────────────┐ ▼ (active, holds the lease) ▼ (standby, stopped) bihar 100.103.44.48 karnataka 100.66.158.127 @@ -39,7 +40,7 @@ The OMP auth broker still runs on spindle at `http://100.113.75.88:8765`, reache | Path | Contents | |---|---| -| `/var/srv/hives/` | Hive data and configuration, copied from spindle's `/srv/hives`: `compose.yaml`, `compose.override.yaml`, `.env`, `{hub,school,reef}/{config,data,secrets}`, `hub-front/`, `proxy/`, `bin/dashboard-*.html` | +| `/var/srv/hives/` | Hive data and configuration, copied from spindle's `/srv/hives`: `compose.yaml`, `compose.override.yaml`, `.env`, `{hub,school,reef,hanthor}/{config,data,secrets}`, `hub-front/`, `proxy/`, `bin/dashboard-*.html` | | `/var/srv/hives-host/compose..yaml` | Host layer: ports, memory limits, broker URL, image digests, and the amd64 binary mount | | `/var/srv/hives-host/bin/hive-2ef7dc7c0-amd64` | `tuna-os/hive` at `2ef7dc7c0` (branch `deploy/upstream-v6-20261113`) built with `CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath ./cmd/hive` | | `/var/srv/hives-host/hives` | Wrapper that runs docker-compose against rootful podman with the three compose files | @@ -56,11 +57,12 @@ Each host publishes its ports on its own tailnet IP only: | 3101 | hub | | 3102 | school | | 3103 | reef | +| 3104 | hanthor | A firewalld rich rule in the `public` zone accepts these ports only from spindle (`100.113.75.88`). Inside the compose network the services reach each other by name, for example `http://hub:3001`. The podman network is named `spindle_default` so that `compose.yaml` works unchanged. Images are pinned by digest, and the pin must match the binary: -- `ghcr.io/hivecommons/hive@sha256:d56425fb…` (school and reef) +- `ghcr.io/hivecommons/hive@sha256:d56425fb…` (school, reef and hanthor) - `ghcr.io/hivecommons/hive-hub@sha256:3c0162eb…` (hub) Do not change to `:v6-latest`. On 2026-11-18 a newer image made the hub log "failed to write token cache". **Upgrade the binary and the images together.** @@ -185,7 +187,7 @@ On a new host, also do these steps: 3. Create the podman network `spindle_default`. 4. Pull the pinned images. 5. Copy `bin/hive-2ef7dc7c0-amd64` from the other host. If neither host has it, build it from `tuna-os/hive` at `2ef7dc7c0`. -6. Create the SSH keys in `/var/srv/hives-host/keys/`, and add the firewalld rule for ports 3100-3103 from spindle. +6. Create the SSH keys in `/var/srv/hives-host/keys/`, and add the firewalld rule for ports 3100-3104 from spindle. Also install `/etc/sysctl.d/90-hives-inotify.conf` (`fs.inotify.max_user_instances = 1024`, `fs.inotify.max_user_watches = 1048576`): with five containers the per-agent permission watchers exhaust the default of 128 instances. 7. Label the data and compose files (not the scripts): `chcon -R -t container_file_t /var/srv/hives /var/srv/hives-host/compose.*.yaml /var/srv/hives-host/bin`. 8. Enable `hive-reconcile.timer`, `hive-sync.timer` and `hive-cleanup.timer`. Start each service one time and make sure that its `Result` is `success`. diff --git a/docs/src/servers/spindle-vps/README.md b/docs/src/servers/spindle-vps/README.md index d911959e..cdb2620d 100644 --- a/docs/src/servers/spindle-vps/README.md +++ b/docs/src/servers/spindle-vps/README.md @@ -8,10 +8,10 @@ terminated 2026-10-25. **Spindle**, **Postgres**, standalone **LiveKit** and **l **The hives moved off this host on 2026-11-18.** School, Reef, Hub and the hub frontend now run on [bihar](../bihar.md), with [karnataka](../karnataka.md) as a cold -standby. Spindle keeps three hive roles: +standby. The `hanthor` spoke (`hive.reilly.asia`) joined them on 2026-11-19. Spindle keeps three hive roles: -- **Caddy front door:** `school`, `reef` and `hub.tunaos.org` go to the active host's - tailnet IP on ports 3100-3103. `hive-failover` rewrites these upstreams. +- **Caddy front door:** `school`, `reef`, `hub.tunaos.org` and `hive.reilly.asia` go to the active host's + tailnet IP on ports 3100-3104 (`hive.reilly.asia` on 3104). `hive-failover` rewrites these upstreams. - **Failover coordinator:** the lease in `/var/lib/hive-failover/active`, `/usr/local/sbin/hive-failover`, and `hive-failover.timer`. - **OMP auth broker:** the `omp-auth-broker` user service on `100.113.75.88:8765`. The hives diff --git a/hives/README.md b/hives/README.md index 204f8448..96edbdbd 100644 --- a/hives/README.md +++ b/hives/README.md @@ -13,6 +13,8 @@ the design, the install locations, daily operations, failover and the gotchas. | `compose..yaml` | `/var/srv/hives-host/` on that host | | `hive-cleanup` | `/var/srv/hives-host/` on bihar and karnataka | | `systemd/hive-{reconcile,sync,cleanup}.{service,timer}` | `/etc/systemd/system/` on bihar and karnataka | +| `hanthor/hive.yaml` | seed for the hanthor spoke: `/var/srv/hives/hanthor/config/hive.yaml` (synced to the standby by `hive-sync`) | +| `hanthor/compose.override.snippet.yaml` | the `hanthor` service, appended to `/var/srv/hives/compose.override.yaml` | | `hive-failover` | `/usr/local/sbin/` on spindle | | `systemd/hive-failover.{service,timer}` | `/etc/systemd/system/` on spindle | diff --git a/hives/compose.bihar.yaml b/hives/compose.bihar.yaml index f6d7be67..19458304 100644 --- a/hives/compose.bihar.yaml +++ b/hives/compose.bihar.yaml @@ -35,3 +35,11 @@ services: OMP_AUTH_BROKER_URL: "http://100.113.75.88:8765" volumes: - /var/srv/hives-host/bin/hive-2ef7dc7c0-amd64:/usr/local/bin/hive:ro + hanthor: + image: ghcr.io/hivecommons/hive@sha256:d56425fbaa1b5854a96356d56ebc2f371f337038339c56c94dcf3f2ec29ead51 + ports: ["100.103.44.48:3104:3001"] + mem_limit: 4g + environment: + OMP_AUTH_BROKER_URL: "http://100.113.75.88:8765" + volumes: + - /var/srv/hives-host/bin/hive-2ef7dc7c0-amd64:/usr/local/bin/hive:ro diff --git a/hives/compose.karnataka.yaml b/hives/compose.karnataka.yaml index 1d928b59..e70fcad6 100644 --- a/hives/compose.karnataka.yaml +++ b/hives/compose.karnataka.yaml @@ -35,3 +35,11 @@ services: OMP_AUTH_BROKER_URL: "http://100.113.75.88:8765" volumes: - /var/srv/hives-host/bin/hive-2ef7dc7c0-amd64:/usr/local/bin/hive:ro + hanthor: + image: ghcr.io/hivecommons/hive@sha256:d56425fbaa1b5854a96356d56ebc2f371f337038339c56c94dcf3f2ec29ead51 + ports: ["100.66.158.127:3104:3001"] + mem_limit: 4g + environment: + OMP_AUTH_BROKER_URL: "http://100.113.75.88:8765" + volumes: + - /var/srv/hives-host/bin/hive-2ef7dc7c0-amd64:/usr/local/bin/hive:ro diff --git a/hives/hanthor/compose.override.snippet.yaml b/hives/hanthor/compose.override.snippet.yaml new file mode 100644 index 00000000..4670257f --- /dev/null +++ b/hives/hanthor/compose.override.snippet.yaml @@ -0,0 +1,32 @@ + # hanthor: hive.reilly.asia, the hanthor/* personal-account hive. Same App as + # school, installation 158987150 (hanthor account). Rebuilt 2026-11-19; the + # AWS-era data was not kept. Host files publish it on :3104. + hanthor: + image: ghcr.io/hivecommons/hive:v6-latest + restart: unless-stopped + command: ["--config", "/etc/hive/hive.yaml"] + environment: + HIVE_ID: "hanthor" + HIVE_HUB_URL: "http://hub:3001" + HIVE_HUB_SECRET: "${HIVE_HUB_SECRET}" + HIVE_DASHBOARD_TOKEN: "${HIVE_DASHBOARD_TOKEN}" + HIVE_SHARED_AGENT_POOL_ENABLED: "true" + OMP_AUTH_BROKER_TOKEN: "${OMP_AUTH_BROKER_TOKEN}" + GH_APP_ID: "3942065" + GH_APP_INSTALLATION_ID: "158987150" + GH_APP_KEY_FILE: "/secrets/gh-app-key.pem" + NTFY_TOPIC: "${NTFY_TOPIC}" + CLAUDE_CODE_OAUTH_TOKEN: "${CLAUDE_CODE_OAUTH_TOKEN}" + HIVE_ACTIONABLE_PRIORITY_LABELS: "priority/p0,triage/accepted,ai-fix-requested,approved-direction,kind/bug,bug,priority/critical-urgent,priority/important-soon,help wanted,good first issue" + volumes: + - ./bin/dashboard-f3e6d836a.html:/opt/hive/proxy/public/index.html:ro + - ./hanthor/config:/etc/hive:ro + - ./hanthor/data:/data + - ./hanthor/secrets:/secrets:ro + depends_on: + - hub + networks: + - default + mem_limit: 2560m + cap_add: + - NET_ADMIN diff --git a/hives/hanthor/hive.yaml b/hives/hanthor/hive.yaml new file mode 100644 index 00000000..35f86a2e --- /dev/null +++ b/hives/hanthor/hive.yaml @@ -0,0 +1,94 @@ +# hanthor: hive.reilly.asia. The personal-account hive (hanthor/*), split from +# school because one hive.yaml holds one GitHub App installation. Same App as +# school (hanthor-hive-agent), its installation on the hanthor account. +# Rebuilt on bihar 2026-11-19 from the AWS-era seed (talos-k8s/hive/spokes/ +# hanthor); the AWS data volume was not kept, so state started fresh. +# The live config is /data/hive.yaml.runtime; dashboard changes win over this. +# +# L5: agents open PRs labelled `hold`; the hive merges only `lgtm` PRs. +# dotfiles auto-merge is gated repo-side (CODEOWNERS + agent-automerge.yml). +acmm_level: 5 +project: + org: hanthor + name: hanthor + repos: + - hanthor/dotfiles + - hanthor/reilly.asia + - hanthor/homebrew-tap + - hanthor/shrimply + - hanthor/indiafoss-chat-android + ai_author: "" + primary_repo: hanthor/dotfiles +hub: + enabled: true + url: http://hub:3001 +agents: + supervisor: + enabled: true + backend: claude + model: claude-sonnet-5-5 + reasoning_effort: high + beads_dir: /data/beads/supervisor + clear_on_kick: false + bead_role: supervisor + caveman_mode: full + backend_owner: operator + model_owner: operator + scanner: + enabled: true + backend: claude + model: claude-haiku-5-5 + reasoning_effort: high + mode: ISSUES_AND_PRS + beads_dir: /data/beads/scanner + clear_on_kick: false + caveman_mode: full + backend_owner: operator + model_owner: operator + quality: + enabled: true + backend: claude + model: claude-sonnet-5-5 + reasoning_effort: high + mode: ISSUES_AND_PRS + beads_dir: /data/beads/quality + clear_on_kick: false + caveman_mode: full + backend_owner: operator + model_owner: operator + guide: + enabled: true + backend: claude + model: claude-sonnet-5-5 + reasoning_effort: high + beads_dir: /data/beads/guide + clear_on_kick: false + caveman_mode: full + backend_owner: operator + model_owner: operator +governor: + eval_interval_s: 300 + modes: + surge: {threshold: 70, supervisor: 10m, scanner: 30m, quality: 1h, guide: 2h} + busy: {threshold: 10, supervisor: 15m, scanner: 1h, quality: 2h, guide: 4h} + quiet: {threshold: 2, supervisor: 30m, scanner: 2h, quality: 4h, guide: 6h} + idle: {threshold: 0, supervisor: 1h, scanner: 4h, quality: 6h, guide: 8h} +github: + app_id: 3942065 + installation_id: 158987150 + app_slug: hanthor-hive-agent + app_authored_prs: true + key_file: /secrets/gh-app-key.pem +dashboard: + port: 3002 + hub_proxied: false + authorized_users: + - hanthor:owner +backends: + allow: + - claude + - omp + deny: [] +sentinel: + exempt_logins: + - renovate[bot] diff --git a/hives/hive-cleanup b/hives/hive-cleanup index 523925bb..366fbb47 100755 --- a/hives/hive-cleanup +++ b/hives/hive-cleanup @@ -21,7 +21,7 @@ WORKDIR_MIN=${WORKDIR_MIN:-2880} BACKUP_DAYS=${BACKUP_DAYS:-14} MIN_FREE_GB=${MIN_FREE_GB:-10} HIVES=/var/srv/hives -CONTAINERS=${CONTAINERS:-"hives-school-1 hives-reef-1"} +CONTAINERS=${CONTAINERS:-"hives-school-1 hives-reef-1 hives-hanthor-1"} log() { echo "hive-cleanup: $*"; } free_gb() { df -BG --output=avail /var | tail -1 | tr -dc 0-9; } diff --git a/hives/hive-failover b/hives/hive-failover index 38368bd0..6be4d7bd 100755 --- a/hives/hive-failover +++ b/hives/hive-failover @@ -44,7 +44,7 @@ set_lease() { echo "$1" > "$LEASE.tmp" && chmod 644 "$LEASE.tmp" && mv "$LEASE.t set_caddy() { local ip=${IP[$1]} tmp tmp=$(mktemp) - sed -E "s#reverse_proxy 100\.(103\.44\.48|66\.158\.127):(310[0-3])#reverse_proxy $ip:\2#" "$CADDYFILE" > "$tmp" + sed -E "s#reverse_proxy 100\.(103\.44\.48|66\.158\.127):(310[0-4])#reverse_proxy $ip:\2#" "$CADDYFILE" > "$tmp" cmp -s "$tmp" "$CADDYFILE" || cat "$tmp" > "$CADDYFILE" # in place: keep the bind-mounted inode rm -f "$tmp" docker cp "$CADDYFILE" "$CADDY_CTR:/tmp/Caddyfile" >/dev/null && @@ -125,7 +125,7 @@ cmd_status() { for h in bihar karnataka; do printf '%-10s healthy=%s running=%s\n' "$h" "$(healthy $h && echo yes || echo no)" "$(running_count $h 2>/dev/null || echo ?)" done - grep -oE 'reverse_proxy 100\.[0-9.]+:310[0-3]' "$CADDYFILE" | sort -u + grep -oE 'reverse_proxy 100\.[0-9.]+:310[0-4]' "$CADDYFILE" | sort -u } case "${1:-}" in