Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions docs/src/automation.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,14 +50,14 @@ manifest on fleet machines.

Each live spoke persists its configuration to `data/hive.yaml.runtime` next to
the seed `config/hive.yaml`, and the runtime file wins on restart, so roster or
level changes made in the dashboard stick. The `hive-config` ConfigMap in
[`talos-k8s/hive/spokes/hanthor/manifest.yaml`](https://github.com/hanthor/dotfiles/blob/master/talos-k8s/hive/spokes/hanthor/manifest.yaml)
was the seed of the former `hive.reilly.asia` spoke on the terminated AWS
cluster (exported 2026-10-01); it does not describe a running Hive. For the
current school and reef files, read-only inspection and the safe edit
procedure, see the
level changes made in the dashboard stick. hive.reilly.asia is the `hanthor`
spoke on bihar (rebuilt 2026-11-19); its seed is
[`hives/hanthor/hive.yaml`](https://github.com/hanthor/dotfiles/blob/master/hives/hanthor/hive.yaml).
The AWS-era ConfigMap in `talos-k8s/hive/spokes/hanthor/manifest.yaml` is
history. For the live files, read-only inspection and the safe edit procedure,
see the
[Hive operations guide](cluster/hive.md#current-deployment-bihar-karnataka-standby):

```bash
ssh ubuntu@100.113.75.88 'docker ps --filter name=hives-; ls /srv/hives/*/data/hive.yaml.runtime'
ssh james@bihar 'sudo podman ps --filter name=hives-; sudo cat /var/srv/hives/hanthor/data/hive.yaml.runtime'
```
26 changes: 21 additions & 5 deletions docs/src/cluster/hive.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,36 @@

## Current deployment: bihar (karnataka standby)

Two Hive spokes, **school** and **reef**, and their **hub** run as compose
Three Hive spokes, **school**, **reef** and **hanthor**, and their **hub** run as compose
services under rootful podman on [bihar](../servers/bihar.md), in
`/var/srv/hives` (`ssh james@bihar`, commands need `sudo`). Until 2026-11-18
they ran on the [Spindle VPS](../servers/spindle-vps/README.md) under
`/srv/hives`. Spindle's Caddy still routes
[school.tunaos.org](https://school.tunaos.org),
[reef.tunaos.org](https://reef.tunaos.org) and
[reef.tunaos.org](https://reef.tunaos.org),
[hive.reilly.asia](https://hive.reilly.asia) and
[hub.tunaos.org](https://hub.tunaos.org), over the tailnet to the active host.
[karnataka](../servers/karnataka.md) is a cold standby. A lease on spindle
selects the active host. Run every command below on **the host the lease
names** (`ssh spindle sudo hive-failover status`). Hosting, failover and the
host-specific compose layer are described in
[Hives hosting and failover](../servers/hives.md).

### hanthor (hive.reilly.asia)

The personal-account spoke from the old AWS cluster was rebuilt on bihar on
2026-11-19. It is the `hanthor` service in `compose.override.yaml` (port 3104),
with its data in `/var/srv/hives/hanthor`. It uses the same GitHub App as
school (`hanthor-hive-agent`, App ID 3942065) through the App's installation on
the hanthor account (158987150), and the same key file. Its seed is
[`hives/hanthor/hive.yaml`](https://github.com/hanthor/dotfiles/blob/master/hives/hanthor/hive.yaml):
ACMM level 5, four agents (supervisor, scanner, quality, guide) on the `claude`
backend, and the repositories `dotfiles`, `reilly.asia`, `homebrew-tap`,
`shrimply` and `indiafoss-chat-android`. Compared with the AWS seed it drops
`indiafoss-companion` (archived), `hummingbird-github` (deleted) and
`rust-wayland-desktop` (now `tuna-os/tuna-desktop`, which school covers). The
AWS data volume was not kept, so the hive started with empty state.

The layout and roster below were read on spindle on 2026-11-13. The move
changed the paths and the commands, but not the hive configuration. Container
state alone does not prove that agents are healthy. See [Checking health](#checking-health).
Expand All @@ -29,7 +45,8 @@ Kubernetes deployment and must not be applied here.
```text
/var/srv/hives/ # on the active host; the host layer is /var/srv/hives-host/compose.<host>.yaml
├── compose.yaml # services: hub, hub-front, school, reef (network spindle_default)
├── compose.override.yaml # pins the running binary/dashboard, memory limits, session pool
├── compose.override.yaml # pins the running binary/dashboard, memory limits, session pool;
│ # also defines the hanthor spoke (added 2026-11-19)
├── .env # secrets and NTFY topic (mode 600; never print or commit)
├── bin/hive-<sha> # fork builds; compose.yaml mounts bin/hive, the override re-pins one
├── <spoke>/config/hive.yaml # seed, mounted read-only at /etc/hive
Expand Down Expand Up @@ -294,8 +311,7 @@ PRs and issues (`gh search prs --author "hanthor-hive-agent[bot]"` and
- The spokes call the `omp-auth-broker` on spindle (`OMP_AUTH_BROKER_URL`,
`http://100.113.75.88:8765`). Caddy routing and failover are described in
[Hives hosting and failover](../servers/hives.md). This page does not describe
the hub's own configuration. The `hive.reilly.asia` hanthor spoke of the old AWS
cluster has no container on this host; its current status is unverified. The
the hub's own configuration. The
[hive_ops role](../roles/hive_ops.md) targets the retired cluster, not this
deployment.

Expand Down
2 changes: 1 addition & 1 deletion docs/src/fleet.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Internet ─▶ Cloudflare ─▶ spindle (Oracle, Frankfurt): Caddy, Matrix, Li
| **pixel-9a** (`termux`) | Phone | Android + Termux | Ansible `termux` host | [Inventory](inventory.md) |

Cross-host services:
- The hives (school, reef, hub) are on bihar. If bihar fails, they move to karnataka. See [Hives hosting](servers/hives.md).
- The hives (school, reef, hanthor, hub) are on bihar. If bihar fails, they move to karnataka. See [Hives hosting](servers/hives.md).
- KubeStellar is described in [Home k3s and KubeStellar](servers/home-k3s.md).

## Intermittent (desktops and laptops)
Expand Down
14 changes: 8 additions & 6 deletions docs/src/servers/hives.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,27 +4,28 @@
> For the hives themselves (agents, configuration, health), see the
> [Hive operations guide](../cluster/hive.md).

Last updated: 2026-11-18. The source for the scripts, compose layers and systemd
Last updated: 2026-11-19 (hanthor added). The source for the scripts, compose layers and systemd
units is [`hives/`](https://github.com/hanthor/dotfiles/tree/master/hives) in this
repository. Each host also has a copy of this runbook at
`/var/srv/hives-host/RUNBOOK.md` (on spindle: `/var/lib/hive-failover/RUNBOOK.md`).

## Overview

The tuna-os hives run as one compose project named `hives`, with four containers:
The hives run as one compose project named `hives`, with five containers:

| Service | Purpose | Public URL |
|---|---|---|
| `hub` | Hub API that all spokes report to | `https://hub.tunaos.org/api/*`, `/login/*` |
| `hub-front` | Static hub page (nginx) | `https://hub.tunaos.org/` |
| `school` | Spoke for the tuna-os org (GitHub App 3942065) | `https://school.tunaos.org` |
| `reef` | Spoke (GitHub App 5546574) | `https://reef.tunaos.org` |
| `hanthor` | Spoke for the `hanthor/*` personal repositories (App 3942065, installation 158987150). Added 2026-11-19 | `https://hive.reilly.asia` |

Until 2026-11-18 the hives ran on spindle (Oracle ARM VM, Frankfurt, 2 cores and 11 GB of RAM). They now run on **bihar** (x86, 24 cores, 62 GB of RAM). **karnataka** is a cold standby. **spindle** keeps the public IP and Caddy, and it coordinates failover.

```
Internet ─▶ Cloudflare ─▶ spindle Caddy (92.5.82.232)
│ tailnet, ports 3100-3103
│ tailnet, ports 3100-3104
┌──────────────┴──────────────┐
▼ (active, holds the lease) ▼ (standby, stopped)
bihar 100.103.44.48 karnataka 100.66.158.127
Expand All @@ -39,7 +40,7 @@ The OMP auth broker still runs on spindle at `http://100.113.75.88:8765`, reache

| Path | Contents |
|---|---|
| `/var/srv/hives/` | Hive data and configuration, copied from spindle's `/srv/hives`: `compose.yaml`, `compose.override.yaml`, `.env`, `{hub,school,reef}/{config,data,secrets}`, `hub-front/`, `proxy/`, `bin/dashboard-*.html` |
| `/var/srv/hives/` | Hive data and configuration, copied from spindle's `/srv/hives`: `compose.yaml`, `compose.override.yaml`, `.env`, `{hub,school,reef,hanthor}/{config,data,secrets}`, `hub-front/`, `proxy/`, `bin/dashboard-*.html` |
| `/var/srv/hives-host/compose.<host>.yaml` | Host layer: ports, memory limits, broker URL, image digests, and the amd64 binary mount |
| `/var/srv/hives-host/bin/hive-2ef7dc7c0-amd64` | `tuna-os/hive` at `2ef7dc7c0` (branch `deploy/upstream-v6-20261113`) built with `CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath ./cmd/hive` |
| `/var/srv/hives-host/hives` | Wrapper that runs docker-compose against rootful podman with the three compose files |
Expand All @@ -56,11 +57,12 @@ Each host publishes its ports on its own tailnet IP only:
| 3101 | hub |
| 3102 | school |
| 3103 | reef |
| 3104 | hanthor |

A firewalld rich rule in the `public` zone accepts these ports only from spindle (`100.113.75.88`). Inside the compose network the services reach each other by name, for example `http://hub:3001`. The podman network is named `spindle_default` so that `compose.yaml` works unchanged.

Images are pinned by digest, and the pin must match the binary:
- `ghcr.io/hivecommons/hive@sha256:d56425fb…` (school and reef)
- `ghcr.io/hivecommons/hive@sha256:d56425fb…` (school, reef and hanthor)
- `ghcr.io/hivecommons/hive-hub@sha256:3c0162eb…` (hub)

Do not change to `:v6-latest`. On 2026-11-18 a newer image made the hub log "failed to write token cache". **Upgrade the binary and the images together.**
Expand Down Expand Up @@ -185,7 +187,7 @@ On a new host, also do these steps:
3. Create the podman network `spindle_default`.
4. Pull the pinned images.
5. Copy `bin/hive-2ef7dc7c0-amd64` from the other host. If neither host has it, build it from `tuna-os/hive` at `2ef7dc7c0`.
6. Create the SSH keys in `/var/srv/hives-host/keys/`, and add the firewalld rule for ports 3100-3103 from spindle.
6. Create the SSH keys in `/var/srv/hives-host/keys/`, and add the firewalld rule for ports 3100-3104 from spindle. Also install `/etc/sysctl.d/90-hives-inotify.conf` (`fs.inotify.max_user_instances = 1024`, `fs.inotify.max_user_watches = 1048576`): with five containers the per-agent permission watchers exhaust the default of 128 instances.
7. Label the data and compose files (not the scripts): `chcon -R -t container_file_t /var/srv/hives /var/srv/hives-host/compose.*.yaml /var/srv/hives-host/bin`.
8. Enable `hive-reconcile.timer`, `hive-sync.timer` and `hive-cleanup.timer`. Start each service one time and make sure that its `Result` is `success`.

Expand Down
6 changes: 3 additions & 3 deletions docs/src/servers/spindle-vps/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ terminated 2026-10-25. **Spindle**, **Postgres**, standalone **LiveKit** and **l

**The hives moved off this host on 2026-11-18.** School, Reef, Hub and the hub
frontend now run on [bihar](../bihar.md), with [karnataka](../karnataka.md) as a cold
standby. Spindle keeps three hive roles:
standby. The `hanthor` spoke (`hive.reilly.asia`) joined them on 2026-11-19. Spindle keeps three hive roles:

- **Caddy front door:** `school`, `reef` and `hub.tunaos.org` go to the active host's
tailnet IP on ports 3100-3103. `hive-failover` rewrites these upstreams.
- **Caddy front door:** `school`, `reef`, `hub.tunaos.org` and `hive.reilly.asia` go to the active host's
tailnet IP on ports 3100-3104 (`hive.reilly.asia` on 3104). `hive-failover` rewrites these upstreams.
- **Failover coordinator:** the lease in `/var/lib/hive-failover/active`,
`/usr/local/sbin/hive-failover`, and `hive-failover.timer`.
- **OMP auth broker:** the `omp-auth-broker` user service on `100.113.75.88:8765`. The hives
Expand Down
2 changes: 2 additions & 0 deletions hives/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ the design, the install locations, daily operations, failover and the gotchas.
| `compose.<host>.yaml` | `/var/srv/hives-host/` on that host |
| `hive-cleanup` | `/var/srv/hives-host/` on bihar and karnataka |
| `systemd/hive-{reconcile,sync,cleanup}.{service,timer}` | `/etc/systemd/system/` on bihar and karnataka |
| `hanthor/hive.yaml` | seed for the hanthor spoke: `/var/srv/hives/hanthor/config/hive.yaml` (synced to the standby by `hive-sync`) |
| `hanthor/compose.override.snippet.yaml` | the `hanthor` service, appended to `/var/srv/hives/compose.override.yaml` |
| `hive-failover` | `/usr/local/sbin/` on spindle |
| `systemd/hive-failover.{service,timer}` | `/etc/systemd/system/` on spindle |

Expand Down
8 changes: 8 additions & 0 deletions hives/compose.bihar.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,3 +35,11 @@ services:
OMP_AUTH_BROKER_URL: "http://100.113.75.88:8765"
volumes:
- /var/srv/hives-host/bin/hive-2ef7dc7c0-amd64:/usr/local/bin/hive:ro
hanthor:
image: ghcr.io/hivecommons/hive@sha256:d56425fbaa1b5854a96356d56ebc2f371f337038339c56c94dcf3f2ec29ead51
ports: ["100.103.44.48:3104:3001"]
mem_limit: 4g
environment:
OMP_AUTH_BROKER_URL: "http://100.113.75.88:8765"
volumes:
- /var/srv/hives-host/bin/hive-2ef7dc7c0-amd64:/usr/local/bin/hive:ro
8 changes: 8 additions & 0 deletions hives/compose.karnataka.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,3 +35,11 @@ services:
OMP_AUTH_BROKER_URL: "http://100.113.75.88:8765"
volumes:
- /var/srv/hives-host/bin/hive-2ef7dc7c0-amd64:/usr/local/bin/hive:ro
hanthor:
image: ghcr.io/hivecommons/hive@sha256:d56425fbaa1b5854a96356d56ebc2f371f337038339c56c94dcf3f2ec29ead51
ports: ["100.66.158.127:3104:3001"]
mem_limit: 4g
environment:
OMP_AUTH_BROKER_URL: "http://100.113.75.88:8765"
volumes:
- /var/srv/hives-host/bin/hive-2ef7dc7c0-amd64:/usr/local/bin/hive:ro
32 changes: 32 additions & 0 deletions hives/hanthor/compose.override.snippet.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# hanthor: hive.reilly.asia, the hanthor/* personal-account hive. Same App as
# school, installation 158987150 (hanthor account). Rebuilt 2026-11-19; the
# AWS-era data was not kept. Host files publish it on :3104.
hanthor:
image: ghcr.io/hivecommons/hive:v6-latest
restart: unless-stopped
command: ["--config", "/etc/hive/hive.yaml"]
environment:
HIVE_ID: "hanthor"
HIVE_HUB_URL: "http://hub:3001"
HIVE_HUB_SECRET: "${HIVE_HUB_SECRET}"
HIVE_DASHBOARD_TOKEN: "${HIVE_DASHBOARD_TOKEN}"
HIVE_SHARED_AGENT_POOL_ENABLED: "true"
OMP_AUTH_BROKER_TOKEN: "${OMP_AUTH_BROKER_TOKEN}"
GH_APP_ID: "3942065"
GH_APP_INSTALLATION_ID: "158987150"
GH_APP_KEY_FILE: "/secrets/gh-app-key.pem"
NTFY_TOPIC: "${NTFY_TOPIC}"
CLAUDE_CODE_OAUTH_TOKEN: "${CLAUDE_CODE_OAUTH_TOKEN}"
HIVE_ACTIONABLE_PRIORITY_LABELS: "priority/p0,triage/accepted,ai-fix-requested,approved-direction,kind/bug,bug,priority/critical-urgent,priority/important-soon,help wanted,good first issue"
volumes:
- ./bin/dashboard-f3e6d836a.html:/opt/hive/proxy/public/index.html:ro
- ./hanthor/config:/etc/hive:ro
- ./hanthor/data:/data
- ./hanthor/secrets:/secrets:ro
depends_on:
- hub
networks:
- default
mem_limit: 2560m
cap_add:
- NET_ADMIN
94 changes: 94 additions & 0 deletions hives/hanthor/hive.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# hanthor: hive.reilly.asia. The personal-account hive (hanthor/*), split from
# school because one hive.yaml holds one GitHub App installation. Same App as
# school (hanthor-hive-agent), its installation on the hanthor account.
# Rebuilt on bihar 2026-11-19 from the AWS-era seed (talos-k8s/hive/spokes/
# hanthor); the AWS data volume was not kept, so state started fresh.
# The live config is /data/hive.yaml.runtime; dashboard changes win over this.
#
# L5: agents open PRs labelled `hold`; the hive merges only `lgtm` PRs.
# dotfiles auto-merge is gated repo-side (CODEOWNERS + agent-automerge.yml).
acmm_level: 5
project:
org: hanthor
name: hanthor
repos:
- hanthor/dotfiles
- hanthor/reilly.asia
- hanthor/homebrew-tap
- hanthor/shrimply
- hanthor/indiafoss-chat-android
ai_author: ""
primary_repo: hanthor/dotfiles
hub:
enabled: true
url: http://hub:3001
agents:
supervisor:
enabled: true
backend: claude
model: claude-sonnet-5-5
reasoning_effort: high
beads_dir: /data/beads/supervisor
clear_on_kick: false
bead_role: supervisor
caveman_mode: full
backend_owner: operator
model_owner: operator
scanner:
enabled: true
backend: claude
model: claude-haiku-5-5
reasoning_effort: high
mode: ISSUES_AND_PRS
beads_dir: /data/beads/scanner
clear_on_kick: false
caveman_mode: full
backend_owner: operator
model_owner: operator
quality:
enabled: true
backend: claude
model: claude-sonnet-5-5
reasoning_effort: high
mode: ISSUES_AND_PRS
beads_dir: /data/beads/quality
clear_on_kick: false
caveman_mode: full
backend_owner: operator
model_owner: operator
guide:
enabled: true
backend: claude
model: claude-sonnet-5-5
reasoning_effort: high
beads_dir: /data/beads/guide
clear_on_kick: false
caveman_mode: full
backend_owner: operator
model_owner: operator
governor:
eval_interval_s: 300
modes:
surge: {threshold: 70, supervisor: 10m, scanner: 30m, quality: 1h, guide: 2h}
busy: {threshold: 10, supervisor: 15m, scanner: 1h, quality: 2h, guide: 4h}
quiet: {threshold: 2, supervisor: 30m, scanner: 2h, quality: 4h, guide: 6h}
idle: {threshold: 0, supervisor: 1h, scanner: 4h, quality: 6h, guide: 8h}
github:
app_id: 3942065
installation_id: 158987150
app_slug: hanthor-hive-agent
app_authored_prs: true
key_file: /secrets/gh-app-key.pem
dashboard:
port: 3002
hub_proxied: false
authorized_users:
- hanthor:owner
backends:
allow:
- claude
- omp
deny: []
sentinel:
exempt_logins:
- renovate[bot]
2 changes: 1 addition & 1 deletion hives/hive-cleanup
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ WORKDIR_MIN=${WORKDIR_MIN:-2880}
BACKUP_DAYS=${BACKUP_DAYS:-14}
MIN_FREE_GB=${MIN_FREE_GB:-10}
HIVES=/var/srv/hives
CONTAINERS=${CONTAINERS:-"hives-school-1 hives-reef-1"}
CONTAINERS=${CONTAINERS:-"hives-school-1 hives-reef-1 hives-hanthor-1"}

log() { echo "hive-cleanup: $*"; }
free_gb() { df -BG --output=avail /var | tail -1 | tr -dc 0-9; }
Expand Down
4 changes: 2 additions & 2 deletions hives/hive-failover
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ set_lease() { echo "$1" > "$LEASE.tmp" && chmod 644 "$LEASE.tmp" && mv "$LEASE.t
set_caddy() {
local ip=${IP[$1]} tmp
tmp=$(mktemp)
sed -E "s#reverse_proxy 100\.(103\.44\.48|66\.158\.127):(310[0-3])#reverse_proxy $ip:\2#" "$CADDYFILE" > "$tmp"
sed -E "s#reverse_proxy 100\.(103\.44\.48|66\.158\.127):(310[0-4])#reverse_proxy $ip:\2#" "$CADDYFILE" > "$tmp"
cmp -s "$tmp" "$CADDYFILE" || cat "$tmp" > "$CADDYFILE" # in place: keep the bind-mounted inode
rm -f "$tmp"
docker cp "$CADDYFILE" "$CADDY_CTR:/tmp/Caddyfile" >/dev/null &&
Expand Down Expand Up @@ -125,7 +125,7 @@ cmd_status() {
for h in bihar karnataka; do
printf '%-10s healthy=%s running=%s\n' "$h" "$(healthy $h && echo yes || echo no)" "$(running_count $h 2>/dev/null || echo ?)"
done
grep -oE 'reverse_proxy 100\.[0-9.]+:310[0-3]' "$CADDYFILE" | sort -u
grep -oE 'reverse_proxy 100\.[0-9.]+:310[0-4]' "$CADDYFILE" | sort -u
}

case "${1:-}" in
Expand Down
Loading