Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 64 additions & 3 deletions docs/src/roles/omp.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,10 @@ Each step is independent: a failure prints `FAIL <step>`, fails the unit, and t
6. `kiro.json`: `~/.config/omp/kiro.json` (kiro-cli ACP fallback on 401/403), creating `~/.config/omp`.
7. `omp-route`: the quota-aware launcher at `~/.local/bin/omp-route`.
8. `routing merged`: deep-merge `units/omp/routing.yml` into `~/.omp/agent/config.yml` with `yq` (lists replace).
9. `broker url`: set `auth.broker.url`; config mode `0600`.
10. `broker token`: set `auth.broker.token` from Bitwarden when the vault is unlocked. If no token ends up in the config the unit prints `PARTIAL: broker token not written ...`.
11. `broker status` (soft): print `omp auth-broker status`.
9. `shared models`: run `units/omp/models.sh` to merge `units/omp/models.yml` into the active `~/.omp/agent/models.yml` or existing `models.yaml`, preserving unrelated providers. Legacy `models.json` is converted first when neither YAML file exists. Files are `0600`; non-empty model allowlists gain `lemonade/*` without restricting an empty/unset allowlist.
10. `broker url`: set `auth.broker.url`; config mode `0600`.
11. `broker token`: set `auth.broker.token` from Bitwarden when the vault is unlocked. If no token ends up in the config the unit prints `PARTIAL: broker token not written ...`.
12. `broker status` (soft): print `omp auth-broker status`.

## Auth broker

Expand All @@ -38,6 +39,66 @@ To seed the token, run `just seed-omp-broker` on a client with the vault unlocke

`DEEPSEEK_API_KEY` and the other provider keys are exposed to the shell by the [`shell-ai` unit](shell_ai.md)'s secret files, not by this unit.

## Karnataka coding model

The `lemonade` provider connects directly to the private, Tailscale-encrypted
endpoint `http://100.66.158.127:13305/api/v1`. The IP avoids requiring MagicDNS
on clients such as Spindle. It has `auth: none`: tailnet access, not a broker
credential, controls connectivity. The broker distributes credentials only;
custom provider/model definitions are distributed by this unit.

```bash
just apply-tags omp
omp --model lemonade/Qwen3-Coder-Next-GGUF
```

This registers Qwen3-Coder-Next MXFP4 (80B total / 3B active), with a 32K
context and 8192-token output limit. Existing cloud model roles and the
quota-aware launcher's starting model are unchanged. The provider is allowed
by the shared model filter. No API key or new Bitwarden item is needed.

See the [Karnataka runbook](../servers/talos-k8s/karnataka/README.md#lemonade-coding-server)
for server persistence and recovery. Native-unit provisioning includes this
configuration on future applies, including no-secrets applies.

### Automatic reconnect rollout

Punjab runs the dedicated user timer `lemonade-rollout.timer`, every 30
minutes and after boot, with user linger enabled. This is model-only
provisioning, not the broad fleet-update timer: it does not pull or apply
the unmerged native-unit migration, install packages, rotate credentials,
or change model roles.

`units/omp/rollout.sh` uses the existing inventory/online-peer intersection,
then transfers only `units/lib.sh`, `units/omp/models.sh`, and the model
definition into a private temporary directory on each reachable client.
The same helper used by the normal unit preserves other custom providers,
broker settings, and cloud availability. Clients need working SSH, `omp`,
and `yq`; failures remain pending for the next timer run. Raw Termux is
excluded because OMP has no Android build; the phone's Debian guest is
the supported `mumbai` target.

The controller uses an isolated payload at `~/.local/lib/omp-lemonade/`
on Punjab, containing `inventory.yml`, `scripts/online_hosts.py`,
`scripts/inventory_parser.py`, and the three files above plus
`units/omp/rollout.sh`. Refresh those files from the approved checkout when
changing the model definition or fleet inventory. The tracked service/timer
templates in `units/omp/` are installed in `~/.config/systemd/user/`.

```bash
systemctl --user list-timers lemonade-rollout.timer
journalctl --user -u lemonade-rollout.service
# Reconcile currently online clients now:
systemctl --user start lemonade-rollout.service
# Stop model-only reconnect provisioning:
systemctl --user disable --now lemonade-rollout.timer
```

Punjab, Karnataka, and Spindle were configured directly. The controller
also successfully re-applied the model-only payload to Karnataka. Mumbai,
Himachal, Kerala, Dilli, and Kanpur were SSH-unreachable during setup and
are not claimed as provisioned; they will be attempted when online.

## Kiro authentication

Plugin installation is unattended. OAuth is per user and must be completed on each computer:
Expand Down
52 changes: 52 additions & 0 deletions docs/src/servers/talos-k8s/karnataka/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,58 @@ updates from the unmerged checkout.
- GPU: AMD integrated
- LAN: DHCP on the home network

## Lemonade coding server

Lemonade Server `2026.44.1` runs as the root-managed Podman Quadlet
`/etc/containers/systemd/lemonade.container`, tracked in
`home-k3s/lemonade.container`. Its generated `lemonade.service` is attached to
`multi-user.target`. The image is digest-pinned; no host OS layering or driver
replacement is required on immutable Utah.

The downloaded model is `Qwen3-Coder-Next-GGUF`:
`unsloth/Qwen3-Coder-Next-GGUF:Qwen3-Coder-Next-MXFP4_MOE.gguf`
(44.7 GiB of weights, 80B total / 3B active parameters). The ROCm stable
backend uses Radeon 8060S (`gfx1151`); context is limited to 32768 tokens and
only one model is loaded. Named volumes `lemonade-cache`, `lemonade-llama`,
`lemonade-data`, and `lemonade-config` retain weights, backends, and settings.
Do not remove these volumes during a service restart or upgrade.

Podman publishes only `127.0.0.1:13305`. Tailscale Serve persistently forwards
tailnet TCP 13305 to that listener:

```bash
sudo tailscale serve --bg --tcp=13305 tcp://127.0.0.1:13305
```

Clients use `http://100.66.158.127:13305/api/v1` over the encrypted tailnet.
There is no API key: all tailnet peers permitted by Tailscale policy can use
the API, including model-management endpoints. Do not publish this port on
LAN/public interfaces or enable Funnel. Corral's existing HTTPS route is
separate and remains intact.

To restore the server, install the tracked Quadlet at the path above, then:

```bash
sudo systemctl daemon-reload
sudo systemctl start lemonade.service
sudo podman exec lemonade-server ./lemonade config set llamacpp.backend=rocm ctx_size=32768 max_loaded_models=1
sudo podman exec lemonade-server ./lemonade backends install llamacpp:rocm
sudo podman exec lemonade-server ./lemonade pull Qwen3-Coder-Next-GGUF
```

Restore the Tailscale Serve rule separately with the command above. Verify
with `sudo podman exec lemonade-server ./lemonade status --json`,
`sudo tailscale serve status`, and `systemctl status lemonade.service`.
The model loads on the first inference request, not merely on service start.
The service has lower CPU/IO weights and a 64 GiB cgroup memory ceiling;
GPU allocations may not count against that ceiling, so monitor host memory
alongside Kubernetes and VM workloads.

On 2026-10-31, direct coding output and a structured tool-call/result round
trip passed with the GPU/ROCm backend. Warm generation was about 36–39
tokens/second on these short requests, not a concurrency benchmark. See the
[`omp` unit](../../../roles/omp.md#karnataka-coding-model) for client configuration.

## Current K3s cluster

Read-only inspection on **2026-10-29** found:
Expand Down
30 changes: 30 additions & 0 deletions home-k3s/lemonade.container
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
[Unit]
Description=Lemonade coding model server (AMD ROCm)
Wants=network-online.target tailscaled.service
After=network-online.target tailscaled.service

[Container]
Image=ghcr.io/lemonade-sdk/lemonade-server@sha256:8ae593e309c5f8f348ea7404f20e27db83d0d247cc9ba3b2bf7fb6310e13e73c
ContainerName=lemonade-server
PublishPort=127.0.0.1:13305:13305
Volume=lemonade-cache:/opt/lemonade/.cache/huggingface
Volume=lemonade-llama:/opt/lemonade/llama
Volume=lemonade-data:/opt/lemonade/.cache/lemonade
Volume=lemonade-config:/opt/lemonade/.config/lemonade
AddDevice=/dev/kfd
AddDevice=/dev/dri
GroupAdd=105
GroupAdd=39
SecurityLabelDisable=true
Exec=./lemond --host 0.0.0.0 --no-broadcast

[Service]
Restart=on-failure
RestartSec=5
TimeoutStartSec=900
MemoryMax=64G
CPUWeight=50
IOWeight=50

[Install]
WantedBy=multi-user.target
52 changes: 50 additions & 2 deletions tests/test_omp_unit.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,6 @@ def test_fresh_home_is_configured(env, tmp_path):
assert (home / ".local/bin/omp-route").stat().st_mode & 0o111
assert broker_url(home) == BROKER_URL
assert (home / ".omp/agent/config.yml").stat().st_mode & 0o777 == 0o600
assert "PARTIAL:" in r.stdout


def test_second_run_changes_nothing(env, tmp_path):
Expand All @@ -69,6 +68,55 @@ def test_failing_step_does_not_stop_later_steps(env, tmp_path):
(home / ".claude").write_text("not a directory")
r = run_unit(env)
assert r.returncode == 1
assert "FAIL claude skill links" in r.stdout
assert (home / ".config/omp/kiro.json").is_file()
assert broker_url(home) == BROKER_URL


@pytest.mark.parametrize("filename", ["models.yml", "models.yaml", "models.json"])
def test_existing_custom_providers_survive_model_provisioning(env, tmp_path, filename):
import json

agent = tmp_path / "home/.omp/agent"
agent.mkdir(parents=True)
existing = {
"providers": {
"private-server": {
"baseUrl": "http://127.0.0.1:8000/v1",
"auth": "none",
"models": [{"id": "private-code", "contextWindow": 4096}],
}
}
}
source = agent / filename
source.write_text(json.dumps(existing) if filename.endswith(".json") else yaml.safe_dump(existing))
for _ in range(2):
result = run_unit(env)
assert result.returncode == 0, result.stdout + result.stderr
active = source if filename == "models.yaml" else agent / "models.yml"
merged = yaml.safe_load(active.read_text())
assert merged["providers"]["private-server"] == existing["providers"]["private-server"]
assert "lemonade" in merged["providers"]
if filename == "models.yaml":
assert not (agent / "models.yml").exists()


@pytest.mark.parametrize("enabled", [None, [], ["anthropic/*"], ["lemonade/*", "anthropic/*"]])
def test_model_only_apply_preserves_cloud_availability_and_settings(env, tmp_path, enabled):
agent = tmp_path / "home/.omp/agent"
agent.mkdir(parents=True)
cfg = agent / "config.yml"
original = {"modelRoles": {"default": "private-server/private-code"}, "auth": {"broker": {"url": BROKER_URL}}}
if enabled is not None:
original["enabledModels"] = enabled
cfg.write_text(yaml.safe_dump(original))
expected = dict(original)
if enabled and "lemonade/*" not in enabled:
expected["enabledModels"] = [*enabled, "lemonade/*"]
for _ in range(2):
result = subprocess.run(
["bash", str(REPO / "units/omp/models.sh")],
env={**env, "DOTFILES_DIR": str(REPO), "UNITS_NO_SECRETS": "1"},
capture_output=True, text=True,
)
assert result.returncode == 0, result.stdout + result.stderr
assert yaml.safe_load(cfg.read_text()) == expected
1 change: 1 addition & 0 deletions units/omp.sh
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ soft "ccleft" ccleft_install
step "kiro.json" install_if_changed "$DOTFILES_DIR/units/omp/kiro.json" "$HOME/.config/omp/kiro.json" 0644
step "omp-route" install_if_changed "$DOTFILES_DIR/units/omp/omp-route" "$HOME/.local/bin/omp-route" 0755
step "routing merged" routing_merge
step "shared models" bash "$DOTFILES_DIR/units/omp/models.sh"
step "broker url" broker_url
step "broker token" broker_token
if [ "$(yq '.auth.broker | has("token")' "$cfg" 2>/dev/null)" != true ]; then
Expand Down
9 changes: 9 additions & 0 deletions units/omp/lemonade-rollout.service
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[Unit]
Description=Provision the Karnataka coding model on online fleet computers
ConditionPathExists=%h/.local/lib/omp-lemonade/units/omp/rollout.sh

[Service]
Type=oneshot
ExecStart=/bin/bash %h/.local/lib/omp-lemonade/units/omp/rollout.sh
Environment="PATH=%h/.local/bin:/home/linuxbrew/.linuxbrew/bin:/usr/local/bin:/usr/bin:/bin"
TimeoutStartSec=10min
12 changes: 12 additions & 0 deletions units/omp/lemonade-rollout.timer
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
[Unit]
Description=Retry model-only provisioning when fleet computers reconnect

[Timer]
OnBootSec=2min
OnCalendar=*-*-* *:00,30:00
RandomizedDelaySec=60
Persistent=true
Unit=lemonade-rollout.service

[Install]
WantedBy=timers.target
34 changes: 34 additions & 0 deletions units/omp/models.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/usr/bin/env bash
# Install only the shared model definition; safe without vault access or package changes.
set -uo pipefail
. "$(dirname "${BASH_SOURCE[0]}")/../lib.sh"

models_merge() {
local models_cfg="$HOME/.omp/agent/models.yml"
mkdir -p "$(dirname "$models_cfg")" || return 1
if [ ! -f "$models_cfg" ] && [ -f "$HOME/.omp/agent/models.yaml" ]; then
models_cfg="$HOME/.omp/agent/models.yaml"
fi
if [ ! -f "$models_cfg" ] && [ -f "$HOME/.omp/agent/models.json" ]; then
yq -P '.' "$HOME/.omp/agent/models.json" | write_atomic "$models_cfg" || return 1
fi
[ -f "$models_cfg" ] || : > "$models_cfg" || return 1
yq -i ". *= (load(\"$DOTFILES_DIR/units/omp/models.yml\") | explode(.) | ... comments=\"\")" "$models_cfg" || return 1
chmod 600 "$models_cfg"
}

models_enable() {
local cfg="$HOME/.omp/agent/config.yml"
[ -f "$cfg" ] || return 0
local count
count="$(yq '(.enabledModels // []) | length' "$cfg")" || return 1
[ "$count" -gt 0 ] || return 0
if [ "$(yq '.enabledModels | contains(["lemonade/*"])' "$cfg")" != true ]; then
yq -i '.enabledModels += ["lemonade/*"]' "$cfg" || return 1
fi
chmod 600 "$cfg"
}

step "models merged" models_merge
step "model filter updated" models_enable
finish
18 changes: 18 additions & 0 deletions units/omp/models.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Merged into the active ~/.omp/agent/models.yml or models.yaml; other providers are retained.
providers:
lemonade:
baseUrl: http://100.66.158.127:13305/api/v1
api: openai-completions
auth: none
models:
- id: Qwen3-Coder-Next-GGUF
name: Qwen3 Coder Next (Karnataka)
reasoning: false
input: [text]
contextWindow: 32768
maxTokens: 8192
cost:
input: 0
output: 0
cacheRead: 0
cacheWrite: 0
37 changes: 37 additions & 0 deletions units/omp/rollout.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
#!/usr/bin/env bash
# Reconcile only the shared model config on online, SSH-accessible OMP clients.
set -uo pipefail
root="${DOTFILES_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)}"
export DOTFILES_DIR="$root"
ssh_opts=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o ServerAliveCountMax=2)
rc=0
for host in $(python3 "$root/scripts/online_hosts.py"); do
# Raw Android has no OMP build; the phone's supported Debian guest is mumbai.
[ "$host" != termux ] || continue
if ! stage=$(ssh "${ssh_opts[@]}" "$host" '
export PATH="$HOME/.bun/bin:$HOME/.local/bin:/home/linuxbrew/.linuxbrew/bin:/opt/homebrew/bin:$PATH"
command -v omp >/dev/null && command -v yq >/dev/null || exit 1
stage=$(mktemp -d /tmp/omp-lemonade.XXXXXXXX) || exit 1
mkdir -p "$stage/units/omp" || exit 1
printf "%s\n" "$stage"
'); then
echo "Pending: $host (SSH, omp, or yq unavailable)" >&2
rc=1
continue
fi
if [[ ! "$stage" =~ ^/tmp/omp-lemonade\.[[:alnum:]]+$ ]]; then
echo "Invalid staging path from $host" >&2
rc=1
continue
fi
if scp -q "${ssh_opts[@]}" "$root/units/lib.sh" "$host:$stage/units/lib.sh" &&
scp -q "${ssh_opts[@]}" "$root/units/omp/models.sh" "$root/units/omp/models.yml" "$host:$stage/units/omp/" &&
ssh "${ssh_opts[@]}" "$host" "export PATH=\"\$HOME/.bun/bin:\$HOME/.local/bin:/home/linuxbrew/.linuxbrew/bin:/opt/homebrew/bin:\$PATH\"; DOTFILES_DIR='$stage' UNITS_NO_SECRETS=1 bash '$stage/units/omp/models.sh'"; then
echo "Configured: $host"
else
echo "Pending: $host (model provisioning failed)" >&2
rc=1
fi
ssh "${ssh_opts[@]}" "$host" "rm -rf -- '$stage'" || rc=1
done
exit "$rc"
1 change: 1 addition & 0 deletions units/omp/routing.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ enabledModels:
- kiro/*
- meta/muse-spark-1.3-contributor
- local/*
- lemonade/*
- web/*
disabledProviders:
- amazon-bedrock
Expand Down