Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions Justfile
Original file line number Diff line number Diff line change
Expand Up @@ -537,6 +537,18 @@ seed-kube:
seed-omp-broker:
@{{ dotfiles_dir }}/scripts/bw-seed-omp-broker.sh

# Tailnet hygiene (scripts/tailnet.py): audit is read-only; the others dry-run unless `--apply`
tailnet-audit:
@{{ dotfiles_dir }}/scripts/tailnet.py audit

# gc + rename + inventory alignment, e.g. `just tailnet-sync --apply`
tailnet-sync *args:
@{{ dotfiles_dir }}/scripts/tailnet.py sync {{ args }}

# Store the Tailscale OAuth client secret (device gc/rename + join key) in Bitwarden
seed-tailnet-oauth:
@{{ dotfiles_dir }}/scripts/bw-seed-tailnet.sh

# Show all online fleet machines' doctor status in parallel
doctor-fleet:
#!/usr/bin/env bash
Expand Down
1 change: 1 addition & 0 deletions docs/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

- [Architecture](architecture.md)
- [Units](units.md)
- [Tailnet hygiene](tailnet.md)
- [Inventory & Groups](inventory.md)
- [Secrets with Bitwarden](bitwarden.md)
- [Onboarding a New Machine](onboarding.md)
Expand Down
4 changes: 2 additions & 2 deletions docs/src/roles/tailscale.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Installs Tailscale, fetches a reusable auth key from BW, joins the network, conf
1. Configures `systemd-resolved` so MagicDNS works (link `/etc/resolv.conf` to the stub resolver, drop in `DNSStubListenerExtra=::1` for container DNS).
2. Sets `net.ipv4.conf.{all,default}.rp_filter=2` and IP forwarding on, so exit-node and subnet-router traffic actually routes.
3. If `tailscale status` reports connected → updates hostname, prefs, operator, systray, then exits the role.
4. If not connected and the vault is unlocked → fetches `tailscale-apikey` from BW, deletes any stale device with this hostname via the Tailscale API, then `tailscale up --authkey …`.
4. If not connected and the vault is unlocked → fetches `tailscale-apikey` (an OAuth client secret) from BW, runs `scripts/tailnet.py release <hostname>` to delete stale devices that still hold the hostname (see [Tailnet hygiene](../tailnet.md)), then `tailscale up --authkey …`.
5. If not connected and the vault is **locked** → emits a one-line warning and ends the role. Joining without a key would either fail noisily or, worse, succeed with an empty key.

## Failure modes
Expand All @@ -20,7 +20,7 @@ Installs Tailscale, fetches a reusable auth key from BW, joins the network, conf
|---|---|---|
| Role aborts with "vault locked" warning, host is not on tailnet | First-time apply on this host without BW unlocked | Unlock BW (`bw unlock --raw | tee /tmp/bw_session > /dev/null`), then `just apply-tags tailscale` |
| Tailscale connected but MagicDNS names don't resolve | `systemd-resolved` link to `/etc/resolv.conf` was overwritten by NetworkManager / DHCP client | The role re-applies it; just re-run `dots-apply`. To make it sticky, check `network-manager` settings (Bluefin does this correctly out of the box) |
| Duplicate device entries in admin console after rebuild | API-key device cleanup didn't run (vault locked at the time) | Manually delete in [tailscale admin console](https://login.tailscale.com/admin/machines) or unlock vault and re-apply |
| Duplicate device entries / `<host>-1` names after rebuild | The BW credential is dead or lacks scopes, so `tailnet.py release` could not delete the old device | `just seed-tailnet-oauth`, then `just tailnet-sync --apply` |
| Joins but no routes advertised | `tailscale_advertise_routes` not set in `host_vars` | Add it: `tailscale_advertise_routes: "<lan-cidr>"` |

## How to verify
Expand Down
31 changes: 31 additions & 0 deletions docs/src/tailnet.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Tailnet hygiene

`scripts/tailnet.py` keeps the tailnet and `inventory.yml` in agreement. Everything is a dry run unless `--apply` is given.

| Command | What it does |
|---|---|
| `just tailnet-audit` | Read-only report; exit 1 when anything needs doing |
| `just tailnet-sync [--apply]` | GC, then rename, then inventory alignment, planned as one sequence |
| `scripts/tailnet.py gc [--apply] [--days N] [--k8s-days N]` | Delete duplicate devices and stale test VMs |
| `scripts/tailnet.py rename [--apply]` | Rename live devices to their intended name |
| `scripts/tailnet.py fix-inventory [--apply]` | Make `ansible_host` IPs and `tailnet_name` overrides follow the live devices |
| `scripts/tailnet.py release NAME` | Delete offline devices called `NAME` or `NAME-<n>`; the `tailscale` role runs it before joining |

## Rules

- A device's tailnet name is the first MagicDNS label. An inventory host's **intended** name is its `tailnet_name` var, else the inventory name. A `tailnet_name` of the form `<host>-<n>` is the auto-suffix Tailscale adds when a stale device still holds `<host>`, so it means "wants `<host>`" and is removed once the rename has happened.
- **Duplicates:** several devices named `<intended>` / `<intended>-<n>`: the live (else most recently seen) one is kept; older offline ones are deleted.
- **Test VMs:** offline longer than `--days` (default 14), named `gate-*`, `repro-*`, `tuna-v1-preflight-*` or `*-vm`, and not an inventory host.
- **k8s proxies** (`tag:k8s*`) are only deleted with `--k8s-days`. The home cluster is powered down and expected back, so its proxies are deliberately kept by default.
- Never touched: devices that are online, this machine, devices shared in from other tailnets, and the retired `vps` group.
- `fix-inventory` run alone reflects the live state, so it will not drop a `tailnet_name` override before the rename has actually happened.

## Credential

Reads fall back to the local `tailscale status --json`; writes need the API. The credential is a Tailscale **OAuth client** (scopes `devices:core` write and `auth_keys` write, tag `tag:fleet`) created at <https://login.tailscale.com/admin/settings/oauth>. Store it with `just seed-tailnet-oauth`: the secret is verified against Tailscale before it is written to Bitwarden item `tailscale-apikey`. `TS_OAUTH_CLIENT_SECRET` overrides the Bitwarden lookup.

The same secret is the join auth key used by the [`tailscale`](roles/tailscale.md) role. The previous cleanup in that role treated it as a classic API key, got a 401 that `failed_when: false` hid, and never deleted anything; that is why re-joined machines came back as `himachal-1` and `kerala-1`.

## Caveat when renaming

After `rename --apply`, `himachal-1` stops resolving; use the new name. Do not rename while something is using the old MagicDNS name (an rsync or ssh session, for example).
45 changes: 12 additions & 33 deletions roles/tailscale/tasks/connect.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,39 +23,18 @@
PATH: "{{ ansible_env_path }}"
BW_SESSION: "{{ bw_session }}"

- name: Remove old tailscale device with same hostname
block:
- name: Fetch tailscale device list
ansible.builtin.uri:
url: https://api.tailscale.com/api/v2/tailnet/-/devices
method: GET
user: "{{ tailscale_apikey.stdout }}"
password: ""
force_basic_auth: true
status_code: 200
headers:
Content-Type: application/json
register: ts_devices
failed_when: false
no_log: true

- name: Find old device matching our hostname
ansible.builtin.set_fact:
ts_old_device_id: >-
{{ (ts_devices.json.devices | default([])
| selectattr('name', 'equalto', inventory_hostname + '.' + tailscale_domain)
| map(attribute='id') | first) | default('') }}

- name: Delete old tailscale device
ansible.builtin.uri:
url: https://api.tailscale.com/api/v2/device/{{ ts_old_device_id }}
method: DELETE
user: "{{ tailscale_apikey.stdout }}"
password: ""
force_basic_auth: true
status_code: 200
when: ts_old_device_id | length > 0
no_log: true
# A stale device that still holds this hostname makes Tailscale join us as
# <hostname>-1. scripts/tailnet.py exchanges the OAuth client secret for an API
# token (the old inline cleanup used it as a classic API key, got a silent 401,
# and never deleted anything).
- name: Remove stale tailscale devices with this hostname
ansible.builtin.command: python3 {{ playbook_dir }}/scripts/tailnet.py release {{ inventory_hostname }}
environment:
TS_OAUTH_CLIENT_SECRET: "{{ tailscale_apikey.stdout }}"
register: ts_release
changed_when: "'deleted' in ts_release.stdout"
failed_when: false
no_log: true
when: tailscale_apikey.stdout | default('') | length > 0

- name: Join tailscale network
Expand Down
49 changes: 49 additions & 0 deletions scripts/bw-seed-tailnet.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
#!/usr/bin/env bash
# Store the Tailscale OAuth client secret in Bitwarden item `tailscale-apikey`.
# Used by the tailscale role (as the join auth key) and scripts/tailnet.py (device gc/rename).
#
# Create the client first: https://login.tailscale.com/admin/settings/oauth
# scopes: devices:core (write), auth_keys (write); tag: tag:fleet
# Usage: ./scripts/bw-seed-tailnet.sh (prompts for the secret, no echo)
# printf %s "$SECRET" | ./scripts/bw-seed-tailnet.sh
# The secret is checked against Tailscale before anything is written.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
source "$SCRIPT_DIR/bw-item.sh"

if [ -z "${BW_SESSION:-}" ]; then
BW_SESSION=$("$SCRIPT_DIR/bw-unlock.sh")
export BW_SESSION
fi

if [ -t 0 ]; then
read -rs -p "Tailscale OAuth client secret (tskey-client-...): " secret; echo >&2
else
secret=$(cat)
fi
secret="${secret//[[:space:]]/}"
case "$secret" in
tskey-client-*) ;;
*) echo "that is not an OAuth client secret (expected tskey-client-...)" >&2; exit 1 ;;
esac
cid=$(cut -d- -f3 <<<"$secret")

code=$(curl -s -o /dev/null -w '%{http_code}' -X POST https://api.tailscale.com/api/v2/oauth/token \
--data-urlencode "client_id=$cid" --data-urlencode "client_secret=$secret")
if [ "$code" != 200 ]; then
echo "Tailscale rejected the secret (HTTP $code); nothing written." >&2
exit 1
fi

payload=$(bw get template item \
| jq --arg tok "$secret" --arg cid "$cid" '
. + {
name: "tailscale-apikey",
type: 1,
notes: "Tailscale OAuth client (devices:core, auth_keys; tag:fleet). Used as the join auth key by the tailscale role and by scripts/tailnet.py. Managed by scripts/bw-seed-tailnet.sh.",
login: {username: $cid, password: $tok}
}')

bw_upsert_item "tailscale-apikey" "$payload" 1
bw sync >/dev/null
echo "Done. Check the tailnet with: just tailnet-audit"
60 changes: 60 additions & 0 deletions scripts/inventory_parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,66 @@ def parse_inventory(text: str):
return all_hosts, vps


def parse_host_vars(text: str):
"""{host: {var: value}} for the 6-space `key: value` lines under each all.hosts entry.

Values are stripped of inline comments and quotes; no PyYAML (see parse_inventory).
"""
hosts, section, host = {}, None, None
for line in text.splitlines():
if not line.strip() or line.lstrip().startswith("#"):
continue
indent = len(line) - len(line.lstrip())
if indent == 2:
section = line.strip().rstrip(":")
elif indent == 4 and section == "hosts":
host = line.strip().rstrip(":")
hosts[host] = {}
elif indent == 6 and section == "hosts" and host is not None:
m = re.match(r"([A-Za-z0-9_]+):\s*(.*?)\s*(?:#.*)?$", line.strip())
if m and m.group(2):
hosts[host][m.group(1)] = m.group(2).strip("'\"")
return hosts


def set_host_var(text: str, host: str, key: str, value):
"""Return `text` with `host`'s `key` set to `value` (or removed when value is None).

Edits only the host's own block under all.hosts, keeping its inline comment.
A missing key is appended after the block's last variable line. Unknown hosts
leave the text unchanged.
"""
lines = text.split("\n")
start = None
for i, line in enumerate(lines):
if line.startswith(" children:"):
break
if re.fullmatch(rf" {re.escape(host)}:\s*(#.*)?", line):
start = i
break
if start is None:
return text
end = start + 1
while end < len(lines) and (lines[end].startswith(" ") or not lines[end].strip() or lines[end].lstrip().startswith("#")):
if lines[end].startswith(" ") and not lines[end].startswith(" "):
break
end += 1
last_var = start
for i in range(start + 1, end):
if lines[i].startswith(" ") and not lines[i].lstrip().startswith("#"):
last_var = i
m = re.match(rf"( {re.escape(key)}:\s*)([^#\n]*?)(\s*(?:#.*)?)$", lines[i])
if m:
if value is None:
del lines[i]
else:
lines[i] = f"{m.group(1)}{value}{m.group(3) if m.group(3).strip() else ''}"
return "\n".join(lines)
if value is not None:
lines.insert(last_var + 1, f" {key}: {value}")
return "\n".join(lines)


def register_host(name: str, mtype: str = "desktop", inv_path: str = "inventory.yml") -> bool:
"""Add `name` to all.hosts and to the `mtype` group.

Expand Down
Loading