From adf88491cbfb45fac5b4ab0f602610b16db515c0 Mon Sep 17 00:00:00 2001 From: Devinder Singh Date: Thu, 29 Oct 2026 12:29:48 +0100 Subject: [PATCH 1/2] infra: capture the server's base setup and a rebuild runbook --- infra/server/README.md | 47 ++++++++++++++++++++++++++++++++ infra/server/firewall.sh | 22 +++++++++++++++ infra/server/k3s-config.yaml | 6 ++++ infra/server/sshd-hardening.conf | 6 ++++ 4 files changed, 81 insertions(+) create mode 100644 infra/server/README.md create mode 100644 infra/server/firewall.sh create mode 100644 infra/server/k3s-config.yaml create mode 100644 infra/server/sshd-hardening.conf diff --git a/infra/server/README.md b/infra/server/README.md new file mode 100644 index 0000000..f749bcb --- /dev/null +++ b/infra/server/README.md @@ -0,0 +1,47 @@ +# Rebuilding the server + +How to get from a fresh VPS back to a running cluster, in order. Every setting that lives on the +server is in this folder or in `k8s/`. Only the web panels (one.com, get.tech) and GitHub's +settings are outside git. + +Copy a file to the server with `ssh relay 'cat > /tmp/' < infra/server/`, then +run the command shown on the server. Commands that ask questions (`adduser`, the first `sudo`) +run alone, never inside a pasted block. + +## 1. Outside the server (web panels) + +- **one.com:** a VPS with Ubuntu 26.04 (4 vCPU, 8 GB). Two-step login on the account. Network + firewall: allow TCP 22, 80 and 443. +- **get.tech DNS:** A records `api`, `cdn` and `grafana` → the VPS's IPv4 address. No AAAA + records: the cluster is IPv4-only, and Let's Encrypt prefers IPv6 when it exists. + +## 2. Base system (steps 0.3–0.7) + +| What | How | +|---|---| +| Personal users | `sudo adduser ` (alone), `sudo usermod -aG sudo `, then `ssh-copy-id` from each laptop | +| Updates | `sudo apt update && sudo apt full-upgrade -y`, then `sudo dpkg-reconfigure -plow unattended-upgrades` (daily security updates) | +| SSH | `sudo install -m 644 /tmp/sshd-hardening.conf /etc/ssh/sshd_config.d/00-hardening.conf && sudo sshd -t && sudo systemctl restart ssh`. Test a new login in a second terminal before closing the first | +| Firewall | `sudo bash /tmp/firewall.sh` | + +## 3. Kubernetes (steps 0.8–0.11) + +| What | How | +|---|---| +| k3s settings | `sudo install -D -m 644 /tmp/k3s-config.yaml /etc/rancher/k3s/config.yaml`, **before** installing k3s | +| k3s v1.36.5 | Download `https://get.k3s.io` to a file, read it, then run it with `INSTALL_K3S_VERSION=v1.36.5+k3s1` | +| kubectl for each admin | `mkdir -p ~/.kube && sudo install -m 600 -o "$USER" /etc/rancher/k3s/k3s.yaml ~/.kube/config`. It's an admin password: never copy it off the server | +| cert-manager v1.21.2 | `kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.21.2/cert-manager.yaml` | +| Platform files | `k8s/platform/*.yaml`, applied by hand (see [`k8s/README.md`](../../k8s/README.md)) | + +## 4. The deploy path (step 1.6) + +1. `setup-deploy-user.sh` and `relay-deploy.sh`: see [`k8s/README.md`](../../k8s/README.md). +2. GitHub environment `production` (only `main` may use it): the secret `DEPLOY_SSH_KEY` + (the new key's private half, with `gh secret set … --env production < key`), and the variables + `DEPLOY_HOST` and `DEPLOY_KNOWN_HOSTS` (the server's new host key). +3. Merge anything to `main` (or re-run the last pipeline): the app deploys itself. + +## 5. Data (Phase 6) + +Restore the newest database dump from S3. Until then, a lost server means lost data. diff --git a/infra/server/firewall.sh b/infra/server/firewall.sh new file mode 100644 index 0000000..2624761 --- /dev/null +++ b/infra/server/firewall.sh @@ -0,0 +1,22 @@ +#!/usr/bin/env bash +# The server's own firewall, ufw (step 0.7). one.com's network firewall sits in front of it +# and is set in their web panel (TCP 22, 80, 443): two layers, so one mistake exposes nothing. +# +# Run on the VPS: sudo bash firewall.sh +# Safe to run again: ufw skips rules it already has. +set -euo pipefail + +ufw default deny incoming # everything not listed below is dropped +ufw default allow outgoing + +ufw allow 22/tcp # SSH, keys only +ufw allow 80/tcp # HTTP: Let's Encrypt's check; everything else redirects to HTTPS +ufw allow 443/tcp # HTTPS + +# k3s's own networks: pods (10.42) and Services (10.43) talk to each other through the host. +ufw allow from 10.42.0.0/16 to any +ufw allow from 10.43.0.0/16 to any + +# Never opened: 5432 (Postgres) and 6443 (the Kubernetes API). +ufw --force enable +ufw status verbose diff --git a/infra/server/k3s-config.yaml b/infra/server/k3s-config.yaml new file mode 100644 index 0000000..489a655 --- /dev/null +++ b/infra/server/k3s-config.yaml @@ -0,0 +1,6 @@ +# k3s settings (step 0.8). Installed as /etc/rancher/k3s/config.yaml BEFORE installing k3s, +# so the setting is there from the first start. +# Encrypts Kubernetes Secrets at rest. By default they're only base64, which anyone who can +# read the disk can decode. Limit: the key is on the same disk, so a copy of the whole disk +# can still decrypt them. +secrets-encryption: true diff --git a/infra/server/sshd-hardening.conf b/infra/server/sshd-hardening.conf new file mode 100644 index 0000000..9c713c3 --- /dev/null +++ b/infra/server/sshd-hardening.conf @@ -0,0 +1,6 @@ +# SSH hardening (step 0.6). Installed as /etc/ssh/sshd_config.d/00-hardening.conf. +# sshd reads these files in alphabetical order and keeps the FIRST value it sees, so "00-" +# makes ours win over one.com's 50-cloud-init.conf, which turns passwords on. +PermitRootLogin no +PasswordAuthentication no +KbdInteractiveAuthentication no From c8a667e113fd0442868b70096770cdae96ef8b4b Mon Sep 17 00:00:00 2001 From: Devinder Singh Date: Thu, 29 Oct 2026 13:10:48 +0100 Subject: [PATCH 2/2] docs: record the firewall fix and match the docs to the live server --- docs/CONTEXT.md | 26 ++++++++++++++++---------- infra/server/README.md | 28 +++++++++++++++++++++++++--- infra/server/firewall.sh | 7 ++++++- 3 files changed, 47 insertions(+), 14 deletions(-) diff --git a/docs/CONTEXT.md b/docs/CONTEXT.md index 8423217..85a5f5e 100644 --- a/docs/CONTEXT.md +++ b/docs/CONTEXT.md @@ -123,8 +123,12 @@ encrypting the signing-secret column (we protect the backups instead), OpenAPI t - **Access:** personal users `devinder` and `abdikhani`, SSH keys only, no root login. Log in with `ssh relay` (alias in each person's `~/.ssh/config`). `administrator` is one.com's default account, kept as a backup. -- **Firewalls:** one.com's network firewall + ufw. ufw allows 22, 80, 443 and the k3s networks - 10.42.0.0/16 and 10.43.0.0/16. one.com's firewall allows 22, 80 and 443. +- **Firewalls:** two layers. one.com's network firewall (web panel) allows only TCP 22, 80 and + 443 over IPv4, and nothing over IPv6. ufw on the server allows 22, 80, 443 and the k3s + networks 10.42.0.0/16 and 10.43.0.0/16. ufw can't filter ports that Kubernetes publishes + itself (NodePorts, such as Traefik's 30795 and 32338), so only one.com's firewall blocks + those. It must never get an "All" rule again (it had one until 29 Oct). Check from outside: + `nc -z -G 3 81.88.23.71 30795` must fail. - **kubectl:** only on the server (`ssh relay`, then `kubectl …`). Port 6443 stays closed. - **Deploy path:** GitHub Actions (environment `production`, main only) logs in as the Linux user `deploy`, whose key can only start `/usr/local/bin/relay-deploy`. The script deploys a @@ -134,8 +138,9 @@ encrypting the signing-secret column (we protect the backups instead), OpenAPI t and `grafana` (the ops dashboard: metrics and logs) all point to the VPS. **No AAAA records**: the cluster is IPv4-only, and Let's Encrypt prefers IPv6 when it exists. - **HTTP → HTTPS:** `k8s/platform/traefik-config.yaml` (a HelmChartConfig) redirects port 80 to - 443 for every host with a 308, except `/.well-known/acme-challenge/`, so certificate checks still - work over plain HTTP. + 443 for every host with a permanent redirect (301 for GET, 308 for other methods, so a POST + keeps its method and body), except `/.well-known/acme-challenge/`, so certificate checks + still work over plain HTTP. ## Git workflow @@ -152,12 +157,13 @@ Phase 0 (server foundation) is done: VPS hardened, k3s, cert-manager with Let's issuers, and a temporary hello-world app on https://api.myrelay.tech with a trusted certificate. -Phase 1 (walking skeleton) is well under way. Done: HTTP → HTTPS redirect (1.1), DNS for cdn -and grafana (1.2), the API skeleton in `api/` with its Docker image (1.3–1.4, PR #11), the CI -pipeline that tests every PR and pushes the image from main (1.5, PR #12), and the deploy path -(1.6): a `deploy` user whose key can only run `relay-deploy`, and a `deployer` account limited -to namespace `relay`. Next: 1.7, the API manifests in `k8s/app/`, then 1.8, the first deploy -through the pipeline. +Phase 1 (walking skeleton) is nearly done. Merged: HTTP → HTTPS redirect (1.1), DNS for cdn +and grafana (1.2), the API skeleton in `api/` with its Docker image (1.3–1.4, #11), and the +deploy path (1.6, #13): a `deploy` user whose key can only run `relay-deploy`, and a +`deployer` account limited to namespace `relay`. In review, to be merged in this order: the CI +pipeline (1.5, #12), rollback to the last good commit (#14), the API manifests (1.7, #15) and +the deploy job (1.8, #16). Then `api.myrelay.tech` switches from the hello app to the API with +the first deploy through the pipeline. The plan in order: 1 walking skeleton · 2 database foundation · 3 core API · 4 worker · 5 dashboard on cdn (Abdikhani, in parallel) · 6 backups + Terraform (Abdikhani, in parallel) · diff --git a/infra/server/README.md b/infra/server/README.md index f749bcb..b7db845 100644 --- a/infra/server/README.md +++ b/infra/server/README.md @@ -10,8 +10,12 @@ run alone, never inside a pasted block. ## 1. Outside the server (web panels) -- **one.com:** a VPS with Ubuntu 26.04 (4 vCPU, 8 GB). Two-step login on the account. Network - firewall: allow TCP 22, 80 and 443. +- **one.com:** a VPS with Ubuntu 26.04 (4 vCPU, 8 GB). Two-step login on the account. +- **one.com network firewall:** exactly three rules, IPv4 TCP 22, 80 and 443. No `All` rule + and nothing for IPv6. ufw can't filter ports that Kubernetes publishes (NodePorts), so this + firewall is the only thing blocking them. When changing rules, add the new one before + deleting an old one, so SSH never drops. Check from outside: `nc -z -G 3 30795` + must fail, and 22, 80, 443 must answer. - **get.tech DNS:** A records `api`, `cdn` and `grafana` → the VPS's IPv4 address. No AAAA records: the cluster is IPv4-only, and Let's Encrypt prefers IPv6 when it exists. @@ -42,6 +46,24 @@ run alone, never inside a pasted block. `DEPLOY_HOST` and `DEPLOY_KNOWN_HOSTS` (the server's new host key). 3. Merge anything to `main` (or re-run the last pipeline): the app deploys itself. -## 5. Data (Phase 6) +## 5. GitHub settings (not in git; made with `gh`, stored by GitHub) + +- Ruleset `protect-main`: PR + 1 approval, approval after the last push, no force-push, no + deletion. Squash merges only. Required check `api-test`. +- Environment `production`: only `main` may use it. Secret `DEPLOY_SSH_KEY`, variables + `DEPLOY_HOST` and `DEPLOY_KNOWN_HOSTS`. +- GHCR package `relay-api`: public, so the cluster pulls it without a password. + +## 6. Data (Phase 6) Restore the newest database dump from S3. Until then, a lost server means lost data. + +k3s's own data (the cluster state) isn't backed up on purpose: everything in it is rebuilt +from git and these scripts. Only the database holds data that can't be rebuilt. + +## Reboots + +`unattended-upgrades` installs security updates every day but never reboots by itself +(`Automatic-Reboot` stays off, so a demo is never interrupted). After a kernel update +(`/var/run/reboot-required` exists), reboot by hand at a quiet time, never in the week of the +defense, and check that `/version` answers again afterwards. diff --git a/infra/server/firewall.sh b/infra/server/firewall.sh index 2624761..c5a34d4 100644 --- a/infra/server/firewall.sh +++ b/infra/server/firewall.sh @@ -1,6 +1,11 @@ #!/usr/bin/env bash # The server's own firewall, ufw (step 0.7). one.com's network firewall sits in front of it -# and is set in their web panel (TCP 22, 80, 443): two layers, so one mistake exposes nothing. +# and is set in their web panel: exactly IPv4 TCP 22, 80 and 443, no "All" rule, nothing for +# IPv6. Two layers, so one mistake exposes nothing. +# +# Limit: ufw can't filter ports that Kubernetes publishes itself (NodePorts such as Traefik's +# 30795/32338, and hostPorts). Kubernetes redirects that traffic before ufw's rules see it, +# so for those ports one.com's firewall is the only layer. # # Run on the VPS: sudo bash firewall.sh # Safe to run again: ufw skips rules it already has.