Skip to content

infra: capture the server's base setup, a rebuild runbook and the firewall fix #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
devinder-dev merged 2 commits into from
Oct 29, 2026
Merged

infra: capture the server's base setup, a rebuild runbook and the firewall fix #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
adf8849
infra: capture the server's base setup and a rebuild runbook
devinder-dev Oct 29, 2026
c8a667e
docs: record the firewall fix and match the docs to the live server
devinder-dev Oct 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 16 additions & 10 deletions docs/CONTEXT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,8 +123,12 @@ encrypting the signing-secret column (we protect the backups instead), OpenAPI t
- **Access:** personal users `devinder` and `abdikhani`, SSH keys only, no root login.
Log in with `ssh relay` (alias in each person's `~/.ssh/config`). `administrator` is
one.com's default account, kept as a backup.
- **Firewalls:** one.com's network firewall + ufw. ufw allows 22, 80, 443 and the k3s networks
10.42.0.0/16 and 10.43.0.0/16. one.com's firewall allows 22, 80 and 443.
- **Firewalls:** two layers. one.com's network firewall (web panel) allows only TCP 22, 80 and
443 over IPv4, and nothing over IPv6. ufw on the server allows 22, 80, 443 and the k3s
networks 10.42.0.0/16 and 10.43.0.0/16. ufw can't filter ports that Kubernetes publishes
itself (NodePorts, such as Traefik's 30795 and 32338), so only one.com's firewall blocks
those. It must never get an "All" rule again (it had one until 29 Oct). Check from outside:
`nc -z -G 3 81.88.23.71 30795` must fail.
- **kubectl:** only on the server (`ssh relay`, then `kubectl …`). Port 6443 stays closed.
- **Deploy path:** GitHub Actions (environment `production`, main only) logs in as the Linux
user `deploy`, whose key can only start `/usr/local/bin/relay-deploy`. The script deploys a
Expand All @@ -134,8 +138,9 @@ encrypting the signing-secret column (we protect the backups instead), OpenAPI t
and `grafana` (the ops dashboard: metrics and logs) all point to the VPS. **No AAAA records**:
the cluster is IPv4-only, and Let's Encrypt prefers IPv6 when it exists.
- **HTTP → HTTPS:** `k8s/platform/traefik-config.yaml` (a HelmChartConfig) redirects port 80 to
443 for every host with a 308, except `/.well-known/acme-challenge/`, so certificate checks still
work over plain HTTP.
443 for every host with a permanent redirect (301 for GET, 308 for other methods, so a POST
keeps its method and body), except `/.well-known/acme-challenge/`, so certificate checks
still work over plain HTTP.

## Git workflow

Expand All @@ -152,12 +157,13 @@ Phase 0 (server foundation) is done: VPS hardened, k3s, cert-manager with Let's
issuers, and a temporary hello-world app on https://api.myrelay.tech with a trusted
certificate.

Phase 1 (walking skeleton) is well under way. Done: HTTP → HTTPS redirect (1.1), DNS for cdn
and grafana (1.2), the API skeleton in `api/` with its Docker image (1.3–1.4, PR #11), the CI
pipeline that tests every PR and pushes the image from main (1.5, PR #12), and the deploy path
(1.6): a `deploy` user whose key can only run `relay-deploy`, and a `deployer` account limited
to namespace `relay`. Next: 1.7, the API manifests in `k8s/app/`, then 1.8, the first deploy
through the pipeline.
Phase 1 (walking skeleton) is nearly done. Merged: HTTP → HTTPS redirect (1.1), DNS for cdn
and grafana (1.2), the API skeleton in `api/` with its Docker image (1.3–1.4, #11), and the
deploy path (1.6, #13): a `deploy` user whose key can only run `relay-deploy`, and a
`deployer` account limited to namespace `relay`. In review, to be merged in this order: the CI
pipeline (1.5, #12), rollback to the last good commit (#14), the API manifests (1.7, #15) and
the deploy job (1.8, #16). Then `api.myrelay.tech` switches from the hello app to the API with
the first deploy through the pipeline.

The plan in order: 1 walking skeleton · 2 database foundation · 3 core API · 4 worker ·
5 dashboard on cdn (Abdikhani, in parallel) · 6 backups + Terraform (Abdikhani, in parallel) ·
Expand Down
69 changes: 69 additions & 0 deletions infra/server/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# Rebuilding the server

How to get from a fresh VPS back to a running cluster, in order. Every setting that lives on the
server is in this folder or in `k8s/`. Only the web panels (one.com, get.tech) and GitHub's
settings are outside git.

Copy a file to the server with `ssh relay 'cat > /tmp/<name>' < infra/server/<name>`, then
run the command shown on the server. Commands that ask questions (`adduser`, the first `sudo`)
run alone, never inside a pasted block.

## 1. Outside the server (web panels)

- **one.com:** a VPS with Ubuntu 26.04 (4 vCPU, 8 GB). Two-step login on the account.
- **one.com network firewall:** exactly three rules, IPv4 TCP 22, 80 and 443. No `All` rule
and nothing for IPv6. ufw can't filter ports that Kubernetes publishes (NodePorts), so this
firewall is the only thing blocking them. When changing rules, add the new one before
deleting an old one, so SSH never drops. Check from outside: `nc -z -G 3 <vps-ip> 30795`

@AbdikhaniAbdirashid AbdikhaniAbdirashid Oct 29, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test can never fail, so it can never catch a regression.

-G isn't a netcat-openbsd flag (that's the BSD/macOS nc). On Ubuntu it's -w. With -G, nc exits on a usage error before it connects to anything — and the runbook's criterion is "must fail", so it looks like it passed.

I ran it against an open port to be sure:

nc -z -G 3 <vps-ip> 443   exit=1    <- 443 is open
nc -z -w 3 <vps-ip> 443   exit=0    <- correct flag sees it

So nc -z -G 3 <vps-ip> 30795 would report "blocked" even if that NodePort were wide open again.

That's the same shape as the lesson this PR records — "in 0.12 we tested that 80/443 worked, but never that other ports were blocked" — so I'd rather we fix it here than inherit a check we trust and shouldn't.

-w works on both the server and my laptop:

nc -z -w 3 <vps-ip> 30795   # must fail  (exit 1)
nc -z -w 3 <vps-ip> 443     # must work  (exit 0)

Worth running both directions, not just the negative one — a check that only ever asserts failure is how we got here.

must fail, and 22, 80, 443 must answer.
- **get.tech DNS:** A records `api`, `cdn` and `grafana` → the VPS's IPv4 address. No AAAA
records: the cluster is IPv4-only, and Let's Encrypt prefers IPv6 when it exists.

## 2. Base system (steps 0.3–0.7)

| What | How |
|---|---|
| Personal users | `sudo adduser <name>` (alone), `sudo usermod -aG sudo <name>`, then `ssh-copy-id` from each laptop |
| Updates | `sudo apt update && sudo apt full-upgrade -y`, then `sudo dpkg-reconfigure -plow unattended-upgrades` (daily security updates) |
| SSH | `sudo install -m 644 /tmp/sshd-hardening.conf /etc/ssh/sshd_config.d/00-hardening.conf && sudo sshd -t && sudo systemctl restart ssh`. Test a new login in a second terminal before closing the first |
| Firewall | `sudo bash /tmp/firewall.sh` |

## 3. Kubernetes (steps 0.8–0.11)

| What | How |
|---|---|
| k3s settings | `sudo install -D -m 644 /tmp/k3s-config.yaml /etc/rancher/k3s/config.yaml`, **before** installing k3s |
| k3s v1.36.5 | Download `https://get.k3s.io` to a file, read it, then run it with `INSTALL_K3S_VERSION=v1.36.5+k3s1` |
| kubectl for each admin | `mkdir -p ~/.kube && sudo install -m 600 -o "$USER" /etc/rancher/k3s/k3s.yaml ~/.kube/config`. It's an admin password: never copy it off the server |
| cert-manager v1.21.2 | `kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.21.2/cert-manager.yaml` |
| Platform files | `k8s/platform/*.yaml`, applied by hand (see [`k8s/README.md`](../../k8s/README.md)) |

## 4. The deploy path (step 1.6)

1. `setup-deploy-user.sh` and `relay-deploy.sh`: see [`k8s/README.md`](../../k8s/README.md).
2. GitHub environment `production` (only `main` may use it): the secret `DEPLOY_SSH_KEY`
(the new key's private half, with `gh secret set … --env production < key`), and the variables
`DEPLOY_HOST` and `DEPLOY_KNOWN_HOSTS` (the server's new host key).
3. Merge anything to `main` (or re-run the last pipeline): the app deploys itself.

## 5. GitHub settings (not in git; made with `gh`, stored by GitHub)

- Ruleset `protect-main`: PR + 1 approval, approval after the last push, no force-push, no
deletion. Squash merges only. Required check `api-test`.

@AbdikhaniAbdirashid AbdikhaniAbdirashid Oct 29, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two of these don't match what's configured right now. I read the ruleset back with gh api repos/:owner/:repo/rulesets:

Documented Actual
approval after the last push dismiss_stale_reviews_on_push: false — an approval survives later pushes
Required check api-test no required status checks at all

The api-test one I think is deliberate: #16's merge plan says to add it last, after #14–#16 are in, so none of them needs a fresh approval. So this line describes the target, not today.

The stale-approval one I haven't seen planned anywhere, and it's the one that worries me more. Right now I can approve a PR and you can push anything to it afterwards and still merge — which is roughly what happened with my #9, where your review sits on a commit two pushes old.

Could this section say which settings are current and which are the end state? A rebuild runbook that quietly makes us stricter is fine; one that claims protections we don't have is worse than saying nothing, because we'd both repeat the claim at the defense.

@AbdikhaniAbdirashid AbdikhaniAbdirashid Oct 29, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correcting myself: the first half of this is wrong, and your doc was right.

I checked dismiss_stale_reviews_on_push and found it false, and concluded the "approval after the last push" line was aspirational. I'd checked the wrong setting. The full pull_request rule is:

"dismiss_stale_reviews_on_push": false,
"require_last_push_approval": true,
"require_extra_approval_for_unattributed_changes": true,
"required_approving_review_count": 1

require_last_push_approval: true is the one that matters. An existing approval isn't deleted on push, but the PR still can't merge until someone who isn't the pusher approves the latest push. Net effect is exactly what the runbook says.

I found out the hard way: #16 went to REVIEW_REQUIRED the moment you pushed d00063d, even with my approval of 67b29dd still on it. The protection works, and I'd just told you it didn't exist.

Sorry — and specifically withdraw the line about #9. Nothing could have been pushed past my approval there.

The second half still stands. Required status checks are genuinely empty (required checks: []), so the api-test line in the runbook is the target state rather than today's. That's deliberate per #16's merge plan, which adds it last. Worth marking which of the two it is, but the stale-approval row needs no change at all.

- Environment `production`: only `main` may use it. Secret `DEPLOY_SSH_KEY`, variables
`DEPLOY_HOST` and `DEPLOY_KNOWN_HOSTS`.
- GHCR package `relay-api`: public, so the cluster pulls it without a password.

## 6. Data (Phase 6)

Restore the newest database dump from S3. Until then, a lost server means lost data.

k3s's own data (the cluster state) isn't backed up on purpose: everything in it is rebuilt
from git and these scripts. Only the database holds data that can't be rebuilt.

## Reboots

`unattended-upgrades` installs security updates every day but never reboots by itself
(`Automatic-Reboot` stays off, so a demo is never interrupted). After a kernel update
(`/var/run/reboot-required` exists), reboot by hand at a quiet time, never in the week of the
defense, and check that `/version` answers again afterwards.
27 changes: 27 additions & 0 deletions infra/server/firewall.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#!/usr/bin/env bash
# The server's own firewall, ufw (step 0.7). one.com's network firewall sits in front of it
# and is set in their web panel: exactly IPv4 TCP 22, 80 and 443, no "All" rule, nothing for
# IPv6. Two layers, so one mistake exposes nothing.
#
# Limit: ufw can't filter ports that Kubernetes publishes itself (NodePorts such as Traefik's
# 30795/32338, and hostPorts). Kubernetes redirects that traffic before ufw's rules see it,
# so for those ports one.com's firewall is the only layer.
#
# Run on the VPS: sudo bash firewall.sh
# Safe to run again: ufw skips rules it already has.
set -euo pipefail

ufw default deny incoming # everything not listed below is dropped
ufw default allow outgoing

ufw allow 22/tcp # SSH, keys only
ufw allow 80/tcp # HTTP: Let's Encrypt's check; everything else redirects to HTTPS
ufw allow 443/tcp # HTTPS

# k3s's own networks: pods (10.42) and Services (10.43) talk to each other through the host.
ufw allow from 10.42.0.0/16 to any
ufw allow from 10.43.0.0/16 to any

# Never opened: 5432 (Postgres) and 6443 (the Kubernetes API).
ufw --force enable
ufw status verbose
6 changes: 6 additions & 0 deletions infra/server/k3s-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# k3s settings (step 0.8). Installed as /etc/rancher/k3s/config.yaml BEFORE installing k3s,
# so the setting is there from the first start.
# Encrypts Kubernetes Secrets at rest. By default they're only base64, which anyone who can
# read the disk can decode. Limit: the key is on the same disk, so a copy of the whole disk
# can still decrypt them.
secrets-encryption: true
6 changes: 6 additions & 0 deletions infra/server/sshd-hardening.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# SSH hardening (step 0.6). Installed as /etc/ssh/sshd_config.d/00-hardening.conf.
# sshd reads these files in alphabetical order and keeps the FIRST value it sees, so "00-"
# makes ours win over one.com's 50-cloud-init.conf, which turns passwords on.
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no