_____ ____  ___   ____________   _____ __________ _    ____________ 
  / ___// __ \/   | / ____/ ____/  / ___// ____/ __ \ |  / / ____/ __ \
  \__ \/ /_/ / /| |/ /   / __/     \__ \/ __/ / /_/ / | / / __/ / /_/ /
 ___/ / ____/ ___ / /___/ /___    ___/ / /___/ _, _/| |/ / /___/ _, _/ 
/____/_/   /_/  |_\____/_____/   /____/_____/_/ |_| |___/_____/_/ |_|  

Space Server

My completely self-hosted personal infrastructure. From an old laptop to a production VPS — mail, blog, portfolio, monitoring, and alerting, all dockerized.

Why this exists

I was tired of depending on third-party services for basic things like email and hosting. I wanted full control over my data and to learn how modern web infrastructure actually works. So I turned an old laptop into a home server and eventually migrated it to a VPS when the hardware couldn't keep up anymore.

This repo is the source of truth for that infrastructure — every service, every config, every lesson learned along the way.

What runs here

Current production: Hetzner VPS (8GB RAM, Intel Xeon, Ubuntu 24.04). 49 containers on one host. This repo manages the platform layer (edge, observability, mail, error pages) and the portfolio, blog and landing stacks; the rest are other projects that share the same Traefik. Of the subdomains below, 4 are public and 8 are admin UIs behind the WireGuard VPN.

Subdomains managed by this repo

Public

URL	Service
cativo.dev	Portfolio (Nuxt 4)
api.cativo.dev	Portfolio API (NestJS)
blog.cativo.dev	Ghost 5 blog
devi.cativo.dev	Hello Kitty landing

Admin UIs, VPN-only (internal-only middleware, see docs/runbooks/admin-vpn.md). Off the VPN they return 403.

URL	Service
mail.cativo.dev	Roundcube webmail
grafana.cativo.dev	Grafana — metrics + dashboards
prometheus.cativo.dev	Prometheus UI (+ basic auth)
alertmanager.cativo.dev	Alertmanager UI (+ basic auth)
dozzle.cativo.dev	Dozzle — live container logs (+ basic auth)
uptime.cativo.dev	Uptime Kuma — external probes
traefik.cativo.dev	Traefik dashboard (+ basic auth)
errors.cativo.dev	Bugsink — error tracking

status.cativo.dev redirects to the Uptime Kuma status page on uptime.cativo.dev, which sits behind the VPN gate, so off-VPN visitors currently get 403 after the redirect.

Plus SMTP/IMAPS on the standard mail ports.

Stack layers

Layer	What
Edge	Traefik v3.6 — automatic Let's Encrypt SSL, HTTP/3, WireGuard VPN gate on admin UIs (basic auth as a second layer on some), security headers, bcrypt hashes only
Mail	docker-mailserver (Postfix + Dovecot + SpamAssassin) with SPF/DKIM/DMARC; Roundcube webmail over STARTTLS; outbound via Resend SMTP relay (Hetzner blocks port 25)
Apps	Ghost 5 (MySQL backing it), Nuxt 4 + NestJS portfolio (the API has MariaDB + Redis), hello-kitty landing
Observability	Prometheus (30-day retention), node_exporter, cAdvisor, Alertmanager → Discord webhook (via alertmanager-discord adapter), Loki + Promtail for logs (7-day retention), Bugsink for application errors, Grafana with provisioned dashboards (Node Exporter Full, Traefik standalone, cAdvisor), Uptime Kuma for external probes, Dozzle for live container logs
Plumbing	docker-socket-proxy with minimal API surface (Swarm endpoints disabled), git-on-prod deploy (git pull && docker compose up -d)

Databases and caches (Ghost's MySQL, portfolio's MariaDB, portfolio's Redis) live on internal-only docker networks — they're not reachable from the edge.

Architecture

graph LR
    Internet["🌐 Internet"]
    Discord["💬 Discord<br/>#alerts"]
    Resend["📤 Resend<br/>(SMTP relay)"]
    StorageBox["💾 Hetzner<br/>Storage Box"]

    Internet -->|"80/443"| Traefik
    Internet -->|"25/465/587/993"| Mail

    subgraph Polaris2["Hetzner VPS · polaris2"]
        Traefik["Traefik v3.6<br/>(TLS termination)"]
        DockerProxy["docker-socket-proxy<br/>(read-only API)"]
        Traefik -.->|tcp:2375| DockerProxy

        subgraph Apps[" Apps "]
            Ghost["Ghost 5 + MySQL<br/>blog.cativo.dev"]
            Portfolio["Nuxt 4 + NestJS<br/>cativo.dev"]
            Mail["docker-mailserver<br/>mail.cativo.dev"]
            Webmail["Roundcube"]
        end

        subgraph Observability[" Observability "]
            Prom["Prometheus<br/>(30d retention)"]
            Node["node_exporter"]
            CAdv["cAdvisor"]
            Grafana["Grafana<br/>(provisioned)"]
            AM["Alertmanager"]
            AMDiscord["alertmanager-discord<br/>(adapter)"]
            Uptime["Uptime Kuma"]
            Dozzle["Dozzle<br/>(live logs)"]
        end

        Traefik --> Ghost & Portfolio & Webmail & Grafana & Prom & AM & Uptime & Dozzle
        Webmail -->|STARTTLS| Mail
        Mail -->|outbound| Resend
        Prom -->|scrapes| Node & CAdv & Traefik & AM
        Prom -->|alerts| AM
        AM --> AMDiscord
        AMDiscord -->|webhook| Discord
    end

    StorageBox -.->|"(planned)"| Polaris2

Loading

Screenshots

Node Exporter Full	Traefik Standalone
live host metrics — CPU 6%, RAM 29.1%, disk 36.4%, uptime 3.8 weeks	HTTP code distribution, requests per entrypoint, Apdex, top slow services

See docs/screenshots/ for the full set + capture guide.

Production stories

A few real incidents from running this. Each one taught me something I now apply by default.

🔐 The basic-auth hash that was sitting in the public repo

While preparing to make the production server a proper git working copy, I discovered the apr1 (Apache MD5) hash protecting traefik.cativo.dev, prometheus.cativo.dev, and dozzle.cativo.dev had been committed to the public repo for three weeks. The $$ in the YAML was Docker Compose env-var escape syntax — to Traefik's file provider it reads as a single $, so the hash was real.

Resolved by rotating to bcrypt, dropping the auth file (and the mail account files, same risk) from tracking, adding .example templates with placeholder hashes, and switching Traefik's volume bind from file-scope to directory-scope so future sed -i edits don't break the bind mount via inode replacement. The new credential lives in a 600-perm file outside any repo until it makes it to a password manager.

🌐 The network that worked by accident for three weeks

Mail routing broke right after a Traefik recreate. Turned out mail-server/docker-compose.yml referenced web: external: true (a literal network name web) while every other stack — ghost, portfolio, Dozzle, Uptime Kuma — used the project-prefixed space-server_web. The routing had been held together by a manual docker network connect web traefik that someone (past me) had run out-of-band; the Traefik recreate dropped that side-effect and exposed the latent mismatch. Fix was to point the mail-server stack's network reference at the canonical space-server_web via Compose's name: override, then delete the orphan web network. Now there's exactly one network and zero magic.

📧 Webmail "login failed" after a clean container recreate

After fixing the network issue, Roundcube started rejecting every login with dovecot's disable_plaintext_auth=yes kicking in even though both containers shared the docker network. docker-mailserver doesn't auto-populate dovecot's login_trusted_networks, so plaintext IMAP from webmail was being refused. Rather than poke holes in dovecot's defaults, I switched Roundcube's imap_host and smtp_host to tls://mail (STARTTLS) in both the env vars and the inline entrypoint heredoc. Cert is self-signed for internal traffic, and verify_peer = false was already set. Verified end-to-end with openssl s_client -starttls imap returning OK Pre-login.

More incidents and their fixes live in commit messages — they're all fix(scope): description (Fxx) and reference the improvement plan.

How I work

The IMPROVEMENT-PLAN.md tracks every finding from a structured architecture review: P0 critical bugs and security, P1 resilience and observability, P2 hardening, P3 quality-of-life. Items get checked off with a one-line note explaining what shipped and a reference back to the commit. It's the same artifact I'd expect to keep on a small team.

Documentation

• ARCHITECTURE.md — Detailed topology, request flows (HTTPS, mail, alerts), network boundaries, persistence layer, deployment flow
• docs/decisions/ — Architecture Decision Records: single source of truth, observability adoption, SMTP relay choice. Nygard format, current index in docs/decisions/README.md
• docs/runbooks/ — Step-by-step operational procedures (status page setup, adding a new service, container log rotation)
• docs/screenshots/ — Visual evidence of the stack (Grafana dashboards, Discord alerts, Traefik routing)
• IMPROVEMENT-PLAN.md — Active findings roadmap with status per item
• CHANGELOG.md — Keep-a-Changelog format

Tech stack

• Reverse proxy: Traefik v3.6 (Let's Encrypt, file + docker providers, VPN allowlist + basic auth + security-headers middleware)
• Mail: docker-mailserver, Roundcube (STARTTLS internal), Resend SMTP relay (outbound)
• Monitoring: Prometheus, Alertmanager, node_exporter, cAdvisor, Loki + Promtail, Bugsink, Grafana with provisioning, alertmanager-discord sidecar
• Apps: Ghost 5, Nuxt 4 frontend, NestJS API, MySQL/MariaDB, Redis
• Ops: Docker Compose v2, git-on-prod, Discord alerting

Quick start

git clone https://github.com/cativo23/space-server.git
cd space-server

# Configure secrets (none are committed)
cp .env.example .env
cp traefik/dynamic/auth.yml.example traefik/dynamic/auth.yml
# Edit .env and auth.yml with your own values
# Generate a bcrypt hash for traefik auth:
#   docker run --rm -i httpd:2-alpine htpasswd -niB admin <<< 'your-strong-password'

# Start the stack
docker compose up -d

Important: the mail server needs DNS (MX/SPF/DKIM/DMARC). If your provider blocks outbound port 25 (Hetzner does), configure the RELAY_* env vars in mail-server/.env to route through Resend / Mailgun / Postmark / etc.

Things I learned

What worked:

• Automated migration scripts pay for themselves the moment you need to recover
• Traefik + Let's Encrypt makes SSL a non-issue across a dozen subdomains
• Docker Compose is enough for a single-node setup — Kubernetes would be cargo culting
• Provisioning Grafana from files in git makes dashboards reproducible
• Conventional commit messages with the why in the body pay off when you git log six months later

What didn't work:

• Committing credential templates with real-looking values (any plausible-looking hash is assumed exposed once it's public)
• File-scope bind mounts in Docker — sed -i replaces the inode and the container keeps reading the old one
• Mixing project-prefixed and literal network names across compose files
• Trusting that "PERMIT_DOCKER=network" propagates to every protocol in docker-mailserver (it doesn't; STARTTLS instead of plaintext is the right answer anyway)

Roadmap

Tracked in IMPROVEMENT-PLAN.md. Currently open:

• F8 — DNS-01 challenge (Cloudflare) for a wildcard cert
• F4 — automated backups via restic → Hetzner Storage Box
• F16 — SOPS/age for encrypted secrets in git
• F46 — offsite dead-man switch so a full-host outage still alerts Discord
• Eventually — Ansible playbook for full reproducibility from a fresh VPS

Image retention

polaris2 is shared by every project's deploy, and each release pulls a new image tag that nothing used to delete. One host-level job owns the cleanup for all stacks, including ones added later; no deploy workflow has to remember it.

scripts/docker-retention.sh runs daily at 04:30 UTC from systemd/docker-retention.{service,timer}. It is dry-run by default and deletes only with --apply. It keeps an image when:

Kept because	Rule
in-use	any container uses it, running or stopped (docker ps -a)
compose-ref	an image: line in a compose file under the stacks' directories names it (${VAR:-default} resolves to its default)
protected	a reference matches PROTECT_REGEX (set it during a rollback window)
newest-N	it is among the KEEP_LAST (default 3) most recently pulled images of its repository
too-young	it was pulled less than MIN_AGE_HOURS (default 72) ago

Things that are easy to get wrong:

• KEEP_LAST counts the image in use: 3 is the current image plus 2 rollbacks. A repository that only ever has a :latest tag keeps no rollback copy, because the next pull replaces it.
• Everything else is removed with docker image rm <reference>, never -f, so the daemon still refuses an image a container uses. Usage, pull time and the reference itself are checked again right before each removal, so an image a deploy or a rollback pull touched since the plan is skipped, and so is a tag that was re-pushed and now names a different image (a sliver of time remains between that last look and the removal).
• It also prunes dangling images older than MIN_AGE_HOURS and build cache unused for 7 days or above 2 GB. Docker documents the dangling filter (until=) as an image creation time, not a pull time; how it behaves on this host's containerd image store has not been verified, so treat that age as approximate. It never runs system prune or image prune -a, and never touches volumes, networks or containers.
• It stops instead of guessing: exit 70 when docker ps, inspect or image ls/image inspect fail or containers exist but none resolved to an image, 75 when another run holds the lock or a container is created/restarting/removing (a deploy in flight), 64 for a bad flag, setting, COMPOSE_ROOTS entry or PROTECT_REGEX, 1 when a removal or a prune failed.
• Repositories that no container uses are listed as NO-CONTAINER for their owner to decide; they are not removed.
• Nothing alerts when the unit fails. Check systemctl status docker-retention.service and journalctl -u docker-retention after changing anything here.

Install once on the host, as the deploy user (the script and the units are copies, not links into the checkout):

cd ~/space-server
sudo install -m 0755 scripts/docker-retention.sh /usr/local/sbin/docker-retention
sudo install -m 0644 systemd/docker-retention.service systemd/docker-retention.timer /etc/systemd/system/
systemd-analyze verify /etc/systemd/system/docker-retention.{service,timer}
sudo systemctl daemon-reload && sudo systemctl enable --now docker-retention.timer
sudo systemctl start docker-retention.service    # one dry-run now
journalctl -u docker-retention -n 80

After a git pull, repeat the install lines to update the copies. Host-only overrides go in /etc/default/docker-retention (see systemd/docker-retention.default.example).

Going from dry-run to apply. The unit ships with --dry-run. After a week of plans in the journal look right (every run exit 0, every REMOVE a superseded versioned tag), change that word to --apply in a PR, then on the host reinstall the unit, sudo systemctl daemon-reload, and run the first apply with sudo systemctl start docker-retention.service, never by calling the script with sudo: the unit supplies the deploy user and the private lock directory, which a hand-run --apply does not have (it refuses to guess a lock path). If you must run --apply by hand, give it its own DOCKER_RETENTION_LOCK; that lock does not exclude the timer, because the unit's lock directory exists only while the unit runs. systemd itself never runs the oneshot unit twice at once, so the rule stays: go through systemctl start.

Rollback. sudo systemctl disable --now docker-retention.timer stops it. An image removed by mistake comes back with docker pull <reference> only if it came from a registry. An image built on the host cannot be pulled back; the job never removes one that is the only tag of its repository (newest-N keeps it), which is the case for locally built :latest images, but a rebuild is the only way back if one is removed anyway.

Tests: bats tests/docker-retention.bats (bats-core 1.7 or newer; CI installs the pinned v1.13.0) runs the script against a stub docker and checks, among other things, that dry-run never issues a destructive command and that --apply issues only the expected ones. CI also runs shellcheck on the script and the tests, and systemd-analyze verify on the units.

Container log rotation

Docker keeps container logs forever by default. docker/daemon.json caps them at 10 MB x 5 files per container and enables live-restore. It is not applied by any deploy: it goes onto the host by hand, in a specific order, and existing containers keep their old log config until recreated. Follow docs/runbooks/container-log-rotation.md, including its rollback, which does not mean deleting the file. CI validates the file with dockerd --validate and with scripts/check-daemon-json.sh, because dockerd --validate does not check the log option values.

Repo structure

space-server/
├── docker-compose.yml          # Root stack: edge + observability
├── mail-server/                # Mail (Postfix/Dovecot + Roundcube)
├── traefik/                    # Reverse proxy + Let's Encrypt
├── prometheus/                 # Scrape config + alert rules
├── alertmanager/               # Discord routing + inhibit rules
├── grafana/                    # Provisioned datasource + dashboards
├── dozzle/                     # Live container logs UI
├── uptime-kuma/                # External uptime probes
├── docker/                     # daemon.json (log rotation, live-restore)
├── scripts/                    # Migration automation + docker-retention.sh, check-daemon-json.sh
├── systemd/                    # Host timers (docker-retention)
├── tests/                      # bats suites (docker-retention with a docker stub, daemon-json)
├── IMPROVEMENT-PLAN.md         # Architecture roadmap (active)
└── CHANGELOG.md                # Keep-a-Changelog format

Requirements

• VPS with at least 4GB RAM (8GB recommended)
• Domain with DNS access (Cloudflare works great; pure DNS-only mode for mail records)
• Docker + Docker Compose v2
• An SMTP relay account if your provider blocks outbound port 25

Contributing

Mostly a personal project, but if you spot a bug or want to suggest something, open an issue. See CONTRIBUTING.md.

License

MIT — do whatever you want with this. If it helps you, great.