Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 36 additions & 6 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ AUDIT_DB_NAME=backup_audit
AUDIT_DB_USER=audit_user
AUDIT_DB_PASSWORD=audit_secret

# Hetzner Storage Box (shared)
# Hetzner Storage Box (shared) — only required when a project uses storage.type: sftp
HETZNER_SSH_HOST=u123456.your-storagebox.de
HETZNER_SSH_USER=u123456
HETZNER_SSH_PORT=23
Expand All @@ -20,6 +20,30 @@ HETZNER_SSH_KEY_PATH=/home/node/.ssh/id_ed25519
# Restic (global defaults, overridable per project)
RESTIC_PASSWORD=default-restic-repo-password

# ── Storage backend credentials ───────────────────────────────────────────────
# Referenced from projects.yml via ${} under storage.config. Only set what you use.

# Backblaze B2 / S3-compatible (storage.type: s3) — the recommended cloud option.
# Works for Backblaze B2, Wasabi, Cloudflare R2, MinIO, and AWS S3.
# Left uncommented because projects-example.yml references them: every ${} is resolved
# for all projects at load time, so an unset one fails the whole config load.
B2_ENDPOINT=https://s3.eu-central-003.backblazeb2.com
B2_KEY_ID=your-application-key-id
B2_APP_KEY=your-application-key

# Backblaze B2 native API (storage.type: b2). Restic recommends the S3 API above instead.
# B2_ACCOUNT_ID=your-account-id
# B2_ACCOUNT_KEY=your-account-key

# rclone (storage.type: rclone) — Google Drive, OneDrive, Dropbox.
# Authorize the remote on a machine with a browser, then paste it in:
# rclone authorize "drive" # on your laptop
# docker exec -it backupctl rclone config # paste the token here
# The remote name becomes the repository prefix, e.g. repository: gdrive:backups/myproject
# Defaults to /home/node/.config/rclone/rclone.conf, which the compose file mounts
# read-write from ./rclone-config — rclone rewrites refreshed OAuth tokens there.
# RCLONE_CONFIG_PATH=/home/node/.config/rclone/rclone.conf

# Global fallback notification (slack | email | webhook | none)
NOTIFICATION_TYPE=slack
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/DEFAULT/WEBHOOK/URL
Expand Down Expand Up @@ -60,12 +84,18 @@ LOG_MAX_FILES=5
# GPG keys directory (auto-imported on startup)
GPG_KEYS_DIR=/app/gpg-keys

# Heartbeat monitoring (optional — only if using push monitors like Uptime Kuma)
# UPTIME_KUMA_BASE_URL=https://kuma.example.com
# Heartbeat monitoring — only needed if a project has a monitor block, which the
# example projects do. Drop it along with those blocks if you don't use Uptime Kuma.
UPTIME_KUMA_BASE_URL=https://kuma.example.com

# Project DB passwords (referenced in projects.yml via ${})
VINSWARE_DB_PASSWORD=secret
VINSWARE_RESTIC_PASSWORD=restic-secret
# Project secrets (referenced in projects.yml via ${})
# Every ${} in projects.yml is resolved at load time for all projects, including
# disabled ones — an unset variable fails the whole config load, not just that project.
VINELAB_DB_PASSWORD=secret
VINELAB_RESTIC_PASSWORD=restic-secret
PROJECTX_DB_PASSWORD=secret
PROJECTY_DB_PASSWORD=secret
PROJECTZ_DB_PASSWORD=secret
PROJECTZ_RESTIC_PASSWORD=restic-secret
EMAIL_PASSWORD=email-secret
SMTP_PASSWORD=smtp-secret
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ config/projects.yml
# SSH & GPG keys
ssh-keys/
gpg-keys/
rclone-config/

# Backup artifacts
*.sql.gz
Expand Down
24 changes: 14 additions & 10 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Backup orchestration for databases, files, or both. Database-agnostic, NestJS 11
- **Entry points**: `src/main.ts` (HTTP), `src/cli.ts` (CLI via nest-commander)
- **Config**: `config/projects.yml` (per-project YAML) + `.env` (global secrets/defaults)
- **Audit DB**: PostgreSQL 16 via TypeORM with explicit migrations (separate container)
- **Remote storage**: Restic over SFTP to Hetzner Storage Box
- **Remote storage**: Restic, backend per project — sftp (Hetzner Storage Box), s3 (Backblaze B2/Wasabi/R2/MinIO/AWS), b2, rclone (Drive/OneDrive/Dropbox), local

## Git Workflow

Expand Down Expand Up @@ -112,7 +112,7 @@ src/
│ │ ├── infrastructure/ # Adapters implementing ports
│ │ │ ├── adapters/
│ │ │ │ ├── dumpers/ # postgres, mysql, mongo
│ │ │ │ ├── storage/ # restic + factory + tagging
│ │ │ │ ├── storage/ # restic adapter + factory + backends/ (per-type resolvers)
│ │ │ │ ├── encryptors/ # gpg + key manager
│ │ │ │ ├── cleanup/ # file cleanup
│ │ │ │ ├── hooks/ # shell hook executor
Expand Down Expand Up @@ -285,6 +285,8 @@ Modules may only import another module's **`application/ports/`** or **`domain/`
| Ports in `application/ports/` (not `domain/`) | Ports define outbound contracts; application layer owns the orchestration interface |
| `DumperRegistry` + `NotifierRegistry` | Dynamic adapter resolution by project config type |
| `BackupLockPort` — file-based `.lock` | Survives crashes, visible on disk, cleaned on startup recovery |
| Storage backend per project (`storage.type`) | `ResticBackendRegistry` resolves a `StorageConfig` to a restic repository string + env. Adapter stays backend-neutral; SSH lives only in the sftp resolver |
| `/health/live` vs `/health` | Docker HEALTHCHECK uses `/health/live` (container-local). `/health` probes remote storage, cached for `HEALTH_STORAGE_CHECK_TTL_SECONDS` (default 300) — a remote outage must not restart the container |
| `AuditLogPort` — `startRun`/`trackProgress`/`finishRun` | Real-time progress visibility + crash detection via orphaned records |
| `FallbackWriterPort` — JSONL format | Append-only, replayed on startup. Backup success never lost to infra failure |
| TypeORM entities as `*.record.ts` | Infrastructure concern, named "record" not "entity" to avoid DDD confusion |
Expand All @@ -310,9 +312,10 @@ Modules may only import another module's **`application/ports/`** or **`domain/`
3. Secrets always in `.env`, referenced via `${}` in YAML
4. Missing `notification` → global `NOTIFICATION_TYPE` + config from `.env`
5. Missing `encryption` → global `ENCRYPTION_ENABLED` / `ENCRYPTION_TYPE` / `GPG_RECIPIENT`
6. Missing `restic.password` → global `RESTIC_PASSWORD`
7. `compression.enabled` defaults to `true` (always compress)
8. Config changes require explicit `backupctl config reload` — no hot-reload
6. Missing `storage.password` → global `RESTIC_PASSWORD`
7. Legacy `restic:` block → read as `storage:` with `type: sftp` (deprecated, warns on load). Both keys present is a validation error
8. `compression.enabled` defaults to `true` (always compress)
9. Config changes require explicit `backupctl config reload` — no hot-reload

## Tech Stack

Expand All @@ -328,7 +331,7 @@ Modules may only import another module's **`application/ports/`** or **`domain/`
| Logging | Winston (nest-winston) with rotation |
| Testing | Jest |
| Container | Docker + Docker Compose |
| Remote storage | Restic → Hetzner Storage Box (SFTP) |
| Remote storage | Restic → sftp / s3 / b2 / rclone / local |
| Encryption | GPG |

## Development Commands
Expand Down Expand Up @@ -364,7 +367,7 @@ scripts/backupctl-manage.sh check # validate prerequisites

# Inside container
docker exec backupctl node dist/cli.js health
docker exec backupctl node dist/cli.js run vinsware --dry-run
docker exec backupctl node dist/cli.js run vinelab --dry-run

# Migrations — dev (manual via scripts/dev.sh)
scripts/dev.sh migrate:run # run pending
Expand Down Expand Up @@ -574,7 +577,7 @@ Explain **why**, not obvious **what**. No comments on self-evident code.
|---------|-------------|
| `run <project> [--all] [--dry-run]` | Trigger backup or simulate |
| `status [project] [--last n]` | Backup status (shows current_stage) |
| `health` | Audit DB, restic repos, disk (`HEALTH_DISK_MIN_FREE_GB`), SSH |
| `health` | Audit DB, disk (`HEALTH_DISK_MIN_FREE_GB`), per-project storage reachability |
| `restore <project> <snap> <path> [--only db/assets] [--decompress] [--guide]` | Restore + guidance |
| `snapshots <project> [--last n]` | List snapshots with tags |
| `prune <project> / --all` | Manual restic prune |
Expand All @@ -599,10 +602,10 @@ Explain **why**, not obvious **what**. No comments on self-evident code.
## Docker

Two containers via `docker-compose.yml`:
- `backupctl` — Node.js 20 Alpine + database clients + restic + GPG
- `backupctl` — Node.js 20 Alpine + database clients + restic + rclone + GPG
- `backupctl-audit-db` — PostgreSQL 16 Alpine

Volumes: `${BACKUP_BASE_DIR}`, `./config:ro`, `./ssh-keys:ro`, `./gpg-keys:ro`, asset paths
Volumes: `${BACKUP_BASE_DIR}`, `./config:ro`, `./ssh-keys:ro`, `./gpg-keys:ro`, `./rclone-config` (read-write — rclone rewrites refreshed OAuth tokens), asset paths

Host scripts: `scripts/backupctl-manage.sh` (prod), `scripts/dev.sh` (dev)

Expand All @@ -611,6 +614,7 @@ Host scripts: `scripts/backupctl-manage.sh` (prod), `scripts/dev.sh` (dev)
- `.env` (secrets)
- `ssh-keys/` (SSH private keys)
- `gpg-keys/`
- `rclone-config/` (rclone OAuth tokens)
- `node_modules/`, `dist/`
- `*.sql.gz`, `*.gpg` (backup artifacts)
- `*.lock` (backup lock files)
Expand Down
12 changes: 8 additions & 4 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ RUN apk upgrade --no-cache \
openssh-client \
gnupg \
fuse3 \
rclone \
docker-cli \
curl \
tini \
Expand All @@ -71,9 +72,10 @@ RUN mkdir -p /data/backups/.logs /data/backups/.fallback-audit \
&& chown -R node:node /data/backups

# Run as non-root — SSH keys mounted to /home/node/.ssh
RUN mkdir -p /home/node/.ssh /home/node/.gnupg \
&& chmod 700 /home/node/.gnupg \
&& chown -R node:node /home/node/.ssh /home/node/.gnupg
# rclone rewrites refreshed OAuth tokens into its config, so .config/rclone must stay writable.
RUN mkdir -p /home/node/.ssh /home/node/.gnupg /home/node/.config/rclone \
&& chmod 700 /home/node/.gnupg /home/node/.config/rclone \
&& chown -R node:node /home/node/.ssh /home/node/.gnupg /home/node/.config

WORKDIR /app
ENV NODE_ENV=production
Expand All @@ -86,7 +88,9 @@ RUN chmod +x /usr/local/bin/docker-entrypoint.sh \
USER node

EXPOSE 3100
# /health/live is container-local only. /health additionally probes remote storage,
# which must not restart the container when a cloud backend has an outage.
HEALTHCHECK --interval=30s --timeout=5s --start-period=60s --retries=3 \
CMD curl -f http://localhost:${APP_PORT:-3100}/health || exit 1
CMD curl -f http://localhost:${APP_PORT:-3100}/health/live || exit 1
ENTRYPOINT ["/sbin/tini", "--", "docker-entrypoint.sh"]
CMD ["node", "dist/main.js"]
8 changes: 5 additions & 3 deletions Dockerfile.dev
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,16 @@ RUN apk upgrade --no-cache \
openssh-client \
gnupg \
fuse3 \
rclone \
curl

COPY --from=restic-builder /restic /usr/local/bin/restic

# Run as non-root — SSH keys mounted to /home/node/.ssh
RUN mkdir -p /home/node/.ssh /home/node/.gnupg \
&& chmod 700 /home/node/.gnupg \
&& chown -R node:node /home/node/.ssh /home/node/.gnupg
# rclone rewrites refreshed OAuth tokens into its config, so .config/rclone must stay writable.
RUN mkdir -p /home/node/.ssh /home/node/.gnupg /home/node/.config/rclone \
&& chmod 700 /home/node/.gnupg /home/node/.config/rclone \
&& chown -R node:node /home/node/.ssh /home/node/.gnupg /home/node/.config

WORKDIR /app

Expand Down
8 changes: 4 additions & 4 deletions _docs/monitoring/uptime-kuma-prd/prd.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ This creates a blind spot: the absence of a backup going undetected until someon
2. Run `docker compose up -d` — Kuma container starts alongside existing containers
3. Access Kuma UI at the configured port (default `3001`)
4. Complete Kuma's first-time setup (create admin account)
5. Create a Push Monitor for each backup project (e.g., "vinsware-backup"), set heartbeat interval to match the project's cron schedule + grace period
5. Create a Push Monitor for each backup project (e.g., "vinelab-backup"), set heartbeat interval to match the project's cron schedule + grace period
6. Copy the push token from each monitor into `projects.yml` under `monitor.config.push_token`
7. Run `backupctl config validate` to verify the monitor config is valid

Expand Down Expand Up @@ -174,7 +174,7 @@ This creates a blind spot: the absence of a backup going undetected until someon
- FR-5.2: No HTTP call is made to Kuma during dry runs

**Acceptance Criteria:**
- AC-5.1: Running `backupctl run vinsware --dry-run` with monitor config does NOT send a heartbeat to Kuma
- AC-5.1: Running `backupctl run vinelab --dry-run` with monitor config does NOT send a heartbeat to Kuma
- AC-5.2: Kuma's heartbeat timer is unaffected by dry runs

**Dependencies:** None
Expand Down Expand Up @@ -224,14 +224,14 @@ UPTIME_KUMA_PORT=3001

```yaml
projects:
- name: vinsware
- name: vinelab
enabled: true
cron: '0 0 * * *'
# ... existing config ...
notification:
type: slack
config:
webhook_url: https://hooks.slack.com/services/VINSWARE/SPECIFIC/HOOK
webhook_url: https://hooks.slack.com/services/VINELAB/SPECIFIC/HOOK
monitor:
type: uptime-kuma
config:
Expand Down
8 changes: 4 additions & 4 deletions _docs/monitoring/uptime-kuma-prd/setup-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ For each backup project that should report heartbeats:

1. In Kuma UI, click **Add New Monitor**
2. Set **Monitor Type** to **Push**
3. Set a descriptive **Friendly Name** (e.g., `vinsware-backup`)
3. Set a descriptive **Friendly Name** (e.g., `vinelab-backup`)
4. Set **Heartbeat Interval** to match the project's cron schedule:

| Cron Schedule | Description | Interval | Grace Period |
Expand Down Expand Up @@ -221,7 +221,7 @@ For each project, add a `monitor` block with the push token from step 5:

```yaml
projects:
- name: vinsware
- name: vinelab
enabled: true
cron: '0 0 * * *'
# ... existing database, assets, restic, notification config ...
Expand All @@ -248,10 +248,10 @@ This verifies that `UPTIME_KUMA_BASE_URL` is set when any project has a `monitor
Trigger a backup and confirm the heartbeat reaches Kuma:

```bash
backupctl run vinsware
backupctl run vinelab
```

In the Kuma dashboard, the `vinsware-backup` monitor should show a green UP status with the backup duration displayed as response time.
In the Kuma dashboard, the `vinelab-backup` monitor should show a green UP status with the backup duration displayed as response time.

To test failure detection, stop sending heartbeats (or run a backup that fails). After the configured interval + grace period, Kuma will mark the monitor as DOWN and send alerts through its notification channels.

Expand Down
Loading