feat(restic-adapter)!: add pluggable storage backends for S3, B2, and rclone - #96
Merged
Merged
Conversation
Every spec that needed a ProjectConfig built one inline, duplicating the full 19-field literal across 19 files. Extract test/support/project-config.builder.ts and route all fixtures through it, so upcoming changes to the ProjectConfig shape are a single edit rather than a 19-file sweep. Specs keep explicit overrides only where the values are load-bearing: the restore guide asserts on its database host and user, so those stay inline. Adds a @test/* path alias to tsconfig and both jest configs.
The password-precedence and fallback tests asserted only toBeDefined(), so they passed regardless of which password actually won. RESTIC_SSH_COMMAND had no coverage at all, and the factory's own "password not configured" error was untested. Drive a real restic invocation through the created port and assert the resulting env, which covers factory-to-adapter wiring end to end. Verified by inverting the precedence in the factory: the old spec stayed green, this one fails. Prep for making the storage backend pluggable.
Replace the SFTP-shaped `restic:` config block with a `storage:` block carrying a `type` discriminator (sftp, s3, b2, rclone, local), a `repository`, and a per-type `config` credential bag. Follows the existing `monitor:` precedent. This is the config half of making the remote storage backend pluggable; the factory still only builds SFTP repositories and is refactored next. The `restic:` block keeps working: the loader maps it onto `type: sftp` and warns. Carrying both keys is a validation error. Validation gains per-backend required-key checks (an s3 project needs its endpoint and credentials; only an sftp project needs the HETZNER_SSH_* vars) and a snapshot_mode enum check, which was previously cast unvalidated so a typo like "combinedd" passed straight through. `config show` claims to mask secrets but printed notification.config raw, leaking Slack webhook URLs and SMTP passwords. Since storage.config is a credential bag with the same problem, add a shared maskSecretValues helper and apply it to the storage, notification, and monitor bags. Also restores `monitor`, which the hand-built allowlist had silently dropped from the output.
Make the remote storage backend pluggable. Previously the factory read the HETZNER_SSH_* vars with getOrThrow for every project and the adapter hardcoded the sftp: URL scheme, so a Storage Box was the only possible target. Add a resolver per backend that turns a StorageConfig into the repository string and env restic needs, dispatched through ResticBackendRegistry (mirroring DumperRegistry). The adapter now takes an already-resolved repository plus an env bag, so sync/prune/listSnapshots/restore stay backend-neutral and the SSH command lives only in the sftp resolver. Backends: sftp, s3 (any S3-compatible: Backblaze B2, Wasabi, R2, MinIO, AWS), b2, rclone (Google Drive, OneDrive, Dropbox), and local. Verified the real Nest DI graph resolves all five to the correct repository strings, and that restic 0.18.1 round-trips init/backup/snapshots/restore against the generated format. rclone needs its binary and a writable config mount, which land next.
EnvValidationService required HETZNER_SSH_HOST/USER/KEY_PATH in production unconditionally, so an S3-only deployment could not boot without setting dummy Storage Box values. Require them only when an enabled project actually resolves to storage.type: sftp. A config that cannot be read falls back to skipping the backend-specific checks, since a malformed projects.yml is `config validate`'s problem and must not surface as a misleading missing-env-var error at boot. Install rclone in both images so the rclone backend (Google Drive, OneDrive, Dropbox) works. Unlike ssh-keys and gpg-keys, its config mount is deliberately read-write: rclone rewrites refreshed OAuth tokens into rclone.conf, and /home/node is not persisted, so a read-only mount would silently drop the refreshed token on every container recreate and fail much later as an expired token. The entrypoint warns if the directory is not writable.
The health check never touched the storage layer: it ran `ssh <host> exit` against
the Hetzner vars and reported that result as "restic repos healthy". For an S3-only
deployment it took the unconfigured path and reported healthy without ever
verifying a repository.
Move connectivity behind RemoteStoragePort.checkConnectivity(), implemented as
`restic cat config`, which proves reachability, credentials and the repository
password in one round trip for whichever backend the project uses. CheckHealthUseCase
now probes each enabled project and isolates failures per project.
Results are cached (HEALTH_STORAGE_CHECK_TTL_SECONDS, default 300) because Docker's
HEALTHCHECK polls /health every 30s: uncached, that is ~2880 restic round trips per
project per day against a billed B2 or rate-limited Drive, each rclone probe also
spawning an `rclone serve restic` subprocess. The audit DB and disk checks stay
uncached.
The SSH probes in SystemHealthPort had no callers left and are removed (knip clean).
Also adds the per-project try/catch that clear-cache --clear-all was missing, so one
misconfigured project no longer aborts the whole run.
BREAKING CHANGE: the /health response replaces checks.ssh and checks.resticRepos with
a checks.storage array of per-project results ({ project, backendType, reachable,
error? }). Alerts querying checks.ssh.connected or checks.resticRepos will silently
match nothing — repoint them at checks.storage[].reachable. `backupctl health` prints
one line per project instead of the SSH/restic lines.
Once /health probes remote storage, using it for the Docker HEALTHCHECK is wrong in two ways: a Backblaze or Drive outage would mark the container unhealthy and trigger restarts that cannot possibly fix a remote outage, and the probe can outlast the healthcheck's 5s timeout. Measured: restic does not fail fast on bad S3 credentials, retrying with backoff for the full 100s until killed, and it exposes no retry-count flag, so the process timeout is the only bound. Add CheckLivenessUseCase behind GET /health/live covering only container-local state (audit DB, disk), and point HEALTHCHECK there. GET /health keeps the deep per-project storage checks for operators and `backupctl health`. Verified against a real MinIO S3 backend and the real audit DB: /health/live answers in 6ms and stays 200 while a storage backend is down, while /health correctly reports 503 with the failing project isolated from the healthy one.
Cover the per-project storage block and each backend (sftp, s3, b2, rclone, local) with the required config keys, steer cloud users to s3 over b2's native API as restic recommends, and document the legacy restic: block with a before/after migration. Record the caveats that are easy to hit: the rclone config mount must stay writable or refreshed OAuth tokens are lost on container recreate; consumer Drive/OneDrive/ Dropbox APIs are rate-limited and slow for restic's many small pack files; iCloud is not supported and why; the HETZNER_SSH_* vars are now only needed for sftp projects. Document the breaking /health change (checks.ssh and checks.resticRepos become checks.storage[]), the new /health/live used by the Docker healthcheck, and why the storage probe is cached. The example config now demonstrates every backend and keeps one project on the legacy block; verified it loads through `config validate`, and that `config show` masks the S3 credentials while leaving the endpoint readable.
install.sh still emitted the deprecated restic: block, so every fresh install would have produced config that logs a deprecation warning on first boot. Emit storage: with type: sftp instead, and update the repo-init step that greps the generated YAML for the path — it read repository_path and would have silently skipped creating the remote directory. Verified the generated shape passes `config validate` with no deprecation warning.
Self-review of the branch turned up three real defects and some duplication. Every sftp example in this repo writes an absolute-looking repository (/backups/vinsware), so converting a project to s3 naturally carries the leading slash — which produced "s3:https://endpoint//bucket/path" and silently targeted an unintended repository. Strip it. Conversely b2 needs bucket:path and accepted the s3-style bucket/path without complaint, failing only at the first scheduled backup; reject it up front like the rclone resolver already does. The deprecation warning fired on every loadAll(), and loadAll() re-reads the file on every call — with the new health probe calling it every 5 minutes, a legacy config would have logged the same warning ~288 times a day forever. Warn once per project, reset on explicit reload. REQUIRED_STORAGE_CONFIG_KEYS was defined in the loader while each resolver independently required the same keys, so the two could drift into a config that validates but fails at run time. Move it to the shared storage-config model that both layers read, and have requireStorageConfig refuse to read an undeclared key. Also drops a redundant getProject() lookup in the health probe (the ProjectConfig was already in hand) and collapses the duplicated legacy-block mapping.
Rename the example project placeholder across config, docs, tests and scripts:
vinsware -> vinelab, VINSWARE -> VINELAB, along with the names derived from it
(vinelab_db, postgres-vinelab, VINELAB_RESTIC_PASSWORD, /data/vinelab/...).
The one real branded address, vineeth.nk@vinsware.com, becomes vineeth.nk@vinelab.in
to match the domain the rest of the repo already uses — a mechanical rename would
have invented vinelab.com.
Cross-checking every ${} in projects-example.yml against .env.example turned up
variables with no uncommented value: B2_ENDPOINT/B2_KEY_ID/B2_APP_KEY, EMAIL_PASSWORD
(both pre-existing) and PROJECTZ_* (added with the rclone example earlier on this
branch). Because ${} is resolved for every project at load time — including disabled
ones — any one of them being unset failed the whole config load, so copying the
shipped example could not work. Uptime Kuma's base URL had the same mismatch on main:
the example projects declare monitor blocks while the variable stayed commented out,
so validation always failed.
Verified the example pair now loads and validates with nothing but .env.example set.
vineethkrishnan
force-pushed
the
feat/pluggable-storage-backends
branch
from
July 16, 2026 05:56
d4badf3 to
d6cc1d4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
restic:config block with a typedstorage:block carrying atypediscriminator. The oldrestic:block keeps working — it is read astype: sftpand warns./healthnow probes each project's real repository instead of runningssh <host> exit, and a new/health/livebacks the Docker healthcheck.Type of Change
feat— New feature⚠ Breaking Change
/healthreplaceschecks.sshandchecks.resticReposwith achecks.storagearray of per-project results:Alerts querying
checks.ssh.connectedorchecks.resticReposwill silently match nothing rather than error — repoint them atchecks.storage[].reachable.backupctl healthprints one line per project instead of the SSH/restic lines.Nothing else breaks: existing
projects.ymlfiles need no edits.Why
/health/liveexistsThe Docker
HEALTHCHECKnow points at/health/live(audit DB + disk only, no remote calls). Pointing it at a storage-probing/healthwould restart the container whenever a remote provider had an outage, which cannot fix anything. It is also too slow: I measuredrestic cat configretrying bad S3 credentials for the full 100s until killed, with no retry-count flag available — well past the healthcheck's 5s timeout.Storage probes are cached (
HEALTH_STORAGE_CHECK_TTL_SECONDS, default 300) because Docker polls every 30s; uncached that is ~2880 billable B2 transactions per project per day.Not included: iCloud
rclone's iCloud backend is Tier 4 (experimental), rejects app-specific passwords, and needs interactive 2FA with a 30-day trust token — that breaks unattended cron and would put a primary Apple ID password in
.env.Test Plan
npm test) — 621 tests, 66 suitesnpm run build)npm run lint:check,lint:strict)npm run test:cov) — 94.1% statements / 80.8% branchesknip) and duplication (jscpd) cleanBeyond the suite, exercised against real infrastructure:
init→backup→snapshots→restore, payload verified byte-for-byte./healthagainst real MinIO + the real audit DB, with a legacyrestic:project alongside: the s3 project reported reachable while the broken sftp one reported its error, isolated per project./health/liveanswered in 6ms and stayed200throughout./healthcalls produced exactly 1 restic probe.projects-example.yml+.env.example) loads and validates with nothing else set.Checklist
!for the breaking changeNotes for review
rclone-config/is mounted read-write on purpose — rclone rewrites refreshed OAuth tokens into it, and/home/nodeis not persisted, so a:romount would silently drop the token on every container recreate and fail much later as an expired token.