Add a SQLite storage backend, with bulk import/export - #27
Merged
Merged
Conversation
The link table is a single three-column relation, which does not need a managed Postgres instance to serve it. SQLiteStore implements the existing Store interface so the choice is a deployment detail rather than a rewrite. The driver is modernc.org/sqlite rather than mattn/go-sqlite3 because the Dockerfile builds with CGO_ENABLED=0 into distroless/static; a cgo driver would not link. WAL mode keeps redirect reads from blocking on writes, and the schema is applied on open so a freshly provisioned volume needs no manual setup. Postgres has no equivalent of SQLite for reporting insert-vs-update, so where PostgresStore uses "returning (xmax = 0)", SQLiteStore does the existence check and the write in one transaction. Storage precedence is --ephemeral, then SQLITE_PATH, then DATABASE_URL, so unsetting one variable reverts to Postgres. cmd/migrate copies Postgres to SQLite and verifies every entry against the source. It never writes to Postgres and is safe to rerun. It ships in the image so it can run on a machine that reaches both the database and the volume. Startup no longer logs the database connection string. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GET /api/links already returned the whole database; this adds the missing half so a link set can be saved off and reloaded. POST /api/links accepts the same Links proto that the GET returns, and the client grows --export and --import flags that read and write it as a file. Imports are additive: links already stored that the body does not mention are left alone, so a restore cannot silently drop entries. Every link is validated before the first write, so one malformed URI fails the request instead of half-applying. Keys that collide only after hyphen normalization are rejected for the same reason. The validation put() performed inline moves to validateLink so that single and bulk writes cannot drift apart. Together with the SQLite backend this makes migrating between stores a matter of exporting, pointing the server at an empty database, and importing -- no direct database access required. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Bulk import/export over the REST API covers the migration, so the direct database-to-database path is dead weight before it has ever been used. It was not free: shipping a second binary in the image took it from ~42MB to ~75MB, and the command had no automated test coverage. The one thing it did that import does not -- copy `segments` verbatim rather than recomputing it -- turned out not to matter. requiredPaths is deterministic from the URI, so a recomputed value is identical, and it is self-healing if a stored value were ever stale. pgx stays regardless: PostgresStore remains as the rollback path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The link database is a single three-column table:
That does not need a managed Postgres instance behind it. This adds a SQLite backend so the table can live in a file on a small volume instead, which suits a low-traffic personal redirector. Postgres stays supported as the fallback.
What
SQLite store implementing the existing
Storeinterface, so the choice is a deployment detail rather than a rewrite.modernc.org/sqlite, notmattn/go-sqlite3— the Dockerfile buildsCGO_ENABLED=0intodistroless/static, so a cgo driver would not link.Putreports created-vs-updated via a transaction, since SQLite has no equivalent of the Postgresxmax = 0trick.--ephemeral, thenSQLITE_PATH, thenDATABASE_URL. Unsetting one variable reverts to Postgres.The tradeoff: a file on a volume pins the app to one machine in one region, with no replication. Fine for this workload.
Bulk import/export.
GET /api/linksalready returned everything; this addsPOST /api/linksplus--export/--importclient flags. Imports are additive (a restore can't silently drop links) and fully validated before the first write (one bad URI fails the request rather than half-applying). The validationput()did inline moved tovalidateLinkso single and bulk writes can't drift.That makes moving between backends: export, restart with
SQLITE_PATHset against an empty file, import. No direct database access required. It doubles as an ordinary backup/restore path.Testing
Beyond unit tests, the switch was rehearsed locally end to end: server on real Postgres in Docker → export via the REST API → second server on an empty SQLite file (verified 0 links,
/rfc/5280→ 404) → import →The
.indexroot redirect and{1}/{0}substitution both survive. Re-importing is idempotent, and exporting back out of SQLite diffs identical to the original Postgres export.Also verified: the production Docker image builds and runs with the pure-Go driver, data survives a container restart on a mounted volume, and
test.shpasses against both backends (./sqlite_test.shand./docker_test.sh) — the latter being the CI configuration.15 new tests.
Not included
fly.tomlis unchanged. SettingSQLITE_PATHthere would switch storage on the next deploy, before any data exists — every link would 404. Provisioning the volume and adding a[[mounts]]block is a separate deliberate step.A direct Postgres-to-SQLite
cmd/migratetool was written and then removed in ca655d3 — the REST path covers it, and shipping the extra binary took the image from ~42MB to ~75MB. It's in the branch history if it's ever wanted.🤖 Generated with Claude Code