Skip to content

Add a SQLite storage backend, with bulk import/export - #27

Merged
jdtw merged 3 commits into
mainfrom
feat/sqlite-store
Aug 1, 2026
Merged

jdtw merged 3 commits into
mainfrom
feat/sqlite-store

Conversation

@jdtw

@jdtw jdtw commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Why

The link database is a single three-column table:

create table if not exists links (
  path text primary key,
  link text not null,
  segments int not null
);

That does not need a managed Postgres instance behind it. This adds a SQLite backend so the table can live in a file on a small volume instead, which suits a low-traffic personal redirector. Postgres stays supported as the fallback.

What

SQLite store implementing the existing Store interface, so the choice is a deployment detail rather than a rewrite.

  • Driver is modernc.org/sqlite, not mattn/go-sqlite3 — the Dockerfile builds CGO_ENABLED=0 into distroless/static, so a cgo driver would not link.
  • WAL mode so redirect reads don't block on writes; schema applied on open so a fresh volume needs no setup.
  • Put reports created-vs-updated via a transaction, since SQLite has no equivalent of the Postgres xmax = 0 trick.
  • Precedence: --ephemeral, then SQLITE_PATH, then DATABASE_URL. Unsetting one variable reverts to Postgres.

The tradeoff: a file on a volume pins the app to one machine in one region, with no replication. Fine for this workload.

Bulk import/export. GET /api/links already returned everything; this adds POST /api/links plus --export / --import client flags. Imports are additive (a restore can't silently drop links) and fully validated before the first write (one bad URI fails the request rather than half-applying). The validation put() did inline moved to validateLink so single and bulk writes can't drift.

That makes moving between backends: export, restart with SQLITE_PATH set against an empty file, import. No direct database access required. It doubles as an ordinary backup/restore path.

Testing

Beyond unit tests, the switch was rehearsed locally end to end: server on real Postgres in Docker → export via the REST API → second server on an empty SQLite file (verified 0 links, /rfc/5280 → 404) → import →

/rfc/5280   → 302 https://datatracker.ietf.org/doc/html/rfc5280
/swap/a/b   → 302 https://example.com/b/a
/plain      → 302 https://example.com/plain
/(root)     → 302 https://jdtw.us/

The .index root redirect and {1}/{0} substitution both survive. Re-importing is idempotent, and exporting back out of SQLite diffs identical to the original Postgres export.

Also verified: the production Docker image builds and runs with the pure-Go driver, data survives a container restart on a mounted volume, and test.sh passes against both backends (./sqlite_test.sh and ./docker_test.sh) — the latter being the CI configuration.

15 new tests.

Not included

fly.toml is unchanged. Setting SQLITE_PATH there would switch storage on the next deploy, before any data exists — every link would 404. Provisioning the volume and adding a [[mounts]] block is a separate deliberate step.

A direct Postgres-to-SQLite cmd/migrate tool was written and then removed in ca655d3 — the REST path covers it, and shipping the extra binary took the image from ~42MB to ~75MB. It's in the branch history if it's ever wanted.

🤖 Generated with Claude Code

jdtw and others added 3 commits July 31, 2026 21:49
The link table is a single three-column relation, which does not need a
managed Postgres instance to serve it. SQLiteStore implements the existing
Store interface so the choice is a deployment detail rather than a rewrite.

The driver is modernc.org/sqlite rather than mattn/go-sqlite3 because the
Dockerfile builds with CGO_ENABLED=0 into distroless/static; a cgo driver
would not link. WAL mode keeps redirect reads from blocking on writes, and
the schema is applied on open so a freshly provisioned volume needs no
manual setup.

Postgres has no equivalent of SQLite for reporting insert-vs-update, so
where PostgresStore uses "returning (xmax = 0)", SQLiteStore does the
existence check and the write in one transaction.

Storage precedence is --ephemeral, then SQLITE_PATH, then DATABASE_URL, so
unsetting one variable reverts to Postgres.

cmd/migrate copies Postgres to SQLite and verifies every entry against the
source. It never writes to Postgres and is safe to rerun. It ships in the
image so it can run on a machine that reaches both the database and the
volume.

Startup no longer logs the database connection string.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GET /api/links already returned the whole database; this adds the missing
half so a link set can be saved off and reloaded. POST /api/links accepts
the same Links proto that the GET returns, and the client grows --export
and --import flags that read and write it as a file.

Imports are additive: links already stored that the body does not mention
are left alone, so a restore cannot silently drop entries. Every link is
validated before the first write, so one malformed URI fails the request
instead of half-applying. Keys that collide only after hyphen
normalization are rejected for the same reason.

The validation put() performed inline moves to validateLink so that single
and bulk writes cannot drift apart.

Together with the SQLite backend this makes migrating between stores a
matter of exporting, pointing the server at an empty database, and
importing -- no direct database access required.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Bulk import/export over the REST API covers the migration, so the direct
database-to-database path is dead weight before it has ever been used.

It was not free: shipping a second binary in the image took it from ~42MB
to ~75MB, and the command had no automated test coverage.

The one thing it did that import does not -- copy `segments` verbatim
rather than recomputing it -- turned out not to matter. requiredPaths is
deterministic from the URI, so a recomputed value is identical, and it is
self-healing if a stored value were ever stale.

pgx stays regardless: PostgresStore remains as the rollback path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jdtw
jdtw force-pushed the feat/sqlite-store branch from ca655d3 to 92c6253 Compare August 1, 2026 04:49
@jdtw
jdtw merged commit 46cdf19 into main Aug 1, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant