recueil

recueil is a self-hosted personal web archiver.

Note

This README is for building and hacking on recueil. Looking to self-host or use it instead? See the docs.

Building from source

Prerequisites:

• Go, matching the version in go.mod
• Node.js (any current LTS) and pnpm
• sqlc 1.31.1 (generates internal/db from migrations//queries/ which isn't checked in, see below)
• jq (reads the build version out of package.json)

Clone with submodules — the build embeds internal/urlnorm/clearurls-rules (a pinned snapshot of the ClearURLs ruleset) directly into the Go binary, and go:embed fails without it checked out:

git clone --recurse-submodules https://github.com/mfinelli/recueil.git
# or, if already cloned:
git submodule update --init

To pull in a newer ruleset snapshot later: cd internal/urlnorm/clearurls-rules && git pull origin master (or pin to a specific commit/tag), then commit the resulting submodule pointer change as its own commit.

Then:

make

Runs sqlc generate, builds the dashboard frontend, and compiles recueil with version info baked in via -ldflags. The resulting binary is the same thing server/agent/gc/user/device/auth/enqueue all live inside of — see CLI Reference and Administration for what each subcommand does.

Repository layout

A monorepo — everything lives here rather than split across repos, including the parts with their own independent build tooling.

• cmd/ — the CLI's subcommands (server, agent, gc, user, device, auth, enqueue), all one binary.
• internal/ — the Go backend, see below.
• src/ — the Svelte dashboard, served by server via go:embed.
• extension/ — the browser extension. It has its own build tooling and README — see extension/README.md.
• terraform/ — the Cloudflare Worker (plain JS, no build step) and the OpenTofu/Terraform module that provisions it, D1, and R2. It has its own README — see terraform/README.md.
• www/ — the docs site and the marketing site, both Zola.
• migrations/ — Postgres schema migrations (goose).
• queries/ — SQL queries sqlc generates internal/db from.
• scripts/ — small standalone maintenance scripts (e.g. checking that pinned tool versions agree across go.mod/Dockerfile/CI).

internal/

Package	What it is
ai	The async AI enrichment job: summarize a capture's extracted text.
archive	The local, canonical disk store for captures.
auth	Password hashing, session tokens, pairing-token encryption, the bootstrap flow.
clicreds	Where recueil auth stores, and recueil enqueue reads, this device's pairing credential.
config	Loads backend configuration via Viper — TOML file, env vars, defaults.
d1migrate	Applies pending D1 schema migrations at backend startup.
dbtest	The Postgres integration-test harness.
deviceapi	What a paired device (the CLI) uses to talk to the Worker's device-facing endpoints.
devices	The backend's client for the dashboard's Manage Devices screen.
gc	Reclaims disk space that deleting pages, captures, and links deliberately leaves behind, on demand.
httpapi	The dashboard-facing HTTP API.
ingest	The ingestion pipeline: pulls completed captures and saved links in from R2/D1.
linkicons	The local disk store for link favicons, kept apart from the capture archive.
mcpapi	The MCP-facing surface over a user's archive — read-only tools.
metrics	Builds the Prometheus registry served at /metrics.
mirror	Pushes backend-owned data (page and link bookmark lists) outward to D1, via the Worker.
pendingcaptures	The backend's client for the Worker's pending-captures endpoints.
pendinglinks	The backend's client for the Worker's saved-links endpoints.
pgmigrate	Applies pending Postgres schema migrations via goose.
preflight	Startup checks server and agent run first, such as that their storage directories exist.
queueitems	The backend's client for the dashboard's Queue screen.
r2	The backend's R2 client (distinct from the Worker's presigned-upload path).
readability	The async reader-text extraction job.
screenshot	The async screenshot/thumbnail job.
sidecar	Plumbing shared by every job that renders through the headless-Chrome sidecar.
slug	Generates and validates the URL-facing slugs stored on tags/collections.
urlnorm	Computes normalized_url for captures/pages.
wayback	Submits URLs to the Wayback Machine on request, and records how each submission went.

internal/db (sqlc-generated query code) isn't checked in — make generates it. See DESIGN.md for the architecture and reasoning behind how these fit together; this list is just a map.

Local development

Postgres via Docker Compose:

just compose local   # or: just compose test, for the test-profile database

Then, for the backend itself:

just serve   # recueil server, against local.toml
just agent   # recueil agent, against local.toml

Both rebuild (make all) before running, so they always reflect your current changes, and create the tmp/archive and tmp/links directories that local.toml defaults to, as necessary.

Important

local.toml as committed will let server/agent start, but not do anything Cloudflare-facing. Postgres-only things (the dashboard, logging in) work as-is; pairing, enqueuing, and the rest of the capture flow need a real Worker/D1/R2 to talk to. The "todo" values (worker_url, worker_service_secret, cloudflare_account_id, cloudflare_d1_database_id, cloudflare_api_token, r2_*) need to point at an actual Cloudflare deployment — see Deploying recueil for provisioning one; a small deployment kept separate from any real archive works fine for this. AI enrichment (ai_api_key) is optional and only needed if you're testing that specifically — ai_base_url/ai_model are already set for OpenRouter, swap them if you use something else.

Caution

local.toml is a tracked file, not gitignored — be careful not to commit real values into it. Check git diff local.toml before committing anything that touches it. This is a real rough edge that I'll smooth out at some point (e.g., untracking it in favor of a checked-in local.toml.example), just not done yet.

For frontend work on the dashboard specifically, pnpm run dev (Vite, with HMR) is faster than just serve — no Go rebuild or manual page refresh needed for every change.

just www-serve is for the docs/marketing site specifically (Zola), not general frontend work.

Testing, linting, formatting

just test
just lint
just fmt

These closely mirror what CI runs and should be a good indicator if CI will eventually pass on a changeset..

License

recueil: self-hosted webpage bookmarker and archiver
Copyright © 2026 Mario Finelli

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.