diff --git a/CHANGELOG.md b/CHANGELOG.md index c3b49ff..49ae6ee 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,26 +1,40 @@ # Changelog -User-visible changes are recorded here. Versions correspond to -[GitHub releases](https://github.com/0then0/form-forge/releases); detailed upgrade -instructions accompany each release. Unreleased changes are not part of the latest -tagged source archive. +All notable user-visible changes are recorded here. The format follows +[Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Versions match +[GitHub releases](https://github.com/0then0/form-forge/releases), where each +release also includes detailed upgrade instructions. ## [Unreleased] ### Added -- Manual GitHub Actions workflow for Neon migrations with a separate database - secret, and a separate manual Netlify production build trigger. -- Repository contribution and security policies, community templates, editor - settings and Git file handling conventions. +- Manually started GitHub Actions workflows for Neon database migrations and + Netlify production deploys. +- Contribution guide, security policy, code of conduct, issue and pull request + templates, and a releasing guide for maintainers. + +### Changed + +- Docker images and the Netlify build use Node.js 24.22.0. The pinned pnpm + version is 12.17.0. +- Dependencies updated, including Next.js 16.4.5 and the Sentry SDK. +- Documentation rewritten for new users and operators. ### Fixed -- Migration command handles transport errors, limits connection and SQL waits, - closes connections and reports errors without printing SQL or credentials. - It preserves existing Drizzle migrations and history and does not retry - automatically. Remote migration execution can bypass a failing local network - path; the cause of local Neon `ETIMEDOUT` failures remains unconfirmed. +- `pnpm db:migrate` closes its connection on network errors, limits connection, + lock and statement waits, and reports failures without printing SQL or + credentials. It does not retry automatically and keeps the existing migration + history. + +### Security + +- Removed vulnerable transitive versions of glob, minimatch and esbuild + ([GHSA-3g3m-2fjr-6pw4](https://github.com/advisories/GHSA-3g3m-2fjr-6pw4), + [GHSA-g3hv-75wg-m249](https://github.com/advisories/GHSA-g3hv-75wg-m249), + [GHSA-67mh-4wv8-2f99](https://github.com/advisories/GHSA-67mh-4wv8-2f99)). + None of the affected code paths were reachable in production. ## [0.2.0] - 2026-10-18 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8e0bab3..738d408 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,51 +1,57 @@ # Contributing to Form Forge -Form Forge is a small libre/self-hosted application. Keep changes focused and -prefer existing services, workspace packages and platform APIs over new dependencies -or external infrastructure. Discuss substantial features, dependencies and public -contract changes in an issue before implementing them. - -By participating, follow the [Code of conduct](CODE_OF_CONDUCT.md). Report -vulnerabilities through the [security policy](SECURITY.md), not a public bug report. - -## Local setup - -Follow [Getting started](docs/getting-started.md) for Node.js, the pinned pnpm -version, PostgreSQL, environment variables and OAuth configuration. Use pnpm and -commit intentional manifest changes together with `pnpm-lock.yaml`. Do not commit -credentials, local environment files, database dumps or submission data. Environment -examples must contain placeholders only. - -The [development guide](docs/development.md) describes commands, CI and test -databases. `.editorconfig`, Biome and Prettier define formatting. Use `pnpm format` -on your own changes; avoid unrelated formatting or snapshot updates. - -## Changes and pull requests - -- Search existing issues and PRs before opening a duplicate. Include a minimal - reproduction for bugs and the user problem for feature requests. -- Work on a branch in your fork and open a focused PR against `main`. Explain - behavior before and after, compatibility implications and checks actually run. - Add screenshots for meaningful UI changes, with personal data removed. -- Add regression tests for meaningful behavior. Preserve server-side workspace - authorization, immutable published schemas and answers, idempotency, transactional - outbox semantics and compatibility with existing forms. -- Generate additive migrations with `pnpm db:generate`; commit SQL and Drizzle - metadata together. Do not edit applied migrations or reset migration history. - Test both empty-database installation and upgrades on disposable local databases. -- Use Conventional Commits in English, such as `fix: preserve submission filters`. - No CLA is required; contributions are provided under the project's - [Apache License 2.0](LICENSE). - -The sole maintainer may push directly to `main`; external and Dependabot changes -use PRs. This does not waive verification. Follow the -[dependency maintenance policy](docs/dependency-maintenance.md) for toolchain, -major and security updates. Do not merge unreviewed dependency changes just to -clear the update queue. - -## Verification - -Run checks relevant to the change and report any checks you could not run: +Thank you for your interest in Form Forge. Bug reports, documentation fixes and +focused improvements are all welcome. + +Form Forge is a small self-hosted application, so we prefer existing services, +workspace packages and platform APIs over new dependencies or external +infrastructure. Please open an issue to discuss substantial features, new +dependencies or changes to public contracts before you start working on them. + +By participating, you agree to follow the [Code of conduct](CODE_OF_CONDUCT.md). +Report vulnerabilities through the [security policy](SECURITY.md), not in a public +issue. + +## Set up your environment + +Follow [Getting started](docs/getting-started.md) to install Node.js and pnpm, +start PostgreSQL, configure environment variables and GitHub sign-in. The +[development guide](docs/development.md) describes the available commands, tests +and CI. + +- Use pnpm, and commit intended manifest changes together with `pnpm-lock.yaml`. +- Never commit credentials, local environment files, database dumps or response + data. Example environment files contain placeholders only. +- `.editorconfig`, Biome and Prettier define formatting. Run `pnpm format` on + your changes and avoid unrelated formatting changes. + +## Make a change + +- Search existing issues and pull requests first. For bugs, include a minimal + reproduction; for features, describe the problem you want to solve. +- Work on a branch in your fork and open a focused pull request against `main`. + Describe the behavior before and after, compatibility implications and the + checks you ran. Add screenshots for visible UI changes, with personal data + removed. +- Add regression tests for meaningful behavior. Keep these guarantees intact: + server-side workspace authorization, immutable published schemas and answers, + idempotent submissions, the transactional outbox, and compatibility with + existing forms. +- For database changes, generate a new migration with `pnpm db:generate` and + commit the SQL and Drizzle metadata together. Never edit an applied migration + or reset migration history. Test both a fresh installation and an upgrade on + disposable local databases. +- Write commit messages in English using + [Conventional Commits](https://www.conventionalcommits.org/), for example + `fix: preserve submission filters`. + +All changes to `main`, including the maintainer's, go through pull requests with +passing CI. Dependency updates follow the +[dependency maintenance policy](docs/dependency-maintenance.md). + +## Verify your change + +Run the checks relevant to your change and mention any you could not run: ```sh pnpm format:check @@ -56,30 +62,31 @@ pnpm test pnpm test:deployment ``` -For runtime, database or UI changes, also run the applicable PostgreSQL, +For changes to runtime behavior, the database or the UI, also run the PostgreSQL, Playwright and build checks from [Development](docs/development.md) and -[Deployment verification](deploy/VERIFICATION.md). `pnpm test` skips PostgreSQL -integration tests unless `TEST_DATABASE_URL` is set. Integration and browser -suites delete fixture data: use separate disposable local databases, never a -shared or production database. A successful build alone does not verify a change. - -## Automation and releases - -The existing [workflows](.github/workflows) have distinct responsibilities: - -- **CI** runs on PRs and pushes to `main`, using disposable PostgreSQL and test - credentials. It checks formatting, types, lint, unused code, tests and builds. -- **Monthly dependency review** opens a review issue; Dependabot opens update PRs. - Neither automatically merges changes. -- **Database migrations** runs manually from `main`, using the dedicated Neon - secret. See the [Neon migration guide](deploy/NETLIFY.md). It never runs on push. -- **Deploy production** is a separate manual Netlify build trigger. Success means - Netlify accepted the request, not that deployment finished. -- **Release** publishes committed source and checksums when a `v*` tag is pushed - and matches `deploy/VERSION`. A release does not require a PR and does not - deploy the application or migrate its database. - -Maintainers update [CHANGELOG.md](CHANGELOG.md), `deploy/VERSION` and -`deploy/RELEASE_NOTES.md` for a release after checking the intended revision. -Detailed release notes must describe migrations and downgrade limits. Do not -create tags or run production workflows as part of an ordinary contribution. +[Deployment verification](deploy/VERIFICATION.md). `pnpm test` skips the +PostgreSQL integration tests unless `TEST_DATABASE_URL` is set. Integration and +browser tests delete their data, so use separate disposable local databases and +never a shared or production database. A successful build alone does not verify a +change. + +## Automation + +The [GitHub Actions workflows](.github/workflows) are: + +- **CI:** runs on every pull request and push to `main` with a disposable + PostgreSQL database and test credentials. It checks formatting, types, lint and + unused code, runs all test suites and builds the application. +- **Dependency review:** opens a monthly issue listing major and toolchain + updates. Dependabot opens update pull requests. Neither merges anything + automatically. +- **Database migrations** and **Deploy production:** started manually by the + maintainer for the Netlify installation. +- **Release:** publishes a source release when a version tag is pushed. + +See [Releasing](docs/releasing.md) for how releases are made. + +## License + +There is no contributor license agreement. By submitting a contribution, you +agree that it is provided under the project's [Apache License 2.0](LICENSE). diff --git a/README.md b/README.md index 7ce67e2..a13d04c 100644 --- a/README.md +++ b/README.md @@ -54,23 +54,33 @@ For production, use the [Docker Compose guide](deploy/README.md) or [Netlify gui ## Documentation +Using Form Forge: + - [Getting started](docs/getting-started.md): requirements, environment, database, OAuth and your first form. -- [User guide](docs/user-guide.md): account linking, roles, archives, intake, privacy and response management. -- [Public API, OpenAPI and MCP](docs/api.md): submission contracts, idempotency, quotas and authenticated integrations. -- [Embedding](docs/embedding.md): iframe setup, automatic height, lifecycle events and cleanup. -- [Webhooks](docs/webhooks.md): signature verification, retries, outbox delivery and recovery. -- [Architecture](docs/architecture.md): workspace packages, data model and project scope. -- [Development](docs/development.md): commands, CI, database tests and Playwright. -- [Dependency maintenance](docs/dependency-maintenance.md): Dependabot schedules, security fixes and monthly reviews. -- [Deployment and operations](docs/operations.md): production configuration, upgrades, backups and monitoring. +- [User guide](docs/user-guide.md): forms, sign-in methods, intake controls, privacy, responses and roles. +- [Public API, OpenAPI and MCP](docs/api.md): submission contract, idempotency, limits and integrations. +- [Embedding](docs/embedding.md): iframe setup, automatic height and events. +- [Webhooks](docs/webhooks.md): payload format, signature verification, retries and deduplication. -## Contributing +Running an installation: + +- [Deployment and operations](docs/operations.md): configuration, reverse proxy, upgrades, backups and monitoring. +- [Docker Compose](deploy/README.md): single-host installation, backup and restore. +- [Netlify](deploy/NETLIFY.md): Netlify with Neon and Inngest Cloud. +- [Deployment verification](deploy/VERIFICATION.md): automated checks and what they prove. -Read [CONTRIBUTING.md](CONTRIBUTING.md) for setup, verification and pull request guidance, and follow the [Code of conduct](CODE_OF_CONDUCT.md). Integration and browser tests require disposable databases and remove test data. +Working on the project: + +- [Architecture](docs/architecture.md): repository layout, technology and data flow. +- [Development](docs/development.md): commands, tests, CI and Playwright. +- [Dependency maintenance](docs/dependency-maintenance.md): Dependabot, security updates and toolchain changes. +- [Releasing](docs/releasing.md): versioning and the release workflow. + +## Contributing -Report bugs and discuss changes through [GitHub Issues](https://github.com/0then0/form-forge/issues). +Contributions are welcome. Read [CONTRIBUTING.md](CONTRIBUTING.md) for setup, checks and pull request guidance, and follow the [Code of conduct](CODE_OF_CONDUCT.md). -For vulnerabilities, follow [SECURITY.md](SECURITY.md) instead of posting details publicly. See [CHANGELOG.md](CHANGELOG.md) for release history and unreleased changes. +Report bugs and suggest features through [GitHub Issues](https://github.com/0then0/form-forge/issues). To report a vulnerability, follow [SECURITY.md](SECURITY.md) instead of opening a public issue. See [CHANGELOG.md](CHANGELOG.md) for release history. ## License diff --git a/SECURITY.md b/SECURITY.md index c90f5b9..89016e6 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -2,37 +2,43 @@ ## Supported versions -Security fixes target the latest stable release. Older releases and prereleases -do not have a separate maintenance commitment; upgrade to the latest stable -version when a fix is available. See [releases](https://github.com/0then0/form-forge/releases) -and [CHANGELOG.md](CHANGELOG.md). There is no guaranteed response or fix timeline. - -## Report a vulnerability privately - -Do not disclose exploitable vulnerabilities, credentials or submission data in -public issues, PRs or discussions. - -Use **Security → Advisories → Report a vulnerability** on -[GitHub](https://github.com/0then0/form-forge/security/advisories/new) when private -vulnerability reporting is enabled. If that option is unavailable, contact -[the maintainer](https://github.com/0then0) through an available private contact -method. If no private contact is listed, open a minimal issue requesting a private -security contact, without describing the vulnerability or sharing a proof of concept. -Private reporting must be enabled in GitHub repository settings; this file alone -does not enable it. - -Include the affected version or commit, deployment type, prerequisites, expected -and observed behavior, impact and a minimal reproduction using synthetic data. -Avoid live credentials and real responses. Test only installations you own or -have permission to assess. Coordinate public disclosure after maintainers have -assessed the report and users have a fix or mitigation. - -## Operating an installation - -Deployment owners are responsible for access control, TLS, secrets, backups, -dependency updates and monitoring. Follow the [operations guide](docs/operations.md) -and the [dependency maintenance policy](docs/dependency-maintenance.md). -Dependabot alerts and security updates require separate repository settings. -Rotate exposed credentials at their source; removing them from a commit is not -enough. Report an incident involving your own stored data to your installation's -operator, and report a reproducible software vulnerability to the project. +Security fixes are made for the latest stable release. Older releases and +prereleases are not maintained separately; upgrade to the latest stable version +when a fix is available. See the +[releases](https://github.com/0then0/form-forge/releases) and +[CHANGELOG.md](CHANGELOG.md). + +There is no guaranteed response or fix time. + +## Reporting a vulnerability + +Please do not disclose vulnerabilities, credentials or response data in public +issues, pull requests or discussions. + +Report vulnerabilities privately through +[GitHub Security Advisories](https://github.com/0then0/form-forge/security/advisories/new) +(**Security → Advisories → Report a vulnerability**). If that option is not +available, contact [the maintainer](https://github.com/0then0) through a private +contact method listed on their profile. If you cannot find one, open an issue +asking for a private security contact, without describing the vulnerability. + +Please include: + +- the affected version or commit and deployment type; +- prerequisites, expected and observed behavior, and impact; +- a minimal reproduction using synthetic data. + +Do not use live credentials or real responses. Test only installations you own or +have permission to test. We will coordinate public disclosure with you once the +report has been assessed and a fix or mitigation is available. + +## Running an installation + +Operators are responsible for access control, TLS, secrets, backups, dependency +updates and monitoring of their installation. Follow the +[operations guide](docs/operations.md). If a credential is exposed, rotate it at +its source; removing it from a commit is not enough. + +Report incidents involving data stored in a particular installation to that +installation's operator. Report reproducible vulnerabilities in the software to +this project. diff --git a/apps/web/e2e/form-lifecycle.spec.ts b/apps/web/e2e/form-lifecycle.spec.ts index 3776523..a1e605d 100644 --- a/apps/web/e2e/form-lifecycle.spec.ts +++ b/apps/web/e2e/form-lifecycle.spec.ts @@ -202,7 +202,7 @@ test("creates, publishes, submits, fails, retries, and succeeds", async ({ }); await expect(page.getByRole("status")).toBeVisible(); - // The new after() path must finish its failed ingestion attempt before the + // The after() path must finish its failed ingestion attempt before the // fixture advances persisted scheduling and takes over as the test scheduler. await expect .poll( diff --git a/apps/web/src/lib/openapi.ts b/apps/web/src/lib/openapi.ts index 892c090..c463e0f 100644 --- a/apps/web/src/lib/openapi.ts +++ b/apps/web/src/lib/openapi.ts @@ -111,7 +111,7 @@ export const openApiDocument = { title: "Form Forge product API", version: "1.0.0", description: - "Public submissions and workspace product operations. Admin access uses the existing NextAuth session; owner/editor/viewer permissions are enforced by each service. OAuth callbacks, scheduler and internal test controls are outside this document. MCP transport is documented separately in README.", + "Public submissions and workspace product operations. Admin access uses the NextAuth session; owner/editor/viewer permissions are enforced by each service. OAuth callbacks, scheduler and internal test controls are outside this document. The MCP endpoint is documented in docs/api.md.", }, servers: [{ url: "/" }], components: { diff --git a/deploy/NETLIFY.md b/deploy/NETLIFY.md index 5378de0..d54c569 100644 --- a/deploy/NETLIFY.md +++ b/deploy/NETLIFY.md @@ -1,56 +1,58 @@ # Deploy to Netlify with Neon and Inngest Cloud -Netlify runs Next.js through its automatically installed OpenNext adapter. Neon -stores PostgreSQL data, GitHub OAuth provides sign-in, and Inngest Cloud runs -background delivery and recovery. The Docker Compose stack is not used here. +In this setup, Netlify runs the Next.js application through its automatically +installed OpenNext adapter, [Neon](https://neon.com/) hosts PostgreSQL, GitHub +OAuth provides sign-in, and [Inngest Cloud](https://www.inngest.com/) runs +background delivery and recovery. The Docker Compose stack is not used. + +Steps: + +1. [Create and migrate the Neon database](#1-create-and-migrate-the-neon-database) +2. [Import the repository into Netlify](#2-import-the-repository-into-netlify) +3. [Set production environment variables](#3-set-production-environment-variables) +4. [Configure GitHub OAuth](#4-configure-github-oauth) +5. [Deploy and sync Inngest](#5-deploy-and-sync-inngest) ## 1. Create and migrate the Neon database -Create a Neon project and database close to your Netlify Functions region. Copy -both connection strings from **Connect**, preserving their SSL parameters: +Create a Neon project and database in a region close to your Netlify Functions +region. From **Connect**, copy both connection strings, keeping their SSL +parameters: -- **Pooled**, with `-pooler` in the hostname: application `DATABASE_URL`. -- **Direct**, without `-pooler`: migration commands. +- **Pooled** (hostname contains `-pooler`): the application's `DATABASE_URL`. +- **Direct** (no `-pooler`): for running migrations. -The existing `pg` and Drizzle client supports Neon without an additional driver. -Use **`sslmode=verify-full`** in the migration URL and preserve other Neon -parameters, including `channel_binding=require` when present. This explicitly -keeps certificate and hostname verification and avoids the `pg` warning about -the future meaning of `sslmode=require`. Do not disable TLS verification to fix -connection timeouts. +In the migration URL, use `sslmode=verify-full` and keep other parameters, such as +`channel_binding=require`. This keeps certificate and hostname verification and +avoids the `pg` driver's warning about `sslmode=require`. Never disable TLS +verification to work around connection problems. -### Manual GitHub Actions run (recommended for Neon) +You can run migrations from GitHub Actions or from your computer. + +### Option A: GitHub Actions (recommended) 1. In GitHub, open **Settings → Secrets and variables → Actions → New repository - secret**. Name it **`NEON_MIGRATION_DATABASE_URL`** and store the **direct** - connection URL, with `sslmode=verify-full`. Netlify's pooled URL remains separate. -2. Ensure the desired migrations are committed on `main` and its CI checks pass. - Back up the database, stop application writes and pause workers before upgrading - an existing installation. See [upgrade precautions](../docs/operations.md). -3. Open **Actions → Database migrations → Run workflow**, select **main**, and run - it. The workflow applies all pending committed migrations from that commit, - including `0006` if only `0000`–`0005` are recorded. It does not generate SQL or - apply only one arbitrarily selected migration. -4. Confirm **Apply committed migrations** succeeds before deploying the matching - application version and resuming writes and workers. - -The [workflow](../.github/workflows/migrate.yml) runs only on manual dispatch from -`main`, never on push, PR, release, Netlify build or function startup. Concurrent -workflow runs are serialized. Do not run local migrations at the same time. The -database secret is supplied only to the migration step, after dependencies are -installed with a frozen lockfile and lifecycle scripts disabled. That step runs -Node directly, avoiding pnpm's automatic dependency installation before scripts. -GitHub runners -must have network access to Neon; IP restrictions must allow that path. - -This bypasses a failing local network path. A local `read ETIMEDOUT` is not an SQL -migration failure merely because small queries work. The SSL warning and transport -timeout are separate issues; changing `sslmode` does not repair the network path. - -### Local alternative - -From the repository root, with the required Node.js/pnpm versions and dependencies -installed: + secret**. Create **`NEON_MIGRATION_DATABASE_URL`** with the **direct** connection + URL, using `sslmode=verify-full`. +2. Make sure the migrations you want are committed on `main` and CI passes. When + upgrading an existing installation, back up the database and stop writes and + workers first; see [Upgrades](../docs/operations.md#upgrades). +3. Open **Actions → Database migrations → Run workflow**, select `main` and run it. + The workflow applies every committed migration that is not yet recorded in the + database. +4. Confirm that **Apply committed migrations** succeeded before deploying the + matching application version. + +The [workflow](../.github/workflows/migrate.yml) runs only when started manually +from `main`, never on push, pull requests, releases, Netlify builds or function +start. Runs are serialized; do not run local migrations at the same time. The +database secret is available only to the migration step, after dependencies are +installed from the lockfile with lifecycle scripts disabled. If your Neon project +restricts IP addresses, allow GitHub-hosted runners. + +### Option B: from your computer + +From the repository root, with Node.js, pnpm and dependencies installed: ```sh export DATABASE_URL='postgresql://USER:PASSWORD@DIRECT_HOST/DB?sslmode=verify-full&channel_binding=require' @@ -58,98 +60,94 @@ pnpm db:migrate unset DATABASE_URL ``` -The migration runner reads the exported `DATABASE_URL`, not Next.js env files or variables -named `DIRECT_URL` or `DATABASE_URL_UNPOOLED`. Pass the direct connection as -`DATABASE_URL` for this command. Keep migrations out of build commands and function -startup to avoid concurrent migration runs and accidental production changes from -Deploy Previews. For later updates, follow the [maintenance precautions](../docs/operations.md). -Preserve installation keys and transfer existing data before enabling -writes and workers when moving an existing installation. - -The runner uses the existing Drizzle migration files and `drizzle.__drizzle_migrations` -history, including installations initialized through Neon SQL Editor with the -original hashes and timestamps. No history reset, manual recreation of `0006`, or -new migration is needed. Connection attempts are limited to 15 seconds, lock waits -to 15 seconds and each SQL statement to five minutes. For a deliberately longer -maintenance operation, set `MIGRATION_STATEMENT_TIMEOUT_MS` explicitly; the manual -workflow also has a 15-minute overall limit. Errors close the connection and return -a nonzero exit status without printing SQL, answer values or credentials. There -are no automatic retries. After a dropped connection, inspect migration history -and database state before rerunning: losing the `COMMIT` acknowledgement makes -the result uncertain. Do not insert history rows by hand to suppress an error. +The migration command reads `DATABASE_URL` from the shell, not from Next.js +environment files or variables such as `DIRECT_URL`. Pass the direct connection +here. Do not run migrations in the build command or at function start, which can +cause concurrent runs and apply production changes from Deploy Previews. + +### Migration behavior + +Migrations use the committed Drizzle files and the +`drizzle.__drizzle_migrations` history table. The command limits connection +attempts and lock waits to 15 seconds each and every statement to five minutes; +set `MIGRATION_STATEMENT_TIMEOUT_MS` for a longer planned operation. The GitHub +workflow has a 15-minute overall limit. + +On error, the command closes the connection and exits with a nonzero status +without printing SQL, answers or credentials. It does not retry. If the +connection drops, the result can be uncertain, because the `COMMIT` +acknowledgement may have been lost: check the migration history and database +state before running it again. Never insert history rows by hand to hide an error. See [Neon's Drizzle guide](https://neon.com/docs/guides/drizzle). -## 2. Import the Git repository into Netlify +## 2. Import the repository into Netlify -Use these build settings: +Import the Git repository and use these build settings: - Base directory: repository root (`.`). - Package directory: `apps/web`, set in the Netlify UI. - Build command: `pnpm exec turbo run build --filter=@form-forge/web...`. -- Publish directory: `apps/web/.next`, relative to the base directory. +- Publish directory: `apps/web/.next`. -The root `netlify.toml` supplies the base, build command, publish path, Node.js -24.22.0, and frozen pnpm install flags. The `packageManager` field pins pnpm -12.17.0 through Corepack. Hoisting is enabled only for the Netlify install, as -recommended by Netlify for pnpm/Next.js. Do not set `NODE_ENV=production` during -dependency installation: the build needs development dependencies. +[`netlify.toml`](../netlify.toml) sets the base directory, build command, publish +directory, Node.js 24.22.0 and frozen pnpm install flags. The `packageManager` +field pins pnpm 12.17.0 through Corepack. Dependency hoisting is enabled only for +the Netlify install, as Netlify recommends for pnpm with Next.js. Do not set +`NODE_ENV=production` for the install, because the build needs development +dependencies. -Keep the automatic Next.js/OpenNext adapter enabled. No manually installed -framework plugin, Dockerfile, static export, SPA catch-all redirect, or -`pnpm start` command is needed. Do not override the Node.js Functions runtime to -an older version. Choose a Functions region close to Neon in Netlify settings. +Keep Netlify's automatic Next.js adapter enabled. You do not need a framework +plugin, Dockerfile, static export, SPA redirect or `pnpm start` command. Do not +pin the Functions runtime to an older Node.js version. Choose a Functions region +close to your Neon database. See [Next.js on Netlify](https://docs.netlify.com/build/frameworks/framework-setup-guides/nextjs/overview/), -[monorepos](https://docs.netlify.com/build/configure-builds/monorepos/), and +[monorepos](https://docs.netlify.com/build/configure-builds/monorepos/) and [dependency management](https://docs.netlify.com/build/configure-builds/manage-dependencies/). -### Manual production deploys +### Production deploys -The repository's `ignore = "exit 0"` skips Git-triggered builds, including push -and PR builds. Keep **Active builds** enabled in Netlify; **Stopped builds** also -disables build hooks. Build hook requests bypass the ignore command. +`netlify.toml` sets `ignore = "exit 0"`, so pushes and pull requests do not +trigger builds. Deploys are started manually through a build hook, which bypasses +the ignore command. Keep **Active builds** enabled in Netlify: **Stopped builds** +also disables build hooks. One-time setup: 1. In Netlify, open **Project configuration → Developer settings → Continuous deployment → Build hooks → Add build hook**. Name it `Manual production`, - choose `main`, and save. -2. Copy the resulting URL without adding query parameters. In GitHub repository - **Settings → Secrets and variables → Actions → New repository secret**, save - it as **`NETLIFY_BUILD_HOOK`**. Do not commit or share the URL. - -To deploy, push the desired code to `main`, then open **GitHub Actions → Deploy -production → Run workflow**, select `main`, and run it. The workflow sends a -single request to Netlify for the latest remote `main` at build time. It does not -pin the workflow's SHA: avoid pushing more changes while the deploy is starting -if you need to deploy a particular revision. Dispatches from other branches are -skipped. No checkout, dependency installation, migration, or GitHub token -permissions are needed for this workflow. - -A green workflow means Netlify accepted the request, not that deployment -succeeded. Check **Netlify → Deploys** for the build, deployed SHA and publication -status. Requests are not automatically retried to avoid duplicate builds; inspect -Deploys before rerunning after a network error. Workflow concurrency serializes -trigger requests only, not the entire Netlify build. Keep automatic publishing -enabled if successful manual builds should immediately update production. - -Database migrations remain a separate manual workflow and are not run by this -deploy trigger. Resync Inngest after changing function definitions, as described -below. The old Active/Stopped toggle sequence is no longer needed. Use this -workflow for routine deploys rather than relying on the Netlify UI to bypass the -ignore rule. + choose `main` and save. +2. Copy the URL without adding query parameters. In GitHub, open **Settings → + Secrets and variables → Actions → New repository secret** and save it as + **`NETLIFY_BUILD_HOOK`**. Do not commit or share this URL. + +To deploy, push to `main`, then open **Actions → Deploy production → Run +workflow**, select `main` and run it. Netlify builds the latest `main` at the time +the build starts, not a fixed commit, so avoid pushing while a deploy you care +about is starting. Runs from other branches are skipped. The workflow does not +check out code, install dependencies or run migrations. + +A successful workflow run means that Netlify accepted the request, not that the +deploy succeeded. Check **Netlify → Deploys** for the build result and deployed +commit. Requests are not retried automatically, to avoid duplicate builds; check +Deploys before running the workflow again after a network error. Keep automatic +publishing enabled if successful builds should go live immediately. + +Migrations are a separate workflow and are not run by deploys. After changing +background function definitions, [sync Inngest](#5-deploy-and-sync-inngest) +again. See [ignore commands](https://docs.netlify.com/build/configure-builds/ignore-builds/) and [build hooks](https://docs.netlify.com/build/configure-builds/build-hooks/). -## 3. Set the Production environment +## 3. Set production environment variables Choose a permanent public URL, such as `https://PROJECT.netlify.app` or a custom -domain. In Netlify's environment-variable settings, supply real values using -`apps/web/.env.production.example` as the checklist. Make application variables -available to **both Builds and Functions** (or all scopes), with values specific -to the Production deploy context: +domain. In Netlify's environment variable settings, add real values, using +[`apps/web/.env.production.example`](../apps/web/.env.production.example) as a +checklist. Make them available to **both Builds and Functions**, with values for +the Production deploy context: ```dotenv DATABASE_URL= @@ -168,92 +166,90 @@ INNGEST_SIGNING_KEY= INNGEST_SERVE_ORIGIN=https://forms.example.org ``` -Replace every example origin with the same permanent URL. Generate each of -`AUTH_SECRET`, `FINGERPRINT_SECRET`, and `WEBHOOK_ENCRYPTION_KEY` separately: +Replace every example URL with your permanent URL. Generate `AUTH_SECRET`, +`FINGERPRINT_SECRET` and `WEBHOOK_ENCRYPTION_KEY` separately: ```sh openssl rand -base64 32 ``` -Keep keys stable across deploys and securely backed up. Changing the encryption -key makes existing webhook secrets unreadable. Sentry DSNs are optional. Leave -`INNGEST_BASE_URL` unset for Inngest Cloud. Do not set `TEST_DATABASE_URL` or -`E2E_PIPELINE_TOKEN` in deployed environments. +Keep these keys unchanged across deploys and back them up; changing the +encryption key makes stored webhook secrets unreadable. Sentry DSNs are optional. +Leave `INNGEST_BASE_URL` unset for Inngest Cloud. Never set `TEST_DATABASE_URL` +or `E2E_PIPELINE_TOKEN` in a deployed environment. -Do not put secrets in `netlify.toml`: variables declared there are build-only, -not available to Functions. After changing environment values, deploy again. -`NEXT_PUBLIC_APP_URL` is embedded in the browser bundle during compilation. +Do not put secrets in `netlify.toml`: variables set there are available to builds +only, not to Functions. Redeploy after changing environment variables; +`NEXT_PUBLIC_APP_URL` is embedded in the browser bundle at build time. -Netlify provides the reserved `SITE_ID` in both builds and Functions. The app uses -it to select the platform trust boundary and enforce production configuration -without `next start`. Turborepo forwards it and includes it in the cache key. -Do not set `SITE_ID` on a self-hosted installation. The build-only `NETLIFY` -variable is not used to detect the Functions runtime. +Netlify sets the reserved `SITE_ID` variable in builds and Functions. Form Forge +uses it to apply Netlify's client IP handling and to enforce the production +configuration checks, since Netlify does not run `next start`. Never set +`SITE_ID` on a self-hosted installation. -Production on Netlify reads only `x-nf-client-connection-ip` with `TRUST_PROXY=1`. -It ignores client-supplied `X-Form-Forge-Client-IP` and `X-Forwarded-For`. -Self-hosted production continues to require the dedicated ingress header. +On Netlify, with `TRUST_PROXY=1`, the client IP address comes only from the +`x-nf-client-connection-ip` header; client-supplied `X-Form-Forge-Client-IP` and +`X-Forwarded-For` are ignored. -See [Functions environment variables](https://docs.netlify.com/build/functions/environment-variables/) -and [Netlify's client IP header guidance](https://answers.netlify.com/t/is-the-client-ip-header-going-to-be-supported-long-term/11203). +See [Functions environment variables](https://docs.netlify.com/build/functions/environment-variables/). ## 4. Configure GitHub OAuth -Create a production GitHub OAuth app using your permanent public hostname: +Create a GitHub OAuth app for your permanent hostname: -- Homepage: `https://forms.example.org`. -- Callback: `https://forms.example.org/api/auth/callback/github`. +- Homepage: `https://forms.example.org` +- Callback: `https://forms.example.org/api/auth/callback/github` -Enter its client ID and secret in the Netlify environment settings. +Add its client ID and secret to the Netlify environment variables. ## 5. Deploy and sync Inngest -Deploy once database migrations and environment values are ready. In the Inngest -production dashboard, select **Apps > Sync new app** and enter -`https://forms.example.org/api/inngest`. Ensure site/password protection does not -block Inngest requests. Inngest verifies requests using `INNGEST_SIGNING_KEY`. -Keep the same origin in `INNGEST_SERVE_ORIGIN`. +Deploy once the database is migrated and the environment variables are set. In +the Inngest production dashboard, select **Apps → Sync new app** and enter +`https://forms.example.org/api/inngest`. Make sure site or password protection +does not block Inngest. Inngest signs its requests with `INNGEST_SIGNING_KEY`; +keep the same origin in `INNGEST_SERVE_ORIGIN`. -After deploying changes to function definitions, sync the app again. Automatic -sync is not configured: the optional `netlify-plugin-inngest` is not required and -is not installed. This keeps deployment free of extra build dependencies. -The existing `/api/inngest` Node.js route serves the functions; no separate -Netlify function endpoint or Netlify Scheduled Function is needed. +Sync the app again after deploying changes to background functions; automatic sync +is not configured. The `/api/inngest` route serves all functions, so no separate +Netlify Function or Scheduled Function is needed. -Confirm three functions are registered and observe an actual completed minute -cron run: +Confirm that three functions are registered and wait for a completed run of the +five-minute cron job: -- `process-submission`. -- `process-webhook-delivery`. -- `drain-outbox` (`*/5 * * * *`). +- `process-submission` +- `process-webhook-delivery` +- `drain-outbox` (`*/5 * * * *`) -Sign in through GitHub, publish a form, submit a response, and inspect Submissions. -Configure an owner-controlled public HTTPS webhook, submit again, and verify a -successful delivery attempt and the corresponding Inngest runs. A successful -build or stored response alone does not verify background delivery. +Then sign in with GitHub, publish a form and submit a response. Add a webhook +endpoint you control, submit again, and check for a successful delivery attempt +and the matching Inngest runs. A successful build or stored response does not +confirm that background delivery works. See [Inngest app sync](https://www.inngest.com/docs/platform-and-operations/apps-and-syncs) -and [Netlify deployment options](https://www.inngest.com/docs/durable-execution/deploying-functions/platforms/netlify). +and [deploying to Netlify](https://www.inngest.com/docs/durable-execution/deploying-functions/platforms/netlify). + +## Preview deploys -## Preview isolation, limits, and backups +Deploy Previews and branch deploys need their own configuration: a separate Neon +branch or database, matching application URLs, separate GitHub OAuth credentials +and a separate Inngest environment. Migrate that database explicitly. Never give +previews production database credentials or sync them into production Inngest. A +stable staging hostname is easier to use with OAuth than changing preview URLs. +The repository does not provision preview resources automatically, so a preview +without valid configuration fails to build. -Deploy Previews and branch deploys need isolated configuration: a separate Neon -branch/database, matching application URLs, separate GitHub OAuth credentials, -and an isolated Inngest environment. Explicitly migrate that database. Never let -previews inherit production database credentials or sync them into production -Inngest. A stable staging hostname is easier for OAuth than changing preview URLs. -Automatic per-preview database/environment provisioning is not configured here; -previews without valid configuration intentionally fail the build. +## Limits and backups Netlify synchronous Functions have a 60-second execution limit and a 6 MB -buffered request/response limit; binary encoding can reduce the effective request -limit to about 4.5 MB. These are below the app's existing 8,000,000-byte submission -and 16,000,000-byte schema limits. Oversized requests can be rejected by Netlify -before app validation. The self-hosted payload allowance is not preserved here. -The Next.js route's `maxDuration` setting does not configure Netlify limits. -See [Functions configuration](https://docs.netlify.com/build/functions/configuration/). - -Use Neon backups/restore for application data and retain installation keys. -Inngest Cloud manages scheduler state/history. Docker Compose backup scripts do -not apply to these managed services. Delivery remains at-least-once; receivers -must deduplicate `X-Form-Forge-Id`. +request and response limit; binary encoding can reduce the usable request size to +about 4.5 MB. Both are below Form Forge's own limits of 8,000,000 bytes per +submission and 16,000,000 bytes per schema, so Netlify can reject large requests +before the application sees them. A route's `maxDuration` setting does not change +Netlify's limits. See +[Functions configuration](https://docs.netlify.com/build/functions/configuration/). + +Use Neon's backup and restore features for application data, and keep copies of +your installation keys. Inngest Cloud manages scheduler state. The Docker Compose +backup scripts do not apply to this setup. Delivery is at least once, so +receivers must deduplicate by `X-Form-Forge-Id`. diff --git a/deploy/README.md b/deploy/README.md index 8c93bbb..52a3c3d 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -1,51 +1,54 @@ -# Form Forge 0.2.0 self-hosting - -This is one single-host Docker Compose deployment, built from source for the -installation's public hostname. See [release notes](RELEASE_NOTES.md) for the supported scope and -[verification](VERIFICATION.md) for reproducible checks and their limits. - -## Contract and prerequisites - -- A dedicated Linux Docker host with Compose v2 or newer, persistent local disk, - and a public DNS hostname pointing to that host. The reference configuration is verified with Linux arm64 containers. - Validate other architectures on a disposable host before deployment. -- An owner-managed, publicly trusted TLS certificate and private key for the - hostname. Renew them outside this stack and reload Nginx after renewal. -- Python 3.9 or newer, OpenSSL, and `shasum` for packaging and backups. - Automated acceptance also requires curl. -- A GitHub OAuth application and a controlled public HTTPS webhook receiver for - installation-specific acceptance. The local acceptance fixture requires none - of these external accounts and does not validate live OAuth. -- Docker registry and package registry access at build/pull time. Production - startup installs no packages. Dependencies are pinned by `pnpm-lock.yaml` and - installed with pnpm 12.17.0 and `--frozen-lockfile`. - -Runtime images are pinned by tag **and digest** in Compose/Dockerfile: Node -24.22.0 (Debian bookworm slim), PostgreSQL 16.14, Redis 7.4.8, Inngest 1.46.0, -Nginx 1.30.5. Next.js 16.4.0 and Inngest SDK 4.23.0 come from the existing lockfile. -No additional dependency is required. - -The application database is `form_forge` in PostgreSQL, with its own credentials -and `application-data` volume. Scheduler configuration and history live in -SQLite under `/var/lib/inngest` in `scheduler-history`. Queue and run state live -in a separate password-protected Redis in `scheduler-queue`, with AOF enabled, -`appendfsync always`, and `noeviction`. The SQLite filesystem boundary replaces -a scheduler database credential; Redis credentials are separate from application -database credentials. Inngest's default in-memory Redis is not used. - -This configuration follows the [Inngest self-hosting documentation](https://www.inngest.com/docs/self-hosting) -and the flags verified against [Inngest 1.46.0](https://github.com/inngest/inngest/releases/tag/v1.46.0). -It provides storage across container recreation and measured graceful scheduler -restart. It makes no power-loss, HA, multi-host, or point-in-time recovery claim. -Redis fsync cannot by itself establish a cross-store consistency guarantee. - -## Obtain the source - -Use a reviewed commit from the [repository](https://github.com/0then0/form-forge) -or a source archive supplied by your release maintainer. Keep the commit ID and -checksums with the installation record; a moving branch does not identify a release. - -For a source checkout, replace `` with the reviewed commit ID: +# Self-hosting with Docker Compose + +This guide installs Form Forge on a single Linux host with Docker Compose. The +images are built from source for your public hostname. For managed hosting, see +the [Netlify guide](NETLIFY.md) instead. + +The stack contains: + +- **nginx:** the only public service. It terminates TLS on port 443. +- **web:** the Next.js application. +- **postgres:** the application database (`form_forge`), stored in the + `application-data` volume. +- **inngest:** a self-hosted [Inngest](https://www.inngest.com/docs/self-hosting) + scheduler. Its configuration and history use SQLite in the `scheduler-history` + volume. +- **redis:** Inngest's queue and run state, password-protected, in the + `scheduler-queue` volume. It uses AOF persistence with `appendfsync always` and + the `noeviction` policy. +- **migrate:** a one-off service that applies database migrations. + +Images are pinned by tag and digest: Node.js 24.22.0 (Debian bookworm slim), +PostgreSQL 16.14, Redis 7.4.8, Inngest 1.46.0 and Nginx 1.30.5. Application +dependencies are installed from `pnpm-lock.yaml` with pnpm 12.17.0 and +`--frozen-lockfile`; nothing is installed when containers start. + +This is a single-host setup. It survives container recreation and graceful +restarts, but does not provide high availability, multi-host operation or +point-in-time recovery, and is not designed to survive power loss without data +loss. + +## Requirements + +- A dedicated Linux host with Docker and Compose v2, persistent local disk, and a + public DNS name pointing to it. The reference configuration is tested with + Linux arm64 containers; test other architectures on a disposable host first. +- A publicly trusted TLS certificate and private key for that name. Renew them + outside this stack and reload Nginx afterwards. +- Python 3.9 or newer, OpenSSL and `shasum` for backups and source packages, and + curl for the automated acceptance test. +- A GitHub OAuth application. +- Access to Docker Hub and the npm registry while building. + +Node.js and pnpm are not needed on the host. + +## 1. Get the source + +Use a [release](https://github.com/0then0/form-forge/releases) or a reviewed +commit. Record the commit ID or the release checksums with your installation; a +branch name does not identify a version. + +From Git, replace `` with a release tag or commit ID: ```sh git clone https://github.com/0then0/form-forge.git @@ -53,41 +56,19 @@ cd form-forge git checkout ``` -Continue with **Configure once** below. Local Node.js and pnpm are not required -for the Docker build. - -### Install from the source artifact - -The artifact contract is `form-forge-0.2.0.tar.gz`, `SOURCE.json`, and -`SHA256SUMS`. `SOURCE.json` records the commit ID and the SHA-256 hash of each -source file. Checksums verify integrity, not publisher identity. Verify and extract -the artifact supplied by your release maintainer: +From a release archive, download `form-forge-.tar.gz`, `SOURCE.json` and +`SHA256SUMS` from the same release, then verify and extract: ```sh shasum -a 256 -c SHA256SUMS -tar -xzf form-forge-0.2.0.tar.gz -cd form-forge-0.2.0 +tar -xzf form-forge-.tar.gz +cd form-forge- ``` -Published artifacts are available on the -[GitHub Releases page](https://github.com/0then0/form-forge/releases). Verify all -three files from the same release before extracting the archive. - -For maintainers, generate this artifact from a clean, committed checkout with -`python3 deploy/package.py`. It does not commit, create a tag, push, publish an -image, or create a GitHub Release. Re-running it against identical source produces -the same checksums. The packager refuses dirty checkouts and includes only tracked regular files. The application and -workspace npm packages keep their existing internal `0.0.0` versions; the -deployment release version is `deploy/VERSION`. - -To publish a reviewed revision, push a tag named `v`. The -[release workflow](../.github/workflows/release.yml) checks that the tag matches -that version, packages the committed source, verifies checksums, uploads the -three artifacts to a draft GitHub Release, then publishes it. Versions containing -a hyphen, such as `0.2.0-rc.1`, are published as prereleases. The workflow uses -GitHub's scoped job token and requires no registry or personal access credentials. +`SOURCE.json` records the commit ID and the SHA-256 hash of every source file. +Checksums verify integrity, not who published the files. -### Configure once +## 2. Configure ```sh cp deploy/.env.example deploy/.env @@ -95,42 +76,48 @@ chmod 600 deploy/.env mkdir -m 700 deploy/tls ``` -Edit `deploy/.env`. Set `PUBLIC_HOST` to the bare hostname, such as -`forms.your-domain.org`, without a protocol, port, path or trailing slash. This -deployment supports the HTTPS root origin on port 443. Set a **different** value -for each of `POSTGRES_PASSWORD`, `REDIS_PASSWORD`, `AUTH_SECRET`, -`FINGERPRINT_SECRET`, `INNGEST_EVENT_KEY` and `INNGEST_SIGNING_KEY`, generating each -with `openssl rand -hex 32`. The passwords must be hex to avoid URI escaping -ambiguity. Generate `WEBHOOK_ENCRYPTION_KEY` with `openssl rand -base64 32`. -Copy the resulting values into the file without printing it in logs. +Edit `deploy/.env`: -Create a [GitHub OAuth application](https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/creating-an-oauth-app) with: +- `PUBLIC_HOST`: the bare hostname, such as `forms.example.org`, without protocol, + port, path or trailing slash. The site is served at the root of + `https://` on port 443. +- `POSTGRES_PASSWORD`, `REDIS_PASSWORD`, `AUTH_SECRET`, `FINGERPRINT_SECRET`, + `INNGEST_EVENT_KEY` and `INNGEST_SIGNING_KEY`: a different value for each, + generated with `openssl rand -hex 32`. Use hex values for the passwords so they + need no escaping in connection URLs. +- `WEBHOOK_ENCRYPTION_KEY`: generate with `openssl rand -base64 32`. -- Homepage: `https://`. -- Callback: `https:///api/auth/callback/github`. +Create a +[GitHub OAuth application](https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/creating-an-oauth-app) +with the homepage `https://` and the callback +`https:///api/auth/callback/github`, and set `AUTH_GITHUB_ID` and +`AUTH_GITHUB_SECRET`. Google and GitLab sign-in are optional; see +[sign-in methods](../docs/user-guide.md#sign-in-methods-and-account-linking). -Set `AUTH_GITHUB_ID` and `AUTH_GITHUB_SECRET` to its real client ID and secret. -GitHub is required; Google and GitLab are optional sign-in providers, configured with the credential pairs documented in the root README. Put the TLS certificate chain in -`deploy/tls/fullchain.pem` and its private key in `deploy/tls/privkey.pem`. -Protect these files and the Docker host. Environment values are visible to Docker -administrators; `.env` is an installation secret store, not a multi-tenant vault. +Put the certificate chain in `deploy/tls/fullchain.pem` and the private key in +`deploy/tls/privkey.pem`. -Back up this configuration securely. Keep `AUTH_SECRET`, `FINGERPRINT_SECRET`, -`WEBHOOK_ENCRYPTION_KEY`, and both Inngest keys stable across restarts and upgrades. -Do not regenerate them at startup. Changing the encryption key makes existing -endpoint secrets unreadable; changing the fingerprint key changes rate identities. -Updating PostgreSQL/Redis passwords also requires updating their existing stores, -not just editing `.env` (PostgreSQL initialization only runs on an empty volume). +Back up `deploy/.env` and the TLS files securely. Keep `AUTH_SECRET`, +`FINGERPRINT_SECRET`, `WEBHOOK_ENCRYPTION_KEY` and both Inngest keys unchanged +across restarts and upgrades: -The build receives only `NEXT_PUBLIC_APP_URL=https://`, never these -secrets. Next.js inlines public configuration at build time. Rebuild for a changed -hostname; this is not a universal image that can be moved between hostnames. -The startup wrapper enforces the same production guard as `next start` and rejects -a runtime URL that differs from the baked URL. +- changing `WEBHOOK_ENCRYPTION_KEY` makes stored webhook secrets unreadable; +- changing `FINGERPRINT_SECRET` resets rate-limit identities and deletion markers; +- changing a database or Redis password in `.env` does not change it in the + existing store, because PostgreSQL applies its password only when initializing + an empty volume. -### Build, migrate, start +Docker administrators can read these values; treat the host accordingly. -Use this shell function from the extracted source root for the commands below: +Only `NEXT_PUBLIC_APP_URL=https://` is passed to the image build; +secrets are not. Because Next.js embeds the public URL at build time, rebuild after +changing the hostname. The container refuses to start if its runtime URL differs +from the URL it was built with. + +## 3. Build, migrate and start + +Run these commands from the source root. The `dc` function keeps the commands +short: ```sh dc() { docker compose --env-file deploy/.env -f deploy/compose.yaml "$@"; } @@ -142,31 +129,31 @@ dc up -d --wait dc ps ``` -`migrate` is an operations-profile service. It is not started by ordinary `up` and -is not an application entrypoint. It runs the committed Drizzle migrations with -the same locked toolchain. Only one operator should run it at a time. Reapplying -it is safe; do not squash, edit, delete or reset migration history. +`migrate` belongs to the `operations` profile, so `dc up` does not start it. It +applies the committed migrations and can be run again safely, but only one +operator should run it at a time. Never edit, squash or delete migration history. -Normal stop/start preserves all three named volumes: +To stop and start the stack while keeping all data: ```sh dc stop dc up -d --wait ``` -Do **not** use `down --volumes` on an installation whose data you want to retain. -The application readiness check tests PostgreSQL connectivity, the exact latest -committed migration hash and a rendered HTTP route. It does not prove pipeline -health. Inngest's own healthcheck only proves its HTTP surfaces are responding. +Do **not** run `dc down --volumes` on an installation whose data you want to +keep. -### Verify registration and the five-minute cron +The application's health check confirms the database connection, that the latest +migration is applied and that a page renders. It does not confirm background +delivery; the next step does. -Inngest polls the internal serve URL every 30 seconds. The app uses -`INNGEST_DEV=0`, `INNGEST_BASE_URL=http://inngest:8288`, the matching event/signing -keys, and `INNGEST_SERVE_ORIGIN=http://web:3000`. Incoming invocations retain the -SDK's signature validation. No unauthenticated sync switch is enabled. +## 4. Verify background jobs -Inspect the private scheduler from the web container: +The application registers its functions with the private Inngest service, which +polls `http://web:3000/api/inngest` every 30 seconds. Requests between them are +signed with the Inngest keys. + +List the registered functions and recent runs from inside the stack: ```sh dc exec -T web node --input-type=module <<'JS' @@ -179,191 +166,194 @@ console.log(JSON.stringify(await response.json(), null, 2)); JS ``` -Require `form-forge-process-submission`, `form-forge-process-webhook-delivery`, -and `form-forge-drain-outbox`, with cron `*/5 * * * *`. Wait for an actual completed -`drain-outbox` run. Registration and a healthy process alone are insufficient. -Do not invoke internal recovery functions manually in an acceptance drill. +Confirm that these functions are registered: + +- `form-forge-process-submission` +- `form-forge-process-webhook-delivery` +- `form-forge-drain-outbox`, with the cron trigger `*/5 * * * *` -Open `https:///login`, sign in through GitHub, publish a small form, -configure your controlled HTTPS endpoint, and submit synthetic data. Verify a -persisted delivery and HMAC over original request bytes at the receiver. -Use the [installation verification procedure](acceptance/README.md) to finish the public -networking and OAuth checks for your installation. +Wait until at least one `drain-outbox` run has completed; a registered function +alone is not enough. + +Then open `https:///login`, sign in with GitHub, publish a small +form, add a webhook endpoint you control, and submit test data. Check that the +delivery succeeds and that your receiver can verify its signature. The +[installation checklist](acceptance/README.md) covers TLS, OAuth, networking and +recovery in more detail. ## Network boundary -Only Nginx port 443 is published. There are no host ports for web, PostgreSQL, -Redis, Inngest dashboard/API, metrics, or Connect. The application origin is -accessible to containers on the private stack network and to Docker administrators. -Do not attach untrusted containers or publish those origin ports. - -This Nginx directly faces clients and terminates TLS. It **replaces** incoming -`X-Form-Forge-Client-IP` and `X-Forwarded-For` with `$remote_addr`; it does not append -or trust client values. `TRUST_PROXY=1` is preserved. This deployment is not -configured for a CDN or another proxy in front: that requires an explicit trusted -address chain before public use. Do not blindly trust forwarded client headers. - -Nginx allows 8,000,000-byte bodies, matching the submission limit. Body upload -timeout is 15 seconds and upstream read/send timeout is 75 seconds. Webhook -transport has its existing 10-second timeout; worker lease is 60 seconds. This -does not impose a new application quota or modify target restrictions. The public -ingress returns 404 for `/api/inngest` and `/api/internal/`; Inngest uses signed -private calls. Authentication and workspace authorization are unchanged. - -Access scheduler diagnostics with `dc exec`, or a temporary administrator-only -SSH tunnel and an explicitly loopback-bound port if you need its UI. No permanent -admin port or anonymous public dashboard is part of this deployment. Backlog -warnings only run while the scheduler is running, so also monitor missing cron -runs, disk space and growing history. Error reporting is disabled in this reference -configuration; the existing backlog warning remains in application logs. - -The upstream uses Docker's internal DNS resolver and refreshes changed web -container addresses. This supports container recreation during maintenance -without retaining an obsolete origin IP; see the -[Nginx upstream documentation](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#resolve). +Only Nginx port 443 is published. PostgreSQL, Redis, the application port and the +Inngest dashboard and API have no host ports and are reachable only from the +private Compose network. Do not attach untrusted containers to that network or +publish those ports. + +Nginx faces clients directly. It **replaces** the incoming +`X-Form-Forge-Client-IP` and `X-Forwarded-For` headers with the connecting +address, and the application runs with `TRUST_PROXY=1`. If you put a CDN or +another proxy in front of Nginx, configure a trusted address chain first; +otherwise every client appears to come from the proxy. + +Nginx accepts request bodies up to 8,000,000 bytes, matching the submission +limit, with a 15-second body upload timeout and 75-second upstream timeouts. It +returns 404 for `/api/inngest` and `/api/internal/`; Inngest reaches the +application over the private network. + +To open the Inngest dashboard, use `dc exec` or a temporary SSH tunnel to a port +bound to loopback. Do not expose the dashboard publicly. + +Sentry is not configured in this stack; the delivery backlog warning appears in +the application logs. Because that warning only runs while the scheduler is +running, also monitor for missing cron runs, disk space and history growth. ## Upgrade and rollback -Before accepting a new artifact, verify its checksum, release limitations and -source provenance. Extract to a new release directory. Copy the existing `.env` -and TLS material securely, without regenerating keys. Keep the same Compose -project name so the same named volumes are used. Build the new `web` and `migrate` -images for the same public hostname before the maintenance window. - -1. Schedule downtime, stop ingress/writes and workers, and take the backup below. -2. With PostgreSQL available and workers stopped, run the new `dc run --rm migrate`. -3. Start Redis, application, scheduler and ingress with `dc up -d --wait`. -4. Verify all three functions, an actual completed five-minute cron and a synthetic - delivery before resuming client traffic. `up --wait` alone is insufficient. - -For a same-release recreation drill, `dc stop; dc up -d --force-recreate --wait` -must retain existing data and keys. The automated fixture checks worker restarts, -storage restarts and application recreation during restore; the entire maintenance -upgrade sequence remains an owner-run check. A future-version upgrade cannot be -certified before its schema and release exist. Roll application code back only when it is compatible with the -already applied migrations and scheduler configuration. There are no destructive -down migrations. Otherwise restore a complete backup into a new destination; -do not drop tables or delete history to force a rollback. +Before upgrading, read the release notes and verify the new release's checksums. + +1. Extract the new release into a new directory. Copy `deploy/.env` and + `deploy/tls/` securely, without regenerating any keys. +2. Keep the same Compose project name so that the existing volumes are used. +3. Build the new images for the same hostname: `dc build --no-cache web migrate`. +4. Schedule downtime, stop the ingress, application and workers, and take a + [backup](#backup-and-isolated-restore). +5. With PostgreSQL running and workers stopped, run `dc run --rm migrate`. +6. Start everything with `dc up -d --wait`. +7. Before resuming traffic, repeat the checks in + [Verify background jobs](#4-verify-background-jobs), including a completed cron + run and a test delivery. `up --wait` alone does not confirm this. + +Roll application code back only to a version that is compatible with the applied +migrations. There are no down migrations; otherwise restore a complete backup into +a new installation. Never drop tables or delete migration history to force a +rollback. ## Backup and isolated restore -Backups cover **three stores plus installation keys**. An application `pg_dump` -alone does not capture scheduler queue, run state, configuration or history. -For a controlled maintenance snapshot, with no other database writers: +A complete backup contains **three stores and the installation keys**: the +PostgreSQL database, the Inngest SQLite directory, the Redis data directory, and +the configuration. A `pg_dump` alone does not include the scheduler's queue, runs, +configuration or history. -```sh -sh deploy/backup.sh /absolute/new/form-forge-backup -``` +### Create a backup -The script stops ingress, web, Inngest and Redis, verifies they are stopped, and creates a PostgreSQL custom dump, -archives the entire stopped SQLite directory (including WAL files) and entire -Redis data directory (including multipart AOF), and copies the installation -environment, release provenance, Compose files, Nginx template and TLS files. Every copied file is covered by the checksum manifest. An `INCOMPLETE` marker remains until dump/archive validation and checksum verification finish; a failed backup must not be restored. The destination must be outside the project directory, including when reached -through a symlink. The script refuses to overwrite a directory and leaves -services stopped for inspection. For an installation with a custom project or -Compose overlay, pass the same standard Compose selectors to the script, for example -`COMPOSE_PROJECT_NAME=my-installation COMPOSE_FILE=deploy/compose.yaml:deploy/local.yaml sh deploy/backup.sh /absolute/new/backup`. -The files must live inside the installation; preserve the backed-up overlays and -review installation-specific addresses and mounts for the destination. Encrypt the entire backup at rest and -move it to restricted storage. Dumps and scheduler history can contain personal -data; never attach them to evidence or release artifacts. Resume with `dc up -d --wait`, then repeat registration/cron checks. - -Rehearse restore on a separate Docker project, never over the source. Extract the -**same** source release to a new directory, securely copy `installation.env` to -`deploy/.env` and backup TLS to `deploy/tls`. Bind `HTTPS_BIND=127.0.0.1` and set an -unused `HTTPS_PORT` (for example 19443) so it cannot accept public traffic. Use a -separate function. From the new source root, verify the **entire** backup before -creating a destination. The strengthened checksum/provenance set is produced by -this revised script; original rc.1 backups omit these metadata and TLS coverage -and cannot pass this stricter verification unchanged. `backup_manifest.py verify` refuses an incomplete set, -uncovered/missing files, checksum mismatches and unsafe scheduler archives. -`shasum` alone cannot establish that required files were not omitted. +Run the backup during a maintenance window, with no other database writers: ```sh -python3 deploy/backup_manifest.py verify /backup -(cd /backup && shasum -a 256 -c SHA256SUMS) -restore_dc() { docker compose --env-file deploy/.env -p form-forge-restore -f deploy/compose.yaml "$@"; } -restore_dc config --quiet -restore_dc up -d --wait postgres -restore_dc exec -T postgres pg_restore -U form_forge -d form_forge --no-owner --exit-on-error < /backup/application.dump -restore_dc run --rm --no-deps -T --entrypoint sh inngest -c 'cd /var/lib/inngest && tar -xzf -' < /backup/scheduler-history.tar.gz -restore_dc run --rm --no-deps -T --entrypoint sh redis -c 'cd /data && tar -xzf -' < /backup/scheduler-queue.tar.gz +sh deploy/backup.sh /absolute/path/to/new/form-forge-backup ``` -These commands require all three destination volumes to be new and empty and all -writers to be stopped. Refuse a pre-existing project/volume; inspect the resolved -Compose configuration and container mounts to rule out source volume names, bind -mounts, external application/scheduler stores and source service addresses. Stop -the source after backup but retain it until the drill finishes. The disposable -runner verifies this isolation; it shares only the controlled receiver network. -The keys come from `installation.env`; keep `VERSION`, `SOURCE.json`, `BACKUP.json`, -`compose.yaml`, `nginx.conf.template` and `configuration/` with the backup. Match -its source hashes/version with the original source release. For custom overlays, -copy the recorded `configuration/` files and adjust only destination project, -paths and local ingress binding. No migration or fixture SQL repairs scheduler -registration during a same-release restore. Match container ownership and image -versions; do not import a live Redis directory or only SQLite's `main.db` while -its writers are running. Build for the preserved hostname and use local DNS -resolution to reach the loopback HTTPS port. Before workers start, compare published schema IDs/hashes, individual submission -and endpoint identities, outbox states/due times, logical delivery IDs/retry budgets -and attempt history, and verify every endpoint secret decrypts with the preserved -key. A one-off verification process may read the restored database without starting -workers. Refuse decryption failure before allowing delivery processing. Inspect the -restored scheduler directories/history before starting Inngest; application outbox -recovery is not evidence that scheduler archives were restored. - -Then run `restore_dc up -d --wait`, verify the same registered function identities, -new completed five-minute cron runs, and real HTTPS delivery under restored logical IDs. -A due retry may already be overdue: application cron dispatches the stored outbox -and repairs due deliveries, rather than preserving a future Inngest sleep for each -application retry. -Do not resume a restore against real receiver endpoints without an agreed -redelivery window; use an owner-controlled isolated receiver for drills. - -The automated test runs this complete cold restore using `deploy/backup.sh` in a -disposable installation and a second Compose project. One snapshot contains pending -outbox, a persisted 503 with pending retry, and completed delivery history. It checks -new empty stores, full restored scheduler-directory file hashes, SQLite identities -and history, surviving Redis values before workers, application invariants and -secret decryption. Corrupted archive and wrong-key copies must fail before workers. -It then requires real cron recovery and raw-byte HTTPS HMAC validation and observes -completed logical work across two additional completed recovery ticks. This finite -observation is not a guarantee of no HTTP replay forever. - -Application outbox recovery also repairs incomplete processing: pending/failed -events retry; sent submissions without deliveries can be requeued after five -minutes; expired 60-second worker leases are recovered by the five-minute cron. Outbox -dispatch backs off from one minute to one hour and sends up to 50 events per cron -run. Delivery retry delays are 1 minute, 5 minutes, 30 minutes and 2 hours, within -five initial attempts. Recovery may take several ticks and is not instantaneous. -Using fresh scheduler state can lose history and schedules; it is not equivalent -to restoring a matching snapshot. Do not clear scheduler volumes as a recovery -shortcut or promise a recovery point without measuring it. - -Delivery is at-least-once. A restore or lost acknowledgement can repeat HTTP -delivery under the same logical `X-Form-Forge-Id`. Receivers must durably deduplicate -that ID before performing side effects. Restoring an old backup may repeat work -accepted after that backup. No exactly-once network delivery or point-in-time -guarantee is claimed. - -## Repeat the automated acceptance - -On a disposable Docker host with Python 3.9 or newer, OpenSSL and curl: +The script: + +1. stops Nginx, the application, Inngest and Redis, and verifies that they are + stopped; +2. creates a PostgreSQL custom-format dump; +3. archives the whole SQLite directory, including WAL files, and the whole Redis + data directory, including multipart AOF files; +4. copies the installation environment, release provenance, Compose files, Nginx + template and TLS files; +5. writes a checksum manifest covering every file. + +The destination must be a new directory outside the source tree, including +through symlinks. An `INCOMPLETE` marker stays in the backup until all files are +validated; never restore a backup that still has it. The script leaves services +stopped; resume with `dc up -d --wait` and repeat the +[background job checks](#4-verify-background-jobs). + +If your installation uses a custom project name or additional Compose files, pass +the same standard Compose variables to the script, for example +`COMPOSE_PROJECT_NAME=my-installation COMPOSE_FILE=deploy/compose.yaml:deploy/local.yaml sh deploy/backup.sh /absolute/path/to/backup`. +The extra files must be inside the installation directory and are included in the +backup. + +Backups contain secrets and personal data. Encrypt the whole backup at rest and +move it to restricted storage. + +### Restore into a separate installation + +Rehearse restores in a separate Compose project, never over the source +installation. Backups made with 0.1.0-rc.1 lack the metadata that the +verification step requires. + +1. Extract the **same** release into a new directory. Copy + `installation.env` from the backup to `deploy/.env` and the backup's TLS files to + `deploy/tls/`. +2. In `deploy/.env`, set `HTTPS_BIND=127.0.0.1` and an unused `HTTPS_PORT`, such as + `19443`, so the copy cannot receive public traffic. +3. Verify the whole backup before creating anything: + + ```sh + python3 deploy/backup_manifest.py verify /backup + (cd /backup && shasum -a 256 -c SHA256SUMS) + ``` + + `backup_manifest.py verify` rejects incomplete backups, missing or uncovered + files, checksum mismatches and unsafe archive contents. `shasum` alone cannot + detect a missing file. + +4. Restore the three stores into new, empty volumes of a separate project: + + ```sh + restore_dc() { docker compose --env-file deploy/.env -p form-forge-restore -f deploy/compose.yaml "$@"; } + restore_dc config --quiet + restore_dc up -d --wait postgres + restore_dc exec -T postgres pg_restore -U form_forge -d form_forge --no-owner --exit-on-error < /backup/application.dump + restore_dc run --rm --no-deps -T --entrypoint sh inngest -c 'cd /var/lib/inngest && tar -xzf -' < /backup/scheduler-history.tar.gz + restore_dc run --rm --no-deps -T --entrypoint sh redis -c 'cd /data && tar -xzf -' < /backup/scheduler-queue.tar.gz + ``` + +5. Before starting workers, check the restored data: published schemas, response + and endpoint counts, pending outbox events and deliveries, and that every + webhook endpoint secret decrypts with the restored key. Stop if any secret fails + to decrypt. +6. Build for the original hostname, resolve it locally to the loopback port, and + start the stack with `restore_dc up -d --wait`. Confirm the registered + functions, new completed cron runs and a real HTTPS delivery. + +Keep the source installation stopped but intact until the rehearsal is finished. +Check that the restore project uses only its own volumes, mounts and service +addresses. Never copy a Redis directory or SQLite database while their writers are +running. With custom Compose files, use the copies stored in the backup's +`configuration/` directory and change only the project name, paths and local +port binding. + +Use a test receiver for restore drills. A restore can resend deliveries that were +already sent, and retries that were due during the outage are sent by the next +recovery run. + +### Recovery behavior + +The recovery job, which runs every five minutes, repairs interrupted work: + +- pending and failed outbox events are retried, backing off from one minute to one + hour, up to 50 events per run; +- responses whose deliveries were never created are requeued after five minutes; +- deliveries whose 60-second worker lease expired are picked up again. + +Webhook retries are spaced 1 minute, 5 minutes, 30 minutes and 2 hours apart, +within five attempts. Recovery can take several runs. Starting with empty +scheduler volumes loses scheduler history and is not equivalent to restoring a +backup; do not clear them as a shortcut. + +Delivery is at least once. A restore or a lost acknowledgement can repeat a +delivery with the same `X-Form-Forge-Id`, and restoring an older backup can repeat +work accepted after it was taken. Receivers must store that ID and skip +duplicates before performing side effects. + +## Automated acceptance test + +To test the whole stack on a disposable Docker host with Python 3.9 or newer, +OpenSSL and curl: ```sh python3 deploy/acceptance/run.py --cold-restore-only --output deploy/evidence/local.json ``` -The runner creates a random project and new volumes, builds from locked source, -generates temporary credentials/certificates, migrates an empty database, exercises -real signed Inngest calls and real HTTPS transport, and cleans only its resources -in `finally`. Loopback ports 18443 and 19443 and the fixture subnet must be free. Fixture identities, -schemas and endpoints are inserted directly into the disposable database; this is -not a live OAuth test. No production authentication bypass is introduced. - -The receiver address `192.0.3.10` exists only on an `internal: true` Docker network -with no external route. A `.invalid` host is mapped inside the web fixture, and a -temporary CA is trusted only by the test container. No request reaches that public -IP's real-world owner or a third-party webhook service. The production transport -and target policy are unchanged. This proves controlled HTTPS transport, not -public DNS, public CA issuance or installation-owner networking. +The test builds the images, creates a separate Compose project with temporary +keys and certificates, and exercises real Inngest and HTTPS delivery, a backup and +a restore into a second project. It removes only the resources it created. +Loopback ports 18443 and 19443 and the test subnet must be free. + +The test receiver runs on an isolated internal Docker network, so no requests +leave the host. The test inserts its users and forms directly into its own +database; it does not test GitHub sign-in, public DNS or certificate issuance. +See [Deployment verification](VERIFICATION.md) for the full list of checks. diff --git a/deploy/VERIFICATION.md b/deploy/VERIFICATION.md index 3b2fa02..e1fd9d2 100644 --- a/deploy/VERIFICATION.md +++ b/deploy/VERIFICATION.md @@ -1,7 +1,13 @@ -# Verification +# Deployment verification -Run the repository checks before packaging a release. Use disposable databases -for integration and browser tests; those suites remove fixture data. +This page lists the checks to run before a release or deployment and explains +what each one does and does not prove. + +## Repository checks + +Run these from the repository root. Integration and browser tests delete their +data, so point them at disposable databases only; see +[Development](../docs/development.md) for database setup. ```sh pnpm format:check @@ -15,111 +21,100 @@ pnpm build pnpm test:production ``` -Database setup and environment requirements are described in the -[development guide](../docs/development.md). Deployment regression tests -use Python 3.9 or newer and its standard library, temporary Git repositories, and an isolated Docker -build context containing fake secrets. The Docker-context test is skipped when -Docker is unavailable. Receiver tests use temporary TLS certificates and real -HTTPS requests. Neither suite uses installation credentials. +`pnpm test:deployment` tests the packaging, backup and acceptance utilities. It +needs Python 3.9 or newer and uses only the standard library, temporary Git +repositories, fake secrets and temporary TLS certificates. The test that builds +the Docker context is skipped when Docker is unavailable. None of these tests use +installation credentials. + +## Automated acceptance test -## Real scheduler and transport +The acceptance runner deploys the real Docker Compose stack on a disposable host: ```sh python3 deploy/acceptance/run.py --cold-restore-only --output deploy/evidence/local.json ``` -Omit `--cold-restore-only` to also run the existing retained-store restart and -lost-acknowledgement drills. The full cold-restore milestone uses the command above. - -The runner builds the application and migration images, creates a unique Compose -project with empty volumes and temporary keys, and exercises: - -- Runtime production guards, hostname consistency, migration readiness and - repeatable migrations. -- Private services, ingress client-IP replacement, signed Inngest invocations, - registration of all three functions and actual completed five-minute cron runs. -- Atomic submission/outbox persistence during a scheduler outage and normal - cron recovery after restart. -- Real HTTPS delivery with raw-byte HMAC verification, a persisted 503 and - restored retry history/budget. The default extended run also checks graceful - scheduler restart, lost ACK/lease recovery and retained PostgreSQL/Redis restarts. -- A full maintenance cold - restore of PostgreSQL, SQLite and Redis into a separate project with three new - empty volumes, using the real backup script and preserved installation keys. -- Pending outbox, pending retry with retained logical ID/history/budget, completed - delivery preservation and two additional completed recovery ticks after pending - work finishes. The shared controlled receiver retains its ordinals/observations. -- Complete backup checksum coverage, corrupted-archive refusal before destination - startup, and wrong-key decryption refusal before workers. -- Source/destination volume, mount, network, container and database-address isolation; - full scheduler-directory file hashes, SQLite registration/event/run identities and - surviving Redis content digests checked before scheduler startup. - -The report records source hashes, commit, dirty-checkout state, component versions, -architecture, UTC times, database transitions, receiver metadata and cleanup. -A successful run requires all scenarios and resource cleanup to pass. A failed -run returns a nonzero exit code and retains its report. Images and build cache -remain available; the runner never performs a global Docker prune. - -Results are installation-specific artifacts under ignored `deploy/evidence/`. -Do not commit reports, credentials, submission bodies or database dumps. Record -checks as passed, failed or not run; do not substitute test timestamps or generated -IDs for evidence from your own installation. - -## Limits - -The fixture maps an HTTPS hostname onto an isolated internal Docker network and -trusts its temporary CA only inside the test web container. It verifies the -production transport and target policy, but not public DNS, certificate issuance -or GitHub OAuth. Follow [public installation verification](acceptance/README.md) -on a disposable installation with your own hostname, OAuth app and receiver. - -The runner performs the -[complete three-store backup and restore](README.md#backup-and-isolated-restore), -including pending work. It checks the pinned 1.46.0 SQLite schema and Redis -content diagnostically; those internal structures are not a cross-version API. -Redis keys expiring during restore are reported separately. Exact archived files -are compared before Redis starts, so expiration cannot hide an omitted archive. -The retry delay lives in application PostgreSQL/outbox; the drill does not claim -that every application retry corresponds to a restored sleeping Inngest run. -No source containers/volumes are used after backup except the explicitly shared -external receiver fixture. Browser tests use fixture sessions and pipeline -controls. Unit/integration tests cannot replace real scheduler acceptance. - -The reference deployment targets one host and at-least-once delivery. Validate -other CPU architectures, future-version upgrades and expected capacity in your environment. Graceful restart results do not prove -power-loss durability, point-in-time recovery or high availability. Receivers must -persistently deduplicate logical delivery IDs before performing side effects. - -## Product controls after 0.1.0 - -Migration `0006_product_controls` is additive: it preserves metadata collection on -pre-existing forms, defaults new forms to collection off, initializes lifetime -accepted counts from existing responses and adds independent processing status, -expiring deletion-key hashes and a PostgreSQL GIN value-search index. Published -schemas and prior values are not rewritten. Apply migrations to a disposable local -database for verification; production upgrades require the normal operator-led -maintenance/backup procedure. The index build uses the normal migration transaction, -so large installations should schedule a maintenance window. - -The integration suite upgrades a real pre-0006 fixture and checks privacy in both -the receive service and HTTP route, rolling abuse counters, last-place admission, -old-version closure, deletion rollback/worker conflicts (including cron paused -between due selection and outbox insertion), key expiry, roles/workspace -isolation and list/CSV consistency. Component tests exercise optional collection, -embed origin/source/channel validation and observer cleanup, while Playwright -exercises intake changes in an already-open page, immutable processing updates, -confirmed deletion, fresh cached lists after client navigation and multiple cross-origin iframe heights through validation, -conditional fields and success. MCP/OpenAPI tests cover session/origin protection, -bounded shared arguments and actual request schemas. No snapshots are regenerated. - -The real scheduler outage fixture explicitly waits for the failed immediate -ingestion attempt before checking atomicity or taking the cold backup boundary; -its proxy-fingerprint scenario explicitly opts into metadata collection. -The development browser fixture waits for the immediate ingestion outage attempt -before advancing the test-only scheduling clock. This is not evidence of real -Inngest transport. Continue to run the real scheduler/HTTPS acceptance drill for -a deployment; unit, integration, component and Playwright checks do not validate -Netlify's installed adapter or an external receiver. The immediate path uses -Next.js `after` plus Inngest steps, bounded HTTP attempts, and the same claims and -stable event IDs as the five-minute recovery path. +Without `--cold-restore-only`, it also runs restart and lost-acknowledgement +drills on the original installation before the backup and restore. + +The runner builds the application and migration images, creates a Compose +project with empty volumes and temporary keys, and checks: + +- production configuration guards, hostname consistency, migration readiness and + that migrations can be applied again; +- that only the ingress is public, that the ingress replaces the client IP header, + that Inngest calls are signed, that all three functions are registered and that + the five-minute cron job completes; +- that a response and its outbox event are saved together while the scheduler is + stopped, and are delivered after it restarts; +- real HTTPS delivery with signature verification over the raw request bytes, + including a `503` response followed by a successful retry; +- in the extended run: graceful scheduler restart, recovery after a worker is + killed before acknowledging a delivery, and PostgreSQL and Redis restarts; +- a full backup with `deploy/backup.sh` and a restore of PostgreSQL, SQLite and + Redis into a second project with new, empty volumes and the preserved keys; +- after the restore: pending outbox events and retries keep their delivery IDs, + history and attempt budgets, completed deliveries are not resent, and two more + recovery runs complete; +- that the backup checksums cover every file, that a corrupted archive is rejected + before anything is restored, and that a wrong encryption key is detected before + workers start; +- that the restored project shares no volumes, mounts, networks, containers or + database addresses with the source, and that the restored scheduler files, + SQLite records and Redis contents match the backup. + +The JSON report records source hashes, the commit and whether the checkout had +local changes, component versions, architecture, UTC times, database transitions, +receiver metadata and cleanup. The run passes only when every scenario and the +cleanup pass. A failed run exits with a nonzero status and keeps its report. +Images and the build cache are kept; the runner never prunes Docker globally. + +Reports are written to `deploy/evidence/`, which Git ignores. Do not commit +reports, credentials, response bodies or database dumps. + +### Limits of the automated test + +The test receiver uses an HTTPS hostname on an internal Docker network with a +temporary certificate authority that only the test container trusts. This checks +the production HTTPS client and webhook target rules, but not public DNS, +certificate issuance or GitHub sign-in. Complete the +[installation checklist](acceptance/README.md) on a disposable installation with +your own hostname, OAuth app and receiver. + +The SQLite and Redis comparisons depend on the internal formats of Inngest 1.46.0 +and are diagnostic, not a compatibility guarantee across versions. Retry timing +is stored in the application database; the test does not claim that each pending +retry has a matching restored Inngest run. + +The reference deployment runs on one host with at-least-once delivery. Validate +other CPU architectures, upgrades between future versions and your expected load +yourself. Graceful restarts do not prove durability after power loss, +point-in-time recovery or high availability. + +## What the application tests cover + +Unit, component, PostgreSQL integration and Playwright tests cover the product +behavior. In particular: + +- **Intake and privacy:** source metadata collection is enforced by both the + submission service and the HTTP route; rate-limit counters; two visitors + competing for the last place; closing a form that visitors already have open. +- **Response deletion:** rollback when deletion conflicts with a worker, including + a recovery job that has selected due work but not yet inserted its outbox row; + expiry of deletion markers. +- **Access:** workspace roles and isolation between workspaces. +- **Responses:** list and CSV export consistency, processing status updates that + never change submitted values, and fresh lists after client-side navigation. +- **Embedding:** origin, window and channel validation, observer cleanup, and + height changes of several cross-origin iframes through validation errors, + conditional fields and the success message. +- **OpenAPI and MCP:** session and origin protection, bounded arguments, and that + the published schemas match the request validators. +- **Upgrades:** applying migrations to a database with data from before the + intake and privacy controls. + +Browser tests use test sessions and a test-only control for scheduling, so they +do not exercise real Inngest transport. Only the acceptance test above, or your +own installation checks, verify real scheduling, Netlify's adapter or an external +receiver. diff --git a/deploy/acceptance/README.md b/deploy/acceptance/README.md index c8c0502..19e175a 100644 --- a/deploy/acceptance/README.md +++ b/deploy/acceptance/README.md @@ -1,17 +1,20 @@ -# Public installation verification +# Installation checklist -Do this on a disposable installation, with your own public domain, GitHub OAuth -application and controlled public HTTPS receiver. Do not use shared/production -resources or someone else's endpoint. The [automated runner](run.py) performs the complete maintenance cold restore, -including pending outbox/retry and completed delivery, on two independent Compose -projects. Run `python3 deploy/acceptance/run.py --cold-restore-only --output deploy/evidence/local.json` -from the source root. It requires free loopback ports 18443 and 19443 and the fixture -subnet. No live installation is required for this automated drill; -public DNS, TLS and OAuth require these installation-specific checks. +Use this checklist to verify a real installation: public TLS, GitHub sign-in, +webhook delivery to a public receiver, and recovery. Run it on a disposable +installation with your own domain, GitHub OAuth app and an HTTPS receiver you +control. Never use shared or production resources, or someone else's endpoint. + +The [automated acceptance test](../VERIFICATION.md#automated-acceptance-test) +covers delivery, backup and restore on a local, isolated network, but cannot +check public DNS, TLS certificates or live OAuth. This checklist covers those. + +The commands below use the `dc` function from the +[installation guide](../README.md#3-build-migrate-and-start). ## Public TLS and OAuth -Install using [the guide](../README.md), then verify the certificate without +Install using [the guide](../README.md), then check the certificate without `--insecure`: ```sh @@ -21,9 +24,9 @@ curl -i https:///api/inngest The first must render and the second must return 404 at ingress. From another machine verify origin port 3000 and ports 5432, 6379, 8288, 8289 are unreachable. -Sign in through the real GitHub OAuth flow; verify callback and a new personal -workspace, then test owner/editor/viewer permissions with real accounts. Fixture -sessions and a production configuration check are not live OAuth evidence. +Sign in through GitHub and check that the callback succeeds and a personal +workspace is created. Then test owner, editor and viewer permissions with real +accounts. ## Public receiver @@ -92,5 +95,4 @@ Record the release/commit/source checksum, component digests, UTC times, logical event/delivery IDs, transitions, completed cron run IDs, receiver verification and recovery duration relative to due time and cron tick. Exclude credentials, OAuth tokens, sessions and submission bodies. Mark each check passed, failed or not run, -with the exact missing prerequisite. Public networking and live OAuth are not -certified by the local runner or the existing Playwright tests. +with the exact missing prerequisite. diff --git a/deploy/acceptance/run.py b/deploy/acceptance/run.py index 3ed5b62..13b53ae 100755 --- a/deploy/acceptance/run.py +++ b/deploy/acceptance/run.py @@ -118,15 +118,6 @@ def wait(label, probe, timeout=720): names = list(source_record["files"]) else: names = run(["git", "-C", str(ROOT.parent), "ls-files"]).splitlines() - # New acceptance utilities are intentionally uncommitted during review. - names = sorted( - set(names) - | { - "deploy/backup_manifest.py", - "deploy/acceptance/cold_restore.py", - "deploy/tests/test_cold_restore.py", - } - ) source_record = { "commit": run(["git", "-C", str(ROOT.parent), "rev-parse", "HEAD"]), "workingTree": run( @@ -386,7 +377,7 @@ def fixture(path): return slug, version def persisted_outage(submission): - # after() now attempts ingestion after commit. Require the real failed + # after() attempts ingestion right after commit. Require the real failed # attempt to settle before outage/backup assertions, rather than treating # an intermediate lease as a completed backup boundary. rows = sql( diff --git a/docs/api.md b/docs/api.md index eb46cf6..4b0af17 100644 --- a/docs/api.md +++ b/docs/api.md @@ -1,16 +1,23 @@ -# Public API +# Public API, OpenAPI and MCP [Project overview](../README.md) · [Documentation](../README.md#documentation) -Public routes do not require an admin session: +## Public endpoints -- `GET /api/forms/:slug` returns `{ data: { form, schema, versionId, versionNumber } }` for the current published form. -- `POST /api/forms/:slug/submit` accepts `{ versionId, values, context? }` and requires an `Idempotency-Key` header. -- `/f/:slug` serves the hosted form; `?embed=1` selects the iframe layout. The editor provides a ready-to-copy embed snippet; see [embedding](embedding.md). +These routes do not require signing in: + +- `GET /api/forms/:slug` returns the current published form: + `{ data: { form, schema, versionId, versionNumber } }`. +- `POST /api/forms/:slug/submit` accepts `{ versionId, values, context? }` and + requires an `Idempotency-Key` header. +- `/f/:slug` serves the hosted form page. Add `?embed=1` for the iframe layout; + see [Embedding](embedding.md). ## Submit a response -Fetch the published schema first. Use its `versionId` and field keys in the submission. Replace the placeholders below; the form must contain a field with the key `name`: +Fetch the published schema first, then submit values keyed by its field keys, +together with its `versionId`. In this example, the form has a field with the key +`name`: ```sh curl 'http://localhost:3000/api/forms/' @@ -21,65 +28,129 @@ curl -X POST 'http://localhost:3000/api/forms//submit' \ -d '{"versionId":"","values":{"name":"Ada"}}' ``` -A new submission returns `201` with `{ data: { duplicate: false, status: "accepted", submissionId } }`. Acceptance means the response was stored, not that webhook delivery has completed. +A new response returns `201` with +`{ data: { duplicate: false, status: "accepted", submissionId } }`. This means +the response was stored; webhook delivery happens afterwards in the background. + +### Idempotency + +Use a new idempotency key for each new response, and reuse the same key when +retrying that response after a network failure: + +- An identical retry returns `200` with `duplicate: true`. +- Reusing a key with different values or a different version returns `409`. +- Replaying the key of a response that an owner deleted returns `409` for 30 + days; see [deleting a response](user-guide.md#deleting-a-response). + +### Versions + +A submission must reference a version of the same form. Older published versions +remain valid while the form is available, so a page that a visitor opened before +you published a new version can still be submitted. Editing the draft never +changes a published schema. -Reuse the same idempotency key when retrying the same response after a network failure. An identical retry returns `200` with `duplicate: true`; reusing the key for different values or a different version returns `409`. Use a new key for each new response. +### Optional context -A previously published version remains valid for submissions from an already-open hosted page, provided the form is still available. A version belonging to another form is rejected. Published schemas are not modified when the draft changes. +`context` can include `visitorId`, `referrer` and `utm`, with the standard +`utm_source`, `utm_medium`, `utm_campaign`, `utm_term` and `utm_content` keys. +The server stores this context only when the form's **Intake and privacy** +settings enable source metadata, which is off by default. See +[privacy and source metadata](user-guide.md#privacy-and-source-metadata). -Optional `context` supports `visitorId`, `referrer`, and `utm` with the standard `utm_source`, `utm_medium`, `utm_campaign`, `utm_term`, and `utm_content` keys. Collection is controlled by the **Intake and privacy** settings, independently of immutable schemas. New forms default to collection off: the hosted client does not read/create a persistent visitor ID or send optional context, and the server stores no fingerprint, UTM, referrer or user agent even if a direct caller supplies them. Enabling collection preserves the previous hosted collection behavior. Existing forms keep collection enabled when migration `0006` runs; existing schemas and stored answers/metadata are unchanged. Disabling collection does not retroactively erase saved metadata. +### Errors -Abuse protection uses form-scoped HMAC markers with a rolling ten-minute expiry in the existing PostgreSQL usage store, separate from responses. These contain no IP, visitor ID or submission ID and expire independently of response deletion. The five-minute recovery job removes expired markers; expired rows never count against the budget even before cleanup. The request budget and idempotency do not depend on metadata stored in an answer. +Errors use `{ error: { code, message, requestId, fieldErrors? } }`: -API errors use `{ error: { code, message, requestId, fieldErrors? } }`. Validation errors return `422`; oversized bodies return `413`; throttled submissions return `429` with `Retry-After: 600`. +- `409`: idempotency conflict, or a temporary conflict with another operation. +- `413`: the request body is too large. +- `422`: validation failed; `fieldErrors` lists problems by field key. +- `429`: a rate limit or quota was exceeded; `Retry-After` gives the wait in + seconds. ## Limits - Submission body: 8,000,000 bytes. -- Idempotency key: 1–200 characters. +- Idempotency key: 1 to 200 characters. - Fields per form or submission: 100. -- Short and long text: 10,000 characters per value; email: 320 characters. +- Short and long text: 10,000 characters per value. Email: 320 characters. - Combined submitted text: 64,000 characters. - UTM values: normalized to at most 200 characters each. -- When a trusted address or client visitor ID is available: 20 submissions per form and subject in a rolling 10-minute window, including when optional metadata is disabled. -- CSV export: 10,000 submissions and 500 distinct submitted field keys. - -Every incoming submit attempt, including malformed requests, consumes a shared PostgreSQL budget before its JSON body is read: 60 requests per IP per fixed UTC minute. Production requires `TRUST_PROXY=1`. On Netlify (`SITE_ID` set), the application uses the platform's `x-nf-client-connection-ip` header. Self-hosted production requires the proxy-only `X-Form-Forge-Client-IP` header; the proxy must overwrite it and prevent direct access to the application. Development may use `X-Forwarded-For` with a trusted proxy; without one, requests share a local-development bucket. The visitor ID is metadata, not an identity for this request budget. - -Each workspace may accept 1,000 new submissions and reserve 5,000 new webhook deliveries per UTC day. Reservations share the submission transaction; failed transactions and idempotent replays do not consume the daily budget. Quota failures return `429` with `Retry-After` indicating the time until the window resets. Daily delivery reservations count logical deliveries, not HTTP attempts or manual retries. - -A form may have five non-archived webhook endpoints, including disabled endpoints. Archive an endpoint to free a slot. Migration `0005` preserves existing data and initializes the current day's usage. If an existing form exceeds the endpoint cap, archive the excess before applying the migration; the migration stops without deleting endpoints. - -CSV export preserves the active form, delivery status, processing status, date range and answer-word filters. It freezes selected records and display metadata before streaming, so new responses and later status changes do not alter an export in progress. Spreadsheet-formula prefixes are escaped. - -## OpenAPI and MCP - -`GET /api/openapi` serves an OpenAPI 3.1 JSON document for the public form/submit -contracts and product administration operations in this stage, including draft -publication, intake/privacy policy, filtered responses, CSV, status and deletion. -Schemas for requests/filters come from the same Zod validators as the application. -The document describes the existing NextAuth session cookies and role requirements. -Webhook signatures retain their existing envelope/API version; OAuth callbacks, -Inngest/internal test controls, webhook-endpoint administration, version-history -and membership administration are not included in this product document. - -`POST /api/admin/mcp` is a read-only, stateless -[Streamable HTTP MCP endpoint](https://modelcontextprotocol.io/specification/2025-11-25/basic/transports). -It supports protocol versions `2025-11-25`, `2025-06-18` and `2025-03-26`, with JSON -responses, `initialize`, `ping`, `tools/list`, `tools/call`, and notification -acknowledgements. It has no SSE stream or MCP session store (`GET`/`DELETE` return -`405` after authentication). Each call needs the existing authenticated NextAuth -session cookie. This is intended for session-aware first-party/local integrations; -there is no new API key, OAuth authorization server or anonymous data access. -Clients expecting automatic remote OAuth discovery need a separate future integration. - -Send `Content-Type: application/json`, `Accept: application/json, text/event-stream` -and the negotiated `MCP-Protocol-Version` after initialization. The 64 KiB body cap, -origin checks on every method and workspace permissions remain enforced. Available -tools are `list_forms` (up to 100, with a truncation flag), `get_form`, -`list_submissions` (the same bounded cursor/filters as HTTP) and `get_submission` -(bounded history). Arguments include `workspaceSlug` plus the resource ID or filters. -Tools only call existing authorized services; viewers cannot gain write access. -Response values returned to an authorized client are untrusted submitted content. -There are no tools for deletion, publication, delivery, or changing status/policy, -and the application adds no LLM or external service. +- CSV export: 10,000 responses and 500 distinct field keys. + +### Rate limits + +Every submit request, including malformed ones, counts against a budget of 60 +requests per client IP address per UTC minute. This check runs before the request +body is read. + +In addition, each form accepts at most 20 responses from the same client in a +rolling ten-minute window; exceeding it returns `429` with `Retry-After: 600`. +The client is identified by its trusted IP address or, when that is unavailable, +by the `visitorId` sent in `context`. This applies even when the form does not collect +source metadata. The limit uses short-lived, keyed markers that contain no IP +address, visitor ID or response ID and are kept separately from responses. + +How the client IP address is determined depends on the deployment: + +- **Production** requires `TRUST_PROXY=1`. +- **Netlify** (detected by `SITE_ID`): the platform's + `x-nf-client-connection-ip` header. +- **Self-hosted:** the `X-Form-Forge-Client-IP` header, which your reverse proxy + must overwrite. See [Deployment and operations](operations.md#reverse-proxy). +- **Development:** `X-Forwarded-For` when `TRUST_PROXY=1`; otherwise all requests + share one local bucket. + +### Workspace quotas + +Each workspace can accept 1,000 new responses and create 5,000 new webhook +deliveries per UTC day. Quotas are reserved in the same transaction as the +response, so failed requests and idempotent retries do not consume them. +Deliveries count once per logical delivery, not per HTTP attempt or manual retry. +When a quota is exhausted, the API returns `429` with `Retry-After` set to the +time until the next UTC day. + +A form can have up to five webhook endpoints that are not archived, including +disabled ones. + +## OpenAPI + +`GET /api/openapi` returns an OpenAPI 3.1 document. It covers the public form and +submit endpoints and the main administration operations: drafts and publishing, +intake and privacy settings, response lists and filters, CSV export, processing +status and response deletion. Request schemas are generated from the same Zod +validators the server uses. + +Administration endpoints use the NextAuth session cookie of a signed-in user and +enforce workspace roles. The document does not describe OAuth callbacks, +Inngest and internal test routes, webhook endpoint administration, version +history or member management. + +## MCP + +`POST /api/admin/mcp` is a read-only +[Model Context Protocol](https://modelcontextprotocol.io/specification/2025-11-25/basic/transports) +endpoint using the Streamable HTTP transport. It is stateless and returns JSON +responses only: there is no SSE stream or session store, and `GET` and `DELETE` +return `405`. Supported protocol versions are `2025-11-25`, `2025-06-18` and +`2025-03-26`. It implements `initialize`, `ping`, `tools/list`, `tools/call` and +acknowledges notifications. + +Each request must carry the NextAuth session cookie of a signed-in user and pass +the same origin checks as the administration interface. API keys and OAuth +authorization for MCP clients are not supported, so the endpoint suits +first-party and local integrations that can use a browser session. + +Send `Content-Type: application/json`, +`Accept: application/json, text/event-stream` and, after initialization, the +negotiated `MCP-Protocol-Version` header. Request bodies are limited to 64 KiB. + +Available tools, each taking `workspaceSlug` plus resource IDs or filters: + +- `list_forms`: up to 100 forms, with a flag when the list is truncated. +- `get_form`: one form. +- `list_submissions`: responses, with the same cursor and filters as the HTTP API. +- `get_submission`: one response with a bounded delivery history. + +Tools use the same services and permissions as the application, so a viewer +cannot gain write access. There are no tools for publishing, deleting, delivery or +changing settings. Treat returned answers as untrusted user content. diff --git a/docs/architecture.md b/docs/architecture.md index 4c15c4a..f960f11 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -2,21 +2,69 @@ [Project overview](../README.md) · [Documentation](../README.md#documentation) -The pnpm workspace keeps domain logic, rendering, UI, and infrastructure separate: +## Repository layout -- `apps/web`: Next.js App Router, admin and public routes, Auth.js, PostgreSQL, Drizzle migrations, Inngest, and Sentry. -- `packages/form-schema`: domain schemas, DTOs, visibility rules, normalization, and validation. TypeScript and Zod only, without framework or platform runtime dependencies. -- `packages/form-renderer`: headless React Hook Form renderer with replaceable UI adapters. -- `packages/ui`: shared UI primitives without form domain logic. +Form Forge is a pnpm workspace built with Turborepo: -The frontend uses React 19, Tailwind CSS, Radix UI, TanStack Query for server state, and TanStack Table for tabular views. Editor state stays in React Hook Form and local React state. +```txt +apps/web/ Next.js application: admin UI, public forms, API routes + app/ App Router pages and route handlers + src/auth/ Auth.js configuration and request-origin checks + src/db/ Drizzle schema and database client + src/services/ Domain services: forms, submissions, deliveries, outbox + src/inngest/ Background functions + src/integration/ Webhook transport, URL policy, secret encryption, DB tests + drizzle/ Committed SQL migrations + e2e/ Playwright tests + scripts/ Migration, startup and readiness scripts +packages/form-schema/ Form schema, validation and visibility rules +packages/form-renderer/ Headless React Hook Form renderer +packages/ui/ Shared UI primitives +deploy/ Docker Compose deployment, backups and acceptance tests +docs/ User and contributor documentation +``` -Drafts are validated JSONB snapshots. Publishing creates an immutable `form_versions` record, and every submission references its exact version. Submission persistence and outbox creation share one PostgreSQL transaction. Inngest processes the outbox; delivery outcomes and attempt history are database records, not just logs. +The packages keep domain logic, rendering and UI separate: -## Scope and limitations +- `packages/form-schema` defines schemas, DTOs, visibility rules, normalization and + validation with TypeScript and Zod only, without framework or platform + dependencies. The editor, public form and server share it. +- `packages/form-renderer` renders a schema with React Hook Form through + replaceable UI components. +- `packages/ui` contains UI primitives without form domain logic. -Automated checks cover the form lifecycle, PostgreSQL invariants, and a built application smoke test. They do not certify an installation's networking, live OAuth, cloud Inngest, backup recovery, or capacity. Verify those deployment-specific controls before accepting public traffic. +Workspace packages are internal and are not published to npm. -The application supports GitHub, Google and GitLab sign-in, plain iframe embedding and an automatic-height embed script. It does not include a visual page builder, custom domains, API-key authentication, email invitations, or a general-purpose embed SDK. Workspace packages are internal and are not published to npm. +## Technology -The local Docker credentials and Inngest `local` values are not deployment configuration. Workspace limits bound new logical work, but manual retries still consume resources and stored history grows over time; monitor usage and provision storage for your retention needs. +- **Application:** Next.js App Router, React 19, Tailwind CSS and Radix UI. + TanStack Query manages server state and TanStack Table renders tables. Editor + state lives in React Hook Form and local React state. +- **Authentication:** Auth.js (NextAuth) with GitHub, Google and GitLab. +- **Database:** PostgreSQL through Drizzle ORM and the `pg` driver. +- **Background jobs:** [Inngest](https://www.inngest.com/), self-hosted or Inngest + Cloud. +- **Error reporting:** optional Sentry, configured by the deployment owner. + +## Data flow + +Drafts are validated JSONB snapshots. Publishing creates an immutable +`form_versions` record, and every response references the exact version it was +submitted against. + +Accepting a response writes the response, quota reservations and an outbox event +in one PostgreSQL transaction. Inngest then processes the outbox and sends +webhooks. Delivery state and attempt history are stored as database records, and +a recovery job repairs work that was interrupted. See [Webhooks](webhooks.md#how-delivery-works). + +## Scope + +Form Forge supports GitHub, Google and GitLab sign-in, hosted forms, plain iframe +embedding and an automatic-height embed script. It does not include a visual page +builder, custom domains, API-key authentication, email invitations or a +general-purpose embed SDK. + +Automated tests cover the form lifecycle, PostgreSQL invariants and a smoke test +of the production build. They do not verify an installation's networking, live +OAuth, Inngest Cloud, backup recovery or capacity; check those for your +deployment before accepting public traffic. diff --git a/docs/dependency-maintenance.md b/docs/dependency-maintenance.md index 556fd4f..843ce0b 100644 --- a/docs/dependency-maintenance.md +++ b/docs/dependency-maintenance.md @@ -2,45 +2,64 @@ [Project overview](../README.md) · [Documentation](../README.md#documentation) -[Dependabot](../.github/dependabot.yml) checks the shared pnpm workspace every Monday -at 09:00 Europe/Moscow. Ordinary patch releases age three days and minor releases -seven days before becoming eligible. Related runtime, authentication, database, -UI and test packages form small groups; unrelated dependencies remain separate. -At most five ordinary npm update PRs stay open at once. TypeScript and -`@types/node` are excluded from those groups and get separate PRs, with shared -workspace pins updated together. Every PR runs the existing full CI workflow. - -Security alerts and security update PRs must be enabled once in the repository's -**Settings → Advanced Security / Code security → Dependabot** settings, together -with the dependency graph. The YAML alone does not enable those repository -features. Security PRs do not wait for the weekly schedule or cooldown, including -fixes requiring a major version. Review actual exposure and prioritize applicable -critical fixes immediately. Keep security updates ungrouped. If pnpm's own release -age policy blocks an urgent reviewed fix, use a narrowly versioned temporary -`minimumReleaseAgeExclude` entry; do not disable the workspace policy globally. -These semantics follow the [GitHub Dependabot reference](https://docs.github.com/en/code-security/reference/supply-chain-security/dependabot-options-reference). - -Dependabot cannot use two schedules for the same ecosystem/directory. The -[monthly review workflow](../.github/workflows/dependency-review.yml) therefore -checks stable npm major versions and pnpm, Node.js LTS and TypeScript on the first -day of each month at 10:00 Europe/Moscow. It opens one review issue with a -checklist and publishes the report in the workflow summary, without installing -dependencies or changing manifests. Repeated runs preserve the existing issue's -review notes and closed/deferred decisions. It can also run manually through -`workflow_dispatch`. The workflow needs Issues enabled and `issues: write` for -the repository's `GITHUB_TOKEN`; no additional secret is required. - -Major migrations remain separate reviewed PRs after checking release notes, -breaking changes and compatibility, and may be deferred. GitHub Actions updates -are individual monthly PRs with a seven-day cooldown. Node Docker image updates -within the current major are monthly and separate from npm changes. pnpm and -Node.js changes need their own PRs coordinating all pins: `packageManager`, the -lockfile environment document, Docker install commands/images/digests, Netlify, -engine requirements, CI and deployment fixtures/documentation. The monthly issue -reminds maintainers about these pins; Dependabot does not synchronize all of them. -No workflow automatically approves or merges dependency changes. - -The configuration takes effect after reaching the default branch. Confirm the -first hosted Dependabot run succeeds with the project's pnpm 12 lockfile; local -report checks cannot establish hosted updater compatibility. Do not downgrade -pnpm or rewrite the lockfile just to accommodate the bot. +This page describes how the project keeps dependencies current. It is mainly for +maintainers and contributors reviewing dependency pull requests. + +## Automated updates + +[Dependabot](../.github/dependabot.yml) checks the pnpm workspace every Monday at +09:00 Europe/Moscow and opens pull requests for patch and minor releases: + +- Patch releases wait three days and minor releases seven days after publication + before they are proposed. +- Related packages are grouped: React and Next.js, authentication, database, + Tailwind, Radix UI, TanStack, unit testing and component testing. Other + packages, including TypeScript and `@types/node`, get individual pull requests. +- At most five npm update pull requests are open at a time. +- GitHub Actions are updated monthly, one pull request per action. +- The Node.js Docker image is updated monthly within its current major version, + separately from npm changes. + +Every pull request runs the full CI workflow. No workflow approves or merges +dependency changes automatically. + +## Security updates + +Dependabot security alerts and security update pull requests are enabled for the +repository. Security updates do not wait for the weekly schedule or cooldown and +can include major versions. They are not grouped with other updates. + +For each alert, check whether the vulnerable code is reachable in this project, +and fix applicable critical issues first. When a transitive dependency has no +fixed upstream release, use a narrowly scoped entry in `overrides` in +[`pnpm-workspace.yaml`](../pnpm-workspace.yaml), with a comment explaining when +to remove it. If pnpm's minimum release age blocks an urgent reviewed fix, add a +version-specific `minimumReleaseAgeExclude` entry instead of disabling the policy. + +## Monthly review + +Major versions and the toolchain are reviewed by hand. On the first day of each +month at 10:00 Europe/Moscow, the +[dependency review workflow](../.github/workflows/dependency-review.yml) opens an +issue that lists stable major updates for npm packages, and new versions of pnpm, +Node.js LTS and TypeScript. It does not install anything or change manifests, and +it can also be run manually. + +For each item, read the release notes and breaking changes, then either open a +separate pull request or record why it is deferred. A listed version is not a +requirement to upgrade. + +## Toolchain changes + +pnpm and Node.js versions are pinned in several places that must change together +in one pull request: + +- **pnpm:** `packageManager` in `package.json`, the `packageManagerDependencies` + section of `pnpm-lock.yaml`, the install command in the `Dockerfile`, and the + versions mentioned in the documentation. +- **Node.js:** Docker images and digests in the `Dockerfile` and Compose files, + `NODE_VERSION` in `netlify.toml`, `engines` in `package.json`, the CI matrix, + deployment test fixtures and documentation. + +Dependabot does not update all of these pins. Do not downgrade pnpm or rewrite the +lockfile to suit the bot. diff --git a/docs/development.md b/docs/development.md index 9d3a93b..5a567b8 100644 --- a/docs/development.md +++ b/docs/development.md @@ -24,7 +24,7 @@ pnpm db:studio # Open Drizzle Studio Database commands require `DATABASE_URL` in the shell. Biome formats and lints JavaScript, TypeScript, JSON, and CSS. Prettier formats Markdown and YAML. TypeScript 7.0.2 checks types through the native compiler CLI, including during `next build`. Biome enables Promise checks, but these rules are experimental and do not guarantee identical coverage to typescript-eslint. Lefthook checks staged code with Biome and formats staged Markdown and YAML with Prettier; it does not run tests. -`db:migrate` runs the existing Drizzle ORM migrator through `apps/web/scripts/migrate.mjs`, with explicit connection cleanup, bounded waits and redacted errors. It preserves the SQL files, journal hashes, timestamps and migration history used by Drizzle Kit. `db:generate` and `db:studio` still use Drizzle Kit. Neon installations can use the [manual GitHub Actions workflow](../deploy/NETLIFY.md#manual-github-actions-run-recommended-for-neon) instead of connecting from a local computer. It is separate from CI and never runs automatically on push. +`db:migrate` runs the existing Drizzle ORM migrator through `apps/web/scripts/migrate.mjs`, with explicit connection cleanup, bounded waits and redacted errors. It preserves the SQL files, journal hashes, timestamps and migration history used by Drizzle Kit. `db:generate` and `db:studio` still use Drizzle Kit. Neon installations can use the [manual GitHub Actions workflow](../deploy/NETLIFY.md#option-a-github-actions-recommended) instead of connecting from a local computer. It is separate from CI and never runs automatically on push. ## Continuous integration diff --git a/docs/embedding.md b/docs/embedding.md index ef2e27b..10f91eb 100644 --- a/docs/embedding.md +++ b/docs/embedding.md @@ -2,8 +2,28 @@ [Project overview](../README.md) · [Documentation](../README.md#documentation) -Plain iframe snippets remain supported. For automatic height, add `data-form-forge` -and load the script once (multiple frames and dynamically inserted frames work): +You can place a published form on another site with an iframe. The form editor +provides a ready-to-copy snippet. Add `?embed=1` to the hosted form URL to use the +compact iframe layout. + +## Plain iframe + +A plain iframe needs no script and keeps the height you set: + +```html + +``` + +## Automatic height + +To resize the iframe to its content, add the `data-form-forge` attribute and load +`embed.js` from your Form Forge installation once per page: ```html