Run the tests, database checks and secret scan on every pull request (app review batch 6a)

.github/workflows/ci.yml

@@ -0,0 +1,176 @@
+# Checks every pull request and every push to main.
+#
+#   web       typechecks the web app, local API and reminder worker, runs the
+#             web tests and builds the web app
+#   database  builds disposable databases from the migrations on PostgreSQL 16
+#             and runs the database tests, the local API's HTTP tests and the
+#             reminder worker's tests on them
+#   secrets   scans the files and every commit behind them for secrets
+#
+# Nothing here deploys, uses a secret or reaches Neon, Vercel or Resend. Each
+# step is a command you can also run locally.
+name: CI
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+permissions:
+  contents: read
+
+# A new push to a pull request cancels its older run. Every push to main runs.
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
+  cancel-in-progress: true
+
+defaults:
+  run:
+    shell: bash
+
+jobs:
+  web:
+    name: Web app
+    runs-on: ubuntu-24.04
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      # Installs the pnpm version named by packageManager in package.json.
+      - uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: 22
+          cache: pnpm
+      - name: Install
+        run: pnpm install --frozen-lockfile --filter @commune/web --filter @commune/local-api --filter @commune/reminder-worker
+      - name: Typecheck
+        run: pnpm --filter @commune/web --filter @commune/local-api --filter @commune/reminder-worker run typecheck
+      - name: Web tests
+        run: pnpm --filter @commune/web test
+      - name: Build
+        run: pnpm --filter @commune/web run build
+
+  database:
+    name: Database, local API and worker
+    runs-on: ubuntu-24.04
+    timeout-minutes: 20
+    services:
+      postgres:
+        image: postgres:16.15
+        env:
+          POSTGRES_USER: commune
+          POSTGRES_PASSWORD: commune
+          # Sort text as the local servers the tests are written against do
+          # (the image would otherwise use en_US.utf8).
+          POSTGRES_INITDB_ARGS: --locale=C.UTF-8
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 5s
+          --health-timeout 5s
+          --health-retries 12
+    env:
+      PGHOST: 127.0.0.1
+      PGPORT: 5432
+      PGUSER: commune
+      PGPASSWORD: commune
+      # Every database this job makes has a name starting with this.
+      SQL_TEST_PREFIX: commune_ci
+      # The local API's tests refuse its everyday port, 3002.
+      API_PORT: 3102
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: 22
+          cache: pnpm
+      # The same install as the web job, so the two share one pnpm cache.
+      - name: Install
+        run: pnpm install --frozen-lockfile --filter @commune/web --filter @commune/local-api --filter @commune/reminder-worker
+      - name: PostgreSQL client
+        run: |
+          command -v psql >/dev/null || { sudo apt-get update -q && sudo apt-get install -y -q postgresql-client; }
+          psql --version
+      # Every test in infra/neon/tests, the lock order test included, each on
+      # a fresh database. The script's header explains the rest.
+      - name: Database tests
+        run: infra/neon/tests/run-local.sh
+      - name: Local API tests
+        run: |
+          db="${SQL_TEST_PREFIX}_api"
+          infra/neon/tests/run-local.sh --create "$db" local
+          log="$RUNNER_TEMP/local-api.log"
+          (cd apps/local-api && DATABASE_URL="postgres://127.0.0.1:$PGPORT/$db" exec node --import tsx src/server.ts) >"$log" 2>&1 &
+          api=$!
+          trap 'kill "$api" 2>/dev/null || true; wait "$api" 2>/dev/null || true; dropdb --if-exists --force "$db" || true' EXIT
+          for _ in $(seq 1 60); do
+            curl -sf "http://127.0.0.1:$API_PORT/api/health" >/dev/null && break
+            sleep 0.5
+          done
+          if ! curl -sf "http://127.0.0.1:$API_PORT/api/health" >/dev/null; then
+            echo "The local API did not start:"; cat "$log"; exit 1
+          fi
+          status=0
+          for test in apps/local-api/tests/*.mjs; do
+            name=$(basename "$test" .mjs)
+            case "$name" in *fixture*) continue ;; esac
+            echo "::group::$name"
+            if API_BASE="http://127.0.0.1:$API_PORT" timeout 300 node "$test"; then result="PASS $name"; else result="FAIL $name"; status=1; fi
+            echo "::endgroup::"
+            echo "$result"
+          done
+          if [ "$status" -ne 0 ]; then echo "::group::Local API log"; cat "$log"; echo "::endgroup::"; fi
+          exit "$status"
+      - name: Reminder worker tests
+        working-directory: apps/reminder-worker
+        run: |
+          node --test tests/email.mjs tests/unsubscribe.mjs
+          status=0
+          for test in integration invitations webhooks monthly-limits-concurrency; do
+            db="${SQL_TEST_PREFIX}_worker_${test//-/_}"
+            # The invitations test runs only on a database named for it.
+            if [ "$test" = invitations ]; then db="${SQL_TEST_PREFIX}_invite_email_test"; fi
+            ../../infra/neon/tests/run-local.sh --create "$db" stub
+            echo "::group::$test"
+            if DATABASE_URL="postgres://127.0.0.1:$PGPORT/$db" timeout 300 node "tests/$test.mjs"; then result="PASS $test"; else result="FAIL $test"; status=1; fi
+            echo "::endgroup::"
+            echo "$result"
+            dropdb --force "$db"
+          done
+          exit "$status"
+
+  secrets:
+    name: Secret scan
+    runs-on: ubuntu-24.04
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          fetch-depth: 0
+          persist-credentials: false
+      - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
+        with:
+          go-version: '1.27.x'
+          cache: false
+      # The gitleaks command line tool, built from its published source, which
+      # Go checks against its checksum database. Unlike the gitleaks GitHub
+      # Action it needs no licence for an organisation's repository.
+      - name: Install gitleaks
+        run: go install github.com/zricethezav/gitleaks/v8@v8.30.1
+      # A finding shows the file, line and rule, with the secret redacted.
+      - name: Scan the files
+        run: gitleaks dir . --redact --no-banner --verbose
+      # Every commit behind this checkout, so a secret added and removed again
+      # is caught however the change was pushed. These are gitleaks' own
+      # history options, limited to this checkout rather than every branch.
+      # .gitleaksignore lists the old commits that hold the Google API key from
+      # app review finding 1 and the earlier app's web push key, by fingerprint
+      # (commit, file, rule and line; no secret). Revoking both is still needed.
+      - name: Scan the history
+        run: gitleaks git --log-opts="--full-history --diff-filter=tuxdb HEAD" --redact --no-banner --verbose .

.gitleaksignore

@@ -0,0 +1,11 @@
+# Findings the secret scan (.github/workflows/ci.yml) leaves out, by
+# fingerprint: commit:file:rule:line. None of them is in the files any more,
+# but history keeps them, so each must be revoked where it was issued.
+#
+# The Google API key from app review finding 1, in three old commits.
+23b3eeb5a19ccc38fb74b2755ce754496111e06b:.migration-backup/docs/COMMUNE-BUILD-BIBLE.md:gcp-api-key:619
+c5fae888cafd3074cfa4891184a3d61790b7eb2d:docs/SESSION-REFERENCE.md:gcp-api-key:619
+da953a1499af8b6c0d3bd8df2f6d2ea8ec4040e3:docs/COMMUNE-BUILD-BIBLE.md:gcp-api-key:619
+# The web push (VAPID) private key of the earlier Supabase app, committed to an
+# example environment file in March 2026. The current app sends no web push.
+fb1d3c8dd0cea11791c140db10159592bbddcb66:apps/web/.env.example:generic-api-key:7

apps/local-api/README.md

@@ -120,6 +120,10 @@ request ids, and pledge limits and notices. It also refuses port 3002.
 signed-in one (`X-Commune-Viewer`, sent by each tab) is refused with 401, while
 the signed-in account and requests without the header are still served.
 
+`sign-in-lockout.mjs` takes about 40 seconds (1,000 failed sign-ins).
+`invitation-email-opt-in.mjs` needs no API:
+`node --test apps/local-api/tests/invitation-email-opt-in.mjs`.
+
 Search integration requires migration 0014. It covers all eight result types,
 active membership, selected/all-commune scopes, removed-member denial, literal
 wildcards, result caps and private-note masking. Its loopback guard refuses

apps/local-api/src/invitation-email.ts

@@ -0,0 +1,18 @@
+// The local API emails an invitation only when told to in so many words. The
+// Resend settings alone are not enough: they may be exported in a shell for
+// the worker, and an invitation made while testing would then reach a real
+// inbox. LOCAL_API_SEND_EMAIL=yes is the opt-in, and the server logs which
+// way it is set when it starts.
+type Env = Record<string, string | undefined>;
+
+export function invitationEmailSetting(env: Env): { send: boolean; notice: string } {
+  const configured = Boolean(env.RESEND_API_KEY && env.RESEND_FROM && env.APP_URL);
+  if (env.LOCAL_API_SEND_EMAIL !== "yes") {
+    return { send: false, notice: configured
+      ? "Invitation email is off: the Resend settings are ignored unless LOCAL_API_SEND_EMAIL=yes is set as well."
+      : "Invitation email is off. Invitation links can still be copied and shared." };
+  }
+  return configured
+    ? { send: true, notice: "Invitation email is on (LOCAL_API_SEND_EMAIL=yes): invitations are emailed through Resend to the address invited." }
+    : { send: false, notice: "Invitation email is off: LOCAL_API_SEND_EMAIL=yes also needs RESEND_API_KEY, RESEND_FROM and APP_URL." };
+}

apps/local-api/src/server.ts

@@ -4,6 +4,7 @@ import type { PoolClient } from "pg";
 import { changePassword, createAccount, getUser, login, logout, logoutOthers, setSessionCookie, type User } from "./auth.js";
 import { pool, withActor } from "./db.js";
 import { InvitationEmailError, invitationMailConfig, sendInvitationEmail } from "../../reminder-worker/src/invitations.js";
+import { invitationEmailSetting } from "./invitation-email.js";
 
 const host = "127.0.0.1";
 const port = Number(process.env.API_PORT ?? "3002");
@@ -180,7 +181,12 @@ async function actor(req: IncomingMessage): Promise<User> {
   return user;
 }
 
-// A bounded local throttle protects password verification from quick guessing.
+// A local throttle protects password verification from quick guessing: ten
+// failures for one email from one client lock that pair out until 15 minutes
+// after the latest. Entries stay in order of their latest failure, so expired
+// ones are always first and each failure drops them. A lockout still running
+// is never dropped, however many other failures arrive; each one costs a
+// password hash, which bounds how many can build up in 15 minutes.
 const failures = new Map<string, { count: number; until: number }>();
 function checkThrottle(req: IncomingMessage, address: string): string {
   const key = `${req.socket.remoteAddress ?? "?"}:${address}`;
@@ -191,11 +197,16 @@ function checkThrottle(req: IncomingMessage, address: string): string {
   return key;
 }
 function failed(key: string): void {
-  if (failures.size > 1000) failures.clear();
+  const now = Date.now();
+  for (const [stale, item] of failures) {
+    if (item.until > now) break;
+    failures.delete(stale);
+  }
   const existing = failures.get(key);
+  failures.delete(key);
   failures.set(key, {
-    count: existing && existing.until > Date.now() ? existing.count + 1 : 1,
-    until: Date.now() + 15 * 60 * 1000,
+    count: existing && existing.until > now ? existing.count + 1 : 1,
+    until: now + 15 * 60 * 1000,
   });
 }
 
@@ -829,11 +840,17 @@ route("GET", /^\/api\/invites\/([0-9a-f]{64})$/, async (req, res, match) => {
   } : null });
 });
 
+// Real email needs LOCAL_API_SEND_EMAIL=yes as well as the Resend settings.
+// Without both, the sender is told email isn't set up here (503).
+const invitationEmail = invitationEmailSetting(process.env);
+console.log(invitationEmail.notice);
+
 route("POST", /^\/api\/invitation-email$/, async (req, res) => {
   const user = await actor(req);
   const input = await body(req);
   try {
-    send(res, await sendInvitationEmail(pool, user.id, text(input.token, "token", 64), invitationMailConfig()));
+    send(res, await sendInvitationEmail(pool, user.id, text(input.token, "token", 64),
+      invitationEmail.send ? invitationMailConfig() : null));
   } catch (error) {
     if (error instanceof InvitationEmailError) throw new ApiError(error.status, error.message);
     throw error;

apps/local-api/tests/invitation-email-opt-in.mjs

@@ -0,0 +1,34 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+import { invitationEmailSetting } from "../src/invitation-email.ts";
+
+// Whether the local API may email an invitation. No API or database needed:
+//   node --test apps/local-api/tests/invitation-email-opt-in.mjs
+const resend = { RESEND_API_KEY: "test-only", RESEND_FROM: "Commune <test@example.test>", APP_URL: "https://example.test" };
+
+test("the Resend settings alone send nothing, and the log says how to opt in", () => {
+  const setting = invitationEmailSetting(resend);
+  assert.equal(setting.send, false);
+  assert.equal(setting.notice, "Invitation email is off: the Resend settings are ignored unless LOCAL_API_SEND_EMAIL=yes is set as well.");
+});
+
+test("sends only with the opt-in and every Resend setting", () => {
+  const setting = invitationEmailSetting({ ...resend, LOCAL_API_SEND_EMAIL: "yes" });
+  assert.equal(setting.send, true);
+  assert.match(setting.notice, /^Invitation email is on/);
+  for (const missing of Object.keys(resend)) {
+    const partial = invitationEmailSetting({ ...resend, LOCAL_API_SEND_EMAIL: "yes", [missing]: "" });
+    assert.equal(partial.send, false, `sent without ${missing}`);
+    assert.match(partial.notice, /also needs RESEND_API_KEY, RESEND_FROM and APP_URL/);
+  }
+});
+
+test("only the exact opt-in counts", () => {
+  for (const value of ["1", "true", "YES", " yes", "no", ""]) {
+    assert.equal(invitationEmailSetting({ ...resend, LOCAL_API_SEND_EMAIL: value }).send, false, `sent with ${JSON.stringify(value)}`);
+  }
+});
+
+test("off by default", () => {
+  assert.deepEqual(invitationEmailSetting({}), { send: false, notice: "Invitation email is off. Invitation links can still be copied and shared." });
+});

apps/local-api/tests/sign-in-lockout.mjs

@@ -0,0 +1,49 @@
+import assert from "node:assert/strict";
+import { randomUUID } from "node:crypto";
+
+// Ten wrong passwords lock that email out of signing in for 15 minutes, and
+// failures for other addresses never lift the lock early. Creates an account,
+// so it refuses the normal application API. About 1,000 failed sign-ins, each
+// a password hash, so it takes half a minute or so.
+const base = process.env.API_BASE;
+if (!base) throw new Error("Set API_BASE to an isolated local test API, not the user's database.");
+const url = new URL(base);
+if (url.protocol !== "http:" || !["127.0.0.1", "localhost"].includes(url.hostname) || !url.port ||
+    url.port === "3002" || url.pathname !== "/" || url.search || url.hash || url.username || url.password) {
+  throw new Error("Sign-in lockout tests require an isolated loopback API port (not 3002).");
+}
+const suffix = randomUUID().slice(0, 8);
+const password = "DisposableLockoutFixture-2026!";
+async function call(path, input) {
+  const response = await fetch(`${base}/api${path}`, {
+    method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(input),
+  });
+  return { status: response.status, ...(await response.json()) };
+}
+const signIn = (email, attempt = password) => call("/auth/login", { email, password: attempt });
+
+const avery = `lockout-avery-${suffix}@example.test`;
+assert.equal((await call("/auth/signup", { name: "Avery Lockout", email: avery, password })).status, 201);
+for (let attempt = 1; attempt <= 10; attempt += 1) {
+  assert.equal((await signIn(avery, `Wrong-guess-${attempt}`)).status, 401, `wrong password ${attempt}`);
+}
+// Locked: even the right password is refused without being checked.
+const locked = await signIn(avery);
+assert.equal(locked.status, 429, JSON.stringify(locked));
+assert.equal(locked.error, "Try signing in again later");
+
+// The lockout list used to be emptied once it held more than 1,000 entries,
+// so 1,001 failures for other addresses gave Avery's guesser ten more tries.
+const others = Array.from({ length: 1001 }, (_, index) => `lockout-other-${index}-${suffix}@example.test`);
+for (let start = 0; start < others.length; start += 8) {
+  const statuses = await Promise.all(others.slice(start, start + 8).map(async (email) => (await signIn(email)).status));
+  assert.ok(statuses.every((status) => status === 401), `other addresses: ${statuses.join(", ")}`);
+}
+assert.equal((await signIn(avery)).status, 429, "Failures for other addresses lifted the lockout");
+
+// Other people sign in as usual.
+const blake = `lockout-blake-${suffix}@example.test`;
+assert.equal((await call("/auth/signup", { name: "Blake Lockout", email: blake, password })).status, 201);
+assert.equal((await signIn(blake, "Wrong-guess-1")).status, 401);
+assert.equal((await signIn(blake)).status, 200);
+console.log("Sign-in lockout: ten failures lock an address out, and failures for 1,001 other addresses leave it locked.");

docs/HANDOFF.md

@@ -121,6 +121,10 @@ install was run.
 - Avoid `pnpm run` and `pnpm install` here for now. pnpm 11 checks the whole
   workspace first, the untracked apps are not in the lockfile, and the last
   attempt ran out of disk.
+- CI: `.github/workflows/ci.yml` (web, database, secrets) runs on pull
+  requests and pushes to main. pnpm is pinned to 10.34.6 through
+  `packageManager`; pnpm 11 hands over to it here. vite is 7.3.7; reinstall
+  `.deps-web` to pick it up.
 - A proper `pnpm install` needs a decision about the untracked workspace apps
   and several GB free. **Free disk fell to about 1.6 GB during this session**
   (from 4.4 GB); this work accounts for under 200 MB of that.

infra/local/README.md

@@ -61,12 +61,14 @@ suppressed locally, so no email is claimed or sent. Contributions and settled
 shares are confirmations recorded by the relevant admin/payer, not payment
 transfers; Stripe is not yet connected.
 
-Invitation emails now have a separate explicit send action. The local server
-returns an honest unavailable message unless server-only `RESEND_API_KEY`,
-`RESEND_FROM`, and `APP_URL` are configured. It still creates a real copyable
-invite link. Never point `APP_URL` at production while testing local invites:
-the email's HTTPS origin must serve this same database. Default local setup
-sends no email. Delivery service tests use a mocked provider and isolated DB.
+Invitation emails now have a separate explicit send action. The local API
+emails invitations only when `LOCAL_API_SEND_EMAIL=yes` is set as well as
+server-only `RESEND_API_KEY`, `RESEND_FROM` and `APP_URL`, and says at start-up
+which way it is set. Otherwise it returns an honest unavailable message and
+still creates a real invite link to copy and share. Never point `APP_URL` at
+production while testing local invites: the email's HTTPS origin must serve
+this same database. Default local setup sends no email. Delivery service tests
+use a mocked provider and isolated DB.
 
 ## 2 October local workflow verification