Skip to content

feat(ci): audit the fleet for the Supabase we retired #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
catomean merged 1 commit into from
Aug 28, 2026
Merged

feat(ci): audit the fleet for the Supabase we retired #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,9 @@ jobs:
- name: The model-pin audit still detects, and still stays quiet
run: node scripts/ci/test-model-pin-audit.mjs

- name: The hosted-Supabase audit still detects, and still stays quiet
run: bash scripts/ci/test-hosted-supabase-audit.sh

# Drift guard. The fleet is on v7; templates handing out v4 is exactly
# how this repo fell behind the repos it governs.
- name: No stale action versions
Expand Down
80 changes: 80 additions & 0 deletions .github/workflows/hosted-supabase.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# The fleet self-hosts Supabase. This proves no repo still says otherwise.
#
# botsmann served PGRST205 from /api/health for months because its setup doc
# described a managed-cloud project we had already left, and told you to apply
# migrations by pasting SQL into a dashboard that does not exist for us. Nobody
# pasted anything, so its eleven migrations were never applied — while every
# deploy went green. The docs were not stale clutter; they were the outage.
#
# Fixing those files once clears today's copies. This is what stops the next one.
name: Hosted Supabase audit

on:
schedule:
# Weekly. A hosted reference enters on the timescale of someone writing a
# doc or copying an old script, not of individual commits — daily would be
# noise, and noise gets muted.
- cron: '27 6 * * 1'
workflow_dispatch:

permissions:
contents: read

concurrency:
group: hosted-supabase-audit
cancel-in-progress: true

jobs:
audit:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v7

# Pure, no checkout of anything else. If the audit's own judgement breaks,
# the sweep below would still print a confident tick.
- name: The audit still detects, and still stays quiet
run: bash scripts/ci/test-hosted-supabase-audit.sh

# GitHub code search returns nothing for these repos, so a full-text sweep
# cannot be done through the API. Shallow-cloning is the cheap honest
# alternative: --depth 1 --filter=blob:none keeps it to seconds per repo,
# and the audit reads `origin/main`, never a working tree.
- name: Clone the fleet
env:
# Same gap as the shared inventory, stated rather than hidden: the
# default token reads public repos only, so private ones are silently
# omitted. The audit prints how many repos it swept — a drop in that
# number is the tell. Set FLEET_READ_TOKEN to cover all of them.
GH_TOKEN: ${{ secrets.FLEET_READ_TOKEN || secrets.GITHUB_TOKEN }}
run: |
set -uo pipefail
mkdir -p "$RUNNER_TEMP/dev"
names=$(gh repo list bitbaum --limit 200 --no-archived --json name --jq '.[].name')
[ -n "$names" ] || { echo "::error::no repos listed — the token cannot see the fleet"; exit 1; }
n=0
for name in $names; do
if gh repo clone "bitbaum/$name" "$RUNNER_TEMP/dev/$name" -- \
--depth 1 --filter=blob:none --quiet 2>/dev/null; then
n=$((n + 1))
else
echo "::warning::could not clone $name — it will not be audited"
fi
done
echo "cloned $n repo(s)"
echo "DEV_ROOT=$RUNNER_TEMP/dev" >> "$GITHUB_ENV"

- name: No repo points at a hosted Supabase
run: |
set -uo pipefail
{
echo '## Hosted Supabase audit'
echo
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
set +e
bash scripts/ci/hosted-supabase-audit.sh --check 2>&1 | tee -a "$GITHUB_STEP_SUMMARY"
rc=${PIPESTATUS[0]}
set -e
echo '```' >> "$GITHUB_STEP_SUMMARY"
exit "$rc"
1 change: 1 addition & 0 deletions SHARED.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,7 @@ fleet-wide checker.
| `scripts/ci/verify-floor-audit.sh` | does every repo's `verify` actually run lint + typecheck + test? |
| `scripts/ci/model-pin-audit.mjs` | is any model id the fleet pins no longer served by its vendor? Zero tokens — one `GET /models` per vendor — so it runs DAILY. Uses `ai-kit`'s `checkCatalog` rather than a second vendor query. Self-tested by `scripts/ci/test-model-pin-audit.mjs` — **102 checks, no network, no key, no checkout**, every fixture the real code that fooled it. Using it to repair seven repos on 2026-08-27 exposed nine faults in both directions, and the blind spots were not random: they mirrored the shapes people write (`GROQ_MODELS = {` defeats `\bmodels?\b`; `models: AIModel[] = [` defeats an array pattern; `modelId` is not `model`). It also read ids out of COMMENTS — reporting a retired id in the very commit that removed it. Never trust its first clean run after widening; re-run the live sweep and read every line. |
| `scripts/ci/ui-defect-audit.mjs` | do any live sites ship an interactive label below its WCAG AA floor, or a stack whose rows start at different x? Renders each site; no repo checkout involved. Self-tested by `scripts/ci/test-ui-defect-audit.mjs`, which pins BOTH sides — the real defect is still caught, correct markup stays silent. |
| `scripts/ci/hosted-supabase-audit.sh` | does any repo still point at a Supabase we retired? The fleet self-hosts on bitbaum; two managed-cloud projects are dead. botsmann kept a setup doc opening with "Completed Setup" for one of them, telling you to apply migrations by pasting SQL into a dashboard that does not exist for us — so nobody pasted anything, its eleven migrations were never applied, and `/api/health` served PGRST205 for months while every deploy went green. The docs were not stale clutter; they were the outage. Allows `supabase.com/docs` (the product documentation is still correct) and `your-project.supabase.co` (a placeholder misleads nobody) — a gate that fires on those gets muted. Legitimate mentions, like a decommission runbook naming what it decommissioned, live in `hosted-supabase.baseline` **with a reason**: a ratchet that may fall or hold, never rise. GitHub code search returns nothing for these repos, so the workflow shallow-clones the fleet and runs the same script — and says SKIPPED, loudly, when it sweeps nothing, because a vacuous pass reads exactly like coverage. Self-tested by `scripts/ci/test-hosted-supabase-audit.sh` — **28 checks, no network, no checkout**, pinning both sides. |

Both report into a weekly workflow's job summary rather than only a log.

Expand Down
174 changes: 174 additions & 0 deletions scripts/ci/hosted-supabase-audit.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,174 @@
#!/usr/bin/env bash
#
# Fleet audit: does any repo still point at a HOSTED Supabase?
#
# hosted-supabase-audit.sh [--check] [--list]
#
# WHY
#
# The fleet self-hosts Supabase on bitbaum. Two managed-cloud projects were
# retired — orangecat's `ohkueislstxomdjavyhs` in 2026-06 and botsmann's
# `jkjmhtirxwhljpkcfxqe` before it — but the repos kept describing them in the
# present tense. botsmann's setup doc opened with "Completed Setup" for the dead
# project and told you to apply migrations by pasting SQL into a dashboard we do
# not have. Nobody pasted anything, its eleven migrations were never applied,
# and /api/health served PGRST205 for months while every deploy went green.
#
# So this is not tidying. A wrong pointer to a real-looking place is worse than
# no pointer: no pointer makes someone ask, a plausible one makes them assume it
# was handled. The first live sweep, 2026-08-28, found one more of exactly that
# shape — printcraft/scripts/seed-roli-project.ts hardcoded the retired host AND
# omitted `db: { schema }`, so repointing it naively at the box would have
# written one customer's rows into orangecat's `public`.
#
# CENTRAL, NOT A COPY PER REPO — the rule this repo already lives by. Three
# repos use Supabase and thirty do not; a gate copied thirty times is the
# duplication SHARED.md measures. botsmann additionally keeps a local
# `check:selfhost` in its own verify, deliberately: it is the repo the outage
# happened in, and blocking the commit beats finding it a week later.
#
# WHY LOCAL CHECKOUTS RATHER THAN `gh api`, unlike verify-floor-audit.sh —
# that one reads ONE small file per repo, which is cheap remotely. This is a
# full-text sweep, and GitHub code search returns nothing for these repos
# (verified 2026-08-28: even a known-present token finds no hit), so the only
# remote option would be fetching every text file of every repo. Instead the
# workflow shallow-clones the fleet into a temp DEV_ROOT and runs this
# unchanged — the script never needs to know which it is looking at.
#
# THE BASELINE IS A RATCHET
#
# Some references are legitimate: a decommission runbook has to name the host it
# decommissioned, a migration history has to name what it moved off. Those live
# in hosted-supabase.baseline with a reason, decided once by a human. The list
# may FALL or hold; it may never RISE without that decision being visible in the
# same PR that adds the line.
set -euo pipefail

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
DEV_ROOT="${DEV_ROOT:-$HOME/dev}"
BASELINE="${BASELINE:-$HERE/hosted-supabase.baseline}"

# ---------------------------------------------------------------- pure helpers

# hosted_pattern — what counts as pointing at a hosted Supabase.
#
# `supabase.com/docs` is deliberately absent: the product documentation is still
# correct for us, only the control plane is not. A project ref is a long opaque
# token, so `your-project.supabase.co` in an example file stays legal — a
# placeholder misleads nobody, and a gate that fires on one gets muted.
hosted_pattern() {
printf '%s' 'supabase\.com/dashboard|app\.supabase\.com|pooler\.supabase\.com|[a-z0-9]{15,}\.supabase\.co'
}

# is_exempt_path <path> — paths whose whole job is to describe the old world.
is_exempt_path() {
case "$1" in
docs/archive/*|*/docs/archive/*) return 0 ;;
*hosted-supabase-audit.sh|*hosted-supabase.baseline) return 0 ;;
*test-hosted-supabase-audit.sh|*no-hosted-supabase.sh) return 0 ;;
*) return 1 ;;
esac
}

# baseline_keys <file> — `repo/path` per line, comments and blanks dropped.
baseline_keys() {
[ -f "$1" ] || return 0
sed -e 's/#.*//' -e 's/[[:space:]]*$//' -e '/^[[:space:]]*$/d' "$1"
}

# in_baseline <key> <keys...> — exact match, so a prefix cannot smuggle a file in.
in_baseline() {
local key="$1"; shift
local k
for k in "$@"; do [ "$k" = "$key" ] && return 0; done
return 1
}

# repo_ref <dir> — audit what is SHARED, not what a session happens to have
# checked out. A stale local main reports violations already fixed upstream:
# the first run of this audit did exactly that and blamed four clean repos.
repo_ref() {
git -C "$1" rev-parse --verify -q origin/main >/dev/null 2>&1 && { printf 'origin/main'; return; }
git -C "$1" rev-parse --verify -q origin/master >/dev/null 2>&1 && { printf 'origin/master'; return; }
printf 'HEAD'
}

if [ -n "${HOSTED_SUPABASE_AUDIT_LIB_ONLY:-}" ]; then return 0; fi

# ---------------------------------------------------------------------- sweep

MODE=check
case "${1:-}" in
--check|"") MODE=check ;;
--list) MODE=list ;;
*) echo "unknown flag: $1" >&2; exit 2 ;;
esac

mapfile -t KEYS < <(baseline_keys "$BASELINE")
PATTERN="$(hosted_pattern)"

new_hits=(); seen_keys=(); scanned_repos=()

for gitdir in "$DEV_ROOT"/*/.git; do
# A linked worktree's .git is a FILE, not a directory. Skipping them stops one
# repo being audited twice under two names — the first run reported fleetcrown
# and fleetcrown-scripts as separate offenders for a single line.
[ -d "$gitdir" ] || continue
repo_dir="${gitdir%/.git}"
repo="$(basename "$repo_dir")"
ref="$(repo_ref "$repo_dir")"
scanned_repos+=("$repo")

while IFS= read -r line; do
[ -n "$line" ] || continue
path="${line%%:*}"
rest="${line#*:}"
is_exempt_path "$path" && continue
key="$repo/$path"
seen_keys+=("$key")
in_baseline "$key" "${KEYS[@]:-}" || new_hits+=("$key:$rest")
done < <(git -C "$repo_dir" grep -nEI "$PATTERN" "$ref" 2>/dev/null | sed "s|^$ref:||" || true)
done

# A runner with only this repo checked out would sweep nothing and pass. A
# vacuous pass reads exactly like coverage — the failure this audit exists to
# prevent — so say so out loud instead of printing a tick.
if [ ${#scanned_repos[@]} -eq 0 ]; then
echo "⊘ hosted-Supabase audit SKIPPED — no fleet checkout under $DEV_ROOT."
echo " This is not a pass. Run it where the repos live."
exit 0
fi

if [ "$MODE" = list ]; then
printf '%s\n' "${seen_keys[@]:-}" | sort -u
exit 0
fi

# A baseline entry whose reference is gone is a licence nobody needs. Only prune
# entries for repos actually scanned — an absent checkout is not proof.
stale=()
for k in "${KEYS[@]:-}"; do
[ -n "$k" ] || continue
[ -d "$DEV_ROOT/${k%%/*}/.git" ] || continue
in_baseline "$k" "${seen_keys[@]:-}" || stale+=("$k")
done

if [ ${#new_hits[@]} -gt 0 ]; then
echo "✗ hosted-Supabase reference(s) not in the baseline:" >&2
printf ' %s\n' "${new_hits[@]}" >&2
echo >&2
echo " The fleet self-hosts at supabase.orangecat.ch. If this is a live" >&2
echo " instruction, fix it. If it is a historical record, add the 'repo/path'" >&2
echo " line to $(basename "$BASELINE") WITH A REASON, in the same PR, so the" >&2
echo " exception is a decision and not an inheritance." >&2
exit 1
fi

if [ ${#stale[@]} -gt 0 ]; then
echo "✗ baseline entries with no matching reference — the ratchet must fall:" >&2
printf ' %s\n' "${stale[@]}" >&2
echo " Delete these lines from $(basename "$BASELINE")." >&2
exit 1
fi

echo "✓ ${#scanned_repos[@]} repos swept, no hosted-Supabase reference outside the baseline (${#seen_keys[@]} allowed)"
13 changes: 13 additions & 0 deletions scripts/ci/hosted-supabase.baseline
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# References to the RETIRED managed-cloud Supabase projects that are allowed to
# stay, because their job is to record that we left. Everything else fails
# hosted-supabase-audit.sh.
#
# Format: <repo>/<path> # why this one is legitimate
#
# This list may FALL or hold. It may never RISE without a human deciding so in
# the same PR that adds the line.

fleetcrown/docs/infrastructure/hetzner-migration.md # the migration's own record of what we moved off
orangecat/.claude/CLAUDE.md # warns the cloud project is retired — that is the point
orangecat/docs/operations/DECOMMISSION-CLOUD.md # the decommission runbook must name what it decommissioned
orangecat/next.config.js # comment marking the retired host, beside the rule that replaced it
92 changes: 92 additions & 0 deletions scripts/ci/test-hosted-supabase-audit.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
#!/usr/bin/env bash
#
# Tests for the hosted-Supabase audit — chiefly the two judgements that decide
# whether it is useful or merely loud.
#
# 1. It must catch a REAL pointer: a dashboard link, a pooler host, a project
# ref. botsmann's setup doc carried all three shapes, and the app served
# PGRST205 for months because nobody could tell they were dead.
# 2. It must NOT catch a placeholder or the product docs. A gate that fires on
# `your-project.supabase.co`, or on a link to supabase.com/docs, gets muted
# — and a muted gate protects nothing.
#
# Pure: no network, no box, no fleet checkout.

set -uo pipefail

HERE="$(cd "$(dirname "$0")" && pwd)"
SCRIPT="$HERE/hosted-supabase-audit.sh"

PASS=0; FAIL=0
ok() { printf ' ✓ %s\n' "$1"; PASS=$((PASS + 1)); }
no() { printf ' ✗ %s\n' "$1"; FAIL=$((FAIL + 1)); }
eq() { [ "$1" = "$2" ] && ok "$3" || no "$3 (want '$1', got '$2')"; }
matches() { printf '%s' "$1" | grep -qE "$(hosted_pattern)" && ok "$2" || no "$2 (should match)"; }
no_match() { printf '%s' "$1" | grep -qE "$(hosted_pattern)" && no "$2 (should NOT match)" || ok "$2"; }

export HOSTED_SUPABASE_AUDIT_LIB_ONLY=1
# shellcheck source=/dev/null
source "$SCRIPT"
unset HOSTED_SUPABASE_AUDIT_LIB_ONLY

echo "hosted_pattern — the shapes that actually misled someone"
matches 'https://supabase.com/dashboard/project/_/sql' "a dashboard link, botsmann's documented migration step"
matches 'https://supabase.com/dashboard/account/tokens' "the account-tokens link orangecat's runbook carried"
matches 'https://app.supabase.com' "the older app. host"
matches 'psql -h aws-0-eu-central-1.pooler.supabase.com -p 6543' "a pooler connection string"
matches 'https://jkjmhtirxwhljpkcfxqe.supabase.co' "botsmann's retired project ref"
matches 'https://ckpynkpsfnuqndplaapc.supabase.co' "printcraft's retired project ref"
matches 'db.ohkueislstxomdjavyhs.supabase.co:5432' "orangecat's, inside a pg_dump URL"

echo
echo "hosted_pattern — what must stay legal, or the gate gets muted"
no_match 'https://supabase.com/docs/guides/storage' "product docs are still correct for us"
no_match 'https://supabase.com/docs' "a bare docs link"
no_match 'NEXT_PUBLIC_SUPABASE_URL="https://your-project.supabase.co"' "a placeholder misleads nobody"
no_match 'https://supabase.orangecat.ch' "our own self-hosted host"
no_match 'import { createClient } from "@supabase/supabase-js"' "the package name is not a host"

echo
echo "is_exempt_path — paths whose job is to describe the old world"
is_exempt_path 'docs/archive/2026-h1/NOTES.md' && ok "docs/archive is history, not instruction" || no "docs/archive should be exempt"
is_exempt_path 'app/docs/archive/OLD.md' && ok "a nested docs/archive too" || no "nested archive should be exempt"
is_exempt_path 'scripts/ci/hosted-supabase.baseline' && ok "the baseline names the refs it allows" || no "baseline should be exempt"
is_exempt_path 'scripts/ci/no-hosted-supabase.sh' && ok "botsmann's local gate states the patterns" || no "local gate should be exempt"
is_exempt_path 'docs/operations/DECOMMISSION.md' && no "a live runbook must NOT be auto-exempt" || ok "a runbook is judged, not waved through"
is_exempt_path '.env.example' && no ".env.example must NOT be auto-exempt" || ok ".env.example is judged"

echo
echo "baseline_keys — comments and blanks are not licences"
TMP="$(mktemp)"
printf '# a header\n\nfoo/bar.md # why it is allowed\n\n \nbaz/qux.ts\n' > "$TMP"
eq 'foo/bar.md
baz/qux.ts' "$(baseline_keys "$TMP")" "reasons stripped, blanks dropped, keys kept"
eq '' "$(baseline_keys /nonexistent/baseline)" "a missing baseline allows nothing, and does not crash"

echo
echo "in_baseline — exact keys, so a prefix cannot smuggle a file through"
in_baseline 'a/b.md' 'a/b.md' 'c/d.md' && ok "an exact key is allowed" || no "exact key should match"
in_baseline 'a/b.md' 'a/b.md.bak' && no "a longer path must not match" || ok "no substring match"
in_baseline 'a/b.md' 'x/y.md' && no "an unrelated key must not match" || ok "unrelated key rejected"
in_baseline 'a/b.md' && no "an empty baseline allows nothing" || ok "empty baseline allows nothing"

echo
echo "repo_ref — audit what is SHARED, not a session's stale checkout"
D="$(mktemp -d)"
trap 'rm -rf "$D" "$TMP"' EXIT
git -C "$D" init -q 2>/dev/null
git -C "$D" commit -q --allow-empty -m init 2>/dev/null
eq HEAD "$(repo_ref "$D")" "no remote falls back to HEAD rather than failing the sweep"
git -C "$D" update-ref refs/remotes/origin/main HEAD
eq origin/main "$(repo_ref "$D")" "origin/main wins — a local main 4 commits stale reports fixed files as broken"

echo
echo "the sweep must never pass vacuously"
out="$(DEV_ROOT=/nonexistent bash "$SCRIPT" --check 2>&1)"; rc=$?
eq 0 "$rc" "no checkout exits 0, so a runner without the fleet is not a red herring"
printf '%s' "$out" | grep -q 'SKIPPED' && ok "but it says SKIPPED — a vacuous pass would read as coverage" \
|| no "must announce the skip, not print a tick"

echo
printf '%d passed, %d failed\n' "$PASS" "$FAIL"
[ "$FAIL" -eq 0 ]
Loading