fix(migrations): stop swallowing failed migrations in run-migrations.sh

[sjawhar]

Problem

docker/run-migrations.sh runs psql -f "$migration" without -v ON_ERROR_STOP=1. psql's default behavior is to print a failing statement's error, keep executing the rest of the script, and still exit 0 — so a migration whose statements partially failed still gets recorded as applied in _migrations. Verified locally: a migration file with CREATE TABLE; INSERT; <bad statement>; INSERT; runs to completion, both INSERTs commit, and the script exits 0.

This specifically endangers #811's idx_request_logs_created_at index migration: CREATE INDEX CONCURRENTLY IF NOT EXISTS matches an existing index by name only. If a concurrent build is ever interrupted (dropped connection, OOM, crashed backend — not a SQL error), Postgres leaves an INVALID index under that name. A retried deploy's IF NOT EXISTS then finds the name already taken, prints a NOTICE, and reports success with exit 0 — ON_ERROR_STOP can't catch this because psql never sees an error on that run. Reproduced locally by killing the backend mid-build, then re-running the same CREATE INDEX CONCURRENTLY IF NOT EXISTS statement: it reports success while the index stays permanently invalid.

Fix

• Every psql invocation now goes through a run_psql() wrapper that adds -v ON_ERROR_STOP=1.
• The "already applied?" check previously piped psql -t <<EOF | tr -d ' ', which discards psql's exit status behind tr's (always 0). Replaced with psql -t -A so there's no pipe to mask it.
• After a migration applies without error, the runner now scans the file for CREATE INDEX CONCURRENTLY statements (comment-stripped, so a comment merely mentioning the phrase isn't misparsed) and checks pg_index.indisvalid for each named index before recording the migration as applied. An invalid index now hard-fails the run instead of silently "succeeding."
• Did not add set -o pipefail: this repo's CI runs the script directly via its #!/bin/sh shebang on ubuntu-latest, where /bin/sh is dash, which doesn't support pipefail (confirmed locally — it's a fatal "Illegal option" under set -e). The Alpine (ash) image this also ships in does support it, but the fix had to work in both, so the one masking pipe was removed instead.

Tests

Added tests/luthien_proxy/integration_tests/test_migration_runner.py — 12 tests against a real Postgres (each in its own scratch DB), covering: aborting on a failing statement (and not recording it, and not running later statements in the same file or later files), the CONCURRENTLY/indisvalid retry trap (planted deterministically via a duplicate-data unique-index build, no timing races), no false positives on a clean CONCURRENTLY build or on comments mentioning the phrase, the pre-existing drift checks (missing-file / hash-mismatch), skip-if-already-applied, and a full run of the real migrations/postgres/ set.

Wired into CI in .github/workflows/dev-checks.yaml, scoped to this one file rather than the whole integration marker: that marker is currently excluded from every workflow (pyproject.toml's default addopts filters it out, and no workflow passes -m integration), so the pre-existing integration_tests/ suite has never actually run in CI — confirmed locally, where test_migration_sync.py::test_schemas_match fails against current main right now. That's a separate, pre-existing gap outside this PR's scope; flagging it here rather than silently expanding the diff to fix unrelated tests.

Related

• fix(db): add missing index on request_logs.created_at #811 (the created_at index migration this protects)