feat(findings): collation_version_mismatch — the collation library changed under the data

CHANGELOG.md

@@ -3,7 +3,7 @@
 All notable changes to pgbot are documented here. The format follows
 [Keep a Changelog](https://keepachangelog.com/), and the project aims for
 [Semantic Versioning](https://semver.org/). The `--json` contract is versioned
-separately by `model.SchemaVersion` (currently 1.2.0).
+separately by `model.SchemaVersion` (currently 1.3.0).
 
 ## [Unreleased]
 
@@ -21,6 +21,16 @@ separately by `model.SchemaVersion` (currently 1.2.0).
   `pgbot ask "why is it slow?"`.
 
 ### Added
+- **`collation_version_mismatch` finding** (PG15+). The collation library
+  (libc or ICU) that defines text sort order changed version under the data —
+  an OS upgrade, a new base image, a restore onto a different host — so every
+  btree over text sorted by it may be silently out of order: lookups miss rows
+  and `UNIQUE` stops catching duplicates. Read from `pg_database.datcollversion`
+  and `pg_collation.collversion` against the library's actual version; critical
+  when it is the database default, warn for a named collation. The remediation
+  is REINDEX **then** `REFRESH COLLATION VERSION`, in that order — the caveat
+  says why. New `collation` section in `--json`; `SchemaVersion` → **1.3.0**
+  (additive; a 1.2.0 consumer parses it unchanged).
 - **`$PGSERVICE` as a connection fallback** (#25). When no connection string
   is passed and neither `$DATABASE_URL` nor `$PGBOT_DATABASE_URL` is set,
   pgbot now checks `$PGSERVICE` too, so a

README.md

@@ -29,7 +29,7 @@
   <a href="docs/providers.md">Provider notes</a>
 </p>
 
-> **Status: beta.** The `--json` contract is versioned (currently `1.2.0`, JSON
+> **Status: beta.** The `--json` contract is versioned (currently `1.3.0`, JSON
 > Schema published in [`schema/`](schema/)) and breaking changes to it are
 > treated as breaking changes to the tool. The human-readable report is **not**
 > a stable interface — parse `--json`, not the terminal output.
@@ -816,20 +816,21 @@ All from SQL — connections, cache-hit ratio, TPS and rollback ratio, WAL and I
 rates, checkpoints, locks and blocking chains, replication lag, replication-slot
 WAL retention and logical-subscription health, top queries
 (`pg_stat_statements`), table/index sizes, dead tuples and vacuum activity,
-unused and missing indexes, and non-default settings. Counters
+unused and missing indexes, non-default settings, and collation version drift
+(PG15+). Counters
 (`pg_stat_database`, `pg_stat_wal`, IO) are **double-sampled** to produce live
 rates; the rest are point-in-time reads trended against the baseline.
 
 ## The `--json` contract
 
 `--json` (and `--format=json`) is the interface to build on — a versioned,
-PII-free document (`schema_version`, currently `1.2.0`) whose machine-checkable
+PII-free document (`schema_version`, currently `1.3.0`) whose machine-checkable
 JSON Schema is published in [`schema/`](schema/). Every section carries an
 `exactness` label — `sampled`, `cumulative`, `scraped`, or `unavailable` — so a
 consumer never mistakes a cumulative total for a live rate.
 
 Versioning policy: additive fields bump the minor version and are not breaking —
-a `1.1.0` consumer parses `1.2.0` output unchanged; breaking changes to the
+a `1.2.0` consumer parses `1.3.0` output unchanged; breaking changes to the
 contract are treated as breaking changes to the tool. `pgbot advise --json` has
 its own schema
 ([`schema/pgbot-advise-1.0.0.json`](schema/pgbot-advise-1.0.0.json)).

docs/findings/README.md

@@ -23,6 +23,7 @@ Lost durability, corruption, wraparound, replication — things that end in an o
 - **[index_invalid](index_invalid.md)** · Critical — a failed CREATE INDEX CONCURRENTLY left an invalid index — critical if it's still maintained on writes, warn if it's failed-build debris
 - **[sync_rep_degraded](sync_rep_degraded.md)** · Critical — fewer synchronous standbys connected than the config requires
 - **[archiving_disabled](archiving_disabled.md)** · Warn — archive_mode is off — no continuous WAL archive for PITR
+- **[collation_version_mismatch](collation_version_mismatch.md)** · Warn — the collation library changed version under the data — text indexes may be silently out of order
 - **[connection_saturation](connection_saturation.md)** · Warn — connections approaching max_connections
 - **[idle_in_transaction](idle_in_transaction.md)** · Warn — sessions idle inside an open transaction, holding locks and the xmin horizon
 - **[int4_identity_column](int4_identity_column.md)** · Warn — a sequence-backed int2/int4 column that will wrap (int4 at 2.1B) regardless of current value

docs/findings/collation_version_mismatch.md

@@ -0,0 +1,125 @@
+---
+id: collation_version_mismatch
+severity: warn
+critical_when: "the database's default collation is the one that changed"
+dimension: risk
+object: db
+scope: infra
+requires: [PG15+]
+thresholds: []
+related: []
+---
+
+# collation_version_mismatch
+
+**Severity:** warn (critical when the database's default collation is the one that changed) · **Dimension:** risk · **Object identity:** `db:<database>` (see [configuration](../configuration.md)) · **Requires:** PostgreSQL 15+
+
+## What pgbot observed
+
+The collation version the catalog recorded no longer matches what the server's
+collation library reports now — for the database default
+(`pg_database.datcollversion` vs `pg_database_collation_actual_version()`) or for
+a named collation (`pg_collation.collversion` vs `pg_collation_actual_version()`).
+Only this database is checked; collations whose provider records no version
+(`C`, `POSIX`) cannot drift and are never reported.
+
+Critical when the **database default** drifted, because every text index that
+does not name a collation uses it. Warn when only named collations drifted.
+
+## Why it matters
+
+Text sort order is not defined by Postgres — it is defined by the OS's libc or
+by ICU, and a btree index is only valid for the order that library produced when
+the index was built. When the library changes underneath (an OS upgrade, a new
+container base image, a restore onto a different host; glibc 2.28 changed the
+order for most locales), the index is **silently out of order**: equality
+lookups miss rows that exist, range scans skip them, and `UNIQUE` constraints
+stop catching duplicates. Nothing errors. Postgres emits a `WARNING` at connect
+time that nobody reads, and repairs nothing.
+
+## How to verify it yourself
+
+```sql
+SELECT datname,
+       datcollversion                              AS recorded,
+       pg_database_collation_actual_version(oid)   AS actual
+FROM pg_database
+WHERE datname = current_database();
+```
+
+```sql
+SELECT n.nspname || '.' || c.collname AS collation,
+       c.collversion                     AS recorded,
+       pg_collation_actual_version(c.oid) AS actual
+FROM pg_collation c
+JOIN pg_namespace n ON n.oid = c.collnamespace
+WHERE c.collversion IS NOT NULL
+  AND c.collversion IS DISTINCT FROM pg_collation_actual_version(c.oid);
+```
+
+A fresh `psql` session to the database prints the same thing Postgres sees:
+`WARNING: database "app" has a collation version mismatch`.
+
+## How to fix it
+
+Rebuild first, then tell Postgres the new version is the right one. In that order.
+
+1. **Reindex everything that sorts text with the affected collation.** For the
+   database default that is every btree over a `text`/`varchar`/`char` column
+   without an explicit `COLLATE`; the simple, safe answer is the whole database,
+   online:
+
+   ```sql
+   REINDEX DATABASE CONCURRENTLY app;
+   ```
+
+   For a named collation, reindex the indexes whose columns use it. If you would
+   rather check than rebuild, `amcheck` can verify btree order first
+   (`CREATE EXTENSION amcheck; SELECT bt_index_check('index_name', true);`) — an
+   index that passes is fine, one that fails must be rebuilt.
+
+2. **Record the new version** so the warning stops and pgbot clears the finding:
+
+   ```sql
+   ALTER DATABASE app REFRESH COLLATION VERSION;
+   -- or, for a named collation:
+   ALTER COLLATION public.de_phonebook REFRESH VERSION;
+   ```
+
+Refreshing *before* reindexing only updates the catalog: the warning goes away
+and the indexes stay corrupt. On a managed provider, check whether the provider
+handled this as part of a major-version upgrade before doing it yourself; it is
+still your indexes.
+
+## When to ignore it
+
+Only once the reindex is done and the refresh is scheduled, or when you have
+verified with `amcheck` that every affected index is in order. A suppressed
+critical still renders in the report; it only drops out of the exit code.
+
+```toml
+[[ignore]]
+finding = "collation_version_mismatch"
+object  = "db:app"
+reason  = "reindexed after the glibc upgrade on 2026-09-10; REFRESH COLLATION VERSION in the next window"
+expires = "2026-10-01"
+```
+
+## What pgbot cannot see
+
+- Whether the sort order **actually changed** between the two versions for your
+  locale — Postgres records versions, not orderings. A mismatch is a "may be
+  corrupt", not a "is corrupt"; the only proof either way is `amcheck` or a
+  rebuild.
+- Which indexes use the affected collation. It reports the collation; mapping it
+  to indexes is the reindex step.
+- Other databases in the cluster. Each records its own `datcollversion`; run
+  `--all-databases` to check them all.
+- PostgreSQL 14 and older, which do not record a database-level version.
+
+## Related
+
+- [checksum_failures](checksum_failures.md) — the other silent-corruption signal,
+  from the storage side rather than the collation library.
+- [index_invalid](index_invalid.md) — an index Postgres already knows is unusable;
+  a collation mismatch is one it still trusts.

internal/collect/collation.go

@@ -0,0 +1,51 @@
+package collect
+
+import (
+	"context"
+	_ "embed"
+	"time"
+
+	"github.com/pgrundev/pgbot/internal/conn"
+	"github.com/pgrundev/pgbot/internal/model"
+)
+
+//go:embed sql/collation.sql
+var sqlCollation string
+
+// collation = catalog objects whose recorded collation version no longer matches
+// the library the server runs against. PG15+, when pg_database began recording
+// datcollversion. Empty list = healthy.
+type collationCollector struct{}
+
+type collationRow struct {
+	Kind     string `db:"kind"`
+	Name     string `db:"name"`
+	Provider string `db:"provider"`
+	Recorded string `db:"recorded"`
+	Actual   string `db:"actual"`
+}
+
+func (collationCollector) Name() string { return "collation" }
+func (collationCollector) Kind() Kind   { return KindGauge }
+func (collationCollector) Available(caps conn.Capabilities) bool {
+	return caps.VersionNum >= 150000 // pg_database.datcollversion + pg_database_collation_actual_version()
+}
+
+func (collationCollector) Sample(ctx context.Context, t *conn.Target, _ conn.Capabilities) (any, error) {
+	return queryMany[collationRow](ctx, t, sqlCollation)
+}
+
+func (collationCollector) Assemble(c *model.Context, _ conn.Capabilities, s sampled, _ time.Duration, _ Options) {
+	rows, ok := s.A.([]collationRow)
+	if s.Err != nil || !ok {
+		c.Collation = &model.Collation{Section: unavail(s.Err, "collation versions need PostgreSQL 15+")}
+		return
+	}
+	col := &model.Collation{Section: model.Section{Exactness: model.ExactnessScraped}}
+	for _, r := range rows {
+		col.Mismatches = append(col.Mismatches, model.CollationMismatch{
+			Kind: r.Kind, Name: r.Name, Provider: r.Provider, Recorded: r.Recorded, Actual: r.Actual,
+		})
+	}
+	c.Collation = col
+}

internal/collect/collation_integration_test.go

@@ -0,0 +1,106 @@
+package collect_test
+
+import (
+	"context"
+	"os"
+	"testing"
+	"time"
+
+	"github.com/jackc/pgx/v5"
+	"github.com/pgrundev/pgbot/internal/collect"
+	"github.com/pgrundev/pgbot/internal/conn"
+	"github.com/pgrundev/pgbot/internal/findings"
+	"github.com/pgrundev/pgbot/internal/model"
+)
+
+// A collation version mismatch can't be produced by upgrading glibc inside a
+// test, but the catalog state it leaves behind can: pg_database.datcollversion
+// is what Postgres compares against the library at connect time, and a superuser
+// can set it directly. Forge a stale version, run the real collector, check the
+// collected row and the finding, then confirm ALTER DATABASE … REFRESH COLLATION
+// VERSION — the step the remediation ends with — clears it.
+func TestIntegration_collationVersionMismatch(t *testing.T) {
+	su := os.Getenv("PGBOT_TEST_SUPERUSER_DSN")
+	if su == "" {
+		t.Skip("set PGBOT_TEST_SUPERUSER_DSN (a superuser DSN) to run the collation fixture")
+	}
+	ro := dsn(t)
+	ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
+	defer cancel()
+	admin, err := pgx.Connect(ctx, su)
+	if err != nil {
+		t.Fatalf("admin connect: %v", err)
+	}
+	t.Cleanup(func() { admin.Close(context.Background()) })
+
+	var vnum int
+	if err := admin.QueryRow(ctx, `SELECT current_setting('server_version_num')::int`).Scan(&vnum); err != nil {
+		t.Fatal(err)
+	}
+	if vnum < 150000 {
+		t.Skip("pg_database.datcollversion is PG15+")
+	}
+	var db string
+	var recorded *string
+	if err := admin.QueryRow(ctx, `SELECT datname, datcollversion FROM pg_database WHERE datname = current_database()`).
+		Scan(&db, &recorded); err != nil {
+		t.Fatal(err)
+	}
+	if recorded == nil {
+		t.Skip("this database's collation records no version (C/POSIX) — nothing can drift")
+	}
+	refresh := `ALTER DATABASE ` + pgx.Identifier{db}.Sanitize() + ` REFRESH COLLATION VERSION`
+	if _, err := admin.Exec(ctx, `UPDATE pg_database SET datcollversion = '0.0-pgbot-test' WHERE datname = current_database()`); err != nil {
+		t.Fatalf("forge a stale datcollversion: %v", err)
+	}
+	t.Cleanup(func() { _, _ = admin.Exec(context.Background(), refresh) })
+
+	target, err := conn.Connect(ctx, ro)
+	if err != nil {
+		t.Fatalf("connect: %v", err)
+	}
+	defer target.Close()
+	run := func() *model.Context {
+		c, err := collect.Run(ctx, target, collect.Options{Interval: 200 * time.Millisecond, ASHHz: 0})
+		if err != nil {
+			t.Fatalf("run: %v", err)
+		}
+		if c.Collation == nil || c.Collation.Exactness != model.ExactnessScraped {
+			t.Fatalf("collation section must be collected on PG15+, got %+v", c.Collation)
+		}
+		return c
+	}
+
+	c := run()
+	var row *model.CollationMismatch
+	for i := range c.Collation.Mismatches {
+		if c.Collation.Mismatches[i].Kind == "database" {
+			row = &c.Collation.Mismatches[i]
+		}
+	}
+	if row == nil {
+		t.Fatalf("the forged database mismatch must be collected, got %+v", c.Collation.Mismatches)
+	}
+	if row.Name != db || row.Recorded != "0.0-pgbot-test" || row.Actual == "" || row.Actual == row.Recorded {
+		t.Fatalf("collected row must mirror the catalog: %+v", *row)
+	}
+	var f *model.Finding
+	for _, x := range findings.Compute(c) {
+		if x.ID == "collation_version_mismatch" {
+			f = &x
+			break
+		}
+	}
+	if f == nil || f.Severity != model.SeverityCritical || f.Object != "db:"+db {
+		t.Fatalf("a drifted database default must fire critical on db:%s, got %+v", db, f)
+	}
+
+	if _, err := admin.Exec(ctx, refresh); err != nil {
+		t.Fatalf("refresh: %v", err)
+	}
+	for _, m := range run().Collation.Mismatches {
+		if m.Kind == "database" {
+			t.Fatalf("the database mismatch must clear after REFRESH COLLATION VERSION, got %+v", m)
+		}
+	}
+}

internal/collect/collector.go

@@ -104,6 +104,7 @@ var registry = []Collector{
 	progressCollector{},
 	archiverCollector{},
 	checksumsCollector{},
+	collationCollector{},
 	standbyCollector{},
 }
 

internal/collect/sql/collation.sql

@@ -0,0 +1,30 @@
+-- Collation version drift (PG15+): the version of the collation library the
+-- catalog recorded when this database (datcollversion) or a collation object
+-- (collversion) was created, against what the running server's libc/ICU reports
+-- now. A difference means the library that defines text sort order changed under
+-- the data — every btree over text sorted by it may be silently out of order
+-- until REINDEXed. Scoped to current_database(): pg_collation is per-database
+-- and the database row is this one's. NULL versions (C/POSIX, or a provider that
+-- reports none) cannot drift and are excluded.
+SELECT 'database' AS kind,
+       d.datname   AS name,
+       CASE d.datlocprovider WHEN 'c' THEN 'libc' WHEN 'i' THEN 'icu' WHEN 'b' THEN 'builtin'
+            ELSE d.datlocprovider::text END                       AS provider,
+       d.datcollversion                                           AS recorded,
+       coalesce(pg_database_collation_actual_version(d.oid), '') AS actual
+FROM pg_database d
+WHERE d.datname = current_database()
+  AND d.datcollversion IS NOT NULL
+  AND d.datcollversion IS DISTINCT FROM pg_database_collation_actual_version(d.oid)
+UNION ALL
+SELECT 'collation',
+       n.nspname || '.' || c.collname,
+       CASE c.collprovider WHEN 'c' THEN 'libc' WHEN 'i' THEN 'icu' WHEN 'b' THEN 'builtin'
+            ELSE c.collprovider::text END,
+       c.collversion,
+       coalesce(pg_collation_actual_version(c.oid), '')
+FROM pg_collation c
+JOIN pg_namespace n ON n.oid = c.collnamespace
+WHERE c.collversion IS NOT NULL
+  AND c.collversion IS DISTINCT FROM pg_collation_actual_version(c.oid)
+ORDER BY 1, 2;