inspect --all-instances: every Aurora writer and reader behind one endpoint (experimental)

[alexshapalov]

Implements #23 within its own constraints: SQL and DNS only, no AWS credentials, CLI, SDK, or API.

How it works

• aurora_replica_status() lists the members; exactly one writer is expected, anything else is refused rather than mislabeled.
• Each instance endpoint is derived from the entry endpoint's DNS name (<instance>.<cluster-id>.<region>.rds.amazonaws.com). Cluster, reader, custom, and instance endpoints all work as entry points; a custom domain is followed through its CNAME; RDS Proxy and non-RDS names are refused.
• aurora_db_instance_identifier() must confirm each derived endpoint reached the member it names before anything is collected.
• The --all-databases fan-out is generalized to (member, database) targets, so the two flags compose. Writer first, then readers, one member at a time. Cluster-wide findings dedupe per member.
• Output: text banners each target; --json carries server.instance and server.instance_role (contract 1.3.0, additive); SARIF/JUnit objects are prefixed instance:<id>/; Prometheus series gain instance and role labels.
• Any member that can't be reached is reported as partial coverage and the run exits 3.

Needs validation on a real cluster. I have no Aurora to run this against, so it is marked experimental and should not merge until someone with a cluster confirms it. @paul-enz, since you asked for this and validated the PgDog work so carefully, would you try it?

git fetch origin pull/38/head:aurora && git checkout aurora && go build -o /tmp/pgbot ./cmd/pgbot

# 1. discovery + derivation + identity check, no collection
PGBOT_AURORA_TEST_DSN="$AURORA_CLUSTER_URL" go test ./internal/conn/ -run TestIntegration_auroraInstances -v

# 2. the real thing
/tmp/pgbot inspect "$AURORA_CLUSTER_URL" --all-instances
/tmp/pgbot inspect "$AURORA_CLUSTER_URL" --all-instances --all-databases --json | jq '.[] | .server | {instance, instance_role, database}'

What I most want to know: whether every member was found and reached, whether the writer/reader roles are right, and, if derivation fails, the exact shape of your cluster endpoint (hostname with the ids redacted is fine). Also worth trying: the reader endpoint and an instance endpoint as the entry point, and the run through RDS Proxy, which should refuse cleanly.

[paul-enz]

@alexshapalov sorry I haven't got around to this yet. I do intend to test it out tomorrow and I'll get back to you.

[paul-enz]

Hey @alexshapalov Sorry for the wait. I gave this a run against an Aurora PostgreSQL 18.3 cluster, connecting straight to the cluster endpoint, and --all-instances refuses to start:

pgbot: discover instances: not an Aurora cluster (aurora_version() is missing) — --all-instances needs a native Aurora endpoint

It is a real Aurora cluster. The catalog probe in internal/conn/connect.go:249-253 is what misses it:

-- Aurora exposes aurora_version(). Look it up in the catalog rather
-- than CALLING it: on every other server the call fails, which writes
-- an ERROR to the server log and books a rollback in pg_stat_database
-- on each pgbot run — the very counter pgbot reports.
(SELECT count(*) FROM pg_proc WHERE proname = 'aurora_version') > 0

On 18.3 that returns 0. There are no aurora% rows in pg_proc at all, even though the functions are there and return exactly what discovery wants:

select aurora_version();            -> 18.3.4
select server_id, session_id from aurora_replica_status();
  -> one writer row (MASTER_SESSION_ID) plus one reader row

With IsAurora false, detectProvider falls through to rds, so AuroraInstances bails out before it ever runs the query.

Swap the probe to to_regprocedure. It finds them where pg_proc does not, and resolves the signature instead of calling it, so nothing lands in the server log or in pg_stat_database:

--- a/internal/conn/connect.go
+++ b/internal/conn/connect.go
@@ -246,7 +246,11 @@
 		       pg_is_in_recovery(),
-		       -- Aurora exposes aurora_version(). Look it up in the catalog rather
-		       -- than CALLING it: on every other server the call fails, which writes
-		       -- an ERROR to the server log and books a rollback in pg_stat_database
-		       -- on each pgbot run — the very counter pgbot reports.
-		       (SELECT count(*) FROM pg_proc WHERE proname = 'aurora_version') > 0`
+		       -- Aurora exposes aurora_version() and aurora_replica_status(), but
+		       -- does not catalogue them in pg_proc on 18.3. to_regprocedure finds
+		       -- them by signature without CALLING them, so no ERROR is written to
+		       -- the server log and no rollback is booked in pg_stat_database.
+		       to_regprocedure('aurora_version()') IS NOT NULL
+		       OR to_regprocedure('aurora_replica_status()') IS NOT NULL`

and the marker comment goes stale too:

--- a/internal/conn/provider.go
+++ b/internal/conn/provider.go
@@ -30,1 +30,1 @@
-	IsAurora    bool // aurora_version() exists in pg_proc
+	IsAurora    bool // aurora_version() or aurora_replica_status() is resolvable

Both predicates are true on 18.3. I built that locally and discovery gets past detection and on to host derivation.