Recover PostgreSQL session pools after connection failures - #147
Conversation
rajpratham1
left a comment
There was a problem hiding this comment.
This PR significantly improves the resilience of the PostgreSQL layer by introducing automatic pool recovery, transaction abstraction, health checks, graceful shutdown behavior, and comprehensive regression tests. The changes are cohesive and address a real operational issue where transient PostgreSQL failures could permanently poison the application's connection pool.
What looks good
Automatic pool recovery
Introduces retirement of failed pools instead of continuing to use broken connections.
Detects connection-level failures and transparently recreates pools.
Handles idle client errors emitted by pg.Pool.
Centralized transaction API
Replaces repeated manual transaction handling with a shared transaction() helper.
Reduces duplicated BEGIN/COMMIT/ROLLBACK logic across multiple stores.
Makes transaction behavior consistent everywhere.
Graceful shutdown
Prevents reopening pools after shutdown.
Properly drains retired pools before closing them.
Ensures close() behaves deterministically.
Health endpoint improvements
/healthz now validates session-store connectivity rather than always returning success.
Includes timeout protection.
Correctly avoids leaking database details in error responses.
Session store improvements
Adds optional health() and close() interfaces.
Runtime shutdown now attempts to close the session store without blocking indefinitely.
Excellent test coverage
Recovery after idle client failures.
Connection-level failures.
Transaction failures.
Pool retirement.
Shutdown behavior.
Health endpoint success/failure/timeout.
Pool recovery edge cases.
Code quality
The refactoring appears consistent:
repeated withPgTransaction(await pool(), ...) calls are replaced with db.transaction(...)
common functionality is centralized instead of duplicated
connection recovery logic is isolated inside pg-pool.ts rather than spread throughout callers
That makes the design easier to maintain.
Minor observation (non-blocking)
The implementation currently detects recoverable failures using PostgreSQL error codes together with regex matching on error messages. While practical, string-based message matching can be brittle if drivers or upstream libraries change wording. If additional failure modes appear in the future, it may be worth centralizing the detection policy even further or relying primarily on structured error metadata where available. This is an enhancement suggestion rather than a blocker.
Summary
Validation
npm run typechecknpm run lint -- --quietnpm run lint:oxDATABASE_URLnpm run test:pg: 148 passed against PostgreSQL 16npm test: 3,581 passed, 132 skipped; 10 sandbox migration failures reproduce unchanged onupstream/mainin the same environmentFixes #136