Skip to content

CF Workers - every request after the first hangs because cloudflare-hyperdrive-postgresql memoizes one pg.Client for the Database's lifetime #253

Description

@khangln-amv

Environment

db0 0.4.0
node 24.16.0
pnpm 11.21.0

Reproduction

The Workers runtime tears down TCP sockets between requests, so from request #2 the cached client's socket is gone while the object still looks connected. pg.Client.query() on that socket never settles, so the Worker hangs until the runtime cancels it.

Minimal repro — db0@0.4.0, pg@8.23.0, no schema required:

// wrangler.jsonc
{
  "name": "db0-hyperdrive-probe",
  "main": "src/index.ts",
  "compatibility_date": "2025-09-01",
  "compatibility_flags": ["nodejs_compat"],
  "hyperdrive": [
    {
      "binding": "HYPERDRIVE",
      "id": "db0-probe-local",
      "localConnectionString": "postgresql://user:pass@127.0.0.1:5432/mydb"
    }
  ]
}
// src/index.ts — module scope, i.e. the shape `useDatabase()` gives you
import { createDatabase } from "db0";
import cloudflareHyperdrivePostgresql from "db0/connectors/cloudflare-hyperdrive-postgresql";

const db = createDatabase(cloudflareHyperdrivePostgresql({ bindingName: "HYPERDRIVE" }));

let requestNo = 0;

export default {
  async fetch(): Promise<Response> {
    const n = ++requestNo;
    try {
      const rows = await db.prepare("select 1 as n").all();
      const client = (await db.getInstance()) as { processID?: number };
      return Response.json({ request: n, ok: true, rows, pid: client?.processID });
    } catch (error) {
      return Response.json({ request: n, ok: false, error: (error as Error).message });
    }
  },
};

wrangler dev --local, then four requests:

req 1: {"request":1,"ok":true,"rows":[{"n":1}],"pid":102}
req 2: Error: The Workers runtime canceled this request because it detected that your
       Worker's code had hung and would never generate a response.
req 3: (same)
req 4: (same)

Describe the bug

A little context: I have a Cloudflare Workers app on Hyperdrive + Neon Postgres, currently using a hand-rolled module-scoped pg.Pool with maxUses: 1 because when i started the project, db0 did not have drizzle integration for postgres. But I believe the problem is with the connector and not the integration, because my first handroll version also encounter the same problem

The issues:

1. The memoized connection outlives its socket, and the request hangs rather than erroring

The connector builds one pg.Client inside lazyInstance and routes every query through it for the lifetime of the Database (src/connectors/cloudflare-hyperdrive-postgresql.ts:46-62):

const getClient = lazyInstance(async () => {
  const pg = interopDefault(await importLib(CONNECTOR_NAME, "pg", lib, () => import("pg")));
  const hyperdrive = await getHyperdrive(opts.bindingName);
  const client = new pg.Client({ ...config, connectionString: hyperdrive.connectionString });
  await client.connect();
  return client;
});

const query: InternalQuery = async (sql, params) => {
  const client = await getClient();
  return client.query(normalizeParams(sql), params);
};

The Workers runtime tears down TCP sockets between requests, so from request #2 the cached client's socket is gone while the object still looks connected. pg.Client.query() on that socket never settles, so the Worker hangs until the runtime cancels it.

The above reproduction is across three separate wrangler dev sessions. Querying pg_stat_activity afterwards shows 0 remaining backends — the socket is gone, confirming the mechanism.

It is a hang, not a rejection, because the catch never runs, so there is nothing to retry on, nothing to log, and nothing reaches an error tracker. The Worker just burns wall-clock until cancellation.

For contrast, the same worker and the same binding, using the shape Cloudflare's and Neon's Workers docs recommend — a module-scoped Pool that never reuses a connection:

const pool = new Pool({ connectionString: env.HYPERDRIVE.connectionString, maxUses: 1 });
req 1: {"request":1,"ok":true,"rows":[{"n":1,"pid":111}]}
req 2: {"request":2,"ok":true,"rows":[{"n":1,"pid":112}]}
req 3: {"request":3,"ok":true,"rows":[{"n":1,"pid":113}]}
req 4: {"request":4,"ok":true,"rows":[{"n":1,"pid":114}]}
req 5: {"request":5,"ok":true,"rows":[{"n":1,"pid":115}]}

The cached Pool object survives; each request dials a fresh backend (note the incrementing pids). Hyperdrive pools server-side, so that connect is cheap, which is probably why this is the recommended pattern from both neon and CF.

The same lazyInstance-one-connection shape is in five connectors:

  • src/connectors/cloudflare-hyperdrive-postgresql.ts:46 — new pg.Client + connect()
  • src/connectors/postgresql.ts:41 — new pg.Client + connect()
  • src/connectors/neon.ts:41 — new pg.Client
  • src/connectors/cloudflare-hyperdrive-mysql.ts:56 — mysql.createConnection
  • src/connectors/mysql2.ts:35 — mysql.createConnection

The two Hyperdrive connectors and neon is probably the most dangerous, since those exist specifically for the runtimes that recycle sockets. (planetscale.ts:37 memoizes too, but its client is HTTP-based, so there is no socket to lose.)

2. Nothing can clear a connection that broke after it was established

src/connectors/_internal/utils.ts:57-68:

export function lazyInstance<T>(factory: () => Promise<T>): LazyInstance<T> {
  const get = (() =>
    (get.current ??= factory().catch((error) => {
      get.current = undefined;   // ← only when the FACTORY rejects
      throw error;
    }))) as LazyInstance<T>;
  get.current = undefined;
  get.reset = () => { get.current = undefined; };
  return get;
}

Once client.connect() has resolved, the memo holds a fulfilled promise. A query failing — or hanging — on that client never re-enters the factory, so the dead connection is handed back forever. get.reset() exists but the connector only calls it from dispose().

And dispose() is not a reset — it is a one-way latch, so it cannot be used to recover. src/database.ts:29-36:

let _disposed = false;
const checkDisposed = () => {
  if (_disposed) {
    const err = new Error(DISPOSED_ERR);
    Error.captureStackTrace?.(err, checkDisposed);
    throw err;
  }
};

Every getInstance / exec / prepare / sql calls checkDisposed() first, so after dispose() the whole Database is dead. Measured on Node with connectors/postgresql (same code path, easier to script), terminating the backend with pg_terminate_backend() to stand in for the Workers teardown:

1. first query:             ok
2. client constructor:      Client
3. same object twice:       true
4. four queries after kill: ["fail: Client has encountered a connection error and is not queryable", ...×4]
5. still the same object:   true
6. after dispose():         fail: This database instance has been disposed and cannot be used.
7. thrown synchronously:    true
8. fresh createDatabase(): ok

So the only recovery from a broken connection is constructing a whole new createDatabase(...) — per request, on Workers — which makes the memoization pure overhead rather than the connection reuse it is meant to be. A reset() on Database, or a dispose() that clears the connector without latching, would give callers something else to call.

Worth noting line 7: checkDisposed() throws synchronously out of prepare(), while every other failure in the API is a rejection. Code awaiting queries the ordinary way gets a synchronous throw from what looks like an async call, and a promise-tail .catch() misses it.

This one is not Workers-specific — a Postgres restart, a failover, an idle-timeout reaper or an admin pg_terminate_backend poisons a long-running Node process the same way, permanently.

3. The client has no 'error' listener, so a server-side disconnect crashes the process

The connector creates the pg.Client and never attaches an 'error' handler. pg emits one on an unexpected disconnect, and an unhandled 'error' event on an EventEmitter terminates the Node process:

node:events:487
      throw er; // Unhandled 'error' event
      ^
error: terminating connection due to administrator command
    at parseErrorMessage (node_modules/pg-protocol/dist/parser.js:306:11)
    ...
    code: '57P01',

This is what happens on a bare connectors/postgresql when Postgres restarts underneath it — no catch anywhere in user code can prevent it, because nothing is throwing into user code. The measurements in section 2 only ran after adding client.on("error", () => {}) by hand first.

Suggested direction

  1. Use pg.Pool / mysql.createPool in these connectors and let callers pass pool options. maxUses: 1 is what makes it correct on Workers, and a pool is the right default on long-running Node too. It also attaches the error handling a raw Client leaves to the caller. This addresses all three.
  2. If keeping a single connection is deliberate, at minimum attach an 'error' listener that calls getClient.reset() and ends the dead client. That fixes 2 and 3, but not the hang in 1 — a torn-down socket produces no error to react to.
  3. Decouple dispose() from the latch, or add reset(), so recovery does not require rebuilding the Database.

Additional context

Unrelated and lower priority, but noticed nearby: because a single pg.Client serializes queries, independent queries issued through Promise.all run sequentially.

Four concurrent pg_sleep(0.3) queries, same database:

db0 (single Client): 1247ms
pg.Pool (max 4):      349ms

pg also emits a deprecation warning for this pattern — "Calling client.query() when the client is already executing a query is deprecated and will be removed in pg@9.0" — so the connectors will need a pool (or explicit queueing) for pg@9.

Logs

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions