Skip to content

fix(api): recover the session store when MongoDB starts after the API - #3369

Open
tommyzli wants to merge 10 commits into
mainfrom
claude/session-store-mongo-retry
Open

tommyzli wants to merge 10 commits into
mainfrom
claude/session-store-mongo-retry

Conversation

@tommyzli

@tommyzli tommyzli commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

If MongoDB is unreachable when the API starts, every request with a session cookie returns 500 for good, even after MongoDB is back. /ready goes 200, but logged-in pages keep failing until the API restarts.

The cause is the express-session store. new MongoStore({ mongoUrl }) opens its own MongoClient once, at construction, and every session read/write awaits that one connect promise. When it rejects, the store replays the original MongoServerSelectionError: connect ECONNREFUSED on every request. The mongoose connection recovers because connectDBWithRetry retries it (#2966), but the session store never used that connection.

This PR adds MongoSessionStore (packages/api/src/utils/sessionStore.ts), a thin session.Store around the same new MongoStore({ mongoUrl }):

  • It creates the MongoStore on first use once mongoose is connected (the same isMongoConnected() check /ready uses), so the store's connect runs when MongoDB is reachable.
  • If that store's connect or setup (its TTL index) fails, it closes the store and creates a new one after a backoff (1 s, doubling up to 30 s, like connectDBWithRetry), instead of replaying the error forever.
  • While mongoose isn't connected, session operations fail fast with "MongoDB is not connected" instead of hanging.

connect-mongo keeps its own MongoClient on the API's mongodb 6.x driver, as before. Handing it mongoose's client instead would need a cast between mongoose's bundled mongodb 4.x types and the 6.x types connect-mongo is typed against, and it breaks when mongoose reconnects with a new client (a closed client can't be reused).

Server.shutdown now also closes the session store's client, which main never did. close() forgets the closed store, so a server started again in the same process (as the integration tests do) builds a new one.

Tests:

  • New unit tests in utils/__tests__/sessionStore.test.ts.
  • make dev-int FILE=dashboards passes locally (10 suites, 544 tests). It includes a file that starts and stops the server twice and logs in each time.
  • I ran a scripted repro against a throwaway Mongo container: log in, stop Mongo, start the API, wait for the 30 s connect timeout, start Mongo, wait for /ready 200, then GET /me three times with the existing cookie. Result: main returned 500 500 500, and this branch returned 200 200 200.

How to reproduce

Starting point: local full stack with yarn dev (the bug needs the API and MongoDB).

On main:

  1. Run yarn dev, register, and stay logged in.
  2. Stop the dev MongoDB container (docker stop <dev project>-db-1; the dev portal at http://localhost:9900 lists the project name).
  3. Restart the API while MongoDB is down, e.g. touch packages/api/src/index.ts so nodemon restarts it.
  4. Wait until the API logs Initial MongoDB connection failed, retrying, which takes about 30 s.
  5. Start the MongoDB container again and wait for GET /ready on the API to return 200.
  6. Reload the app. Logged-in pages fail, /me returns 500, and the API log shows MongoServerSelectionError: connect ECONNREFUSED from MongoClient.connect.

On this branch:

  1. Repeat steps 1–5.
  2. Reload the app. Logged-in pages load without restarting the API.

How to test on Vercel preview

N/A, non-UI change.

References

🤖 Generated with Claude Code

connect-mongo's `mongoUrl` option opens its own MongoClient once at
construction and caches the connect promise. If MongoDB is down when the
API boots, every session read/write replays that MongoServerSelectionError
and every request with a session cookie returns 500 until a restart, even
after connectDBWithRetry has connected mongoose (#2966).

Back the session store with the mongoose connection instead: build a
MongoStore on mongoose's client once it is connected, rebuild it when
mongoose replaces the client, and fail session operations fast until then.
@vercel

vercel Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Oct 14, 2026 12:48am UTC
hyperdx-storybook Ready Ready Preview Oct 14, 2026 12:48am UTC

Request Review

@changeset-bot

changeset-bot Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 47ff983

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the review/tier-2 Low risk — AI review + quick human skim label Oct 13, 2026
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

🔵 Tier 2 — Low Risk

Small, isolated change with no API route or data model modifications.

Why this tier:

  • Standard feature/fix — introduces new logic or modifies core functionality

Review process: AI review + quick human skim (target: 5–15 min). Reviewer validates AI assessment and checks for domain-specific concerns.
SLA: Resolve within 4 business hours.

Stats
  • Production files changed: 3
  • Production lines changed: 159 (+ 222 in test files, excluded from tier calculation)
  • Branch: claude/session-store-mongo-retry
  • Author: tommyzli

To override this classification, remove the review/tier-2 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 125 in total (1.74%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

Comment thread packages/api/src/utils/sessionStore.ts Resolved
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

1 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 1 minor

None could be anchored to changed lines; they are listed below.

Findings outside the changed lines

1 minor
  • 🔵 packages/api/src/utils/sessionStore.ts:336 — The backoff schedule is copied from connectDBWithRetry and has to be kept in sync by hand → connectDBWithRetry in packages/api/src/models/index.ts:95-96,127 already hardcodes the 1000/30000 defaults and the Math.min(base * 2 ** (attempt - 1), max) formula. The comment here says the two schedules are meant to match, but nothing enforces it, so changing one will silently break that. Fix: move the constants and a backoffDelayMs(attempt) helper into one module, such as models/index.ts or a new utils/backoff.ts, and use it from both places.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@greptile-apps

greptile-apps Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[High impact] The PR appears safe to merge, with no new actionable issues found.

Summary

The PR lets session storage recover when MongoDB starts after the API.

  • Logged-in sessions recover after MongoDB comes back online.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Session operation] --> B{Store exists?}
  B -->|Yes| C[Use current store]
  B -->|No| D{MongoDB connected and retry delay passed?}
  D -->|No| E[Return error]
  D -->|Yes| F[Create MongoStore]
  F --> G{Setup succeeds?}
  G -->|Yes| C
  G -->|No| H{Still the current store?}
  H -->|No| I[Leave newer store untouched]
  H -->|Yes| J[Record failure and close failed store]
  J --> K[Wait before another attempt]
Loading

Reviews (6) · Last reviewed commit: "fix(api): ignore rejections from a sessi..." · Reviewed by Greptile

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. The change is well-scoped and defensively handles the close/rebuild race (with dedicated tests). Shutdown swallows rejections via Promise.allSettled, the module-level sessionStore singleton does no work at construction, and the lazy-connect gate on isMongoConnected() is sound. One recommended reliability item and a couple of nits below.

🟡 P2 -- recommended

  • packages/api/src/utils/sessionStore.ts:51 -- this.failures is incremented on every setup failure but never reset after a successful rebuild (only close() resets it), so the exponential backoff ratchets across the process lifetime: a single brief Mongo blip early on, followed by a healthy period and a later unrelated failure, computes the next retry from the accumulated count and can jump straight to the 30 s cap instead of restarting at 1 s, delaying recovery for the second outage.
    • Fix: Reset this.failures = 0 (and this.retryAt = 0) when a rebuilt store's collectionP resolves successfully, e.g. by attaching a .then beside the existing .catch.
🔵 P3 nitpicks (2)
  • packages/api/src/utils/__tests__/sessionStore.test.ts:1 -- No test asserts that the failure counter resets after a successful recovery, so the backoff-ratchet behavior above is unobserved, and the setupFailureCounter.add(1) instrumentation path has no coverage.
    • Fix: Add a case that fails setup once, recovers, then fails again and asserts the retry delay returns to the base (and optionally assert the counter increments on failure).
  • packages/api/src/utils/sessionStore.ts:10 -- The "Same schedule as connectDBWithRetry's initial-connect retries" comment couples these constants to another module by prose only, so the two can silently drift.
    • Fix: If keeping them in sync matters, import/share the delay constants; otherwise soften the comment to avoid implying an enforced invariant.

Reviewers (8): correctness, reliability, adversarial, kieran-typescript, testing, maintainability, project-standards, previous-comments.

Testing gaps: No coverage for the failure-counter reset-on-success semantics or the setupFailureCounter increment.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 408 passed • 1 skipped • 1329s

Status Count
✅ Passed 408
❌ Failed 0
⚠️ Flaky 1
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

Use the shared isMongoConnected() readiness check so sessions and /ready
agree on what "connected" means, drop a banner comment, and type the test
helper's result instead of using any.
Comment thread packages/api/src/utils/sessionStore.ts Resolved
The mongoose (mongodb 4.x) to connect-mongo (mongodb 6.x types) client cast
needs one eslint-disable; without it the warning pushes the api package over
its --max-warnings ceiling. Raise the api eslint-disable ratchet baseline
from 36 to 37 to match.
Handing mongoose's client to connect-mongo needed a cast between mongoose's
bundled mongodb 4.x types and the 6.x types connect-mongo is typed against.
Instead, keep connect-mongo's own `mongoUrl` client, but only create the
store once mongoose is connected and create a new one if its connect or
setup fails. This drops the cast and its eslint-disable, and connect-mongo
stays on the driver it already used.
A store whose client connected but whose setup (the TTL index) failed was
dropped without closing its MongoClient, leaking a connection pool per
failure. Close it, and wait with capped exponential backoff (1s to 30s, as
connectDBWithRetry does) before building the next store so a persistent
setup failure doesn't open a new client on every session operation.
Comment thread packages/api/src/utils/sessionStore.ts Resolved
Comment thread packages/api/src/utils/sessionStore.ts Resolved
The session store opens its own MongoClient, but Server.shutdown only
closed mongoose's. Add MongoSessionStore.close(), which closes the current
store and forgets it so a restarted server builds a new one, and call it
from Server.shutdown. Also drop a guard in the failed-store handler that
was always true.
Comment thread packages/api/src/utils/sessionStore.ts Resolved
Comment thread packages/api/src/utils/sessionStore.ts Resolved
Comment thread packages/api/src/utils/sessionStore.ts Outdated
this.retryAt =
Date.now() +
Math.min(
REBUILD_BASE_DELAY_MS * 2 ** (this.failures - 1),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Backoff schedule copied from connectDBWithRetry by hand

REBUILD_BASE_DELAY_MS, REBUILD_MAX_DELAY_MS and Math.min(base * 2 ** (n - 1), max) repeat the defaults and the formula in packages/api/src/models/index.ts:95-127. The comment 'Same schedule as connectDBWithRetry' only stays true while both are edited together. Move the defaults and a small backoffDelayMs(attempt, opts) helper into a shared module and use it in both places.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

callback?: (err?: unknown) => void,
) {
this.withStore(callback, store =>
store.touch(sid, sess as Parameters<MongoStore['touch']>[1], callback),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — as cast in touch() goes against the 'Avoid as casts' convention

sess as Parameters<MongoStore['touch']>[1] bypasses type checking. Type touch's sess parameter as Parameters<MongoStore['touch']>[1] (it is compatible with express-session's Store signature) or use satisfies, instead of casting at the call site.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

The failed-store handler dropped the rejection reason, so a persistent
setup failure (missing createIndex privilege, auth error on the store's
client) only surfaced as the generic retry message during backoff. Log it
with the failure count and retry delay, as connectDBWithRetry does, and
count it in hyperdx.session_store.setup_failures.
close() replaces the current store, so a store whose setup rejects after
close() could clear a newer store (leaking its client) and count the
deliberate close as a setup failure, backing off the next start. Skip the
handler when the store is no longer current, and reset the backoff in
close() so a restarted server builds its store straight away.

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 47ff983a Deployed Oct 14, 2026 by vercel[bot]
Preview – hyperdx-storybook — 47ff983a Deployed Oct 14, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-2 Low risk — AI review + quick human skim

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant