Skip to content

test: run the api integration suite on DocumentDB - #161

Merged
catinspace-au merged 11 commits into
mainfrom
fix/documentdb-tests
Oct 23, 2026
Merged

catinspace-au merged 11 commits into
mainfrom
fix/documentdb-tests

Conversation

@catinspace-au

@catinspace-au catinspace-au commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

DFE keeps HyperDX's documents in DocumentDB from 2.2.1. This runs the api integration suite, our dfe/ tests included, against it on every PR, fixes the three races it found in our code, and moves our compose overrides off FerretDB. No upstream file is touched.

The harness:

  • New workflow dfe-documentdb: 4 shards of @hyperdx/api ci:int on documentdb-local 1.0 on PostgreSQL 18.6 (pg18-1.0.0@sha256:03aef53ed72cd3007236f8ce1b4ef7a0c958dbbce1519074b85d8a3e10be89a5, amd64 and arm64), the build dfe-infra and dfe-docker deploy. TLS required, SCRAM-SHA-256.
  • dfe/test/docker-compose.documentdb.yml layers over upstream's docker-compose.ci.yml and swaps its MongoDB out. DFE_INT_MONGO_URI points the suite at it through int-store.setup.ts, which jest.dfe.int.config.js runs after upstream's dotenv loader.
  • tlsAllowInvalidCertificates=true is for the harness only. The image signs its own certificate there.
  • Upstream's chore(dev): run local dev, integration and e2e on DocumentDB ClickHouse/ClickStack#3375 is NOT carried. None of it was needed.
  • Two upstream MCP assertions are skipped by name. They fail on MongoDB too, against the @modelcontextprotocol/sdk 1.32.1 our lockfile carries where upstream's has 1.29.0.
  • Renovate now scopes the harness compose file and the workflow.

The three races, all in dfe/:

  • First logins racing on a NEW team name made several teams: four of five in CI. ensureUniqueNameIndex marked the index ensured before createIndex returned. Every caller now awaits one build through indexOnce.
  • Two upserts racing to insert one unique key fail the loser on DocumentDB with 11000 or 112, where MongoDB retries it itself. retryUpsertRace in dfe/models/duplicate-key.ts retries exactly those codes, four attempts with short jittered backoff, then rethrows. recordDashboardTombstone and the DfeSource upsert behind PUT /dfe/sources/:name both use it, so a concurrent PUT no longer answers 500.
  • Two PUTs of one source name created it twice on a team with a connection, on MongoDB and DocumentDB alike. Our code now builds a unique index on the upstream Source collection over {team, name} (dfe_one_source_name_per_team, nothing in the upstream model), and the find-then-write goes through retryUpsertRace.
  • store-operations.int.test.ts races each of these ten rounds with jest retries OFF, so a lost race fails the first attempt.

Compose and docs:

  • docker-compose.dfe.yml runs documentdb-local pg18-1.0.0 in place of FerretDB and its Postgres. A one-shot documentdb-tls mints a CA and the gateway certificate, and the app verifies the gateway against that CA and logs in with SCRAM-SHA-256.
  • docker-compose.dfe.dev.yml makes the same swap for dev on host port 27017, with the gateway's self-signed certificate because the app runs on the host.
  • ADR 0006 records DocumentDB replacing FerretDB, and 0001 is marked superseded.

Done when two dfe-documentdb runs go green on this head with no retried test.

DFE keeps HyperDX's documents in DocumentDB from 2.2.1, so the api integration suite, our dfe/ tests included, now runs against it on every PR in the new dfe-documentdb workflow.

dfe/test/docker-compose.documentdb.yml layers over upstream's docker-compose.ci.yml and swaps its MongoDB for documentdb-local pg17-1.0.0, digest-pinned, requiring TLS and SCRAM-SHA-256. DFE_INT_MONGO_URI points the suite at it through a setup file jest.dfe.int.config.js runs after upstream's dotenv loader. No upstream file changes.

store-operations.int.test.ts covers the dfe store calls no other integration test reached: the first-login race on the unique team name, tombstone upserts on their compound unique index and the source manifest's Mixed spec. Two racing upserts on one unique key fail the losing call on DocumentDB with 11000 or 112, where MongoDB retries it, and the test pins the one row either way.

Two upstream MCP assertions are skipped by name in the workflow. They fail on MongoDB as well, against the @modelcontextprotocol/sdk 1.32.1 our lockfile carries.
ensureUniqueNameIndex marked the index ensured before createIndex returned, so concurrent first logins skipped it and found-then-created a team each. On a store with no index yet, which is every fresh deploy, five racing logins made four teams on DocumentDB in CI. Every caller now awaits one shared createIndex promise, and a failed build clears it so the next call tries again.
Two upserts racing to insert one unique key fail the loser on DocumentDB with 11000 or 112, where MongoDB retries it itself. retryUpsertRace in dfe/models/duplicate-key.ts retries exactly those two codes up to four attempts with short jittered backoff, then rethrows. recordDashboardTombstone and the DfeSource upsert behind PUT /dfe/sources/:name both go through it, so a concurrent PUT no longer answers 500.

store-operations.int.test.ts races each caller ten rounds and runs with jest retries off, so a lost race fails the first attempt.
The harness moves to documentdb-local pg18-1.0.0, the same 1.0 build on PostgreSQL 18.6 that dfe-infra and dfe-docker deploy, digest-pinned for amd64 and arm64. Renovate now scopes the harness compose file and the dfe-documentdb workflow, so the digest and the action pins stay current.
docker-compose.dfe.yml swaps FerretDB and its Postgres for documentdb-local pg18-1.0.0. A one-shot documentdb-tls mints a CA and the gateway certificate into a volume, the gateway refuses plain connections, and the app gets only the CA, verifying the gateway as documentdb and logging in with SCRAM-SHA-256. Usage telemetry is off.
upsertOnTeam found and then created, so two concurrent PUTs of one source name each created it on a team with a connection: two sources, on MongoDB and DocumentDB alike. The dfe layer now builds a unique index on the upstream Source collection over {team, name}, named dfe_one_source_name_per_team, and the upstream model stays as it ships. The find-then-write goes through retryUpsertRace, so the loser's create fails on the index and its retry replaces the winner's source.

indexOnce in dfe/models/index-once.ts builds such an index once for every caller, and the team-name index now uses it too.
docker-compose.dfe.dev.yml swaps FerretDB and its Postgres for documentdb-local pg18-1.0.0 on host port 27017, requiring TLS with a SCRAM-SHA-256 login and with usage telemetry off. The app runs on the host, which cannot read a CA held in a volume, so dev keeps the gateway's self-signed certificate, and the file's header gives the dev MONGO_URI with tlsAllowInvalidCertificates. docker-compose.dfe.yml loses one non-ASCII dash.
0006 records DocumentDB 1.0 replacing FerretDB, following ClickHouse/ClickStack#3375. 0001 keeps its text and gains a status line pointing at it, and the index marks it superseded.
@catinspace-au
catinspace-au marked this pull request as ready for review October 23, 2026 00:07
A request that sent an object for the source name, such as {"$ne": null}, made the exists, upsert and delete filters match any source. Each filter now compares with $eq.
The upsert's update document carried the caller's spec at the top level, so CodeQL still read it as operator injection. The update now goes through an explicit $set, and the unit tests expect the $eq filters.
@catinspace-au
catinspace-au merged commit f354d43 into main Oct 23, 2026
32 checks passed
@catinspace-au
catinspace-au deleted the fix/documentdb-tests branch October 23, 2026 00:43
@github-actions

Copy link
Copy Markdown
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants