DriftGuard

API documentation goes stale silently. Tests catch code that breaks; nothing catches a doc page that quietly starts lying. DriftGuard watches the gap between a repository's OpenAPI spec and the pages meant to describe it, reports what drifted onto the pull request that caused it, and serves the current docs to AI agents over MCP.

Three surfaces, one detection engine:

• On push — scans the spec and the docs, records drift events, and closes them again once a later scan finds the problem fixed.
• On pull request — posts one comment it rewrites on each push, plus a driftguard/docs commit status a branch protection rule can require.
• For agents — an MCP server with three tools: read the current docs, search them semantically, and report a problem found while using them.

TypeScript · Express · PostgreSQL + pgvector · MongoDB · Redis/BullMQ · Anthropic or Ollama.

An illustrated walkthrough of the pipeline → — six drawn chapters from webhook to MCP, plus a searchable index of the written docs. Source in site/.

Quickstart

cp .env.example .env     # DATABASE_URL, MONGODB_URI, REDIS_URL, API_KEY_SALT
npm install
npm run dev:all          # datastores, migrations, API + worker + MCP server

dev:all validates your .env, brings up Postgres/Mongo/Redis and waits for them to report healthy, applies migrations, checks whether your model backend is reachable, then runs all three services in one terminal. Ctrl-C stops everything.

Then, with nothing else set up:

npm run seed:demo

Repos normally arrive by webhook, so a fresh install has no data and every endpoint correctly returns an empty list. seed:demo builds a complete tenant — a user, an API key, an owned installation, a repo whose spec is parsed by the real parser, doc pages stored as real Mongo snapshots — and then runs the actual drift check over it, so the events it reports are produced by the real pipeline rather than fabricated.

The fixture is rigged to produce both kinds of drift: POST /v1/widgets exists in the spec with no doc mentioning it (structural, no model needed), and the docs for GET /v1/widgets/{id} contradict the spec (semantic, needs a model). It prints an API key and ready-to-paste curl commands for the REST API and the MCP server.

That fixture immediately earned its keep: it exposed that endpoint/doc matching worked on path only, so POST /v1/widgets counted as documented merely because a page mentioned GET /v1/widgets. Adding a method to an existing path — one of the most common real drift cases there is — was undetectable by construction.

To receive real webhooks you also need a public URL and a GitHub App: see docs/github-app-setup.md.

Architecture

GitHub push/PR
      │  webhook (HMAC-verified)
      ▼
Express route ──── verifies, resolves repo+owner, ENQUEUES, returns 202
      │            ~50ms — nothing slow happens in the request
      │  Redis / BullMQ
      ▼
┌─────────────────── Worker process ───────────────────┐
│                                                       │
│  repo-scan queue                drift-check queue     │
│  ───────────────                ─────────────────     │
│  push → canonical scan  ──────► structural diff       │
│    tree, blobs, spec,           + LLM semantic check  │
│    snapshots, embeddings                              │
│                                                       │
│  PR   → preview scan ─────────► same analysis,        │
│    reads only, persists           in memory only      │
│    NOTHING                            │               │
└───────────────────────────────────────┼───────────────┘
       │                    │           │
       ▼                    ▼           ▼
  PostgreSQL            MongoDB     GitHub PR bot
 repos, endpoints,     raw doc +    comment + commit
 doc_pages+vectors,      spec        status check
 drift_events,         snapshots
 users, api_keys
       │
       ├──────────────► REST API   (Express, API-key auth, per-user scoped)
       └──────────────► MCP Server (get_current_docs, search_docs,
                                    report_doc_issue — same API keys)

Two things in that diagram are load-bearing rather than incidental. The webhook route does no content fetching at all, because GitHub closes the delivery window after ten seconds and a real repo's doc tree takes longer than that. And the PR path is read-only by construction — the schema has one doc_pages row per file path with no notion of a branch, so a PR scan that persisted anything would overwrite what the default branch says.

Why two databases

• PostgreSQL — relational state: users, GitHub installations, repos, API endpoints, doc pages, drift events, API keys. Anything that needs foreign keys, joins, or transactional writes. Doc-page embeddings live here too, in a pgvector column.
• MongoDB — large, schema-flexible, versioned blobs: raw doc content and raw OpenAPI spec snapshots, one per commit. Postgres holds a pointer (mongo_snapshot_id) to the relevant Mongo document.

The split isn't arbitrary. One side is heavily joined and constraint-enforced; the other is append-only blobs keyed by commit, kept for every commit rather than overwritten. Storing both in one place means compromising the shape of one of them.

Documentation

docs/getting-started.md	Setting it up for real, end to end
docs/github-app-setup.md	Connecting GitHub, with a check at every step
docs/design-notes.md	How each phase works and why it's shaped that way
docs/api/	The REST API reference — and DriftGuard's own drift-detection fixture
site/	The illustrated field guide, published to GitHub Pages

That last one is worth a word. DriftGuard carries its own openapi.yaml and its own API reference under docs/, so it watches its own documentation for drift. Nine endpoints, all documented — the result that proves it doesn't cry wolf.

Project status

• Phase 0 — scaffold (Express+TS, Postgres+Mongo, Redis/BullMQ, CI)
• Phase 1 — GitHub App ingestion (webhooks → OpenAPI + doc scan → Postgres/Mongo)
• Phase 2 — drift detection worker (structural + LLM semantic diff, Anthropic or Ollama)
• Phase 3 — MCP server (tools + RAG search over docs)
• Phase 4 — REST API + auth (+ 4.1: review fixes — MCP auth, per-user scoping)
• Phase 4.2 — hardening (pagination, per-key rate limiting, request IDs)
• Phase 5 — GitHub PR bot (comment + commit status, non-destructive PR scans)
• Phase 6 — case study harness (npm run case-study against real public repos)
• Phase 7 — packaging (architecture, demo script, GitHub App walkthrough)

What has and hasn't been exercised

124 unit tests, plus five smoke suites that run against real Postgres, Mongo, Redis and Ollama rather than mocks — because the defects that mattered most here were all of the form "this statement touches more rows than intended", which no amount of mocking can catch.

The GitHub App path has now run end to end against a live repository: a push triggers a scan, a pull request gets a comment naming the endpoint that drifted, and the stored default-branch state is untouched by it. That run immediately found a bug two phases old — every job id contained a colon, which BullMQ rejects outright, so no real webhook enqueue had ever succeeded. Nothing caught it because seed:demo calls the drift check directly and the smoke suites drive the repository layer: no test had ever put a job on a real queue.

Still unproven: the Anthropic provider. Every semantic verdict so far has come from a local llama3.1, which produces false positives freely — on DriftGuard's own accurate docs it called 13 of 13 pages mismatched. The structural layer got the same run exactly right, which is the argument for keeping the two layers separate.

docs/github-app-setup.md ends with an honest list of what would still need doing before this served anyone but its author.

Scripts

Command	What it does
npm run dev:all	Datastores, migrations and all three services in one terminal
npm run dev	API only, with hot reload
npm run worker:dev	The repo-scan and drift-check queues
npm run mcp:dev	MCP server on :4100
npm run migrate	Apply Postgres migrations
npm run seed:demo	Seed a full demo tenant — no GitHub App needed
npm run case-study -- <owner/repo>...	Run the real analyser against public repos, write a report
npm run create-api-key -- --username <name>	Bootstrap the first API key for a user
npm run link-installation	List or claim installations that have no owner
npm test	Vitest — 124 tests
npm run typecheck	tsc --noEmit, including scripts/
npm run lint	ESLint over src/ and scripts/
npm run build / npm start	Compile to dist/ and run it

Five smoke suites run against real infrastructure rather than mocks, because the defects that mattered most in this project were all of the form "this statement touches more rows than intended" — which no amount of mocking can catch.

Command	Covers
npm run smoke:db	Repository layer vs. real Postgres
npm run smoke:drift	Drift-event resolution and dedup scoping
npm run smoke:ingestion	Repo and installation reconciliation
npm run smoke:api	Full REST API, auth, per-user scoping
npm run smoke:mcp	Full MCP server vs. Postgres + Mongo + Ollama

The three that need only Postgres are the fastest check that a change hasn't broken the data layer:

npm run migrate && npm run smoke:db && npm run smoke:drift && npm run smoke:ingestion