Sentinel

Watches vendor API changelogs, detects breaking changes before they hit customer codebases, scans a customer repo for affected usages, generates reviewable fix diffs, and opens a PR — before the sunset date, not after something breaks in production.

Quickstart (zero setup, 60 seconds)

See it detect a real breaking change with no account, API key, or database — just Node:

git clone https://github.com/rtsdque/sentinel.git
cd sentinel
npm install
npx tsx src/cli.ts stripe --track acacia

That's it — this scrapes Stripe's real changelog and prints a fully structured breaking-change event to your terminal, deterministically (no LLM call). Everything past this point (interpreting changes with an LLM, scanning a codebase, opening PRs, running on a schedule) layers on top and needs progressively more setup — jump to the phase you want below, or npm test to run the suite (also needs nothing but Node).

See it in action

This is a real diff Sentinel generated and opened as a PR, unedited, against a test repo — Stripe nested three Balance Settings request fields under a new required payments object, and Sentinel found the affected call site and proposed the migration on its own:

 export async function enableNegativeBalanceDebits() {
   return stripe.balanceSettings.update({
-    debit_negative_balances: true,
-    payouts: {
-      schedule: { interval: "daily" },
+    payments: {
+      debit_negative_balances: true,
+      payouts: {
+        schedule: { interval: "daily" },
+      },
     },
   });
 }

The PR body links the upstream changelog entry, states the deadline if there is one, and separately lists anything it couldn't confidently fix — see Phase 4 and Phase 5 below for what makes it refuse rather than guess.

Why not just use Dependabot / Renovate / oasdiff?

Reasonable question — here's what's actually different, based on a real look at the landscape, not a guess:

Tool	What it does	Where it stops
Dependabot (incl. AI-agent remediation)	Reacts to a known CVE or a new package version in your manifest	Anchored to your dependency file — never reads a vendor's changelog prose, only version diffs
Renovate	Same, plus customDatasources for arbitrary version feeds	Still just version tracking — no interpretation of what changed or why it breaks you
oasdiff / openapi-diff	Diffs two OpenAPI specs, flags breaking changes precisely	Needs the vendor's own spec-versioned CI — doesn't touch your codebase at all
Visualping / Theneo / PageCrawl	Watch a changelog page, alert on any change	Stops at a notification. You still read it, find the usages, and fix them yourself
ChangeSpec	An open JSON format for vendors to publish structured change events	A spec waiting for vendor adoption — Stripe doesn't publish in it today, so someone still has to read the prose

Every one of these is genuinely good at one piece. None of them do the thing this YC RFS actually asked for: watch a vendor's unstructured changelog prose, turn it into a structured breaking change, find every place your specific codebase uses the affected surface, and propose the fix — before your build breaks, not after. That's the gap Sentinel targets, and as far as we could find during a real competitive pass (not a hunch), nobody had shipped the full pipeline yet.

Status

Phases 1–6 built, scoped to Stripe + TypeScript/JavaScript customer repos only (see project scope notes — deliberately not broad).

Phase	What it does	Entry point
1. Change Watcher	Scrapes Stripe's changelog, deterministically parses breaking-change entries into structured ChangeEvents	src/cli.ts
2. Change Interpreter	LLM step (claude-opus-4-8) that decomposes prose + the parsed param table into atomic, machine-actionable changes	src/cli.ts --interpret
3. Codebase Scanner	AST-based (ts-morph) scan of a TS/JS repo for usages of the affected API surface, scoped to real Stripe client calls	src/scan.ts
4. Fix Generator	Per-match LLM call that proposes a diff or explicitly declines with a reason — never guesses	src/fix.ts
5. PR Bot	GitHub App that applies approved fixes, formats with Prettier, and opens a PR — every PR states what wasn't auto-fixed	src/open-pr.ts
6. Persistence & Scheduling	Postgres-backed "have I seen this" store + a repo registry, so a run only acts on genuinely new changes	src/run-scheduled.ts

Nothing in this pipeline auto-merges. Every fix is generated for human/PR review; the PR body explicitly separates "proposed changes" from "needs manual review," because there's no customer test suite to verify correctness against.

Structure

src/
  lib/                 fetch wrapper, markdown-table parsing
  watcher/stripe/       Phase 1: changelog fetch + parse + ChangeEvent builder
  interpreter/stripe.ts Phase 2: LLM decomposition into atomic changes
  scanner/typescript.ts Phase 3: ts-morph AST scan for affected usages
  fixer/stripe.ts       Phase 4: per-match fix generation + apply/consolidate/format
  github/
    app.ts               GitHub App installation auth
    pr.ts                 branch + commit + PR creation
  cli.ts / scan.ts / fix.ts / open-pr.ts   entry points for each phase
  types.ts
data/stripe/           structured ChangeEvent JSON, one file per detected entry
examples/sample-customer-repo/  synthetic fixture repo used to validate the scanner/fixer

Setup

npm install
cp .env.example .env

Fill in .env directly in your editor — never paste secrets into chat. ANTHROPIC_API_KEY is needed for Phases 2 and 4; the GITHUB_APP_* vars are needed for Phase 5 (see below).

Note on npm run <script> -- --flags: on Windows, npm's .cmd shim can silently drop --flag-style arguments passed after --. Every command below uses npx tsx directly instead, which doesn't have this problem — pass --env-file=.env if the command needs your API key.

Phase 1 — Change Watcher

npx tsx src/cli.ts stripe --track acacia

Flags: --track <name> (default acacia; current live track is dahlia), --all (include non-breaking entries), --limit N. Output: one JSON file per ChangeEvent under data/stripe/.

Stripe serves clean raw markdown for any doc page via a .md suffix — no HTML scraping. The index page flags each entry Breaking/Non-breaking; detail pages include a per-language parameter table (Added/Removed) that Phase 1 parses deterministically.

Phase 2 — Change Interpreter

npx tsx --env-file=.env src/cli.ts stripe --track acacia --interpret

Attaches an interpreted field to each ChangeEvent: an array of atomic changes (param_renamed, param_removed, method_removed, response_shape_changed, etc., each with from/to/description), a migration summary, and a confidence level. Deduplicates to one canonical entry per target — validated against 10 real breaking changes across 3 release tracks. Stripe's own tables are sometimes incomplete (e.g. list only the new values with no explicit mapping from old ones); the interpreter recovers the full mapping from prose in those cases.

Phase 3 — Codebase Scanner

npx tsx src/scan.ts <path-to-customer-repo> --events <eventId1>,<eventId2>

Omit --events to scan against every interpreted event in data/stripe/. TypeScript/JavaScript only. Uses ts-morph to parse the real AST rather than grep:

• Call-argument matches (request params) are scoped by tracing the call's callee chain back to a known Stripe client variable — either constructed locally (new Stripe(...)) or imported from another local module that constructs one (import stripe from "./server", resolved recursively through re-exports via ts-morph's getExportedDeclarations()). Not just "any object literal in a file that happens to import stripe."
• Both call-argument and property-access matches require the target's parent path segment (when it has one — e.g. billing_mode for billing_mode.type, errors for Account.requirements.errors[].code) to appear in the surrounding context — the receiver expression's text for property access, the enclosing call's callee text for call arguments — guarding generic leaf names like type, status, or code from matching unrelated objects.

Both are AST/text heuristics, not full type-checker provenance — meaningfully better than grep, but not infallible. Validated in two stages:

1. Against examples/sample-customer-repo, a fixture with deliberate false-positive traps (an unrelated local field sharing a param name, an unrelated Stripe Event.type read) — zero false positives.
2. Against a real, unmodified production codebase — Cal.com, a large TypeScript monorepo — which immediately surfaced a real gap the fixture couldn't: Cal.com constructs its Stripe client once in a shared module and imports it everywhere else, a very common production pattern our same-file-only detection completely missed (a false negative, confirmed empirically with a synthetic target before fixing). Fixing that then widened the surface for a second real false positive — a generic leaf name (code) matching an unrelated OAuth call — which is what the context-guard extension above addresses. Re-verified clean on both the fixture and Cal.com after both fixes.
3. Against a second real codebase — Documenso, an npm workspace monorepo — which surfaced two more real findings:
   • The shared Stripe client is imported via a monorepo package alias (@documenso/lib/server-only/stripe), not a relative path. ts-morph's module resolution needs the workspace's node_modules symlinks on disk to follow this — a bare git clone isn't enough; dependencies must be installed (npm install, ideally with --ignore-scripts for read-only static analysis of a repo you don't otherwise trust) before scanning a workspace monorepo, or every cross-package import silently resolves to nothing.
   • Once installed, scanning surfaced a case the context guard can't help with: a bare rename target with no path segments at all (e.g. name, or in a live run, filter) has no context to check, so property-access matching falls back to "anywhere in a file that touches a Stripe client" — which matched an unrelated Prisma call's .name field in testing, and in a real run against the 10 validated events, matched Array.prototype.filter() calls (.data.filter(...)) against the Billing Alerts filter field purely on name collision. Rather than chase a fragile heuristic fix (properly solving this needs type-checker provenance — does the receiver actually originate from a Stripe response? — which is a bigger lift than this pass), matches like this now carry lowConfidence: true and print with an explicit warning in scan.ts/fix.ts output, so they're surfaced for extra scrutiny instead of looking identical to a well-scoped match.

Phase 4 — Fix Generator

npx tsx --env-file=.env src/fix.ts <path-to-customer-repo> --events <eventId1>,<eventId2>

For each scan match, sends the enclosing statement + the atomic change details to claude-opus-4-8 and gets back a structured verdict: canGenerateFix, confidence, a proposed replacement (or null), an explanation, and caveats. Prints two sections: fixes with a proposed diff, and matches flagged for manual review with no diff proposed.

The generator is instructed to refuse rather than guess when a change alters a value's shape, not just its key name (e.g. a boolean becoming a string enum) — validated against exactly that trap and two others; it correctly declined all three with specific, useful reasoning, and correctly generated an accurate diff for a case that was genuinely safe (a pure structural rename with no value-type change).

Phase 5 — PR Bot

npx tsx --env-file=.env src/open-pr.ts <path-to-local-clone> --owner <org> --repo <repo> --base main --events <eventId>

Runs scan → fix → apply (with statement-level deduplication when multiple atomic changes land in the same statement, and a Prettier pass to clean up indentation from spliced-in replacement text) → opens one PR per ChangeEvent. The PR body always includes both the applied changes and the items flagged for manual review — nothing is silently dropped — plus a disclaimer that nothing has been tested.

GitHub App setup (one-time, you do this — not something Claude can do on your behalf)

This touches customer code, so the app is scoped as narrowly as possible:

Permission	Level	Why
Contents	Read & write	Create a branch, commit the fix
Pull requests	Read & write	Open the PR
Metadata	Read (mandatory default)	Required by every GitHub App

No Administration, no Actions, no org-wide access. No webhook subscription needed — this runs on-demand from the CLI, not as an event listener, so you can leave "Webhook" unchecked when registering the app.

1. Go to github.com/settings/apps/new (or your org's /settings/apps/new for an org-owned app).
2. Set the permissions above under "Repository permissions."
3. Uncheck "Active" under Webhook (not needed for this phase).
4. Under "Where can this GitHub App be installed?", choose "Only on this account" unless you specifically want it public.
5. Create the app, note the App ID shown on its settings page.
6. Scroll to "Private keys" → Generate a private key — downloads a .pem file. Save it somewhere local (not inside this repo), and set GITHUB_APP_PRIVATE_KEY_PATH in .env to that file's path. Never paste the key's contents into chat or commit it anywhere.
7. Click Install App, choose a repo you own to test against (ideally a throwaway/test repo, not anything real, for the first run), and select "Only select repositories."
8. After installing, the URL bar shows .../installations/<number> — that number is GITHUB_APP_INSTALLATION_ID.
9. Set GITHUB_APP_ID and GITHUB_APP_INSTALLATION_ID in .env.

I can't do any of this myself — it requires your GitHub account and creates a real, persistent app registration. Once it's set up, open-pr.ts only ever acts within that installation's granted repos.

Phase 6 — Persistence & Scheduling

Postgres, via Drizzle ORM. Two ways to get one:

• Cloud (needed for real scheduling): a free Neon or Supabase project — required if you actually want .github/workflows/scheduled-scan.yml to run unattended, since a GitHub Actions runner needs a database it can reach over the network, not one on your laptop.
• Local, for just trying it out: docker compose up -d starts Postgres on localhost:55432 (deliberately not the default 5432 — many machines already have a local Postgres service bound to that port, and connections would silently land on the wrong server with a confusing auth error instead of an obvious "port in use"). Then set DATABASE_URL=postgresql://sentinel:sentinel@localhost:55432/sentinel in .env. Fine for running run-scheduled.ts by hand; won't work for the cloud cron workflow.

Three tables (src/db/schema.ts):

• change_events — every detected entry, keyed by the same stable id the watcher already computes. Row existence is what "have I seen this before" means.
• watched_repos — registry of which repos to scan for which vendor, replacing manually passing --owner/--repo on the CLI.
• pull_requests — log of PRs Sentinel has opened; prevents re-opening a PR for a (change event, repo) pair already handled, and is the source of truth for the "users" metric (distinct repos with at least one row here).

npm run db:generate   # generate a migration from the schema
npm run db:migrate    # apply it

npx tsx src/backfill.ts <track>   # mark existing history as known WITHOUT interpreting or acting
                                    # on any of it — run this once per track when first adopting
                                    # Sentinel, so a scheduled run only reacts to changes from this
                                    # point forward, not the entire historical backlog. No LLM calls.

node --env-file=.env --import tsx src/manage-repos.ts add --owner <o> --repo <r>
node --env-file=.env --import tsx src/manage-repos.ts list

node --env-file=.env --import tsx src/run-scheduled.ts --track <track>

run-scheduled.ts is the actual orchestrator: check a track for entries not yet in change_events → interpret only the new ones (the cost-saving point of persistence — a backfilled track costs nothing to re-check) → for each active watched repo, clone it fresh via the GitHub App's own installation token (never a persisted local path — a scheduled run has no guarantee of running on the same machine twice, and a cloud runner has no access to anyone's local filesystem at all), npm install --ignore-scripts (needed for monorepo package-alias imports to resolve — see Phase 3), scan, fix, apply, and open a PR — skipping any (event, repo) pair already in pull_requests.

This now genuinely runs on a schedule, not just on-demand: .github/workflows/scheduled-scan.yml runs it every 6 hours via GitHub Actions cron (plus a manual workflow_dispatch trigger for testing), reading secrets from the Actions secrets store rather than a local .env. Secret names can't start with GITHUB_ (a hard GitHub restriction), so they're stored as SENTINEL_* and mapped to the expected env var names inside the workflow's env: block. The private key is provided as raw content via GITHUB_APP_PRIVATE_KEY in CI (no local file to point a path at); local dev keeps using GITHUB_APP_PRIVATE_KEY_PATH — loadPrivateKeyContent() in src/github/app.ts checks the inline content first, falls back to the path.

Validated live in both places, not just typechecked:

• Locally: backfilled the real basil track (33 entries via deterministic parsing, zero LLM cost), removed one known-good entry to simulate it being newly discovered, ran the orchestrator against the real rtsdque/app-test repo — it correctly found exactly the 1 new entry (not the other 32), interpreted only that one, cloned the repo fresh, opened a correct PR, and recorded it. Running it again immediately found 0 new entries and did nothing, confirming the dedup path.
• In the cloud: pushed to a real (private) GitHub repo, rtsdque/sentinel, set the 5 required secrets, and triggered the workflow for real via workflow_dispatch — it connected to Neon from GitHub's own infrastructure, correctly found 0 new entries on basil (already fully processed), and exited clean. Confirms the whole chain — secrets, npm ci, DB connectivity, cloning — works outside this machine, not just here.
• Multi-repo: registered a second fixture, examples/sample-monorepo-repo (pushed to its own repo, rtsdque/sentinel-test-monorepo) — a real npm workspace monorepo with the Stripe client constructed in one package and imported by another via a package alias (@fixture/stripe-client), the exact shape found in Documenso. Until this test, that pattern had only ever been scanned read-only, never run through fix generation and PR creation. Re-ran the same balance-settings change against both watched repos in a single orchestrator invocation — it correctly opened two independent, correctly-scoped PRs (app-test#5, sentinel-test-monorepo#1), each with the right diff, both logged against the right repo id in pull_requests.

Known gap surfaced by this design, not yet resolved: a watched repo is only checked against events discovered after it was registered. Adding a new repo to watched_repos doesn't scan it against events already in change_events — there's no "backfill this repo against the last N known changes" path yet. Fine for the steady-state case (repos stay registered, watcher runs continuously), but a gap for onboarding a repo after some changes have already been seen.

Known limitations / flagged open questions

• This repo's own visibility: rtsdque/sentinel is currently private. Going public is a deliberate later decision, not an accident to guard against — but worth knowing the code, data/stripe/*.json, and migration history are all sitting in a repo that isn't public yet, so nothing here has had public scrutiny.
• Scanner/fixer scope: TypeScript/JavaScript + Stripe only. No Python, no other vendors yet.
• AST heuristics, not type-checker provenance: both the scanner's context guards and the fixer's statement-boundary matching are heuristics that can be fooled by unusual code shapes. Treat every match and diff as a candidate for review, not proof. Cross-file client resolution handles direct exports, re-export chains, and monorepo package-alias imports (confirmed against both Cal.com and Documenso) — but only once the repo's dependencies are actually installed (see Phase 3 above); a bare git clone isn't enough for workspace monorepos. Still untested: computed member access (stripe["oauth"]["token"]), a client factory function instead of a bare exported instance. Bare (path-less) rename targets with a common English-word leaf (name, filter, id) remain a known false-positive risk for property-access matching — surfaced via lowConfidence: true on the ScanMatch/FixCandidate, not eliminated; fully closing this needs type-checker provenance (does the receiver actually originate from a Stripe response?), which is future work.
• One PR per file-touching event, sequential: open-pr.ts doesn't yet dedupe against a PR that already exists for the same branch beyond a clear error message — no update-existing-PR flow yet.

License

MIT