Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions content/posts/43-red-dependabot-prs-none-were-the-dependency.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
---
title: "43 red Dependabot PRs, and not one was the dependency's fault"
date: 2026-07-26
description: "My GitHub org had 43 open Dependabot PRs, most of them showing red CI. The obvious story is 'these upgrades break things.' The actual story is that a red check can lie exactly the same way a green one can — and telling the two apart is the whole job."
slug: "43-red-dependabot-prs-none-were-the-dependency"
---

I opened the pull-requests tab across my [GitHub org](https://github.com/Aswincloud) one morning and found **43 open Dependabot PRs**, most of them flying red CI. The tempting read is the cynical one: dependency upgrades are a treadmill, half of them break your build, this is just the tax. Merge the green ones, ignore the rest, move on.

I nearly did exactly that. I'm glad I didn't, because when I actually dug in, the punchline was almost funny: **of ~25 PRs showing failing CI, not one failed because of the dependency it was bumping.**

## What a red check actually told me

Here's the thing I keep having to relearn. I've written before — about [a status page that was green for the wrong reason](/posts/status-page-that-survives-home/) — that a check passing tells you less than you think. This was the mirror image: a check *failing* also tells you less than you think. A red X means "this branch, as it exists right now, doesn't pass." It does **not** mean "the change in this PR is the reason."

Dependabot opens a PR by branching off your default branch at some moment in time and applying one dependency bump. If `main` was healthy at that moment, a red check genuinely implicates the bump. But if `main` itself was *broken* when the branch was cut, every PR branched from that point inherits the breakage — and the red X is pointing at a bug that has nothing to do with the dependency at all.

> A failing check on a stale branch is a photograph of a problem that may already be fixed upstream. You're not looking at your code; you're looking at the past.

So before triaging a single PR, I asked a different question: **when were these branches cut, and what did `main` look like then?**

```bash
# Group the org's Dependabot PRs by the day their branch was created.
# A tight cluster of "failing since <date>" is the tell: they share a cause,
# and that cause is almost certainly the state of main on that date — not
# 25 unrelated dependencies all breaking at once.
gh search prs --owner Aswincloud --author "app/dependabot" --state open \
--json number,repository,createdAt,title \
--jq 'group_by(.createdAt[0:10])[] | {day: .[0].createdAt[0:10], count: length}'
```

The failures clustered. A big batch of them, across two different repos, had all branched on the *same day* — a day when, checking the history, both of those repos' `main` branches were themselves red. One had a lint error in a UI banner component that had since been fixed. The other had a broken deploy step, also since fixed. The dependencies were innocent bystanders. The PRs were just **stale** — carrying a months-old breakage that no longer existed on `main`.

## The one-word fix

Once you know a branch is stale rather than broken, the fix is almost insultingly simple: rebase it onto current `main` and let CI run again against reality.

```
@dependabot rebase
```

That comment tells Dependabot to recreate the branch on top of the current default branch and re-run everything. It's the cleanest possible experiment, because it **isolates the variable you care about**: same dependency bump, but now on top of a healthy base. Whatever CI says *after* the rebase is finally an honest verdict on the upgrade itself.

I fired `@dependabot rebase` at every PR in the stale clusters and waited.

## Now the failures meant something

After the rebases, the board sorted itself into two clean piles, and *this* is where the real information was:

- **The ones that went green.** These were never the dependency's fault; they were hostages of a broken base. They rebased, CI passed against current `main`, and I approved them and added them to the merge queue. About 25 PRs. The queue re-runs the required checks one more time before it merges, so "green after rebase" gets a second, serialized confirmation on its way in — I'm not trusting a single snapshot.

- **The ones that stayed red.** *Now* a red X was worth reading, because the only thing that had changed was the base — and it was healthy. These were the genuine major-version incompatibilities: a React major that needed code changes, an ESLint 8→10 jump that broke config in two repos, a Next.js major, a couple of runtime majors (Express 4→5, a Node engine bump) that pass CI but change behavior in ways I want to read before shipping. That's a real short-list — a handful of PRs that actually deserve a human — instead of a wall of 43 red Xs I'd have rubber-stamped or ignored wholesale.

```bash
# Enqueue a verified-safe PR. No --squash flag on purpose: the merge queue owns
# the merge method, and the queue re-runs required checks before it merges — so
# this is a second gate, not a rubber stamp. Nothing here arms auto-merge; I
# still click "Merge when ready" on the majors myself.
gh pr merge <number> --repo Aswincloud/<repo>
```

## The lesson I keep paying tuition on

The failure that looks like the dependency is usually the base. The green check that looks like health is sometimes a ping answered by the wrong host. In both directions, the CI signal is a *starting point for a question*, not the answer — and the actual work is figuring out **why** it's the color it is.

Forty-three red PRs sounds like forty-three problems. It was really two — a lint error and a broken deploy step, both already fixed — wearing forty-three costumes. The costume was the stale branch. `@dependabot rebase` took the costumes off, and then, for the first time, the handful of PRs still standing there in red were telling me the truth.

I did this triage by hand this time. The next post is about what happened when I got tired of doing org-wide chores by hand and [built a reconciler to do them on a schedule](/posts/gh-org-guard-self-healing-governance/) — because "a human notices the board is red and investigates" is not a control you can rely on at 43 PRs, let alone 430.
71 changes: 71 additions & 0 deletions content/posts/beg-bounty-and-the-token-that-wasnt-dead.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
---
title: "A stranger emailed me about a leaked token. The scary part was the second one."
date: 2026-07-25
description: "An unsolicited 'security alert' about a secret in one of my public repos turned out to be right — but chasing it down taught me two things worth more than the fix: 'revoked' in a dashboard doesn't mean a key is dead, and the leak I already knew about was hiding a second one I didn't."
slug: "beg-bounty-and-the-token-that-wasnt-dead"
---

I got an email from someone I didn't know, flagging a hardcoded API token in one of my public GitHub repos. It had the shape of the genre: a free-mail sender, a display name dressed up to look like an automated scanner, a helpful offer to tell me more. This is a whole cottage industry — scrape public repos for anything token-shaped, mass-mail the owners, hope a few panic and pay. Most of it is noise.

The problem is that "mostly noise" is not "always wrong." This one was right. There *was* a live token sitting in a public repo. And the interesting part of this story isn't the leak — it's the two things I got wrong while cleaning it up.

## Wrong assumption #1: I thought it was already dead

The token was a [Telegram](https://telegram.org/) bot credential for one of my little side bots. When I first went to triage it, I looked at the repo's secret-scanning alert and saw it had been marked **resolved — "revoked"** months earlier. Case closed, I thought. Old news. Some past version of me had already handled this.

Then I actually tested the key.

```bash
# The only honest way to know if a token is dead: ask the API that owns it.
# 200 + bot JSON => STILL LIVE. 401 Unauthorized => actually dead.
curl -s "https://api.telegram.org/bot<redacted-token>/getMe"
# {"ok":true,"result":{"id":...,"is_bot":true,"username":"..."}} <- live.
```

It answered. `ok: true`. The token I'd been told, by my own dashboard, was "revoked" — was fully, cheerfully alive, and had been for months.

> Marking an alert "revoked" in a UI is a note to yourself. It changes a label in a database. It does **not** reach out and invalidate the secret. The only thing that kills a credential is rotating it *at the source that issued it* — here, [BotFather](https://core.telegram.org/bots#botfather). I had confused "I filed the paperwork" with "the threat is gone."

This is the same disease as [the status page that pinged the wrong host](/posts/status-page-that-survives-home/): a check that reports success for a reason unrelated to the thing you care about. There, a ping answered by Cloudflare instead of my origin. Here, a dashboard status that reflects a human clicking a dropdown, not the actual state of the key. Both are green. Both are lying.

So I did it in the correct order, which I'll never do out of order again:

1. **Rotate at the source first.** Revoke the token in BotFather; get the new one.
2. **Confirm the old one is dead.** `getMe` on the old token now returns `401`. *Now* it's actually revoked.
3. **Only then** mark the alert resolved — because now the label is finally true.

## Wrong assumption #2: this was a one-off

Having been humbled once, I stopped trusting my sense of "I'd know if there were others." I ran a sweep across every repo in my [GitHub org](https://github.com/Aswincloud) — not just "is there an open alert," but "is secret scanning even *turned on* here, and are there live keys nobody's looking at."

That last question is the one that mattered, because it exposed the trap: **secret scanning only alerts on repos where it's enabled.** A repo with scanning switched off isn't reported as risky. It's reported as *nothing at all*. Silence. And silence reads exactly like safety.

The sweep found a second live Telegram token — a different bot, a different repo — one where scanning had never been enabled. GitHub had never warned me about it because I'd never asked it to look. If I'd only chased the single token in the stranger's email and stopped, I'd have "fixed the problem" and left an identical one live three repos over.

```bash
# The sweep's real job wasn't "list open alerts" (that only covers repos already
# being watched). It was "which public repos have scanning OFF" — because those
# are the ones that can't generate an alert no matter what's committed to them.
gh api "orgs/<org>/repos?type=public&per_page=100" --paginate \
--jq '.[] | select(.archived==false)
| {name, scanning: .security_and_analysis.secret_scanning.status}'
# ...any "status":"disabled" line is a blind spot, not a clean bill of health.
```

Second token: rotate at source, confirm `401`, mark resolved, turn scanning **on** for that repo so the next mistake actually gets caught.

## What I actually changed (so it's not just a story)

Fixing two keys is not a fix. Fixing the *conditions that let two keys sit live* is. So:

- **Secret scanning + push protection on, org-wide** — including a default for new repos, so "I forgot to enable it" stops being a category of failure. Push protection is the good kind of annoying: it blocks the commit that introduces a secret, at push time, before it's ever public.
- **A weekly automated sweep** that re-asks the boring questions — is scanning on everywhere, is push protection on, are there open alerts — and files an issue if the answer drifts. I wrote about the reconcile-don't-configure philosophy behind this in [the org-guard post](/posts/gh-org-guard-self-healing-governance/); secret hygiene is just one more thing that has to be *re-checked on a schedule*, not set once and trusted forever.
- **A written triage runbook**: rotate-at-source → confirm dead → mark resolved, in that order, every time. The order is the whole point.

## The part I want to remember

The stranger's email was, functionally, spam. The genre it belongs to preys on exactly the panic reflex that makes people pay up, and the right response to "pay me to learn more" is to not. But treating the *message* as spam and treating the *claim* as false are two different moves, and only the first one was safe.

The email was worth precisely one thing: a nudge to go look. Everything valuable after that came from me looking properly — testing the key instead of trusting the label, and sweeping for the blind spots instead of fixing the one thing I'd been handed. The token in the email was never the real risk. The real risk was the one I was sure I didn't have.

Everything here is redacted — the tokens are long dead, rotated at the source, confirmed with a `401`. Which is the only version of "revoked" I trust now.
Loading
Loading