diff --git a/content/posts/43-red-dependabot-prs-none-were-the-dependency.md b/content/posts/43-red-dependabot-prs-none-were-the-dependency.md new file mode 100644 index 0000000..2d577b4 --- /dev/null +++ b/content/posts/43-red-dependabot-prs-none-were-the-dependency.md @@ -0,0 +1,68 @@ +--- +title: "43 red Dependabot PRs, and not one was the dependency's fault" +date: 2026-07-26 +description: "My GitHub org had 43 open Dependabot PRs, most of them showing red CI. The obvious story is 'these upgrades break things.' The actual story is that a red check can lie exactly the same way a green one can — and telling the two apart is the whole job." +slug: "43-red-dependabot-prs-none-were-the-dependency" +--- + +I opened the pull-requests tab across my [GitHub org](https://github.com/Aswincloud) one morning and found **43 open Dependabot PRs**, most of them flying red CI. The tempting read is the cynical one: dependency upgrades are a treadmill, half of them break your build, this is just the tax. Merge the green ones, ignore the rest, move on. + +I nearly did exactly that. I'm glad I didn't, because when I actually dug in, the punchline was almost funny: **of ~25 PRs showing failing CI, not one failed because of the dependency it was bumping.** + +## What a red check actually told me + +Here's the thing I keep having to relearn. I've written before — about [a status page that was green for the wrong reason](/posts/status-page-that-survives-home/) — that a check passing tells you less than you think. This was the mirror image: a check *failing* also tells you less than you think. A red X means "this branch, as it exists right now, doesn't pass." It does **not** mean "the change in this PR is the reason." + +Dependabot opens a PR by branching off your default branch at some moment in time and applying one dependency bump. If `main` was healthy at that moment, a red check genuinely implicates the bump. But if `main` itself was *broken* when the branch was cut, every PR branched from that point inherits the breakage — and the red X is pointing at a bug that has nothing to do with the dependency at all. + +> A failing check on a stale branch is a photograph of a problem that may already be fixed upstream. You're not looking at your code; you're looking at the past. + +So before triaging a single PR, I asked a different question: **when were these branches cut, and what did `main` look like then?** + +```bash +# Group the org's Dependabot PRs by the day their branch was created. +# A tight cluster of "failing since " is the tell: they share a cause, +# and that cause is almost certainly the state of main on that date — not +# 25 unrelated dependencies all breaking at once. +gh search prs --owner Aswincloud --author "app/dependabot" --state open \ + --json number,repository,createdAt,title \ + --jq 'group_by(.createdAt[0:10])[] | {day: .[0].createdAt[0:10], count: length}' +``` + +The failures clustered. A big batch of them, across two different repos, had all branched on the *same day* — a day when, checking the history, both of those repos' `main` branches were themselves red. One had a lint error in a UI banner component that had since been fixed. The other had a broken deploy step, also since fixed. The dependencies were innocent bystanders. The PRs were just **stale** — carrying a months-old breakage that no longer existed on `main`. + +## The one-word fix + +Once you know a branch is stale rather than broken, the fix is almost insultingly simple: rebase it onto current `main` and let CI run again against reality. + +``` +@dependabot rebase +``` + +That comment tells Dependabot to recreate the branch on top of the current default branch and re-run everything. It's the cleanest possible experiment, because it **isolates the variable you care about**: same dependency bump, but now on top of a healthy base. Whatever CI says *after* the rebase is finally an honest verdict on the upgrade itself. + +I fired `@dependabot rebase` at every PR in the stale clusters and waited. + +## Now the failures meant something + +After the rebases, the board sorted itself into two clean piles, and *this* is where the real information was: + +- **The ones that went green.** These were never the dependency's fault; they were hostages of a broken base. They rebased, CI passed against current `main`, and I approved them and added them to the merge queue. About 25 PRs. The queue re-runs the required checks one more time before it merges, so "green after rebase" gets a second, serialized confirmation on its way in — I'm not trusting a single snapshot. + +- **The ones that stayed red.** *Now* a red X was worth reading, because the only thing that had changed was the base — and it was healthy. These were the genuine major-version incompatibilities: a React major that needed code changes, an ESLint 8→10 jump that broke config in two repos, a Next.js major, a couple of runtime majors (Express 4→5, a Node engine bump) that pass CI but change behavior in ways I want to read before shipping. That's a real short-list — a handful of PRs that actually deserve a human — instead of a wall of 43 red Xs I'd have rubber-stamped or ignored wholesale. + +```bash +# Enqueue a verified-safe PR. No --squash flag on purpose: the merge queue owns +# the merge method, and the queue re-runs required checks before it merges — so +# this is a second gate, not a rubber stamp. Nothing here arms auto-merge; I +# still click "Merge when ready" on the majors myself. +gh pr merge --repo Aswincloud/ +``` + +## The lesson I keep paying tuition on + +The failure that looks like the dependency is usually the base. The green check that looks like health is sometimes a ping answered by the wrong host. In both directions, the CI signal is a *starting point for a question*, not the answer — and the actual work is figuring out **why** it's the color it is. + +Forty-three red PRs sounds like forty-three problems. It was really two — a lint error and a broken deploy step, both already fixed — wearing forty-three costumes. The costume was the stale branch. `@dependabot rebase` took the costumes off, and then, for the first time, the handful of PRs still standing there in red were telling me the truth. + +I did this triage by hand this time. The next post is about what happened when I got tired of doing org-wide chores by hand and [built a reconciler to do them on a schedule](/posts/gh-org-guard-self-healing-governance/) — because "a human notices the board is red and investigates" is not a control you can rely on at 43 PRs, let alone 430. diff --git a/content/posts/beg-bounty-and-the-token-that-wasnt-dead.md b/content/posts/beg-bounty-and-the-token-that-wasnt-dead.md new file mode 100644 index 0000000..ccd5575 --- /dev/null +++ b/content/posts/beg-bounty-and-the-token-that-wasnt-dead.md @@ -0,0 +1,71 @@ +--- +title: "A stranger emailed me about a leaked token. The scary part was the second one." +date: 2026-07-25 +description: "An unsolicited 'security alert' about a secret in one of my public repos turned out to be right — but chasing it down taught me two things worth more than the fix: 'revoked' in a dashboard doesn't mean a key is dead, and the leak I already knew about was hiding a second one I didn't." +slug: "beg-bounty-and-the-token-that-wasnt-dead" +--- + +I got an email from someone I didn't know, flagging a hardcoded API token in one of my public GitHub repos. It had the shape of the genre: a free-mail sender, a display name dressed up to look like an automated scanner, a helpful offer to tell me more. This is a whole cottage industry — scrape public repos for anything token-shaped, mass-mail the owners, hope a few panic and pay. Most of it is noise. + +The problem is that "mostly noise" is not "always wrong." This one was right. There *was* a live token sitting in a public repo. And the interesting part of this story isn't the leak — it's the two things I got wrong while cleaning it up. + +## Wrong assumption #1: I thought it was already dead + +The token was a [Telegram](https://telegram.org/) bot credential for one of my little side bots. When I first went to triage it, I looked at the repo's secret-scanning alert and saw it had been marked **resolved — "revoked"** months earlier. Case closed, I thought. Old news. Some past version of me had already handled this. + +Then I actually tested the key. + +```bash +# The only honest way to know if a token is dead: ask the API that owns it. +# 200 + bot JSON => STILL LIVE. 401 Unauthorized => actually dead. +curl -s "https://api.telegram.org/bot/getMe" +# {"ok":true,"result":{"id":...,"is_bot":true,"username":"..."}} <- live. +``` + +It answered. `ok: true`. The token I'd been told, by my own dashboard, was "revoked" — was fully, cheerfully alive, and had been for months. + +> Marking an alert "revoked" in a UI is a note to yourself. It changes a label in a database. It does **not** reach out and invalidate the secret. The only thing that kills a credential is rotating it *at the source that issued it* — here, [BotFather](https://core.telegram.org/bots#botfather). I had confused "I filed the paperwork" with "the threat is gone." + +This is the same disease as [the status page that pinged the wrong host](/posts/status-page-that-survives-home/): a check that reports success for a reason unrelated to the thing you care about. There, a ping answered by Cloudflare instead of my origin. Here, a dashboard status that reflects a human clicking a dropdown, not the actual state of the key. Both are green. Both are lying. + +So I did it in the correct order, which I'll never do out of order again: + +1. **Rotate at the source first.** Revoke the token in BotFather; get the new one. +2. **Confirm the old one is dead.** `getMe` on the old token now returns `401`. *Now* it's actually revoked. +3. **Only then** mark the alert resolved — because now the label is finally true. + +## Wrong assumption #2: this was a one-off + +Having been humbled once, I stopped trusting my sense of "I'd know if there were others." I ran a sweep across every repo in my [GitHub org](https://github.com/Aswincloud) — not just "is there an open alert," but "is secret scanning even *turned on* here, and are there live keys nobody's looking at." + +That last question is the one that mattered, because it exposed the trap: **secret scanning only alerts on repos where it's enabled.** A repo with scanning switched off isn't reported as risky. It's reported as *nothing at all*. Silence. And silence reads exactly like safety. + +The sweep found a second live Telegram token — a different bot, a different repo — one where scanning had never been enabled. GitHub had never warned me about it because I'd never asked it to look. If I'd only chased the single token in the stranger's email and stopped, I'd have "fixed the problem" and left an identical one live three repos over. + +```bash +# The sweep's real job wasn't "list open alerts" (that only covers repos already +# being watched). It was "which public repos have scanning OFF" — because those +# are the ones that can't generate an alert no matter what's committed to them. +gh api "orgs//repos?type=public&per_page=100" --paginate \ + --jq '.[] | select(.archived==false) + | {name, scanning: .security_and_analysis.secret_scanning.status}' +# ...any "status":"disabled" line is a blind spot, not a clean bill of health. +``` + +Second token: rotate at source, confirm `401`, mark resolved, turn scanning **on** for that repo so the next mistake actually gets caught. + +## What I actually changed (so it's not just a story) + +Fixing two keys is not a fix. Fixing the *conditions that let two keys sit live* is. So: + +- **Secret scanning + push protection on, org-wide** — including a default for new repos, so "I forgot to enable it" stops being a category of failure. Push protection is the good kind of annoying: it blocks the commit that introduces a secret, at push time, before it's ever public. +- **A weekly automated sweep** that re-asks the boring questions — is scanning on everywhere, is push protection on, are there open alerts — and files an issue if the answer drifts. I wrote about the reconcile-don't-configure philosophy behind this in [the org-guard post](/posts/gh-org-guard-self-healing-governance/); secret hygiene is just one more thing that has to be *re-checked on a schedule*, not set once and trusted forever. +- **A written triage runbook**: rotate-at-source → confirm dead → mark resolved, in that order, every time. The order is the whole point. + +## The part I want to remember + +The stranger's email was, functionally, spam. The genre it belongs to preys on exactly the panic reflex that makes people pay up, and the right response to "pay me to learn more" is to not. But treating the *message* as spam and treating the *claim* as false are two different moves, and only the first one was safe. + +The email was worth precisely one thing: a nudge to go look. Everything valuable after that came from me looking properly — testing the key instead of trusting the label, and sweeping for the blind spots instead of fixing the one thing I'd been handed. The token in the email was never the real risk. The real risk was the one I was sure I didn't have. + +Everything here is redacted — the tokens are long dead, rotated at the source, confirmed with a `401`. Which is the only version of "revoked" I trust now. diff --git a/content/posts/gh-org-guard-self-healing-governance.md b/content/posts/gh-org-guard-self-healing-governance.md new file mode 100644 index 0000000..fdc147b --- /dev/null +++ b/content/posts/gh-org-guard-self-healing-governance.md @@ -0,0 +1,97 @@ +--- +title: "Governing a one-person GitHub org without locking myself out" +date: 2026-07-27 +description: "I wanted every public repo in my org to enforce branch protection and required reviews. The catch: require one approval on a solo org and the owner's own PRs can never be approved — the repo bricks itself. The fix is a governance system that's careful about the exact order it does things." +slug: "gh-org-guard-self-healing-governance" +--- + +I run a GitHub org with something north of twenty public repos and exactly one human in it: me. I wanted the boring, grown-up guarantees on all of them — no force-pushing `main`, no deleting branches, pull requests required, reviews required, secret scanning on. The kind of thing an org admin sets up once and forgets. + +The problem is that "sets up once" is precisely the wrong mental model, and the moment I tried to apply "require one approving review" org-wide, I walked straight into a trap that nearly bricked a repo. That trap, and the way out of it, is the whole reason [gh-org-guard](https://github.com/Aswincloud/gh-org-guard) exists. + +## The solo-owner deadlock + +Here's the trap, and it's a good one because every individual piece is a best practice. + +Require at least one approving review before a PR can merge: sensible. Don't let people approve their own pull requests: obviously sensible — it's the entire point of review. Now put those two rules on a repo where you are the only person with access. + +You open a PR. It needs one approval. The only person who can approve is you. You are not allowed to approve your own PR. **The PR can never merge.** The repo is now read-only to its own owner, and the settings that did it are the settings you'd get by clicking every checkbox a security guide told you to click. + +> Any system powerful enough to lock a branch is powerful enough to lock *you* out of it. If there's no deliberate escape hatch, you didn't build governance — you built a very compliant brick. + +I hit this, backed it out, and realized the fix couldn't be "remember not to turn that rule on." It had to be a system that understands the deadlock and refuses to create it. + +## Two baselines: FLOOR and REVIEW + +The design that came out of it is two baselines and a strict rule about which one a repo is allowed to be on. + +**FLOOR** is the safe-on-anything baseline: block force-push and branch deletion, require a PR, require conversation resolution — but **zero required approvals**. A solo owner can always merge their own work. FLOOR can never deadlock, so it's safe to apply blindly to every repo in the org. + +**REVIEW** is FLOOR *plus* the grown-up part: one required approval and a code-owner review. This is the baseline that would have bricked me — *unless* something can supply that approval without me clicking "approve" on my own PR. + +So the rule is: a repo is only allowed onto REVIEW once it has both a `CODEOWNERS` file **and** a working auto-approver. If either is missing, it stays on FLOOR. And critically, the reconciler will drop a REVIEW repo *back down* to FLOOR if it ever needs to rewrite one of those files — never leaving a gap where the rule is on but the escape hatch is off. + +``` +Does the repo have BOTH a CODEOWNERS and a working auto-approver? + │ │ + no yes + ▼ ▼ + FLOOR REVIEW + 0 approvals, 1 approval + code-owner, + owner can always merge owner's PR gets auto-approved +``` + +Plus a break-glass rule I consider non-negotiable: org admins can always bypass. If some required status check gets misconfigured and starts failing forever, I am not locked out of my own repos while I fix it. The escape hatch is load-bearing. + +## Why a bot can't just approve the PR + +"Fine," you say, "just have a bot approve your PRs." I tried. GitHub has a rule waiting for you here too: **an approval from an app or bot does not count toward a ruleset's required-approval count.** Only a review from a real user with write access counts. A bot can click approve all day; the PR stays blocked. + +So the auto-approver uses **two identities on purpose**: + +- A **GitHub App token** to *read* team membership — is the author of this PR actually an admin who should get an auto-approval? +- A **real write-collaborator account's token** to *cast* the approval — because only a real user's review satisfies the requirement. + +```yaml +# The membership check runs under the App token (it only needs to read teams). +# The approval itself runs under WRITE_ACCESS_PAT — a real user with write +# access — because GitHub silently ignores a bot's approval toward required +# reviews. Two identities, two jobs. It ONLY ever approves; it never merges. +- name: Approve the PR + if: steps.member.outputs.is_admin == 'true' + env: + GH_TOKEN: ${{ secrets.WRITE_ACCESS_PAT }} # a user, not the app + run: | + gh pr review "${PR}" --repo "${REPO}" --approve \ + --body "Auto-approved: @${AUTHOR} is a member of @${ORG}/${TEAM}." +``` + +Three guardrails on that approver, because an auto-approver is a scary thing to have lying around: + +1. **It only approves — it never merges.** I still click "Merge when ready" myself. No auto-merge, ever. The system decides a PR is *allowed* to merge; a human decides *when*. +2. **It won't self-approve.** If the token's user happens to be the PR author, GitHub blocks it anyway, so it detects that case and leaves a comment instead of silently doing nothing. +3. **It approves once per PR, not once per push** — no timeline spam. + +## Reconcile, don't configure + +The deeper idea — the one this whole project is really about — is in that word from the deadlock section: *once*. Settings configured once **drift**. A new repo starts unprotected. Someone toggles a setting to debug something and forgets to toggle it back. Secret scanning gets left off on the one repo that ends up holding a live token (which is [exactly how I got here](/posts/beg-bounty-and-the-token-that-wasnt-dead/)). + +gh-org-guard doesn't configure. It **reconciles**: a scheduled job that, every week, re-asserts the desired state across every public repo. Already correct? No-op. Drifted? Corrected. It's the same discipline I'd just been applying by hand to [a wall of stale Dependabot PRs](/posts/43-red-dependabot-prs-none-were-the-dependency/) — except a schedule doesn't get tired, doesn't skip the boring repo, and doesn't forget that "I set this up in March" is not the same as "this is true today." + +Two things make running it safe rather than terrifying: + +- **Dry-run by default.** A scheduled run enforces; a manual run only *prints what it would do* unless I explicitly flip a flag. A misclick can't rewrite the whole org. +- **Idempotent everything.** Every step is safe to repeat. "Already in the desired state" is a no-op, not an error — which is what lets it run unattended forever. + +``` +### org reconcile — DRY-RUN (no changes) +| repo | action | note | +|-------------|---------|----------------------------------------| +| blog | in-sync | | +| new-service | FLOOR | would add CODEOWNERS; would add caller | +| api | REVIEW | raised to require-review | + +review=1 floor=1 in-sync=1 skip=4 blocked=0 +``` + +I eventually packaged it as a composite GitHub Action so it's a few lines of YAML to drop into another org, defaulting to dry-run so nobody's first run is a surprise. But the Action is just packaging. The idea is the thing worth stealing: **posture is a state you continuously reconcile toward, not a checkbox you tick.** A rule you set once and trust forever is just a slower way of finding out it drifted — and if that rule is powerful enough to lock a branch, make very sure it can never lock out the one person holding the keys.