Skip to content

Wire Bellwether Ingest Action - #237

Open
ghost wants to merge 2 commits into
mainfrom
bellwether/setup-ingest
Open

Wire Bellwether Ingest Action#237
ghost wants to merge 2 commits into
mainfrom
bellwether/setup-ingest

Conversation

@ghost

@ghost ghost commented May 24, 2026

Copy link
Copy Markdown

Bellwether — Test Failure Ingest

This PR wires the
Bellwether Ingest Action
into your CI so failed test runs flow into the Bellwether triage inbox with
full per-test detail (stack traces, assertion messages, test names).

⚠️ Required: add the BELLWETHER_TOKEN repo secret BEFORE merging

The workflow reads ${{ secrets.BELLWETHER_TOKEN }} to authenticate
uploads to the Bellwether API. Without it, the action runs but every
upload fails with a 401.

  1. Mint a token from the Bellwether ingest-action page:
    https://bellwether.run/onboarding/ingest-action
  2. In this repo: Settings → Secrets and variables → Actions →
    New repository secret
    with name BELLWETHER_TOKEN.
  3. Then merge this PR.

What this changes

  • Adds one new file: .github/workflows/bellwether-ingest.yml
  • Does NOT modify any of your existing workflows
  • Listens via workflow_run: completed for failed runs across all workflows
  • Forwards artifact data only when a run fails or is cancelled

Why

Without this: test failures land in the Bellwether inbox with no
per-test detail, the classifier can't tell regressions from flakes, and
every failure gets stuck at "Needs Review."

With this: classifier sees JUnit / Playwright JSON, predictions land
above the confidence threshold, the inbox surfaces the actual triage
signal.

Safe to merge as-is once the secret is set. Customize the branches
filter, the if: predicate, or the action version any time — Bellwether
reads from whichever workflow file is named bellwether-ingest.yml.

Manage your install at https://bellwether.run.


🤖 Created automatically by Bellwether on repo connect (BEL-90). Closing
this PR is fine — Bellwether will fall back to the per-failure
"Set up Ingest Action" CTA.

@ghost
ghost requested a review from jcast90 as a code owner May 24, 2026 17:28
…kflows

Two hardening changes before this goes near main.

Pin `bellwether-run/ingest-action@v1` to its current commit SHA
(92000d98..., v1.0.0). This job runs on `workflow_run`, which executes in the
base repo's context with access to repo secrets — that is what lets it read
runs triggered from fork PRs, and it means whatever the step resolves to can
see BELLWETHER_TOKEN. `@v1` is a mutable tag the upstream owner can repoint at
any commit at any time; a SHA cannot be repointed. Bump it deliberately.

Scope `workflows:` to ["CI", "Integration"] instead of "*". Release and
Changesets don't run tests, so there is nothing in them for Bellwether to
classify and no reason to hand the action their run context.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jcast90

jcast90 commented Jul 11, 2026

Copy link
Copy Markdown
Owner

@jcast90 confirmed he signed up for Bellwether, so keeping this — with two hardening changes pushed to the branch.

1. Pinned the action to a commit SHA. This job runs on workflow_run, which executes in the base repo's context with access to repo secrets — that is precisely what lets it read runs triggered from fork PRs (we have open ones from chrisbremmer and SHRIKEN117). So whatever bellwether-run/ingest-action@v1 resolves to at run time can read BELLWETHER_TOKEN. @v1 is a mutable tag: the upstream owner can repoint it at any commit, at any time, and this workflow would silently pick it up. Now pinned to 92000d98c24f826bbb02507ec13872a0453d75ef (v1.0.0). Bump it deliberately.

2. Scoped workflows: from "*" to ["CI", "Integration"]. Release and Changesets don't run tests — nothing for Bellwether to classify in them, and no reason to hand the action their run context.

⚠️ Blocking: BELLWETHER_TOKEN is not set on this repo. I checked — it does not exist. Per your own PR body, every upload 401s without it, so this must land before merge and only you can add it:

Settings → Secrets and variables → Actions → New repository secret

Ping me once it's in and I'll merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant