diff --git a/AGENTS.md b/AGENTS.md index 0c73c0d..286f887 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ jev-samples/ fetch_prs.py builds the dataset from the GitHub API, needs no Jev key pr_triage.py the Jev call, the tiers, the tables data/ triage reports, committed; the datasets they scored, gitignored - assets/ the pipeline diagram: HTML source, rendered PNG, renderer + assets/ the flow diagram: HTML source, rendered PNG, renderer go/ both halves in one static binary, github.com and GHES vend/ a SKILL.md and an installer, so any repo can use it ``` diff --git a/samples/pr-triage/README.md b/samples/pr-triage/README.md index cedeab5..54b1ee9 100644 --- a/samples/pr-triage/README.md +++ b/samples/pr-triage/README.md @@ -14,13 +14,13 @@ Before any of that, a pull request has to be worth reading. One call to the chec | `ci-pending` | a check is still running | nobody, come back later | | `draft` | the author marked it draft | the author, who is not asking | -`ci-failing` is a fact rather than a verdict. On the eighteen airflow pull requests below, eight were failing and **five of those failed only on a check that also fails on unrelated pull requests**, including a boto3 version bump that cannot break Postgres serialization. That is the check being broken, not each change breaking it, and the output says so. Detecting it costs nothing extra, because the names are already in the dataset. +`ci-failing` states a fact and never judges the author. A check name that fails on several unrelated pull requests is the check being broken, and the output marks those failures as the checks' own. -Skipping those states took ten of eighteen out of the queue before a single question was asked. +Those three states took ten of eighteen out of the queue before Jev saw a single one. The pair picks a route, and the route answers one question: **what would be enough to merge this change?** Enough for a green tick to settle it, or enough that it wants a test, an AI review, or a person. -None of them says merge it now, and none of them merges anything. TypeSafe reports 67.8% accuracy on their own benchmark, published September 2026, which is enough to decide how much evidence to demand and nowhere near enough to be the last gate before main. +The tool merges nothing and tells nobody to merge now. TypeSafe reports 67.8% accuracy on their own benchmark, published September 2026, which is enough to decide how much evidence to demand and nowhere near enough to be the last gate before main. | Route | Enough to merge on | | --- | --- | @@ -32,13 +32,25 @@ None of them says merge it now, and none of them merges anything. TypeSafe repor Consequence sets the floor and effort can only raise it. Of those eighteen, 8 were reviewable and cost about two tenths of a cent to route. +## What each stage costs + +Of the eighteen pull requests below, a frontier review is the deciding evidence on one. Ten never reach a model, removed by rules about drafts and check runs. Jev reads the remaining eight, at 3,039 input tokens and 164 ms each. + +Pre-triage is `if` statements over the checks API. No model can do that stage, because no model knows whether CI passed. + +Routing the eight is a closed question: eleven judgements, two numbers, one route. A frontier model can answer it, reading the same diff and returning the same eleven answers at its own price. Routing one pull request costs about 3,000 input tokens whoever does it, which is $0.00013 at TypeSafe's $0.042 per million, published September 2026. Divide your model's input price by 0.042 for the multiple. + +Reading code for defects needs a model that can read code, which on this queue is one pull request in eighteen. Routing one takes 164 ms, so 710 of them took two minutes. + +The saving holds while the cheap routes are right. A pull request sent to `green-is-enough` that needed reading was false economy whatever it cost, so the number to measure is precision on that route. + ## What it asks Eleven questions go to Jev in one call: how far the change reaches, whether it touches auth or secrets, how many judgment calls a reviewer has to agree with, and eight more. Nine carry a weight in [questions.yml](questions.yml). Four of those nine also feed the consequence axis, marked below. - + -That diagram is built from [assets/pipeline.html](assets/pipeline.html). Edit the HTML and run `python3 assets/render.py` to rebuild the PNG. +[assets/flow.html](assets/flow.html) is the source. Edit the HTML and run `python3 assets/render.py` to rebuild the PNG. | Question | Type | Weight | What a high answer means | | --- | --- | --- | --- | @@ -54,7 +66,7 @@ That diagram is built from [assets/pipeline.html](assets/pipeline.html). Edit th | `has_tests` | Noul (inverted) | 0.08 | the change comes with tests that exercise it | | `description_quality` | Score (inverted) | 0.08 | the author explained what, why and how they checked | -`questions.yml` inverts three of them, so a mechanical diff, a covered change and a thorough description each lower the effort. The four marked **(c)** also feed consequence, where the *max* of them is taken rather than the mean: `security_surface` at 0.98 is not offset by `infra_surface` at 0.04. A score feeding consequence reads the probability on its *top* level rather than its normalised score, because `blast_radius` level 1 is "confined to one module", which is ordinary work rather than half a catastrophe. +`questions.yml` inverts three of them, so a mechanical diff, a covered change and a thorough description each lower the effort. The four marked **(c)** also feed consequence, which takes the *max* of them rather than the mean: `security_surface` at 0.98 survives an `infra_surface` of 0.04, where a mean would drag it to the middle. A score feeding consequence reads the probability on its *top* level rather than its normalised score, because `blast_radius` level 1 is "confined to one module", which is ordinary work rather than half a catastrophe. ## Running it @@ -77,7 +89,7 @@ uv run pr_triage.py --dataset data/owner-repo-open-all.json uv run pr_triage.py --dataset data/owner-repo-open-all.json --explain 1693 ``` -`fetch_prs.py` needs a GitHub token and no Jev key. It reads `GITHUB_TOKEN` or `GH_TOKEN`, and falls back to `gh auth token`. Unauthenticated callers get sixty requests an hour and one pull request costs two of them, so twenty-six pull requests want a token. +`fetch_prs.py` needs a GitHub token and no Jev key. It reads `GITHUB_TOKEN` or `GH_TOKEN`, and falls back to `gh auth token`. Unauthenticated callers get sixty requests an hour, and one pull request costs three of them: the listing, its files, and its check runs. Any real queue wants a token. `pr_triage.py` needs a Jev key in `TYPESAFE_API_KEY`, from the environment or from `.env` beside the sample or at the repo root. @@ -85,7 +97,7 @@ uv run pr_triage.py --dataset data/owner-repo-open-all.json --explain 1693 [`go/`](go/) holds a Go port that compiles both halves into one static binary, which is what the skill installs and what a CI runner wants. It works against github.com and a GitHub Enterprise Server host. -The Python stays canonical. `go/questions.yml` is a copy, `build.sh` refreshes it before every build, and `payload_test.go` fails when the two drift, so the two tools cannot score one pull request differently in silence. +The Python stays canonical. `go/questions.yml` is a copy, `build.sh` refreshes it before every build, and `payload_test.go` fails when the two drift, so the two tools cannot disagree about one pull request in silence. ## Install it as a Claude Code skill @@ -120,7 +132,7 @@ Then start a new Claude Code session and ask for a triage in your own words: The skill picks the flags, runs the binary, reads the report and tells you which pull requests need an hour and which clear on a glance. It writes a JSON report and a markdown one you can paste into an issue. -For CI rather than a conversation, [`go/README.md`](go/README.md) covers the binary on its own, including `-fail-on-tier` for failing a job and the three ways to point it at a GitHub Enterprise Server host. +For CI rather than a conversation, [`go/README.md`](go/README.md) covers the binary on its own, including `-fail-on-route` for failing a job and the three ways to point it at a GitHub Enterprise Server host. ## What it prints @@ -184,7 +196,7 @@ The eight that were reviewable: | #73722 | 1 | +2/-1 | docs | 0.20 | 0.04 | green-is-enough | Clarify max_db_retries doc wording to avoid off-by-one ... | ``` -`Load` is effort and `Cons` is consequence. A load marked `(size)` was raised by the file and line counts. Either number marked `(on a cut)` sits within 0.02 of a threshold, which is about how far repeat calls move it, so read it as either side. +`Load` is effort and `Cons` is consequence. A load marked `(size)` means the file and line counts raised it. Either number marked `(on a cut)` sits within 0.02 of a threshold, which is about how far repeat calls move it, so read it as either side. Then the same pull requests grouped by route, cheapest evidence last: @@ -221,34 +233,36 @@ Every run writes two reports into [data/](data/). The JSON one holds each answer ### One pull request, from eleven answers to one route -Airflow #73698 binds the AWS auth manager's SAML response to the browser that started the login: 8 files, +626/-73, and Jev read 88% of the diff. The nine weighted answers, with the four that also feed consequence marked **(c)**: +Airflow #73741 improves the error a DAG tag length check raises. Two files, +11/-3, and Jev read all of the diff. The nine weighted answers, heaviest contribution first, with the four that also feed consequence marked **(c)**: | Question | Jev returned | Weight | Credit | Adds to effort | Feeds consequence | | --- | --- | --- | --- | --- | --- | -| `security_surface` **(c)** | 0.99 | 0.15 | 0.99 | 0.1485 | **0.99** | -| `design_decisions` | 2.00 / 2 | 0.14 | 1.00 | 0.1400 | | -| `mechanical` (inverted) | 0.29 | 0.12 | 0.71 | 0.0852 | | -| `blast_radius` **(c)** | 1.02 / 2 | 0.15 | 0.51 | 0.0765 | 0.02 | -| `breaking_change` **(c)** | 0.69 | 0.11 | 0.69 | 0.0759 | 0.69 | -| `infra_surface` **(c)** | 0.70 | 0.08 | 0.70 | 0.0560 | 0.70 | -| `scope_creep` | 0.48 | 0.09 | 0.48 | 0.0432 | | -| `has_tests` (inverted) | 0.95 | 0.08 | 0.05 | 0.0040 | | -| `description_quality` (inverted) | 1.98 / 2 | 0.08 | 0.01 | 0.0008 | | -| | | **1.00** | | **0.6301** | **max 0.99** | +| `security_surface` **(c)** | 0.97 | 0.15 | 0.97 | 0.1455 | **0.97** | +| `blast_radius` **(c)** | 1.21 / 2 | 0.15 | 0.60 | 0.0907 | 0.22 | +| `mechanical` (inverted) | 0.39 | 0.12 | 0.61 | 0.0732 | | +| `design_decisions` | 0.98 / 2 | 0.14 | 0.49 | 0.0686 | | +| `breaking_change` **(c)** | 0.07 | 0.11 | 0.07 | 0.0077 | 0.07 | +| `scope_creep` | 0.08 | 0.09 | 0.08 | 0.0072 | | +| `has_tests` (inverted) | 0.97 | 0.08 | 0.03 | 0.0024 | | +| `infra_surface` **(c)** | 0.02 | 0.08 | 0.02 | 0.0016 | 0.02 | +| `description_quality` (inverted) | 2.00 / 2 | 0.08 | 0.00 | 0.0000 | | +| | | **1.00** | | **0.3969** | **max 0.97** | + +Effort is that column added up: 0.3969, a short read of two files. The report records 0.3970, because it rounds the total rather than adding up nine rounded rows. Consequence is the max of the four marked entries: 0.97, set by `security_surface`. Effort clears no raise, consequence clears its top cut, so the route is `human-required` and the consequence axis decided it alone. -Effort is that column added up: 0.6301, a long read. Consequence is the max of the four marked entries: 0.99, set by `security_surface`. Both clear their top cut, so the route is `human-plus-author`, decided by both axes. +The title reads like an error-message tweak, and the diff is eleven added lines. It still needs a person, because a length check on user input is input validation, and `security_surface` came back 0.97. That is the case a single blended score buries: averaging the nine puts this at 0.40, in the middle of the queue, where nobody would look twice. -Two rows show why the axes need different arithmetic. `has_tests` came back 0.95, so the change ships tests, and inverted that contributes almost nothing to effort: a tested change is quicker to review. It does not touch consequence at all, because tests are evidence against regression and not against an authentication mistake. +Three rows show the axes reading one answer two ways. `blast_radius` scored 1.21 out of 2, so it contributes 0.60 of credit to effort, a fair chunk of a short read. Its probability on the top level, "reaches shared code paths that many callers depend on", is 0.22, so it contributes 0.22 to consequence and stops there. `has_tests` came back 0.97, and inverted that adds 0.0024 to effort, because a tested change is quicker to read. It feeds consequence not at all, since tests are evidence against regression and not against a validation mistake. `description_quality` at 2.00 out of 2 contributes 0.0000, which is what a thorough description earns. -`blast_radius` is the sharper case. It scored 1.02 out of 2, which is 0.51 of credit toward effort, a middling read. But its probability on the *top* level, "reaches shared code paths that many callers depend on", is only 0.02: Jev is confident this stays inside one module. So it contributes 0.51 to effort and 0.02 to consequence, from one answer. Reading the normalised score into consequence would have called a contained change half a catastrophe, which is the bug that made every airflow pull request look dangerous in an earlier run. +The run also prints the one thing that would move it: a test exercising that security surface, or a reviewer who owns the path. ## What to notice -**Most of a queue is not waiting on review at all.** Of those eighteen, ten never reached Jev: one draft, eight with a failing check, one still running. The eight that were reviewable came out 2 `green-is-enough`, 3 `tests-are-enough`, 1 `ai-review-is-enough`, 2 `human-required`. The cheapest pull requests in a queue are usually the ones nobody needs to read, and the loudest are usually waiting on the build rather than on a person. +**Ten of eighteen were waiting on a build or their author.** One draft, eight with a failing check, one still running, and none of them reached Jev. The eight that were reviewable came out 2 `green-is-enough`, 3 `tests-are-enough`, 1 `ai-review-is-enough`, 2 `human-required`. A queue hides two cheap groups: the changes nobody needs to read, and the ones stuck behind a build. -**The spread is a fact about the repository, not about the thresholds.** The same cuts over 710 airflow pull requests opened in the last 100 days took 342 of them, 48%, off the human queue. Run against a repository where every change touches authentication or deployment credentials, they took none of 24, because that queue contains no low-consequence work to find. Airflow's cheap tail is its 48 docs pull requests, 20 dependency bumps and 22 test-only changes. A repository without those has no cheap tail, and a router that invented one would be wrong. +**One set of cuts produces two different distributions.** The same cuts over 710 airflow pull requests opened in the last 100 days took 342 of them, 48%, off the human queue. Run against a repository where every change touches authentication or deployment credentials, they took none of 24, because that queue contains no low-consequence work to find. Airflow's cheap tail is its 48 docs pull requests, 20 dependency bumps and 22 test-only changes. A repository without those has no cheap tail, and a router that invented one would be wrong. -**Nine files can outrank fifty-four.** #73704 changes 89 lines across 9 files and routes `human-required` at consequence 0.84, because it fixes socket leaks and adds request timeouts across providers and `security_surface` came back 0.84. #73713 changes 598 lines across 54 files and its effort is only 0.37, because `mechanical` came back 0.81 on one locale-formatting edit repeated through the UI. Size is not review cost, and neither is line count. +**Nine files can outrank fifty-four.** #73704 changes 89 lines across 9 files and routes `human-required` at consequence 0.84, because it fixes socket leaks and adds request timeouts across providers and `security_surface` came back 0.84. #73713 changes 598 lines across 54 files and its effort is only 0.37, because `mechanical` came back 0.81 on one locale-formatting edit repeated through the UI. The eleven questions sort by what a review has to catch, and the driver line names the one that did it. **Arithmetic stays in code.** Jev cannot count, so the file and line totals reach it as a sentence for context, and every threshold on a number lives in [pr_triage.py](pr_triage.py) and its Go twin. `SIZE_FLOORS` names the lowest effort tier a change of a given size can land in: over 30 files or 1,500 lines is high whatever Jev returned. `CONSEQUENCE_FLOORS` and `EFFORT_RAISES` turn the two axes into a route. Every one of them only ever raises, and the output marks the rows where it did, so a reader sees the disagreement instead of inheriting it. @@ -262,6 +276,6 @@ Two rows show why the axes need different arithmetic. `has_tests` came back 0.95 ## Calibrate before you trust a route -TypeSafe reports 67.8% accuracy on their own benchmark, published September 2026. A router can live with that, because its errors are bounded: send something to a human who did not need to look and you waste twenty minutes, while CI, the tests and an AI review all still stand behind a cheaper route. A merge decision cannot live with it, because nothing stands behind that. +TypeSafe reports 67.8% accuracy on their own benchmark, published September 2026. A router survives that, because a routing mistake costs little: send something to a human who did not need to look and you waste twenty minutes, while CI, the tests and an AI review all still stand behind a cheaper route. A merge decision cannot live with it, because nothing stands behind that. -So the number to measure is not accuracy across the queue. It is **precision on `green-is-enough`**, where a false "this is safe" is the only expensive mistake the tool can make. Replay fifty pull requests your team already merged, count how many the sample would have sent to `green-is-enough`, and check how many of those were later reverted or hot-fixed. Move `CONSEQUENCE_FLOORS` until that count is zero. [jevcal](https://github.com/abhixhek/jevcal) turns that comparison into a measurement. +The number to measure is **precision on `green-is-enough`**, where a false "this is safe" is the only expensive mistake the tool can make. Replay fifty pull requests your team already merged, count how many the sample would have sent to `green-is-enough`, and check how many of those were later reverted or hot-fixed. Move `CONSEQUENCE_FLOORS` until that count is zero. [jevcal](https://github.com/abhixhek/jevcal) turns that comparison into a measurement. diff --git a/samples/pr-triage/assets/flow.html b/samples/pr-triage/assets/flow.html new file mode 100644 index 0000000..12ea858 --- /dev/null +++ b/samples/pr-triage/assets/flow.html @@ -0,0 +1,192 @@ + + + +
+ + + + + +| Route | What the repo owner does |
|---|---|
| green-is-enough | merge it on a green tick, nobody needs to read it |
| tests-are-enough | merge it once a test covers the change |
| ai-review-is-enough | run an AI review; if it finds nothing, merge it |
| human-required | put it in front of a person before it merges |
| human-plus-author | a person reads it with the author walking them through |
Jev answers eleven questions in one call. Code does every piece of arithmetic, and the output is a route rather than a merge.
- -Title, description, the file list, and as much of the diff as the budget holds. A patch - too large to send whole arrives as whole files first, then paths and line counts.
-One call to the checks API per pull request, before any Jev call. Three states are - terminal: no questions asked, no route computed, nothing spent. On eighteen - apache/airflow pull requests this skipped ten of them.
-| State | Means | Waiting on |
|---|---|---|
| ci-failing | a check concluded failure, timed out, or wants action | the checks, or the branch |
| ci-pending | a check is still running | nobody, come back later |
| draft | the author marked it draft | the author, who is not asking |
A name that fails on several unrelated pull requests is the check being - broken rather than each change breaking it: a boto3 bump cannot break Postgres - serialization. Five of eight failures were shared that way, and the output says so, - because the state is a fact and never a verdict on the author.
-Every question sees the same pull request. None sees another's answer, so the eleven - cost the tokens of eleven and the wall clock of about one.
-Nine scored, and weighted
-blast radius how far it reaches · security surface auth, secrets, - tokens, input validation · breaking change a caller stops working · - infra surface deploy, images, CI · design decisions judgment calls to - agree with · mechanical one edit repeated · scope creep the diff - exceeds the description · has tests the change is covered · - description quality what, why, and how they checked
-Two labels, never scored
-What kind of change: docs, tests, dependency, config, bugfix, feature or
- refactor.
- What to check first: correctness, security, compatibility, tests, deployment
- or readability.
- Both print beside the route.
Effort and consequence are different questions, and they take different arithmetic.
-Effort: the weighted mean of nine
-All nine, at the weights in questions.yml: blast radius - 0.15, security surface 0.15, design decisions 0.14, mechanical 0.12, breaking - change 0.11, scope creep 0.09, infra surface 0.08, has tests 0.08, description - quality 0.08.
-Averaging is right here, because a diff that is long in one - way and short in three really is a medium read. Mechanical, has tests and - description quality count inverted, so a repeated edit, a covered change and - a thorough description each pull it down.
-Consequence: the max of four
-Four of the same nine, read differently: security surface (auth, secrets, - tokens), breaking change (an existing caller stops working), blast - radius (reaches shared code paths), infra surface (deploy, images, CI).
-The max, never the mean, because a change that is safe - in three ways and dangerous in one is a dangerous change. A score reads the - probability on its top level only: blast radius level 1 is "one module", - which is ordinary work.
-Consequence sets the floor. Effort can raise it and never lower it. Nothing here - says "merge", because a model reported at 67.8% accuracy can decide how much - evidence to demand and cannot be the last gate before main.
-| Route | Enough to merge on | Reached when |
|---|---|---|
| green-is-enough | CI passing | consequence under 0.15 |
| tests-are-enough | CI, and a test that exercises it | consequence 0.15, or effort 0.42 |
| ai-review-is-enough | the above, and an AI review finding nothing | consequence 0.35, or effort 0.62 |
| human-required | a person reads it | consequence 0.60 |
| human-plus-author | a person reads it, author walking through | consequence 0.60 and effort 0.62 |
Jev reads a truncated diff, so a 54-file change can read as one repeated edit. - Size is arithmetic, Python and Go own it, and it only ever raises.
-Grouped by route, with what set the consequence, the questions carrying the effort, - and the single signal that would drop it a route: "a test that exercises the - unconnected-identity path". A tier tells an author nothing to do. That line does.
-