diff --git a/AGENTS.md b/AGENTS.md index 0c73c0d..286f887 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ jev-samples/ fetch_prs.py builds the dataset from the GitHub API, needs no Jev key pr_triage.py the Jev call, the tiers, the tables data/ triage reports, committed; the datasets they scored, gitignored - assets/ the pipeline diagram: HTML source, rendered PNG, renderer + assets/ the flow diagram: HTML source, rendered PNG, renderer go/ both halves in one static binary, github.com and GHES vend/ a SKILL.md and an installer, so any repo can use it ``` diff --git a/samples/pr-triage/README.md b/samples/pr-triage/README.md index cedeab5..54b1ee9 100644 --- a/samples/pr-triage/README.md +++ b/samples/pr-triage/README.md @@ -14,13 +14,13 @@ Before any of that, a pull request has to be worth reading. One call to the chec | `ci-pending` | a check is still running | nobody, come back later | | `draft` | the author marked it draft | the author, who is not asking | -`ci-failing` is a fact rather than a verdict. On the eighteen airflow pull requests below, eight were failing and **five of those failed only on a check that also fails on unrelated pull requests**, including a boto3 version bump that cannot break Postgres serialization. That is the check being broken, not each change breaking it, and the output says so. Detecting it costs nothing extra, because the names are already in the dataset. +`ci-failing` states a fact and never judges the author. A check name that fails on several unrelated pull requests is the check being broken, and the output marks those failures as the checks' own. -Skipping those states took ten of eighteen out of the queue before a single question was asked. +Those three states took ten of eighteen out of the queue before Jev saw a single one. The pair picks a route, and the route answers one question: **what would be enough to merge this change?** Enough for a green tick to settle it, or enough that it wants a test, an AI review, or a person. -None of them says merge it now, and none of them merges anything. TypeSafe reports 67.8% accuracy on their own benchmark, published September 2026, which is enough to decide how much evidence to demand and nowhere near enough to be the last gate before main. +The tool merges nothing and tells nobody to merge now. TypeSafe reports 67.8% accuracy on their own benchmark, published September 2026, which is enough to decide how much evidence to demand and nowhere near enough to be the last gate before main. | Route | Enough to merge on | | --- | --- | @@ -32,13 +32,25 @@ None of them says merge it now, and none of them merges anything. TypeSafe repor Consequence sets the floor and effort can only raise it. Of those eighteen, 8 were reviewable and cost about two tenths of a cent to route. +## What each stage costs + +Of the eighteen pull requests below, a frontier review is the deciding evidence on one. Ten never reach a model, removed by rules about drafts and check runs. Jev reads the remaining eight, at 3,039 input tokens and 164 ms each. + +Pre-triage is `if` statements over the checks API. No model can do that stage, because no model knows whether CI passed. + +Routing the eight is a closed question: eleven judgements, two numbers, one route. A frontier model can answer it, reading the same diff and returning the same eleven answers at its own price. Routing one pull request costs about 3,000 input tokens whoever does it, which is $0.00013 at TypeSafe's $0.042 per million, published September 2026. Divide your model's input price by 0.042 for the multiple. + +Reading code for defects needs a model that can read code, which on this queue is one pull request in eighteen. Routing one takes 164 ms, so 710 of them took two minutes. + +The saving holds while the cheap routes are right. A pull request sent to `green-is-enough` that needed reading was false economy whatever it cost, so the number to measure is precision on that route. + ## What it asks Eleven questions go to Jev in one call: how far the change reaches, whether it touches auth or secrets, how many judgment calls a reviewer has to agree with, and eight more. Nine carry a weight in [questions.yml](questions.yml). Four of those nine also feed the consequence axis, marked below. -![The pr-triage pipeline: read one pull request, stop early on anything not reviewable because CI is failing or pending or it is a draft, ask eleven questions in one call, read the answers on two axes as an effort mean and a consequence max, turn the pair into a route naming what evidence is sufficient, let the file and line counts raise the effort tier, then print the route with the one thing that would change it.](assets/pipeline.png) +![The pr-triage flow: a pull request goes through pre-triage, which is plain rules and no model. Any of draft, a failing check or a running check stops there and reports that state. Otherwise one Jev call asks eleven questions, nine weighted and two labels, the nine become an effort and consequence pair, and the pair picks one route saying what the repo owner should do. Two worked examples end the diagram.](assets/flow.png) -That diagram is built from [assets/pipeline.html](assets/pipeline.html). Edit the HTML and run `python3 assets/render.py` to rebuild the PNG. +[assets/flow.html](assets/flow.html) is the source. Edit the HTML and run `python3 assets/render.py` to rebuild the PNG. | Question | Type | Weight | What a high answer means | | --- | --- | --- | --- | @@ -54,7 +66,7 @@ That diagram is built from [assets/pipeline.html](assets/pipeline.html). Edit th | `has_tests` | Noul (inverted) | 0.08 | the change comes with tests that exercise it | | `description_quality` | Score (inverted) | 0.08 | the author explained what, why and how they checked | -`questions.yml` inverts three of them, so a mechanical diff, a covered change and a thorough description each lower the effort. The four marked **(c)** also feed consequence, where the *max* of them is taken rather than the mean: `security_surface` at 0.98 is not offset by `infra_surface` at 0.04. A score feeding consequence reads the probability on its *top* level rather than its normalised score, because `blast_radius` level 1 is "confined to one module", which is ordinary work rather than half a catastrophe. +`questions.yml` inverts three of them, so a mechanical diff, a covered change and a thorough description each lower the effort. The four marked **(c)** also feed consequence, which takes the *max* of them rather than the mean: `security_surface` at 0.98 survives an `infra_surface` of 0.04, where a mean would drag it to the middle. A score feeding consequence reads the probability on its *top* level rather than its normalised score, because `blast_radius` level 1 is "confined to one module", which is ordinary work rather than half a catastrophe. ## Running it @@ -77,7 +89,7 @@ uv run pr_triage.py --dataset data/owner-repo-open-all.json uv run pr_triage.py --dataset data/owner-repo-open-all.json --explain 1693 ``` -`fetch_prs.py` needs a GitHub token and no Jev key. It reads `GITHUB_TOKEN` or `GH_TOKEN`, and falls back to `gh auth token`. Unauthenticated callers get sixty requests an hour and one pull request costs two of them, so twenty-six pull requests want a token. +`fetch_prs.py` needs a GitHub token and no Jev key. It reads `GITHUB_TOKEN` or `GH_TOKEN`, and falls back to `gh auth token`. Unauthenticated callers get sixty requests an hour, and one pull request costs three of them: the listing, its files, and its check runs. Any real queue wants a token. `pr_triage.py` needs a Jev key in `TYPESAFE_API_KEY`, from the environment or from `.env` beside the sample or at the repo root. @@ -85,7 +97,7 @@ uv run pr_triage.py --dataset data/owner-repo-open-all.json --explain 1693 [`go/`](go/) holds a Go port that compiles both halves into one static binary, which is what the skill installs and what a CI runner wants. It works against github.com and a GitHub Enterprise Server host. -The Python stays canonical. `go/questions.yml` is a copy, `build.sh` refreshes it before every build, and `payload_test.go` fails when the two drift, so the two tools cannot score one pull request differently in silence. +The Python stays canonical. `go/questions.yml` is a copy, `build.sh` refreshes it before every build, and `payload_test.go` fails when the two drift, so the two tools cannot disagree about one pull request in silence. ## Install it as a Claude Code skill @@ -120,7 +132,7 @@ Then start a new Claude Code session and ask for a triage in your own words: The skill picks the flags, runs the binary, reads the report and tells you which pull requests need an hour and which clear on a glance. It writes a JSON report and a markdown one you can paste into an issue. -For CI rather than a conversation, [`go/README.md`](go/README.md) covers the binary on its own, including `-fail-on-tier` for failing a job and the three ways to point it at a GitHub Enterprise Server host. +For CI rather than a conversation, [`go/README.md`](go/README.md) covers the binary on its own, including `-fail-on-route` for failing a job and the three ways to point it at a GitHub Enterprise Server host. ## What it prints @@ -184,7 +196,7 @@ The eight that were reviewable: | #73722 | 1 | +2/-1 | docs | 0.20 | 0.04 | green-is-enough | Clarify max_db_retries doc wording to avoid off-by-one ... | ``` -`Load` is effort and `Cons` is consequence. A load marked `(size)` was raised by the file and line counts. Either number marked `(on a cut)` sits within 0.02 of a threshold, which is about how far repeat calls move it, so read it as either side. +`Load` is effort and `Cons` is consequence. A load marked `(size)` means the file and line counts raised it. Either number marked `(on a cut)` sits within 0.02 of a threshold, which is about how far repeat calls move it, so read it as either side. Then the same pull requests grouped by route, cheapest evidence last: @@ -221,34 +233,36 @@ Every run writes two reports into [data/](data/). The JSON one holds each answer ### One pull request, from eleven answers to one route -Airflow #73698 binds the AWS auth manager's SAML response to the browser that started the login: 8 files, +626/-73, and Jev read 88% of the diff. The nine weighted answers, with the four that also feed consequence marked **(c)**: +Airflow #73741 improves the error a DAG tag length check raises. Two files, +11/-3, and Jev read all of the diff. The nine weighted answers, heaviest contribution first, with the four that also feed consequence marked **(c)**: | Question | Jev returned | Weight | Credit | Adds to effort | Feeds consequence | | --- | --- | --- | --- | --- | --- | -| `security_surface` **(c)** | 0.99 | 0.15 | 0.99 | 0.1485 | **0.99** | -| `design_decisions` | 2.00 / 2 | 0.14 | 1.00 | 0.1400 | | -| `mechanical` (inverted) | 0.29 | 0.12 | 0.71 | 0.0852 | | -| `blast_radius` **(c)** | 1.02 / 2 | 0.15 | 0.51 | 0.0765 | 0.02 | -| `breaking_change` **(c)** | 0.69 | 0.11 | 0.69 | 0.0759 | 0.69 | -| `infra_surface` **(c)** | 0.70 | 0.08 | 0.70 | 0.0560 | 0.70 | -| `scope_creep` | 0.48 | 0.09 | 0.48 | 0.0432 | | -| `has_tests` (inverted) | 0.95 | 0.08 | 0.05 | 0.0040 | | -| `description_quality` (inverted) | 1.98 / 2 | 0.08 | 0.01 | 0.0008 | | -| | | **1.00** | | **0.6301** | **max 0.99** | +| `security_surface` **(c)** | 0.97 | 0.15 | 0.97 | 0.1455 | **0.97** | +| `blast_radius` **(c)** | 1.21 / 2 | 0.15 | 0.60 | 0.0907 | 0.22 | +| `mechanical` (inverted) | 0.39 | 0.12 | 0.61 | 0.0732 | | +| `design_decisions` | 0.98 / 2 | 0.14 | 0.49 | 0.0686 | | +| `breaking_change` **(c)** | 0.07 | 0.11 | 0.07 | 0.0077 | 0.07 | +| `scope_creep` | 0.08 | 0.09 | 0.08 | 0.0072 | | +| `has_tests` (inverted) | 0.97 | 0.08 | 0.03 | 0.0024 | | +| `infra_surface` **(c)** | 0.02 | 0.08 | 0.02 | 0.0016 | 0.02 | +| `description_quality` (inverted) | 2.00 / 2 | 0.08 | 0.00 | 0.0000 | | +| | | **1.00** | | **0.3969** | **max 0.97** | + +Effort is that column added up: 0.3969, a short read of two files. The report records 0.3970, because it rounds the total rather than adding up nine rounded rows. Consequence is the max of the four marked entries: 0.97, set by `security_surface`. Effort clears no raise, consequence clears its top cut, so the route is `human-required` and the consequence axis decided it alone. -Effort is that column added up: 0.6301, a long read. Consequence is the max of the four marked entries: 0.99, set by `security_surface`. Both clear their top cut, so the route is `human-plus-author`, decided by both axes. +The title reads like an error-message tweak, and the diff is eleven added lines. It still needs a person, because a length check on user input is input validation, and `security_surface` came back 0.97. That is the case a single blended score buries: averaging the nine puts this at 0.40, in the middle of the queue, where nobody would look twice. -Two rows show why the axes need different arithmetic. `has_tests` came back 0.95, so the change ships tests, and inverted that contributes almost nothing to effort: a tested change is quicker to review. It does not touch consequence at all, because tests are evidence against regression and not against an authentication mistake. +Three rows show the axes reading one answer two ways. `blast_radius` scored 1.21 out of 2, so it contributes 0.60 of credit to effort, a fair chunk of a short read. Its probability on the top level, "reaches shared code paths that many callers depend on", is 0.22, so it contributes 0.22 to consequence and stops there. `has_tests` came back 0.97, and inverted that adds 0.0024 to effort, because a tested change is quicker to read. It feeds consequence not at all, since tests are evidence against regression and not against a validation mistake. `description_quality` at 2.00 out of 2 contributes 0.0000, which is what a thorough description earns. -`blast_radius` is the sharper case. It scored 1.02 out of 2, which is 0.51 of credit toward effort, a middling read. But its probability on the *top* level, "reaches shared code paths that many callers depend on", is only 0.02: Jev is confident this stays inside one module. So it contributes 0.51 to effort and 0.02 to consequence, from one answer. Reading the normalised score into consequence would have called a contained change half a catastrophe, which is the bug that made every airflow pull request look dangerous in an earlier run. +The run also prints the one thing that would move it: a test exercising that security surface, or a reviewer who owns the path. ## What to notice -**Most of a queue is not waiting on review at all.** Of those eighteen, ten never reached Jev: one draft, eight with a failing check, one still running. The eight that were reviewable came out 2 `green-is-enough`, 3 `tests-are-enough`, 1 `ai-review-is-enough`, 2 `human-required`. The cheapest pull requests in a queue are usually the ones nobody needs to read, and the loudest are usually waiting on the build rather than on a person. +**Ten of eighteen were waiting on a build or their author.** One draft, eight with a failing check, one still running, and none of them reached Jev. The eight that were reviewable came out 2 `green-is-enough`, 3 `tests-are-enough`, 1 `ai-review-is-enough`, 2 `human-required`. A queue hides two cheap groups: the changes nobody needs to read, and the ones stuck behind a build. -**The spread is a fact about the repository, not about the thresholds.** The same cuts over 710 airflow pull requests opened in the last 100 days took 342 of them, 48%, off the human queue. Run against a repository where every change touches authentication or deployment credentials, they took none of 24, because that queue contains no low-consequence work to find. Airflow's cheap tail is its 48 docs pull requests, 20 dependency bumps and 22 test-only changes. A repository without those has no cheap tail, and a router that invented one would be wrong. +**One set of cuts produces two different distributions.** The same cuts over 710 airflow pull requests opened in the last 100 days took 342 of them, 48%, off the human queue. Run against a repository where every change touches authentication or deployment credentials, they took none of 24, because that queue contains no low-consequence work to find. Airflow's cheap tail is its 48 docs pull requests, 20 dependency bumps and 22 test-only changes. A repository without those has no cheap tail, and a router that invented one would be wrong. -**Nine files can outrank fifty-four.** #73704 changes 89 lines across 9 files and routes `human-required` at consequence 0.84, because it fixes socket leaks and adds request timeouts across providers and `security_surface` came back 0.84. #73713 changes 598 lines across 54 files and its effort is only 0.37, because `mechanical` came back 0.81 on one locale-formatting edit repeated through the UI. Size is not review cost, and neither is line count. +**Nine files can outrank fifty-four.** #73704 changes 89 lines across 9 files and routes `human-required` at consequence 0.84, because it fixes socket leaks and adds request timeouts across providers and `security_surface` came back 0.84. #73713 changes 598 lines across 54 files and its effort is only 0.37, because `mechanical` came back 0.81 on one locale-formatting edit repeated through the UI. The eleven questions sort by what a review has to catch, and the driver line names the one that did it. **Arithmetic stays in code.** Jev cannot count, so the file and line totals reach it as a sentence for context, and every threshold on a number lives in [pr_triage.py](pr_triage.py) and its Go twin. `SIZE_FLOORS` names the lowest effort tier a change of a given size can land in: over 30 files or 1,500 lines is high whatever Jev returned. `CONSEQUENCE_FLOORS` and `EFFORT_RAISES` turn the two axes into a route. Every one of them only ever raises, and the output marks the rows where it did, so a reader sees the disagreement instead of inheriting it. @@ -262,6 +276,6 @@ Two rows show why the axes need different arithmetic. `has_tests` came back 0.95 ## Calibrate before you trust a route -TypeSafe reports 67.8% accuracy on their own benchmark, published September 2026. A router can live with that, because its errors are bounded: send something to a human who did not need to look and you waste twenty minutes, while CI, the tests and an AI review all still stand behind a cheaper route. A merge decision cannot live with it, because nothing stands behind that. +TypeSafe reports 67.8% accuracy on their own benchmark, published September 2026. A router survives that, because a routing mistake costs little: send something to a human who did not need to look and you waste twenty minutes, while CI, the tests and an AI review all still stand behind a cheaper route. A merge decision cannot live with it, because nothing stands behind that. -So the number to measure is not accuracy across the queue. It is **precision on `green-is-enough`**, where a false "this is safe" is the only expensive mistake the tool can make. Replay fifty pull requests your team already merged, count how many the sample would have sent to `green-is-enough`, and check how many of those were later reverted or hot-fixed. Move `CONSEQUENCE_FLOORS` until that count is zero. [jevcal](https://github.com/abhixhek/jevcal) turns that comparison into a measurement. +The number to measure is **precision on `green-is-enough`**, where a false "this is safe" is the only expensive mistake the tool can make. Replay fifty pull requests your team already merged, count how many the sample would have sent to `green-is-enough`, and check how many of those were later reverted or hot-fixed. Move `CONSEQUENCE_FLOORS` until that count is zero. [jevcal](https://github.com/abhixhek/jevcal) turns that comparison into a measurement. diff --git a/samples/pr-triage/assets/flow.html b/samples/pr-triage/assets/flow.html new file mode 100644 index 0000000..12ea858 --- /dev/null +++ b/samples/pr-triage/assets/flow.html @@ -0,0 +1,192 @@ + + + + + + + + + +

pr-triage: from a pull request to one instruction

+ +
+
A pull request
+
title, description, file list, diff, and its check runs
+
+ +
+ +
+
Pre-triage: plain rules, no model
+
Is it a draft? Is a check failing? Is a check still running?
+
+ +
+
+
any of them yes
+
+
Stop. Report the state.
+
draft · ci-failing · ci-pending
+ No model call, nothing spent, nothing to review yet.
+
+
+
+
all of them no
+
+
the only model call
+
One Jev call, eleven questions
+
9 weighted feed the numbers:
+ blast radius, security surface, has tests, and six more
+ 2 labels are reported, never scored:
+ what kind of change it is, and what to check first
+
+
+
+ +
+ +
+
The nine become a pair
+
+
effortmean of all nine
how long to read
+
consequencemax of four
what breaks if wrong
+
+
+ +
+ +
+
The pair picks one route, and the route is the instruction
+ + + + + + + +
RouteWhat the repo owner does
green-is-enoughmerge it on a green tick, nobody needs to read it
tests-are-enoughmerge it once a test covers the change
ai-review-is-enoughrun an AI review; if it finds nothing, merge it
human-requiredput it in front of a person before it merges
human-plus-authora person reads it with the author walking them through
+
+ +
+
What the skill reports, from two real runs
+
+
+ apache/airflow #73729  "Fix clearing with upstream and downstream selecting unrelated tasks"
+ bugfix · check correctness first  |  2 files, +51/-2
+ security surface 0.06 · blast radius 1.1/2 · has tests 0.98 + ⟶ (0.27, 0.15) + ⟶ tests-are-enough +
+
+ mcp-gateway-registry #1693  "feat(security): Asymmetric (ES256) internal JWT signing"
+ feature · check security first  |  51 files, +3052/-626
+ security surface 1.00 · blast radius 2.0/2 · has tests 0.96 + ⟶ (0.74, 1.00) + ⟶ human-plus-author +
+
+ the tuple is (effort, consequence) +
+
+
+ + + diff --git a/samples/pr-triage/assets/flow.png b/samples/pr-triage/assets/flow.png new file mode 100644 index 0000000..95eebb5 Binary files /dev/null and b/samples/pr-triage/assets/flow.png differ diff --git a/samples/pr-triage/assets/pipeline.html b/samples/pr-triage/assets/pipeline.html deleted file mode 100644 index f9ca5cf..0000000 --- a/samples/pr-triage/assets/pipeline.html +++ /dev/null @@ -1,249 +0,0 @@ - - - - - - - - - -

pr-triage: one pull request, one call, one route

-

Jev answers eleven questions in one call. Code does every piece of arithmetic, and the output is a route rather than a merge.

- -
-
1
-
-

Read one pull request

-

Title, description, the file list, and as much of the diff as the budget holds. A patch - too large to send whole arrives as whole files first, then paths and line counts.

-
-
- -
- -
-
2
-
-

Stop early on anything that is not reviewable

-

One call to the checks API per pull request, before any Jev call. Three states are - terminal: no questions asked, no route computed, nothing spent. On eighteen - apache/airflow pull requests this skipped ten of them.

- - - - - -
StateMeansWaiting on
ci-failinga check concluded failure, timed out, or wants actionthe checks, or the branch
ci-pendinga check is still runningnobody, come back later
draftthe author marked it draftthe author, who is not asking
-

A name that fails on several unrelated pull requests is the check being - broken rather than each change breaking it: a boto3 bump cannot break Postgres - serialization. Five of eight failures were shared that way, and the output says so, - because the state is a fact and never a verdict on the author.

-
-
- -
- -
-
3
-
-

Ask eleven questions, in one call

-

Every question sees the same pull request. None sees another's answer, so the eleven - cost the tokens of eleven and the wall clock of about one.

-
-
-

Nine scored, and weighted

-

blast radius how far it reaches · security surface auth, secrets, - tokens, input validation · breaking change a caller stops working · - infra surface deploy, images, CI · design decisions judgment calls to - agree with · mechanical one edit repeated · scope creep the diff - exceeds the description · has tests the change is covered · - description quality what, why, and how they checked

-
-
-

Two labels, never scored

-

What kind of change: docs, tests, dependency, config, bugfix, feature or - refactor.
- What to check first: correctness, security, compatibility, tests, deployment - or readability.
- Both print beside the route.

-
-
-
-
- -
- -
-
4
-
-

Read the answers on two axes, not one

-

Effort and consequence are different questions, and they take different arithmetic.

-
-
-

Effort: the weighted mean of nine

-

All nine, at the weights in questions.yml: blast radius - 0.15, security surface 0.15, design decisions 0.14, mechanical 0.12, breaking - change 0.11, scope creep 0.09, infra surface 0.08, has tests 0.08, description - quality 0.08.

-

Averaging is right here, because a diff that is long in one - way and short in three really is a medium read. Mechanical, has tests and - description quality count inverted, so a repeated edit, a covered change and - a thorough description each pull it down.

-
-
-

Consequence: the max of four

-

Four of the same nine, read differently: security surface (auth, secrets, - tokens), breaking change (an existing caller stops working), blast - radius (reaches shared code paths), infra surface (deploy, images, CI).

-

The max, never the mean, because a change that is safe - in three ways and dangerous in one is a dangerous change. A score reads the - probability on its top level only: blast radius level 1 is "one module", - which is ordinary work.

-
-
-
-
- -
- -
-
5
-
-

Turn the pair into a route: what would be enough to merge

-

Consequence sets the floor. Effort can raise it and never lower it. Nothing here - says "merge", because a model reported at 67.8% accuracy can decide how much - evidence to demand and cannot be the last gate before main.

- - - - - - - -
RouteEnough to merge onReached when
green-is-enoughCI passingconsequence under 0.15
tests-are-enoughCI, and a test that exercises itconsequence 0.15, or effort 0.42
ai-review-is-enoughthe above, and an AI review finding nothingconsequence 0.35, or effort 0.62
human-requireda person reads itconsequence 0.60
human-plus-authora person reads it, author walking throughconsequence 0.60 and effort 0.62
-
-
- -
- -
-
6
-
-

Let the file and line counts raise the effort tier

-
-
over 30 files or 1,500 lines → high
-
over 12 files or 500 lines → medium
-
over 4 files or 150 lines → low
-
-

Jev reads a truncated diff, so a 54-file change can read as one repeated edit. - Size is arithmetic, Python and Go own it, and it only ever raises.

-
-
- -
- -
-
7
-
-

Print the route, the reason, and the one thing that would change it

-

Grouped by route, with what set the consequence, the questions carrying the effort, - and the single signal that would drop it a route: "a test that exercises the - unconnected-identity path". A tier tells an author nothing to do. That line does.

-
-
- - - diff --git a/samples/pr-triage/assets/pipeline.png b/samples/pr-triage/assets/pipeline.png deleted file mode 100644 index f1af26a..0000000 Binary files a/samples/pr-triage/assets/pipeline.png and /dev/null differ diff --git a/samples/pr-triage/assets/render.py b/samples/pr-triage/assets/render.py index 9cf086b..220c613 100644 --- a/samples/pr-triage/assets/render.py +++ b/samples/pr-triage/assets/render.py @@ -1,4 +1,4 @@ -"""Render pipeline.html into the PNG the sample README embeds. +"""Render flow.html into the PNG the sample README embeds. The HTML is the source. Edit it, run this, and commit both files. @@ -25,8 +25,8 @@ logger = logging.getLogger(__name__) ASSETS: pathlib.Path = pathlib.Path(__file__).parent -SOURCE: pathlib.Path = ASSETS / "pipeline.html" -TARGET: pathlib.Path = ASSETS / "pipeline.png" +SOURCE: pathlib.Path = ASSETS / "flow.html" +TARGET: pathlib.Path = ASSETS / "flow.png" # The body sets its own width, so the viewport only has to be wide enough not to # wrap it. Two device pixels per CSS pixel keeps the text crisp on a retina screen diff --git a/samples/pr-triage/go/README.md b/samples/pr-triage/go/README.md index 2529962..8174835 100644 --- a/samples/pr-triage/go/README.md +++ b/samples/pr-triage/go/README.md @@ -44,7 +44,7 @@ Without a GitHub token the API allows sixty requests an hour, and one pull reque ## GitHub Enterprise Server -The API base is a variable, not a glued-together path, so an enterprise host needs no patch. Three ways to set it, highest precedence first: +Every endpoint hangs off one base URL that the caller can set, so an enterprise host needs no patch. Three ways to set it, highest precedence first: ```bash pr-triage owner/repo -api-base https://ghe.example.com/api/v3 # the flag @@ -60,12 +60,26 @@ The progress line says which API it chose, so a wrong guess shows up before any repository acme/widgets through https://ghe.example.com/api/v3 ``` +## It stops before Jev on anything not reviewable + +One call to the checks API per pull request settles three states, and none of them spends a question: + +| State | Means | Waiting on | +| --- | --- | --- | +| `ci-failing` | a check concluded failure, timed out, or wants action | the checks, or the branch | +| `ci-pending` | a check is still running | nobody, come back later | +| `draft` | the author marked it draft | the author, who is not asking | + +`ci-failing` states a fact and never judges the author. A check name that fails on several unrelated pull requests is the check being broken rather than each change breaking it, so the binary counts the names across the queue and says which failures belong to the checks. That costs no extra call, because the names are already in the dataset. + +On the eighteen most recent open `apache/airflow` pull requests this skipped ten. + ## What it writes Two files per run, into `-out` (default `./data`), plus the dataset when it fetched one: -- `--triage.json`, every answer as Jev sent it, beside the credit, the tier and the size floor -- `--triage.md`, the table and the tier sections, ready to paste into a pull request +- `--triage.json`, every answer as Jev sent it, beside the credit, the effort, the consequence, the route and the size floor, with `route_counts` and `state_counts` rolling the queue up so a job reads its shape without walking every entry +- `--triage.md`, the table and the route sections, ready to paste into a pull request Both match the Python sample's reports field for field, and the dataset matches too, so a dataset or report written by either tool reads in the other. @@ -77,7 +91,7 @@ Both match the Python sample's reports field for field, and the dataset matches pr-triage owner/repo -all -fail-on-route human-required -quiet ``` -`-fail-on-tier` does the same on the effort tier, for a job that cares about review time rather than about what evidence is required. +`-fail-on-tier` does the same on the effort tier, for a job that cares about review time rather than about what a merge would need. | Exit | Meaning | | --- | --- | @@ -109,7 +123,7 @@ go build -o pr-triage . && ./pr-triage -version ../questions.yml and the embedded copy differ: run ./build.sh, which copies the canonical file in ``` -Without that test the two tools would score the same pull request differently, for a reason nobody could see from either side. +Without that test the two tools would disagree about one pull request, for a reason nobody could see from either side. `-questions path.yml` points at a payload on disk instead, which is how you try a different weight without rebuilding. @@ -117,6 +131,6 @@ Without that test the two tools would score the same pull request differently, f Nothing in the judgment: the same eleven questions, the same weights, the same two axes, the same route cuts, the same 0.02 deadband on both, and the same size floor. Two differences worth knowing: -**The legend is read loosely.** Jev echoes each rubric level back in a `legend`, and one level in the current payload arrives as an object rather than a string, because `- None: some text` is YAML for a mapping. The Go side accepts any value there, so a released binary keeps working against an older payload. +**The legend takes any value.** Jev echoes each rubric level back in a `legend`, and one level in the current payload arrives as an object rather than a string, because `- None: some text` is YAML for a mapping. The Go side accepts any value there, so a released binary keeps working against an older payload. -**Repeat runs move a load by about 0.02**, the same as the Python. A tier within the deadband of a cut prints `(on a cut)` rather than picking a side, so two runs of one queue agree with each other. +**Repeat runs move an effort by about 0.03 and a consequence by about 0.07**, the same as the Python. Either number within 0.02 of a cut prints `(on a cut)` rather than picking a side, so two runs of one queue do not contradict each other. Running both implementations over one dataset agreed on 16 of 18 routes, and both disagreements sat on a cut.