diff --git a/.env.example b/.env.example index ccfcc3c..0285181 100644 --- a/.env.example +++ b/.env.example @@ -83,6 +83,45 @@ PRXREF_CONFIDENCE_FLOOR=0.6 # (Legacy alias: PRXREF_MAX_ERRORS.) PRXREF_MAX_ERROR_FINDINGS=10 +# Per-severity caps (0.15.0): max warning-severity findings reported per +# review, the excess dropped lowest-confidence-first. Must be >= 0; 0 caps +# every warning. Empty (the default) = unlimited. +# PRXREF_MAX_WARNING_FINDINGS= + +# Per-severity caps (0.15.0): the same cap for the minor tier, findings of +# severity "outofscope". That is a severity, NOT ticket scope "out" from +# PRXREF_TICKET_CONTEXT_FILE. "spec" findings are never capped. Must be >= 0; +# empty (the default) = unlimited. +# PRXREF_MAX_OUTOFSCOPE_FINDINGS= + +# Per-rule cap (0.15.0): the most findings one team rule may produce in a +# review, across files. Applies only when a review rules file +# (PRXREF_REVIEW_RULES or PRXREF_SCOPED_RULES) is loaded, and then the model +# is asked to name the rule it applied. The excess fold into the best one +# (highest severity, then confidence), which lists them as "Also at: ..."; +# findings with no rule fold on their normalized title. Whole-PR sweep +# findings are never capped. Must be >= 0; 0 turns it off. Default 2. +# PRXREF_MAX_FINDINGS_PER_RULE=2 + +# Finding grouping (0.15.0): set to exactly 1 to fold findings that break the +# same rule in the same file into one comment listing the other locations +# ("Also at: ..."); without a rule, the normalized title groups them. Runs +# before the caps, so PRXREF_MAX_ERROR_FINDINGS counts groups. The rule label +# is one line (whitespace runs collapse to a space) of at most 120 characters; +# a longer one, or one holding a control, surrogate, private-use or bidi +# control character, is dropped and the finding groups by title. Every +# --format json finding row carries "rule" and "locations" (null when unset). +# Default off: prompts and output are unchanged, apart from those two JSON +# keys being null. +# PRXREF_GROUP_FINDINGS=0 + +# Reworded-duplicate dedup (0.15.0): two findings on the same file and line +# whose titles score at or above this Jaccard similarity (and share at least +# 3 title words) are one finding; a chunk copy beats a sweep copy of equal or +# lower severity. Greater than 0 and at most 1.0. Empty (the default) = off: +# only the exact-title dedup runs. Not yet tuned on real review data. +# PRXREF_DEDUP_SIMILARITY= + # Max diff chunks reviewed per PR (default 8). Must be > 0. PRXREF_MAX_CHUNKS=8 @@ -204,6 +243,31 @@ PRXREF_MAX_CHUNKS=8 # a longer body is truncated with a warning. Must be > 0. # PRXREF_REVIEW_RULES_MAX_CHARS=12000 +# Path-scoped review rules (0.15.0): rules files and directories (*.md, one +# level deep) whose `applies_to:` front-matter globs pick the chunks each file +# reaches; the whole-PR sweep gets the union. Added to PRXREF_REVIEW_RULES, +# never replacing it. A file without applies_to reaches every unit; a URL, +# `applies_to: []` or a malformed file is a configuration error (exit 2). +# Comma- or whitespace-separated; repeatable `--scoped-rules PATH` flags +# replace this list. Empty (the default) = off. Read from a trusted checkout. +# PRXREF_SCOPED_RULES= + +# Path-scoped review rules (0.15.0): characters of scoped-rules text one +# review unit (a chunk or the sweep) receives. Whole files go in, in load +# order, while they fit; the file that overflows is truncated to the room +# left (or left out when no room is left), every later file is left out, and +# one warning per run names this variable. Must be > 0. +# PRXREF_SCOPED_RULES_MAX_CHARS=24000 + +# Prompt template overrides (0.15.0): directory of replacement worker.md, +# systemic.md and summary.md; an absent file keeps the packaged template. Each +# one is validated before any network call, and a bad one is a configuration +# error (exit 2). `--prompts-dir DIR` wins for one run; `prxref prompts export +# DIR` writes the packaged templates to start from. Read it from a trusted +# checkout: a PR that commits a template rewrites its own review. Empty (the +# default) = the packaged templates. +# PRXREF_PROMPTS_DIR= + # Plain-text or Markdown file holding the ticket this PR implements. When set, # every finding is marked in, out of, or of unknown ticket scope. An empty # file means "this PR has no ticket". A missing or non-UTF-8 file is a diff --git a/.gitignore b/.gitignore index 165d860..d226341 100644 --- a/.gitignore +++ b/.gitignore @@ -6,6 +6,7 @@ __pycache__/ dist/ *.egg-info/ .env +prxref-eval/ .superpowers/ .claude/ docs/superpowers/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 453e21c..4fc9ff0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,399 @@ this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm Issue numbers in entries before 0.14.0 refer to the project's previous issue tracker. +## [0.15.0] — 2026-09-24 + +The tuning release. A team can now replace the review prompts, scope its review +rules to paths, fold findings that break the same rule into one comment, cap +warnings and minor findings, cap how many findings one rule may produce, and +measure a configuration against labelled human findings with `prxref eval`. A +`--pr-url` replay shows the title and description the PR had at review time, +and GitHub pull requests past GitHub's diff size limit are reviewed. Each new +option is off until you configure it, except the per-rule cap, which loading a +review rules file turns on. + +### Added + +- **Reworded-duplicate dedup (#10).** Set `PRXREF_DEDUP_SIMILARITY` (above 0, + at most 1.0) to make the sweep dedup also drop reworded restatements: two + active findings in the same file and on the same line whose titles reach that + Jaccard similarity over a dedicated title tokenizer, and share at least 3 + title tokens, count as one. Across the chunk/sweep boundary the chunk copy + always survives, and the sweep copy is dropped only when it is no more + severe, so the review's worst severity never drops. On one side of the + boundary the more severe copy is kept, then the more confident one. + File-level findings are never compared. The dropped copy is kept for audit in + the run record, with the drop reason `duplicate of chunk finding (reworded, + similarity 0.57)` or `duplicate of sweep finding (reworded, similarity + 0.57)`. It is an environment variable only, with no CLI flag, and the webhook + server reads it the same way. Unset, the default, only the exact-title dedup + runs, as before. +- **Prompt-template overrides (#11).** `PRXREF_PROMPTS_DIR`, or `--prompts-dir + DIR` for one run, names a directory that may replace any of `worker.md`, + `systemic.md` and `summary.md`; a template it lacks falls back to the packaged + one. The worker template reaches every chunk, the systemic template the + whole-PR sweep, and the summary template every posted summary, the empty-diff + summary and the inline-accounting re-post included, but not the error notice. + Overrides are checked before any network call. A review template must keep + the `## Review Context` marker and every packaged placeholder below it + (`{scope_example}` and `{rule_example}` are optional), and `summary.md` must + keep `{findings}`. A URL, a missing directory, a template over 256 KiB, text + that is not UTF-8 or holds NUL bytes, and a symlink out of the directory are + each a configuration error (exit 2) naming the flag or the variable. An + unknown placeholder, a placeholder above the marker and an unrecognised file + each log a warning. The flag wins over the variable, `--prompts-dir ""` turns + it off, and the webhook server re-reads the variable on every webhook. The + run record and `--format json` gain `prompt_templates` (always present, + `null` when unset), holding the path, raw-byte SHA-256 and character count of + each template file the directory holds, edited or not; the trace gains a + `prompts ok` event, and `-v` prints a `prompts:` line with the first 12 + characters of each hash. See `docs/prompt-templates.md`. +- **`prxref prompts export DIR [--force]` (#11).** It writes the packaged + `worker.md`, `systemic.md` and `summary.md` into `DIR` byte for byte, as the + starting point for an override directory, creating `DIR` when missing and + printing each path it wrote. If any of the three already exists, nothing is + written and the command exits 2 naming the file; `--force` overwrites. The + `prxref eval` judge prompt cannot be overridden and is not exported. +- **Path-scoped review rules (#12).** `PRXREF_SCOPED_RULES`, or the repeatable + `--scoped-rules PATH` that replaces it for one run (`--scoped-rules ""` turns + it off), names rules files and directories. A directory is read one level + deep for `*.md` files in name order, and a run loads at most 50 files. Each + file's `applies_to:` front matter (alias `applyTo`, the `.github/instructions` + spelling) decides which chunks receive it: a glob, a comma-separated string, + a one-line `["...", "..."]` list, or indented `- ` lines. Globs match + the whole path case-sensitively, `*` crosses `/`, a `!` glob excludes + wherever it sits in the list, and `**/` also matches zero directories, so + `**/*.java` selects a root-level `Foo.java`; a renamed file's old path + selects too. A file without the key reaches every chunk, and the whole-PR + sweep receives the union. Scoped files add to the always-on + `PRXREF_REVIEW_RULES` file, and their `severity:` maps merge into one + run-wide map, which applies even without an always-on file; one word mapped + to two tiers across the files is a configuration error. Each file is capped + at `PRXREF_REVIEW_RULES_MAX_CHARS`, and `PRXREF_SCOPED_RULES_MAX_CHARS` + (default 24000) caps the scoped text one review unit receives: files go in + whole in load order, the first that does not fit is truncated, and later ones + are omitted behind a marker, with one WARNING per run. Every file is checked + before any network call. A URL, a path that escapes the working directory, + text that is not UTF-8 or holds NUL bytes, and malformed front matter + (`applies_to: []`, an entry that is not a string, a glob starting with `/`, + or only `!` globs) each exit 2 naming the flag or the variable, the path and, + where known, the line. The run record and `--format json` gain + `scoped_rules` (`null` when off; otherwise the entries, each file with its + SHA-256 and globs, the per-unit cap, and the files each unit received), `-v` + prints a `scoped rules:` line, and the trace gains a `scoped_rules ok` event + and a `rules` field on each `chunk start` and `sweep start` event. The + webhook server re-reads the variable and the files on every webhook. See + `docs/review-rules.md` "Path-scoped rules". +- **Finding grouping (#13).** With `PRXREF_GROUP_FINDINGS=1`, chunk findings + in one file that break the same rule, or that name no rule and share a + normalized title, fold into one comment at the group's first line. That + comment takes the group's highest severity and highest confidence and ends + with `Also at:` and the group's other lines as `file:line`; the other + findings are kept for audit with the drop reason `grouped into + :`. The worker and sweep prompts ask the model to name, in a + per-finding `rule` key, the team review rule or standard a finding applies, + and the example finding shows the key through a new optional + `{rule_example}` slot in both packaged templates. A rule is kept as a + one-line label of at most 120 characters with whitespace collapsed, and + compared without regard to case. It is dropped, never truncated, when it is + not a string, is empty or too long, or holds a control, surrogate or + private-use character or a bidi embedding, override or isolate control + (U+202A to U+202E, U+2066 to U+2069); a zero-width joiner or non-joiner is + kept. A prompt override without the slot still loads and still asks for a + rule, but its example shows no `rule` key, and a review with grouping on logs + one WARNING naming each such override file; re-export with `prxref prompts + export DIR --force` and re-apply your edits to pick the slot up. Grouping + runs before the finding caps, so every cap, `PRXREF_MAX_ERROR_FINDINGS` + included, counts a group once. Whole-PR sweep findings are never grouped, + and a sweep finding that restates a grouped finding is still dropped as + `duplicate of chunk finding`. Off by default: unless a review rules file + turns on the per-rule cap (#18), the model is not asked for a rule, and + output is unchanged apart from the `rule` and `locations` JSON keys being + `null`. +- **Per-severity finding caps (#13).** `PRXREF_MAX_WARNING_FINDINGS` and + `PRXREF_MAX_OUTOFSCOPE_FINDINGS` cap how many `warning` and how many + `outofscope` findings one review posts, ranked like + `PRXREF_MAX_ERROR_FINDINGS`: the most confident survive, then by file and + line. The rest are kept for audit with the drop reason `warning cap exceeded + (max N)` or `outofscope cap exceeded (max N)`, and `0` drops every finding of + that severity; `spec` findings are never capped. `outofscope` here is the + minor severity (style and nits), not the ticket scope `out`: a finding the + ticket puts out of scope is capped by its severity like any other. Both caps + also apply to a release-only PR's summary-only review. Under + `PRXREF_FAIL_ON=any`, a cap of `0` can turn exit 1 into exit 0, since a cap + narrows the gate and never widens it. Unset means unlimited. +- **`rule` and `locations` in the output (#13).** Every `--format json` + finding row carries two more keys after `scope`: `rule`, the rule the finding + names (`null` when it names none), and `locations`, which on a grouped + finding lists the other places its `Also at:` paragraph names as `{"file", + "line"}` objects, in that order. The per-rule cap (#18) sets it too, on the + best finding of each rule it folds, and it is `null` on every other row, + including each member dropped as `grouped into :`. In text + output, a finding that names a rule ends its line with ` [rule: ]`, + after any scope tag. +- **`prxref eval run` (#14).** `prxref eval run --cases PATH --label NAME + [--out DIR] [--resume]` replays every labelled case in process, never + posting, and writes the run to `DIR/NAME/` (default `--out` is + `./prxref-eval/`). Cases come from a `cases.json` file (`{"version": 1, + "cases": [...]}`) or a directory of `case-*/` directories in the + `tests/evals` layout, and every case is validated before the first review; a + bad one exits 2 naming `--cases`, the case and the field. `--rules-file`, the + repeatable `--scoped-rules` and `--prompts-dir` give every case the same team + rules, path-scoped rules and prompt templates, as `review` takes them, and + `PRXREF_SCOPED_RULES` and `PRXREF_PROMPTS_DIR` reach every case unless those + flags override them. `run.json` records the case ids, the SHA-256 of each + packaged prompt template and any overrides, the reviewer's sampling, review + rules and scoped rules, and an allowlist of non-secret settings, so no + credential is written; each case keeps its `--format json` record and a + trace. A crash or a per-case configuration error is recorded as that case's + `error.json`, the next case still runs, and the command exits 0. A bad + `--label`, or an existing run without `--resume`, exits 2 naming the flag, + and `--resume` skips every case already recorded. A bare `prxref eval` + prints usage and exits 2, and `eval` adds no environment variable. See + `docs/evals.md`. +- **`prxref eval score` (#14).** `prxref eval score --label NAME + [--judge-model MODEL] [--out DIR]` grades a run against its human labels and + writes `score.json` and `score.md` into the run directory. A label with a + `must_match` predicate is graded in code with no LLM call: the same file, + within 5 lines, and a match on the finding's title and body (a plain value + ignores case and markup, a `re:` pattern is case-insensitive). Every other + label goes to one single-shot judge call per case, which grades it `full`, + `partial` or `none` under a versioned judge prompt. The judge runs on the + review's own backend, base URL and credentials with only the model changed; + `--judge-model` is required whenever such a label exists, and leaving it out + exits 2 before any call. Judge replies are cached under + `DIR/NAME/judge-cache/`, so a rescore makes no call, and a failed or + malformed judge reply counts as `judge_error`, left out of every denominator + rather than scored as a miss. One AI finding credits at most two labels, and + a grouped finding is credited at each of its locations in both tiers. A case + whose review failed is scored with every label a miss. The report gives + micro recall with per-case rows, recall by severity, by category and over + accepted labels (a `partial` grade counts 0.5), unmatched AI findings per + PR, severity agreement (a human `minor` counts as `warning`), chunks failed, + elapsed time, and review and judge cost, where an unknown cost is shown as + unknown and never summed. A judge model that is also a reviewer model logs a + warning and is stamped `self_judged`. +- **`prxref eval compare` (#14).** `prxref eval compare A B [--out DIR]` + compares two scored runs, each named by a label under `--out` or by a run + directory; a label wins over a same-named directory. It prints a Markdown + report with no timestamps, so the same two runs always give byte-identical + output: a metrics table with A, B and the change (recall in percentage + points, and `unknown` rather than a number when either side is unknown), + every label whose grade or credit changed, and the cases and labels only one + run has. It warns when the runs used a different judge prompt or judge + model, cover different cases, or carry different replay `description` + stamps for a case, since a pinned run against a live one is not a + like-for-like comparison. A run with no `score.json` exits 2 naming `A` or + `B`. +- **Replay description flags (#16).** `prxref review --as-of TIME`, + `--description-file PATH` and `--no-description` choose which PR description + a replay shows. Each makes the run a replay with posting off, and they + exclude each other: giving two or more exits 2 naming each one. `--as-of` + takes an ISO-8601 time with a UTC offset (`2026-05-01T09:30:00Z`) and needs + `--pr-url`; a date alone or a time without an offset exits 2 rather than + being read in the local time zone. `--description-file` is read like + `--rules-file`, so a missing, unreadable or non-regular file, text that is + not UTF-8 or holds NUL bytes, and a path under the working directory that + symlinks out of it each exit 2 naming the flag. `--no-description` reviews + with an empty description. +- **Replays record the title and description they showed (#16).** The + `replay` stamp in the run record and `--format json` gains `description` + (`pinned`, `live`, `file` for `--description-file` or a `--diff-file` without + `--pr-url`, `none` for `--no-description`), `as_of` (the cutoff as a UTC + ISO-8601 time ending in `Z`, else `null`) and `as_of_source` (`flag`, + `first-review` or `head-commit`, else `null`). All seven keys are present on + every replay, and an `as_of` passed back as `--as-of` names the same instant. + The text `replay:` line gains `description=` and, when a cutoff was + chosen, `as_of=