Skip to content

ci: review pull requests against documented conventions - #220

Open
SpiGAndromeda wants to merge 14 commits into
mainfrom
ci/conventions-check
Open

SpiGAndromeda wants to merge 14 commits into
mainfrom
ci/conventions-check

Conversation

@SpiGAndromeda

@SpiGAndromeda SpiGAndromeda commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Conventions that span several files (the Pi compatibility floor, brand asset references, Node.js versions, the changelog rules, and a few more) were only checked by tests or stated in AGENTS.md and CONTRIBUTING.md prose. This pull request documents them and reviews each pull request against them with pi.

Conventions

  • docs/conventions/ holds one file per convention, <slug>.md.
  • Its paths frontmatter lists the globs (git glob pathspecs) of the files the convention applies to; a leading ! excludes.
  • docs/conventions/README.md describes the format and shows a sample convention file. It defines a change, the unit a convention speaks of, as the diff of a pull request in CI or the edits an agent or contributor makes before committing, so the conventions also serve a check outside CI.
  • AGENTS.md becomes a router to the conventions and the other docs.
  • CONTRIBUTING.md drops the rules that moved and keeps the contributor how-to.

Review

.github/workflows/conventions.yml runs pi as a read-only reviewer (read, grep, find, ls) on the pull request's diff, one run per convention whose paths match a changed file. The review instructions are the pi skill .agents/skills/checking-convention/SKILL.md. The reviewer:

  • works from the diff hunks;
  • gives not_applicable without reading anything to a statement sentence that no added or removed line concerns;
  • otherwise reads only a file the diff changes or a file the convention names, each once;
  • returns one verdict per sentence, with the line that makes the sentence true or false as evidence and, for a violation, the reason.

The diff the reviewer gets carries each added or context line's line number in the head revision in front of it, so the evidence needs no extra read. The job installs ripgrep and fd-find first, because pi's grep otherwise depends on downloading them from GitHub at first use.

The conventions and the skill are read from the checkout, which is GitHub's merge commit for the pull request, so a pull request is checked against the conventions as they read once it is merged.

Workflow

It runs on pull requests to main from branches of this repository; forks get no secrets. Its steps call git, jq, node and pi directly, with no script file of their own.

Job What it does
List conventions Turns each convention's paths into git pathspecs and lists the conventions that match a changed file. Fails on a malformed convention file (frontmatter holding anything but double-quoted paths items, a brace in a glob, or not exactly one H1 outside code blocks) and on one without an including glob.
One job per listed convention Runs with fail-fast: false, so every convention reports. Skipped when none is listed.
Conventions Fails unless the list succeeded and the convention jobs succeeded or were skipped. It is the job to make required.

A convention job fails:

  • on a violation;
  • when the verdicts do not cover each sentence of the convention's statement exactly once;
  • on a non-zero pi exit;
  • when pi writes no verdict. pi writes the verdict only when it is valid against scripts/conventions-verdict.schema.json, and a turn budget of 30 stops a run that does not get there.

Output

  • Violations become ::error annotations at the offending line, at the file for a removed line, or at the convention file when no file shows the violation (a missing changelog entry), with the sentence, the reason and, when there is one, the quoted line.
  • pi's stderr tail is printed inside a ::stop-commands:: window, so text from the pull request cannot run workflow commands.
  • Each job uploads pi's event stream as a conventions-events-<slug>-attempt-<n> artifact kept for 7 days.
  • The pi extensions are cached in the agent directory, and the cache is saved even when the check fails. A Renovate regex manager updates their pinned versions.

Setup

The model is deepseek-flash through pi's built-in DeepSeek provider, which reads the DEEPSEEK_API_KEY repository secret. pi records each response's cost in the event stream.

Also included

  • The Renovate change that stops updates for the Pi packages and typebox, because the Pi compatibility floor convention states it.
  • import-x/no-named-export for index.ts in eslint.config.mjs: the extension's public surface is the default export of index.ts, and lint enforces it.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Unit test coverage

lines statements functions branches
Total 88.65% (3689/4161) 88.78% (3815/4297) 93.34% (743/796) 79.96% (2619/3275)
Base (6e8cd36) 88.65% (3689/4161) 88.78% (3815/4297) 93.34% (743/796) 79.96% (2619/3275)
Change ±0.00 ±0.00 ±0.00 ±0.00
Coverage per file (52)
File lines statements functions branches
as-promise.ts 100% (2/2) 100% (2/2) 100% (2/2) 100% (0/0)
frontmatter-cache.ts 95.83% (92/96) 95.83% (92/96) 100% (23/23) 93.91% (108/115)
handoff/boundary.ts 96.42% (135/140) 96.57% (141/146) 100% (23/23) 93.87% (92/98)
handoff/command.ts 100% (16/16) 100% (16/16) 100% (3/3) 95% (19/20)
handoff/copy.ts 100% (16/16) 100% (16/16) 100% (0/0) 100% (0/0)
handoff/eligibility.ts 100% (18/18) 100% (18/18) 100% (5/5) 100% (30/30)
handoff/format.ts 100% (17/17) 100% (17/17) 100% (7/7) 100% (12/12)
handoff/tool.ts 100% (32/32) 100% (32/32) 100% (5/5) 100% (31/31)
index.ts 95.98% (359/374) 96.08% (368/383) 91.93% (57/62) 92.83% (246/265)
model-groups/autocomplete.ts 93.02% (40/43) 92% (46/50) 94.44% (17/18) 76.66% (23/30)
model-groups/availability.ts 100% (2/2) 100% (2/2) 100% (2/2) 100% (2/2)
model-groups/command.ts 78.57% (11/14) 78.57% (11/14) 80% (4/5) 87.5% (7/8)
model-groups/constraints/engine.ts 91.66% (11/12) 92.85% (13/14) 100% (7/7) 87.5% (7/8)
model-groups/constraints/modalities.ts 92.15% (47/51) 92.3% (60/65) 90.24% (37/41) 91.52% (54/59)
model-groups/constraints/presentation.ts 100% (16/16) 100% (16/16) 100% (6/6) 62.5% (5/8)
model-groups/constraints/registry.ts 100% (9/9) 100% (12/12) 100% (4/4) 100% (4/4)
model-groups/constraints/resolution.ts 100% (1/1) 100% (2/2) 100% (2/2) 100% (0/0)
model-groups/constraints/types.ts 0% (0/0) 0% (0/0) 0% (0/0) 0% (0/0)
model-groups/display.ts 98.14% (53/54) 98.18% (54/55) 100% (3/3) 98% (49/50)
model-groups/modalities.ts 100% (7/7) 100% (7/7) 100% (3/3) 100% (0/0)
model-groups/modality.ts 100% (29/29) 100% (30/30) 100% (2/2) 100% (34/34)
model-groups/names.ts 100% (1/1) 100% (1/1) 100% (1/1) 100% (0/0)
model-groups/prompt.ts 100% (7/7) 100% (7/7) 100% (3/3) 83.33% (5/6)
model-groups/router.ts 98.24% (112/114) 98.36% (120/122) 100% (30/30) 92.59% (100/108)
model-groups/store.ts 94.69% (250/264) 94.77% (254/268) 100% (51/51) 90.09% (200/222)
model-groups/tui.ts 90.05% (616/684) 90.2% (645/715) 89.47% (119/133) 80.32% (351/437)
model-groups/types.ts 100% (5/5) 100% (5/5) 100% (1/1) 100% (0/0)
notebook/pager.ts 94.33% (50/53) 94.33% (50/53) 90% (9/10) 85.36% (35/41)
notebook/prompt.ts 100% (12/12) 100% (12/12) 100% (6/6) 85.71% (12/14)
notebook/rehydration.ts 100% (57/57) 100% (58/58) 100% (6/6) 92% (46/50)
notebook/store.ts 96.82% (61/63) 96.96% (64/66) 100% (14/14) 89.47% (34/38)
notebook/tools.ts 96.72% (59/61) 96.72% (59/61) 100% (13/13) 78.43% (40/51)
notebook/topic-tool.ts 100% (9/9) 100% (9/9) 100% (3/3) 100% (8/8)
notebook/topic.ts 100% (14/14) 100% (14/14) 100% (3/3) 100% (6/6)
notifications.ts 100% (44/44) 100% (44/44) 100% (21/21) 100% (2/2)
os-sandbox.ts 80.51% (62/77) 80.76% (63/78) 94.11% (16/17) 84.21% (48/57)
prompt-sections.ts 100% (21/21) 100% (22/22) 100% (6/6) 100% (10/10)
readonly-bash.ts 62.8% (439/699) 62.98% (451/716) 76.47% (52/68) 53.5% (359/671)
readonly-boundary.ts 94.73% (18/19) 94.73% (18/19) 100% (4/4) 86.66% (13/15)
readonly-rehydration.ts 100% (12/12) 100% (13/13) 100% (1/1) 100% (10/10)
resolve-path.ts 88.88% (8/9) 88.88% (8/9) 100% (1/1) 50% (1/2)
runtime-singletons.ts 100% (9/9) 100% (9/9) 80% (4/5) 100% (4/4)
spawn/index.ts 98.69% (227/230) 98.75% (238/241) 100% (49/49) 92.22% (178/193)
spawn/renderer.ts 89.34% (503/563) 89.35% (512/573) 91.66% (77/84) 74.27% (332/447)
spawn/shared.ts 100% (13/13) 100% (14/14) 100% (2/2) 100% (8/8)
state.ts 96.55% (56/58) 96.55% (56/58) 83.33% (10/12) 100% (2/2)
system-prompt.ts 100% (3/3) 100% (5/5) 100% (4/4) 100% (0/0)
temp-dir.ts 80% (4/5) 80% (4/5) 100% (1/1) 100% (0/0)
thinking-levels.ts 100% (2/2) 100% (2/2) 100% (1/1) 100% (0/0)
time/format.ts 93.33% (28/30) 93.54% (29/31) 100% (10/10) 80% (12/15)
tui.ts 100% (25/25) 100% (25/25) 100% (1/1) 92.85% (26/28)
watchdog.ts 100% (49/49) 100% (49/49) 100% (12/12) 96.42% (54/56)

Commit 38edba5, attempt 1 of the workflow run; the HTML report is in that attempt's coverage artifact.

The Pi compatibility floor is a deliberate decision, so the four `@earendil-works/pi-*` packages and `typebox` are disabled in Renovate and move only through a manual change. `pi-node-floor` keeps its dashboard approval and now forms its own "Node.js floor" group, since the Pi group no longer exists.
Repository conventions that span several files (the Pi compatibility floor, brand asset references, Node.js versions, changelog and lint exemption rules) were only checked by tests or stated in `AGENTS.md` and `CONTRIBUTING.md` prose. `docs/conventions.md` now states each one with the files it applies to, and `AGENTS.md` becomes a router to it and to the other docs. `CONTRIBUTING.md` drops the rules that moved and keeps the contributor how-to.

`scripts/check-conventions.mjs` runs pi as a read-only reviewer (read, grep, find, ls) on the pull request's diff against the base branch's `docs/conventions.md` and fails on any violation. It also fails when pi exits non-zero, when the verdict is missing, unparsable or does not match `scripts/conventions-verdict.schema.json`, or when a convention heading has no result. Under GitHub Actions, text from the model and from pi is printed inside a `::stop-commands::` window, so it cannot inject workflow commands, and violations become `::error` annotations at the offending line.

`.github/workflows/conventions.yml` runs the check on pull requests to main from branches of this repository, since fork pull requests get no secrets. The model is reached through the CLIProxyAPI provider extension, configured by the `CLIPROXYAPI_BASE_URL` and `CLIPROXYAPI_API_KEY` secrets. The pi extensions install into the agent directory once and are cached by the hash of the script that pins their versions. While the base branch has no `docs/conventions.md`, the check uses the pull request's own copy and says so in a notice.
The first run on a large pull request stopped after 15 requests with "Response incomplete without a provider reason", and the log could not show what the turns did or why the last response ended. pi now runs with `--mode json`, and `scripts/check-conventions.mjs` writes the event stream to the file given by the new required `--events` argument. After every run it prints one summary line per turn (the tool calls with their main argument, and the tokens) plus totals. Under GitHub Actions the summary sits in a collapsed `pi turns` group, with its model-derived text inside the stop-commands window.

In json mode pi exits 0 even when its last response ends in an error, so the script now reads the stop reason from the last assistant message and fails with that reason and its error message. A `pi-budget` "reached turns limit" notice always fails the run, also when a verdict was written before it. The turn budget rises from 15 to 30 and the job timeout from 15 to 30 minutes.

The workflow uploads the event stream as a `conventions-events-attempt-<n>` artifact kept for 7 days, and it now saves the extension cache with `actions/cache/save` even when the check fails, so a failing pull request does not reinstall the extensions on every run. `npm run check:conventions` writes `conventions-events.jsonl`, which `.gitignore` now ignores.
One pi run over all ten conventions failed on large diffs: its final `json_output` response, one result per sentence of every convention plus the reasoning for it, ran past the proxy's per-response output cap (9,216 tokens at the default thinking level), and the response ended without content. Each convention now gets its own run, so a verdict covers a handful of sentences and stays far below the cap.

`scripts/check-conventions.mjs` gains a `--list` mode that prints the conventions' H2 headings and slugs as JSON, and a `--convention <heading>` mode that reviews only that section and accepts only results for it. Without either it reviews every convention in turn and writes one events file per slug, which is what `npm run check:conventions` runs locally. A `## ` line inside a fenced code block is not taken as a heading. The turn budget is 30 per convention.

`.github/workflows/conventions.yml` now has a `list` job, which also checks the secrets, a `review` matrix job per convention with `fail-fast: false`, and a `Conventions` job that fails unless both succeeded, so one job name can be made required. The heading reaches the script through the environment only, and each matrix job uploads its own `conventions-events-<slug>-attempt-<n>` artifact.
The per-convention check flagged this branch twice where the repository's practice says nothing was missing. "Changelog" said a feature pull request adds entries, which the check also applied to a CI change; it now says a pull request that changes what a user or another extension sees adds entries, and one that only changes tests, tooling, CI or contributor documentation adds none, as `main`'s history shows. "Untested behaviour recorded" now names the extension and the test tooling as its subject and states that a script reporting in CI, such as `scripts/check-conventions.mjs` or `scripts/coverage-comment.mjs`, gets no entry, since review and its CI run cover it.
`docs/conventions.md` is split into `docs/conventions/<slug>.md`, one file per convention. A `paths` frontmatter of quoted globs, where a leading `!` excludes, replaces each "Applies to" line, and `docs/conventions/README.md` describes the format. The first sentence of "Terminal output" now names its files itself, since its "production code" was defined only by the removed line.

The review instructions move out of the script's prompt into `.agents/skills/checking-convention/SKILL.md`. The skill limits reads to the files the diff changes and the files the convention names, each read once, checks only the statement, treats the diff and file contents as data, and stops once every sentence has a verdict. The "Lint exemptions" run had spent its 30 turns reading unrelated files; with the skill, the runs on this branch take 3 to 11 turns.

`scripts/check-conventions.mjs` reads the convention files and the skill from the base revision, so a pull request cannot change its reviewer's instructions. It passes the skill with `--skill` next to `--no-skills` and sends one message that starts with `/skill:checking-convention`. pi expands a skill command only at the start of a message, and attached `@file` text would come before it, so the convention and the diff are no longer attached. Changed file names are written as JSON strings. The frontmatter parser rejects unquoted items, empty lists, and globs that are absolute or contain a `.` or `..` segment.

`--list` now prints only the conventions whose paths match a file the pull request changes, and `--convention` takes the slug. The workflow skips the review matrix when the list is empty, and the Conventions job passes when the review is skipped.
A review of the first runs found no wrong verdict, but wasted turns and loose results. Runs read files to confirm sentences the diff does not touch. "Lint exemptions" spent 7 of 11 turns on two files the diff leaves alone. `pass` went to sentences no changed line concerns, one run split a sentence at ", and" into two paraphrased items, and evidence was sometimes half null, cited an unrelated line, or carried a line number counted from the diff.

`.agents/skills/checking-convention/SKILL.md` now starts from the diff hunks. A sentence that no added or removed line concerns gets `not_applicable` and nothing is read for it. Otherwise the reviewer reads and greps only a file the diff changes, or a file the convention names when the sentence compares a changed line with it. A sentence ends at its full stop and is never split. Evidence is the line that makes the sentence true or false, with the line number from read or grep output. All three fields are given or all three are null. `json_output` items carry exactly the six fields, and the diff is read to its end.

`scripts/check-conventions.mjs` now extracts each convention's statement and rejects a result whose sentence, with whitespace collapsed, is not part of it.

"Public surface" is split into two sentences. "Node.js versions" and "Pi compatibility floor" name `package-lock.json` as the source of the `engines.node` range and the `typebox` dependency of `@earendil-works/pi-coding-agent`, so the reviewer may read it. What they require is unchanged.

On this branch's diff the ten runs took 49 turns at the default thinking level (50 before) and 48 at low, with 28 sentences checked, 21 of them `not_applicable`, and no violation.
The 604-line `scripts/check-conventions.mjs` did little that git, sed and jq cannot do in a step, so `.github/workflows/conventions.yml` now holds the check itself. The list job reads each convention's `paths` with sed, turns them into git pathspecs (`:(glob)` and `:(glob,exclude)`) and selects the conventions whose pathspecs match a file in `git diff origin/<base>...HEAD`. It fails on a convention file without an including glob, since git would otherwise match every changed file outside the excludes, and it strips carriage returns from the convention name before printing it in a notice.

The review job writes the diff, runs pi and turns each `violation` into an `::error` annotation with jq, which escapes the message and the properties. The job fails when pi exits non-zero or writes no verdict, and prints the stderr tail inside a `::stop-commands::` window. pi-json-schema already checks the verdict against the schema before writing it, so the script's own shape check, its verbatim-sentence check and its turn summary are gone, along with the flags a fresh CI agent directory does not need (`--no-mcp`, `--no-skills`, `--no-prompt-templates`, `--no-approve`, `--no-session`, `--json-fallback none`). The extension cache key now hashes the workflow, where the extension versions are pinned.

Conventions and the skill are read from the checkout, which is GitHub's merge commit for the pull request, so a pull request is checked against the conventions as they read once it is merged. The workflow and the schema already came from the pull request, so reading the conventions from the base protected nothing, and a pull request that changes a convention together with the code is now checked against the new text.

A verdict now carries `reason`, which the schema requires for a `violation`, and the annotation shows it. The `convention` field and the skill's convention-name and matching-files inputs are dropped: each run checks one convention, and the skill derives the files from the diff. `npm run check:conventions` and its git-ignored output directory are removed.
…check

A review of the CI runs of the conventions check found three causes of wasted or failed turns.

pi looks for `rg` and `fd` on PATH and otherwise downloads them from GitHub on first use. In one run that download failed, and every grep call of that run failed with "ripgrep (rg) is not available and could not be downloaded". The review job now installs `ripgrep` and `fd-find` with apt before pi starts.

Five of nine runs spent one or two turns re-reading or grepping a changed file only to find the line number of a line the diff already showed. The diff given to pi now carries each added or context line's line number in the head revision in front of it. Removed lines get none. The raw diff is written to its own file first, so a failing `git diff` still fails the step. The skill takes the evidence line number from the diff when the diff shows one, and quotes the line without the number and the `+`.

Five runs read the diff 200 to 500 lines at a time, which cost one run three turns and let another skip most of the diff. The skill now says to read the convention file and the diff without a `limit`.

On this branch's diff the nine selected conventions took 26 turns in a local run, against 42 in the previous CI run, with every cited line number matching the file.
The conventions said "a pull request", which ties them to the CI check although an agent or contributor can check its edits against them before committing. The touched conventions and the checking skill now speak of a change, and `docs/conventions/README.md` defines it as a set of edits checked as one: the diff of a pull request in CI, or the edits made before committing.

"What is tested" and "Untested behaviour recorded" contradicted each other: one required tests for every behaviour change, the other allowed an entry in the "Untested behaviour" list instead. "Tested behaviour" replaces both and accepts either. It no longer covers the test tooling.

The public surface convention is gone. `eslint.config.mjs` enables `import-x/no-named-export` for `index.ts`, which fails on a named export there.

The changelog convention states that only a release changes `version` in `package.json`, and drops the clause that counted a non-`index.ts` module's lost export as a breaking change, which contradicted a public surface limited to the default export.

"HOME in tests" also covers `.husky/`, the root `.mjs` loader scripts and `tests/`, where a suite-wide `HOME` could be set too.

The list job read `paths` items only in the form `  - "<glob>"` and dropped any other line silently, so a typo narrowed which changes a convention reviewed. It now fails on a convention file whose frontmatter holds anything but double-quoted `paths` items or that has no or several H1s. The README says the check skips it and that a convention file of another format fails the check.
The conventions check reached `deepseek-flash` through a CLIProxyAPI endpoint, which reports no cost, so the event logs could not show what a run costs. pi ships a `deepseek` provider that reads `DEEPSEEK_API_KEY` and fills each response's cost into the event stream from its price list.

The review job now runs `deepseek/deepseek-flash` with the `DEEPSEEK_API_KEY` repository secret and no longer loads the CLIProxyAPI provider extension. `CONTRIBUTING.md` names the new secret.
The verdict schema checked each row's shape, so one `not_applicable` row passed a convention with several statement sentences. The report step now splits the statement the way `checking-convention` defines it (a full stop inside backticks does not end a sentence) and fails unless the verdicts cover each sentence once.

A violation without a file, such as a missing changelog entry, is annotated at the convention file instead of the job. A violation caused by a removed line gives its file and the removed text with no line number.

The format check rejects braces in `paths`, because git glob pathspecs have no brace alternation and such a glob matched nothing, so the convention was skipped. The frontmatter is read only up to its first closing `---`, so a horizontal rule in the body no longer adds globs, and the H1 count skips fenced code blocks.

A Renovate regex manager covers the pinned `-e npm:` extensions in `conventions.yml`.

The `home-in-tests` and `lint-exemptions` conventions are removed because a suite-wide `HOME` under the temp directory already fails the readonly tests, and the lint exemption rules are either enforced by lint or a visible change under the policy comment in `eslint.config.mjs`. The remaining conventions no longer state what a squashed diff cannot show (a raise "by hand, in one commit", "the current LTS"), drop the version sentence from the changelog convention, and name `notebook/pager.ts` as the one place that hands the terminal to a child.
@SpiGAndromeda SpiGAndromeda self-assigned this Nov 17, 2026
@SpiGAndromeda SpiGAndromeda modified the milestones: v0.6.0, 0.7.2 Nov 17, 2026

@grzegorznowak grzegorznowak left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool stuff!
Posting some early birds Skill-related observations as a first round.

Co-Authored-By: Claude <noreply@anthropic.com>
@SpiGAndromeda

SpiGAndromeda commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator Author

Thanks for the review. I’ve replied inline: statement extraction, diff generation, and diff scope.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants