Skip to content

docs(agents): record test-gate regression and stale-PR merge mechanics #1914

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
seonghobae merged 6 commits into from
Sep 5, 2026
+67 −0
Merged

docs(agents): record test-gate regression and stale-PR merge mechanics #1914

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
57210f8
docs(agents): record test-gate regression and stale-PR merge mechanics
seonghobae Sep 5, 2026
b35d1f6
Merge origin/main into docs/test-gate-and-merge-mechanics
seonghobae Sep 5, 2026
0ff7ed2
docs(agents): tighten test-gate rules after adversarial review
seonghobae Sep 5, 2026
e280c4f
docs(agents): name the real categories of forty-hex literal in tests/
seonghobae Sep 5, 2026
741acc7
Merge origin/main (#1906) into docs/test-gate-and-merge-mechanics
seonghobae Sep 5, 2026
6ecc526
Merge origin/main (#1909) into docs/test-gate-and-merge-mechanics
seonghobae Sep 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,3 +168,47 @@ them alone proves succession.
attributed a "timeout defaults to null" sentence to it that appears only in
`#1891`'s PR body — both caught by grepping the file instead of trusting the
summary that introduced them.

## Test-gate regressions and stale-PR merges

- A red `tests`, coverage, or `interrogate` gate on your pull request is not proof that your
diff caused it. Full-suite execution on a push to `main` is not guaranteed: the workflows
that run `pytest tests` on push are `paths:`-filtered, so a pairing broken outside their
declared paths reaches `main` with no full-suite run. The breakage then surfaces on the
next pull request whose review dispatch does run the suite, and fails it regardless of
that request's own diff. This procedure covers the suite gates only; a red Semgrep,
CodeQL, Strix, or Scorecard check is a different diagnosis.
- Reproduce a suspect failure on a clean baseline before repairing it. Run
`git worktree add /tmp/baseline <the PR's base ref> --detach`, then `cd /tmp/baseline`
and run `python3 -m pytest tests -q`; that takes roughly four minutes and needs no
virtualenv. You must `cd` into the worktree: over thirty test files read repository files
through working-directory-relative paths such as `Path(".github/workflows/...")`, so
pointing pytest at the baseline directory from your own checkout silently tests your tree
and reports a green baseline that proves nothing. Baseline the pull request's actual base
or merge-base rather than `origin/main` once `main` has moved past it. If the failure
reproduces on the baseline it is pre-existing: repair it as its own pull request and name
the change that introduced it.
- When you change a workflow file or a `scripts/ci/` module, grep the whole `tests/` tree
for every literal you touched — event-type strings, cron expressions, environment-variable
names, tuple members, pinned digests — not only the obviously named sibling test. A change
can satisfy one oracle and still leave a second, independent one stale.
- Read a stale pull request's own changes with a three-dot diff —
`git diff <base>...<head>` — or with `gh pr diff`, which is already three-dot. A two-dot
`git diff <base> <head>` renders everything the base gained since the fork point as though
this branch deleted it, so an untouched branch reads as a mass revert.
- Content-hash pins exist under `tests/`; find them before editing a workflow. Run
`grep -rn 'hash-object' tests/` — today that is the `git hash-object` pin of
`.github/workflows/opencode-review-dispatch.yml`. Any byte change to a pinned file makes
its constant stale and fails a required gate for every open pull request, reverts included,
because a revert restores the original bytes while the pin stays on the reverted value.
Recompute only with `git hash-object <path>`, and only for a constant you have confirmed is
a blob pin. Nearly every other forty-hex literal under `tests/` is something else — a
pinned action SHA, a vendored-revision pin, a synthetic fixture head, or an assertion that
a SHA appears in a document — and pointing `hash-object` at any of those produces a wrong
value that breaks what it replaces. A second contract re-derives the dispatch pin by
regular expression from the first, so keep the assignment on one line and correct it in one
place.
- Production code under `scripts/ci/` branches on `GITHUB_ACTIONS`, and pytest inherits that
variable in CI, so a failure class exists that cannot reproduce locally. Before calling a
scheduler change clean, run the affected tests both ways, including
`GITHUB_ACTIONS=true python3 -m pytest <paths>`.
23 changes: 23 additions & 0 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,3 +196,26 @@ repeatable compile command.
longer than it is. Querying `status=success` and `status=failure` directly cuts through the churn
to the most recent real conclusion of each kind. Those are historical signals about pipeline
liveness only — they never substitute for exact-current-head evidence on the PR you are acting on.
- **Do not assume `interrogate` skips private helpers.** `[tool.interrogate]` here sets no
`ignore-*` flags and the tool defaults them off, so a docstring-less `_helper` or `__helper` in
`scripts/ci/` counts against the 100% gate — it is the stricter docstring check, not the laxer
one. Sibling repositories configure this differently (`contextual-orchestrator` enables six
`ignore-*` flags and does skip them), so read the target repo's `pyproject.toml` rather than
carrying a docstring habit across repositories. Note also that `ignore-private` would cover only
double-underscore names; single-underscore needs `ignore-semiprivate`.
- **A stale PR's conflict scope is a snapshot, not a property of the PR.** Any advance of the base
between measuring the conflicts and resolving them invalidates the list, and base advances land in
the same directories conflicts do (`.github/workflows/`, `scripts/ci/`, `docs/doctoring/`). Scope
grows as often as it shrinks — a branch that merged cleanly can become conflicted with no change
to the branch at all — so re-run the merge yourself immediately before resolving and treat any
earlier measurement, including your own from minutes ago, as expired. Resolving against a stale
smaller scope silently leaves conflicts unhandled.
- **No test parses fenced code blocks.** The doc-contract tests match exact prose in specific files;
none of them check Markdown structure, and `ARCHITECTURE.md` (five mermaid diagrams) is read by no
test at all. A conflict resolution that splits a fenced block into two fragments therefore ships
green, rendering the diagram source as a plain code block. After resolving a conflict in a
document containing fenced blocks, re-read the whole enclosing section rather than the diff hunk,
and confirm each block has one opening fence carrying its language tag and one matching closing
fence. Do not check by counting fences — a split leaves four where there were two, so an even
count proves nothing. The damage can also arrive inherited, from an earlier commit on the same
branch or from the autofix flow's conflict-marker resolution.
Loading