Skip to content

docs: trim AGENTS.md to rules no tool enforces and drop vendor names - #624

Merged
leehack merged 9 commits into
mainfrom
docs/agents-md-trim
Sep 23, 2026
Merged

leehack merged 9 commits into
mainfrom
docs/agents-md-trim

Conversation

@leehack

@leehack leehack commented Sep 23, 2026 •

Copy link
Copy Markdown
Owner

Summary

AGENTS.md loads into every agent session. This trims it from 406 lines (~5.0k tokens) to 159 (~2.3k) and removes vendor names.

  • Kept: rules that change agent behavior and that no lint, test or CI job enforces. Duplicates are merged and each rule is shorter.
  • Dropped: rules that tooling already fails on with a clear message (e.g. prepare_workspace.dart package checks, the coverage gate, broken docs links), generic design advice, the literal Flutter version and incident history.
  • Moved: procedures stay in the docs AGENTS.md now points to (doc/testing_matrix.md, doc/high_risk_pre_merge_readiness.md, native-and-web-sync.md, release-workflow.md, doc/webgpu_bridge.md). Every moved rule was already in its destination doc.
  • Added: check_platform_boundaries.dart in the gate block (a CI gate the old list left out), litert-lm-native in the owner list, and a note that the tooling tests pin phrases in this file.
  • Restored after audit: naming conventions (directories are not linted), useful Dartdoc, conditional imports for platform backends, the two-mode verify_release_docs_versions.dart run before install-snippet changes, validation before the final commit of a cross-repo change, the owner-repo commit-and-push step, the source-wide TODO/FIXME rule, the obvious-type condition on local inference, and explicit parameter, return and field types on all public API (not only new API).
  • Vendor-neutral auditor: AGENTS.md, CONTRIBUTING.md and doc/testing_matrix.md now describe the high-risk auditor as "an operator or a fresh agent session that took no part in the implementation, never the PR author". doc/high_risk_pre_merge_readiness.md says which existing audit_kind each records. The schema and evaluator are unchanged.

Production-readiness scope

  • User-facing scope: contributor and agent guidance only.
  • Supported platforms/paths: docs-only.
  • Unsupported or intentionally unavailable paths: N/A.
  • Out of scope / follow-ups: CI skips the tests that pin this wording on docs-only PRs: #623. Audit findings left as is (one P2, five P3s) are listed in the High-risk block.

Completeness checklist

  • Declared scope is fully implemented, or explicitly reduced above.
  • Unsupported platform/option combinations fail loudly with actionable diagnostics or are clearly documented as unavailable.
  • Public API docs, README/website docs, examples, support matrices, and changelog entries are updated where relevant.
  • Regression coverage covers the original issue plus key negative/version-skew paths where relevant.
  • Security/privacy review completed: no secrets, bearer tokens, signed URLs, or raw secret-bearing paths leak through logs, cache keys, metadata, errors, or snapshots.
  • Follow-up work that is useful but not required for this PR is tracked in GitHub Issues, or explicitly marked "None" above.

PR type guidance

  • Docs-only PR: no runtime behavior changes. No changelog entry: contributor guidance only.

Test Plan

  • dart run tool/prepare_workspace.dart
  • dart format --output=none --set-exit-if-changed .
  • dart analyze
  • dart run tool/testing/check_platform_boundaries.dart
  • dart test -p vm -j 1 --exclude-tags local-only: N/A for docs; the suites that read these files ran (below).
  • dart test -p chrome --exclude-tags local-only: N/A, no Dart change.
  • Other:
    • All at 0252bf820: dart test -p vm test/unit/tooling/ test/unit/test_structure/ test/unit/hook/native_release_tag_test.dart: 568 passed, 4 skipped.
    • verify_release_docs_versions.dart and check_webgpu_bridge_tag.dart: pass. verify_release_docs_versions.dart --release-prep: fails on the pending llamadart_llama_cpp_flutter bump from #630; main at ae7b74528 fails with the same output, so it is not caused by this PR.
    • Mutations at 0252bf820: 15 of 20 fail their test (high_risk_review_policy_test, safe_pr_head_update_test, verify_release_docs_companion_pins_test): pinned phrases in all four files, and deleting the whole AGENTS.md Pull requests section. The native-tag ones also fail verify_release_docs_versions.dart. The 5 survivors pass the three suites above, verify_release_docs_versions.dart and check_webgpu_bridge_tag.dart: the audit_kind sentence in the runbook and the four AGENTS.md sentences fixed after audit (TODO/FIXME scope, owner-repo push step, local inference, explicit types on all public API). No test or tool reads them, so the audits checked them.
    • Every backticked repo path in AGENTS.md exists.

Matrix Evidence

Matrix row Scope covered Platform / model / backend Result Evidence / notes
static-format-analyze format, analyzer, boundaries macOS, Flutter 3.47.1 PASS local, 0252bf820
tooling policy tests wording pins in AGENTS.md, CONTRIBUTING.md, doc/ VM PASS 568 passed; 15/20 mutants killed
release-doc-version-consistency native tag grammar pins macOS PASS (default); FAIL (--release-prep) --release-prep fails the same way on main (pending companion bump from #630); not PR-caused
webgpu-bridge-tag-consistency no stray pins macOS PASS
root-vm / root-chrome N/A no Dart change

Review Notes

  • Independent review status: every old rule was mapped to keep, drop-enforced, move or stale; an adversarial review found 1 P1 (the rule that a moved head voids earlier evidence was lost) and several P2s (changed meanings), all fixed before the PR opened.
  • CI status / head SHA: 0252bf820fdf539c397cc1062738f4f0ba7b2e59: 8 checks pass, 14 skipped (docs-only selection; see #623).

The maintainer confirmed audit_kind: operator-owned for a fresh non-Codex agent session.

High-risk regression review

  • Classification: high-risk (classify_high_risk_changes.dart: regressionPolicy)
  • Implementation task: this session's AGENTS.md workflow
  • Independent blocking QA task: fresh sessions run by the operator, no prior reports. Through a214f62ba each mapped every old rule; later rounds are blocking-only. Through b7519af2c: Codex (gpt-6-astra, reasoning effort xhigh, per their launch headers). From a214f62ba: fresh Claude workflow subagents (claude-opus-5-5), recorded as operator-owned.
    • cdd9b150d: ACCEPT, 0 P1/P2, five P3s; four fixed in 1251a2f96, the fifth in 8e903d5cd.
    • 1251a2f96: ACCEPT, one P2 (weaker two-mode release-docs check) and one P3 (validation timing); both fixed in 8e903d5cd.
    • 8e903d5cd: ACCEPT, 0 P1/P2. One P3 kept: the naming line partly repeats lints, which don't fail CI outside lib/.
    • 454f99d8b (merge of #606): ACCEPT, 0 P1. P2: this body named the old pair (fixed here). P3: owner-repo commit/push step no longer explicit.
    • 4f07e3334 (merge of #616): ACCEPT, 0 P1. P2: the stale body pair again (fixed here). P3s: that push step and the narrowed TODO/FIXME scope; both fixed in b7519af2c.
    • b7519af2c: ACCEPT, 0 P1/P2. P3: local inference lost "when the type is obvious"; fixed in a214f62ba.
    • a214f62ba (Claude): ACCEPT, 0 P1/P2, 130 rules mapped; merge ancestry, four-file diff and the #606/#616 content checked. P3-1 (explicit types only on new public API) fixed in 331217e15; P3-2 (classifier-only high-risk trigger) kept.
    • 331217e15 (Claude, blocking-only): ACCEPT, 0 P1/P2. P3-2 again, and a P3: the release-prep publish boundary and the advisory-check rule now live only in linked docs. Voided when main moved (#630).
    • Exact head 0252bf820 (merge of #630; Claude, audit:624i, blocking-only): ACCEPT, 0 P1, one P2, five P3s. It checked the four-file diff, the head/base refs at start and end, and AGENTS.md against main's v0.4.1-1 native and bridge tag rules. Findings below.
  • Exact head / current base: 0252bf820fdf539c397cc1062738f4f0ba7b2e59 / main at ae7b745280321a41d78c9d5bd2600d7e08f79570 (merge base)
  • Readiness evaluator: high_risk_readiness.dart on the exact pair, bindings read live from GitHub: exit 2, unverifiedPrerequisites (the local ceiling). The evidence cites the three tests above with their exact-head mutants and the 0252bf820 audit as operator-owned.
  • Fixed after audit: P3-1 (a214f62ba) in 331217e15: AGENTS.md again asks for explicit parameter, return and field types on all public API.
  • Known non-blocking (left as is; the 0252bf820 verdict is ACCEPT):
    • P2 (was P3-2): AGENTS.md names only the path-based classifier as the high-risk trigger, not the kinds of change. The classifier returns standard for lib/src/core/decision/decision_engine.dart (capability probe, backend gating). The list by meaning stays in CONTRIBUTING.md and the PR template, which AGENTS.md requires filling. The auditor suggests one AGENTS.md line or a wider classifier.
    • P3: the rule that release-prep PRs do not publish (merge is the approval boundary) and the post-release checklist are no longer in AGENTS.md; they are in the linked release-workflow.md.
    • P3: AGENTS.md no longer names the retired qa identity; the evaluator rejects it (retiredQaProfileProhibited), and CONTRIBUTING.md and the docs still prohibit it.
    • P3: supplying repository, PR, author, head and base "from an independent source" is no longer in AGENTS.md; it is in the linked readiness doc and doc/testing_matrix.md.
    • P3: no doc names check_webgpu_bridge_tag.dart now; CI still runs it, including on docs-only PRs, and it passes on head and main.
    • P3: the runbook's audit_kind sentence matches the schema and evaluator, which check only the enum, so the mapping is procedure, not code.
  • Main-state condition (not PR-caused): verify_release_docs_versions.dart --release-prep fails on the pending llamadart_llama_cpp_flutter bump from #630, with the same output and exit 1 on main at ae7b74528.
  • Production-branch deletion, bypass, or miswire proof: N/A, no production code; the wording pins are mutation-checked at the exact head (Test Plan).
  • Affected-family real-model/artifact evidence: N/A, docs-only.
  • Explicit unavailable-family or other N/A evidence: N/A.
  • Known PR-caused P1 regressions: 0
  • Unresolved review threads: 0

AGENTS.md loads into every agent session. Keep only rules that change
behavior and that no lint, test or CI job already enforces; point to the
docs that hold procedures. Describe the independent high-risk auditor
without naming a vendor, in AGENTS.md, CONTRIBUTING.md and
doc/testing_matrix.md, and say which audit_kind each auditor records.
@leehack
leehack marked this pull request as ready for review September 23, 2026 20:36
@leehack
leehack merged commit 7556d26 into main Sep 23, 2026
22 checks passed
@leehack
leehack deleted the docs/agents-md-trim branch September 23, 2026 21:56
@github-actions

Copy link
Copy Markdown
Contributor

Chat app preview removed for leehack/llamadart-chat-pr-624.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant