diff --git a/AGENTS.md b/AGENTS.md index 6c24b9f..886e698 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -44,7 +44,7 @@ uvx ruff check --fix . && uvx ruff format . # lint and format ## Testing instructions -The repo has no test suite and no CI. Until one exists, a change is done when all of this passes: +One check runs in CI: `.github/workflows/agents-md-readiness.yml` scores this file on any pull request that touches it, and fails below 0.85 readiness or on a line that looks like a credential. The repo has no test suite, so a change is done when all of this passes: 1. `uv run python -m py_compile ` on every Python file you touched. 2. `uvx ruff check .` and `uvx ruff format --check .` both report clean. @@ -52,7 +52,7 @@ The repo has no test suite and no CI. Until one exists, a change is done when al 4. Any output shown in a README comes from a run you just did, with the date next to it. 5. `samples/agents-md-readiness/go/` compiles that sample into one static binary, so a change to its `questions.yml` or its Go files means `gofmt -l .`, `go vet ./...`, `go test ./...` and `./build.sh ` in that folder, then one live run of the binary you built. The Python stays canonical, and `payload_test.go` fails when the embedded copy of the payload drifts. -If you add a test suite, use pytest, mock the client rather than calling the API, and replace this section. +Run the whole list before you open a pull request, because CI covers the AGENTS.md check and nothing else. If you add a test suite, use pytest, mock the client rather than calling the API, and wire it into that workflow. ## Code style diff --git a/samples/agents-md-readiness/README.md b/samples/agents-md-readiness/README.md index 256f39c..1ef41e6 100644 --- a/samples/agents-md-readiness/README.md +++ b/samples/agents-md-readiness/README.md @@ -92,7 +92,28 @@ curl -fsSL https://raw.githubusercontent.com/aarora79/jev-samples/main/samples/a agents-md-readiness -fail-under 0.8 -fail-on-credential AGENTS.md ``` -Exit codes are 0 scored, 1 error, 2 gate failed. [0.1.0](https://github.com/aarora79/jev-samples/releases/tag/agents-md-readiness/0.1.0) ships binaries for linux and macOS on amd64 and arm64, plus windows amd64, and the installer checks them against the published `SHA256SUMS`. The Python here stays canonical, and a test in that folder fails when its copy of the payload drifts. [go/README.md](go/README.md) covers building, releasing and the caveats. +Exit codes are 0 scored, 1 error, 2 gate failed. [0.2.0](https://github.com/aarora79/jev-samples/releases/tag/agents-md-readiness/0.2.0) ships binaries for linux and macOS on amd64 and arm64, plus windows amd64, and the installer checks them against the published `SHA256SUMS`. The Python here stays canonical, and a test in that folder fails when its copy of the payload drifts. [go/README.md](go/README.md) covers building, releasing and the caveats. + +### This repo runs the check on itself + +[`.github/workflows/agents-md-readiness.yml`](../../.github/workflows/agents-md-readiness.yml) installs that release and scores this repo's AGENTS.md on any pull request touching it, failing below 0.85 readiness or on a credential reading `suspect` or worse. Copy it, change the bar, and the check is yours. + +```yaml +- run: curl -fsSL https://raw.githubusercontent.com/aarora79/jev-samples/main/samples/agents-md-readiness/go/install.sh | sh + env: + VERSION: "0.2.0" + BINDIR: ${{ runner.temp }}/bin + +- run: ${{ runner.temp }}/bin/agents-md-readiness -fail-under 0.85 -fail-on-credential -json report AGENTS.md + env: + TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }} +``` + +Three details the workflow settles, each from a run rather than a guess: + +- **The bar leaves room for the model.** This file scores 0.92 with a spread of 0.01 across repeat runs, so 0.85 passes today and still fails a real regression. On a GitHub runner the check took 641 ms and 3,109 input tokens. +- **A failing gate has to fail the job.** `set -o pipefail` keeps the exit code through the `tee` that feeds the job summary. A throwaway commit raising the bar to 0.99 failed with `readiness 0.92 is under the 0.99 bar` and exit code 2, and the summary and the JSON artifact still appeared, because both steps run under `always()`. +- **A fork pull request gets no secrets.** The job checks for the key first and writes a line in the summary saying it skipped, instead of failing on a missing key and teaching everyone to ignore a red check. ## What it asks diff --git a/samples/agents-md-readiness/go/README.md b/samples/agents-md-readiness/go/README.md index 34915f4..3a06a7f 100644 --- a/samples/agents-md-readiness/go/README.md +++ b/samples/agents-md-readiness/go/README.md @@ -10,7 +10,7 @@ agents-md-readiness -fail-under 0.8 https://github.com/apache/airflow ## Install it -[0.1.0](https://github.com/aarora79/jev-samples/releases/tag/agents-md-readiness/0.1.0) carries binaries for linux and macOS on amd64 and arm64, plus windows amd64: +[0.2.0](https://github.com/aarora79/jev-samples/releases/tag/agents-md-readiness/0.2.0) carries binaries for linux and macOS on amd64 and arm64, plus windows amd64: ```bash curl -fsSL https://raw.githubusercontent.com/aarora79/jev-samples/main/samples/agents-md-readiness/go/install.sh | sh @@ -39,7 +39,7 @@ go build -o agents-md-readiness . `build.sh` cross-compiles every platform `install.sh` knows about from whichever machine you run it on, and writes `SHA256SUMS` beside the binaries: ```bash -./build.sh 0.1.0 +./build.sh 0.2.0 ``` That produced five binaries on 19 September 2026, 6.4M to 7.2M each, in 4.4 seconds on this machine: linux amd64 and arm64, darwin amd64 and arm64, windows amd64. `-trimpath` drops the build paths, and Go stamps the git revision and a dirty flag into the binary, so the same commit in a clean tree reproduces the same hashes and a dirty tree does not. `go version -m ` prints both. The release assets came from a fresh clone of the tagged commit, which is why their hashes differ from a build in a working tree that has edits. @@ -147,17 +147,19 @@ Build from a clean clone of the commit you are tagging, so the revision Go stamp ```bash git clone --depth 1 https://github.com/aarora79/jev-samples.git /tmp/relbuild cd /tmp/relbuild/samples/agents-md-readiness/go -./build.sh 0.1.0 -gh release create agents-md-readiness/0.1.0 dist/* \ +./build.sh 0.2.0 +gh release create agents-md-readiness/0.2.0 dist/* \ --repo aarora79/jev-samples \ --target "$(git rev-parse HEAD)" \ - --title "agents-md-readiness 0.1.0" \ + --title "agents-md-readiness 0.2.0" \ --notes "Static binaries for linux, macOS and Windows." ``` -Versions are plain [semver](https://semver.org), `0.1.0` rather than `v0.1.0`, in the tag, the asset names and what `-version` prints. The tag carries the `agents-md-readiness/` prefix because `install.sh` resolves the newest tag with that prefix, which leaves room for another sample to ship its own binary. `dist/` is gitignored: the release holds the binaries and the repo holds the source. +Versions are plain [semver](https://semver.org), `0.2.0` rather than `v0.2.0`, in the tag, the asset names and what `-version` prints. The tag carries the `agents-md-readiness/` prefix because `install.sh` resolves the newest tag with that prefix, which leaves room for another sample to ship its own binary. `dist/` is gitignored: the release holds the binaries and the repo holds the source. -0.1.0 came out of `3bfae70` on 19 September 2026, built in a clean clone, and every asset stamps that revision with `vcs.modified=false`. The install path ran end to end from an empty directory: `install.sh` resolved 0.1.0 off the tag, downloaded the linux amd64 asset, printed `checksum ok`, and installed a binary that reports `agents-md-readiness 0.1.0`. It then scored `langchain-ai/langchain` in 385 ms for 5,723 input tokens and exited 2 on `microsoft/vscode` under `-fail-under 0.8`. +0.2.0 came out of `2d28560` on 19 September 2026, built in a clean clone, and every asset stamps that revision with `vcs.modified=false`. It carries the deadband; 0.1.0, cut from `3bfae70`, does not, which is why the minor version moved rather than the patch. The install path ran end to end from an empty directory: `install.sh` resolved 0.2.0 off the tag, downloaded the linux amd64 asset, printed `checksum ok`, and installed a binary that reports `agents-md-readiness 0.2.0`. + +This repo's own pull request check installs that release and scores this repo's AGENTS.md, so [the workflow](../../../.github/workflows/agents-md-readiness.yml) is the worked example: `checksum ok`, readiness 0.93, 3,109 input tokens, 641 ms on a GitHub runner. Raising its bar to 0.99 on a throwaway commit failed the job with `readiness 0.92 is under the 0.99 bar` and exit code 2. ## What to notice