docs: add governance shell and close documentation drift gaps

.github/pull_request_template.md

@@ -0,0 +1,22 @@
+# Pull request
+
+## What changed
+
+<!-- Why this change exists. The diff already shows what; explain why. -->
+
+## How it was verified
+
+<!-- The commands you actually ran, and what they printed. -->
+
+## Checklist
+
+Tick each item, or replace it with one line saying why it does not apply. These
+mirror the gates in `pr.yml` and the expectations in `CONTRIBUTING.md`.
+
+- [ ] Scenario catalog (`scenarios/catalog.yml`) updated, or this change adds or removes no scenarios.
+- [ ] Matrix regenerated with `python3 tools/check-scenarios.py --write-matrix`, and the README section 5 embed matches it.
+- [ ] Every command I changed in the documentation was executed, not just edited.
+- [ ] Documentation updated for this change type, per the lifecycle table in [`docs/README.md`](../docs/README.md).
+- [ ] Workflow changes under `.github/workflows/` are mirrored in `docs/ci-workflows.md` in this pull request.
+- [ ] No credentials or tokens, private endpoints, unredacted traces or screenshots, or real user data, per the secrets and captures rules in `CONTRIBUTING.md`.
+- [ ] ADR added under `docs/adr/`, or this decision is not durable enough to need one.

.github/workflows/pr.yml

@@ -66,6 +66,43 @@ jobs:
       - name: Lint Markdown
         run: git ls-files -z '*.md' | xargs -0 npx --yes markdownlint-cli2@0.23.0
 
+      - name: Check Markdown links
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: >-
+          git ls-files -z '*.md' | xargs -0
+          docker run --rm
+          -v "$PWD:/repo:ro"
+          -w /repo
+          -e GITHUB_TOKEN
+          lycheeverse/lychee:0.24.2@sha256:e2d19e57cf6ab037026f20b8e449a1f30d9d7f81eef4194763aab2eab20bd28d
+          --no-progress
+          --exclude-loopback
+          --max-retries 3
+          --retry-wait-time 2
+          --timeout 20
+
+      - name: Validate README fast start
+        run: python3 tools/check-fast-start.py
+
+      - name: Require CI guide updates with workflow changes
+        if: >-
+          github.event_name == 'pull_request' &&
+          !contains(github.event.pull_request.title, '[ci-guide-exempt]')
+        env:
+          BASE_SHA: ${{ github.event.pull_request.base.sha }}
+          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+        run: |
+          changed="$(git diff --name-only "$BASE_SHA"..."$HEAD_SHA")"
+          if printf '%s\n' "$changed" | grep -q '^\.github/workflows/' &&
+             ! printf '%s\n' "$changed" | grep -qx 'docs/ci-workflows.md'; then
+            echo "::error file=docs/ci-workflows.md::This pull request changes .github/workflows/ but not docs/ci-workflows.md"
+            echo "docs/ci-workflows.md mirrors the workflows at job and step granularity, and nothing else keeps it honest."
+            echo "Update it in this pull request. If the change has no semantic effect on the guide, add [ci-guide-exempt] to the pull request title."
+            exit 1
+          fi
+          echo "CI guide sync ok"
+
   scenario-catalog:
     name: "Scenario catalog · drift check"
     runs-on: ubuntu-24.04

.lycheeignore

@@ -0,0 +1,17 @@
+# Exclusions for the link check in the repo-hygiene job of
+# .github/workflows/pr.yml. One regular expression per line.
+#
+# Only genuinely unverifiable or volatile targets belong here. Everything else,
+# including every relative link between documents, is checked strictly: a
+# broken relative link must fail the pull request.
+
+# The public demo app under test. Its availability is upstream and outside this
+# repository's control, so an outage there must never fail an unrelated pull
+# request. The pinned container image, not this host, is what CI actually runs
+# against.
+https://the-internet\.herokuapp\.com
+
+# Workflow status badges. The badge endpoint answers per request and is prone to
+# rate limiting and redirects, which would make the gate flaky. The Actions
+# links rendered beside each badge point at the same workflows and are checked.
+https://github\.com/jsugg/the-internet-tests/actions/workflows/[^/]+/badge\.svg

CONTRIBUTING.md

@@ -0,0 +1,250 @@
+# Contributing
+
+This is a teaching repository: the tests, the catalog, and the CI wiring are the
+product. A change is finished when someone reading the documentation can
+reproduce it, not when it passes locally. Read [`README.md`](README.md) for what
+the repo is and is not before proposing new scope.
+
+## Repository setup
+
+Start the demo app under test from the repository root. Every stack runs against
+it on `http://localhost:7080`:
+
+```bash
+docker compose -f docker/compose.yml up -d website
+```
+
+Set up only the stacks you are changing. Each stack is self-contained and runs
+from its own directory:
+
+```bash
+cd stacks/ts-playwright
+npm ci
+npx playwright install --with-deps chromium
+```
+
+```bash
+cd stacks/python-playwright
+python3 -m venv .venv
+source .venv/bin/activate
+python -m pip install --upgrade pip
+python -m pip install -e ".[dev]"
+python -m playwright install --with-deps chromium
+```
+
+The Java stack needs no install step; Maven resolves dependencies on first run,
+and Selenium Manager provisions the browser drivers.
+
+Stop the app from the repository root when you are done:
+
+```bash
+docker compose -f docker/compose.yml down
+```
+
+Each stack README is the authority on running that stack:
+[Java](stacks/java-selenium-testng/README.md),
+[TypeScript](stacks/ts-playwright/README.md),
+[Python](stacks/python-playwright/README.md).
+
+## Fast validation loop
+
+Run these two from the repository root before every push. They are the same
+checks the `repo-hygiene` and `scenario-catalog` PR jobs run, and they need no
+browsers and no app:
+
+```bash
+python3 tools/check-scenarios.py
+git ls-files -z '*.md' | xargs -0 npx --yes markdownlint-cli2@0.23.0
+```
+
+Then run the static checks for the stacks you touched:
+
+```bash
+mvn -B -f stacks/java-selenium-testng/pom.xml clean test-compile
+```
+
+```bash
+cd stacks/ts-playwright
+npm run typecheck
+npm run lint
+npm run format:check
+```
+
+```bash
+cd stacks/python-playwright
+ruff check .
+ruff format --check .
+mypy src tests
+```
+
+If you changed a workflow, lint it with the same pinned image CI uses, from the
+repository root:
+
+```bash
+docker run --rm -v "$PWD:/repo:ro" -w /repo rhysd/actionlint:1.7.12 -color
+```
+
+## Adding or changing a scenario
+
+The catalog is the canonical list of scenarios; the tests reference it, and
+`tools/check-scenarios.py` reconciles the two in both directions. A scenario that
+is not in the catalog is not covered, no matter how many tests exist. Work in
+this order:
+
+1. Add or edit the row in [`scenarios/catalog.yml`](scenarios/catalog.yml). Give
+   it a unique ID, a priority, a title, and a `coverage:` entry per stack.
+2. Add the tests, referencing the scenario ID. Java requires the ID in the
+   TestNG `testName="…"` attribute; TypeScript and Python carry it in the test
+   title or marker.
+3. Set `coverage:` to `true` only for the stacks that actually have a test. The
+   checker fails in both directions: a catalog row marked covered with no test,
+   and a test whose ID is missing or not marked covered.
+4. Regenerate the matrix and sync the README embed from the repository root:
+
+   ```bash
+   python3 tools/check-scenarios.py --write-matrix
+   ```
+
+Never hand-edit [`docs/scenario-matrix.md`](docs/scenario-matrix.md); it is
+generated, and the `scenario-catalog` job fails on any drift.
+
+## Commit conventions
+
+Commits are linted by commitlint in CI against
+[`commitlint.config.cjs`](commitlint.config.cjs), which is the authority. The
+rules:
+
+- Conventional commits: `type(scope): subject`.
+- Type is one of `feat`, `fix`, `docs`, `ci`, `build`, `refactor`, `test`,
+  `chore`, `perf`, lower-case.
+- Scope is optional, lower-case, and if present must be one of `java`, `ts`,
+  `py`, `catalog`, `docker`, `actions`, `deps`, `readme`. There is deliberately
+  no `docs` or `ci` scope: use a scopeless `docs: …` for governance docs and
+  `ci(actions): …` for workflow changes.
+- The header is at most 72 characters, and the subject takes no trailing period.
+- Body and footer lines are at most 100 characters.
+
+Explain why in the body, not what; the diff already shows what. Do not add
+attribution trailers, and do not name tooling vendors or assistants in branch
+names, commit messages, or pull requests.
+
+## Pull requests
+
+Fill in the [pull request template](.github/pull_request_template.md); it is the
+change checklist, not a formality. Squash-merge is the norm, so the PR title
+becomes the squashed subject and must satisfy the same commitlint rules.
+
+Every pull request runs the always-on fast gate (`pr.yml`): repository hygiene,
+scenario-catalog drift, and a compile-plus-smoke slice per stack. Regression
+workflows are path-filtered and only run when your change touches their stack.
+See [`docs/ci-workflows.md`](docs/ci-workflows.md) for the full trigger fan-out.
+
+## Secrets and captures
+
+This is a teaching repository with no deployed service, no users, and no data of
+its own, so the realistic risk here is not an exploit. It is committing something
+that should never have been public, into a repository that is public on purpose.
+
+- **Never commit a credential or token.** Not an API key, a provider token, a
+  session cookie, or a private key — including expired and throwaway ones. Secret
+  scanning and push protection are enabled on this repository, so a detected
+  secret blocks the push. Treat that as the backstop, not the plan.
+- **Point the stacks at the demo app, not at a real system.** The moment a stack
+  runs against something real, its traces and screenshots inherit that system's
+  secrets. Use the pinned container on `http://localhost:7080`.
+- **Redact captures before attaching them.** A Playwright trace records request
+  headers in full. Open a trace, screenshot, or log before you put it in a pull
+  request.
+- **Keep `artifacts/` out of Git.** It is git-ignored staging for CI upload, and
+  CI artifacts on a public repository are downloadable by anyone.
+- **Keep remote-grid credentials outside the repository**, injected only through
+  a private runner or a maintainer-approved workflow secret, as the
+  [Java stack README](stacks/java-selenium-testng/README.md) says. No
+  checked-in workflow needs one.
+
+The demo app's `tomsmith` / `SuperSecretPassword!` login is a published fixture
+of a public teaching app, not a secret. It belongs in the tests.
+
+## Documentation compliance
+
+A change is documentation-compliant when all five hold:
+
+1. A new contributor can reproduce the affected behavior from the documented
+   commands, without reading the diff.
+2. Canonical documentation agrees with the executable configuration it
+   describes.
+3. Cross-stack design changes are recorded, so the reasoning outlives the PR.
+4. Documentation responsibility for the change is identifiable.
+5. No required documentation validation fails.
+
+## Documentation expectations
+
+Update the documentation in the same pull request as the change. What to update
+depends on what you changed. [`docs/README.md`](docs/README.md) carries the full
+lifecycle table — every document's audience, update trigger, and review cadence —
+plus the authority order to apply when two sources disagree, and the
+documentation conventions to follow when writing: terminology, headings, dates,
+shell assumptions, source references, diagrams, and generated files.
+
+| Change | Update |
+| --- | --- |
+| A workflow under `.github/workflows/` | [`docs/ci-workflows.md`](docs/ci-workflows.md), in the same pull request. CI enforces this. |
+| A scenario or the catalog | Regenerate `docs/scenario-matrix.md` and sync the README §5 embed. |
+| A stack's commands, runtime, or reports | That stack's README, and the root README stack matrix if the headline command changed. |
+| Retry, tagging, quarantine, or artifact policy | [`docs/flakiness-guide.md`](docs/flakiness-guide.md). |
+| A durable cross-stack, CI, or catalog decision | A new ADR under `docs/adr/`. |
+| Scope, quick start, or supported stacks | [`README.md`](README.md). |
+
+## Waivers
+
+A waiver is a decision not to meet a requirement, written down so it stays a
+decision instead of decaying into an oversight. The difference matters: an
+undocumented gap looks identical to a mistake, and the next reader cannot tell
+whether it was considered or forgotten.
+
+To record one, copy
+[`docs/templates/waiver-template.md`](docs/templates/waiver-template.md), fill
+every field, and add it to the register below. A waiver carries the requirement,
+its scope, the reason, the owner, the approver, the revisit trigger, and a
+tracking issue.
+
+Two rules keep this from becoming a way to opt out of anything inconvenient:
+
+- **The reason must say why the requirement is wrong *here*, not that meeting it
+  is expensive.** Cost alone is a backlog item, not a waiver.
+- **The revisit trigger must be an observable event, not a date.** "First
+  sustained external contributor" is something anyone can check; "later" and
+  "Q3" are not.
+
+A waiver is not permanent, and it is not an argument. When its trigger fires, the
+requirement is back on the table.
+
+### Recorded waivers
+
+Requirements this repository deliberately does not meet, with the condition that
+would reopen each one. Both exist because this is a single-maintainer repository;
+neither is an oversight.
+
+#### W-001 — `CODEOWNERS` omitted
+
+| Field | Value |
+| --- | --- |
+| Requirement | Code-owner review via a `CODEOWNERS` file |
+| Scope | Whole repository |
+| Reason | The only owner entry possible today is `* @jsugg`. Code-owner review would then demand an approval that the sole maintainer cannot give on their own pull requests, deadlocking every change. |
+| Owner | @jsugg |
+| Approver | @jsugg |
+| Revisit trigger | First sustained external contributor |
+| Tracking issue | None; repository issues are disabled, so the trigger above is the record |
+
+#### W-002 — Reviewer-assignment machinery omitted
+
+| Field | Value |
+| --- | --- |
+| Requirement | Owner maps, required reviewers, and automatic reviewer assignment |
+| Scope | Whole repository |
+| Reason | There is one maintainer and no reviewer pool to route to. Required reviewers would block every pull request for the same reason as W-001, and an owner map would encode a single name already visible in the repository metadata. |
+| Owner | @jsugg |
+| Approver | @jsugg |
+| Revisit trigger | First sustained external contributor |
+| Tracking issue | None; repository issues are disabled, so the trigger above is the record |

README.md

@@ -54,7 +54,9 @@ Then use the stack READMEs and guides for deeper runs:
 
 ## 5. Scenario matrix
 
-Embedded from [`docs/scenario-matrix.md`](docs/scenario-matrix.md), generated by `tools/check-scenarios.py --write-matrix`.
+Embedded from [`docs/scenario-matrix.md`](docs/scenario-matrix.md), generated by `tools/check-scenarios.py --write-matrix`. This copy is checked against the catalog in CI, so edit the catalog and regenerate rather than editing the table below.
+
+Legend: ✅ = covered by that stack · — = not covered.
 
 | ID | Priority | Scenario | Java | TS | Python |
 | --- | --- | --- | --- | --- | --- |
@@ -137,3 +139,8 @@ CI stages report files under `artifacts/<stack>/<run-id>/<slice>/` before upload
 | TypeScript full gate | [![TypeScript](https://github.com/jsugg/the-internet-tests/actions/workflows/ts.yml/badge.svg?branch=master)](https://github.com/jsugg/the-internet-tests/actions/workflows/ts.yml?query=branch%3Amaster) |
 | Python full gate | [![Python](https://github.com/jsugg/the-internet-tests/actions/workflows/python.yml/badge.svg?branch=master)](https://github.com/jsugg/the-internet-tests/actions/workflows/python.yml?query=branch%3Amaster) |
 | Nightly matrix | [![Nightly](https://github.com/jsugg/the-internet-tests/actions/workflows/nightly.yml/badge.svg?branch=master)](https://github.com/jsugg/the-internet-tests/actions/workflows/nightly.yml?query=branch%3Amaster) |
+
+## 9. Contributing
+
+- [Contributing guide](CONTRIBUTING.md) — setup, the fast validation loop, the scenario-catalog workflow, commit and pull-request conventions, and the recorded waivers.
+- [Documentation index](docs/README.md) — every document, its audience and update trigger, and which source of truth wins when two disagree.