From 22f9227d38620f60dcda7de047196de333b1c70e Mon Sep 17 00:00:00 2001 From: Chizoba Jude Ojini Date: Thu, 29 Oct 2026 10:33:05 +0000 Subject: [PATCH] robots.txt: allow search, block AI training crawlers; noindex the pages.dev addresses --- AGENTS.md | 1 + apps/web/public/_headers | 7 +++ apps/web/public/robots.txt | 58 ++++++++++++++++++++++++ apps/web/src/crawlers.test.ts | 83 +++++++++++++++++++++++++++++++++++ docs/deploy.md | 8 ++++ scripts/nginx-conf.ts | 4 ++ 6 files changed, 161 insertions(+) create mode 100644 apps/web/public/robots.txt create mode 100644 apps/web/src/crawlers.test.ts diff --git a/AGENTS.md b/AGENTS.md index e472ce4..7ee21a2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -157,6 +157,7 @@ To add a tool: manifest, run, panel, three registry entries, and tests. Reuse th - Tools home (`Home.tsx`, at most 1600 px wide) is drop-first: the `Dropzone` `bar` layout, and dropped files handed to a tool through `handoff.ts`, the same mechanism as "Use in another tool". - Docker: `Dockerfile` builds the site with Node (on the build machine's platform) and serves it with `nginxinc/nginx-unprivileged` (non-root, port 8080). Its NGINX site is generated from `apps/web/public/_headers` by `scripts/nginx-conf.ts`, so change headers there only. The image build skips the typecheck (tests are left out by `.dockerignore`; CI typechecks anyway). `check` builds and runs it and checks the page, the headers, `sw.js` not being cached, the manifest's type, a 404 and a non-root user; the release pushes amd64 and arm64 images to `ghcr.io/deityhub/ream` and, with `DOCKERHUB_USERNAME`/`DOCKERHUB_TOKEN` set, `docker.io//ream`, tagged with the version and `latest`. Containers can't start in this Mac's Docker; test a server inside a throwaway `docker build` (start it in a `RUN`, query it with curl) or rely on CI. - Versions: the root `package.json` `version` (semver, starting at 0.1.0) is the release version. A pull request into `main` must raise it and have its `CHANGELOG.md` section: `yarn release patch|minor|major` (`scripts/release.ts prepare`) writes both, the section from the pull requests merged since the last tag (first-parent merges of `dev`, their titles and links; earlier releases skipped), and `node scripts/release.ts check` fails on a version already tagged or older or one without a section. The merge deploys production, tags `v` and creates a GitHub Release whose notes are that section (`release.ts notes`). `.github/workflows/prepare-release.yml` (Actions → Prepare release) runs `prepare` on `dev` and opens the "Release v" pull request into `dev`, as the `RELEASE_TOKEN` secret so `check` and the preview run on it. The version is in the footer (`__APP_VERSION__`, defined in `vite.config.ts`, declared in `version.d.ts`) and in the offline cache's name. `.github/workflows/deploy-version.yml` builds a released tag again and deploys it to production (rollback when the Cloudflare dashboard's can't). +- Crawlers: `apps/web/public/robots.txt` allows search engines and blocks AI training crawlers only (AI search and assistant fetchers and link-preview bots stay allowed, on purpose); `crawlers.test.ts` pins both lists, so adding a bot to one list means changing the test too. `_headers` ends with host rules (`https://:project.pages.dev/*`) that make the `pages.dev` addresses `noindex`; `scripts/nginx-conf.ts` skips rules for a whole address. - Hosting: Cloudflare Pages, deployed by `.github/workflows/web.yml`: a pull request into `dev` runs `check` and gets a preview (its link is commented on the PR), a pull request into `main` runs `check`, a merge into `main` runs `check` and deploys production. Plain pushes run nothing. `check` is the status `.github/rulesets/protect-dev-main.json` requires (import it under Settings → Rules once the repo is public: rulesets aren't free on private repos). `apps/web/public/_headers` holds the security headers: the CSP allows only the site's own files, plus `api.fontsource.org` and `cdn.jsdelivr.net` in `connect-src` for Download free fonts, and no inline scripts or styles (`style-src 'self'`: React's `style` props are fine, `