Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,7 @@ To add a tool: manifest, run, panel, three registry entries, and tests. Reuse th
- Tools home (`Home.tsx`, at most 1600 px wide) is drop-first: the `Dropzone` `bar` layout, and dropped files handed to a tool through `handoff.ts`, the same mechanism as "Use in another tool".
- Docker: `Dockerfile` builds the site with Node (on the build machine's platform) and serves it with `nginxinc/nginx-unprivileged` (non-root, port 8080). Its NGINX site is generated from `apps/web/public/_headers` by `scripts/nginx-conf.ts`, so change headers there only. The image build skips the typecheck (tests are left out by `.dockerignore`; CI typechecks anyway). `check` builds and runs it and checks the page, the headers, `sw.js` not being cached, the manifest's type, a 404 and a non-root user; the release pushes amd64 and arm64 images to `ghcr.io/deityhub/ream` and, with `DOCKERHUB_USERNAME`/`DOCKERHUB_TOKEN` set, `docker.io/<user>/ream`, tagged with the version and `latest`. Containers can't start in this Mac's Docker; test a server inside a throwaway `docker build` (start it in a `RUN`, query it with curl) or rely on CI.
- Versions: the root `package.json` `version` (semver, starting at 0.1.0) is the release version. A pull request into `main` must raise it and have its `CHANGELOG.md` section: `yarn release patch|minor|major` (`scripts/release.ts prepare`) writes both, the section from the pull requests merged since the last tag (first-parent merges of `dev`, their titles and links; earlier releases skipped), and `node scripts/release.ts check` fails on a version already tagged or older or one without a section. The merge deploys production, tags `v<version>` and creates a GitHub Release whose notes are that section (`release.ts notes`). `.github/workflows/prepare-release.yml` (Actions → Prepare release) runs `prepare` on `dev` and opens the "Release v<version>" pull request into `dev`, as the `RELEASE_TOKEN` secret so `check` and the preview run on it. The version is in the footer (`__APP_VERSION__`, defined in `vite.config.ts`, declared in `version.d.ts`) and in the offline cache's name. `.github/workflows/deploy-version.yml` builds a released tag again and deploys it to production (rollback when the Cloudflare dashboard's can't).
- Crawlers: `apps/web/public/robots.txt` allows search engines and blocks AI training crawlers only (AI search and assistant fetchers and link-preview bots stay allowed, on purpose); `crawlers.test.ts` pins both lists, so adding a bot to one list means changing the test too. `_headers` ends with host rules (`https://:project.pages.dev/*`) that make the `pages.dev` addresses `noindex`; `scripts/nginx-conf.ts` skips rules for a whole address.
- Hosting: Cloudflare Pages, deployed by `.github/workflows/web.yml`: a pull request into `dev` runs `check` and gets a preview (its link is commented on the PR), a pull request into `main` runs `check`, a merge into `main` runs `check` and deploys production. Plain pushes run nothing. `check` is the status `.github/rulesets/protect-dev-main.json` requires (import it under Settings → Rules once the repo is public: rulesets aren't free on private repos). `apps/web/public/_headers` holds the security headers: the CSP allows only the site's own files, plus `api.fontsource.org` and `cdn.jsdelivr.net` in `connect-src` for Download free fonts, and no inline scripts or styles (`style-src 'self'`: React's `style` props are fine, `<style>` elements and `style="..."` strings are not). A new network host or inline style needs the policy changed, and the user asked first (the network rule above). Check with `wrangler pages dev` and a `securitypolicyviolation` listener.
- PWA: `manifest.webmanifest` and `icons/` in `apps/web/public` (PNGs from `apps/web/icons/icon.svg` by `node scripts/icons.ts`; square, full bleed, the mark inside the maskable safe zone). `apps/web/sw.js` is plain JS, outside `src`: the `serviceWorker` plugin in `vite.config.ts` writes it into `dist` after the build with every file's name (minus `.woff` fallbacks and the non-Latin Office fonts, which it keeps when first used) and a version hashed from their contents. The page is kept as `./` because Pages redirects `index.html` to `/`. Navigations are network first, everything else cache first. A new version waits until the user clicks Reload in `UpdateBar.tsx` (`pwa.ts` posts `switch`; the page reloads on `controllerchange` only after that, since the first worker's `clients.claim` changes the controller too). `pwa.ts` also keeps the `beforeinstallprompt` event for the landing page's Install Ream button. The worker is registered in production builds only: `yarn dev` has none. To test offline for real, serve the build with `wrangler pages dev`, let the worker fill its cache, then stop the server (CDP's offline emulation doesn't reach the worker's own requests).

Expand Down
7 changes: 7 additions & 0 deletions apps/web/public/_headers
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,10 @@
# the service worker and the page must always be checked for a newer version
/sw.js
Cache-Control: no-cache

# Only ream.deityhub.com is for search engines: the same site on Cloudflare's own addresses (the project's, and every
# preview deployment's) would be an indexable duplicate
https://:project.pages.dev/*
X-Robots-Tag: noindex
https://:version.:project.pages.dev/*
X-Robots-Tag: noindex
58 changes: 58 additions & 0 deletions apps/web/public/robots.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# Ream's robots.txt. Search engines are welcome; crawlers that gather pages to train AI models are not.
#
# This is a request, not a lock: well-behaved crawlers follow it, others may not. Cloudflare's "Block AI bots" setting
# (Security, Bots) is what enforces it. The bots below are the training crawlers; AI search and assistant fetchers
# (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Claude-User) and link-preview bots are not blocked.
# Applebot (Siri and Spotlight search) and Googlebot are allowed; only their AI-training opt-out tokens are blocked.
# The list follows https://github.com/ai-robots-txt/ai.robots.txt, kept to training crawlers.

User-agent: *
Allow: /
# Cloudflare's content signals: found by search and quoted by assistants, but not used to train models
Content-Signal: search=yes, ai-input=yes, ai-train=no

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Claude-Web
User-agent: CCBot
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: FacebookBot
User-agent: Bytespider
User-agent: TikTokSpider
User-agent: Amazonbot
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: MistralAI-Training
User-agent: Diffbot
User-agent: ImagesiftBot
User-agent: omgili
User-agent: omgilibot
User-agent: Timpibot
User-agent: PanguBot
User-agent: Kangaroo Bot
User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
User-agent: Webzio-Extended
User-agent: img2dataset
User-agent: LAIONDownloader
User-agent: ICC-Crawler
User-agent: VelenPublicWebCrawler
User-agent: SBIntuitionsBot
User-agent: DeepSeekBot
User-agent: QwenBot
User-agent: MoonshotBot
User-agent: DoubaoBot
User-agent: TongyiBot
User-agent: ChatGLM-Spider
User-agent: YiBot
User-agent: Cotoyogi
User-agent: Scrapy
Disallow: /

Sitemap: https://ream.deityhub.com/sitemap.xml
83 changes: 83 additions & 0 deletions apps/web/src/crawlers.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
import { execFileSync } from 'node:child_process'
import { readFileSync } from 'node:fs'
import { describe, expect, test } from 'vitest'

const root = new URL('../../../', import.meta.url)
const robots = readFileSync(new URL('apps/web/public/robots.txt', root), 'utf8')

/** the groups of a robots.txt: the agents that share the rules, and the rules */
function groups(text: string) {
const out: { agents: string[]; rules: [string, string][] }[] = []
let current: (typeof out)[number] | null = null
let lastWasAgent = false
for (const raw of text.split('\n')) {
const line = raw.replace(/#.*/, '').trim()
const at = line.indexOf(':')
if (at < 0) continue
const key = line.slice(0, at).trim().toLowerCase()
const value = line.slice(at + 1).trim()
if (key === 'sitemap') continue
if (key === 'user-agent') {
if (!current || !lastWasAgent) out.push((current = { agents: [], rules: [] }))
current.agents.push(value.toLowerCase())
lastWasAgent = true
} else if (current) {
current.rules.push([key, value])
lastWasAgent = false
}
}
return out
}

/** whether the whole site is closed to this crawler: its own group if it has one, else the `*` group (RFC 9309) */
function blocked(agent: string) {
const all = groups(robots)
const own = all.find((g) => g.agents.includes(agent.toLowerCase())) ?? all.find((g) => g.agents.includes('*'))!
return own.rules.some(([k, v]) => k === 'disallow' && v === '/')
}

describe('robots.txt', () => {
test('search engines and link-preview bots are welcome', () => {
for (const agent of ['Googlebot', 'Bingbot', 'Applebot', 'DuckDuckBot', 'Twitterbot', 'LinkedInBot', 'facebookexternalhit', 'Slackbot', 'WhatsApp']) {
expect(blocked(agent), agent).toBe(false)
}
})

test('AI training crawlers are blocked', () => {
for (const agent of ['GPTBot', 'ClaudeBot', 'anthropic-ai', 'CCBot', 'Google-Extended', 'Applebot-Extended', 'Bytespider', 'Meta-ExternalAgent', 'cohere-ai', 'Diffbot', 'Amazonbot']) {
expect(blocked(agent), agent).toBe(true)
}
})

test('AI search and assistant fetchers are not blocked, as chosen', () => {
for (const agent of ['OAI-SearchBot', 'ChatGPT-User', 'PerplexityBot', 'Claude-SearchBot', 'Claude-User']) {
expect(blocked(agent), agent).toBe(false)
}
})

test('says AI training is off in the content signals, names the sitemap and lists nobody twice', () => {
expect(robots).toMatch(/^Content-Signal: .*\bai-train=no\b/m)
expect(robots).toMatch(/^Sitemap: https:\/\/ream\.deityhub\.com\/sitemap\.xml$/m)
const agents = groups(robots).flatMap((g) => g.agents)
expect(agents.length).toBe(new Set(agents).size)
})
})

describe('search engines and the other addresses of the site', () => {
// the rules, without their comments
const headers = readFileSync(new URL('apps/web/public/_headers', root), 'utf8').split('\n').filter((l) => !l.trimStart().startsWith('#')).join('\n')

test("Cloudflare's own addresses (and previews) are noindex, the custom domain is not", () => {
for (const host of ['https://:project.pages.dev/*', 'https://:version.:project.pages.dev/*']) {
expect(headers).toMatch(new RegExp(`^${host.replace(/[.*]/g, '\\$&')}\\n\\s+X-Robots-Tag: noindex$`, 'm'))
}
expect(headers).not.toMatch(/ream\.deityhub\.com/)
expect(headers.match(/X-Robots-Tag/g)).toHaveLength(2) // nothing on the shared /* rule
})

test('the self-hosted NGINX site has no location for those addresses', () => {
const conf = execFileSync(process.execPath, ['scripts/nginx-conf.ts'], { cwd: root, encoding: 'utf8' })
expect(conf).not.toMatch(/pages\.dev|X-Robots-Tag|location [^\n]*https?:/)
expect(conf).toMatch(/location = \/sw\.js/) // the other rules are still there
})
})
8 changes: 8 additions & 0 deletions docs/deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,14 @@ The site is https://ream.deityhub.com, on Cloudflare Pages. `.github/workflows/w

`apps/web/public/_headers` sets the response headers: a content security policy that lets the page load only its own files (plus Fontsource and jsDelivr, for Download free fonts), no inline scripts or styles, no framing; no referrer; and a year's caching for the content-named files in `/assets`. To try the headers locally: `yarn build`, then `yarn dlx wrangler@4 pages dev apps/web/dist`.

### Search engines and AI crawlers

`apps/web/public/robots.txt` welcomes search engines and asks AI **training** crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, Meta-ExternalAgent and others; the list follows the [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) project) to stay out. AI search and assistant fetchers (OAI-SearchBot, PerplexityBot, Claude-SearchBot and the like) and link-preview bots are allowed, so Ream can be found and quoted but not trained on. It also sends Cloudflare's `Content-Signal: ai-train=no`. To change the choice, edit the lists in that file; `apps/web/src/crawlers.test.ts` checks which crawlers are allowed and which are blocked.

robots.txt is a request that well-behaved crawlers follow, not a lock. The setting that enforces it is Cloudflare's, and it is free: dashboard → ream.deityhub.com → Security → Bots → **Block AI bots** (or AI Crawl Control). If you also turn on Cloudflare's *managed robots.txt*, it adds its own rules in front of this file.

The Cloudflare addresses of the site (`ream-bqb.pages.dev` and each preview) send `X-Robots-Tag: noindex` (host rules at the end of `_headers`), so search engines list only ream.deityhub.com. The self-hosted NGINX site leaves those rules out.

## Installable and offline (PWA)

Ream is a progressive web app. `apps/web/public/manifest.webmanifest` and the icons in `apps/web/public/icons` (drawn from `apps/web/icons/icon.svg` by `node scripts/icons.ts`) make it installable: Chrome and Edge show Install Ream on the landing page, Safari has File → Add to Dock, iPhone and iPad have Share → Add to Home Screen. The service worker (`apps/web/sw.js`, written into the build as `sw.js` with its file list by the `serviceWorker` plugin in `vite.config.ts`) keeps the app on the device:
Expand Down
4 changes: 4 additions & 0 deletions scripts/nginx-conf.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,10 @@ for (const line of readFileSync(file, 'utf8').split('\n')) {
}
}

// rules for a whole address (https://:project.pages.dev/*) are Cloudflare's: a self-hosted copy has one host, and a
// location can't match one
for (let i = rules.length; i--; ) if (/^https?:\/\//.test(rules[i].path)) rules.splice(i, 1)

const every = rules.filter((r) => r.path === '/*').flatMap((r) => r.headers)
// pages and anything without a rule of its own are checked for a newer version each time, like Cloudflare Pages does
const fallback: [string, string][] = [['Cache-Control', 'public, max-age=0, must-revalidate']]
Expand Down
Loading