Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/deploy-web.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ on:
branches: [main]
paths:
- 'web/**'
- 'src/sitemap.js'
- 'src/**'
- '.github/workflows/deploy-web.yml'
workflow_dispatch: {}

Expand Down
22 changes: 22 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,28 @@ All notable changes to `@geosuite/sitemap-builder` are documented here.
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and the project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Changed

- Same results as the GeoSuite platform's Sitemap Builder, checked by golden
cases both repos run (`test/fixtures/golden-sitemap.json`, `test/golden.test.js`).
- The crawler uses `fetch` instead of `node:http`, so the hosted page runs the
package's own `crawlSite()` instead of a separate copy.
- Redirects are followed and pages are listed at their final URL; a start URL
redirecting to another host (apex to `www`) moves the crawl there.
- Only 2xx HTML pages go in the sitemap: 404s, 500s and non-HTML responses no
longer do.
- Links are normalized (lowercase host, no default port, `&` decoded) and
followed in sorted order; pages are sorted by depth, then by URL in code-point
order (it was locale order).
- `<lastmod>` accepts only HTTP dates and is the UTC date.
- `crawlSite()` accepts a bare domain and returns the `origin` it crawled;
`--respect-robots` follows `Allow` rules and groups of several `User-agent`
lines (longest match wins, `Allow` on a tie).
- Hosted page: depth 3 (was 2), the platform's timeouts and budget, and
`<lastmod>` in the output; still at most 40 pages.

## [0.3.2] - 2026-05-23

### Changed
Expand Down
38 changes: 26 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,16 +80,30 @@ geosuite-sitemap-builder https://example.com --json

## What it actually does

1. Starts at the URL you pass.
2. BFS-crawls **same-origin** `<a href>` links only (never wanders off the host).
3. Drops obvious non-HTML extensions (`.png`, `.css`, `.pdf`, …) so the
1. Starts at the URL you pass (a bare `example.com` is crawled over https).
2. BFS-crawls **same-origin** `<a href>` links only (never wanders off the host),
in sorted order. Links are normalized the way a browser does: lowercase
host, no default port, no `#fragment`, `.`/`..` resolved, `&amp;` decoded,
path and query percent-encoded.
3. Follows redirects and lists each page at its **final URL**. If the start URL
redirects to another host (`example.com` to `www.example.com`), the crawl
continues there; any other page that redirects off the site is dropped.
4. Lists only pages that answer **2xx with HTML**: a 404, a 500 or a PDF
served without an extension stays out.
5. Drops obvious non-HTML extensions (`.png`, `.css`, `.pdf`, …) so the
sitemap doesn't get polluted with assets.
4. Skips fragment-only links (`#section`), `mailto:`, `tel:`, and `javascript:`.
5. Stops at any of three caps (whichever fires first):
6. Skips fragment-only links (`#section`), `mailto:`, `tel:`, and `javascript:`.
7. Stops at any of three caps (whichever fires first):
- page count (`--max-pages`)
- BFS depth (`--max-depth`)
- wall-clock budget (`--budget-s`)
6. Renders the discovered URLs as a sitemaps.org-compliant `<urlset>`.
8. Renders the discovered URLs as a sitemaps.org-compliant `<urlset>`, by
depth and then by URL, with `<lastmod>` from the `Last-Modified` header
when the site sends one.

The same rules run in the [GeoSuite](https://trygeosuite.it) platform's
Sitemap Builder: both run the cases in `test/fixtures/golden-sitemap.json`, so
for the same site and the same caps the two give the same sitemap.

The output is intentionally minimal: `<loc>` plus an optional `<lastmod>`. We
skip `<changefreq>` and `<priority>` — the spec deprecates them and the major
Expand All @@ -102,12 +116,12 @@ search engines have ignored them for years.
- **JavaScript rendering**. The crawler is HTTP + regex. Single-page apps
whose links only appear after client-side hydration won't be discovered.
Build-time pre-rendering or an SSR layer is the right fix.
- **Robots.txt awareness**. By default the tool runs against the site
owner's own domain and honoring `robots.txt` would silently strip the
pages they want to publish. (Adding an opt-in `--respect-robots` flag is
on the roadmap.)
- **`<lastmod>` accuracy**. Today we don't fill `<lastmod>` from `Last-Modified`
response headers. Coming in 0.2.
- **Robots.txt by default**. The tool runs against the site owner's own
domain, and honoring `robots.txt` would silently strip the pages they want
to publish. Pass `--respect-robots` to skip the paths disallowed for
`User-agent: *` (longest match wins, `Allow` on a tie).
- **`<lastmod>` beyond the header**. `<lastmod>` comes only from the
`Last-Modified` response header; pages that don't send it get none.
- **LLM-powered grouping or summaries**. The deterministic 0.1 ships without
a network dependency on any model. An opt-in `--ai` mode is on the
roadmap (provide `OPENAI_API_KEY` or `ANTHROPIC_API_KEY` to enable).
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
"CHANGELOG.md"
],
"scripts": {
"lint": "node --check src/index.js && node --check src/crawler.js && node --check src/sitemap.js && node --check src/ai.js && node --check bin/cli.js && node --check test/test-sitemap.js && node --check test/test-crawler.js && node --check test/test-ai.js",
"lint": "node --check src/index.js && node --check src/crawler.js && node --check src/robots.js && node --check src/sitemap.js && node --check src/ai.js && node --check bin/cli.js && node --check test/test-sitemap.js && node --check test/test-crawler.js && node --check test/test-ai.js && node --check test/golden.test.js && node --check test/golden-run.js && node --check scripts/update-golden.mjs",
"test": "node --test test/*.js",
"coverage": "node --experimental-test-coverage --test test/*.js"
},
Expand Down
21 changes: 21 additions & 0 deletions scripts/update-golden.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#!/usr/bin/env node
// Recompute the `expected` blocks of test/fixtures/golden-sitemap.json from
// the current code. Run it only after an intentional behavior change, read the
// diff, then copy the file to the platform repo
// (geo: backend/tests/fixtures/golden-sitemap.json) and make its Python
// service give the same answers.
//
// node scripts/update-golden.mjs

import { readFile, writeFile } from 'node:fs/promises';
import { fileURLToPath } from 'node:url';
import { dirname, join } from 'node:path';
import { SECTIONS, computeGolden } from '../test/golden-run.js';

const file = join(dirname(fileURLToPath(import.meta.url)), '..', 'test', 'fixtures', 'golden-sitemap.json');
const golden = JSON.parse(await readFile(file, 'utf8'));
for (const section of SECTIONS) {
for (const c of golden[section]) c.expected = await computeGolden(section, c);
}
await writeFile(file, JSON.stringify(golden, null, 2) + '\n');
process.stdout.write(`updated ${file}\n`);
Loading
Loading