Skip to content

feat: one crawler for CLI and web, same pages as the platform - #7

Merged
matte97p merged 1 commit into
mainfrom
matte97p/align-platform
Oct 19, 2026
Merged

matte97p merged 1 commit into
mainfrom
matte97p/align-platform

Conversation

@matte97p

Copy link
Copy Markdown
Member

Same input, same result as the GeoSuite platform (owner's rule: the free tools on tools.trygeosuite.it and the platform always agree). A golden fixture in test/fixtures is shared byte for byte with the platform, which runs it too: drift fails a test on the side that changed.

  • The web page uses the same crawlSite() as the CLI, with the platform's settings (depth 3, 6 s per page, 25 s total) and lastmod. The free page stays capped at 40 pages (Workers subrequest limit).
  • Only 2xx HTML pages are listed, with the final URL after redirects; links normalised like a browser and sorted.
  • deploy-web.yml redeploys on src/** too.
  • test/fixtures/golden-sitemap.json: 35 cases (node scripts/update-golden.mjs).

Same input, same result as the GeoSuite platform (owner's rule: the free tools on tools.trygeosuite.it and the platform always agree). A golden fixture in test/fixtures is shared byte for byte with the platform, which runs it too: drift fails a test on the side that changed.

- The web page uses the same crawlSite() as the CLI, with the platform's settings (depth 3, 6 s per page, 25 s total) and lastmod. The free page stays capped at 40 pages (Workers subrequest limit).
- Only 2xx HTML pages are listed, with the final URL after redirects; links normalised like a browser and sorted.
- deploy-web.yml redeploys on src/** too.
- test/fixtures/golden-sitemap.json: 35 cases (node scripts/update-golden.mjs).
@matte97p
matte97p merged commit 5512c33 into main Oct 19, 2026
2 checks passed
@matte97p
matte97p deleted the matte97p/align-platform branch October 19, 2026 08:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant