Skip to content

Fix audit engine hang and real-site crawling; expand rule set to 35 - #3

Merged
zimkk merged 1 commit into
mainfrom
feat/engine-fixes-and-expansion
Oct 24, 2026
Merged

zimkk merged 1 commit into
mainfrom
feat/engine-fixes-and-expansion

Conversation

@zimkk

@zimkk zimkk commented Oct 24, 2026

Copy link
Copy Markdown
Owner

Summary

Makes the deterministic engine actually work end to end, then fills in the parts of the master plan that were stubbed.

Fixes

  • Audit never finished: the crawler never closed its queue, so rankcore audit hung until Ctrl+C. The old integration test only passed via its 20s timeout; it now completes in ~1s.
  • Compressed pages parsed as garbage: setting Accept-Encoding manually disabled Go's transparent gzip.
  • CLI flags after the target were ignored (rankcore audit <url> --out x --json, the form the skill uses), and --json mixed progress text into stdout.
  • Link handling: absolute same-origin links were never crawled; //cdn… links were treated as internal; the link graph compared raw hrefs to absolute URLs, so every page was an orphan.
  • :443 stripping left example.com:.
  • Broken-link and canonical-target rules were stubs; both are now real site-wide checks.
  • verify marked unreached pages "fixed" and lost track of findings when the dev-server port changed.
  • Redirect chains recorded hop by hop; the SSRF guard dials the IP it validated.
  • Agent detection looked in the wrong folder; install.ps1 closed the PowerShell session on failure.

Additions

  • Sitemap discovery (robots.txt, indexes, gzip) seeds the crawl.
  • New rules (35 total): non-indexable sitemap entries, missing sitemap, orphans, redirecting internal links, multiple canonicals, duplicate titles/descriptions, hreflang return links, structured-data required properties, search/AI-search crawler blocks (training blocks never flagged).
  • Exit codes per §12.9 plus --fail-on <severity>; audit ./dist serves a local build; --max-depth, --ignore-robots, --timeout, --max-render.
  • Findings sorted deterministically and normalized to catalog category / automation class; report includes per-page summary, origin and sitemaps.
  • x/net v0.61.0; CI on Linux/Windows/macOS using the go.mod version, race detector on Linux; release builds stamp the version.

Behaviour changes to note

  • Thin content now uses a 200-word threshold (per the plan) instead of 200 characters.
  • FacebookBot is classed as a training crawler, matching Meta's documentation.

Testing

  • gofmt, go vet, go test ./... pass locally (Windows).
  • Smoke-tested the binary: local-directory audit, verify, exit codes, live crawl of example.com, rendered mode with Chrome.
  • Race detector not run locally (no gcc); covered by the Linux CI job.

🤖 Generated with Claude Code

…ules

Fixes
- Crawler never closed its queue, so `rankcore audit` hung until interrupted.
- Manual Accept-Encoding disabled Go's gzip handling; compressed pages parsed as garbage.
- CLI flags after the target were ignored; --json mixed progress into stdout.
- Absolute same-origin links were never crawled; protocol-relative links were treated as internal.
- Link graph keyed raw hrefs against absolute URLs, so every page was an orphan.
- :443 port stripping left a trailing colon.
- Broken-link and canonical-target rules were stubs; now real site-wide checks.
- verify marked unreached pages as fixed and broke when the origin port changed.
- Redirect chains are recorded hop by hop; SSRF guard dials the validated IP.
- Agent detection checked the wrong directory; install.ps1 closed the shell on failure.

Additions
- Sitemap discovery (robots.txt, indexes, gzip) seeds the crawl.
- New rules: sitemap entries, orphans, redirecting links, multiple canonicals,
  duplicate titles/descriptions, hreflang return links, structured-data required
  properties, search-crawler blocks (35 rules total).
- Exit codes and --fail-on gate, local directory targets, new crawl flags.
- Deterministic sorted findings with catalog category/automation, per-page report.
- x/net v0.61.0, CI matrix with race detector, versioned release builds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zimkk
zimkk merged commit 315fe5e into main Oct 24, 2026
3 checks passed
@zimkk
zimkk deleted the feat/engine-fixes-and-expansion branch October 24, 2026 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant