seo-crawler

A concurrent Go CLI that crawls a site and audits its SEO correctness — canonical tags, hreflang reciprocity, broken links, meta, and structured data.

Background

This started as a recurring manual chore: auditing a product site published in five locales — English, German, Turkish, Chinese, and Spanish (en / de / tr / zh / es). The failures that actually hurt rankings were the cross-locale ones — an hreflang on the German page pointing at the English page that never pointed back, a zh page missing its x-default, a canonical on the Spanish page silently collapsing it into another URL. Checking that by hand across hundreds of pages is exactly the kind of tedious, error-prone work worth automating, so hreflang reciprocity is the centerpiece the rest of the tool is built around.

How it works

flowchart LR
    subgraph engine["crawl engine (bounded)"]
        F["frontier queue<br/>dedup + visited set"] --> WP["worker pool<br/>N workers + politeness"]
        WP -->|new in-scope links| F
    end
    WP --> FE["Fetcher port<br/>HTTP · robots.txt · timeout"]
    FE --> PA["parser<br/>HTML → Page"]
    PA --> PG["pages"]
    PG --> PC["per-page checkers<br/>canonical · meta · structured data"]
    PG --> CC["cross-page checkers<br/>hreflang · duplicate titles · broken links"]
    PC --> RP["Reporter<br/>console + JSON"]
    CC --> RP

Loading

The engine and the SEO rules are decoupled and all I/O sits behind ports (Fetcher, Reporter). The crawl engine knows nothing about specific rules; a checker knows nothing about HTTP. cmd/crawler is the only place concrete adapters and checkers are wired together.

Design decisions & trade-offs

Bounded concurrency + politeness. A fixed pool of N workers does the fetching — never one goroutine per URL. A global limiter spaces fetch starts a configurable delay apart (not per-worker, so raising concurrency doesn't defeat it), robots.txt is respected, each fetch carries a context timeout, and crawling is scoped to the root's host. This is an auditor, not a load generator: throughput is deliberately capped so it never hammers the target.

Pluggable checkers. Every rule implements a small Checker interface and is registered in one place. Adding a rule is a registration, not a change to the engine — the engine never learns what "canonical" or "hreflang" mean.

Per-page vs. cross-page checks. The split is explicit. Per-page checks (canonical, meta, structured data) need only the page in front of them and run during the crawl. Cross-page checks (hreflang reciprocity, duplicate titles, broken internal links) need the whole corpus and run after the crawl completes. Keeping the two kinds separate is what lets reciprocity be checked correctly — you can't confirm B links back to A until you've seen every page.

Centralized URL normalization. Everything dedups on one canonical form: scheme/host lowercased, default ports and fragments dropped, relative refs resolved, trailing slashes normalized, query params sorted. Two URLs that address the same resource collapse to the same key, which is what makes the visited set and hreflang matching reliable. The trade-off is that normalization is conservative — it can't know a given server's own canonicalization quirks (e.g. case-sensitive query values), so it errs toward treating URLs as distinct rather than wrongly merging them.

Static HTML only. The crawler fetches and parses server-rendered HTML; there is no headless browser or JavaScript execution. Search engines index primarily what's in the server response, so that's what the audit targets — and it keeps the tool fast, deterministic, and dependency-free. The cost is that content rendered entirely client-side (a SPA that ships an empty shell) is invisible to the audit. Sitemap generation is likewise out of scope.

Usage

go build -o bin/seo-crawler ./cmd/crawler

./bin/seo-crawler \
  --url https://example.com/ \
  --concurrency 8 \
  --delay 200ms \
  --max-pages 500 \
  --max-depth 10 \
  --out report.json

Or via the Makefile: make run URL=https://example.com.

Flag	Default	Meaning
--url	(required)	Root URL to crawl
--concurrency	8	Number of concurrent fetch workers
--delay	200ms	Minimum spacing between requests (politeness)
--max-pages	500	Stop after this many pages
--max-depth	10	Maximum link depth from the root
--out	report.json	Path for the JSON report

Sample report

{
  "root": "https://example.com/",
  "pages_crawled": 128,
  "issues": [
    {
      "severity": "error",
      "code": "hreflang.no-reciprocity",
      "page": "https://example.com/de/preise",
      "message": "hreflang de→en points to https://example.com/en/pricing, which has no return link to this page"
    },
    {
      "severity": "error",
      "code": "hreflang.no-x-default",
      "page": "https://example.com/tr/fiyatlar",
      "message": "alternate set has no x-default entry"
    },
    {
      "severity": "warning",
      "code": "canonical.not-self",
      "page": "https://example.com/es/precios",
      "message": "canonical points to https://example.com/es/, not a self-reference"
    },
    {
      "severity": "warning",
      "code": "meta.description.missing",
      "page": "https://example.com/zh/jiage",
      "message": "<meta name=\"description\"> is missing"
    },
    {
      "severity": "error",
      "code": "links.broken",
      "page": "https://example.com/en/blog",
      "message": "links to https://example.com/en/old-post, which returned 404"
    }
  ]
}

Development

make test   # go test ./... -race
make lint   # golangci-lint run