AI Content Audit Crawler

Audit every page of your nonprofit's website — automatically. Sitemap or no sitemap. Hundreds of pages or thousands. One command, one Excel report.

demo-full.mp4

▲ 3-minute demo: setup, sitemap mode, BFS fallback, AI scoring, and the Excel report.

Two ways it finds your pages

Your site has a sitemap	Your site has no sitemap
Auto-discovers sitemap.xml, sitemap index files, and platform-specific locations like Zendesk's /hc/sitemap.xml. Nested sitemap-of-sitemaps are resolved recursively.	Automatically switches to BFS link discovery: starts at your homepage, follows every internal link breadth-first, dedupes URL variants, skips login/admin/cart/asset junk, and caps the queue so it always finishes.

Built with Playwright (crawling) + Claude AI (scoring) + openpyxl (Excel reporting).

---

What this tool does

Nonprofits, charities, libraries, and NGOs run websites full of product listings, resource hubs, blog posts, and help articles — and content quietly goes stale: discontinued products stay listed, brand names get outdated ("G Suite" instead of "Google Workspace"), promotional deadlines expire, and pages break. Manually auditing hundreds or thousands of pages is not realistic for a small team.

This crawler automates the entire audit:

1. Discovers every page on your site automatically via the sitemap (sitemap.xml, sitemap index files, Zendesk hc/sitemap.xml, etc.). If no sitemap exists, it falls back to breadth-first link discovery.
2. Extracts clean content from each page using a real headless Chromium browser — it strips navigation, footers, scripts, and styles, and even force-opens hidden tab panels (Bootstrap, jQuery UI, Drupal, ARIA tabs) so no content is missed.
3. Scores every page with Claude AI on three dimensions, each 0–10:
   • Relevance — Does the page serve its stated purpose and your nonprofit audience?
   • Freshness — Are brand names, products, dates, and offers current?
   • Quality — Is the page complete for its type, error-free, and genuinely valuable?
4. Classifies each page by type (form, thank-you page, product page, blog article, resource hub, campaign landing, redirect stub) so a 40-word contact form is never unfairly penalised for being short.
5. Detects discontinued/404 pages by pattern matching — these are flagged instantly without spending an API call.
6. Applies "Top Partner" logic — pages belonging to vendors you designate get stricter rating thresholds, so problems on your most important partner pages surface first.
7. Generates a professional Excel report with executive summaries, per-page details, an issues list, score breakdowns with a chart, and a prioritised action plan with effort estimates.

---

The Excel report

Each run produces audit_YYYYMMDD_HHMMSS.xlsx containing 8 sheets:

Sheet	Contents
Exec Summary – Overall	KPI counts (Good / Needs Update / Critical / Discontinued), average scores, partner vs. standard breakdown, top 10 issues, 15 lowest-scoring pages
Exec Summary – Freshness	Same view focused on the Freshness dimension
Exec Summary – Relevance	Same view focused on the Relevance dimension
Exec Summary – Quality	Same view focused on the Quality dimension
Page Details	One row per page: URL, title, page type, partner tag, all scores, rating, priority, summary, strengths, issues, suggestions, and full extracted content
Issues & Actions	Every issue found, sorted by priority, with the suggested fix
Score Breakdown	All pages sorted best → worst, with a bar chart
Action Plan	A remediation checklist with category (Bug Fix / Content / Technical), effort estimate, and a status column your team can track

---

Ratings explained

Claude scores each dimension 0–10; the average drives the overall rating:

Rating	Meaning
Good	Page works. Minor improvements at most.
Needs Update	Noticeable gaps or stale content — schedule work.
Critical	Broken, misleading, or empty — act now. (Only possible for Top Partner pages; see below.)
Discontinued	Page shows a "no longer available" / 404 message. Flagged without an API call.

Top Partner logic: In sites.json you can list your most important vendors or program partners under top_partners. Pages matched to those partners (by URL, title, or content) use stricter thresholds and can be rated Critical. All other pages bottom out at Needs Update. Leave the list empty ([]) to treat all pages the same.

Average score	Top Partner page	Standard page
< 6.0	Critical	Needs Update
6.0 – 7.0	Needs Update	Needs Update
> 7.0	Good	Good

---

Requirements

• Python 3.10+
• An Anthropic API key — get one at console.anthropic.com
• Internet access (the crawler visits your live site)

---

Setup

1. Clone the repository

git clone https://github.com/<your-org>/ai-content-audit-crawler.git
cd ai-content-audit-crawler

2. (Recommended) Create a virtual environment

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate

3. Install dependencies

pip install -r requirements.txt
playwright install chromium

4. Add your API key

cp .env.example .env
# then edit .env and paste your key:
# ANTHROPIC_API_KEY=sk-ant-...

Alternatively, set it as an environment variable:

export ANTHROPIC_API_KEY=sk-ant-...     # Windows: set ANTHROPIC_API_KEY=sk-ant-...

⚠️ Never commit .env to git. The included .gitignore already excludes it.

5. Configure your site(s)

Open sites.json and replace the example.org entries with your own website. Minimal example:

{
  "top_partners": [],

  "my_site": {
    "base_url": "https://www.mynonprofit.org",
    "start_path": "/",
    "allowed_paths": ["*"],
    "max_pages": 200
  }
}

---

Running the crawler

Test with a single page first (cheapest way to verify everything works):

python crawler.py --url https://www.mynonprofit.org/about

Small trial run (first 10 pages of one site entry):

python crawler.py --site my_site --max-pages 10

Full audit (every site entry in sites.json, respecting each max_pages):

python crawler.py

Batch mode — for large sites, crawl in resumable chunks so a crash or interruption never loses hours of work:

python crawler.py --site my_site --batch-size 100 --batch-offset 0 --output batch1.xlsx
python crawler.py --site my_site --batch-size 100 --batch-offset 100 --output batch2.xlsx
python crawler.py --site my_site --batch-size 100 --batch-offset 200 --output batch3.xlsx

All CLI options

Flag	Description	Default
--url URL	Audit a single URL only	—
--site NAME	Audit one named entry from sites.json	all entries
--max-pages N	Override max_pages for the chosen site	from sites.json
--batch-size N	Pages per run (0 = unlimited)	0
--batch-offset N	Skip this many URLs before starting	0
--output FILE	Output filename	audit_<timestamp>.xlsx
--sites-file FILE	Alternate config file	sites.json

Before crawling, the tool prints a sitemap analysis: how many URLs it found, how they break down by category (Products, Support, Blog, …), and exactly which URLs are queued. If your path filters match zero pages, it prints a diagnosis table of the real path prefixes on your site and tells you what to paste into allowed_paths.

---

Configuration files

sites.json — what to crawl

Each top-level key (other than top_partners and keys starting with _) is an independent site entry:

Key	Required	Purpose
base_url	✔	Root URL. The sitemap is auto-discovered from here.
sitemap_path		Override sitemap location, e.g. /hc/sitemap.xml for Zendesk Help Centers.
allowed_domains		Extra domains to accept from the sitemap (subdomains, country TLDs).
allowed_paths		Only crawl URLs containing these path fragments. ["*"] = catch-all.
exclude_paths		Skip these paths — combine with ["*"] so a catch-all entry doesn't re-crawl sections handled by other entries.
start_path		Starting URL if no sitemap is found (link-discovery mode).
max_pages		Hard cap on pages per run.

prompt.txt — how pages are scored

This file is the audit criteria, written in plain English. Non-developers can edit it freely: change the organisation description, adjust score thresholds, add outdated brand names or discontinued products to flag, or reword any dimension. Lines starting with # are comments and are stripped before the prompt is sent to Claude. The only untouchable part is the JSON template at the bottom.

selectors.txt — what to strip from pages

One CSS selector per line. Matching elements (navigation bars, headers, footers) are removed before content extraction so menus don't pollute the audit. Add selectors for your site's chrome, e.g. .cookie-banner or #sidebar.

.env — your API key

ANTHROPIC_API_KEY=sk-ant-...

---

Cost control

• Each page costs one Claude API call (page content is truncated to 6,000 characters).
• The default model is set in crawler.py (CLAUDE_MODEL). Switching to a Haiku-class model cuts costs roughly 3× for large audits.
• Discontinued/404 pages are detected by pattern matching and never hit the API.
• Use --max-pages for trial runs and batch mode for large sites.

---

How it works (architecture)

sites.json ──► fetch_sitemap() ──► filter_urls() ──► [URL queue]
                     │ (no sitemap?)
                     └──► crawl_by_links()  (BFS fallback with skip rules)

For each URL:
  goto_with_retry()          3 attempts: domcontentloaded → load → commit
  extract_page_content()     strip selectors.txt + scripts/styles,
                             force-open all tab panels, extract clean text
  is_discontinued()          pattern check — skip API if page is dead
  score_page_with_claude()   prompt.txt + page content → structured JSON
  compute_rating()           apply Top Partner threshold logic

build_excel_report()         8-sheet formatted workbook

Key robustness features:

• Retry navigation with progressively looser wait conditions (45s → 60s → 90s).
• Browser crash recovery — if a tab dies, a fresh one is opened and the crawl continues.
• URL normalisation — http/https variants, fragments, query strings, and trailing slashes are canonicalised so pages are never crawled twice.
• Sitemap index support — nested sitemap-of-sitemaps files are resolved recursively.
• Noise filtering — lines that look like leaked JavaScript/CSS are dropped before scoring.

---

Troubleshooting

Problem	Fix
ERROR: ANTHROPIC_API_KEY not set	Create .env from .env.example, or export the variable in your shell.
0 URLs matched allowed_paths	The console prints your site's real path prefixes — copy one into allowed_paths in sites.json.
No sitemap found	The crawler automatically switches to link-discovery (BFS) mode from start_path. Set sitemap_path if your sitemap lives at a non-standard location.
Pages timing out	Some servers throttle bots. Reduce max_pages, run in batches, or check whether the page requires login/geo-access.
playwright: command not found	Run pip install -r requirements.txt then playwright install chromium.
Nav menus appearing in extracted content	Add your site's menu/footer CSS selectors to selectors.txt.

---

Responsible crawling

• Crawl your own website or sites you have permission to audit.
• Respect the target site's robots.txt and terms of service.
• Keep max_pages reasonable and use batch mode for large runs to avoid hammering servers.

---

Contributing

Contributions are welcome! Ideas that would help the nonprofit community:

• CSV/JSON export alongside Excel
• robots.txt compliance and configurable crawl delay
• Support for additional AI providers
• Localised prompt templates

Please open an issue or pull request.

License

Released under the MIT License — free for any nonprofit, charity, library, or NGO (or anyone else) to use, modify, and share.