Audit your own site's SEO health — broken links, titles, meta, alt text, schema & sitemap —
and export a clean HTML & PDF report. Python (FastAPI) + React, politely rate-limited.

A small, dependency-light Python tool that crawls a website you own and produces a readable HTML report on common on-page and technical SEO issues:

• 🏷️ Title tags — missing, duplicate, too short / too long
• 📝 Meta descriptions — missing, empty, duplicate, length
• 🔗 Broken internal links — 4xx/5xx targets, with the pages that link to them
• 🖼️ Image alt text — missing or empty alt attributes
• ⏱️ Page load — slow server responses, server errors, heavy HTML payloads
• 🧩 Schema markup — pages missing structured data (JSON-LD / microdata / RDFa)
• 🗺️ Sitemap & robots.txt — missing/invalid files, sitemap not referenced, accidental site-wide Disallow: /, off-domain sitemap URLs
• 🔑 Inferred keywords — focus keyword, top phrases (TF-IDF), density and cannibalization, from on-page signals only (never ranking data)
• 🧰 Generators — build a sitemap.xml from the crawl and fetch your robots.txt

It uses requests + BeautifulSoup, enforces a configurable rate limit so you don't hammer your own server, obeys your robots.txt by default, and writes a single self-contained HTML report.

Two ways to use it:

• 🖥️ a CLI (seo-auditor https://your-site.com), and
• 🌐 a web app — a FastAPI backend + a React (Vite) frontend where you enter a URL, watch live crawl progress, and browse the results in the browser.

⚠️ Only audit domains you own or are authorised to test. This tool sends real HTTP requests. The defaults are polite (1 request/second, robots-aware), but crawling sites you don't control may violate their terms of service.

---

Project layout

seo-auditor/
├── src/seo_auditor/
│   ├── cli.py            # command-line interface
│   ├── audit.py          # orchestrates crawl + checks
│   ├── crawler.py        # rate-limited BFS crawler + HTML extraction
│   ├── http.py           # Fetcher: requests.Session + the rate limit
│   ├── urls.py           # URL normalisation / same-site helpers
│   ├── models.py         # Page, Issue, AuditReport dataclasses
│   ├── config.py         # defaults + YAML + CLI override merging
│   ├── report.py         # HTML report renderer
│   ├── checks/           # one module per category of check
│   │   ├── titles.py  links.py  images.py
│   │   ├── performance.py  schema.py  site_files.py
│   └── api/              # FastAPI backend
│       ├── app.py        # routes + CORS
│       ├── jobs.py       # background audit jobs (thread pool)
│       ├── schemas.py    # pydantic request/response models
│       └── serialize.py  # AuditReport -> JSON
├── frontend/             # React + Vite single-page app
│   └── src/  components/ App.jsx  api.js  styles.css
├── tests/                # offline unit tests (pytest, incl. API)
├── config/config.example.yaml
├── requirements.txt          # core (CLI) deps
├── requirements-api.txt      # + FastAPI / uvicorn
├── pyproject.toml
├── run.py                # run the CLI without installing
└── README.md

---

Setup

Requires Python 3.10+.

# 1. From the project folder, create and activate a virtual environment
python -m venv .venv

#    Windows (PowerShell):
.venv\Scripts\Activate.ps1
#    Windows (Git Bash) / macOS / Linux:
source .venv/Scripts/activate      # Git Bash on Windows
source .venv/bin/activate          # macOS / Linux

# 2. Install dependencies
pip install -r requirements.txt

#    (optional) install as an editable package to get the `seo-auditor` command:
pip install -e .

---

Quick tutorial

1. Run your first audit

Point it at your site. The simplest run uses all defaults:

python run.py https://your-domain.com

…or, if you ran pip install -e .:

seo-auditor https://your-domain.com

You'll see live progress as it crawls:

Auditing https://your-domain.com (max 100 pages, 1.0s delay)…
  [200] https://your-domain.com
  [200] https://your-domain.com/about
  [404] https://your-domain.com/old-page
  ...

  Domain        : your-domain.com
  Pages crawled : 24
  Errors        : 3
  Warnings      : 11
  Notices       : 6
  Report        : reports/seo-report.html

2. Open the report

Open reports/seo-report.html in any browser. You'll get:

• summary cards (pages crawled, errors, warnings, notices, failed requests),
• findings grouped by category, each with a severity badge, the affected URL, and a short explanation,
• a table of every crawled page (status, response time, whether it has a title / meta description / schema, image count).

3. Tune the crawl

Common options (CLI flags override the config file):

# Crawl more pages, be extra gentle (2s between requests)
python run.py https://your-domain.com --max-pages 500 --delay 2

# Write the report somewhere specific
python run.py https://your-domain.com -o reports/run-2024.html

# Faster pass on your own staging box (0.2s delay), custom UA
python run.py https://staging.your-domain.com -d 0.2 --user-agent "MyAudit/1.0"

Full list: python run.py --help.

4. Use a config file (optional)

For repeatable runs, copy the example and edit it:

cp config/config.example.yaml config/config.local.yaml
python run.py -c config/config.local.yaml

The config file sets the start URL, crawl limits, rate-limit delay, and the thresholds used to decide what counts as "too long", "slow", etc.

---

Web app (FastAPI + React)

Prefer a UI? Run the API and the React frontend side by side.

1. Start the backend (FastAPI)

# from the project root, with your virtualenv active
pip install -r requirements-api.txt
python -m seo_auditor.api          # serves http://127.0.0.1:8000
#   …or, if you ran `pip install -e ".[api]"`:  seo-auditor-api

Interactive API docs are at http://127.0.0.1:8000/docs.

Method	Endpoint	Purpose
POST	/api/audits	Start an audit, returns a job id
GET	/api/audits/{id}	Poll status + result (when finished)
GET	/api/audits	List recent jobs
GET	/api/audits/{id}/report	The full HTML report
GET	/api/health	Health check

Audits run in a background thread pool, so POST /api/audits returns immediately and the client polls GET /api/audits/{id} for progress.

2. Start the frontend (React + Vite)

cd frontend
npm install
npm run dev                        # serves http://localhost:5173

Open http://localhost:5173, enter your site's URL, and run an audit. The Vite dev server proxies /api to the backend on port 8000 (see vite.config.js), so no CORS setup is needed in development. Build for production with npm run build.

The web app adds a guided walkthrough of the audit steps, live crawl progress, a one-click PDF export, and sitemap.xml / robots.txt downloads alongside the full HTML report.

---

Generate a sitemap.xml & robots.txt

The crawler already maps your site, so it can also build a sitemap.xml from the pages it found and fetch your robots.txt (or write a sensible template if you don't have one).

Standalone script:

python scripts/generate_sitemap.py https://your-domain.com -o out/
#   out/sitemap.xml : every crawlable, 200-OK HTML page
#   out/robots.txt  : your existing robots.txt, or a generated template

Options: -m/--max-pages, -d/--delay, -o/--out-dir, -q/--quiet.

In the web app, the results view has sitemap.xml and robots.txt download buttons that generate the files from the audit in your browser.

---

Inferred keyword analysis

From the crawl data only (no Google Search Console, Ahrefs, SEMrush or any external API), the auditor infers what each page is optimised for using its on-page signals — title, headings, URL slug, first paragraph, body text, image alt, anchor text and meta keywords.

It produces, per page:

• an inferred focus keyword (weighted: title › H1 › URL › first paragraph › body),
• top keyword phrases via TF-IDF over 1/2/3-grams (stop-words removed, light stemming),
• keyword density, an SEO score, and concrete recommendations,
• on-page keyword issues (focus keyword missing from title/H1/URL/first paragraph, keyword stuffing, no clear focus, multiple H1, thin content);

and, site-wide:

• a keyword index (every inferred keyword → pages, average density, variations),
• keyword cannibalization (pages targeting the same focus keyword, with a similarity score and a suggested primary page).

In the web app this appears as a Keywords section: a focus-keyword card, a density chart, the cannibalization table, and a searchable per-page explorer with score/keyword filters. It's also included in the HTML report.

These are inferred / target keywords derived from on-page SEO signals — not actual search-engine ranking data.

---

Configuration (config.yaml)

All weights and thresholds live in config.yaml, loaded once — no magic numbers in code. It covers crawl limits + allowlist, rate limiting, the report paths, the (preserved) keyword-engine weights, and each pillar's thresholds. Crawling is white-hat: it stays on the start host (plus any crawl.allowlist hosts), obeys robots.txt and <meta robots> (noindex/nofollow), and is rate-limited.

python run.py https://your-domain.com -c config.yaml

Audit pillars

Pillar 1 — Crawl & indexability

Reuses the crawl to report, per page: HTTP status (4xx/5xx), redirect chains & loops, <meta robots> noindex/nofollow, canonical health (missing / self / cross-domain / different-URL), crawl depth, and orphan pages. Site-level: validates robots.txt and sitemap.xml (well-formed, reachable, in sync with the crawl) and checks hreflang reciprocity + x-default. Every finding is an Issue with a severity (critical/warning/notice), the pillar, the page_url, a message, and a concrete recommended_fix. Thresholds: indexability.* in config.yaml.

How the checks work (notes & caveats)

• Rate limiting. Every request — crawling, link verification, robots and sitemap fetches — goes through one Fetcher that waits delay seconds between calls. Lower it only against servers you control.
• Crawl scope. Stays on the same registrable host (www. is treated as the same site), skips mailto: / tel: / javascript: links, and stops at max_pages. It obeys robots.txt unless you pass --ignore-robots.
• Broken links. Targets already fetched during the crawl reuse their status. Internal links the crawl didn't reach (e.g. beyond max_pages, or PDFs) get a cheap HEAD request (falling back to GET).
• Page load. Measures server response + download time (close to TTFB) — not full browser render time. It's a lightweight proxy to surface slow endpoints and heavy HTML, not a Lighthouse replacement.
• Schema markup. Detects JSON-LD, microdata (itemscope) and RDFa (typeof). "Missing schema" is a notice, not an error — it's a suggestion.

---

Running the tests

The unit tests are fully offline (a fake fetcher stands in for the network):

pip install pytest        # or: pip install -e ".[dev]"
pytest

---

Exit codes

0 clean (no errors), 1 hard errors were found (useful in CI), 2 configuration/usage error.

Author

Aamir El Amiri — aamirelamiri3@gmail.com

• GitHub: github.com/ISTIFANO
• Instagram: instagram.com/aamir_el_amiri
• Discord: aamirelamiri202

---

License

MIT.