Skip to content

development.md

Latest commit

 

History

History
54 lines (39 loc) · 1.45 KB

development.md

File metadata and controls

54 lines (39 loc) · 1.45 KB
Raw

Development Guide

Environment

uv sync --dev
uv run playwright install chromium

The project also works with a standard virtualenv, but uv is the supported development path because it keeps dependency resolution reproducible through uv.lock.

Common Commands

uv run ruff check src tests schemas
uv run pytest
uv run python -m json.tool schemas/source_manifest.schema.json >/dev/null
uv run python -m json.tool schemas/source_relationship.schema.json >/dev/null
uv run python -m json.tool schemas/scrape_manifest.schema.json >/dev/null

Or use:

make check
make test

Test Policy

  • Unit tests must not require live Substack network access.
  • Tests should use small fixtures or fake clients.
  • Paid content, cookies, and generated scrape output must never be committed.
  • Live target checks belong in private operator runbooks, not CI.

Config Policy

Tracked config files must be templates only:

  • config/*.example.yml
  • examples/**/config.example.yml

Private config files should use names such as config/my-target.local.yml, which are ignored by Git.

Scraper Design Constraints

  • Preserve source text verbatim after format conversion.
  • Do not summarize, paraphrase, or build wiki atoms.
  • Use stable Substack user/profile IDs for author-comment attribution.
  • Prefer structured Substack JSON/API data before rendered-page fallbacks.
  • Fail or warn explicitly when access, attribution, or parsing is incomplete.