A scraping and analysis pipeline for opportunity.org.nz. Web content is scraped, cleaned into canonical markdown, and built into a static site — ready for future analysis or introspection of downstream tooling on the party's people, policies, or manfiest. See LICENSE for terms.
Requires macOS or Linux, Homebrew, and Python 3.12+.
brew bundle --file=scripts/Brewfile # direnv, fnm, uv, just, lefthook, …
direnv allow # load .envrc (DAGSTER_HOME + venv PATH)
just install # uv sync — Python deps
just dev # open the Dagster UIThis project is shaped for AI-assisted development: every workflow is observable, every command is reproducible, every tool choice is deliberate. The selection of building blocks of which tools + dependencies are used are more significant. Anything that breaks I can verify through dagster.io, with manual validation of transformations via obsidian.md.
flowchart LR
raw[data/sources/<br/>raw ingestor output] --> scrape[scrape]
scrape --> transform[transform]
transform --> clean[data/clean/<br/>canonical markdown]
clean --> build[site build]
build --> site[static site]
Policy detail PDFs from opportunity.org.nz (hosted on Google Drive) flow through a parallel pipeline alongside the scraped HTML — downloaded, extracted as structured markdown, validated, and rendered as HTML. Source PDFs are never served; only the extracted content reaches the site. See docs/pdf-extraction.md for the full process (tools, output paths, two-pass validation, and where to find the content for hosting on opportunity.org.nz). The per-PDF coverage report lives at docs/pdf-pipeline.md.
Three things to understand before you touch anything:
Dagster — the Python orchestrator. Every scrape, transform, and build step is an asset under pipeline/defs/assets/. New sources, transforms, and consumers are added by writing new assets and wiring them into a job. The UI (just dev) shows the full lineage of any output back to its raw source — useful for visualisation. AI agents can use the dg utility enabled by direnv.
direnv — auto-loads .envrc when you cd into the project. Combined with uv, every shell session gets the exact pinned toolchain (and a stable DAGSTER_HOME) without global installs. Run direnv allow once after cloning.
pi.dev — the AI coding harness this project is shaped for. Skills, agent context, and the Dagster observability surface are designed so an agent can pick up any task without re-explaining the codebase. Additionally I have used npx skills@latest for downloading useful skills for project development.
data/
├── sources/ # gitignored, raw ingestor output — write-only
└── clean/ # tracked, canonical markdown + meta.json — read by all consumers
Ingestors write to data/sources/; everyone else reads from data/clean/. Adding a new source or consumer, schema details, and the layer invariants all live in docs/data-architecture.md.
| Recipe | What it does |
|---|---|
just install |
uv sync — install Python deps |
just dev |
Launch the Dagster UI |
just check |
ruff check + ruff format --check + ty check (read-only, CI-safe) |
just fix |
Auto-fix lint and reformat |
just validate |
Verify links in data/clean/**/*.md |
just hooks-install |
Wire lefthook into .git/hooks (once after cloning) |
Only valiadation is running just check before opening a PR — it must pass. The same checks run automatically on git pre-commit. To glance over the project structure, start with docs/data-architecture.md for architecture and docs/data-schema.md for schema questions.
The Opportunity Party is one voice among many. The most useful analysis often comes from combining this corpus with other sources rather than reading it in isolation. Candidate sources worth adding if possible:
- News coverage that mentions the party or its policies
- YouTube/podcast transcripts (
youtubeingestor is already in place) - Social feeds (X, Facebook, SubStack) via API clients
- Parliamentary records, select-committee submissions
- External newsletters and policy commentary
A browsable static site that mirrors the party's public-facing pages as plain markdown would be useful for archive and research access — particularly when the live marketing site changes and a snapshot is needed for citation. I'd host this if there's interest. For anything else, feel free to fork or open an issue with the source URL and what you want to extract — the pattern is small and well-defined.