Skip to content

SPEC.md

Latest commit

 

History

History
56 lines (41 loc) · 2.43 KB

SPEC.md

File metadata and controls

56 lines (41 loc) · 2.43 KB
Raw

SPEC.md — Reddit Data Analyser

Version: v0.1.0-poc Status: Working POC. Schema may break between versions.

Storage schema (corpus.db)

posts

id (PK, base36 without t3_ prefix), subreddit, author, title, selftext, score, num_comments, created_utc, permalink, url, is_self, over_18, fetched_at, source. Indexed on subreddit, created_utc, author.

comments

id (PK, without t1_ prefix), post_id, parent_id, subreddit, author, body, score, created_utc, permalink, fetched_at, source. Indexed on post_id, subreddit, created_utc, author.

subreddits

name (PK), subscribers, description, lang, over_18, discovered_via, discovered_at, selected.

pull_runs

id (PK), kind, target, started_at, finished_at, posts_added, comments_added, error, params.

CLI surface

Command Effect
init Create corpus.db
discover --query Q [--limit N] Reddit subreddit search, upsert into subreddits
select <name> Mark sub for default pulls
unselect <name> Remove from default pulls
list-subs Print config + discovered subs
pull [--subreddit S] [--since X] [--max-posts N] [--max-comments N] Bulk pull
pull-thread <url> Single thread (post via Reddit, comments via PullPush)
pull-seeds pull-thread for every URL in seeds.txt
stats Row counts + recent runs
scan <pattern> [--subreddit S] [--limit N] LIKE search
surface [--subreddit S] Per-bucket keyword hits + top threads
export --kind {posts|comments|both} [--subreddit S] --output FILE JSONL dump

Config contract (config.yaml)

  • subreddits: list of {name, notes} dicts
  • default_window: {since, max_posts_per_sub, max_comments_per_sub}
  • rate_limit: {pullpush_delay_seconds, reddit_delay_seconds, max_retries, backoff_base}
  • taxonomy_keywords: {bucket_name: [keyword, ...]}

since format: <n><unit> where unit ∈ {h,d,w,m}, or "all".

Versioning

vMAJOR.MINOR.PATCH-stage. MAJOR for incompatible schema/CLI changes, MINOR for additive, PATCH for fixes. Bump on every PR that changes schema or CLI surface.

Known gaps (deliberate)

  • No backfill-posts command (see .claude/commands/backfill-posts.md)
  • No FTS5 — LIKE is fine until ~500k comments
  • No comment dedup across buckets (intentional — co-occurrence is signal)
  • No Arctic Shift integration yet
  • No incremental pull