Skip to content

CLAUDE.md

Latest commit

 

History

History
81 lines (59 loc) · 4.32 KB

CLAUDE.md

File metadata and controls

81 lines (59 loc) · 4.32 KB
Raw

CLAUDE.md — Reddit Data Analyser

A retrieval pipeline that pulls Reddit threads/comments via PullPush + Reddit live JSON, stores them in SQLite, and surfaces candidate threads for Pakka's behavior-taxonomy refinement.

This is a corpus tool, not a detector. It retrieves & ranks. Pakka's LiteLLM-based detector (separate repo) decides what's a pattern.

Hard rules (do not violate)

  1. No detection/classification logic in this repo. No LLM calls for behavior labeling. LLM calls for data curation (cleaner tool, web/bff/services/cleaner_service.py) are allowed — they prune noise, not label behaviors. Keyword LIKE matches are retrieval helpers, not detectors.
  2. No personality-trait language anywhere. No "narcissist", "avoidant", "manipulator", "toxic". Taxonomy buckets describe observable behaviors, not the people who exhibit them.
  3. Rate limits are non-negotiable. PullPush is one person's free service. Default delays in config.yaml are floors, not targets.
  4. Idempotent writes only. Every insert is UPSERT.
  5. No author-level profiling. Authors stored for thread reconstruction. Don't build aggregate-by-author features.
  6. Don't reproduce Reddit content verbatim outside this tool. Exports for internal Pakka work only. Aggregate patterns and paraphrased excerpts only.

Architecture (keep it boring)

  • Flat layout. One module per concern at repo root. No src/, no nested packages.
  • Stdlib first: argparse, sqlite3, logging. External: requests, PyYAML only.
  • Two HTTP clients (pullpush.py, reddit_api.py). Never call each other.
  • One storage layer (store.py). Owns schema + writes. No raw SQL elsewhere.
  • CLI is the only entry point. No Flask, no Celery, no daemon.

Module responsibilities

File Owns Imports
corpus.py CLI subcommands; orchestration only all others
pullpush.py PullPush HTTP, pagination, backoff requests
reddit_api.py Reddit JSON: subreddit search, single thread, URL parsing requests
store.py SQLite schema, UPSERTs, reads, run audit sqlite3 (stdlib)
analyze.py Keyword surfacing, stats formatting store
config.yaml Subreddits, rate limits, taxonomy keywords —
seeds.txt User-supplied URLs for pull-seeds —

If a change blurs these lines, the change is wrong.

Conventions

  • Type hints on all function signatures. from __future__ import annotations at module top.
  • One-line docstring summary; no Sphinx/Google rituals.
  • Logging via log = logging.getLogger(__name__). No print() in library modules — CLI layer only.
  • Custom exceptions per source (PullPushError, RedditError). Don't swallow silently.
  • No tests yet (POC stage). Add when a bug recurs.
  • No mypy/ruff config. Solo project.

Common tasks

  • Add taxonomy bucket: edit config.yaml only. analyze.py is generic.
  • Add subcommand: function in corpus.py + subparser block + storage methods in store.py if needed.
  • Change schema: update SCHEMA in store.py, affected upsert_* / iter_* methods, bump SPEC.md. No migration system — document rm corpus.db && python corpus.py init in PR if breaking.
  • Add data source: new module (e.g., arctic_shift.py) mirroring pullpush.py shape. Wire through corpus.py.

Out of scope

  • Web UI (build a separate static frontend on corpus.db if you need one)
  • Multi-user / concurrent writes
  • Cloud storage, real-time ingestion
  • ML training inside this repo
  • AI explanations of bucket assignments (Pakka's job)

If a request involves the above: "different repo or different tool."

Diff hygiene

  • One concern per change.
  • 3+ file diffs are probably two changes.
  • Prefer new module over expanding existing one past ~400 lines.
  • corpus.py allowed to grow with new subcommands; others stay small.

When in doubt

The user (founder) prefers blunt pushback over agreeable execution. If a request violates a rule above, say so before doing it. If a rule itself seems wrong for the situation, propose changing the rule, then make the change.

Available slash commands

  • /add-taxonomy-bucket <bucket> [keywords...] — config-only edit
  • /add-subcommand <name> <intent> — scaffold new CLI command
  • /backfill-posts — implement orphan-post backfill
  • /audit-pulls — analyze pull_runs, suggest tuning
  • /verify-rate-limits — regression check on rate-limit code