Add Reddit and Bluesky platform support (+ relevance.py None-fix) - #1
Open
billhector wants to merge 8 commits into
Open
billhector wants to merge 8 commits into
billhector wants to merge 8 commits into
Conversation
`post.get("description", "")` returns the default only when the key is
missing — but the platform fetchers can return posts where `description`
or `transcript` is present with a `None` value (e.g. Instagram captions,
YouTube videos with no transcript yet). That makes the str concatenation
in score_relevance crash with TypeError mid-run.
Use `(post.get(k) or "")` so an explicit None is treated the same as a
missing key. Defensive, no behavior change for valid posts.
Discovered while running a 7-day scrape across IG + YouTube on 2026-06-03;
script aborted in analyze_results before scoring/output.
Two new fetcher modules plus registry / scoring / URL-detection wiring, covering the two most-requested platforms not previously supported. **Reddit** (subreddit-centric tracking) - ScrapeCreators does not expose `/user/posts` for reddit, so reddit tracking is by subreddit, not by user. `handle` passed to fetch_profile is a subreddit name (with or without `r/` prefix); `author` on each returned post is the post submitter. - Endpoints used: /v1/reddit/subreddit (profile), /v1/reddit/post/comments (single post + comments — same endpoint serves both), /v1/reddit/post/transcript. - Engagement: ups/downs/score/comments/upvote_ratio. Scoring weights score × upvote_ratio + 3× comments so controversial high-vote posts don't dominate. - Post URL: canonical https://www.reddit.com/r/{sub}/comments/{id}/ **Bluesky** (user-centric, like X/IG/YT) - Endpoints used: /v1/bluesky/user/posts (profile), /v1/bluesky/post (single). - No comment endpoint — replyCount is included on each post but reply *text* is not retrievable. No transcript endpoint (Bluesky is text-only). Both registries intentionally omit bluesky. - Engagement: likes / reposts / replies / quotes. Scoring weights reposts + quotes 2×, replies 3× (active amplification > passive likes). - Post URL: derived from AT-proto URI → https://bsky.app/profile/{handle}/post/{rkey} **Wiring** - platforms.py — both registered in PROFILE_FETCHERS + POST_FETCHERS; reddit also in COMMENT_FETCHERS + TRANSCRIPT_FETCHERS. - scoring.py — added per-platform branches. - urls.py — detect_platform recognises reddit.com / redd.it / bsky.app; extract_handle_from_url returns the subreddit for reddit URLs and the bluesky handle for bsky URLs. - __init__.py — module docstring updated. Verified via smoke test that all six platforms are now in the registries and URL detection returns the expected platform for canonical URLs.
ScrapeCreators returns `credits_remaining` on most responses but the library used to throw it away. Now captured at the http layer and surfaced at the end of every scrape run — no extra API call, no quota cost. **lib/http.py** - Module-level `_last_credits_remaining` updates whenever a response contains the field. - New `get_last_credits_remaining()` reader. **scripts/scrape.py** - New `_emit_credits()` runs at end of both profile-mode and urls-mode. - Logs `[credits] credits_remaining: N` to stderr (visible alongside existing progress). - Writes a tiny single-int sidecar at `$CONTENT_HOME/.last-credits` when `CONTENT_HOME` is set, so external wrappers can read post-run quota without parsing log output. - Sidecar write is best-effort; failures log to stderr and don't abort the run. **SKILL.md** - New "wrapper-first invocation" guidance in Step 3: prefer `$CONTENT_HOME/bin/scrape.sh` if present, fall back to bare scrape.py. Lets per-user wrappers add credits-tracking / run-logging / alerting without forking the skill itself. Use case that motivated this: I'm on the paid plan with thousands of credits and want a daily run to auto-update an `api-usage.md` file with the new remaining balance and warn me when I drop below a threshold. The wrapper that does that lives outside the plugin (per-user), but it needs the credits to be reachable without a redundant probe call.
Adds an opt-in multi-project mode for users running multiple
non-overlapping content lines (a SaaS site + a personal brand +
a side project — each with different niche, pillars, competitors,
goals, and ideally a separate API key for attribution).
**Behavior in the two modes:**
Single-project (existing, default, no breaking change):
- One CONTENT_HOME, one ~/.config/content/.env, one brand/.
- Everything works exactly as before.
Multi-project (new, opt-in):
- /content-ideas <slug> → that project's feed
- /content-ideas → last-used slug from
~/.config/content/last-project
- Per-project state at
<CI_ROOT>/projects/<slug>/{brand,research}/
- Per-project key at ~/.config/content/<slug>.env
- Shared wrapper at <CI_ROOT>/bin/scrape.sh (rewritten in the
user's vault to be slug-aware; the wrapper passes wrapper exit
codes 3/4/5 back so the skill can distinguish missing-project,
missing-key, and empty-key states and trigger the right recovery
flow).
**SKILL.md changes:**
1. New "Resolve the project (multi-project mode)" section before
"Resolve the content home". Parses slug from $ARGUMENTS, falls
back to last-project file, discovers wrapper at conventional
locations (CONTENT_IDEAS_HOME, mise vault, ~/Documents/Content,
or $CONTENT_HOME/bin).
2. "Resolve the content home" gated on single-project mode (multi-
project resolves CONTENT_HOME from the slug above).
3. Step 0 now branches: detect first-run for single-project; detect
new-project for multi-project (project dir + key file existence
check). Setup writes to the right paths in either mode.
4. Step 3 wrapper-first invocation expanded: shows the
multi-project signature (`scrape.sh <slug> '<json>' ...`),
the single-project signature, and the bare-fallback. Documents
the wrapper's exit codes that callers should handle vs treat
as generic failures.
**commands/content-ideas.md:**
- argument-hint updated to mention the project slug
- skill invocation prose updated to describe the routing step
**Backward compatibility:**
Users with no slug and no wrapper continue using the original flow
unchanged — the new section short-circuits to "skip ahead to Step 0"
when neither slug nor wrapper is found.
**Note re relevance.py + reddit + bluesky:**
This commit stacks on the previous three on this branch (None-fix +
reddit/bluesky platforms + credits surfacing). Together they form
the full "I want to track multiple projects with proper credit
attribution and not lose work to plugin updates" story.
Adds a "What's new in this fork" section between the install/quick-start block and "The problem it solves". Strictly additive — Brad's existing prose is untouched. The new section walks through the four commits on this PR (bradautomates#1) at a level appropriate for a casual reader of the GitHub repo page: 1. Reddit + Bluesky platform support — including the subreddit-centric vs user-centric tracking distinction, what's missing (no Bluesky comments endpoint, no `/v1/reddit/user/posts`), and the engagement scoring weights chosen. 2. Credits surfacing — what credits_remaining lookup looks like (stderr log line + sidecar file), why it matters (zero extra calls for usage tracking), where the sidecar lands. 3. Multi-project mode — slug-aware routing, opt-in via wrapper detection, per-project state + per-project API key + shared credit pool with attribution. Reference wrapper kept in user-vault layer (not the plugin) so per-user logging concerns stay out of the plugin's scope. 4. relevance.py None-fix — one-line defensive fix; brief mention. Anyone landing on the fork on GitHub now sees the additions without having to read commit messages or the PR diff. Brad sees the docs land with the code so the PR is self-documenting.
…tent)
Every new install used to silently default to ~/Documents/Content with
no way to know that was the choice until files started landing there.
On macOS in particular this surprises users who keep their durable state
in iCloud Drive, an Obsidian vault, ~/Content, or some XDG-style hidden
location. By the time you notice, you've already accumulated a brand/
profile + a few research/ days in the wrong place and either have to
move them by hand or live with the default.
This change makes the first invocation ask explicitly.
**SKILL.md — new Step 0a / 0b / 0c**
Step 0 now begins with two short prompts BEFORE the existing API-key
flow:
0a. "Where should content-ideas store your files?" — only fires on a
truly fresh install (no ~/.config/content/.env, no <slug>.env
files, no CONTENT_IDEAS_HOME set anywhere). Offers
~/Documents/Content (default), ~/Content, ~/.local/share/
content-ideas, or paste-a-path. The chosen path is written as
CONTENT_IDEAS_HOME=... in ~/.config/content/.env so it persists
across shells/sessions without modifying the user's rc files.
0b. "What should we call this project?" — slug name. This is the
project identifier you pass as /content-ideas <slug>. Solo-feed
users pick anything that fits their brand; the multi-project
layout (projects/<slug>/) is a no-cost upgrade path for when
they add a second feed later. Validates slug shape.
0c. Existing-install / new-project detection — branches between
"skip to Step 1" (existing slug + key) and "run 0d-0g" (need
to set up this project).
The old 0a-0d (Welcome+API-key / Manual / Profile / Tracked accounts)
become 0d-0g, with refs across the file updated.
**Resolve-the-project section** updated to describe the new mode
defaults (multi-project for fresh installs, single-project legacy for
pre-multi-project installs) and to note that CONTENT_IDEAS_HOME is
resolved from BOTH the env var AND the .env file (env.py:
content_ideas_home).
**env.py — new resolver chain**
- `content_ideas_home()` — new public function. Resolves the
install root: env var → .env file → default ~/Documents/Content.
Used by Step 0a to find the persisted path; also used as
content_home()'s fallback so single-project users get the right
behavior automatically.
- `content_home()` — now respects CONTENT_HOME env (per-invocation
in multi-project mode) but falls back through
`content_ideas_home()` instead of the hard-coded default. No
behavior change for users who don't set CONTENT_IDEAS_HOME.
- Extracted `_read_env_file_value()` helper so both the new path
resolver and the existing API-key loader share one .env-parse
implementation.
- Updated module docstring to explain the two concepts (install
root vs per-run base dir) and how they relate to single vs
multi-project mode.
**Backward compat:** users whose install already exists at
~/Documents/Content/ (or whose CONTENT_HOME env var was already set)
are not re-prompted — the install-path check in Step 0a only fires when
there's NO sign of an existing install. They keep working unchanged.
**Why land this in PR bradautomates#1 instead of a separate PR:** continues the
pattern set by the multi-project-routing commit two commits back. The
two changes are the "set up + use multiple projects from day 1" story
together. If reviewers prefer to split later, the individual commits
are clean to cherry-pick.
Adds section 5 "Install-path setup on first run" under "What's new in this fork", documenting: - the day-1 pain point upstream creates (silent default to ~/Documents/Content, only noticed once files already accumulated) - the new Step 0a prompt — when it fires, what it offers (~/Documents/Content default, ~/Content, ~/.local/share/ content-ideas, paste-a-path), and where the choice is persisted (CONTENT_IDEAS_HOME=... in ~/.config/content/.env, durable across shells without touching rc files) - the auto-continuation into multi-project mode (Step 0b — name the first project) - the new content_ideas_home() resolver chain - the backward-compat guarantee (existing installs never re-prompted) Also: section intro updated from "the four commits" to "the commits" so it doesn't drift as the PR grows.
…ences/ Token-cache optimization. SKILL.md was 29.4KB; Step 0 (7 substeps for install-path + slug + API key + brand-profile + competitor seeding) is ~7.5KB of bootstrap content that only fires on the first run per project, not on the daily-feed common path. New: references/first-run-setup.md — full 0a-0g flow. SKILL.md: 29.4KB → 22.5KB. Saves ~7KB per daily-run fire after first run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds two new platforms to the scraper: Reddit (subreddit-centric) and Bluesky (user-centric), plus a small defensive fix in
relevance.py. Both new platforms use ScrapeCreators endpoints already available on the same API key — no new dependencies, no new secrets, no changes to existing platform behavior.Sent because it felt like a gap a lot of users would hit: I just wanted to track a couple of subreddits and one Bluesky creator alongside my IG/YT tracking, and adding both was straightforward following the existing per-platform module pattern.
Commits
fix: tolerate None values in relevance scoring(47d90df)post.get("description", "")returns the default only when the key is missing — but the platform fetchers can legitimately return posts wheredescriptionortranscriptis explicitlyNone. That crashesscore_relevancemid-pipeline withTypeError: can only concatenate str (not "NoneType") to str. Hit this on a real run before I'd touched any code. One-line change:(post.get(k) or "").feat: add Reddit and Bluesky platform support(eea427f)Reddit (
scripts/lib/reddit.py)/v1/reddit/user/posts, sohandleis a subreddit name (with or withoutr/prefix).authoron each returned post is the post submitter, used as metadata./v1/reddit/subreddit(profile),/v1/reddit/post/comments(single post + comments; same endpoint serves both),/v1/reddit/post/transcript(video transcripts, rare).ups,downs,comments,score,upvote_ratio. Scoring isscore × upvote_ratio + 3×commentsso controversial high-vote posts don't dominate.https://www.reddit.com/r/{sub}/comments/{id}/.Bluesky (
scripts/lib/bluesky.py)handleis a Bluesky handle (e.g.pfrazee.com); leading@is tolerated./v1/bluesky/user/posts(profile),/v1/bluesky/post(single).replyCountis on each post but reply text is not retrievable. No transcript endpoint either (Bluesky is text-only). Both registries intentionally omit bluesky.likes,reposts,replies,quotes. Scoring weights reposts + quotes 2× (active amplification) and replies 3×.https://bsky.app/profile/{handle}/post/{rkey}.Wiring
lib/platforms.py— both registered inPROFILE_FETCHERS+POST_FETCHERS; reddit also inCOMMENT_FETCHERS+TRANSCRIPT_FETCHERS.lib/scoring.py— added per-platform branches.lib/urls.py—detect_platformrecognisesreddit.com/redd.it/bsky.app;extract_handle_from_urlreturns the subreddit for reddit URLs and the bluesky handle for bsky URLs.lib/__init__.py— module docstring updated.Test plan
from lib.platforms import *clean, all six platforms in registries, URL detection returns expected platform for canonical URLs.r/Solopreneur(24 posts) +rikschennink.com(30 posts) over 7 days. Both populated correctly —subredditfield on reddit posts,bsky.appURLs constructed, outlier detection working, comments fetched on top reddit posts, no exceptions.pipeline.py,analyze.py) unchanged; new platforms slot in via the registry pattern.Notes
fetch_commentstobluesky.pyand register inCOMMENT_FETCHERS./v1/reddit/user/postsstyle), happy to follow up with a second tracking-mode that lets users be tracked alongside subreddits.