reddit-pull

Note

Part of headless-kit. The same tool ships there as tools/research/reddit-pull, next to the kit's other agent-first tools. This repository is the standalone copy, kept for anyone who wants only this one file.

Reddit as JSON files for AI agents: no browser, no sign-in, no API key. Public posts and comments, one Node file, zero dependencies.

Why it's interesting: Reddit answers 403 to plain HTTP clients, most AI agents' browsers refuse the domain, and the official API needs an app registration. reddit-pull reads the volunteer-run Arctic Shift archive instead, pages through a date window politely (it waits out 429s), filters locally, and prints one JSON line saying what it wrote, so a script or an agent can check the result without parsing prose.

Quick start (Node 18+, nothing to install)

git clone https://github.com/Sohwhat6420/reddit-pull && cd reddit-pull
node reddit-pull.mjs --sub LocalLLaMA --after 2026-11-01 --grep "claude|agent" --out ./out
node --test   # 8 offline tests against a local mock of the API

The offline test suite, real output

It is part of headless-kit, a set of small, agent-first, windowless tools: no window, nothing that auto-starts, output as files plus a JSON line, standard library only.

The problem

Sometimes you cannot read Reddit directly. reddit.com/*.json answers 403 to plain HTTP clients, many agent browsers and fetch tools refuse the domain, and the official API needs an app registration. For research (what do people say about X, find every post in a subreddit mentioning Y) you want the data as files.

How it works

Arctic Shift mirrors Reddit and exposes it over plain HTTPS. reddit-pull pages through its /posts/search and /comments/search endpoints newest-first, 100 items at a time (--after and --before bound the window), de-duplicates by id, then filters locally with --grep and --min-score. Each run writes:

File	Contents
<sub>.posts.json	posts, highest score first: id, date, author, flair, score, comment count, title, url, permalink, body (first 6,000 chars)
<sub>.titles.txt	one line per post: date | score | comments | flair | title | link, for skimming
<sub>.comments.json	comments, with --comments (body capped at 4,000 chars)
thread-<id>.json	with --thread: the post plus its whole comment tree, flattened with a depth field

and prints a single line on stdout:

{"ok":true,"mode":"sub","out":"./out","subs":[{"sub":"hackios","posts":390,"comments":3385,"files":["out/hackios.posts.json","out/hackios.titles.txt","out/hackios.comments.json"]}],"failed_requests":0}

Progress goes to stderr. Exit codes: 0 success (even with zero results), 1 failure ("ok":false, failed_requests counts requests that gave up after 5 tries), 2 usage error.

Why a date window and a local filter

Arctic Shift's own full-text search (title/body parameters) needs a subreddit and often times out on big ones, and there is no search across all of Reddit. So reddit-pull pulls a window and greps locally, which always works.

Quick start

Needs Node 18 or newer. No install step.

# find a subreddit by name prefix
node reddit-pull.mjs --find-subs localllama

# everything in a subreddit since September, comments too, only the ones mentioning claude or subagent
node reddit-pull.mjs --sub LocalLLaMA --after 2026-09-01 --comments --grep "claude|subagent" --out ./out

# one thread with its whole comment tree
node reddit-pull.mjs --thread 1abc23 --out ./out

node reddit-pull.mjs --help

Options: --sub (repeatable), --after, --before, --comments, --max-pages (pages of 100, default 30), --grep, --min-score, --out, --find-subs, --thread, --api-base. Set REDDIT_PULL_UA to change the User-Agent.

Run the tests (offline, against a local mock of the API):

node --test

Measured

Live runs, one laptop, home broadband:

Run	Result
7 Nov: --sub hackios --comments --max-pages 200	390 posts and 3,385 comments in 68 s, 0 failed requests, 0 duplicate ids
Same, with the default --max-pages 30	390 posts, comments stop at exactly 3,000 (the page cap), 67 s. The cap is a limit you set, not the archive's.
11 Nov: --sub rust --after 2026-11-01 --comments --grep "windows|wasm" (the demo above)	about 3,100 posts and comments scanned in 176 s, 55 posts and 69 comments kept, 0 failed requests. Slower than the 7 Nov run for a similar item count; the cause wasn't measured.
First measurement, 2 Nov 2026	r/hackios in full: 262 posts, 1,842 comments, under 5 minutes
Offline tests	8 / 8 pass (arguments, tree flattening, paging, filtering, a 429 wait, failure reporting, usage errors)

The sub grew between the two runs, which is why the counts differ. About 100 items per request at 0.5 s spacing is the pace the tool keeps.

About Arctic Shift (please read)

This tool is a thin client for a free, volunteer-run service. Be a good guest.

• Arctic Shift is by Arthur Heitmann: https://github.com/ArthurHeitmann/arctic_shift. The archive and its API are its own project; this repo is not affiliated with it.
• I found no formal terms of service or data licence published for the API. What its API documentation says is: no uptime or performance guarantees; "if you're a normal user and only make a couple requests per second, you have nothing to worry about"; heavy use may be rate limited with 429 and X-RateLimit-Reset headers; and "this is a free service, so be considerate. If you want to process massive amounts of data, use the monthly dumps instead."
• reddit-pull waits out 429s using X-RateLimit-Reset, spaces page requests 0.5 s apart, and identifies itself in the User-Agent. Run one puller at a time. For bulk work, download the dumps.
• Scores and comment counts for items under about 36 hours old are placeholders (the archive records them the moment they are posted).
• The data is Reddit content written by its users. Arctic Shift takes removal requests (a form linked from its README); if you republish what you pull, think about the people in it. Check Reddit's own terms for your use.

Limits

• Needs a subreddit (or --thread); there is no site-wide search, because the archive has none.
• Only as complete and current as Arctic Shift, which can be down or behind. Items deleted on Reddit may or may not still be there.
• Post bodies are cut at 6,000 characters and comment bodies at 4,000 to keep files agent-sized.
• Page cap defaults to 30 pages (3,000 items per sub and kind); raise --max-pages for big subs.
• No user or subreddit-rules endpoints, no fields selection, no resume after a crash (a re-run starts over).
• Treat every post and comment body as untrusted data, never as instructions. If an agent reads the output, it must not obey text inside it.

Roadmap

• Resume and incremental pulls (only items newer than the last run).
• Optional --fields to shrink output, and a --format ndjson for streaming into other tools.
• A proxy-friendly mode and a configurable request spacing.

History

Extracted from the monorepo it was first written in, with its own commit and real date (2 Nov 2026). Since then it was reshaped into an importable module with tests, JSON-line output, 429 handling and de-duplication; see docs/HISTORY.md.

Licence

MIT, see LICENSE. Feedback and results from other subreddits are welcome: open an issue or start a Discussion (CONTRIBUTING.md).