paper-ingest

Daily ML/AI paper + blog digest. Cron → fetch arXiv & RSS → Claude Code categorizes and writes notes → Slack digest.

Setup (new machine)

git clone https://github.com/layer6ai-labs/paper-ingest.git
cd paper-ingest
./setup.sh          # installs deps, captures your Slack webhook, prints a crontab line

Then do one manual bash scripts/run_ingest.sh before adding the printed cron line. The detailed manual walkthrough (and per-step verification) is in Quickstart below.

Syncthing / Obsidian users: if you clone this into a Syncthing-synced vault, add path/to/paper-ingest/.git to the vault's .stignore so Syncthing never replicates git internals (that can corrupt the repo). The generated notes/ are gitignored and stay local.

Quickstart

One-time setup

1. Install Python deps (only third-party dep is feedparser and pyyaml):

   uv pip install --user feedparser pyyaml
   # or: pipx install feedparser pyyaml  /  pip install --user feedparser pyyaml

2. Slack webhook:

   echo "https://hooks.slack.com/services/XXX/YYY/ZZZ" > ~/.config/paper-ingest/slack_webhook
   chmod 600 ~/.config/paper-ingest/slack_webhook

3. Curate config/feeds.yaml — the seed list is a starting point. Replace <handle> placeholders, comment out anything you don't want.

4. Verify pieces independently before letting cron run it:

   # A. Fetchers
   python3 scripts/fetch_arxiv.py
   python3 scripts/fetch_feeds.py
   ls -lh ~/.cache/paper-ingest/
   
   # B. Claude step interactively (no -p) the first time so you can watch it
   claude "$(cat prompts/ingest.md)"
   # inspect: notes/$(date +%Y-%m-%d)/_summary.md and a sample per-paper note
   
   # C. Slack post in isolation
   python3 scripts/post_slack.py
   
   # D. Full wrapper end-to-end
   bash scripts/run_ingest.sh
   
   # E. Cron sanity — run with cron's environment to catch PATH issues
   env -i HOME=$HOME PATH=/home/barry/miniconda3/bin:/home/barry/.local/bin:/usr/local/bin:/usr/bin:/bin \
     /bin/bash -lc '$HOME/Documents/Obsidian/projects/paper-ingest/scripts/run_ingest.sh'

5. Install cron (after a successful manual run). The PATH must include both miniconda (for python3 + feedparser/pyyaml) and ~/.local/bin (for claude):

   PATH=/home/barry/miniconda3/bin:/home/barry/.local/bin:/usr/local/bin:/usr/bin:/bin
   30 6 * * * /bin/bash -lc '$HOME/Documents/Obsidian/projects/paper-ingest/scripts/run_ingest.sh' >> $HOME/.cache/paper-ingest/cron.log 2>&1

   Verify the binaries are reachable under cron's environment with the dry-run command at the bottom of "Verify pieces independently". If claude or python3 aren't found, either fix PATH in the crontab or hardcode absolute paths inside run_ingest.sh.

Layout

projects/paper-ingest/
├── scripts/   fetch_arxiv.py, fetch_feeds.py, post_slack.py, run_ingest.sh
├── prompts/   ingest.md  — the headless Claude prompt
├── config/    feeds.yaml, arxiv_categories.yaml
└── notes/     YYYY-MM-DD/<category>/<slug>.md  (generated)

~/.cache/paper-ingest/         arxiv.json, feeds.json, seen_*.json, slack_summary.txt, run-*.log
~/.config/paper-ingest/        slack_webhook (mode 600)

Operational notes

• Persistent dedup: seen_arxiv.json and seen_feeds.json live in ~/.cache/paper-ingest/. Deleting them re-ingests everything within the 36h freshness window.
• RSSHub flakiness: public rsshub.app instances rate-limit aggressively. If a URL stops working, swap it; the fetcher already tolerates per-feed failures. For long-term Twitter/X reliability, self-host RSSHub on a small VPS.
• First-run advice: invoke claude interactively (no -p) on the first run to watch what it writes. Once happy, switch to run_ingest.sh.
• Watch the first 3 days before walking away — RSS edge cases (republished GUIDs, redirects, charset issues) only surface across runs.

Troubleshooting

Symptom	Likely cause
arxiv.json empty but no error	All papers older than 36h cutoff — fine on quiet days.
Cron run silent, no notes written	claude not on cron's PATH; hardcode absolute path in run_ingest.sh.
Slack post fails	Check ~/.config/paper-ingest/slack_webhook is readable and the URL is current.
Notes have category unknown/	Prompt couldn't classify — usually low-signal items; review and tighten prompt categories.
feedparser ImportError	Reinstall deps for the Python interpreter cron uses. head -1 scripts/fetch_arxiv.py shows the shebang.