A swarm intelligence engine that deploys AI agent "bees" to autonomously gather information from across the internet and feed it into a Karpathy-style LLM Wiki raw/ folder.
One command. One topic. Your knowledge base grows while you sleep.
hivemind deploy --topic "silicon chips"👑 Queen planning strategy for: "silicon chips"
👑 Queen planned: 5 listeners, 15 searchers, 5 agent missions
🔍 Registered searcher S-01: "TSMC advanced process node 2026"
🔍 Registered searcher S-02: "EUV lithography ASML latest developments"
📧 Registered listener L-01: "SEMI Newsletter" (bee-01@aliaskit.com)
✓ Auto-signed up via generic-form
🚀 Launching 5 agent bees for initial harvest...
🐝 Agent A-01: searching "TSMC 1.4nm process node" → 10 results
📄 Agent A-01: fetched https://www.tsmc.com/...
🐝 Agent A-01: spawning searcher → "SMEE Shanghai lithography"
✓ Agent A-01 complete: 14 files, 2 bees spawned
📊 Pulse: 23 new files. Coverage focused on TSMC N2 timeline and CHIPS Act.
🔔 Emerging: "Rapidus" — deploying agent...
🟢 Hive active. 24 bees. Autonomous mode.
156 markdown files in raw/ from one deploy. The swarm grows itself.
Karpathy's LLM Wiki pattern (17K+ GitHub stars) lets an LLM compile a structured wiki from raw source material. But someone has to find and dump that material manually.
Hivemind automates that. It deploys a swarm of bees that search the web, subscribe to newsletters, follow leads, and dump everything as markdown into your raw/ folder. Your existing Karpathy brain (Claude Code, wiki-skills, whatever you use) compiles it into wiki/.
Hivemind is the eyes and ears. Your Karpathy brain is the compiler. They connect through raw/.
Hivemind (this project) Your Karpathy brain (not this project)
┌─────────────────────┐ ┌──────────────────────────┐
│ Queen plans │ │ │
│ Searchers search │──raw/──>│ Claude Code compiles │
│ Agents dive deep │ │ into wiki/ with backlinks│
│ Listeners catch email│ │ and cross-references │
└─────────────────────┘ └──────────────────────────┘
The queen is the strategist. When you run deploy, she uses extended thinking to deeply analyze your topic and produce a comprehensive plan: which newsletters to subscribe to, which search queries to run, which angles need deep-dive research. She runs once at deploy, and can be re-invoked with hivemind review to reassess the swarm. The queen is the only component that thinks about the big picture.
Searcher bees run web searches on a schedule and save new results to raw/. No LLM needed. They're the backbone, the workhorse. They use a search API (Brave, Firecrawl, or SearXNG) to find content, then fetch full pages and convert them to markdown.
Agent bees are LLM-powered researchers. They chain searches, follow leads, and go deep on a specific mission. An agent might search for a company, find their CTO's name, search for their publications, find a related arXiv paper, discover a newsletter, and sign up for it. They dump everything they find to raw/ and spawn new searcher/listener bees for ongoing monitoring. They self-terminate when done. When an agent bee hits a paywall or registration wall, it can provision an AliasKit identity to sign up and access the content.
Listener bees have real email addresses (via AliasKit) and subscribe to newsletters. When an email arrives, it's converted to markdown and saved to raw/. Passive after setup.
AliasKit gives bees real email addresses, phone numbers, and signup capability. It's used by:
- Listener bees (required) to receive newsletter emails
- Agent bees (optional fallback) when they hit a registration wall during research
- Searcher bees (optional fallback) if a fetched page requires signup to access
Without AliasKit, searcher and agent bees still work fully via web search and public page fetching. AliasKit unlocks the ability to get past signup walls and subscribe to push sources.
Hivemind is the collection engine. It's designed to be coupled with other tools:
- Browser Use for complex signup flows with CAPTCHAs or multi-step forms
- Playwright for scraping JS-heavy sites that direct fetch can't handle
- Crawl4AI as a self-hosted alternative to Jina Reader for page-to-markdown conversion
- Any Karpathy wiki compiler (Claude Code, wiki-skills, llm-wiki-compiler) for turning
raw/into a structured wiki
The architecture is open. Bees dump markdown to raw/. What consumes it is up to you.
After you run deploy, the hive runs itself:
- Searcher bees run on cron schedules (every few hours), finding new content
- Pulse detector runs every 30 minutes, scanning new file titles for emerging entities
- When the pulse finds something new mentioned across 3+ sources, it auto-deploys an agent bee to research it
- The agent bee dives deep, dumps everything to
raw/, and spawns new searcher bees for ongoing monitoring - Health monitor kills dead bees (silent listeners, empty searchers)
- The swarm grows toward signal and contracts away from silence
You don't touch it. It expands and contracts on its own.
git clone https://github.com/Nalin-Atmakur/hivemind.git
cd hivemind
npm installnpx tsx src/cli.ts initThis creates two files:
.envfor API keys (gitignored, never committed)hivemind.yamlfor settings (safe to commit, no secrets)
Open .env and add at minimum:
# Pick one LLM provider
ANTHROPIC_API_KEY=sk-ant-...
# or
OPENAI_API_KEY=sk-...
# Pick one search provider
BRAVE_API_KEY=BSA... # Free 2K queries/month at api.search.brave.com
# or
FIRECRAWL_API_KEY=fc-... # Free 500 credits at firecrawl.devIf you want listener bees (email newsletter subscriptions):
ALIASKIT_API_KEY=ak_live_... # Get at aliaskit.comnpx tsx src/cli.ts deploy --topic "AI agent infrastructure"That's it. The queen plans, bees deploy, agents harvest, and raw/ fills up.
However you compile your wiki, point it at the raw/ directory:
# If using Claude Code with a wiki skill:
# Just tell it: "ingest everything in ./raw"
# If using a custom setup, set raw_dir in hivemind.yaml:
raw_dir: /path/to/your/wiki/raw# LLM Provider (pick one)
ANTHROPIC_API_KEY= # Claude models
OPENAI_API_KEY= # GPT models
# Search Provider (pick one)
BRAVE_API_KEY= # Free 2K queries/month (recommended)
FIRECRAWL_API_KEY= # Free 500 credits
# SearXNG needs no key, just set searxng_url in hivemind.yaml
# AliasKit (optional, for listener bees)
ALIASKIT_API_KEY= # Real email inboxes for newsletter subscriptionsllm_provider: anthropic # anthropic | openai
raw_dir: ./raw # Where bees dump markdown files
search:
provider: brave # brave | firecrawl | searxng
# searxng_url: http://localhost:8080
models:
queen: claude-sonnet-4-6 # Smart model for strategy (or o3)
agent: claude-haiku-4-5-20251001 # Cheap model for research (or gpt-4.1-mini)
pulse: claude-haiku-4-5-20251001 # Cheap model for entity extraction
autonomy:
auto_deploy_agents: true # Pulse auto-deploys agents for emerging entities
auto_kill: true # Health monitor auto-kills dead bees
max_concurrent_agents: 3 # Max parallel agent bees
max_total_bees: 100 # Swarm size cap
pulse:
interval_minutes: 30 # How often to scan for emerging entities
threshold_sources: 3 # Entity must appear in 3+ different sources
health:
kill_listener_silent_days: 21
kill_searcher_empty_polls: 5
check_interval_hours: 6
deploy:
initial_agents: 5 # Parallel agent bees for initial harvest
initial_searchers: 15 # Search queries the queen should generate# Setup
hivemind init # Create .env + hivemind.yaml
# Deploy (starts the hive, runs until Ctrl+C)
hivemind deploy --topic "your topic" # Queen plans + full deploy
# While the hive is running (in another terminal):
hivemind hunt "NeuralMesh founding team" # Deploy an agent on a specific mission
hivemind search "TSMC 1.4nm" --schedule "0 */6 * * *" # Add a searcher manually
hivemind kill S-14 # Kill a specific bee
hivemind review # Queen reassesses the full swarm
# Anytime:
hivemind status # Show all bees, agents, tracked terms
hivemind logs # Show last 50 log lines
hivemind logs -f # Live follow mode
hivemind config # Print current settingsBees need to turn web pages into markdown. Hivemind uses a fallback chain so it never fails:
- Jina Reader (free, 200/day) — best markdown quality
- Firecrawl scrape (if you have a key) — handles JS-heavy sites
- Direct fetch + turndown (always works) — raw HTTP + HTML-to-markdown conversion
If Jina hits its rate limit, the system silently falls through to the next option. No bee ever fails to fetch a page.
When the queen identifies a newsletter, Hivemind:
- Creates an AliasKit identity (real email address)
- Detects the platform (Substack, Beehiiv, ConvertKit, Buttondown)
- POSTs directly to the platform's signup API
- Falls back to parsing the HTML form and submitting it
No headless browser. No Python. Pure HTTP. Works for ~80% of dev newsletters.
For the other 20% (CAPTCHAs, custom flows), hivemind status shows the email address created for each listener. Visit the signup page and paste it in manually.
src/
├── cli.ts # CLI entry point
├── queen.ts # Strategic planner (LLM, runs once at deploy)
├── agent.ts # Deep researcher (LLM, temporary, self-terminates)
├── searcher.ts # Web search + fetch (no LLM, runs on cron)
├── listener.ts # Email inbox processing (no LLM, passive)
├── hive.ts # Orchestrator: cron, websocket, pulse, health, IPC
├── pulse.ts # Emerging entity detector (tiny LLM call on titles)
├── llm.ts # Unified Anthropic/OpenAI abstraction
├── signup.ts # Auto newsletter signup (Substack, Beehiiv, etc.)
├── search-api.ts # Search providers (Brave, Firecrawl, SearXNG) + smart fetch
├── dedup.ts # URL deduplication
├── raw-writer.ts # Consistent markdown output with frontmatter
├── state.ts # JSON/YAML state management + config + logging
└── types.ts # TypeScript interfaces
What uses LLM: Queen (strategy), Agent bees (research), Pulse (entity extraction from titles)
What doesn't: Searcher bees, Listener bees, Hive, Health monitor, Dedup, Signup, Fetch
Every file has consistent YAML frontmatter regardless of which bee type created it:
---
source: search:S-07
bee: S-07
type: searcher
date: 2026-04-15T14:22:00Z
url: https://example.com/article
query: "TSMC 1.4nm process 2026"
title: "TSMC Reveals N1.4 Details at Symposium"
---
TSMC revealed plans for its N1.4 node at its annual
technology symposium. Key details include...Your Karpathy compiler reads these files, extracts concepts and entities, and builds wiki articles with cross-references. The frontmatter tells it where the information came from.
All state lives in state/ as JSON/YAML files. No database.
state/
├── source-map.yaml # Queen's plan
├── bees.json # Active bees and stats
├── agents.json # Agent missions (running/completed)
├── seen-urls.json # URL dedup set
├── tracked-terms.json # Entities the pulse has flagged
├── agent-queue.json # Missions waiting for agent slots
├── pending-bees.json # Bees spawned by agents, waiting for registration
└── hivemind-log.md # Append-only activity log
The hive survives restarts. Stop with Ctrl+C, start again with deploy, it reads bees.json and picks up where it left off.
| Provider | Free tier | Key needed | Best for |
|---|---|---|---|
| Brave (recommended) | 2,000 queries/month | Yes, free at api.search.brave.com | General use |
| Firecrawl | 500 credits on signup | Yes, free at firecrawl.dev | JS-heavy sites, better extraction |
| SearXNG | Unlimited | No (self-hosted) | Power users with Docker |
| Provider | Models | Best for |
|---|---|---|
| Anthropic | Queen: claude-sonnet-4-6, Agent/Pulse: claude-haiku-4-5-20251001 |
Extended thinking for queen planning |
| OpenAI | Queen: o3, Agent/Pulse: gpt-4.1-mini |
If you already have an OpenAI key |
Set llm_provider in hivemind.yaml and the matching API key in .env. If both keys are set, it uses Anthropic by default.
Day 0: hivemind deploy --topic "silicon chips"
Queen plans: 5 listeners + 15 searchers + 5 agent missions
Agents run in parallel, dump 60+ pages to raw/
Agents spawn 8 new searchers from discoveries
Swarm: 5 listeners, 23 searchers
Day 3: Pulse detects "Rapidus" mentioned in 4 sources
Auto-deploys agent bee → researches Rapidus → 12 pages
Agent discovers Rapidus newsletter → spawns listener
Agent finds 2 new subtopics → spawns 2 searchers
Swarm: 6 listeners, 25 searchers
Day 7: Health monitor kills 2 dead searchers (0 new results)
Pulse deploys agent for "gallium nitride"
Swarm: 6 listeners, 25 searchers (grew, pruned, grew)
Day 30: 900+ files in raw/
Your wiki has 130+ articles
You never typed a command after day 0
- Node.js 18+
- One LLM key (Anthropic or OpenAI)
- One search key (Brave or Firecrawl) or self-hosted SearXNG
- Optional: AliasKit key for listener bees
Does this replace my Karpathy wiki setup?
No. Hivemind only fills raw/. Your existing compiler (Claude Code, wiki-skills, whatever) still handles raw/ to wiki/ compilation. They connect through the raw/ folder.
How much does it cost to run? LLM costs: ~$1-2 for initial deploy (queen + agents), ~$0.001 per pulse scan (every 30 min), ~$0.05-0.10 per agent bee mission. Searcher and listener bees use zero LLM. A typical month: $5-15 in LLM costs.
Can I use it for niche topics? Yes. The queen generates search queries for any topic. "Silicon chips", "rare earth mining in Mongolia", "UK fintech regulation" all work. For niche topics, there may be fewer newsletters to subscribe to, but searcher bees find content via web search regardless.
What if I only have an OpenAI key?
Set OPENAI_API_KEY in .env and llm_provider: openai in hivemind.yaml. Change model names to OpenAI models (o3, gpt-4.1-mini). Everything else works the same.
Can I run multiple hiveminds on different topics?
Yes. Create separate directories, each with their own hivemind.yaml, .env, state/, and raw/. Each runs independently.
How do I stop it?
Ctrl+C. The state is saved. Run deploy again to resume.
MIT