CrabberBot 🦀

CrabberBot is a Telegram bot that downloads videos, photos, and galleries from sites supported by yt-dlp.

Try CrabberBot

Features

• Downloads supported videos, photos, galleries, and short playlists.
• Validates duration, file size, and playlist length before downloading.
• Caches Telegram uploads and audio so repeated requests avoid duplicate work.
• Offers optional audio extraction, transcription, and summarization through Telegram Stars subscriptions and top-ups.
• Uses a local Telegram Bot API server for uploads larger than the public Bot API limit.

Commands

• /start — usage guide
• /version — running version
• /subscribe — plans and top-ups
• /terms — Terms of Service
• /support <message> — contact support
• /refundme — request a refund for the most recent eligible purchase

Architecture

Docker Compose runs the following services:

Service	Responsibility	Network access
crabberbot	Rust webhook application, PostgreSQL access, Telegram uploads, and premium APIs	backend, internet
downloader-worker	yt-dlp, ffmpeg, and ffprobe behind a Unix socket	downloader only
egress-proxy	The worker's sole Internet path	downloader, internet
telegram-bot-api	Local Telegram Bot API server for large uploads	backend, internet
cloudflared	Cloudflare Tunnel for Telegram webhooks	backend, internet
postgres	Media cache, requests, billing, and callback state	backend only

The app and downloader communicate through /downloader/downloader.sock; the app image contains no yt-dlp or ffmpeg fallback. The worker passes all yt-dlp traffic through egress-proxy, so it cannot directly reach either the backend network or the Internet. The proxy is intentionally deployed as part of the stack, not as an optional application setting.

The app exposes GET /healthz, returning 204 only when PostgreSQL accepts SELECT 1; Docker uses it as the application health check.

Premium features

After a supported video download, the bot can offer:

• Extract Audio — cached MP3 produced by the worker.
• Transcribe — Deepgram transcription of the cached audio.
• Summarize — Deepgram transcription followed by OpenRouter summarization.

The public products are Basic (50 Stars/month, 60 AI Video Minutes), Pro (150 Stars/month, 200 AI Video Minutes plus unlimited audio extraction), and a 60-minute top-up (50 Stars). AI Video Minutes are charged by video duration. Top-ups work without a subscription and expire 365 days after the latest top-up purchase. The bot's /terms command is the authoritative customer-facing policy.

Premium callback contexts are bound to the Telegram user who requested the download. Quota reservations, payment fulfilment, and refunds are persisted in PostgreSQL to prevent duplicate charges or delivery.

Self-hosting

Prerequisites

• Docker Compose
• Telegram bot token from @BotFather
• Telegram API ID and hash from my.telegram.org
• Cloudflare Tunnel token and public webhook URL

Clone the repository:

git clone https://github.com/Enucatl/crabberbot.git
cd crabberbot

Create these secret files:

• secrets/telegram_bot_token
• secrets/telegram_api_id
• secrets/telegram_api_hash
• secrets/tunnel_token
• secrets/postgres_password
• secrets/deepgram_api_key
• secrets/openrouter_api_key
• secrets/owner_chat_id

Keep an empty .env file in the project directory because the Puppet deployment wrapper includes it when invoking Compose. Non-sensitive runtime defaults are defined directly in docker-compose.yml.

Compose mounts all sensitive values under /run/secrets; they aren't exposed in the application or Telegram Bot API container environments.

DEEPGRAM_API_KEY and OPENROUTER_API_KEY are required only for transcription and summarization. OPENROUTER_MODEL is optional and defaults to openai/gpt-6-luna; overrides must select an OpenRouter model with structured-output support. OWNER_CHAT_ID enables owner-only grants, support replies, and refunds.

Start the stack:

docker compose up -d

Development

Run formatting, Clippy, and the test suite with Docker available:

bash scripts/test.sh

The script starts a disposable PostgreSQL 18 container with its data in tmpfs, uses an automatically assigned local port, and removes it on exit. SQLx creates an isolated database for each storage test. PostgreSQL is required to test the application's SQL and locking behavior; SQLite in memory is not interchangeable.

For a local Compose build, use the supplied override and test environment:

CARGO_PACKAGE_VERSION=$(git describe --long | sed 's/-/./') \
  docker compose --env-file .env.test up --build

The worker builds yt-dlp from source. To refresh that Docker layer only when upstream changes, provide the current commit hash:

YT_DLP_COMMIT_HASH=$(git ls-remote https://github.com/Enucatl/yt-dlp.git refs/heads/master | cut -f1) \
  CARGO_PACKAGE_VERSION=$(git describe --long | sed 's/-/./') \
  docker compose build downloader-worker

Security

The Compose services extend the shared docker-compose-security-baseline for non-root execution, reduced capabilities, no-new-privileges, and memory, swap, and PID limits. See SECURITY.md for dependency-audit exceptions.

Contributing

Contributions are welcome. Please format the code and run the test suite before opening a pull request.