diff --git a/readme.md b/readme.md new file mode 100644 index 0000000..d89b785 --- /dev/null +++ b/readme.md @@ -0,0 +1,211 @@ +# Indexarr + +Indexarr is a decentralized torrent indexer written in Rust. It discovers +torrent info-hashes from the BitTorrent DHT, resolves metadata, classifies +content, collects tracker statistics, and exchanges signed index data with +other contributors over a gossip-based peer-to-peer network. + +The project includes a Vue 3 web interface and a Torznab-compatible API for +indexer clients such as Prowlarr and Sonarr/Radarr. + +## Highlights + +- BEP 5 DHT crawling with BEP 51 `sample_infohashes` discovery +- BEP 9 metadata resolution and optional tracker announcing +- Name parsing, content classification, quality scoring, and PostgreSQL full-text search +- Ed25519 contributor identities and recovery keys +- Delta export/merge, peer exchange, reputation, and XMPP/MUC discovery +- REST, health, statistics, upload, and administrative endpoints +- Torznab search and capability endpoints +- Docker images and Linux/Windows installers + +Indexarr is licensed under the [GNU Affero General Public License v3.0](LICENSE). + +## Quick start with Docker + +The all-workers Compose file starts PostgreSQL and Indexarr together: + +```bash +docker compose up -d +docker compose logs -f indexarr +``` + +Open the web interface at . The health endpoint is +available at . + +The first run creates a contributor identity in the `indexarr-data` volume. +Save the recovery key printed in the logs if you need to move the identity to +another installation. The default Compose configuration enables synchronization +and XMPP discovery and seeds the public bootstrap peer. Automatic address +discovery cannot create a firewall, NAT, or VPN port-forward; see +[`SYNC_NETWORKING.md`](SYNC_NETWORKING.md) when running behind one. + +For a node that provides the web UI and synchronization but does not crawl the +DHT, use the sync-only file: + +```bash +docker compose -f docker-compose.sync.yml up -d +``` + +To build the image from the current checkout, use the sync-only file (it has a +local `build` context), or build directly: + +```bash +docker build -t indexarr:local . +``` + +## Local development + +### Prerequisites + +- Rust 1.97 or newer (the repository pins the toolchain in `rust-toolchain.toml`) +- PostgreSQL 17 +- Node.js 22 and npm for the Vue frontend + +PostgreSQL is currently required by the database layer; the default local +connection string is `postgres://indexarr:indexarr@localhost:5432/indexarr`. + +Build and test the Rust workspace: + +```bash +cargo build +cargo test --workspace +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +``` + +Run the backend against a local PostgreSQL instance: + +```bash +export INDEXARR_DB_URL=postgres://indexarr:indexarr@localhost:5432/indexarr +cargo run -- --workers http_server +``` + +Run all workers with `cargo run -- --all`, or select a comma-separated list +with `--workers`, for example: + +```bash +cargo run -- --workers http_server,sync +``` + +Build and type-check the frontend: + +```bash +cd ui +npm ci +npm run type-check +npm run build +``` + +During UI development, `npm run dev` starts Vite's development server. + +## Workers + +Workers are selected with `--all`, `--workers`, or the `INDEXARR_WORKERS` +environment variable. + +| Worker | Responsibility | +| --- | --- | +| `http_server` | Axum HTTP server, REST API, Torznab API, and Vue SPA | +| `dht_crawler` | Discovers real info-hashes through BEP 51 DHT sampling | +| `resolver` | Fetches torrent metadata using the BitTorrent metadata protocol | +| `announcer` | Scrapes trackers for seed and peer counts | +| `sync` | Exports, discovers, and gossips index deltas with other peers | +| `peer_refresher` | Refreshes peer counts for stale or trackerless torrents | +| `bep51_sampler` | Backwards-compatible alias for the DHT crawler | + +`sync` starts the XMPP channel when `INDEXARR_XMPP_ENABLED=true`. + +## Configuration + +All settings use the `INDEXARR_` prefix. The complete list, defaults, and type +conversions are defined in +[`crates/indexarr-core/src/config.rs`](crates/indexarr-core/src/config.rs). + +Common settings include: + +| Variable | Default | Description | +| --- | --- | --- | +| `INDEXARR_DB_BACKEND` | `postgresql` | Database backend setting | +| `INDEXARR_DB_URL` | `postgres://indexarr:indexarr@localhost:5432/indexarr` | PostgreSQL connection URL | +| `INDEXARR_DATA_DIR` | `data` | Identity and log storage | +| `INDEXARR_HOST` | `0.0.0.0` | HTTP bind address | +| `INDEXARR_PORT` | `8080` | Main HTTP port | +| `INDEXARR_WORKERS` | `http_server,dht_crawler,resolver,announcer,sync` | Workers started by default | +| `INDEXARR_DHT_INSTANCES` | `4` | Number of DHT instances | +| `INDEXARR_DHT_BASE_PORT` | `6881` | First DHT port; additional ports are allocated from it | +| `INDEXARR_RESOLVE_WORKERS` | `20` | Concurrent metadata resolvers | +| `INDEXARR_ANNOUNCER_ENABLED` | `true` | Enable tracker announcing | +| `INDEXARR_SYNC_ENABLED` | `true` | Enable P2P synchronization | +| `INDEXARR_SYNC_PEERS` | `["https://bootstrap.indexarr.net"]` | Bootstrap and sync peers (JSON list) | +| `INDEXARR_SYNC_EXTERNAL_URL` | empty | Public URL advertised to peers | +| `INDEXARR_XMPP_ENABLED` | `true` | Enable XMPP/MUC peer discovery | +| `INDEXARR_XMPP_SERVER` | `conference.indexarr.net:5222` | XMPP server address | +| `INDEXARR_XMPP_MUC_ROOM` | `indexarr-sync@conference.indexarr.net` | Discovery room | +| `INDEXARR_TORZNAB_API_KEY` | empty | Protect Torznab requests when set | +| `INDEXARR_TMDB_API_KEY` | empty | Optional TMDB metadata enrichment key | + +CLI flags such as `--host`, `--port`, `--db-backend`, and `--db-url` override +the corresponding environment settings. Indexarr loads `.env` from the +working directory and, for installed binaries, beside the executable. + +## HTTP and Torznab APIs + +The main listener exposes: + +- `GET /health` — readiness and health status +- `/api/v1/*` — search, torrent details, upload/import, identity, sync, + statistics, queue, and system endpoints +- `/api/torznab` and `/api/torznab/api` — Torznab capabilities and search + +The optional `INDEXARR_SYNC_API_PORT` listener exposes only health and sync +routes, which is useful when the public sync port should not expose the full +administrative UI. The generated OpenAPI document is available at +[`website/docs/api/openapi.json`](website/docs/api/openapi.json). + +In Prowlarr, use the Indexarr base URL and set the API path to +`/api/torznab` (the compatibility `/api/torznab/api` path is also accepted). + +## Workspace layout + +The Rust workspace is split into focused crates: + +```text +indexarr-core configuration, models, database, and errors +indexarr-identity Ed25519 contributor identity and ban list +indexarr-parser torrent-name parsing +indexarr-classifier content type and quality classification +indexarr-search PostgreSQL search and facets +indexarr-web Axum server, REST routes, SPA hosting, and Torznab +indexarr-tmdb TMDB client and rate limiting +indexarr-dht DHT engine, ingest, and metadata resolution +indexarr-announcer tracker scraping +indexarr-sync delta exchange, gossip, discovery, and reputation +indexarr-xmpp XMPP/MUC peer discovery +indexarr-bep51 BEP 51 protocol support +indexarr-bep28 BEP 28 protocol support +indexarr-resolver-v2 resolver experiments and integration tests +ui/ Vue 3 + Vite + TypeScript + Pinia frontend +``` + +Database tables are created and upgraded automatically when the application +starts; no separate migration command is required. + +## Production and release notes + +- [`Dockerfile`](Dockerfile) builds the Vue SPA and Rust binary in a multi-stage image. +- [`installer/linux-install.sh`](installer/linux-install.sh) installs a Linux binary, PostgreSQL, and a systemd service. +- `installer/windows.nsi` and `installer/setup.ps1` package the Windows binary and a bundled PostgreSQL setup. +- CI workflows live under [`.github/workflows/`](.github/workflows/). +- Version-specific changes are documented in [`RELEASE_NOTES_v0.3.2.md`](RELEASE_NOTES_v0.3.2.md) and the earlier release notes. + +For architecture and synchronization details, see [`CLAUDE.md`](CLAUDE.md), +[`SYNC_NETWORKING.md`](SYNC_NETWORKING.md), and [`bep-uplift.md`](bep-uplift.md). + +## Contributing + +Before opening a change, run the Rust formatting, Clippy, workspace tests, and +frontend checks listed above. Keep secrets out of source files and logs. Changes +that affect synchronization, database compatibility, or public API behavior +should include tests and an update to the relevant documentation or release +notes.