A premium, state-of-the-art agentic platform designed to streamline academic research. Shouko-AI automatically discovers daily research papers, builds deep-dive interactive artifacts, and enables real-time, high-performance RAG chat.
- 🤖 Agentic Daily Discovery & Digests: Automatically scans ArXiv by user-defined categories, scoring and filtering new papers against personal interest profiles using LLM-based analysis (Groq/Llama 3.1 8B as primary, falling back to moonshotai/kimi-k2.6 via OpenRouter, and a deterministic local matching fallback).
- 📄 Deep PDF Ingestion & Storage Pipeline: Downloads PDFs, extracts high-fidelity text via PyMuPDF, chunks text using a sliding window (512 words, 50 overlap) with section-boundary heuristics, stores them as vector embeddings (nvidia/llama-nemotron-embed-vl-1b-v2:free 2048d) into pgvector, and persists PDFs in Cloudflare R2 (falls back to local filesystem).
- 🔬 High-Fidelity Artifacts: Uses advanced agentic workflows (moonshotai/kimi-k2.6 via OpenRouter, falling back to Llama-3.3-70b-versatile via Groq) to analyze paper chunks section-by-section, creating structured JSONB summaries, key takeaways, auto-generated Q&As, and suggested follow-up experiments. Includes a self-recovery retry mechanism (
POST /artifacts/{id}/retry). - 💬 Context-Aware RAG Streaming: High-performance Server-Sent Events (SSE) streaming chat that queries paper chunks using pgvector cosine similarity, with semantic caching (Redis) for fast response times.
- 📂 Collections & Library: Real-time organization tools allowing users to catalog, tag, and group paper artifacts into color-coded collections.
- 🔒 Token-Based Security & Rate Limiting: Robust JWT validation (HS256) with local user profile persistence. Includes Redis-backed sliding-window rate limiters and database-level usage tracking enforcing plan quotas.
- ✉️ Notification Sandbox: Compilation of daily digest summaries written directly to local sandbox HTML email files.
| Landing Page | Interactive Mindmap Canvas |
|---|---|
 |
 |
| Artifact Deep-Dive | Library View |
|---|---|
 |
 |
| Daily Digest & Chat | Settings & Interests |
|---|---|
 |
 |
Shouko-AI is structured as a modern monorepo with a Next.js frontend and FastAPI backend, coordinated via Celery background tasks and Redis.
┌──────────────────────────────────────────────────────────────┐
│ FRONTEND │
│ Next.js 14 (Vercel Edge Network) │
│ │
│ Dashboard │ Artifact View │ Chat Interface │ Digest Feed │
└────────────────────────┬─────────────────────────────────────┘
│ HTTPS / SSE
┌────────────────────────▼─────────────────────────────────────┐
│ BACKEND API │
│ FastAPI (Railway / AWS ECS) │
│ │
│ routers/ │ services/ │ core/ │ agents/ │
└─────┬──────────┬──────────┬──────────────────────────────────┘
│ │ │
▼ ▼ ▼
Postgres Upstash Cloudflare
pgvector Redis R2
(data + (cache + (PDF files)
vectors) task queue)
│
▼
┌─────────────────────────────────┐
│ CELERY WORKERS │
│ (Railway / AWS ECS Fargate) │
│ │
│ discovery_agent │
│ ingestion_agent │
│ artifact_agent │
└────────────┬────────────────────┘
│
┌───────┴────────┐
▼ ▼
OpenRouter Groq API
Kimi K2.6 Llama 3.1
│
▼
ArXiv API
- Frontend: Next.js 14 (TypeScript, App Router), Tailwind CSS, TanStack Query (React Query), Zustand, shadcn/ui.
- Backend: FastAPI (Python), SQLAlchemy, Alembic (Migrations), Celery (Background Tasks), Redis (Task Broker, Caching, Rate Limiting), pgvector (Vector Search).
- AI Engines: OpenRouter (nvidia/llama-nemotron-embed-vl-1b-v2:free for embeddings, moonshotai/kimi-k2.6 for completions/chat), Groq (llama-3.1-8b-instant / llama-3.3-70b-versatile as fallback).
paperbrain-ai/
├── apps/
│ ├── api/ # FastAPI Backend
│ │ ├── agents/ # LLM Agents (Discovery, Artifact Generation)
│ │ ├── alembic/ # Database Migrations
│ │ ├── core/ # Config, DB connections, Rate Limiting, Security
│ │ ├── models/ # SQLAlchemy Database Models (9 models)
│ │ ├── prompts/ # Versioned LLM prompts
│ │ ├── routers/ # API Endpoints (Papers, Chat, Users, Digests, Artifacts, Collections, Annotations)
│ │ ├── services/ # Business Logic (Ingestion, RAG, PDF, Embeddings, Storage, Digest)
│ │ ├── tasks/ # Celery Workers & Tasks
│ │ └── tests/ # Integration & Smoke pytest test suite
│ └── web/ # Next.js 14 Frontend
│ ├── app/ # App Router Pages & Layouts (Dashboard, Library, Artifact, Digest, Collections, Settings)
│ ├── components/ # UI, Chat, Artifact, & Layout Components
│ ├── hooks/ # TanStack Query & SSE Hooks
│ ├── lib/ # API Client
│ └── types/ # TypeScript Interfaces
├── docs/ # Architecture, Database Schema, AI architecture, API reference, Deployment docs
├── Images/ # Branding & Asset Gallery
└── infrastructure/ # Docker Compose configurations (Dev and Production), Nginx Config
sequenceDiagram
participant User
participant Router as Papers & Artifacts Router
participant Ingestion as Ingestion Service
participant PDF as PDF Service
participant Embed as Embedding Service
participant DB as PostgreSQL (pgvector)
participant Celery as Celery Worker
participant Agent as ArtifactAgent
participant LLM as LLMService (OpenRouter)
User->>Router: POST /papers/ingest
Router->>Ingestion: process paper
Ingestion->>Ingestion: download PDF
Ingestion->>PDF: extract_text_by_page()
PDF-->>Ingestion: pages
Ingestion->>PDF: chunk_text(pages)
PDF-->>Ingestion: text chunks
Ingestion->>Embed: get_embeddings(chunks)
Embed-->>Ingestion: vectors
Ingestion->>DB: store PaperChunks
Ingestion-->>Router: return paper info
Router-->>User: HTTP 201 Created
User->>Router: POST /artifacts
Router->>DB: create artifact (status: queued)
Router->>Celery: queue generate_artifact task
Router-->>User: HTTP 201 Created
Celery->>Celery: update status -> ingesting
Celery->>Agent: generate()
Celery->>Celery: update status -> generating
Agent->>LLM: generate_completion()
LLM-->>Agent: JSON output
Agent->>Agent: parse ArtifactOutput
Agent->>DB: store to Artifact table
Agent->>DB: update status -> ready
sequenceDiagram
participant User
participant Frontend as ChatPanel.tsx
participant Router as Chat Router
participant RAG as RAG Service
participant DB as PostgreSQL (pgvector)
participant LLM as LLMService (OpenRouter)
User->>Frontend: types question
Frontend->>Router: POST /conversations/{id}/chat (useSSEChat)
Router->>Router: check rate_limit
Router->>RAG: retrieve_context_chunks(question, artifact_id)
RAG->>DB: cosine similarity search
DB-->>RAG: top K chunks
RAG->>RAG: compile_rag_prompt(chunks, question)
RAG->>LLM: stream_chat_response(prompt)
LLM-->>Router: OpenRouter SSE stream
Router-->>Frontend: Server-Sent Events stream
Frontend-->>User: render in MessageList.tsx
Create a .env file in apps/api/ with the following variables:
| Variable | Required | Description |
|---|---|---|
ENVIRONMENT |
Yes | App environment (development | production | test) |
APP_SECRET_KEY |
Yes | Secret key used for signing JWT auth tokens |
DATABASE_URL |
Yes | PostgreSQL connection URI |
REDIS_URL |
Yes | Redis instance URI |
OPENROUTER_API_KEY |
Yes | OpenRouter key (used for embeddings & completions) |
GROQ_API_KEY |
No | Groq key (used for fast classification & fallback) |
R2_ACCOUNT_ID |
No | Cloudflare R2 Account ID (falls back to local filesystem if missing) |
R2_ACCESS_KEY_ID |
No | Cloudflare R2 Access Key ID |
R2_SECRET_ACCESS_KEY |
No | Cloudflare R2 Secret Access Key |
R2_BUCKET_NAME |
No | Cloudflare R2 Bucket Name |
Create a .env.local file in apps/web/ with the following variables:
| Variable | Required | Description |
|---|---|---|
NEXT_PUBLIC_API_URL |
Yes | Base URL of the FastAPI backend (e.g. http://localhost:8000) |
- Docker & Docker Compose
- Python 3.11+
- Node.js 18+
Spin up the local database and cache services:
docker compose -f infrastructure/docker-compose.yml up -d- Navigate to the API directory:
cd apps/api - Create and activate a Python virtual environment:
python3 -m venv venv source venv/bin/activate - Install dependencies:
pip install -r requirements.txt
- Configure environment secrets:
cp .env.example .env # Open .env and fill in: APP_SECRET_KEY, DATABASE_URL, REDIS_URL, and OPENROUTER_API_KEY - Apply database migrations:
alembic upgrade head
- Run the FastAPI development server:
uvicorn main:app --reload
In a new terminal window (with the virtual environment active in apps/api/):
celery -A tasks worker --loglevel=info- Navigate to the web app directory:
cd apps/web - Install npm packages:
npm install
- Run the Next.js development server:
npm run dev
- Access the web interface at
http://localhost:3000.
The backend includes a comprehensive pytest suite covering routing logic, artifact retries, auth profiles, and collections.
Ensure you have a local postgres instance running, then run:
cd apps/api
python3 -m pytestA production-ready multi-container configuration is provided. It sets up network isolation, persistent volumes, restart policies, and healthchecks.
To deploy the entire production stack (Postgres + pgvector, Redis, FastAPI, Next.js standalone runner, Celery Worker, Celery Beat):
docker compose -f infrastructure/docker-compose.production.yml up -dThis project is licensed under the MIT License.