Paper Summary System

A modular Python pipeline for scraping academic papers from DBLP, building a searchable vector database, and generating AI-powered summaries using Ollama LLM.

Korean : https://github.com/hyeeeee-kim/AutoSummary-Kor

---

Pipeline Overview

The system operates as a 4-step modular pipeline where each step is independent and can be executed in sequence:

Step 1: Build/Rebuild Database

• Scrapes enabled conferences from DBLP
• Fetches paper titles, abstracts, and links
• Stores papers in SQLite database with deduplication
• Detects duplicates based on title hash (MD5)

Step 2: Build Search Index

• Loads all papers from database into memory
• Creates vector embeddings using sentence-transformers (UAE-Large-V1 model)
• Stores embeddings in Chroma vector database
• Builds BM25 sparse index for keyword matching

Step 3: Search Papers

• Performs hybrid search (BM25 sparse + dense embeddings)
• Returns top 50 results ranked by combined score
• Combines sparse weight (0.3) and dense weight (0.7)
• Displays results with relevance scores and links

Step 4: Generate Summaries

• Reads PDF files from a user-specified directory
• Extracts text from PDF files
• Generates AI-powered summaries using Ollama LLM for each PDF
• Creates summaries: Overview, Method, Experiments
• Automatically generates markdown file (summary_pdf_[timestamp].md) in data/output/
• Uses filename as paper title

---

Installation and Setup

Prerequisites

• Python 3.8+
• Conda (Anaconda or Miniconda)
• Ollama (for LLM functionality)

Environment Setup

1. Create Conda environment

conda create -n paper_summary python=3.10
conda activate paper_summary

2. Install dependencies

pip install -r requirements.txt

Configuration Files

1. Create .env file

Copy .env.example to .env and configure with your settings

cp .env.example .env

Then edit .env with your actual values (see Environment Configuration section below).

• .env.example is included in the repository as a template

2. Configure conferences.yaml

Edit config/conferences.yaml to select which conferences to scrape (see Conference Configuration section below).

---

Running the Application

Basic Usage

python main.py

This starts the interactive menu where you can select steps 1-4.

Running Specific Steps Programmatically

from module.pipeline import Pipeline

pipeline = Pipeline()

# Step 1: Build database
pipeline.run_step_1_build_db()

# Step 2: Build search index
pipeline.run_step_2_build_index()

# Step 3: Search
pipeline.run_step_3_search()

# Step 4: Summarize from PDFs (generates markdown automatically)
pipeline.run_step_4_summarize()
# Prompts: "Enter directory path containing PDF files: "
# Output: Creates summary_pdf_[timestamp].md in data/output/

PDF Processing Workflow

Step 4 processes PDF files with streaming output:

Select step (0-5): 4

Enter directory path containing PDF files: D:\papers

[Processing PDFs with streaming output]
  Processing PDFs: 100%|████████| 10/10 [02:30<00:00, 15.00s/it]

OK: Processed 10 PDFs
Output: data/output/summary_pdf_1777856449.md

Key Features:

• Markdown file is written directly during processing (no memory buffering)
• Each PDF is processed and immediately written to file
• Memory usage stays constant regardless of PDF count
• Output path shown upon completion

---

Environment Configuration (.env)

Required Settings

HF_TOKEN (Optional)

HF_TOKEN=your_huggingface_token_here

• Used for accessing HuggingFace models
• Required if using specific gated models from HuggingFace
• Leave as placeholder if not needed

LLM Configuration

LLM_SERVER_URL=http://localhost:11434
LLM_MODEL=mistral

• LLM_SERVER_URL: URL where Ollama API is running

  • Default: http://localhost:11434 (local Ollama)
  • Can be changed to remote server: http://remote-host:11434

• LLM_MODEL: Model to use for summarization

  • Default: mistral
  • Alternative options: llama2, neural-chat, phi, etc.
  • Must be available in your Ollama installation

Embedding Model Configuration

EMBEDDING_MODEL=UAE-Large-V1

• Model for generating paper embeddings
• Default: UAE-Large-V1 (WhereIsAI/UAE-Large-V1)
• Maps internally to HuggingFace model identifiers

ChromaDB Telemetry

CHROMA_TELEMETRY_DISABLED=True

• Disables telemetry collection by Chroma database
• Keep as True to avoid compatibility issues
• Can be set to False if you want to help Chroma developers

Example .env File

# Copy this file to .env and fill in your actual values

HF_TOKEN=your_huggingface_token_here

LLM_SERVER_URL=http://localhost:11434
LLM_MODEL=mistral

EMBEDDING_MODEL=UAE-Large-V1

CHROMA_TELEMETRY_DISABLED=True # Default

---

Conference Configuration (conferences.yaml)

Structure

The config/conferences.yaml file defines which academic conferences to scrape:

conferences:
  conference_key:
    name: "Full Conference Name YYYY"
    url: "https://dblp.org/db/conf/xxx/xxxYYYY.html"
    enabled: true/false

Fields Explanation

Field	Type	Description
conference_key	string	Unique identifier (used internally, not displayed)
name	string	Full conference name with year (displayed to users)
url	string	DBLP conference page URL
enabled	boolean	true to include in scraping, false to skip

Modifying Existing Conferences

To disable a conference from being scraped, change enabled to false:

neurips:
  name: "NeurIPS 2024"
  url: "https://dblp.org/db/conf/nips/neurips2024.html"
  enabled: false  # This conference will be skipped

Adding New Conferences

1. Find the DBLP page for the conference

   • Go to https://dblp.org/
   • Search for the conference name
   • Get the URL for the specific year

2. Add to config/conferences.yaml

your_new_conference:
  name: "Conference Name YYYY"
  url: "https://dblp.org/db/conf/xxx/xxxYYYY.html"
  enabled: true

---

LLM Setup and Configuration

Installing Ollama

Windows

1. Download from https://ollama.ai/download
2. Run the installer
3. Ollama will start as a background service

macOS

1. Download from https://ollama.ai/download
2. Run the .dmg file
3. Ollama will run as a service

Linux

curl https://ollama.ai/install.sh | sh

Starting the LLM Server

Method 1: Automatic (Recommended)

Ollama runs as a background service by default after installation. Verify it's running:

curl http://localhost:11434/api/tags

Method 2: Manual Start

# Terminal 1: Start Ollama server
ollama serve

# Terminal 2: Pull and run model
oollama run mistral

Available Models

Check available models:

ollama list

Pull a new model:

ollama pull mistral      # Faster, good for most tasks
ollama pull llama2       # Default, balanced performance
ollama pull neural-chat  # Conversational, smaller

Changing the LLM Model

To use a different model, update .env

LLM_MODEL=mistral

Then ensure the model is available in Ollama

ollama pull mistral

---

LLM Prompt Customization

Where Prompts Are Stored

Prompts are defined in module/summarizer.py in the SummaryGenerator class.

Current Summary Format

The default summarization generates 3 parts: Overview, Method, Experiments

Modifying Prompts

Simple Method: Edit the prompt template

Edit module/summarizer.py:

def generate(self, title: str, abstract: str, context: str = None) -> str:
    """Generate summary from title and abstract"""
    prompt = f"""
Please provide a concise summary of this paper in 3 parts:

Title: {title}
Abstract: {abstract}

Summary format:
1. Overview: What is this paper about?
2. Method: How did they approach the problem?
3. Experiments: What experiments were conducted?

Keep each section to 2-3 sentences maximum.
"""
    # ... rest of code

Tuning LLM Parameters

Edit the API call in summarizer.py:

response = requests.post(
    f"{self.server_url}/api/generate",
    json={
        "model": self.model,
        "prompt": prompt,
        "temperature": 0.7,        # 0.0 = deterministic, 1.0 = creative
        "num_predict": 500,         # Max tokens in response
        "top_p": 0.95,              # Nucleus sampling
        "top_k": 40,                # Top-k sampling
    }
)

---

Project Structure

paper_summary/
├── main.py                 # Entry point
├── requirements.txt        # Python dependencies
├── .env                    # Environment variables
├── .gitignore             
├── README.md              
│
├── config/
│   ├── __init__.py
│   ├── settings.py        # Configuration from environment
│   └── conferences.yaml   # Conference definitions
│
├── module/
│   ├── database.py        # SQLite database operations
│   ├── scrapers.py        # DBLP web scraping
│   ├── rag.py             # Vector search and hybrid retrieval
│   ├── summarizer.py      # LLM-based summarization
│   ├── pdf_processor.py   # PDF text extraction and processing
│   ├── outputs.py         # Markdown export
│   └── pipeline.py        # Main orchestrator
│
└── data/
    ├── papers.db          # SQLite database
    ├── chroma/            # Vector database storage
    └── output/            # Generated markdown files

---

Notes

• All data is stored locally in data/ directory
• Database is in SQLite format (portable, no server needed)
• Embeddings are cached in Chroma for fast retrieval
• Conference scraping respects DBLP rate limits (with retry logic)
• LLM prompts can be customized without code changes (edit summarizer.py)