Mastra RAG Example - Multi-Agent Systems Research

A Retrieval-Augmented Generation (RAG) system built with Mastra that enables intelligent querying of research documents about multi-agent systems using a local vector database.

Overview

This project demonstrates how to build a RAG system that:

• Processes PDF research documents from the data/ folder
• Chunks and cleans text content, removing irrelevant characters
• Generates embeddings and stores them in a local LibSQL vector database
• Provides an AI agent for querying the research content through Mastra's playground

Prerequisites

• Node.js >= 20.9.0
• Bun (recommended) or npm/yarn
• Git

Installation

1. Clone the repository (if not already done):

   git clone <your-repo-url>
   cd mastraragexample

2. Install dependencies:

   # Using Bun (recommended)
   bun install
   
   # Or using npm
   npm install

3. Set up environment variables: Create a .env file in the project root:

   DATA_FOLDER=./data
   VECTOR_DB=./db/vector.db
   OPENAI_API_KEY=your_openai_api_key_here

Usage

Step 1: Add Research Documents

Place your PDF research documents in the data/ folder. The system currently supports:

• PDF files (.pdf)
• The system will automatically process all PDFs in the folder

Note: The project comes with sample research papers about multi-agent systems. You can add your own PDF documents to expand the knowledge base.

Step 2: Generate Embeddings

Run the data processing script to generate embeddings and store them in the vector database:

# Using Bun (recommended)
npx bun src/dataTransform/store.ts

# Or using npm
npx tsx src/dataTransform/store.ts

This will:

• Load all PDF documents from the data/ folder
• Extract and normalize text content
• Chunk the text into manageable segments for better retrieval
• Clean the text by removing irrelevant characters and formatting
• Generate embeddings using OpenAI's embedding model
• Store everything in the local LibSQL vector database (db/vector.db)

Step 3: Start the Mastra Server

Initialize the local Mastra server:

# Using Bun
bun run dev

# Or using npm
npm run dev

This will start the Mastra development server and playground.

Step 4: Query the Research Data

1. Open your browser and navigate to the Mastra playground (usually http://localhost:4111/)
2. Use the research agent to ask questions about the multi-agent systems research documents
3. The agent will retrieve relevant information from the vector database and provide contextual answers

Project Structure

mastraragexample/
├── data/                    # PDF research documents
├── db/                      # Local vector database files
├── src/
│   ├── dataTransform/       # Data processing scripts
│   │   ├── generateEmbeddings.ts
│   │   ├── loadData.ts
│   │   ├── store.ts
│   │   └── textNormalizer.ts
│   └── mastra/              # Mastra configuration
│       ├── agents/
│       │   └── researchAgent.ts
│       └── index.ts
├── package.json
└── README.md

Features

• Local Vector Database: Uses LibSQL for local vector storage
• PDF Processing: Automatically extracts text from PDF documents
• Text Chunking: Breaks down documents into manageable segments for optimal retrieval
• Text Cleaning: Removes irrelevant characters and normalizes text for better embedding quality
• Research Agent: AI agent specialized for research document queries
• Mastra Integration: Leverages Mastra's RAG capabilities

Reset the Database

To regenerate the vector database:

# Delete the existing database
rm -rf db/vector.db*

# Regenerate embeddings
npx bun src/dataTransform/store.ts

Dependencies

• @mastra/core: Core Mastra framework
• @mastra/rag: RAG capabilities
• @mastra/libsql: Local vector database
• @ai-sdk/openai: OpenAI integration
• mammoth: PDF text extraction
• pdf-ts: PDF processing