🔎 RepoLens

AI-Powered Codebase Intelligence & Developer Onboarding

RepoLens helps developers explore and understand unfamiliar GitHub repositories through conversational AI.

Instead of treating source code as plain text, RepoLens parses supported programming languages using Tree-sitter, breaks the repository into meaningful code units such as functions and classes, converts them into vector embeddings, and retrieves the most relevant source code when a developer asks a question.

The result is a codebase-aware assistant that can answer questions such as:

• Where is authentication implemented?
• What does this service/function do?
• How is this repository structured?
• Which files are relevant to a particular feature?
• Where is a specific function or class defined?
• How does a particular part of the application work?

Responses are streamed to the interface and accompanied by references to the relevant source files and line ranges.

---

💡 Why RepoLens?

Understanding a new repository usually involves jumping between folders, searching for symbols, tracing functions, and reading large amounts of unrelated code.

RepoLens reduces that exploration process by combining:

AST-aware code understanding + semantic retrieval + symbol lookup + LLM reasoning

Rather than sending an entire repository to an LLM, RepoLens retrieves only the code that is most relevant to the developer's question.

This keeps the context focused while allowing the model to answer using the actual indexed source code.

---

⚙️ How It Works

RepoLens operates in two main stages.

1. Repository Indexing

GitHub Repository
        │
        ▼
Repository Validation
        │
        ▼
Shallow Git Clone
        │
        ▼
Tree-sitter Parsing
        │
        ▼
AST-aware Code Chunking
        │
        ▼
Vector Embeddings
        │
        ▼
Qdrant Vector Database

When a repository is submitted:

1. Repository metadata is validated.
2. The repository is shallow-cloned temporarily.
3. Supported source files are parsed using Tree-sitter.
4. Functions, classes, methods and other meaningful code structures are extracted as chunks.
5. Embeddings are generated for those chunks.
6. Embeddings and source metadata are stored in Qdrant.
7. Repository metadata, indexing statistics and generated architecture information are stored in MongoDB.
8. The temporary cloned repository is removed.

---

2. Codebase Question Answering

Developer Question
        │
        ▼
Question Embedding
        │
        ├───────────────┐
        ▼               ▼
Semantic Search    Symbol Lookup
        │               │
        └───────┬───────┘
                ▼
        Relevant Code Chunks
                │
                ▼
        Context-Augmented Prompt
                │
                ▼
               LLM
                │
                ▼
     Streamed Answer + Sources

For each question, RepoLens performs semantic vector retrieval to locate relevant source-code chunks.

When the query contains a recognizable function, class or identifier, RepoLens also performs an exact symbol-name lookup and combines those results with semantic retrieval.

The selected code is then supplied to the language model as context.

This allows the model to generate an answer grounded in the indexed repository rather than relying only on its pretrained knowledge.

---

✨ Core Capabilities

🌳 AST-Aware Code Indexing

RepoLens uses Tree-sitter rather than splitting code at arbitrary line or character boundaries.

This helps preserve structures such as:

• Functions
• Classes
• Methods
• Interfaces
• Type aliases
• Exported declarations

Currently supported languages include:

• JavaScript
• JSX
• TypeScript
• TSX
• Python

---

🧠 Semantic Code Retrieval

Source-code chunks are converted into embeddings and stored inside Qdrant.

Questions are embedded using the same embedding model, allowing RepoLens to retrieve code based on semantic similarity rather than requiring exact text matches.

---

🔍 Symbol-Aware Retrieval

Semantic search is complemented by exact identifier lookup.

For example, a question containing:

parseGithubUrl

can directly retrieve a chunk associated with that symbol even when vector similarity alone does not rank it highly.

---

💬 Conversational Code Exploration

Developers can ask natural-language questions about the repository without manually navigating every file.

Conversation information is persisted through MongoDB chat sessions so previous interactions can be supplied to the language model during a conversation.

---

⚡ Streaming Responses

LLM responses are streamed using Server-Sent Events (SSE).

Instead of waiting for the complete generated response, the frontend displays tokens as they arrive.

---

🔗 Source-Grounded Answers

Retrieved chunks retain metadata including:

• File path
• Symbol name
• Programming language
• Starting line
• Ending line

Relevant source references are displayed alongside generated responses, making it easier to verify the answer against the actual repository.

---

📊 Repository Overview

After indexing, RepoLens provides information about the repository such as:

• Number of processed files
• Number of generated code chunks
• Total indexed lines
• Language distribution
• AI-generated architecture summary

This gives developers an initial overview before they begin exploring individual components.

---

🛠️ Tech Stack

Layer	Technology
Frontend	React, Vite, Tailwind CSS
Backend	Node.js, Express.js
Database	MongoDB
Vector Database	Qdrant
Code Parsing	Tree-sitter
GitHub Integration	Octokit, Simple Git
Embeddings	OpenAI / OpenAI-compatible embedding models
LLM	OpenAI / Ollama-compatible models
Streaming	Server-Sent Events
Deployment	Vercel + Render

---

🏗️ System Architecture

flowchart LR
    USER[Developer] --> UI[React Frontend]

    UI -->|Repository URL| API[Express API]

    API --> GH[GitHub]
    API --> PARSER[Tree-sitter Parser]

    PARSER --> CHUNKS[AST Code Chunks]
    CHUNKS --> EMB[Embedding Model]
    EMB --> VECTOR[(Qdrant)]

    API --> DB[(MongoDB)]

    UI -->|Question| API
    API --> QEMB[Query Embedding]
    QEMB --> VECTOR

    VECTOR -->|Relevant Code| API
    API --> LLM[Language Model]

    LLM -->|SSE Stream| API
    API --> UI

Loading

The architecture separates repository ingestion from repository querying.

Indexing path

Clone → Parse → Chunk → Embed → Store

Query path

Question → Retrieve → Augment → Generate → Stream

---

🚀 Running RepoLens Locally

Prerequisites

Make sure you have:

• Node.js 20
• npm
• Git
• Docker, or hosted MongoDB and Qdrant instances
• OpenAI API access or an OpenAI-compatible local model provider

---

1. Clone the Repository

git clone <your-repository-url>
cd repolens

---

2. Install Backend Dependencies

cd server
npm install

Create the environment file:

cp .env.example .env

Configure the required environment variables inside .env.

---

3. Install Frontend Dependencies

cd ../client
npm install

---

4. Start MongoDB and Qdrant

From the project root:

docker compose up -d

---

5. Start the Backend

cd server
npm run dev

---

6. Start the Frontend

Open another terminal:

cd client
npm run dev

Then open the local Vite application and submit a small public GitHub repository for testing.

---

🗂️ Project Structure

repolens/
│
├── client/
│   └── src/
│       ├── api/
│       ├── components/
│       ├── pages/
│       ├── App.jsx
│       └── main.jsx
│
├── server/
│   └── src/
│       ├── config/
│       ├── models/
│       ├── routes/
│       ├── services/
│       │   ├── github.js
│       │   ├── chunker.js
│       │   ├── embedder.js
│       │   ├── vectorStore.js
│       │   ├── llm.js
│       │   └── indexer.js
│       │
│       ├── utils/
│       └── index.js
│
├── docker-compose.yml
└── README.md

---

🎯 Project Goal

RepoLens is designed to explore how Retrieval-Augmented Generation can be applied to structured source code rather than traditional documents.

The project combines compiler-style source parsing with semantic retrieval and generative AI to create a practical developer tool for:

codebase exploration, developer onboarding and repository intelligence.