A self-hosted API data aggregation dashboard — configure external REST endpoints, fetch & store their data into MongoDB, browse and search records, all from a clean web UI.
Check out the live deployment of ThiruXDB here:
- URL: thiruxdb.indevs.in
- Username:
demo - Password:
demo@123
- Endpoint Manager — Add, edit, activate/deactivate, and delete API endpoints with support for
None,Bearer,API Key, andBasicauthentication - Dynamic Path Variables — Automatically orchestrate mass API fetching by injecting variables (like
{id}) into URLs from another MongoDB collection's fields. - Custom Collections — Store fetched data in a central
data_recordscollection or automatically route it to dedicated custom collections. - Field Mappings — Map and transform response fields (
string,number,boolean,date) to a unified schema - One-click Fetch — Pull data from a single endpoint or all active endpoints at once, with robust background fault tolerance and error skipping.
- Cancellable & Resumable Sync — Safely cancel active data syncs without corrupting data. Progress states (like
partial) are stored persistently and seamlessly resume on refresh. - Ultra-Fast Database Insertion — Automatically batches operations into memory and leverages MongoDB's
bulkWriteAPI to insert thousands of records simultaneously at absolute maximum wire speed. - Data Browser — Paginated table/grid view with date range filters, endpoint filtering, and full-text search.
- Fetch Logs — Complete history of every fetch operation with status, record counts, duration, and errors. Includes granular deletion and "Clear All" features to manage history.
- Live Terminal Logs — Real-time, streaming terminal view of your background Sync Engine operations, just like deployment logs, with auto-scrolling and log size monitoring.
- High-Performance Dashboard — Live stats cached directly on endpoint documents for instantaneous rendering of total records, active endpoints, and distributions.
- Export — Download current view as JSON or CSV
- Mobile Responsive — Fully optimized UI with sidebar overlays and responsive grids for managing data on the go.
- Role-Based Access Control (RBAC) — Three-tier permission system (
admin,editor,viewer) utilizing bcrypt password hashing. Includes granular, per-user page restrictions (e.g., blocking access to Endpoints, Fetch Data, Data Browser, or Fetch Logs). - Zero-Config Dynamic JWT Manager — The backend utilizes the
joselibrary to automatically generate a secure cryptographic secret, export it as a JWK (JSON Web Key), and persist it securely in MongoDB. No need to manage or configure aJWT_SECRETenvironment variable! - Dynamic Session Timeout — Admins can change the global session expiration time directly from the "System Security" panel in the UI (e.g.,
30mfor 30 minutes,12hfor 12 hours,7dfor 7 days). - Session Hijacking Prevention — JWT tokens are cryptographically fingerprinted using a hash of the user's IP Address and Browser User-Agent. Stolen or forged session tokens will instantly be invalidated if used on a different device or network.
- Anti-Export & Screenshot Security — If a user is restricted from exporting, the UI activates robust deterrence: text selection and right-clicking are disabled, Developer Tools are trapped, keyboard shortcuts (Ctrl+C, F12, Snipping Tool) are blocked, and the screen aggressively blurs to 0% opacity if the window loses focus to deter screen recording software.
- Public REST API Gateway — Generate secure API keys to expose aggregated data to external applications via
/api/v1/public/:collection. Includes dynamic filtering, pagination, MongoDB query translation, and highly granular rate limiting (per second/minute/hour) and usage quotas (per day/week/month). - Activity & IP Auditing — Comprehensive user activity logs with automatic IP Geolocation (
ip-api.com), tracking ISPs, Cities, and Device information. All API requests to third-party IP services are strictly proxied through the Node.js backend to bypass aggressive browser CORS and CSP restrictions. - Frappe UI "Espresso" Aesthetic — Sleek, flat design layout natively built in React + Tailwind.
- Global Light/Dark Mode — Fully responsive theme switching that persists in local storage.
- Local Auth & Root Admin Protection — Simple session-based login (no external auth service required). The primary Admin account's credentials are strictly enforced via the
.envfile, meaning the root password bypasses the database and cannot be tampered with or locked out via the UI.
ThiruXDB is designed to solve a very specific problem: Consolidating scattered data from dozens of different REST APIs into a single, highly searchable MongoDB database without writing custom scripts for every integration.
- Data Analysts & Scientists: Who need to pull massive amounts of data from third-party APIs (like weather data, financial stocks, or sports statistics) into a database for analysis, without needing to write Node.js or Python scraper scripts.
- Backend Developers: Who are migrating legacy systems or aggregating data from microservices and need a reliable, fault-tolerant background engine to sync the data without building custom ETL (Extract, Transform, Load) pipelines from scratch.
- OSINT Researchers: Who need to continuously poll public APIs, handle rate limits, and map varying JSON structures into a unified format for investigation.
- Agencies & Freelancers: Who want to hand off a beautiful, permission-controlled, white-labeled dashboard to clients so they can browse aggregated data and export it to CSV.
- No More One-Off Scripts: Stop writing brittle
fetch()loops in Python or Node.js every time you need to pull data from a new API. Just paste the URL into ThiruXDB, map the fields, and click fetch. - Built-in Fault Tolerance: Writing a script to download 10,000 items is hard because networks fail. ThiruXDB automatically handles connection drops, skips malformed data, and lets you safely pause/resume massive sync operations.
- Instant Admin Panel: Getting data into a database is only half the battle. ThiruXDB instantly gives you and your team a lightning-fast, searchable frontend to actually look at the data you just downloaded.
- Public REST API Gateway: Turn ThiruXDB into your organization's central Data Hub. Securely distribute aggregated data to your external apps or third-party clients via API Keys and
GET /api/v1/public/...endpoints, complete with built-in pagination, filtering, and API Rate Limits to prevent abuse. - Military-Grade Security: With built-in RBAC, JWT fingerprinting, dynamic cryptographic keys, and anti-export protections, you can safely give external clients or junior employees access to view the data without worrying about them deleting it, stealing it, or hijacking sessions.
| Layer | Technology |
|---|---|
| Runtime | Bun |
| Frontend | React 18 + TypeScript + Vite |
| Styling | Tailwind CSS |
| Backend | Express.js |
| Database | MongoDB (via native mongodb driver) |
| Icons | Lucide React |
ThiruXDB/
├── server/ # Express API backend
│ ├── index.js # Server entry point (port 3001)
│ ├── db.js # MongoDB connection + index setup
│ └── routes/
│ ├── endpoints.js # CRUD for api_endpoints collection
│ ├── records.js # Paginated records, upsert, text search
│ ├── logs.js # Fetch logs with endpoint name join
│ └── dashboard.js # Aggregated stats in one round-trip
├── src/ # React frontend
│ ├── lib/
│ │ └── api.ts # Typed fetch client (replaces Supabase client)
│ ├── types/
│ │ └── database.ts # Shared TypeScript interfaces
│ ├── context/
│ │ └── AuthContext.tsx # Local session auth
│ └── components/
│ ├── DashboardPage.tsx
│ ├── EndpointsPage.tsx
│ ├── EndpointForm.tsx
│ ├── FetchPage.tsx
│ ├── DataBrowserPage.tsx
│ ├── LogsPage.tsx
│ ├── LiveLogsPage.tsx
│ ├── LoginPage.tsx
│ └── Layout.tsx
├── .env.example # Environment variable template
├── vite.config.ts # Vite + /api proxy config
└── package.json # Bun scripts
ThiruXDB can be easily deployed to modern serverless and PaaS platforms. Below are the recommended deployment strategies, ranging from Serverless functions to Persistent Docker environments.
For production environments handling continuous background syncs, Persistent Servers are highly recommended. Unlike Serverless, persistent servers do not sleep between requests, allowing ThiruXDB's internal setInterval sync engine to run reliably.
- Render.com / Railway.app
- Build Command:
bun run build - Start Command:
bun run start(ornode server/index.js) - Note: ThiruXDB includes a built-in Anti-Sleep mechanism specifically designed for Render's Free Tier! It will automatically self-ping every 14 minutes to prevent your backend from sleeping during long background syncs.
- Build Command:
- VPS / Docker (DigitalOcean, AWS, Linode)
- Run via
pm2 start server/index.jsor build the included Dockerfile.
- Run via
ThiruXDB is natively configured to run on Serverless platforms like Vercel and Netlify out of the box! The repository includes a vercel.json config and an api/index.js serverless endpoint that perfectly bridges the Express backend with Vercel's Node.js runtime.
- Connect your GitHub repository to Vercel or Netlify.
- Set your Environment Variables (
MONGODB_URI,VITE_ADMIN_USERNAME,VITE_ADMIN_PASS). - Deploy! The platform will automatically build the React frontend and configure the Serverless API endpoints.
Note on Serverless Syncing: Because serverless functions spin down after responding to a request, ThiruXDB's internal background sync engine will only trigger when the API is actively receiving traffic. For strict cron-based syncing, a persistent server is required.
Currently, ThiruXDB cannot be deployed directly to Cloudflare Workers or Cloudflare Pages Functions. This limitation stems from the MongoDB Node.js Driver relying heavily on native Node.js core modules (such as net, tls, dns, and crypto) to establish binary TCP connections, which Cloudflare's V8 Isolates do not fully support.
- Bun
>= 1.0 - A MongoDB instance — MongoDB Atlas free tier works great
git clone https://github.com/ThiruXD/Endpoint-URL-Migration-To-MongoDB.git
cd Endpoint-URL-Migration-To-MongoDB
bun installcp .env.example .envEdit .env:
MONGODB_URI=mongodb+srv://<user>:<password>@cluster0.xxxxx.mongodb.net/?retryWrites=true&w=majority
MONGODB_DB=thiruXDB
PORT=3001
VITE_ADMIN_USERNAME=your_username_here
VITE_ADMIN_PASS=your_password_hereNote: Bun loads
.envautomatically — nodotenvpackage required.
bun run devThis starts both servers concurrently:
- Vite (frontend) →
http://localhost:5173 - Express API →
http://localhost:3001
Vite proxies all /api/* requests to the Express server, so the frontend just calls /api/....
Default credentials (change in src/context/AuthContext.tsx):
| Field | Value |
|---|---|
| Username | your_username |
| Password | your_password |
Stores configured API sources.
| Field | Type | Description |
|---|---|---|
_id |
ObjectId | Auto-generated |
name |
string | Display name |
collection_name |
string | Optional custom target collection (defaults to data_records) |
base_url |
string | The API URL to fetch |
path_variables |
array | Dynamic variables for URL injection (variable, source_collection, source_field) |
auth_type |
string | none | api_key | bearer | basic |
auth_config |
object | Auth credentials (headers, token, etc.) |
field_mappings |
array | Source → target field transform rules |
response_path |
string | Dot-path to array in response (e.g. data.results) |
pagination_type |
string | none | offset | cursor | page |
is_active |
boolean | Whether endpoint is enabled |
record_count |
number | Cached total of records belonging to this endpoint |
last_fetched_at |
Date | Timestamp of last successful fetch |
last_error |
string | Last error message, if any |
Stores fetched records from all endpoints.
| Field | Type | Description |
|---|---|---|
endpoint_id |
ObjectId | Reference to api_endpoints |
external_id |
string | Original id / _id from source API |
raw_data |
object | Complete original response item |
mapped_data |
object | Transformed data per field mappings |
_search_text |
string | Stringified raw_data for $text index |
fetched_at |
Date | When this record was fetched |
Tracks every fetch operation.
| Field | Type | Description |
|---|---|---|
endpoint_id |
ObjectId | Reference to api_endpoints |
status |
string | success | error | partial |
records_fetched |
number | Total items in API response |
records_created |
number | New records inserted |
records_updated |
number | Existing records updated |
duration_ms |
number | Total fetch duration |
error_message |
string | Error detail, if any |
All routes are prefixed with /api.
| Method | Path | Description |
|---|---|---|
GET |
/api/health |
Health check |
GET |
/api/dashboard |
All dashboard stats in one call |
GET |
/api/endpoints |
List all endpoints |
POST |
/api/endpoints |
Create endpoint |
POST |
/api/endpoints/sync-stats |
Re-calculates and repairs the record_count cache across all endpoints |
PUT |
/api/endpoints/:id |
Update endpoint |
PATCH |
/api/endpoints/:id/toggle |
Toggle active state |
PATCH |
/api/endpoints/:id/status |
Update last_fetched_at / last_error |
DELETE |
/api/endpoints/:id |
Delete endpoint + cascade records & logs |
GET |
/api/records |
List records (paginated, filterable) |
GET |
/api/records/search |
Full-text search |
GET |
/api/records/counts |
Total + per-endpoint record counts |
POST |
/api/records |
Upsert a record |
PUT |
/api/records/:id |
Update mapped_data |
DELETE |
/api/records/:id |
Delete record |
GET |
/api/logs |
List fetch logs |
POST |
/api/logs |
Create fetch log entry |
GET |
/api/endpoints/:id/live-logs |
Stream live fetch logs for an endpoint |
DELETE |
/api/endpoints/:id/live-logs |
Clear live fetch logs for an endpoint |
bun run dev # Start Vite + Express concurrently
bun run dev:client # Vite only
bun run dev:server # Express only (with --watch hot reload)
bun run build # Production Vite build
bun run typecheck # TypeScript type checking
bun run lint # ESLintContributions are highly encouraged and always welcome! We believe in the power of open-source and community collaboration. Whether you are fixing a bug, adding a new feature, or improving the documentation, your help is appreciated.
How to contribute:
- Fork the repository.
- Create your feature branch:
git checkout -b feature/my-new-feature - Commit your changes:
git commit -m 'feat: Add some amazing feature' - Push to the branch:
git push origin feature/my-new-feature - Open a Pull Request!
If you're not a developer, you can still contribute by opening issues for bugs or suggesting new features.
- Author: ThiruXD
- AI Assistance: Built with the help of Gemini AI and Claude AI for architecture, complex logic refactoring, and UI enhancements.
MIT