Live at:
Automated Web Scraper is a full-stack web application that implements a structured and measurable web scraping pipeline.
It allows authenticated users to:
- 🌐 Scrape structured data from websites
- 📊 Measure runtime and memory usage
- 📈 Calculate scraping efficiency
- 📁 Export results as CSV
- 🗂 Store scraping history in MongoDB
The system follows a clear:
Fetching → Extraction → Execution architecture model.
flowchart TD
A[React Frontend<br/>Vite + Tailwind] --> B[FastAPI Backend]
B --> C[Authentication Layer<br/>JWT + Bcrypt]
B --> D[Scraping Pipeline]
D --> E[Fetch Stage<br/>Requests]
D --> F[Extract Stage<br/>BeautifulSoup + lxml]
D --> G[Execution Stage<br/>JSON + CSV Output]
G --> H[Performance Engine<br/>Runtime + Memory + Efficiency]
B --> I[MongoDB]
I --> J[Users Collection]
I --> K[Scraping History Collection]
subgraph Frontend
A
end
subgraph Backend Core
B
C
D
end
subgraph Scraping Engine
E
F
G
H
end
subgraph Database
I
J
K
end
- Built using React + Vite
- Built with FastAPI
- 🔐 JWT Authentication
- 📊 Runtime Monitoring
- 🧠 Memory Usage Tracking
- 📈 Efficiency Ratio Calculation
- 📁 CSV Export
- 🗂 Scraping History Persistence
- ⚙ Optional Selenium Fallback
- 🛡 Structured API Error Handling
- 🏗 Modular Backend Architecture
Recommended production architecture:
| Component | Platform |
|---|---|
| Frontend | Vercel |
| Backend | Render |
| Database | MongoDB Atlas |
- Python
- FastAPI
- React (Vite)
- MongoDB
- BeautifulSoup
- lxml
- JWT Authentication
This is just the initial stage of this project ,it is not the final or complete version of it, We will be enhancing it with course of time.
If you feel there is a better direction to approach the idea , you can reach out to me , I am open to suggestions. 😊
If you like the idea , give it a star and keep watching this repo. Thanks 😁 !
Sayantan Dutta
MIT License