Website Content Scraper

A beautifully designed, production-ready web scraping application built with React and Express.js. Extract, analyze, and export website content with Apple-inspired design and enterprise-grade functionality.

🚀 Features

Core Functionality

• Smart Web Scraping: Extract titles, descriptions, content, headings, images, and links
• Batch Processing: Scrape multiple URLs simultaneously (up to 10)
• Real-time Validation: URL validation with visual feedback
• Multiple Export Formats: JSON, CSV, and Markdown export options
• Content Search: Full-text search across all scraped content
• Analytics Dashboard: Comprehensive insights and performance metrics

Design & UX

• Apple-Inspired Design: Clean, minimal interface with glassmorphism effects
• Dark Mode: System preference detection with manual toggle
• Responsive Design: Mobile-first approach with perfect desktop experience
• Smooth Animations: Framer Motion powered micro-interactions
• Toast Notifications: Real-time feedback for all user actions

Technical Features

• Rate Limiting: Respectful crawling with built-in rate limiting
• Error Handling: Comprehensive error handling with user-friendly messages
• Content Caching: Intelligent caching to avoid redundant scraping
• Progress Tracking: Real-time progress indicators for batch operations
• Data Persistence: Supabase database for reliable data storage

🛠 Tech Stack

Frontend

• React 18 with TypeScript
• Tailwind CSS for styling
• Framer Motion for animations
• React Query for state management
• React Hook Form for form handling
• Recharts for analytics visualization

Backend

• Node.js with Express.js
• Cheerio for HTML parsing
• Supabase with custom database layer
• Axios for HTTP requests
• Winston for logging

📦 Installation

1. Clone the repository

   git clone <repository-url>
   cd contentharvest

2. Install dependencies

   npm install

3. Start the development server

   npm run dev

The application will be available at http://localhost:5173 with the API server running on http://localhost:3001.

🔧 Configuration

Environment Variables

Create a .env file in the root directory:

NODE_ENV=development
PORT=3001
DATABASE_PATH=./contentharvest.db

Database Setup

The database is automatically initialized on first run. No manual setup required.

📖 Usage

Single URL Scraping

1. Navigate to the Web Scraper tab
2. Enter a valid URL in the input field
3. Click "Scrape Content" to extract data
4. View results in the Results tab

Batch URL Scraping

1. Switch to "Batch URLs" mode
2. Enter multiple URLs (one per line)
3. Use "Paste URLs" to import from clipboard
4. Click "Scrape All URLs" to process

Viewing Results

1. Go to the Results tab to see all scraped content
2. Use the search bar to filter results
3. Select multiple items for bulk operations
4. Export data in JSON, CSV, or Markdown formats

Analytics Dashboard

1. Visit the Analytics tab for insights
2. View scraping trends and performance metrics
3. Analyze top domains and content distribution
4. Monitor success rates and content quality

🎨 Design System

Colors

• Primary Blue: #007AFF
• Light Gray: #F5F5F7
• Dark Gray: #1D1D1F
• Success: #32D74B
• Warning: #FF9500
• Error: #FF3B30

Typography

• Primary Font: SF Pro Display (fallback to system fonts)
• Font Weights: 300, 400, 500, 600, 700
• Spacing: 8px grid system

Components

• Glass Effects: Backdrop blur with translucent backgrounds
• Cards: Rounded corners with subtle shadows
• Buttons: Three variants (primary, secondary, ghost)
• Inputs: Consistent styling with focus states

🔒 Security

• Input Sanitization: All user inputs are validated and sanitized
• Rate Limiting: 30 requests per minute per IP address
• CORS Protection: Configured for secure cross-origin requests
• Helmet Integration: Security headers for production deployment
• Content Security Policy: XSS and injection attack prevention

📊 API Documentation

Scraper Endpoints

POST /api/scraper/scrape

Scrape a single URL

{
  "url": "https://example.com"
}

POST /api/scraper/batch-scrape

Scrape multiple URLs

{
  "urls": ["https://example1.com", "https://example2.com"]
}

GET /api/scraper/content

Get all scraped content with optional search

?search=keyword&limit=50&offset=0

POST /api/scraper/export

Export content in various formats

{
  "format": "json|csv|markdown",
  "ids": ["id1", "id2"]
}

Analytics Endpoints

GET /api/analytics/dashboard

Get comprehensive analytics data

🚀 Deployment

Production Build

npm run build

Docker Deployment

FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
RUN npm run build
EXPOSE 3001
CMD ["npm", "start"]

Environment Setup

• Set NODE_ENV=production
• Configure proper database path
• Set up reverse proxy (nginx recommended)
• Enable HTTPS in production

🧪 Testing

# Run linting
npm run lint

# Run type checking
npx tsc --noEmit

# Test API endpoints
curl -X POST http://localhost:3001/api/scraper/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

🛣 Roadmap

• Chrome extension for one-click scraping
• Webhook notifications for completed scrapes
• Team collaboration features
• API rate limiting per user
• Scheduled scraping (cron jobs)
• Advanced content analysis with AI
• Browser bookmarklet
• Slack/Discord bot integration

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Design inspiration from Apple's Human Interface Guidelines
• Icons provided by Lucide React
• Charts powered by Recharts
• Animation library by Framer Motion

📞 Support

For support, email support@contentharvest.com or create an issue on GitHub.

---

Built with ❤️ and attention to detail. Happy scraping! 🕷️