A powerful and user-friendly web scraping tool built with Streamlit, Selenium, and Google Gemini AI. This application allows you to scrape content from any website and extract specific information using natural language descriptions.
- Web Scraping: Uses Selenium to render and scrape dynamic web content.
- AI Parsing: Leverages Google's Gemini 1.5 Flash model to intelligently extract data based on your instructions.
- Clean UI: A modern, dark-themed interface built with Streamlit.
- DOM View: Inspect the raw and cleaned DOM content before parsing.
- Robust Handling: Handles various response formats and ensures reliable data extraction.
Before running the application, ensure you have the following installed:
- Python 3.8+
- Google Chrome
- ChromeDriver (Must match your Chrome version and be placed in the project root)
-
Clone the repository (if applicable) or navigate to the project directory.
-
Install dependencies:
pip install -r requirements.txt
-
Set up Environment Variables:
- Create a
.envfile in the root directory. - Add your Google Gemini API key:
GOOGLE_API_KEY=your_api_key_here
- You can get an API key from Google AI Studio.
- Create a
-
Start the application:
streamlit run main.py
-
Scrape a Website:
- Enter the URL in the input field.
- Click "Scrape Website".
-
Parse Content:
- Once scraped, expand "View DOM Content" to see what was captured.
- In the "Parse Content" section, describe what you want to extract (e.g., "Extract all product names and prices").
- Click "Parse Content" to let Gemini do the magic.
main.py: The main Streamlit application and UI logic.scrape.py: Handles Selenium web scraping and content cleaning.parse.py: Interfaces with Google Gemini for content parsing.requirements.txt: List of Python dependencies..env: Configuration file for API keys (not committed).
- Ensure
chromedriver.exeis in the root folder or updated inscrape.pypath. - The application uses
gemini-flash-latestfor fast and efficient parsing.