diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ab53047..f89efb6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -2,7 +2,7 @@ name: CI Pipeline on: push: - branches: [ main, develop ] + branches: [ main ] pull_request: branches: [ main ] @@ -13,25 +13,20 @@ jobs: steps: - uses: actions/checkout@v4 - - name: Set up Python - uses: actions/setup-python@v4 - with: - python-version: '3.10' - - name: Install dependencies run: | python -m pip install --upgrade pip - pip install -r requirements.txt + pip install pytest + - name: Set up Python + uses: actions/setup-python@v4 + with: + python-version: '3.12' + - name: Run tests run: | pytest test.py -v - - name: Test Streamlit app startup - run: | - timeout 10s streamlit run app.py --headless --server.port 8501 || true - echo "Streamlit app startup test completed" - code-quality: runs-on: ubuntu-latest @@ -41,7 +36,7 @@ jobs: - name: Set up Python uses: actions/setup-python@v4 with: - python-version: '3.10' + python-version: '3.12' - name: Install dependencies run: | diff --git a/README.md b/README.md index bf90be5..27606f1 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,7 @@ The system is split into three main containerized services: 1. **ML Model Backend**: A Python FastAPI application that: - Loads the Matrix Factorization model from S3 - Serves predictions via `/predict` endpoint + - Serves a `/feedback` endpoint for saving user feedback to DynamoDB. - Logs all requests to DynamoDB for monitoring - Handles model inference with supporting lookup tables @@ -50,45 +51,48 @@ The system is split into three main containerized services: - Provides a user-friendly interface for inputting favorite books - Displays real-time recommendations from the FastAPI backend - Handles user interactions and API communication + - Provides an option for feedback on recommendations. 3. **Model Monitoring Dashboard**: A Streamlit dashboard that: - Connects directly to DynamoDB to visualize prediction logs - Tracks prediction latency over time - - Monitors data drift and prediction distribution - Collects and displays user feedback for model accuracy + - Calculates Recall at K based on new feedback. ## Technology Stack -- **Backend**: FastAPI, Python 3.11+ +- **Backend**: FastAPI, Python 3.12+ - **Frontend**: Streamlit -- **ML Framework**: scikit-learn, joblib +- **ML Framework**: scikit-learn, joblib, implicit for ALS model - **Cloud Services**: AWS (S3, DynamoDB, EC2) - **Experiment Tracking**: Weights & Biases (W&B) -- **Containerization**: Docker, Docker Compose +- **Containerization**: Docker - **Data Processing**: pandas, numpy -- **Visualization**: matplotlib, seaborn +- **Visualization**: matplotlib, seaborn, plotly ## Prerequisites -Before you begin, ensure you have the following installed and configured: +Before running the code locally, ensure you have the following installed and configured: ### System Requirements -- **Python**: 3.11 or higher -- **Docker**: 20.10 or higher -- **Docker Compose**: 2.0 or higher +- **Python**: 3.12 or higher +- **Docker**: 20.10 or higher (if running locally) - **Git**: For cloning the repository +- **AWS**: If planning to run using EC2 and AWS. ### AWS Account Setup 1. **AWS Account**: Create an AWS account if you don't have one -2. **AWS CLI**: Install and configure AWS CLI (optional, for local development) +2. **AWS CLI**: Install and configure AWS CLI (optional for local development) 3. **Required AWS Services**: - **S3 Bucket**: Named `readcrumbs` (or update code to use your bucket name) - Store model files: `models/als_model-small-v1.pkl` - Store lookup tables: `data/v1/index_to_title.json`, `data/v1/title_to_index.json` - - **DynamoDB Table**: Named `readcrumbs-logs` (or set via `DDB_TABLE` env var) + - **DynamoDB Table**: Named `readcrumbs-logs` - Used for storing prediction logs + - **DynamoDB Table**: Named `readcrumbs-feedback` + - Used to store user feedback on provided predictions. - **EC2 Instance**: For production deployment - Recommended: t3.medium or larger - Ubuntu 22.04 LTS or Amazon Linux 2 @@ -96,7 +100,7 @@ Before you begin, ensure you have the following installed and configured: ### Weights & Biases Setup 1. Create a W&B account at [wandb.ai](https://wandb.ai) -2. Install W&B: `pip install wandb` +2. Install W&B: `pip install wandb` or run in Docker container 3. Login: `wandb login` 4. Create a project named `readcrumbs` (or update in code) @@ -112,7 +116,7 @@ Your AWS credentials need the following permissions: ## Environment Variables -Create a `.env` file in the project root (never commit this file to git). Here's the structure: +If not running inside Docker container on EC2 with IAM role, you should create a `.env` file in each separate part of the project (never commit this file to git). Here's the structure: ```bash # AWS Credentials @@ -192,70 +196,23 @@ aws s3 ls s3://readcrumbs/ aws dynamodb describe-table --table-name readcrumbs-logs ``` -## Running the Project Locally +## Running the Project With EC2 -### Using Docker Compose (Recommended) - -The entire system is containerized and managed via Docker Compose. - -#### 1. Build Containers - -Build the Docker images for all services: - -```bash -docker compose build -``` - -#### 2. Run All Services - -Start the entire MLOps system in detached mode: - -```bash -docker compose up -d -``` - -To see logs: - -```bash -docker compose up -``` - -#### 3. Verify Services are Running - -Check that all containers are up: - -```bash -docker compose ps -``` - -You should see three services running: -- `backend` (FastAPI) -- `frontend` (Streamlit) -- `monitoring` (Streamlit Dashboard) +- Create four EC2 containers in AWS for the frontend, monitoring dashboard, backend server, and training the model, respectively. +- SSH into each container, add the files using the public IP, and create a Docker container. +- Run the Docker container, which should start each service. #### 4. Access the Services Once running, access the services: -- **FastAPI Backend API**: http://localhost:8000 +- **FastAPI Backend API**: http://localhost:8000 (or http://ec2-ip-address:8000) - Health Check: http://localhost:8000/health - API Docs: http://localhost:8000/docs -- **Frontend Interface**: http://localhost:8080/ -- **Monitoring Dashboard**: http://localhost:8081/ - -#### 5. Stop Services - -To stop and remove containers: - -```bash -docker compose down -``` - -To stop and remove containers with volumes: - -```bash -docker compose down -v -``` + - Prediction endpoint: http://localhost:8000/predict + - Feedback endpoint: http://localhost:8000/feedback +- **Frontend Interface**: http://localhost:8501/ (or http://ec2-ip-address:8501) +- **Monitoring Dashboard**: http://localhost:8501/ (or http://ec2-ip-address:8501) ### Running Individual Services @@ -274,7 +231,7 @@ docker run -p 8000:8000 --env-file ../.env readcrumbs-backend ```bash cd frontend docker build -t readcrumbs-frontend . -docker run -p 8080:8501 readcrumbs-frontend +docker run -p 8501:8501 readcrumbs-frontend ``` ## AWS EC2 Deployment @@ -290,9 +247,8 @@ docker run -p 8080:8501 readcrumbs-frontend 2. **Security Group Configuration**: - Open the following ports: - **Port 8000**: FastAPI backend (HTTP) - - **Port 8080**: Frontend interface (HTTP) - - **Port 8081**: Monitoring dashboard (HTTP) - **Port 22**: SSH (for initial setup) + - Allow access via HTTP and SSH. 3. **IAM Role** (Recommended): - Attach an IAM role to your EC2 instance with S3 and DynamoDB permissions @@ -315,16 +271,12 @@ sudo apt-get update # Install Docker sudo apt-get install -y docker.io -# Install Docker Compose -sudo apt-get install -y docker-compose - # Add your user to docker group (to run without sudo) sudo usermod -aG docker $USER newgrp docker # Verify installation docker --version -docker compose version ``` #### 3. Clone Repository @@ -334,6 +286,11 @@ git clone https://github.com/smiley-maker/readcrumbs cd readcrumbs ``` +Or you can copy the files from your local computer to the EC2 container using: +```bash +scp -r -i path/to/your/key.pem folder/to/pass ubuntu@ipv4-address:~/ +``` + #### 4. Set Up Environment Variables ```bash @@ -345,88 +302,9 @@ Add your environment variables (see [Environment Variables](#environment-variabl **Note**: If using IAM roles, you may only need `AWS_REGION` and `DDB_TABLE`. -#### 5. Build and Run Containers - -```bash -# Build containers -docker compose build - -# Run in detached mode -docker compose up -d - -# Check status -docker compose ps - -# View logs -docker compose logs -f -``` - -#### 6. Verify Deployment - -Test each service: - -```bash -# Backend health check -curl http://localhost:8000/health - -# Or from your local machine -curl http://your-ec2-ip:8000/health -``` - -#### 7. Set Up as Systemd Service (Optional) - -For automatic startup on reboot: - -```bash -# Create systemd service file -sudo nano /etc/systemd/system/readcrumbs.service -``` - -Add the following: - -```ini -[Unit] -Description=ReadCrumbs MLOps Application -Requires=docker.service -After=docker.service - -[Service] -Type=oneshot -RemainAfterExit=yes -WorkingDirectory=/home/ubuntu/readcrumbs -ExecStart=/usr/bin/docker compose up -d -ExecStop=/usr/bin/docker compose down -User=ubuntu -Group=docker - -[Install] -WantedBy=multi-user.target -``` - -Enable and start the service: - -```bash -sudo systemctl daemon-reload -sudo systemctl enable readcrumbs -sudo systemctl start readcrumbs -sudo systemctl status readcrumbs -``` ### Troubleshooting Deployment -**Issue: Containers won't start** - -```bash -# Check logs -docker compose logs - -# Check if ports are already in use -sudo netstat -tulpn | grep -E '8000|8080|8081' - -# Restart Docker daemon -sudo systemctl restart docker -``` - **Issue: AWS credentials not working** ```bash @@ -470,7 +348,7 @@ Check if the API is running. } ``` -**cURL Example**: +**CURL Example**: ```bash curl http://localhost:8000/health ``` @@ -613,100 +491,6 @@ FastAPI provides automatic interactive documentation: - **Swagger UI**: http://localhost:8000/docs - **ReDoc**: http://localhost:8000/redoc -## Frontend Usage - -The frontend is a Streamlit application that provides a user-friendly interface. - -### Accessing the Frontend - -- **Local**: http://localhost:8080/ -- **Production**: http://your-ec2-ip:8080/ - -### How to Use - -1. **Enter Favorite Books**: - - In the text area, enter your favorite book titles - - Separate multiple books with commas - - Example: `The Great Gatsby, 1984, To Kill a Mockingbird` - -2. **Get Recommendations**: - - Click the "Analyze Sentiment" button (button text may vary) - - Wait for the API to process your request - - View your personalized book recommendations - -3. **View Results**: - - Recommendations appear as a numbered list - - Each recommendation is a book title - -### Input Format - -- Books should be entered as plain text titles -- Separate multiple books with commas -- Case-insensitive -- The system will match titles from the model's vocabulary - -### Expected Output - -The frontend displays: -- A list of 10 recommended book titles -- Based on your input favorites -- Ranked by relevance - -## Monitoring Dashboard - -The monitoring dashboard provides real-time insights into model performance and system health. - -### Accessing the Dashboard - -- **Local**: http://localhost:8081/ -- **Production**: http://your-ec2-ip:8081/ - -### Features - -#### 1. Prediction Latency Over Time - -- Visualizes the time taken to process predictions -- Helps identify performance degradation -- Shows trends over time - -#### 2. Prediction Distribution (Target Drift) - -- Displays the distribution of predicted book titles -- Helps detect data drift -- Shows which books are being recommended most frequently - -#### 3. User Feedback Collection - -- Allows users to provide feedback on predictions -- Tracks model accuracy based on user feedback -- Calculates live accuracy metrics - -#### 4. Live Model Accuracy - -- Displays accuracy percentage based on user feedback -- Updates in real-time as feedback is collected -- Helps monitor model performance - -### How to Use - -1. **View Metrics**: The dashboard automatically loads and displays metrics from DynamoDB - -2. **Provide Feedback**: - - Enter a User ID in the text input - - View the most recent prediction for that user - - Select whether the prediction was correct - - Click "Submit Feedback" - -3. **Monitor Performance**: - - Check the "Live Model Accuracy" metric - - Review latency trends - - Monitor prediction distributions - -### Interpreting Metrics - -- **High Latency**: May indicate model or infrastructure issues -- **Skewed Distribution**: Could indicate data drift or model bias -- **Low Accuracy**: May require model retraining or data quality improvements ## Project Structure @@ -723,24 +507,19 @@ readcrumbs/ │ ├── Dockerfile # Frontend container configuration │ └── requirements.txt # Python dependencies ├── monitoring/ -│ ├── app.py # Streamlit monitoring dashboard +│ ├── dashboard_app.py # Streamlit monitoring dashboard │ ├── Dockerfile # Monitoring container configuration │ └── requirements.txt # Python dependencies -├── experiment-tracking/ -│ └── wandb.py # W&B experiment tracking and model registry ├── experiments/ │ ├── training/ │ │ ├── preprocess.py # Data preprocessing utilities │ │ ├── train_model.py # Model training script -│ │ └── utils.py # Training utilities │ └── notebooks/ │ └── eda.ipynb # Exploratory data analysis -├── tests/ -│ └── test_preprocess.py # Preprocessing tests +| |-- tracking/ +│ └── wandb_tracking.py # W&B experiment tracking and model registry ├── data/ │ └── README.md # Data documentation -├── docker-compose.yml # Multi-container orchestration -├── requirements.txt # Root-level dependencies └── README.md # This file ``` @@ -748,9 +527,8 @@ readcrumbs/ - **`backend/api.py`**: Main FastAPI application with prediction endpoints - **`frontend/readcrumbs_app.py`**: User-facing Streamlit interface -- **`monitoring/app.py`**: Monitoring and analytics dashboard -- **`experiment-tracking/wandb.py`**: W&B integration for model tracking -- **`docker-compose.yml`**: Container orchestration configuration +- **`monitoring/dashboard_app.py`**: Monitoring and analytics dashboard +- **`experiments/tracking/wandb_tracking.py`**: W&B integration for model tracking ## Testing @@ -766,15 +544,10 @@ pytest tests/test_api.py -v #### Preprocessing Tests ```bash +cd experiments/training pytest tests/test_preprocess.py -v ``` -#### Run All Tests - -```bash -pytest tests/ -v -``` - ### Test Coverage The test suite includes: @@ -796,146 +569,6 @@ backend/tests/test_api.py::test_predict PASSED ======================== 3 passed in 2.34s ======================== ``` -## Troubleshooting - -### Common Issues - -#### 1. Docker Containers Won't Start - -**Symptoms**: `docker compose up` fails or containers exit immediately - -**Solutions**: -```bash -# Check logs -docker compose logs - -# Rebuild containers -docker compose build --no-cache - -# Check if ports are in use -sudo lsof -i :8000 -sudo lsof -i :8080 -sudo lsof -i :8081 -``` - -#### 2. AWS Credentials Not Working - -**Symptoms**: S3 or DynamoDB access errors - -**Solutions**: -```bash -# Verify credentials in .env file -cat .env | grep AWS - -# Test AWS CLI access -aws s3 ls s3://readcrumbs/ -aws dynamodb list-tables - -# Check IAM permissions -aws iam get-user -``` - -#### 3. Model Not Loading - -**Symptoms**: Backend starts but predictions fail - -**Solutions**: -- Verify model file exists in S3: `aws s3 ls s3://readcrumbs/models/` -- Check model file path in `backend/api.py` -- Verify S3 bucket permissions - -#### 4. DynamoDB Connection Issues - -**Symptoms**: Logs not saving or monitoring dashboard empty - -**Solutions**: -```bash -# Verify table exists -aws dynamodb describe-table --table-name readcrumbs-logs - -# Check table permissions -aws iam get-role-policy --role-name YourRoleName --policy-name YourPolicyName - -# Verify table name in environment variables -echo $DDB_TABLE -``` - -#### 5. Frontend Not Connecting to Backend - -**Symptoms**: Frontend shows errors when requesting predictions - -**Solutions**: -- Check API URL in `frontend/readcrumbs_app.py` -- Verify backend is running: `curl http://localhost:8000/health` -- Check CORS settings if needed -- Verify network connectivity between containers - -#### 6. Monitoring Dashboard Shows No Data - -**Symptoms**: Dashboard loads but shows empty charts - -**Solutions**: -- Verify DynamoDB table has data: `aws dynamodb scan --table-name readcrumbs-logs --limit 5` -- Check table name matches in `monitoring/app.py` -- Verify AWS credentials for monitoring container -- Make some predictions first to generate data - -### Debugging Tips - -1. **View Container Logs**: - ```bash - docker compose logs backend - docker compose logs frontend - docker compose logs monitoring - ``` - -2. **Access Container Shell**: - ```bash - docker compose exec backend bash - docker compose exec frontend bash - ``` - -3. **Check Environment Variables**: - ```bash - docker compose exec backend env | grep AWS - ``` - -4. **Test API Manually**: - ```bash - curl -X POST http://localhost:8000/predict \ - -H "Content-Type: application/json" \ - -d '{"items": ["test"], "userid": "123"}' - ``` - -## Contributing - -### Development Workflow - -1. Fork the repository -2. Create a feature branch: `git checkout -b feature/your-feature-name` -3. Make your changes -4. Add tests for new functionality -5. Ensure all tests pass: `pytest` -6. Commit your changes: `git commit -m "Add your feature"` -7. Push to the branch: `git push origin feature/your-feature-name` -8. Open a Pull Request - -### Code Style - -- Follow PEP 8 for Python code -- Use type hints where appropriate -- Add docstrings to functions and classes -- Keep functions focused and small - -### Reporting Issues - -If you encounter a bug or have a feature request, please open an issue on GitHub with: -- Description of the problem -- Steps to reproduce -- Expected behavior -- Actual behavior -- Environment details (OS, Python version, Docker version) - ## License This project is open source and available for educational and research purposes. @@ -944,14 +577,4 @@ This project is open source and available for educational and research purposes. Developed by **Jordan Sinclair** and **Jordan Larson** -- GitHub: [smiley-maker/readcrumbs](https://github.com/smiley-maker/readcrumbs) - -## Acknowledgments - -- Built with FastAPI, Streamlit, and Docker -- Model tracking powered by Weights & Biases -- Deployed on AWS infrastructure - ---- - -For questions or support, please open an issue on the GitHub repository. +- GitHub: [smiley-maker/readcrumbs](https://github.com/smiley-maker/readcrumbs) \ No newline at end of file diff --git a/backend.ppk b/backend.ppk deleted file mode 100644 index 9bbbd7e..0000000 --- a/backend.ppk +++ /dev/null @@ -1,26 +0,0 @@ -PuTTY-User-Key-File-2: ssh-rsa -Encryption: none -Comment: backend -Public-Lines: 6 -AAAAB3NzaC1yc2EAAAADAQABAAABAQDYFtyIaC0Mr2z/oeVLARxPvLBC2g7C4Juf -XhN6KHTq3U9niZrKY4pC9uYX3L7FgPGHd0QICztSRFQ3wEaQ51/87RpKogS+S4lN -YYYqEJLUoScR/uq3ooi4YI+YRgv61RY3iTd7fSoht3xhpTbwbXodq/pRi8jv8/Kz -BFPbNScU8AepaIPAtqlyzlhx2+5r/xW//TMkKHUvWsB1ZHuKkvYlNdZpEFrsOVL7 -7bbnFZhn7A9FYgz5XViZYH5v6LOvhxJD6Qq1Bx9d+A1DwmMQxb+0AV0MK4eWthaM -FQ7c824si6QgnquzvT9uev5e3lWpviKWKGpoB+M7jsUj9VRcIcMz -Private-Lines: 14 -AAABAQClWhEVHEpkq5RHpLXViBsG9QcXkM681qye7ZkP4AdfdUv+mXBmMHcrOMzn -M5aTgVDQ7TWUxit1jy22n54f4b00yKZWt9XTW+/L07WbWKVSqaJBBgTL0ka2d8VJ -q0gdf5MJmilniGaF+GboPUll/w/zxpa8ca+n1c9Apy4XznuCa6Yh7Y8kYbgT5wke -wVXaVbWL89K0SUCJHZDtimLjWeRNQwvdMkjt+GnPET8PiQACvMLS2B2U6sRLWlY+ -AeKH6QwwvWzwHf/qjAb+BRojYyindI/BljxFVuGz3WETneN8jKdR8oV3zr/cnjYj -00uXtV+U94NRNDUofIo3z/7tHeDhAAAAgQDuzss0AuSsmXs19K/zbZ1cN93MuZXa -tEFIrhrADyIag87ftRLZTjp8OOtUedP4K6b6sE/zxEdZbtauo/jiR3/rdGJQ8Qer -tgEFm5Lj2WFG0KqIVk3jjVWyb7boYcnLC7YywwopAgNDbyvfbqWtWPPB1gnu4+c+ -YMZ0hmdkxsUfIwAAAIEA56VeN0jlU8+YLUbEi0wpF8NCffwLRXVwELZWqRyEuPFS -K9Zr2cvEsk5Wz9lZlHR1m9c9QI4ONKyHTV4XYJoP5pQEQQeOEGajp+AnaP9ylOZl -Vm65urQ89BJ9VfSlhQnUtAvbvudOJG4x5F72XVTS0I3FIxUMvRF0z/KNnEm9lLEA -AACABuvc8kP4dvk+an8iydDopJ/2v4Et3f9qKNYnsuPgU+qwH3x2fburMkXv49oj -Lrn5NKRui/qWCell6rTWU3i+91c8vhtfaAVdgutQjSMyVXgfzFXRkvaPvOKnrt9k -a3kneUkD2dyx6RhLEpPQI6Rb/8il8hBEwiN+LTGqqVlg0RA= -Private-MAC: 25566de206acf221830b84f25c13174a6998d597 diff --git a/backend/Dockerfile b/backend/Dockerfile index e69de29..2d8b0f8 100644 --- a/backend/Dockerfile +++ b/backend/Dockerfile @@ -0,0 +1,19 @@ +FROM python:3.12 + +# Set the working directory in the container +WORKDIR /app + +# Copy the requirements file into the container at /app +COPY requirements.txt /app/ + +# Install any needed packages specified in requirements.txt +RUN pip install --no-cache-dir -r requirements.txt + +# Copy the main application code into the container at /app +COPY api.py /app/ + +# Make port 8000 available to the world outside this container +EXPOSE 8000 + +# Run the API server when the container launches +CMD ["uvicorn", "api:app", "--host", "0.0.0.0", "--port", "8000"] \ No newline at end of file diff --git a/backend/api.py b/backend/api.py index 36eb72a..4f489fa 100644 --- a/backend/api.py +++ b/backend/api.py @@ -3,46 +3,79 @@ import os import datetime import random -import pickle import joblib import json import numpy as np from typing import List from pydantic import BaseModel import io +import pandas as pd +from decimal import Decimal + class MyFavorites(BaseModel): - items: List[str] - userid: str + items: List[str] # List of favorite book titles + userid: str # Unique user identifier class PredictionResponse(BaseModel): -# user_id: int -# req: MyFavorites recs: List[str] # titles of the recommended books -''' -To-Do: -- [ ] Connect to S3 w/ model -- [ ] Create a function to load the model from S3 -- [ ] Create a function to predict using the model -''' +class FeedbackItem(BaseModel): + userid: str + reccommendations: List[str] + feedback: int # 1 = like + title: str + position: int + timestamp: datetime.datetime + + ## Helper Functions -def load_supporting_tables_from_s3(table_name: str): - """ - Download and load supporting tables from S3 into memory without persisting it to disk. +def load_supporting_tables_from_s3(table_name: str, client = None) -> dict: + """Download and load supporting tables from S3 into memory without persisting it to disk. + + Args: + table_name (str): Name of the table in S3. + + Returns: + dict: Dictionary representing the table. """ - s3 = boto3.client("s3") + if client is None: + client = boto3.client("s3") + s3_bucket = 'readcrumbs' - # Download model object as bytes into memory - response = s3.get_object(Bucket=s3_bucket, Key=table_name) + # Download the object as bytes into memory + response = client.get_object(Bucket=s3_bucket, Key=table_name) table_bytes = response['Body'].read().decode('utf-8') table = json.loads(table_bytes) return table -def load_model_from_s3(model_name: str): +def load_dataset_from_s3(bucket_name: str, file_key: str, client = None) -> pd.DataFrame: + """Download and load a dataset file from S3 into memory without persisting it to disk. + + Args: + bucket_name (str): The S3 bucket name. + file_key (str): The key/path of the dataset file in the S3 bucket. + + Returns: + pd.DataFrame: The loaded dataset as a pandas DataFrame. + """ + if client is None: + client = boto3.client('s3') + + # Get the object from S3 + response = client.get_object(Bucket=bucket_name, Key=file_key) + + # Load the dataset from the bytes in memory + buffer = io.BytesIO(response['Body'].read()) + df = pd.read_parquet(buffer) + + return df + + +def load_model_from_s3(model_name: str, client = None): """ Download and load an ML model file from S3 into memory without persisting it to disk. @@ -65,25 +98,47 @@ def load_model_from_s3(model_name: str): """ s3_bucket = 'readcrumbs' - # Download model object as bytes into memory - s3_client = boto3.client('s3') - response = s3_client.get_object(Bucket=s3_bucket, Key=model_name) -# model_data = response['Body'].read() + # Check if client is none, then create one + if client is None: + client = boto3.client('s3') + + # Get the model object from S3 + response = client.get_object(Bucket=s3_bucket, Key=model_name) + + # Load the model from the bytes in memory buffer = io.BytesIO(response['Body'].read()) model = joblib.load(buffer) -# model_file = io.BytesIO(model_data) -# model = pickle.load(io.BytesIO(model_data)) -# model = joblib.load(loaded_model) -# model = pickle.loads(loaded_model) + return model -def predict_using_model(data: MyFavorites, n_recs: int = 10): - my_favs_ids = [title_to_index[f] for f in data.items] - fav_vectors = [model.item_factors[i] for i in my_favs_ids] - #Average the vectors +def predict_using_model(data: MyFavorites, n_recs: int = 10) -> tuple[list[str], float, float]: + """Gets n_recs recommendations based on the users favorite books. + + Args: + data (MyFavorites): The users favorite book titles. + n_recs (int, optional): Number of recommendations to return. Defaults to 10. + + Returns: + list[str]: List of recommended book titles. + """ + + # Get the book ids for the user's favorite books + my_favs_ids = [title_to_index[f] for f in data] + # Get the item vectors for the user's favorite books using the model + fav_vectors = [model.item_factors[int(i)] for i in my_favs_ids] + #Average the vectors together to get a single user vector avg_vec = np.average(np.stack(fav_vectors), axis=0) - recommendations = np.argsort(np.dot(avg_vec, model.item_factors.T))[:n_recs] - return [index_to_title[i] for i in recommendations] + # Calculate similarities using a dot product between the user vector and all item vectors + sims = np.dot(avg_vec, model.item_factors.T) + # Get the top n_recs recommendations by finding the closest item vectors to the user vector + recommendations = np.argsort(sims)[:n_recs] + # Get an estimate of confidence for this request + confidence = np.average([sims[r] for r in recommendations]) + # Get max similarity score + max_sim = sims[recommendations[0]] + # Convert the item ids back to titles using the index_to_title mapping + # Return the list of recommended titles, confidence, and max similarity + return [index_to_title[str(i)] for i in recommendations], confidence, max_sim def serialize_for_dynamodb(data): """ @@ -99,17 +154,15 @@ def serialize_for_dynamodb(data): else: return data -def get_dynamodb_table(): + + +def get_dynamodb_table(table_name: str = "readcrumbs-logs"): """ Get a DynamoDB table resource with proper credentials. Returns: boto3 DynamoDB Table resource """ - table_name = os.environ.get("DDB_TABLE") - if not table_name: -# raise ValueError("DDB_TABLE environment variable not set.") - table_name = "readcrumbs-logs" region = os.environ.get("AWS_REGION", "us-east-1") @@ -140,11 +193,9 @@ def get_random_item_from_ddb(): Returns: dict: A random item from the table, or None if table is empty """ - table = get_dynamodb_table() + table = get_dynamodb_table("readcrumbs-logs") # Scan the table to get all items - # Note: For very large tables, this could be expensive. - # Consider optimizing with pagination or sampling if needed. response = table.scan() items = response.get('Items', []) @@ -159,48 +210,45 @@ def get_random_item_from_ddb(): # Return a random item return random.choice(items) -def save_to_ddb(data): + +def save_to_ddb(data, table_name: str = None): """ Save or update a dictionary of data to DynamoDB. - Uses user_id (integer) as the primary key. If user_id already exists, the item will be updated. - - Uses AWS credentials from environment variables if available: - - AWS_ACCESS_KEY_ID - - AWS_SECRET_ACCESS_KEY - - AWS_SESSION_TOKEN (optional, for temporary credentials) - - Falls back to IAM roles (if running on EC2, Lambda, ECS, etc.) or ~/.aws/credentials - - Required environment variables: - - DDB_TABLE: DynamoDB table name - - AWS_REGION: AWS region (optional, defaults to us-east-1) + Uses userid (integer) as the primary key. If userid already exists, the item will be updated. Args: - data: Dictionary containing user_id (int) and other fields. user_id is used as primary key. + data: Dictionary containing userid (int) and other fields. userid is used as primary key. + table_name (str, optional): Name of the DynamoDB table. + If None, defaults to "readcrumbs-logs". + + Returns: + dict: Response from DynamoDB put_item operation. """ - table = get_dynamodb_table() + + # Get the DynamoDB table with the table name + table = get_dynamodb_table(table_name) - # Serialize data for DynamoDB (convert datetime objects, etc.) + # Serialize data for DynamoDB (convert datetime objects, lists, etc.) serialized_data = serialize_for_dynamodb(data) - # Ensure user_id exists (required as primary key) - if 'user_id' not in serialized_data: - raise ValueError("user_id is required in the request body") + # Ensure userid exists (required as primary key) + if 'userid' not in serialized_data: + raise ValueError("userid is required in the request body") - # Map user_id to pred-id (the table's primary key field name) - # Keep user_id in the data as well for reference -# serialized_data['pred-id'] = serialized_data['user_id'] - response = table.put_item(Item=serialized_data) return response -## API +## FastAPI app app = FastAPI() -model = load_model_from_s3("models/als_model-small-v1.pkl") -index_to_title = load_supporting_tables_from_s3("data/v1/index_to_title.json") -title_to_index = load_supporting_tables_from_s3("data/v1/title_to_index.json") +# Set up a client so we don't have to keep recreating it +s3_client = boto3.client("s3") + +# Load the model and supporting tables into memory +model = load_model_from_s3("models/als_model-small-v1.pkl", client=s3_client) +index_to_title = load_supporting_tables_from_s3("data/v1/index_to_title.json", client=s3_client) +title_to_index = load_supporting_tables_from_s3("data/v1/title_to_index.json", client=s3_client) @app.get("/health") def health_check(): @@ -210,39 +258,87 @@ def health_check(): def get_random(): """ Get a random item from the DynamoDB table. + + Raises: + HTTPException404: 404 error if no items found in table. + HTTPException500: 500 error for any issues during retrieval. Returns: dict: A random item from the table """ - random_item = get_random_item_from_ddb() - if random_item is None: - raise HTTPException(status_code=404, detail="No items found in table") - return random_item + try: + random_item = get_random_item_from_ddb() + if random_item is None: + raise HTTPException(status_code=404, detail="No items found in table") + return random_item + except Exception as e: + raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail=f"Error retrieving random item: {str(e)}") + + +@app.post("/feedback") +def submit_feedback(feedback: FeedbackItem): + """Submit user feedback (like) to DynamoDB table. + + Args: + feedback (FeedbackItem): Feedback data to store in DynamoDB. + + Raises: + HTTPException500: 500 error for any issues during saving feedback. + + Returns: + dict: Response from DynamoDB put_item operation. + """ + try: + response = save_to_ddb(feedback, table_name="readcrumbs-feedback") + return response + except Exception as e: + raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail=f"Error saving feedback: {str(e)}") + @app.post("/predict") def predict(request: MyFavorites) -> PredictionResponse: - # Create a dictionary from the request and other metrics. - if len(request.items) < 1 or type(request.items) != List: - raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Must enter at least one favorite book.") + """Get book recommendations from the users favorite titles. + + Args: + request (MyFavorites): Includes items, which is a list of favorite book titles, + and userid, a unique user identifier. + + Raises: + HTTPException400: 400 error if no favorite books are provided. + HTTPException500: 500 error for any other issues during prediction. + + Returns: + PredictionResponse: The predicted book recommendations. + """ + if len(request.items) < 1: + raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=f"Must enter at least one favorite book. You gave: {request.items}") try: - recs = predict_using_model(request.items) + # Get recommendations using the model + # Calculate processing time + timestart = datetime.datetime.now() + recs, confidence, max_sim = predict_using_model(request.items) + timeend = datetime.datetime.now() + latency = (timeend - timestart).total_seconds() * 1000 # in milliseconds + + # Log the request and prediction to DynamoDB logs = { "items": request.items, - "user_id": request.userid, + "userid": request.userid, "timestamp": datetime.datetime.now(datetime.timezone.utc), - "prediction": recs + "prediction": recs, + "confidence": Decimal(str(confidence)), + "max_similarity": Decimal(str(max_sim)), + "latency": Decimal(str(latency)), } - save_to_ddb(logs) + # Save the logs to DynamoDB + save_to_ddb(logs, table_name="readcrumbs-logs") + # Return the recommendations as a PredictionResponse return { "recs": recs } + except Exception as e: - raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail=str(e)) -# data = request.model_dump() -# data['timestamp'] = datetime.datetime.now(datetime.timezone.utc) -# data['prediction'] = predict_using_model(model, data) -# save_to_ddb(data) -# return {"status": "ok"} \ No newline at end of file + raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail=f"{str(e)}\Request: {request}") \ No newline at end of file diff --git a/backend/app/core/config.py b/backend/app/core/config.py deleted file mode 100644 index e69de29..0000000 diff --git a/backend/app/core/database.py b/backend/app/core/database.py deleted file mode 100644 index e69de29..0000000 diff --git a/backend/app/main.py b/backend/app/main.py deleted file mode 100644 index e69de29..0000000 diff --git a/backend/app/services/model_service.py b/backend/app/services/model_service.py deleted file mode 100644 index e69de29..0000000 diff --git a/backend/requirements.txt b/backend/requirements.txt index e69de29..f550d85 100644 --- a/backend/requirements.txt +++ b/backend/requirements.txt @@ -0,0 +1,8 @@ +fastapi>=0.109.0 +boto3>=1.34.0 +joblib>=1.3.2 +numpy>=1.26.4 +pydantic>=2.6.0 +uvicorn[standard]>=0.27.0 +implicit +pandas>=2.2.4 \ No newline at end of file diff --git a/data/README.md b/data/README.md index 0f051c0..f9361da 100644 --- a/data/README.md +++ b/data/README.md @@ -6,13 +6,9 @@ No large raw or processed data files are committed to Git. These files are eithe ## Data Source Details -The foundation of our recommendation model is the Amazon Review Data. +We originally planned to use the complete Amazon Book Reviews dataset, but at about 20 GB it was intractable for smaller AWS computes to preprocess and run training with this dataset. Some experimentation with portions of this dataset can be found the exploratory analysis notebook. The dataset came from Julian McAuley's Amazon Review Dataset, and included ratings, books, and users files in a JSONL format. The link is: https://amazon-reviews-2023.github.io/. -| Attribute | Details | -| :------- | :------: | -| Dataset Name | Amazon Review Data — Books Subset | -| Original Source | Julian McAuley's Amazon Review Dataset | -| Dataset Components | Ratings, Books, and Users files (specific format depends on chosen subset) | -| Size | 10.3 million users, 4.4 million items, and 29.5 million ratings | -| License | Open access for non-commercial research purposes. | -| Link | https://amazon-reviews-2023.github.io/ | \ No newline at end of file + +## Smaller Dataset + +Due to the size of the dataset above, we decided to use a smaller dataset from Kaggle for testing on smaller computes in AWS. This dataset included book details and reviews CSV files, but only the reviews data was used to train a collaborative filtering model, although it would be worth exploring a hybrid using both datasets and content based filtering as well. The link is: https://www.kaggle.com/datasets/mohamedbakhet/amazon-books-reviews. \ No newline at end of file diff --git a/docker-compose.yml b/docker-compose.yml deleted file mode 100644 index e69de29..0000000 diff --git a/backend/app/api/__init__.py b/experiments/__init__.py similarity index 100% rename from backend/app/api/__init__.py rename to experiments/__init__.py diff --git a/experiments/tracking/wandb.py b/experiments/tracking/wandb.py deleted file mode 100644 index e69de29..0000000 diff --git a/experiment-tracking/wandb.py b/experiments/tracking/wandb_tracking.py similarity index 64% rename from experiment-tracking/wandb.py rename to experiments/tracking/wandb_tracking.py index 9c56d00..7283e71 100644 --- a/experiment-tracking/wandb.py +++ b/experiments/tracking/wandb_tracking.py @@ -121,6 +121,9 @@ def promote_model_to_stage(registered_model_name, alias="staging", metric_name=" """ Promote a model version to a specific stage (staging/production) in the Model Registry. + Note: This function works by finding artifacts linked to the registered model and updating + their aliases. The model must have been previously linked using run.link_artifact(). + Args: registered_model_name: Name of the registered model in Model Registry alias: Stage alias to assign ('staging' or 'production') @@ -154,55 +157,139 @@ def promote_model_to_stage(registered_model_name, alias="staging", metric_name=" if project_name is None: project_name = wandb.run.project if wandb.run else "readcrumbs" - # Access registered model - registered_model_path = f"{project_name}/{registered_model_name}" - registered_model = api.registered_model(registered_model_path) + entity = None + if wandb.run: + entity = wandb.run.entity if hasattr(wandb.run, 'entity') else None + + # Since api.registered_model() doesn't exist, we access artifacts directly + # When an artifact is linked to a registered model via run.link_artifact(), + # it becomes accessible through the registered model name path + + project_path = f"{entity}/{project_name}" if entity else project_name + + # Try to access artifacts linked to the registered model + # The pattern is typically: entity/project/registered_model_name:alias + artifacts = [] - if metric_value is not None: + # First, check if we're in the current run and can access the artifact directly + if wandb.run: + try: + # Try to get the artifact from the current run + current_run = api.run(f"{project_path}/{wandb.run.id}") + for artifact_name in current_run.used_artifacts(): + artifact_str = str(artifact_name) + if registered_model_name in artifact_str.lower(): + try: + artifact = api.artifact(artifact_str) + if artifact.type == "model": + artifacts.append(artifact) + except Exception: + continue + except Exception: + pass + + # Try accessing via the registered model name directly + # This works when artifacts are linked to the model registry + if not artifacts: + try: + # Try with :latest alias first + artifact = api.artifact(f"{project_path}/{registered_model_name}:latest") + if artifact.type == "model": + artifacts.append(artifact) + except Exception: + pass + + # If that didn't work, search through recent runs for linked artifacts + if not artifacts: + print(f"Searching for artifacts linked to registered model '{registered_model_name}'...") + runs = api.runs(project_path, per_page=20) # Check recent runs + + for run in runs: + try: + # Check if this run has artifacts linked to our registered model + # We'll look for artifacts that might be model artifacts + for artifact_collection in run.used_artifacts(): + artifact_str = str(artifact_collection) + # Try to get the artifact and check if it's a model + try: + artifact = api.artifact(artifact_str) + # Check if this artifact is linked to our registered model + # by checking if the registered model name appears in aliases or path + if (artifact.type == "model" and + (registered_model_name in artifact_str.lower() or + any(registered_model_name.lower() in str(alias).lower() + for alias in (artifact.aliases or [])))): + artifacts.append(artifact) + except Exception: + continue + except Exception: + continue + + if not artifacts: + print(f"Note: No model artifacts found for registered model '{registered_model_name}'. " + f"Make sure the model has been registered using run.link_artifact().") + print(f"Tried accessing: {project_path}/{registered_model_name}:latest") + return False + + # Sort artifacts by creation time (newest first) + try: + artifacts.sort(key=lambda a: a.created_at if hasattr(a, 'created_at') else 0, reverse=True) + except Exception: + pass + + if metric_value is None: + # Promote the latest version + latest_artifact = artifacts[0] + current_aliases = list(latest_artifact.aliases) if latest_artifact.aliases else [] + if alias not in current_aliases: + current_aliases.append(alias) + latest_artifact.aliases = current_aliases + latest_artifact.save() + print(f"Promoted latest model version to '{alias}' stage") + return True + else: + print(f"Model already has '{alias}' alias") + return True + else: # Find the best model based on metric - best_version = None + best_artifact = None best_metric = float('-inf') if comparison == "max" else float('inf') - for version in registered_model.versions: - # Get metadata from the artifact + for artifact in artifacts: try: - artifact = version.artifact version_metadata = artifact.metadata or {} version_metric = version_metadata.get(metric_name) if version_metric is not None: if comparison == "max" and version_metric > best_metric: best_metric = version_metric - best_version = version + best_artifact = artifact elif comparison == "min" and version_metric < best_metric: best_metric = version_metric - best_version = version + best_artifact = artifact except Exception: continue - if best_version: - # Update aliases - current_aliases = list(best_version.aliases) if best_version.aliases else [] + if best_artifact: + current_aliases = list(best_artifact.aliases) if best_artifact.aliases else [] if alias not in current_aliases: current_aliases.append(alias) - best_version.aliases = current_aliases - best_version.update() - print(f"Promoted model version {best_version.version} to '{alias}' stage " - f"(metric: {metric_name}={best_metric})") - return True - else: - # Promote the latest version - if registered_model.versions: - latest_version = registered_model.versions[0] - current_aliases = list(latest_version.aliases) if latest_version.aliases else [] - if alias not in current_aliases: - current_aliases.append(alias) - latest_version.aliases = current_aliases - latest_version.update() - print(f"Promoted latest model version {latest_version.version} to '{alias}' stage") + best_artifact.aliases = current_aliases + best_artifact.save() + print(f"Promoted model to '{alias}' stage (metric: {metric_name}={best_metric})") return True + else: + print(f"Could not find a model with metric '{metric_name}' in metadata") + return False return False + + except AttributeError as e: + print(f"Error: API method not available in this wandb version: {e}") + print(f"Consider updating wandb: pip install --upgrade wandb") + print(f"Note: Model linking via run.link_artifact() should still work. " + f"Promotion to stages may need to be done manually in the wandb UI.") + return False except Exception as e: print(f"Error promoting model: {e}") print(f"Note: Make sure the registered model '{registered_model_name}' exists in the Model Registry.") @@ -316,42 +403,4 @@ def _get_model_extension(model_type): } wandb.log(final_metrics) -# Save the trained model as an artifact and register it in Model Registry -# Example usage (uncomment and modify based on your model): -# model = your_trained_model # Replace with your actual model -# -# # Prepare metadata with performance metrics -# model_metadata = { -# "final_accuracy": final_metrics["final_accuracy"], -# "final_f1_score": final_metrics["final_f1_score"], -# "epochs": hyperparameters["epochs"], -# "learning_rate": hyperparameters["learning_rate"], -# "batch_size": hyperparameters["batch_size"], -# "code_version": wandb.config.get("code_version", "unknown"), -# "data_version": wandb.config.get("data_version", "unknown"), -# } -# -# # Save and register model with automatic promotion to staging if it's the best -# artifact, promoted = save_and_register_model( -# model=model, -# model_name="readcrumbs-model", -# model_type="pytorch", # or "tensorflow", "sklearn", "pickle" -# registered_model_name="readcrumbs-model", # Name in Model Registry -# metadata=model_metadata, -# auto_promote=True, # Automatically promote to staging if best model -# promotion_stage="staging", # or "production" -# promotion_metric="f1_score" # Metric to use for comparison -# ) -# -# if promoted: -# print(f"Model automatically promoted to staging based on {promotion_metric}") -# -# # Alternatively, manually promote to production after review: -# # promote_model_to_stage( -# # registered_model_name="readcrumbs-model", -# # alias="production", -# # metric_name="f1_score", -# # comparison="max" -# # ) - -wandb.finish() +wandb.finish() \ No newline at end of file diff --git a/.env.example b/experiments/training/__init__.py similarity index 100% rename from .env.example rename to experiments/training/__init__.py diff --git a/experiments/training/preprocess.py b/experiments/training/preprocess.py index e69de29..ef1eef0 100644 --- a/experiments/training/preprocess.py +++ b/experiments/training/preprocess.py @@ -0,0 +1,207 @@ +import pandas as pd +import json +import boto3 +from typing import Union +from io import BytesIO + + +def load_csv_data(bucket : str, objectkey: str, client) -> pd.DataFrame: + """Loads CSV dataset from S3 + + Args: + bucket (str): The bucket name where the data is stored. + objectkey (str): The path to the file in the bucket. + client: boto3 client to handle data reading. + + Returns: + pd.DataFrame: The dataset read in from S3. + """ + + try: + # Get the object from S3 + print("Getting data from S3") + response = client.get_object(Bucket=bucket, Key=objectkey) + + # The 'Body' is a streamable object (a botocore.response.StreamingBody). + # pandas.read_csv can handle this stream directly, which avoids loading + # the entire file into a Python string variable first. + print("Reading CSV directly from S3 stream...") + df = pd.read_csv(response['Body']) + + print(f"Dataframe with {len(df)} rows created.") + print(df.head()) + + return df + except Exception as e: + print(f"Error reading CSV from S3 storage: {e}") + + + +def save_data(data : Union[pd.DataFrame, dict], bucket : str, objectkey : str, client) -> bool: + """Saves either dataframe or dictionary data to S3. + + Args: + data (DataFrame or dict): The data to save. + bucket (str): The bucket name where the data is stored. + objectkey (str): The path to the file in the bucket. + client: boto3 client to handle data reading. + + Returns: + bool: True if successful upload, false otherwise. + """ + + try: + # Check if data is a DataFrame or dictionary + if type(data) == pd.DataFrame: + # We want to save as a parquet file. + # Need to create a buffer for pandas conversion process + buffer = BytesIO() + + # Write the parquet file to the buffer + data.to_parquet(buffer) + + # Set the dataset equal to the value of the buffer. + dataset = buffer.getvalue() + + elif type(data) == dict: + # We want to save the data as JSON + dataset = json.dumps(data) + + else: + # Unsupported type + print(f"Unsupported type: {type(data)}") + return False + + client.put_object( + Bucket=bucket, + Key=objectkey, + Body=dataset + ) + + print(f"Successfully saved data to s3://{bucket}/{objectkey}") + return True + except Exception as e: + print(f"Error saving {type(data)} data to S3 url: s3://{bucket}/{objectkey}.") + print(e) + return False + + +def preprocess_data(df : pd.DataFrame, feats : list[str] = None) -> pd.DataFrame: + """Preprocesses the reviews dataset by selecting relevant + columns and dropping null values. + + Args: + df (pd.DataFrame): original dataframe + + Returns: + pd.DataFrame: processed dataset. + """ + + # Select features relevant for collaborative filtering, or + # use provided ones. + if not feats: + feats = ["User_id", "Title", "review/score"] + df = df[feats] + + # Drop any rows containing null values because all the features are necessary + df = df.dropna(axis=0) + + # We only want to keep 4 and 5 star reviews because the implicit library works on + # the basis of interactions, and while the ratings will sway the model more towards + # higher numbers, lower values aren't considered 'negative' and the model could + # still recommend those books. + df = df[df["review/score"] >= 4] + + # Change the user id and title to categorical columns + df["User_id"] = df["User_id"].astype("category") + df["Title"] = df["Title"].astype("category") + + return df + + +def create_mappings(df : pd.DataFrame, title_col : str = "title_book") -> tuple[dict, dict]: + """Creates mapping dictionaries for book titles to indices and vice versa. + + Args: + df (pd.DataFrame): Dataframe containing book titles. + title_col (str, optional): The column name containing book titles. Defaults to "title_book". + Returns: + tuple[dict, dict]: A tuple containing two dictionaries: + - title_to_index: Maps book titles to their corresponding indices. + - index_to_title: Maps indices back to their corresponding book titles. + """ + + # Convert the book title into a categorical column (each name = new book) + titles = df[title_col].astype("category") + + # Create a title to code mapping (for looking up factors) + title_to_index = dict(zip(titles.cat.categories, titles.cat.codes)) + + # Create the reverse mapping for code to title, used in displaying results + index_to_title = dict(enumerate(titles.cat.categories)) + + # Return dictionaries + return title_to_index, index_to_title + + +if __name__=="__main__": + print("Starting dataset processing...") + # Create an S3 client + print("Creating client") + client = boto3.client("s3") + + # Load dataset from S3 + print("Loading dataset") + df = load_csv_data( + bucket="readcrumbs", + objectkey="dataset/raw/book_ratings.csv", + client=client + ) + + # Preprocess dataset + print("Preprocessing data") + df = preprocess_data( + df=df + ) + + # Save processed dataset back to S3 + print("Saving Dataset") + res = save_data( + data=df, + bucket="readcrumbs", + objectkey="data/processed/ratings-small-v1.parquet", + client=client + ) + + assert res + + if not res: + print(f"Error saving processed dataset.") + + # Create and save mapping tables + print("Creating Mapping Tables") + tindex, indext = create_mappings(df, title_col="Title") + print("Saving mapping tables") + res = save_data( + data=tindex, + bucket="readcrumbs", + objectkey="data/v1/title_to_index.json", + client=client + ) + + assert res + + if not res: + print(f"Error saving the title to index dictionary.") + + res = save_data( + data=indext, + bucket="readcrumbs", + objectkey="data/v1/index_to_title.json", + client=client + ) + + assert res + + if not res: + print(f"Error saving the index to title dictionary.") \ No newline at end of file diff --git a/backend/app/api/endpoints.py b/experiments/training/tests/__init__.py similarity index 100% rename from backend/app/api/endpoints.py rename to experiments/training/tests/__init__.py diff --git a/experiments/training/tests/test_preprocess.py b/experiments/training/tests/test_preprocess.py new file mode 100644 index 0000000..b5f2fde --- /dev/null +++ b/experiments/training/tests/test_preprocess.py @@ -0,0 +1,31 @@ +import pandas as pd +from experiments.training.preprocess import preprocess_data + +# Unit test +## testing if we get the right feature columns after preprocessing +## Test if there are any null values in the data remaining +def test_preprocess_data(): + # Create a sample dataframe + df = pd.read_csv("./data/raw/Books_rating.csv") + + # Get just a small sample for testing + df = df.sample(n=100, random_state=42) + + # Preprocess the data + processed_df = preprocess_data(df) + + # Check if the processed dataframe has the expected columns + expected_columns = ["User_id", "Title", "review/score"] + assert all(col in processed_df.columns for col in expected_columns), "Not all expected columns are present." + + # Check if there are any null values remaining + assert not processed_df.isnull().values.any(), "There are still null values in the processed data." + + # Check if there are any reviews with a score less than 4. + assert (processed_df["review/score"] >= 4).all(), "There are reviews with a score less than 4." + + print("test_preprocess_data passed.") + + +if __name__ == "__main__": + test_preprocess_data() \ No newline at end of file diff --git a/experiments/training/train_model.py b/experiments/training/train_model.py index e69de29..e9bb761 100644 --- a/experiments/training/train_model.py +++ b/experiments/training/train_model.py @@ -0,0 +1,160 @@ +import pandas as pd +import numpy as np +import matplotlib.pyplot as plt +from implicit.als import AlternatingLeastSquares +from implicit.evaluation import precision_at_k, train_test_split +import scipy.sparse as sparse +import boto3 +import io +import joblib +import wandb +# When running this in EC2, wandb_tracking.py will be in the same folder... +from wandb_tracking import * + + +def load_data_from_s3(bucket : str, objectkey : str, client) -> pd.DataFrame: + """Loads a parquet dataset from S3 + + Args: + bucket (str): The bucket name where the data is stored. + objectkey (str): The path to the file in the bucket. + client: boto3 client to handle data reading. + + Returns: + _type_: Dataframe loaded from parquet data in S3. + """ + + # Gets the object using the client + obj = client.get_object(Bucket=bucket, Key=objectkey) + + # Creates a buffer so that pandas can read in the data + # (since it's not saved anywhere) + buffer = io.BytesIO(obj['Body'].read()) + + # Returns a pandas dataframe based on the parquet file read + # from the buffer. + return pd.read_parquet(buffer) + + +def save_model_to_s3(model, bucket : str, objectkey : str, client): + """Saves a trained model to S3 as a pickle file. + + Args: + model (implicit model): Trained model + bucket (str): The bucket name where the model should be stored. + objectkey (str): The path where the model should be stored. + """ + + # Creates a buffer that the model can be placed in + buffer = io.BytesIO() + + # Uses joblib to dump the model into the buffer as a pickle file. + joblib.dump(model, buffer) + + # Moves the pointer in the buffer back to the beginning (where the model is) + buffer.seek(0) + + # Puts the model file into S3 using the client. + client.put_object(Bucket=bucket, Key=objectkey, Body=buffer) + +def train_als_model( + user_item_matrix, factors : int=50, regularization : float=0.01, iterations : int=15 + ): + """Trains an ALS model using provided data and hyperparameters. + + Args: + user_item_matrix (sparse csr user-book matrix): User-item CSR matrix. + factors (int, optional): The number of latent factors to compute. Defaults to 50. + regularization (float, optional): The regularization factor to use. Defaults to 0.01. + iterations (int, optional): The number of training iterations. Defaults to 15. + + Returns: + implicit model: Returns the trained model + """ + + # Initialize the ALS model + model = AlternatingLeastSquares(factors=factors, regularization=regularization, iterations=iterations) + + # Train the model + model.fit(user_item_matrix) + + return model + +if __name__ == "__main__": + # Create a client using boto3 + client = boto3.client("s3") + + # Load data + bucket_name = 'readcrumbs' + file_key = 'data/processed/ratings-small-v1.parquet' + data = load_data_from_s3( + bucket=bucket_name, + objectkey=file_key, + client=client + ) + print(data.head()) + print(data.dtypes) + + + # Create a user-item interaction matrix + user_item_matrix = sparse.csr_matrix((data['review/score'], (data['User_id'].cat.codes, data['Title'].cat.codes))) + + # Split into training and testing CSR's + train, test = train_test_split(user_item_matrix, train_percentage=0.9, random_state=42) + + # Train the ALS model using default params and training subset + factors = 50 + regularization = 0.01 + iterations = 15 + model = train_als_model( + user_item_matrix=train, + factors=factors, + regularization=regularization, + iterations=iterations + ) + + # Evaluate the model + p_at_k = precision_at_k(model, train, test, K=10) + print(f"Factors: {factors}, Reg: {regularization} -> Precision@10: {p_at_k}") + + # Save the trained model to a pickle file on s3 + model_file = "models/als_model-small-v1.pkl" + + save_model_to_s3( + model=model, + bucket=bucket_name, + objectkey=model_file, + client=client + ) + + print("Model trained and saved successfully.") + + # Initialize wandb + run = wandb.init( + project="readcrumbs", + name="small-v1", + config={ + "regularization": regularization, + "factors": factors, + "iterations": iterations, + "model_file": model_file, + "data_version": "data/processed/ratings-small-v1.parquet", # Update with your actual data path + } + ) + + # Register model + artifact = wandb.Artifact( + name="readcrumbs-model-small-v1", + type="model" + ) + + # Add the model file to the artifact + artifact.add_file(model_file) + + # Log the artifact to wandb + run.log_artifact(artifact) + + # Log precision at k + run.log({"precision_at_k": p_at_k}) + + run.finish() \ No newline at end of file diff --git a/experiments/training/utils.py b/experiments/training/utils.py deleted file mode 100644 index e69de29..0000000 diff --git a/frontend/Dockerfile b/frontend/Dockerfile index 69416e0..2f01a57 100644 --- a/frontend/Dockerfile +++ b/frontend/Dockerfile @@ -4,17 +4,17 @@ FROM python:3.12-slim WORKDIR /app # Copy the requirements file into the container at /app -COPY frontend/requirements.txt /app/ +COPY requirements.txt /app/ # Install any needed packages specified in requirements.txt RUN pip install --no-cache-dir -r requirements.txt # Copy the application code into the container at /app -COPY frontend/readcrumbs_app.py /app/ +COPY readcrumbs_app.py /app/ # Make port 8501 available to the world outside this container EXPOSE 8501 # Run app.py when the container launches -CMD ["streamlit", "run", "app.py"] \ No newline at end of file +CMD ["streamlit", "run", "readcrumbs_app.py"] \ No newline at end of file diff --git a/frontend/readcrumbs_app.py b/frontend/readcrumbs_app.py index 1a407b6..8e356dc 100644 --- a/frontend/readcrumbs_app.py +++ b/frontend/readcrumbs_app.py @@ -1,8 +1,10 @@ import streamlit as st import requests +import uuid +import datetime # API URL (probably need to change) -API_URL = "http://54.91.115.10:8000" +API_URL = "http://44.201.69.213:8000" # Setup the streamlit page title = "Readcrumbs" @@ -11,34 +13,64 @@ st.title(title) st.write(description) +if 'userid' not in st.session_state: + st.session_state.userid = str(uuid.uuid4()) + +if 'prediction' not in st.session_state: + st.session_state.prediction = None + input_text = st.text_area("What are some of your favorite books?", height=150) -# Button and Prediction Logic: # Use the return value of st.button() to control when the prediction is made. -if st.button("Analyze Sentiment"): - # If no review is entered, show a warning. +if st.button("Get Recommendations"): + # If no favorite books are entered, show a warning. if input_text.strip() == "": - st.warning("Please enter a movie review to analyze.") + st.warning("Please at least one favorite book.") else: # Connect with the API to make a prediction. Should pass a list of titles. + # The titles do have to match exactly what is in the dataset. input_list = input_text.split(", ") - # Create json data from input_list. + # Create json data from input_list. This should match the expected input of the API. data_to_send = { - "items": input_list + "items": input_list, + "userid": st.session_state.userid } response = requests.post(f"{API_URL}/predict", json=data_to_send) if response.status_code == 200: prediction = response.json() - prediction = prediction["recs"] + st.session_state.prediction = prediction["recs"] + else: + print("API problem...") + st.session_state.prediction = [] - # Display result - st.subheader("Your Recommendations") - for r in prediction: - # display title - st.text(r) +# Display the results if they exist +if st.session_state.prediction: + st.header("Recommended Books") + for idx, book in enumerate(st.session_state.prediction): + cols = st.columns([3, 1]) + # Shows the book title and a like button + with cols[0]: + st.markdown(f"**{idx+1}. {book}**") + with cols[1]: + if st.button("👍", key=f"like_{idx}"): + feedback_data = { + "userid": st.session_state.userid+"_"+book.replace(" ", "_"), # Create unique user-book id + "recommendations": st.session_state.prediction, # Full list of recommendations + "feedback": 1, # 1 = like + "title": book, # The book being liked + "position": idx + 1, # Position in the recommendation list + "timestamp": str(datetime.datetime.now()), # Current timestamp + } + + # Save feedback to DynamoDB using the API endpoint + feedback_response = requests.post(f"{API_URL}/feedback", json=feedback_data) + if feedback_response.status_code == 200: + st.markdown("Thanks for your feedback! :smiley:") + else: + st.markdown("There was an issue submitting your feedback. Please try again.") # Footer with name and link to GitHub repository st.divider() diff --git a/images/backend_running.png b/images/backend_running.png new file mode 100644 index 0000000..5198b5b Binary files /dev/null and b/images/backend_running.png differ diff --git a/images/frontend_input.png b/images/frontend_input.png new file mode 100644 index 0000000..e8c9fe2 Binary files /dev/null and b/images/frontend_input.png differ diff --git a/images/frontend_running.png b/images/frontend_running.png new file mode 100644 index 0000000..65e943e Binary files /dev/null and b/images/frontend_running.png differ diff --git a/images/model_monitoring.png b/images/model_monitoring.png new file mode 100644 index 0000000..be14b4d Binary files /dev/null and b/images/model_monitoring.png differ diff --git a/images/model_training.png b/images/model_training.png new file mode 100644 index 0000000..c08fdf0 Binary files /dev/null and b/images/model_training.png differ diff --git a/images/running_monitoring.png b/images/running_monitoring.png new file mode 100644 index 0000000..016e9ff Binary files /dev/null and b/images/running_monitoring.png differ diff --git a/images/test_preprocess.png b/images/test_preprocess.png new file mode 100644 index 0000000..7e57516 Binary files /dev/null and b/images/test_preprocess.png differ diff --git a/images/wandbexample.png b/images/wandbexample.png new file mode 100644 index 0000000..3ae3f23 Binary files /dev/null and b/images/wandbexample.png differ diff --git a/images/wandbmodel.png b/images/wandbmodel.png new file mode 100644 index 0000000..fdf3174 Binary files /dev/null and b/images/wandbmodel.png differ diff --git a/monitoring/Dockerfile b/monitoring/Dockerfile index badb45c..d65254c 100644 --- a/monitoring/Dockerfile +++ b/monitoring/Dockerfile @@ -1,5 +1,4 @@ -# Use Python 3.11 slim image as base -FROM python:3.11-slim +FROM python:3.12-slim # Set working directory WORKDIR /app @@ -11,11 +10,10 @@ COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt # Copy application code -COPY app.py . +COPY dashboard_app.py . # Expose Streamlit port EXPOSE 8501 # Run Streamlit app -CMD ["streamlit", "run", "app.py", "--server.port=8501", "--server.address=0.0.0.0"] - +CMD ["streamlit", "run", "dashboard_app.py"] \ No newline at end of file diff --git a/monitoring/app.py b/monitoring/app.py deleted file mode 100644 index a2b6d30..0000000 --- a/monitoring/app.py +++ /dev/null @@ -1,98 +0,0 @@ -import streamlit as st -import boto3 -import pandas as pd -import matplotlib.pyplot as plt -import seaborn as sns - -s3 = boto3.client('s3') -dynamodb = boto3.client('dynamodb') - -# Helper functions -def get_data_from_dynamodb(table_name): - response = dynamodb.scan(TableName=table_name) - return response['Items'] - -def convert_to_df(items): - return pd.DataFrame(items) - - -df = convert_to_df(get_data_from_dynamodb('prediction-logs')) - -# ---------------------------- Streamlit app ---------------------------- -st.title("Monitoring Dashboard") - -df = convert_to_df(get_data_from_dynamodb('prediction-logs')) - -# Convert columns to proper types -df['datetime'] = pd.to_datetime(df['datetime'], errors='coerce') -df['user_id'] = pd.to_numeric(df['user_id'], errors='coerce') -df['prediction'] = df['prediction'].astype(str) -if 'req' in df: - df['req'] = df['req'].astype(str) - -st.header('Prediction Latency Over Time') - -if 'latency' in df.columns: - # Plot latency over time if available - fig1, ax1 = plt.subplots() - sns.lineplot(x='datetime', y='latency', data=df, ax=ax1) - ax1.set_title('Prediction Latency Over Time') - st.pyplot(fig1) -else: - st.info("No latency field present in data. Please ensure the backend logs the latency of predictions.") - -st.header('Prediction Distribution (Target Drift)') - -fig2, ax2 = plt.subplots() -sns.countplot(x='prediction', data=df, ax=ax2) -ax2.set_title('Distribution of Predicted Classes') -st.pyplot(fig2) - -st.header('Collect User Feedback') - -st.write("Click below to rate the most recent model prediction and help track accuracy.") - -user_id_input = st.text_input("User ID", "") -recent = None -if user_id_input: - try: - uid = int(user_id_input) - cur_user_rows = df[df['user_id'] == uid] - if not cur_user_rows.empty: - recent = cur_user_rows.sort_values('datetime', ascending=False).iloc[0] # get latest - st.write(f"Last prediction for User {user_id_input}:") - st.code(dict(recent), language='json') - except Exception: - st.warning("Please enter a valid numeric user ID.") - -if recent is not None: - feedback = st.radio("Are these recommendations relevant to you?", ['Yes', 'No']) - feedback_submitted = st.button("Submit Feedback") - if feedback_submitted: - - feedback_table = 'prediction-feedback' - record = { - 'user_id': {'N': str(recent['user_id'])}, - 'datetime': {'S': str(recent['datetime'])}, - 'prediction': {'S': str(recent['prediction'])}, - 'feedback': {'S': feedback} - } - try: - dynamodb.put_item(TableName=feedback_table, Item=record) - st.success("Thank you for your feedback!") - except Exception as e: - st.error(f"Failed to submit feedback: {e}") - -# Calculate live accuracy from feedback - feedback_items = convert_to_df(get_data_from_dynamodb('prediction-feedback')) - feedback_items['feedback'] = feedback_items['feedback'].astype(str) - if not feedback_items.empty: - acc = (feedback_items['feedback'] == 'Yes').mean() - st.metric("Live Model Accuracy (from feedback)", f"{acc:.2%}") - else: - st.info("No feedback yet; accuracy cannot be computed.") - - - - - diff --git a/monitoring/dashboard_app.py b/monitoring/dashboard_app.py index e69de29..468b849 100644 --- a/monitoring/dashboard_app.py +++ b/monitoring/dashboard_app.py @@ -0,0 +1,120 @@ +import streamlit as st +import boto3 +import pandas as pd +import matplotlib.pyplot as plt +import plotly.express as px +import seaborn as sns +from decimal import Decimal + +def fetch_dynamodb_table(table_name: str, region_name='us-east-1') -> pd.DataFrame: + # Get the DynamoDB resource + dynamodb = boto3.resource('dynamodb', region_name=region_name) + # Retrieve the table + table = dynamodb.Table(table_name) + + # Scan the table to get all items + response = table.scan() + # Collect all items + data = response['Items'] + + while 'LastEvaluatedKey' in response: + # Continue scanning if there are more items + response = table.scan(ExclusiveStartKey=response['LastEvaluatedKey']) + # Append new items to data list + data.extend(response['Items']) + + # Convert list of items to DataFrame + # Handle empty table case + if not data: + return pd.DataFrame() + + df = pd.DataFrame(data) + + # This explicitly targets columns that are Decimals and turns them into floats/ints + for col in df.columns: + # Check if the first non-null element is a Decimal to determine column type + sample = df[col].dropna().iloc[0] if not df[col].dropna().empty else None + if isinstance(sample, Decimal): + # Convert entire column from Decimal to float + df[col] = df[col].apply(lambda x: float(x) if x else x) + + return df + + +# Get data from DynamoDB tables +logs = fetch_dynamodb_table('readcrumbs-logs') +feedback = fetch_dynamodb_table('readcrumbs-feedback') + +# Start Streamlit app +st.title("Readcrumbs Monitoring Dashboard") + +st.header("Item Coverage") +# Spread out the predictions because they are stored as a list in each dataframe entry. +df = logs.explode('prediction') +unique_books = df['prediction'].nunique() +st.metric("Unique Recommended Books", unique_books) + +st.header("Input Diversity") +df = logs.explode('items') +unique_inputs = df['items'].nunique() +st.metric("Unique Input Combinations", unique_inputs) + +st.metric("Total Predictions Made", len(logs)) + +st.header("User Engagement") +unique_users = df['userid'].nunique() +st.metric("Unique Users", unique_users) + + +st.header("Average Latency") +avg_latency = logs['latency'].mean() +st.metric("Average Prediction Latency (ms)", f"{avg_latency:.2f}") + + +st.header('Prediction Latency Over Time') + +if 'latency' in logs.columns: + # plot using plotly for interactivity + fig = px.line(logs, x=logs.index, y='latency', title='Prediction Latency Over Time') + st.plotly_chart(fig, use_container_width=True) + fig.update_layout(xaxis_title='Timestamp', yaxis_title='Latency (ms)' ) + +else: + st.info("No latency field present in data.") + +st.header("Feedback Summary") +st.subheader("Recall At K=10") +if not feedback.empty: + # Get user ids from userid+name format + feedback['real_user_id'] = feedback['userid'].apply(lambda x: x.split("_")[0]) + # Calculate number of likes per user + likes_per_user = feedback.groupby('real_user_id')['feedback'].sum() + # Remove nonnumeric entries if any + likes_per_user = pd.to_numeric(likes_per_user, errors='coerce').fillna(1) + st.metric("Average Likes per User", f"{likes_per_user.mean():.2f}") + st.metric("Total Likes", int(likes_per_user.sum())) + # Each user got 10 recommendations, so recall@10 is likes/10 + recall_at_10 = likes_per_user.sum() / (len(likes_per_user) * 10) + st.metric("Recall@10", f"{recall_at_10:.4f}") +else: + st.info("No feedback data available.") + + + +st.header("Top Recommended Books") +df = logs.explode('prediction') +top_books = df['prediction'].value_counts().head(10) + +# Plot top recommended books +fig = px.bar(x=top_books.values, y=top_books.index, orientation='h', title='Top 10 Most Recommended Books') +st.plotly_chart(fig, use_container_width=True) +fig.update_layout(xaxis_title='Number of Recommendations', yaxis_title='Book Title') + +st.header("Top Input Books") +df = logs.explode('items') +top_inputs = df['items'].value_counts().head(10) + +# Plot top input books +fig = px.bar(x=top_inputs.values, y=top_inputs.index, orientation='h', title='Top 10 Input Books') +st.plotly_chart(fig, use_container_width=True) +fig.update_layout(xaxis_title='Number of Times Input', yaxis_title='Book Title') \ No newline at end of file diff --git a/monitoring/requirements.txt b/monitoring/requirements.txt index 261174b..6db6c00 100644 --- a/monitoring/requirements.txt +++ b/monitoring/requirements.txt @@ -1,5 +1,6 @@ streamlit boto3 pandas -matplotlib.pyplot -seaborn \ No newline at end of file +matplotlib +seaborn +plotly-express \ No newline at end of file diff --git a/prediction-table.pem b/prediction-table.pem deleted file mode 100644 index d852dd2..0000000 --- a/prediction-table.pem +++ /dev/null @@ -1,27 +0,0 @@ ------BEGIN RSA PRIVATE KEY----- -MIIEpAIBAAKCAQEA0SR5XQ2Ar4KW7f1Th3jG1IJ9oORR/841XhiUSGJhwtLRjuOr -qy9Mu4XXRHypQ0ZK8J7tRdw2pXpE9XWHtjM+hu16G5lYEfc4sQOApRfpgW/PMlUg -KBWceV+MoQB/NmQWmA6U1wMeYvPqGoG/yH6JGKQEC0iKhfSYBIhYpl0zU82XCGM+ -nTM7+jjJSzokJDDgDffqCBrzgAdTvWLmGg52iI5bDMe6Dgd7A1lS/vh3vcteNRgN -4Fx9t7H+JXf0AR5TqzV8VuPYxw3/ScUKOGkVyh+eWSYG5n+BivPnKdu2tWF370Hw -VrpJR2uQbP1lbenf2SfpiEvNGHP/ER1XuW7I8wIDAQABAoIBAQDOoysGNYEf5/cX -zWPqRfqtnQBjJzOdezBfeAmKOyo8Q++pLmk/CczWuramhET4o0sH0v68N4gGl3fq -zeT4sEjnJ1uuSSQrHAh3XO6OL8IWkVI2eMT81d10TmOz77nBE8L/GekVR4+OVVDI -P8otXlg2cFdOjq3PDIvmbpCoTw2XjZAcAkbkr7+kqc9Y126PgvcGS3sLkhUmtykb -xCTbZ4fEfLd1dZCv3xGAUcJ3NSsZhysmkEVel6+vLPch0zb2aRAKf0FLyFgNGflK -52mblsrBfbDYkM3NI1sKkHksYrrwA5rBa6tDhire/TexMIGT7JkubVLaQ1VL/R72 -2ZHCm9aBAoGBAPKWqBukvHvNoB51ksEIdJQzxqcgMuzpnRlkKhaGBws0eY2kDxkp -bfYpSd3Xjd7BUYkYty2jT3GF4s5roxF3XiTodMBj3DCuwfJQ1omYC8qwRLa8FZ8w -YkgcD0wajCJNj6fQ6O/C3pK3ub4b6U3oAFl/qEoeuAL+M/h7g38YojxlAoGBANy0 -dRmUvrg/6mERFqBgbSLjezQDPp8QlAqT/4+goaBqpWxdWMUuS3RoT9eLxk+M4c7b -fr/K0imU8iR4kq8CUWUFXXAPlXaLGZwaZETGEFkcen4DaC4n6I/6AyTYa5LQMu7H -N74I4KdLycT50KY2eA29GidiZ6NULweUrn9C7353AoGBAL58SHK0b4BzXUitn9fN -kOUSpulyoipf4okemuHmyj8lLFFpQqXKX1sM3sDA0tjYSfLyIlxGwUnuDMNzx68e -YSFwGsU7ZJohj497pIqUhqXYtYwbsoq2jmX7CpQCwIjrCGOI6m/iP61LcSFzf0Y6 -Z5PfZsEUz/8hpqN2MTIqoLH1AoGARCCYPQtDTBC+wrPJrjvVtH1P3KBbxjIR4KoK -q0VEXwZMhgTSkBtYQ1invLtyvb+ZPIdYus9azGcjz8pATTGD+pELZLoKwwrxHtSu -uuQAy+EUlq1qjUTYbwkXy1na6vjFoBtyw4BuCHZGlD0hAQ2zRVpoJlwj7bDgy5BD -xRjeYMUCgYAx6ftKAkhWZy1fGLobFnYBEMVIJEfyh3Qp/a2TOjO/Ky5kDBbbLeR3 -Mau0q4/848q14knJeJT2RS9uS/GseYFxp6egLk1FL63mf74NNzgQmK1snw/fqI1U -+t64/J9B52upx2nRrDyue4ruQdXcpEGukqYPW/VZGJ7lmcV6V6DpYw== ------END RSA PRIVATE KEY----- \ No newline at end of file diff --git a/requirements.txt b/requirements.txt deleted file mode 100644 index e69de29..0000000 diff --git a/tests/test_preprocess.py b/tests/test_preprocess.py deleted file mode 100644 index e69de29..0000000