From 0c02b96b1005d7841a548c3c99bb9f99c37a29e4 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 8 May 2026 07:06:52 +0000 Subject: [PATCH] Rewrite README with full project overview Agent-Logs-Url: https://github.com/MohanedAwad/eg-soccer-analytics/sessions/7e81bd19-1b62-472d-a193-9db7629dedbe Co-authored-by: MohanedAwad <125491698+MohanedAwad@users.noreply.github.com> --- README.md | 345 +++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 274 insertions(+), 71 deletions(-) diff --git a/README.md b/README.md index 1376da4..f567dc1 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,188 @@ # eg-soccer-analytics -Analytics and exploratory research on **Egyptian soccer** using Python and Jupyter Notebooks. -This repository is primarily notebook-based (Jupyter Notebook + Python) and is intended for data exploration, cleaning, visualization, and (optionally) modeling. +An end-to-end analytics and machine learning project focused on **Egyptian Premier League players** and the problem of **predicting market value direction**. -## Contents +This repository is not a toy example. It combines real football data collection, cross-source player matching, ETL, feature engineering, model training, testing, and API inference into one pipeline. -Typical things you’ll find / add in this project: +## Project goal -- **Notebooks** for exploration and analysis (EDA) -- **Python helpers** (utilities for loading/cleaning data) -- **Datasets** (raw and/or processed) -- **Outputs** (figures, tables, exported CSVs) +The main question behind this project is simple: -> If your repo uses specific folder names (e.g., `notebooks/`, `data/`, `src/`, `reports/`), update the sections below to match your structure. +> **Can we predict whether a player's market value will increase, decrease, or stay almost constant?** -## Getting Started +Instead of predicting one exact price, the project frames the problem as a **3-class classification task**: + +- `increase` +- `decrease` +- `constant` + +That decision is practical for clubs, scouts, and analysts because in real football operations they often need a directional signal more than a fragile exact number. + +## Why this project matters + +A player's price is **not determined by stats alone**. + +On-pitch performance is important, but market value is also affected by: + +- age and development stage +- injury history and availability +- club context +- playing time +- public popularity +- team strength / ranking context +- league-level competitiveness + +This project already covers a strong part of that picture, and it proves that a useful football valuation pipeline can be built for the Egyptian league. At the same time, it is honest about what is still missing and what should be improved next. + +## What the project currently includes + +The current pipeline is already built across the main stages: + +1. **Data collection** + - Opta feed scraping for player and match statistics + - Transfermarkt scraping for player profiles, market value history, injuries, transfers, awards, and club standings + +2. **Entity matching** + - Fuzzy matching between Opta players and Transfermarkt players + - Egyptian/Arabic name normalization and club normalization + +3. **ETL and validation** + - cleaning + - schema validation + - quarantine files for bad records + - unified player master table + +4. **Feature engineering** + - performance features + - availability and injury features + - market context features + - age-based features + - club/context encodings + +5. **Modeling** + - multiple classifiers + - MLflow experiment tracking + - prediction pipeline + +6. **Serving** + - FastAPI endpoints for prediction + +7. **Testing** + - unit tests + - integration tests + - business/acceptance tests + +## Data sources + +### 1. Opta feed + +Used for rich football performance data, including many match-level indicators across seasons. + +Examples of useful signal: + +- minutes played +- goals +- assists +- passing +- duels +- recoveries +- defensive actions + +### 2. Transfermarkt + +Used for real-world football market context: + +- player profile data +- market value history +- injury history +- transfer history +- awards +- club standings + +This is a key reason the project is strong: it does not rely on only one source. + +### 3. Wikipedia page views (experimental) + +We also tried to add **popularity** through Wikipedia page views, because popularity can clearly affect player valuation. + +That work is **partially done but not production-ready yet**. + +What happened: + +- we reached roughly **70%** of the scraping goal +- Arabic player names created major matching problems +- players with the same or very similar Arabic names reduced reliable coverage +- the scraper and player-merging logic still need optimization + +So the idea was correct, but the implementation needs another iteration before it can be trusted in the final modeling pipeline. + +## Important project insight + +One of the main findings from this work is: + +> **Stats alone are not enough to determine a player's price.** + +A realistic football valuation model should include more than match performance. + +The next valuable factors to strengthen the project even more are: + +- popularity / media attention +- stronger team rank signal +- stronger league rank signal +- club prestige and competitive context +- possibly transfer news or public sentiment later + +## Main features already engineered + +The project already engineers a broad set of features, including: + +- age and age interactions +- previous market value context +- minutes played and match participation +- attacking output per 90 +- passing and progression quality +- defensive and duel metrics +- ball security and discipline +- injury and availability indicators +- club-related context + +This means the project is already much deeper than a simple stats table or notebook-only experiment. + +## Pipeline overview + +High-level flow: + +```text +Opta + Transfermarkt data + ↓ +player matching across sources + ↓ +ETL cleaning and validation + ↓ +feature building and feature engineering + ↓ +classification model training + ↓ +prediction API / evaluation +``` + +## Repository structure + +```text +eg-soccer-analytics/ +├── api/ FastAPI prediction service +├── notebooks/ EDA and experimentation +├── src/ scrapers, ETL, features, models, validators +├── tests/ unit, integration, and business tests +├── utils/ shared helpers and config loading +├── report.md full technical/project report +├── testing_report.md +├── config.toml non-secret configuration +├── pyproject.toml project dependencies +└── Makefile pipeline commands +``` + +## How to use the project ### 1) Clone the repository @@ -23,107 +191,142 @@ git clone https://github.com/MohanedAwad/eg-soccer-analytics.git cd eg-soccer-analytics ``` -### 2) Create a Python environment (recommended) +### 2) Install dependencies + +This project targets **Python 3.11+**. -Using `venv`: +Recommended: ```bash -python -m venv .venv -# macOS/Linux -source .venv/bin/activate -# Windows (PowerShell) -.venv\Scripts\Activate.ps1 +poetry install ``` -### 3) Install dependencies - -If you have a `requirements.txt`: +If you only want to run the API layer, there is also: ```bash -pip install -r requirements.txt +pip install -r requirements-api.txt ``` -If you *don’t* have one yet, you can start with: +### 3) Add secrets for Opta access + +Create a `.env` file in the project root and add the credentials needed by the Opta scraper: + +```env +OPTA_AUTH_TOKEN=your_token_here +OPTA_DOMAIN=your_domain_here +REFERER_DOMAIN=your_referer_here +``` + +### 4) Create the data directories ```bash -pip install jupyter pandas numpy matplotlib seaborn scikit-learn +make dirs ``` -(Adjust based on what your notebooks actually import.) +### 5) Run the scraping and pipeline stages -### 4) Run Jupyter +#### Scraping ```bash -jupyter notebook +make scrape-opta-raw +make scrape-tm +make etl-match +make scrape-opta-extended ``` -or: +#### ETL ```bash -jupyter lab +make etl ``` -Open the main notebook(s) and run cells top-to-bottom. +#### Feature generation -## Data +```bash +make features +``` -This project may use soccer data such as: +#### Train the models -- match results -- team/player stats -- event data (passes, shots, etc.) if available -- scraped public sources or manually curated datasets +```bash +make train +``` -### Suggested structure (optional) +#### Run tests +```bash +make test +make test-cov ``` -data/ - raw/ # original data dumps (do not modify) - interim/ # intermediate transformed data - processed/ # clean datasets ready for analysis/modeling -notebooks/ # Jupyter notebooks -src/ # reusable Python code (loading, cleaning, features) -reports/ - figures/ # saved plots - tables/ # exported summary tables + +### 6) Run the prediction API + +```bash +uvicorn api.main:app --reload ``` -### Notes -- If you commit data, consider keeping **raw data immutable** and generating processed outputs via notebooks/scripts. -- If the dataset is large or has licensing restrictions, add instructions for how to obtain it. +Then open: + +- `http://127.0.0.1:8000/docs` +- `http://127.0.0.1:8000/health` + +## Available API endpoints + +- `GET /health` +- `GET /models` +- `POST /predict` +- `POST /predict/batch` + +## Current limitations + +This project is complete across the main pipeline stages, but there are still realistic limitations: + +- popularity is not fully integrated yet +- Arabic-name normalization and duplicate-name player merging still need optimization +- some valuable context factors should be expanded further, especially **team rank** and **league rank** +- Transfermarkt values are estimates, not true transfer fees +- the project is currently focused on one league context + +These are not signs of weakness. They are exactly the kind of real-world data problems serious sports analytics projects face. + +## Why this project is strong -## Reproducibility +- it uses **multiple real football data sources** +- it solves a hard **cross-source matching** problem +- it includes **data validation**, not just scraping +- it has a full **feature engineering + modeling** pipeline +- it includes **testing** +- it exposes predictions through an **API** +- it documents limitations instead of hiding them -To make notebook runs reproducible, consider adding: -- `requirements.txt` or `environment.yml` -- a “Run order” section (which notebook to run first) -- fixed random seeds for modeling -- exported outputs (figures/tables) to `reports/` +In short: this project took real effort, real data work, and real iteration. -## Results / Outputs +## Read the full report -Common outputs include: -- charts (team performance trends, form, goal distributions) -- tables (league standings, player rankings, xG summaries if available) -- exported datasets (`.csv`) after cleaning/feature engineering +For the complete project explanation, methodology, model framing, feature details, limitations, and future work, read: -If you generate outputs, document where they are saved (e.g., `reports/figures/`). +- `report.md` +- `testing_report.md` ## Contributing -Contributions are welcome. A typical workflow: +Contributions are welcome. -1. Fork the repo -2. Create a feature branch: `git checkout -b feature/my-change` -3. Commit changes -4. Open a pull request +If you want to help, useful contribution areas include: -## License +- improving Arabic name matching and player merging +- optimizing the Wikipedia popularity scraper +- adding stronger team-rank and league-rank features +- improving evaluation and temporal validation +- extending the project to more leagues +- improving documentation and reproducibility -Add a license if you plan to share/redistribute. -If unsure, consider adding an MIT License (for code) and documenting any dataset licenses/attribution requirements. +Typical workflow: ---- +1. Fork the repository +2. Create a branch +3. Make your changes +4. Run the relevant tests +5. Open a pull request -### Maintainer -- GitHub: https://github.com/MohanedAwad"# eg-soccer-analytics" +If you are interested in football analytics, data engineering, sports scraping, or machine learning for valuation problems, your contribution would be very valuable.