eg-soccer-analytics

An end-to-end analytics and machine learning project focused on Egyptian Premier League players and the problem of predicting market value direction.

This repository is not a toy example. It combines real football data collection, cross-source player matching, ETL, feature engineering, model training, testing, and API inference into one pipeline.

Project goal

The main question behind this project is simple:

Can we predict whether a player's market value will increase, decrease, or stay almost constant?

Instead of predicting one exact price, the project frames the problem as a 3-class classification task:

• increase
• decrease
• constant

That decision is practical for clubs, scouts, and analysts because in real football operations they often need a directional signal more than a fragile exact number.

Why this project matters

A player's price is not determined by stats alone.

On-pitch performance is important, but market value is also affected by:

• age and development stage
• injury history and availability
• club context
• playing time
• public popularity
• team strength / ranking context
• league-level competitiveness

This project already covers a strong part of that picture, and it proves that a useful football valuation pipeline can be built for the Egyptian league. At the same time, it is honest about what is still missing and what should be improved next.

What the project currently includes

The current pipeline is already built across the main stages:

1. Data collection

   • Opta feed scraping for player and match statistics
   • Transfermarkt scraping for player profiles, market value history, injuries, transfers, awards, and club standings

2. Entity matching

   • Fuzzy matching between Opta players and Transfermarkt players
   • Egyptian/Arabic name normalization and club normalization

3. ETL and validation

   • cleaning
   • schema validation
   • quarantine files for bad records
   • unified player master table

4. Feature engineering

   • performance features
   • availability and injury features
   • market context features
   • age-based features
   • club/context encodings

5. Modeling

   • multiple classifiers
   • MLflow experiment tracking
   • prediction pipeline

6. Serving

   • FastAPI endpoints for prediction

7. Testing

   • unit tests
   • integration tests
   • business/acceptance tests

Data sources

1. Opta feed

Used for rich football performance data, including many match-level indicators across seasons.

Examples of useful signal:

• minutes played
• goals
• assists
• passing
• duels
• recoveries
• defensive actions

2. Transfermarkt

Used for real-world football market context:

• player profile data
• market value history
• injury history
• transfer history
• awards
• club standings

This is a key reason the project is strong: it does not rely on only one source.

3. Wikipedia page views (experimental)

We also tried to add popularity through Wikipedia page views, because popularity can clearly affect player valuation.

That work is partially done but not production-ready yet.

What happened:

• we reached roughly 70% of the scraping goal
• Arabic player names created major matching problems
• players with the same or very similar Arabic names reduced reliable coverage
• the scraper and player-merging logic still need optimization

So the idea was correct, but the implementation needs another iteration before it can be trusted in the final modeling pipeline.

Important project insight

One of the main findings from this work is:

Stats alone are not enough to determine a player's price.

A realistic football valuation model should include more than match performance.

The next valuable factors to strengthen the project even more are:

• popularity / media attention
• stronger team rank signal
• stronger league rank signal
• club prestige and competitive context
• possibly transfer news or public sentiment later

Main features already engineered

The project already engineers a broad set of features, including:

• age and age interactions
• previous market value context
• minutes played and match participation
• attacking output per 90
• passing and progression quality
• defensive and duel metrics
• ball security and discipline
• injury and availability indicators
• club-related context

This means the project is already much deeper than a simple stats table or notebook-only experiment.

Pipeline overview

High-level flow:

Opta + Transfermarkt data
        ↓
player matching across sources
        ↓
ETL cleaning and validation
        ↓
feature building and feature engineering
        ↓
classification model training
        ↓
prediction API / evaluation

Repository structure

eg-soccer-analytics/
├── api/            FastAPI prediction service
├── notebooks/      EDA and experimentation
├── src/            scrapers, ETL, features, models, validators
├── tests/          unit, integration, and business tests
├── utils/          shared helpers and config loading
├── report.md       full technical/project report
├── testing_report.md
├── config.toml     non-secret configuration
├── pyproject.toml  project dependencies
└── Makefile        pipeline commands

How to use the project

1) Clone the repository

git clone https://github.com/MohanedAwad/eg-soccer-analytics.git
cd eg-soccer-analytics

2) Install dependencies

This project targets Python 3.11+.

Recommended:

poetry install

If you only want to run the API layer, there is also:

pip install -r requirements-api.txt

3) Add secrets for Opta access

Create a .env file in the project root and add the credentials needed by the Opta scraper:

OPTA_AUTH_TOKEN=your_token_here
OPTA_DOMAIN=your_domain_here
REFERER_DOMAIN=your_referer_here

4) Create the data directories

make dirs

5) Run the scraping and pipeline stages

Scraping

make scrape-opta-raw
make scrape-tm
make etl-match
make scrape-opta-extended

ETL

make etl

Feature generation

make features

Train the models

make train

Run tests

make test
make test-cov

6) Run the prediction API

uvicorn api.main:app --reload

Then open:

• http://127.0.0.1:8000/docs
• http://127.0.0.1:8000/health

Available API endpoints

• GET /health
• GET /models
• POST /predict
• POST /predict/batch

Current limitations

This project is complete across the main pipeline stages, but there are still realistic limitations:

• popularity is not fully integrated yet
• Arabic-name normalization and duplicate-name player merging still need optimization
• some valuable context factors should be expanded further, especially team rank and league rank
• Transfermarkt values are estimates, not true transfer fees
• the project is currently focused on one league context

These are not signs of weakness. They are exactly the kind of real-world data problems serious sports analytics projects face.

Why this project is strong

• it uses multiple real football data sources
• it solves a hard cross-source matching problem
• it includes data validation, not just scraping
• it has a full feature engineering + modeling pipeline
• it includes testing
• it exposes predictions through an API
• it documents limitations instead of hiding them

In short: this project took real effort, real data work, and real iteration.

Read the full report

For the complete project explanation, methodology, model framing, feature details, limitations, and future work, read:

• report.md
• testing_report.md

Contributing

Contributions are welcome.

If you want to help, useful contribution areas include:

• improving Arabic name matching and player merging
• optimizing the Wikipedia popularity scraper
• adding stronger team-rank and league-rank features
• improving evaluation and temporal validation
• extending the project to more leagues
• improving documentation and reproducibility

Typical workflow:

1. Fork the repository
2. Create a branch
3. Make your changes
4. Run the relevant tests
5. Open a pull request

If you are interested in football analytics, data engineering, sports scraping, or machine learning for valuation problems, your contribution would be very valuable.