The goal of this project is to predict the probability that an NBA field-goal attempt is made using the provided shot-context and player-tracking data.
The final output is a make_prob between 0 and 1 for every shot in testing.csv.gz.
Because the project is evaluated using log-loss, I treated this as a probability-estimation problem rather than a make/miss classification problem. Log-loss evaluates the probability assigned to the observed outcome rather than only whether a classification was correct [1].
No external basketball data were used.
The labeled training data contains two seasons, while the testing dataset comes from a different season.
Rather than randomly splitting individual shots, I validated across seasons:
- Train on Season 1 → validate on Season 2
- Train on Season 2 → validate on Season 1
This more closely matches the final task, where the model must generalize to a season it was not trained on. It also follows the general grouped cross-validation principle of keeping meaningful groups separate between training and validation data [2].
All model-selection decisions were based on held-out-season log-loss.
I started with simple baselines and added complexity only when it could be tested under the same validation framework.
| Model | Mean Held-Out-Season Log-Loss |
|---|---|
| CatBoost + tracking features | 0.627153 |
| CatBoost basic | 0.628034 |
| Logistic regression | 0.641976 |
| CatBoost + historical priors | 0.684050 |
| Constant probability | 0.689642 |
The final tracking-enhanced CatBoost model reduced log-loss by approximately 9.1% relative to the constant-probability baseline.
More importantly, the tracking features improved the otherwise similar CatBoost model from 0.628034 to 0.627153.
The first baseline predicts the training-season average make probability for every validation shot.
Mean held-out-season log-loss:
0.689642
I next trained a logistic regression model using basic shot and game-context variables.
Mean held-out-season log-loss:
0.641976
This confirmed that the supplied shot-context variables contained substantial predictive information before introducing a nonlinear model.
The dataset contains a mix of continuous variables and categorical identifiers such as shooter, defender, and team IDs.
CatBoost supports categorical variables directly and was designed to handle them within gradient-boosted trees without requiring conventional one-hot encoding of high-cardinality categories [3][4].
The basic CatBoost model achieved:
0.628034
This improved substantially on the logistic regression baseline.
One of the most useful supplied variables was closestdefapproach.
It contains four sequential defender-distance measurements leading into the shot.
Example:
{20.05,18.82,16.57,13.86}
I parsed this sequence into four individual measurements:
def_dist_t0
def_dist_t1
def_dist_t2
def_dist_t3
I then derived features describing how defensive pressure changed before the attempt:
- total defender closing distance
- interval-by-interval closing distance
- mean defender distance
- minimum and maximum defender distance
- defender-distance range
- defender-distance standard deviation
- defender-approach slope
- shooter-speed × defender-closing interaction
I also created additional shot-context features including:
- defender distance relative to shot distance
- shot distance × defender distance
- nearest contester distance
- mean contester distance
- contester-distance variability
- low-shot-clock indicators
- log-transformed dribbles
- absolute court location
The purpose was to represent not only how close a defender was, but how defensive pressure developed leading into the shot.
To test whether the additional engineering actually improved the model, I compared two otherwise similar CatBoost models.
| Feature Set | Mean Log-Loss |
|---|---|
| Basic CatBoost | 0.628034 |
| CatBoost + tracking features | 0.627153 |
The tracking-enhanced model produced the lower held-out-season log-loss, so I retained those features in the final model.
I also tested whether explicit historical performance summaries would improve the model.
I created smoothed estimates for:
- shooter make rate
- closest defender opponent make rate
- offensive team make rate
- defensive team opponent make rate
To reduce target leakage, training observations used leave-one-out calculations, while validation observations received values calculated only from the opposite training season. Cross-fitting or leave-one-out approaches are commonly used when creating target-based encodings so that an observation's own target does not directly determine its feature value [5].
This model performed substantially worse:
0.684050 mean held-out-season log-loss
I therefore removed these features from the final model.
This was an important negative result: features that appeared intuitively useful did not necessarily generalize well across seasons.
The final model relied most heavily on a combination of shot type, shot geometry, defensive pressure, player movement, and player identity.
The highest-ranked features included:
shottypedefender_dist_per_shot_distdistancelocationxshooterspeednum_contestersshooter_idshotclock
Several features derived from the defender-approach sequence also appeared among the most important predictors.
The engineered defender_dist_per_shot_dist variable ranked second in the final model.
Feature importance is used here to describe which variables the fitted model relied on most heavily. It should not be interpreted as evidence of a causal relationship.
Because the project is evaluated using predicted probabilities, I also examined whether predicted make probabilities corresponded to observed make rates on the held-out seasons.
A well-calibrated binary classifier should produce outcomes at approximately the same frequency as the probability it assigns [6].
The calibration curve remained close to the ideal diagonal across the observed probability range.
Observed make rates were slightly higher than predicted rates in several bins, indicating mild underprediction, but the differences were small.
I did not apply an additional calibration model because only two labeled seasons were available, and further fitting to the same validation data could reduce generalization to the unseen testing season.
The final model is the tracking-enhanced CatBoost model.
It was selected because it:
- produced the lowest held-out-season log-loss
- improved on the same CatBoost approach without the added tracking features
- remained well calibrated on held-out seasons
- was simpler than alternative approaches that did not improve validation performance
After model selection, I retrained the model using all labeled training shots and generated a make probability for every shot in testing.csv.gz.
The final predictions are stored in:
submission.csv
with the required columns:
shot_id,make_prob
The analysis script verifies that:
- every test shot receives a prediction
- the
shot_idorder matches the provided submission template - no probabilities are missing
- all probabilities fall between 0 and 1
Install the required Python packages:
pip install pandas numpy matplotlib scikit-learn catboostRun:
python analysis.pyThe script performs:
- data loading
- feature engineering
- whole-season validation
- baseline comparison
- CatBoost experiments
- historical-prior testing
- blend testing
- calibration analysis
- feature-importance analysis
- final model training
- test prediction generation
The final tracking-enhanced CatBoost model achieved a mean held-out-season log-loss of:
0.627153
compared with:
- 0.628034 — basic CatBoost
- 0.641976 — logistic regression
- 0.689642 — constant-probability baseline
The final model was chosen because it produced the strongest out-of-season probability estimates while avoiding additional complexity that did not improve validation performance.
[1] Scikit-learn developers. sklearn.metrics.log_loss.
https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html
[2] Scikit-learn developers. GroupKFold.
https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupKFold.html
[3] CatBoost documentation. Categorical features.
https://catboost.ai/docs/en/features/categorical-features
[4] Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2018). CatBoost: unbiased boosting with categorical features. Advances in Neural Information Processing Systems 31.
https://proceedings.neurips.cc/paper/2018/hash/14491b756b3a51daac41c24863285549-Abstract.html
[5] Scikit-learn developers. Target Encoder's Internal Cross Fitting.
https://scikit-learn.org/stable/auto_examples/preprocessing/plot_target_encoder_cross_val.html
[6] Scikit-learn developers. Probability calibration.
https://scikit-learn.org/stable/modules/calibration.html