Add in 2021 Weeks 1-8 location tracking data

.gitattributes

@@ -0,0 +1 @@
+*.parquet filter=lfs diff=lfs merge=lfs -text

.github/workflows/pr-ci.yml

@@ -50,6 +50,10 @@ jobs:
       - name: Test transformer training
         run: uv run src/train_transformer.py --profile --ci
 
-        # This will use best_model.pth for inference
+        # This will use transformer.pt for inference, will make predictions for all of week 9
       - name: Test prediction generation
-        run: uv run src/generate_predictions.py --profile
+        run: uv run src/generate_predictions.py --profile
+
+        # This will use the predictions saved as parquet in stage above, only 1 plot, one animated play, and one confusion matrix is created
+      - name: Test process predictions
+        run: uv run src/process_predictions.py --profile --ci

.gitignore

@@ -194,6 +194,7 @@ log
 
 # Artifacts
 *.pkl
+artifacts/
 
 # Any mp4s that sneak in
 *.mp4

README.md

@@ -79,6 +79,22 @@ git lfs pull
 ```
 
 This will fully populate `nfl/parquet/*` with:
+
+`2021/`:
+- `games.parquet`
+- `pffScoutingData.parquet`
+- `players.parquet`
+- `plays.parquet`
+- `tracking_week_1.parquet`
+- `tracking_week_2.parquet`
+- `tracking_week_3.parquet`
+- `tracking_week_4.parquet`
+- `tracking_week_5.parquet`
+- `tracking_week_6.parquet`
+- `tracking_week_7.parquet`
+- `tracking_week_8.parquet`
+
+`2022/`:
 - `games.parquet`
 - `player_play.parquet`
 - `players.parquet`
@@ -93,7 +109,6 @@ This will fully populate `nfl/parquet/*` with:
 - `tracking_week_8.parquet`
 - `tracking_week_9.parquet`
 
-
 ## Setup python virtual environment
 This project uses `uv` for managing the python virtual environment. To install uv please see the official [documentation](https://docs.astral.sh/uv/getting-started/installation/) or install via CLI below:
 ```bash
@@ -119,9 +134,10 @@ uv run src/train_transformer.py
 uv run src/train_transformer.py --tune
 
 # Run inference on week 9
-uv run src/generate_predictions.py
+uv run src/generate_predictions.py 
 
-# From this point you can explore the notebooks/predictions.ipynb, adjust the data path as neccesary and see the animations and plots
+# Create artifacts (plot on accuracy for week 9, classification report, and animations of plays)
+uv run src/process_predictions.py
 ```
 
 ## Activating MLflow
@@ -144,11 +160,19 @@ The last command will start the MLflow GUI at your local host loopback on port 5
 ## Profiler
 The PyTorch profiler is used for examining the training process and run time inference.
 
+### Miscellaneous Profiling
+You can profile any core scripts with:
+```bash
+# Generic profiling (uses the decorators to shown time the fcns take)
+uv run src/the_script.py --profile
+
+```
 
 ### Profiling Training
+
 You can profile the training process with:
 ```bash
-uv run src/train_transformer --profile
+uv run src/train_transformer.py --profile
 ```
 
 This will do three things:
@@ -363,11 +387,8 @@ Note, if you followed instructions above to setup the git hook, ruff will run th
   - See if I can build a model to determine blitzer like red circle AWS at post snap
   - Add in confusion matrix, ROC curve, and prediction post snap accuracy to automatic predictions and not just in notebook
 - Infra
-  - Profiler
-    - Add in PyTorch profiler with new tensorboard (new thing, this is deprecated) traces into Chrome UI, built into CI for profiler
   - Optimization
     - Switch to flash attention layer which can accelerate training
   - General
     - Get docker working and implemented in CI
-    - Add git hook for ruff fix
     - Look back on mypy for type

data/parquet/2021/README.md

@@ -0,0 +1 @@
+From the 2023 bowl: https://www.kaggle.com/competitions/nfl-big-data-bowl-2023/data?select=games.csv

notebooks/eda.ipynb

@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 1,
    "id": "0fe17705",
    "metadata": {},
    "outputs": [],
@@ -17,6 +17,74 @@
     "games_df, plays_df, players_df, location_data_df = rawLoader.get_data(weeks=[i for i in range(1, 10)])"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "15113b4e",
+   "metadata": {},
+   "source": [
+    "# Exploring plays_df"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "db093750",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "       gameId  playId                                    playDescription  \\\n",
+      "0  2022102302    2655  (1:54) (Shotgun) J.Burrow pass short middle to...   \n",
+      "1  2022091809    3698  (2:13) (Shotgun) J.Burrow pass short right to ...   \n",
+      "2  2022103004    3146  (2:00) (Shotgun) D.Mills pass short right to D...   \n",
+      "3  2022110610     348  (9:28) (Shotgun) P.Mahomes pass short left to ...   \n",
+      "4  2022102700    2799  (2:16) (Shotgun) L.Jackson up the middle to TB...   \n",
+      "\n",
+      "   quarter  down  yardsToGo possessionTeam defensiveTeam yardlineSide  \\\n",
+      "0        3     1         10            CIN           ATL          CIN   \n",
+      "1        4     1         10            CIN           DAL          CIN   \n",
+      "2        4     3         12            HOU           TEN          HOU   \n",
+      "3        1     2         10             KC           TEN          TEN   \n",
+      "4        3     2          8            BAL            TB           TB   \n",
+      "\n",
+      "   yardlineNumber  ... yardsGained  homeTeamWinProbabilityAdded  \\\n",
+      "0              21  ...           9                     0.004634   \n",
+      "1               8  ...           4                     0.002847   \n",
+      "2              20  ...           6                     0.000205   \n",
+      "3              23  ...           4                    -0.001308   \n",
+      "4              27  ...          -1                     0.027141   \n",
+      "\n",
+      "   visitorTeamWinProbilityAdded expectedPointsAdded  isDropback  \\\n",
+      "0                     -0.004634            0.702717        True   \n",
+      "1                     -0.002847           -0.240509        True   \n",
+      "2                     -0.000205           -0.218480        True   \n",
+      "3                      0.001308           -0.427749        True   \n",
+      "4                     -0.027141           -0.638912       False   \n",
+      "\n",
+      "   pff_runConceptPrimary  pff_runConceptSecondary  pff_runPassOption  \\\n",
+      "0                   None                     None                  0   \n",
+      "1                   None                     None                  0   \n",
+      "2                   None                     None                  0   \n",
+      "3                   None                     None                  0   \n",
+      "4                    MAN              READ OPTION                  0   \n",
+      "\n",
+      "  pff_passCoverage pff_manZone  \n",
+      "0          Cover-3        Zone  \n",
+      "1         Quarters        Zone  \n",
+      "2         Quarters        Zone  \n",
+      "3         Quarters        Zone  \n",
+      "4          Cover-1         Man  \n",
+      "\n",
+      "[5 rows x 50 columns]\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(plays_df.head())"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "dad4cbc3",