Portfolio cleanup: fix CLI and pbp bugs, stop publishing third-party data, remove leftovers

.gitignore

@@ -52,6 +52,20 @@ __pycache__/
 #   apps/*/models/**   the 10 trained model artifacts
 # Trevor cleared public real data and USD branding on 2026-09-12. A repo whose apps
 # cannot start from a fresh clone is a demo nobody can run.
+#
+# Except third-party data, ignored since 2026-10-27: that clearance covers USD's own
+# TrackMan and branding, not data licensed or scraped from others. 6-4-3 Charts exports
+# and the bios built from PerfectGame, Sidearm and D1Baseball stay local, and so do the
+# generated boards, which name athletes and carry fit and likelihood scores.
+data/643_exports/**/*.csv
+data/hot_board.csv
+data/board_by_role.csv
+data/california_board.csv
+data/ca_ties_baseline.csv
+data/portal_board.html
+data/portal_board.xlsx
+data/portal_data_gaps.csv
+apps/hitter/docs/D1+Baseball+Reference+Docs.pdf
 
 # Raw baseballr pulls, rebuilt by research/r/mlb/fetch_cache.R
 data/mlb/raw/
@@ -77,3 +91,6 @@ data/mlb/parquet/
 *.bak
 *.err
 tmp_*
+
+# Raw HTML dumped by the pbp scrapers when parsing fails
+ncaa_pbp_debug.html

README.md

@@ -1,5 +1,7 @@
 # baseball-analytics
 
+By Trevor White ([@tdub29](https://github.com/tdub29) on GitHub).
+
 Baseball scouting, player evaluation, and MLB game modeling. The NCAA platform combines
 transfer-portal scouting, pitch-level models, hitter and pitcher Streamlit apps, and play-by-play
 reports used by the University of San Diego coaching staff. The MLB work covers Statcast pitch
@@ -11,6 +13,24 @@ Stuff+, whiff, and pitch-location models. The [model index](models/README.md) co
 artifacts to their source notebooks. Game-level MLB research has its own methods, experiments,
 and evaluation reports below.
 
+## Highlights
+
+- **Can a public-data model beat Vegas? No, and the test says how close it came.** A model
+  built only from Retrosheet and MLB StatsAPI was frozen, then scored once on 6,451 games from
+  2023 to 2025. Its log loss trails the no-vig closing line by 0.0036 per game (0.0018, borderline,
+  once 12 dates with late-scraped "closing" lines are dropped). Its bets at the opening line moved
+  toward the close by about 2 probability points, but the return cannot be told apart from zero.
+  The rules for every verdict were written down before the test data was scored.
+  [Read the report](research/r/mlb/REPORT.md).
+- **Coaches use these tools every week.** USD's staff uses the hitter and pitcher apps, the
+  transfer-portal board and the per-game PDF reports ([samples](docs/samples/README.md)).
+- **One method at two levels.** Stuff+ and whiff models are trained on NCAA TrackMan. The xSLG
+  and swing-decision models are trained on MLB Statcast and applied to NCAA pitches in the hitter
+  app ([model index](models/README.md)).
+- **Designed around a 2FA wall.** The NCAA portal login needs a coach's 2FA code every time, so
+  about 90 percent of the portal workflow runs from public sources with no login at all
+  ([design](docs/transfer-portal-pipeline.md)).
+
 ## Scouting and pitch models
 
 | Work | What it provides | Start here |
@@ -75,15 +95,15 @@ Related projects: [original R pipeline](research/r/legacy/) and
 | Area | Size | Notes |
 |---|---|---|
 | `src/ncaa/portal/` | 28 files, 5,296 lines | Portal ingest, dedupe, entity resolution, enrichment, geo, program tier, rollup, fit scoring, alerts |
-| `src/ncaa/pbp/` | 25 files, 6,516 lines | NCAA play-by-play acquisition (Playwright plus `ncaa-api.henrygd.me`), situational "battle" scoring, season PDF reports |
+| `src/ncaa/pbp/` | 26 code files, 6,550 lines | NCAA play-by-play acquisition (Playwright plus `ncaa-api.henrygd.me`), situational "battle" scoring, season PDF reports |
 | `src/ncaa/cli.py` | 336 lines | `init`, `import-excel`, `ingest`, `enrich-643`, `evaluate`, `board`, `run-all` |
 | `warehouse/` | 18 SQL files, 925 lines | dbt-duckdb rebuild of the transform layer: 3 staging, 1 intermediate and 7 mart models (dims, `fct_pitch`, season marts), 3 singular tests, a pitch-type seed, model contracts, source freshness and sample fixtures |
 | `apps/hitter/`, `apps/pitcher/` | 5,275 lines | The two Streamlit apps coaches open, with their 10 model artifacts |
 | `scripts/` | 22 files | NCAA bio fetchers (D1/D2/D3/portal), school geocoding, Excel import and export, board HTML build, Netlify publish, Supabase RLS verify |
-| `tests/` plus the pbp tests | 15 pytest files, 98 test functions | bio, board, d1baseball, dedupe, geo, overlay, RLS schema, pipeline, program tier, resolve, rollup, speed, status, battle calc, and a guard that the battle converter stays linear in input size |
-| `research/notebooks/` | 20 notebooks | 8 MLB, 12 NCAA, outputs stripped (verified 2026-09-16: 0 of the 20 carry embedded outputs). Every shipped model was trained in one of these |
+| `tests/` plus the pbp tests | 16 pytest files, 103 test functions | bio, board, d1baseball, dedupe, geo, overlay, paths, RLS schema, pipeline, program tier, resolve, rollup, speed, status, battle calc, and a guard that the battle converter stays linear in input size |
+| `research/notebooks/` | 20 notebooks | 8 MLB, 12 NCAA, outputs stripped (verified 2026-09-16: none carry embedded outputs). Every shipped model was trained in one of these |
 | `research/r/` | MLB pipelines, studies, simulator, and testthat suite | Run expectancy, recency and matchup models, bullpen usage, Monte Carlo game simulation, market comparisons, and forward-season evaluation. The original pipeline is preserved in `research/r/legacy/`. |
-| `docs/` | 12 design docs | Pipeline design, 2026 sourcing, capabilities and blockers, Netlify and Supabase recipes, the consolidation plan |
+| `docs/` | 10 design docs | Pipeline design, 2026 sourcing, capabilities and blockers, Netlify and Supabase recipes, the consolidation plan |
 
 Model provenance, which notebook trained which artifact: [`models/README.md`](models/README.md).
 
@@ -97,14 +117,20 @@ Everything under `research/` is exactly that, and several notebooks depend on a
 
 ## Quickstart
 
-Real data ships with this repo, so a fresh clone runs without credentials.
+The pipeline runs from a fresh clone without credentials, but it needs portal entries to rank.
+Those come from the coaching staff's transfer-portal workbook (sheets `Names + Notes`,
+`All Divisions Pitcher Data`, `Hitter Hot Board Data`, `RosterInfo`), which is not committed, or
+from the live sources. Without either, `evaluate` and `board` run and produce an empty board.
+6-4-3 Charts stat exports are licensed and are not committed either; `data/643_exports/README.md`
+gives the format to drop your own into.
 
 ```bash
 pip install -e ".[scoring,ncaa-stats]"
-python -m ncaa.cli init --force            # create db/baseball.db from db/schema.sql
-python -m ncaa.cli import-excel            # seed from the committed workbook
-python -m ncaa.cli evaluate                # compute fit scores
-python -m ncaa.cli board --limit 150       # write data/hot_board.csv
+python -m ncaa.cli init --force                     # create db/baseball.db from db/schema.sql
+python -m ncaa.cli import-excel path/to/workbook.xlsx   # seed portal entries
+python -m ncaa.cli enrich-643 --dir data/643_exports    # load your own 6-4-3 exports, if licensed
+python -m ncaa.cli evaluate                         # compute fit scores
+python -m ncaa.cli board --limit 150                # write data/hot_board.csv
 ```
 
 Live sources need a `config.yaml` (copy `config.example.yaml`). The D1Baseball adapter needs a
@@ -139,7 +165,8 @@ Rscript research/r/mlb/run_season.R 2019   # one season end to end
 ## Sample output
 
 `docs/samples/` holds real per-game battle reports the pbp pipeline generated for the 2026
-season. `data/portal_board.html` is a published hot board.
+season. The coach board itself (`scripts/build_html.py`) is generated locally and not committed,
+because it names athletes and carries fit scores.
 
 ## What this repo replaced
 
@@ -151,9 +178,11 @@ fields for the top call targets. `docs/transfer-portal-pipeline.md` has the full
 
 ## Data and attribution
 
-Real TrackMan and portal exports are committed on purpose, because an app nobody can start is a
-screenshot. The files name individual athletes and carry player IDs. USD branding, names and
-data are used with permission.
+USD's own TrackMan data is committed on purpose, because an app nobody can start is a
+screenshot; USD branding, names and data are used with permission. Data licensed or scraped from
+third parties (6-4-3 Charts exports, PerfectGame, Sidearm and D1Baseball bios) and the generated
+recruiting boards are kept local and gitignored. The NCAA play-by-play under `data/pbp/` comes
+from public box scores.
 
 Secrets are a separate matter and are not committed: `config.yaml`, `secrets/`, `.env`, keys and
 service-account JSON are all ignored, and `git ls-files` is checked against that list before any