Skip to content

SPEC.md

Latest commit

 

History

History
194 lines (147 loc) · 8.02 KB

SPEC.md

File metadata and controls

194 lines (147 loc) · 8.02 KB
Raw

SPEC — Forecasting Research Loop Prototype (M5 / Walmart)

Purpose: prove that an agent wired to a reproducible backtest harness can run scored forecasting experiments unattended, that an adversary agent catches leakage, and that the loop produces a candidate that beats a baseline on a frozen holdout. Public data only. This is a mechanics test, not an accuracy claim.

Hand this file to Claude Code in an empty directory and say: "Read SPEC.md and execute Phase 0 through Phase 3 in order. Stop at the end of each phase and report against the acceptance criteria before continuing."

Prerequisites (human does these once, before starting)

  1. GitHub CLI installed and authenticated: gh auth status returns logged in.
  2. Kaggle account, competition rules accepted for "M5 Forecasting - Accuracy" at https://www.kaggle.com/competitions/m5-forecasting-accuracy/rules
  3. Kaggle API token placed at ~/.kaggle/kaggle.json with permissions 600. The agent must never read, print, or move this file.
  4. Python 3.11+ and uv or pip available. 8 GB RAM is sufficient for the subset defined below.

The agent does not create accounts, does not enter credentials, and does not modify anything under ~/.kaggle.

Phase 0 — Repository and environment

Tasks

  • Create a private GitHub repository named kepler under the authenticated user with gh repo create kepler --private --clone.
  • Inside it, create the layout below. Commit as chore: scaffold.
kepler/
  SPEC.md                  (this file)
  CLAUDE.md                (researcher instructions, Phase 2)
  adversary/CLAUDE.md      (adversary instructions, Phase 3)
  Makefile
  pyproject.toml
  src/
    data.py                (download, subset, snapshot)
    features.py            (feature construction; the researcher edits this)
    model.py               (model definition; the researcher edits this)
    backtest.py            (rolling-origin folds; the researcher edits this only to add models, never the fold logic)
    scorer.py              (metrics; FROZEN, never edited by any agent)
    report.py              (aggregates runs.csv; FROZEN)
  data/
    raw/                   (gitignored)
    snapshot/              (parquet subset, committed via git-lfs or regenerated by make data)
  holdout/                 (FROZEN, gitignored, created once by make holdout)
  runs/
    runs.csv               (append-only experiment log)
  findings/                (one markdown file per experiment)
  hypotheses/              (human-written hypotheses for debate mode)
  .gitignore
  • .gitignore must include data/raw/, holdout/, .env, *.kaggle*.
  • Python dependencies: pandas, numpy, pyarrow, lightgbm, scikit-learn, statsmodels. Pin versions in pyproject.toml.
  • Add a CODEOWNERS file assigning src/scorer.py, src/report.py, and src/backtest.py fold logic to the human.

Acceptance

  • gh repo view shows the private repo.
  • make env installs dependencies and python -c "import lightgbm" succeeds.

Phase 1 — Data, harness, baseline

Data (make data)

  • Download M5 with the Kaggle CLI: kaggle competitions download -c m5-forecasting-accuracy -p data/raw and unzip.
  • Build the subset: store CA_1, department FOODS_3 only. Roughly 800 series, 1,913 days. Write to data/snapshot/sales.parquet, data/snapshot/calendar.parquet, data/snapshot/prices.parquet.
  • Record a SHA-256 of each snapshot file in data/snapshot/MANIFEST.txt. Every backtest verifies the manifest before running and aborts on mismatch.

Holdout (make holdout, run once, by the human)

  • Cut the final 28 days of the snapshot into holdout/. Remove those days from the snapshot the agents see. The agents never read holdout/. Scoring on holdout happens only via make score-holdout, which the human runs.

Backtest (make backtest MODEL=<name>)

  • Rolling-origin evaluation. Four folds, each forecasting a 28-day horizon, fold origins spaced 28 days apart, ending at the last day of the agent-visible snapshot.
  • Training data for each fold is strictly prior to the fold origin. Features may not use any information after the origin. Feature construction receives the origin date explicitly and must respect it.
  • Outputs per fold and aggregated: WRMSSE (the M5 competition metric, computed at the item-store level with the standard weights), WAPE, and bias (sum of forecast minus actual, divided by sum of actual). Also report WAPE by horizon bucket (days 1-7, 8-14, 15-28).
  • Runtime target: under 10 minutes per full backtest on a laptop. If a model exceeds 20 minutes, the run is marked timeout and discarded.

Run log (runs/runs.csv, append-only)

Columns, exactly:

run_id, timestamp, git_commit, model_name, config_hash, fold_count,
wrmsse, wape, bias, wape_h1_7, wape_h8_14, wape_h15_28,
seconds, status, findings_file, author

author is human, researcher, or adversary. status is ok, timeout, error, or rejected.

Baseline

  • seasonal_naive: value from 7 days prior. Log it.
  • lgbm_baseline: LightGBM with lags 7, 14, 28; rolling means over 7 and 28; day-of-week; month; SNAP flag; event flag; sell price. One model across all series, item and store as categorical. Log it.
  • Reproduce lgbm_baseline twice with a fixed seed. Metrics must match to four decimals. If they do not, fix determinism before proceeding.

Acceptance

  • make backtest MODEL=lgbm_baseline runs twice with identical logged metrics.
  • runs.csv has three rows: seasonal_naive, lgbm_baseline, lgbm_baseline (repeat).
  • make report prints a table of all runs sorted by WRMSSE.
  • Commit as feat: harness and baseline. Tag v0-harness.

Phase 2 — Researcher loop

Write CLAUDE.md at repo root (content in the spec handed to the operator).

Tasks

  • Start a Claude Code session in the repo root and say: "Read CLAUDE.md and run the loop for two hours."
  • Human records wall-clock start and end.

Acceptance

  • At least 10 scored runs logged with author=researcher in the two-hour window.
  • Every logged run has a matching findings file and, if kept, a branch.
  • Zero edits to frozen files (verify with git diff v0-harness -- src/scorer.py src/report.py).
  • The session report names best WRMSSE, baseline WRMSSE, and runs per hour.

Phase 3 — Adversary

Write adversary/CLAUDE.md (content in the spec handed to the operator).

Planted leak test

  • Human creates a branch exp/planted-leak that adds a feature using the rolling 7-day mean centered on the forecast date (which reaches into the future). Do not tell the adversary which branch is planted.
  • Run the adversary on all exp/* branches.

Acceptance

  • The adversary FAILs exp/planted-leak on checklist item 1 with the specific feature named.
  • The adversary produces a review for every researcher branch.
  • At least one researcher branch receives PASS.

Phase 4 — Holdout gate (human only)

  • make score-holdout MODEL=lgbm_baseline and make score-holdout MODEL=<best PASS branch>.
  • Record both in runs/holdout.csv. This file is written by the human only.
  • Do not iterate on holdout. One scoring per candidate. If the candidate loses, the loop resumes on the backtest and a new candidate is scored later.

Numbers to carry out of this test

Record in RESULTS.md at the end:

  1. Researcher runs per hour.
  2. Compute seconds per run and, if using API billing, cost per run.
  3. Did the adversary catch the planted leak (yes/no) and how many real branches did it reject.
  4. Holdout WRMSSE: baseline versus best PASS candidate.
  5. The single hypothesis the researcher flagged as most promising and unexplored.

These five numbers are the input to the compute test and the team conversation. Nothing else from this prototype is a claim about PepsiCo.

Out of scope for this prototype

  • Any PepsiCo data, code, or credentials.
  • Hierarchical reconciliation beyond the item-store level.
  • Cold-start or new-product forecasting. That is a separate harness with a different fold design.
  • Scheduling the loop overnight. Run it interactively first; add a cron job only after Phase 3 passes.