cag-nasbench — does CAG γ predict ANN training dynamics on NAS-Bench-201?

Extension of the cag-snn result (γ predicts SNN convergence rate on MNIST with ρ=−0.894, R²=0.92) to ANN architectures via NAS-Bench-201. Tests whether the Fiedler value of a cell DAG's normalized Laplacian predicts convergence rate and final accuracy across 15,625 pre-trained architectures, without training anything.

Core claim to test

For each NAS-Bench-201 architecture, compute γ (Fiedler value of normalized Laplacian of the cell DAG) at init, then correlate against stored metrics:

• Epochs-to-convergence (first epoch at ≥ 95% of final accuracy)
• Final test accuracy
• Standard zero-cost proxies (NTK condition number, synflow, snip, etc.)

Hypothesis: γ is a competitive zero-cost proxy, possibly capturing information orthogonal to existing proxies.

See PRECOMMIT.md for the full locked experimental protocol and decision rules.

Setup

git clone https://github.com/tomdif/cag-nasbench.git
cd cag-nasbench
pip install -r requirements.txt

# Download NAS-Bench-201 from https://github.com/D-X-Y/NAS-Bench-201
# (~2GB .pth file — instructions in their README)
# Place at ./NAS-Bench-201-v1_1-096897.pth

Run

# Step 0: sanity-check γ computation (no benchmark needed)
python gamma_cell.py

# Step 1: sweep 15,625 architectures (few minutes, CPU-only)
python run_experiment.py --api_path ./NAS-Bench-201-v1_1-096897.pth --out results.csv

# Step 1 quick test first (100 archs, ~30 sec):
python run_experiment.py --api_path ... --max_arch 100 --out smoke.csv

# Step 2: compute zero-cost proxies via Abdelfattah et al.'s repo
#   https://github.com/SamsungLabs/zero-cost-nas
#   Run their scripts to produce proxies.csv with (arch_idx, grad_norm, snip,
#   grasp, fisher, synflow, jacob_cov, plain, naswot, ntk_cond)
#   A few hours on one GPU.

# Step 3: analyze
python analyze.py --results results.csv --proxies proxies.csv --out_dir ./out

Outputs:

• out/correlations.json — all pairwise Spearman ρ + partials
• out/depth_slopes.json — log-log slopes within each depth bin
• out/verdict.txt — automated go/no-go per the precommit rules

What to look for

From the locked precommit:

1. |ρ(γ, cifar10_conv_epoch)| > 0.3, p < 0.004 → H1 hits (primary claim)
2. |ρ(γ, cifar10_final_acc)| > 0.3 → H2 hits (secondary)
3. ρ(γ, ntk_cond) determines framing: novel (< 0.5), related (0.5–0.9), or redundant (> 0.9)
4. slope ≈ −D within each depth bin → H4 theory generalizes

Compute budget

• γ computation: minutes (eigenvalues of 4×4 matrices)
• NAS-Bench-201 metric lookup: ~5 min across all 15k archs
• Zero-cost proxy computation (using SamsungLabs repo): overnight on a single GPU
• Analysis: seconds
• Total: ~1 day of calendar time

Files

File	Purpose
PRECOMMIT.md	Locked hypotheses + decision rules
gamma_cell.py	γ_topo and γ_weighted computation for a NAS-Bench-201 cell
run_experiment.py	Sweep 15k architectures, join with benchmark metrics
analyze.py	Spearman correlations, partials, depth slopes, verdict
requirements.txt	Python deps

Status

Code staged 2026-04-23. Not yet run — see PRECOMMIT.md for what the results would need to look like to support each level of claim.

License

MIT