AccelEval

A benchmark for evaluating LLMs on CPU-to-CUDA code acceleration.

AccelEval measures whether LLMs can translate sequential CPU programs into efficient CUDA kernels. Unlike prior benchmarks that focus on isolated GPU primitives or pure syntax, AccelEval stresses end-to-end acceleration across 42 production-style workloads drawn from established HPC suites (HPCG, NAS Parallel Benchmarks, Rodinia, GAPBS, miniWeather), domain-specific scientific codes (DualSPHysics, Box2D), and industrial financial / operations-research workloads (FinanceBench, OR-Tools, recent OR papers).

• 📦 Test data: Accel-Eval/AccelEval-data on Hugging Face — input binaries, expected outputs, CPU baseline times, plus a tasks.parquet manifest with the full cpu_reference.c and prompt_template.yaml for every task.
• 🔬 Tasks: 42 tasks × 3 input scales (small / medium / large)
• 🧪 Interface: every task is a single solution_compute(...) function, timed end-to-end (allocation, H↔D copy, kernel, library, cleanup) so there is no place to hide work in an untimed setup phase.
• 🧠 Decomposition pipeline: every passing solution is labelled against a 43-pattern CUDA optimization catalog and fed back as natural-language guidance for cross-model strategy transfer.

---

Quick start

# 1. Install
pip install -r requirements.txt

# 2. Pull the benchmark data from Hugging Face
python3 scripts/download_data.py small      # smoke (~50 MB)
python3 scripts/download_data.py medium     # leaderboard (~1.8 GB)
python3 scripts/download_data.py large      # stress (~2.5 GB)

# 3. Configure API keys (.env): OPENROUTER_API_KEY, ANTHROPIC_API_KEY, ...
cp .env.example .env

# 4. Generate + evaluate + analyze for one model
python3 scripts/run_all_tasks.py \
    --models gemini-3.1-pro-preview-openrouter \
    --levels 3 --samples 1 --sizes small --yes

Solutions and per-cell results are written to runs/<model>_l<level>_<timestamp>/; consolidated leaderboards land in runs/reports/.

Task categories

42 tasks across six top-level categories:

Category	Count	Examples
HPC reference kernels	7	hpcg_spmv_27pt, hpcg_symgs_sweep, hpcg_mg_vcycle, npb_cg_sparse_solve, npb_lu_ssor_structured, npb_sp_adi_pentadiagonal, miniWeather
Scientific simulation	4	sph_cell_index, sph_forces, sph_position, hotspot_2d
Graph algorithms	7	bellman_ford, held_karp_tsp, max_flow_push_relabel, rodinia_bfs_levels, gapbs_cc_afforest, gapbs_pagerank_pullgs, gapbs_triangle_orderedcount
Spatial–temporal	7	dbscan, dtw_distance, euclidean_distance_matrix, hausdorff_distance, collision_detection, smith_waterman, regex_match
Financial computing	5	black_scholes, bonds_pricing, monte_carlo, repo_pricing, batched_lhpca_portfolio
Operations research	12	crew_pairing, gittins_index, hawkes_dynamic_pricing_hjb, inventory_replenishment_dp, motzkin_straus_blp_eval, nash_flows_over_time, network_rm_dp, pathfinder_grid_dp, pdlp, robust_value_iteration_hypercube, self_exciting_pricing_dp, thompson_sampling

The full per-task manifest (source repo, brief description) is in tasks/<task_id>/task.json.

Unified solution_compute interface

Every task exposes a single function:

extern "C" void solution_compute(
    /* inputs */  int N, const float* xs, const float* ys, float eps, int minPts,
    /* output */  int* labels);

The harness passes full host-side inputs every call; the LLM-generated CUDA must do H2D copy + kernel launch + D2H copy and synchronise before returning. solution_compute is called with three warmups and five timed trials; the full wall time of every call is measured via CUDA Events, so allocation cost cannot be hidden in an untimed init() phase.

A dedicated timing-audit layer (next section) verifies that every timed call is self-contained — no allocation or compute smuggled into untimed warmup.

Timing audit

The benchmark's contract is that every solution_compute call carries its own full cost. The audit enforces and verifies this with independent layers, and reports per solution exactly which check failed:

• Per-call transparency (S0). The harness times every call individually — warmup included — and archives all of them in timing.json (warmup_ms[] + trial_ms[]). Cost hidden in an early call is visible as asymmetry in the archived series; used as a screen, never to convict.
• Reset-and-revalidate (S1). --audit-rounds N wipes device state (cudaDeviceReset) between extra rounds, rebuilds the context untimed, and re-validates the output each round. A solution that caches device pointers or results across calls dereferences dangling state — CUDA error or output mismatch. Deterministic; no threshold involved.
• Cold/warm ratio (S2). K fresh-process single-call runs (--warmup 0 --trials 1, context pre-created, CUDA_MODULE_LOADING=EAGER) measure a zero-prior-call invocation. Compliant solutions allocate inside every call, so cold ≈ warm (median ratio ≈ 1.0–1.1 in practice); a solution whose timed calls skip work done in earlier calls shows cold ≫ warm and is flagged for source inspection.
• Cold-process validation. output.txt is produced by a single cold call in a fresh process, so correctness can never rely on state accumulated across the warm calls.

python3 run.py audit --run <run_dir> --sizes medium
# → per-sample verdict + audit_results_<ts>.json
#   (violations / provisional / advisories, with the violated check named)

Deliberately non-compliant reference solutions (static-pointer caching, host-side result caching, deferred correctness) live in tests/audit_controls/ together with the expected verdict for each; the compliant baseline passes all layers with warmup semantics unchanged.

Prompt levels

Level	Includes	Purpose
L1	Task + interface + CPU code + full optimization guide	Ceiling with scaffolding
L2	Task + interface + CPU code + brief hints	Optimization selection
L3	Task + interface + CPU code only	Autonomous capability (default)

Prompts assemble from tasks/<id>/prompt_template.yaml via framework/generate_prompt.py.

Directory layout

AccelEval/
├── run.py                  # CLI entry-point (single model / single task)
├── framework/
│   ├── benchmark.py        # CUDA-Event end-to-end timing + cold-process validation
│   ├── audit.py            # Timing audit: S1 reset-revalidate, S2 cold/warm ratio
│   ├── compile.py          # nvcc compile (auto-injects weak solution_free)
│   ├── validate.py         # Output comparison (per-task tolerance)
│   ├── generate.py         # LLM dispatcher
│   ├── generate_prompt.py  # L1 / L2 / L3 prompt assembly
│   ├── run_all_tasks.py    # Generate → eval → analyze pipeline
│   ├── llm/                # Provider clients (OpenAI, Anthropic, Google, OpenRouter)
│   ├── knowledge/          # Pattern decomposition + LLM analyzer
│   └── harness_{gpu,cpu}.{cu,c}   # Timing + validation skeleton
├── tasks/<task_id>/
│   ├── task.json               # Metadata: category, difficulty, sizes, tolerance
│   ├── prompt_template.yaml    # Task description + interface + hints
│   ├── cpu_reference.c         # CPU baseline
│   ├── task_io.{cu,c}          # I/O adapter
│   ├── gen_data.py             # Generate input.bin + expected_output.txt
│   └── data/{small,medium,large}/   # ← `python3 scripts/download_data.py`
├── scripts/
│   ├── download_data.py            # Pull benchmark data from Hugging Face
│   ├── upload_to_hf.py             # Maintainer: push data to HF
│   ├── gen_all_data.sh / gen_data.sh
│   ├── run_all_tasks.py / run_dual_gpu_clean_eval.sh / clean_eval_*.sh
│   ├── consolidate_eval_data.py    # Build per-(model, task, size) leaderboard JSON
│   ├── export_xlsx.py / export_xlsx_from_consolidated.py
│   ├── analyze_pattern_impact.py   # Per-pattern within-task LIFT
│   ├── analyze_s2_control.py       # Strategy-transfer ablation (treatment vs control)
│   ├── compute_passk.py            # pass@k aggregates from a single k-sample run
│   ├── plot_pattern_cooccurrence.py / plot_scale_*.py
│   └── run_human_baselines.sh
├── tests/
│   └── audit_controls/     # Planted-cheat + compliant solutions validating the audit
├── docs/                   # REPRODUCE.md, task_porting_guide.md
└── runs/                   # Generated solutions + eval results (gitignored)

Common workflows

# Eval already-generated .cu files against a fresh data download (no API calls)
python3 run.py eval --run runs/<model>_<config>_<date> --sizes medium

# Timing audit of the passing solutions (names the violated check per sample)
python3 run.py audit --run runs/<model>_<config>_<date> --sizes medium

# Cross-model summary
python3 scripts/consolidate_eval_data.py
python3 scripts/export_xlsx_from_consolidated.py

# Best-of-k pass@k leaderboard (after generating k samples per task in ONE run)
python3 scripts/compute_passk.py --runs runs/<model>_<config>_<date> --k 3

# Decomposition pipeline: pattern attribution + LIFT analysis
python3 scripts/analyze_pattern_impact.py
python3 scripts/plot_pattern_cooccurrence.py

# Strategy-transfer Stage-2 (treatment + length-matched control)
python3 scripts/analyze_s2_control.py

The end-to-end re-run that produced the public leaderboard is in docs/REPRODUCE.md.

Adding a new task

1. Create tasks/<task_id>/
2. Write task.json — metadata including "interface_mode": "compute_only"
3. Write prompt_template.yaml — description + single solution_compute signature + L1 / L2 hints
4. Write cpu_reference.c — pure computation, one solution_compute(...) function, no I/O
5. Write task_io.cu and task_io_cpu.c — read input.bin into ctx, call solution_compute
6. Write gen_data.py — produce input.bin and the expected output via the CPU baseline
7. Run python3 tasks/<task_id>/gen_data.py small tasks/<task_id>/data/small --with-expected

The full porting workflow is documented in docs/task_porting_guide.md.

Environment requirements

• Python 3.10+
• CUDA Toolkit 12.0+ (nvcc on PATH)
• NVIDIA GPU; targets sm_80 and newer (default sm_89 — H200 / RTX 4090)
• nsys (optional, for kernel-level profiling)
• pip install huggingface_hub for data download

Contributing

Bug reports, new tasks, and new LLM-provider integrations are welcome. For large task additions, please include a gen_data.py that produces deterministic output and keeps the medium size under three minutes of single-thread CPU baseline time.

License

Apache 2.0 — see LICENSE.