[codex] Consolidate benchmark support changes - #144
Conversation
bernalde
left a comment
There was a problem hiding this comment.
GitHub would not accept REQUEST_CHANGES from the PR author account, so this is submitted as COMMENT. An eligible reviewer should treat the blocking item below as a request-changes review.
Blocking issues:
- The packaged benchmark summary path now depends on pandas.to_markdown(), but tabulate is not declared as a runtime dependency. Fresh pip installs can finish benchmark solves and then fail during the default summary step.
Nonblocking issues:
- The no-results summary guard still treats Path.glob() iterators as truthy, so campaigns with no JSON results can still raise during summary generation.
Questions: None.
Tests run and outcomes:
- /home/bernalde/.pixi/bin/pixi run pytest tests/test_benchmark.py -v --tb=short: passed, 20 tests.
- /home/bernalde/.pixi/bin/pixi run test: passed, 287 passed, 1 skipped.
- /home/bernalde/.pixi/bin/pixi run lint: passed; the second flake8 inventory reported existing style findings under --exit-zero.
- gh pr checks 144 --repo SECQUOIA/gdplib: all reported checks passed.
- Additional focused smoke with tabulate blocked: generate_benchmark_summary raises ImportError from pandas.to_markdown().
The PR should not be merged as-is. I would not merge this until the blocking issues above are addressed.
|
|
||
| os.makedirs('benchmark_summary', exist_ok=True) | ||
|
|
||
| combined_problem_df.to_markdown( |
There was a problem hiding this comment.
Blocking: DataFrame.to_markdown() imports pandas' optional tabulate dependency, but tabulate is only in requirements-dev.txt/Pixi and is not declared in pyproject.toml, setup.py, or requirements.txt. Because gdplib-benchmark run now calls this packaged summary path by default, a normal pip installation can complete the benchmark solves and then fail while generating summaries. Add tabulate>=0.9.0 to the runtime dependency metadata, or make Markdown output conditional while still writing the CSV summaries when tabulate is absent. Please add a fresh-install or metadata regression test for whichever behavior is intended.
There was a problem hiding this comment.
Addressed in f8b4ff9 by adding tabulate>=0.9.0 to runtime dependency metadata (pyproject.toml, setup.py, requirements.txt) and covering it in CI/installation metadata tests. pixi.lock was updated for the new runtime metadata.
| @@ -1256,7 +1260,7 @@ def _generate_summary(run_id, instances): | |||
| folders = [str(_result_dir(instance, run_id)) for instance in instances] | |||
| if not any(Path(folder).glob("*.json") for folder in folders): | |||
There was a problem hiding this comment.
Nonblocking: this guard does not actually check whether any JSON files exist. Path.glob() returns an iterator object, so the outer any() is truthy for any non-empty folder list and _generate_summary() still calls generate_benchmark_summary() with no frames after a campaign that produced no JSON results. Consider if not any(any(Path(folder).glob('*.json')) for folder in folders): return False and add a no-results regression test.
There was a problem hiding this comment.
Addressed in f8b4ff9 by checking for actual JSON files before summary generation and adding test_generate_summary_returns_false_without_json_results.
|
Pushed commit:
Main changes:
Tests run:
Comments intentionally not addressed:
Remaining risks or follow-up:
|
Summary
gdplib.benchmark_summaryand keepgenerate_benchmark_summary_all.pyas a wrapper.optcr=1e-6, record it in benchmark metadata, and update the PR58 med-term solver routing rows.gurobienvironment metadata/docs and focused benchmark coverage from the open model PRs.Validation
python -m pytest tests/test_benchmark.py -v --tb=shortpython -m black -S -C --target-version py310 --check --diff benchmark.py gdplib/benchmark.py gdplib/benchmark_summary.py generate_benchmark_summary_all.py tests/test_benchmark.pyflake8 gdplib/ --count --select=E9,F63,F7,F82 --show-source --statisticsgit diff --checkNotes