Skip to content

[codex] Consolidate benchmark support changes - #144

Merged
bernalde merged 2 commits into
mainfrom
codex/consolidate-ancillary-pr-changes
May 14, 2026
Merged

bernalde merged 2 commits into
mainfrom
codex/consolidate-ancillary-pr-changes

Conversation

@bernalde

@bernalde bernalde commented May 14, 2026

Copy link
Copy Markdown
Member

Summary

  • Extract benchmark summary generation into gdplib.benchmark_summary and keep generate_benchmark_summary_all.py as a wrapper.
  • Standardize GAMS benchmark optcr=1e-6, record it in benchmark metadata, and update the PR58 med-term solver routing rows.
  • Add the optional Pixi gurobi environment metadata/docs and focused benchmark coverage from the open model PRs.

Validation

  • python -m pytest tests/test_benchmark.py -v --tb=short
  • python -m black -S -C --target-version py310 --check --diff benchmark.py gdplib/benchmark.py gdplib/benchmark_summary.py generate_benchmark_summary_all.py tests/test_benchmark.py
  • flake8 gdplib/ --count --select=E9,F63,F7,F82 --show-source --statistics
  • git diff --check

Notes

@bernalde
bernalde marked this pull request as ready for review May 14, 2026 14:42

@bernalde bernalde left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GitHub would not accept REQUEST_CHANGES from the PR author account, so this is submitted as COMMENT. An eligible reviewer should treat the blocking item below as a request-changes review.

Blocking issues:

  1. The packaged benchmark summary path now depends on pandas.to_markdown(), but tabulate is not declared as a runtime dependency. Fresh pip installs can finish benchmark solves and then fail during the default summary step.

Nonblocking issues:

  1. The no-results summary guard still treats Path.glob() iterators as truthy, so campaigns with no JSON results can still raise during summary generation.

Questions: None.

Tests run and outcomes:

  • /home/bernalde/.pixi/bin/pixi run pytest tests/test_benchmark.py -v --tb=short: passed, 20 tests.
  • /home/bernalde/.pixi/bin/pixi run test: passed, 287 passed, 1 skipped.
  • /home/bernalde/.pixi/bin/pixi run lint: passed; the second flake8 inventory reported existing style findings under --exit-zero.
  • gh pr checks 144 --repo SECQUOIA/gdplib: all reported checks passed.
  • Additional focused smoke with tabulate blocked: generate_benchmark_summary raises ImportError from pandas.to_markdown().

The PR should not be merged as-is. I would not merge this until the blocking issues above are addressed.


os.makedirs('benchmark_summary', exist_ok=True)

combined_problem_df.to_markdown(

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: DataFrame.to_markdown() imports pandas' optional tabulate dependency, but tabulate is only in requirements-dev.txt/Pixi and is not declared in pyproject.toml, setup.py, or requirements.txt. Because gdplib-benchmark run now calls this packaged summary path by default, a normal pip installation can complete the benchmark solves and then fail while generating summaries. Add tabulate>=0.9.0 to the runtime dependency metadata, or make Markdown output conditional while still writing the CSV summaries when tabulate is absent. Please add a fresh-install or metadata regression test for whichever behavior is intended.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in f8b4ff9 by adding tabulate>=0.9.0 to runtime dependency metadata (pyproject.toml, setup.py, requirements.txt) and covering it in CI/installation metadata tests. pixi.lock was updated for the new runtime metadata.

Comment thread gdplib/benchmark.py Outdated
@@ -1256,7 +1260,7 @@ def _generate_summary(run_id, instances):
folders = [str(_result_dir(instance, run_id)) for instance in instances]
if not any(Path(folder).glob("*.json") for folder in folders):

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nonblocking: this guard does not actually check whether any JSON files exist. Path.glob() returns an iterator object, so the outer any() is truthy for any non-empty folder list and _generate_summary() still calls generate_benchmark_summary() with no frames after a campaign that produced no JSON results. Consider if not any(any(Path(folder).glob('*.json')) for folder in folders): return False and add a no-results regression test.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in f8b4ff9 by checking for actual JSON files before summary generation and adding test_generate_summary_returns_false_without_json_results.

@bernalde

Copy link
Copy Markdown
Member Author

Pushed commit:

  • f8b4ff9 Address benchmark summary review feedback

Main changes:

  • Added tabulate>=0.9.0 as a runtime dependency in pyproject.toml, setup.py, and requirements.txt; removed the duplicate dev-only requirement and updated the Pixi lock metadata.
  • Updated CI and installation metadata checks so tabulate stays covered as a runtime dependency.
  • Fixed _generate_summary() so it returns False when the selected run has no JSON result files.
  • Added a no-results regression test for summary generation.

Tests run:

  • /home/bernalde/.pixi/bin/pixi run pytest tests/test_benchmark.py::test_generate_summary_uses_packaged_summary_module tests/test_benchmark.py::test_generate_summary_returns_false_without_json_results tests/test_pip_installation.py::TestPipInstallation::test_runtime_dependencies_specified tests/test_pip_installation.py::TestPipInstallation::test_dependency_list_consistency tests/test_installation.py::TestInstallation::test_pip_installation_dependencies -v --tb=short: passed, 5 tests.
  • /home/bernalde/.pixi/bin/pixi run test: passed, 288 passed, 1 skipped.
  • /home/bernalde/.pixi/bin/pixi run lint: passed; the second flake8 inventory reported existing findings under --exit-zero.
  • git diff --check: passed.

Comments intentionally not addressed:

  • None. Both unresolved inline comments were addressed.

Remaining risks or follow-up:

  • No solver-backed benchmark campaign was run; the changes are limited to dependency metadata and summary-generation control flow.

@bernalde
bernalde merged commit 07c6ce7 into main May 14, 2026
6 checks passed
@bernalde
bernalde deleted the codex/consolidate-ancillary-pr-changes branch May 14, 2026 16:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant