Follow-up items from Plan B Task 5 (#129, SWE-bench runtime measurement). All non-blocking; documented for later.
1. MEASURE_SAMPLES>1 status-label aggregation (deploy/knative/measure-swebench-runtimes.sh)
With more than one sample per instance, the per-sample status label reflects only the last sample, and median could mix a real runtime with the timeout-cap value. Purely cosmetic at the default MEASURE_SAMPLES=1 (the only exercised path). If multi-sample is ever used, treat any timeout sample as dominating and aggregate ms only within a single status class.
2. CI shellcheck for the measurement driver
Add deploy/knative/measure-swebench-runtimes.sh to the CI shellcheck coverage (it was shellcheck-clean at authoring, but isn't gated in CI).
3. matplotlib collection-as-error under newer setuptools
matplotlib__matplotlib-24177's measured runtime is import+collection only: its pytest collection errors on a DeprecationWarning-as-error at base_commit (newer setuptools in the baked env), so 0 tests are collected. The recorded time is faithful to the gold command but undervalues the instance. If a truer matplotlib number is wanted, relax the warning filter for the measurement run (e.g. PYTHONWARNINGS=default) — note this deviates from the exact gold test_cmd.
4. test_cmd-as-list defensive branch (scripts/gen_swebench_deck.py)
The list→string join branch is unexercised with swebench 4.1.0 (returns str); kept for forward-compat. Add a unit case if a future swebench version returns a list.
Assisted-By: Claude Code
Follow-up items from Plan B Task 5 (#129, SWE-bench runtime measurement). All non-blocking; documented for later.
1.
MEASURE_SAMPLES>1status-label aggregation (deploy/knative/measure-swebench-runtimes.sh)With more than one sample per instance, the per-sample
statuslabel reflects only the last sample, andmediancould mix a real runtime with the timeout-cap value. Purely cosmetic at the defaultMEASURE_SAMPLES=1(the only exercised path). If multi-sample is ever used, treat anytimeoutsample as dominating and aggregate ms only within a single status class.2. CI shellcheck for the measurement driver
Add
deploy/knative/measure-swebench-runtimes.shto the CI shellcheck coverage (it was shellcheck-clean at authoring, but isn't gated in CI).3. matplotlib collection-as-error under newer setuptools
matplotlib__matplotlib-24177's measured runtime is import+collection only: its pytest collection errors on aDeprecationWarning-as-error atbase_commit(newer setuptools in the baked env), so 0 tests are collected. The recorded time is faithful to the gold command but undervalues the instance. If a truer matplotlib number is wanted, relax the warning filter for the measurement run (e.g.PYTHONWARNINGS=default) — note this deviates from the exact goldtest_cmd.4.
test_cmd-as-list defensive branch (scripts/gen_swebench_deck.py)The list→string join branch is unexercised with swebench 4.1.0 (returns
str); kept for forward-compat. Add a unit case if a future swebench version returns a list.Assisted-By: Claude Code