Repository navigation
Conversation
`agg_supplycurve()` assigned `dfin['bin'] = []` for an empty input supply
curve, which pandas types as float64. pandas 3.0 stopped excluding empty
frames from dtype resolution in `pd.concat`, so stacking an empty offshore
curve onto a populated onshore one upcast the `bin` index level to float64.
`"wsc" + bin.astype(str)` then produced `wsc1.0` instead of `wsc1`, the
`{"wsc1": "bin1", ...}` rename matched nothing, and the downstream pivot --
which selects value columns with `startswith("bin")` -- silently dropped
every wind row.
The result was `rsc_combined.csv` containing only `cost_cap`/`cost_trans`
for `wind-ons` and no `cap`/`cost`, leaving the model with zero buildable
wind. Runs completed with no error or warning. Hit `st-AZNM` (landlocked,
so its offshore curve is header-only, but `GSw_OfsWind=1` still processes
it); coastal cases are unaffected because their offshore curves are
non-empty.
Type the empty column to the int64 that `reeds.inputs.get_bin` produces in
the non-empty branch. Fixed at the dtype source rather than at the
`.astype(str)` call: the same trap sits one line below on `class`, and
`agg_supplycurve()` also feeds the geothermal `pd.concat(geo, axis=0)`,
which is one switch change away from failing the same way. (UPV and CSP
call `agg_supplycurve()` standalone, with no populated sibling for an empty
frame to corrupt, so they cannot hit this.)
Verified against `runs/v20260924fix_st-AZNM_baseline` inputs: restores 377
`cap` and 377 `cost` rows for `wind-ons` and 920.3 GW of buildable
resource, with `cap` byte-identical to the pre-pandas-3
`v20260916v4_st-AZNM_baseline` run. UPV output unchanged.
Pre-existing upstream bug, not a regression from our pandas pin. Upstream
has the same bare `[]` at tag 2026.08.03 and on upstream/main, and has
pinned `pandas=3.0` since 2026-05-08 -- four months before this fork did --
so the bug is live there too. They have not hit it because their default
and test cases (`cendiv/Pacific`, `country/USA`) are coastal or national
and always have a populated offshore curve.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two corrections to the entries added in d2af269. Upstream is NOT on pandas 2.x. `environment.yml` on ReEDS-Model/ReEDS has pinned `pandas=3.0` since 2026-05-08 (`2ff493b5`) -- four months before this fork moved to it in `129ecdbc`. The bug is therefore live upstream right now, not latent: they satisfy the pandas and code-path conditions already and have simply not hit an empty offshore curve, because their default and test cases (`cendiv/Pacific`, `country/USA`) are coastal or national. Also confirmed not raised on their issue tracker as of 2026-09-25, having searched all 86 issues plus PR history; the nearest hits (#27, #28) are about offshore *zones*, not supply curves. Scope was stated too loosely as "any run where a supply curve passed to `agg_supplycurve()` is empty while its technology is enabled". That overstates it. The bug needs four conditions at once, and in particular needs a `pd.concat(dict, axis=0)` over sub-techs where one member is empty and another is populated. Only wind (`ons`/`ofs`) and geothermal (`geohydro`/`egs`) do that; UPV and CSP call `agg_supplycurve()` standalone and cannot hit it. Verified the switch boundary against the unfixed code on the same st-AZNM inputs: `GSw_OfsWind=1` gives 0 cap/0 cost wind rows, `GSw_OfsWind=0` gives 1640/1640. Records which regions are exposed, measured from existing runs: `st/AZ.NM` and `nercr/WECC_SW` have zero offshore rows (exposed); `st/VA`, `st/MS.AL.GA`, `transreg/SERTP` and `country/USA` do not. WECC_SW had not been re-run since the pandas upgrade, so it was queued up to hit this next. Also notes the geothermal site is latent only because current switches leave `rev_geo_types` single-element. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request documents and addresses a critical bug in the ReEDS input processing pipeline that caused all new onshore wind capacity to be silently dropped in certain cases when using pandas 3.x. The bug was triggered when a landlocked region with no offshore wind resource had offshore wind enabled, resulting in a mismatch of data types during concatenation of supply curve dataframes. The fix ensures the correct data type is assigned for empty supply curve bins, preventing the loss of buildable onshore wind resource. The changes are thoroughly documented in multiple project logs and issue trackers, and the batch results confirm the fix restores expected model behavior.
Bug fix and documentation of pandas 3.x supply-curve issue:
reeds/input_processing/writesupplycurves.pywhere, for empty supply curves, thebincolumn is now explicitly typed asint64(usingpd.Series([], dtype='int64')) instead of a bare list. This prevents pandas 3.x from upcasting integer bin labels to floats duringpd.concat, which previously caused onshore wind to be dropped from the model in landlocked regions with offshore wind enabled.CEPM/known-reeds-issues.md, including instructions for identifying affected runs and verifying the fix.CEPM/reeds-to-cepm-log.mdto log the issue and its resolution, describing the technical details and the specific code changes made to address the problem. [1] [2]Batch log and results:
CEPM/batch-log.mddocumenting the re-run of affected cases, the restoration of onshore wind capacity, and the impact of the fix. The entry also summarizes the change, its rationale, results, and next steps for merging the fix. [1] [2]These changes ensure that the ReEDS model correctly processes wind supply curves under pandas 3.x, restoring model fidelity for landlocked regions and providing clear documentation for future maintenance and upstream contribution.
results-st-AZNM_baseline,st-AZNM_limitre,st-AZNM_optimized.pptx