diff --git a/docs/translation_mappings/translation-from-plexos-to-sienna-investments.md b/docs/translation_mappings/translation-from-plexos-to-sienna-investments.md new file mode 100644 index 0000000..c9d2729 --- /dev/null +++ b/docs/translation_mappings/translation-from-plexos-to-sienna-investments.md @@ -0,0 +1,293 @@ +# Translation from PLEXOS to a Sienna investments portfolio + +This document tells you what each part of your PLEXOS expansion plan becomes in the Sienna +portfolio. It gives the source of each field. + +> **Scope:** the translator accepts electricity-only models. It translates what your plan may +> build, not when it builds it: it does not carry the investment periods, the representative +> days or their weights. It does not translate an expansion of a transmission line, and it +> writes no aggregation of candidates by category. Refer to +> [Not translated](#not-translated) and to +> [the gap analysis](plexos-to-sienna-gap-analysis.md), which states what each loss does to an +> expansion. + +One run of `plexos-to-sienna-investments` writes **two documents**. `system.json` is the base +power system: the fleet that already runs. `portfolio.json` is the expansion problem: the +technologies a plan may build, the demand they have to meet, and the caps they run under. The +portfolio names the system in its `base_system_file`, so the two are read together. + +The pipeline runs through a PyPSA network on the way. This document does not describe that +network. It states the mapping as one step, because that is what you give and what you get. +Where the intermediate form loses something, this document says so. + +--- + +## What becomes what + +| PLEXOS | Sienna investments | +| --- | --- | +| [`Generator` that may be built](#generator-that-may-be-built--supplytechnology) | `SupplyTechnology` | +| [`Battery` that may be built](#battery-or-pumped-storage-that-may-be-built--storagetechnology) | `StorageTechnology` | +| [Pumped-storage turbine that may be built](#battery-or-pumped-storage-that-may-be-built--storagetechnology) | `StorageTechnology` | +| [`Region`](#region--the-portfolios-regions) | The region each technology sits in, and a `TopologyMapping` naming its nodes | +| [Region `Load`](#region-load--demandrequirement) | `DemandRequirement` | +| [`Constraint`](#constraint--carboncaps) | `CarbonCaps`, but only where its members cover the whole model | +| [A plant that already runs](#a-plant-that-already-runs--existingdevices-and-retirementpotential) | `ExistingDevices` and `RetirementPotential` on the technology of its carrier | +| Everything else | A component of the base system. Refer to [translation-from-plexos-to-sienna.md](translation-from-plexos-to-sienna.md). | + +## Reading the tables + +| Mapping | Meaning | +| --- | --- | +| `direct` | The translator uses your PLEXOS value without a change. | +| `derived` | The translator calculates the value from your values, with the formula that the table gives. | +| `default` | PLEXOS has no such data. The translator supplies this value. | +| `dropped` | The Sienna investments schema has no equivalent. The value does not go into the portfolio. | +| `mapped by you` | Your carrier mappings file gives the value. | +| `parameter` | The run gives the value. Refer to [The base year](#the-base-year). | + +## Across all components + +- **`Max Units Built` is what makes an object a candidate.** An object stating a count above + zero may be built and becomes a technology. An object stating none may not, and its capacity + is fixed: it belongs to the base system alone. +- **One candidate becomes one technology, named after the object.** The translator does not + join a category of objects into one technology, because no PLEXOS field states how to + average unequal build costs, discount rates and lifetimes. +- **A build the plan has yet to decide is not a plant.** An object that runs no units yet is + in the portfolio and not in the base system. An object that runs some units and may build + more is in both: the base system rates it at the units it runs, and the portfolio's + `capacity_limits.min` is that same capacity, which a build cannot take away. +- **A candidate whose carrier your mappings file does not name is left out.** The run + completes and `decisions.md` names each one. +- **A generator candidate names one of four base system types.** Your mappings file must send + its carrier to `ThermalStandard`, `RenewableDispatch`, `RenewableNonDispatch` or + `HydroDispatch`, and a `ThermalStandard` row also states the `sienna_fuel_type`. A `Battery` + candidate and a pumped-storage candidate need no row of their own, because the mappings + pipeline supplies a `storage_kind` row for all three storage kinds. +- **A candidate whose carrier your mappings file sends to another kind's type is left out.** + A generator becomes a `SupplyTechnology` and a battery or a pumped-storage turbine becomes a + `StorageTechnology`, so a carrier sent to a base system type the other kind holds names a + type the technology never becomes. The run completes and `decisions.md` names each one. +- **A candidate that prices no build is already gone.** The PLEXOS to PyPSA leg leaves out a + candidate with no `Build Cost`, no `WACC` or no `Economic Life`, because PyPSA cannot + annuitise a cost without all three. `decisions.md` names each one. + +## `Generator` that may be built → `SupplyTechnology` + +| Sienna field | Unit | From | Mapping | +| --- | --- | --- | --- | +| `name` | | `Generator.name` | `direct` | +| `power_systems_type` | | The base system type your mappings file sends the generator's `Fuel` or category to | `mapped by you` | +| `prime_mover_type` | | Your mappings file | `mapped by you` | +| `fuel` | | Your mappings file, for a thermal type only. A renewable or hydro type names none. | `mapped by you` | +| `region` | | The `Region` that contains the generator's `Node` | `derived` | +| `capacity_limits.min` | MW | `Max Capacity × Units`: the capacity a build cannot take away | `derived` | +| `capacity_limits.max` | MW | `Max Capacity × (Units + Max Units Built)` | `derived` | +| `unit_size` | MW | `Max Capacity`: what one unit of the candidate is | `direct` | +| `capital_costs.capital_cost` | $/MW | `Build Cost`, as the slope of a linear cost curve | `derived` | +| `capital_costs.interconnection_cost` | $ | `0.0`. PLEXOS prices no last-mile connection separately. | `default` | +| `operation_costs.fixed` | $/MW/yr | `FO&M Charge` | `direct` | +| `operation_costs.cost_type` | | `THERMAL` where the base system type is `ThermalStandard`, `HYDRO_GEN` where it is `HydroDispatch`, `RENEWABLE` otherwise | `derived` | +| `operation_costs.start_up`, `shut_down` | $ | `0.0`, and only for a `THERMAL` cost. The other two cost representations state neither. | `default` | +| `operation_costs.variable_operation_cost` | | A zero curve. The `VO&M Charge` and the fuel price price the component in the base system, not the build. | `default` | +| `lifetime` | yr | `Technical Life`: how long a built unit runs | `direct` | +| `financial_data.capital_recovery_period` | yr | `Economic Life`: the period the build cost is recovered over | `direct` | +| `financial_data.return_on_equity` | | `WACC`. Refer to [WACC is written as all-equity financing](#wacc-is-written-as-all-equity-financing). | `derived` | +| `financial_data.debt_fraction`, `debt_rate`, `tax_rate` | | `0`, as the same rule requires | `default` | +| `financial_data.technology_base_year` | | The run's `base_year` | `parameter` | +| `available` | | `true` | `default` | +| `outage_factor` | | `Forced Outage Rate` and `Maintenance Rate` derate the component in the base system, not the technology. | `dropped` | +| `min_generation_fraction`, `ramp_limits`, `time_limits`, `start_fuel_mmbtu_per_mw` | | The base system carries the operating characteristics. | `dropped` | +| `cofire_start_limits`, `cofire_level_limits` | | PLEXOS multi-fuel blending is not translated. | `dropped` | +| `requirements` | | A `CarbonCaps` this translator writes holds the whole portfolio, so it names no members and no technology names it. | `dropped` | + +## `Battery` or pumped storage that may be built → `StorageTechnology` + +A Sienna storage technology adds charge power, discharge power and energy independently. A +PLEXOS `Battery` states one `Max Power` and one `Capacity`, and a pumped-storage plant one +`Max Capacity` and one reservoir volume, so the discharge side carries the rating and the +cost and the energy side is derived from the two. + +| Sienna field | Unit | From | Mapping | +| --- | --- | --- | --- | +| `name` | | `Battery.name`, or the turbine `Generator.name` | `direct` | +| `power_systems_type` | | The base system type your mappings file sends the storage kind to | `mapped by you` | +| `prime_mover_type` | | Your mappings file (`BA` for a `Battery`, `PS` for pumped storage, by the rows the mappings pipeline derives) | `mapped by you` | +| `storage_tech` | | `OTHER_MECH`. PLEXOS names no chemistry, and `StorageTech` has no value for an unstated one. | `default` | +| `region` | | The `Region` that contains the object's `Node` | `derived` | +| `capacity_limits_discharge.min` / `.max` | MW | `Max Power × Units` and `Max Power × (Units + Max Units Built)` | `derived` | +| `capacity_limits_energy.min` / `.max` | MWh | The discharge limits multiplied by the object's storage hours (`Capacity ÷ Max Power` for a `Battery`) | `derived` | +| | | A candidate whose storage hours come out at zero is left out, since a build could add power it can never charge. A candidate whose storage hours are not a finite number is left out too, since the energy a build may add has no upper bound. `decisions.md` names each one. | | +| `unit_size_discharge` | MW | `Max Power`: what one unit of the candidate is | `direct` | +| `capital_costs.discharge_capital_cost` | $/MW | `Build Cost`, as the slope of a linear cost curve | `derived` | +| `capital_costs.charge_capital_cost`, `energy_capital_cost` | | Zero curves. PLEXOS prices the object by its power, so it states no separate price for charging or for energy. | `default` | +| `operation_costs.fixed` | $/MW/yr | `FO&M Charge` | `direct` | +| `efficiency.in` / `.out` | | `Charge Efficiency` or `Pump Efficiency`, read as a round trip and split evenly | `derived` | +| `lifetime` | yr | `Technical Life` | `direct` | +| `financial_data` | | As for a [`SupplyTechnology`](#generator-that-may-be-built--supplytechnology) | | +| `unit_size_charge`, `unit_size_energy` | | PLEXOS sizes one unit by its power alone. | `dropped` | +| `capacity_limits_charge` | | PLEXOS builds charging and discharging together. | `dropped` | +| `duration_limits`, `losses`, `min_discharge_fraction` | | The base system carries the operating characteristics. | `dropped` | + +## `Region` → the portfolio's regions + +The portfolio does not hold its regions as components of its own: `aggregation` names the base +system type they are, which is `Area`, and each technology's `region` is a list of `Area` ids +in that system. Each `Region` of your model is one `Area` there. + +Each region also gets a `TopologyMapping`, a supplemental attribute holding the names of the +nodes in it, so a consumer can place the region's technologies on the base system's buses. + +## Region `Load` → `DemandRequirement` + +The translator writes one `DemandRequirement` for each demand of the base system, naming the +region it is drawn in. + +| Sienna field | Unit | From | Mapping | +| --- | --- | --- | --- | +| `name` | | `_load`, the name the demand has in the base system | `direct` | +| `power_systems_type` | | The base system type the demand is held as: `InterruptiblePowerLoad` where the region prices a shortfall, `PowerLoad` otherwise | `derived` | +| `region` | | The `Region` that contains the demand's node | `derived` | +| `available` | | `true` | `default` | +| `new_demand_mw`, `new_construction_year`, `growth_rate`, `conformity` | | These describe a demand a plan adds. The demand your model states is one the base system already holds, with its own profile. | `dropped` | +| `value_of_lost_load`, `unserved_demand_curve` | | The region's `VoLL` prices the load in the base system on a reliability run. | `dropped` | + +## `Constraint` → `CarbonCaps` + +`CarbonCaps` carries a target year and a limit, and **names no members and no region**: a cap +in a portfolio holds the whole portfolio. So only a `Constraint` whose members cover the whole +model becomes one. The whole model is every generator and storage object the two documents +hold: the plants in the base system and the candidates in the portfolio. An object neither +document holds emits nothing a cap could bound, so a constraint need not name it. + +| Sienna field | Unit | From | Mapping | +| --- | --- | --- | --- | +| `name` | | `Constraint.name` | `direct` | +| `max_mtons` | Mt | The `RHS Year`, or the `RHS` where the constraint states no yearly limit | `derived` | +| `available` | | `true` | `default` | +| `target_year` | | PLEXOS states the span a right-hand side applies over, not the year it applies in. | `dropped` | +| `max_tons_mwh` | | PLEXOS has no rate-based right-hand side. | `dropped` | + +Five kinds of `Constraint` are left out, each named in `decisions.md`: + +- one whose `Include in LT Plan` is false, because the expansion plan does not have to meet + it, and a cap written from it would bound a problem the model leaves free; +- one whose members cover only part of the model, because a cap written from it would hold + more than the model meant; +- one whose `Sense` is not `<=`, because a cap is a ceiling and nothing else; +- one stating neither an `RHS Year` nor an `RHS`, because no other span bounds the whole run; +- one whose right-hand side is not a finite number, because a cap states a number of + million tonnes, and neither NaN nor Infinity is one. + +Every `Constraint` still reaches `extensions.json` unchanged, whether or not it became a cap. + +## A plant that already runs → `ExistingDevices` and `RetirementPotential` + +A technology stands for more of what its own region already runs. So for each technology, the +translator writes two supplemental attributes naming the base system's components that share +both its carrier and its region: + +| Sienna field | From | Mapping | +| --- | --- | --- | +| `ExistingDevices.existing_devices` | The base system components sharing the technology's carrier and region | `derived` | +| `RetirementPotential.eligible_generators` | The same list | `derived` | +| `RetirementPotential.build_year` | The first year the object's dated `Units` rise above zero, per object | `derived` | +| `RetirementPotential.planned_retirement_year` | The first year the object's dated `Units` fall back to zero, per object | `derived` | +| `RetirementPotential.retirement_cost` | A zero curve. PLEXOS prices no retirement. | `default` | + +A technology that no such base system component matches gets neither attribute. + +## Special business rules + +### `WACC` is written as all-equity financing + +`TechnologyFinancialData` requires all six of its fields and has none for a weighted average +cost of capital. So the portfolio states `debt_fraction: 0`, `debt_rate: 0`, `tax_rate: 0` and +a `return_on_equity` equal to your `WACC`. With no debt the weighted average cost of capital +equals the return on equity, so the number your model states is the number the solver uses. +It is a derivation, not a default, and `decisions.md` records it as one. + +### A scoped `Constraint` is left out + +Refer to [`Constraint` → `CarbonCaps`](#constraint--carboncaps). Writing a cap from a +constraint over part of the model would apply that limit to the whole portfolio, which is a +different problem from the one your model states. Leaving it out and naming it is the safer +answer. + +### The base year + +`PortfolioFinancialData.base_year` and each technology's `financial_data.technology_base_year` +are the economic year a cost is quoted in. No PLEXOS field and no PyPSA field states one, so +both come from the `base_year` parameter of the `pypsa_to_sienna_investments_map_technologies` +step. It defaults to **2020**, which is the year SiennaSchemas itself defaults a construction +year to. Set it to the dollar year your `Build Cost` and `FO&M Charge` are quoted in. + +A chained pipeline prompts for the source of its first leg and the sinks of its last, and for +no step in between, so a `plexos-to-sienna-investments` run always takes the default. To state +another year, run the two legs yourself: `plexos-to-pypsa`, then `pypsa-to-sienna-investments` +over the network and the sidecar it wrote. The second leg prompts for its steps, so the base +year is the `base_year` of `step[2]`. That leg also asks for a mappings file in PyPSA words, +which the chain derives from your PLEXOS file and a lone run cannot, so write a `carriers` +file with one `pypsa_carrier` row for each carrier the first leg wrote, the storage carriers +included. + +### What the PyPSA hub cannot carry + +The route runs through a PyPSA network, and three values have no PyPSA column: + +| PLEXOS | How it crosses the hub | +| --- | --- | +| `Max Capacity` or `Max Power` as the size of one unit | The extensions sidecar, as `unit_size_mw`. PyPSA sizes a candidate by `p_nom_max` alone. | +| `Technical Life` | The extensions sidecar, as `technical_life_years`. PyPSA has one lifetime field, and the capital recovery period claims it. | +| The year a dated `Units` falls to zero | The extensions sidecar, as `retirement_year`. PyPSA carries a build year and nothing for the other end of a life. | + +The build year rides the hub as PyPSA's own `build_year`, so it survives the netCDF round trip +as a real field. A run that loses the sidecar loses the three values above, and the portfolio +then states no unit size, takes the schema's own default lifetime of 100 years, and names no +planned retirement. + +### Units and rounding + +The unit rules of the dispatch translation apply here too: refer to +[translation-from-plexos-to-sienna.md](translation-from-plexos-to-sienna.md#unit-conversions). +Two more apply to the expansion fields: + +| Value | Read in | +| --- | --- | +| `Build Cost` | $/MW, from the unit your model declares for it | +| `FO&M Charge` | $/MW/yr, from the unit your model declares for it | +| `WACC` | A fraction, read from a percentage where your model declares one | + +`max_mtons` is read in million tonnes. The translator applies **no conversion** to the +constraint's right-hand side: it writes the number your model states and records the unit your +model named it in. + +## Not translated + +| PLEXOS | Effect | +| --- | --- | +| Investment periods, and the years a plan steps through | The portfolio states one expansion problem, not a schedule. These belong on the solve request beside the model. | +| Representative days and their weights | As above. | +| An expandable `Line` or transformer | No transport technology is written, so the plan cannot build transmission. | +| `Max Units Built` on anything other than a `Generator`, a `Battery` or a pumped-storage turbine | The object's capacity is fixed. | +| A category of candidates as one technology | Each candidate is its own technology. | +| `Constraint` over part of the model | Refer to [`Constraint` → `CarbonCaps`](#constraint--carboncaps). | +| Emission objects and their prices | The carbon price reaches the base system's operating cost. No `Emission` becomes a cap. | +| Retirement costs | `RetirementPotential.retirement_cost` is zero. | + +## Assumptions worth checking against your model + +The translator reads these values in a particular way. If your model uses a different +convention, the translator gives wrong values. + +| Value | Assumed as | +| --- | --- | +| `Max Units Built` | The count of units the plan may add on top of `Units`, not the total it may reach | +| `Max Capacity`, `Max Power` | The rating of one unit, so a candidate's ceiling is that rating times the units it may hold | +| `Build Cost` | An overnight cost per MW, which the solver annuitises. It is not an annuitised cost already. | +| `Economic Life` | The period the build cost is recovered over, which is not how long the plant runs | +| `Technical Life` | How long a built unit runs | +| `WACC` | A cost of capital with no debt in it | +| A `Constraint` right-hand side | Already in million tonnes, if you read the cap as a mass of CO2 | +| A `Fuel` name or a category | One technology. Two names are two technologies, even where they are one chemistry. | diff --git a/docs/translation_mappings/translation-from-pypsa-to-sienna.md b/docs/translation_mappings/translation-from-pypsa-to-sienna.md index 230ae6c..cfb8a18 100644 --- a/docs/translation_mappings/translation-from-pypsa-to-sienna.md +++ b/docs/translation_mappings/translation-from-pypsa-to-sienna.md @@ -11,7 +11,7 @@ Concretely, versus `main` (≈280 insertions / 294 deletions): - **Time series.** Replaced the `SingleTimeSeries`/`add_time_series!` framing with SiennaSchemas `TimeSeriesAssociation` records (integer `owner_id`); value arrays are **retained in HDF5** keyed by `time_series_uuid` (the schema's named external store). Standardised on the **per-unit shape + `scaling_factor_multiplier`** convention. - **Component extensions.** PyPSA fields with no SiennaSchemas home now go in a separate `extensions.json` sidecar, a document keyed by kind whose records are identified by `name`, replacing the PowerSystems.jl `ext` dict. - **References & ordering.** `get_component(...)` / "added to the `System`" became integer-`id` references with an id-assignment ordering (the lists themselves are unordered). -- **Investments boundary.** Added that SiennaSchemas' `Investments/` namespace is a separate, parallel model (the home for capacity expansion); this document targets **Operations** (PowerSimulations.jl). +- **Investments boundary.** Added that SiennaSchemas' `Investments/` namespace is a separate, parallel model (the home for capacity expansion); this document targets **Operations** (PowerSimulations.jl), and `pypsa-to-sienna-investments` translates the expansion problem. --- @@ -79,7 +79,7 @@ This document targets the SiennaSchemas **Operations** namespace (the dispatchab - It adds policy `Requirements` (`CarbonCaps`, `CarbonTax`, `CapacityReserveMargin`, `EnergyShareRequirements`, …) and `Financials`. - It bridges to Operations through each technology's `power_systems_type` (the Operations type it realises into) and `TopologyMapping` (region → buses). -**Implication for this translation.** PyPSA's capacity-expansion fields (`p_nom_extendable`, `capital_cost`, `p_nom_min`/`p_nom_max`, `build_year`, `lifetime`) belong to **Investments**, not extra fields on an Operations component. This document is the Operations translation, so it rates each component by the capacity an operations model may dispatch: `effective_p_nom` reads `p_nom_opt`, then `p_nom_min`, then `p_nom`, as the paragraph below states. Every other expansion parameter is dropped, or kept in the `ext` sidecar for the round trip. +**Implication for this translation.** PyPSA's capacity-expansion fields (`p_nom_extendable`, `capital_cost`, `p_nom_min`/`p_nom_max`, `build_year`, `lifetime`) belong to **Investments**, not extra fields on an Operations component. This document is the Operations translation, so it rates each component by the capacity an operations model may dispatch: `effective_p_nom` reads `p_nom_opt`, then `p_nom_min`, then `p_nom`, as the paragraph below states. Every other expansion parameter is dropped, or kept in the `ext` sidecar for the round trip. The `pypsa-to-sienna-investments` pipeline translates the expansion problem itself, and writes a portfolio document beside the operations system. It is the second leg of `plexos-to-sienna-investments`, so [Translation from PLEXOS to a Sienna investments portfolio](translation-from-plexos-to-sienna-investments.md) states the source of each technology, requirement and attribute field the portfolio holds. The `From` column of that document names the PLEXOS property behind each field, not the PyPSA one. Throughout this document, **`effective_p_nom`** denotes the capacity an operations model may dispatch: `p_nom_opt` where an extendable component states one, `p_nom_min` where it states a capacity a build cannot take away, and `p_nom` otherwise. All capacity-derived fields — `base_power`, `active_power`, `active_power_limits`, `ramp_limits`, and the hydro energy budget scaling factor — use `effective_p_nom`. For a non-extendable component `effective_p_nom = p_nom`. A solve that builds none of an extendable component writes `p_nom_opt` 0, and that 0 is the capacity to translate: the plan rejected the build. A network no solve has touched leaves `p_nom_opt` out of the file altogether. An extendable component there is rated at the lower of its `p_nom_min` and its `p_nom`, because a `p_nom_min` above `p_nom` is a build the model forces rather than capacity the component already runs. An extendable component whose lower value is 0 is a candidate rather than a plant: the generator, renewable, hydro, storage and link mappings leave it out and record it as skipped. diff --git a/interop/pipelines/plexos-to-sienna-investments.yaml b/interop/pipelines/plexos-to-sienna-investments.yaml new file mode 100644 index 0000000..1352d43 --- /dev/null +++ b/interop/pipelines/plexos-to-sienna-investments.yaml @@ -0,0 +1,13 @@ +source_framework: plexos +destination_framework: sienna +mappings: + - pipeline: derive-plexos-sienna-mappings +compose: + - pipeline: plexos-to-pypsa + params: + emit_pypsa_network.output_path: network.nc + emit_extensions_json.output_path: extensions.json + - pipeline: pypsa-to-sienna-investments + params: + stage_pypsa_network_file.path: $plexos-to-pypsa.emit_pypsa_network.output_path + stage_pypsa_network_file.extensions_json_path: $plexos-to-pypsa.emit_extensions_json.output_path diff --git a/interop/pipelines/pypsa-to-sienna-investments.yaml b/interop/pipelines/pypsa-to-sienna-investments.yaml new file mode 100644 index 0000000..f412727 --- /dev/null +++ b/interop/pipelines/pypsa-to-sienna-investments.yaml @@ -0,0 +1,23 @@ +source_framework: pypsa +destination_framework: sienna +source: + name: stage_pypsa_network_file +steps: + - name: pypsa_to_sienna_map_components + - name: pypsa_to_sienna_relate_components + - name: pypsa_to_sienna_investments_map_technologies +validators: + - name: pypsa_bus_reference_integrity + - name: pypsa_generators + - name: pypsa_storage_units + - name: pypsa_loads + - name: pypsa_lines + - name: pypsa_links + - name: pypsa_unique_names +sinks: + - name: emit_sienna_files + params: + output_system_json_file_path: outputs/system.json + - name: emit_sienna_portfolio + params: + output_path: outputs/portfolio.json diff --git a/interop/plugins/shared/pypsa_sienna_investments_translations/__init__.py b/interop/plugins/shared/pypsa_sienna_investments_translations/__init__.py new file mode 100644 index 0000000..8c575d9 --- /dev/null +++ b/interop/plugins/shared/pypsa_sienna_investments_translations/__init__.py @@ -0,0 +1,71 @@ +from ._associations import build_association_rows +from ._carbon_caps import ( + CARBON_CAP_SKIPS, + build_carbon_cap_translations, + build_carbon_caps_source_table, +) +from ._demand import LOAD_TYPE_COL, build_demand_translations +from ._existing import ( + TECHNOLOGY_NAME, + TECHNOLOGY_TYPE, + CandidateTechnology, + build_existing_devices_translations, + build_existing_fleet_source_table, + build_retirement_potential_translations, +) +from ._financials import build_financial_data_events, build_portfolio_financial_data +from ._shared import ( + FOM_CHARGE_COL, + FOM_CHARGE_DERIVATION, + FUEL_COL, + POWER_SYSTEMS_TYPE_COL, + PRIME_MOVER_COL, + REGION_COL, + TECHNICAL_LIFE_COL, + UNIT_SIZE_COL, + build_scope_skips, + yearly_fixed_charge, +) +from ._storage import ( + STORAGE_SKIPS, + build_storage_technology_translations, + fill_storage_technology_defaults, +) +from ._supply import SUPPLY_SKIPS, build_supply_translations, fill_supply_defaults +from ._topology import AREA_NAME, build_topology_mapping_translations, build_topology_source_table + +__all__ = [ + "AREA_NAME", + "CARBON_CAP_SKIPS", + "FUEL_COL", + "LOAD_TYPE_COL", + "POWER_SYSTEMS_TYPE_COL", + "PRIME_MOVER_COL", + "REGION_COL", + "STORAGE_SKIPS", + "SUPPLY_SKIPS", + "FOM_CHARGE_COL", + "FOM_CHARGE_DERIVATION", + "yearly_fixed_charge", + "TECHNICAL_LIFE_COL", + "TECHNOLOGY_NAME", + "TECHNOLOGY_TYPE", + "UNIT_SIZE_COL", + "CandidateTechnology", + "build_association_rows", + "build_carbon_cap_translations", + "build_carbon_caps_source_table", + "build_demand_translations", + "build_existing_devices_translations", + "build_existing_fleet_source_table", + "build_financial_data_events", + "build_portfolio_financial_data", + "build_retirement_potential_translations", + "build_scope_skips", + "build_storage_technology_translations", + "build_supply_translations", + "build_topology_mapping_translations", + "build_topology_source_table", + "fill_storage_technology_defaults", + "fill_supply_defaults", +] diff --git a/interop/plugins/shared/pypsa_sienna_investments_translations/_associations.py b/interop/plugins/shared/pypsa_sienna_investments_translations/_associations.py new file mode 100644 index 0000000..8d6b237 --- /dev/null +++ b/interop/plugins/shared/pypsa_sienna_investments_translations/_associations.py @@ -0,0 +1,37 @@ +"""The rows linking each supplemental attribute to the component it describes. + +The attributes travel in one flat array, so the association carries the type of the +attribute as well as its id, and the type of the component as well as its name. The sink +resolves the name to an id. +""" + +from __future__ import annotations + +from collections.abc import Sequence + +import polars as pl + +from interop.plugins.shared.sienna_investments_constants import ( + SUPPLEMENTAL_ATTRIBUTE_ASSOCIATION_SCHEMA, + SiennaSupplementalAttribute, + SiennaSupplementalAttributeAssociationCol, +) + +A = SiennaSupplementalAttributeAssociationCol + + +def build_association_rows( + attribute_type: SiennaSupplementalAttribute, + attribute_ids: Sequence[int], + component_names: Sequence[str], + component_types: Sequence[str], +) -> pl.DataFrame: + return pl.DataFrame( + { + A.COMPONENT_NAME: list(component_names), + A.COMPONENT_TYPE: list(component_types), + A.ATTRIBUTE_ID: list(attribute_ids), + A.ATTRIBUTE_TYPE: [str(attribute_type)] * len(attribute_ids), + }, + schema=SUPPLEMENTAL_ATTRIBUTE_ASSOCIATION_SCHEMA, + ) diff --git a/interop/plugins/shared/pypsa_sienna_investments_translations/_carbon_caps.py b/interop/plugins/shared/pypsa_sienna_investments_translations/_carbon_caps.py new file mode 100644 index 0000000..1981cdd --- /dev/null +++ b/interop/plugins/shared/pypsa_sienna_investments_translations/_carbon_caps.py @@ -0,0 +1,227 @@ +"""Translation objects for an extensions-sidecar constraint -> Sienna CarbonCaps.""" + +from __future__ import annotations + +from collections.abc import Sequence +from functools import partial +from typing import Any + +import polars as pl + +from interop.core.extensions import ( + ConstraintExtension, + ConstraintLimit, + ConstraintPeriod, + ConstraintSense, + ExtensionKind, +) +from interop.plugins.shared.pypsa_sienna_investments_translations._shared import ( + PORTFOLIO_ID_NOTE, + investments_skip_report, +) +from interop.plugins.shared.pypsa_sienna_translations._shared import ( + pypsa_source_field, + sienna_dest_field, +) +from interop.plugins.shared.sienna_constants import SIENNA_TYPE_ATTRIBUTE +from interop.plugins.shared.sienna_investments_constants import ( + SiennaCarbonCapsCol, + SiennaInvestmentsComponent, +) +from interop.plugins.shared.translation_runner import ( + SkipRule, + Translation, + default_translation, + direct_translation, + row_position_id_translation, +) +from interop.ports.outbound.reporting import EventKind, TranslationEvent + +# Source-table columns read off a sidecar record, none of which is a schema field. +CONSTRAINT_NAME = "name" +CONSTRAINT_SENSE = "sense" +LIMIT_PERIOD = "limit_period" +LIMIT_VALUE = "limit_value" +LIMIT_UNIT = "limit_unit" +COVERS_MODEL = "covers_model" +APPLIES_TO_PLAN = "applies_to_expansion_plan" + +CARBON_CAPS_SOURCE_SCHEMA: dict[str, pl.DataType | type[pl.DataType]] = { + CONSTRAINT_NAME: pl.Utf8, + CONSTRAINT_SENSE: pl.Utf8, + LIMIT_PERIOD: pl.Utf8, + LIMIT_VALUE: pl.Float64, + LIMIT_UNIT: pl.Utf8, + COVERS_MODEL: pl.Boolean, + APPLIES_TO_PLAN: pl.Boolean, +} + +# The spans a cap is read from, most specific first: no other span bounds the whole run. +_CAP_PERIODS: tuple[ConstraintPeriod, ...] = (ConstraintPeriod.YEAR, ConstraintPeriod.HORIZON) + +_source = partial(pypsa_source_field, ExtensionKind.CONSTRAINT) +_dest = partial(sienna_dest_field, SiennaInvestmentsComponent.CARBON_CAPS) + +C = SiennaCarbonCapsCol + +_direct = partial(direct_translation, _source, _dest, name_col=CONSTRAINT_NAME) +_default = partial(default_translation, _dest, name_col=CONSTRAINT_NAME) + +_constraint_skip = partial( + investments_skip_report, + component=ExtensionKind.CONSTRAINT, + name_col=CONSTRAINT_NAME, + counted_noun="constraint(s)", +) + +SCOPED_CONSTRAINT_SKIP = _constraint_skip( + reason="weight a named subset of the model rather than all of it", + note=( + "CarbonCaps names no members and no region, so a cap written from this constraint " + "would hold the whole portfolio rather than the components the constraint names" + ), +) + +WRONG_SENSE_SKIP = _constraint_skip( + reason="hold their weighted sum to something other than a ceiling", + note="CarbonCaps states an upper limit, and this constraint does not", + attribute_col=CONSTRAINT_SENSE, +) + +NO_LIMIT_SKIP = _constraint_skip( + reason="state no yearly or horizon-wide right-hand side", + note="a cap with no limit bounds nothing, and no other span bounds the whole run", +) + +NOT_IN_PLAN_SKIP = _constraint_skip( + reason="the expansion plan does not have to meet", + note=( + "the source states that the plan need not meet this constraint, so a cap written " + "from it would bound an expansion problem the model leaves free" + ), + attribute_col=APPLIES_TO_PLAN, +) + +NON_FINITE_LIMIT_SKIP = _constraint_skip( + reason="state a right-hand side that is not a finite number", + note="max_mtons would be NaN or Infinity, which no JSON reader accepts as a number", + attribute_col=LIMIT_VALUE, +) + +CARBON_CAP_SKIPS: tuple[SkipRule, ...] = ( + # A source that states nothing about the plan leaves every constraint in it. + SkipRule(keep=pl.col(APPLIES_TO_PLAN).fill_null(value=True), report=NOT_IN_PLAN_SKIP), + SkipRule(keep=pl.col(CONSTRAINT_SENSE) == ConstraintSense.AT_MOST, report=WRONG_SENSE_SKIP), + SkipRule(keep=pl.col(LIMIT_VALUE).is_not_null(), report=NO_LIMIT_SKIP), + # is_finite answers null for a null limit, so this rule follows the one that drops those. + SkipRule(keep=pl.col(LIMIT_VALUE).is_finite(), report=NON_FINITE_LIMIT_SKIP), + SkipRule(keep=pl.col(COVERS_MODEL), report=SCOPED_CONSTRAINT_SKIP), +) + + +def build_carbon_caps_source_table( + records: Sequence[ConstraintExtension], model_components: set[tuple[str, str]] +) -> pl.DataFrame: + """One row per sidecar constraint, with the limit a cap would read and its reach. + + ``model_components`` is every component of the network a constraint could weight, each + named with the class it belongs to. A constraint naming all of them holds the whole + model; one naming fewer holds a subset. Two classes can hold an object of one name, so a + member counts only where its class matches as well, and a member whose class the network + states differently leaves the constraint short of the whole model. + """ + rows = [_read_record(record, model_components) for record in records] + return pl.DataFrame(rows, schema=CARBON_CAPS_SOURCE_SCHEMA) + + +def _read_record( + record: ConstraintExtension, model_components: set[tuple[str, str]] +) -> dict[str, Any]: + limit = _choose_limit(record) + members = {(member.name, member.member_class) for member in record.members} + return { + CONSTRAINT_NAME: record.name, + CONSTRAINT_SENSE: None if record.sense is None else str(record.sense), + LIMIT_PERIOD: None if limit is None else str(limit.period), + LIMIT_VALUE: None if limit is None else limit.value, + LIMIT_UNIT: None if limit is None else limit.unit, + COVERS_MODEL: bool(model_components) and model_components <= members, + APPLIES_TO_PLAN: record.applies_to_expansion_plan, + } + + +def _choose_limit(record: ConstraintExtension) -> ConstraintLimit | None: + for period in _CAP_PERIODS: + for limit in record.limits: + if limit.period == period: + return limit + return None + + +CARBON_CAP_NAME = _direct(source_col=CONSTRAINT_NAME, dest_col=C.NAME) + +CARBON_CAP_AVAILABLE = _default( + dest_col=C.AVAILABLE, + value=True, + note="a constraint the sidecar carries is one the model applies", +) + +CARBON_CAP_SIENNA_TYPE = Translation( + exprs=[], + make_events=lambda old, _: [ + TranslationEvent( + kind=EventKind.VALUE_DERIVED, + sources=[_source(old[CONSTRAINT_NAME], CONSTRAINT_SENSE, old[CONSTRAINT_SENSE])], + destinations=[ + _dest( + old[CONSTRAINT_NAME], + SIENNA_TYPE_ATTRIBUTE, + SiennaInvestmentsComponent.CARBON_CAPS, + ) + ], + derivation="a ceiling over every component of the model -> CarbonCaps", + ) + ], +) + +CARBON_CAP_MAX_MTONS = Translation( + exprs=[pl.col(LIMIT_VALUE).alias(C.MAX_MTONS)], + make_events=lambda old, new: [ + TranslationEvent( + kind=EventKind.VALUE_DERIVED, + sources=[ + _source( + old[CONSTRAINT_NAME], + f"{LIMIT_VALUE} ({old[LIMIT_PERIOD]})", + old[LIMIT_VALUE], + old[LIMIT_UNIT], + ) + ], + destinations=[_dest(old[CONSTRAINT_NAME], C.MAX_MTONS, new[C.MAX_MTONS])], + derivation=f"the {old[LIMIT_PERIOD]} right-hand side of the constraint", + note=( + "max_mtons is read in million tonnes; the constraint states " + f"{old[LIMIT_UNIT]}, and no conversion is applied" + if old[LIMIT_UNIT] + else "max_mtons is read in million tonnes, and the constraint names no unit" + ), + ) + ], +) + +CARBON_CAP_TRANSLATIONS: list[Translation] = [ + CARBON_CAP_NAME, + CARBON_CAP_AVAILABLE, + CARBON_CAP_SIENNA_TYPE, + CARBON_CAP_MAX_MTONS, +] + + +def build_carbon_cap_translations(start: int) -> list[Translation]: + """Every CarbonCaps translation, including the one the shared id counter decides.""" + return [ + row_position_id_translation( + _dest, dest_name_col=C.NAME, id_col=C.ID, note=PORTFOLIO_ID_NOTE, start=start + ), + *CARBON_CAP_TRANSLATIONS, + ] diff --git a/interop/plugins/shared/pypsa_sienna_investments_translations/_demand.py b/interop/plugins/shared/pypsa_sienna_investments_translations/_demand.py new file mode 100644 index 0000000..1a16056 --- /dev/null +++ b/interop/plugins/shared/pypsa_sienna_investments_translations/_demand.py @@ -0,0 +1,107 @@ +"""Translation objects for a PyPSA Load -> Sienna DemandRequirement. + +A portfolio states the demand an expansion has to meet. The load's own profile stays with +the base system, so the requirement here names the load and the region it draws in, and the +base system carries the megawatts. +""" + +from __future__ import annotations + +from functools import partial + +import polars as pl + +from interop.plugins.shared.pypsa_constants import PyPSAComponent, PyPSALoadCol +from interop.plugins.shared.pypsa_sienna_investments_translations._shared import ( + PORTFOLIO_ID_NOTE, + REGION_COL, +) +from interop.plugins.shared.pypsa_sienna_translations._shared import ( + pypsa_source_field, + sienna_dest_field, +) +from interop.plugins.shared.sienna_constants import SIENNA_TYPE_ATTRIBUTE +from interop.plugins.shared.sienna_investments_constants import ( + SiennaDemandRequirementCol, + SiennaInvestmentsComponent, +) +from interop.plugins.shared.translation_runner import ( + Translation, + default_translation, + direct_translation, + row_position_id_translation, +) +from interop.ports.outbound.reporting import EventKind, TranslationEvent + +_source = partial(pypsa_source_field, PyPSAComponent.LOAD) +_dest = partial(sienna_dest_field, SiennaInvestmentsComponent.DEMAND_REQUIREMENT) + +D = SiennaDemandRequirementCol + +# The Sienna type the base system wrote a load as, which the step enriches the table with. +LOAD_TYPE_COL = "_load_type" + +_direct = partial(direct_translation, _source, _dest, name_col=PyPSALoadCol.NAME) +_default = partial(default_translation, _dest, name_col=PyPSALoadCol.NAME) + +DEMAND_NAME = _direct(source_col=PyPSALoadCol.NAME, dest_col=D.NAME) + +DEMAND_AVAILABLE = _default( + dest_col=D.AVAILABLE, + value=True, + note="PyPSA Load has no availability field; the demand has to be met", +) + +DEMAND_SIENNA_TYPE = Translation( + exprs=[], + make_events=lambda old, _: [ + TranslationEvent( + kind=EventKind.VALUE_DERIVED, + sources=[_source(old[PyPSALoadCol.NAME], PyPSALoadCol.NAME, old[PyPSALoadCol.NAME])], + destinations=[ + _dest( + old[PyPSALoadCol.NAME], + SIENNA_TYPE_ATTRIBUTE, + SiennaInvestmentsComponent.DEMAND_REQUIREMENT, + ) + ], + derivation="each Load is one demand the expansion has to meet", + ) + ], +) + +DEMAND_POWER_SYSTEMS_TYPE = _direct( + source_col=PyPSALoadCol.NAME, + dest_col=D.POWER_SYSTEMS_TYPE, + expr=pl.col(LOAD_TYPE_COL), + derivation="the base system type the demand is served as", + note=( + "a load whose bus prices a shortfall is written as a type a solve may cut, so the " + "requirement names whichever of the two load types the base system holds it as" + ), +) + +DEMAND_REGION = _direct( + source_col=PyPSALoadCol.BUS, + dest_col=D.REGION_NAME, + expr=pl.col(REGION_COL), + derivation="the area of the bus the load sits on", +) + +DEMAND_TRANSLATIONS: list[Translation] = [ + DEMAND_NAME, + DEMAND_AVAILABLE, + DEMAND_SIENNA_TYPE, + DEMAND_POWER_SYSTEMS_TYPE, + DEMAND_REGION, +] + + +def build_demand_translations(start: int) -> list[Translation]: + """Every DemandRequirement translation, including the one the shared id counter decides.""" + return [ + row_position_id_translation( + _dest, dest_name_col=D.NAME, id_col=D.ID, note=PORTFOLIO_ID_NOTE, start=start + ), + *DEMAND_TRANSLATIONS, + ] diff --git a/interop/plugins/shared/pypsa_sienna_investments_translations/_existing.py b/interop/plugins/shared/pypsa_sienna_investments_translations/_existing.py new file mode 100644 index 0000000..c7c0d68 --- /dev/null +++ b/interop/plugins/shared/pypsa_sienna_investments_translations/_existing.py @@ -0,0 +1,239 @@ +"""The base-system fleet one technology stands for: ExistingDevices and RetirementPotential. + +Both attributes are lists of names in the base system, so they are built from the technology +table and the components the operations steps wrote, not from the PyPSA source alone. +""" + +from __future__ import annotations + +from collections.abc import Mapping, Sequence +from functools import partial +from typing import Any, NamedTuple + +import polars as pl + +from interop.plugins.shared.constants import UNIT_YEARS +from interop.plugins.shared.pypsa_constants import PyPSAGeneratorCol +from interop.plugins.shared.pypsa_sienna_translations._shared import ( + ZERO_IO_CURVE, + pypsa_source_field, + sienna_dest_field, +) +from interop.plugins.shared.sienna_investments_constants import ( + NAMED_YEAR_DTYPE, + NamedYearField, + SiennaExistingDevicesCol, + SiennaRetirementPotentialCol, + SiennaSupplementalAttribute, +) +from interop.plugins.shared.translation_runner import Translation, row_position_id_translation +from interop.ports.outbound.reporting import EventKind, TranslationEvent + +# Source-table columns, none of which is a schema field. +TECHNOLOGY_NAME = "technology_name" +TECHNOLOGY_TYPE = "technology_type" +DEVICE_CLASS = "device_class" +DEVICES = "devices" +BUILD_YEARS = "build_years" +RETIREMENT_YEARS = "retirement_years" + +EXISTING_FLEET_SOURCE_SCHEMA: dict[str, pl.DataType | type[pl.DataType]] = { + TECHNOLOGY_NAME: pl.Utf8, + TECHNOLOGY_TYPE: pl.Utf8, + DEVICE_CLASS: pl.Utf8, + DEVICES: pl.List(pl.Utf8), + BUILD_YEARS: NAMED_YEAR_DTYPE, + RETIREMENT_YEARS: NAMED_YEAR_DTYPE, +} + + +class CandidateTechnology(NamedTuple): + """One technology in the portfolio, and the fleet it stands for. + + ``carrier`` and ``region`` are what its base-system devices share, and ``device_class`` + is the PyPSA class those devices belong to, so the report names each one by the class it + came from. + """ + + name: str + component_type: str + device_class: str + carrier: str + region: str | None + + +def build_existing_fleet_source_table( + technologies: Sequence[CandidateTechnology], + devices_by_carrier_and_region: Mapping[tuple[str, str | None], Sequence[str]], + build_years: Mapping[str, int], + retirement_years: Mapping[str, int], +) -> pl.DataFrame: + """One row per technology that has a fleet in the base system already. + + A technology stands for more of what its own region already runs, so a device counts only + where it shares both the carrier and the region. A technology no such device matches has + no fleet to name, so it gets no row and neither attribute. + """ + rows: list[dict[str, Any]] = [] + for technology in technologies: + fleet = (technology.carrier, technology.region) + devices = list(devices_by_carrier_and_region.get(fleet, [])) + if not devices: + continue + rows.append( + { + TECHNOLOGY_NAME: technology.name, + TECHNOLOGY_TYPE: technology.component_type, + DEVICE_CLASS: technology.device_class, + DEVICES: devices, + BUILD_YEARS: _named_years(devices, build_years), + RETIREMENT_YEARS: _named_years(devices, retirement_years), + } + ) + return pl.DataFrame(rows, schema=EXISTING_FLEET_SOURCE_SCHEMA) + + +def _named_years(devices: Sequence[str], years: Mapping[str, int]) -> list[dict[str, Any]]: + """The year each device states, as the name/year pairs the sink writes as an object.""" + return [ + {NamedYearField.NAME: device, NamedYearField.YEAR: years[device]} + for device in devices + if device in years + ] + + +_existing_dest = partial(sienna_dest_field, SiennaSupplementalAttribute.EXISTING_DEVICES) +_retirement_dest = partial(sienna_dest_field, SiennaSupplementalAttribute.RETIREMENT_POTENTIAL) + +# The extensions sidecar field carrying the year a device leaves service. +_RETIREMENT_YEAR_FIELD = "retirement_year" + +E = SiennaExistingDevicesCol +R = SiennaRetirementPotentialCol + + +EXISTING_DEVICES_LIST = Translation( + exprs=[pl.col(DEVICES).alias(E.EXISTING_DEVICES)], + make_events=lambda old, new: [ + TranslationEvent( + kind=EventKind.VALUE_DERIVED, + sources=[pypsa_source_field(old[DEVICE_CLASS], device) for device in old[DEVICES]], + destinations=[ + _existing_dest(old[TECHNOLOGY_NAME], E.EXISTING_DEVICES, new[E.EXISTING_DEVICES]) + ], + derivation="the base system components whose carrier this technology builds more of", + ) + ], +) + + +def build_existing_devices_translations(start: int) -> list[Translation]: + """The ExistingDevices attributes, taking ids from a counter the flat array shares.""" + return [ + row_position_id_translation( + _existing_dest, + dest_name_col=TECHNOLOGY_NAME, + id_col=E.ID, + note="assigned by position in the portfolio's flat supplemental attribute array", + start=start, + ), + EXISTING_DEVICES_LIST, + ] + + +RETIREMENT_ELIGIBLE = Translation( + exprs=[pl.col(DEVICES).alias(R.ELIGIBLE_GENERATORS)], + make_events=lambda old, new: [ + TranslationEvent( + kind=EventKind.VALUE_DERIVED, + sources=[pypsa_source_field(old[DEVICE_CLASS], device) for device in old[DEVICES]], + destinations=[ + _retirement_dest( + old[TECHNOLOGY_NAME], R.ELIGIBLE_GENERATORS, new[R.ELIGIBLE_GENERATORS] + ) + ], + derivation="every base system component this technology stands for may retire", + ) + ], +) + +RETIREMENT_BUILD_YEAR = Translation( + exprs=[pl.col(BUILD_YEARS).alias(R.BUILD_YEAR)], + make_events=lambda old, new: [ + TranslationEvent( + kind=EventKind.VALUE_DERIVED, + sources=[ + pypsa_source_field( + old[DEVICE_CLASS], + entry[NamedYearField.NAME], + PyPSAGeneratorCol.BUILD_YEAR, + entry[NamedYearField.YEAR], + UNIT_YEARS, + ) + for entry in old[BUILD_YEARS] + ], + destinations=[_retirement_dest(old[TECHNOLOGY_NAME], R.BUILD_YEAR, new[R.BUILD_YEAR])], + derivation="build_year, for each device that states one", + ) + ], +) + +RETIREMENT_PLANNED_YEAR = Translation( + exprs=[pl.col(RETIREMENT_YEARS).alias(R.PLANNED_RETIREMENT_YEAR)], + make_events=lambda old, new: [ + TranslationEvent( + kind=EventKind.VALUE_DERIVED, + sources=[ + pypsa_source_field( + old[DEVICE_CLASS], + entry[NamedYearField.NAME], + _RETIREMENT_YEAR_FIELD, + entry[NamedYearField.YEAR], + UNIT_YEARS, + ) + for entry in old[RETIREMENT_YEARS] + ], + destinations=[ + _retirement_dest( + old[TECHNOLOGY_NAME], + R.PLANNED_RETIREMENT_YEAR, + new[R.PLANNED_RETIREMENT_YEAR], + ) + ], + derivation="the retirement year the extensions sidecar carries, per device", + ) + ], +) + +RETIREMENT_COST = Translation( + exprs=[ZERO_IO_CURVE.alias(R.RETIREMENT_COST)], + make_events=lambda old, new: [ + TranslationEvent( + kind=EventKind.TRANSLATOR_DEFAULT_APPLIED, + destinations=[ + _retirement_dest(old[TECHNOLOGY_NAME], R.RETIREMENT_COST, new[R.RETIREMENT_COST]) + ], + note=( + "RetirementPotential requires a retirement cost and PyPSA prices no " + "retirement, so retiring a device costs nothing" + ), + ) + ], +) + + +def build_retirement_potential_translations(start: int) -> list[Translation]: + """The RetirementPotential attributes, taking ids from the same counter.""" + return [ + row_position_id_translation( + _retirement_dest, + dest_name_col=TECHNOLOGY_NAME, + id_col=R.ID, + note="assigned by position in the portfolio's flat supplemental attribute array", + start=start, + ), + RETIREMENT_ELIGIBLE, + RETIREMENT_BUILD_YEAR, + RETIREMENT_PLANNED_YEAR, + RETIREMENT_COST, + ] diff --git a/interop/plugins/shared/pypsa_sienna_investments_translations/_financials.py b/interop/plugins/shared/pypsa_sienna_investments_translations/_financials.py new file mode 100644 index 0000000..fe18ad2 --- /dev/null +++ b/interop/plugins/shared/pypsa_sienna_investments_translations/_financials.py @@ -0,0 +1,75 @@ +"""The portfolio's own financial data: the year every cost is discounted to. + +``PortfolioFinancialData`` requires a base year and three rates. No PyPSA field states any of +them: the rate a source does state belongs to one component, and this translation writes it +into that technology's own financial data instead. So the document-level record carries the +base year the run is given and the schema's own rates. +""" + +from __future__ import annotations + +import polars as pl + +from interop.plugins.shared.constants import Framework +from interop.plugins.shared.sienna_investments_constants import ( + DEFAULT_PORTFOLIO_RATE, + PORTFOLIO_FINANCIAL_DATA_DESTINATION_SCHEMA, + PORTFOLIO_FINANCIAL_DATA_TABLE, + SiennaPortfolioFinancialDataCol, +) +from interop.ports.outbound.reporting import DestinationField, EventKind, TranslationEvent + +F = SiennaPortfolioFinancialDataCol + +# The record identifies the portfolio itself rather than a component, so the report names it +# after the field it is. +_RECORD_NAME = "financial_data" + + +def build_portfolio_financial_data(base_year: int) -> pl.DataFrame: + return pl.DataFrame( + { + F.ID: [1], + F.DISCOUNT_RATE: [DEFAULT_PORTFOLIO_RATE], + F.INFLATION_RATE: [DEFAULT_PORTFOLIO_RATE], + F.INTEREST_RATE: [DEFAULT_PORTFOLIO_RATE], + F.BASE_YEAR: [base_year], + }, + schema=PORTFOLIO_FINANCIAL_DATA_DESTINATION_SCHEMA, + ) + + +def build_financial_data_events(base_year: int) -> list[TranslationEvent]: + """What the report carries for a record no source states a field of.""" + return [ + TranslationEvent( + kind=EventKind.TRANSLATOR_DEFAULT_APPLIED, + destinations=[_field(F.BASE_YEAR, base_year)], + note=( + "no PyPSA field states the year a cost is quoted in; taken from the step's " + "base_year parameter" + ), + ), + TranslationEvent( + kind=EventKind.TRANSLATOR_DEFAULT_APPLIED, + destinations=[ + _field(F.DISCOUNT_RATE, DEFAULT_PORTFOLIO_RATE), + _field(F.INFLATION_RATE, DEFAULT_PORTFOLIO_RATE), + _field(F.INTEREST_RATE, DEFAULT_PORTFOLIO_RATE), + ], + note=( + "PyPSA states a discount rate per component, which each technology's own " + "financial data carries, so the portfolio states no rate of its own" + ), + ), + ] + + +def _field(attribute: str, value: object) -> DestinationField: + return DestinationField( + framework=Framework.SIENNA, + component=PORTFOLIO_FINANCIAL_DATA_TABLE, + name=_RECORD_NAME, + attribute=attribute, + value=value, + ) diff --git a/interop/plugins/shared/pypsa_sienna_investments_translations/_shared.py b/interop/plugins/shared/pypsa_sienna_investments_translations/_shared.py new file mode 100644 index 0000000..2bf873e --- /dev/null +++ b/interop/plugins/shared/pypsa_sienna_investments_translations/_shared.py @@ -0,0 +1,345 @@ +"""Primitives every PyPSA -> Sienna investments translation module shares.""" + +from __future__ import annotations + +from collections.abc import Sequence +from functools import partial +from typing import Any + +import polars as pl + +from interop.plugins.shared.constants import UNIT_YEARS, Framework +from interop.plugins.shared.pypsa_constants import PyPSAComponentNaming +from interop.plugins.shared.sienna_constants import MinMaxField +from interop.plugins.shared.sienna_investments_constants import ( + ALL_EQUITY_DEBT_FRACTION, + ALL_EQUITY_DEBT_RATE, + ALL_EQUITY_TAX_RATE, + CAPACITY_LIMITS_DTYPE, + TECHNOLOGY_FINANCIAL_DATA_DTYPE, + SiennaTechnologyFinancialDataField, +) +from interop.plugins.shared.translation_runner import ( + DestinationFieldFactory, + SkippedNames, + SkipReport, + SkipRule, + SourceFieldFactory, + Translation, +) +from interop.ports.outbound.reporting import EventKind, TranslationEvent + +PYPSA_TO_SIENNA_INVESTMENTS = "pypsa-to-sienna-investments" + +# Enrichment columns the step adds to a source table, which finalise() drops. +REGION_COL = "_region_name" +POWER_SYSTEMS_TYPE_COL = "_power_systems_type" +PRIME_MOVER_COL = "_prime_mover" +FUEL_COL = "_fuel" +UNIT_SIZE_COL = "_unit_size_mw" +TECHNICAL_LIFE_COL = "_technical_life_years" +FOM_CHARGE_COL = "_fom_charge_per_mw_year" + +FOM_CHARGE_DERIVATION = ( + "the yearly fixed charge, from the extensions sidecar where a PLEXOS model stated one and " + "from fom_cost otherwise, as the fixed term of the operation cost" +) + + +def yearly_fixed_charge(fom_cost_col: str) -> pl.Expr: + """What one MW costs to keep for a year. + + A network this translator wrote carries the charge in the sidecar, because PyPSA reads + ``fom_cost`` as a charge for the whole modelled horizon rather than a yearly one. A + network a user wrote states no sidecar record, so ``fom_cost`` is all there is. + """ + return pl.coalesce(pl.col(FOM_CHARGE_COL), pl.col(fom_cost_col), pl.lit(0.0)) + + +PORTFOLIO_ID_NOTE = "assigned by position in the portfolio's components, which share one counter" + +investments_skip_report = partial( + SkipReport, pipeline=PYPSA_TO_SIENNA_INVESTMENTS, framework=Framework.PYPSA +) + +UNNAMED_CARRIER_REASON = "may be built and have a carrier the user mappings file does not name" +UNNAMED_CARRIER_NOTE = ( + "the portfolio reads a technology's Sienna type, prime mover and fuel off the carrier, " + "and the user mappings file names no such carrier" +) +UNSUPPORTED_TARGET_REASON = ( + "may be built and have a carrier the mappings file sends to a Sienna type this kind of " + "candidate never becomes" +) +UNSUPPORTED_TARGET_NOTE = ( + "power_systems_type names the base system type a build becomes, and the user mappings " + "file sends this carrier to a type this candidate never holds" +) +NOT_AN_ELECTRICITY_BUS_REASON = "may be built and sit on a bus that is not an electricity bus" +NOT_AN_ELECTRICITY_BUS_NOTE = "bus is not an electricity (AC) bus, so it is in no region" + + +def build_scope_skips( + naming: PyPSAComponentNaming, + *, + name_col: str, + carrier_col: str, + bus_col: str, + carriers: Sequence[str], + translated_carriers: Sequence[str], + bus_names: Sequence[str], +) -> list[SkipRule]: + """The drops every candidate table shares, in the order they apply. + + Order matters: a row the mappings file never names must not also report an unusable + target type or an unusable bus. + """ + skip = partial( + investments_skip_report, + component=naming.display, + name_col=name_col, + counted_noun=naming.plural, + ) + listed = SkippedNames(column=carrier_col, label="The carriers") + return [ + SkipRule( + keep=pl.col(carrier_col).is_in(list(carriers)), + report=skip( + reason=UNNAMED_CARRIER_REASON, + note=UNNAMED_CARRIER_NOTE, + listed=listed, + ), + ), + SkipRule( + keep=pl.col(carrier_col).is_in(list(translated_carriers)), + report=skip( + reason=UNSUPPORTED_TARGET_REASON, + note=UNSUPPORTED_TARGET_NOTE, + listed=listed, + ), + ), + SkipRule( + keep=pl.col(bus_col).is_in(list(bus_names)), + report=skip(reason=NOT_AN_ELECTRICITY_BUS_REASON, note=NOT_AN_ELECTRICITY_BUS_NOTE), + ), + ] + + +UNBOUNDED_BUILD_REASON = "are extendable and put no upper bound on the capacity a build may add" +UNBOUNDED_BUILD_NOTE = ( + "p_nom_max is not a finite number of MW, so the technology has no maximum installed " + "capacity to state" +) +UNBOUNDED_LIFETIME_REASON = "are extendable and state no finite lifetime" +UNBOUNDED_LIFETIME_NOTE = ( + "lifetime is not a finite number of years, so the technology has no capital recovery " + "period to annuitise its overnight cost across" +) +SHORT_LIFETIME_REASON = "are extendable and state a lifetime below one year" +SHORT_LIFETIME_NOTE = ( + "lifetime is below one year, so the capital recovery period truncates to zero or less " + "and no annuity can recover the overnight cost across it" +) +UNPRICED_BUILD_REASON = "are extendable and put no overnight cost on the capacity a build adds" +UNPRICED_BUILD_NOTE = ( + "PyPSA prices a build through overnight_cost or through the annuity in capital_cost, and " + "an annuity cannot be undone into the two terms a capital cost curve states, so the " + "technology has no price to build at" +) +CAPACITY_FLOOR_ABOVE_CEILING_REASON = ( + "are extendable and state a capacity floor above the capacity a build may reach" +) +CAPACITY_FLOOR_ABOVE_CEILING_NOTE = ( + "p_nom_min is above p_nom_max, so capacity_limits.min would be above capacity_limits.max " + "and no capacity could meet the technology's own bounds" +) +NO_DISCOUNT_RATE_REASON = "are extendable and state no discount rate" +NO_DISCOUNT_RATE_NOTE = ( + "TechnologyFinancialData requires a return on equity, and the discount rate is the only " + "cost of capital PyPSA states" +) + + +def build_expansion_skips( + naming: PyPSAComponentNaming, + *, + name_col: str, + build_limit_col: str, + capacity_floor_col: str, + lifetime_col: str, + overnight_cost_col: str, + discount_rate_col: str, +) -> tuple[SkipRule, ...]: + skip = partial( + investments_skip_report, + component=naming.display, + name_col=name_col, + counted_noun=naming.plural, + ) + return ( + SkipRule( + keep=pl.col(build_limit_col).is_finite(), + report=skip( + reason=UNBOUNDED_BUILD_REASON, + note=UNBOUNDED_BUILD_NOTE, + attribute_col=build_limit_col, + ), + ), + SkipRule( + keep=pl.col(capacity_floor_col) <= pl.col(build_limit_col), + report=skip( + reason=CAPACITY_FLOOR_ABOVE_CEILING_REASON, + note=CAPACITY_FLOOR_ABOVE_CEILING_NOTE, + attribute_col=capacity_floor_col, + ), + ), + SkipRule( + keep=pl.col(lifetime_col).is_finite(), + report=skip( + reason=UNBOUNDED_LIFETIME_REASON, + note=UNBOUNDED_LIFETIME_NOTE, + attribute_col=lifetime_col, + ), + ), + SkipRule( + keep=pl.col(lifetime_col) >= 1, + report=skip( + reason=SHORT_LIFETIME_REASON, + note=SHORT_LIFETIME_NOTE, + attribute_col=lifetime_col, + ), + ), + SkipRule( + keep=pl.col(overnight_cost_col).is_not_null(), + report=skip( + reason=UNPRICED_BUILD_REASON, + note=UNPRICED_BUILD_NOTE, + attribute_col=overnight_cost_col, + ), + ), + SkipRule( + keep=pl.col(discount_rate_col).is_not_null(), + report=skip( + reason=NO_DISCOUNT_RATE_REASON, + note=NO_DISCOUNT_RATE_NOTE, + attribute_col=discount_rate_col, + ), + ), + ) + + +WACC_DERIVATION = ( + "discount_rate as an all-equity cost of capital: with no debt the weighted average " + "cost of capital equals the return on equity" +) +ALL_EQUITY_NOTE = ( + "TechnologyFinancialData has no field for a weighted average cost of capital and " + "requires all six of its own, so the financing is written as all equity and the " + "return on equity carries the rate the source states" +) + + +def finite_or_null(column: pl.Expr) -> pl.Expr: + """A sidecar is JSON, and json.load reads the NaN and Infinity tokens, so a sidecar number + is not always finite. A cast to a whole number raises on one, and the sink writes a token + no strict JSON reader accepts. + """ + return pl.when(column.is_finite()).then(column) + + +def capacity_limits_struct(minimum: pl.Expr, maximum: pl.Expr) -> pl.Expr: + return pl.struct( + minimum.cast(pl.Float64).alias(MinMaxField.MIN), + maximum.cast(pl.Float64).alias(MinMaxField.MAX), + ).cast(CAPACITY_LIMITS_DTYPE) + + +def financial_data_struct( + capital_recovery_period: pl.Expr, return_on_equity: pl.Expr, base_year: int +) -> pl.Expr: + field = SiennaTechnologyFinancialDataField + return pl.struct( + capital_recovery_period.cast(pl.Int64).alias(field.CAPITAL_RECOVERY_PERIOD), + pl.lit(base_year, dtype=pl.Int64).alias(field.TECHNOLOGY_BASE_YEAR), + pl.lit(ALL_EQUITY_DEBT_FRACTION).alias(field.DEBT_FRACTION), + pl.lit(ALL_EQUITY_DEBT_RATE).alias(field.DEBT_RATE), + return_on_equity.cast(pl.Float64).alias(field.RETURN_ON_EQUITY), + pl.lit(ALL_EQUITY_TAX_RATE).alias(field.TAX_RATE), + ).cast(TECHNOLOGY_FINANCIAL_DATA_DTYPE) + + +def build_financial_data_translation( + source_field: SourceFieldFactory, + dest_field: DestinationFieldFactory, + *, + name_col: str, + lifetime_col: str, + discount_rate_col: str, + dest_col: str, + base_year: int, +) -> Translation: + """The six fields ``TechnologyFinancialData`` requires, from the two PyPSA states. + + Two of the six come from the network: the lifetime PyPSA annuitises across becomes the + capital recovery period, and the discount rate becomes the return on equity. The other + four are the translator's own, and each gets its own event. + """ + + def make_events(old: dict[str, Any], new: dict[str, Any]) -> Sequence[TranslationEvent]: + name = old[name_col] + written = new[dest_col] + field = SiennaTechnologyFinancialDataField + return [ + TranslationEvent( + kind=EventKind.VALUE_DERIVED, + sources=[source_field(name, lifetime_col, old[lifetime_col], UNIT_YEARS)], + destinations=[ + dest_field( + name, + f"{dest_col}.{field.CAPITAL_RECOVERY_PERIOD}", + written[field.CAPITAL_RECOVERY_PERIOD], + UNIT_YEARS, + ) + ], + derivation="the period PyPSA annuitises an overnight cost across", + ), + TranslationEvent( + kind=EventKind.VALUE_DERIVED, + sources=[source_field(name, discount_rate_col, old[discount_rate_col])], + destinations=[ + dest_field( + name, + f"{dest_col}.{field.RETURN_ON_EQUITY}", + written[field.RETURN_ON_EQUITY], + ) + ], + derivation=WACC_DERIVATION, + note=ALL_EQUITY_NOTE, + ), + TranslationEvent( + kind=EventKind.TRANSLATOR_DEFAULT_APPLIED, + destinations=[ + dest_field(name, f"{dest_col}.{field.DEBT_FRACTION}", ALL_EQUITY_DEBT_FRACTION), + dest_field(name, f"{dest_col}.{field.DEBT_RATE}", ALL_EQUITY_DEBT_RATE), + dest_field(name, f"{dest_col}.{field.TAX_RATE}", ALL_EQUITY_TAX_RATE), + ], + note=ALL_EQUITY_NOTE, + ), + TranslationEvent( + kind=EventKind.TRANSLATOR_DEFAULT_APPLIED, + destinations=[ + dest_field(name, f"{dest_col}.{field.TECHNOLOGY_BASE_YEAR}", base_year) + ], + note="no PyPSA field states the year a cost is quoted in; taken from the " + "step's base_year parameter", + ), + ] + + return Translation( + exprs=[ + financial_data_struct(pl.col(lifetime_col), pl.col(discount_rate_col), base_year).alias( + dest_col + ) + ], + make_events=make_events, + ) diff --git a/interop/plugins/shared/pypsa_sienna_investments_translations/_storage.py b/interop/plugins/shared/pypsa_sienna_investments_translations/_storage.py new file mode 100644 index 0000000..a4e9a1a --- /dev/null +++ b/interop/plugins/shared/pypsa_sienna_investments_translations/_storage.py @@ -0,0 +1,383 @@ +"""Translation objects for an extendable PyPSA StorageUnit -> Sienna StorageTechnology. + +A Sienna storage technology adds charge power, discharge power and energy independently, so +each has its own capital cost, unit size and capacity limits. A PyPSA StorageUnit states one +power rating and an energy stated as hours of it, so the discharge side carries the rating +and the cost, and the energy side carries the rating multiplied by the hours. +""" + +from __future__ import annotations + +from functools import partial + +import polars as pl + +from interop.plugins.shared.constants import ( + UNIT_DOLLARS_PER_MW, + UNIT_DOLLARS_PER_MW_YEAR, + UNIT_HOURS, + UNIT_MW, + UNIT_MWH, + UNIT_YEARS, +) +from interop.plugins.shared.pypsa_constants import ( + PYPSA_COMPONENT_NAMING, + PyPSAComponent, + PyPSAStorageUnitCol, + PyPSATable, +) +from interop.plugins.shared.pypsa_sienna_investments_translations._shared import ( + FOM_CHARGE_COL, + FOM_CHARGE_DERIVATION, + PORTFOLIO_ID_NOTE, + POWER_SYSTEMS_TYPE_COL, + PRIME_MOVER_COL, + REGION_COL, + TECHNICAL_LIFE_COL, + UNIT_SIZE_COL, + build_expansion_skips, + build_financial_data_translation, + capacity_limits_struct, + finite_or_null, + investments_skip_report, + yearly_fixed_charge, +) +from interop.plugins.shared.pypsa_sienna_translations._shared import ( + ZERO_IO_CURVE, + linear_value_curve, + pypsa_source_field, + sienna_dest_field, +) +from interop.plugins.shared.sienna_constants import ( + EFFICIENCY_DTYPE, + PRIME_MOVERS_DTYPE, + SIENNA_TYPE_ATTRIBUTE, + SiennaCostType, + SiennaStorageTech, + SiennaStructField, +) +from interop.plugins.shared.sienna_investments_constants import ( + STORAGE_CAPITAL_COST_DTYPE, + STORAGE_OPERATION_COST_DTYPE, + SiennaInvestmentsComponent, + SiennaOperationCostField, + SiennaStorageCapitalCostField, + SiennaStorageTechnologyCol, +) +from interop.plugins.shared.translation_runner import ( + SkipRule, + Translation, + default_translation, + direct_translation, + fill_defaults, + row_position_id_translation, +) +from interop.ports.outbound.reporting import EventKind, TranslationEvent + +_source = partial(pypsa_source_field, PyPSAComponent.STORAGE_UNIT) +_dest = partial(sienna_dest_field, SiennaInvestmentsComponent.STORAGE_TECHNOLOGY) + +S = SiennaStorageTechnologyCol + +_direct = partial(direct_translation, _source, _dest, name_col=PyPSAStorageUnitCol.NAME) +_default = partial(default_translation, _dest, name_col=PyPSAStorageUnitCol.NAME) + +NO_ENERGY_SKIP = investments_skip_report( + component=PyPSAComponent.STORAGE_UNIT, + name_col=PyPSAStorageUnitCol.NAME, + counted_noun=PYPSA_COMPONENT_NAMING[PyPSATable.STORAGE_UNITS].plural, + reason="are extendable and hold no energy", + note=lambda row: ( + f"max_hours is {row[PyPSAStorageUnitCol.MAX_HOURS]}, so the energy capacity limits a " + "build could add are zero MWh" + ), + attribute_col=PyPSAStorageUnitCol.MAX_HOURS, +) + +UNBOUNDED_ENERGY_SKIP = investments_skip_report( + component=PyPSAComponent.STORAGE_UNIT, + name_col=PyPSAStorageUnitCol.NAME, + counted_noun=PYPSA_COMPONENT_NAMING[PyPSATable.STORAGE_UNITS].plural, + reason="are extendable and put no upper bound on the energy a build may add", + note=( + "max_hours is not a finite number of hours, so the technology has no energy " + "capacity limits to state" + ), + attribute_col=PyPSAStorageUnitCol.MAX_HOURS, +) + +STORAGE_SKIPS: tuple[SkipRule, ...] = ( + *build_expansion_skips( + PYPSA_COMPONENT_NAMING[PyPSATable.STORAGE_UNITS], + name_col=PyPSAStorageUnitCol.NAME, + build_limit_col=PyPSAStorageUnitCol.P_NOM_MAX, + capacity_floor_col=PyPSAStorageUnitCol.P_NOM_MIN, + lifetime_col=PyPSAStorageUnitCol.LIFETIME, + overnight_cost_col=PyPSAStorageUnitCol.OVERNIGHT_COST, + discount_rate_col=PyPSAStorageUnitCol.DISCOUNT_RATE, + ), + SkipRule( + keep=pl.col(PyPSAStorageUnitCol.MAX_HOURS).is_finite(), + report=UNBOUNDED_ENERGY_SKIP, + ), + SkipRule(keep=pl.col(PyPSAStorageUnitCol.MAX_HOURS) > 0, report=NO_ENERGY_SKIP), +) + + +def fill_storage_technology_defaults(table: pl.DataFrame) -> pl.DataFrame: + """Add the expansion columns PyPSA omits when every storage unit shares its default. + + PyPSA gives ``max_hours`` a default of one hour of the power rating, and defaults an + overnight cost and a discount rate to NaN rather than to a number, so both of the latter + stay null here and a unit stating neither is dropped rather than built free. + """ + return fill_defaults( + table, + [ + (PyPSAStorageUnitCol.P_NOM_MIN, 0.0), + (PyPSAStorageUnitCol.P_NOM_MAX, float("inf")), + (PyPSAStorageUnitCol.MAX_HOURS, 1.0), + (PyPSAStorageUnitCol.EFFICIENCY_STORE, 1.0), + (PyPSAStorageUnitCol.EFFICIENCY_DISPATCH, 1.0), + (PyPSAStorageUnitCol.OVERNIGHT_COST, None), + (PyPSAStorageUnitCol.DISCOUNT_RATE, None), + (PyPSAStorageUnitCol.LIFETIME, float("inf")), + (PyPSAStorageUnitCol.FOM_COST, 0.0), + ], + [(PyPSAStorageUnitCol.P_NOM_EXTENDABLE, False)], + ) + + +def storage_capital_cost_struct(overnight_cost: pl.Expr) -> pl.Expr: + """A Sienna ``StorageCapitalCost`` pricing the discharge side alone.""" + return pl.struct( + ZERO_IO_CURVE.alias(SiennaStorageCapitalCostField.CHARGE_CAPITAL_COST), + linear_value_curve(overnight_cost, input_at_zero=pl.lit(None, dtype=pl.Float64)).alias( + SiennaStorageCapitalCostField.DISCHARGE_CAPITAL_COST + ), + ZERO_IO_CURVE.alias(SiennaStorageCapitalCostField.ENERGY_CAPITAL_COST), + pl.lit(0.0).alias(SiennaStorageCapitalCostField.INTERCONNECTION_COST), + ).cast(STORAGE_CAPITAL_COST_DTYPE) + + +STORAGE_NAME = _direct(source_col=PyPSAStorageUnitCol.NAME, dest_col=S.NAME) + +STORAGE_AVAILABLE = _default( + dest_col=S.AVAILABLE, + value=True, + note="PyPSA states no availability for a candidate; the technology may be built", +) + +STORAGE_SIENNA_TYPE = Translation( + exprs=[], + make_events=lambda old, _: [ + TranslationEvent( + kind=EventKind.VALUE_DERIVED, + sources=[ + _source( + old[PyPSAStorageUnitCol.NAME], + PyPSAStorageUnitCol.P_NOM_EXTENDABLE, + old[PyPSAStorageUnitCol.P_NOM_EXTENDABLE], + ) + ], + destinations=[ + _dest( + old[PyPSAStorageUnitCol.NAME], + SIENNA_TYPE_ATTRIBUTE, + SiennaInvestmentsComponent.STORAGE_TECHNOLOGY, + ) + ], + derivation="an extendable StorageUnit is a candidate, and each becomes one technology", + ) + ], +) + +STORAGE_POWER_SYSTEMS_TYPE = _direct( + source_col=PyPSAStorageUnitCol.CARRIER, + dest_col=S.POWER_SYSTEMS_TYPE, + expr=pl.col(POWER_SYSTEMS_TYPE_COL), + derivation="carrier -> the base system type a build becomes, via the user mappings file", +) + +STORAGE_REGION = _direct( + source_col=PyPSAStorageUnitCol.BUS, + dest_col=S.REGION_NAME, + expr=pl.col(REGION_COL), + derivation="the area of the bus the storage unit sits on", +) + +STORAGE_PRIME_MOVER = _direct( + source_col=PyPSAStorageUnitCol.CARRIER, + dest_col=S.PRIME_MOVER_TYPE, + expr=pl.col(PRIME_MOVER_COL).cast(PRIME_MOVERS_DTYPE), + derivation="carrier -> PrimeMovers via the user mappings file", +) + +STORAGE_TECH = _default( + dest_col=S.STORAGE_TECH, + value=SiennaStorageTech.OTHER_MECH, + note="PyPSA names no storage chemistry, and StorageTech has no value for an unstated one", +) + +STORAGE_CAPITAL_COSTS = _direct( + source_col=PyPSAStorageUnitCol.OVERNIGHT_COST, + dest_col=S.CAPITAL_COSTS, + expr=storage_capital_cost_struct(pl.col(PyPSAStorageUnitCol.OVERNIGHT_COST)), + unit=UNIT_DOLLARS_PER_MW, + derivation="overnight_cost as the discharge capital cost; charge and energy cost nothing", +) + +STORAGE_OPERATION_COSTS = _direct( + source_col=FOM_CHARGE_COL, + dest_col=S.OPERATION_COSTS, + expr=pl.struct( + pl.lit(SiennaCostType.STORAGE).alias(SiennaOperationCostField.COST_TYPE), + yearly_fixed_charge(PyPSAStorageUnitCol.FOM_COST).alias(SiennaOperationCostField.FIXED), + pl.lit(0.0).alias(SiennaOperationCostField.START_UP), + pl.lit(0.0).alias(SiennaOperationCostField.SHUT_DOWN), + ).cast(STORAGE_OPERATION_COST_DTYPE), + unit=UNIT_DOLLARS_PER_MW_YEAR, + derivation=FOM_CHARGE_DERIVATION, +) + +STORAGE_UNIT_SIZE_DISCHARGE = _direct( + source_col=UNIT_SIZE_COL, + dest_col=S.UNIT_SIZE_DISCHARGE, + expr=finite_or_null(pl.col(UNIT_SIZE_COL)), + unit=UNIT_MW, + derivation="the size of one unit, from the extensions sidecar", +) + +STORAGE_CAPACITY_LIMITS_DISCHARGE = Translation( + exprs=[ + capacity_limits_struct( + pl.col(PyPSAStorageUnitCol.P_NOM_MIN), pl.col(PyPSAStorageUnitCol.P_NOM_MAX) + ).alias(S.CAPACITY_LIMITS_DISCHARGE) + ], + make_events=lambda old, new: [ + TranslationEvent( + kind=EventKind.VALUE_DERIVED, + sources=[ + _source( + old[PyPSAStorageUnitCol.NAME], + PyPSAStorageUnitCol.P_NOM_MIN, + old[PyPSAStorageUnitCol.P_NOM_MIN], + UNIT_MW, + ), + _source( + old[PyPSAStorageUnitCol.NAME], + PyPSAStorageUnitCol.P_NOM_MAX, + old[PyPSAStorageUnitCol.P_NOM_MAX], + UNIT_MW, + ), + ], + destinations=[ + _dest( + old[PyPSAStorageUnitCol.NAME], + S.CAPACITY_LIMITS_DISCHARGE, + new[S.CAPACITY_LIMITS_DISCHARGE], + UNIT_MW, + ) + ], + derivation="p_nom_min, p_nom_max -> capacity_limits_discharge.{min, max}", + ) + ], +) + +STORAGE_CAPACITY_LIMITS_ENERGY = Translation( + exprs=[ + capacity_limits_struct( + pl.col(PyPSAStorageUnitCol.P_NOM_MIN) * pl.col(PyPSAStorageUnitCol.MAX_HOURS), + pl.col(PyPSAStorageUnitCol.P_NOM_MAX) * pl.col(PyPSAStorageUnitCol.MAX_HOURS), + ).alias(S.CAPACITY_LIMITS_ENERGY) + ], + make_events=lambda old, new: [ + TranslationEvent( + kind=EventKind.VALUE_DERIVED, + sources=[ + _source( + old[PyPSAStorageUnitCol.NAME], + PyPSAStorageUnitCol.MAX_HOURS, + old[PyPSAStorageUnitCol.MAX_HOURS], + UNIT_HOURS, + ), + _source( + old[PyPSAStorageUnitCol.NAME], + PyPSAStorageUnitCol.P_NOM_MIN, + old[PyPSAStorageUnitCol.P_NOM_MIN], + UNIT_MW, + ), + _source( + old[PyPSAStorageUnitCol.NAME], + PyPSAStorageUnitCol.P_NOM_MAX, + old[PyPSAStorageUnitCol.P_NOM_MAX], + UNIT_MW, + ), + ], + destinations=[ + _dest( + old[PyPSAStorageUnitCol.NAME], + S.CAPACITY_LIMITS_ENERGY, + new[S.CAPACITY_LIMITS_ENERGY], + UNIT_MWH, + ) + ], + derivation="max_hours x the power capacity limits -> capacity_limits_energy", + ) + ], +) + +STORAGE_EFFICIENCY = _direct( + source_col=PyPSAStorageUnitCol.EFFICIENCY_STORE, + dest_col=S.EFFICIENCY, + expr=pl.struct( + pl.col(PyPSAStorageUnitCol.EFFICIENCY_STORE).alias(SiennaStructField.IN), + pl.col(PyPSAStorageUnitCol.EFFICIENCY_DISPATCH).alias(SiennaStructField.OUT), + ).cast(EFFICIENCY_DTYPE), + derivation="(in=efficiency_store, out=efficiency_dispatch)", +) + +STORAGE_LIFETIME = _direct( + source_col=TECHNICAL_LIFE_COL, + dest_col=S.LIFETIME, + expr=finite_or_null(pl.col(TECHNICAL_LIFE_COL)).cast(pl.Int64), + unit=UNIT_YEARS, + derivation="the technical life, from the extensions sidecar", + note="how long a built unit runs, which is not the period its cost is recovered over", +) + +STORAGE_TECHNOLOGY_TRANSLATIONS: list[Translation] = [ + STORAGE_NAME, + STORAGE_AVAILABLE, + STORAGE_SIENNA_TYPE, + STORAGE_POWER_SYSTEMS_TYPE, + STORAGE_REGION, + STORAGE_PRIME_MOVER, + STORAGE_TECH, + STORAGE_CAPITAL_COSTS, + STORAGE_OPERATION_COSTS, + STORAGE_UNIT_SIZE_DISCHARGE, + STORAGE_CAPACITY_LIMITS_DISCHARGE, + STORAGE_CAPACITY_LIMITS_ENERGY, + STORAGE_EFFICIENCY, + STORAGE_LIFETIME, +] + + +def build_storage_technology_translations(base_year: int, start: int) -> list[Translation]: + """Every StorageTechnology translation, including the two the caller's numbers decide.""" + return [ + row_position_id_translation( + _dest, dest_name_col=S.NAME, id_col=S.ID, note=PORTFOLIO_ID_NOTE, start=start + ), + *STORAGE_TECHNOLOGY_TRANSLATIONS, + build_financial_data_translation( + _source, + _dest, + name_col=PyPSAStorageUnitCol.NAME, + lifetime_col=PyPSAStorageUnitCol.LIFETIME, + discount_rate_col=PyPSAStorageUnitCol.DISCOUNT_RATE, + dest_col=S.FINANCIAL_DATA, + base_year=base_year, + ), + ] diff --git a/interop/plugins/shared/pypsa_sienna_investments_translations/_supply.py b/interop/plugins/shared/pypsa_sienna_investments_translations/_supply.py new file mode 100644 index 0000000..aa3d02c --- /dev/null +++ b/interop/plugins/shared/pypsa_sienna_investments_translations/_supply.py @@ -0,0 +1,324 @@ +"""Translation objects for an extendable PyPSA Generator -> Sienna SupplyTechnology. + +A generator whose capacity the network lets a plan change is a candidate, and each candidate +becomes one technology named after it. What the plan may build is the gap between +``p_nom_min`` and ``p_nom_max``; what building it costs is the overnight cost, the fixed +O&M and the discount rate and lifetime PyPSA annuitises them with. +""" + +from __future__ import annotations + +from functools import partial + +import polars as pl + +from interop.plugins.shared.constants import ( + UNIT_DOLLARS_PER_MW, + UNIT_DOLLARS_PER_MW_YEAR, + UNIT_MW, + UNIT_YEARS, +) +from interop.plugins.shared.pypsa_constants import ( + PYPSA_COMPONENT_NAMING, + PyPSAComponent, + PyPSAGeneratorCol, + PyPSATable, +) +from interop.plugins.shared.pypsa_sienna_investments_translations._shared import ( + FOM_CHARGE_COL, + FOM_CHARGE_DERIVATION, + FUEL_COL, + PORTFOLIO_ID_NOTE, + POWER_SYSTEMS_TYPE_COL, + PRIME_MOVER_COL, + REGION_COL, + TECHNICAL_LIFE_COL, + UNIT_SIZE_COL, + build_expansion_skips, + build_financial_data_translation, + capacity_limits_struct, + finite_or_null, + yearly_fixed_charge, +) +from interop.plugins.shared.pypsa_sienna_translations._shared import ( + linear_value_curve, + pypsa_source_field, + sienna_dest_field, + variable_cost_curve, +) +from interop.plugins.shared.sienna_constants import ( + PRIME_MOVERS_DTYPE, + SIENNA_TYPE_ATTRIBUTE, + SiennaComponent, + SiennaCostType, +) +from interop.plugins.shared.sienna_investments_constants import ( + CAPITAL_COST_DTYPE, + GENERIC_OPERATION_COST_DTYPE, + SiennaCapitalCostField, + SiennaInvestmentsComponent, + SiennaOperationCostField, + SiennaSupplyTechnologyCol, +) +from interop.plugins.shared.translation_runner import ( + SkipRule, + Translation, + default_translation, + direct_translation, + fill_defaults, + row_position_id_translation, +) +from interop.ports.outbound.reporting import EventKind, TranslationEvent + +_source = partial(pypsa_source_field, PyPSAComponent.GENERATOR) +_dest = partial(sienna_dest_field, SiennaInvestmentsComponent.SUPPLY_TECHNOLOGY) + +S = SiennaSupplyTechnologyCol + +_direct = partial(direct_translation, _source, _dest, name_col=PyPSAGeneratorCol.NAME) +_default = partial(default_translation, _dest, name_col=PyPSAGeneratorCol.NAME) + +SUPPLY_SKIPS: tuple[SkipRule, ...] = build_expansion_skips( + PYPSA_COMPONENT_NAMING[PyPSATable.GENERATORS], + name_col=PyPSAGeneratorCol.NAME, + build_limit_col=PyPSAGeneratorCol.P_NOM_MAX, + capacity_floor_col=PyPSAGeneratorCol.P_NOM_MIN, + lifetime_col=PyPSAGeneratorCol.LIFETIME, + overnight_cost_col=PyPSAGeneratorCol.OVERNIGHT_COST, + discount_rate_col=PyPSAGeneratorCol.DISCOUNT_RATE, +) + + +def fill_supply_defaults(table: pl.DataFrame) -> pl.DataFrame: + """Add the expansion columns PyPSA omits when every generator shares its default. + + PyPSA defaults an overnight cost and a discount rate to NaN rather than to a number, so + both stay null here and a generator stating neither is dropped rather than built free. + """ + return fill_defaults( + table, + [ + (PyPSAGeneratorCol.P_NOM_MIN, 0.0), + (PyPSAGeneratorCol.P_NOM_MAX, float("inf")), + (PyPSAGeneratorCol.OVERNIGHT_COST, None), + (PyPSAGeneratorCol.DISCOUNT_RATE, None), + (PyPSAGeneratorCol.LIFETIME, float("inf")), + (PyPSAGeneratorCol.FOM_COST, 0.0), + ], + [(PyPSAGeneratorCol.P_NOM_EXTENDABLE, False)], + ) + + +def capital_cost_struct(overnight_cost: pl.Expr) -> pl.Expr: + """A Sienna ``CapitalCost`` whose curve prices one MW of new capacity linearly.""" + return pl.struct( + linear_value_curve(overnight_cost, input_at_zero=pl.lit(None, dtype=pl.Float64)).alias( + SiennaCapitalCostField.CAPITAL_COST + ), + pl.lit(0.0).alias(SiennaCapitalCostField.INTERCONNECTION_COST), + ).cast(CAPITAL_COST_DTYPE) + + +def operation_cost_struct(fixed: pl.Expr, cost_type: pl.Expr) -> pl.Expr: + """A Sienna ``GenericOperationCost`` carrying the fixed O&M of new capacity. + + A candidate's variable cost belongs to the component the build becomes in the base + system, so the curve here is zero and only the fixed term carries a number. Only + ``ThermalGenerationCost`` states a start-up and a shut-down cost, so the other two + variants leave both null and the sink writes neither. + """ + thermal_only = ( + pl.when(cost_type == SiennaCostType.THERMAL) + .then(pl.lit(0.0)) + .otherwise(pl.lit(None, dtype=pl.Float64)) + ) + return pl.struct( + cost_type.alias(SiennaOperationCostField.COST_TYPE), + fixed.alias(SiennaOperationCostField.FIXED), + thermal_only.alias(SiennaOperationCostField.START_UP), + thermal_only.alias(SiennaOperationCostField.SHUT_DOWN), + variable_cost_curve(pl.lit(0.0)).alias(SiennaOperationCostField.VARIABLE_OPERATION_COST), + ).cast(GENERIC_OPERATION_COST_DTYPE) + + +SUPPLY_NAME = _direct(source_col=PyPSAGeneratorCol.NAME, dest_col=S.NAME) + +SUPPLY_AVAILABLE = _default( + dest_col=S.AVAILABLE, + value=True, + note="PyPSA states no availability for a candidate; the technology may be built", +) + +SUPPLY_SIENNA_TYPE = Translation( + exprs=[], + make_events=lambda old, _: [ + TranslationEvent( + kind=EventKind.VALUE_DERIVED, + sources=[ + _source( + old[PyPSAGeneratorCol.NAME], + PyPSAGeneratorCol.P_NOM_EXTENDABLE, + old[PyPSAGeneratorCol.P_NOM_EXTENDABLE], + ) + ], + destinations=[ + _dest( + old[PyPSAGeneratorCol.NAME], + SIENNA_TYPE_ATTRIBUTE, + SiennaInvestmentsComponent.SUPPLY_TECHNOLOGY, + ) + ], + derivation="an extendable Generator is a candidate, and each becomes one technology", + ) + ], +) + +SUPPLY_POWER_SYSTEMS_TYPE = _direct( + source_col=PyPSAGeneratorCol.CARRIER, + dest_col=S.POWER_SYSTEMS_TYPE, + expr=pl.col(POWER_SYSTEMS_TYPE_COL), + derivation="carrier -> the base system type a build becomes, via the user mappings file", +) + +SUPPLY_REGION = _direct( + source_col=PyPSAGeneratorCol.BUS, + dest_col=S.REGION_NAME, + expr=pl.col(REGION_COL), + derivation="the area of the bus the generator sits on", +) + +SUPPLY_PRIME_MOVER = _direct( + source_col=PyPSAGeneratorCol.CARRIER, + dest_col=S.PRIME_MOVER_TYPE, + expr=pl.col(PRIME_MOVER_COL).cast(PRIME_MOVERS_DTYPE), + derivation="carrier -> PrimeMovers via the user mappings file", +) + +SUPPLY_FUEL = _direct( + source_col=PyPSAGeneratorCol.CARRIER, + dest_col=S.FUEL, + expr=pl.when(pl.col(FUEL_COL).is_not_null()) + .then(pl.concat_list(pl.col(FUEL_COL))) + .otherwise(pl.lit(None, dtype=pl.List(pl.Utf8))), + derivation="carrier -> the one ThermalFuels entry the user mappings file names, or none", +) + +SUPPLY_CAPITAL_COSTS = _direct( + source_col=PyPSAGeneratorCol.OVERNIGHT_COST, + dest_col=S.CAPITAL_COSTS, + expr=capital_cost_struct(pl.col(PyPSAGeneratorCol.OVERNIGHT_COST)), + unit=UNIT_DOLLARS_PER_MW, + derivation="overnight_cost as the proportional term of a linear capital cost curve", +) + +_cost_type = ( + pl.when(pl.col(POWER_SYSTEMS_TYPE_COL) == SiennaComponent.THERMAL_STANDARD) + .then(pl.lit(SiennaCostType.THERMAL)) + .when(pl.col(POWER_SYSTEMS_TYPE_COL) == SiennaComponent.HYDRO_DISPATCH) + .then(pl.lit(SiennaCostType.HYDRO_GEN)) + .otherwise(pl.lit(SiennaCostType.RENEWABLE)) +) + +SUPPLY_OPERATION_COSTS = _direct( + source_col=FOM_CHARGE_COL, + dest_col=S.OPERATION_COSTS, + expr=operation_cost_struct(yearly_fixed_charge(PyPSAGeneratorCol.FOM_COST), _cost_type), + unit=UNIT_DOLLARS_PER_MW_YEAR, + derivation=FOM_CHARGE_DERIVATION, + note=( + "the cost representation follows the base system type a build becomes: thermal for a " + "ThermalStandard, hydro for a HydroDispatch and renewable for the rest, and each " + "carries the fixed term" + ), +) + +SUPPLY_UNIT_SIZE = _direct( + source_col=UNIT_SIZE_COL, + dest_col=S.UNIT_SIZE, + expr=finite_or_null(pl.col(UNIT_SIZE_COL)), + unit=UNIT_MW, + derivation="the size of one unit, from the extensions sidecar", +) + +SUPPLY_CAPACITY_LIMITS = Translation( + exprs=[ + capacity_limits_struct( + pl.col(PyPSAGeneratorCol.P_NOM_MIN), pl.col(PyPSAGeneratorCol.P_NOM_MAX) + ).alias(S.CAPACITY_LIMITS) + ], + make_events=lambda old, new: [ + TranslationEvent( + kind=EventKind.VALUE_DERIVED, + sources=[ + _source( + old[PyPSAGeneratorCol.NAME], + PyPSAGeneratorCol.P_NOM_MIN, + old[PyPSAGeneratorCol.P_NOM_MIN], + UNIT_MW, + ), + _source( + old[PyPSAGeneratorCol.NAME], + PyPSAGeneratorCol.P_NOM_MAX, + old[PyPSAGeneratorCol.P_NOM_MAX], + UNIT_MW, + ), + ], + destinations=[ + _dest( + old[PyPSAGeneratorCol.NAME], + S.CAPACITY_LIMITS, + new[S.CAPACITY_LIMITS], + UNIT_MW, + ) + ], + derivation="p_nom_min, p_nom_max -> capacity_limits.{min, max}", + ) + ], +) + +SUPPLY_LIFETIME = _direct( + source_col=TECHNICAL_LIFE_COL, + dest_col=S.LIFETIME, + expr=finite_or_null(pl.col(TECHNICAL_LIFE_COL)).cast(pl.Int64), + unit=UNIT_YEARS, + derivation="the technical life, from the extensions sidecar", + note="how long a built unit runs, which is not the period its cost is recovered over", +) + +SUPPLY_TRANSLATIONS: list[Translation] = [ + SUPPLY_NAME, + SUPPLY_AVAILABLE, + SUPPLY_SIENNA_TYPE, + SUPPLY_POWER_SYSTEMS_TYPE, + SUPPLY_REGION, + SUPPLY_PRIME_MOVER, + SUPPLY_FUEL, + SUPPLY_CAPITAL_COSTS, + SUPPLY_OPERATION_COSTS, + SUPPLY_UNIT_SIZE, + SUPPLY_CAPACITY_LIMITS, + SUPPLY_LIFETIME, +] + + +def build_supply_translations(base_year: int, start: int) -> list[Translation]: + """Every SupplyTechnology translation, including the two the caller's numbers decide.""" + return [ + row_position_id_translation( + _dest, + dest_name_col=S.NAME, + id_col=S.ID, + note=PORTFOLIO_ID_NOTE, + start=start, + ), + *SUPPLY_TRANSLATIONS, + build_financial_data_translation( + _source, + _dest, + name_col=PyPSAGeneratorCol.NAME, + lifetime_col=PyPSAGeneratorCol.LIFETIME, + discount_rate_col=PyPSAGeneratorCol.DISCOUNT_RATE, + dest_col=S.FINANCIAL_DATA, + base_year=base_year, + ), + ] diff --git a/interop/plugins/shared/pypsa_sienna_investments_translations/_topology.py b/interop/plugins/shared/pypsa_sienna_investments_translations/_topology.py new file mode 100644 index 0000000..51dcfe5 --- /dev/null +++ b/interop/plugins/shared/pypsa_sienna_investments_translations/_topology.py @@ -0,0 +1,75 @@ +"""The buses each region holds: the Sienna TopologyMapping supplemental attribute. + +A portfolio groups its technologies by region and the base system holds the buses, so the +mapping between the two travels as an attribute of the region rather than as a component. +""" + +from __future__ import annotations + +from functools import partial + +import polars as pl + +from interop.plugins.shared.constants import Framework +from interop.plugins.shared.pypsa_sienna_translations._shared import sienna_dest_field +from interop.plugins.shared.sienna_constants import SiennaACBusCol, SiennaComponent +from interop.plugins.shared.sienna_investments_constants import ( + SiennaSupplementalAttribute, + SiennaTopologyMappingCol, +) +from interop.plugins.shared.translation_runner import Translation, row_position_id_translation +from interop.ports.outbound.reporting import EventKind, SourceField, TranslationEvent + +# Source-table column holding the region a mapping describes; not a schema field. +AREA_NAME = "area_name" + +T = SiennaTopologyMappingCol + +_dest = partial(sienna_dest_field, SiennaSupplementalAttribute.TOPOLOGY_MAPPING) + + +def build_topology_source_table(buses: pl.DataFrame) -> pl.DataFrame: + """One row per area of the base system, holding the names of the buses in it.""" + return ( + buses.filter(pl.col(SiennaACBusCol.AREA).is_not_null()) + .group_by(SiennaACBusCol.AREA) + .agg(pl.col(SiennaACBusCol.NAME).alias(T.BUSES)) + .sort(SiennaACBusCol.AREA) + .rename({SiennaACBusCol.AREA: AREA_NAME}) + ) + + +TOPOLOGY_BUSES = Translation( + exprs=[pl.col(T.BUSES)], # identity: the aggregation already named the column + make_events=lambda old, new: [ + TranslationEvent( + kind=EventKind.VALUE_DERIVED, + sources=[ + SourceField( + framework=Framework.SIENNA, + component=SiennaComponent.AC_BUS, + name=bus, + attribute=SiennaACBusCol.AREA, + value=old[AREA_NAME], + ) + for bus in old[T.BUSES] + ], + destinations=[_dest(old[AREA_NAME], T.BUSES, new[T.BUSES])], + derivation="the buses of the base system that sit in this area", + ) + ], +) + + +def build_topology_mapping_translations(start: int) -> list[Translation]: + """The TopologyMapping attributes, taking ids from the counter the flat array shares.""" + return [ + row_position_id_translation( + _dest, + dest_name_col=AREA_NAME, + id_col=T.ID, + note="assigned by position in the portfolio's flat supplemental attribute array", + start=start, + ), + TOPOLOGY_BUSES, + ] diff --git a/interop/plugins/shared/sienna_constants.py b/interop/plugins/shared/sienna_constants.py index c06c667..89b3796 100644 --- a/interop/plugins/shared/sienna_constants.py +++ b/interop/plugins/shared/sienna_constants.py @@ -90,6 +90,14 @@ class SiennaComponent(StrEnum): SIENNA_TYPE_ATTRIBUTE = "type" """Attribute name used in event-only sienna_type Translation objects.""" +SIENNA_NAME_COLUMN = "name" + +SIENNA_ID_COLUMN = "id" + +SIENNA_REGION_NAME_COLUMN = "region_name" + +SIENNA_REGION_COLUMN = "region" + class SiennaACBusCol: """Sienna ACBus column names.""" diff --git a/interop/plugins/shared/sienna_investments_constants.py b/interop/plugins/shared/sienna_investments_constants.py new file mode 100644 index 0000000..a1d00fa --- /dev/null +++ b/interop/plugins/shared/sienna_investments_constants.py @@ -0,0 +1,438 @@ +"""Sienna investments vocabulary: the types a portfolio document is made of. + +Every name and every required/optional status here comes from the ``Investments/`` namespace +of SiennaSchemas. A portfolio is a separate document from a system: it holds the candidate +technologies, the policy requirements and the regional aggregations of an expansion problem, +and it names the base power system it expands rather than embedding it. +""" + +from __future__ import annotations + +from enum import StrEnum + +import polars as pl + +from interop.plugins.shared.sienna_constants import ( + COST_CURVE_DTYPE, + COST_TYPE_DTYPE, + EFFICIENCY_DTYPE, + IO_CURVE_DTYPE, + MIN_MAX_DTYPE, + PRIME_MOVERS_DTYPE, + SIENNA_REGION_NAME_COLUMN, +) + + +class SiennaInvestmentsComponent(StrEnum): + """The investments types this translation writes into ``PortfolioDocument.components``.""" + + SUPPLY_TECHNOLOGY = "SupplyTechnology" + STORAGE_TECHNOLOGY = "StorageTechnology" + DEMAND_REQUIREMENT = "DemandRequirement" + CARBON_CAPS = "CarbonCaps" + + +class SiennaSupplementalAttribute(StrEnum): + """The supplemental attribute types a portfolio carries in its flat array. + + Each one holds names of things in the base system, so it describes a component rather + than standing beside it: the association table says which component each describes. + """ + + EXISTING_DEVICES = "ExistingDevices" + RETIREMENT_POTENTIAL = "RetirementPotential" + TOPOLOGY_MAPPING = "TopologyMapping" + + +class PortfolioDocument: + """Top-level keys of a SiennaSchemas ``PortfolioDocument``.""" + + AGGREGATION = "aggregation" + FINANCIAL_DATA = "financial_data" + COMPONENTS = "components" + SUPPLEMENTAL_ATTRIBUTES = "supplemental_attributes" + SUPPLEMENTAL_ATTRIBUTE_ASSOCIATIONS = "supplemental_attribute_associations" + TIME_SERIES_ASSOCIATIONS = "time_series_associations" + BASE_SYSTEM_FILE = "base_system_file" + TIME_SERIES_STORAGE_FILE = "time_series_storage_file" + + +PORTFOLIO_JSON_FILENAME = "portfolio.json" +"""Default basename of the portfolio document written beside the base system.""" + +PORTFOLIO_AGGREGATION = "Area" +"""The base system type a portfolio's regions are grouped by. + +``PortfolioDocument.aggregation`` names a type the consumer resolves rather than a component +of the document, and the regions a PyPSA network states are the areas of the base system. +""" + + +class SiennaSupplyTechnologyCol: + """SiennaSchemas ``SupplyTechnology`` field names. + + ``region_name`` is not a schema field: it holds the area name until the sink resolves it + to the ``region`` list of integer ids, the way a load carries ``bus_name``. + """ + + ID = "id" + NAME = "name" + AVAILABLE = "available" + POWER_SYSTEMS_TYPE = "power_systems_type" + REGION_NAME = SIENNA_REGION_NAME_COLUMN + PRIME_MOVER_TYPE = "prime_mover_type" + FUEL = "fuel" + COFIRE_START_LIMITS = "cofire_start_limits" + COFIRE_LEVEL_LIMITS = "cofire_level_limits" + CAPITAL_COSTS = "capital_costs" + OPERATION_COSTS = "operation_costs" + UNIT_SIZE = "unit_size" + CAPACITY_LIMITS = "capacity_limits" + OUTAGE_FACTOR = "outage_factor" + MIN_GENERATION_FRACTION = "min_generation_fraction" + RAMP_LIMITS = "ramp_limits" + TIME_LIMITS = "time_limits" + START_FUEL_MMBTU_PER_MW = "start_fuel_mmbtu_per_mw" + LIFETIME = "lifetime" + REQUIREMENTS = "requirements" + FINANCIAL_DATA = "financial_data" + + +class SiennaStorageTechnologyCol: + """SiennaSchemas ``StorageTechnology`` field names. + + Charge, discharge and energy capacity are added independently, so each has its own + capital cost, unit size and capacity limits. + """ + + ID = "id" + NAME = "name" + AVAILABLE = "available" + REGION_NAME = SIENNA_REGION_NAME_COLUMN + POWER_SYSTEMS_TYPE = "power_systems_type" + MIN_DISCHARGE_FRACTION = "min_discharge_fraction" + PRIME_MOVER_TYPE = "prime_mover_type" + STORAGE_TECH = "storage_tech" + CAPITAL_COSTS = "capital_costs" + OPERATION_COSTS = "operation_costs" + UNIT_SIZE_DISCHARGE = "unit_size_discharge" + UNIT_SIZE_CHARGE = "unit_size_charge" + UNIT_SIZE_ENERGY = "unit_size_energy" + CAPACITY_LIMITS_CHARGE = "capacity_limits_charge" + CAPACITY_LIMITS_DISCHARGE = "capacity_limits_discharge" + CAPACITY_LIMITS_ENERGY = "capacity_limits_energy" + DURATION_LIMITS = "duration_limits" + EFFICIENCY = "efficiency" + LOSSES = "losses" + LIFETIME = "lifetime" + REQUIREMENTS = "requirements" + FINANCIAL_DATA = "financial_data" + + +class SiennaDemandRequirementCol: + ID = "id" + NAME = "name" + AVAILABLE = "available" + POWER_SYSTEMS_TYPE = "power_systems_type" + CONFORMITY = "conformity" + GROWTH_RATE = "growth_rate" + NEW_DEMAND_MW = "new_demand_mw" + NEW_CONSTRUCTION_YEAR = "new_construction_year" + REGION_NAME = SIENNA_REGION_NAME_COLUMN + VALUE_OF_LOST_LOAD = "value_of_lost_load" + UNSERVED_DEMAND_CURVE = "unserved_demand_curve" + REQUIREMENTS = "requirements" + + +class SiennaCarbonCapsCol: + """SiennaSchemas ``CarbonCaps`` field names. + + The type names no members and no region: a cap holds the whole portfolio. + """ + + ID = "id" + NAME = "name" + AVAILABLE = "available" + TARGET_YEAR = "target_year" + MAX_TONS_MWH = "max_tons_mwh" + MAX_MTONS = "max_mtons" + + +class SiennaExistingDevicesCol: + ID = "id" + EXISTING_DEVICES = "existing_devices" + + +class SiennaRetirementPotentialCol: + """SiennaSchemas ``RetirementPotential`` field names. + + ``build_year`` and ``planned_retirement_year`` are objects keyed by device name. A + Polars struct has fixed field names, so both travel as a list of name/year pairs and the + sink writes each list out as the object the schema states. + """ + + ID = "id" + ELIGIBLE_GENERATORS = "eligible_generators" + PLANNED_RETIREMENT_YEAR = "planned_retirement_year" + BUILD_YEAR = "build_year" + RETIREMENT_COST = "retirement_cost" + + +class NamedYearField: + """Field names of one entry in a name-keyed year object.""" + + NAME = "name" + YEAR = "year" + + +class SiennaTopologyMappingCol: + ID = "id" + BUSES = "buses" + + +class SiennaPortfolioFinancialDataCol: + ID = "id" + DISCOUNT_RATE = "discount_rate" + INFLATION_RATE = "inflation_rate" + INTEREST_RATE = "interest_rate" + BASE_YEAR = "base_year" + + +class SiennaTechnologyFinancialDataField: + """SiennaSchemas ``TechnologyFinancialData`` field names. All six are required.""" + + CAPITAL_RECOVERY_PERIOD = "capital_recovery_period" + TECHNOLOGY_BASE_YEAR = "technology_base_year" + DEBT_FRACTION = "debt_fraction" + DEBT_RATE = "debt_rate" + RETURN_ON_EQUITY = "return_on_equity" + TAX_RATE = "tax_rate" + + +class SiennaCapitalCostField: + CAPITAL_COST = "capital_cost" + INTERCONNECTION_COST = "interconnection_cost" + + +class SiennaStorageCapitalCostField: + CHARGE_CAPITAL_COST = "charge_capital_cost" + DISCHARGE_CAPITAL_COST = "discharge_capital_cost" + ENERGY_CAPITAL_COST = "energy_capital_cost" + INTERCONNECTION_COST = "interconnection_cost" + + +class SiennaOperationCostField: + """Field names of the Sienna ``GenericOperationCost`` variants this translation writes.""" + + COST_TYPE = "cost_type" + FIXED = "fixed" + START_UP = "start_up" + SHUT_DOWN = "shut_down" + VARIABLE_OPERATION_COST = "variable_operation_cost" + + +class SiennaSupplementalAttributeAssociationCol: + """SiennaSchemas ``SupplementalAttributeAssociation`` field names. + + ``component_name`` is not a schema field: it holds the described component's name until + the sink resolves it to ``component_id``. + """ + + COMPONENT_ID = "component_id" + COMPONENT_NAME = "component_name" + COMPONENT_TYPE = "component_type" + ATTRIBUTE_ID = "attribute_id" + ATTRIBUTE_TYPE = "attribute_type" + + +# --- Nested struct dtypes --- + +CAPITAL_COST_DTYPE: pl.DataType = pl.Struct( + { + SiennaCapitalCostField.CAPITAL_COST: IO_CURVE_DTYPE, + SiennaCapitalCostField.INTERCONNECTION_COST: pl.Float64, + } +) + +STORAGE_CAPITAL_COST_DTYPE: pl.DataType = pl.Struct( + { + SiennaStorageCapitalCostField.CHARGE_CAPITAL_COST: IO_CURVE_DTYPE, + SiennaStorageCapitalCostField.DISCHARGE_CAPITAL_COST: IO_CURVE_DTYPE, + SiennaStorageCapitalCostField.ENERGY_CAPITAL_COST: IO_CURVE_DTYPE, + SiennaStorageCapitalCostField.INTERCONNECTION_COST: pl.Float64, + } +) + +GENERIC_OPERATION_COST_DTYPE: pl.DataType = pl.Struct( + { + SiennaOperationCostField.COST_TYPE: COST_TYPE_DTYPE, + SiennaOperationCostField.FIXED: pl.Float64, + SiennaOperationCostField.START_UP: pl.Float64, + SiennaOperationCostField.SHUT_DOWN: pl.Float64, + SiennaOperationCostField.VARIABLE_OPERATION_COST: COST_CURVE_DTYPE, + } +) + +STORAGE_OPERATION_COST_DTYPE: pl.DataType = pl.Struct( + { + SiennaOperationCostField.COST_TYPE: COST_TYPE_DTYPE, + SiennaOperationCostField.FIXED: pl.Float64, + SiennaOperationCostField.START_UP: pl.Float64, + SiennaOperationCostField.SHUT_DOWN: pl.Float64, + } +) + +TECHNOLOGY_FINANCIAL_DATA_DTYPE: pl.DataType = pl.Struct( + { + SiennaTechnologyFinancialDataField.CAPITAL_RECOVERY_PERIOD: pl.Int64, + SiennaTechnologyFinancialDataField.TECHNOLOGY_BASE_YEAR: pl.Int64, + SiennaTechnologyFinancialDataField.DEBT_FRACTION: pl.Float64, + SiennaTechnologyFinancialDataField.DEBT_RATE: pl.Float64, + SiennaTechnologyFinancialDataField.RETURN_ON_EQUITY: pl.Float64, + SiennaTechnologyFinancialDataField.TAX_RATE: pl.Float64, + } +) + +NAMED_YEAR_DTYPE: pl.DataType = pl.List( + pl.Struct({NamedYearField.NAME: pl.Utf8, NamedYearField.YEAR: pl.Int64}) +) + +CAPACITY_LIMITS_DTYPE: pl.DataType = MIN_MAX_DTYPE + + +# --- Destination table schemas --- + +SUPPLY_TECHNOLOGY_DESTINATION_SCHEMA: dict[str, pl.DataType | type[pl.DataType]] = { + SiennaSupplyTechnologyCol.ID: pl.Int64, + SiennaSupplyTechnologyCol.NAME: pl.Utf8, + SiennaSupplyTechnologyCol.AVAILABLE: pl.Boolean, + SiennaSupplyTechnologyCol.POWER_SYSTEMS_TYPE: pl.Utf8, + SiennaSupplyTechnologyCol.REGION_NAME: pl.Utf8, + SiennaSupplyTechnologyCol.PRIME_MOVER_TYPE: PRIME_MOVERS_DTYPE, + SiennaSupplyTechnologyCol.FUEL: pl.List(pl.Utf8), + SiennaSupplyTechnologyCol.COFIRE_START_LIMITS: pl.Utf8, + SiennaSupplyTechnologyCol.COFIRE_LEVEL_LIMITS: pl.Utf8, + SiennaSupplyTechnologyCol.CAPITAL_COSTS: CAPITAL_COST_DTYPE, + SiennaSupplyTechnologyCol.OPERATION_COSTS: GENERIC_OPERATION_COST_DTYPE, + SiennaSupplyTechnologyCol.UNIT_SIZE: pl.Float64, + SiennaSupplyTechnologyCol.CAPACITY_LIMITS: CAPACITY_LIMITS_DTYPE, + SiennaSupplyTechnologyCol.OUTAGE_FACTOR: pl.Utf8, + SiennaSupplyTechnologyCol.MIN_GENERATION_FRACTION: pl.Float64, + SiennaSupplyTechnologyCol.RAMP_LIMITS: pl.Utf8, + SiennaSupplyTechnologyCol.TIME_LIMITS: pl.Utf8, + SiennaSupplyTechnologyCol.START_FUEL_MMBTU_PER_MW: pl.Float64, + SiennaSupplyTechnologyCol.LIFETIME: pl.Int64, + SiennaSupplyTechnologyCol.REQUIREMENTS: pl.List(pl.Int64), + SiennaSupplyTechnologyCol.FINANCIAL_DATA: TECHNOLOGY_FINANCIAL_DATA_DTYPE, +} + +STORAGE_TECHNOLOGY_DESTINATION_SCHEMA: dict[str, pl.DataType | type[pl.DataType]] = { + SiennaStorageTechnologyCol.ID: pl.Int64, + SiennaStorageTechnologyCol.NAME: pl.Utf8, + SiennaStorageTechnologyCol.AVAILABLE: pl.Boolean, + SiennaStorageTechnologyCol.REGION_NAME: pl.Utf8, + SiennaStorageTechnologyCol.POWER_SYSTEMS_TYPE: pl.Utf8, + SiennaStorageTechnologyCol.MIN_DISCHARGE_FRACTION: pl.Float64, + SiennaStorageTechnologyCol.PRIME_MOVER_TYPE: PRIME_MOVERS_DTYPE, + SiennaStorageTechnologyCol.STORAGE_TECH: pl.Utf8, + SiennaStorageTechnologyCol.CAPITAL_COSTS: STORAGE_CAPITAL_COST_DTYPE, + SiennaStorageTechnologyCol.OPERATION_COSTS: STORAGE_OPERATION_COST_DTYPE, + SiennaStorageTechnologyCol.UNIT_SIZE_DISCHARGE: pl.Float64, + SiennaStorageTechnologyCol.UNIT_SIZE_CHARGE: pl.Float64, + SiennaStorageTechnologyCol.UNIT_SIZE_ENERGY: pl.Float64, + SiennaStorageTechnologyCol.CAPACITY_LIMITS_CHARGE: CAPACITY_LIMITS_DTYPE, + SiennaStorageTechnologyCol.CAPACITY_LIMITS_DISCHARGE: CAPACITY_LIMITS_DTYPE, + SiennaStorageTechnologyCol.CAPACITY_LIMITS_ENERGY: CAPACITY_LIMITS_DTYPE, + SiennaStorageTechnologyCol.DURATION_LIMITS: CAPACITY_LIMITS_DTYPE, + SiennaStorageTechnologyCol.EFFICIENCY: EFFICIENCY_DTYPE, + SiennaStorageTechnologyCol.LOSSES: pl.Float64, + SiennaStorageTechnologyCol.LIFETIME: pl.Int64, + SiennaStorageTechnologyCol.REQUIREMENTS: pl.List(pl.Int64), + SiennaStorageTechnologyCol.FINANCIAL_DATA: TECHNOLOGY_FINANCIAL_DATA_DTYPE, +} + +DEMAND_REQUIREMENT_DESTINATION_SCHEMA: dict[str, pl.DataType | type[pl.DataType]] = { + SiennaDemandRequirementCol.ID: pl.Int64, + SiennaDemandRequirementCol.NAME: pl.Utf8, + SiennaDemandRequirementCol.AVAILABLE: pl.Boolean, + SiennaDemandRequirementCol.POWER_SYSTEMS_TYPE: pl.Utf8, + SiennaDemandRequirementCol.REGION_NAME: pl.Utf8, + SiennaDemandRequirementCol.CONFORMITY: pl.Utf8, + SiennaDemandRequirementCol.GROWTH_RATE: pl.Float64, + SiennaDemandRequirementCol.NEW_DEMAND_MW: pl.Float64, + SiennaDemandRequirementCol.NEW_CONSTRUCTION_YEAR: pl.Int64, + SiennaDemandRequirementCol.VALUE_OF_LOST_LOAD: pl.Float64, + SiennaDemandRequirementCol.UNSERVED_DEMAND_CURVE: IO_CURVE_DTYPE, + SiennaDemandRequirementCol.REQUIREMENTS: pl.List(pl.Int64), +} + +CARBON_CAPS_DESTINATION_SCHEMA: dict[str, pl.DataType | type[pl.DataType]] = { + SiennaCarbonCapsCol.ID: pl.Int64, + SiennaCarbonCapsCol.NAME: pl.Utf8, + SiennaCarbonCapsCol.AVAILABLE: pl.Boolean, + SiennaCarbonCapsCol.TARGET_YEAR: pl.Int64, + SiennaCarbonCapsCol.MAX_TONS_MWH: pl.Float64, + SiennaCarbonCapsCol.MAX_MTONS: pl.Float64, +} + +EXISTING_DEVICES_DESTINATION_SCHEMA: dict[str, pl.DataType | type[pl.DataType]] = { + SiennaExistingDevicesCol.ID: pl.Int64, + SiennaExistingDevicesCol.EXISTING_DEVICES: pl.List(pl.Utf8), +} + +RETIREMENT_POTENTIAL_DESTINATION_SCHEMA: dict[str, pl.DataType | type[pl.DataType]] = { + SiennaRetirementPotentialCol.ID: pl.Int64, + SiennaRetirementPotentialCol.ELIGIBLE_GENERATORS: pl.List(pl.Utf8), + SiennaRetirementPotentialCol.PLANNED_RETIREMENT_YEAR: NAMED_YEAR_DTYPE, + SiennaRetirementPotentialCol.BUILD_YEAR: NAMED_YEAR_DTYPE, + SiennaRetirementPotentialCol.RETIREMENT_COST: IO_CURVE_DTYPE, +} + +TOPOLOGY_MAPPING_DESTINATION_SCHEMA: dict[str, pl.DataType | type[pl.DataType]] = { + SiennaTopologyMappingCol.ID: pl.Int64, + SiennaTopologyMappingCol.BUSES: pl.List(pl.Utf8), +} + +PORTFOLIO_FINANCIAL_DATA_DESTINATION_SCHEMA: dict[str, pl.DataType | type[pl.DataType]] = { + SiennaPortfolioFinancialDataCol.ID: pl.Int64, + SiennaPortfolioFinancialDataCol.DISCOUNT_RATE: pl.Float64, + SiennaPortfolioFinancialDataCol.INFLATION_RATE: pl.Float64, + SiennaPortfolioFinancialDataCol.INTEREST_RATE: pl.Float64, + SiennaPortfolioFinancialDataCol.BASE_YEAR: pl.Int64, +} + +SUPPLEMENTAL_ATTRIBUTE_ASSOCIATION_SCHEMA: dict[str, pl.DataType | type[pl.DataType]] = { + SiennaSupplementalAttributeAssociationCol.COMPONENT_NAME: pl.Utf8, + SiennaSupplementalAttributeAssociationCol.COMPONENT_TYPE: pl.Utf8, + SiennaSupplementalAttributeAssociationCol.ATTRIBUTE_ID: pl.Int64, + SiennaSupplementalAttributeAssociationCol.ATTRIBUTE_TYPE: pl.Utf8, +} + +# The order the flat supplemental attribute array lists its types in. The step numbering the +# attributes and the sink writing them both read it. +SUPPLEMENTAL_ATTRIBUTE_ORDER: tuple[SiennaSupplementalAttribute, ...] = ( + SiennaSupplementalAttribute.EXISTING_DEVICES, + SiennaSupplementalAttribute.RETIREMENT_POTENTIAL, + SiennaSupplementalAttribute.TOPOLOGY_MAPPING, +) + +# Keys for the destination tables that are not components of the document: the association +# rows and the portfolio-wide financial data. +SUPPLEMENTAL_ATTRIBUTE_ASSOCIATIONS_TABLE = "supplemental_attribute_associations" +PORTFOLIO_FINANCIAL_DATA_TABLE = "PortfolioFinancialData" + +# A cost of capital PyPSA states as one discount rate, written as all-equity financing. +# TechnologyFinancialData requires all six of its fields and has none for a weighted average +# cost of capital; with no debt that average equals the return on equity, so the rate the +# source states is the rate the solver uses. +ALL_EQUITY_DEBT_FRACTION: float = 0.0 +ALL_EQUITY_DEBT_RATE: float = 0.0 +ALL_EQUITY_TAX_RATE: float = 0.0 + +# The base economic year every cost is discounted to, where no source states one. It matches +# the year SiennaSchemas itself defaults a construction year to. +DEFAULT_BASE_YEAR: int = 2020 + +# A portfolio this translation writes carries no rate of its own beside the per-technology +# return on equity, so the document-level rates take the schema's own defaults. +DEFAULT_PORTFOLIO_RATE: float = 0.0 diff --git a/interop/plugins/sinks/_sienna_files.py b/interop/plugins/sinks/_sienna_files.py index 21a2293..ba2bbd3 100644 --- a/interop/plugins/sinks/_sienna_files.py +++ b/interop/plugins/sinks/_sienna_files.py @@ -47,6 +47,15 @@ def _write_sienna_files( self._fs.write_bytes(paths.system_json, _serialise(paths, payload, indent)) +def validate_refs(ref_col: str, needed: list[str], available: set[str], context: str) -> None: + missing = set(needed) - available + if missing: + raise ValueError( + f"{context}: {len(missing)} reference(s) in {ref_col!r} " + f"not found in parent table: {sorted(missing)}" + ) + + def _serialise(paths: SiennaFilePaths, payload: dict[str, Any], indent: int) -> bytes: """The system JSON, naming its companions by the bare filenames a reader expects.""" named = { diff --git a/interop/plugins/sinks/emit_sienna_portfolio.py b/interop/plugins/sinks/emit_sienna_portfolio.py new file mode 100644 index 0000000..fd1c93b --- /dev/null +++ b/interop/plugins/sinks/emit_sienna_portfolio.py @@ -0,0 +1,217 @@ +"""Write the SiennaSchemas portfolio document beside the base system it expands.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any, ClassVar + +import polars as pl +from pydantic import BaseModel, Field + +from interop.core.pipeline import Sink, State +from interop.plugins.shared.sienna_constants import ( + SIENNA_ID_COLUMN, + SIENNA_NAME_COLUMN, + SIENNA_REGION_COLUMN, + SIENNA_REGION_NAME_COLUMN, + SiennaCompanionFilename, + SiennaComponent, +) +from interop.plugins.shared.sienna_investments_constants import ( + PORTFOLIO_AGGREGATION, + PORTFOLIO_FINANCIAL_DATA_TABLE, + PORTFOLIO_JSON_FILENAME, + SUPPLEMENTAL_ATTRIBUTE_ASSOCIATIONS_TABLE, + SUPPLEMENTAL_ATTRIBUTE_ORDER, + NamedYearField, + PortfolioDocument, + SiennaInvestmentsComponent, + SiennaRetirementPotentialCol, + SiennaSupplementalAttributeAssociationCol, +) +from interop.plugins.sinks._sienna_files import SYSTEM_JSON_FILENAME, validate_refs +from interop.ports.outbound.filesystem import FilesystemPort, Location + +_DEFAULT_OUTPUT_DIR = Path("outputs") + +_NAMED_YEAR_FIELDS: tuple[str, ...] = ( + SiennaRetirementPotentialCol.BUILD_YEAR, + SiennaRetirementPotentialCol.PLANNED_RETIREMENT_YEAR, +) + +_COMPONENT_TYPES: tuple[SiennaInvestmentsComponent, ...] = ( + SiennaInvestmentsComponent.SUPPLY_TECHNOLOGY, + SiennaInvestmentsComponent.STORAGE_TECHNOLOGY, + SiennaInvestmentsComponent.DEMAND_REQUIREMENT, + SiennaInvestmentsComponent.CARBON_CAPS, +) + + +class EmitSiennaPortfolioParams(BaseModel): + output_path: Location = Field( + default=_DEFAULT_OUTPUT_DIR / PORTFOLIO_JSON_FILENAME, + description="the SiennaSchemas portfolio document to write", + ) + base_system_file: str = Field( + default=SYSTEM_JSON_FILENAME, + description="basename of the base system document the portfolio expands", + ) + time_series_storage_file: str = Field( + default=SiennaCompanionFilename.TIME_SERIES_H5, + description="basename of the HDF5 companion holding the base system's time series", + ) + indent: int = Field(default=2, description="JSON indent width") + + +class EmitSiennaPortfolio(Sink): + name: ClassVar[str] = "emit_sienna_portfolio" + params_schema: ClassVar[type[BaseModel] | None] = EmitSiennaPortfolioParams + + def __init__(self, fs: FilesystemPort) -> None: + self._fs = fs + + def write(self, state: State, params: BaseModel | None) -> None: + if not isinstance(params, EmitSiennaPortfolioParams): + raise TypeError( + f"{type(self).__name__} requires {EmitSiennaPortfolioParams.__name__}, " + f"got {type(params).__name__}" + ) + payload = self.build_payload( + state, params.base_system_file, params.time_series_storage_file + ) + serialised = json.dumps(payload, indent=params.indent, default=str).encode("utf-8") + self._fs.write_bytes(params.output_path, serialised) + + @staticmethod + def build_payload( + state: State, base_system_file: str, time_series_storage_file: str + ) -> dict[str, Any]: + area_ids = _name_to_id(state.destination_tables.get(SiennaComponent.AREA)) + components: dict[str, Any] = {} + component_ids: dict[str, dict[str, int]] = {} + for component in _COMPONENT_TYPES: + table = state.destination_tables.get(component) + objects = _build_components(table, area_ids) + if objects: + components[str(component)] = objects + component_ids[str(component)] = _name_to_id(table) + component_ids[str(SiennaComponent.AREA)] = area_ids + + attributes, attribute_ids = _build_attributes(state) + payload: dict[str, Any] = { + PortfolioDocument.AGGREGATION: PORTFOLIO_AGGREGATION, + PortfolioDocument.COMPONENTS: components, + PortfolioDocument.SUPPLEMENTAL_ATTRIBUTES: attributes, + PortfolioDocument.SUPPLEMENTAL_ATTRIBUTE_ASSOCIATIONS: _build_associations( + state.destination_tables.get(SUPPLEMENTAL_ATTRIBUTE_ASSOCIATIONS_TABLE), + component_ids, + attribute_ids, + ), + PortfolioDocument.TIME_SERIES_ASSOCIATIONS: [], + PortfolioDocument.BASE_SYSTEM_FILE: base_system_file, + PortfolioDocument.TIME_SERIES_STORAGE_FILE: time_series_storage_file, + } + financial_data = state.destination_tables.get(PORTFOLIO_FINANCIAL_DATA_TABLE) + if financial_data is not None and not financial_data.is_empty(): + payload[PortfolioDocument.FINANCIAL_DATA] = _stated( + next(financial_data.iter_rows(named=True)) + ) + return payload + + +def _name_to_id(table: pl.DataFrame | None) -> dict[str, int]: + if table is None or table.is_empty(): + return {} + return dict( + zip(table[SIENNA_NAME_COLUMN].to_list(), table[SIENNA_ID_COLUMN].to_list(), strict=True) + ) + + +def _build_components(table: pl.DataFrame | None, area_ids: dict[str, int]) -> list[dict[str, Any]]: + """One object per row, with the region name resolved to the area ids it stands for.""" + if table is None or table.is_empty(): + return [] + holds_region = SIENNA_REGION_NAME_COLUMN in table.columns + if holds_region: + validate_refs( + SIENNA_REGION_NAME_COLUMN, + table[SIENNA_REGION_NAME_COLUMN].drop_nulls().unique().to_list(), + set(area_ids), + "technologies -> areas", + ) + objects: list[dict[str, Any]] = [] + for row in table.iter_rows(named=True): + component = _stated({k: v for k, v in row.items() if k != SIENNA_REGION_NAME_COLUMN}) + area_name = row.get(SIENNA_REGION_NAME_COLUMN) if holds_region else None + if area_name is not None: + component[SIENNA_REGION_COLUMN] = [area_ids[area_name]] + objects.append(component) + return objects + + +def _build_attributes(state: State) -> tuple[list[dict[str, Any]], set[int]]: + """The flat array, in the order the step numbered its types, and the ids it holds.""" + attributes: list[dict[str, Any]] = [] + ids: set[int] = set() + for attribute in SUPPLEMENTAL_ATTRIBUTE_ORDER: + table = state.destination_tables.get(attribute) + if table is None: + continue + for row in table.iter_rows(named=True): + ids.add(row[SIENNA_ID_COLUMN]) + attributes.append(_stated({k: _named_years(k, v) for k, v in row.items()})) + return attributes, ids + + +def _named_years(field: str, value: Any) -> Any: + """A list of name/year pairs, as the object keyed by name that the schema states.""" + if field not in _NAMED_YEAR_FIELDS or value is None: + return value + years = {entry[NamedYearField.NAME]: entry[NamedYearField.YEAR] for entry in value} + return years or None + + +def _build_associations( + table: pl.DataFrame | None, + component_ids: dict[str, dict[str, int]], + attribute_ids: set[int], +) -> list[dict[str, Any]]: + """One row per (attribute, component) pair, with the component's name resolved to its id.""" + if table is None or table.is_empty(): + return [] + A = SiennaSupplementalAttributeAssociationCol + rows: list[dict[str, Any]] = [] + for row in table.iter_rows(named=True): + component_type = row[A.COMPONENT_TYPE] + name = row[A.COMPONENT_NAME] + by_name = component_ids.get(component_type, {}) + validate_refs(A.COMPONENT_NAME, [name], set(by_name), f"attributes -> {component_type}") + if row[A.ATTRIBUTE_ID] not in attribute_ids: + raise ValueError( + f"attributes -> supplemental_attributes: no attribute with id {row[A.ATTRIBUTE_ID]}" + ) + rows.append( + { + A.COMPONENT_ID: by_name[name], + A.COMPONENT_TYPE: component_type, + A.ATTRIBUTE_ID: row[A.ATTRIBUTE_ID], + A.ATTRIBUTE_TYPE: row[A.ATTRIBUTE_TYPE], + } + ) + return rows + + +def _stated(row: dict[str, Any]) -> dict[str, Any]: + """The fields a row actually holds, at every depth of a nested struct. + + A field no table wrote is absent rather than null, so the schema's own default applies + to it rather than a null a consumer would have to read as one. A cost struct carries + every variant's fields, so this is also what leaves a renewable technology's cost + without the two a thermal one alone states. + """ + return { + key: _stated(value) if isinstance(value, dict) else value + for key, value in row.items() + if value is not None + } diff --git a/interop/plugins/sinks/emit_sienna_system_json.py b/interop/plugins/sinks/emit_sienna_system_json.py index da25952..c416662 100644 --- a/interop/plugins/sinks/emit_sienna_system_json.py +++ b/interop/plugins/sinks/emit_sienna_system_json.py @@ -20,6 +20,7 @@ SiennaThermalGeneratorCol, SiennaTimeSeriesAssociationCol, ) +from interop.plugins.sinks._sienna_files import validate_refs from interop.ports.outbound.filesystem import FilesystemPort, Location # Internal-only columns in TIME_SERIES_ASSOCIATION that are not part of the JSON schema. @@ -189,7 +190,7 @@ def _build_bus_components( if buses_df is None: return [], {} referenced_areas = buses_df[SiennaACBusCol.AREA].drop_nulls().unique().to_list() - _validate_refs(SiennaACBusCol.AREA, referenced_areas, set(area_ids), "buses -> areas") + validate_refs(SiennaACBusCol.AREA, referenced_areas, set(area_ids), "buses -> areas") components: list[dict[str, Any]] = [] bus_ids: dict[str, int] = {} for row in buses_df.iter_rows(named=True): @@ -216,7 +217,7 @@ def _build_bus_attached_components( """ if df is None: return [] - _validate_refs(name_col, df[name_col].unique().to_list(), set(bus_ids), context) + validate_refs(name_col, df[name_col].unique().to_list(), set(bus_ids), context) components: list[dict[str, Any]] = [] for row in df.iter_rows(named=True): component: dict[str, Any] = {k: v for k, v in row.items() if k != name_col} @@ -262,7 +263,7 @@ def _build_branch_components( .unique() .to_list() ) - _validate_refs( + validate_refs( f"{SiennaLineCol.BUS0}/{SiennaLineCol.BUS1}", referenced_buses, set(bus_ids), @@ -292,17 +293,3 @@ def _build_time_series_associations( .with_row_index(name=SiennaTimeSeriesAssociationCol.ID, offset=1) .iter_rows(named=True) ) - - -def _validate_refs( - ref_col: str, - needed: list[str], - available: set[str], - context: str, -) -> None: - missing = set(needed) - available - if missing: - raise ValueError( - f"{context}: {len(missing)} reference(s) in {ref_col!r} " - f"not found in parent table: {sorted(missing)}" - ) diff --git a/interop/plugins/steps/pypsa_to_sienna_investments_map_technologies.py b/interop/plugins/steps/pypsa_to_sienna_investments_map_technologies.py new file mode 100644 index 0000000..9d817a1 --- /dev/null +++ b/interop/plugins/steps/pypsa_to_sienna_investments_map_technologies.py @@ -0,0 +1,635 @@ +"""Turn what a PyPSA network says about its own expansion into a Sienna portfolio. + +The operations steps run first and write the base system, leaving out every component whose +capacity a build has yet to decide. This step reads the same network again and writes what +those steps had no home for: the candidates as technologies, the demand they have to meet, +the caps they run under, and the attributes tying each technology to the fleet the base +system already holds. +""" + +from __future__ import annotations + +from collections.abc import Callable, Mapping, Sequence +from functools import partial +from typing import ClassVar, Literal, NamedTuple + +import polars as pl +from pydantic import BaseModel, Field + +from interop.core.extensions import ( + ExtensionKind, + ExtensionReader, +) +from interop.core.pipeline import State, TranslationStep +from interop.core.reporting import ScopedRecorder +from interop.plugins.shared.pypsa_constants import ( + PYPSA_COMPONENT_NAMING, + PYPSA_NAME_COLUMN, + PyPSAComponent, + PyPSAComponentCol, + PyPSAGeneratorCol, + PyPSALoadCol, + PyPSAStorageUnitCol, + PyPSATable, +) +from interop.plugins.shared.pypsa_sienna_investments_translations import ( + AREA_NAME, + CARBON_CAP_SKIPS, + FOM_CHARGE_COL, + FUEL_COL, + LOAD_TYPE_COL, + POWER_SYSTEMS_TYPE_COL, + PRIME_MOVER_COL, + REGION_COL, + STORAGE_SKIPS, + SUPPLY_SKIPS, + TECHNICAL_LIFE_COL, + TECHNOLOGY_NAME, + TECHNOLOGY_TYPE, + UNIT_SIZE_COL, + CandidateTechnology, + build_association_rows, + build_carbon_cap_translations, + build_carbon_caps_source_table, + build_demand_translations, + build_existing_devices_translations, + build_existing_fleet_source_table, + build_financial_data_events, + build_portfolio_financial_data, + build_retirement_potential_translations, + build_scope_skips, + build_storage_technology_translations, + build_supply_translations, + build_topology_mapping_translations, + build_topology_source_table, + fill_storage_technology_defaults, + fill_supply_defaults, +) +from interop.plugins.shared.pypsa_sienna_user_mappings import CarrierMappings +from interop.plugins.shared.sienna_constants import ( + SIENNA_ID_COLUMN, + SIENNA_NAME_COLUMN, + SiennaACBusCol, + SiennaComponent, +) +from interop.plugins.shared.sienna_investments_constants import ( + CARBON_CAPS_DESTINATION_SCHEMA, + DEFAULT_BASE_YEAR, + DEMAND_REQUIREMENT_DESTINATION_SCHEMA, + EXISTING_DEVICES_DESTINATION_SCHEMA, + PORTFOLIO_FINANCIAL_DATA_TABLE, + RETIREMENT_POTENTIAL_DESTINATION_SCHEMA, + STORAGE_TECHNOLOGY_DESTINATION_SCHEMA, + SUPPLEMENTAL_ATTRIBUTE_ASSOCIATION_SCHEMA, + SUPPLEMENTAL_ATTRIBUTE_ASSOCIATIONS_TABLE, + SUPPLEMENTAL_ATTRIBUTE_ORDER, + SUPPLY_TECHNOLOGY_DESTINATION_SCHEMA, + TOPOLOGY_MAPPING_DESTINATION_SCHEMA, + SiennaInvestmentsComponent, + SiennaSupplementalAttribute, +) +from interop.plugins.shared.translation_runner import ( + SkipRule, + Translation, + apply_translations, + filter_component, + finalise, +) + +_GENERATOR_BASE_TYPES: tuple[SiennaComponent, ...] = ( + SiennaComponent.THERMAL_STANDARD, + SiennaComponent.RENEWABLE_DISPATCH, + SiennaComponent.RENEWABLE_NON_DISPATCH, + SiennaComponent.HYDRO_DISPATCH, +) + +# The base system types a PyPSA StorageUnit becomes, which is the storage type plus the hydro +# type the operations steps write a storage unit with an inflow as. +_STORAGE_BASE_TYPES: tuple[SiennaComponent, ...] = ( + SiennaComponent.ENERGY_RESERVOIR_STORAGE, + SiennaComponent.HYDRO_DISPATCH, +) + +# The base system types each PyPSA class alone writes, which a fleet is matched against. Only +# a StorageUnit becomes HydroDispatch, and a Generator may carry the name of a StorageUnit. +_GENERATOR_FLEET_TYPES: tuple[SiennaComponent, ...] = ( + SiennaComponent.THERMAL_STANDARD, + SiennaComponent.RENEWABLE_DISPATCH, + SiennaComponent.RENEWABLE_NON_DISPATCH, +) + +_BASE_LOAD_TYPES: tuple[SiennaComponent, ...] = ( + SiennaComponent.POWER_LOAD, + SiennaComponent.INTERRUPTIBLE_POWER_LOAD, +) + +# A PyPSA build year of zero is what the network writes when it states none. +_UNSTATED_BUILD_YEAR = 0 + +_Schema = dict[str, pl.DataType | type[pl.DataType]] +_FillDefaults = Callable[[pl.DataFrame], pl.DataFrame] +_BuildTranslations = Callable[[int], list[Translation]] +_CandidateTranslations = Callable[[int, int], list[Translation]] + + +class _CandidateKind(NamedTuple): + """Everything one candidate table does differently from the other.""" + + source_table: str + extendable_col: str + fill: _FillDefaults + skips: tuple[SkipRule, ...] + extension: Literal[ExtensionKind.GENERATOR, ExtensionKind.STORAGE] + build: _CandidateTranslations + schema: _Schema + component: SiennaInvestmentsComponent + base_types: tuple[SiennaComponent, ...] + fleet_types: tuple[SiennaComponent, ...] + device_class: str + build_year_col: str + + +_SUPPLY = _CandidateKind( + source_table=PyPSATable.GENERATORS, + extendable_col=PyPSAGeneratorCol.P_NOM_EXTENDABLE, + fill=fill_supply_defaults, + skips=SUPPLY_SKIPS, + extension=ExtensionKind.GENERATOR, + build=build_supply_translations, + schema=SUPPLY_TECHNOLOGY_DESTINATION_SCHEMA, + component=SiennaInvestmentsComponent.SUPPLY_TECHNOLOGY, + base_types=_GENERATOR_BASE_TYPES, + fleet_types=_GENERATOR_FLEET_TYPES, + device_class=PyPSAComponent.GENERATOR, + build_year_col=PyPSAGeneratorCol.BUILD_YEAR, +) + +_STORAGE = _CandidateKind( + source_table=PyPSATable.STORAGE_UNITS, + extendable_col=PyPSAStorageUnitCol.P_NOM_EXTENDABLE, + fill=fill_storage_technology_defaults, + skips=STORAGE_SKIPS, + extension=ExtensionKind.STORAGE, + build=build_storage_technology_translations, + schema=STORAGE_TECHNOLOGY_DESTINATION_SCHEMA, + component=SiennaInvestmentsComponent.STORAGE_TECHNOLOGY, + base_types=_STORAGE_BASE_TYPES, + fleet_types=_STORAGE_BASE_TYPES, + device_class=PyPSAComponent.STORAGE_UNIT, + build_year_col=PyPSAStorageUnitCol.BUILD_YEAR, +) + +# In the order they take their ids, which one counter hands out in turn. +_CANDIDATE_KINDS: tuple[_CandidateKind, ...] = (_SUPPLY, _STORAGE) + + +class _Attribute(NamedTuple): + schema: _Schema + build: _BuildTranslations + name_col: str + describes: Callable[[pl.DataFrame], list[str]] + + +# An attribute takes its id from its place in the flat array, so the step numbers the types in +# the order the sink writes them, SUPPLEMENTAL_ATTRIBUTE_ORDER. +_ATTRIBUTES: dict[SiennaSupplementalAttribute, _Attribute] = { + SiennaSupplementalAttribute.EXISTING_DEVICES: _Attribute( + schema=EXISTING_DEVICES_DESTINATION_SCHEMA, + build=build_existing_devices_translations, + name_col=TECHNOLOGY_NAME, + describes=lambda source: source[TECHNOLOGY_TYPE].to_list(), + ), + SiennaSupplementalAttribute.RETIREMENT_POTENTIAL: _Attribute( + schema=RETIREMENT_POTENTIAL_DESTINATION_SCHEMA, + build=build_retirement_potential_translations, + name_col=TECHNOLOGY_NAME, + describes=lambda source: source[TECHNOLOGY_TYPE].to_list(), + ), + SiennaSupplementalAttribute.TOPOLOGY_MAPPING: _Attribute( + schema=TOPOLOGY_MAPPING_DESTINATION_SCHEMA, + build=build_topology_mapping_translations, + name_col=AREA_NAME, + describes=lambda source: [str(SiennaComponent.AREA)] * source.height, + ), +} + + +class PypsaToSiennaInvestmentsMapTechnologiesParams(BaseModel): + base_year: int = Field( + default=DEFAULT_BASE_YEAR, + description=( + "the economic year every cost in the portfolio is quoted in; no PyPSA field states one" + ), + ) + + +class _CandidateScope(NamedTuple): + """What every candidate table reads off the base system, and the year its costs are in.""" + + area_by_bus: Mapping[str, str | None] + bus_names: Sequence[str] + base_year: int + + +class _Years(NamedTuple): + """The two ends of a base-system component's life, for the devices that state them.""" + + built: dict[str, int] + retired: dict[str, int] + + +class _Numbering: + """One id counter, handed to each table written under it in turn. + + An association names a component by id alone, and an attribute by id alone, so ids are + unique across a portfolio's components and across its flat attribute array rather than + within either's own type. + """ + + def __init__(self) -> None: + self.next_id = 1 + + +class PypsaToSiennaInvestmentsMapTechnologies(TranslationStep): + name: ClassVar[str] = "pypsa_to_sienna_investments_map_technologies" + params_schema: ClassVar[type[BaseModel] | None] = PypsaToSiennaInvestmentsMapTechnologiesParams + + def __init__(self, recorder: ScopedRecorder, carrier_mappings: CarrierMappings) -> None: + self._recorder = recorder + self._carrier_mappings = carrier_mappings + + def run(self, state: State, params: BaseModel | None) -> State: + base_year = ( + params.base_year + if isinstance(params, PypsaToSiennaInvestmentsMapTechnologiesParams) + else DEFAULT_BASE_YEAR + ) + buses = state.destination_tables.get(SiennaComponent.AC_BUS) + if buses is None: + return state + reader = state.extension_reader() + bus_names = buses[SiennaACBusCol.NAME].to_list() + area_by_bus = dict(zip(bus_names, buses[SiennaACBusCol.AREA].to_list(), strict=True)) + numbering = _Numbering() + scope = _CandidateScope(area_by_bus=area_by_bus, bus_names=bus_names, base_year=base_year) + candidates = [ + (kind, self._map_candidates(state, reader, scope, kind, numbering)) + for kind in _CANDIDATE_KINDS + ] + self._map_demand(state, area_by_bus, numbering) + self._map_carbon_caps(state, reader, numbering, candidates) + self._map_supplemental_attributes(state, buses, reader, scope, candidates) + state.destination_tables[PORTFOLIO_FINANCIAL_DATA_TABLE] = build_portfolio_financial_data( + base_year + ) + for event in build_financial_data_events(base_year): + self._recorder.append(event) + return state + + def _map_candidates( + self, + state: State, + reader: ExtensionReader, + scope: _CandidateScope, + kind: _CandidateKind, + numbering: _Numbering, + ) -> pl.DataFrame: + """The rows one candidate table was translated from, once the table is written.""" + table = self._candidate_rows(state, kind, scope.bus_names) + if table.is_empty(): + return table + lookup = reader.read(kind.extension) + names = table[PyPSAComponentCol.NAME].to_list() + table = self._enrich_candidate( + table, + scope.area_by_bus, + unit_sizes=[lookup.get(name).unit_size_mw for name in names], + technical_lives=[lookup.get(name).technical_life_years for name in names], + fom_charges=[lookup.get(name).fom_charge_per_mw_year for name in names], + ) + self._write_table( + state, + table, + partial(kind.build, scope.base_year), + kind.schema, + kind.component, + numbering, + ) + return table + + def _candidate_rows( + self, state: State, kind: _CandidateKind, bus_names: Sequence[str] + ) -> pl.DataFrame: + """The rows of one source table that state a build, less what cannot be translated. + + A component whose capacity the network fixes is not a candidate and belongs to the + base system alone, so it is filtered out silently rather than reported as dropped. + """ + src = state.source_topology.get(kind.source_table) + if src is None: + return pl.DataFrame() + table = kind.fill(src.collect()).filter(pl.col(kind.extendable_col)) + rules = [ + *build_scope_skips( + PYPSA_COMPONENT_NAMING[kind.source_table], + name_col=PyPSAComponentCol.NAME, + carrier_col=PyPSAComponentCol.CARRIER, + bus_col=PyPSAComponentCol.BUS, + carriers=sorted(self._carrier_mappings.get_carriers()), + translated_carriers=sorted(self._translated_carriers(kind)), + bus_names=bus_names, + ), + *kind.skips, + ] + for rule in rules: + table, _ = filter_component(table, rule.keep, rule.report, self._recorder) + return table + + def _translated_carriers(self, kind: _CandidateKind) -> set[str]: + """The carriers the mappings file sends to a type this kind of candidate becomes.""" + carriers: set[str] = set() + for component in kind.base_types: + carriers |= self._carrier_mappings.get_carriers(component) + return carriers + + def _enrich_candidate( + self, + table: pl.DataFrame, + area_by_bus: Mapping[str, str | None], + *, + unit_sizes: Sequence[float | None], + technical_lives: Sequence[float | None], + fom_charges: Sequence[float | None], + ) -> pl.DataFrame: + """Add what the carrier, the bus and the sidecar say about a candidate.""" + component_types = { + mapping.pypsa_carrier: str(mapping.sienna_component_type) + for mapping in self._carrier_mappings.carriers + } + prime_movers = { + carrier: str(mover) + for carrier, mover in self._carrier_mappings.get_prime_mover_map().items() + } + fuels = { + carrier: str(target[0]) + for carrier, target in self._carrier_mappings.get_thermal_carrier_map().items() + } + carrier_col = PyPSAComponentCol.CARRIER + return table.with_columns( + _looked_up(carrier_col, component_types, POWER_SYSTEMS_TYPE_COL), + _looked_up(carrier_col, prime_movers, PRIME_MOVER_COL), + _looked_up(carrier_col, fuels, FUEL_COL), + _looked_up(PyPSAComponentCol.BUS, dict(area_by_bus), REGION_COL), + pl.Series(UNIT_SIZE_COL, list(unit_sizes), dtype=pl.Float64), + pl.Series(TECHNICAL_LIFE_COL, list(technical_lives), dtype=pl.Float64), + pl.Series(FOM_CHARGE_COL, list(fom_charges), dtype=pl.Float64), + ) + + def _write_table( + self, + state: State, + table: pl.DataFrame, + build: _BuildTranslations, + schema: _Schema, + component: str, + numbering: _Numbering, + ) -> pl.DataFrame: + """Translate one source table into a destination table, and take its share of the ids.""" + out = finalise( + apply_translations(table, build(numbering.next_id), self._recorder), + schema, + self._recorder, + component, + ) + state.destination_tables[component] = out + numbering.next_id += out.height + return out + + def _map_demand( + self, state: State, area_by_bus: Mapping[str, str | None], numbering: _Numbering + ) -> None: + """One requirement per load the base system holds, named as the type it holds it as.""" + src = state.source_topology.get(PyPSATable.LOADS) + if src is None: + return + load_types = _base_load_types(state) + table = src.collect().filter(pl.col(PyPSALoadCol.NAME).is_in(list(load_types))) + if table.is_empty(): + return + table = table.with_columns( + _looked_up(PyPSALoadCol.BUS, dict(area_by_bus), REGION_COL), + _looked_up(PyPSALoadCol.NAME, load_types, LOAD_TYPE_COL), + ) + self._write_table( + state, + table, + build_demand_translations, + DEMAND_REQUIREMENT_DESTINATION_SCHEMA, + SiennaInvestmentsComponent.DEMAND_REQUIREMENT, + numbering, + ) + + def _map_carbon_caps( + self, + state: State, + reader: ExtensionReader, + numbering: _Numbering, + candidates: Sequence[tuple[_CandidateKind, pl.DataFrame]], + ) -> None: + records = reader.read(ExtensionKind.CONSTRAINT).read_all() + if not records: + return + table = build_carbon_caps_source_table(records, _model_components(state, candidates)) + for rule in CARBON_CAP_SKIPS: + table, _ = filter_component(table, rule.keep, rule.report, self._recorder) + if table.is_empty(): + return + self._write_table( + state, + table, + build_carbon_cap_translations, + CARBON_CAPS_DESTINATION_SCHEMA, + SiennaInvestmentsComponent.CARBON_CAPS, + numbering, + ) + + def _map_supplemental_attributes( + self, + state: State, + buses: pl.DataFrame, + reader: ExtensionReader, + scope: _CandidateScope, + candidates: Sequence[tuple[_CandidateKind, pl.DataFrame]], + ) -> None: + """The three attributes, numbered from the one counter the flat array shares.""" + fleet = self._build_fleet(state, reader, scope, candidates) + sources: dict[SiennaSupplementalAttribute, pl.DataFrame] = { + SiennaSupplementalAttribute.EXISTING_DEVICES: fleet, + SiennaSupplementalAttribute.RETIREMENT_POTENTIAL: fleet, + SiennaSupplementalAttribute.TOPOLOGY_MAPPING: build_topology_source_table(buses), + } + associations: list[pl.DataFrame] = [] + numbering = _Numbering() + for kind in SUPPLEMENTAL_ATTRIBUTE_ORDER: + attribute = _ATTRIBUTES[kind] + source = sources[kind] + if source.is_empty(): + continue + out = self._write_table( + state, source, attribute.build, attribute.schema, kind, numbering + ) + associations.append( + build_association_rows( + kind, + out[SIENNA_ID_COLUMN].to_list(), + source[attribute.name_col].to_list(), + attribute.describes(source), + ) + ) + state.destination_tables[SUPPLEMENTAL_ATTRIBUTE_ASSOCIATIONS_TABLE] = ( + pl.concat(associations) + if associations + else pl.DataFrame(schema=SUPPLEMENTAL_ATTRIBUTE_ASSOCIATION_SCHEMA) + ) + + def _build_fleet( + self, + state: State, + reader: ExtensionReader, + scope: _CandidateScope, + candidates: Sequence[tuple[_CandidateKind, pl.DataFrame]], + ) -> pl.DataFrame: + """The base-system devices each technology stands for, and the years they state.""" + fleets: list[pl.DataFrame] = [] + for kind, found in candidates: + source = state.source_topology.get(kind.source_table) + devices = pl.DataFrame() if source is None else source.collect() + years = self._years(devices, reader, kind) + fleets.append( + build_existing_fleet_source_table( + _technologies(found, kind.component, kind.device_class), + _fleet_groups(devices, _base_names(state, kind.fleet_types), scope.area_by_bus), + years.built, + years.retired, + ) + ) + return pl.concat(fleets) + + @staticmethod + def _years(devices: pl.DataFrame, reader: ExtensionReader, kind: _CandidateKind) -> _Years: + """Each device's build year, from the network, and its retirement year, from the sidecar. + + PyPSA carries a build year on the component and nothing for the other end of a life, + which is why one of the pair comes off the sidecar. A generator and a storage unit may + share a name, so each source table is read on its own. + """ + built: dict[str, int] = {} + retired: dict[str, int] = {} + if devices.is_empty(): + return _Years(built=built, retired=retired) + lookup = reader.read(kind.extension) + has_build_year = kind.build_year_col in devices.columns + for row in devices.iter_rows(named=True): + name = row[PYPSA_NAME_COLUMN] + year = row[kind.build_year_col] if has_build_year else None + if year is not None and int(year) != _UNSTATED_BUILD_YEAR: + built[name] = int(year) + retirement_year = lookup.get(name).retirement_year + if retirement_year is not None: + retired[name] = retirement_year + return _Years(built=built, retired=retired) + + +def _looked_up(name_col: str, values: Mapping[str, str | None], dest_col: str) -> pl.Expr: + """What a mapping says about each row, null where the mapping names no such row.""" + return ( + pl.col(name_col) + .replace_strict(dict(values), default=None, return_dtype=pl.Utf8) + .alias(dest_col) + ) + + +def _model_components( + state: State, candidates: Sequence[tuple[_CandidateKind, pl.DataFrame]] +) -> set[tuple[str, str]]: + """Every component of the network a constraint could weight, with the class that wrote it. + + The devices the base system holds, and the candidates the portfolio holds. A generator + neither document carries emits nothing a cap could bound, so a constraint reaches the + whole model without naming it. Each name travels with its PyPSA class, because a + constraint can weight an object of another class that carries the same name. + """ + components: set[tuple[str, str]] = set() + for kind, found in candidates: + names = set(_base_names(state, kind.fleet_types)) + if not found.is_empty(): + names |= set(found[PyPSAComponentCol.NAME].to_list()) + components |= {(name, kind.device_class) for name in names} + return components + + +def _base_load_types(state: State) -> dict[str, str]: + """The Sienna type the base system wrote each load as, by load name.""" + types: dict[str, str] = {} + for component in _BASE_LOAD_TYPES: + table = state.destination_tables.get(component) + if table is not None: + types |= {name: str(component) for name in table[SIENNA_NAME_COLUMN].to_list()} + return types + + +def _base_names(state: State, types: tuple[SiennaComponent, ...]) -> set[str]: + """Every component of the given types the base system holds.""" + names: set[str] = set() + for key in types: + table = state.destination_tables.get(key) + if table is not None: + names |= set(table[SIENNA_NAME_COLUMN].to_list()) + return names + + +def _fleet_groups( + devices: pl.DataFrame, + in_base_system: set[str], + area_by_bus: Mapping[str, str | None], +) -> dict[tuple[str, str | None], list[str]]: + """The base-system components of each carrier and area, in the order the network states them. + + A technology sits in one region, so a device of its carrier in another region is not a + plant it adds to and not a plant a build of it may retire. + """ + groups: dict[tuple[str, str | None], list[str]] = {} + if devices.is_empty(): + return groups + for name, carrier, bus in zip( + devices[PYPSA_NAME_COLUMN].to_list(), + devices[PyPSAComponentCol.CARRIER].to_list(), + devices[PyPSAComponentCol.BUS].to_list(), + strict=True, + ): + if name in in_base_system: + groups.setdefault((carrier, area_by_bus.get(bus)), []).append(name) + return groups + + +def _technologies( + candidates: pl.DataFrame, component: SiennaInvestmentsComponent, device_class: str +) -> list[CandidateTechnology]: + """Each translated technology, beside the fleet its base-system devices make up.""" + if candidates.is_empty(): + return [] + return [ + CandidateTechnology( + name=name, + component_type=str(component), + device_class=device_class, + carrier=carrier, + region=region, + ) + for name, carrier, region in zip( + candidates[PyPSAComponentCol.NAME].to_list(), + candidates[PyPSAComponentCol.CARRIER].to_list(), + candidates[REGION_COL].to_list(), + strict=True, + ) + ] diff --git a/libs/interop-testing/src/interop_testing/builders/sienna_documents.py b/libs/interop-testing/src/interop_testing/builders/sienna_documents.py index 0969ac3..1ff4de2 100644 --- a/libs/interop-testing/src/interop_testing/builders/sienna_documents.py +++ b/libs/interop-testing/src/interop_testing/builders/sienna_documents.py @@ -300,3 +300,24 @@ def sienna_time_series_uuid(sienna_type: str, name: str, attribute: str) -> str: would pass however the scheme changed. Keep the two in sync by hand. """ return str(uuid.uuid5(_TIME_SERIES_UUID_NAMESPACE, f"ts.{sienna_type}.{name}.{attribute}")) + + +def portfolio_attributes_for( + data: dict[str, Any], attribute_type: str, component_type: str, component_id: int +) -> list[dict[str, Any]]: + """The supplemental attributes of one type describing one component, by id. + + The document links the two through `supplemental_attribute_associations`, so this walks + the same route a consumer does: the association carrying the component id and the + attribute type, then the attribute of that id in the flat array. An attribute may + describe a component of the base system rather than of the portfolio, which is why the + id is the argument. + """ + attribute_ids = { + row["attribute_id"] + for row in data.get("supplemental_attribute_associations", []) + if row["component_id"] == component_id + and row["component_type"] == component_type + and row["attribute_type"] == attribute_type + } + return [a for a in data.get("supplemental_attributes", []) if a["id"] in attribute_ids] diff --git a/libs/interop-testing/src/interop_testing/steps/__init__.py b/libs/interop-testing/src/interop_testing/steps/__init__.py index b962beb..ae681f9 100644 --- a/libs/interop-testing/src/interop_testing/steps/__init__.py +++ b/libs/interop-testing/src/interop_testing/steps/__init__.py @@ -16,6 +16,7 @@ "interop_testing.steps.pypsa_network", "interop_testing.steps.caiso_stack_model", "interop_testing.steps.sienna_system", + "interop_testing.steps.sienna_portfolio", "interop_testing.steps.sienna_results", "interop_testing.steps.plexos_model", "interop_testing.steps.plexos_resources", diff --git a/libs/interop-testing/src/interop_testing/steps/pypsa_network/build_network.py b/libs/interop-testing/src/interop_testing/steps/pypsa_network/build_network.py index cd51d1c..63f1501 100644 --- a/libs/interop-testing/src/interop_testing/steps/pypsa_network/build_network.py +++ b/libs/interop-testing/src/interop_testing/steps/pypsa_network/build_network.py @@ -222,7 +222,7 @@ def given_network_contains_storage_unit( p_min_pu_m = re.search(r"p_min_pu (-?[\d.]+)", rest) p_max_pu_m = re.search(r"p_max_pu ([\d.]+)", rest) marginal_cost_m = re.search(r"marginal_cost ([\d.]+)", rest) - max_hours_m = re.search(r"max_hours ([\d.]+)", rest) + max_hours_m = re.search(r"max_hours ([\d.]+|inf)", rest) eff_store_m = re.search(r"efficiency_store ([\d.]+)", rest) eff_dispatch_m = re.search(r"efficiency_dispatch ([\d.]+)", rest) soc_initial_m = re.search(r"state_of_charge_initial ([\d.]+)", rest) diff --git a/libs/interop-testing/src/interop_testing/steps/sienna_portfolio.py b/libs/interop-testing/src/interop_testing/steps/sienna_portfolio.py new file mode 100644 index 0000000..db387d2 --- /dev/null +++ b/libs/interop-testing/src/interop_testing/steps/sienna_portfolio.py @@ -0,0 +1,77 @@ +"""pytest-bdd vocabulary for a Sienna investments portfolio a pipeline wrote. + +A portfolio groups its components by type name the way a system does, but it also carries a +flat array of supplemental attributes linked to those components by an association table. +""" + +from __future__ import annotations + +import json + +from pytest_bdd import parsers, then + +from interop_testing.builders.sienna_documents import ( + find_sienna_component, + portfolio_attributes_for, + sienna_components_of_type, +) +from interop_testing.files import navigate_json, read_json + + +@then( + parsers.parse( + 'the file "{path}" parses as a portfolio with no component "{sienna_type}" named "{name}"' + ) +) +def assert_portfolio_component_absent(path: str, sienna_type: str, name: str) -> None: + components = sienna_components_of_type(read_json(path), sienna_type) + matching = [c for c in components if c.get("name") == name] + assert not matching, ( + f"expected no portfolio component type={sienna_type!r} name={name!r} in {path}, " + f"got {matching!r}" + ) + + +@then( + parsers.parse( + 'the file "{path}" parses as a portfolio where the "{attribute_type}" of' + ' "{component_type}" "{component_name}" has "{field_path}" set to {value}' + ) +) +def assert_portfolio_attribute_field( + path: str, + attribute_type: str, + component_type: str, + component_name: str, + field_path: str, + value: str, +) -> None: + component_id = find_sienna_component(read_json(path), component_type, component_name)["id"] + assert_base_system_attribute_field( + path, attribute_type, component_type, component_id, field_path, value + ) + + +@then( + parsers.parse( + 'the file "{path}" parses as a portfolio where the "{attribute_type}" of base system' + ' "{component_type}" {component_id:d} has "{field_path}" set to {value}' + ) +) +def assert_base_system_attribute_field( + path: str, + attribute_type: str, + component_type: str, + component_id: int, + field_path: str, + value: str, +) -> None: + """An attribute describing a component of the base system, which the portfolio names by id.""" + expected = json.loads(value) + matching = portfolio_attributes_for( + read_json(path), attribute_type, component_type, component_id + ) + context = f"{path}[{attribute_type} of {component_type} {component_id}]" + assert len(matching) == 1, f"expected 1 {context}, got {len(matching)}: {matching!r}" + actual = navigate_json(matching[0], field_path, context) + assert actual == expected, f"expected {context}.{field_path!r} = {expected!r}, got {actual!r}" diff --git a/libs/interop-testing/src/interop_testing/steps/sienna_system.py b/libs/interop-testing/src/interop_testing/steps/sienna_system.py index c5ce6d6..0629c1b 100644 --- a/libs/interop-testing/src/interop_testing/steps/sienna_system.py +++ b/libs/interop-testing/src/interop_testing/steps/sienna_system.py @@ -448,16 +448,19 @@ def assert_sienna_component_field( @then( parsers.parse( 'the file "{path}" parses as JSON with component "{sienna_type}" named "{name}"' - ' without field "{field}"' + ' without field "{field_path}"' ) ) def assert_sienna_component_field_absent( - path: str, sienna_type: str, name: str, field: str -) -> None: - data = read_json(path) - component = find_sienna_component(data, sienna_type, name) - assert field not in component, ( - f"expected field {field!r} absent from [{sienna_type}:{name}] in {path}, got {component!r}" + path: str, sienna_type: str, name: str, field_path: str +) -> None: + """A field the component does not hold, named by a dotted path into its nested structs.""" + component = find_sienna_component(read_json(path), sienna_type, name) + context = f"{path}[{sienna_type}:{name}]" + parent_path, _, field = field_path.rpartition(".") + holder = navigate_json(component, parent_path, context) if parent_path else component + assert field not in holder, ( + f"expected field {field_path!r} absent from {context}, got {holder!r}" ) diff --git a/tests/features/plexos_to_sienna_investments.feature b/tests/features/plexos_to_sienna_investments.feature new file mode 100644 index 0000000..396352b --- /dev/null +++ b/tests/features/plexos_to_sienna_investments.feature @@ -0,0 +1,134 @@ +@slow @fork_unsafe +Feature: a PLEXOS expansion plan becomes a Sienna investments portfolio + plexos-to-sienna-investments is a composed pipeline, not a translator of its own: it runs + plexos-to-pypsa and then pypsa-to-sienna-investments over the network the first leg wrote. + The second leg writes two documents, the base system holding the fleet that already runs + and the portfolio holding what the plan may build beside it. + + An object stating Max Units Built may be built, and each one becomes a technology named + after it. What building it costs travels as PyPSA's own expansion fields; the size of one + unit and the Technical Life have no PyPSA column, so both cross the hub in the extensions + sidecar and are read back here. + + Scenario: a candidate generator becomes a SupplyTechnology + Given a Plexos model + And the model states "Build Cost" in "$/kW" + And the model states "FO&M Charge" in "$/kW/yr" + And the model states "WACC" in "%" + And the model contains region "North" + And the model contains node "North_Node" in region "North" + And the model contains generator "REZ_Solar" with "node=North_Node, category=Solar, Max Capacity=100, Units=0, Max Units Built=5, Build Cost=1200, WACC=7, Economic Life=25, Technical Life=30, FO&M Charge=15" + And the model is saved as "inputs/model.xml" + And a PLEXOS mappings file: + | plexos_concept | plexos_name | sienna_component_type | sienna_fuel_type | sienna_prime_mover_type | + | category | Solar | RenewableDispatch | | PVe | + When I run the plexos-to-sienna-investments chain against "inputs/model.xml" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as valid JSON + And the file "outputs/portfolio.json" parses as JSON with 1 component of type "SupplyTechnology" + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "power_systems_type" set to "RenewableDispatch" + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "prime_mover_type" set to "PVe" + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "available" set to true + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "region" set to [1] + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "capacity_limits.min" set to 0.0 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "capacity_limits.max" set to 500.0 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "capital_costs.capital_cost.function_data.proportional_term" set to 1200000.0 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "operation_costs.fixed" set to 15000.0 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "unit_size" set to 100.0 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "lifetime" set to 30 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "financial_data.capital_recovery_period" set to 25 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "financial_data.return_on_equity" set to 0.07 + And the file "outputs/system.json" parses as JSON with 0 components of type "RenewableDispatch" + And the file "outputs/system.json" parses as JSON with 1 component of type "Area" + + Scenario: a candidate Battery becomes a StorageTechnology + Given a Plexos model + And the model contains region "North" + And the model contains node "North_Node" in region "North" + And the model contains battery "NewBattery" on node "North_Node" with max_power 50 capacity 200 charge_efficiency 90 initial_soc 50 + And battery "NewBattery" has property "Units" 0 + And battery "NewBattery" has property "Max Units Built" 6 + And battery "NewBattery" has property "Build Cost" 800000 + And battery "NewBattery" has property "WACC" 0.07 + And battery "NewBattery" has property "Economic Life" 20 + And battery "NewBattery" has property "Technical Life" 15 + And the model is saved as "inputs/battery.xml" + And a PLEXOS mappings file: + | plexos_concept | plexos_name | sienna_component_type | sienna_fuel_type | sienna_prime_mover_type | + | category | Solar | RenewableDispatch | | PVe | + When I run the plexos-to-sienna-investments chain against "inputs/battery.xml" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as JSON with 1 component of type "StorageTechnology" + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "power_systems_type" set to "EnergyReservoirStorage" + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "prime_mover_type" set to "BA" + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "storage_tech" set to "OTHER_MECH" + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "capacity_limits_discharge.max" set to 300.0 + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "capacity_limits_energy.max" set to 1200.0 + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "capital_costs.discharge_capital_cost.function_data.proportional_term" set to 800000.0 + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "unit_size_discharge" set to 50.0 + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "lifetime" set to 15 + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "financial_data.capital_recovery_period" set to 20 + + Scenario: a plant that already runs is the fleet its technology adds to + Given a Plexos model + And the model contains region "North" + And the model contains node "North_Node" in region "North" + And the model contains generator "OldSolar" with "node=North_Node, category=Solar, Max Capacity=80, Units=1" + And the model contains generator "REZ_Solar" with "node=North_Node, category=Solar, Max Capacity=100, Units=0, Max Units Built=5, Build Cost=1200000, WACC=0.07, Economic Life=25" + And the model is saved as "inputs/fleet.xml" + And a PLEXOS mappings file: + | plexos_concept | plexos_name | sienna_component_type | sienna_fuel_type | sienna_prime_mover_type | + | category | Solar | RenewableDispatch | | PVe | + When I run the plexos-to-sienna-investments chain against "inputs/fleet.xml" writing "outputs/portfolio.json" + Then the file "outputs/system.json" parses as JSON with 1 component of type "RenewableDispatch" + And the file "outputs/portfolio.json" parses as a portfolio where the "ExistingDevices" of "SupplyTechnology" "REZ_Solar" has "existing_devices" set to ["OldSolar"] + And the file "outputs/portfolio.json" parses as a portfolio where the "RetirementPotential" of "SupplyTechnology" "REZ_Solar" has "eligible_generators" set to ["OldSolar"] + + Scenario: a Constraint over the whole model becomes a CarbonCaps + Given a Plexos model + And the model contains region "North" + And the model contains node "North_Node" in region "North" + And the model contains fuel "Natural Gas" with price 3 + And the model contains generator "GasPlant" with "node=North_Node, fuel=Natural Gas, Max Capacity=500, Heat Rate=9, Units=1" + And the model contains generator "REZ_Solar" with "node=North_Node, category=Solar, Max Capacity=100, Units=0, Max Units Built=5, Build Cost=1200000, WACC=0.07, Economic Life=25" + And the model contains constraint "CarbonBudget" over: + | class | name | coefficient property | coefficient | + | Generator | GasPlant | Generation Coefficient | 1 | + | Generator | REZ_Solar | Generation Coefficient | 1 | + And constraint "CarbonBudget" states "Sense" of -1 + And constraint "CarbonBudget" states "RHS Year" of 20 + And the model is saved as "inputs/cap.xml" + And a PLEXOS mappings file: + | plexos_concept | plexos_name | sienna_component_type | sienna_fuel_type | sienna_prime_mover_type | + | fuel | Natural Gas | ThermalStandard | NATURAL_GAS | CC | + | category | Solar | RenewableDispatch | | PVe | + When I run the plexos-to-sienna-investments chain against "inputs/cap.xml" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as JSON with 1 component of type "CarbonCaps" + And the file "outputs/portfolio.json" parses as JSON with component "CarbonCaps" named "CarbonBudget" having "max_mtons" set to 20.0 + And the file "outputs/portfolio.json" parses as JSON with component "CarbonCaps" named "CarbonBudget" having "available" set to true + And the file "outputs/portfolio.json" parses as JSON with component "CarbonCaps" named "CarbonBudget" without field "target_year" + And the file "decisions.md" contains "the year right-hand side of the constraint" + + Scenario: a Constraint over part of the model is left out, and the run completes + CarbonCaps names no members and no region, so a cap written from a constraint over some + of the model would hold all of it. + Given a Plexos model + And the model contains region "North" + And the model contains node "North_Node" in region "North" + And the model contains fuel "Natural Gas" with price 3 + And the model contains generator "GasPlant" with "node=North_Node, fuel=Natural Gas, Max Capacity=500, Heat Rate=9, Units=1" + And the model contains generator "REZ_Solar" with "node=North_Node, category=Solar, Max Capacity=100, Units=0, Max Units Built=5, Build Cost=1200000, WACC=0.07, Economic Life=25" + And the model contains constraint "GasCap" over: + | class | name | coefficient property | coefficient | + | Generator | GasPlant | Generation Coefficient | 1 | + And constraint "GasCap" states "Sense" of -1 + And constraint "GasCap" states "RHS Year" of 5 + And the model is saved as "inputs/scoped.xml" + And a PLEXOS mappings file: + | plexos_concept | plexos_name | sienna_component_type | sienna_fuel_type | sienna_prime_mover_type | + | fuel | Natural Gas | ThermalStandard | NATURAL_GAS | CC | + | category | Solar | RenewableDispatch | | PVe | + When I run the plexos-to-sienna-investments chain against "inputs/scoped.xml" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as JSON with 0 components of type "CarbonCaps" + And the file "outputs/portfolio.json" parses as JSON with 1 component of type "SupplyTechnology" + And the file "decisions.md" contains "`pypsa.constraint.GasCap`" + And the file "decisions.md" contains "CarbonCaps names no members and no region" + And the log contains "1 constraint(s) weight a named subset of the model rather than all of it, so each is left out" diff --git a/tests/features/pypsa_to_sienna_investments/portfolio.feature b/tests/features/pypsa_to_sienna_investments/portfolio.feature new file mode 100644 index 0000000..9ff917c --- /dev/null +++ b/tests/features/pypsa_to_sienna_investments/portfolio.feature @@ -0,0 +1,531 @@ +@slow @fork_unsafe +Feature: a PyPSA network that states its own expansion becomes a Sienna portfolio + pypsa-to-sienna-investments runs the operations steps and the investments step over one + network and writes two documents: the base system, holding the fleet that already runs, + and the portfolio, holding what a plan may build beside it. + + A component whose capacity a plan may change becomes a technology named after it. What the + plan may build is the gap between p_nom_min and p_nom_max; what building it costs is the + overnight cost, the fixed O&M, and the discount rate and lifetime PyPSA annuitises them + with. The size of one unit and the technical life have no PyPSA column, so both reach the + portfolio through the extensions sidecar. + + Scenario: an extendable generator becomes a SupplyTechnology + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains generator "REZ_Solar" on "North_bus" carrier "solar" p_nom 0 p_nom_extendable True + And generator "REZ_Solar" has p_nom_min 0 + And generator "REZ_Solar" has p_nom_max 500 + And generator "REZ_Solar" has overnight_cost 1200000 + And generator "REZ_Solar" has discount_rate 0.07 + And generator "REZ_Solar" has lifetime 25 + And generator "REZ_Solar" has fom_cost 15000 + And the network is saved as "inputs/candidate.nc" + And a file "inputs/extensions.json" containing the lines: + | line | + | { | + | "generator": [ | + | { | + | "name": "REZ_Solar", | + | "unit_size_mw": 100.0, | + | "technical_life_years": 30.0 | + | } | + | ] | + | } | + When I run the pypsa investments translation with sidecar "inputs/extensions.json" against "inputs/candidate.nc" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as JSON with 1 component of type "SupplyTechnology" + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "id" set to 1 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "available" set to true + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "power_systems_type" set to "RenewableDispatch" + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "prime_mover_type" set to "PVe" + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "region" set to [1] + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "capacity_limits.min" set to 0.0 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "capacity_limits.max" set to 500.0 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "capital_costs.capital_cost.function_data.proportional_term" set to 1200000.0 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "capital_costs.interconnection_cost" set to 0.0 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "operation_costs.cost_type" set to "RENEWABLE" + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "operation_costs.fixed" set to 15000.0 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" without field "operation_costs.start_up" + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" without field "operation_costs.shut_down" + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "unit_size" set to 100.0 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "lifetime" set to 30 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "financial_data.capital_recovery_period" set to 25 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "financial_data.return_on_equity" set to 0.07 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "financial_data.debt_fraction" set to 0.0 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "financial_data.debt_rate" set to 0.0 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "financial_data.tax_rate" set to 0.0 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "financial_data.technology_base_year" set to 2020 + # A carrier the mappings file sends to a renewable type names no fuel, and no PyPSA field + # states a ramp for a technology, so neither is written at all. + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" without field "fuel" + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" without field "ramp_limits" + # The build the plan has yet to decide is not a component of the operations system. + And the file "outputs/system.json" parses as JSON with 0 components of type "RenewableDispatch" + And the file "decisions.md" contains "`pypsa.Generator.REZ_Solar.discount_rate` = 0.07" + And the file "decisions.md" contains "with no debt the weighted average cost of capital equals the return on equity" + + Scenario: the portfolio names the base system it expands, its regions and its demand + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains load "North_load" on "North_bus" with static p_set 100 + And the network contains generator "REZ_Wind" on "North_bus" carrier "onwind" p_nom 0 p_nom_extendable True + And generator "REZ_Wind" has p_nom_max 300 + And generator "REZ_Wind" has overnight_cost 900000 + And generator "REZ_Wind" has discount_rate 0.06 + And generator "REZ_Wind" has lifetime 20 + And the network is saved as "inputs/regions.nc" + When I run the pypsa investments translation against "inputs/regions.nc" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as JSON with "aggregation" set to "Area" + And the file "outputs/portfolio.json" parses as JSON with "base_system_file" set to "system.json" + And the file "outputs/portfolio.json" parses as JSON with "time_series_storage_file" set to "system_time_series_storage.h5" + And the file "outputs/portfolio.json" parses as JSON with "financial_data.base_year" set to 2020 + And the file "outputs/portfolio.json" parses as JSON with "financial_data.discount_rate" set to 0.0 + And the file "outputs/portfolio.json" parses as JSON with 1 component of type "DemandRequirement" + And the file "outputs/portfolio.json" parses as JSON with component "DemandRequirement" named "North_load" having "power_systems_type" set to "PowerLoad" + And the file "outputs/portfolio.json" parses as JSON with component "DemandRequirement" named "North_load" having "region" set to [1] + And the file "outputs/portfolio.json" parses as a portfolio where the "TopologyMapping" of base system "Area" 1 has "buses" set to ["North_bus"] + And the file "decisions.md" contains "the buses of the base system that sit in this area" + + Scenario: an existing plant of the same carrier is the technology's fleet + A technology stands for more of what a carrier already runs, so the base system components + of that carrier are the devices it may add to and the devices a plan may retire. + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains generator "OldSolar" on "North_bus" carrier "solar" p_nom 200 + And generator "OldSolar" has build_year 1995 + And the network contains generator "REZ_Solar" on "North_bus" carrier "solar" p_nom 0 p_nom_extendable True + And generator "REZ_Solar" has p_nom_max 500 + And generator "REZ_Solar" has overnight_cost 1200000 + And generator "REZ_Solar" has discount_rate 0.07 + And generator "REZ_Solar" has lifetime 25 + And the network is saved as "inputs/fleet.nc" + And a file "inputs/extensions.json" containing the lines: + | line | + | { | + | "generator": [ | + | { | + | "name": "OldSolar", | + | "retirement_year": 2040 | + | } | + | ] | + | } | + When I run the pypsa investments translation with sidecar "inputs/extensions.json" against "inputs/fleet.nc" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as a portfolio where the "ExistingDevices" of "SupplyTechnology" "REZ_Solar" has "existing_devices" set to ["OldSolar"] + And the file "outputs/portfolio.json" parses as a portfolio where the "RetirementPotential" of "SupplyTechnology" "REZ_Solar" has "eligible_generators" set to ["OldSolar"] + And the file "outputs/portfolio.json" parses as a portfolio where the "RetirementPotential" of "SupplyTechnology" "REZ_Solar" has "build_year.OldSolar" set to 1995 + And the file "outputs/portfolio.json" parses as a portfolio where the "RetirementPotential" of "SupplyTechnology" "REZ_Solar" has "planned_retirement_year.OldSolar" set to 2040 + And the file "outputs/portfolio.json" parses as a portfolio where the "RetirementPotential" of "SupplyTechnology" "REZ_Solar" has "retirement_cost.function_data.proportional_term" set to 0.0 + And the file "outputs/system.json" parses as JSON with 1 component of type "RenewableDispatch" + + Scenario: an extendable storage unit becomes a StorageTechnology + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains storage unit "NewBattery" on "North_bus" carrier "PHS" p_nom 0 max_hours 4.0 efficiency_store 0.9 efficiency_dispatch 0.95 p_nom_extendable True + And storage unit "NewBattery" has p_nom_max 300 + And storage unit "NewBattery" has overnight_cost 800000 + And storage unit "NewBattery" has discount_rate 0.07 + And storage unit "NewBattery" has lifetime 20 + And storage unit "NewBattery" has fom_cost 12000 + And the network is saved as "inputs/battery.nc" + And a file "inputs/extensions.json" containing the lines: + | line | + | { | + | "storage": [ | + | { | + | "name": "NewBattery", | + | "unit_size_mw": 50.0, | + | "technical_life_years": 15.0 | + | } | + | ] | + | } | + When I run the pypsa investments translation with sidecar "inputs/extensions.json" against "inputs/battery.nc" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as JSON with 1 component of type "StorageTechnology" + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "power_systems_type" set to "EnergyReservoirStorage" + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "storage_tech" set to "OTHER_MECH" + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "prime_mover_type" set to "PS" + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "capacity_limits_discharge.max" set to 300.0 + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "capacity_limits_energy.max" set to 1200.0 + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "capital_costs.discharge_capital_cost.function_data.proportional_term" set to 800000.0 + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "capital_costs.charge_capital_cost.function_data.proportional_term" set to 0.0 + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "operation_costs.cost_type" set to "STORAGE" + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "operation_costs.fixed" set to 12000.0 + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "unit_size_discharge" set to 50.0 + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "efficiency.in" set to 0.9 + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "efficiency.out" set to 0.95 + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "lifetime" set to 15 + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "financial_data.capital_recovery_period" set to 20 + + Scenario: a candidate that puts no ceiling on its build is left out, and the run completes + A technology states the most capacity it may hold, and a p_nom_max PyPSA left at infinity + is no such number. + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains generator "Unbounded_REZ" on "North_bus" carrier "solar" p_nom 0 p_nom_extendable True + And generator "Unbounded_REZ" has overnight_cost 1200000 + And generator "Unbounded_REZ" has discount_rate 0.07 + And generator "Unbounded_REZ" has lifetime 25 + And the network contains generator "Bounded_REZ" on "North_bus" carrier "solar" p_nom 0 p_nom_extendable True + And generator "Bounded_REZ" has p_nom_max 400 + And generator "Bounded_REZ" has overnight_cost 1200000 + And generator "Bounded_REZ" has discount_rate 0.07 + And generator "Bounded_REZ" has lifetime 25 + And the network is saved as "inputs/unbounded.nc" + When I run the pypsa investments translation against "inputs/unbounded.nc" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as JSON with 1 component of type "SupplyTechnology" + And the file "outputs/portfolio.json" parses as a portfolio with no component "SupplyTechnology" named "Unbounded_REZ" + And the file "decisions.md" contains "`pypsa.Generator.Unbounded_REZ.p_nom_max`" + And the file "decisions.md" contains "p_nom_max is not a finite number of MW" + And the log contains "1 Generator(s) are extendable and put no upper bound on the capacity a build may add" + + Scenario: a candidate that names a thermal or a hydro carrier gets that cost representation + GenericOperationCost is chosen by cost_type, and only the thermal variant states a + start-up and a shut-down cost. + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains generator "New_CCGT" on "North_bus" carrier "CCGT" p_nom 0 p_nom_extendable True + And generator "New_CCGT" has p_nom_max 400 + And generator "New_CCGT" has overnight_cost 900000 + And generator "New_CCGT" has discount_rate 0.08 + And generator "New_CCGT" has lifetime 30 + And the network contains generator "New_Hydro" on "North_bus" carrier "hydro" p_nom 0 p_nom_extendable True + And generator "New_Hydro" has p_nom_max 200 + And generator "New_Hydro" has overnight_cost 2000000 + And generator "New_Hydro" has discount_rate 0.05 + And generator "New_Hydro" has lifetime 40 + And the network is saved as "inputs/cost_types.nc" + When I run the pypsa investments translation against "inputs/cost_types.nc" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "New_CCGT" having "power_systems_type" set to "ThermalStandard" + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "New_CCGT" having "operation_costs.cost_type" set to "THERMAL" + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "New_CCGT" having "operation_costs.start_up" set to 0.0 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "New_CCGT" having "operation_costs.shut_down" set to 0.0 + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "New_CCGT" having "fuel" set to ["NATURAL_GAS"] + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "New_Hydro" having "power_systems_type" set to "HydroDispatch" + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "New_Hydro" having "operation_costs.cost_type" set to "HYDRO_GEN" + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "New_Hydro" without field "operation_costs.start_up" + + Scenario: every component of the portfolio takes its id from one counter + An association names the component it describes by id alone, so two components of + different types may not share one. + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains load "North_load" on "North_bus" with static p_set 100 + And the network contains generator "REZ_Solar" on "North_bus" carrier "solar" p_nom 0 p_nom_extendable True + And generator "REZ_Solar" has p_nom_max 500 + And generator "REZ_Solar" has overnight_cost 1200000 + And generator "REZ_Solar" has discount_rate 0.07 + And generator "REZ_Solar" has lifetime 25 + And the network contains storage unit "NewBattery" on "North_bus" carrier "PHS" p_nom 0 max_hours 4.0 efficiency_store 0.9 efficiency_dispatch 0.95 p_nom_extendable True + And storage unit "NewBattery" has p_nom_max 300 + And storage unit "NewBattery" has overnight_cost 800000 + And storage unit "NewBattery" has discount_rate 0.07 + And storage unit "NewBattery" has lifetime 20 + And the network is saved as "inputs/ids.nc" + When I run the pypsa investments translation against "inputs/ids.nc" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" having "id" set to 1 + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewBattery" having "id" set to 2 + And the file "outputs/portfolio.json" parses as JSON with component "DemandRequirement" named "North_load" having "id" set to 3 + + Scenario: a load whose bus prices a shortfall is a demand the base system may cut + The base system writes such a load as an InterruptiblePowerLoad, and power_systems_type + names the type the base system holds it as. + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains load "North_load" on "North_bus" with static p_set 100 + And the network is saved as "inputs/voll.nc" + And a file "inputs/extensions.json" containing the lines: + | line | + | {"bus": [{"name": "North_bus", "value_of_lost_load": 9000.0}]} | + When I run the pypsa investments translation with sidecar "inputs/extensions.json" against "inputs/voll.nc" writing "outputs/portfolio.json" + Then the file "outputs/system.json" parses as JSON with 1 components of type "InterruptiblePowerLoad" + And the file "outputs/portfolio.json" parses as JSON with component "DemandRequirement" named "North_load" having "power_systems_type" set to "InterruptiblePowerLoad" + + Scenario: a candidate the network prices nothing for is left out, and the run completes + PyPSA leaves overnight_cost and discount_rate at NaN rather than at zero, and a + technology written from either would build for free or discount at nothing. + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains generator "Free_REZ" on "North_bus" carrier "solar" p_nom 0 p_nom_extendable True + And generator "Free_REZ" has p_nom_max 400 + And generator "Free_REZ" has discount_rate 0.07 + And generator "Free_REZ" has lifetime 25 + And the network contains generator "Unrated_REZ" on "North_bus" carrier "solar" p_nom 0 p_nom_extendable True + And generator "Unrated_REZ" has p_nom_max 400 + And generator "Unrated_REZ" has overnight_cost 1200000 + And generator "Unrated_REZ" has lifetime 25 + And the network contains generator "Priced_REZ" on "North_bus" carrier "solar" p_nom 0 p_nom_extendable True + And generator "Priced_REZ" has p_nom_max 400 + And generator "Priced_REZ" has overnight_cost 1200000 + And generator "Priced_REZ" has discount_rate 0.07 + And generator "Priced_REZ" has lifetime 25 + And the network is saved as "inputs/unpriced.nc" + When I run the pypsa investments translation against "inputs/unpriced.nc" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as JSON with 1 component of type "SupplyTechnology" + And the file "outputs/portfolio.json" parses as a portfolio with no component "SupplyTechnology" named "Free_REZ" + And the file "outputs/portfolio.json" parses as a portfolio with no component "SupplyTechnology" named "Unrated_REZ" + And the file "decisions.md" contains "`pypsa.Generator.Free_REZ.overnight_cost`" + And the file "decisions.md" contains "`pypsa.Generator.Unrated_REZ.discount_rate`" + And the log contains "1 Generator(s) are extendable and put no overnight cost on the capacity a build adds" + And the log contains "1 Generator(s) are extendable and state no discount rate" + + Scenario: a sidecar number that is not finite leaves its field out, and the run completes + A sidecar is JSON, and a JSON reader accepts the words NaN and Infinity. + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains generator "REZ_Solar" on "North_bus" carrier "solar" p_nom 0 p_nom_extendable True + And generator "REZ_Solar" has p_nom_max 500 + And generator "REZ_Solar" has overnight_cost 1200000 + And generator "REZ_Solar" has discount_rate 0.07 + And generator "REZ_Solar" has lifetime 25 + And the network is saved as "inputs/non_finite_sidecar.nc" + And a file "inputs/extensions.json" containing the lines: + | line | + | {"generator": [{"name": "REZ_Solar", "unit_size_mw": Infinity, "technical_life_years": NaN}]} | + When I run the pypsa investments translation with sidecar "inputs/extensions.json" against "inputs/non_finite_sidecar.nc" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as JSON with 1 component of type "SupplyTechnology" + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" without field "unit_size" + And the file "outputs/portfolio.json" parses as JSON with component "SupplyTechnology" named "REZ_Solar" without field "lifetime" + And the file "decisions.md" contains "`pypsa.Generator.REZ_Solar._unit_size_mw` = inf" + + Scenario: a candidate whose lifetime is below one year is left out, and the run completes + A capital recovery period is a whole number of years, so a lifetime below one year leaves + no years to recover the overnight cost across. + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains generator "REZ_Solar" on "North_bus" carrier "solar" p_nom 0 p_nom_extendable True + And generator "REZ_Solar" has p_nom_max 500 + And generator "REZ_Solar" has overnight_cost 1200000 + And generator "REZ_Solar" has discount_rate 0.07 + And generator "REZ_Solar" has lifetime 0.5 + And the network is saved as "inputs/short_lifetime.nc" + When I run the pypsa investments translation against "inputs/short_lifetime.nc" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as JSON with 0 components of type "SupplyTechnology" + And the log contains "1 Generator(s) are extendable and state a lifetime below one year" + And the file "decisions.md" contains "lifetime is below one year" + + Scenario: a candidate whose floor is above its ceiling is left out, and the run completes + A technology states a capacity floor and a capacity ceiling, and no capacity can meet a + floor above the ceiling. + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains generator "REZ_Solar" on "North_bus" carrier "solar" p_nom 0 p_nom_extendable True + And generator "REZ_Solar" has p_nom_min 600 + And generator "REZ_Solar" has p_nom_max 500 + And generator "REZ_Solar" has overnight_cost 1200000 + And generator "REZ_Solar" has discount_rate 0.07 + And generator "REZ_Solar" has lifetime 25 + And the network is saved as "inputs/inverted_limits.nc" + When I run the pypsa investments translation against "inputs/inverted_limits.nc" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as JSON with 0 components of type "SupplyTechnology" + And the log contains "1 Generator(s) are extendable and state a capacity floor above the capacity a build may reach" + And the file "decisions.md" contains "p_nom_min is above p_nom_max" + + Scenario: a storage candidate with no energy ceiling is left out, and the run completes + A unit whose max_hours is not a finite number puts no bound on the energy a build may add. + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains storage unit "NewPHS" on "North_bus" carrier "PHS" p_nom 0 max_hours inf efficiency_store 0.9 efficiency_dispatch 0.95 p_nom_extendable True + And storage unit "NewPHS" has p_nom_max 300 + And storage unit "NewPHS" has overnight_cost 800000 + And storage unit "NewPHS" has discount_rate 0.07 + And storage unit "NewPHS" has lifetime 20 + And the network is saved as "inputs/unbounded_energy.nc" + When I run the pypsa investments translation against "inputs/unbounded_energy.nc" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as JSON with 0 components of type "StorageTechnology" + And the log contains "1 StorageUnit(s) are extendable and put no upper bound on the energy a build may add" + And the file "decisions.md" contains "max_hours is not a finite number of hours" + + Scenario: a storage candidate that holds no energy is left out, and the run completes + A StorageTechnology states the energy a build may add as max_hours of its power, so a + unit stating no hours could build power it can never charge. + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains storage unit "Flat_PHS" on "North_bus" carrier "PHS" p_nom 0 max_hours 0.0 efficiency_store 0.9 efficiency_dispatch 0.95 p_nom_extendable True + And storage unit "Flat_PHS" has p_nom_max 300 + And storage unit "Flat_PHS" has overnight_cost 800000 + And storage unit "Flat_PHS" has discount_rate 0.07 + And storage unit "Flat_PHS" has lifetime 20 + And the network is saved as "inputs/no_energy.nc" + When I run the pypsa investments translation against "inputs/no_energy.nc" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as JSON with 0 components of type "StorageTechnology" + And the file "decisions.md" contains "`pypsa.StorageUnit.Flat_PHS.max_hours`" + And the file "decisions.md" contains "so the energy capacity limits a build could add are zero MWh" + And the log contains "1 StorageUnit(s) are extendable and hold no energy" + + Scenario: a candidate whose carrier names another kind's Sienna type is left out + A StorageTechnology states the base system type a build becomes, and the mappings file may + send a carrier to a type no storage unit is ever written as. + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains storage unit "Odd_PHS" on "North_bus" carrier "solar" p_nom 0 max_hours 4.0 efficiency_store 0.9 efficiency_dispatch 0.95 p_nom_extendable True + And storage unit "Odd_PHS" has p_nom_max 300 + And storage unit "Odd_PHS" has overnight_cost 800000 + And storage unit "Odd_PHS" has discount_rate 0.07 + And storage unit "Odd_PHS" has lifetime 20 + And the network is saved as "inputs/wrong_target.nc" + When I run the pypsa investments translation against "inputs/wrong_target.nc" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as JSON with 0 components of type "StorageTechnology" + And the file "decisions.md" contains "the user mappings file sends this carrier to a type this candidate never holds" + And the log contains "1 StorageUnit(s) may be built and have a carrier the mappings file sends to a Sienna type this kind of candidate never becomes" + + Scenario: a generator and a storage unit of one name keep their own build years + PyPSA names a generator and a storage unit independently, so one name may belong to both + and each technology reads the year its own class states. + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains generator "Shared" on "North_bus" carrier "solar" p_nom 200 + And generator "Shared" has build_year 1995 + And the network contains storage unit "Shared" on "North_bus" carrier "PHS" p_nom 100 max_hours 4.0 efficiency_store 0.9 efficiency_dispatch 0.95 + And storage unit "Shared" has build_year 2005 + And the network contains generator "REZ_Solar" on "North_bus" carrier "solar" p_nom 0 p_nom_extendable True + And generator "REZ_Solar" has p_nom_max 500 + And generator "REZ_Solar" has overnight_cost 1200000 + And generator "REZ_Solar" has discount_rate 0.07 + And generator "REZ_Solar" has lifetime 25 + And the network contains storage unit "NewPHS" on "North_bus" carrier "PHS" p_nom 0 max_hours 4.0 efficiency_store 0.9 efficiency_dispatch 0.95 p_nom_extendable True + And storage unit "NewPHS" has p_nom_max 300 + And storage unit "NewPHS" has overnight_cost 800000 + And storage unit "NewPHS" has discount_rate 0.07 + And storage unit "NewPHS" has lifetime 20 + And the network is saved as "inputs/shared_names.nc" + When I run the pypsa investments translation against "inputs/shared_names.nc" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as a portfolio where the "RetirementPotential" of "SupplyTechnology" "REZ_Solar" has "build_year.Shared" set to 1995 + And the file "outputs/portfolio.json" parses as a portfolio where the "RetirementPotential" of "StorageTechnology" "NewPHS" has "build_year.Shared" set to 2005 + + Scenario: a hydro storage candidate stands for the hydro units already running + A StorageUnit whose carrier the mappings file sends to HydroDispatch is still a storage + technology, and the base system holds the units of that carrier as HydroDispatch, so + those are the devices the technology adds to. + Given a PyPSA network + And the network has 3 snapshots at 60 minute intervals + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains storage unit "OldHydro" on "North_bus" carrier "hydro" p_nom 200 max_hours 6.0 efficiency_dispatch 0.9 inflow 10.0 20.0 30.0 + And storage unit "OldHydro" has build_year 1980 + And the network contains storage unit "NewHydro" on "North_bus" carrier "hydro" p_nom 0 max_hours 6.0 efficiency_store 0.9 efficiency_dispatch 0.95 p_nom_extendable True + And storage unit "NewHydro" has p_nom_max 300 + And storage unit "NewHydro" has overnight_cost 900000 + And storage unit "NewHydro" has discount_rate 0.07 + And storage unit "NewHydro" has lifetime 30 + And the network is saved as "inputs/hydro_fleet.nc" + When I run the pypsa investments translation against "inputs/hydro_fleet.nc" writing "outputs/portfolio.json" + Then the file "outputs/system.json" parses as JSON with 1 components of type "HydroDispatch" + And the file "outputs/portfolio.json" parses as JSON with component "StorageTechnology" named "NewHydro" having "power_systems_type" set to "HydroDispatch" + And the file "outputs/portfolio.json" parses as a portfolio where the "ExistingDevices" of "StorageTechnology" "NewHydro" has "existing_devices" set to ["OldHydro"] + And the file "outputs/portfolio.json" parses as a portfolio where the "RetirementPotential" of "StorageTechnology" "NewHydro" has "eligible_generators" set to ["OldHydro"] + And the file "outputs/portfolio.json" parses as a portfolio where the "RetirementPotential" of "StorageTechnology" "NewHydro" has "build_year.OldHydro" set to 1980 + + Scenario: a technology's fleet is the plant of its own region + A portfolio groups its technologies by region, so a technology stands for more of what its + own region already runs. A plant of the same carrier in another region is not one it adds + to, and not one a build of it may retire. + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains bus "South_bus" carrier "AC" v_nom 380.0 location "South" + And the network contains generator "OldSolar_North" on "North_bus" carrier "solar" p_nom 100 + And the network contains generator "OldSolar_South" on "South_bus" carrier "solar" p_nom 200 + And the network contains generator "REZ_Solar_North" on "North_bus" carrier "solar" p_nom 0 p_nom_extendable True + And generator "REZ_Solar_North" has p_nom_max 500 + And generator "REZ_Solar_North" has overnight_cost 1200000 + And generator "REZ_Solar_North" has discount_rate 0.07 + And generator "REZ_Solar_North" has lifetime 25 + And the network is saved as "inputs/two_regions.nc" + When I run the pypsa investments translation against "inputs/two_regions.nc" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as a portfolio where the "ExistingDevices" of "SupplyTechnology" "REZ_Solar_North" has "existing_devices" set to ["OldSolar_North"] + And the file "outputs/portfolio.json" parses as a portfolio where the "RetirementPotential" of "SupplyTechnology" "REZ_Solar_North" has "eligible_generators" set to ["OldSolar_North"] + + Scenario: a supply technology takes only the devices of its own PyPSA class + A Generator and a StorageUnit may share a name, and only a StorageUnit becomes a + HydroDispatch, so a supply technology's fleet is the generators of its carrier alone. + Given a PyPSA network + And the network has 3 snapshots at 60 minute intervals + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains storage unit "Shared" on "North_bus" carrier "hydro" p_nom 200 max_hours 6.0 efficiency_dispatch 0.9 inflow 10.0 20.0 30.0 + And the network contains generator "OldSolar" on "North_bus" carrier "solar" p_nom 200 + And the network contains generator "Shared" on "North_bus" carrier "solar" p_nom 0 p_nom_extendable True + And generator "Shared" has p_nom_max 500 + And generator "Shared" has overnight_cost 1200000 + And generator "Shared" has discount_rate 0.07 + And generator "Shared" has lifetime 25 + And the network is saved as "inputs/shared_hydro_name.nc" + When I run the pypsa investments translation against "inputs/shared_hydro_name.nc" writing "outputs/portfolio.json" + Then the file "outputs/system.json" parses as JSON with 1 component of type "HydroDispatch" + And the file "outputs/portfolio.json" parses as a portfolio where the "ExistingDevices" of "SupplyTechnology" "Shared" has "existing_devices" set to ["OldSolar"] + And the file "outputs/portfolio.json" parses as a portfolio where the "RetirementPotential" of "SupplyTechnology" "Shared" has "eligible_generators" set to ["OldSolar"] + + Scenario: a constraint whose limit is not a finite number is left out, and the run completes + A cap states a number of million tonnes, and neither NaN nor Infinity is one. + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains generator "GasPlant" on "North_bus" carrier "CCGT" p_nom 500 + And the network contains generator "REZ_Solar" on "North_bus" carrier "solar" p_nom 0 p_nom_extendable True + And generator "REZ_Solar" has p_nom_max 500 + And generator "REZ_Solar" has overnight_cost 1200000 + And generator "REZ_Solar" has discount_rate 0.07 + And generator "REZ_Solar" has lifetime 25 + And the network is saved as "inputs/non_finite_cap.nc" + And a file "inputs/extensions.json" containing the lines: + | line | + | {"constraint": [{"name": "CarbonBudget", "sense": "<=", "limits": [{"period": "year", "value": Infinity}], "members": [{"name": "GasPlant", "member_class": "Generator"}, {"name": "REZ_Solar", "member_class": "Generator"}]}]} | + When I run the pypsa investments translation with sidecar "inputs/extensions.json" against "inputs/non_finite_cap.nc" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as JSON with 0 components of type "CarbonCaps" + And the log contains "1 constraint(s) state a right-hand side that is not a finite number" + + Scenario: a constraint the expansion plan need not meet is left out, and the run completes + A source says whether its expansion plan has to meet a constraint. A cap written from a + constraint the plan need not meet would bound a problem the model leaves free. + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains generator "GasPlant" on "North_bus" carrier "CCGT" p_nom 500 + And the network contains generator "REZ_Solar" on "North_bus" carrier "solar" p_nom 0 p_nom_extendable True + And generator "REZ_Solar" has p_nom_max 500 + And generator "REZ_Solar" has overnight_cost 1200000 + And generator "REZ_Solar" has discount_rate 0.07 + And generator "REZ_Solar" has lifetime 25 + And the network is saved as "inputs/dispatch_only_cap.nc" + And a file "inputs/extensions.json" containing the lines: + | line | + | {"constraint": [{"name": "DispatchOnlyBudget", "sense": "<=", "limits": [{"period": "year", "value": 20.0}], "applies_to_expansion_plan": false, "members": [{"name": "GasPlant", "member_class": "Generator"}, {"name": "REZ_Solar", "member_class": "Generator"}]}]} | + When I run the pypsa investments translation with sidecar "inputs/extensions.json" against "inputs/dispatch_only_cap.nc" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as JSON with 0 components of type "CarbonCaps" + And the log contains "1 constraint(s) the expansion plan does not have to meet" + + Scenario: a constraint over another class of one name does not reach the whole model + Two classes may hold an object of one name, so a member counts against the model only + where its class matches as well. + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains generator "Shared" on "North_bus" carrier "CCGT" p_nom 500 + And the network contains generator "REZ_Solar" on "North_bus" carrier "solar" p_nom 0 p_nom_extendable True + And generator "REZ_Solar" has p_nom_max 500 + And generator "REZ_Solar" has overnight_cost 1200000 + And generator "REZ_Solar" has discount_rate 0.07 + And generator "REZ_Solar" has lifetime 25 + And the network is saved as "inputs/name_clash_cap.nc" + And a file "inputs/extensions.json" containing the lines: + | line | + | {"constraint": [{"name": "NodeFlowLimit", "sense": "<=", "limits": [{"period": "year", "value": 20.0}], "members": [{"name": "Shared", "member_class": "Bus"}, {"name": "REZ_Solar", "member_class": "Generator"}]}]} | + When I run the pypsa investments translation with sidecar "inputs/extensions.json" against "inputs/name_clash_cap.nc" writing "outputs/portfolio.json" + Then the file "outputs/portfolio.json" parses as JSON with 0 components of type "CarbonCaps" + And the log contains "1 constraint(s) weight a named subset of the model rather than all of it" + + Scenario: a generator neither document holds leaves a whole-model constraint whole + A cap holds every component of the portfolio and the base system it expands. A generator + an earlier hop of this translator added to shed load reaches neither, so a constraint + naming everything both documents hold still covers the model. + Given a PyPSA network + And the network contains bus "North_bus" carrier "AC" v_nom 380.0 location "North" + And the network contains load "North_load" on "North_bus" with static p_set 100 + And the network contains generator "GasPlant" on "North_bus" carrier "CCGT" p_nom 500 + And the network contains generator "North_bus_load_shedding" on "North_bus" carrier "load_shedding" p_nom 100 marginal_cost 9000 + And the network contains generator "REZ_Solar" on "North_bus" carrier "solar" p_nom 0 p_nom_extendable True + And generator "REZ_Solar" has p_nom_max 500 + And generator "REZ_Solar" has overnight_cost 1200000 + And generator "REZ_Solar" has discount_rate 0.07 + And generator "REZ_Solar" has lifetime 25 + And the network is saved as "inputs/whole_model_cap.nc" + And a file "inputs/extensions.json" containing the lines: + | line | + | {"bus": [{"name": "North_bus", "value_of_lost_load": 9000.0}], "constraint": [{"name": "CarbonBudget", "sense": "<=", "limits": [{"period": "year", "value": 20.0}], "members": [{"name": "GasPlant", "member_class": "Generator"}, {"name": "REZ_Solar", "member_class": "Generator"}]}]} | + When I run the pypsa investments translation with sidecar "inputs/extensions.json" against "inputs/whole_model_cap.nc" writing "outputs/portfolio.json" + Then the file "outputs/system.json" parses as JSON with 1 components of type "ThermalStandard" + And the file "outputs/portfolio.json" parses as JSON with 1 component of type "CarbonCaps" + And the file "outputs/portfolio.json" parses as JSON with component "CarbonCaps" named "CarbonBudget" having "max_mtons" set to 20.0 diff --git a/tests/step_defs/pypsa_to_sienna_investments/conftest.py b/tests/step_defs/pypsa_to_sienna_investments/conftest.py new file mode 100644 index 0000000..0709535 --- /dev/null +++ b/tests/step_defs/pypsa_to_sienna_investments/conftest.py @@ -0,0 +1,52 @@ +from __future__ import annotations + +from pathlib import Path + +import pytest +from pytest_bdd import parsers, when + +from tests.step_defs.conftest import ( + STANDARD_CARRIER_MAP, + STANDARD_PRIME_MOVER_MAP, + invoke_translate, + write_user_mappings, +) + +_MAPPINGS_PATH = "user_mappings.yaml" + +_CARRIERS = ("solar", "onwind", "hydro", "PHS") +_THERMAL_CARRIER = "CCGT" + + +@pytest.fixture(autouse=True) +def carrier_mappings_file() -> None: + write_user_mappings( + {_THERMAL_CARRIER: STANDARD_CARRIER_MAP[_THERMAL_CARRIER]}, + prime_mover={carrier: STANDARD_PRIME_MOVER_MAP[carrier] for carrier in _CARRIERS}, + ) + + +@when( + parsers.re( + r"I run the pypsa investments translation" + r'(?: with sidecar "(?P[^"]+)")?' + r' against "(?P[^"]+)" writing "(?P[^"]+)"' + ) +) +def run_investments_translation( + monkeypatch: pytest.MonkeyPatch, + nc_path: str, + portfolio_path: str, + extensions_path: str | None, +) -> None: + sidecar = {"source_extensions_json_path": str(Path(extensions_path))} if extensions_path else {} + invoke_translate( + monkeypatch, + "pypsa", + "sienna", + "pypsa-to-sienna-investments", + user_mappings_path=_MAPPINGS_PATH, + source_path=str(Path(nc_path)), + sink_1_output_path=portfolio_path, + **sidecar, + ) diff --git a/tests/step_defs/pypsa_to_sienna_investments/test_portfolio.py b/tests/step_defs/pypsa_to_sienna_investments/test_portfolio.py new file mode 100644 index 0000000..cb61f2f --- /dev/null +++ b/tests/step_defs/pypsa_to_sienna_investments/test_portfolio.py @@ -0,0 +1,3 @@ +from pytest_bdd import scenarios + +scenarios("../features/pypsa_to_sienna_investments/portfolio.feature") diff --git a/tests/step_defs/test_plexos_to_sienna_investments.py b/tests/step_defs/test_plexos_to_sienna_investments.py new file mode 100644 index 0000000..0ef7675 --- /dev/null +++ b/tests/step_defs/test_plexos_to_sienna_investments.py @@ -0,0 +1,28 @@ +from pathlib import Path + +import pytest +from pytest_bdd import parsers, scenarios, when + +from tests.step_defs.conftest import PLEXOS_MAPPINGS_PATH, invoke_translate + +scenarios("../features/plexos_to_sienna_investments.feature") + + +@when( + parsers.parse( + 'I run the plexos-to-sienna-investments chain against "{xml_path}" ' + 'writing "{portfolio_path}"' + ) +) +def run_plexos_to_sienna_investments( + monkeypatch: pytest.MonkeyPatch, xml_path: str, portfolio_path: str +) -> None: + invoke_translate( + monkeypatch, + "plexos", + "sienna", + "plexos-to-sienna-investments", + user_mappings_path=PLEXOS_MAPPINGS_PATH, + source_path=str(Path(xml_path)), + sink_1_output_path=portfolio_path, + )