add read/write support for .surge.json

README.md

@@ -1,12 +1,12 @@
 # PowerIO
 
 Lossless IO and format conversion for power system case files. Parse MATPOWER
-`.m`, PSS/E `.raw`, PowerWorld `.aux`, PowerModels JSON, and EGRET JSON into one
-format neutral `Network`; write any of them back (same-format round trips are byte
-for byte); convert between them with explicit fidelity reporting; and emit the
-sparse matrices and graph views a solver needs. The same Rust core is callable
-from Rust, Python, C/C++, and Julia. The core crate has six dependencies and no
-matrix or solver stack.
+`.m`, PSS/E `.raw`, PowerWorld `.aux`, PowerModels JSON, EGRET JSON, and Surge
+JSON into one format neutral `Network`; write any of them back (same format
+round trips are byte for byte); convert between them with explicit fidelity
+reporting; and emit the sparse matrices and graph views a solver needs. The
+same Rust core is callable from Rust, Python, C/C++, and Julia. The core crate
+has six dependencies and no matrix or solver stack.
 
 ## Workspace
 
@@ -36,29 +36,34 @@ Every reader produces a `Network` and every writer consumes one, so a new format
 is one module at the hub, not an N×M matrix of pairwise converters.
 
 **Readers and writers**: MATPOWER `.m`, PowerModels JSON, PSS/E `.raw` (v33),
-PowerWorld `.aux`, and EGRET JSON.
+PowerWorld `.aux`, EGRET JSON, and Surge JSON.
 
 Legend: 🟩 byte-exact · 🟦 full · 🟨 partial (drops are logged in `Conversion::warnings`)
 
-| reader ↓ \ writer → | MATPOWER | PowerModels JSON | PSS/E | PowerWorld | EGRET JSON |
-| --- | --- | --- | --- | --- | --- |
-| **MATPOWER** | 🟩 | 🟦 | 🟨 | 🟨 | 🟨 |
-| **PowerModels JSON** | 🟦 | 🟩 | 🟨 | 🟨 | 🟨 |
-| **PSS/E** | 🟦 | 🟦 | 🟩 | 🟨 | 🟨 |
-| **PowerWorld** | 🟦 | 🟦 | 🟨 | 🟩 | 🟨 |
-| **EGRET JSON** | 🟦 | 🟦 | 🟨 | 🟨 | 🟩 |
+| reader ↓ \ writer → | MATPOWER | PowerModels JSON | PSS/E | PowerWorld | EGRET JSON | Surge JSON |
+| --- | --- | --- | --- | --- | --- | --- |
+| **MATPOWER** | 🟩 | 🟦 | 🟨 | 🟨 | 🟨 | 🟨 |
+| **PowerModels JSON** | 🟦 | 🟩 | 🟨 | 🟨 | 🟨 | 🟨 |
+| **PSS/E** | 🟦 | 🟦 | 🟩 | 🟨 | 🟨 | 🟦 |
+| **PowerWorld** | 🟦 | 🟦 | 🟨 | 🟩 | 🟨 | 🟦 |
+| **EGRET JSON** | 🟦 | 🟦 | 🟨 | 🟨 | 🟩 | 🟦 |
+| **Surge JSON** | 🟨 | 🟨 | 🟨 | 🟨 | 🟨 | 🟩 |
 
 **🟩 byte-exact**: writing back to the source format reproduces the file verbatim,
 comments and exact tokens like `7e-05` included. **🟦 full**: every field the source
 carries survives. **🟨 partial**: the target cannot represent some fields (PSS/E and
 PowerWorld have no cost curves; EGRET has no HVDC or storage), and each dropped
-field is reported in `Conversion::warnings`, not dropped silently. Two target
-caveats fold into this: canonical MATPOWER output omits dcline and storage, and the
-PowerModels writer maps them best-effort.
-
-Every reader and writer is validated against an independent tool, PowerModels.jl,
-the EGRET package, ExaPowerIO.jl, and pandapower, over the full conversion matrix.
-See [benchmarks/RESULTS.md](benchmarks/RESULTS.md) and
+field is reported in `Conversion::warnings`, not dropped silently. Three target
+caveats fold into this: canonical MATPOWER output omits dcline and storage, the
+PowerModels writer maps them best-effort, and Surge JSON carries the core network
+profile only. Rich Surge dispatch, result, market, controls, ZIP load,
+converter, and DC grid data are reported on cross format writes. Compressed
+`.surge.json.zst` and `.surge.bin` files are out of scope for the runtime reader.
+
+Every reader and writer is validated against an independent tool where one exists:
+PowerModels.jl, the EGRET package, ExaPowerIO.jl, pandapower, and an optional
+Surge CLI oracle through `SURGE_BIN` or `SURGE_CHECKOUT`. See
+[benchmarks/RESULTS.md](benchmarks/RESULTS.md) and
 [docs/format-fidelity.md](docs/format-fidelity.md).
 
 ## Matrices

benchmarks/run_validation.sh

@@ -12,13 +12,17 @@
 #                                                   benchmarks/validate_exapowerio.jl
 #   pp      — powerio's parse + Y_bus vs pandapower (_m2ppc + makeYbus).
 #                                                   benchmarks/validate_pandapower.py
+#   Surge   — optional: powerio's Surge JSON writer vs Surge's own parser.
+#             Set SURGE_BIN=/path/to/surge-solve or SURGE_CHECKOUT=/path/to/surge.
+#                                                   benchmarks/validate_surge.py
 #
 # Then the read sides and the full conversion matrix:
 #   PSSE-read   — powerio reads a real PSS/E .raw, emits PowerModels JSON, compared
 #                 against PowerModels.jl reading the same .raw.
 #   EGRET-read  — powerio reads a real EGRET .json (egret's own output), emits
 #                 PowerModels JSON, checked against the matching MATPOWER case.
-#   matrix(5x5) — every reader -> every writer over the fixtures, each output's
+#   matrix(5x5) — every reader -> every writer over the fixtures covered by the
+#                 independent PowerModels and egret oracles, each output's
 #                 electrical core checked against the ground-truth MATPOWER case
 #                 (PowerModels.jl for MATPOWER/PowerModels/PSS-E/PowerWorld, the
 #                 egret package for EGRET), byte-exact on the diagonal.
@@ -28,7 +32,8 @@
 # built into .venv (`maturin develop --release`), the Julia env instantiated
 # (`julia --project=benchmarks -e 'using Pkg; Pkg.instantiate()'`), and the Python
 # oracle tools (`pip install -r benchmarks/requirements.txt`, for the pandapower
-# and EGRET checks). All oracle tools are benchmark-scoped, not powerio deps.
+# and EGRET checks). Surge is optional through SURGE_BIN or SURGE_CHECKOUT. All
+# oracle tools are benchmark-scoped, not powerio deps.
 #
 #   bash benchmarks/run_validation.sh
 #
@@ -48,6 +53,11 @@ trap 'rm -rf "$TMP"' EXIT
 HAVE_EGRET=1
 "$PY" -c "import egret" 2>/dev/null || HAVE_EGRET=0
 
+HAVE_SURGE=0
+if [ -n "${SURGE_BIN:-}" ] || [ -n "${SURGE_CHECKOUT:-}" ]; then
+    HAVE_SURGE=1
+fi
+
 MCASES=(
     tests/data/case9.m
     tests/data/case14.m
@@ -126,6 +136,17 @@ for m in "${MCASES[@]}"; do
     mark "$PY" benchmarks/validate_pandapower.py "$m"
     row+="  pp:$MARK"
 
+    if [ "$HAVE_SURGE" -eq 0 ]; then
+        MARK="SKIP"
+    elif [ "$base" = "case2869pegase" ]; then
+        MARK="SKIP(nonfinite)"
+    elif convert "$m" surge-json "$TMP/$base.surge.json" 2>"$TMP/err"; then
+        mark "$PY" benchmarks/validate_surge.py "$m" "$TMP/$base.surge.json"
+    else
+        echo "  Surge: convert failed"; cat "$TMP/err"; MARK="FAIL"; fails=$((fails + 1))
+    fi
+    row+="  Surge:$MARK"
+
     rows+=("$row")
 done
 

benchmarks/validate_surge.py

@@ -0,0 +1,103 @@
+#!/usr/bin/env python
+"""Optional Surge oracle for powerio's Surge JSON writer.
+
+Usage:
+  SURGE_BIN=/path/to/surge-solve benchmarks/validate_surge.py ref.m out.surge.json
+  SURGE_CHECKOUT=/path/to/surge benchmarks/validate_surge.py ref.m out.surge.json
+
+The oracle loads powerio's `.surge.json` output with Surge's own CLI
+`--parse-only --output json` and compares counts and totals against the
+reference case parsed by powerio. It is benchmark scoped: no Surge crate or
+Python package is a dependency of powerio.
+"""
+
+from __future__ import annotations
+
+import json
+import math
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+import powerio
+
+
+def surge_command() -> list[str] | None:
+    if bin_path := os.environ.get("SURGE_BIN"):
+        return [bin_path]
+    if checkout := os.environ.get("SURGE_CHECKOUT"):
+        manifest = Path(checkout) / "Cargo.toml"
+        return [
+            "cargo",
+            "run",
+            "--quiet",
+            "--manifest-path",
+            str(manifest),
+            "-p",
+            "surge-bindings",
+            "--bin",
+            "surge-solve",
+            "--",
+        ]
+    return None
+
+
+def powerio_core(path: str) -> dict[str, float]:
+    case = powerio.parse(path)
+    return {
+        "n_buses": case.n,
+        "n_branches": case.n_branches,
+        "n_generators": case.n_gens,
+        "total_load_mw": sum(load["p"] for load in case.loads),
+        "total_gen_mw": sum(gen["pg"] for gen in case.gens),
+        "base_mva": case.base_mva,
+    }
+
+
+def surge_core(path: str) -> dict[str, float]:
+    cmd = surge_command()
+    if cmd is None:
+        print("SKIP: set SURGE_BIN or SURGE_CHECKOUT for the Surge oracle")
+        raise SystemExit(77)
+    out = subprocess.run(
+        [*cmd, path, "--parse-only", "--output", "json"],
+        capture_output=True,
+        text=True,
+    )
+    if out.returncode != 0:
+        print(out.stderr or out.stdout, file=sys.stderr)
+        raise SystemExit(out.returncode)
+    data = json.loads(out.stdout)
+    return {
+        "n_buses": data["n_buses"],
+        "n_branches": data["n_branches"],
+        "n_generators": data["n_generators"],
+        "total_load_mw": data["total_load_mw"],
+        "total_gen_mw": data["total_gen_mw"],
+        "base_mva": data["base_mva"],
+    }
+
+
+def main() -> None:
+    if len(sys.argv) != 3:
+        print("usage: validate_surge.py <reference case> <powerio surge json>", file=sys.stderr)
+        raise SystemExit(2)
+    ref, out = sys.argv[1], sys.argv[2]
+    ref_core = powerio_core(ref)
+    out_core = surge_core(out)
+    problems = []
+    for key in ("n_buses", "n_branches", "n_generators"):
+        if int(ref_core[key]) != int(out_core[key]):
+            problems.append(f"{key} {ref_core[key]}!={out_core[key]}")
+    for key in ("total_load_mw", "total_gen_mw", "base_mva"):
+        if not math.isclose(ref_core[key], out_core[key], rel_tol=1e-6, abs_tol=1e-6):
+            problems.append(f"{key} {ref_core[key]}!={out_core[key]}")
+    if problems:
+        print("; ".join(problems), file=sys.stderr)
+        raise SystemExit(1)
+    print("ok")
+
+
+if __name__ == "__main__":
+    main()

docs/format-fidelity.md

@@ -6,8 +6,9 @@ this document covers the conventions and the proof behind it.
 
 ## Conventions
 
-powerio's numeric conventions match MATPOWER and PowerModels.jl. The reference
-implementations and the matching powerio code:
+powerio's numeric conventions match MATPOWER and PowerModels.jl. Surge stores
+angles in radians; powerio normalizes them to the same degree based model used by
+the other formats. The reference implementations and the matching powerio code:
 
 | Quantity | Convention | Reference | powerio |
 | --- | --- | --- | --- |
@@ -16,9 +17,12 @@ implementations and the matching powerio code:
 | Line charging `b` | split half to each end (`b_fr = b_to = BR_B/2`) | PowerModels `matpower.jl` | `format::powermodels` |
 | Tap ratio | `0` means a line (treated as `1`); nonzero is a transformer | MATPOWER `idx_brch` `TAP` | `Branch::effective_tap` |
 | Phase shift, angle | degrees in the model; PowerModels JSON carries radians | PowerModels `make_per_unit!` | `format::powermodels` |
+| Surge phase shift, angle | degrees in the model; Surge JSON carries bus angles, branch shifts, and branch angle limits in radians | Surge data model | `format::surge` |
 | Angle limits | `angmin`/`angmax` default ±360 (unconstrained) | MATPOWER `idx_brch` `ANGMIN`/`ANGMAX` | `Branch::has_angle_limits` |
+| Surge tap ratio | `1.0` on a plain line maps back to MATPOWER raw tap `0`; nonzero transformer taps are preserved | Surge data model | `format::surge` |
 | dcline `Pt`/`Qf`/`Qt` | sign flips vs MATPOWER | PowerModels `matpower.jl` | `format::powermodels` |
 | Generator cost | `c2 p² + c1 p` → `q = 2c2`, `c = c1`; coefficients high order first | MATPOWER `idx_cost`, EGRET `matpower_parser` | `GenCost::quadratic` |
+| Surge generator cost | polynomial coefficients are high order first; piecewise curves are point lists | Surge data model | `format::surge` |
 | `source_id` | `["bus", id]` for bus-tied elements | PowerModels `matpower.jl` | `format::powermodels` |
 
 EGRET's own MATPOWER parser uses the same reductions (bus type as
@@ -28,8 +32,9 @@ MATPOWER case taken through powerio to EGRET matches egret's direct import.
 
 ## Validation
 
-The harness `benchmarks/run_validation.sh` checks powerio against four independent
-tools. Every reader and writer, and every conversion pair, is exercised.
+The harness `benchmarks/run_validation.sh` checks powerio against independent
+tools. The Rust suite exercises every reader and writer, and every conversion
+pair, including Surge.
 
 - **PowerModels.jl** (`validate_powermodels.jl`, `validate_psse.jl`,
   `core_json.jl`). Reads MATPOWER, PowerModels JSON, and PSS/E. The MATPOWER to
@@ -41,23 +46,29 @@ tools. Every reader and writer, and every conversion pair, is exercised.
 - **ExaPowerIO.jl** (`validate_exapowerio.jl`). Reads MATPOWER through powerio's C
   ABI and compares value for value.
 - **pandapower** (`validate_pandapower.py`). Cross-checks the parse and the `Y_bus`.
+- **Surge CLI** (`validate_surge.py`). Optional. Set `SURGE_BIN=/path/to/surge-solve`
+  or `SURGE_CHECKOUT=/path/to/surge`; the script loads powerio's Surge JSON
+  output with Surge's parser and compares counts and totals.
 
 ### The conversion matrix
 
-`benchmarks/validate_matrix.py` converts each source to every target and checks
-the electrical core of the output (bus/branch/generator counts and the per unit
-demand, generation, and shunt totals) against the source's own core, read by an
-independent oracle. The diagonal is checked byte-exact: writing back to the source
-format reproduces the file. Sources use the real native files where they exist
-(the vendored PSS/E `.raw` and EGRET `.json`) and representative MATPOWER cases
-otherwise: basic (`case9`), shunts and transformers (`case14`, `case30`), size
-(`case118`, `case2869pegase`), HVDC with a mixed piecewise/polynomial gencost
+`benchmarks/validate_matrix.py` converts each source to every target covered by
+the PowerModels and egret oracles and checks the electrical core of the output
+(bus/branch/generator counts and the per unit demand, generation, and shunt
+totals) against the source's own core, read by an independent oracle. The
+diagonal is checked byte-exact: writing back to the source format reproduces the
+file. Sources use the real native files where they exist (the vendored PSS/E
+`.raw` and EGRET `.json`) and representative MATPOWER cases otherwise: basic
+(`case9`), shunts and transformers (`case14`, `case30`), size (`case118`,
+`case2869pegase`), HVDC with a mixed piecewise/polynomial gencost
 (`t_case9_dcline`), and a piecewise-cost case (`pglib_opf_case5_pjm`).
 
-All 65 cells pass (13 source cases × 5 targets). The core is preserved by every
-writer regardless of fidelity tier, so it is the invariant checked across the
-whole matrix; cost, HVDC, and angle limits are tier-specific and covered by the
-dedicated checks above and the Rust suite.
+The benchmark matrix has 65 cells (13 source cases × 5 oracle targets). The Rust
+all pairs suite adds Surge as a sixth format and checks byte-exact same format
+echo plus approximate JSON equality where numeric formatting changes. The core
+is preserved by every writer regardless of fidelity tier, so it is the invariant
+checked across the whole matrix; cost, HVDC, storage, source-only fields, and
+angle limits are covered by dedicated checks and the Rust suite.
 
 ### Running it
 
@@ -69,9 +80,10 @@ pip install -r benchmarks/requirements.txt         # pandapower + egret oracles
 bash benchmarks/run_validation.sh
 ```
 
-The oracle tools (PowerModels.jl, egret, ExaPowerIO.jl, pandapower) are
-benchmark-scoped: they are declared in `benchmarks/Project.toml` and
-`benchmarks/requirements.txt`, never as dependencies of the powerio package.
+The oracle tools (PowerModels.jl, egret, ExaPowerIO.jl, pandapower, and optional
+Surge) are benchmark scoped: they are declared in `benchmarks/Project.toml`,
+`benchmarks/requirements.txt`, or supplied by `SURGE_BIN`/`SURGE_CHECKOUT`, never
+as dependencies of the powerio package.
 
 ## Known limits
 
@@ -90,3 +102,14 @@ These are reported in `Conversion::warnings`, not dropped silently.
 - **EGRET** output drops HVDC and storage. The reader takes the power flow
   ModelData subset (numeric bus ids, scalar values); unit commitment cases
   (`system.time_keys`) are rejected.
+- **Surge** runtime support is limited to decompressed `.surge.json` network
+  documents. `.surge.json.zst`, `.surge.bin`, native dispatch profiles, and
+  native result profiles are out of scope. If a Surge JSON document carries
+  `dispatch` or `solution`, the reader models `network` and cross format writers
+  report the ignored sections. ZIP and CMPLDW load fields, frequency,
+  ownership, classification, market data, topology metadata, branch control
+  bounds, richer storage metadata, DC grid data, converter details, reactive
+  limits, losses, and controls are warned on cross format conversion. Surge
+  tagged nonfinite floats are rejected by the reader; the writer emits JSON
+  `null` for nonfinite values through the shared JSON number helper and reports
+  format loss where applicable.

powerio-capi/README.md

@@ -7,6 +7,9 @@ Python ctypes, …) can drive powerio through it.
 
 The header is [`include/powerio.h`](include/powerio.h).
 
+Format names are the same aliases the Rust hub accepts, including `matpower`,
+`powermodels-json`, `egret-json`, `surge-json`, `psse`, and `powerworld`.
+
 ## Build
 
 ```

powerio-capi/include/powerio.h

@@ -34,8 +34,9 @@ extern "C" {
 typedef struct PioCase PioCase;
 
 /* Parse `path`; format from the file extension, or forced by `from`
- * ("matpower","powermodels","psse","powerworld") when non-NULL. Returns NULL on
- * error and writes the message into errbuf (a char[errlen]). */
+ * ("matpower","powermodels","egret","surge","psse","powerworld") when
+ * non-NULL. Returns NULL on error and writes the message into errbuf
+ * (a char[errlen]). */
 PioCase *pio_parse(const char *path, const char *from, char *errbuf, size_t errlen);
 void pio_case_free(PioCase *c);
 

powerio-cli/src/main.rs

@@ -152,6 +152,8 @@ enum FormatArg {
     PowerModelsJson,
     #[value(name = "egret-json", alias = "egret")]
     EgretJson,
+    #[value(name = "surge-json", alias = "surge")]
+    SurgeJson,
     #[value(name = "psse", alias = "raw")]
     Psse,
     #[value(name = "powerworld", alias = "aux")]
@@ -164,6 +166,7 @@ impl From<FormatArg> for powerio_matrix::TargetFormat {
             FormatArg::Matpower => Self::Matpower,
             FormatArg::PowerModelsJson => Self::PowerModelsJson,
             FormatArg::EgretJson => Self::EgretJson,
+            FormatArg::SurgeJson => Self::SurgeJson,
             FormatArg::Psse => Self::Psse,
             FormatArg::PowerWorld => Self::PowerWorld,
         }
@@ -177,6 +180,7 @@ impl FormatArg {
             FormatArg::Matpower => "matpower",
             FormatArg::PowerModelsJson => "powermodels-json",
             FormatArg::EgretJson => "egret-json",
+            FormatArg::SurgeJson => "surge-json",
             FormatArg::Psse => "psse",
             FormatArg::PowerWorld => "powerworld",
         }

powerio-matrix/src/lib.rs

@@ -40,8 +40,8 @@ pub use powerio::{
     GenCost, Generator, Hvdc, IndexCore, IndexedNetwork, Load, Network, Result, ScenarioMismatch,
     Shunt, SourceFormat, Storage, TargetFormat, error, format, indexed, network, parse,
     parse_matpower, parse_matpower_file, parse_powermodels_json, parse_powerworld, parse_psse,
-    parse_str, read_path, target_format_from_name, write_as, write_egret_json, write_matpower,
-    write_powermodels_json, write_powerworld, write_psse,
+    parse_str, parse_surge_json, read_path, target_format_from_name, write_as, write_egret_json,
+    write_matpower, write_powermodels_json, write_powerworld, write_psse, write_surge_json,
 };
 
 pub mod io;