Skip to content

feat(data): import pathway metadata from the v2 workbook - #1000

Merged
jacobvjk merged 8 commits into
epic/v2from
feat/import-metadata-pathways
Oct 23, 2026
Merged

jacobvjk merged 8 commits into
epic/v2from
feat/import-metadata-pathways

Conversation

@jacobvjk

@jacobvjk jacobvjk commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Imports the pathway descriptions from the authored workbook (pathway_data_prepared.xlsx, draft_ tabs): 75 pathways, no unmatched rows, no authoring errors. Refs #978: 24 of the 26 pathways still on metadata v1 are migrated, so they now appear on the site.

What changes on the site

Pathways
New NGFS Phase V, 21: one per scenario and model (GCAM, MESSAGE, REMIND). IEA WEO 2025 CPS, STEPS, NZE. AGF Malaysia OEP, OMAP. AGF Philippines MAP3, MAP4
Updated and now shown (were v1) SDSN Philippines (4), Singapore (6), CGS, IESR (2), UTS-ISF, TransitionZero (4), Viet Nam EREA/DEA (5) and others
Updated ACE, IEA WEO 2024, JRC GECO, PH DOE, JETP, SDSN Thailand, Myanmar, Laos
Removed The 7 per-scenario NGFS files, replaced by the 21. They were v1, so the site never showed them

Malaysia: EP MY now describes the Phase 2.2 report, which replaces 2.1. The two remaining v1 files, MAP MY and MAP1 MY (2.1), are removed in a separate PR.

Commits

  1. Importer (scripts/import-pathway-data.ts): every workbook row now has a target.
    • IEA keys carry the WEO edition year, because WEO 2024 and 2025 share scenario codes. Without it, the 2024 rows were merged into the 2025 files.
    • The 21 NGFS entries are generated from the scenario × model grid.
    • New publishers: CGS/IESR, UTS-ISF, TransitionZero, EREA/DEA, NGFS.
  2. Metadata import: the files written by the importer.
  3. JRC timeseries: re-imported from RMI/tpr_benchmark_data_preparation#28. GECO publishes "Rest South East Asia", not the South East Asia total the pipeline used to compute. The JRC metadata now uses GECO's region names.

Left out, listed in the import report

The TPR has no vocabulary yet for passenger transport metrics or Aviation and Steel technologies, so those values are dropped. Some workbook fixes are also still open, for example "Emissions intensity (total)" in the Metric column. None of these block the import.

Checks

  • npm run schema:check: 108 valid, 0 invalid.
  • npx vitest run: 1348 passing, including 15 new importer tests.
  • Re-running the import produces byte-identical files.

Related: #993, #994.

🤖 Generated with Claude Code

jacobvjk and others added 3 commits October 22, 2026 10:23
The workbook now carries 75 pathways; 34 of its rows matched no TARGETS
entry and were skipped. Every row now has one.

* IEA: keys carry the WEO edition year from "Name of publication", since
  WEO 2024 and WEO 2025 share scenario codes. Without it the 2024 STEPS
  and NZE rows were merged into the 2025 files. A row whose publication
  names no year stays unmatched rather than being guessed. WEO 2024
  STEPS, APS and NZE update the existing files.
* NGFS Phase V: 21 new pathways, one per scenario and model (GCAM,
  MESSAGE, REMIND), generated from the 7 x 3 grid. They replace the seven
  per-scenario files; Below 2°C is "B2C". The template supplies only the
  publication block and is one of the 21 itself.
* Malaysia (AGF Phase 2.2 replaces 2.1): EP MY is rebuilt as "new" over
  its existing file, so name, description and publication come from the
  workbook; OEP and OMAP MY are new. Philippines: SEP, OEP, MAP1, MAP2
  updated; MAP3, MAP4 new.
* JRC: the workbook's codes are now "(Reference)" and "(1.5°C)".
* New publishers: CGS/IESR, UTS-ISF, TransitionZero, EREA/DEA, NGFS, each
  updating its existing files. Viet Nam's "(BSL)" maps to the existing
  baseline, which keeps its short name "Baseline".
* The report's "#801 not addressed" note is gone: the workbook now
  carries NGFS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Imported with scripts/import-pathway-data.ts from pathway_data_prepared.xlsx
(draft_ tabs, 22 October): 75 pathways, no unmatched rows, no authoring
errors. 46 files updated, 16 of them migrated from v1, and 29 created.

* New: NGFS Phase V (21, one per scenario and model), IEA WEO 2025 CPS,
  STEPS and NZE, AGF Malaysia OEP and OMAP, AGF Philippines MAP3 and MAP4.
* Removed: the seven per-scenario NGFS files, replaced by the 21. They
  were v1, so the site never loaded them.
* Malaysia EP MY now describes the Phase 2.2 report, which replaces 2.1;
  MAP MY and MAP1 MY (2.1) are removed separately.
* JRC regions use GECO's own names, e.g. "Rest South East Asia". The
  computed 13-country "South East Asia" region is gone.

Values the workbook carries that the TPR has no vocabulary for yet are
left out and listed in the import report: passenger transport metrics,
Aviation and Steel technologies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GECO publishes its South East Asian countries and "Rest South East Asia",
but no South East Asia total. The preparation pipeline used to build that
total by summing GECO's regions, publish it as "South East Asia", and drop
GECO's own region.

* Timeseries: re-imported from RMI/tpr_benchmark_data_preparation (run
  20261021_T170540Z, PR #28), which now publishes "Rest South East Asia"
  instead of the total. Only the three JRC files change; the other 19
  imported datasets are identical to epic/v2.
* index.gen.ts regenerated with npm run build:timeseries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Expected version change and release notes

🚨 WARNING: This PR is not expected to trigger a new version

To trigger a version bump, use at least one conventional commit message in this branch. See: https://www.conventionalcommits.org/en/v1.0.0/

@github-actions

Copy link
Copy Markdown

Azure Static Web Apps: Your stage site is ready! Visit it here: https://proud-glacier-0f640931e-1000.westus2.2.azurestaticapps.net

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Incorrect country descriptions and incomplete coverage metadata would publish misleading information and exclude supported metrics from search.

5 open issues
What changed in this PR

Imports authored v2 pathway metadata into the site's data catalog, making previously hidden pathways available and expanding scenario coverage toward #978.

Changes:

  • Extends importer mappings, separates WEO editions, and adds tests for newly supported pathways.
  • Migrates and refreshes pathway descriptions, scoped features, dependencies, and data availability.
  • Removes legacy NGFS entries and updates the JRC timeseries index to use “Rest South East Asia.”
File Description
src/​data/​un-sdsn-cw/​SDSN-CW-SEP-PH-2024.json Migrates SEP to v2 metadata.
src/​data/​un-sdsn-cw/​SDSN-CW-OMAP-MM-2025.json Expands sectors and refreshes overview.
src/​data/​un-sdsn-cw/​SDSN-CW-OMAP-LA-2025.json Updates technology-cost availability.
src/​data/​un-sdsn-cw/​SDSN-CW-OEPP-MM-2025.json Expands sectors and refreshes narrative.
src/​data/​un-sdsn-cw/​SDSN-CW-OEP-PH-2024.json Migrates OEP to v2 metadata.
src/​data/​un-sdsn-cw/​SDSN-CW-OEP-LA-2025.json Updates technology-cost availability.
src/​data/​un-sdsn-cw/​SDSN-CW-NZE-TH-2024.json Revises emissions scope and cost resolution.
src/​data/​un-sdsn-cw/​SDSN-CW-MAP4-PH-2024.json Adds MAP4 pathway metadata.
src/​data/​un-sdsn-cw/​SDSN-CW-MAP3-PH-2024.json Adds MAP3 pathway metadata.
src/​data/​un-sdsn-cw/​SDSN-CW-MAP2-PH-2024.json Migrates MAP2 to v2 metadata.
src/​data/​un-sdsn-cw/​SDSN-CW-MAP1-PH-2024.json Migrates MAP1 to v2 metadata.
src/​data/​un-sdsn-cw/​SDSN-CW-MAP-TH-2024.json Revises emissions trajectory and scope.
src/​data/​un-sdsn-cw/​SDSN-CW-MAP-LA-2025.json Replaces overview and updates cost availability.
src/​data/​un-sdsn-cw/​SDSN-CW-HA2S-SG-2024.json Expands scenario dependencies.
src/​data/​un-sdsn-cw/​SDSN-CW-HA2O-SG-2024.json Expands and revises dependencies.
src/​data/​un-sdsn-cw/​SDSN-CW-HA1S-SG-2024.json Expands scenario dependencies.
src/​data/​un-sdsn-cw/​SDSN-CW-HA1O-SG-2024.json Adds policy, market, and technology dependencies.
src/​data/​un-sdsn-cw/​SDSN-CW-EXT-TH-2024.json Revises emissions trajectory and scope.
src/​data/​un-sdsn-cw/​SDSN-CW-EPP-MM-2025.json Expands sectors and clarifies emissions boundaries.
src/​data/​un-sdsn-cw/​SDSN-CW-EP-LA-2025.json Replaces overview and updates cost availability.
src/​data/​un-sdsn-cw/​SDSN-CW-BAU-SG-2024.json Expands scenario dependencies.
src/​data/​ngfs/​NGFS-NZ2050-2024.json Removes legacy per-scenario NZ2050 metadata.
src/​data/​ngfs/​NGFS-NDC-2024.json Removes legacy per-scenario NDC metadata.
src/​data/​ngfs/​NGFS-LD-2024.json Removes legacy per-scenario LD metadata.
src/​data/​ngfs/​NGFS-FW-2024.json Removes legacy per-scenario FW metadata.
src/​data/​ngfs/​NGFS-DT-2024.json Removes legacy per-scenario DT metadata.
src/​data/​ngfs/​NGFS-CP-2024.json Removes legacy per-scenario CP metadata.
src/​data/​ngfs/​NGFS-B2DS-2024.json Removes legacy per-scenario B2DS metadata.
src/​data/​jetp-id/​JETP-CIPP-2023.json Refreshes drivers, dependencies, and wording.
src/​data/​index.gen.ts Updates JRC regional geography labels.
src/​data/​cgs-iesr/​CGS-IESR-BAU-2024.json Migrates BAU to authored v2 metadata.
scripts/​import-pathway-data.ts Expands targets and adds edition-aware matching.
scripts/​import-pathway-data.test.ts Tests publisher matching, targets, editions, and templates.

Review effort: Balanced

Comment thread src/data/un-sdsn-cw/SDSN-CW-EP-LA-2025.json Outdated
Comment thread src/data/un-sdsn-cw/SDSN-CW-MAP-LA-2025.json Outdated
Comment thread src/data/un-sdsn-cw/SDSN-CW-MAP3-PH-2024.json Outdated
Comment thread src/data/un-sdsn-cw/SDSN-CW-MAP4-PH-2024.json Outdated
Comment thread src/data/un-sdsn-cw/SDSN-CW-SEP-PH-2024.json Outdated
jacobvjk and others added 2 commits October 22, 2026 11:50
parseSectorGroups split Technology coverage and Metric cells on every
";" before looking at brackets, so three of the workbook's four
spellings lost items without a word in the report:

* `Power: [a; b]` and `Power: a; b` kept only the first item;
* `a; b`, naming no sector, kept nothing.

That left 23 pathways with one technology or none (ACE, SDSN Laos,
Singapore, Myanmar and Philippines), and AGF Philippines MAP3/MAP4 with
one of their five metrics. ";" now separates groups only outside
brackets, an unlabelled item belongs to the last sector named, and an
unlabelled list goes to the pathway's only sector.

So it can't go silent again, the report now notes a list that names no
sector on a multi-sector pathway, and a cell from which no technology
could be read.

Found by review of #1000.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-imported from the workbook (22 October, 11:48) with the fixed list
parser:

* Technology coverage is now complete for ACE, SDSN Laos, Singapore,
  Myanmar and Philippines; AGF Philippines MAP3 and MAP4 list all five
  metrics.
* Workbook corrections: Laos EP and MAP had Myanmar's overviews; Myanmar
  names the sector of its technologies; Philippines MAP2 and MAP4 no
  longer list nuclear, which they exclude.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Expected version change and release notes

🚨 WARNING: This PR is not expected to trigger a new version

To trigger a version bump, use at least one conventional commit message in this branch. See: https://www.conventionalcommits.org/en/v1.0.0/

@github-actions

Copy link
Copy Markdown

Azure Static Web Apps: Your stage site is ready! Visit it here: https://proud-glacier-0f640931e-1000.westus2.2.azurestaticapps.net

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unresolved metadata inconsistencies affect emissions classifications, capacity search results, and policy descriptions.

No open issues 🎉

3 previously missed issues

In code that hasn't changed since last review

Medium severity Reconcile phaseout classification with no phase-out date

src/​data/​cgs-iesr/​CGS-IESR-BAU-2024.json:139

This new driver says no phase-out date applies, while keyFeatures.policyTypes classifies the same Power/ID pathway as using “Phaseout dates”. Users therefore get contradictory descriptions of the baseline's policy assumptions. Reconcile the workbook fields against the BAU source: remove the phaseout classification if retirement is purely age-based, or correct the assertion here if a policy phaseout applies, then regenerate the metadata.

Medium severity Add Capacity to the top-level metric array

src/​data/​cgs-iesr/​CGS-IESR-BAU-2024.json:209

This new availability row declares covered Capacity data, but the top-level metric array at line 58 contains only Absolute Emissions. The metric filter in src/utils/searchUtils.ts reads that array, so Capacity searches exclude this pathway despite the capacity figures described here. Add Capacity to the top-level list and keep the workbook/import behavior consistent so subsequent imports preserve the correction.

Medium severity Reconcile Philippine scope coverage and emissions trends

src/​data/​un-sdsn-cw/​SDSN-CW-MAP1-PH-2024.json:70

All six Philippine records (SEP, OEP, and MAP1–MAP4) declare only Power but import different emissions trends for across sectors and Power at geography PH. Those scopes cover the same declared sector: for example, MAP1 and MAP2 assign both “Minor increase” and “Significant decrease”, and OEP assigns both an increase and a decrease. The scope helpers select the aggregate for the overview and include both values in Power-filtered searches, producing misleading classifications. Reconcile the workbook scopes with the declared coverage across all six records and regenerate them. Remove out-of-coverage aggregate entries, or declare the additional sectors if the source genuinely covers them.

5 resolved since last review

Review effort: Balanced

jacobvjk and others added 2 commits October 22, 2026 17:57
The Metric column was read only for new pathways; an update kept the
metric list already in its file. That list drives the site's metric
search, so a stale one hid pathways from searches for metrics they cover:
CGS/IESR's workbook lists Capacity, but the site never showed it.

* Updates now take their metrics from the workbook too. An update whose
  cell yields nothing keeps its own list, and the same metrics in a
  different order keep the file's order, so a re-import touches only
  files whose metrics actually changed.
* The report notes each metric that data availability marks as covered
  but the Metric column leaves out, the mismatch that hides a pathway
  from search.

Found by review of #1000.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-imported from the workbook (22 October, 17:52), now reading the
Metric column for updates:

* CGS/IESR BAU and 1.5°C add Capacity, and SDSN Laos EP, OEP and MAP add
  Emissions Intensity, all of which their data availability covers.
* SDSN Singapore BAS drops Capacity, which it doesn't cover.
* CGS/IESR BAU's policy driver says what is absent ("no early retirement
  date for existing plants"), no longer contradicting its "Phaseout
  dates" policy type: the ban on new coal plants is the phase-out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jacobvjk

Copy link
Copy Markdown
Collaborator Author

Phaseout classification (CGS-IESR-BAU): The classification is intended: the ban on new coal plants, together with existing plants retiring at end of life, effectively sets a phase-out date. The driver text said "no phase-out date applies", which read as a contradiction. It now says "no early retirement date for existing plants" (5767bda).

Capacity missing from metric (CGS-IESR-BAU): Right, but the cause was the importer, not the workbook: for existing pathways it kept the old metric list and ignored the workbook's Metric column, which does list Capacity. Fixed in 8a665fe, and the report now flags any metric that data availability covers but the Metric column omits. Re-imported in 5767bda, which also added Capacity to CGS-IESR 1.5°C and Emissions Intensity to Laos EP/OEP/MAP.

Philippine "across sectors" vs Power: Leaving this as is for now. These AGF pathways only cover Power in a way that fits our sector list; the across-sector information is economy-wide and doesn't map to further sectors, so none are declared. We'll revisit how across-sector entries are shown for single-sector pathways separately.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Conflicting emissions scopes in six Philippine pathways produce incorrect displayed classifications and contradictory search matches.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Correct emissions scopes for six Philippine pathways

src/​data/​un-sdsn-cw/​SDSN-CW-MAP1-PH-2024.json:70

All six Philippine pathways (SEP, OEP and MAP1–MAP4) declare only Power, but their new across sectors emissions entries disagree with their Power entries. In src/utils/keyFeatureScope.ts, across sectors means the union of declared sectors, so these scopes are equivalent. widestValue() prefers the aggregate entry: MAP1 and MAP2 display “Minor increase” instead of “Significant decrease”, and OEP displays “Moderate increase” instead of “Minor decrease”. SEP, MAP3 and MAP4 likewise display the aggregate classification, while Power-filtered searches match both values. Correct the emissions scopes in the workbook and regenerate all six records, retaining the Power trajectories for these Power-only pathways.

Review effort: Balanced

@github-actions

Copy link
Copy Markdown

Expected version change and release notes

🚨 WARNING: This PR is not expected to trigger a new version

To trigger a version bump, use at least one conventional commit message in this branch. See: https://www.conventionalcommits.org/en/v1.0.0/

@github-actions

Copy link
Copy Markdown

Azure Static Web Apps: Your stage site is ready! Visit it here: https://proud-glacier-0f640931e-1000.westus2.2.azurestaticapps.net

@jacobvjk

Copy link
Copy Markdown
Collaborator Author

Review guide

There's no need to read the authored text line by line. Content accuracy is the workbook authors' responsibility, and CI enforces the formal rules. What does need a human eye, in priority order:

1. Importer logic (scripts/import-pathway-data.ts)

This is the only code change. Four decisions are worth checking:

  • WEO editions (canonicalKey): IEA rows are matched by the year in Name of publication, on every tab. If one tab drops the year, that tab's rows for the pathway go unmatched. The import report then says "unmatched" or "no key_features rows", but nothing fails.
  • List parsing (parseSectorGroups): it reads all four spellings the workbook uses for Technology coverage and Metric. Before, three of them were silently cut short. Should the importer accept all four, or should authors use a single format?
  • Metrics for updates: these now come from the workbook. The file keeps its own list only if the cell is empty or lists the same metrics in a different order. This is a deliberate change in behaviour.
  • New vs. update (TARGETS): this decides file identities.
    • EP MY and IEA WEO 2025 are rebuilt as "new" over their existing files.
    • Viet Nam's "BSL" maps to the existing "Baseline" file.
    • The 21 NGFS pathways are generated from the scenario × model grid and share one template.

2. Which pathways exist afterwards

  • Added (25): NGFS (21, one per scenario and model), Malaysia OEP and OMAP, Philippines MAP3 and MAP4.
  • Deleted (7): the per-scenario NGFS files. They were v1 and never shown, so no live links break.
  • No existing pathway changes its ID.
  • 24 pathways become visible for the first time (26 pathways are still on metadata v1 and don't appear on the site #978), so the preview will look noticeably fuller.

3. Report details on new pathways

For new pathways, the publisher, licence and links are copied from a template rather than authored. I checked every link in the changed files:

  • All links on the new pathways work.
  • Three links were already broken and are not introduced here:
    • The IEA WEO 2024 "free dataset" page returns 404.
    • Both Philippines DOE links (legacy.doe.gov.ph) don't respond.
  • ngfs.net (403) and TransitionZero (429) block scripts, so check those two in a browser.
  • To confirm: should all 21 NGFS pathways share the report-level DOI and licence?

4. Region membership

Regions are rebuilt from the workbook: JRC now uses GECO's own names, and IEA and others are re-grouped. That changes the region filter in search. CI checks that the published data only uses declared regions. It does not check that the right countries are in each region.

5. Spot checks on the preview (about 15 minutes)

  • One NGFS scenario across its three models: are the values distinct?
  • IEA STEPS 2024 vs. STEPS 2025: two separate pages with their own content?
  • Malaysia EP MY: does it show the Phase 2.2 report and link?
  • A pathway that was v1 until now, e.g. TransitionZero or UTS-ISF.
  • JRC charts: is "Rest South East Asia" there?
  • Search: does Metric = Capacity include both CGS, IESR pathways, and Technology = Solar include ACE, Laos and Singapore?

Covered by CI and tests, so you can skip these

  • Schema validity, allowed values and character limits.
  • Key-feature and data-availability scopes against the declared sectors and regions.
  • Timeseries regions against the declared regions.
  • Formatting and the generated index.
  • A second import producing identical files (checked locally).

Known and out of scope, so no need to flag

  • No TPR vocabulary yet for passenger transport metrics or for Aviation and Steel technologies, so those values are dropped.
  • "AN" (Netherlands Antilles) in the NGFS regions.
  • The Philippines "across sectors" emissions.
  • "Emissions intensity (total)" in the workbook.
  • Removing Malaysia 2.1 (MAP MY, MAP1 MY), which is a separate PR.

@jacobvjk
jacobvjk requested a review from AlexAxthelm October 22, 2026 16:57
From review of #1000:

* JRC names written without a code in brackets fell back to "REFERENCE"
  and "1.5C", which no longer match the TARGETS keys. The fallback now
  uses the workbook's codes ("Reference", "1.5°C").
* Sub-sheet rows that belong to no metadata row were dropped without a
  word, e.g. a key features row whose WEO title lacks the year. The
  report now lists them.
* A Metric cell with items outside the vocabulary was only partly read,
  and the shorter list replaced an update's metrics. Such a cell may now
  add metrics but never remove any; the report names what was kept.
* A missing template only produced a note, and the new file was written
  without its publication block. It is now an authoring error.
* The WEO edition year is taken after "Outlook" or "WEO", falling back
  to the first year in the title.

Re-importing the current workbook leaves every data file unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Expected version change and release notes

🚨 WARNING: This PR is not expected to trigger a new version

To trigger a version bump, use at least one conventional commit message in this branch. See: https://www.conventionalcommits.org/en/v1.0.0/

@github-actions

Copy link
Copy Markdown

Azure Static Web Apps: Your stage site is ready! Visit it here: https://proud-glacier-0f640931e-1000.westus2.2.azurestaticapps.net

@AlexAxthelm AlexAxthelm left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments from Claude below. I let it take a first pass at this, but I'm not sure which of these are actual issues, without knowing more about the data.


Review of PR #1000 (import pathway metadata). Your latest commits fix the per-sector list parsing, and the re-imported metrics match the timeseries. These are still open, most important first:

  1. JRC fallback codes no longer match the TARGETS keys (import-pathway-data.ts:645). scenarioParts still falls back to "REFERENCE" and "1.5C", but the keys are now "JRC:Reference:" and "JRC:1.5°C:". A JRC row with no code in parentheses (e.g. "Reference Scenario" in a sub-sheet) matches nothing and is dropped without a report.

  2. Unmatched sub-sheet rows are dropped without a report (import-pathway-data.ts:1811). groupByKey skips sub-sheet rows that match no target. IEA keys now take the year from each sub-sheet's publication cell, so a blank or year-less one keys to "IEA:STEPS:" and its key features disappear. The "Unmatched Excel rows" report only covers the metadata sheet.

  3. A partial workbook metric list overwrites an update's metrics (import-pathway-data.ts:1700). If one Metric entry fails the vocabulary check (e.g. the still-open "Emissions intensity (total)"), the shorter list replaces the file's correct one, and the import only adds a note. The pathway then disappears from that metric's search. Current data isn't affected (BAS-SG dropping Capacity is correct). Suggest making this an error, or keeping the existing list.

  4. The NGFS template is one of its own outputs (import-pathway-data.ts:609). All 21 NGFS targets use NGFS-NZ2050-GCAM-2024.json as their template, and EP MY uses itself. If the template is missing, the import still exits successfully but writes all 21 files without a publication block. Suggest making a missing template an error.

  5. The bundle grows by about 4 MB (pathwayMetadata.ts:14). Metadata JSON goes from about 0.9 MB to 4.8 MB, 3.1 MB of it NGFS (about 138 KB of keyFeatures per file). All of it loads up front on every page. Worth considering lazy loading for keyFeatures/dataAvailability.

  6. A mixed technology cell loses items silently (import-pathway-data.ts:1500). For a single-sector pathway, "Coal; Power: [Gas, Solar]" drops Coal with no note. A colon in a bare item ("Gas: CCGT") is read as a sector label, and that item is lost.

  7. The IEA edition year is the first 20xx in the publication title (import-pathway-data.ts:669). A title that mentions another year first would key the row to the wrong WEO edition. Suggest matching the year right after "Outlook" or "WEO-".

  8. The link-year update can corrupt a URL (import-pathway-data.ts:1788). url.replace(oldYear, year) changes only the first match, wherever it is in the URL, so a link like /2024/10/weo-2024.pdf would be only partly updated.

  9. JRC no longer has an overall South East Asia region (JRC-GECO-*-2025.json). Only "Rest South East Asia" remains, which leaves out ID, MY, TH and VN, so a JRC regional value next to IEA's "Southeast Asia" or ACE's "ASEAN" covers different countries. Worth confirming that this is intended.

  10. No test checks that sub-sheet keys match their metadata row (import-pathway-data.test.ts:104). Such a test would catch #1 and #2. Minor: the empty sub fixture is repeated three times in the new tests.

@jacobvjk

Copy link
Copy Markdown
Collaborator Author

@AlexAxthelm thanks, all ten checked against the code and the current workbook. Six are fixed in 3d2df76, three are tracked as issues, and one is intended.

Fixed in 3d2df76. Re-importing the current workbook leaves every data file byte-identical, so none of these changes any data.

  1. JRC fallback codes: the fallback now returns the workbook's codes (Reference, NDC-LTS, 1.5°C), so those rows reach their target. Covered by a test.
  2. Unmatched sub-sheet rows: the report now lists every sub-sheet row whose pathway has no metadata row ("Sub-sheet rows with no matching metadata row, not imported"). Covered by a test with a year-less WEO title.
  3. Partial metric list: fixed slightly differently from your suggestion. A Metric cell with unrecognised items may still add metrics to an update, but never removes any, and the report names what was kept. Keeping the old list outright would have frozen 11 pathways whose cells list transport and steel metrics the TPR doesn't define yet.
  4. Missing template: now an authoring error, so nothing is written.
  5. IEA edition year: the year after "Outlook" or "WEO" wins, with the first year as fallback.
  6. Test for sub-sheet keys: added (see 2), and the empty fixture is now one shared constant.

Tracked separately
5. Bundle size: #1003 (lazy-load key features and data availability).
6. Mixed or colon-containing technology cells: #1004. Neither occurs in the current workbook; an audit found no truncated lists.
8. Link-year rewrite: #1005. All links on the new files were checked and resolve. A general fix needs a decision: take links from the workbook, or flag rewritten links for a manual check.

Intended
9. JRC South East Asia: the TPR only hosts regions the publication itself reports (RMI/tpr_benchmark_data_preparation#28). GECO publishes "Rest South East Asia", not a total, so a JRC regional value isn't directly comparable to IEA's "Southeast Asia" or ACE's "ASEAN".

@jacobvjk
jacobvjk requested a review from AlexAxthelm October 23, 2026 13:59

@AlexAxthelm AlexAxthelm left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, thanks

@jacobvjk
jacobvjk merged commit f35d37d into epic/v2 Oct 23, 2026
11 checks passed
@jacobvjk
jacobvjk deleted the feat/import-metadata-pathways branch October 23, 2026 15:18
@github-actions github-actions Bot mentioned this pull request Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants