dataAvailability field + download/link table #870

Activity

1. AlexAxthelm

   self-assigned this

   on Aug 26, 2026
2. AlexAxthelm

   added 2 commits that reference this issue on Aug 27, 2026

   Add authored per-metric dataAvailability (#870)

   ...

   7dbb439

   Merge branch 'main' into feat/metadata-v2-schema

   ...

   7f9e1c2
3. jacobvjk

   added a commit that references this issue on Sep 11, 2026

   feat(ui): source segment tooltips from the timeseries taxonomy

   ...

   bd75a8b
4. jacobvjk commented on Sep 11, 2026

   

   jacobvjk

   on Sep 11, 2026

   Collaborator

   More actions

   Status after #936 and #946.

   • The table is built (src/components/DataAvailabilityTable.tsx, Feat/UI tabs #936) and is
     now filtered by the detail page's scope selection (Feat/detail scope filters #946), as the schema
     comment on byMetric[].geography anticipated.
   • Feat/detail scope filters #946 also fixed a latent bug: rows were keyed by (metricName, sectorSegment,
     geography), omitting sector from the uniqueness tuple the schema enforces,
     so a multi-sector pathway reporting the same metric twice rendered one row
     twice. Sector and geography scope are now visible as conditional columns.

   Blocker on observing any of it: no pathway in src/data/ has
   byMetric populated. Only testdata/valid/pathwayMetadata_v2_full.json and
   test fixtures exercise the table, so this ships invisible in the running app.
   Authoring it for one real pathway is probably worth its own task.

   Vocabulary change to be aware of (#936): sectorSegment renamed Storage
   → Energy storage, and a new Fuel extraction and processing segment was
   added. Schema enum, generated types and fixtures updated; no authored data
   needed migrating because none used a named segment.
5. jacobvjk commented on Sep 24, 2026

   

   jacobvjk

   on Sep 24, 2026

   Last edited by jacobvjk

   Collaborator

   More actions

   This PR splits dataFormat into two separate columns: dataFromat and access: https://github.com/RMI/tpr/pull/898/changes#diff-0dc51340471f147cebec4629b65693571cc18b10b34e328fb7ddac1e5f1e011b

   The rational is that one describes the actual format that the data is available in and the other describes how users can access that information. That is a good idea. However, we only host data and pathways that are available for free at this point anyway, and the "free" value in the access variable does not really convey the details that may come with data licenses anyway. Likewise, there is no point in mixing information on data format/access within this tool (value: "in tool") with source data format information. So "in tool" should be dropped. The final version of this variable should instead be the following:

   Field	Type	Notes
   metricName	enum	sector-conditional; shares vocabulary with metric
   sector	enum	existing sector enum
   sectorSegment	enum	sector-conditional (e.g. Power: Power generation / Energy storage / T&D / …)
   geographyCoverage	enum	global / regional / country
   timeResolution	enum	2050 point / medium-term / 10-yr / 5-yr / 1-yr / 5–10-yr / other / unavailable
   dataFormat	enum	tabular / text / figure
   granularity	string[] | null	metric-conditional (e.g. Power capacity: Coal/Gas/Wind/Solar/…)
   scopeLimitations	string | null	free text
6. jacobvjk commented on Sep 24, 2026

   

   jacobvjk

   on Sep 24, 2026

   Collaborator

   More actions

   Following up on my comment above — dataFormat is now authored in the cookbook, so the vocabulary is settled. RMI/tpr-tpc#20.

   The allowed values

   dataFormat is a single-valued enum with exactly four members, spelled as follows:

   Value	Meaning
   Tabular	Values are in a table or a machine-readable data file, readable directly
   Text	Values appear only in prose
   Figure	Values appear only in a chart, plot or map, and must be read off the graphic
   Not covered	The sector-metric combination is not covered

   Spelling matters — these are the literal strings. Two differences from what's in dataAvailability.v1.json on the #898 branch:

   • Not Tabular in publication / Text in publication. With In tool gone, every value is in-publication by definition, so the qualifier carries nothing.
   • Capitalised first letter only. My earlier comment wrote tabular / text / figure as shorthand; the cookbook's house style for allowed values is Tabular, Text, Figure, matching Not covered, Medium-term and the rest.

   What needs to change on this side

   1. Drop access entirely, including the validateScopedEntries biconditional that ties it to In tool. There's nothing left for it to hold.
   2. Drop In tool. dataFormat describes the source publication only. Whether we host a file is something the repository knows about itself, not a classification judgement about the pathway — that was the whole point of separating the two. If the "we serve this / go to the publisher" signal is still wanted in the UI, it should be derived from the timeseries index as it is today, or become its own field. It should not come back as a dataFormat member.
   3. Add Not covered. This one isn't in Feat/metadata v2 schema #898 at all and is easy to miss. The cookbook requires a row for every allowable (sector, metric) pair in a covered sector, even when the pathway reports nothing for it — an omitted row is a completeness defect, not an absence. So Not covered is a real value that will appear in authored data, not a null.
   4. There is deliberately no Unspecified, unlike timeResolution. A metric whose values you can see at all has a visible form.

   One rule that affects authoring

   When a metric appears in more than one form, the most extractable wins: Tabular > Text > Figure. This comes up immediately — AGF Laos justifies eight rows with "Technology-level reporting in tables/figures", so a publication that prints a chart and ships the underlying spreadsheet is Tabular.

   Related: judge where the metric's values are, not where it's mentioned. JETP CIPP's Power technology cost is Figure — the LCOE numbers are in Figure 5.2-10 — even though surrounding prose references cost baselines.

   Provenance

   Cookbook: docs/cookbook/split/data_availability/data_format.md, cookbook version tpr_cookbook_20260924. Reasoning and the rejected alternatives are in decision record 0021. The classification template gained Data format and Data format_Justification.

   Worth knowing for planning: the classification side has the columns and the validator enforces the enum, but no pathway has authored Data format values yet — the gold workbooks carry NULL pending an authoring pass. So this unblocks the schema and UI work, but real values arrive separately, same as the byMetric situation flagged in the #936/#946 status above.
7. jacobvjk

   added a commit that references this issue on Sep 25, 2026

   feat(schema): align dataAvailability with the cookbook vocabulary

   ...

   b01b0b7
8. jacobvjk commented on Sep 25, 2026

   

   jacobvjk

   on Sep 25, 2026

   Collaborator

   More actions

   The vocabulary is in, but the table can't be filled in yet

   The schema and UI now match cookbook tpr_cookbook_20260924 — dataFormat is
   Tabular / Text / Figure / Not covered, access and In tool are gone, the time
   resolutions use the cookbook's exact wording, geography coverage is the list of
   geographies rather than a Global/Regional/Country label, and Unspecified vs
   Not covered replaces blank cells everywhere. Pushed to #898, #936, #946 and #954.

   Before anyone starts an authoring pass, though, there's a gap worth knowing about. It
   isn't a problem with this ticket's design — it's a collision with something #860 parked.

   The tool only knows 5 metrics; the cookbook's table needs up to 13

   The Data Availability section has its own metric list — the cookbook calls it
   "Metrics – Extended" and is explicit that it's a different thing from the pathway-level
   Metric field: it's what defines a row in the table.

   The tool currently accepts five metrics anywhere: Emissions Intensity, Capacity,
   Generation, Technology Mix, Absolute Emissions. The cookbook's table wants:

   Sector	Metrics the table needs	We can record
   Power	9	5
   Steel	10	2
   Aviation	13	0

   (The Steel count is generous: it assumes our single "Emissions Intensity" stands in for
   the cookbook's three — total, primary and secondary — which it can't distinguish.)

   Missing across every sector: Technology cost, Investment requirement, Asset
   lifetime. Plus Transmission lines for Power; Scrap share, Steel production by
   technology and the split Emissions intensity (total / primary / secondary) for
   Steel; and all the Aviation demand metrics.

   In practice: a row for one of those is rejected when the file is saved. So today you
   could fill in about half of a Power pathway's rows and none of the Steel- or
   Aviation-specific ones.

   A second, less obvious blocker

   The tool also currently requires every Data Availability row to name a metric the
   pathway already lists in its own top-level "Metrics" field. The cookbook says the
   opposite: a covered sector gets a row for every metric in the extended list,
   whether or not the pathway reports it — an unreported one is recorded as Not covered.

   So that rule has to go as well. While it stands, Not covered rows are impossible to
   write, which is exactly the completeness rule the section is built around.

   Why this wasn't just fixed alongside the rest

   Two reasons, and both need a decision rather than code.

   It's user-facing. The metric list isn't only used here — it's the Metrics filter
   dropdown on the search page. Adding nine-plus values changes what every visitor sees,
   so it isn't a quiet internal change.

   The cookbook hasn't settled it either. Its own register (T1) flags that the Data
   Availability metric list has Transmission lines where the pathway-level list has
   Storage capacity, notes the two disagree, and records it as unresolved.

   This is the "sector-aware restructuring of the metric field" that #860 lists under
   Out of scope ("separate future release; structure TBD"). So it belongs in its own
   ticket rather than being smuggled in here.

   Suggestion

   Raise it as a ticket now and sequence it before any authoring pass, since the
   cookbook's "a row for every allowable pair" rule can't be met until it lands. Two things
   to decide:

   1. One metric list or two? Is Data Availability's list a superset of the
      pathway-level one, or a genuinely separate vocabulary that happens to overlap? That
      determines whether the search dropdown grows.
   2. Transmission lines or Storage capacity for Power — the cookbook's two lists
      disagree, and we'd be picking a side.

   What did land

   Everything else in the section. granularity also gained the emissions-scope values
   (Scope 1, Scope 1 & 2, …) that Absolute Emissions and Emissions Intensity need,
   since those two metrics do exist. The rest of the cookbook's per-metric granularity
   vocabularies — the Steel production routes, the Aviation propulsion types — are waiting
   on the same decision, because they hang off metrics the tool can't name yet.

   Worth noting alongside the byMetric situation already flagged above: no pathway in
   src/data has any of this authored, so nothing in the running app changes yet.
9. jacobvjk

   added 3 commits that reference this issue on Oct 2, 2026

   feat(schema): complete the sector-conditional vocabularies

   ...

   57d93c6

   feat(import): import data availability, and stop the import losing data

   ...

   1a610a3

   data: import data availability, restore 13 descriptions, add Singapore

   ...

   956a0cc
10. jacobvjk commented on Oct 2, 2026

    

    jacobvjk

    on Oct 2, 2026

    Collaborator

    More actions

    Correction, and the table now has real data

    Correction to my earlier comment. I said the extra metrics (Transmission lines, Technology cost, Investment requirement, Asset lifetime, and the Steel and Aviation lists) were blocked behind #860's deferred restructure of the metric field, because adding them would change the search filter. That was wrong. Cookbook register item D16 says the pathway-level Metric list and the Data Availability Metrics – Extended list are meant to differ. So the availability table got its own metric list, and the search filter is unchanged. Nothing was blocked. This is done on #898.

    Where things stand. The import on #935 now fills the table from the draft_data_availability sheet: 171 rows across 24 pathways. You can see it on the #937 preview, for example ACE-ATS-2024 → Scope & Granularity.

    Held back, and why. 45 of the sheet's 216 rows are not imported yet, each named in the import report:

    Rows	Pathways	What needs authoring
    35	all six Singapore pathways	Data Format is blank. It's required, and I didn't want to guess it. Until it's filled in, Singapore shows only its uncovered metrics, which reads as if it covers almost nothing.
    10	IEA CPS / STEPS / NZE 2025 (2 each), the four Laos pathways (1 each)	Data Format says "Not covered" while the rest of the row describes a covered metric. In the cookbook, "Not covered" means the sector–metric pair isn't covered at all, so these rows contradict themselves. Probably a real format (Tabular / Text / Figure) was meant.

    One more to check: for Singapore BAS SG, the metadata lists Capacity as a reported metric, but its availability row says Capacity is "Not covered".