feat(emissionFactors)!: identify factors by reporting year and unit family by mrivas00 · Pull Request #627 · undp/carbon-footprint-program

mrivas00 · 2026-09-02T12:39:01Z

What & why

The reporting year lived inside the free-text source ("DEFRA 2025"), so the catalog could not hold two vintages of the same factor, and uniqueness was enforced on the wrong key.
Makes the year a first-class column, keys factor identity on (dimensions, year, source, unit family), and gives maintainers a separate year field instead of asking them to type it into the name.
Plan and rationale: openspec/changes/add-emission-factor-year/ (merged by docs/mati/emission-factor-year-plan).

Risk / blast radius — read first

Breaking (DB): drops emission_factor_unique_subcategory_dims_source and replaces it with a partial unique index; adds two NOT NULL columns backfilled in-migration. Deploy order and the parked-migration procedure are in rollout.md; verify-migration.sh is the preflight.
Breaking (types): EmissionFactor responses and factor create/update forms gain year. No consumer outside this PR reads it yet.
Both seed catalogs (base + testing) are reclassified — ~1.8k lines of the diff is that data.
The line-sync request contract is not touched here; that is the next PR.

Key changes & decisions

1. Unit-family magnitudes denormalized onto the factor, derived server-side

schema.prisma, migration.sql, measurementUnits/helpers.ts (+4) · 5ceb39c, a4e9c4f

Trigger — the unique index has to compare unit families, but a partial index cannot join rate_measurement_unit.
Change — numerator_magnitude_id / denominator_magnitude_id are stored on the factor, always derived from the selected rate unit by a shared helper reused by create, update and seed.
Why this way — denormalizing is what makes the constraint expressible in the database rather than in application code that a second write path could bypass. They are deliberately not exposed as writable client fields: letting a client send them would let it forge factor identity.

2. Partial unique index with `NULLS NOT DISTINCT`, excluding soft-deleted rows

migration.sql, schema.prisma · 5ceb39c

Why this way — default SQL null semantics would let unlimited rows with a null dimension or null year bypass uniqueness entirely; and scoping to status <> 'DELETED' keeps a soft-deleted row from permanently blocking re-creation of the same factor.

3. `validateSourceConsistency` removed

emissionFactors/helpers.ts, errors.ts (+3) · a4e9c4f

Trigger — it rejected a subcategory holding factors from two different sources.
Change — deleted, together with its error type and message mapping.
Why this way — with year as its own axis, DEFRA (2025) and IPCC in the same subcategory is a legitimate catalog, not a conflict. The uniqueness index is now the only authority on duplicates, so keeping a second overlapping rule would only produce false rejections.

4. Duplicate lookup omits optional dimension slots instead of matching them as null

emissionFactors/helpers.ts · 6cb25f5

Why this way — a subcategory with one required dimension stores nothing in slot 2; matching dimensionValue2Id: null explicitly made the lookup miss real duplicates. Omitting the unused slot matches what the normalized index compares.

5. Maintainer: year is its own field; the name-contains-a-year check is a warning, not a validation

useEmissionFactorColumns.tsx, emissionFactorSourceGuidance.ts (+10) · 75ea42e, c49cf52, 90c62a3

Change — separate nullable year column requiring an explicit blank for transversal, persistent helper text asking that the source name omit the year, and a non-blocking warning when a four-digit year is detected. Catalog duplicate errors now explain the conflict by source, year, dimensions and unit family, in Spanish.
Why this way — a blocking API validation on "source looks like it has a year" would reject legitimate names and fight the maintainer during the transition, while the data is already protected by the index. The warning nudges without gating.

6. Seed year is required-but-nullable

seedEmissionFactors.ts, shared.ts, both methodologies.json · c582806

Why this way — an omitted year fails validation while an explicit "year": null passes. This forces the classification from decision 1 of the plan to be stated per factor instead of defaulting silently to transversal.

Deliberate deviations

478f1bf patches only the fixtures of syncCarbonInventoryLines/integration.test.ts (adds the derived magnitudes) so the suite compiles against the new NOT NULL columns. The full rewrite of that test for the new request contract lands in the next PR. Without this the intermediate branch would not type-check — it is the cost of splitting the original single branch, not a design choice.

Test plan
Automated gates that actually ran on this branch:

pnpm format:check && pnpm lint && pnpm type-check — green.
pnpm test:api -- /emissionFactors --coverage=false — 60/60 (includes the new emissionFactorVintages suite).
pnpm test:api -- /carbonInventories --coverage=false — 534/534.
pnpm test:web — 837/837. pnpm test:seed — 4/4.
Every api run applies prisma migrate deploy + seeds the testing dataset on a fresh Postgres, so the migration and the reclassified testing catalog are proven against the new index on a fresh database. The remaining api domains run in the CI matrix.

Manual, by risk — not run in this session:

openspec/changes/add-emission-factor-year/verify-migration.sh against a restored current-data database: unclassified factor fails the preflight; provider/year split lands as expected; no non-required dimension slot is erased.
Maintainer → emission factors: create two factors same source different year → both saved; same source, year and unit family → duplicate error naming all four axes; type DEFRA 2025 in the name → warning appears, save still allowed.
Methodology Excel export: year is its own column, transversal rows render an empty year cell.

Chain — merge in order:

docs(openspec): plan emission factors by reporting year #626 — openspec plan — decisions, specs, rollout (docs only)
feat(emissionFactors)!: identify factors by reporting year and unit family #627 — schema, catalog identity, maintainer year ← this PR
feat(carbonInventory)!: make the server authoritative over catalog factor snapshots #628 — server-authoritative snapshots, vintage ranking, mismatch warning
feat(carbonInventory): keep a line's stored factor when the request says unchanged #629 — UNCHANGED variant + batched catalog lookups

Out of scope

Line-sync contract, server-authoritative snapshots, vintage ranking in capture, mismatch warning → feat/mati/emission-factor-server-authority.

Scope note: consolidation now covers seven industrial subcategories

The unit-family consolidation in this change originally covered four
subcategories (16 collision groups): Acero 4, Cemento 1, Cinc 3, Vidrio 8.

Rebasing onto the current base brought in the per-process industrial catalogue
main added in 9ccbf3a, which carries each factor twice — once as kg/ton and
once as the equivalent kg/kg. That adds three subcategories and 19 groups:
Ferroaleaciones y otros metales 14, Cal 3, Aluminio 2. Consolidation therefore
covers seven subcategories and 35 groups.

They are treated by the same rule already applied here: kg/ton survives and the
kg/kg twin is retired, which is also the migration's own tie-break (the larger
denominator base factor wins, keeping the most significant digits inside
Decimal(28, 10)). Every pair is an exact equivalent (750 kg/ton is 0.75 kg/kg),
so no group needed methodology review. Their 38 rows arrived without a
reporting year and all carry source IPCC, already reviewed here as transversal,
so the year is written out as an explicit null.

Seeded active factors after consolidation: 249 in the base dataset, 248 in
testing, with no index collisions.

No migration code changed for this: it groups by unit family and names no
subcategory. What is now stale is the plan's audit write-up in
openspec/changes/add-emission-factor-year/factor-classification.md, which still
says "16 collision groups", lists only the four subcategories and says "All four
subcategories allow g, kg and ton" (analysis.html repeats the 16). That file
belongs to the base PR, so it is flagged here rather than edited.

mrivas00

Automated review pass over the whole branch: 14 findings (3 alta, 5 media, 6 baja). 11 are inline below; the 3 that land on files or lines outside this PR's diff are grouped here.

For context, four of the baja findings are chain-sequencing observations rather than defects: I verified against feat/mati/emission-factor-server-authority (PR 628) that isSameMagnitudeFamily, the CATALOG_EMISSION_FACTOR_* codes, appliedFactorYear writes and the year read paths all acquire their consumers there. They are flagged only because this branch is reviewable and mergeable on its own.

Grouped findings (outside this diff)

[alta] apps/web/src/screens/Maintainer/components/cells/EditableNumberCell.tsx:50 — a malformed year (202S, 2o25) goes through Number(localValue) to NaN and then onChange(null), and null in the new model means "confirmed transversal", so a typo is silently stored as a deliberate declaration; the value column is shielded from this only because its schema rejects null, while year accepts it. — Fix: stop collapsing unparseable input to null (forward NaN so the resolver rejects it, or reject the keystroke) and keep the empty string as the only path to null.

[baja] apps/api/src/features/carbonInventories/syncCarbonInventoryLines/helper.ts — within this branch alone nothing ever writes carbon_inventory_line_factor.applied_factor_year: it is set by the migration's backfill and by nothing else, so the column is accurate only until the first re-sync. PR 628 adds the writer (appliedFactorYear: factor.year) and the duplicateCarbonInventory copy, so this resolves in the chain — worth noting only because 627 is independently mergeable. — Fix: none needed if 627 and 628 land together; otherwise pull the two writes back into this PR.

[baja] tools/seed/src/data/base/methodologies.json / read schemas — this PR strips the year out of every source string while no read path outside the maintainer exposes year, so on 627 alone a user who saw "DEFRA 2025" in the factor selector now sees "DEFRA" with the vintage nowhere on screen. PR 628 adds appliedFactorYear to getEmissionFactors and a dedicated year column in the summary grid, which closes it. — Fix: none needed if the two PRs ship together; flagging it as a deploy-order constraint, not a code change.

… family A factor used to carry its vintage informally inside `source` ("DEFRA 2025"), and the index keyed identity on (subcategory, dimensions, source). That made a second vintage for the same activity impossible, and let two rate units in the same physical family — kg/kg and kg/ton are both mass/mass — sit in the catalog as separate rows. `emission_factor` now stores `year` explicitly, where NULL means the factor is confirmed transversal and applies to every reporting year; it never means "unknown". It also denormalizes the numerator/denominator magnitude pair, because Postgres cannot reach those through `rate_measurement_unit` from a unique index, and identity has to compare the unit *family* rather than the exact unit. `carbon_inventory_line_factor` gains `applied_factor_year` so a saved line stays reproducible; whether it disagrees with the footprint year is derived at read time, never stored, or it would go stale the moment someone re-dated the inventory. The migration refuses to guess. An unclassified source aborts the whole transaction rather than defaulting to transversal, and two same-family factors whose values disagree after conversion abort too — that is a methodology question, not a migration one. Duplicate unit representations are retired by soft delete after captured lines are repointed at the survivor, since the FK is ON DELETE SET NULL and would otherwise drop those lines' catalog link. No applied value, unit, source or result is rewritten. BREAKING CHANGE: the API requires the new columns, so the migration must run before the new API starts. See openspec/changes/add-emission-factor-year/rollout.md.

Applies the reviewed classification to both datasets: `source` keeps only the provider name and `year` carries the reporting year, written out explicitly as an integer or as null. Seed `year` is required-but-nullable so omitting it fails validation — null has to be a deliberate statement that the factor is transversal, not the result of leaving a key out. The 16 duplicate unit representations are consolidated, keeping kg/ton. Both rows were the same IPCC factor written twice, and the methodology payload already expands a factor into every unit of its family, so the kg/kg reading is generated by conversion instead of stored. kg/ton keeps the larger numeric value (1460 rather than 1.46), which leaves the most significant digits inside Decimal(28, 10). The magnitude pair is derived from each rate unit here, the same way every production write path derives it, so a freshly seeded catalog and a migrated one end up with identical business keys — verified: the only difference between them is the one factor `base` has and `testing` does not. The row-count check now counts ACTIVE factors only. The migration retires duplicate representations by soft delete, so a migrated database keeps rows this seed data no longer describes and would otherwise fail the check while being perfectly correct.

`checkDuplicateEmissionFactor` now compares the full identity — required dimension values, year, source and the magnitude pair — matching null explicitly, because `year = null` is a transversal factor rather than a wildcard and has to collide with another transversal factor from the same provider. Normalization of non-required dimension slots applies to the comparison only, never to what is stored. A value a maintainer entered in an optional slot is real catalog data; it simply does not earn the factor a separate identity. That leaves the check marginally stricter than the index, which compares raw columns — the same asymmetry the catalog already had, and safe in that direction. `validateSourceConsistency` is removed outright rather than scoped by year. It forced every active factor in a subcategory to share one source, which made the accepted multi-provider case impossible; source is part of a factor's identity now, so DEFRA (2025) and IPCC (2025) are two legitimate alternatives. The magnitude pair is derived from the selected rate unit by a shared helper and re-derived on every update, not only when the rate unit changes — a stale pair would leave the row outside the identity the index is enforcing. Methodology duplication carries year and both magnitude IDs across, or the copy would collapse several vintages into one row.

…olumn Year is a nullable column of its own, where a blank cell is the explicit choice that declares a factor transversal. Persistent helper text asks for the provider name without the year — capture appends it, so "DEFRA 2025" would display as "DEFRA 2025 (2025)" — and a four-digit year in the name raises an advisory icon beside the field. It never blocks saving: a provider whose name really contains a number is legitimate, so this is data quality guidance, not a domain rule. Source locking is removed. It pinned every factor in a subcategory to the first one's source, which is exactly what made a second provider impossible to enter, and its tooltip claimed a subcategory-wide invariant that no longer exists. The duplicate-factor message now explains the real conflict — source, year, required dimensions and unit family, noting that kg/kg and kg/ton are one family — and the source-conflict message is gone with the validation behind it. The spreadsheet gets a separate year column, blank rather than "-" for a transversal factor: an empty year cell is how transversal reads, and it must not look like missing data.

Pinning a non-required slot to null asks for rows whose slot is also empty, which is not the same as ignoring it. An existing factor with a value parked in that slot was invisible to the check, and since the unique index compares the raw columns it did not catch the insert either — leaving two ACTIVE rows sharing one identity, two indistinguishable options in the capture selector, and a tie that stops the recommendation from firing. The slot is now left out of the query entirely, which is what the function's own doc comment already claimed: reject a little more than the constraint would, never less. The existing test covered the direction that happened to work — stored slot empty, candidate populated. Adds the one that did not.

The year was an unbounded integer, so a typo in the maintainer's numeric cell was stored as-is. It then became part of the factor's identity, sorted to the top of the capture selector, won the vintage ranking, and flagged every line using it as mismatched — with no way to notice from the UI. The bounds live in @repo/constants because the plausible range is a per-deployment judgement, and the upper one is fixed rather than derived from today so the same payload does not start being rejected overnight.

The four CATALOG_EMISSION_FACTOR_* codes had no entry, so every one of them reached the user as "Error al guardar la huella" — with nothing to indicate the line at fault or that the fix is re-picking a factor. Each message names what happened and what to do. The wording is deliberately the same shape across the four, because from the organization's side they are one situation: the factor on that line can no longer be applied.

The warning detected the year with a pattern that captures its surrounding context, then removed it with a plain substring replace. That only works when the year sits bare at the end: "IPCC (2019)" was turned into "IPCC ()", and "2020 EcoAct 2020" kept the very year the message asks the maintainer to remove. The removal now uses the pattern, keeps the character before the year so neighbours survive, repeats for every occurrence, and cleans up the emptied brackets and stranded separators. Suggesting an empty name still falls back to the original. The test covered only the one shape that worked by accident; it now covers the ones that did not.

EmissionFactorYearSchema is also the maintainer form's resolver, so its bound messages were rendering as English labels inside the Año cell of a Spanish-only UI. Adds a message to the type check as well: the cell now forwards an unparseable entry as NaN, and that has to read as a year problem rather than fall through to Zod's default.

Blur folded a NaN into null, and for the new Año column null is a real answer: it declares the factor transversal. A typo like 2o25 was therefore stored as a deliberate declaration, with nothing downstream able to flag it. The empty string stays the only input that means null; anything unparseable goes through as NaN so the resolver rejects it.

PAS 2050 is a real standard the maintainer already offers as a methodology regulation, and the heuristic told the user to move its 2050 into the Año column. 2050 passes the year bounds, so following that advice stored a factor that outranks every real vintage — worse than no warning at all. Names that match a known standard are now exempt, and extractYearFromSource is the single gate so every caller inherits it.

The duplicate lookup ignores a dimension slot the subcategory does not require, while the unique index compares the raw columns. Two ACTIVE rows differing only in such a slot are therefore legal, and the migration deliberately leaves them alone — but running the check on every PATCH found each row's sibling and made both permanently un-editable, even for a change to the value alone. The check now runs only when the update actually moves the factor to a different identity. If it does not, any conflict predates the request and this update cannot be what introduced it.

resolveTestRateUnitMagnitudes repeated resolveRateUnitMagnitudeFamily query for query, which is the drift that helper exists to prevent: a second copy would keep building factors in the old family the day the derivation changes, and the suite would quietly stop covering the constraint. Only the seeded-database hint stays test-specific.

The seed schema restated the year as a bare nullable integer, so the bounds the API enforces did not apply to the path that writes the whole catalog. A seed typo like 20255 sorts to the top of the vintage ranking and marks every line that uses it as mismatched, permanently. Shares EmissionFactorYearSchema instead of keeping a second definition.

The reviewed list was hardcoded twice, as the preflight allowlist and as three independent UPDATEs. Adding a source to the allowlist and forgetting its UPDATE would pass the preflight and leave the factor at year = NULL — "confirmed transversal", which this migration's own header says a missing suffix must never imply. Both steps now read a single mapping table, so extending it for another catalog is one row.

The bounds lived only in the API Zod schema, so the seeds — which write straight through Prisma — could store a year the HTTP layer would have rejected. A CHECK mirroring EMISSION_FACTOR_YEAR_MIN/MAX closes the last write path that could park a typo in the vintage ranking.

count(DISTINCT base_value) demanded exact numeric equality, so two representations that are equal in reality but rounded differently aborted the whole migration. measurement_units.json carries base factors that are not powers of ten — a day is 24 hours, a gallon 3.78541 litres — so the same factor stored as 1 kg/d and as 0.0416666667 kg/h looked like a methodology conflict on a catalog that is perfectly consistent. Only the power-of-ten pairs in the current seeds hid this in testing. The comparison is now relative, and the report names the actual range instead of a count.

The GROUP BY included the numerator and denominator magnitudes but the message did not, so two different families conflicting under the same subcategory, dimensions, year and source printed as two identical lines and left the operator with no way to tell which family to review.

…versal The per-process subcategories that main added (Ferroaleaciones, Cal, Aluminio) predate the reporting year, so their 38 rows arrived without one. All 38 carry source IPCC, which this change already reviewed and confirmed transversal, so the year is written out as an explicit null rather than left absent — omitting the key is a data error here, not a shorthand for transversal.

The per-process subcategories main added carry each factor twice, once as kg/ton and once as the equivalent kg/kg: 14 pairs in Ferroaleaciones, 3 in Cal, 2 in Aluminio. Under the new identity they are one factor expressed two ways, so the unique index keeps only one and the seed's own count check fails on the other 19. kg/ton survives, matching what Cinc, Cemento, Vidrio and Acero already do here and the migration's own rule that the representation with the larger denominator base factor wins — it leaves the most significant digits inside Decimal(28, 10). The values are exact equivalents (750 kg/ton is 0.75 kg/kg), so nothing is rounded away.

… table The check compared a global ACTIVE count against this dataset's row count, so two unrelated facts could break it on a correct database: a migrated database keeps the duplicate unit representations the migration soft-deleted, and a factor a maintainer created through the UI is ACTIVE and counted too. Filtering to ACTIVE only addressed the first. It now asserts that every row the dataset describes is present, keyed by the identity the unique index enforces, and names the missing ones when it fails.

Every other field on an added row defaults to something the schema refuses — empty ids, an empty source, a zero value — so the row cannot be saved until the maintainer answers. year was the exception: null is a positive claim that the factor applies to every reporting year, so the default published that claim on the maintainer's behalf, and it was indistinguishable from choosing it deliberately. A forgotten year also takes its provider's transversal slot and blocks a genuinely transversal factor later. The row now starts on NaN, which renders as an empty cell and fails the resolver. Typing a year answers it; clearing the cell maps the empty input to null, which is the transversal declaration made on purpose.

The backfill filled applied_factor_year while applied_factor_source kept the un-split legacy string, so every historical line carried its vintage twice in two columns that would then have to agree forever, and any reader rendering the snapshot as source plus year would print "DEFRA 2025 (2025)". The same reviewed map now splits the snapshot in the same statement, so it is the only place that needs to know the legacy format. A source the map does not cover is left byte for byte as captured: those are strings a user typed for a custom factor, not catalog provenance.

mrivas00 added priority: medium Medium priority type: feature new user-facing capability or enhancement area: web apps/web (React frontend) area: api apps/api (Fastify backend) area: database packages/database (Prisma schema / migrations) labels Sep 2, 2026

This was referenced Sep 2, 2026

docs(openspec): plan emission factors by reporting year #626

Closed

feat(carbonInventory)!: make the server authoritative over catalog factor snapshots #628

Closed

feat(carbonInventory): keep a line's stored factor when the request says unchanged #629

Closed

mrivas00 commented Sep 9, 2026

View reviewed changes

Comment thread apps/web/src/screens/Maintainer/screens/EmissionFactorsMaintainerScreen.tsx Outdated

mrivas00 commented Sep 9, 2026

View reviewed changes

Comment thread tools/seed/src/scripts/seedMethodologyData/seedEmissionFactors.ts Outdated

mrivas00 force-pushed the feat/mati/emission-factor-year-schema branch from 4670e67 to 2de0a90 Compare September 10, 2026 11:30

mrivas00 force-pushed the feat/mati/emission-factor-year-schema branch from 2de0a90 to fda7a94 Compare September 10, 2026 11:53

mrivas00 added 16 commits September 10, 2026 14:50

feat(types): add the reporting year to the emission factor contracts

44dbc63

test(api): cover emission factor vintages and uniqueness

0ff83f1

test(api): supply the derived magnitudes in the existing sync fixtures

bd32ab6

mrivas00 added 10 commits September 10, 2026 14:50

mrivas00 force-pushed the feat/mati/emission-factor-year-schema branch from c5e0dfc to a00092e Compare September 10, 2026 18:13

mrivas00 closed this Sep 15, 2026

Navigation Menu

Repository navigation

feat(emissionFactors)!: identify factors by reporting year and unit family - #627

feat(emissionFactors)!: identify factors by reporting year and unit family#627
mrivas00 wants to merge 26 commits into
docs/mati/emission-factor-year-planfrom
feat/mati/emission-factor-year-schema

mrivas00 commented Sep 2, 2026 •

edited

Loading

Uh oh!

mrivas00 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

mrivas00 commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. Unit-family magnitudes denormalized onto the factor, derived server-side

2. Partial unique index with NULLS NOT DISTINCT, excluding soft-deleted rows

3. validateSourceConsistency removed

4. Duplicate lookup omits optional dimension slots instead of matching them as null

5. Maintainer: year is its own field; the name-contains-a-year check is a warning, not a validation

6. Seed year is required-but-nullable

Scope note: consolidation now covers seven industrial subcategories

Uh oh!

mrivas00 left a comment

Choose a reason for hiding this comment

Grouped findings (outside this diff)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mrivas00 commented Sep 2, 2026 •

edited

Loading

2. Partial unique index with `NULLS NOT DISTINCT`, excluding soft-deleted rows

3. `validateSourceConsistency` removed