Skip to content

v1.1: align with Data Package v2 - #35

Merged
Rafnuss merged 25 commits into
mainfrom
v1.1-resource-schema
Sep 18, 2026
Merged

Rafnuss merged 25 commits into
mainfrom
v1.1-resource-schema

Conversation

@Rafnuss

@Rafnuss Rafnuss commented Sep 17, 2026 •

Copy link
Copy Markdown
Collaborator

v1.0 puts the GeoLocator DP table schema URL in resource.$schema, but in the Data Package standard $schema is the profile of the descriptor and the Table Schema belongs in schema. A resource written exactly as v1.0 prescribes cannot be read:

Resource "tags" must have a schema property.

Real GeoLocatoR output failed the profile too — 8/8 resources — because it writes an inline schema and no $schema. The two errors cancelled out and nothing validated a complete package, so neither was noticed.

This brings v1.1 into line with Data Package v2 and adds an example package plus a CI check, so a mistake of this shape cannot go unnoticed again.

Resources

A tabular resource now requires $schema, name, path, type and schema:

{
  "$schema": "https://datapackage.org/profiles/2.0/dataresource.json",
  "name": "tags",
  "path": "tags.csv",
  "type": "table",
  "schema": "https://raw.githubusercontent.com/GeoPressure/GeoLocator-DP/v1.1/tags-table-schema.json"
}

schema also accepts an inline descriptor, so packages stay self-contained. type: "table" replaces the v1 profile: "tabular-data-resource". Package-level $schema is unchanged.

$schema is required, not optional. Without it frictionless::version() reads a resource as Data Package v1 even inside a package declaring v2 — the package reports >=2.0 while every resource in it reports 1.0 — and upgrade_resource() branches on exactly that value. GeoLocatoR sets the property itself rather than waiting for frictionless 2.0 to reach CRAN.

Table schemas

  • fieldsMatch is a string, not an array. The array form came from an error in the published Table Schema profile (frictionlessdata/datapackage#965), fixed in v2.1.
  • tags moves to fieldsMatch: partial, so a package may carry extra attributes on a tag. The other six tables keep superset.
  • primaryKey and foreignKeys fields are arrays of strings, the v2 spelling.
  • categories on every categorical field, giving each allowed value a label. The EURING age_class and catching_method codes were previously only partly documented in prose; measurements.sensor carries its vocabulary term per category. enum is kept alongside, as the standard allows, because no implementation enforces categories yet.
  • Dropped the unique constraint on tags.tag_id, which duplicated the primary key; fixed the observations.sex and pressurepaths.type examples, which were not valid values of their own fields.

One thing worth recording: fieldsMatch says nothing about which columns are required, so on its own every mode here lets a table omit a column carrying a required constraint — including a primary key. The standard cannot yet express "the required columns must be present, the optional ones may be absent" (frictionlessdata/datapackage#1126), so GeoLocatoR::validate_gldp() checks that separately.

Profile

Resource schemas are selected with if/then on the resource name rather than oneOf. The accepted descriptors are unchanged, but a validation error now names the property at fault instead of reporting that the whole resource matched none of the branches.

Website and CI

  • Each table page shows primaryKey, foreignKeys, missingValues and fieldsMatch, lists categories with their labels, links SKOS terms by identifier, and says which columns may be left out and which must be present — the * marking a required column was previously unexplained.
  • Fixed a broken SKOS link on observations.datetime and restored the _data symlink for the measurements schema, so the site renders the schema the standard ships.
  • Added example/datapackage.json with example/measurements.csv, and tests/validate_profile_and_schemas.py, which validates the example package against the profile along with the spelling of fieldsMatch and of the keys. It runs on every PR.
  • Added the changelog page.

Compatibility

Breaking for archived packages, which carry profile and no type — hence v1.1. GeoLocatoR upgrades them on read.

Resources still carry profile: "tabular-data-resource" alongside $schema and type. It is the v1 spelling those two replace, but frictionless 1.3.0, the current CRAN release, refuses to read a resource without it. It can be dropped once frictionless 2.0 ships.

Tracked separately: #36 (lat/lon vs latitude/longitude) and uniqueKeys adoption, which needs the j column to stop being null outside simulations.

🤖 Generated with Claude Code

Rafnuss and others added 4 commits September 17, 2026 20:26
The Data Package standard gives a resource two distinct properties:
`$schema` is the profile that validates the resource descriptor, and
`schema` is the Table Schema describing the columns. v1.0 required the
GeoLocator DP table schema URL in `$schema` and never mentioned
`schema`, so a resource written exactly as the profile prescribes
declares no table schema at all and frictionless refuses to read it:

    Resource "tags" must have a schema property.

A tabular resource now requires `name`, `path`, `type` and `schema`.
`schema` accepts either a versioned GeoLocator DP table schema URL or an
inline descriptor, so a package can stay self-contained. `$schema` stays
optional but is pinned to the v2 data resource profile when present.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
No version has shipped a `datapackage.json`, only the CSVs it indexes.
Add one, plus the `measurements.csv` needed to reach the three resources
the profile requires.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Only the GeoLocator DP half of the profile (`allOf[1]`) is applied, so
the check stays offline; the other half is a `$ref` to datapackage.org.
Also check that every file and table schema a resource references exists
in the repository, which works before a version is tagged.

Nothing validated a complete package until now, which is how the
`$schema` and `schema` mix-up went unnoticed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Collects the GitHub releases from v0.1 to v1.0 in one place, and
documents v1.1. URLs are normalised from the former Rafnuss/ org to
GeoPressure/.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Rafnuss
Rafnuss force-pushed the v1.1-resource-schema branch from dc8f219 to 4554a0b Compare September 17, 2026 18:26
@Rafnuss

Rafnuss commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator Author

@peterdesmet, Could I ask you opinion on the required property on $schema? I'm not really seeing the point of forcing it as I can control it in creation, but I'm not sure I have a wide enough understanding on the issue.

I'm also considering to set tags resources as fieldMatch: partial to be more flexible to possible specific user case. Curious to hear if you have any opnion on this too?

@Rafnuss
Rafnuss marked this pull request as ready for review September 17, 2026 18:32
Rafnuss and others added 10 commits September 17, 2026 21:25
The Table Schema specification is normative: a descriptor "MAY contain a
property `fieldsMatch` that MUST be a string". The array form came from the
published `tableschema.json` profile, which types `fieldsMatch` as an array
because of a typo (`item` instead of `items`, so the enum never applies).

That is frictionlessdata/datapackage#965, acknowledged upstream and fixed in
Data Package v2.1.

Until v2.1 ships there is no spelling that satisfies both the prose and the
profile, so follow the prose. Nothing in the toolchain objects: frictionless-py
accepts either, frictionless-r does not read `fieldsMatch` at all
(frictionlessdata/frictionless-r#216), and GeoLocatoR reads it through
`as.character(...)[1]`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Data Package v2 changelog on `primaryKey`: "should now always be an array
of strings, not a string", and the same for `foreignKeys`. The published
profile still allows the v1 string form through a `oneOf`, so this was valid
but drifting.

`tags.tag_id` also dropped `constraints.unique`, which duplicated the primary
key.

GeoLocatoR is unaffected: `validate_gldp_coherence()` already unlists the
field lists.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both spellings the previous two commits corrected are accepted by the
published `tableschema.json` profile, so `Schema.validate_descriptor()` does
not catch a regression. Check them here instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The resource schema was a `oneOf` over three branches (fixed table, `params`,
custom). A resource that fails one property fails all three branches, and the
validator can only report that the whole descriptor "is not valid under any of
the given schemas", dumping the descriptor without naming the problem.

Guard each branch with `if` on `name` instead. The accepted set of descriptors
is unchanged -- checked against 17 cases, valid and invalid -- but errors now
point at the property:

  before: {'$schema': '...', 'name': 'tags', 'path': 'tags.csv', 'format': ...}
          is not valid under any of the given schemas
  after:  resources/0: 'type' is a required property

`name` stays inside each branch as well as in its guard, so it is still
documented and the website template can keep building the resource table from
the branch properties.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every other schema in `_data/` is a symlink to the one at the repository root,
so the website renders the same file the standard ships. This one was replaced
by a real copy in 0732e4d, a bulk URL rewrite.

The copy happened to stay in step until now, but it had already stopped being
the same file: the `fieldsMatch` and key changes on this branch only landed at
the root, so the measurements page would have rendered the old spelling.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`observations.datetime` had `skos:closeMatch` set to " http://vocab.nerc...",
which the website rendered as a broken link.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A table page rendered only the fields. `primaryKey`, `foreignKeys`,
`missingValues` and `fieldsMatch` were declared in the schema and shown
nowhere, so the field descriptions had taken to restating the foreign keys in
prose -- which can drift from the keys themselves.

They are now listed above the fields table, with foreign keys linking to the
target table's page.

Two changes in the fields table itself:

- `required` is dropped from the constraint list. It was rendered on all 74
  fields, as `required: true` or `required: false`, next to a field name whose
  `*` already says the same thing.
- The five SKOS relations are rendered from one list rather than five
  near-identical blocks, and a term links by its identifier
  (`dwc:decimalLatitude`, `MVB000074`) instead of by its full URL, which was
  long enough to crowd out the definition above it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@peterdesmet

Copy link
Copy Markdown
Contributor
  1. Question about the issue title. Did geolocatoR 1.0 put the table schema in $schema? Cause that is wrong, table schema should be included or referenced in schema.

I'm not really seeing the point of forcing it as I can control it in creation

  1. What do you mean by forcing it? When reading? Note that frictionless will use that property to assess the version of the Data Package standard used.

  2. If you’re using Data Package v2 properties like fieldMatch, then you should set $schema to the v2 value. Definitely for schema, and also for the resource.

I'm also considering to set tags resources

  1. What do you mean by tags?

—

A minimal json example of a package with a resource and schema might help 😊

Rafnuss and others added 5 commits September 17, 2026 22:03
`observations.sex` gave `female` as its example where the field takes the
EURING codes `U`, `M` and `F`, and `pressurepaths.type` gave `3` where the
field takes `tag`, `most_likely`, `simulation` or `geopressureviz`.

Both were surfaced by checking examples against the enumerated values.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Data Package v2 adds `categories`, which gives each allowed value a label.
Until now the labels lived in the field `description` as a Markdown list, and
for the two EURING-coded fields that list was incomplete: `age_class`
documented 7 of its 18 codes and trailed off with "- ...", `catching_method` 3
of 20. A single-character EURING code means nothing without its label, so most
of these values were undocumented.

The missing labels come from the EURING Exchange Code 2020 the fields already
cite (catching method p.12, sex p.13, age p.14, condition p.23). That document
spells out age codes 0-B and then says only "C, D, E, F, G, H onwards et seq.",
so those six are derived from the stated rule -- odd positions are an exact
year, even positions "after" that year. The derivation agrees with the BTO's
own table, which continues the same scheme in numerals: C..H are BTO 12..17,
"hatched five or more years ago" through "definitely hatched seven years ago".

`measurements.sensor` also carries its vocabulary term per category, lifted out
of the description, where each of the 13 terms had been written as a raw
<span> of inline HTML around a full URL.

The `enum` constraint stays alongside, which the standard allows as long as it
is a subset. It is kept because nothing enforces `categories` yet:
frictionless-py 5.19 reads the property but accepts a value outside it without
an error, so dropping `enum` would quietly drop the validation.

`categoriesOrdered` is deliberately not set. `age_class` looks ordinal but is
not: `0` is "age unknown" and `2` is "full-grown, year of hatching unknown".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`enum` and `categories` are declared together and can drift apart. Check they
list the same values in the same order, that no value repeats, and that a
field's `example` is one of them -- which is what caught the two examples
fixed in 09e623a.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The enum constraint rendered as a bare comma-separated list, so
`catching_method` was twenty single letters with no indication of what any of
them meant. Categories are listed with their labels instead, and the enum is
skipped where it would only repeat them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Rafnuss Rafnuss changed the title v1.1: move table schemas from resource.$schema to resource.schema v1.1: align with Data Package v2 Sep 17, 2026
Rafnuss and others added 2 commits September 17, 2026 22:36
Implementations read `$schema` to decide which version of the standard a
resource follows, so leaving it optional meant a package could declare v2 while
every resource in it read as v1:

    package           -> version(): >=2.0
    its only resource -> version(): 1.0

frictionless branches on exactly that: `upgrade_resource()` returns early
unless the resource reports 1.0, so a v1.1 resource without `$schema` is
treated as a v1 descriptor to be rewritten.

Raised by @peterdesmet in #35.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Rafnuss

Rafnuss commented Sep 17, 2026 •

Copy link
Copy Markdown
Collaborator Author

Thanks @peterdesmet, this is helpful.

  1. Yes, it was wrong: v1.0 had required: ["name", "path", "$schema"] with $schema pinned to a *-table-schema.json URL, and no schema property at all. That's what this PR fixes 1062e75.

2./3. Ok, make sense. Implemented in a87f327

  1. tags is one of our resources (tags.csv). Some users want to carry extra Movebank attributes on it, which superset forbids.

Example — example/datapackage.json in this PR:

{
  "$schema": "https://raw.githubusercontent.com/GeoPressure/GeoLocator-DP/v1.1/geolocator-dp-profile.json",
  "resources": [
    {
      "$schema": "https://datapackage.org/profiles/2.0/dataresource.json",
      "name": "tags",
      "path": "tags.csv",
      "type": "table",
      "format": "csv",
      "mediatype": "text/csv",
      "encoding": "utf-8",
      "schema": "https://raw.githubusercontent.com/GeoPressure/GeoLocator-DP/v1.1/tags-table-schema.json"
    }
  ]
}

Add a .ruby-version file set to ruby-3.3.4 and update README to note the GitHub Pages Jekyll environment runs Ruby 3.3.4. Instructs users (chruby/other managers) to confirm ruby --version before installing dependencies.
@peterdesmet

Copy link
Copy Markdown
Contributor

Your geolocator-dp-profile.json looks good:

  • Extends "https://datapackage.org/profiles/2.0/datapackage.json", so the package is defined as v2
  • Requires package.$schema with a URL referring to your geolocator-dp-profile. This is self-fulfilling: if you refer to it, it will be there.
  • Requires resource.$schema with the value "https://datapackage.org/profiles/2.0/dataresource.json", so resources are defined as v2
  • Requires resource.schema referring to a table-schema for that resource
  • Requires resource.type with the value "table". This is good practice (documents that it is tabular), but not absolutely needed. Good to have it.

The table schemas (e.g tags-table-schema.json) look good too:

  • Have schema.$schema with the value https://datapackage.org/profiles/2.0/tableschema.json, so resources are defined as v2
  • Have fieldMatch which is now a string rather than an array.

The fieldMatch you currently have is:

  • edges: superset
  • measurements: superset
  • observations: superset
  • paths: superset
  • pressurepaths: partial
  • staps: superset
  • tags: superset
  • twilights: superset

So all tables can MAY fewer fields than in the schema, except for pressurepaths, which may have fewer fields + extra undefined fields.


For all of the above version() in {frictionless} should recognize the version as:

  • package: >=2.0 (notices a custom $schema, doesn't dive deeper, but knows it's at least 2.0)
  • resource: 2.0
  • schema: 2.0

read_resource() in {frictionless} will throw an error, because we haven't implemented fieldsMatch yet (frictionlessdata/frictionless-r#216). But I think you catch that with your R package.

Rafnuss and others added 3 commits September 18, 2026 10:00
`tags` declared `fieldsMatch: superset`, which lets a table leave columns
out but not add any. Users carrying extra Movebank attributes on a tag had
nowhere to put them, so it becomes `partial`, as `pressurepaths` already
was.

`fieldsMatch` is a set rule over all the fields and says nothing about which
ones are required, so neither mode expresses what these tables actually
want: the required columns present, the optional ones free to be absent.
That gap is frictionlessdata/datapackage#1126. Presence of the required
columns is checked by GeoLocatoR instead, so widening `tags` does not give
up the guarantee that `tag_id` is there.

The table pages said which columns were allowed but never which had to be
present, leaving the `*` beside a required field unexplained. They now say
both, and the `partial` wording mentions that columns may also be left out.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The sentence explaining what implementations do with the property was
written for a87f327 but did not survive the commit, leaving the description
stating the requirement without its reason. The changelog entry refers to
it, and GeoLocatoR's bundled copy already carried it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every released section carries the date and a link to its release; v1.1 was
the only one without.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Rafnuss
Rafnuss merged commit 6885bba into main Sep 18, 2026
1 check passed
@Rafnuss
Rafnuss deleted the v1.1-resource-schema branch October 16, 2026 06:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants