Repository navigation
v1.1: align with Data Package v2 - #35
Conversation
The Data Package standard gives a resource two distinct properties:
`$schema` is the profile that validates the resource descriptor, and
`schema` is the Table Schema describing the columns. v1.0 required the
GeoLocator DP table schema URL in `$schema` and never mentioned
`schema`, so a resource written exactly as the profile prescribes
declares no table schema at all and frictionless refuses to read it:
Resource "tags" must have a schema property.
A tabular resource now requires `name`, `path`, `type` and `schema`.
`schema` accepts either a versioned GeoLocator DP table schema URL or an
inline descriptor, so a package can stay self-contained. `$schema` stays
optional but is pinned to the v2 data resource profile when present.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
No version has shipped a `datapackage.json`, only the CSVs it indexes. Add one, plus the `measurements.csv` needed to reach the three resources the profile requires. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Only the GeoLocator DP half of the profile (`allOf[1]`) is applied, so the check stays offline; the other half is a `$ref` to datapackage.org. Also check that every file and table schema a resource references exists in the repository, which works before a version is tagged. Nothing validated a complete package until now, which is how the `$schema` and `schema` mix-up went unnoticed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Collects the GitHub releases from v0.1 to v1.0 in one place, and documents v1.1. URLs are normalised from the former Rafnuss/ org to GeoPressure/. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dc8f219 to
4554a0b
Compare
|
@peterdesmet, Could I ask you opinion on the required property on I'm also considering to set |
The Table Schema specification is normative: a descriptor "MAY contain a property `fieldsMatch` that MUST be a string". The array form came from the published `tableschema.json` profile, which types `fieldsMatch` as an array because of a typo (`item` instead of `items`, so the enum never applies). That is frictionlessdata/datapackage#965, acknowledged upstream and fixed in Data Package v2.1. Until v2.1 ships there is no spelling that satisfies both the prose and the profile, so follow the prose. Nothing in the toolchain objects: frictionless-py accepts either, frictionless-r does not read `fieldsMatch` at all (frictionlessdata/frictionless-r#216), and GeoLocatoR reads it through `as.character(...)[1]`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Data Package v2 changelog on `primaryKey`: "should now always be an array of strings, not a string", and the same for `foreignKeys`. The published profile still allows the v1 string form through a `oneOf`, so this was valid but drifting. `tags.tag_id` also dropped `constraints.unique`, which duplicated the primary key. GeoLocatoR is unaffected: `validate_gldp_coherence()` already unlists the field lists. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both spellings the previous two commits corrected are accepted by the published `tableschema.json` profile, so `Schema.validate_descriptor()` does not catch a regression. Check them here instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The resource schema was a `oneOf` over three branches (fixed table, `params`,
custom). A resource that fails one property fails all three branches, and the
validator can only report that the whole descriptor "is not valid under any of
the given schemas", dumping the descriptor without naming the problem.
Guard each branch with `if` on `name` instead. The accepted set of descriptors
is unchanged -- checked against 17 cases, valid and invalid -- but errors now
point at the property:
before: {'$schema': '...', 'name': 'tags', 'path': 'tags.csv', 'format': ...}
is not valid under any of the given schemas
after: resources/0: 'type' is a required property
`name` stays inside each branch as well as in its guard, so it is still
documented and the website template can keep building the resource table from
the branch properties.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every other schema in `_data/` is a symlink to the one at the repository root, so the website renders the same file the standard ships. This one was replaced by a real copy in 0732e4d, a bulk URL rewrite. The copy happened to stay in step until now, but it had already stopped being the same file: the `fieldsMatch` and key changes on this branch only landed at the root, so the measurements page would have rendered the old spelling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`observations.datetime` had `skos:closeMatch` set to " http://vocab.nerc...", which the website rendered as a broken link. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A table page rendered only the fields. `primaryKey`, `foreignKeys`, `missingValues` and `fieldsMatch` were declared in the schema and shown nowhere, so the field descriptions had taken to restating the foreign keys in prose -- which can drift from the keys themselves. They are now listed above the fields table, with foreign keys linking to the target table's page. Two changes in the fields table itself: - `required` is dropped from the constraint list. It was rendered on all 74 fields, as `required: true` or `required: false`, next to a field name whose `*` already says the same thing. - The five SKOS relations are rendered from one list rather than five near-identical blocks, and a term links by its identifier (`dwc:decimalLatitude`, `MVB000074`) instead of by its full URL, which was long enough to crowd out the definition above it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
— A minimal json example of a package with a resource and schema might help 😊 |
`observations.sex` gave `female` as its example where the field takes the EURING codes `U`, `M` and `F`, and `pressurepaths.type` gave `3` where the field takes `tag`, `most_likely`, `simulation` or `geopressureviz`. Both were surfaced by checking examples against the enumerated values. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Data Package v2 adds `categories`, which gives each allowed value a label. Until now the labels lived in the field `description` as a Markdown list, and for the two EURING-coded fields that list was incomplete: `age_class` documented 7 of its 18 codes and trailed off with "- ...", `catching_method` 3 of 20. A single-character EURING code means nothing without its label, so most of these values were undocumented. The missing labels come from the EURING Exchange Code 2020 the fields already cite (catching method p.12, sex p.13, age p.14, condition p.23). That document spells out age codes 0-B and then says only "C, D, E, F, G, H onwards et seq.", so those six are derived from the stated rule -- odd positions are an exact year, even positions "after" that year. The derivation agrees with the BTO's own table, which continues the same scheme in numerals: C..H are BTO 12..17, "hatched five or more years ago" through "definitely hatched seven years ago". `measurements.sensor` also carries its vocabulary term per category, lifted out of the description, where each of the 13 terms had been written as a raw <span> of inline HTML around a full URL. The `enum` constraint stays alongside, which the standard allows as long as it is a subset. It is kept because nothing enforces `categories` yet: frictionless-py 5.19 reads the property but accepts a value outside it without an error, so dropping `enum` would quietly drop the validation. `categoriesOrdered` is deliberately not set. `age_class` looks ordinal but is not: `0` is "age unknown" and `2` is "full-grown, year of hatching unknown". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`enum` and `categories` are declared together and can drift apart. Check they list the same values in the same order, that no value repeats, and that a field's `example` is one of them -- which is what caught the two examples fixed in 09e623a. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The enum constraint rendered as a bare comma-separated list, so `catching_method` was twenty single letters with no indication of what any of them meant. Categories are listed with their labels instead, and the enum is skipped where it would only repeat them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Implementations read `$schema` to decide which version of the standard a
resource follows, so leaving it optional meant a package could declare v2 while
every resource in it read as v1:
package -> version(): >=2.0
its only resource -> version(): 1.0
frictionless branches on exactly that: `upgrade_resource()` returns early
unless the resource reports 1.0, so a v1.1 resource without `$schema` is
treated as a v1 descriptor to be rewritten.
Raised by @peterdesmet in #35.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Thanks @peterdesmet, this is helpful.
2./3. Ok, make sense. Implemented in a87f327
Example — example/datapackage.json in this PR: {
"$schema": "https://raw.githubusercontent.com/GeoPressure/GeoLocator-DP/v1.1/geolocator-dp-profile.json",
"resources": [
{
"$schema": "https://datapackage.org/profiles/2.0/dataresource.json",
"name": "tags",
"path": "tags.csv",
"type": "table",
"format": "csv",
"mediatype": "text/csv",
"encoding": "utf-8",
"schema": "https://raw.githubusercontent.com/GeoPressure/GeoLocator-DP/v1.1/tags-table-schema.json"
}
]
} |
Add a .ruby-version file set to ruby-3.3.4 and update README to note the GitHub Pages Jekyll environment runs Ruby 3.3.4. Instructs users (chruby/other managers) to confirm ruby --version before installing dependencies.
|
Your
The table schemas (e.g
The fieldMatch you currently have is:
So all tables can MAY fewer fields than in the schema, except for pressurepaths, which may have fewer fields + extra undefined fields. For all of the above
|
`tags` declared `fieldsMatch: superset`, which lets a table leave columns out but not add any. Users carrying extra Movebank attributes on a tag had nowhere to put them, so it becomes `partial`, as `pressurepaths` already was. `fieldsMatch` is a set rule over all the fields and says nothing about which ones are required, so neither mode expresses what these tables actually want: the required columns present, the optional ones free to be absent. That gap is frictionlessdata/datapackage#1126. Presence of the required columns is checked by GeoLocatoR instead, so widening `tags` does not give up the guarantee that `tag_id` is there. The table pages said which columns were allowed but never which had to be present, leaving the `*` beside a required field unexplained. They now say both, and the `partial` wording mentions that columns may also be left out. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The sentence explaining what implementations do with the property was written for a87f327 but did not survive the commit, leaving the description stating the requirement without its reason. The changelog entry refers to it, and GeoLocatoR's bundled copy already carried it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every released section carries the date and a link to its release; v1.1 was the only one without. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
v1.0 puts the GeoLocator DP table schema URL in
resource.$schema, but in the Data Package standard$schemais the profile of the descriptor and the Table Schema belongs inschema. A resource written exactly as v1.0 prescribes cannot be read:Real GeoLocatoR output failed the profile too — 8/8 resources — because it writes an inline
schemaand no$schema. The two errors cancelled out and nothing validated a complete package, so neither was noticed.This brings v1.1 into line with Data Package v2 and adds an example package plus a CI check, so a mistake of this shape cannot go unnoticed again.
Resources
A tabular resource now requires
$schema,name,path,typeandschema:{ "$schema": "https://datapackage.org/profiles/2.0/dataresource.json", "name": "tags", "path": "tags.csv", "type": "table", "schema": "https://raw.githubusercontent.com/GeoPressure/GeoLocator-DP/v1.1/tags-table-schema.json" }schemaalso accepts an inline descriptor, so packages stay self-contained.type: "table"replaces the v1profile: "tabular-data-resource". Package-level$schemais unchanged.$schemais required, not optional. Without itfrictionless::version()reads a resource as Data Package v1 even inside a package declaring v2 — the package reports>=2.0while every resource in it reports1.0— andupgrade_resource()branches on exactly that value. GeoLocatoR sets the property itself rather than waiting for frictionless 2.0 to reach CRAN.Table schemas
fieldsMatchis a string, not an array. The array form came from an error in the published Table Schema profile (frictionlessdata/datapackage#965), fixed in v2.1.tagsmoves tofieldsMatch: partial, so a package may carry extra attributes on a tag. The other six tables keepsuperset.primaryKeyandforeignKeysfields are arrays of strings, the v2 spelling.categorieson every categorical field, giving each allowed value a label. The EURINGage_classandcatching_methodcodes were previously only partly documented in prose;measurements.sensorcarries its vocabulary term per category.enumis kept alongside, as the standard allows, because no implementation enforcescategoriesyet.uniqueconstraint ontags.tag_id, which duplicated the primary key; fixed theobservations.sexandpressurepaths.typeexamples, which were not valid values of their own fields.One thing worth recording:
fieldsMatchsays nothing about which columns are required, so on its own every mode here lets a table omit a column carrying arequiredconstraint — including a primary key. The standard cannot yet express "the required columns must be present, the optional ones may be absent" (frictionlessdata/datapackage#1126), soGeoLocatoR::validate_gldp()checks that separately.Profile
Resource schemas are selected with
if/thenon the resourcenamerather thanoneOf. The accepted descriptors are unchanged, but a validation error now names the property at fault instead of reporting that the whole resource matched none of the branches.Website and CI
primaryKey,foreignKeys,missingValuesandfieldsMatch, listscategorieswith their labels, links SKOS terms by identifier, and says which columns may be left out and which must be present — the*marking a required column was previously unexplained.observations.datetimeand restored the_datasymlink for the measurements schema, so the site renders the schema the standard ships.example/datapackage.jsonwithexample/measurements.csv, andtests/validate_profile_and_schemas.py, which validates the example package against the profile along with the spelling offieldsMatchand of the keys. It runs on every PR.Compatibility
Breaking for archived packages, which carry
profileand notype— hence v1.1. GeoLocatoR upgrades them on read.Resources still carry
profile: "tabular-data-resource"alongside$schemaandtype. It is the v1 spelling those two replace, but frictionless 1.3.0, the current CRAN release, refuses to read a resource without it. It can be dropped once frictionless 2.0 ships.Tracked separately: #36 (
lat/lonvslatitude/longitude) anduniqueKeysadoption, which needs thejcolumn to stop being null outside simulations.🤖 Generated with Claude Code