From 21c9919eda4e22ab06cfc45d71d3be4d6c6c94ba Mon Sep 17 00:00:00 2001 From: timlichtenberg Date: Fri, 11 Sep 2026 21:19:38 +0200 Subject: [PATCH 1/2] Document the data-governance model Add an Explanations page that states the policy the manifest-and-registry mechanism enforces: what counts as a data change, who may make one, and how it is reviewed and tested. A dataset is pinned by two committed files, a fetch reads the committed registry rather than the live Zenodo record, and a data change takes the same pull-request path as a code change. The page states the testing asymmetry plainly: the scheduled drift check covers only the shared manifest, and a model's own datasets get the pull-request review but no scheduled drift check in fwl-io. It links to the design and manifests pages for the mechanism and adds one nav entry under Explanations. --- docs/Explanations/governance.md | 27 +++++++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 28 insertions(+) create mode 100644 docs/Explanations/governance.md diff --git a/docs/Explanations/governance.md b/docs/Explanations/governance.md new file mode 100644 index 0000000..70f7855 --- /dev/null +++ b/docs/Explanations/governance.md @@ -0,0 +1,27 @@ +# Data governance + +This page states who may change the data the ecosystem depends on, and how such a change is reviewed and tested. The mechanism behind it is described in [Design decisions](design.md) and [Manifests and registries](manifests.md); this page states the policy that the mechanism enforces. + +## What a data change is + +A dataset is pinned by two committed files: the Zenodo version DOI in a manifest, and the registry file beside it that lists the dataset's files and their checksums. A data change is a change to those. Nothing else alters the data a model receives, because a fetch reads the committed registry, not the live Zenodo record. + +fwl-io provides the manifest for shared datasets. A model's own datasets are declared in that model's manifest, installed with the model through the `fwl_io.manifests` entry point. Ownership follows the manifest: a shared dataset is changed in fwl-io, a model dataset in the model that owns it. + +## Who may change data, and how + +A data change is a code change and takes the same path. There is no separate data-admin role: anyone who can open a pull request on the owning repository can propose one, and it is reviewed and merged under that repository's normal permissions. + +To change a dataset: + +1. Edit the `zenodo` version DOI in the owning manifest. +2. Run `fwl-io sync ` to regenerate the committed registry from the Zenodo record. +3. Commit the manifest and the registry together, and open a pull request. + +The registry diff shows the new file list and checksums, so a reviewer sees exactly which files and hashes change. `fwl-io sync` rejects a concept DOI, so a manifest can pin only a fixed version, and the data cannot change under pinned code without a visible manifest edit. + +## How data is tested + +The committed registry is the contract a fetch trusts, so the test is whether that contract still matches its source. A scheduled workflow runs the slow test tier once a week. It fetches the live Zenodo registry of every dataset in the shared manifest and compares it to the committed registry, and it fails if the two have drifted. A change made to a Zenodo record outside a reviewed `fwl-io sync` is therefore caught by the scheduled run, not by a user's failing fetch. + +The scheduled drift check in fwl-io covers only the shared manifest. It runs no equivalent check against a model manifest, so drift protection for a model's own datasets is the responsibility of that model's repository, where the same pull-request review already applies to every data change. diff --git a/mkdocs.yml b/mkdocs.yml index 7d5c850..cafafe8 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -20,6 +20,7 @@ nav: - Explanations: - Design decisions: Explanations/design.md - Manifests and registries: Explanations/manifests.md + - Data governance: Explanations/governance.md - Reference: - CLI reference: Reference/cli.md From 74a46bba33eada3151589e77e8005292e9ef6612 Mon Sep 17 00:00:00 2001 From: timlichtenberg Date: Tue, 15 Sep 2026 12:28:19 +0200 Subject: [PATCH 2/2] Rework governance.md's Dataverse mirror and archive-checksum notes Move the archive-dataset checksum caveat from the definitional section to How data is tested, since it describes a verification gap rather than what counts as a data change, and note that the scheduled drift check does not cover on-disk corruption either. Move the Dataverse mirror paragraph to the end of the pull-request procedure instead of the middle of it, so the numbered steps read as a single continuous path. Describe the mirror's actual access control: the DATAVERSE_TOKEN is a protected secret bound to a GitHub deployment environment and never distributed outside CI, and the workflow that publishes it runs on manual dispatch rather than pull-request review. Fix the ambiguous pronoun in the opening paragraph. --- docs/Explanations/governance.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/Explanations/governance.md b/docs/Explanations/governance.md index 70f7855..c781170 100644 --- a/docs/Explanations/governance.md +++ b/docs/Explanations/governance.md @@ -1,10 +1,10 @@ # Data governance -This page states who may change the data the ecosystem depends on, and how such a change is reviewed and tested. The mechanism behind it is described in [Design decisions](design.md) and [Manifests and registries](manifests.md); this page states the policy that the mechanism enforces. +This page states who may change the data the ecosystem depends on, and how such a change is reviewed and tested. The mechanism behind this policy is described in [Design decisions](design.md) and [Manifests and registries](manifests.md); this page states the policy that the mechanism enforces. ## What a data change is -A dataset is pinned by two committed files: the Zenodo version DOI in a manifest, and the registry file beside it that lists the dataset's files and their checksums. A data change is a change to those. Nothing else alters the data a model receives, because a fetch reads the committed registry, not the live Zenodo record. +A dataset is pinned by two committed files: the Zenodo version DOI in a manifest, and the registry file beside it that lists the dataset's files and their checksums. A data change is a change to those: nothing else changes which dataset version is pinned, because a fetch reads the committed registry, not the live Zenodo record. fwl-io provides the manifest for shared datasets. A model's own datasets are declared in that model's manifest, installed with the model through the `fwl_io.manifests` entry point. Ownership follows the manifest: a shared dataset is changed in fwl-io, a model dataset in the model that owns it. @@ -20,8 +20,12 @@ To change a dataset: The registry diff shows the new file list and checksums, so a reviewer sees exactly which files and hashes change. `fwl-io sync` rejects a concept DOI, so a manifest can pin only a fixed version, and the data cannot change under pinned code without a visible manifest edit. +The Dataverse mirror is a separate access surface. Publishing or updating the mirrored copy requires the `DATAVERSE_TOKEN`, a protected secret bound to a `dataverse` GitHub deployment environment and never distributed outside CI; the workflow that runs it is triggered manually, not through pull-request review. It hosts a copy of already-pinned data and does not change which version a fetch resolves, so it is not part of the data-change path above. + ## How data is tested The committed registry is the contract a fetch trusts, so the test is whether that contract still matches its source. A scheduled workflow runs the slow test tier once a week. It fetches the live Zenodo registry of every dataset in the shared manifest and compares it to the committed registry, and it fails if the two have drifted. A change made to a Zenodo record outside a reviewed `fwl-io sync` is therefore caught by the scheduled run, not by a user's failing fetch. +For an archive dataset, the registry records member names rather than a per-file digest, so a file that is corrupted or truncated on disk after its first fetch is not re-verified by a later fetch that finds the extracted tree already in place. That check does not cover this case either, since it compares registry contents to the Zenodo record, not files already on disk. + The scheduled drift check in fwl-io covers only the shared manifest. It runs no equivalent check against a model manifest, so drift protection for a model's own datasets is the responsibility of that model's repository, where the same pull-request review already applies to every data change.