From 82b13020c0d21980c3aac9d923db81a1f7b4f7a0 Mon Sep 17 00:00:00 2001 From: Andrey Kislyuk Date: Thu, 11 Jul 2019 14:50:45 -0700 Subject: [PATCH 1/9] WIP --- rfcs/text/0000-bundle-types.md | 90 ++++++++++++++++++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100644 rfcs/text/0000-bundle-types.md diff --git a/rfcs/text/0000-bundle-types.md b/rfcs/text/0000-bundle-types.md new file mode 100644 index 00000000..a9c80150 --- /dev/null +++ b/rfcs/text/0000-bundle-types.md @@ -0,0 +1,90 @@ +### DCP PR: + +***Leave this blank until the RFC is approved** then the **Author(s)** must create a link between the assigned RFC number and this pull request in the format:* + +`[dcp-community/rfc#](https://github.com/HumanCellAtlas/dcp-community/pull/)` + +# RFC: Bundle Types + +## Summary + +Formalizes data bundle types, definitions, target users/consumers, and specific use cases to improve clarity and +confidence among DCP developers when working with the DCP data model. + +## Author(s) + +* [Mallory Freeberg](mailto:mfreeberg@ebi.ac.uk) +* [Brian Hannafious](mailto:bhannafi@ucsc.edu) +* [Hannes Schmidt](mailto:hannes@ucsc.edu) +* [Andrey Kislyuk](mailto:akislyuk@chanzuckerberg.com) + +## Shepherd +***Leave this blank.** This role is assigned by DCP PM to guide the **Author(s)** through the RFC process.* + +*Recommended format for Shepherds:* + + `[Name](mailto:username@example.com)` + +## Motivation + +There are currently no documented definitions for what a "bundle" is in the HCA DCP, including no definitions of what +they are, what they contain, or who they support. Lack of clarity causes confusion for DCP developers and data consumers +about how the data and metadata in the DCP are structured and organized. + +### User Stories + +1. As a tool developer, I would like to identify and get raw data in the DSS so that I can run my data processing and + analysis tools. + +1. As a tool developer, I would like to identify and get alignment results and count matrices in order to produce + expression matrices. + +1. As a computational biologist, I would like to get expression matrices for the Immune Cell Atlas project so that I can + do my research. + +1. As an external user of the DSS who is developing analysis tools, I would like to browse, query, and get raw and + processed data based on some rough criteria so that I can understand what the data are and develop my tool. + +1. As an Azul developer I want to be able to distinguish between different types of bundles in order to know which + bundle to index, link, etc. + +1. As a computational biologist, I would like to know where the reference data is. + +## Detailed Design + +Introduce a new field, `bundle-type`, in the DSS bundle manifest (the response to `GET bundle` in DSS). The field value +is a string formatted in a similar manner +to [DCP Media Types](https://docs.google.com/document/d/1TqihrgXjct9aDmTJO52_gE2WlpFysB1OkG9C8exmWTw) and consistent +with the syntax defined in [RFC 7231](https://tools.ietf.org/html/rfc7231#section-3.1.1.1). DSS must require the bundle +type to be specified when creating a new bundle, but should not enforce the vocabulary. DSS must allow subscriptions +and search results to be predicated on bundle type using the same string predicates available with other metadata and +manifest string fields. + +### Registry of Bundle Types + +Introduce a registry of bundle types to be maintained by the Metadata team. The initial contents of the registry are: + +| Bundle Type | Example `bundle-type` value | +|--------------------------|-----------------------------------------------------------| +| Project Metadata Bundle | `hca/project` | +| Primary Sequence Bundle | `hca/primary-data; hca-data-type:SmartSeq2` | +| Primary Imaging Bundle | `hca/primary-data; hca-data-type:sptx` | +| Secondary Analysis Bundle| `hca/analysis-output; hca-pipeline:snap-atac` | +| Reference Resource Bundle| `hca/analysis-support; hca-resource-type:reference-genome`| +| Expression Matrix Bundle | `hca/analysis-output; hca-resource-type:expression-matrix`| + +### Unresolved Questions + +- Bundle degrees: Useful? Who assigns them? Who uses them? +- Is there actually added value to not having to update all bundles when e.g. the project is updated? +- Versioned/unversioned references? + +### Prior Art + +[Submission to bundles SOP](https://docs.google.com/document/d/1x8mYLU8ubpZtTrzkwJrft1heqReX-pJLjJfb2X0fd2w) (Apr 2018) +[Bundle metadata schema update ticket](https://github.com/HumanCellAtlas/metadata-schema/issues/986) + +### Alternatives + +- Status Quo: no well-defined bundle types. Existing DCP components continue to infer bundle types via ad hoc methods + and included metadata. From 4734480eadb0ccd7f797132de4d0ffdb6ab883d9 Mon Sep 17 00:00:00 2001 From: Andrey Kislyuk Date: Mon, 15 Jul 2019 11:37:54 -0700 Subject: [PATCH 2/9] Wording fixup --- rfcs/text/0000-bundle-types.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfcs/text/0000-bundle-types.md b/rfcs/text/0000-bundle-types.md index a9c80150..d881086f 100644 --- a/rfcs/text/0000-bundle-types.md +++ b/rfcs/text/0000-bundle-types.md @@ -52,8 +52,8 @@ about how the data and metadata in the DCP are structured and organized. ## Detailed Design -Introduce a new field, `bundle-type`, in the DSS bundle manifest (the response to `GET bundle` in DSS). The field value -is a string formatted in a similar manner +A new field, `bundle-type`, will be introduced to the DSS bundle manifest (the response to `GET bundle` in DSS). The +field value is a string formatted in a similar manner to [DCP Media Types](https://docs.google.com/document/d/1TqihrgXjct9aDmTJO52_gE2WlpFysB1OkG9C8exmWTw) and consistent with the syntax defined in [RFC 7231](https://tools.ietf.org/html/rfc7231#section-3.1.1.1). DSS must require the bundle type to be specified when creating a new bundle, but should not enforce the vocabulary. DSS must allow subscriptions From c53a3ce7d9528e8d992171c4fe3f6fa8b6254653 Mon Sep 17 00:00:00 2001 From: Andrey Kislyuk Date: Mon, 15 Jul 2019 12:49:10 -0700 Subject: [PATCH 3/9] Add Query Service use case --- rfcs/text/0000-bundle-types.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfcs/text/0000-bundle-types.md b/rfcs/text/0000-bundle-types.md index d881086f..20cd7e66 100644 --- a/rfcs/text/0000-bundle-types.md +++ b/rfcs/text/0000-bundle-types.md @@ -45,8 +45,8 @@ about how the data and metadata in the DCP are structured and organized. 1. As an external user of the DSS who is developing analysis tools, I would like to browse, query, and get raw and processed data based on some rough criteria so that I can understand what the data are and develop my tool. -1. As an Azul developer I want to be able to distinguish between different types of bundles in order to know which - bundle to index, link, etc. +1. As an Azul or Query Service developer I want to be able to distinguish between different types of bundles in order to + know which bundle to index, link, etc. or to provide bundle types to users as a search criterion. 1. As a computational biologist, I would like to know where the reference data is. From b4d5f5e2448b635cf5cdf0a9275e2129b6c0808a Mon Sep 17 00:00:00 2001 From: Andrey Kislyuk Date: Wed, 11 Sep 2019 11:11:20 -0700 Subject: [PATCH 4/9] Add shepherd --- rfcs/text/0000-bundle-types.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/rfcs/text/0000-bundle-types.md b/rfcs/text/0000-bundle-types.md index 20cd7e66..9956d9ec 100644 --- a/rfcs/text/0000-bundle-types.md +++ b/rfcs/text/0000-bundle-types.md @@ -19,11 +19,8 @@ confidence among DCP developers when working with the DCP data model. * [Andrey Kislyuk](mailto:akislyuk@chanzuckerberg.com) ## Shepherd -***Leave this blank.** This role is assigned by DCP PM to guide the **Author(s)** through the RFC process.* -*Recommended format for Shepherds:* - - `[Name](mailto:username@example.com)` +* [Andrey Kislyuk](mailto:akislyuk@chanzuckerberg.com) ## Motivation From 43e0d86f2b48bc085513f477a6f155286563c1a4 Mon Sep 17 00:00:00 2001 From: Andrey Kislyuk Date: Wed, 11 Sep 2019 11:13:34 -0700 Subject: [PATCH 5/9] Expand reference bundle user story to clarify --- rfcs/text/0000-bundle-types.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/text/0000-bundle-types.md b/rfcs/text/0000-bundle-types.md index 9956d9ec..a4ae82b1 100644 --- a/rfcs/text/0000-bundle-types.md +++ b/rfcs/text/0000-bundle-types.md @@ -45,7 +45,7 @@ about how the data and metadata in the DCP are structured and organized. 1. As an Azul or Query Service developer I want to be able to distinguish between different types of bundles in order to know which bundle to index, link, etc. or to provide bundle types to users as a search criterion. -1. As a computational biologist, I would like to know where the reference data is. +1. As a computational biologist, I would like to use the same references that the HCA DCP uses in their analysis. ## Detailed Design From 4d0a5885d700436d0d1f86e00bd5f459783a7518 Mon Sep 17 00:00:00 2001 From: Andrey Kislyuk Date: Wed, 11 Sep 2019 11:14:50 -0700 Subject: [PATCH 6/9] Rename analysis-support to analysis-reference --- rfcs/text/0000-bundle-types.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/rfcs/text/0000-bundle-types.md b/rfcs/text/0000-bundle-types.md index a4ae82b1..48eeaf63 100644 --- a/rfcs/text/0000-bundle-types.md +++ b/rfcs/text/0000-bundle-types.md @@ -61,14 +61,14 @@ manifest string fields. Introduce a registry of bundle types to be maintained by the Metadata team. The initial contents of the registry are: -| Bundle Type | Example `bundle-type` value | -|--------------------------|-----------------------------------------------------------| -| Project Metadata Bundle | `hca/project` | -| Primary Sequence Bundle | `hca/primary-data; hca-data-type:SmartSeq2` | -| Primary Imaging Bundle | `hca/primary-data; hca-data-type:sptx` | -| Secondary Analysis Bundle| `hca/analysis-output; hca-pipeline:snap-atac` | -| Reference Resource Bundle| `hca/analysis-support; hca-resource-type:reference-genome`| -| Expression Matrix Bundle | `hca/analysis-output; hca-resource-type:expression-matrix`| +| Bundle Type | Example `bundle-type` value | +|--------------------------|-------------------------------------------------------------| +| Project Metadata Bundle | `hca/project` | +| Primary Sequence Bundle | `hca/primary-data; hca-data-type:SmartSeq2` | +| Primary Imaging Bundle | `hca/primary-data; hca-data-type:sptx` | +| Secondary Analysis Bundle| `hca/analysis-output; hca-pipeline:snap-atac` | +| Reference Resource Bundle| `hca/analysis-reference; hca-resource-type:reference-genome`| +| Expression Matrix Bundle | `hca/analysis-output; hca-resource-type:expression-matrix` | ### Unresolved Questions From 2c4a65e4b499299916e09fb64c41bea149207dac Mon Sep 17 00:00:00 2001 From: Andrey Kislyuk Date: Sun, 15 Sep 2019 13:41:20 -0700 Subject: [PATCH 7/9] Clarify bundle type registry ownership --- rfcs/text/0000-bundle-types.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/rfcs/text/0000-bundle-types.md b/rfcs/text/0000-bundle-types.md index 48eeaf63..e5ddc429 100644 --- a/rfcs/text/0000-bundle-types.md +++ b/rfcs/text/0000-bundle-types.md @@ -59,7 +59,8 @@ manifest string fields. ### Registry of Bundle Types -Introduce a registry of bundle types to be maintained by the Metadata team. The initial contents of the registry are: +Introduce a registry of bundle types to be maintained by the Data Store team with specification input from the Metadata +team. The initial contents of the registry are: | Bundle Type | Example `bundle-type` value | |--------------------------|-------------------------------------------------------------| From 6f3c135c04b020388eb70644c0e828ef60594345 Mon Sep 17 00:00:00 2001 From: Andrey Kislyuk Date: Mon, 16 Sep 2019 08:38:43 -0700 Subject: [PATCH 8/9] Address comments in review --- rfcs/text/0000-bundle-types.md | 53 +++++++++++++++++++++++++++------- 1 file changed, 43 insertions(+), 10 deletions(-) diff --git a/rfcs/text/0000-bundle-types.md b/rfcs/text/0000-bundle-types.md index e5ddc429..e198aa34 100644 --- a/rfcs/text/0000-bundle-types.md +++ b/rfcs/text/0000-bundle-types.md @@ -8,8 +8,8 @@ ## Summary -Formalizes data bundle types, definitions, target users/consumers, and specific use cases to improve clarity and -confidence among DCP developers when working with the DCP data model. +Formalizes *de facto* data bundle types, definitions, target users/consumers, and specific use cases to improve clarity +and confidence among DCP developers when working with the DCP data model. ## Author(s) @@ -49,6 +49,39 @@ about how the data and metadata in the DCP are structured and organized. ## Detailed Design +### What is a bundle? + +A bundle is a unit of data organization in the DCP, roughly equivalent to a folder or directory. Bundles in DCP can be +categorized into data bundles and utility bundles. + +* Data bundles contain the results of a single data transaction. They can be viewed as the digital equivalent of an + "instrument run" - a quantity of data gathered from a single process instance. Data bundles are annotated with + metadata using a procedure designed by the metadata team. + +* Utility bundles exist to organize (group) data bundles and represent data of non-DCP origin (such as reference data) + that is used in the course of analysis. + +### How are bundle types defined? + +Operationally, bundle types can be defined like file types or media types: they are used as a data type identifier for +applications to pattern match on whether data should be considered as suitable for input to the application. + +### When is a new bundle type needed? + +* There are 3 classes of data bundles currently provided or concretely envisioned in DCP: primary data bundles, + secondary analysis bundles, and tertiary (meta-analysis or aggregate analysis) bundles. A new class of bundle could be + needed if a new analysis modality or data aggregation method were to be introduced. + +* As denoted in the bundle type notation sketch below, [RFC 7231](https://tools.ietf.org/html/rfc7231) parameters can be + used to designate the process that generated a bundle. A new class of process (for example, a new secondary analysis + pipeline) could cause a new `bundle-type` value with the same bundle type and subtype as previous secondary analysis + data bundles, but with a new parameter value for `hca-pipeline`. + +* Similarly, a new modality or format of primary experimental output data would cause a new primary data bundle + `bundle-type` value. + +### Conveying bundle types + A new field, `bundle-type`, will be introduced to the DSS bundle manifest (the response to `GET bundle` in DSS). The field value is a string formatted in a similar manner to [DCP Media Types](https://docs.google.com/document/d/1TqihrgXjct9aDmTJO52_gE2WlpFysB1OkG9C8exmWTw) and consistent @@ -62,14 +95,14 @@ manifest string fields. Introduce a registry of bundle types to be maintained by the Data Store team with specification input from the Metadata team. The initial contents of the registry are: -| Bundle Type | Example `bundle-type` value | -|--------------------------|-------------------------------------------------------------| -| Project Metadata Bundle | `hca/project` | -| Primary Sequence Bundle | `hca/primary-data; hca-data-type:SmartSeq2` | -| Primary Imaging Bundle | `hca/primary-data; hca-data-type:sptx` | -| Secondary Analysis Bundle| `hca/analysis-output; hca-pipeline:snap-atac` | -| Reference Resource Bundle| `hca/analysis-reference; hca-resource-type:reference-genome`| -| Expression Matrix Bundle | `hca/analysis-output; hca-resource-type:expression-matrix` | +| Bundle Type | Bundle Class | Example `bundle-type` value | +|--------------------------|---------------------|-------------------------------------------------------------| +| Project Metadata Bundle | Utility bundle | `hca/project` | +| Primary Sequence Bundle | Primary data bundle | `hca/primary-data; hca-data-type:SmartSeq2` | +| Primary Imaging Bundle | Primary data bundle | `hca/primary-data; hca-data-type:sptx` | +| Secondary Analysis Bundle| Analysis data bundle| `hca/analysis-output; hca-pipeline:snap-atac` | +| Reference Resource Bundle| Utility bundle | `hca/analysis-reference; hca-resource-type:reference-genome`| +| Expression Matrix Bundle | Analysis data bundle| `hca/analysis-output; hca-resource-type:expression-matrix` | ### Unresolved Questions From f6d2cadda9d8b66b5a98ff37afd4b302527d2e24 Mon Sep 17 00:00:00 2001 From: Andrey Kislyuk Date: Mon, 16 Sep 2019 08:48:47 -0700 Subject: [PATCH 9/9] WIP --- rfcs/text/0000-bundle-types.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/rfcs/text/0000-bundle-types.md b/rfcs/text/0000-bundle-types.md index e198aa34..16501af8 100644 --- a/rfcs/text/0000-bundle-types.md +++ b/rfcs/text/0000-bundle-types.md @@ -104,11 +104,14 @@ team. The initial contents of the registry are: | Reference Resource Bundle| Utility bundle | `hca/analysis-reference; hca-resource-type:reference-genome`| | Expression Matrix Bundle | Analysis data bundle| `hca/analysis-output; hca-resource-type:expression-matrix` | +The registry would be maintained either as part of this git repository (HumanCellAtlas/dcp-community) or as part of a +repository chosen by the Data Store team, subject to a pull request-based contribution model. + ### Unresolved Questions -- Bundle degrees: Useful? Who assigns them? Who uses them? -- Is there actually added value to not having to update all bundles when e.g. the project is updated? -- Versioned/unversioned references? +- What defines a "correct" bundle type? + +- Is the level of detail for the operational requirements of a bundle registry sufficient? ### Prior Art