Skip to content

Publish standalone YAML schema files as single source of truth - #12

Open
matsen wants to merge 1 commit into
mainfrom
11-publish-schema-yaml-files
Open

Publish standalone YAML schema files as single source of truth#12
matsen wants to merge 1 commit into
mainfrom
11-publish-schema-yaml-files

Conversation

@matsen

@matsen matsen commented Apr 3, 2026

Copy link
Copy Markdown
Contributor

🤖

Summary

  • Introduces olmsted_cli/schemas/{node,tree,clone,dataset}.schema.yaml as the single authoritative source for all Olmsted JSON schemas. Python loads these at import time, so the validation dicts are always identical to the published artifacts — structural divergence is impossible.
  • Resolves the duplicate timepoint_multiplicity_spec bug (lines 49 and 757 in old schemas.py): the canonical definition now lives in each YAML file's definitions block with a title, description, minimum: 0, nullable timepoint_id, and additionalProperties: false.
  • Generates *.schema.json files alongside the YAML for linking from AIRR Standards proposals. A make schemas target regenerates JSON from YAML; a new test_schema_freshness.py test fails if committed JSON is stale.

Deliverables checklist

  • olmsted_cli/schemas/node.schema.yaml — standalone with AIRR/extension annotations per field
  • olmsted_cli/schemas/tree.schema.yaml
  • olmsted_cli/schemas/clone.schema.yaml
  • olmsted_cli/schemas/dataset.schema.yamlfield_metadata.properties patched dynamically from FIELD_LEVELS
  • olmsted_cli/schemas/*.schema.json — generated artifacts
  • schemas.py replaced with YAML-loading code + dynamic patching
  • make schemas Makefile target
  • tests/test_schema_freshness.py — asserts JSON matches YAML
  • pyproject.toml updated to include schemas/*.yaml and schemas/*.json in package data

Test plan

  • All existing tests pass (190 pass; 2 pre-existing floating-point time-tree failures unaffected)
  • pytest tests/test_schema_freshness.py passes for all 4 schemas
  • make schemas regenerates JSON correctly (no-op when JSON is current)

Closes #11

🤖 Generated with Claude Code

@matsen
matsen requested a review from davidrich27 April 3, 2026 21:53
@matsen

matsen commented Apr 3, 2026

Copy link
Copy Markdown
Contributor Author

@davidrich27 the reason I went ahead and set this up is because I want the AIRR people to consider our schema. Can you try it out and make sure it works, please?

@matsen

matsen commented Apr 3, 2026

Copy link
Copy Markdown
Contributor Author

🤖

Nice work on the core mechanics — YAML-as-source-of-truth loaded at import time is exactly right, and the schemas.py reduction from 660→39 lines is clean.

Two things to revisit:

1. Single file vs four files. The official AIRR schema is one monolithic YAML (airr-schema.yaml) with all objects at root level and $ref: '#/Node'-style references between them. Our PR splits into four separate files, which means the node definition is duplicated inline in tree.schema.yaml and clone.schema.yaml (the comment even says "Inline copy of node.schema.yaml; kept in sync via YAML authoring convention" — that's exactly the kind of manual sync we were trying to eliminate). We should match the AIRR convention: one olmsted-schema.yaml with Node, Tree, Clone, and Dataset as root-level objects, cross-referenced via $ref.

2. x-airr extensions. The AIRR schema annotates every field with x-airr: blocks (nullable, identifier, adc-query-support, etc.). Our YAML uses # comments for the [AIRR]/[Olmsted extension] annotations, which are great for humans but invisible to tooling and get stripped from the generated JSON. If we want the Standards Committee to be able to mechanically diff our extensions against the base schema, we should adopt x-airr: (or an x-olmsted: namespace) for machine-readable annotation of which fields are standard vs extended.

Introduces olmsted_cli/schemas/olmsted-schema.yaml as the authoritative
source for the Node/Tree/Clone/Dataset schemas (AIRR-style monolithic
file with #/Object cross-refs and x-olmsted annotations). schemas.py
loads it at import and patches the dynamic fragments (field_metadata
levels, field type/display enums) from constants.py. A generated
olmsted-schema.json is committed alongside for linking from AIRR
Standards proposals; 'make schemas' regenerates it and
test_schema_freshness.py fails if it drifts.

Resolves the duplicate timepoint_multiplicity_spec definition (#11).

Rebased onto main and re-synced the schema to the post-#34 field set:
cdr3_sequence/cdr3_sequence_aa (node), min/max_cdr3_length and
reconstruction_method (tree), cdr3_alignment_start/end and
cdr1/cdr2/cdr3_length (clone), plus the expanded dataset.type text.

Closes #11

Co-Authored-By: David Rich <david.rich27@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@davidrich27
davidrich27 force-pushed the 11-publish-schema-yaml-files branch from bb87493 to 24991b2 Compare July 18, 2026 03:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Publish standalone schema files for the Olmsted AIRR extensions

1 participant