Skip to content

OssieSource: read current-spec Ossie documents, and never load less than they declare (v0.63.0) - #164

Merged
flyersworder merged 4 commits into
mainfrom
fix/ossie-flat-documents
Nov 7, 2026
Merged

flyersworder merged 4 commits into
mainfrom
fix/ossie-flat-documents

Conversation

@flyersworder

@flyersworder flyersworder commented Nov 7, 2026 •

Copy link
Copy Markdown
Owner

Why

The Ossie spec moved under OssieSource (last changed 2026-09-05):

The contract's allowlist and forbidden operations were enforced throughout; what the agent lost was the metrics and joins.

What changes

Document shape and silent drops

  • Flat documents are read; the legacy semantic_model list still loads, with a warning.
  • Refused at load instead of loading empty or partial: no model, both shapes, a model with no datasets, a nameless dataset/field/metric, a metric with no expression.
  • Named in warnings: a vendor-block key matching no metric (with a dataset.metric hint), a vendor listed twice, a duplicate metric name.

Physical columns (bug since v0.42.0)

  • A field's column is the one its expression reads (plain, backtick or bracket quoting): NVIDIA's customer_name over name now registers name; a computed field, a keyword like current_date, or an unparseable expression registers none. Relationship endpoints are mapped the same way. No fixture renamed a field, so no test could see this.
  • Metric expressions still name fields, so describe_table carries the bridge: name reads (Ossie field: customer_name), and orders lists net_total = subtotal - discount as computed. Translating metric SQL is deferred: it means generating SQL, which Denodo consumers cannot use.

Expressions

Other

  • Dataset-scoped metrics, named dataset.metric. No metric gets a source_model from Ossie, since its expression names fields, not columns.
  • Vendor blocks on datasets, fields, metrics and relationships carried in get_extras()["ossie_entity_extensions"] (NVIDIA puts each metric's physical SQL there).

Behaviour change

0.63.0, not a patch:

  • a spec-invalid Ossie file that used to load (partially) now raises a ValueError naming what is missing;
  • a drill_by in our vendor block written with a renamed field's name now fails with "unknown column" (it named a column the table does not have).

Testing

  • 61 tests in tests/test_semantic/test_ossie_document_shape.py, each watched failing first. Fixture tests/fixtures/ossie_flat_nvidia_auto_ontology.yml is the converter's real output (apache/ossie-converters@cb4b46b, nvidia/), not a hand-written example.
  • Two older test documents were spec-invalid (models with no datasets, metrics with empty dialects) and were made valid.
  • Full suite: 3245 passed; prek run --all-files clean.
  • End to end through the agent tools on the NVIDIA export: describe_table shows physical columns, lookup_metric returns the metric, its certified SQL runs on DuckDB, allowlist and forbidden operations still block.
  • Three review rounds: a correctness review and a review against the Ossie spec and JSON schema before the PR, and a final correctness review of the PR (10 findings, all addressed in 3429e1d).

Deferred

  • Identifier case-insensitivity (primary_key: [ID] vs to_columns: [id]).
  • Tab-indented JSON files.
  • Translating Ossie field references in metric expressions to physical SQL.

🤖 Generated with Claude Code

flyersworder and others added 4 commits November 7, 2026 08:37
…m empty

apache/ossie#383 made a document one model with `name` and `datasets` at the
root and dropped the `semantic_model` list, which was all OssieSource read.
A current-spec document, such as NVIDIA Auto-Ontology's export, loaded with
no tables, metrics or relationships and no warning. Both shapes are now read,
the legacy one with a warning, and a document holding no model is refused.

Also reads `expression.sql` (#617), which loaded as an empty expression, the
`OSSIE_SQL_2026` dialect spelling, and dataset-scoped metrics (#591), named
`dataset.metric` with the dataset's table as source_model.

The converter's real output is a fixture, so the test pins what a producer
emits rather than our reading of the spec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rops

From two reviews of the flat-document fix, one against the Ossie spec:

- A field's column is the one its expression reads, not its name: NVIDIA's
  `customer_name` over `name` registered a column the table lacks, and
  computed fields were listed as columns. Relationship endpoints follow.
- A model with no datasets, a nameless dataset/field/metric, and a metric
  with no expression are refused instead of loading empty or partial.
- custom_extensions on datasets, fields, metrics and relationships are
  carried in `ossie_entity_extensions`; NVIDIA puts each metric's SQL there.
- Dialect names compare case-folded, the caller's included.
- Dataset-scoped metrics no longer claim a source_model: their expression
  names fields, so "reads this table alone" was unsupported.
- Unmatched vendor-block keys, repeated vendors and duplicate metric names
  are named in warnings.
- Docs: physical columns, entity extensions, the ANSI-first rationale
  (the AGG claim was wrong), ossie-schema.json, additionalProperties scope.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dialect fallback took the first declared expression whatever its
language, so a Tableau export's `SUM([Profit])` reached lookup_metric as
the metric's SQL. The fallback now skips the formula languages in Ossie's
dialect registry (TABLEAU, DAX, MDX, MAQL, SIGMA, THOUGHTSPOT,
HOLISTICS_AQL, SPARQL); an explicit dialect= still returns one, and an
unregistered dialect is still a fallback.

A metric written only in those is kept the way dbt and Cube keep one they
cannot express (#123): empty sql_expression, and an `untranslated` reason
quoting each formula. A field written only in one registers no column.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Field columns are read with sqlglot only (plain SQL, then BigQuery and
  T-SQL quoting); the regex shortcut registered `current_date`, `TRUE` and
  `NULL` as columns, and a tokenizer error crashed the whole load.
- describe_table carries the field vocabulary metrics still use: a renamed
  column notes its Ossie field, and a table lists its computed fields.
- A same-named legacy model no longer overwrites an earlier one's vendor
  blocks; an empty dialect entry no longer hides a real expression; the
  unmatched-overlay warning names the caller's vendor; duplicate check O(1).
- CHANGELOG: drill_by now names a physical column, a loud break for one
  written with a renamed field's name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@flyersworder
flyersworder merged commit ca44258 into main Nov 7, 2026
9 checks passed
@flyersworder
flyersworder deleted the fix/ossie-flat-documents branch November 7, 2026 11:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant