Feat: OSI semantic-model core entity and CRUD by flyrain · Pull Request #4961 · apache/polaris

flyrain · 2026-07-02T23:59:43Z

Turning the 501 scaffolding from #4816 into working create/list/load/update/ drop for OSI semantic models on the JDBC backend.

Checklist

🛡️ Don't disclose security issues! (contact security@apache.org)
🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
🧪 Added/updated tests with good coverage, or manually tested (and explained how)
💡 Added comments for complex logic
🧾 Updated CHANGELOG.md (if needed)
📚 Updated documentation in site/content/in-dev/unreleased (if needed)

adam-christian-software

Overall, this looks good!

adam-christian-software · 2026-07-06T14:40:48Z

+      throw new IllegalStateException(
+          String.format(
+              "Failed to drop semantic model %s error status: %s with extraInfo: %s",
+              identifier, result.getReturnStatus(), result.getExtraInformation()));


Just for my understanding, is all of the extra information returned by this exception guaranteed to not cause any additional information disclosure? Like, we aren't leaking out the DB IP or anything like that?

Good question! The persistence layer implementation is not supposed to leak any sensitive information. It's kind of a convention, please check the other use cases, like https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/LocalIcebergCatalog.java#L541-L541

adam-christian-software · 2026-07-06T14:43:56Z

+   * current catalog. Fails with 400 and a JSON-Pointer to the offending dataset if any source does
+   * not resolve. Column-level checks are deferred (F5).
+   */
+  private void resolveSourcesOrThrow(JsonNode semanticModel) {


Is this method this class's responsibility? I thought that the SemanticDocumentValidator would have been the one who would have validated the syntax and the semantics of the document.

As the design doc(https://docs.google.com/document/d/1ZdI-1w_5LbyCMhvUhLCtOt-N1Z89L2P-oiGLaYayCZg/edit?tab=t.0#heading=h.golo1tnt8wb6) described, there are two types of validations:

Schema validation: Writes validate against the bundled OSI JSON Schema. A new OsiDocumentValidator produces BadRequestException with JSON-Pointer field paths on failure. The validator is strict: unknown top-level fields cause a 400. Forward compatibility with newer OSI versions is handled by upgrading Polaris's bundled schema as an explicit, coordinated action, not by silently accepting unvalidated content. Vendor-specific or experimental fields belong in custom_extensions, which the schema does allow.

Source table validation: Validate that every dataset.source resolves to a Polaris TABLE_LIKE entity at write time; writes with unresolved sources fail with 400.

In this PR, we defer schema validation to a SemanticDocumentValidator, but we still apply source table validation. Theoretically, both are tied to OSI spec versions, but the dataset source concept is much more stable than the overall JSON schema. I think it makes sense to keep it here.
WDYT?

dimas-b · 2026-07-10T18:37:15Z

    return catalogEntity;
  }

-  private boolean shouldDecodeToken() {


Is this related to Semantic Model? Perhaps it is worth removing in a separate PR with a focused commit message 🤔

Reverted the change in the new commit.

dimas-b · 2026-07-10T18:38:08Z

      case PolicyMappingAlreadyExistsException policyMappingAlreadyExistsException ->
          Response.Status.CONFLICT;
      case PolicyInUseException policyInUseException -> Response.Status.BAD_REQUEST;
+      case NoSuchSemanticModelException noSuchSemanticModelException -> Response.Status.NOT_FOUND;


Do we really need a new exception class for this? 🤔

Yeah, this is similar to class NoSuchTableException, NoSuchNamespaceException, NoSuchPolicyException, and etc.

dimas-b · 2026-07-10T18:46:18Z

+ * Thrown when a semantic-model update supplies an {@code entity-version} that does not match the
+ * version currently stored in the catalog (optimistic-concurrency conflict). Maps to HTTP 409.
+ */
+public class SemanticModelVersionMismatchException extends PolarisException {


Who is expected to catch or process these exceptions after they are thrown?

The class PolarisExceptionMapper processes and maps it to a 409, check the line 64.

Implements the design's Phase 0 (spec freeze) and Phase 3 (core entity), turning the 501 scaffolding from apache#4816 into working create/list/load/update/ drop for OSI semantic models on the JDBC backend. Core (polaris-core): - New SEMANTIC_MODEL(11, NAMESPACE) entity type and SemanticModelEntity, storing the OSI document (spec-version + JSON content) in entity properties, mirroring PolicyEntity. - ResolvedPathKey.ofSemanticModel; SEMANTIC_MODEL_MAX_{DOCUMENT,EXPRESSION}_BYTES feature configs; NoSuchSemanticModel/SemanticModelVersionMismatch exceptions wired into PolarisExceptionMapper (404/409). - Five authorizable operations registered in RbacOperationSemantics, gated on CATALOG_MANAGE_CONTENT as an interim (the dedicated SEMANTIC_MODEL_* privilege matrix and read-time enforcement modes land in the authorization phase). Extension (extensions/semantic-models): - Bundled OSI v0.1.1 JSON Schema and OsiDocumentValidator (strict schema validation with JSON-Pointer errors, document/expression size caps) via networknt json-schema-validator. - SemanticModelCatalog/Handler/Factory: optimistic concurrency on entity-version, write-time dataset.source -> TABLE_LIKE resolution (400 on unresolved), list pagination. Adapter now dispatches to the handler. - Unit tests: OsiDocumentValidatorTest and SemanticModelCatalogTest (plain JUnit + Mockito, following the extensions/auth/opa pattern). NoSQL support is deferred to a follow-up; SEMANTIC_MODEL is excluded from the NoSQL entity-type completeness check so the module keeps loading.

- PolarisEntityTypeTest.fromCode: code 11 now maps to SEMANTIC_MODEL (was asserted as null); shift the unknown-code case to 12. - runtime/server/distribution/LICENSE: add Apache-2.0 mentions for the new json-schema-validator dependency and its transitive com.ethlo.time:itu, required by :polaris-server:generateLicenseReport.

Ship only the SemanticDocumentValidator contract in this PR; the concrete OSI schema-validation implementation lands in a follow-up. - Add SemanticDocumentValidator interface (validate(SemanticModelDocument)). - Remove OsiDocumentValidator and its test, the bundled osi-schema.json resource and its RAT exclude, the networknt json-schema-validator dependency, the two LICENSE mentions (json-schema-validator, com.ethlo.time:itu), and the SEMANTIC_MODEL_MAX_*_BYTES feature configs. - SemanticModelCatalog now parses the document body and resolves dataset.source references only (schema/size validation deferred); drop the size-cap constructor params. Malformed JSON still fails with 400.

Address review feedback: define the '.' used to split an OSI dataset.source into a documented SOURCE_SEPARATOR constant so the dot-splitting per the IRC object-identifier scheme is explicit.

Create and update crashed at runtime against the real resolution manifest: both probed keys via getPassthroughResolvedPath that were never registered via addPassthroughPath, and that method hard-throws on an unknown key. The mock-based unit test stubbed the view, so the failure was invisible. - Drop the create existence pre-check that probed the (unregistered) semantic-model path; rely on createEntityIfNotExists returning ENTITY_ALREADY_EXISTS, which is already handled. - Resolve each dataset.source with a fresh single-use PolarisResolutionManifest instead of the request manifest, since sources are parsed from the opaque document and cannot be pre-registered. SemanticModelCatalog now takes a ResolutionManifestFactory and PolarisPrincipal. Add real-manifest handler tests (SemanticModelCatalogHandlerCrudTest, SemanticModelCatalogHandlerAuthzTest) driven through TestServices, sharing a common bootstrap base. These exercise the strict manifest that the Mockito unit test cannot, and cover the CATALOG_MANAGE_CONTENT deny path.

Copilot

Pull request overview

This PR turns the previously scaffolded OSI semantic-model API into a working implementation by introducing a core SEMANTIC_MODEL entity type and wiring create/list/load/update/drop through the catalog handler to persistence (JDBC-backed), including basic optimistic concurrency and REST exception-to-status mapping.

Changes:

Add SEMANTIC_MODEL as a first-class PolarisEntityType, plus ResolvedPathKey helpers and new PolarisAuthorizableOperation entries.
Implement semantic-model CRUD in extensions/semantic-models (adapter + handler + catalog logic), including source-table resolution and optimistic-concurrency conflict handling.
Add unit + “real manifest” handler tests, and extend exception mapping / pagination helper reuse in the runtime service layer.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
runtime/service/src/main/java/org/apache/polaris/service/exception/PolarisExceptionMapper.java	Map semantic-model not-found and version-mismatch exceptions to 404/409.
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java	Remove local pagination-token decode helper in favor of shared base logic.
runtime/service/src/main/java/org/apache/polaris/service/catalog/common/CatalogHandler.java	Add shared `shouldDecodeToken()` honoring `LIST_PAGINATION_ENABLED` with catalog overrides.
polaris-core/src/test/java/org/apache/polaris/core/entity/PolarisEntityTypeTest.java	Update entity-type code mapping tests to include `SEMANTIC_MODEL`.
polaris-core/src/main/java/org/apache/polaris/core/semantic/SemanticModelEntity.java	New entity wrapper + builder for storing semantic-model documents in entity properties.
polaris-core/src/main/java/org/apache/polaris/core/semantic/exceptions/SemanticModelVersionMismatchException.java	New exception for optimistic concurrency conflicts (intended 409).
polaris-core/src/main/java/org/apache/polaris/core/semantic/exceptions/NoSuchSemanticModelException.java	New exception for missing semantic models (intended 404).
polaris-core/src/main/java/org/apache/polaris/core/persistence/resolver/ResolvedPathKey.java	Add `ofSemanticModel(...)` helper for resolver lookups.
polaris-core/src/main/java/org/apache/polaris/core/entity/PolarisEntityType.java	Add `SEMANTIC_MODEL(11, ...)` to the entity-type enum.
polaris-core/src/main/java/org/apache/polaris/core/auth/RbacOperationSemantics.java	Register semantic-model ops (interim) under `CATALOG_MANAGE_CONTENT`.
polaris-core/src/main/java/org/apache/polaris/core/auth/PolarisAuthorizableOperation.java	Add five semantic-model authorizable operations.
persistence/nosql/persistence/metastore-types/src/test/java/org/apache/polaris/persistence/nosql/coretypes/mapping/TestEntityObjMappings.java	Temporarily exclude `SEMANTIC_MODEL` from NoSQL mapping coverage checks.
persistence/nosql/persistence/metastore-types/src/main/java/org/apache/polaris/persistence/nosql/coretypes/mapping/EntityObjMappings.java	Temporarily exclude `SEMANTIC_MODEL` from NoSQL backend type completeness enforcement.
extensions/semantic-models/src/test/java/org/apache/polaris/service/catalog/semanticmodel/SemanticModelCatalogTest.java	New Mockito-based unit tests for semantic-model catalog CRUD and validation.
extensions/semantic-models/src/test/java/org/apache/polaris/service/catalog/semanticmodel/SemanticModelCatalogHandlerCrudTest.java	End-to-end handler CRUD tests via `TestServices` and strict manifest behavior.
extensions/semantic-models/src/test/java/org/apache/polaris/service/catalog/semanticmodel/SemanticModelCatalogHandlerAuthzTest.java	Authz regression tests ensuring unprivileged principals are rejected for all ops.
extensions/semantic-models/src/test/java/org/apache/polaris/service/catalog/semanticmodel/AbstractSemanticModelCatalogHandlerTest.java	Shared `TestServices` bootstrap for handler integration-style tests.
extensions/semantic-models/src/main/java/org/apache/polaris/service/catalog/semanticmodel/SemanticModelCatalogHandlerFactory.java	CDI factory to construct request-scoped semantic-model handlers.
extensions/semantic-models/src/main/java/org/apache/polaris/service/catalog/semanticmodel/SemanticModelCatalogHandler.java	Handler layer performing authz + resolution and delegating to `SemanticModelCatalog`.
extensions/semantic-models/src/main/java/org/apache/polaris/service/catalog/semanticmodel/SemanticModelCatalogAdapter.java	Replace 501 stub with real dispatch to handler, gated by `ENABLE_SEMANTIC_MODELS`.
extensions/semantic-models/src/main/java/org/apache/polaris/service/catalog/semanticmodel/SemanticModelCatalog.java	Implement core semantic-model CRUD, source resolution, and optimistic concurrency behavior.
extensions/semantic-models/src/main/java/org/apache/polaris/service/catalog/semanticmodel/SemanticDocumentValidator.java	Add (future) validation interface for schema/size enforcement.
extensions/semantic-models/build.gradle.kts	Add runtime-service test fixtures + related deps to support handler tests.

+  CREATE_SEMANTIC_MODEL,
+  LOAD_SEMANTIC_MODEL,
+  UPDATE_SEMANTIC_MODEL,
+  DROP_SEMANTIC_MODEL,
+  LIST_SEMANTIC_MODEL,


Revert the extraction of shouldDecodeToken into the base CatalogHandler so this PR stays focused on semantic models. IcebergCatalogHandler keeps its original private method, and SemanticModelCatalogHandler holds its own copy. The dedup can land in a separate, focused PR.

MonkeyCanCode · 2026-07-30T04:08:05Z

+            PolarisCatalogHelpers.identifierToList(namespace, identifier.getName()),
+            PolarisEntityType.SEMANTIC_MODEL,
+            true /* optional */));
+    resolutionManifest.resolveAll();


This is style inconsistency.

Should we not go through authorizer().resolveAuthorizationInputs() SPI? For reference:

polaris/runtime/service/src/main/java/org/apache/polaris/service/catalog/policy/PolicyCatalogHandler.java

Line 153 in 48ddd3d

authorizer()

Which it also means, we will need to update PolarisSecurable as the factory is missing from https://github.com/apache/polaris/blame/48ddd3d103d1c2f7a7d0bc2c974572fbbd62586e/runtime/service/src/main/java/org/apache/polaris/service/catalog/common/PolarisSecurableMapper.java#L36

Good catch. Made the change.

MonkeyCanCode · 2026-07-30T04:28:28Z

+import org.jspecify.annotations.Nullable;
+
+/**
+ * A Polaris entity that stores an Open Semantic Interchange (OSI) semantic-model document.


As OSI is now Apache Ossie, we should avoid using OSI or Open Semantic Interchange. From Apache Ossie ML, @jbonofre is leading the renaming effort via apache/ossie#288

Renamed OSI to Apache Ossie

singhpk234 · 2026-07-30T06:08:51Z

+   * Resolves and validates every {@code dataset.source} in the parsed OSI document against the
+   * current catalog. Every dataset must define a string {@code source}; a missing or non-string
+   * source, or one that does not resolve to a {@code TABLE_LIKE} entity, fails with 400 and a
+   * JSON-Pointer to the offending dataset. Column-level checks are deferred (F5).


is this something a catalog should validate ? or can rely on engine to have done due deligence ?
like normal views ?

Good question. One of problems of a traditional views is its engine-specific/dialectic SQL format, which prevent its interoperability. Ossie semnatic model uses a well defined JSON, which is intended to be interoperable across engines. This interoperability makes catalog level validation practical, while we can still allow engine specific checks stay with each engine. WDYT?

i wonder if we are doing double validation in this case though ? i don't have a strong stance here though ?

I think it is more of a sanity check in the catalog side. It also aligns with the future plan for reverse lookup API(list modes by a given table). I'm OK to remove it if you think it's a blocker.

Its not a blocker to me as i don't have a hard stance 🙂 and since we are in beta we can iterate, if we are keeping the validation we should have a through one that user has access to table too i would say ! this is come back in the MV discussion as well !

singhpk234

LGTM thanks @flyrain !

I had some feebacks about the validation what belongs to catalog and what to engine / caller, since this is beta we can keep iterating

singhpk234 · 2026-08-01T15:21:26Z

+    if (resolved == null
+        || resolved.getRawLeafEntity() == null
+        || resolved.getRawLeafEntity().getType() != PolarisEntityType.TABLE_LIKE) {
+      throw new BadRequestException(
+          "Semantic model source '%s' at %s does not resolve to a table or view in catalog '%s'",
+          source, pointer, catalogEntity.getName());


[ok with follow-up] : if we are checking entity existence, we should ideally also check if caller has access to these entites too ?

MonkeyCanCode

LGTM. Thanks @flyrain

snazy · 2026-08-03T13:18:06Z

Before this merges, I think we need to reconcile it with the dev@ discussion.

We agreed there that the current API is a beta document-hosting/registry surface and that the client discovery and consumption story is still evolving. My concern in that thread was specifically that durable implementation should not get ahead of that story.

This PR now makes SEMANTIC_MODEL a core entity and settles JDBC persistence and lifecycle semantics, while NoSQL support is explicitly excluded from its completeness check.

Could we first state the concrete v1 request flows this persistent model is intended to serve, and why they require this core entity/storage contract now? If v1 is intentionally only namespace/name CRUD for opaque documents, that is a possible answer, but it should be an explicit decision rather than an implementation side effect.

We should also either provide backend-neutral behavior or capability-gate the feature so an unsupported backend cannot advertise or enable it.

github-project-automation Bot added this to Basic Kanban Board Jul 2, 2026

github-project-automation Bot moved this to PRs In Progress in Basic Kanban Board Jul 2, 2026

adam-christian-software reviewed Jul 6, 2026

View reviewed changes

dimas-b reviewed Jul 10, 2026

View reviewed changes

flyrain added 9 commits July 10, 2026 13:06

Split schema validation and dataset parsing

5a60dd8

Extract dataset.source separator into a named constant

a40ffaa

Address review feedback: define the '.' used to split an OSI dataset.source into a documented SOURCE_SEPARATOR constant so the dot-splitting per the IRC object-identifier scheme is explicit.

Refactor

e6ea1ce

add todo

5f50fc8

resolve comments

57cc749

flyrain force-pushed the semantic-models-core-entity branch from 7208b20 to 4a57220 Compare July 10, 2026 21:06

flyrain marked this pull request as ready for review July 10, 2026 21:06

Copilot AI review requested due to automatic review settings July 10, 2026 21:06

Copilot started reviewing on behalf of flyrain July 10, 2026 21:07 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread polaris-core/src/main/java/org/apache/polaris/core/auth/PolarisAuthorizableOperation.java

Comment on lines +118 to +122

CREATE_SEMANTIC_MODEL,

LOAD_SEMANTIC_MODEL,

UPDATE_SEMANTIC_MODEL,

DROP_SEMANTIC_MODEL,

LIST_SEMANTIC_MODEL,

flyrain mentioned this pull request Jul 27, 2026

Refactor PolarisException status mapping into PolarisException hierarchy #5167

Open

MonkeyCanCode self-requested a review July 30, 2026 03:50

MonkeyCanCode reviewed Jul 30, 2026

View reviewed changes

singhpk234 reviewed Jul 30, 2026

View reviewed changes

flyrain added 2 commits July 30, 2026 16:33

Address semantic model review feedback

42c1b02

Resolve comments

7e0d859

flyrain mentioned this pull request Jul 31, 2026

Implement semantic model document validation #5205

Open

Resolve comments

347fd60

singhpk234 approved these changes Aug 1, 2026

View reviewed changes

github-project-automation Bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Aug 1, 2026

MonkeyCanCode reviewed Aug 1, 2026

View reviewed changes

MonkeyCanCode approved these changes Aug 1, 2026

View reviewed changes

Uh oh!

Conversation

flyrain commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

adam-christian-software left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MonkeyCanCode Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

singhpk234 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MonkeyCanCode left a comment

Choose a reason for hiding this comment

Uh oh!

snazy commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

flyrain commented Jul 2, 2026 •

edited

Loading

MonkeyCanCode Jul 30, 2026 •

edited

Loading