Skip to content

[Integration] Publish physical schema observations for OpenMetadata interoperability #1072

Description

@seonghobae

Buyer problem

Customers need PostgreSQL and supported warehouse schemas observed by pg-erd-cloud to become governed metadata without manually recreating tables, columns, constraints, and schema revisions in OpenMetadata.

Boundary

  • pg-erd-cloud owns physical schema inspection, snapshots, diff, ERD, and migration evidence.
  • It emits provider-neutral observed metadata/evidence through a released context-graph-contracts contract.
  • semantic-data-portal admits and projects that evidence and owns the OpenMetadata anti-corruption/write boundary.
  • pg-erd-cloud must not embed an OpenMetadata SDK, store OpenMetadata credentials, write directly to OpenMetadata, or query SDP databases.

Scope

  • Produce a versioned physical-schema observation envelope for database, schema, table, column, key, constraint, index, view, and supported relation identities.
  • Preserve source engine, normalized connection identity without credentials, snapshot ID/hash, inspector version, captured time, and database-native identifiers.
  • Publish schema additions, removals, and modifications as explicit snapshot/diff evidence.
  • Represent physical lineage only where deterministically observed; do not infer business lineage.
  • Emit no sample rows, credential material, DSNs, unbounded DDL, or query text in the general event.
  • Provide an operation receipt so SDP can reconcile every admitted or rejected observation.
  • Add an end-to-end fixture: PostgreSQL schema → pg-erd snapshot → contract event → SDP projection → OpenMetadata-ready table/column representation.

Acceptance

  • Same snapshot event is replay-idempotent.
  • Snapshot and diff hashes are deterministic for equivalent physical schemas.
  • Dropped objects close validity rather than deleting historical observations.
  • Unknown ownership, glossary, quality, and business domain remain unknown; pg-erd-cloud does not invent them.
  • Cross-tenant and cross-connection identity collisions are rejected.
  • Production statement/branch coverage and public API docs: 100%.
  • Real PostgreSQL integration, migration, fuzz, credential-redaction, and large-schema tests.

Dependency order

  1. context-graph-contracts#26 released.
  2. SDP durable admission contract released from semantic-data-portal#95.
  3. Producer implementation and exact-version consumer E2E.

Related: ContextualWisdomLab/semantic-data-portal#95, ContextualWisdomLab/context-graph-contracts#26.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions