Skip to content

feat(framework): introduce typed product framework and Trino reference operator - #646

Open
whg517 wants to merge 1 commit into
zncdatadev:mainfrom
whg517:codex/framework-redesign-pr
Open

whg517 wants to merge 1 commit into
zncdatadev:mainfrom
whg517:codex/framework-redesign-pr

Conversation

@whg517

@whg517 whg517 commented Sep 15, 2026

Copy link
Copy Markdown
Member

Product operators currently have to coordinate configuration inheritance, resource assembly and lifecycle behavior across several extension points. This change introduces a typed product-definition framework: products declare defaults, resolved facts and runtime requirements, while the framework owns folding, overrides, assembly, validation and reconciliation. The Trino example exercises the complete path through generated APIs and controller registration.

What changes

  • Add public product/input/generator/registration APIs and an internal configuration-to-resource pipeline. Preserve field presence, fold role and role-group inputs, apply file/env/CLI overrides, and apply Pod overrides last, including precedence over env overrides.
  • Deliver controller ownership, dependency refresh, pause/stop/resume, role-level PDBs, orphan retirement and retained-volume identity checks.
  • Implement storage, Secret/Listener CSI, authentication, S3 connections, Vector logging, initialization/probes and ordered shutdown. Products remain responsible for native configuration and actual process consumption.
  • Add DataAsset/DataOperation APIs and a separately deployed executor for explicitly authorized adoption, migration and destruction, with identity checks, consumer exclusion, operation locks and retained history.
  • Replace the Trino example's legacy hooks/webhooks with generated registration and product adapters; provide materializer/executor packaging and reusable runtime acceptance tooling.
  • Consolidate formal contracts into architecture/security documentation. Discussion notes, prototypes and run outputs are excluded from both this branch's files and its new commit history; local records are not build dependencies.

Validation

Completed during implementation:

  • Root module tests and lint; Trino module tests and lint; generated-artifact checks; 29 Python harness tests; installer/executor rendering and Linux ARM64 executor build.
  • A complete isolated ARM64 kind run at the pre-publication implementation baseline passed all required stages and cleanup. It exercised Trino SQL and all four overrides, catalog refresh, pause/stop/recovery, retained storage, real Secret/Listener CSI and AutoTLS, PASSWORD authentication and Secret refresh, Hive/MinIO S3 writes/reads and connection refresh, native log delivery/destination refresh, active-query shutdown across controller restart, and explicit data adoption/migration/destruction with backend reclamation.

Rechecked after merging the latest upstream dependency changes:

  • make verify-generate
  • make -C examples/trino-operator test lint build
  • git diff --check
  • Documentation links and local archive integrity.

The full runtime run predates the final documentation cleanup and the upstream OAuth2/protobuf dependency updates; it has not been rerun with those updates. Remote CI results are reported separately by the PR checks. Reproduction commands and environment requirements are in hack/framework-e2e/README.md; private raw run records remain local.

Scope and review

This is a new framework API and a replacement of the Trino reference implementation, without a compatibility layer for the previous example CRD. The existing GenericReconciler SDK remains separately scoped. Start with pkg/framework, then the internal pipeline/controller, the Trino product adapters, and the independent data executor.

Runtime evidence covers the pinned Trino 476/ARM64 fixture and single RWO Filesystem data slots. File-tree migration is not database-consistent backup, and backend deletion is not a media-erasure guarantee. Data execution requires its own deployment and permissions; normal product reconciliation does not authorize it.

…e operator

Deliver generated input and registration, configuration/resource pipeline, platform and lifecycle domains, and independently authorized data operations. Consolidate formal documentation and exclude local discussion and runtime records.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant