Proposal: Add Flink SQL support #280
campossalex
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi all,
I'd like to propose adding Flink SQL as a supported target for Ossie and get feedback before putting together a PR.
Context: every converter merged or in review today (dbt, GoodData, Salesforce, Polaris, Spark) targets a query-at-rest engine, and the project's roadmap names Flink as a future target with nothing concrete started yet. Flink SQL is a natural fit for Ossie specifically because it's a Calcite-based, largely ANSI-SQL-compliant surface - unlike Flink's DataStream or Table APIs, it can consume Ossie's SQL expressions almost directly. I'd propose scoping this to Flink SQL only (vendor_name FLINK_SQL, not just FLINK) to keep that distinction explicit, since the other Flink APIs aren't SQL-expression-based and would need a different kind of converter entirely.
I have a working prototype that converts a semantic model to Flink SQL DDL for Kafka, Pulsar, Postgres CDC, MySQL CDC, and filesystem sources, using a FLINK_SQL custom_extensions block to carry the connector/watermark info core-spec doesn't (and shouldn't) model. Happy to share it if useful for this discussion.
Main question for the group: is anyone already working on this, and does targeting Flink SQL specifically (as opposed to Flink more broadly) seem like the right scope for a first version?
Thanks, Alex Campos
All reactions