Repository navigation
Add Spark 4.x / Scala 2.13 support (#22) - #24
Merged
dolfinus merged 11 commits intoSep 24, 2026
Merged
Conversation
Cross-build the dialect against two Spark / Scala lines, selected with `-PsparkProfile` (default `spark3`, unchanged behaviour): spark3 -> Scala 2.12 / Spark 3.5.x / Java 8-11 -> spark-dialect-extension_2.12 spark4 -> Scala 2.13 / Spark 4.x / Java 17 -> spark-dialect-extension_2.13 Build layout (per maintainer guidance in MTSWebServices#22): - Split the single module into Gradle sub-projects: version-agnostic `common` (runtime registry + shared test fixtures), and one implementation module per Spark line (`spark35`, `spark41`). Only the profile-compatible module is included, and each publishes a single fat jar bundling `common` + its dialect. The registry selects the implementation at runtime from the Spark version, so one Scala 2.13 artifact serves every Spark 4.x minor release. - Tests are split per Spark version (spark35/spark41) instead of using in-test conditionals; shared infrastructure lives in `common` test fixtures. Spark 4 specifics: - Override `JdbcDialect.isObjectNotFoundException` to detect ClickHouse error code 60 (UNKNOWN_TABLE) via the JDBC `SQLException` vendor code (locale independent), so writes to a not-yet-existing table work. - Add the standard `--add-opens` JVM args to the test task on Java 9+. - Version-specific array-read expectations: several `Array(Int16/Int32/Int64/ Float32/Float64)` now read successfully with driver 0.7.x; the rest surface as `COLUMN_ARRAY_ELEMENT_TYPE_MISMATCH`. - Scalafix is limited to the spark3 profile (no semanticdb-scalac for the Scala 2.13 patch Spark 4 pulls in); Scalafmt runs on both. Extract `ClickhouseDialectRegistry.dialectClassName` so the version dispatch is unit-testable without an active Spark session (Spark 4's `getActiveSession` filters out non-usable/mocked sessions). CI test and release workflows run both profiles with the matching Java version.
dolfinus
requested review from
dolfinus and
matveeysv
and removed request for
matveeysv
August 24, 2026 09:41
dolfinus
reviewed
Aug 24, 2026
Co-authored-by: Maxim Martynov <martinov_m_s_@mail.ru>
Co-authored-by: Maxim Martynov <martinov_m_s_@mail.ru>
Co-authored-by: Maxim Martynov <martinov_m_s_@mail.ru>
Co-authored-by: Maxim Martynov <martinov_m_s_@mail.ru>
Co-authored-by: Maxim Martynov <martinov_m_s_@mail.ru>
Co-authored-by: Maxim Martynov <martinov_m_s_@mail.ru>
Co-authored-by: Maxim Martynov <martinov_m_s_@mail.ru>
Co-authored-by: Maxim Martynov <martinov_m_s_@mail.ru>
matveeysv
reviewed
Aug 24, 2026
dolfinus
reviewed
Aug 26, 2026
…bServices#22 review) Address review feedback: avoid copy-pasting the dialect implementation between Spark versions. All type-mapping logic now lives in a single `ClickhouseDialectBase` in the version-agnostic `common` module; the per-version modules are thin subclasses: - spark35: `object ClickhouseDialectExtension extends ClickhouseDialectBase` - spark41: same, plus the Spark-4-only `isObjectNotFoundException` override. The implementation modules now compileOnly-depend on `common` (the fat jar already bundles the `common` classes, so this stays out of the published POM). The spark41 dialect keeps being compiled against Spark 4.1.x (where `isObjectNotFoundException` was introduced) but the artifact also runs on Spark 4.0.x: there `JdbcUtils.tableExists` uses the older `Try { probe }.isSuccess` logic and never calls the method, so it is inert and the older behaviour is unaffected.
matveeysv
approved these changes
Sep 24, 2026
dolfinus
approved these changes
Sep 24, 2026
Member
|
@CMBCKSRL that is awesome, thank you! |
Coverage
Files
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cross-build the dialect against two Spark / Scala lines, selected with
-PsparkProfile(defaultspark3, unchanged behaviour):spark3 -> Scala 2.12 / Spark 3.5.x / Java 8-11 -> spark-dialect-extension_2.12
spark4 -> Scala 2.13 / Spark 4.x / Java 17 -> spark-dialect-extension_2.13
Build layout (per maintainer guidance in #22):
common(runtime registry + shared test fixtures), and one implementation module per Spark line (spark35,spark41). Only the profile-compatible module is included, and each publishes a single fat jar bundlingcommon+ its dialect. The registry selects the implementation at runtime from the Spark version, so one Scala 2.13 artifact serves every Spark 4.x minor release.commontest fixtures.Spark 4 specifics:
JdbcDialect.isObjectNotFoundExceptionto detect ClickHouse error code 60 (UNKNOWN_TABLE) via the JDBCSQLExceptionvendor code (locale independent), so writes to a not-yet-existing table work.--add-opensJVM args to the test task on Java 9+.Array(Int16/Int32/Int64/ Float32/Float64)now read successfully with driver 0.7.x; the rest surface asCOLUMN_ARRAY_ELEMENT_TYPE_MISMATCH.Extract
ClickhouseDialectRegistry.dialectClassNameso the version dispatch is unit-testable without an active Spark session (Spark 4'sgetActiveSessionfilters out non-usable/mocked sessions).CI test and release workflows run both profiles with the matching Java version.
Change Summary
Related issue number
Checklist