[improve][test] Add Canal source integration test by david-streamlio · Pull Request #98 · apache/pulsar-connectors

david-streamlio · 2026-07-10T18:22:36Z

Fixes #48

Motivation

The Canal source connector had no automated coverage at all. This adds the first integration test: a full MySQL-binlog CDC pipeline (MySQL → canal-server → CanalStringSource), verified green on a Linux x86-64 host.

Modifications

Add CanalStringSourceIntegrationTest: two Testcontainers on a shared network — mysql:8.0 with ROW binlog and canal/canal-server:v1.1.7 (matching the module's canal.client/canal.protocol 1.1.7). canal reaches MySQL via a network alias; the source reaches canal via the mapped 11111 port. An INSERT made after the source subscribes is read back on a bounded worker thread with a deadline, so an under-delivering pipeline fails fast instead of hanging CI.
Add the canal-server instance config (instance.properties, ASF-licensed) mounted into the container.
Add test deps: Testcontainers and mysql-connector-j with protobuf-java excluded — see below.

The protobuf story (replaces the earlier module-wide 3.6.1 pin)

An earlier revision of this PR force-pinned protobuf-java to 3.6.1 module-wide, flagged in #99 as needing a maintainer decision (2018 vintage, known CVEs). Root-caused now, the pin is unnecessary:

The protobuf 4.x on the test classpath came from the test-only mysql-connector-j dependency (it pulls protobuf 4.x for X DevAPI, unused by plain JDBC), outranking the platform's 3.25.5.
canal's generated CanalPacket code calls GeneratedMessageV3.makeExtensionsImmutable(), which protobuf 4.x removed but 3.25.5 still has — verified empirically: a standalone canal 1.1.x client against protobuf-java 3.25.5 completes the handshake and delivers CDC entries.
The NAR bundles no protobuf (by convention, com.google.protobuf is excluded from NARs); the Pulsar runtime provides the 3.25.x line, so production is not affected today.
Upgrading to canal 1.1.8 does not help: its CanalPacket$Packet gencode makes the same call (verified via javap).

So the fix is a one-line exclusion of protobuf-java from mysql-connector-j, which makes the test classpath resolve the platform's 3.25.5 — the same protobuf line production uses. Detailed analysis posted on #99.

Verifying this change

Run on Linux x86-64 (Ubuntu 24.04, Docker 29.1.3, OpenJDK 21):

$ ./gradlew :canal:test --tests "*CanalStringSourceIntegrationTest" --rerun-tasks

Gradle suite > Gradle test > org.apache.pulsar.io.canal.CanalStringSourceIntegrationTest > testCanalCdcEvents PASSED

BUILD SUCCESSFUL in 46s

Mutation check (test is not vacuous): with the payload assertion deliberately broken, the test fails showing the real CDC record it received — table products, inserted value canal-widget — then passes again after reverting:

java.lang.AssertionError: CDC message should carry the inserted row value, but was:
[{"data":[...,"columnName":"name","columnValue":"canal-widget",...],"database":"testdb",...,"table":"products","type":"INSERT"}]
expected [true] but found [false]

Documentation

doc-not-needed (test-only change)

Copilot

Pull request overview

Adds the first automated test coverage for the Canal source connector by introducing a full end-to-end CDC integration test (MySQL binlog → canal-server → CanalStringSource) using Testcontainers. It also adjusts the canal Gradle module to add needed test dependencies and attempts to address a protobuf incompatibility between Canal 1.1.7 and protobuf 4.x.

Changes:

Add CanalStringSourceIntegrationTest that stands up MySQL + canal-server via Testcontainers and asserts an INSERT produces a CDC event.
Add a canal-server instance configuration (instance.properties) used by the integration test container.
Update canal/build.gradle.kts to add Testcontainers + MySQL JDBC test deps and pin protobuf-java to 3.6.1 for Canal compatibility.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
canal/src/test/resources/canal/instance.properties	Adds canal-server instance config for the integration test pipeline.
canal/src/test/java/org/apache/pulsar/io/canal/CanalStringSourceIntegrationTest.java	New Testcontainers-based integration test verifying Canal CDC events flow into the source.
canal/build.gradle.kts	Adds test dependencies and pins protobuf to work around Canal/protobuf incompatibility.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+configurations.all {
+    resolutionStrategy {
+        force("com.google.protobuf:protobuf-java:3.6.1")
+    }
+}


david-streamlio · 2026-07-10T18:30:03Z

Two follow-ups on this PR.

1. Fixed the CI failure. Apache Rat flagged the new instance.properties as an unapproved license — it was missing the ASF header. Added it (rat passes locally now) and pushed.

2. The protobuf pin needs a maintainer decision — flagging rather than merging as-is. I dug into the force("protobuf-java:3.6.1") and filed the underlying incompatibility as #99. The key findings:

The canal client 1.1.7 genuinely can't talk to canal-server against modern protobuf-java — its generated code calls GeneratedMessageV3.makeExtensionsImmutable(), removed in protobuf 3.21+/4.x. So this is a real incompatibility, and the test correctly surfaces it.
But the fix here is configurations.all — module-wide, including the shipped NAR — pinning to protobuf 3.6.1 (2018, with known CVEs). The demonstrated breakage is on the test classpath (protobuf 4.31.1 pulled by pulsar-client). The module's runtimeClasspath resolves no external protobuf at all, so whether the NAR actually needs this pin depends on what protobuf the Pulsar runtime provides — likely modern, which would mean the connector is broken in production too, but that should be confirmed.
canal.client 1.1.8 exists. If it's protobuf-4-compatible, upgrading is cleaner than downgrading protobuf across the NAR.

My recommendation: don't merge the module-wide 3.6.1 pin without deciding between (a) canal 1.1.8 upgrade, (b) shading protobuf in the NAR, or (c) a test-scoped pin if production is unaffected. The test itself is sound and mutation-verified — it's only the bundled dependency workaround that warrants the call. Details and CVE references in #99.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

+// canal.protocol/canal.client 1.1.7 were generated against protobuf-java 3.6.1: their generated
+// CanalPacket code calls GeneratedMessageV3.makeExtensionsImmutable(), which was removed in
+// protobuf-java 4.x. The shared dependency platform force-upgrades protobuf-java to 4.31.1, which
+// makes the canal client throw NoSuchMethodError on connect() and leaves the connector unable to
+// talk to canal-server at all. Pin protobuf-java back to the version canal declares so the client
+// (and this module's NAR) works. See CanalStringSourceIntegrationTest.


+configurations.all {
+    resolutionStrategy {
+        force("com.google.protobuf:protobuf-java:3.6.1")
+    }
+}


Fixes apache#48 Stand up a full MySQL-binlog CDC pipeline with Testcontainers and assert that CanalStringSource delivers the change event. The test runs two containers on a shared network: mysql:8.0 with ROW binlog and canal/canal-server:v1.1.7 (matching the module's canal.client/canal.protocol 1.1.7). canal reaches MySQL via a network alias; the source reaches canal via the mapped 11111 port. An INSERT made after the source subscribes is read back on a bounded worker thread with a deadline, so a missing event fails fast instead of hanging CI. The test-only mysql-connector-j dependency excludes protobuf-java: it drags protobuf 4.x (for X DevAPI, unused by plain JDBC) onto the test classpath, where it outranks the platform's 3.25.5. canal's generated CanalPacket code calls GeneratedMessageV3.makeExtensionsImmutable(), which protobuf 4.x removed, so the canal client throws NoSuchMethodError on connect() under protobuf 4. With the exclusion the test runs against the same protobuf line the Pulsar runtime provides to the NAR (which still has the method), matching production. See apache#99 for the analysis.

david-streamlio · 2026-07-11T02:12:17Z

Pushed the resolution to the protobuf question flagged earlier (and analyzed in #99): the module-wide protobuf 3.6.1 force is gone. Root cause turned out to be the test-only mysql-connector-j dependency dragging protobuf 4.x onto the test classpath; production is unaffected because the NAR bundles no protobuf and the Pulsar runtime provides the 3.25.x line, which canal's gencode works with (verified empirically — details in the updated PR description and on #99). The branch is also rebased onto current master and squashed to a single commit.

Re-verified on Linux x86-64 after the rebase: testCanalCdcEvents PASSED, BUILD SUCCESSFUL in 47s, and the mutation check fails correctly when the payload assertion is broken.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

+        // Give the source thread time to connect + subscribe to canal before the mutation, so the
+        // INSERT lands in the stream canal delivers to this subscriber.
+        Thread.sleep(5000);
+
+        try (Connection conn = openMysql();
+             Statement stmt = conn.createStatement()) {
+            stmt.execute("INSERT INTO testdb.products (name, description) "
+                    + "VALUES ('canal-widget', 'inserted after subscription')");
+        }
+
+        Record<CanalMessage> record = readOne();
+        assertNotNull(record.getValue(), "CDC record value should not be null");
+        String message = record.getValue().getMessage();
+        log.info("Received CDC record: key={}, message={}", record.getKey().orElse(null), message);
+        assertNotNull(message, "CDC message payload should not be null");
+        // The FlatMessage JSON carries the table name and the inserted row values.
+        assertTrue(message.contains("products"),
+                "CDC message should reference the products table, but was: " + message);
+        assertTrue(message.contains("canal-widget"),
+                "CDC message should carry the inserted row value, but was: " + message);
+        record.ack();
+    }


david-streamlio requested a review from Copilot July 10, 2026 18:22

Copilot started reviewing on behalf of david-streamlio July 10, 2026 18:23 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread canal/build.gradle.kts Outdated

Comment on lines +30 to +34

configurations.all {

resolutionStrategy {

force("com.google.protobuf:protobuf-java:3.6.1")

}

}

david-streamlio requested a review from Copilot July 10, 2026 18:27

Copilot started reviewing on behalf of david-streamlio July 10, 2026 18:28 View session

david-streamlio mentioned this pull request Jul 10, 2026

[Bug] Canal source client is incompatible with modern protobuf-java (NoSuchMethodError on connect) #99

Open

Copilot AI reviewed Jul 10, 2026

View reviewed changes

david-streamlio requested a review from merlimat July 10, 2026 19:28

david-streamlio force-pushed the test/canal-integration branch from a0723f7 to de74ef1 Compare July 11, 2026 02:12

david-streamlio requested a review from Copilot July 11, 2026 02:12

Copilot started reviewing on behalf of david-streamlio July 11, 2026 02:12 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

david-streamlio mentioned this pull request Jul 11, 2026

[improve][io] Align the OpenMLDB module's forced protobuf with the platform version #106

Open

1 task

Navigation Menu

Uh oh!

[improve][test] Add Canal source integration test - #98

[improve][test] Add Canal source integration test#98
david-streamlio wants to merge 1 commit into
apache:masterfrom
david-streamlio:test/canal-integration

david-streamlio commented Jul 10, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

david-streamlio commented Jul 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

david-streamlio commented Jul 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

david-streamlio commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

The protobuf story (replaces the earlier module-wide 3.6.1 pin)

Verifying this change

Documentation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

david-streamlio commented Jul 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

david-streamlio commented Jul 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

david-streamlio commented Jul 10, 2026 •

edited

Loading