-
Notifications
You must be signed in to change notification settings - Fork 3.8k
[wip][feat][io] Debezium DB2 source connector for Pulsar #19821
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
devinbost
wants to merge
45
commits into
apache:master
Choose a base branch
from
devinbost:db2
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
45 commits
Select commit
Hold shift + click to select a range
061b4df
Added Debezium DB2 connector for Pulsar
devinbost 9972036
Initial commit for DB2 Debezium connector testing
devinbost 4ea5216
Updated db2 source config
devinbost f38a5c9
Removed CI file
devinbost 5f281bb
Updating docs pom
devinbost d3c5bbd
Added missing hooks to execute DB2 tests and updated some test code.
devinbost 0fa7509
Updated source config for doc purposes
devinbost 92dc034
Update pulsar-io/debezium/db2/src/main/java/org/apache/pulsar/io/debe…
devinbost 3e97cc1
Separated db2 connection with subsequent commands
devinbost 817d7dc
Added more startup commands
devinbost 58a4ca0
Updated image for Debezium test container to use the new DB2 image
devinbost c482f3b
Fixed image reference
devinbost e6b5599
Fix license header
devinbost 262d7c6
Fixed format of license header
devinbost d29db07
Fixed license format by running mvn initialize license:format
devinbost e493cf8
Added more license details and updated CI to build docker image for DB2
devinbost fa9188e
Added DB2 image to Pulsar IO step in CI
devinbost 4f05272
Added check to ensure that commands aren't run on DB2 until DB finish…
devinbost b04c23e
Added wait checks to ensure DB2 has initialized before connecting
devinbost 5ab2163
Added more logging to troubleshoot container startup
devinbost c6812ef
Added more logging to troubleshoot container startup if there are errors
devinbost 77557d1
Add more checks since something is timing out in the test
devinbost dd7515e
Increasing timeout since DB2 takes a while to start
devinbost bf0d3e1
Increasing timeout since DB2 takes a while to start.
devinbost 414738c
Increasing wait time between subsequent commands
devinbost a9673de
Catching exception and printing more debug info
devinbost 49c342f
Refactored DB2 testing to use JDBC driver
devinbost 4cdd64b
Added JDBC logging for DB2
devinbost 5ffe1e4
Added test wait strategy based on testcontainers DB2 container
devinbost ab96ad1
fixed typo
devinbost 189ef39
Fixed issue with DB2 calls in test
devinbost e400261
Removed CDC logic from test
devinbost 2b2de0d
Added step to setup CDC
devinbost 970fd53
Refactored cdcsetup.sh execution to be synchronous to prevent race on DB
devinbost f7bcb0d
Fixed style checks
devinbost 992aa52
Fixed case sensitive values
devinbost 1544db3
Fixed case
devinbost 4ce089a
Added more properties to DB2 source to try to fix connector restarts
devinbost faf2eab
Removing additional insert to avoid duplicate write from failing test
devinbost 16170a8
Incremented number of expected messages.
devinbost 972c349
Added db2 bind command
devinbost b437fd5
Fixed bind command
devinbost 2dbd84a
Added more logging
devinbost ce6e395
Added retention
devinbost 45f2fde
Fixed retention limit
devinbost File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| <!-- | ||
|
|
||
| Licensed to the Apache Software Foundation (ASF) under one | ||
| or more contributor license agreements. See the NOTICE file | ||
| distributed with this work for additional information | ||
| regarding copyright ownership. The ASF licenses this file | ||
| to you under the Apache License, Version 2.0 (the | ||
| "License"); you may not use this file except in compliance | ||
| with the License. You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, | ||
| software distributed under the License is distributed on an | ||
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations | ||
| under the License. | ||
|
|
||
| --> | ||
| <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
| xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
| <modelVersion>4.0.0</modelVersion> | ||
| <parent> | ||
| <groupId>org.apache.pulsar</groupId> | ||
| <artifactId>pulsar-io-debezium</artifactId> | ||
| <version>3.0.0-SNAPSHOT</version> | ||
| </parent> | ||
|
|
||
| <artifactId>pulsar-io-debezium-db2</artifactId> | ||
| <name>Pulsar IO :: Debezium :: DB2</name> | ||
|
|
||
| <dependencies> | ||
|
|
||
| <dependency> | ||
| <groupId>${project.groupId}</groupId> | ||
| <artifactId>pulsar-io-core</artifactId> | ||
| <version>${project.version}</version> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>${project.groupId}</groupId> | ||
| <artifactId>pulsar-io-debezium-core</artifactId> | ||
| <version>${project.version}</version> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>io.debezium</groupId> | ||
| <artifactId>debezium-connector-db2</artifactId> | ||
| <version>${debezium.version}</version> | ||
| </dependency> | ||
|
|
||
| </dependencies> | ||
|
|
||
| <build> | ||
| <plugins> | ||
| <plugin> | ||
| <groupId>org.apache.nifi</groupId> | ||
| <artifactId>nifi-nar-maven-plugin</artifactId> | ||
| </plugin> | ||
| </plugins> | ||
| </build> | ||
|
|
||
| </project> |
36 changes: 36 additions & 0 deletions
36
...ar-io/debezium/db2/src/main/java/org/apache/pulsar/io/debezium/db2/DebeziumDB2Source.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, | ||
| * software distributed under the License is distributed on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| * KIND, either express or implied. See the License for the | ||
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
| package org.apache.pulsar.io.debezium.db2; | ||
|
|
||
| import java.util.Map; | ||
| import org.apache.kafka.connect.runtime.TaskConfig; | ||
| import org.apache.pulsar.io.debezium.DebeziumSource; | ||
|
|
||
|
|
||
| /** | ||
| * A pulsar source that runs debezium oracle source. | ||
| */ | ||
| public class DebeziumDB2Source extends DebeziumSource { | ||
| private static final String DEFAULT_TASK = "io.debezium.connector.db2.Db2ConnectorTask"; | ||
|
|
||
| @Override | ||
| public void setDbConnectorTask(Map<String, Object> config) throws Exception { | ||
| throwExceptionIfConfigNotMatch(config, TaskConfig.TASK_CLASS_CONFIG, DEFAULT_TASK); | ||
| } | ||
| } |
19 changes: 19 additions & 0 deletions
19
pulsar-io/debezium/db2/src/main/java/org/apache/pulsar/io/debezium/db2/package-info.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, | ||
| * software distributed under the License is distributed on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| * KIND, either express or implied. See the License for the | ||
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
| package org.apache.pulsar.io.debezium.db2; |
22 changes: 22 additions & 0 deletions
22
pulsar-io/debezium/db2/src/main/resources/META-INF/services/pulsar-io.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| # | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, | ||
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
| # | ||
|
|
||
| name: debezium-db2 | ||
| description: Debezium DB2 Source | ||
| sourceClass: org.apache.pulsar.io.debezium.db2.DebeziumDB2Source |
59 changes: 59 additions & 0 deletions
59
pulsar-io/debezium/db2/src/main/resources/debezium-db2-source-config.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| # | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, | ||
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
| # | ||
|
|
||
| ## config for db2, docker image: ibmcom/db2, which was moved to icr.io/db2_community/db2 | ||
| ## As per note at https://hub.docker.com/r/ibmcom/db2 | ||
| ## docs are being moved to: https://www.ibm.com/docs/en/db2/11.5?topic=deployments-db2-community-edition-docker | ||
| ## For testing, you can use this image that was configured with the required libraries: apachepulsar/debezium-db2-test-image:latest | ||
| ## docker run -itd --name mydb2 --privileged=true -p 50000:50000 -e LICENSE=accept -e DB2INST1_PASSWORD=admin -e DBNAME=testdb -v <db storage dir>:/database ibmcom/db2 | ||
|
|
||
| tenant: "public" | ||
| namespace: "default" | ||
| name: "debezium-db2-source" | ||
| inputs: [ "db2-connect-topic" ] | ||
| topicName: "db2-connect-topic" | ||
| archive: "connectors/pulsar-io-debezium-db2-2.11.0-SNAPSHOT.nar" | ||
| parallelism: 1 | ||
| configs: | ||
| database.hostname: "localhost" | ||
| database.port: "50000" | ||
| database.user: "db2inst1" | ||
| database.password: "admin" | ||
| database.dbname: "mydb2" | ||
| database.server.name: "db2inst1" | ||
| topic.prefix: "stores" # Example for hypothetical stores topics | ||
| table.exclude.list: "DB2INST1.EXCLUDEDSTORES" # Example for hypothetical stores table | ||
|
|
||
| database.history: "org.apache.pulsar.io.debezium.PulsarDatabaseHistory" | ||
| database.history.pulsar.topic: "debezium-db2-source-history-topic" | ||
| database.history.pulsar.service.url: "pulsar://127.0.0.1:6650" | ||
| ## PULSAR_SERVICE_URL_CONFIG | ||
| pulsar.service.url: "pulsar://127.0.0.1:6650" | ||
| topic.namespace: "public/default" | ||
| # CONVERTERS | ||
| key.converter: "org.apache.kafka.connect.json.JsonConverter" | ||
| value.converter: "org.apache.kafka.connect.json.JsonConverter" | ||
| typeClassName: "org.apache.pulsar.common.schema.KeyValue" | ||
|
|
||
| task.class: "io.debezium.connector.db2.Db2ConnectorTask" | ||
|
|
||
| offset.storage.topic: "offset-topic" | ||
| snapshot.mode: "initial" | ||
| database.tcpKeepAlive: "true" | ||
| decimal.handling.mode: "double" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.