[wip][feat][io] Debezium DB2 source connector for Pulsar

[devinbost]

Adds new Debezium DB2 source connector to pulsar-io and includes integration tests.
Fixes #7837

Motivation:
Getting data from IBM DB2 into Pulsar

• doc
• doc-required
• doc-not-needed
• doc-complete

Docs at apache/pulsar-site#494

Does this pull request potentially affect one of the following parts:

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment
• Build process

The biggest change was that we needed to include building a Docker image in the build process for testing the DB2 connector since Debezium hasn't published an actual image with the necessary files. I verified that the DB2 license permits use of these drivers for testing purposes. More details in comments below (#19821 (comment) and #19821 (comment)).

The new Dockerfile was added to tests/docker-images/debezium-db2-test-image and was scoped as part of the integrationTests profile.

Tested against DB2 v11.5

Matching PR in forked repository
devinbost#15

[github-actions]

@devinbost Please add the following content to your PR description and select a checkbox:

- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

[nicoloboschi]

LGTM, just one comment

@dlg99 PTAL

[dlg99]

Great job!
LGTM with couple of nits + needs proper PR description and title

[dlg99]

Please follow up with docs update https://github.com/apache/pulsar-site/blob/main/docs/io-debezium-source.md

[Anonymitaet]

Hi @devinbost, I see you've contributed docs along with code changes, thank you!

A gentle reminder: have you submitted a PIP for this change? Can you attach the PIP vote link in the PR description?

[devinbost]

A gentle reminder: have you submitted a PIP for this change? Can you attach the PIP vote link in the PR description?

Hi @Anonymitaet , I don't think a PIP is typically required for new connectors.

[eolivelli]

I don't think that a PIP is required for a new Connector.

PIP are important for new APIs and for API changes/breaking changes.

Add a new Connector has very little impact, especially for this kind of connectors (based on Debezium)

[eolivelli]

I am generally +1 on this work, but I left a comment about the tests

[github-actions]

The pr had no activity for 30 days, mark with Stale label.

[tisonkun]

cc @devinbost @eolivelli do we still make progress on this PR?

[devinbost]

It's very close to being finished but I just haven't had time to finish it. There's just a test that needs to pass. IIRC there was a mismatch in the number of records the connector was emitting and the number expected due to something being emitted twice.

…

-- Devin Bost Sent from mobile

On Mon, Jun 12, 2023, 7:57 PM tison ***@***.***> wrote: cc @devinbost <https://github.com/devinbost> @eolivelli <https://github.com/eolivelli> do we still make progress on this PR? — Reply to this email directly, view it on GitHub <#19821 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABYTBL3MUEGT24YKV3RYSODXK63H5ANCNFSM6AAAAAAV3VRUT4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[github-actions]

The pr had no activity for 30 days, mark with Stale label.

[dave2wave]

@devinbost @eolivelli It looks like this is close, but now there are CI and TestContainer updates needed.

[github-actions]

The pr had no activity for 30 days, mark with Stale label.