Introduction

Source Connect provides the ETL pipeline from file source to the kafka with Exactly Once Semantics.

PreRequisites

Quick Start in local

(1) Launch kafka and kafka-ui

# Execute kafka brokers and kafka-ui in your host machine
$ docker-compose up -d

Check the kafka-ui at http://localhost:9090

(2) Configure the application.yml

Edit the application.yml file located in source-connector/src/main/resources/application.yml as below:

source:
  storage:
    type: local
    paths:
      - "file:///path/to/your/directory"

    configs:
      recursive: true
      filters:

This configuration let source connector to read files from the specified local directory.
And produce it to the kafka topic

(3) Execute the Source Connector

Set the JOB_INDEX environment variable for specifying single worker index

$ JOB_INDEX=0 ./gradlew :source-connector:bootRun

(4) Verify the produced messages in kafka-ui

Go to the kafka-ui at http://localhost:9090
Select the topic named sink-topic and check the messages produced from the source connector.

(Optional) Execute Offset Manager

$ ./gradlew :offset-manager:bootRun

You should edit the offsetManager property in application.yml in source-connector module as below:

offsetManager:
  type: http
  baseUrl: http://localhost:8080

Documentation

Design notes and usage information can be found in the wiki