INDE-2025

This project is a Scala POC application that demonstrates how to consume data from a Kafka topic, process it, and save it to MinIO as JSON files for further analysis.

The data generator simulates detector data with a temperature field in the raw-detector-data. Both the DataConsumer and AlertConsumer consume this data from the Kafka topic.

• The DataConsumer processes the data and saves it to MinIO
• The AlertConsumer checks if the temperature exceeds a threshold and logs an alert message to the console and saves it to a Postgres database.

Finally, the BronzeToGold object is used to transform the data from the detector-data bucket to the detector-gold bucket in MinIO.

Setup

Java version

You need to have java 17.0.15-albba:

sdk install java 17.0.15-albba

Then, switch to the correct version:

sdk default java 17.0.15-albba

sbt

sbt download

Run the project

Docker

Just run the following command to start the Kafka and MinIO services:

docker compose up

MinIO

On the browser, go to http://localhost:9000 and login with

• Access Key: minioadmin
• Secret Key: minioadmin

Make a bucket called detector-data and detector-gold.

You can also use the MinIO CLI inside of the container:

mc alias set local http://localhost:9000/ minioadmin minioadmin
mc mb local/detector-data
mc mb local/detector-gold

Final steps

Once you have the Docker services running, you can run the Scala project.

Run the data generator:

sbt "runMain pipeline.Generator"

Run the alert AlertConsumer:

sbt "runMain pipeline.AlertConsumer"

Then run the DataConsumer:

sbt "runMain pipeline.DataConsumer"

Monitoring & ETL

Now you have a Kafka consumer that listens to the raw-detector-data topic, processes the data, and saves it to MinIO as JSON files in the detector-data bucket.

If we exceed the temperature threshold, it will log an alert message to the console, and save the logging to the Postgres database.

You can execute the ETL pipeline with the BronzeToGold object:

sbt "runMain etl.BronzeToGold"

Health check

You can run the commands:

# Check the Kafka topics
docker exec -it kafka ./opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

# Check the consumers
docker exec -it kafka ./opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic raw-detector-data --from-beginning