Kafka

[reanguiano]

Please describe the project
Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

URL for the project
https://github.com/apache/kafka

Describe current CI/CD setup
The Kafka project is migrating from using a Jenkins setup to using GitHub runners but is not currently using GitHub runners.

Describe the primary use case for the GitHub Action Runner
It would be nice if a GitHub runner was available for the s390x and ppc64le architectures. This would allow us to compile the code and run some basic tests for those architectures.

Paste a link to the existing actions workflow file(s) or directory
I don't believe there is a current actions workflow file or directory.

How often do you plan on executing the runner?
I've asked the project for this information but I believe the project previously had a daily build job for these architectures.

What is the primary programming language for the project?
Java

Please select desired hardware

• Power 9 (ppc64le)
• IBM Z / LinuxONE (s390x)

Account names of the GitHub repo admins that will need access to setting up the runner

reanguiano will be setting up the runner, but these are my ASF Infra teammates that should also have the same level of access:

• gmcdonald
• Humbedooh
• cwlls
• bobthomson70
• clambertus
• dfoulks1

[pleia2]

Hi @reanguiano Thanks for filling this out!

The existing workflow directory would be this: https://github.com/apache/kafka/tree/trunk/.github

And we need to know which jobs you're planning on running on s390x/ppc64le to determine the capacity requirements for the project. You mentioned having a daily build, if that's all there is, I don't anticipate a problem. But there are also things like regular tests on PR/merge that many projects run, as well as release builds (which sometimes trigger multiple jobs at once during peak times. Whatever insight you can provide into how long the jobs run and confirm their frequency would be a huge help.

[reanguiano]

Hi @pleia2 Ah, thanks for the workflow location tip.

The team working on ppc64le said:

"On our specific case would be nightly and on-demand, when we are working on possible related issues. We could target the job to be quite specific so it only runs when strictly specified on specific PRs/commits. But at least once a day."

I am waiting to hear back if the same applies to the s390x folks.

I'll try to get job run length info from the teams.

Please let me know if I missed anything else you still need to proceed with your analysis.

[reanguiano]

Hi @pleia2

The team responded with two points:

1. The same frequency applies to s390x builds.
2. They expect a job run would takes from 20 minutes to 1 hour, depending on the build flags in use and what components are built and the capacity of the build host. Without an initial runner and some validation it is difficult to estimate how long it will take.

Thanks.

[reanguiano]

Good morning @pleia2, we've had some internal confusion and I have mixed up the CI/CD answers from the Arrow team with the Kafka team. I'll resubmit the template above once I have it sorted and open a new issue for the Arrow team.