Separate Kafka Workers from API Infrastructure

[SB2318]

Description

Separate background Kafka consumers/workers from the main API infrastructure so background processing does not compete directly with API workloads.

Tasks

• Identify existing Kafka consumers.
• Separate worker processes from the API service.
• Configure dedicated worker infrastructure.
• Configure appropriate consumer groups.
• Monitor Kafka consumer lag.
• Test worker failure and recovery.

Expected Benefits

• More consistent API performance
• Independent worker scaling
• Better resource isolation
• Improved background-processing reliability

[github-actions]

Thank you for creating this issue! We'll look into it as soon as possible. Your contributions are highly appreciated! 😊

---

🤖 Assignment Guidelines

To get assigned to this issue, simply leave a comment with or .
The AI bot will automatically assign you if you meet the following eligibility criteria:

1. You do not currently have any other active assigned issues in this repository.
2. If you were previously assigned an issue, you must have submitted a Pull Request for it before requesting a new one.

[github-actions]

This issue has been marked as out of scope for the current contribution program.

Reasoning: The issue is well-defined and provides sufficient details. However, it falls under 'Infrastructure migration' due to separating Kafka workers from the main API infrastructure, which is explicitly out of scope for general issues. It is classified as a backend task and is an advanced-level architectural change, requiring maintainer escalation.