Fix CI failure: RabbitMQ 4.x incompatible with Celery's transient_nonexcl_queues

[CasperWA-ai-bot]

Problem / Goal

The test_celery_remote test fails on all Linux CI runners because pytest-celery uses rabbitmq:latest as its default Docker image for the broker, and RabbitMQ 4.x has removed support for the transient_nonexcl_queues feature that Celery/kombu relies on.
This blocks every PR until the issue is resolved, including auto-update PRs such as #677.

Context

• Failing job: https://github.com/EMMC-ASBL/oteapi-core/actions/runs/25013266104/job/73254343796
• Failing test: tests/strategies/transformation/test_celery_remote.py::test_celery_remote
• pytest-celery vendor default for RabbitMQ image: rabbitmq:latest (see pytest_celery/vendors/rabbitmq/defaults.py)
• The default_rabbitmq_broker_image fixture in pytest-celery is designed to be overridden locally

Reproduction

The failure occurs automatically on all Linux CI runs. The exact error from the Celery worker container is:

amqp.exceptions.InternalError: Queue.declare: (541) INTERNAL_ERROR -
Feature `transient_nonexcl_queues` is deprecated.
By default, this feature is not permitted anymore.

Celery's mingle boot-step calls Queue.declare on a transient, non-exclusive queue (the celery.pidbox broadcast mailbox) during worker startup.
RabbitMQ 4.x rejects this call with a hard 541 error, crashing the worker before it can become ready.
pytest-docker-tools then times out after 30 s waiting for the worker container, surfacing as ContainerNotReady.

Considerations

Three approaches were considered:

1. Pin the RabbitMQ Docker image to rabbitmq:3 (chosen)

Override the default_rabbitmq_broker_image pytest fixture in tests/strategies/transformation/conftest.py to return "rabbitmq:3" instead of "rabbitmq:latest".
This fixture is explicitly provided by pytest-celery as an override point.

• Pro: Minimal, targeted change. No behaviour difference in the tests. Compatible with current Celery/kombu.
• Con: Needs updating again when Celery/kombu gain RabbitMQ 4.x support; rabbitmq:3 will eventually stop receiving security patches.

2. Disable Celery's mingle boot-step in tests

Configure the Celery test app with worker_disable_mingle=True (or CELERY_WORKER_DISABLE_MINGLE=True) to skip the mingle step that performs the problematic Queue.declare.

• Pro: Works with any RabbitMQ version.
• Con: Changes the worker configuration under test, potentially masking real connectivity issues; requires deeper integration with pytest-celery's worker fixture.

3. Wait for an upstream Celery/kombu fix

Track celery/celery#9659 and upgrade once a version of Celery or kombu is released that avoids transient_nonexcl_queues.

• Pro: Correct long-term resolution; no local workaround needed.
• Con: Blocks the CI today and has no fixed timeline.

Approach 1 is the most pragmatic fix and is straightforward to revert once the upstream issue is resolved.

[CasperWA]

Duplicate of #680. Closing — #680 is the canonical issue and is referenced in PR #681.