Monitoring queue length checks can throttle the Azure Service Bus namespace with no log evidence

[ramonsmits]

Description

The ServiceControl Monitoring instance's queue length monitoring routine periodically queries the Azure Service Bus namespace for queue runtime properties. On busy or large namespaces this generates enough management-plane requests to trigger Azure Service Bus request throttling. The throttling is visible on the Azure Service Bus namespace dashboard but produces no corresponding error or warning in the ServiceControl logs.

Expected behavior

Queue length monitoring should not drive the namespace into sustained throttling; and if throttling does occur, it should be observable in the ServiceControl logs so it can be diagnosed.

Actual behavior

• The Azure Service Bus namespace dashboard shows request throttling that begins when the Monitoring instance is started.
• Nothing is written to the ServiceControl Monitoring logs — no errors, warnings, or other indication.
• Stopping the Monitoring instance resolves the throttling (the dashboard shows it subside).

Versions

ServiceControl 6.18.2 (latest). All supported 6.x versions are affected.

Transport: Azure Service Bus (NetStandardAzureServiceBus).

Steps to reproduce

1. Configure a ServiceControl Monitoring instance on the Azure Service Bus transport, pointed at a namespace with many queues / active traffic.
2. Start the Monitoring instance.
3. Observe request throttling on the Azure Service Bus namespace metrics/dashboard.
4. Note that the Monitoring logs contain no related errors or warnings.
5. Stop the Monitoring instance and observe the throttling subside.

Relevant log output

(none — the absence of any log evidence is part of the bug)

Additional Information

Workarounds

• Increase QueueLengthQueryDelayInterval (milliseconds) in the Monitoring instance's transport connection string to reduce polling frequency.
• Isolate the Monitoring instance to a dedicated namespace.

Possible solutions

• Make the throttling observable in the logs.
• Detect throttling and back off the query cadence.

Additional information (root-cause analysis)

• The queue length provider calls GetQueuesRuntimeProperties for all queues in the namespace at a default interval of 500 ms. These are management-plane operations, which Azure Service Bus throttles independently of message traffic.
• The Azure SDK retries throttled (HTTP 429) management requests internally and usually succeeds, so the throttling never surfaces as an exception — which is why the dashboard counts throttled requests while the logs stay silent.