Environment
Sentry 25.x, 26.x, Snuba 26.1/26.2
Self-hosted Kubernete (deplyed using helm chart 28.x and 29.x)
Related discussion: getsentry/sentry#101581 (comment)
Steps to Reproduce
Deploy self-hosted Sentry
Create a metric alert rule (e.g. Number of errors > 10 in 5 minutes)
Send enough events to breach the threshold
Expected Result
Metric alert fires when the threshold is breached.
Actual Result
Alert never fires. After investigation i figured out that this could be happening due to a bug in snuba. errors.yaml sets subscription_synchronization_timestamp: received_p99 (https://github.com/getsentry/snuba/blob/master/snuba/datasets/configuration/events/storages/errors.yaml#L422), however here it hardcodes sentry_received_timestamp=None (https://github.com/getsentry/snuba/blob/master/snuba/datasets/processors/rust_compat_processor.py#L46), Since received_p99 is computed as the p99 of sentry_received_timestamp, and it is always None, no timestamps are ever collected and received_p99 falls back to None on every commit-log message. I believe this is causing the subscription scheduler to silently drop all ticks, so alerts never fire.
from running pods:
$ grep -rn "received_p99\|sentry_received_timestamp" /usr/src/snuba/snuba/ | grep -v ".pyc"
snuba/datasets/processors/rust_compat_processor.py:50: sentry_received_timestamp=None,
snuba/consumers/consumer.py:317: received_p99 = None
snuba/datasets/configuration/events/storages/errors.yaml: subscription_synchronization_timestamp: received_p99
snuba/datasets/configuration/transactions/storages/transactions.yaml:266: subscription_synchronization_timestamp: received_p99
Workaround
Patch errors.yaml in a custom snuba image to use orig_message_ts instead (orig_message_ts is a valid synchronization timestamp per snuba docs: https://getsentry.github.io/snuba/configuration/writable_storage.html):
sed -i 's/subscription_synchronization_timestamp: received_p99/subscription_synchronization_timestamp: orig_message_ts/g' \
/usr/src/snuba/snuba/datasets/configuration/events/storages/errors.yaml
Alerts fire correctly with this workaround. Note that transactions.yaml has the same setting so transaction-based metric alerts are likely affected too.
Should RustCompatProcessor populate sentry_received_timestamp correctly instead of hardcoding None? Thanks! 🙏🏻
Environment
Sentry 25.x, 26.x, Snuba 26.1/26.2
Self-hosted Kubernete (deplyed using helm chart 28.x and 29.x)
Related discussion: getsentry/sentry#101581 (comment)
Steps to Reproduce
Deploy self-hosted Sentry
Create a metric alert rule (e.g. Number of errors > 10 in 5 minutes)
Send enough events to breach the threshold
Expected Result
Metric alert fires when the threshold is breached.
Actual Result
Alert never fires. After investigation i figured out that this could be happening due to a bug in snuba.
errors.yamlsetssubscription_synchronization_timestamp: received_p99(https://github.com/getsentry/snuba/blob/master/snuba/datasets/configuration/events/storages/errors.yaml#L422), however here it hardcodessentry_received_timestamp=None(https://github.com/getsentry/snuba/blob/master/snuba/datasets/processors/rust_compat_processor.py#L46), Since received_p99 is computed as the p99 of sentry_received_timestamp, and it is always None, no timestamps are ever collected and received_p99 falls back to None on every commit-log message. I believe this is causing the subscription scheduler to silently drop all ticks, so alerts never fire.from running pods:
Workaround
Patch errors.yaml in a custom snuba image to use orig_message_ts instead (orig_message_ts is a valid synchronization timestamp per snuba docs: https://getsentry.github.io/snuba/configuration/writable_storage.html):
Alerts fire correctly with this workaround. Note that
transactions.yamlhas the same setting so transaction-based metric alerts are likely affected too.Should RustCompatProcessor populate sentry_received_timestamp correctly instead of hardcoding None? Thanks! 🙏🏻