bench: deliver soak alarms and reaper notices to Slack via AWS Chatbot - #4733
Merged
Conversation
terraform/persistent/slack.tf binds a Chatbot Slack channel configuration to both SNS topics (soak alarms + orphan reaper), gated on TF_VAR_slack_workspace_id / TF_VAR_slack_channel_id so the stack applies before the one-time console OAuth is done. The channel's guardrail and role are CloudWatchReadOnlyAccess — enough to render alarm cards, no mutation surface. Email becomes the optional backup subscriber (empty default), with a cross-variable validation ensuring at least one channel is always configured — an apply with neither fails loudly instead of leaving alarms silently unrouted. Chatbot silently drops plain SNS text, so the reaper now publishes an SNS per-protocol envelope: email subscribers keep the human-readable summary, everything else gets Chatbot's custom-notification schema. Pinned by TestSweep_PublishUsesPerProtocolEnvelope. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
The descriptions render verbatim in the Slack alarm card, where bare issue numbers (#4648) and class jargon meant nothing to a responder. Each now leads with what happened in plain language, gives the first diagnostic step, points at the runbook, and links past examples as full URLs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review fixes for the Slack alerting PR: - The Slack workspace/channel IDs become committed variable defaults (they're not secrets — both appear in every Slack URL), mirroring backend.hcl's account-specific values. Count-gating on ambient TF_VAR_* meant any re-apply whose shell didn't re-export them silently DESTROYED the channel configuration — and SOAK.md's own add-a-connector step documented exactly that invocation. Passing both as "" remains the explicit disable, still guarded by the at-least-one-channel validation. - perProtocolMessage's defensive fallback returned raw text that the caller then published with MessageStructure=json — which SNS rejects, losing the notice entirely. It now returns an error and the caller publishes plain text WITHOUT the structure flag (email-only delivery beats a rejected publish). - SOAK.md's add-a-connector step and one-time-setup bullet (plus the nightly workflow header and the Alerts bullet) now describe the committed-defaults world; the stale 'deliberately undefaulted' email claim is gone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two review findings shared a root cause: the email subscription was count-gated on an ambient TF_VAR_* (self-destructing on any env-less re-apply, and needing a human confirmation click to resurrect), and it only covered the soak-alerts topic — the reaper topic had zero email subscribers, contradicting the never-silently-unrouted claim. A committed address default (the fix used for the Slack IDs) is not available here: no team alias exists and a personal mailbox doesn't belong in the repo. So email backups leave Terraform entirely: they are documented manual subscriptions to BOTH topics (SOAK.md), which no re-apply can unsubscribe. The pre-existing subscription is preserved as exactly that via a removed block (state forget, not destroy). Slack is now the ONLY Terraform-managed channel: its IDs are required by validation (blanking them would leave both topics unrouted), and the obsolete count gating comes off the three resources with moved blocks — verified against real state: 3 moves, 0 add/change/destroy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Six panels over the RedpandaConnect/Bench CloudWatch namespace with connector/scenario template dropdowns (dimension discovery — new connectors joining the rotation appear without dashboard changes), the three alarm thresholds drawn as lines, CloudWatch alarm-state annotations, and a RunActive overlay so absence-of-run reads differently from failure. Binds to any CloudWatch data source via a variable at import time; the bench account's data source is being requested from the Grafana admins separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dering Re-encode all panel targets with explicit queryMode: Metrics / metricQueryType: 0 / metricEditorMode: 0 / matchExact: false — the legacy encoding made Grafana fall back to the data source's default (non-us-east-2) region, rendering every panel as NO DATA. Default the time range to now-7d since soaks run nightly, and set insertNulls: 3600000 so separate runs render as disconnected segments instead of being joined by interpolated lines. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A rewrite left it dangling at the end of the Grafana bullet, reading as if the three channels were a property of the dashboard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The panel drew its red line at 2097152 while the rss-slope alarm fires at 2000000, so a leak between the two paged while rendering below the line. Match the alarm's decimal threshold and say MB in the description. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Leftover from when the subscription was Terraform-managed; the manual `aws sns subscribe` sentence above already covers confirmation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t above Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Leward
approved these changes
Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Observability delivery for the soak pipeline's two SNS topics and CloudWatch metrics:
terraform/persistent/slack.tf): channel configuration bound to both topics (soak alarms + orphan reaper), guardrailed toCloudWatchReadOnlyAccess. Workspace/channel IDs are committed defaults (not secrets; mirrors backend.hcl's account-specific values) and are required by validation — Slack is the only Terraform-managed alert channel.cleanup-lambda/sweep.go): Chatbot silently drops plain SNS text, so thedefaultprotocol carries Chatbot's custom-notification schema whileemailkeeps the human-readable summary. Pinned byTestSweep_PublishUsesPerProtocolEnvelope.removedblock (state forget, not destroy).#4648-style numbers.benchmarking/aws/grafana/soak-dashboard.json): connector/scenario template dropdowns via dimension discovery, alarm thresholds drawn in, CloudWatch alarm annotations, RunActive overlay. Binds to any CloudWatch data source at import time (bench-account data source requested from the Grafana admins separately).Validated live
Both message shapes confirmed rendering in #soak-redpanda-connect (alarm card with graph via a synthetic
set-alarm-state, reaper custom notification via a test publish). Real-state plan for the final Terraform: 3 pure state moves, 0 add/change/destroy.🤖 Generated with Claude Code