bench: deliver soak alarms and reaper notices to Slack via AWS Chatbot

.github/workflows/soak_nightly.yml

@@ -15,11 +15,13 @@ name: Nightly Soak
 # reaches the default branch.
 #
 # One-time setup (operator):
-#   1. TF_VAR_soak_alert_email=<team-alias> task aws:persistent
-#                                     (OIDC provider, provisioner role,
-#                                      archive bucket, dashboards, reaper;
-#                                      the alert email has no default —
-#                                      use a monitored team alias)
+#   1. task aws:persistent — creates the OIDC provider, provisioner role,
+#      archive bucket, dashboards, reaper, and Slack alert delivery (the
+#      workspace/channel IDs are committed defaults in
+#      terraform/persistent/slack.tf; a NEW account/workspace needs the
+#      one-time Chatbot console OAuth first; the IDs are required by
+#      validation). Optional email backups are manual SNS subscriptions
+#      to both topics — see SOAK.md.
 #   2. aws secretsmanager create-secret \
 #        --name redpanda-connect-bench/license \
 #        --secret-string file://rpcn.license --region us-east-2

benchmarking/aws/SOAK.md

@@ -30,8 +30,9 @@ files named below.
    `task aws:validate scenario=<engine>/<name>` must pass.
 3. **Register the dashboard + alarms**: add an entry to `soak_scenarios`
    in `terraform/persistent/variables.tf` (key → connector + scenario
-   name), then `TF_VAR_soak_alert_email=<team-alias> task aws:persistent`.
-   Alarms and the dashboard are generated per entry.
+   name), then `task aws:persistent`. Alarms and the dashboard are
+   generated per entry; Slack delivery is on by default via `slack.tf`'s
+   committed IDs — no extra vars needed.
 4. **First runs**: dispatch the nightly workflow manually with the
    scenario input. The baseline comparator stays advisory until three
    soak-index entries exist.
@@ -53,13 +54,17 @@ files named below.
   the workflows — no relevant merge, no run. Fails open: a missing entry or
   unknown SHA runs the soak.
 - **One-time account setup** (already done in 605419575229, needed again
-  only for a new account): `TF_VAR_soak_alert_email=<team-alias> task
-  aws:persistent` (builds the reaper's `bootstrap.zip` itself; the alert
-  email is deliberately undefaulted — point it at a monitored team
-  alias, never an individual); create the license secret: `aws secretsmanager
+  only for a new account): authorize the Slack workspace in the AWS
+  Chatbot console (OAuth — see `terraform/persistent/slack.tf`) and put
+  the resulting IDs in that file's defaults, then `task aws:persistent`
+  (builds the reaper's `bootstrap.zip` itself); optionally add a manual
+  email backup — `aws sns subscribe --protocol email
+  --notification-endpoint <team-alias> --topic-arn <arn>` for BOTH the
+  soak-alerts and orphans topics, then click each confirmation link
+  (deliberately outside Terraform: no re-apply can unsubscribe it);
+  create the license secret: `aws secretsmanager
   create-secret --name redpanda-connect-bench/license --secret-string
-  file://<license> --region us-east-2`; confirm the SNS email
-  subscription.
+  file://<license> --region us-east-2`.
 - **Laptop runs**: always from a git worktree (never a checkout you might
   branch-switch mid-run), always with credentials that outlive the run —
   `aws-vault exec` static creds die at ~1h; prefer the scheduled workflow.
@@ -72,11 +77,25 @@ files named below.
   sweeping every 15 min. A bench legitimately running past 4h needs the
   rule disabled first (and re-enabled after — set a reminder). The
   persistent stack itself is exempt via its distinct Project tag.
-- **Alerts** land at the `redpanda-connect-bench-soak-alerts` SNS topic
-  (email today; swap the subscription for Chatbot/Slack without touching
-  alarms). Alarm emails during a run are the acute channel; a red nightly
-  workflow is the between-builds channel; the `/soak` comment is the
-  before-merge channel.
+- **Alerts** land at the `redpanda-connect-bench-soak-alerts` SNS topic and
+  deliver to #soak-redpanda-connect via AWS Chatbot
+  (`terraform/persistent/slack.tf` — the workspace/channel IDs are
+  committed defaults, so plain `task aws:persistent` keeps Slack wired).
+  Alarm cards render natively; reaper notices arrive via the
+  custom-notification envelope in `cleanup-lambda/sweep.go` (plain SNS
+  text is silently dropped by Chatbot — keep that envelope). Email backup
+  is a manual SNS subscription to BOTH topics (see one-time setup above) —
+  deliberately unmanaged, so no re-apply can silently unsubscribe it.
+  Slack is the only Terraform-managed channel, so blanking its IDs fails
+  validation instead of leaving the topics unrouted. Alarms during a run
+  are the acute channel; a red nightly workflow is the between-builds
+  channel; the `/soak` comment is the before-merge channel.
+- **Grafana**: `grafana/soak-dashboard.json` is an importable dashboard
+  over the same CloudWatch metrics (template dropdowns for
+  connector/scenario, alarm thresholds drawn in, CloudWatch alarm
+  annotations). It needs a CloudWatch data source for account
+  605419575229 in the Grafana stack — read-only metrics access; the
+  dashboard binds to it via a data-source variable at import time.
 
 ## Known limitations
 

benchmarking/aws/cleanup-lambda/sweep.go

@@ -10,6 +10,7 @@ package main
 
 import (
 	"context"
+	"encoding/json"
 	"errors"
 	"fmt"
 	"log/slog"
@@ -30,6 +31,40 @@ import (
 	"github.com/aws/smithy-go"
 )
 
+// perProtocolMessage wraps the human-readable sweep summary in an SNS
+// per-protocol envelope: email subscribers get the plain text, while every
+// other protocol — which is how AWS Chatbot's Slack delivery subscribes —
+// gets Chatbot's custom-notification schema. Chatbot silently DROPS plain
+// SNS text (it only forwards formats it recognises), so without this the
+// Slack channel would look wired up while never showing a reaper notice.
+//
+// A non-nil error means the caller must publish msg as plain text WITHOUT
+// MessageStructure=json — SNS rejects a structure-json Publish whose body
+// isn't a JSON object with a "default" key, so returning raw text here
+// would lose the notice entirely instead of degrading to email-only.
+func perProtocolMessage(msg string) (string, error) {
+	chatbot, err := json.Marshal(map[string]any{
+		"version": "1.0",
+		"source":  "custom",
+		"content": map[string]string{
+			"textType":    "client-markdown",
+			"title":       "bench orphan-cleanup ran",
+			"description": msg,
+		},
+	})
+	if err != nil {
+		return "", err
+	}
+	envelope, err := json.Marshal(map[string]string{
+		"default": string(chatbot),
+		"email":   msg,
+	})
+	if err != nil {
+		return "", err
+	}
+	return string(envelope), nil
+}
+
 // isGone reports whether err is a not-found-style AWS error: the resource
 // vanished between the tag index (which lags real deletions by minutes to
 // hours) and our describe/delete call. Already-gone is a successful no-op,
@@ -641,11 +676,20 @@ func Sweep(ctx context.Context, api cleanupAPI, now time.Time, ttl time.Duration
 		if len(report.Failed) > 0 {
 			msg += "\n\nfailed (still accruing cost; retried next sweep):\n" + strings.Join(report.Failed, "\n")
 		}
-		if _, err := api.Publish(ctx, &sns.PublishInput{
+		in := &sns.PublishInput{
 			TopicArn: aws.String(snsTopicARN),
 			Subject:  aws.String("bench orphan-cleanup ran"),
-			Message:  aws.String(msg),
-		}); err != nil {
+		}
+		if envelope, err := perProtocolMessage(msg); err != nil {
+			// Unreachable for maps of strings, but if it ever fires,
+			// plain text (email-only delivery) beats a rejected publish.
+			slog.Error("per-protocol envelope failed; publishing plain text", "err", err)
+			in.Message = aws.String(msg)
+		} else {
+			in.Message = aws.String(envelope)
+			in.MessageStructure = aws.String("json")
+		}
+		if _, err := api.Publish(ctx, in); err != nil {
 			slog.Error("sns publish failed", "err", err)
 		}
 	}

benchmarking/aws/cleanup-lambda/sweep_test.go

@@ -10,6 +10,7 @@ package main
 
 import (
 	"context"
+	"encoding/json"
 	"strconv"
 	"testing"
 	"time"
@@ -74,6 +75,7 @@ type FakeAWS struct {
 	DeletedRDSSubnetGroups []string
 	DeletedRDSParamGroups  []string
 	SNSMessages            []string
+	SNSStructures          []string
 	// CallLog records "<Method>:<id>" for every mutating call, in call
 	// order, so tests can assert cross-resource dependency ordering.
 	CallLog []string
@@ -387,6 +389,7 @@ func (f *FakeAWS) DeleteRole(_ context.Context, in *iam.DeleteRoleInput) (*iam.D
 
 func (f *FakeAWS) Publish(_ context.Context, in *sns.PublishInput) (*sns.PublishOutput, error) {
 	f.SNSMessages = append(f.SNSMessages, aws.ToString(in.Message))
+	f.SNSStructures = append(f.SNSStructures, aws.ToString(in.MessageStructure))
 	return &sns.PublishOutput{}, nil
 }
 
@@ -809,6 +812,48 @@ func TestSweep_ErroredDeleteNotReportedDestroyed(t *testing.T) {
 	require.NotContains(t, api.SNSMessages[0], "destroyed:", "must not list the errored resource as destroyed")
 }
 
+// TestSweep_PublishUsesPerProtocolEnvelope pins the SNS message contract:
+// MessageStructure=json, with "email" carrying the plain-text summary and
+// "default" carrying AWS Chatbot's custom-notification schema. Chatbot
+// (the Slack delivery path) silently drops any other shape, so a break
+// here means a Slack channel that looks wired up but never shows a reaper
+// notice.
+func TestSweep_PublishUsesPerProtocolEnvelope(t *testing.T) {
+	now := time.Date(2026, 5, 21, 12, 0, 0, 0, time.UTC)
+	old := now.Add(-4 * time.Hour)
+	api := &FakeAWS{
+		TaggedResources: []rgtatypes.ResourceTagMapping{
+			{ResourceARN: aws.String("arn:aws:ec2:us-east-2:1:instance/i-old")},
+		},
+		EC2Instances: map[string]ec2types.Instance{
+			"i-old": {InstanceId: aws.String("i-old"), LaunchTime: &old},
+		},
+	}
+	_, err := Sweep(t.Context(), api, now, 3*time.Hour, "arn:sns:topic")
+	require.NoError(t, err)
+	require.Len(t, api.SNSMessages, 1)
+	require.Equal(t, []string{"json"}, api.SNSStructures)
+
+	var envelope map[string]string
+	require.NoError(t, json.Unmarshal([]byte(api.SNSMessages[0]), &envelope))
+	require.Contains(t, envelope["email"], "destroyed 1 resources")
+	require.Contains(t, envelope["email"], "ec2:i-old")
+
+	var chatbot struct {
+		Version string `json:"version"`
+		Source  string `json:"source"`
+		Content struct {
+			TextType    string `json:"textType"`
+			Description string `json:"description"`
+		} `json:"content"`
+	}
+	require.NoError(t, json.Unmarshal([]byte(envelope["default"]), &chatbot))
+	require.Equal(t, "1.0", chatbot.Version)
+	require.Equal(t, "custom", chatbot.Source)
+	require.Equal(t, "client-markdown", chatbot.Content.TextType)
+	require.Equal(t, envelope["email"], chatbot.Content.Description)
+}
+
 // TestSweep_StaleTagIndexEntryIsSilent: the tag index lags real deletions
 // by minutes to hours, so a sweep routinely describes resources that no
 // longer exist and gets a NotFound-style error back. That is a successful