bench(soak): add mysql_cdc to the nightly soak rotation (CON-179 R6)

[prakhargarg105]

What

Adds mysql_cdc as the second connector in the CDC soak rotation, following SOAK.md's "Adding a connector to the rotation" recipe end to end:

• Scenarios — scenarios/mysql/orders-soak.yaml: the same sustained-moderate profile as the postgres soak (10K writes/sec × 1.2 KB ≈ 12 MB/s, ~10–15% of mysql_cdc's ~100–115 MB/s measured read ceiling) for 90 minutes at one vCPU point; plus the 30m orders-soak-pr.yaml binary-arm variant for /soak mysql/orders-soak-pr.
• Terraform — stacks/mysql + modules/rds-mysql, recovered from the benchmarking branch and brought up to this tree's conventions: bench-session-id tag (reaper age signal), cloud-nuke-excluded exemption, required_version >= 1.10 (lockfile locking), nullable storage_throughput with the same ≥400 GB gp3 tiering as postgres. binlog_format=ROW / binlog_row_image=FULL are what mysql_cdc requires.
• Runner — the mysql_cdc engineSpec entry and the discrete-flags reset form return from git history, keeping the current rpk-based topic cleanup.
• Seeder — seeders/cdc-rows-mysql is a forward-port of the current postgres seeder (the old benchmarking-branch copy predates the worker/tick remainder fairness fixes and the one-placeholder-per-row change), with InterpolateParams enabled since go-sql-driver, unlike pgx, otherwise wraps every parameterized Exec in a hidden server-side Prepare+Execute+Close.
• Rotation wiring — soak_nightly.yml becomes a matrix resolved by a plan job (the dispatch input is allowlisted BEFORE strategy evaluation, so a crafted value can't fan one dispatch into N paid soak jobs), serialized with max-parallel: 1 — one bench at a time, ever. terraform/persistent soak_scenarios gains the mysql entry (dashboard + stall/rss-slope/backlog alarms).

Landmines defused (found in review)

• RDS binlog retention: backup_retention_period=1 only enables binlog — RDS purges backed-up binlogs within minutes unless CALL mysql.rds_set_configuration('binlog retention hours', 24) is issued, which the reset now does every run. Without it, any mid-soak reconnect dies with ERROR 1236, a red night indistinguishable from a connector regression.
• checkpoint_limit: the 1024 default admits an oversized batch only when nothing is pending, serializing the pipeline to one 5000-row batch in flight; both scenarios set 10000.
• The reset renders mariadb (guaranteed by mariadb1011 on the runner host) rather than betting on the deprecated mysql compat symlink, refuses at render time to emit a command with any empty terraform-output connection part (bash collapses -p"" to a bare -p an hour into a paid session), and rejects multiline reset SQL, which %q quoting would silently alter.

Deliberately deferred

• IAM auth: this soak is the designated home of the ~15-min credential-rotation lever (RDS IAM tokens ride normal-protocol MySQL connections, unlike postgres replication — see mysql_cdc - IAM token expires mid-snapshot (input_mysql_stream.go:335) #4258/postgres_cdc: heartbeat connection never refreshes IAM auth token, fails every tick after ~15min #4668), but it lands as its own increment: AWSAuthenticationPlugin DB user in reset, rds-db:connect policy in the stack, and a live validation pass. Password auth first, exactly how the postgres soak entered rotation.
• Shared with the postgres seeder, deferred to keep the two in lockstep: extracting the duplicated worker/tick/payload math into a shared package, and cancelling sibling workload workers on first error.

Operator steps (BEFORE merge — the cron arms itself on the default branch)

1. TF_VAR_soak_alert_email=<team-alias> task aws:persistent re-apply (creates the mysql dashboard + alarms; until then nothing watches mysql metrics). Note bench: deliver soak alarms and reaper notices to Slack via AWS Chatbot #4733's Slack Chatbot subscription rides the same SNS topic — merge ordering with that branch just needs one persistent apply from main afterwards.
2. Manual nightly dispatch with scenario=mysql/orders-soak for live validation; the baseline comparator stays advisory until three soak-index entries exist.

Verification

• go test -race -shuffle=on ./..., golangci-lint, gofumpt: clean on the benchmarking module
• terraform validate + terraform fmt -check: clean on the new stack/module
• task aws:validate scenario=mysql/orders-soak and =mysql/orders-soak-pr: pass
• Not yet run live — that's operator step 2 above

🤖 Generated with Claude Code

[prakhargarg105]

Live validation: PASSED — run 33441795019 (manual dispatch of mysql/orders-soak from this branch, 2026-08-31)

Throughput	median 12.19 MB/s over 5,399 samples (p5 12.19 / p95 12.20 / peak 12.29)
Records/s	pinned at exactly 10,000 (the declared rate)
Memory	RSS max ~307 MB, no slope
Anomalies	0; zero broker scrape errors; no alarm fired during the run
Baseline comparator	ran advisory as designed (0 prior soak-index entries; arms at 3)
Teardown	terraform destroy complete, workflow's EC2/RDS verification confirmed the account clean

Byte-for-byte the same profile the postgres soak validated at (also 12.19 MB/s median), which is what you'd want from an identical 10K×1.2KB workload.

Operational state: the persistent stack already carries the mysql dashboard (rpcn-bench-soak-mysql) + 3 alarms, and the alarm→SNS→Chatbot→Slack path was confirmed live in #soak-redpanda-connect. The temporary OIDC trust entry used for this branch dispatch has been reverted (trust is back to refs/heads/main only). Branch is rebased onto main post-#4733 (SOAK.md conflict resolved).

🤖 Generated with Claude Code

[claude]

Soak results will land in a new docs/benchmark-results/mysql.md, not the existing mysql-cdc.md

The runner derives the results page from s.Stack, not the connector:

connect/benchmarking/aws/runner/main.go

Lines 567 to 571 in c00b992

	}
	mdPath := filepath.Join(opts.repoRoot, "docs/benchmark-results", s.Stack+".md")
	if err := AppendMarkdown(mdPath, result, strings.TrimSpace(s.Description)); err != nil {
	return err
	}

AppendMarkdown opens that path with os.O_APPEND|os.O_CREATE, so a missing file is silently created rather than erroring:

connect/benchmarking/aws/runner/render.go

Lines 223 to 228 in c00b992

	f, err := os.OpenFile(target, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
	if err != nil {
	return err
	}
	defer f.Close()

With stack: mysql the target is docs/benchmark-results/mysql.md, which does not exist — mysql_cdc's results page is docs/benchmark-results/mysql-cdc.md. The first nightly mysql soak will therefore create a second, parallel results file for the same connector, and the ~100-115 MB/s ceiling this scenario's description calibrates against (recorded in mysql-cdc.md) will sit in a different file from the soak trend line.

This diverges from the documented convention in docs/benchmarking.md — "Each connector gets its own file (e.g. mssqlserver-cdc.md)" — which CONTRIBUTING §1.3.4 points at for the reporting requirements. Postgres does not hit this because stack: postgres happens to match the existing postgres.md.

Suggested fix: rename docs/benchmark-results/mysql-cdc.md to mysql.md so the AWS soak appends to the established page (and update the link in docs/benchmarking.md's connector table), or make the results filename come from the connector rather than the stack. stack can't simply be renamed here since it also selects terraform/stacks/<stack> and the S3 state key.