Skip to content

tests: add goleak leak detection and advisory race-detector CI (CON-179) - #4693

Closed
prakhargarg105 wants to merge 2 commits into
redpanda-data:mainfrom
prakhargarg105:con-179-goleak-race
Closed

tests: add goleak leak detection and advisory race-detector CI (CON-179)#4693
prakhargarg105 wants to merge 2 commits into
redpanda-data:mainfrom
prakhargarg105:con-179-goleak-race

Conversation

@prakhargarg105

@prakhargarg105 prakhargarg105 commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Part of CON-179 (test hardening). This PR adds two complementary checks: goroutine-leak detection inside the test suites of selected connector packages, and a race-detector CI job that starts out advisory and gains teeth package by package.

1. Goroutine-leak detection (goleak)

Adds go.uber.org/goleak and wires it into a package-level TestMain for four packages: internal/impl/mysql, internal/impl/postgresql, internal/impl/protobuf, and internal/impl/sql. After the last test in the package finishes, goleak fails the run if any goroutine started during the tests is still alive.

How it runs: TestMain is Go's standard per-package entry point, so this needs no CI wiring. It fires on every go test of these packages, which means every PR (via the existing test.yml unit suite) and every nightly integration run. A leak is a hard test failure.

Ignore policy: each TestMain carries a small set of narrow, individually commented ignores for goroutines we cannot stop from test code. Examples: prototransform's SchemaWatcher poll loop (its owner exposes no stop hook and the processor's Close is a no-op), and the license service's hourly expiry-metric loop started by InjectTestService. Broad ignores are deliberately avoided; each entry names the library and the reason.

Real leak fixed along the way: TestIntegrationPostgresSnapshotAckBarrier blocked a consumer goroutine on a context that benthos never cancels (consumer funcs receive context.Background()), which leaked the goroutine and the unacked stream behind it past the end of the test. The test now uses its own cancellable context for the simulated crash and explicitly stops the crashed stream with StopWithin. The protobuf mock BSR server also gets a proper Shutdown on cleanup.

All four packages were verified leak-clean on repeated local runs before the check was enabled, so this cannot fail a PR on a pre-existing condition.

2. Advisory race-detector CI job

New workflow .github/workflows/race_test.yml, plus a promotion list at .github/race-blocking-packages.txt.

How it runs: on every PR the job diffs against the merge-base and collects the internal/impl/<pkg> directories the PR touched (same auto-scoping as integration_test.yml, which keeps the 2-10x race-detector runtime cost bounded). If no connector package changed, the job skips entirely. Otherwise it runs go test -count=1 -race -shuffle=on -timeout 10m per touched package and writes a pass/fail table to the job summary. A workflow_dispatch trigger allows manual runs against any package by name.

Advisory-first design: a race failure produces a warning annotation, and the check stays green, unless the failing package is listed in race-blocking-packages.txt. That file starts empty, so on day one nothing can go red from this job. A package is promoted to blocking only after it holds a green -race baseline (repeated clean scoped runs plus a green advisory run in CI). The invariant this preserves: a blocking gate must never be able to fail a PR on a pre-existing race the PR did not introduce.

Supporting changes

  • CONTRIBUTING.md: contributors are asked to run task test:unit-race on packages they touch, with a pointer to the blocking-list semantics.
  • .claude/agents/tester.md: recipe for wiring goleak into a new package, including the ignore policy and the requirement to verify against a clean baseline before merging.
  • go.mod: adds go.uber.org/goleak v1.3.0 (test-only dependency).

🤖 Generated with Claude Code

prakhargarg105 and others added 2 commits August 12, 2026 09:25
… packages

Wire goleak.VerifyTestMain into internal/impl/{protobuf,sql,postgresql,mysql}
— the packages that shipped leak bugs (9caed49, 07c39cd) plus the two
flagship CDC connector packages — with narrow, commented per-package ignore
lists, each verified green on a clean baseline (unit x3, -race x2,
postgres/mysql integration suites under Docker).

Two real test-side leaks surfaced and fixed rather than ignored:
- protobuf: runMockBSRServer used bare http.Serve with no shutdown, leaking
  the accept loop and connection goroutines; now an http.Server closed via
  t.Cleanup.
- postgresql: TestIntegrationPostgresSnapshotAckBarrier's consumer waited on
  a context benthos never cancels (runConsumerFunc passes Background), and
  run 1 was never stopped because Stream.Run returns on ctx cancel without
  tearing the stream down; now selects on the test-owned context and stops
  the crashed stream explicitly.

Also documents the "adding goleak to a package" recipe in the tester agent
guide, per the R2 acceptance criteria.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r packages

Add race_test.yml: on every PR it runs go test -race against only the
internal/impl/<component> packages the PR touches, using the same
change-detection as integration_test.yml to bound the 2-10x race-detector
cost. Failures surface as warning annotations and a step-summary table, not
a red check — unless the package is listed in
.github/race-blocking-packages.txt, the promotion list for packages that
have earned a green -race baseline. The list starts empty so the gate can
never fail on a pre-existing race a PR didn't touch.

Also reference task test:unit-race in the CONTRIBUTING §6 pre-PR checklist,
per the R3 acceptance criteria.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@prakhargarg105 prakhargarg105 changed the title Con 179 goleak race tests: add goleak leak detection and advisory race-detector CI (CON-179) Aug 12, 2026
@prakhargarg105

Copy link
Copy Markdown
Collaborator Author

Superseded by #4721 — remade from a branch directly on this repo instead of a fork.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant