tests: add goleak leak detection and advisory race-detector CI (CON-179) - #4693
Closed
prakhargarg105 wants to merge 2 commits into
Closed
tests: add goleak leak detection and advisory race-detector CI (CON-179)#4693prakhargarg105 wants to merge 2 commits into
prakhargarg105 wants to merge 2 commits into
Conversation
… packages
Wire goleak.VerifyTestMain into internal/impl/{protobuf,sql,postgresql,mysql}
— the packages that shipped leak bugs (9caed49, 07c39cd) plus the two
flagship CDC connector packages — with narrow, commented per-package ignore
lists, each verified green on a clean baseline (unit x3, -race x2,
postgres/mysql integration suites under Docker).
Two real test-side leaks surfaced and fixed rather than ignored:
- protobuf: runMockBSRServer used bare http.Serve with no shutdown, leaking
the accept loop and connection goroutines; now an http.Server closed via
t.Cleanup.
- postgresql: TestIntegrationPostgresSnapshotAckBarrier's consumer waited on
a context benthos never cancels (runConsumerFunc passes Background), and
run 1 was never stopped because Stream.Run returns on ctx cancel without
tearing the stream down; now selects on the test-owned context and stops
the crashed stream explicitly.
Also documents the "adding goleak to a package" recipe in the tester agent
guide, per the R2 acceptance criteria.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r packages Add race_test.yml: on every PR it runs go test -race against only the internal/impl/<component> packages the PR touches, using the same change-detection as integration_test.yml to bound the 2-10x race-detector cost. Failures surface as warning annotations and a step-summary table, not a red check — unless the package is listed in .github/race-blocking-packages.txt, the promotion list for packages that have earned a green -race baseline. The list starts empty so the gate can never fail on a pre-existing race a PR didn't touch. Also reference task test:unit-race in the CONTRIBUTING §6 pre-PR checklist, per the R3 acceptance criteria. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
|
Superseded by #4721 — remade from a branch directly on this repo instead of a fork. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of CON-179 (test hardening). This PR adds two complementary checks: goroutine-leak detection inside the test suites of selected connector packages, and a race-detector CI job that starts out advisory and gains teeth package by package.
1. Goroutine-leak detection (goleak)
Adds
go.uber.org/goleakand wires it into a package-levelTestMainfor four packages:internal/impl/mysql,internal/impl/postgresql,internal/impl/protobuf, andinternal/impl/sql. After the last test in the package finishes, goleak fails the run if any goroutine started during the tests is still alive.How it runs:
TestMainis Go's standard per-package entry point, so this needs no CI wiring. It fires on everygo testof these packages, which means every PR (via the existingtest.ymlunit suite) and every nightly integration run. A leak is a hard test failure.Ignore policy: each
TestMaincarries a small set of narrow, individually commented ignores for goroutines we cannot stop from test code. Examples: prototransform'sSchemaWatcherpoll loop (its owner exposes no stop hook and the processor'sCloseis a no-op), and the license service's hourly expiry-metric loop started byInjectTestService. Broad ignores are deliberately avoided; each entry names the library and the reason.Real leak fixed along the way:
TestIntegrationPostgresSnapshotAckBarrierblocked a consumer goroutine on a context that benthos never cancels (consumer funcs receivecontext.Background()), which leaked the goroutine and the unacked stream behind it past the end of the test. The test now uses its own cancellable context for the simulated crash and explicitly stops the crashed stream withStopWithin. The protobuf mock BSR server also gets a properShutdownon cleanup.All four packages were verified leak-clean on repeated local runs before the check was enabled, so this cannot fail a PR on a pre-existing condition.
2. Advisory race-detector CI job
New workflow
.github/workflows/race_test.yml, plus a promotion list at.github/race-blocking-packages.txt.How it runs: on every PR the job diffs against the merge-base and collects the
internal/impl/<pkg>directories the PR touched (same auto-scoping asintegration_test.yml, which keeps the 2-10x race-detector runtime cost bounded). If no connector package changed, the job skips entirely. Otherwise it runsgo test -count=1 -race -shuffle=on -timeout 10mper touched package and writes a pass/fail table to the job summary. Aworkflow_dispatchtrigger allows manual runs against any package by name.Advisory-first design: a race failure produces a warning annotation, and the check stays green, unless the failing package is listed in
race-blocking-packages.txt. That file starts empty, so on day one nothing can go red from this job. A package is promoted to blocking only after it holds a green-racebaseline (repeated clean scoped runs plus a green advisory run in CI). The invariant this preserves: a blocking gate must never be able to fail a PR on a pre-existing race the PR did not introduce.Supporting changes
CONTRIBUTING.md: contributors are asked to runtask test:unit-raceon packages they touch, with a pointer to the blocking-list semantics..claude/agents/tester.md: recipe for wiring goleak into a new package, including the ignore policy and the requirement to verify against a clean baseline before merging.go.mod: addsgo.uber.org/goleak v1.3.0(test-only dependency).🤖 Generated with Claude Code