fix(metrics): track unique cardinality label tuples#210
Open
ramnnn2006 wants to merge 1 commit into
Open
Conversation
cardinalityOK was incrementing seen[metric] on every event, so a
single busy (syscall, comm) pair could exhaust the 5000 limit in
under a second and permanently silence all observations for that
category, including already-tracked series.
fix: change seen from map[string]int to map[string]map[string]struct{}
and add a labelKey param to cardinalityOK. each caller now passes the
actual label tuple (joined with \x00) so only genuinely new combinations
count toward the cap. known tuples always pass through even when full.
updated TestCardinalityLimit to test new semantics: unique keys fill
the bucket, known keys still pass when full, new keys are rejected,
different metrics stay isolated.
Signed-off-by: ramnnn2006 <phrkvvp@gmail.com>
|
🚀 First PR — welcome aboard! A few things to expect:
If you get stuck, reply here or jump to Discussions. We want this PR to land. |
Contributor
Author
|
hi @btwshivam , could you review one happy to keep working on more issues in the repo, thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
this PR fixes a bug in
cardinalityOKwhereseen[metric]was incremented on every event instead of tracking distinct label combinations. it also refactors the underlying data structure and updates the test suite to cover the correct semantics.Why
Fixes #117
cardinalityOKwas acting as a raw event counter, not a cardinality tracker. one busy(syscall, comm)pair could exhaust the 5000 limit in under a second and permanently silence all observations for that category, including series that were already being tracked fine. the bug meant the cap was effectively unreachable in any real workload.How
the refactor changes
seenfrommap[string]inttomap[string]map[string]struct{}so each metric key maps to a bounded set of distinct label tuples instead of an event count. added alabelKeyparam tocardinalityOKand updated all 5 callers to compute their label values before the guard and pass them joined with\x00:recordSyscallpassessc + "\x00" + commrecordTCPpassessrc + "\x00" + dst + "\x00" + commrecordDiskIOpassesdev + "\x00" + oprecordSchedDelayandrecordFDpasscommknown tuples always pass through even when the set is full. only genuinely new combinations get dropped once the cap is hit.
Testing
rewrote
TestCardinalityLimitto test the actual cardinality verified event counting, which is the buggy behavior):N unique keys fill the bucket, last call returns
trueN+1 unique keys, last one is correctly rejected
known key passes even when the bucket is full (the exact cas
saturating one metric has no effect on a separate metric
go build ./...passes (GOOS=linuxfor eBPF-dependent packages)go test ./...passesgo vet ./...passes (non-eBPF packages)golangci-lint run ./...passesTested locally
N/A — pure refactor/bug fix, no eBPF programs touched
sudo ./bin/bpf-verify --read 5sconfirms 6/6 programs./scripts/verify.shpassesChecklist
fix(metrics): track unique label tuples in cardinalityOK)git commit -s)