fix(metrics): bound the owner-chain climb against circular ownerReferences - #299
Merged
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a depth-bound limit to the Kubernetes owner reference traversal in resolvePodDescriptor to prevent infinite loops and goroutine leaks caused by circular owner references, along with a unit test to verify termination. The reviewer suggests a more robust approach to detect circular references using a map of visited UIDs instead of relying on an arbitrary depth limit, which would also allow removing the maxOwnerChainDepth constant.
…ences OwnerReferences are client-settable, so a cycle is reachable via a buggy controller or a hand-edited ref. The climb had no termination condition other than reaching the top of the chain, so a cycle would spin forever and wedge the resolver goroutine, stalling every metric it feeds. This was true of the previous implementation too. Bound the walk at maxOwnerChainDepth (10). Genuine chains are at most two hops (Pod -> ReplicaSet -> Deployment, Pod -> Job -> CronJob), so the headroom covers nested custom controllers while still terminating. Exhausting the bound is treated as terminal rather than transient: a cycle cannot be fixed by retrying, and leaving the descriptor uncached would re-walk ten levels on every scrape. The identity resolved so far is kept and a warning is logged. Adds a regression test that runs the resolve on a goroutine and fails on a 5s timeout, so removing the bound reintroduces a detectable hang rather than an invisible one.
mayankpande88
force-pushed
the
fix/owner-chain-depth-limit
branch
from
July 29, 2026 08:10
b25c696 to
366ed3a
Compare
blue4209211
approved these changes
Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #298. This change was pushed to that branch in response to review feedback, but #298 was merged from
c7af3d5moments earlier, so the fix did not land onmain.Problem
resolvePodDescriptorclimbs the ownership chain with no termination condition other than reaching the top:ownerReferencesare client-settable, so a cycle is reachable via a buggy controller or a hand-edited ref. A cycle spins forever and wedges the resolver goroutine, stalling every metric it feeds — allcontainer_net_tcp_*,container_net_latency_seconds, and all L7 protocol families.This is pre-existing, not introduced by #298: the original
for owner != nilwas equally unbounded.Raised by
gemini-code-assistin #298 (comment).Change
Bound the walk at
maxOwnerChainDepth = 10. Genuine chains are at most two hops (Pod -> ReplicaSet -> Deployment,Pod -> Job -> CronJob), leaving headroom for nested custom controllers.The exhaustion case is explicit rather than a bare
depth < 10loop condition, because the code after the loop cannot otherwise distinguish "reached the top of the chain" from "gave up" — a malformed chain would be cached as though it had resolved cleanly:Exhausting the bound is treated as terminal, not transient: a cycle cannot be fixed by retrying, and leaving the descriptor uncached would re-walk ten levels on every scrape. The identity resolved so far is kept, and the warning names the pod and the last owner reached so a real cycle is diagnosable rather than silent.
Test
TestResolvePodDescriptor_CircularOwnerChainTerminatesbuilds a true A→B→A cycle, runs the resolve on a goroutine and fails on a 5s timeout — so removing the bound produces a failing test rather than a silent production hang.Full
./common/suite passes;go vet,gofmt,GOOS=linux GOARCH=amd64build all clean.