Skip to content

fix(metrics): bound the owner-chain climb against circular ownerReferences - #299

Merged
blue4209211 merged 1 commit into
mainfrom
fix/owner-chain-depth-limit
Jul 29, 2026
Merged

fix(metrics): bound the owner-chain climb against circular ownerReferences#299
blue4209211 merged 1 commit into
mainfrom
fix/owner-chain-depth-limit

Conversation

@mayankpande88

Copy link
Copy Markdown
Contributor

Follow-up to #298. This change was pushed to that branch in response to review feedback, but #298 was merged from c7af3d5 moments earlier, so the fix did not land on main.

Problem

resolvePodDescriptor climbs the ownership chain with no termination condition other than reaching the top:

for {
    next, err := resolver.getControllerOfOwner(current)
    ...
}

ownerReferences are client-settable, so a cycle is reachable via a buggy controller or a hand-edited ref. A cycle spins forever and wedges the resolver goroutine, stalling every metric it feeds — all container_net_tcp_*, container_net_latency_seconds, and all L7 protocol families.

This is pre-existing, not introduced by #298: the original for owner != nil was equally unbounded.

Raised by gemini-code-assist in #298 (comment).

Change

Bound the walk at maxOwnerChainDepth = 10. Genuine chains are at most two hops (Pod -> ReplicaSet -> Deployment, Pod -> Job -> CronJob), leaving headroom for nested custom controllers.

The exhaustion case is explicit rather than a bare depth < 10 loop condition, because the code after the loop cannot otherwise distinguish "reached the top of the chain" from "gave up" — a malformed chain would be cached as though it had resolved cleanly:

for depth := 0; ; depth++ {
    if depth >= maxOwnerChainDepth {
        klog.Warningf("owner chain for pod %s/%s exceeded max depth %d at %s/%s — possible circular ownerReferences", ...)
        break
    }
    ...
}

Exhausting the bound is treated as terminal, not transient: a cycle cannot be fixed by retrying, and leaving the descriptor uncached would re-walk ten levels on every scrape. The identity resolved so far is kept, and the warning names the pod and the last owner reached so a real cycle is diagnosable rather than silent.

Test

TestResolvePodDescriptor_CircularOwnerChainTerminates builds a true A→B→A cycle, runs the resolve on a goroutine and fails on a 5s timeout — so removing the bound produces a failing test rather than a silent production hang.

Full ./common/ suite passes; go vet, gofmt, GOOS=linux GOARCH=amd64 build all clean.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a depth-bound limit to the Kubernetes owner reference traversal in resolvePodDescriptor to prevent infinite loops and goroutine leaks caused by circular owner references, along with a unit test to verify termination. The reviewer suggests a more robust approach to detect circular references using a map of visited UIDs instead of relying on an arbitrary depth limit, which would also allow removing the maxOwnerChainDepth constant.

Comment thread common/ip_resolver.go
Comment thread common/ip_resolver.go
…ences

OwnerReferences are client-settable, so a cycle is reachable via a buggy
controller or a hand-edited ref. The climb had no termination condition other
than reaching the top of the chain, so a cycle would spin forever and wedge
the resolver goroutine, stalling every metric it feeds. This was true of the
previous implementation too.

Bound the walk at maxOwnerChainDepth (10). Genuine chains are at most two
hops (Pod -> ReplicaSet -> Deployment, Pod -> Job -> CronJob), so the
headroom covers nested custom controllers while still terminating.

Exhausting the bound is treated as terminal rather than transient: a cycle
cannot be fixed by retrying, and leaving the descriptor uncached would
re-walk ten levels on every scrape. The identity resolved so far is kept and
a warning is logged.

Adds a regression test that runs the resolve on a goroutine and fails on a
5s timeout, so removing the bound reintroduces a detectable hang rather than
an invisible one.
@mayankpande88
mayankpande88 force-pushed the fix/owner-chain-depth-limit branch from b25c696 to 366ed3a Compare July 29, 2026 08:10
@blue4209211
blue4209211 merged commit a061fc4 into main Jul 29, 2026
7 checks passed
@blue4209211
blue4209211 deleted the fix/owner-chain-depth-limit branch July 29, 2026 08:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants