Conversation
ClusterCache.Flush set cluster.Id from HashName.StrToNum, which is a plain
map lookup rather than the allocator. HashName.Hash is the only function
that registers a name, and it is called solely on the workload path, so in
kernel-native mode strToNum stays empty and every cluster flushed to the
BPF map got id 0.
Circuit breaker stats are keyed by {netns_cookie, cluster_id}, so all
clusters shared one counter and were throttled against a single combined
limit. clearClusterStats also resolved id 0 and batch-deleted every entry
with ClusterId 0, wiping the connection counters of unrelated clusters
whenever one was removed.
clearClusterStats keeps using StrToNum: it looks up an id that Flush has
already registered, and runs before hashName.Delete on the delete path.
TestClusterFlushAssignsIds covers the two properties the stats map depends
on: two clusters get distinct non-zero ids, and an id is stable across a
later flush of the same cluster, since its stats entries are keyed by it.
Signed-off-by: opx0 <akyv2.5@gmail.com>
The test built its stats keys from HashName.StrToNum on a HashName that nothing had registered into, so all five keys carried ClusterId 0 and collapsed to three distinct entries. The first clearClusterStats call resolved id 0 and deleted all of them, leaving the iteration with nothing to compare, so the assertions never ran for any of the three clusters. Register the names through Hash so the keys carry distinct ids, and assert the whole remaining map rather than only that the cleared id is absent. That covers the property the test was named for: clearing one cluster leaves the other clusters' counters untouched. Signed-off-by: opx0 <akyv2.5@gmail.com>
|
Welcome @opx0! It looks like this is your first PR to kmesh-net/kmesh 🎉 |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Pull request overview
Fixes kernel-native mode cluster flushing so clusters receive stable, non-zero IDs (instead of always 0), preventing circuit-breaker stats collisions keyed by {netns_cookie, cluster_id}.
Changes:
- Update
ClusterCache.Flushto allocate cluster IDs viaHashName.Hash(name)rather than a plain lookup. - Fix
TestClearClusterStatsto register cluster names throughHash()and assert the remaining stats map precisely after deletions. - Add
TestClusterFlushAssignsIdsto assert cluster IDs are non-zero, distinct across clusters, and stable across multiple flushes.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| pkg/cache/v2/cluster.go | Allocate/persist cluster IDs during flush using Hash() so ADS/kernel-native mode no longer emits Id=0 for every cluster. |
| pkg/cache/v2/cluster_test.go | Update stats-clearing test to use registered IDs and add coverage ensuring flush assigns stable, distinct non-zero IDs. |
Suppressed comments (2)
pkg/cache/v2/cluster_test.go:433
- This subtest creates a HashName backed by the global persist file ("/mnt/hash_name.yaml") and deletes it via Reset(). Since CI runs package tests in parallel, this shared file can be mutated concurrently by other packages' tests that also use HashName, leading to flaky results. Prefer injecting a temp persist path per test (t.TempDir) once HashName supports it.
t.Run("clusters get distinct non-zero ids", func(t *testing.T) {
hashName := utils.NewHashName()
defer hashName.Reset()
cache := NewClusterCache(nil, hashName)
pkg/cache/v2/cluster_test.go:449
- Same as above: this subtest uses utils.NewHashName() (global /mnt/hash_name.yaml) and deletes it via Reset(), which can race with other packages' tests under parallel
go test. Once HashName allows configuring the persist path, switch tests to a per-test temp file to avoid cross-package interference.
t.Run("id is stable across flushes", func(t *testing.T) {
hashName := utils.NewHashName()
defer hashName.Reset()
cache := NewClusterCache(nil, hashName)
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| for name, cluster := range cache.apiClusterCache { | ||
| if cluster.GetApiStatus() == core_v2.ApiStatus_UPDATE { | ||
| cluster.Id = cache.hashName.StrToNum(name) | ||
| cluster.Id = cache.hashName.Hash(name) | ||
| err := maps_v2.ClusterUpdate(name, cluster) |
| hashName := utils.NewHashName() | ||
| defer hashName.Reset() | ||
| clusterCache := NewClusterCache(adsObj, hashName) |
Codecov Report✅ All modified and coverable lines are covered by tests.
... and 1 file with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
|
I’ve taken the e2e for point 2 (two clusters with different circuit-breaker limits must not share cluster_stats). The test is TestCircuitBreakerClusterStatsIsolation: one client pod, DestinationRules with maxConnections: 10 and 100 (same reviews/ratings scenario), hold more than 10 connections on the high-limit cluster, then assert the low-limit cluster still accepts. That is the BPF-map property the unit test cannot cover. I will keep this off #1938 so we don’t both land the Hash/Id fix. After this merges I’ll open a follow-up with that test and ping you for review. If you’d rather have the e2e on this PR, say so and I can send the commits here instead. |
|
@Aryan-B2107 Let's do it as a follow-up, the way you suggested. It keeps things clean on both sides — #1938 stays a small, one-line fix with its unit tests, so it's easy to review and merge, and your e2e gets to run against main once the fix is actually there, which is what it needs anyway. Bundling them would just gate the little fix behind the full Istio CI for no real gain. So I'll focus on getting #1938 in, then ping me the moment your follow-up is up and I'll review it right away. Appreciate you jumping on this one. |
|
Sounds good, follow-up after this merges, e2e only. Could you ping me when #1938 is in? I’ll open the PR against main right after and tag you for review. |
What type of PR is this?
/kind bug
What this PR does / why we need it:
In kernel-native mode every cluster is flushed to the BPF map with
Id = 0.ClusterCache.Flushtook the id fromHashName.StrToNum, which is a plain map read rather than the allocator:HashName.Hashis the only function that inserts intostrToNum, and it is called exclusively frompkg/controller/workload/workload_processor.go— the workload path. Nothing on the ads path ever calls it.NewAdsCachebuilds a freshutils.NewHashName()and hands it only toNewClusterCache, and the persist file thatNewHashNamerestores from is written only byHashandDelete, so in kernel-native mode it stays empty too. The lookup therefore returned the zero value for every cluster.That matters because circuit breaker stats are keyed by
{netns_cookie, cluster_id}(bpf/kmesh/ads/include/circuit_breaker.h), andon_cluster_sock_bindreads the id straight off the cluster:So with every id at 0:
cluster_statsentry. A pod callingreviews(maxConnections: 10) andratings(maxConnections: 100) shares one counter: after 10 combined connectionson_cluster_sock_bindreturns-1and further connections toreviewsare rejected, whileratingsis throttled at the wrong threshold.clearClusterStatsresolvesclusterId = 0andBatchDeletes every key withClusterId == 0, which is all of them. Removing one cluster wipes the live connection counters of every cluster in every netns, and subsequent closes then hit the-delta > stats->active_connectionsguard, so the counters stay skewed.The fix is to use the allocator:
Hashreturns the existing id when the name is already registered, so a cluster keeps its id across config pushes and its stats entries stay reachable.The second commit is a test-only fix.
TestClearClusterStatsbuilt its stats keys fromStrToNumon aHashNamenothing had registered into, so all five keys carriedClusterId 0and collapsed to three distinct entries; the firstclearClusterStatscall then deleted all of them and the per-cluster assertions never ran. It now registers the names throughHashand asserts the whole remaining map, which is the property the test is named for: clearing one cluster leaves the other clusters' counters untouched.Which issue(s) this PR fixes:
Fixes #1936
Special notes for your reviewer:
clearClusterStatsdeliberately stays onStrToNum. It resolves an id thatFlushhas already registered, and it runs beforehashName.Delete(name)on the delete path, so the entry is still present. Using the allocator there would mint an id for a cluster that is being removed.Hashpersists new entries, the ads path now writes/mnt/hash_name.yaml, which it never did before./mntis already a hostPath mount in the daemonset (deploy/yaml/kmesh.yaml,deploy/charts/kmesh-helm/templates/daemonset.yaml), and this is what makes cluster ids survive a restart, so thecluster_statsentries written before the restart stay addressable afterwards.HashName.Hash, which has a separate edge case:for num = h.hash.Sum32(); num < math.MaxUint32; num++never enters the body when the hash is exactlymath.MaxUint32, so it returns an id it never registered. On this path that would mean a live cluster whose name is missing fromstrToNum, and a laterclearClusterStatsresolving 0 for it. It is a 1-in-2^32 case and fix(utils): make HashName ring probe reach the final slot #1905 already fixes it, so I have not touched it here; flagging the interaction since it becomes reachable from the ads path with this change.TestClusterFlushAssignsIdsfails on the unmodified line (Should not be zero, but was 0for both clusters, and the two ids equal) and passes with the fix, under the same-race -gcflags=all=-lflags CI uses.pkg/controller/ads/...and the rest ofpkg/cache/v2show no new failures againstmain; five tests inpkg/cache/v2(TestClusterLookupAll,TestClearClusterStats,TestListenerLookupAll,TestListenerFlushAndLookup,TestListenerUpdateAndDeleteFlush) fail identically on unmodifiedmainin my environment because I cannot attach real BPF maps there. That means theTestClearClusterStatschange in the second commit is exercised by CI rather than by me locally.cluster_stats. That property lives in the BPF map, not in the Go cache, so it needs an e2e test rather than a unit test; what the unit test can show is what it now shows, that the two clusters resolve to distinct non-zero ids, with the differingmaxConnectionsin the fixture to keep the scenario recognisable. Happy to follow up with the e2e case if you would like it.Does this PR introduce a user-facing change?: