Skip to content

fix: retry cgroup2/bpf-fs path check to handle mount propagation delay - #1845

Open
bhumikadangayach wants to merge 2 commits into
kmesh-net:mainfrom
bhumikadangayach:fix/cgroup2-stat-race
Open

bhumikadangayach wants to merge 2 commits into
kmesh-net:mainfrom
bhumikadangayach:fix/cgroup2-stat-race

Conversation

@bhumikadangayach

Copy link
Copy Markdown

Fixes #1842

The kmesh-daemon was crashing on startup with "stat /mnt/kmesh_cgroup2: no such file or directory" when running on kind. Traced it to ParseConfig() in daemon/options/bpf.go doing a single os.Stat call with no retry - if the mount isn't ready yet (which happens on kind due to mount propagation delay), it fails immediately instead of waiting.

This PR adds a waitForPath() helper that retries the stat call up to 10 times with 500ms between attempts (5 seconds total) before giving up. Applied to both the cgroup2 path and bpf-fs path checks.

Verified with make build - compiles clean, no errors.

@gemini-code-assist

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@kmesh-bot

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kevin-wangzefeng for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kmesh-bot

Copy link
Copy Markdown
Collaborator

Welcome @bhumikadangayach! It looks like this is your first PR to kmesh-net/kmesh 🎉

@codecov

codecov Bot commented Aug 2, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 39.62%. Comparing base (09e96d3) to head (3284f68).
⚠️ Report is 3 commits behind head on main.
see 1 file with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 60881fc...3284f68. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread daemon/options/bpf.go Outdated
retryDelay = 500 * time.Millisecond
)
var err error
for i := 0; i < maxRetries; i++ {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for i := 0; i < maxRetries; i++ {
for i := 0; i <= maxRetries; i++ {

Ensure that a retry is attempted even after the last failure

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, thanks!

@bhumikadangayach

Copy link
Copy Markdown
Author

The failing E2E appears to be TestCrossNamespace, where the unenrolled workload unexpectedly gets L7 processing (X-Request-Id). TestKmeshRestart passes with 360/360 successful requests. This appears unrelated to the cgroup2/bpf-fs retry change.

@bhumikadangayach

Copy link
Copy Markdown
Author

Same failure on the IPv6 run too - TestCrossNamespace, unenrolled workload getting X-Request-Id set. Unrelated to this change, only touches daemon/options/bpf.go.

Signed-off-by: Bhumika Dangayach <139267865+bhumikadangayach@users.noreply.github.com>
Signed-off-by: Bhumika Dangayach <139267865+bhumikadangayach@users.noreply.github.com>
@bhumikadangayach

Copy link
Copy Markdown
Author

This run hit #1911 - tests all passed (TestCrossNamespace, TestKmeshRestart 350/350), suite still exits 1 on log-collection step. Unrelated to this change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kmesh-daemon crash loops on kind because of a cgroup2 stat race

3 participants