Skip to content

fix(cluster): leave a level someone else changed for the cooldown - #128

Merged
Mearman merged 4 commits into
mainfrom
test/regrade-stepped-nodes
Oct 20, 2026
Merged

Mearman merged 4 commits into
mainfrom
test/regrade-stepped-nodes

Conversation

@Mearman

@Mearman Mearman commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

The regrade scenario flaked three times on different checks (another server stepped while node 2 stepped up, the role moved node 2 off etcd-only, node 3 did not read etcd-only after a step by hand). The cause is in the core, not the scenario: the watcher's calm streak began at its first sample and its cooldown counted only its own steps, so any server stepped down by hand or by another watcher was stepped straight back up on the next sample, by the service's own watcher running beside the one the test drives. A level that is not the one the last sample saw, and that the watcher did not step to, now ends both streaks and starts the cooldown. The scenario also counts only a known reduced level as a step, waits for a level to read instead of asserting a single read, and prints what regrading recorded for each node when it fails. The core change is a self-contained diff of fleet-regrade-core.sh with its test, for the fleet image's in-tree copy.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
🔒 Security Review ✅ Completed 2026-10-20T22:46:00.085604Z 91697b9 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins.

The watcher's calm streak and cooldown belonged to its own steps only.
A server stepped down by hand, or by another watcher, while the
watcher had been calm since its first sample was therefore stepped
straight back up on the next sample, undoing the step. A level that is
not the one the last sample saw, and that this watcher did not step
to, now ends both streaks and starts the cooldown.
@Mearman Mearman changed the title test(mesh): count only a known reduced level as a server having stepped fix(cluster): leave a level someone else changed for the cooldown Oct 20, 2026
@Mearman

Mearman commented Oct 20, 2026

Copy link
Copy Markdown
Member Author

@codex security review

@Mearman
Mearman merged commit 8f9257e into main Oct 20, 2026
23 checks passed
@Mearman
Mearman deleted the test/regrade-stepped-nodes branch October 20, 2026 22:43
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 2.7.4 🎉

The release is available on:

Your semantic-release bot 📦🚀

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 91697b9ede

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant