Skip to content

fix: restore memory limit at 4Gi to prevent node starvation#1046

Open
BATMAN-JD wants to merge 1 commit into
openshift:masterfrom
BATMAN-JD:fix/restore-memory-limit-6gi
Open

fix: restore memory limit at 4Gi to prevent node starvation#1046
BATMAN-JD wants to merge 1 commit into
openshift:masterfrom
BATMAN-JD:fix/restore-memory-limit-6gi

Conversation

@BATMAN-JD

@BATMAN-JD BATMAN-JD commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Summary

PR #1044 removed the memory limit to address OOMKilled issues with 3Gi limit. However, this creates a risk where the memory leak in AAO could consume the entire node's memory and starve other pods.

Changes

Restore memory limit at 4Gi in both deployment manifests:

  • deploy/operator.yaml
  • deploy_pko/Deployment-aws-account-operator.yaml.gotmpl

Rationale

  • Provide headroom: 4Gi accommodates current workload (586 Account CRs in hivei01ue1) with ~33% increase over previous 3Gi
  • Prevent node starvation: Contains memory leak from consuming entire node memory (~25% of smallest 16Gi nodes)
  • Address OOM issues: Increase over previous 3Gi limit reduces OOM risk without consuming too much node capacity

Next Steps

The memory leak itself needs to be investigated and fixed separately.

Related

@coderabbitai

coderabbitai Bot commented Jul 1, 2026

Copy link
Copy Markdown

Walkthrough

Adds a memory: 6Gi resource limit for the aws-account-operator container in both deploy/operator.yaml and deploy_pko/Deployment-aws-account-operator.yaml.gotmpl, extending the existing cpu limits configuration.

Changes

Resource Limit Configuration

Layer / File(s) Summary
Add memory limit to deployment manifests
deploy/operator.yaml, deploy_pko/Deployment-aws-account-operator.yaml.gotmpl
Adds a memory: 6Gi entry under resources.limits for the aws-account-operator container in both files, alongside the existing cpu limit.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Suggested labels: deployment, configuration

Suggested reviewers: none

🐰 A hop, a nibble, a memory boost,
Six gigabytes now, no more roost
For carrots crammed too tight in a jar,
The operator breathes a little easier by far.

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly matches the main change: restoring the 6Gi memory limit to prevent node starvation.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PR only changes deployment YAML memory limits; no Ginkgo test titles were added or modified, so the deterministic-name rule isn’t implicated.
Test Structure And Quality ✅ Passed Only two deployment YAMLs changed; no Ginkgo test files or test code were modified, so the test-structure check is not applicable.
Microshift Test Compatibility ✅ Passed PR only changes two deployment manifests to add memory limits; no Ginkgo e2e tests or MicroShift-sensitive APIs/features were added.
Single Node Openshift (Sno) Test Compatibility ✅ Passed Only deployment manifests changed; no It/Describe/Context/When tests were added, so SNO compatibility checks are not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed Only memory limits changed in the two manifests; no nodeSelector, affinity, anti-affinity, spread constraints, or other topology-sensitive scheduling logic was added.
Ote Binary Stdout Contract ✅ Passed PR only changes two deployment YAML manifests to add a memory limit; no process-level code or stdout logging paths were modified.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No new Ginkgo/e2e tests were added; the branch only changes deployment manifests to set memory limits.
No-Weak-Crypto ✅ Passed Touched manifests only change memory limits; no weak-crypto terms or secret-comparison code appears in the PR files.
Container-Privileges ✅ Passed The changed manifests only add memory: 6Gi; neither file contains privileged, hostPID/hostNetwork/hostIPC, SYS_ADMIN, or allowPrivilegeEscalation settings.
No-Sensitive-Data-In-Logs ✅ Passed The only touched lines add a 6Gi memory limit in two manifests; no logging or sensitive-data fields appear in the changed files.
Description check ✅ Passed The description clearly explains the change, rationale, and related context, though the checklist and manual test steps from the template are missing.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot requested review from dakotalongRH and geowa4 July 1, 2026 14:39
@openshift-ci

openshift-ci Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BATMAN-JD

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 1, 2026
PR openshift#1044 removed the memory limit to address OOMKilled issues with 3Gi limit.
However, this creates a risk where the memory leak in AAO could consume the
entire node's memory and starve other pods.

Restore memory limit at 6Gi (double the previous 3Gi limit) to:
- Provide headroom for the current 586 Account CRs in hivei01ue1
- Prevent unbounded memory growth from starving the node
- Address the OOM issues while containing the memory leak

The memory leak itself needs to be investigated and fixed separately.

Related: openshift#1044
@BATMAN-JD BATMAN-JD force-pushed the fix/restore-memory-limit-6gi branch from 39d1d7b to 5f1f99a Compare July 1, 2026 14:50
@BATMAN-JD BATMAN-JD changed the title fix: restore memory limit at 6Gi to prevent node starvation fix: restore memory limit at 4Gi to prevent node starvation Jul 1, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 46.65%. Comparing base (b5e7e1f) to head (5f1f99a).

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #1046   +/-   ##
=======================================
  Coverage   46.65%   46.65%           
=======================================
  Files          46       46           
  Lines        7033     7033           
=======================================
  Hits         3281     3281           
  Misses       3414     3414           
  Partials      338      338           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@openshift-ci

openshift-ci Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

@BATMAN-JD: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants