Skip to content

Avoid cancelltion checks on every advance() call for posting enums - #22856

Open
sgup432 wants to merge 2 commits into
opensearch-project:mainfrom
sgup432:advance_cancellation_check_fix
Open

Avoid cancelltion checks on every advance() call for posting enums#22856
sgup432 wants to merge 2 commits into
opensearch-project:mainfrom
sgup432:advance_cancellation_check_fix

Conversation

@sgup432

@sgup432 sgup432 commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Description

As part of earlier change(in OS 3.7, PR) done to strengthen the system so that we are able to stop the long running query if it was already cancelled, we ended up adding a check on advance() without sampling. In worst advance() might behave like nextDoc() where we end up iterating all the docs, so doing cancellation checks on every doc can cause high CPU and also unnecessary memory allocation.

This was visible for one of the OpenSearch user. I took a 5min JFR profile for their domain to root cause latency/CPU spikes after upgrade to 3.7.

CPU profile:


  
  ┌──────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
  │  %   │                                                               Method                                                                │
  ├──────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ~36% │ cancellation check: MutableQueryTimeout.checkCancelled → HashMap$HashIterator (<init> 18.8% + nextNode 12.1% + checkCancelled 5.6%) │
  ├──────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ 26%  │ ExitablePostingsEnum.advance (the 11-term must_not disjunction advancing postings)                                                  │
  ├──────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤

Memory allocation:


  ┌──────────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ % of all heap allocation │                                                                 Source                                                                  │
  ├──────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ 74.2%                    │ HashMap$KeySet.iterator() — from checkCancelled()'s for (Runnable : runnables) over a HashSet, allocating a KeyIterator every advance() │
  ├──────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ~4%                      │ BlockPostingsEnum.<init>                                                                                                                │
  ├──────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>
@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

PR Reviewer Guide 🔍

(Review updated until commit 44dc105)

Here are some key observations to aid the review process:

🧪 No relevant tests
🔒 No security concerns identified
✅ No TODO sections
🔀 No multiple PR themes
⚡ No major issues detected

@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 44dc105

@github-actions

Copy link
Copy Markdown
Contributor

❌ Gradle check result for 44dc105: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions

Copy link
Copy Markdown
Contributor

✅ Gradle check result for 44dc105: SUCCESS

@codecov

codecov Bot commented Aug 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 71.64%. Comparing base (7c635d7) to head (44dc105).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main   #22856      +/-   ##
============================================
+ Coverage     71.53%   71.64%   +0.11%     
- Complexity    77249    77390     +141     
============================================
  Files          6170     6170              
  Lines        359710   359710              
  Branches      52460    52460              
============================================
+ Hits         257318   257718     +400     
+ Misses        81965    81589     -376     
+ Partials      20427    20403      -24     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sgup432

sgup432 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

@jainankitk Can we merge this in now?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants