Fix: OpenSearch process does not exit when startup fails due to StartupException by aparajita31pandey · Pull Request #22259 · opensearch-project/OpenSearch

aparajita31pandey · 2026-06-21T07:19:29Z

Description

When a plugin or bootstrap component can throws a RuntimeException during startup, OpenSearch.init() wraps it in a StartupException and rethrows it. StartupException was never caught in the call chain — it escaped OpenSearch.main(String[]) entirely, bypassing the exit(status) call.

This lefts the JVM process hanging indefinitely after startup failure.

Resolves

#22260

Root Cause

StartupException is a RuntimeException. The CLI framework in Command.main() only handles OptionException and UserException — so StartupException propagates uncaught all the way through main(String[], OpenSearch, Terminal) and escapes main(String[]). When non-daemon threads are still alive (as they are during a partial bootstrap), the JVM does not terminate automatically, leaving the process hanging.

// main(String[]) — StartupException escapes here, exit() is never reached
int status = main(args, opensearch, Terminal.DEFAULT);
if (status != ExitCodes.OK) {
    ...
    exit(status);  // never reached
}

Fix

StartupException designed to escape to main() — its printStackTrace() override has a comment: "This logic actually prints the exception to the console, its what is invoked by the JVM when we throw the exception from main()". The formatted output was already correct. The only missing piece was exit().

Catch StartupException in main(String[]) — the correct level where System.err is appropriate and process exit decisions belong — call e.printStackTrace(System.err) to preserve the existing custom-formatted output (which truncates guice frames, etc.), then exit(CODE_ERROR):

try {
    status = main(args, opensearch, Terminal.DEFAULT);
} catch (StartupException e) {
    e.printStackTrace(System.err);
    exit(ExitCodes.CODE_ERROR);
    return;
}

Stack trace (reproducer)

org.opensearch.bootstrap.StartupException: java.lang.RuntimeException: java.lang.RuntimeException: dummy
    at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:173)
    at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:160)
    at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110)
    at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:146)
    at org.opensearch.cli.Command.main(Command.java:101)
    at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:126)

Signed-off-by: Aparajita Pandey aparajita31pandey@gmail.com

github-actions · 2026-06-21T07:20:33Z

PR Reviewer Guide 🔍

(Review updated until commit `2c7bb2b`)

Here are some key observations to aid the review process:

🧪 PR contains tests
🔒 No security concerns identified
✅ No TODO sections
🔀 No multiple PR themes
⚡ No major issues detected

github-actions · 2026-06-21T07:20:43Z

PR Code Suggestions ✨

Latest suggestions up to d270b56

Explore these optional code suggestions:

Category	Suggestion	Impact
General	Flush error writer after printing stack trace After printing the stack trace to the terminal's error writer, the writer should be flushed to ensure the output is actually emitted before the method returns and the process exits. Without flushing, the error output may be lost, defeating the purpose of preserving `StartupException`'s custom formatting. server/src/main/java/org/opensearch/bootstrap/OpenSearch.java [126-133] try { return opensearch.main(args, terminal); } catch (StartupException e) { // StartupException has custom printStackTrace formatting (truncates guice frames, etc.). // Catch it here so the process exits rather than hanging, while preserving that output. e.printStackTrace(terminal.getErrorWriter()); + terminal.getErrorWriter().flush(); return ExitCodes.CODE_ERROR; } Suggestion importance[1-10]: 7 __ Why: Flushing the error writer after `printStackTrace` is a valid concern, as buffered output could be lost before process exit, defeating the purpose of preserving the custom formatting. This is a meaningful reliability improvement.	Medium

Previous suggestions

Suggestions up to commit 07ac748

Category	Suggestion	Impact
General	Flush error writer after printing stack trace After writing the stack trace to the terminal's error writer, the buffered output may not be visible to the user because the writer is not flushed. Flush the error writer after printing to ensure the error output is emitted before the process exits. server/src/main/java/org/opensearch/bootstrap/OpenSearch.java [126-133] static int main(final String[] args, final OpenSearch opensearch, final Terminal terminal) throws Exception { try { return opensearch.main(args, terminal); } catch (StartupException e) { // StartupException has custom printStackTrace formatting (truncates guice frames, etc.). // Catch it here so the process exits rather than hanging, while preserving that output. e.printStackTrace(terminal.getErrorWriter()); + terminal.getErrorWriter().flush(); return ExitCodes.CODE_ERROR; } } Suggestion importance[1-10]: 5 __ Why: Flushing the error writer is a reasonable defensive measure to ensure the stack trace is emitted before process exit, though in practice `PrintWriter` from `Terminal` may auto-flush. Minor robustness improvement.	Low

Suggestions up to commit 7bfb1fb

Category	Suggestion	Impact
General	Flush error writer after printing exception After printing the stack trace to the terminal's error writer, the buffered output may not be flushed before the process exits, potentially causing the error message to be lost. Explicitly flush the error writer after printing. server/src/main/java/org/opensearch/bootstrap/OpenSearch.java [126-133] static int main(final String[] args, final OpenSearch opensearch, final Terminal terminal) throws Exception { try { return opensearch.main(args, terminal); } catch (StartupException e) { // StartupException has custom printStackTrace formatting (truncates guice frames, etc.). // Catch it here so the process exits rather than hanging, while preserving that output. e.printStackTrace(terminal.getErrorWriter()); + terminal.getErrorWriter().flush(); return ExitCodes.CODE_ERROR; } } Suggestion importance[1-10]: 6 __ Why: Flushing the error writer after printing the stack trace is a reasonable defensive measure to ensure the error output is not lost before process exit, though the underlying `Terminal` implementation may already flush on close. Moderate impact on reliability of error reporting.	Low

Suggestions up to commit 23ea016

Category	Suggestion	Impact
General	Flush error writer before returning After printing the stack trace to the terminal's error writer, the writer should be flushed to ensure the output is actually emitted before the process exits. Without flushing, buffered error output may be lost when the JVM terminates via `System.exit`. server/src/main/java/org/opensearch/bootstrap/OpenSearch.java [126-133] static int main(final String[] args, final OpenSearch opensearch, final Terminal terminal) throws Exception { try { return opensearch.main(args, terminal); } catch (StartupException e) { // StartupException has custom printStackTrace formatting (truncates guice frames, etc.). // Catch it here so the process exits rather than hanging, while preserving that output. e.printStackTrace(terminal.getErrorWriter()); + terminal.getErrorWriter().flush(); return ExitCodes.CODE_ERROR; } } Suggestion importance[1-10]: 6 __ Why: Flushing the error writer before returning is a reasonable safeguard to ensure the stack trace output is emitted before the JVM exits, though in practice `System.exit` and shutdown hooks typically flush standard streams.	Low

Suggestions up to commit 61cf667

Category	Suggestion	Impact
General	Flush error writer after printing exception After printing the stack trace to the terminal's error writer, the buffered output may not be flushed before the process exits, causing the diagnostic to be lost. Flush the error writer explicitly after printing the stack trace to ensure the failure details are visible to the user. server/src/main/java/org/opensearch/bootstrap/OpenSearch.java [126-133] try { return opensearch.main(args, terminal); } catch (StartupException e) { // StartupException has custom printStackTrace formatting (truncates guice frames, etc.). // Catch it here so the process exits rather than hanging, while preserving that output. e.printStackTrace(terminal.getErrorWriter()); + terminal.getErrorWriter().flush(); return ExitCodes.CODE_ERROR; } Suggestion importance[1-10]: 6 __ Why: Flushing the error writer after printing the stack trace is a reasonable safeguard to ensure the diagnostic output is not lost before process exit, though the underlying writer may already flush on close/exit. It's a minor but valid improvement.	Low

Suggestions up to commit 84c3696

Category	Suggestion	Impact
General	Flush error writer before returning After printing the stack trace to the terminal's error writer, the writer should be flushed to ensure the error output is actually emitted before the process exits. Without flushing, the error message may be lost if the writer buffers output. server/src/main/java/org/opensearch/bootstrap/OpenSearch.java [126-133] try { return opensearch.main(args, terminal); } catch (StartupException e) { // StartupException has custom printStackTrace formatting (truncates guice frames, etc.). // Catch it here so the process exits rather than hanging, while preserving that output. e.printStackTrace(terminal.getErrorWriter()); + terminal.flush(); return ExitCodes.CODE_ERROR; } Suggestion importance[1-10]: 6 __ Why: Flushing the terminal's error writer before returning is a reasonable defensive practice to ensure error output is not lost due to buffering before process exit. The impact is moderate since the caller may still flush, but explicit flushing improves reliability.	Low

github-actions · 2026-06-21T07:29:10Z

❌ Gradle check result for ee5004b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2026-06-21T08:34:36Z

Persistent review updated to latest commit d669f14

github-actions · 2026-06-21T08:41:00Z

❌ Gradle check result for d669f14: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2026-06-21T09:22:08Z

Persistent review updated to latest commit c6438ee

github-actions · 2026-06-21T09:28:29Z

Persistent review updated to latest commit 8a930f0

github-actions · 2026-06-21T09:31:10Z

Persistent review updated to latest commit 115ca81

github-actions · 2026-06-21T09:39:34Z

❌ Gradle check result for 115ca81: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2026-06-21T10:15:18Z

Persistent review updated to latest commit 8a930f0

github-actions · 2026-06-21T10:21:38Z

❌ Gradle check result for 8a930f0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2026-06-22T04:58:12Z

Persistent review updated to latest commit 0a86cbc

github-actions · 2026-06-22T07:00:12Z

❌ Gradle check result for 0a86cbc: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

msfroh · 2026-07-10T23:30:34Z

+        int status;
+        try {
+            status = main(args, opensearch, Terminal.DEFAULT);
+        } catch (StartupException e) {
+            // StartupException has custom printStackTrace formatting (truncates guice frames, etc.).
+            // Catch it here so the process exits rather than hanging, while preserving that output.
+            e.printStackTrace(System.err);
+            exit(ExitCodes.CODE_ERROR);
+            return;
+        }


[Nitpick] Would it make sense to do something more like:

try { status = main(args, opensearch, Terminal.DEFAULT); } catch (StartupException e) { // StartupException has custom printStackTrace formatting (truncates guice frames, etc.). // Catch it here so the process exits rather than hanging, while preserving that output. e.printStackTrace(System.err); status = ExitCodes.CODE_ERROR; // Continue to the next if-statement to exit(). }

What do you think? It's essentially the same thing (and the same number of lines), but part of me likes having exactly one place where we call exit().

I guess another option would be to modify OpenSearch.main(String[], OpenSearch, Terminal) to be:

static int main(final String[] args, final OpenSearch opensearch, final Terminal terminal) throws Exception { try { return opensearch.main(args, terminal); } catch (StartupException e) { e.printStackTrace(terminal.getErrorWriter()); return ExitCodes.CODE_ERROR; } }

Incidentally, I think the e.printStackTrace(System.err) in your solution should at least be e.printStackTrace(Terminal.DEFAULT.getErrorWriter()). I just read up on System.console() (since the ConsoleTerminal is the one that differentiates from SystemTerminal, which just delegates to System.out and System.err), since I was unfamiliar with it. I think the essential piece is that Console synchronizes its output (and input) methods, so at least a println is guaranteed to be atomic. (I guess lines of a stack trace could get interleaved with other lines.) It looks like localization may also be affected (if the console's locale differs from the system's locale).

went with option 2. moved the catch into the 3-arg main so it uses the injected terminal instead of Terminal.DEFAULT and main(String[]) keeps a single exit() call.

Didn't know about the Console sync and locale difference between System.err and `terminal.getErrorWriter() . Good to know

github-actions · 2026-07-17T18:12:47Z

Persistent review updated to latest commit 7677bcf

github-actions · 2026-07-17T18:22:19Z

❌ Gradle check result for 7677bcf: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2026-07-19T14:08:17Z

Persistent review updated to latest commit d8b7bd5

github-actions · 2026-07-19T14:14:09Z

Persistent review updated to latest commit 84c3696

…tartup failure When a plugin or bootstrap component throws a RuntimeException during startup, OpenSearch.init() wraps it in a StartupException. Previously, StartupException propagated uncaught through execute() and escaped OpenSearch.main(String[]) entirely, bypassing the exit(status) call. This left the JVM process hanging — especially problematic in Docker/k8s where a non-zero exit is required for the orchestrator to detect failure. Fix: catch StartupException in execute() alongside the existing NodeValidationException handler and rethrow it as UserException with ExitCodes.CODE_ERROR so the CLI framework returns a non-OK status and exit(status) is called. Added a regression test in OpenSearchCliTests that simulates a StartupException thrown from init() and asserts ExitCodes.CODE_ERROR is returned. Signed-off-by: Aparajita Pandey <aparajita31pandey@gmail.com>

…Exception Pass the original StartupException as the cause to UserException so the full stack trace is retained for debugging, per reviewer suggestion. Signed-off-by: Aparajita Pandey <aparajita31pandey@gmail.com>

… on startup failure StartupException was designed to escape to main() and be printed via its custom printStackTrace formatter (truncates guice frames, etc.). The bug was that exit() was never called after it escaped, leaving the JVM process hanging when non-daemon threads were still alive. Fix: catch StartupException in main(String[]) — the correct level where System.err is appropriate and process exit decisions belong — call e.printStackTrace(System.err) to preserve the existing formatted output, then exit(CODE_ERROR). The execute() catch is reverted: StartupException is not a UserException and should not flow through the CLI error path. Test updated to assert StartupException propagates through the 3-arg main (test harness path) with the original cause preserved. Signed-off-by: Aparajita Pandey <aparajita31pandey@gmail.com>

Move the StartupException handler from main(String[]) into main(String[], OpenSearch, Terminal) so that: - The terminal parameter is used directly (terminal.getErrorWriter()) instead of hardcoding Terminal.DEFAULT, making it testable. - main(String[]) retains a single exit() call path. Signed-off-by: Aparajita Pandey <aparajita31pandey@gmail.com>

github-actions · 2026-07-19T14:19:32Z

Persistent review updated to latest commit 61cf667

Signed-off-by: Aparajita Pandey <aparajita31pandey@gmail.com>

github-actions · 2026-07-19T14:26:35Z

Persistent review updated to latest commit 23ea016

github-actions · 2026-07-19T14:47:56Z

❌ Gradle check result for 23ea016: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

The 3-arg main() now catches StartupException and returns CODE_ERROR instead of propagating it. Update testStartupExceptionExitsWithCodeError to expect CODE_ERROR exit status and verify the cause message appears in error output, rather than expecting an uncaught StartupException. Signed-off-by: Aparajita Pandey <aparajita31pandey@gmail.com>

github-actions · 2026-07-19T15:12:51Z

Persistent review updated to latest commit 7bfb1fb

github-actions · 2026-07-19T15:20:52Z

Persistent review updated to latest commit 07ac748

github-actions · 2026-07-19T15:31:13Z

❌ Gradle check result for 07ac748: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Aparajita Pandey <aparajita31pandey@gmail.com>

github-actions · 2026-07-19T15:57:35Z

Persistent review updated to latest commit d270b56

github-actions · 2026-07-19T16:56:46Z

✅ Gradle check result for d270b56: SUCCESS

codecov · 2026-07-19T16:57:13Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.52%. Comparing base (c93e9af) to head (d270b56).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #22259      +/-   ##
============================================
+ Coverage     73.43%   73.52%   +0.08%     
- Complexity    76472    76545      +73     
============================================
  Files          6104     6104              
  Lines        346573   346590      +17     
  Branches      49886    49888       +2     
============================================
+ Hits         254514   254833     +319     
+ Misses        71798    71499     -299     
+ Partials      20261    20258       -3

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2026-08-29T06:10:51Z

Persistent review updated to latest commit 2c7bb2b

github-actions · 2026-08-29T06:43:22Z

❌ Gradle check result for 2c7bb2b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

aparajita31pandey requested a review from a team as a code owner June 21, 2026 07:19

aparajita31pandey marked this pull request as draft June 21, 2026 07:56

aparajita31pandey mentioned this pull request Jun 21, 2026

OpenSearch process does not exit when startup fails due to StartupException #22260

Open

aparajita31pandey marked this pull request as ready for review June 21, 2026 09:29

aparajita31pandey force-pushed the fix/startup-exception-process-exit branch from 115ca81 to 8a930f0 Compare June 21, 2026 10:14

msfroh reviewed Jul 10, 2026

View reviewed changes

aparajita31pandey force-pushed the fix/startup-exception-process-exit branch from d8b7bd5 to 84c3696 Compare July 19, 2026 14:12

aparajita31pandey added 4 commits July 19, 2026 19:47

aparajita31pandey force-pushed the fix/startup-exception-process-exit branch from 9eb3606 to 61cf667 Compare July 19, 2026 14:18

Trigger CI

6901b8a

Signed-off-by: Aparajita Pandey <aparajita31pandey@gmail.com>

aparajita31pandey force-pushed the fix/startup-exception-process-exit branch from 887db9c to 7bfb1fb Compare July 19, 2026 15:11

aparajita31pandey force-pushed the fix/startup-exception-process-exit branch from 7bfb1fb to 07ac748 Compare July 19, 2026 15:19

Apply spotless formatting to OpenSearchCliTests

d270b56

Signed-off-by: Aparajita Pandey <aparajita31pandey@gmail.com>

Merge branch 'main' into fix/startup-exception-process-exit

2c7bb2b

Uh oh!

Conversation

aparajita31pandey commented Jun 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Resolves

Root Cause

Fix

Stack trace (reproducer)

Uh oh!

github-actions Bot commented Jun 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Reviewer Guide 🔍

(Review updated until commit 2c7bb2b)

Uh oh!

github-actions Bot commented Jun 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Code Suggestions ✨

Previous suggestions

Uh oh!

github-actions Bot commented Jun 21, 2026

Uh oh!

github-actions Bot commented Jun 21, 2026

Uh oh!

github-actions Bot commented Jun 21, 2026

Uh oh!

github-actions Bot commented Jun 21, 2026

Uh oh!

github-actions Bot commented Jun 21, 2026

Uh oh!

github-actions Bot commented Jun 21, 2026

Uh oh!

github-actions Bot commented Jun 21, 2026

Uh oh!

github-actions Bot commented Jun 21, 2026

Uh oh!

github-actions Bot commented Jun 21, 2026

Uh oh!

github-actions Bot commented Jun 22, 2026

Uh oh!

github-actions Bot commented Jun 22, 2026

Uh oh!

msfroh Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

aparajita31pandey Jul 19, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 17, 2026

Uh oh!

github-actions Bot commented Jul 17, 2026

Uh oh!

github-actions Bot commented Jul 19, 2026

Uh oh!

github-actions Bot commented Jul 19, 2026

Uh oh!

github-actions Bot commented Jul 19, 2026

Uh oh!

github-actions Bot commented Jul 19, 2026

Uh oh!

github-actions Bot commented Jul 19, 2026

Uh oh!

github-actions Bot commented Jul 19, 2026

Uh oh!

github-actions Bot commented Jul 19, 2026

Uh oh!

github-actions Bot commented Jul 19, 2026

Uh oh!

github-actions Bot commented Jul 19, 2026

Uh oh!

github-actions Bot commented Jul 19, 2026

Uh oh!

codecov Bot commented Jul 19, 2026

Codecov Report

Uh oh!

github-actions Bot commented Aug 29, 2026

Uh oh!

github-actions Bot commented Aug 29, 2026

Uh oh!

Reviewers

aparajita31pandey commented Jun 21, 2026 •

edited

Loading

github-actions Bot commented Jun 21, 2026 •

edited

Loading

(Review updated until commit `2c7bb2b`)

github-actions Bot commented Jun 21, 2026 •

edited

Loading