Repository navigation
fix: Restart Backstage when startup fails to reach ready - #82
Merged
Merged
Conversation
ecv
force-pushed
the
fix/exit-on-startup-failure
branch
from
September 16, 2026 01:10
e002be7 to
ba33385
Compare
ecv
force-pushed
the
fix/exit-on-startup-failure
branch
from
September 17, 2026 23:44
ba33385 to
61bdd1f
Compare
ecv
marked this pull request as ready for review
September 17, 2026 23:49
ecv
marked this pull request as draft
September 18, 2026 00:00
A plugin that failed during startup left the process alive and serving nothing: the startup error surfaced as an unhandled rejection that the backend only logged, readiness stayed at 503, and liveness kept passing, so the pod was never restarted. Production sat in that state for nearly three hours after a momentary database outage. Key changes: - Await the backend start and exit non-zero when it fails, so a failed startup becomes a container restart rather than a pod that serves nothing - Replace the root health service so liveness fails once the backend has never reached ready within a bounded deadline, leaving liveness untouched for a backend that has started - Wrap the database service so a plugin retries an unreachable database for a bounded window before giving up, which absorbs a short blip without costing a restart The retry window closes well before the liveness deadline, so a database that stays away still ends the process rather than holding it open.
The retry wrapper called the same plugin database service on every attempt, and that service memoizes its first connection promise for the lifetime of its manager. A rejected promise stays in that cache, so every later attempt resolved the original failure instantly and no second connection was ever opened. A database that returned after a second still failed the startup it was meant to survive. Key changes: - Build a database manager per attempt, so each retry carries its own cache and opens a real connection - Cover the case with a delegate that memoizes its first failure, which is the behaviour the upstream manager has and the previous mock did not
ecv
force-pushed
the
fix/exit-on-startup-failure
branch
from
September 18, 2026 00:02
61bdd1f to
a5de69e
Compare
ecv
marked this pull request as ready for review
September 18, 2026 00:12
ecv
requested review from
0xmc,
JoseSzycho,
kevwilliams,
mattdjenkinson and
scotwells
September 18, 2026 00:14
ecv
enabled auto-merge
September 18, 2026 00:16
kevwilliams
approved these changes
Sep 18, 2026
kevwilliams
left a comment
kevwilliams
left a comment
There was a problem hiding this comment.
Fixes a real incident-causing gap: a failed plugin startup used to leave the process alive serving nothing (readiness 503, liveness 200 forever). Now backend.start() failure exits the process, a new bounded-deadline root health service fails liveness only if the backend never reaches ready within 5 minutes (so it won't flap on a legitimately slow start), and the DB retry wrapper rebuilds its delegate per attempt to avoid replaying a memoized first failure. Retry window (30s) closes well inside the liveness deadline (300s), and tests cover both the healthy and failed-startup paths with real assertions, not just no-throw checks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Backstage was down for nearly three hours because a brief database outage at startup left it running while serving nothing.
The plugins that could not reach the database gave up and asked for a shutdown, yet the process stayed alive, readiness kept answering 503, and liveness kept answering 200, so Kubernetes had no reason to restart the pod.
Startup now ends the process when a plugin fails, liveness fails once the backend has never reached ready within a bounded deadline, and a plugin waits out a short database absence, closing well inside the liveness deadline, before it gives up.
This reaches production only once the deployment repository bumps to an image that carries it.
Test plan
Fixes #81