Retry "Launch Simulation" on server failure

[ezio-melotti]

After clicking on the LAUNCH SIMULATION button in the config wizard, we might sometime get a 5xx error from the server. When that happens, instead of showing a popup with the error, it would be better to tell the user that our server are busy, and automatically retry the request after 10-15 seconds. If after a few attempts we still get an error, a regular popup should tell the user that all our server are busy.

Since we already have a timeout modal, it would be nice to reuse it for the countdown before the retry. I looked into it briefly, however the timeout is not-blocking and returns immediately, so it didn't work.

Click to see a diff of my (not-working) attempt, which outlines the implementation.

diff --git a/src/views/ConfigurationView.vue b/src/views/ConfigurationView.vue
index 96ccb69..a628fe5 100644
--- a/src/views/ConfigurationView.vue
+++ b/src/views/ConfigurationView.vue
@@ -167,7 +167,7 @@ export default {
     },
     methods: {
         ...mapMutations(['SETGAMEID']),
-        ...mapActions('modal', ['alert']),
+        ...mapActions('modal', ['alert', 'timeout']),
 
         toggleMenu() {
             this.menuActive = !this.menuActive
@@ -191,12 +191,19 @@ export default {
         },
 
         handleAxiosError(error) {
-            console.error(error)
+            //console.error(error)
             if (error.response && error.response.status === 401) {
                 this.alert('Please log in again to continue.')
                 this.$router.push('entry')
             } else {
-                this.alert(error)
+                console.log('showing timeout alert')
+                //this.alert('Our servers are currently busy.\nClick "Ok" to try again.')
+                this.timeout({
+                    message: ('Our servers are currently busy.\n' +
+                              'Retrying in 10'),
+                    secondsLeft: 10,
+                    timeoutCallback: () => console.log('timeout done'),
+                })
             }
         },
 
@@ -282,26 +289,30 @@ export default {
                 this.alert(err_msg)
                 return  // abort if there are any errors
             }
-            try {
-                this.awaiting_response = true
-                // Wait for the new game to be created
-                const response = await axios.post('/new_game', configParams)
-                // store the game ID and full game_config from the response
-                this.SETGAMEID(response.data.game_id)
-                this.setGameParams({
-                    game_config: response.data.game_config,
-                    currency_desc: response.data.currency_desc,
-                })
-                this.currentMode = this.currentMode !== 'kiosk' ? 'sim' : 'kiosk'
-                this.isLive = false
-                this.loadFromSimData = false
-                // If all is well then move the user to the dashboard screen
-                this.$router.push('dashboard')
-            } catch (error) {
-                this.handleAxiosError(error)
-            } finally {
-                this.awaiting_response = false
+            for (let attempt = 0; attempt < 3; attempt++) {
+                console.log(`attempt ${attempt}`)
+                try {
+                    this.awaiting_response = true
+                    // Wait for the new game to be created
+                    const response = await axios.post('/new_game', configParams)
+                    // store the game ID and full game_config from the response
+                    this.SETGAMEID(response.data.game_id)
+                    this.setGameParams({
+                        game_config: response.data.game_config,
+                        currency_desc: response.data.currency_desc,
+                    })
+                    this.currentMode = this.currentMode !== 'kiosk' ? 'sim' : 'kiosk'
+                    this.isLive = false
+                    this.loadFromSimData = false
+                    // If all is well then move the user to the dashboard screen
+                    this.$router.push('dashboard')
+                } catch (error) {
+                    this.handleAxiosError(error)
+                } finally {
+                    this.awaiting_response = false
+                }
             }
+            this.alert('All our servers are currently busy.\nPlease try again later.')
         },
     },
 }

This could be implemented in launchSimulation and/or handleAxiosError but, if possible, it should avoid duplicating code (both the one that handles the error, and the one that sends the request). It might also require an udpate to the timeout modal to make it async, so that you can await until the 10-15 pass before retrying and doing another iteration of the loop.

[hiyaryan]

Is there a simple way to replicate this server error? I tried on the ngs and it seems to be working without error. I do get a 500 error when I run simoc locally though (see below). Is this the same one you're referring to?

[hiyaryan]

When I press "Launch Simulation" when running locally, this vite error pops up too. Not sure is its related to the same error though.

[hiyaryan]

I can't seem to replicate the same error on ngs but when I launch SIMOC from ngs on a new browser instance, I get an uncaught promise error. Possibly related to the 500 error thrown locally above but instead of the trace coming from ConfigurationView its from BaseDashboard.

[ezio-melotti]

Both on NGS and locally there should be no errors, so to test this you can just shut down the backend after you logged in so that attempting to launch a simulation will fail.

When you do this, the frontend should tell you that the servers are busy and do a countdown, try again, show the same error/countdown, try again, and eventually give up and tell you that all the servers are busy.

[ezio-melotti]

To clarify, this issue is not about handling a specific error that I'm seeing -- it's just an added measure to handle potential errors that might be raised by the server when e.g. it's under heavy load.

When I press "Launch Simulation" when running locally, this vite error pops up too.

This might be because you have outdated simdata files. You can just delete the frontend/src/assets/simdata dir, and when you launch simoc-web.py shell it should download them again. Also keep in mind that if you select the presets, the frontend won't request step data to the backend, so you won't get a 500 even if you shut down the backend.

when I launch SIMOC from ngs on a new browser instance, I get an uncaught promise error.

This seems unrelated, and I haven't seen this error. Did all the panels look ok or was there something broken?

[hiyaryan]

To clarify, this issue is not about handling a specific error that I'm seeing -- it's just an added measure to handle potential errors that might be raised by the server when e.g. it's under heavy load.

Thanks for the clarification.

You can just delete the frontend/src/assets/simdata dir, and when you launch simoc-web.py shell it should download them again.

I'll try this. However, I'll also be sure not to use presets when testing.

Did all the panels look ok or was there something broken?

All the panels seemed to have been working properly. I tried producing the error again but couldn't. If I do I'll take a closer look at the error.