Add support for 'continue-on-error' in scenario steps and UI compon… by atti92 · Pull Request #112 · openutm/verification

atti92 · 2026-03-20T15:00:43Z

Introduced 'continue_on_error' property to control behavior on step failure.
Updated YAML conversion functions to include 'continue-on-error'.
Enhanced UI components to allow setting 'Continue on failure' for steps.
Added tests to verify the correct parsing and functionality of 'continue_on_error'.

Copilot

Pull request overview

Adds a new per-step failure policy (on_step_result_fail) to allow scenarios (and the web editor) to continue executing subsequent steps after a failure, addressing cases where cleanup/teardown must still run.

Changes:

Introduces on_step_result_fail: continue|stop in execution models and runner control flow, and tracks this via StepResult.continue_on_fail.
Updates condition evaluation (success() / failure()) to ignore failures marked as continue-on-fail.
Extends the web editor (types, YAML<->graph conversion, UI controls/badges) and adds tests + scenario YAML updates demonstrating always() and continue-on-fail.

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
web-editor/src/utils/scenarioConversion.ts	Maps `on_step_result_fail` between YAML and graph node data.
web-editor/src/utils/tests/scenarioConversion.test.ts	Adds conversion tests for emitting/omitting `on_step_result_fail`.
web-editor/src/types/scenario.ts	Adds `onStepResultFail` / `on_step_result_fail` typing.
web-editor/src/styles/Node.module.css	Styles new “continue” badge and inline condition text.
web-editor/src/components/ScenarioEditor/PropertiesPanel.tsx	Adds checkbox UI for continue-on-failure and tweaks condition help text.
web-editor/src/components/ScenarioEditor/CustomNode.tsx	Renders continue-on-fail badge and displays step `if` condition under the title.
web-editor/src/components/ScenarioEditor/tests/PropertiesPanel.test.tsx	Updates props for new `onUpdateOnStepResultFail` callback.
web-editor/src/components/ScenarioEditor.tsx	Wires node updates for `onStepResultFail`.
tests/test_group_execution.py	Adds YAML parsing tests for `on_step_result_fail` (including inside groups).
tests/test_conditions.py	Adds tests ensuring `success()`/`failure()` ignore continue-on-fail failures.
src/openutm_verification/server/runner.py	Implements continue-on-failure behavior during scenario/group execution.
src/openutm_verification/server/router.py	Returns scenario JSON using aliases (e.g., `if`).
src/openutm_verification/core/reporting/reporting_models.py	Adds `continue_on_fail` to `StepResult`.
src/openutm_verification/core/execution/definitions.py	Adds `on_step_result_fail` to `StepDefinition` with default `stop`.
src/openutm_verification/core/execution/conditions.py	Excludes continue-on-fail failures when determining last step status.
scenarios/*/.yaml	Updates scenarios to use `if: always()` and one use of `on_step_result_fail: continue`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

RomanPszonka · 2026-03-30T20:41:57Z

can you add some screenshots from UI please? @atti92

Copilot

Pull request overview

Copilot reviewed 31 out of 31 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…ents - Introduced 'on_step_result_fail' property to control behavior on step failure. - Updated YAML conversion functions to include 'on_step_result_fail'. - Enhanced UI components to allow setting 'on_step_result_fail' for steps. - Added tests to verify the correct parsing and functionality of 'on_step_result_fail'.

Copilot

Pull request overview

Copilot reviewed 32 out of 32 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (1)

web-editor/src/utils/scenarioConversion.ts:86

The new continue-on-error YAML parsing (step['continue-on-error'] -> node.data.continueOnError) is not covered by the web-editor Vitest suite (current tests only validate convertGraphToYaml). Add tests for convertYamlToGraph to ensure both regular steps and group steps correctly set continueOnError when the YAML includes/omits continue-on-error.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-10T19:14:36Z

                    position: { x: CONVERSION_LAYOUT.groupChildOffset, y: groupInternalYPos - yPos }, // Relative to parent
                    data: {
                        label: groupStep.step,
                        stepId: groupStep.id || groupStep.step,
                        operationId: groupOperation?.id,
                        description: groupStep.description || groupOperation?.description || '',
                        phase: groupOperation?.phase,
-                        parameters: groupParameters
+                        parameters: groupParameters,
+                        continueOnError: groupStep['continue-on-error'],
                    }


TypeScript strict mode: groupStep is typed as GroupStepDefinition (see web-editor/src/types/scenario.ts), which currently doesn't declare a 'continue-on-error' field. Indexing with groupStep['continue-on-error'] will fail compilation; the group-step type needs to be extended to include this property (and any other step-level fields supported in groups).

Copilot · 2026-04-10T19:14:36Z

+                            [groupName]: {
+                                ...group,
+                                steps: group.steps.map(s =>
+                                    (s.id || s.step) === stepId
+                                        ? { ...s, 'continue-on-error': value || undefined }
+                                        : s
+                                ),


This update writes a 'continue-on-error' field into group.steps objects, but currentScenarioGroups is typed as Record<string, GroupDefinition> and GroupDefinition.steps items are GroupStepDefinition (which does not currently include 'continue-on-error'). In strict TS this will trigger an excess-property/type error; update the group step type to include 'continue-on-error' (and keep it consistent with what the backend supports).

Copilot · 2026-04-10T19:14:36Z

+    runner.config = type(
+        "C",
+        (),
+        {
+            "suites": {},
+            "reporting": type("R", (), {"output_dir": "/tmp/reports", "formats": []})(),
+        },
+    )()


These integration tests call SessionManager.run_scenario, which triggers setup_logging() (it calls logger.remove() globally) and creates timestamp-based output dirs under a fixed /tmp/reports path. That global logger reconfiguration + filesystem writes can make the wider test suite flaky (especially under parallel runs). Consider patching setup_logging to a no-op for these tests and using tmp_path (or pre-setting runner.current_output_dir/current_start_time) to avoid touching real paths and to isolate runs.

Copilot · 2026-04-10T19:14:37Z

                else:
                    group_results = await self._execute_group(step, scenario)
                    if any(r.status == Status.FAIL for r in group_results):
-                        logger.error(f"Group '{step.id}' failed, breaking scenario")
-                        break
+                        if should_continue_on_error:
+                            logger.warning(f"Group '{step.id}' failed, but continue-on-error=true — continuing")
+                            self._mark_results_continue_on_error(group_results)
+                        else:
+                            logger.error(f"Group '{step.id}' failed, breaking scenario")
+                            break


continue_on_error on group-internal steps is currently not respected at the scenario level. _execute_group can mark a failed StepResult as continue_on_error=True, but run_scenario still treats the group as failed whenever any(r.status == FAIL for r in group_results) and will break the scenario (unless the group reference step also has continue-on-error). To support continue-on-error inside groups, the failure check here (and for loop-for-group) should ignore FAIL results where r.continue_on_error is true (e.g., only break on FAIL && !continue_on_error).

Copilot · 2026-04-10T19:14:37Z

                if step.loop:
                    loop_results = await self._execute_loop_for_group(step, scenario)
                    if loop_results and any(r.status == Status.FAIL for r in loop_results):
-                        logger.error(f"Loop for group '{step.id}' failed, breaking scenario")
-                        break
+                        if should_continue_on_error:
+                            logger.warning(f"Loop for group '{step.id}' failed, but continue-on-error=true — continuing")
+                            self._mark_results_continue_on_error(loop_results)
+                        else:
+                            logger.error(f"Loop for group '{step.id}' failed, breaking scenario")
+                            break


Same as the non-loop group case: loop_results may include FAIL results that were explicitly allowed to fail (StepResult.continue_on_error=True from group-internal continue-on-error), but this condition uses any(r.status == FAIL ...) and will treat that as a hard failure. Consider checking only for failures that are not marked continue_on_error when deciding whether to break the scenario.

atti92 requested review from RomanPszonka and hrishiballal March 20, 2026 18:01

RomanPszonka requested a review from Copilot March 30, 2026 20:36

Copilot started reviewing on behalf of RomanPszonka March 30, 2026 20:36 View session

RomanPszonka reviewed Mar 30, 2026

View reviewed changes

Comment thread scenarios/flight-declarations/add_flight_declaration.yaml

Copilot AI reviewed Mar 30, 2026

View reviewed changes

Comment thread src/openutm_verification/server/runner.py Outdated

Comment thread src/openutm_verification/server/router.py

Comment thread web-editor/src/components/ScenarioEditor/PropertiesPanel.tsx Outdated

Comment thread web-editor/src/components/ScenarioEditor/PropertiesPanel.tsx Outdated

RomanPszonka requested changes Mar 30, 2026

View reviewed changes

atti92 requested a review from RomanPszonka March 30, 2026 22:21

atti92 self-assigned this Mar 30, 2026

atti92 changed the title ~~Add support for 'on_step_result_fail' in scenario steps and UI compon…~~ Add support for 'continue-on-error' in scenario steps and UI compon… Mar 30, 2026

RomanPszonka requested a review from Copilot March 30, 2026 22:24

Copilot started reviewing on behalf of RomanPszonka March 30, 2026 22:25 View session

Copilot AI reviewed Mar 30, 2026

View reviewed changes

Comment thread web-editor/src/components/ScenarioEditor/PropertiesPanel.tsx Outdated

Comment thread web-editor/src/utils/scenarioConversion.ts

Comment thread src/openutm_verification/server/runner.py

atti92 added 2 commits April 10, 2026 20:48

resolve comments

bd2a8dd

atti92 force-pushed the issue-107 branch from 1d5766e to bd2a8dd Compare April 10, 2026 18:48

atti92 added 2 commits April 10, 2026 21:05

resolve comments

ff51208

update

5fca634

atti92 requested a review from Copilot April 10, 2026 19:07

Copilot started reviewing on behalf of atti92 April 10, 2026 19:07 View session

Copilot AI reviewed Apr 10, 2026

View reviewed changes

hrishiballal approved these changes Apr 13, 2026

View reviewed changes

atti92 added 2 commits April 25, 2026 19:34

Merge branch 'main' into issue-107

c94a2c1

fix unit test

1cc1abf

atti92 merged commit 026b541 into main Apr 25, 2026
2 checks passed

atti92 deleted the issue-107 branch April 25, 2026 18:09

Uh oh!

Conversation

atti92 commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RomanPszonka commented Mar 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

atti92 commented Mar 20, 2026 •

edited

Loading