-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
feature: Reverse order #1807
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+304
−154
Merged
feature: Reverse order #1807
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
395fcc3
Add initial TDD view for the reverse ordering
luke-hill 62e3774
Fix up positive scenario to have full timestamp sanitized
luke-hill 12a73df
Add in logic for new reversal filter
luke-hill 743ce09
Fix name of profile and ensure autoload
luke-hill 3d164fc
Fixed test for reverse order regular
luke-hill a90a2bc
Fix up test for reverse with a skipped scenario
luke-hill c27f670
Newly generated todo file
luke-hill 1ef0cdf
Add changelog
luke-hill 20dfbb9
Merge branch 'main' into feature/reverse_order
luke-hill File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,244 @@ | ||
| Feature: Ordering | ||
|
|
||
| Cucumber can run scenarios in different orders. By default, scenarios are run in the order they | ||
| appear in the feature files. Use the `--order random` switch to run scenarios in random order. | ||
|
|
||
| You can also run cucumber in a reverse order using `--order reverse`. | ||
|
|
||
| Using different ordering can help you detect situations where you have state | ||
| leaking between scenarios, which can cause flickering or fragile tests. | ||
|
|
||
| If you do find a random run that exposes dependencies between your tests, | ||
| you can reproduce that run by using the seed that's printed at the end of | ||
| the test run. | ||
|
|
||
| For a given seed, the order of scenarios is constant, i.e. if step A runs | ||
| before step B, it will always run before step B even if other steps are | ||
| skipped. | ||
|
|
||
| Background: | ||
| Given a file named "features/bad_practice_part_1.feature" with: | ||
| """ | ||
| Feature: Bad practice, part 1 | ||
|
|
||
| Scenario: Set state | ||
| Given I set some state | ||
|
|
||
| Scenario: Depend on state from a preceding scenario | ||
| When I depend on the state | ||
| """ | ||
| And a file named "features/bad_practice_part_2.feature" with: | ||
| """ | ||
| Feature: Bad practice, part 2 | ||
|
|
||
| Scenario: Depend on state from a preceding feature | ||
| When I depend on the state | ||
| """ | ||
| And a file named "features/unrelated.feature" with: | ||
| """ | ||
| Feature: Unrelated | ||
|
|
||
| @skipme | ||
| Scenario: Do something unrelated | ||
| When I do something | ||
| """ | ||
| And a file named "features/step_definitions/steps.rb" with: | ||
| """ | ||
| Given('I set some state') do | ||
| $global_state = 'set' | ||
| end | ||
|
|
||
| Given('I depend on the state') do | ||
| raise 'I expect the state to be set!' unless $global_state == 'set' | ||
| end | ||
|
|
||
| Given('I do something') do | ||
| end | ||
| """ | ||
|
|
||
| Scenario: Run scenarios in order | ||
| When I run `cucumber` | ||
| Then it should pass | ||
| And the stdout should contain exactly: | ||
| """ | ||
| Feature: Bad practice, part 1 | ||
|
|
||
| Scenario: Set state # features/bad_practice_part_1.feature:3 | ||
| Given I set some state # features/step_definitions/steps.rb:1 | ||
|
|
||
| Scenario: Depend on state from a preceding scenario # features/bad_practice_part_1.feature:6 | ||
| When I depend on the state # features/step_definitions/steps.rb:5 | ||
|
|
||
| Feature: Bad practice, part 2 | ||
|
|
||
| Scenario: Depend on state from a preceding feature # features/bad_practice_part_2.feature:3 | ||
| When I depend on the state # features/step_definitions/steps.rb:5 | ||
|
|
||
| Feature: Unrelated | ||
|
|
||
| @skipme | ||
| Scenario: Do something unrelated # features/unrelated.feature:4 | ||
| When I do something # features/step_definitions/steps.rb:9 | ||
|
|
||
| 4 scenarios (4 passed) | ||
| 4 steps (4 passed) | ||
| 0m0.012s | ||
| """ | ||
|
|
||
| @global_state | ||
| Scenario: Run scenarios randomized | ||
| When I run `cucumber --order random:41544 -q` | ||
| Then it should fail | ||
| And the stdout should contain exactly: | ||
| """ | ||
| Feature: Bad practice, part 1 | ||
|
|
||
| Scenario: Depend on state from a preceding scenario | ||
| When I depend on the state | ||
| I expect the state to be set! (RuntimeError) | ||
| ./features/step_definitions/steps.rb:6:in `"I depend on the state"' | ||
| features/bad_practice_part_1.feature:7:in `I depend on the state' | ||
|
|
||
| Feature: Unrelated | ||
|
|
||
| @skipme | ||
| Scenario: Do something unrelated | ||
| When I do something | ||
|
|
||
| Feature: Bad practice, part 2 | ||
|
|
||
| Scenario: Depend on state from a preceding feature | ||
| When I depend on the state | ||
| I expect the state to be set! (RuntimeError) | ||
| ./features/step_definitions/steps.rb:6:in `"I depend on the state"' | ||
| features/bad_practice_part_2.feature:4:in `I depend on the state' | ||
|
|
||
| Feature: Bad practice, part 1 | ||
|
|
||
| Scenario: Set state | ||
| Given I set some state | ||
|
|
||
| Failing Scenarios: | ||
| cucumber features/bad_practice_part_1.feature:6 | ||
| cucumber features/bad_practice_part_2.feature:3 | ||
|
|
||
| 4 scenarios (2 failed, 2 passed) | ||
| 4 steps (2 failed, 2 passed) | ||
|
|
||
| Randomized with seed 41544 | ||
| """ | ||
|
|
||
| @force_legacy_loader | ||
| Scenario: Rerun scenarios randomized | ||
| When I run `cucumber --order random --format summary` | ||
| And I rerun the previous command with the same seed | ||
| Then the output of both commands should be the same | ||
|
|
||
| @global_state | ||
| Scenario: Run scenarios randomized with some skipped | ||
| When I run `cucumber --tags "not @skipme" --order random:41544 -q` | ||
| Then it should fail with exactly: | ||
| """ | ||
| Feature: Bad practice, part 1 | ||
|
|
||
| Scenario: Depend on state from a preceding scenario | ||
| When I depend on the state | ||
| I expect the state to be set! (RuntimeError) | ||
| ./features/step_definitions/steps.rb:6:in `"I depend on the state"' | ||
| features/bad_practice_part_1.feature:7:in `I depend on the state' | ||
|
|
||
| Feature: Bad practice, part 2 | ||
|
|
||
| Scenario: Depend on state from a preceding feature | ||
| When I depend on the state | ||
| I expect the state to be set! (RuntimeError) | ||
| ./features/step_definitions/steps.rb:6:in `"I depend on the state"' | ||
| features/bad_practice_part_2.feature:4:in `I depend on the state' | ||
|
|
||
| Feature: Bad practice, part 1 | ||
|
|
||
| Scenario: Set state | ||
| Given I set some state | ||
|
|
||
| Failing Scenarios: | ||
| cucumber features/bad_practice_part_1.feature:6 | ||
| cucumber features/bad_practice_part_2.feature:3 | ||
|
|
||
| 3 scenarios (2 failed, 1 passed) | ||
| 3 steps (2 failed, 1 passed) | ||
|
|
||
| Randomized with seed 41544 | ||
|
|
||
| """ | ||
|
|
||
| @global_state | ||
| Scenario: Run scenarios in reverse order | ||
| When I run `cucumber --order reverse -q` | ||
| Then it should fail | ||
| And the stdout should contain exactly: | ||
| """ | ||
| Feature: Unrelated | ||
|
|
||
| @skipme | ||
| Scenario: Do something unrelated | ||
| When I do something | ||
|
|
||
| Feature: Bad practice, part 2 | ||
|
|
||
| Scenario: Depend on state from a preceding feature | ||
| When I depend on the state | ||
| I expect the state to be set! (RuntimeError) | ||
| ./features/step_definitions/steps.rb:6:in `"I depend on the state"' | ||
| features/bad_practice_part_2.feature:4:in `I depend on the state' | ||
|
|
||
| Feature: Bad practice, part 1 | ||
|
|
||
| Scenario: Depend on state from a preceding scenario | ||
| When I depend on the state | ||
| I expect the state to be set! (RuntimeError) | ||
| ./features/step_definitions/steps.rb:6:in `"I depend on the state"' | ||
| features/bad_practice_part_1.feature:7:in `I depend on the state' | ||
|
|
||
| Scenario: Set state | ||
| Given I set some state | ||
|
|
||
| Failing Scenarios: | ||
| cucumber features/bad_practice_part_2.feature:3 | ||
| cucumber features/bad_practice_part_1.feature:6 | ||
|
|
||
| 4 scenarios (2 failed, 2 passed) | ||
| 4 steps (2 failed, 2 passed) | ||
| """ | ||
|
|
||
| @global_state | ||
| Scenario: Run scenarios in reverse order with some skipped | ||
| When I run `cucumber --tags "not @skipme" --order reverse -q` | ||
| Then it should fail | ||
| And the stdout should contain exactly: | ||
| """ | ||
| Feature: Bad practice, part 2 | ||
|
|
||
| Scenario: Depend on state from a preceding feature | ||
| When I depend on the state | ||
| I expect the state to be set! (RuntimeError) | ||
| ./features/step_definitions/steps.rb:6:in `"I depend on the state"' | ||
| features/bad_practice_part_2.feature:4:in `I depend on the state' | ||
|
|
||
| Feature: Bad practice, part 1 | ||
|
|
||
| Scenario: Depend on state from a preceding scenario | ||
| When I depend on the state | ||
| I expect the state to be set! (RuntimeError) | ||
| ./features/step_definitions/steps.rb:6:in `"I depend on the state"' | ||
| features/bad_practice_part_1.feature:7:in `I depend on the state' | ||
|
|
||
| Scenario: Set state | ||
| Given I set some state | ||
|
|
||
| Failing Scenarios: | ||
| cucumber features/bad_practice_part_2.feature:3 | ||
| cucumber features/bad_practice_part_1.feature:6 | ||
|
|
||
| 3 scenarios (2 failed, 1 passed) | ||
| 3 steps (2 failed, 1 passed) | ||
| """ | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.