Is your feature request related to a problem? Please describe.
Currently, uss_qualifier has a binary notion of "success" for test scenarios and groups of test scenarios (suites, action generators). But, this notion is confusing because there are many different potential definitions for success. For example, uss_qualifier is successful as a tool if it completes the test run as designed and does not crash. Participant X is successful if their system meets the requirements identified by the test designer as necessary to verify via uss_qualifier. But, the various successful and success attributes on test scenarios, suites, and action generators do not match either of those definitions. Instead, they mean "No participant has any findings in this portion of the test run". There are many problems with this definition, including:
- Low-severity findings explicitly do not indicate failure to meet requirements (but will cause these boolean flags to be false)
- Some findings may be acceptable for a successful participant, even Medium or High severity findings
- A failure by Participant X in a particular scenario does not mean Participant Y has any failures in that particular scenario
- A failure by Participant X in a particular scenario does not mean uss_qualifier failed to perform as designed in the test scenario
- A failure by Participant X in a particular scenario does not mean the test scenario did not complete all its intended steps (it did not necessarily stop early)
Describe the solution you'd like
Remove all participant-dependent notions of success from uss_qualifier, except when explicitly defined by the test designer, such as in the tested requirements artifact.
Describe alternatives you've considered
This misleading concept of "success" has not yet caused any acute problems, so perhaps we could leave it in with longer explanations as to what it means exactly. But, it seems like this risks future confusion from regulators and users, so it would be better to pre-emptively solve the problem rather than risk reputational impact from solving it reactively.
Is your feature request related to a problem? Please describe.
Currently, uss_qualifier has a binary notion of "success" for test scenarios and groups of test scenarios (suites, action generators). But, this notion is confusing because there are many different potential definitions for success. For example, uss_qualifier is successful as a tool if it completes the test run as designed and does not crash. Participant X is successful if their system meets the requirements identified by the test designer as necessary to verify via uss_qualifier. But, the various
successfulandsuccessattributes on test scenarios, suites, and action generators do not match either of those definitions. Instead, they mean "No participant has any findings in this portion of the test run". There are many problems with this definition, including:Describe the solution you'd like
Remove all participant-dependent notions of success from uss_qualifier, except when explicitly defined by the test designer, such as in the tested requirements artifact.
Describe alternatives you've considered
This misleading concept of "success" has not yet caused any acute problems, so perhaps we could leave it in with longer explanations as to what it means exactly. But, it seems like this risks future confusion from regulators and users, so it would be better to pre-emptively solve the problem rather than risk reputational impact from solving it reactively.