Feature Request: Persist comparison tolerances with test metadata #15

Activity

1. davidchall commented on Jul 13, 2017

   

   davidchall

   on Jul 13, 2017

   Owner

   More actions

   Hi Michael,

   Just to confirm, you are asking for comparison tolerance fields in the test configuration files? This would be nice because tests can then have personalized tolerances.

   If we do this, I'd probably prefer to remove the ability to set the comparison tolerance from the command line. This would need to override the tolerance in all tests, which is just asking for trouble.
2. michaeltryby commented on Jul 13, 2017

   

   michaeltryby

   on Jul 13, 2017

   Author

   More actions

   @davidchall Yes like you say, comparison tolerance fields in the test configuration file. I also agree that setting the comparison tolerance from the command line could become problematic in that scenario.

   How were you thinking of implementing this? I was thinking maybe add another optional field to the test configuration file. Perhaps, a dictionary were the key was the file type and the value was a list of tolerance values.

   It would be nice to enclose tolerance values in a data structure. That way test functions with different input parameters besides rtol and atol could also be supported. Make sense?
3. davidchall commented on Feb 7, 2018

   

   davidchall

   on Feb 7, 2018

   Owner

   More actions

   I've been thinking about this some more. The only truly comprehensive way to do this would be to change the Test config files in a backwards-incompatible way, so we can add metadata to each result file. Then (rtol, atol) values can be specified independently for each comparison. This can also allow custom arguments to be sent to the comparison routine. I really like this flexibility. The downside is that it would require an awful lot more effort to create a Test config file.

   Something like:

   {
       "name": "Scoring_01",
       "version": "1.0",
       "description": "Basic test shooting a 6cm diameter proton beam into a water phantom.",
       "args": [
           "Scoring_01.txt"
       ],
       "input_files": [
           "Scoring_01.txt",
           "GlobalParameters.txt"
       ],
       "output_files": [
           {
               "filename": "Dose.csv",
               "compare_routine": "topas binned",
               "compare_args": {
                   "rtol": 0.01,
                   "atol": 0
               }
           }
       ]
   }