Test improvements

[reidsunderland]

These are some changes from a few months ago that I never created a PR for. I've done some more testing on them now and I'm pretty sure the changes are good.

The problem with the test code itself is that it did mkdir dir_test/ and rm -r dir_test/, both with trailing slash, and both would get rejected by sr3 so the directory was never created and never removed.

The tests didn't catch MetPX/sarracenia#1503 because the directory comparison was only comparing files (using find -type f). Sometimes shim_dirB would have some extra empty directories that were not present in shim_dirA, but the test never found those. I updated the tree comparison to catch these cases, and moved it into a separate file that can be called from all tests.

I also moved the wait for retries code into a separate script, and added some extra cases where directory fileOps are done with trailing slashes.

The tests won't pass with these changes until corresponding changes are made in sr3.

[petersilva]

fwiw... I tried running it on my laptop... it fails all the tests, except the first one... my environment changes a lot... I don't know what's wrong, it could very easily be my problem, but I hasn't been easy to isolate so far.

[reidsunderland]

Sorry for the confusion. The tests are expected to fail right now, they are updated to catch problems that currently exist and were not caught with the previous versions.

The patch for MetPX/sarracenia#1503 was merged into the sr3 development branch on Jan 21.

But MetPX/sarracenia#1507 still needs to be fixed to get the sr3c tests fully passing. Branch issue1507 for sr3 has a change that I'm working on, but I get inconsistent results when using it. It seems to be a timing issue in sr3, not an sr3c problem, so that's why I thought it made sense to create the sr3c PR.

[reidsunderland]

With the addition of the sleep, the tests except test_shim_race_conditions are now passing consistently for me.

test_shim_race_conditions is failing with RESULT: BAD tree differences in 0 files, and 0 links and these other differences: Only in shim_dirB: dirone which is caused by the sr3 Python problem described here: MetPX/sarracenia#1507 (comment)