feat: add test-runner functionality

[svishnus]

• Test Runner integration with CI

[shwestrick]

Would it be possible to run tests by checking out parallel-ml-bench only within the github action? To avoid duplicating that code here. We can let parallel-ml-bench be the definitive source of the benchmark codes.

[Forthoney]

make test run all the tests with mlton in a push-button manner. The same should apply for make test SMLC=../build/bin/mpl, but some tests like "dedup" apparently use deprecated functionality. If @svishnus can look at such tests and replace the deprecated functionality with their modern counterparts I think we would be ready to publish this PR.

[shwestrick]

@svishnus @Forthoney my one request would just be to keep the benchmark sources outside of MPLLang/mpl version control. It would be a bit of a nightmare to have two separate versions of the benchmarks to maintain. We can let MPLLang/parallel-ml-bench be the source of the benchmarks.

[Forthoney]

@svishnus @Forthoney my one request would just be to keep the benchmark sources outside of MPLLang/mpl version control. It would be a bit of a nightmare to have two separate versions of the benchmarks to maintain. We can let MPLLang/parallel-ml-bench be the source of the benchmarks.

I was actually thinking of freezing the scripts we have in this repo and think of them as rudimentary tests (compiling properly, not segfaulting) rather than benchmarks. It just happened to be that the benchmarks repo had the most diverse set of realistic programs.

If we have new benchmarks or optimizations to the algos in the benchmarks, these would not reflect it since they are for correctness.

[shwestrick]

Gotcha -- this makes sense but I worry that the code duplication will become a problem. Inevitably, updates will happen in one place and we will want to port them across... we'll end up with duplicated work and/or inconsistent versions.

Rather than vendor the frozen benchmarks, we could accomplish the same by hardcoding a particular commit of parallel-ml-bench and git cloning. This would (I think?) be a fairly small change from what you currently have and would allow for updating the commit hash in the future if we update benchmarks.

And, this would move in the direction of making parallel-ml-bench be more useful. We can extend it with additional functionality. Already it is good for performance tests; we could incrementally make it better for correctness tests, too. (Some of the benchmarks already have a --check flag; we could incrementally add this to more benchmarks.)