Add proper regression test suite

[RobinWebbers]

It would be great to have a regression test suite. There are a bunch of things we've added or are planning to add and insight on how it affects performance is becoming increasingly important!