Integrate BI tests with Runt & CI

[ngernest]

This PR addresses Kevin's comment here in #270, adding BI Runt tests to our testing framework / CI:

Can you make a separate PR for adding the bi to the tests and to CI? Then we can iterate on how to best integrate this with the new testing infrastrucutre.

Most files changed in this PR are new .expect files containing the BI output, the key code changes to review are:

• scripts/generate_runt_config.py (I followed Nikil's instructions in Migrate to Runt #252 to add support for Runt tests here. Note that the BI uses the same set of test cases as the monitor, i.e. the BI test suite uses MONITOR_CASES in test_catalog.py. This is possible since BI and monitor share largely the same CLI args.)
• bi/src/main.rs (described below)

There are some minor changes to the bi executable in this PR:

• Added a new CLI arg --color never so that error messages are printed in plaintext (no color) in .expect files, like the monitor
• Following the advice here (https://users.rust-lang.org/t/suppress-panic-message/6303/3), I added a panic hook so that when bi panics, instead of the default Rust error message:

thread 'main' (11301631) panicked at bi/src/bi.rs:628:21:
[add_busy_wait@00!] Cannot fork at step zero!
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

we instead print (and only save in .expect files) the following:

panicked at bi/src/bi.rs:628:21:
[add_busy_wait@00!] Cannot fork at step zero!

This avoids the OS thread ID (the thread 'main' (11301631) ... information) in the default error message from appearing in the .expect files, since the OS thread ID changes every time we run the executable.

Two test infra TODOs for future PRs

1. For the monitor (not BI), when the monitor is expected to fail (e.g. running on a Brave New World buggy waveform), the corresponding .expect file only displays the error code:

---CODE---
124

Ideally, we change this behavior so that monitor error messages are also captured in .expect files (matching what is done for bi in this PR). This is a one-line change in generate_runt_configs.py, but also affects a bunch of .expect files, so I wanted to defer it to a subsequent PR.

2. Kevin mentioned in (WIP) Use same protocol to both drive DUT + infer transactions for BNW axi-lite-s1 bug #270 that:

You should also try to only have a single .prot for each one of the brave new world benchmarks. After all, our claim is that a single correct protocol can distinguish between buggy and fixed version. I think the reason we used to maintain two copies was because of how turnt test discovery works. With the new testing infrastructure, we should not have to do that anymore.

To implement this, we have to adapt generate_runt_configs.py so that for the Brave New World test cases, one single .prot can correspond to multiple .expect files (different output for buggy / fixed waveforms). This is doable, but I'd like to defer this to a future PR since it will modify (remove) a bunch of .expect files.

[Nikil-Shyamsunder]

As mentioned in Slack, this is okay but maybe we can find a way to have passing tests share a .expect between monitor and bi

[ekiwi]

We should handle this in a different way. Probably instead of panicing we would like to exit more cleanly in the error case.

[ekiwi]

Why would we ever need a timeout here?

[ngernest]

Some of the existing monitor test-cases in MONITOR_CASES (on main) have an expected timeout of 5 seconds, e.g. nested_busy_wait.prot where there are nested repeat loops and the monitor fails to infer both
outer_iters and inner_iters:

prot nested_busy_wait<DUT: Adder>(a: u32, b: u32, outer_iters: uint, inner_iters: uint, s: u32) {
  DUT.a := a;
  DUT.b := b;
  repeat outer_iters iterations {
    repeat inner_iters iterations {
      step();
      assert_eq(s, DUT.s);
    }
    step();
    assert_eq(s, DUT.s);
  }
  DUT.a := X;
  DUT.b := X;
  assert_eq(s, DUT.s);
  fork();
  step();
}

Since the bi and monitor share the same test cases in test_catalog.py, I added this here just ot make it consistent with the monitor tests (this is also what is done in the existing monitor_runt_command function in generate_runt_configs.py). I can get rid of this though if it is not needed for the BI -- I am thinking perhaps this is the case and this reveals a bug in the monitor.

I looked at the BI output for these test cases where the monitor times out and it says Cannot fork at step zero!, so I think this is somewhat related to the monitor bug #214 where it doesn't handle the case when a loop takes 0 iterations (i.e. we skip the loop) and we end up having a fork in cycle 0 (for the protocol above).

[ekiwi]

we instead print (and only save in .expect files) the following:

panicked at bi/src/bi.rs:628:21:
[add_busy_wait@00!] Cannot fork at step zero!

This avoids the OS thread ID (the thread 'main' (11301631) ... information) in the default error message from appearing in the .expect files, since the OS thread ID changes every time we run the executable.

We should try to solve these cases instead of working around the problem. Either the protocol or the bi implementation is buggy here.

[ekiwi]

I would like to use the same expect files for monitor and bi, at least for all passing tests.

[ngernest]

@ekiwi Got it, thanks! I'll rework the .expect tests so that the monitor + BI can share the same files for passing tests -- hopefully I can get to this later this week! For examples where the monitor + BI have differing output, I'll mark that as a test failure for now.

[ekiwi]

Got it, thanks! I'll rework the .expect tests so that the monitor + BI can share the same files for passing tests -- hopefully I can get to this later this week! For examples where the monitor + BI have differing output, I'll mark that as a test failure for now.

Yeah, so I am pretty sure that there are some tests were the BI and the monitor disagree and I have not had the time yet to debug which one is wrong.

[ngernest]

Closing this PR as we are removing the monitor for now in #287