[BI] Replace panics with error messages, update Runt output - #289
Conversation
…rk before step' failure
…too many traces from showing up
|
|
||
| if let Some(tu) = cli.time_unit { | ||
| assert_eq!(tu, "ns", "Only nano seconds are supported"); | ||
| if let Some(tu) = cli.time_unit |
| let show_warnings = false; | ||
| let skip_static_step_fork_checks = false; | ||
| let mut d = DiagnosticHandler::new(ColorChoice::Auto, false, show_warnings, false); | ||
| let mut d = DiagnosticHandler::new(cli.color, false, show_warnings, false); |
|
|
||
| for fail in fails { | ||
| let proto = &protos[fail.proto_id]; | ||
| let msg = format!( |
There was a problem hiding this comment.
I would prefer for the error message to stay here and not to move it into a separate method.
The reason is that I would like to separate presentation and representation. I.e., proto_trace.rs should only care about conveying all the info, not how it gets presented to the user.
There was a problem hiding this comment.
Ah I see, that makes sense!
| ForkBeforeStep, | ||
| } | ||
|
|
||
| impl Failure { |
There was a problem hiding this comment.
Let's keep the exact error messages local to where the get emitted.
| @@ -373,6 +373,9 @@ impl StepToTime { | |||
| TimescaleUnit::FemtoSeconds => format!("{}ns", time as f64 / 1000.0 / 1000.0), | |||
| TimescaleUnit::PicoSeconds => format!("{}ns", time as f64 / 1000.0), | |||
| TimescaleUnit::NanoSeconds => format!("{}ns", time), | |||
| self.has_forked = true; | ||
| self.next_stmt = ti.next_stmt[&stmt]; | ||
| self.effectful_stmt_in_step = true; | ||
| Fork |
There was a problem hiding this comment.
Unfortunately, this does not work.
There are two different kinds of “failures” here:
- The protocol execution just does not match the given trace; that is what the
biuses theself.failuresfor. In this case, we keep on exploring other execution paths to see if we can find a matching one. - The protocol is malformed. That is what the
bicurrently handles with panics/assertion failures. In that case the whole execution should be stopped and no traces should be printed. In the short term, I would prefer to only test thebion legal protocols and punt handling ill-formed protocols in a nicer way until later.
There was a problem hiding this comment.
Makes sense, thanks! For nested_busy_wait.prot specifically, I think the protocol (shown below) is malformed, because when outer_iters = 0 and inner_iters = 0, we skip both repeat loops, but then we have a fork() before any step()s have been called.
prot nested_busy_wait<DUT: Adder>(a: u32, b: u32, outer_iters: uint, inner_iters: uint, s: u32) {
DUT.a := a;
DUT.b := b;
repeat outer_iters iterations {
repeat inner_iters iterations {
step();
assert_eq(s, DUT.s);
}
step();
assert_eq(s, DUT.s);
}
DUT.a := X;
DUT.b := X;
assert_eq(s, DUT.s);
fork();
step();
}And in this case, following (2), we should have bi terminate (with an error message instead of a panic). If its okay, I can get rid of the FailureKind enum and just make the bi exit with an error message (e.g. using std::process::exit).
There was a problem hiding this comment.
That would be a better solution, but - imho - the best solution is to just delete these .prot files. If you want to, you could open an issue as a reminder to revisit this problem of repeat loops that are taken zero times. But for now I would rather not have to think about dealing with broken protocols.
There was a problem hiding this comment.
Makes sense -- I looked at all the files where the bi (on main) emits the cannot fork at step zero panic and it looks like all of them suffer from the same "repeat loops taken zero times" issue ({push_pop_loop_empty, push_pop_loop_not_empty, busy_wait, nested_busy_wait, loop_with_assigns}.prot), so I'll just delete these files (& their associated .expect tests) for now.
…rk before step) + their associated tests
…sage formatting should happen
* Update generate_runt_configs.py to add Runt test suite for bi * Update Justfile to run bi tests * Add auto-generated Runt BI config * Add expected test output for BI test cases * Formatting * Redirect stderr to stdout so that bi error messages appear in expected output files * Update auto-generated BI Runt config * Update some .expect files * Add --color CLI arg to BI to suppress colors * Pass --color never to BI when running Runt tests * Update auto-generated Runt BI config * Update .expect files * Register custom panic hook to suppress extraneous information from panic error messages * Update .expect file output * Formatting * Add BI tests to CI * Remove monitor code + test cases * Update scripts for generating runt catalog * update readme to refer to BI instead of monitor * Change .monitor.prot test cases to bi.prot * Update case_stem function to refer to .bi suffix instead of .monitor * Regenerate Runt config * update some outdated comments * [BI] Replace panics with error messages, update Runt output (#289) * Add new FailureKind enum to BI * Error-handling for BI when fork is called before step * Update expect files for tests that previously panicked due to the 'fork before step' failure * Add --max-traces argument to the nested_busy_wait test case to avoid too many traces from showing up * Add --max-traces to push_pop_loop_empty test, update Runt expected output * Propagate --color CLI flag to diagnostic handler * Update Runt expect files (error messages no longer contain color) * Fix remaining failing test case involving a panic due to unsupported time unit * formatting * Add one extra comment * Remove malformed protocols (repeat loops with 0 iterations causing fork before step) + their associated tests * Remove FailureKind enum, address Kevin's comments re: where error message formatting should happen
Note: this PR should be reviewed before #287. #287 removes the monitor and integrates BI with the testing infrastructure, but CI currently fails on #287 due to BI
panics on certain test cases. This PR (#289) fixes these issues, and once this branch is merged into #287's branch, CI should pass.This PR replaces certain
panics in the BI (e.g.encountered fork before a step) into dedicated error messages, per Kevin's comment here:This involved adding a new
FailureKindtype to the BI diagnostics and updating the error message infrastructure -- Runt expect test outputs have also been updated.Note that previously, when the BI panicked (e.g. an execution path reached the
encountered fork before a steppanic), the entire BI executable would terminate. Now though, this is represented as aFailurein the BI (it is still displayed), but if there are other execution paths that succeed, the BI displays the inferred transactions on those execution paths instead.This is why certain expect test outputs (e.g.
push_pop_loop_not_empty.bi.expect) now contain inferred transaction traces instead of just the panic error message. For example, fornested_busy_wait, this is the.txfile supplied to the interpreter:The BI now infers the following:
This PR also changes the
Only nano seconds are supportedpanic to an error message to make theexpecttest output more consistent, as the default output for panic error messages displays the OS thread ID of the running executable, which changes every time the executable is run.Finally, for two test cases where the BI infers many traces (
nested_busy_waitandpush_pop_loop_empty), I have edited the Runt config to pass in the CLI argument--max-traces 20for these two specific test cases (to avoid the.expecttest file from being too long).