Skip to content

[BI] Replace panics with error messages, update Runt output - #289

Merged
ekiwi merged 12 commits into
remove-monitorfrom
bi-panic-to-errors
Jul 20, 2026
Merged

[BI] Replace panics with error messages, update Runt output#289
ekiwi merged 12 commits into
remove-monitorfrom
bi-panic-to-errors

Conversation

@ngernest

@ngernest ngernest commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Note: this PR should be reviewed before #287. #287 removes the monitor and integrates BI with the testing infrastructure, but CI currently fails on #287 due to BI panics on certain test cases. This PR (#289) fixes these issues, and once this branch is merged into #287's branch, CI should pass.

This PR replaces certain panics in the BI (e.g. encountered fork before a step) into dedicated error messages, per Kevin's comment here:

We should handle this in a different way. Probably instead of panicing we would like to exit more cleanly in the error case.

This involved adding a new FailureKind type to the BI diagnostics and updating the error message infrastructure -- Runt expect test outputs have also been updated.

Note that previously, when the BI panicked (e.g. an execution path reached the encountered fork before a step panic), the entire BI executable would terminate. Now though, this is represented as a Failure in the BI (it is still displayed), but if there are other execution paths that succeed, the BI displays the inferred transactions on those execution paths instead.

This is why certain expect test outputs (e.g. push_pop_loop_not_empty.bi.expect‎) now contain inferred transaction traces instead of just the panic error message. For example, for nested_busy_wait, this is the .tx file supplied to the interpreter:

trace {
    reset();
    push_n_times(7, 3);
    pop_n_times(7, 2);
    check_not_empty();          // FIFO is non-empty (has one element remaining)
}

The BI now infers the following:

// trace 0
trace {
    reset();
    push_n_times(7, 1);
    push_n_times(7, 2);
    pop_n_times(7, 2);
    check_not_empty();
}

// trace 1
trace {
    reset();
    push_n_times(7, 1);
    push_n_times(7, 2);
    pop_n_times(7, 1);
    pop_n_times(7, 1);
    check_not_empty();
}

...
// Different combinations of `push_n_times` & `pop_n_times` such that the no. of `repeat` loop iterations 
// adds up to 4 for each of them

This PR also changes the Only nano seconds are supported panic to an error message to make the expect test output more consistent, as the default output for panic error messages displays the OS thread ID of the running executable, which changes every time the executable is run.

Finally, for two test cases where the BI infers many traces (nested_busy_wait and push_pop_loop_empty), I have edited the Runt config to pass in the CLI argument --max-traces 20 for these two specific test cases (to avoid the .expect test file from being too long).

@ngernest
ngernest changed the base branch from main to remove-monitor July 18, 2026 19:53
@ngernest
ngernest marked this pull request as ready for review July 18, 2026 20:06
Comment thread bi/src/main.rs

if let Some(tu) = cli.time_unit {
assert_eq!(tu, "ns", "Only nano seconds are supported");
if let Some(tu) = cli.time_unit

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is good

Comment thread bi/src/main.rs
let show_warnings = false;
let skip_static_step_fork_checks = false;
let mut d = DiagnosticHandler::new(ColorChoice::Auto, false, show_warnings, false);
let mut d = DiagnosticHandler::new(cli.color, false, show_warnings, false);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good

Comment thread bi/src/main.rs

for fail in fails {
let proto = &protos[fail.proto_id];
let msg = format!(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer for the error message to stay here and not to move it into a separate method.
The reason is that I would like to separate presentation and representation. I.e., proto_trace.rs should only care about conveying all the info, not how it gets presented to the user.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see, that makes sense!

Comment thread bi/src/proto_trace.rs Outdated
ForkBeforeStep,
}

impl Failure {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep the exact error messages local to where the get emitted.

Comment thread bi/src/signal_trace.rs
@@ -373,6 +373,9 @@ impl StepToTime {
TimescaleUnit::FemtoSeconds => format!("{}ns", time as f64 / 1000.0 / 1000.0),
TimescaleUnit::PicoSeconds => format!("{}ns", time as f64 / 1000.0),
TimescaleUnit::NanoSeconds => format!("{}ns", time),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great!

Comment thread bi/src/bi.rs
self.has_forked = true;
self.next_stmt = ti.next_stmt[&stmt];
self.effectful_stmt_in_step = true;
Fork

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, this does not work.

There are two different kinds of “failures” here:

  1. The protocol execution just does not match the given trace; that is what the bi uses the self.failures for. In this case, we keep on exploring other execution paths to see if we can find a matching one.
  2. The protocol is malformed. That is what the bi currently handles with panics/assertion failures. In that case the whole execution should be stopped and no traces should be printed. In the short term, I would prefer to only test the bi on legal protocols and punt handling ill-formed protocols in a nicer way until later.

@ngernest ngernest Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, thanks! For nested_busy_wait.prot specifically, I think the protocol (shown below) is malformed, because when outer_iters = 0 and inner_iters = 0, we skip both repeat loops, but then we have a fork() before any step()s have been called.

prot nested_busy_wait<DUT: Adder>(a: u32, b: u32, outer_iters: uint, inner_iters: uint, s: u32) {
  DUT.a := a;
  DUT.b := b;
  repeat outer_iters iterations {
    repeat inner_iters iterations {
      step();
      assert_eq(s, DUT.s);
    }
    step();
    assert_eq(s, DUT.s);
  }
  DUT.a := X;
  DUT.b := X;
  assert_eq(s, DUT.s);
  fork();
  step();
}

And in this case, following (2), we should have bi terminate (with an error message instead of a panic). If its okay, I can get rid of the FailureKind enum and just make the bi exit with an error message (e.g. using std::process::exit).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be a better solution, but - imho - the best solution is to just delete these .prot files. If you want to, you could open an issue as a reminder to revisit this problem of repeat loops that are taken zero times. But for now I would rather not have to think about dealing with broken protocols.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense -- I looked at all the files where the bi (on main) emits the cannot fork at step zero panic and it looks like all of them suffer from the same "repeat loops taken zero times" issue ({push_pop_loop_empty, push_pop_loop_not_empty, busy_wait, nested_busy_wait, loop_with_assigns}.prot), so I'll just delete these files (& their associated .expect tests) for now.

@ekiwi ekiwi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@ekiwi
ekiwi merged commit 5401963 into remove-monitor Jul 20, 2026
18 checks passed
ekiwi pushed a commit that referenced this pull request Jul 20, 2026
* Update generate_runt_configs.py to add Runt test suite for bi

* Update Justfile to run bi tests

* Add auto-generated Runt BI config

* Add expected test output for BI test cases

* Formatting

* Redirect stderr to stdout so that bi error messages appear in expected output files

* Update auto-generated BI Runt config

* Update some .expect files

* Add --color CLI arg to BI to suppress colors

* Pass --color never to BI when running Runt tests

* Update auto-generated Runt BI config

* Update .expect files

* Register custom panic hook to suppress extraneous information from panic error messages

* Update .expect file output

* Formatting

* Add BI tests to CI

* Remove monitor code + test cases

* Update scripts for generating runt catalog

* update readme to refer to BI instead of monitor

* Change .monitor.prot test cases to bi.prot

* Update case_stem function to refer to .bi suffix instead of .monitor

* Regenerate Runt config

* update some outdated comments

* [BI] Replace panics with error messages, update Runt output (#289)

* Add new FailureKind enum to BI

* Error-handling for BI when fork is called before step

* Update expect files for tests that previously panicked due to the 'fork before step' failure

* Add --max-traces argument to the nested_busy_wait test case to avoid too many traces from showing up

* Add --max-traces to push_pop_loop_empty test, update Runt expected output

* Propagate --color CLI flag to diagnostic handler

* Update Runt expect files (error messages no longer contain color)

* Fix remaining failing test case involving a panic due to unsupported time unit

* formatting

* Add one extra comment

* Remove malformed protocols (repeat loops with 0 iterations causing fork before step) + their associated tests

* Remove FailureKind enum, address Kevin's comments re: where error message formatting should happen
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants