Full system mode: RISC-V + Ruby + CHI stuck in livelock, perhaps due to a timing issue.

[deglaus]

Hello gem5 community!

I have issues running multi-threaded applications in gem5 full system mode using RISC-V + Ruby + CHI.
Running applications from Parsec-3.0, splash2x and splash4, seemingly any kind of multithreading can cause a livelock. I used the RiscvTimingSimpleCPU and pretty much all applications get stuck running 4 cores or more. I had better luck using the RiscvO3CPU where more application + core-count combinations seem to work just fine, however, some get stuck at 8 cores. The AtomicSimpleCPU is working.

Using gem5's debug traces, we could observe the following when running 4-cores on the NO-PIE cholesky benchmark:

• Divergence in-between executions that may be caused by timing whenever the benchmark calls a library for instance.
  • We are exploring this in FS mode, but lining up everything is difficult since timing is involved. We cannot tell whether the divergence is an issue, but we're currently dumping per-ASID traces.
• Once stuck, a single core will continue to execute instructions in kernel and machine mode with seemingly no progress being made in the benchmark.
  • Using m5term, we see that we are still executing inside the benchmark, because there are no result prints and the simulation does not terminate. We have ensured that an m5 exit instruction is triggered after the benchmark, but this is not reached.

When stuck, gem5 will accumulate warnings about consecutive SC fails from the riscv isa. However, we do not think this is the issue since one core is still executing instructions outside of the benchmark.

I had to create my own configuration for RISC-V + Ruby + Full system mode, which I can provide, along with details how how to recreate my setup (including disk image and kernel/bootloader). The config uses a RiscvBoard and runs the CHI cache coherence protocol with 3-levels, targeting MOESI behaviour.

In any case, we would like to know if anybody has any suggestions on how to debug this and how to find out what is wrong, or if somebody already knows what is wrong.

Best regards,
Johan