[Feature] Implement DMA support

[BenkangPeng]

Related issue: coredac/CGRA-SoC#2

This PR introduces CgraDmaRTL which integrates the CGRA with a DMA engine, enabling direct memory transfers between external DRAM(don't implement now) and the CGRA's dataSPM.

[tancheng]

Why can't we use the RecvIfcRTL and SendIfcRTL interfaces to connect the DmaRTL?

[BenkangPeng]

Fixed.

[tancheng]

As we discussed before, should we connect the dma to the controller (as intermediate interface/transition)? instead of directly connecting to data spm?

So then we can leverage

VectorCGRA/controller/ControllerRTL.py

Lines 138 to 139 in 44618d5

	s.recv_from_cpu_pkt //= s.recv_from_cpu_pkt_queue.recv
	s.send_to_cpu_pkt //= s.send_to_cpu_pkt_queue.send

to decide the next location (e.g., local spam bank, remote spm)?

[HobbitQia]

Hi @tancheng @BenkangPeng , I summarized two direction of DMA design as below:

• Rely on data controller

  • DMA is added as a new client of the DataMemControllerRTL, where data in the DMA engine communicates directly with the DataMemControllerRTL, and the logic for multiplexing SPM ports is also implemented in that module.

    To initiate DMA, the CPU can send dma_mvin or dma_mvout to the CGRA, after which the controller activates the DMA engine by sending start signals.

  • Pros: Keeps the controller clean; provides a faster path because data does not go through the controller.

  • Cons: Additional logic is required to feed DMA results into the control memory.

  

• All in controller

  • All decoding logic is handled in the controller. The logic for handling the logic port should still reside in the DataMemControllerRTL, since the data memory should have its own port multiplexing logic.

    The logic of packeting should also be implemented in the controller module.

  • Pros: Unifies control and data memory within the controller (the controller is already connected to both control and data memory).

  • Cons: Introduces complex control logic in the controller; results in a slower path.

  

I prefer the second method but I think there are still some logic should be written in DataMemControllerRTL. WDTY?

[tancheng]

Hi @HobbitQia, option 2 looks good to me. Though I am not sure what logic should be additionally in DataMemController?

[tancheng]

All these will change if we go with @HobbitQia's option 2, right?

Moreover, can we use send/recv interfaces, and define msg (https://github.com/tancheng/VectorCGRA/blob/master/lib/messages.py) to encapsulate data, addr, or whatever needed as struct? So we don't need to declare so many ports, and explicitly connect each of them. This CGRA RTL shouldn't see these details, the data struct can be decomposed inside the submodule.

[HobbitQia]

All these will change if we go with @HobbitQia's option 2, right?

Moreover, can we use send/recv interfaces, and define msg (https://github.com/tancheng/VectorCGRA/blob/master/lib/messages.py) to encapsulate data, addr, or whatever needed as struct? So we don't need to declare so many ports, and explicitly connect each of them. This CGRA RTL shouldn't see these details, the data struct can be decomposed inside the submodule.

Yes.

I agree with we can use this helper functions or classes to define these ports. Initially I design the interface only from the side about how to connect the DMA engine and the chipyard so I defined these ports. But I believe these input/output ports can be used in our struct and wrapped by another adapter so that we can connect them. @BenkangPeng

[HobbitQia]

Hi @HobbitQia, option 2 looks good to me. Though I am not sure what logic should be additionally in DataMemController?

I am thinking if we want enable the concurrent running of DMA and traditional load/store, then we need to multiplex the port of Data SPM and I think this logic can be implemented in DataMemController. However, we can also transform the data from DRAM into packets and use the command like CMD_STORE_REQUEST or CMD_LOAD_REQUEST to send to SPM. Then we can entirely write our logic in controller but with higher latency. I think maybe the former method can have better performance with minimal addition to DataMemController.

[tancheng]

Hi @HobbitQia, option 2 looks good to me. Though I am not sure what logic should be additionally in DataMemController?

I am thinking if we want enable the concurrent running of DMA and traditional load/store, then we need to multiplex the port of Data SPM and I think this logic can be implemented in DataMemController. However, we can also transform the data from DRAM into packets and use the command like CMD_STORE_REQUEST or CMD_LOAD_REQUEST to send to SPM. Then we can entirely write our logic in controller but with higher latency. I think maybe the former method can have better performance with minimal addition to DataMemController.

I thought they are the same latency if we can distinguish the CMD_STORE_REQUEST into CMD_STORE_REQUEST_FROM_NOC and CMD_STORE_REQUEST_FROM_CPU (and add another inport on the xbar) in controller?

Adding logic inside the DataMemController kind of bypassing the CGRA controller, which doesn't align with your Option 2, WDYT?

[HobbitQia]

Hi @HobbitQia, option 2 looks good to me. Though I am not sure what logic should be additionally in DataMemController?

I am thinking if we want enable the concurrent running of DMA and traditional load/store, then we need to multiplex the port of Data SPM and I think this logic can be implemented in DataMemController. However, we can also transform the data from DRAM into packets and use the command like CMD_STORE_REQUEST or CMD_LOAD_REQUEST to send to SPM. Then we can entirely write our logic in controller but with higher latency. I think maybe the former method can have better performance with minimal addition to DataMemController.

I thought they are the same latency if we can distinguish the CMD_STORE_REQUEST into CMD_STORE_REQUEST_FROM_NOC and CMD_STORE_REQUEST_FROM_CPU (and add another inport on the xbar) in controller?

Adding logic inside the DataMemController kind of bypassing the CGRA controller, which doesn't align with your Option 2, WDYT?

If the DMA data should go through the controller packet path, there may be extra latency of packeting, and there may be competitions between NoC/CPU/tile request to SPM? Or we have two separate paths in Controller?