Skip to content

Add Linux userspace & kernel driver support for CGRA4ML#20

Open
JDRadatti wants to merge 40 commits into
KastnerRG:masterfrom
JDRadatti:master
Open

Add Linux userspace & kernel driver support for CGRA4ML#20
JDRadatti wants to merge 40 commits into
KastnerRG:masterfrom
JDRadatti:master

Conversation

@JDRadatti

Copy link
Copy Markdown

SUMMARY

This PR adds Linux support for CGRA4ML, enabling inference on the CGRA accelerator from a Linux userspace application on a Zynq FPGA. It introduces a kernel driver with IOCTL-based register access and DMA-coherent buffer management, a userspace platform adapter that overrides the bare-metal runtime.h macros, a shared library (libinference.so), C/Python examples, and a test suite.

CHANGES

New: Linux Kernel Driver (deepsocflow/linux/driver/)

  • Character device (/dev/cgra4ml) with IOCTL commands for register access, buffer query, HW reset/start, wait done, and status dump
  • 5 DMA-coherent buffers (weights, input, output, ocm0, ocm1) allocated in probe, accessible via mmap with vm_pgoff buffer index

New: Linux Userspace Adapter (deepsocflow/c/deepsocflow_linux.h)

  • Macro-override header that replaces direct MMIO register access with IOCTL calls
  • Virtual-to-physical address translation for the DMA-coherent weights buffer
  • flush_cache becomes a no-op (memory is already DMA-coherent)
  • Seamlessly adapts runtim.h; no changes needed to the core inference loop

New: Host Initialization (deepsocflow/linux/host.c)

  • host_setup(): opens device, queries buffers via GET_BUFS ioctl, mmaps DMA buffer, loads wbx.bin
  • host_cleanup(): unmaps buffer, closes device
  • Exported from libinference.so for both C and Python callers

New: Build System (deepsocflow/linux/Makefile)

  • Targets: kernel_prepare (clones and configures linux-xlnx), driver (kernel module), lib (libinference.so), linux_example, bundle (collects deployables indo deploy/ dir), bootgen (bitstream conversion to .bit.bin)
  • Cross-compilation for aarch64-linux-gnu- with a single make linux target for the full build

New: Examples

  • linux_example.c: C inference entry point: host_setup --> run --> print_output --> host_cleanup
  • linux_example.py: Python equivalent using ctypes to call into libinference.so

New: Test Suite (deepsocflow/linux/test/)

  • reg_test.c: register read/write verification
  • ioctl_test.c: IOCTL interface smoke test
  • dma_buf_test.c: DMA buffer mmap and pattern write/readback
  • run_smoke.sh: test runner
  • Comprehensive test README with expected output and test flow diagram

Other fixes & improvements

  • Updated Vivado flow TCL to use relative paths instead of hardcoded absolute paths
  • Device tree example for CGRA4ML node
  • Update README with steps to run on linux

ORGANIZATION

deepsocflow/linux/
├── driver/                # Linux kernel module (cgra4ml_drv.ko)
│   ├── cgra4ml_main.c     # Module probe/remove, ioctl dispatch, mmap 
│   ├── cgra4ml_hw.c       # Register read/write, reset, start, wait done
│   ├── cgra4ml_dma.c      # DMA-coherent buffer alloc/free/mmap
│   ├── cgra4ml_ioctl.h    # Shared userspace-kernel IOCTL definitions
│   ├── cgra4ml_regs.h     # AXI-Lite register map offsets
│   ├── cgra4ml_priv.h     # Driver-private types (cgra4ml_dev, dma_buf)
│   ├── cgra4ml.dts        # Device-tree node example
│   └── Makefile
├── host.c                 # host_setup/host_cleanup (exported from libinference.so)
├── linux_example.c        # C inference example
├── linux_example.py       # Python inference example (ctypes)
├── Makefile               # Top-level Linux build (driver, lib, bundle, bootgen)
└── test/                  # Test suite
    ├── reg_test.c         # Register access test
    ├── ioctl_test.c       # IOCTL interface test
    ├── dma_buf_test.c     # DMA buffer mmap test
    ├── run_smoke.sh       # Automated smoke test
    ├── Makefile           # Test build
    └── README.md

TODO

  • Build and deploy a full Linux image using Yocto for production
  • Benchmark inference performance on hardware
  • Test on PYNQ-Z1 / PYNQ-Z2
  • Test on ZCU102 and other ZynqMP platforms
  • Test with different Vivado versions
  • Add Jupyter notebook example
  • libinference.so target in /linux/test/Makefile could be moved to /linux/Makefile

NEED TO FIX

  • BOARDSTORE_BRANCH is hardcoded in the Makefile (Hardcoded to 2024.2, but works on my machine, running Vivado 2025.2)
  • PROJECT_NAME is hardcoded in the Makefile (Hardcoded to dsf_zcu104)

Contributors:

@allan70627
@JDRadatti

JDRadatti added 30 commits May 27, 2026 08:58
The baremetal runtime.h assumes virtual==physical addressing and
direct register access, which breaks under Linux's virtual memory.

Add deepsocflow_linux.h — a preprocessor-based adapter that redirects
register accesses through driver IOCTLs and places the Memory_st
struct inside the DMA-coherent weights buffer so hardware and CPU
share the same physical memory without copying.

Also:
  - Guard fb_fw_wrap.h baremetal definitions behind
    DEEPSOCFLOW_LINUX_OVERRIDE to avoid macro-recursion during
    compilation
  - Add inference test app (linux_test/inference.c) that loads
    wbx.bin, runs the full bundle loop, and prints results
  - Update linux_test/Makefile to build inference with -lm
- deepsocflow/c/host.c: exports host_setup() and host_cleanup() which
  open /dev/cgra4ml, GET_BUFS ioctl, mmap DMA buffer, load wbx.bin
- linux_test/inference.c: updated to call host_setup/host_cleanup
- linux_test/Makefile: build libinference.so from host.c
- python/run_inference.py: new ctypes runner calling host_setup/run/print_output/host_cleanup
- deepsocflow/c/linux_inference.c: removed (replaced by host.c)
- make bundle: builds libinference.so + test binaries, copies
  libinference.so, inference, run_inference.py, cgra4ml_drv.ko,
  and wbx.bin into deploy/ for a single scp -r deploy/ command
- git mv linux_driver deepsocflow/linux/driver
- Update Makefile driver targets to new path
- Update .gitignore patterns
- git mv linux_test deepsocflow/linux/test
- Update test Makefile include paths (now one level deeper)
- Update top-level Makefile test_app and bundle targets
- Update .gitignore patterns
- git mv python/run_inference.py deepsocflow/linux/runner.py
- Update Makefile bundle target path
- Delete cgra4ml_pynq.py (dead code, replaced by ctypes-based runner.py)
- Delete python_README.md (documented the removed file)
- Update main README.md paths (linux_driver → deepsocflow/linux/driver,
  linux_test → deepsocflow/linux/test)
- Rename linux_driver_README.md → README.md
- Rename linux_test_README.md → README.md
- Update internal path references in both READMEs
Per the reorganization plan, host.c (Linux platform initialization)
belongs in deepsocflow/linux/ alongside the other Linux-specific code,
not in deepsocflow/c/ which contains platform-agnostic firmware.

- git mv deepsocflow/c/host.c deepsocflow/linux/host.c
- Update deepsocflow/linux/test/Makefile: ../../c/host.c → ../host.c
- Add 'make linux' target that builds driver + bundle in one command
- Remove 'driver_install' target (user prefers manual scp)
- Update .PHONY declarations
- Print deployment instructions after build

New workflow:
  make hw       # Build bitstream (already done)
  make linux    # Build everything + create deploy/
  # Manual deployment:
  scp deepsocflow/linux/driver/cgra4ml_drv.ko amd-edf@192.168.2.10:/tmp/
  scp -r deploy amd-edf@192.168.2.10:/home/amd-edf/
Change run/work/wbx.bin to run/work/vectors/wbx.bin to match
the actual location where the model generator places the file.
JDRadatti added 10 commits June 1, 2026 13:39
- Updated Makefile, .gitignore to match
- deepsocflow/linux/test/inference.c -> deepsocflow/linux/linux_example.c
- deepsocflow/linux/runner.py -> deepsocflow/linux/linux_example.py
- Moved linux logic to /deepsocflow/linux/Makefile
- Fixed `overlay` target which enabled removing fragile sed commands
- Other minor fixes

Some things still need cleaning:
- EDF_OVERLAY_TARGET is hardcoded
- PROJECT_NAME is redefined
- BOARDSTORE_BRANCH is hardcoded
Update scp command to store in fpga's home directory
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant