Skip to content
athralkPublic

About

Custom RVV 64 bit vector accelerator for the CVA6 RISC-V CPU. SIMD style layout with 3 memory hierarchies inside the accelerator.

Resources

Stars

30 stars

Watchers

0 watching

Forks

Latest commit

 

History

5 Commits

Folders and files

Repository files navigation

TinyGPU

A small SIMD vector accelerator, attached to a CVA6 RISC-V core as a custom coprocessor extension (CV-X-IF), that runs 4x4 matrix operations (vmacc matrix-multiply, vadd.vv elementwise add) on real Zynq-7000 hardware. CVA6 issues vector-style instructions over the CV-X-IF interface; TinyGPU decodes them, runs four ALU lanes in parallel, and writes results back to memory over AXI.

It accepts 64-bit RISC-V Vector Extension standard instructions like vle64.v vadd.vv vmacc.vv vse64.v
and operates like ARA in tandem with CVA6.

Elaborated design

Overview

TinyGPU is four ALU lanes (design_1 / "the SIMD core") wrapped by a small FSM (tinygpu_fsm.v) that:

  • receives an offloaded instruction from CVA6 over CV-X-IF (tinygpu_cvxif_wrap.v, cva6_sv_shim.sv),
  • pulls the operand matrices (mat_a, mat_b) into a 4-row/4-column register cache (cache.v, register_file.v),
  • runs all 4 lanes' ALUs (alu.v) in parallel per pass, either as a plain elementwise op or, for vmacc, accumulating each lane's product into a dot-product term (dot_reg[] in design_1.v) across 4 passes to produce one row of a real matrix product,
  • and drives an AXI4 write burst (writeback.v) to park the 4x4 result back in memory for the host to read.

The whole thing is built as a Zynq PS7 + custom PL design (zynq_system.bd): CVA6 and TinyGPU live in the PL as fpga_top, boot straight out of a synthesizable BRAM (axi4_bram_slave.v, preloaded via boot_image.hex) instead of an OS, run a fixed instruction sequence (load matrix A, vmacc A×A, store, vadd.vv A+A, store, loop), and the PS7's ARM side (running a bare-metal Vitis app) reads the results straight out of PL memory over the GP0 AXI port and prints them over UART.

Workflow

  1. On reset, CVA6 boots directly from boot_image.hex (no OS, no bootloader) and runs a short RISC-V program, compiled by a real riscv64-unknown-elf- gcc/riscv-none-elf-gcc toolchain from fw/boot_matmul.S (see fw/ below), that:

    • loads a 4x4 matrix A (64-bit elements) via a custom vle64.v load,
    • issues vmacc (matrix multiply, A × A) and stores the 4x4 result C_mul,
    • issues vadd.vv (elementwise add, A + A) and stores the 4x4 result C_add,
    • jumps to itself (jal x0,0) to park.
  2. Every custom vector instruction is caught by CVA6's CV-X-IF offload interface and handed to TinyGPU instead of being executed on the scalar pipeline.

  3. TinyGPU computes the result across 4 lanes/4 passes and writes it back over AXI4 to a fixed address in PL memory.

  4. The Zynq PS7 (a bare-metal Vitis app, vitis/main.c) reads both result matrices straight out of that PL memory through the GP0 AXI aperture (0x4000_0000+) and prints them over UART:

    For vmacc.vv

    C_mul = A x A (vmacc)
    [   90  100  110  120 ]
    [  202  228  254  280 ]
    [  314  356  398  440 ]
    [  426  484  542  600 ]
    

    And for vadd.vv

    
    C_add = A + A (vadd.vv)
    [    2    4    6    8 ]
    [   10   12   14   16 ]
    [   18   20   22   24 ]
    [   26   28   30   32 ]
    

Block design (Vivado)

Block Design

PS7 is configured with the board preset, M_AXI_GP0 enabled, and a 25MHz FPGA clock. fpga_top is wired in as a plain RTL module (not packaged as an IP), with GP0 connected directly to axi4_bram_slave's AXI-lite-style read port (no SmartConnect / interconnect in the datapath) so the PS7 can read TinyGPU's result memory with the simplest possible address decode.

Zynq Implemented design

Simulation

image The initial zero hold is for CVA6 to boot, and then there is the output from matrix multiplication which takes 4 cycles to complete and thenyou have matrix addition taking 1 cycle.

Repo layout

boot_image.hex     -- $readmemh preload for axi4_bram_slave (real-hardware boot program).
                      Generated by fw/build.sh -- do not hand-edit, see fw/ below.
bench_image.hex    -- optional: fw/build.sh bench's output, boots the mcycle
                      benchmark instead of the plain demo -- see Benchmarking below.
build/             -- Vivado Tcl flow, run in numeric order 01..10 (see "How to build
                      in Vivado"), plus fpga_top.xdc (the one constraints file)
docs/              -- design notes (full_pipeline.md: CVA6 -> CV-X-IF -> TinyGPU walkthrough)
fw/                -- real RISC-V toolchain source for boot_image.hex/bench_image.hex (see below)
images/            -- figures used in this README
src/rtl/
  cvxif/           -- CV-X-IF glue: AXI slaves, CVA6<->TinyGPU shim, dispatch FSM
  tinygpu/         -- TinyGPU core: ALU lanes, cache, register file, decoder, writeback
src/tb/            -- testbenches: tb_cva6_boot (main boot+matmul test, hand-pokes
                      instructions directly into memory), tb_boot_image (same test,
                      but $readmemh's the real boot_image.hex through the real
                      axi4_bram_slave.v -- verifies fw/'s compiler output),
                      tb_bench_image (same idea for bench_image.hex), tb_speedup_cycles
                      (sim-only scalar-vs-offload cycle count via internal probes)
vitis/             -- main.c for the PS7 bare-metal app (reads results, prints over UART)

fw/ -- real compiler toolchain

boot_image.hex used to be hand-spliced hex, instruction by instruction. It's now a build artifact:

fw/boot_matmul.S    -- the boot program in real RISC-V assembly
fw/bench_matmul.S   -- csrr-mcycle-bracketed benchmark variant, see Benchmarking below
fw/link.ld          -- places .text at 0x8000_0000, matrix data at 0x8000_1000
fw/build.sh         -- runs the toolchain; `./build.sh` -> boot_image.hex,
                       `./build.sh bench` -> bench_image.hex
fw/elf2hex.py       -- converts the linked ELF into axi4_bram_slave.v's $readmemh format

Requires a riscv64-unknown-elf-*/riscv-none-elf-* GCC toolchain on PATH (e.g. the xPack GNU RISC-V Embedded GCC). Run CROSS_PREFIX=riscv-none-elf- ./fw/build.sh (adjust the prefix to match your toolchain's binary names) to regenerate boot_image.hex.

TinyGPU's decoder repurposes real RVV opcode/funct3/funct6 bit patterns to mean matrix/cache-line IDs, not real per-lane vector-register semantics governed by vtype/vl — letting the compiler auto-generate a vsetvli or real vector code for this program would silently compute the wrong thing. The 4 custom ops (vle64.v/vmacc/ vadd.vv/vse64.v) are emitted as .word literals in the assembly instead; only the surrounding scalar glue (address setup via li, the park loop) comes from the compiler. See boot_matmul.S's header comment for the full reasoning, and src/rtl/cvxif/tinygpu_fsm.v's is_vec_arith comment for the decoder-side half of this (a real vsetvli, if one ever slipped in, now traps instead of corrupting state).

After rebuilding boot_image.hex, simulate tb_boot_image (not just tb_cva6_boot, which never touches the file) before touching synthesis — see build/02_import_sources.tcl's notes on switching the sim top.

TinyGPU needs a checkout of the CVA6 core (not included in this repo). build/02_import_sources.tcl looks for it, in order, at:

  1. the directory named by the CVA6_ROOT environment variable (the CVA6 checkout root -- the folder containing core/ and vendor/);
  2. <parent-of-this-repo>/CVA6/cva6 (the usual layout: this repo and CVA6/ -- a clone of https://github.com/openhwgroup/cva6 -- side by side in the same parent folder);
  3. <grandparent-of-this-repo>/CVA6/cva6.

It stops with a clear error if none of them exist. Every other path in the Tcl scripts is computed from the script's own location ([info script]), so the repo can be cloned anywhere.

Benchmarking on hardware

Method: the RISC-V mcycle CSR (0xB00). Confirmed implemented and free-running on this CVA6 config (cv64a6_imafdc_sv39, PerfCounterEn=1, RVZicntr=1); an M-mode read never traps. fw/bench_matmul.S runs bare-metal in M-mode and brackets two back-to-back phases with csrr:

  • Scalar baseline -- a real triple i/j/k loop computing C_s = A x A on CVA6's own RV64IM pipeline (real address arithmetic + branches, not an idealized unrolled loop). Window: first ld -> loop exit.
  • Offload -- the same vle64.v/vmacc/vse64.v/vadd.vv/vse64.v sequence as boot_matmul.S. Window: issue vle64.v -> CV-X-IF commit of the last vse64.v. The AXI bursts are inside this window even though no code waits on them: CVA6's cvxif_fu only completes a CV-X-IF instruction when the coprocessor returns result_valid, tinygpu_fsm.v asserts that for vse64.v only after the AXI write response, instructions retire in order, and the trailing csrr executes at commit. (Software must not poll the result buffer to wait for the write -- CVA6's dcache has no coherency with TinyGPU's separate AXI master, so the poll reads a stale cached line forever; see fw/bench_matmul.S's header.)

Both deltas land in BRAM (0x8000_4000 scalar / 0x8000_4008 offload); vitis/main.c reads them over the GP0 aperture and prints cycles, speedup, and wall-clock time.

Build and run:

cd fw && CROSS_PREFIX=<your-prefix> ./build.sh bench   # -> ../bench_image.hex

Then simulate src/tb/tb_bench_image.v ($readmemh's bench_image.hex through the real axi4_bram_slave.v, checks results + both deltas, scalar_delta > offload_delta) before touching synthesis. Cross-check the scalar delta against src/tb/tb_speedup_cycles.v's independent $time-based CVA6 matmul-only (first ld -> last add commit) line -- expect roughly a 2x difference, not agreement (measured: 806 vs 1561 cycles): that program's scalar baseline is fully unrolled with immediate-baked offsets and no branches, while bench_matmul.S's is a real i/j/k loop with runtime address arithmetic -- two valid scalar baselines, not a measurement error. The number to compare against is the TinyGPU COMPUTE -> result_valid figure (5 cycles), which matches the RTL's documented 4 passes + 1 settle cycle.

On hardware: flip fpga_top.v's axi4_bram_slave INIT_FILE from "boot_image.hex" to "bench_image.hex" (see the *** MANUAL TOGGLE *** comment there), full OOC bitstream rebuild (build/06_launch_synth.tcl, never a bare reset_run), re-export the .xsa, rebuild the Vitis app, boot, read UART.

Measured result (Zynq-7000, 25 MHz FCLK_CLK0):

scalar  (CVA6 RV64IM, first ld -> last add)                : 1561 cycles
offload (TinyGPU vmacc+vadd, issue -> last vse64.v commit) : 100 cycles
speedup : 15.61x
@ 25 MHz FCLK_CLK0 : scalar 62.44 us, offload 4.00 us

This matches tb_bench_image's simulated result bit-for-cycle -- the RTL model is a faithful predictor of the silicon for this measurement.

How to build in Vivado

Tested with Vivado, targeting a Zynq-7000 (xc7z020clg484, ZC702 board preset). Run the scripts in numeric order from the Vivado Tcl console (source {<path-to-repo>/build/<script>.tcl}); each one prints what to run next. None of them contain machine-specific paths (see "CVA6" above for the one external dependency).

# Script What it does
1 01_create_project.tcl New RTL project for xc7z020clg484-1 / ZC702, in <repo>/vivado_proj/TinyGPU (override with TINYGPU_PROJ_DIR / TINYGPU_PROJ_NAME; on Windows keep the path short)
2 02_import_sources.tcl Adds the CVA6 core files, src/rtl/**, the testbenches, boot_image.hex (and bench_image.hex if present) as Memory Initialization Files, and build/fpga_top.xdc; sets tb_cva6_boot as sim top
3 03_build_bd.tcl PS7 + fpga_top block design at 25 MHz
4 04_connect_gp0.tcl Wires PS7's GP0 straight to fpga_top's AXI read port (no SmartConnect) and validates
5 05_finalize_bd.tcl Generates zynq_system_wrapper and sets it as the synthesis top
6 06_launch_synth.tcl Full out-of-context-safe synthesis launch
7 07_check_synth.tcl Post-synthesis checks and utilization report
8 08_launch_impl.tcl Implementation through route_design
9 09_generate_bitstream.tcl write_bitstream
10 10_export_hardware.tcl Writes the .xsa (with bitstream) for Vitis

Simulate before synthesizing (zero bitstream risk): after step 2, run behavioral simulation on tb_cva6_boot and confirm MUL_RESULT_ADDR+15 == 600 and RESULT_ADDR+15 == 32. To verify a rebuilt boot_image.hex, switch the sim top to tb_boot_image; for the benchmark image, tb_bench_image:

set_property top tb_boot_image [get_filesets sim_1]
launch_simulation; run -all

Do not skip steps 3-5. If synthesis is launched without the block design and wrapper, Vivado synthesizes the bare fpga_top RTL as top: there is no PS7, no clock definition, fpga_top.xdc matches nothing, and the timing report shows WNS = inf / "no user specified timing constraints". A correct run has synth_design -top zynq_system_wrapper in synth_1/runme.log and a clk_fpga_0 25 MHz clock in the timing summary.

Important — out-of-context IP caching: fpga_top lives inside zynq_system.bd and gets its own cached out-of-context synthesis run, separate from the top-level synth_1. If you change boot_image.hex (or any file fpga_top depends on) and just reset_run synth_1, Vivado may not notice the block design's cached IP is stale and will silently reuse the old synthesized fpga_top. 06_launch_synth.tcl handles this correctly (reset_target all + generate_target all -force on the .bd before resetting synth_1) — always rebuild through it after touching a file fpga_top depends on.

Vitis (host-side app)

  1. Create a platform component from the exported .xsa, and an application component (standalone domain, ps7_cortexa9_0, empty C application) pointing at that platform.
  2. Copy vitis/main.c into the application component's src/.
  3. Build the platform, then the application, program the device, open a serial terminal (UART, 115200 8N1 by default on this board), and reset the board — CVA6 reboots from the BRAM image baked into the bitstream and the app prints C_mul and C_add as shown above, plus the benchmark block (or a "benchmark firmware not loaded" note when the bitstream was built with boot_image.hex).

If you change boot_image.hex/bench_image.hex or any RTL fpga_top depends on, you must regenerate the bitstream (see the out-of-context caching note above), re-export hardware, and rebuild the Vitis platform/application against the fresh .xsa — an app built against a stale .xsa will boot the old hardware even if the bitstream on disk is current.

About

Custom RVV 64 bit vector accelerator for the CVA6 RISC-V CPU. SIMD style layout with 3 memory hierarchies inside the accelerator.

Resources

Stars

30 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages