A small SIMD vector accelerator, attached to a CVA6 RISC-V core
as a custom coprocessor extension (CV-X-IF), that runs 4x4 matrix operations
(vmacc matrix-multiply, vadd.vv elementwise add) on real Zynq-7000 hardware.
CVA6 issues vector-style instructions over the CV-X-IF interface; TinyGPU
decodes them, runs four ALU lanes in parallel, and writes results back to
memory over AXI.
It accepts 64-bit RISC-V Vector Extension standard instructions like vle64.v vadd.vv vmacc.vv vse64.v
and operates like ARA in tandem with CVA6.
TinyGPU is four ALU lanes (design_1 / "the SIMD core") wrapped by a small
FSM (tinygpu_fsm.v) that:
- receives an offloaded instruction from CVA6 over CV-X-IF (
tinygpu_cvxif_wrap.v,cva6_sv_shim.sv), - pulls the operand matrices (
mat_a,mat_b) into a 4-row/4-column register cache (cache.v,register_file.v), - runs all 4 lanes' ALUs (
alu.v) in parallel per pass, either as a plain elementwise op or, forvmacc, accumulating each lane's product into a dot-product term (dot_reg[]indesign_1.v) across 4 passes to produce one row of a real matrix product, - and drives an AXI4 write burst (
writeback.v) to park the 4x4 result back in memory for the host to read.
The whole thing is built as a Zynq PS7 + custom PL design
(zynq_system.bd): CVA6 and TinyGPU live in the PL as fpga_top, boot
straight out of a synthesizable BRAM (axi4_bram_slave.v, preloaded via
boot_image.hex) instead of an OS, run a fixed instruction sequence (load
matrix A, vmacc A×A, store, vadd.vv A+A, store, loop), and the PS7's
ARM side (running a bare-metal Vitis app) reads the results straight out of
PL memory over the GP0 AXI port and prints them over UART.
-
On reset, CVA6 boots directly from
boot_image.hex(no OS, no bootloader) and runs a short RISC-V program, compiled by a realriscv64-unknown-elf- gcc/riscv-none-elf-gcctoolchain fromfw/boot_matmul.S(seefw/below), that:- loads a 4x4 matrix
A(64-bit elements) via a customvle64.vload, - issues
vmacc(matrix multiply, A × A) and stores the 4x4 resultC_mul, - issues
vadd.vv(elementwise add, A + A) and stores the 4x4 resultC_add, - jumps to itself (
jal x0,0) to park.
- loads a 4x4 matrix
-
Every custom vector instruction is caught by CVA6's CV-X-IF offload interface and handed to TinyGPU instead of being executed on the scalar pipeline.
-
TinyGPU computes the result across 4 lanes/4 passes and writes it back over AXI4 to a fixed address in PL memory.
-
The Zynq PS7 (a bare-metal Vitis app,
vitis/main.c) reads both result matrices straight out of that PL memory through the GP0 AXI aperture (0x4000_0000+) and prints them over UART:For vmacc.vv
C_mul = A x A (vmacc) [ 90 100 110 120 ] [ 202 228 254 280 ] [ 314 356 398 440 ] [ 426 484 542 600 ]And for vadd.vv
C_add = A + A (vadd.vv) [ 2 4 6 8 ] [ 10 12 14 16 ] [ 18 20 22 24 ] [ 26 28 30 32 ]
PS7 is configured with the board preset, M_AXI_GP0 enabled, and a 25MHz
FPGA clock. fpga_top is wired in as a plain RTL module (not packaged as an
IP), with GP0 connected directly to axi4_bram_slave's AXI-lite-style
read port (no SmartConnect / interconnect in the datapath) so the PS7 can
read TinyGPU's result memory with the simplest possible address decode.
The initial zero hold is for CVA6 to boot, and then there is the output from matrix multiplication which takes 4 cycles to complete
and thenyou have matrix addition taking 1 cycle.
boot_image.hex -- $readmemh preload for axi4_bram_slave (real-hardware boot program).
Generated by fw/build.sh -- do not hand-edit, see fw/ below.
bench_image.hex -- optional: fw/build.sh bench's output, boots the mcycle
benchmark instead of the plain demo -- see Benchmarking below.
build/ -- Vivado Tcl flow, run in numeric order 01..10 (see "How to build
in Vivado"), plus fpga_top.xdc (the one constraints file)
docs/ -- design notes (full_pipeline.md: CVA6 -> CV-X-IF -> TinyGPU walkthrough)
fw/ -- real RISC-V toolchain source for boot_image.hex/bench_image.hex (see below)
images/ -- figures used in this README
src/rtl/
cvxif/ -- CV-X-IF glue: AXI slaves, CVA6<->TinyGPU shim, dispatch FSM
tinygpu/ -- TinyGPU core: ALU lanes, cache, register file, decoder, writeback
src/tb/ -- testbenches: tb_cva6_boot (main boot+matmul test, hand-pokes
instructions directly into memory), tb_boot_image (same test,
but $readmemh's the real boot_image.hex through the real
axi4_bram_slave.v -- verifies fw/'s compiler output),
tb_bench_image (same idea for bench_image.hex), tb_speedup_cycles
(sim-only scalar-vs-offload cycle count via internal probes)
vitis/ -- main.c for the PS7 bare-metal app (reads results, prints over UART)
boot_image.hex used to be hand-spliced hex, instruction by instruction.
It's now a build artifact:
fw/boot_matmul.S -- the boot program in real RISC-V assembly
fw/bench_matmul.S -- csrr-mcycle-bracketed benchmark variant, see Benchmarking below
fw/link.ld -- places .text at 0x8000_0000, matrix data at 0x8000_1000
fw/build.sh -- runs the toolchain; `./build.sh` -> boot_image.hex,
`./build.sh bench` -> bench_image.hex
fw/elf2hex.py -- converts the linked ELF into axi4_bram_slave.v's $readmemh format
Requires a riscv64-unknown-elf-*/riscv-none-elf-* GCC toolchain on
PATH (e.g. the xPack GNU RISC-V Embedded
GCC).
Run CROSS_PREFIX=riscv-none-elf- ./fw/build.sh (adjust the prefix to
match your toolchain's binary names) to regenerate boot_image.hex.
TinyGPU's decoder repurposes real RVV opcode/funct3/funct6
bit patterns to mean matrix/cache-line IDs, not real per-lane
vector-register semantics governed by vtype/vl — letting the compiler
auto-generate a vsetvli or real vector code for this program would
silently compute the wrong thing. The 4 custom ops (vle64.v/vmacc/
vadd.vv/vse64.v) are emitted as .word literals in the
assembly instead; only the surrounding scalar glue (address setup via
li, the park loop) comes from the compiler. See boot_matmul.S's header
comment for the full reasoning, and src/rtl/cvxif/tinygpu_fsm.v's
is_vec_arith comment for the decoder-side half of this (a real
vsetvli, if one ever slipped in, now traps instead of corrupting state).
After rebuilding boot_image.hex, simulate tb_boot_image (not just
tb_cva6_boot, which never touches the file) before touching synthesis —
see build/02_import_sources.tcl's notes on switching the sim top.
TinyGPU needs a checkout of the CVA6 core (not included in this repo).
build/02_import_sources.tcl looks for it, in order, at:
- the directory named by the
CVA6_ROOTenvironment variable (the CVA6 checkout root -- the folder containingcore/andvendor/); <parent-of-this-repo>/CVA6/cva6(the usual layout: this repo andCVA6/-- a clone of https://github.com/openhwgroup/cva6 -- side by side in the same parent folder);<grandparent-of-this-repo>/CVA6/cva6.
It stops with a clear error if none of them exist. Every other path in the
Tcl scripts is computed from the script's own location ([info script]), so
the repo can be cloned anywhere.
Method: the RISC-V mcycle CSR (0xB00). Confirmed implemented and
free-running on this CVA6 config (cv64a6_imafdc_sv39, PerfCounterEn=1,
RVZicntr=1); an M-mode read never traps. fw/bench_matmul.S runs
bare-metal in M-mode and brackets two back-to-back phases with csrr:
- Scalar baseline -- a real triple
i/j/kloop computingC_s = A x Aon CVA6's own RV64IM pipeline (real address arithmetic + branches, not an idealized unrolled loop). Window: firstld-> loop exit. - Offload -- the same
vle64.v/vmacc/vse64.v/vadd.vv/vse64.vsequence asboot_matmul.S. Window: issuevle64.v-> CV-X-IF commit of the lastvse64.v. The AXI bursts are inside this window even though no code waits on them: CVA6'scvxif_fuonly completes a CV-X-IF instruction when the coprocessor returnsresult_valid,tinygpu_fsm.vasserts that forvse64.vonly after the AXI write response, instructions retire in order, and the trailingcsrrexecutes at commit. (Software must not poll the result buffer to wait for the write -- CVA6's dcache has no coherency with TinyGPU's separate AXI master, so the poll reads a stale cached line forever; seefw/bench_matmul.S's header.)
Both deltas land in BRAM (0x8000_4000 scalar / 0x8000_4008 offload);
vitis/main.c reads them over the GP0 aperture and prints cycles, speedup,
and wall-clock time.
Build and run:
cd fw && CROSS_PREFIX=<your-prefix> ./build.sh bench # -> ../bench_image.hexThen simulate src/tb/tb_bench_image.v ($readmemh's bench_image.hex
through the real axi4_bram_slave.v, checks results + both deltas,
scalar_delta > offload_delta) before touching synthesis. Cross-check the
scalar delta against src/tb/tb_speedup_cycles.v's independent
$time-based CVA6 matmul-only (first ld -> last add commit) line --
expect roughly a 2x difference, not agreement (measured: 806 vs 1561
cycles): that program's scalar baseline is fully unrolled with
immediate-baked offsets and no branches, while bench_matmul.S's is a real
i/j/k loop with runtime address arithmetic -- two valid scalar baselines,
not a measurement error. The number to compare against is the TinyGPU
COMPUTE -> result_valid figure (5 cycles), which matches the RTL's
documented 4 passes + 1 settle cycle.
On hardware: flip fpga_top.v's axi4_bram_slave INIT_FILE from
"boot_image.hex" to "bench_image.hex" (see the *** MANUAL TOGGLE ***
comment there), full OOC bitstream rebuild (build/06_launch_synth.tcl, never a
bare reset_run), re-export the .xsa, rebuild the Vitis app, boot, read
UART.
Measured result (Zynq-7000, 25 MHz FCLK_CLK0):
scalar (CVA6 RV64IM, first ld -> last add) : 1561 cycles
offload (TinyGPU vmacc+vadd, issue -> last vse64.v commit) : 100 cycles
speedup : 15.61x
@ 25 MHz FCLK_CLK0 : scalar 62.44 us, offload 4.00 us
This matches tb_bench_image's simulated result bit-for-cycle -- the RTL
model is a faithful predictor of the silicon for this measurement.
Tested with Vivado, targeting a Zynq-7000 (xc7z020clg484, ZC702
board preset). Run the scripts in numeric order from the Vivado Tcl
console (source {<path-to-repo>/build/<script>.tcl}); each one prints what
to run next. None of them contain machine-specific paths (see "CVA6" above
for the one external dependency).
| # | Script | What it does |
|---|---|---|
| 1 | 01_create_project.tcl |
New RTL project for xc7z020clg484-1 / ZC702, in <repo>/vivado_proj/TinyGPU (override with TINYGPU_PROJ_DIR / TINYGPU_PROJ_NAME; on Windows keep the path short) |
| 2 | 02_import_sources.tcl |
Adds the CVA6 core files, src/rtl/**, the testbenches, boot_image.hex (and bench_image.hex if present) as Memory Initialization Files, and build/fpga_top.xdc; sets tb_cva6_boot as sim top |
| 3 | 03_build_bd.tcl |
PS7 + fpga_top block design at 25 MHz |
| 4 | 04_connect_gp0.tcl |
Wires PS7's GP0 straight to fpga_top's AXI read port (no SmartConnect) and validates |
| 5 | 05_finalize_bd.tcl |
Generates zynq_system_wrapper and sets it as the synthesis top |
| 6 | 06_launch_synth.tcl |
Full out-of-context-safe synthesis launch |
| 7 | 07_check_synth.tcl |
Post-synthesis checks and utilization report |
| 8 | 08_launch_impl.tcl |
Implementation through route_design |
| 9 | 09_generate_bitstream.tcl |
write_bitstream |
| 10 | 10_export_hardware.tcl |
Writes the .xsa (with bitstream) for Vitis |
Simulate before synthesizing (zero bitstream risk): after step 2, run
behavioral simulation on tb_cva6_boot and confirm MUL_RESULT_ADDR+15 == 600 and RESULT_ADDR+15 == 32. To verify a rebuilt boot_image.hex, switch
the sim top to tb_boot_image; for the benchmark image, tb_bench_image:
set_property top tb_boot_image [get_filesets sim_1]
launch_simulation; run -allDo not skip steps 3-5. If synthesis is launched without the block design
and wrapper, Vivado synthesizes the bare fpga_top RTL as top: there is no
PS7, no clock definition, fpga_top.xdc matches nothing, and the timing
report shows WNS = inf / "no user specified timing constraints". A correct
run has synth_design -top zynq_system_wrapper in synth_1/runme.log and a
clk_fpga_0 25 MHz clock in the timing summary.
Important — out-of-context IP caching:
fpga_toplives insidezynq_system.bdand gets its own cached out-of-context synthesis run, separate from the top-levelsynth_1. If you changeboot_image.hex(or any filefpga_topdepends on) and justreset_run synth_1, Vivado may not notice the block design's cached IP is stale and will silently reuse the old synthesizedfpga_top.06_launch_synth.tclhandles this correctly (reset_target all+generate_target all -forceon the.bdbefore resettingsynth_1) — always rebuild through it after touching a filefpga_topdepends on.
- Create a platform component from the exported
.xsa, and an application component (standalone domain,ps7_cortexa9_0, empty C application) pointing at that platform. - Copy
vitis/main.cinto the application component'ssrc/. - Build the platform, then the application, program the device, open a
serial terminal (UART, 115200 8N1 by default on this board), and reset
the board — CVA6 reboots from the BRAM image baked into the bitstream and
the app prints
C_mulandC_addas shown above, plus the benchmark block (or a "benchmark firmware not loaded" note when the bitstream was built withboot_image.hex).
If you change boot_image.hex/bench_image.hex or any RTL fpga_top
depends on, you must regenerate the bitstream (see the out-of-context caching
note above), re-export hardware, and rebuild the Vitis
platform/application against the fresh .xsa — an app built against a stale
.xsa will boot the old hardware even if the bitstream on disk is current.


