Unable to match Runai Distributed Bandwidth on VLLM

[bbartels]

We have 400gbit/s networked VAST storage and generally see ~1 minute weight loading times of kimi-k2.5 with Runai streamer and the following settings:

RUNAI_STREAMER_DIST: 1
RUNAI_STREAMER_CHUNK_BYTESIZE: 4194304
--model-loader-extra-config={{\"concurrency\": 140, \"distributed\": true}}

I tried instanttensor on VLLM and noticed it takes around ~2:30 min

I tested both with Kimi-k2.5, 8xH200, on a networked VAST fileshare mounted via NFS readahead: 4096, nconnect: 16, rdma: true. There is no persisted storage, loading directly into mounted RAM disk

Is there anything else that would need configuring?
I also tried with INSTANTTENSOR_USE_CUFILE. I am using the latest lightly VLLM cu13 x86 image with instanttensor installed into it

[bbartels]

[global]
ioengine=io_uring
direct=1
rw=read
bs=4m
iodepth=32
time_based=1
runtime=20
group_reporting=1
EOF
for i in $(seq -w 1 64); do
  cat >> /tmp/multi64_4m_direct_1x_rt20.fio <<EOF
[f${i}]
filename=${base}/model-000${i}-of-000064.safetensors
numjobs=1
EOF
  echo >> /tmp/multi64_4m_direct_1x_rt20.fio
done
fio /tmp/multi64_4m_direct_1x_rt20.fio'
Defaulted container "model-container" out of: model-container, batch-k8s-preflight-checks (init)
f01: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=io_uring, iodepth=32
[...] (abbreviated)
f64: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=io_uring, iodepth=32
fio-3.36
Starting 64 processes
f01: (groupid=0, jobs=64): err= 0: pid=71578: Wed Mar 18 23:36:21 2026
  read: IOPS=9710, BW=37.9GiB/s (40.7GB/s)(769GiB/20276msec)
    slat (nsec): min=270, max=2178.2k, avg=1876.29, stdev=6372.09
    clat (msec): min=4, max=2252, avg=188.31, stdev=241.88
     lat (msec): min=4, max=2252, avg=188.32, stdev=241.88
    clat percentiles (msec):
     |  1.00th=[   19],  5.00th=[   46], 10.00th=[   55], 20.00th=[   66],
     | 30.00th=[   86], 40.00th=[  108], 50.00th=[  126], 60.00th=[  182],
     | 70.00th=[  222], 80.00th=[  271], 90.00th=[  305], 95.00th=[  342],
     | 99.00th=[ 1821], 99.50th=[ 1989], 99.90th=[ 2198], 99.95th=[ 2232],
     | 99.99th=[ 2265]
   bw (  MiB/s): min=10696, max=85688, per=100.00%, avg=47119.23, stdev=345.39, samples=2090
   iops        : min= 2674, max=21422, avg=11779.81, stdev=86.35, samples=2090
  lat (msec)   : 10=0.04%, 20=1.27%, 50=5.84%, 100=28.95%, 250=40.47%
  lat (msec)   : 500=21.37%, 750=0.30%, 2000=1.32%, >=2000=0.44%
  cpu          : usr=0.01%, sys=8.15%, ctx=223803, majf=0, minf=668
  IO depths    : 1=0.1%, 2=0.3%, 4=0.6%, 8=1.2%, 16=1.9%, 32=95.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=196896,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
   READ: bw=37.9GiB/s (40.7GB/s), 37.9GiB/s-37.9GiB/s (40.7GB/s-40.7GB/s), io=769GiB (826GB), run=20276-20276msec

[arlo-scitix]

Thanks. This project is still in its early stages and is not yet robust enough. The current v0.1.6 has an issue with its parameters being overfitted to our own platform. InstantTensor includes three tunable parameters: chunk_size, concurrency, and io_depth. Previously, we set io_depth too optimistically (too low). This is not a problem on platforms with low I/O latency; however, once switching to a high-latency platform, or when background traffic is present (for example, in network storage scenarios) and causes higher latency, this parameter configuration performs poorly. We have also observed performance degradation in some cases on our own platform.

We will release a new version soon to address this issue. We may also adjust some other parts along the way to better accommodate a larger io_depth and provide higher bandwidth on high-latency scenarios. In addition, we may further simplify user-side tuning by exposing these controls through environment variables.

As for runai_model_streamer, in our previous tests we only set {"distributed": true} and left concurrency at its default value. We’ll rerun the benchmark using your configuration next.

[arlo-scitix]

Perhaps you could try the latest commit on our main branch? See build from source for details.

[bbartels]

We run airgapped, would be difficult to build from source, we have pypi package registry passthrough. Could you publish a dev version potentially, happy to give it a try then!

[arlo-scitix]

0.1.7 is now available with wheel support. Feel free to give it a try :)

[bbartels]

@arlo-scitix I gave it a try, while it got a bit better still seems a fair bit slower than runai_streamer for us. Now around 1:30 - 2:00 for instanttensor, with runai_streamer being <0:45

[bbartels]

Is there any settings I should tune?

[arlo-scitix]

Can you provide the logs with the environment variable INSTANTTENSOR_DEBUG=1?