server mode?

[khimaros]

a voxtral-server with openai endpoint support would be lovely

[khimaros]

something similar to whisper-server in whisper.cpp

[kikduck]

#6 check here

[kikduck]

https://github.com/kikduck/voxtral.cpp

fork with voxtral-server.

[khimaros]

@kikduck could you provide a working example?

i tried this (which works with whisper.cpp server)

ffmpeg -i "$input" -ar 16000 -ac 1 -c:a pcm_s16le -f wav pipe:1 2>/dev/null | \
  curl http://localhost:7860/v1/audio/transcriptions \
  -H "Content-Type: multipart/form-data" \
  -F file="@/dev/stdin" \
  -F model="voxtral-mini-4b-realtime-2602:Q8_0" \
  -s -o "$output"

there is nothing interesting on stdout/stderr except i can see that the audio samples are received:

voxtral-server: loading model /home/user/src/voxtral.cpp/models/voxtral/Q8_0.gguf  (gpu=auto, threads=4)
voxtral_I: loading model from /home/user/src/voxtral.cpp/models/voxtral/Q8_0.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Radeon 8060S Graphics (RADV STRIX_HALO) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
voxtral_I: model loaded: enc_layers=32 dec_layers=26 vocab=131072
voxtral_I: model weights: 4713.061888 MB
voxtral_I: encoder: dim=1280 heads=32 head_dim=64 hidden=5120
voxtral_I: decoder: dim=3072 heads=32 head_dim=128 hidden=9216 kv_heads=8
voxtral_I: model load time: 906.60 ms
voxtral_I: backend: VULKAN (CPU fallback 4 threads)
voxtral_I: buffers: encoder_chunk=10.24 MB kv_cache=1744.83 MB
voxtral_I: context initialized
voxtral-server: model loaded in 987 ms

voxtral-server ready
  http://0.0.0.0:7875/health
  http://0.0.0.0:7875/v1/audio/transcriptions

voxtral_I: audio loaded: 2147483647 samples (134217.7 s)

FWIW, i'm using a reverse proxy (llama-swap) and VULKAN backend.

[khimaros]

same issue when hitting the endpoint directly instead of via the llama-swap reverse proxy and the sample wav files in the repo work via the reverse proxy, so i suppose it's an issue with the input audio data.

this ffmpeg should be converting to the format described in the README though, so i'm a bit confused why it's not working.

[khimaros]

ahh, the issue is with streaming input and not knowing the file size.

ffmpeg to a temporary file and then making the API call at least allows it to start converting the first chunk. i wonder what's different from the whisper.cpp server here?

the hint was that the size/sample count was wrong in the logs.

HOWEVER, it still only seems to convert the first chunk and then seems to spin for a long time without converting any of the subsequent chunks.