Load a model from a memory buffer

CMakeLists.txt

@@ -149,6 +149,7 @@ endif()
 set(VOICEDETECT_SRC
     src/voicedetect.cpp
     src/voicedetect_capi.cpp
+    src/gguf_check.cpp
     src/model.cpp
     src/model_loader.cpp
     src/backend.cpp

README.md

@@ -188,6 +188,7 @@ The frozen symbol set:
 
 - `voicedetect_capi_abi_version` - integer LocalAI checks for compatibility; bump it on any breaking change to the frozen symbols (additive functions do not require a bump).
 - `voicedetect_capi_load` / `voicedetect_capi_free` - load a GGUF model, free the context.
+- `voicedetect_capi_load_from_memory` / `voicedetect_capi_load_from_memory_prefixed` / `voicedetect_capi_last_load_error` - load a model from a buffer instead of a path, and read why a load failed (see "Loading from memory" below).
 - `voicedetect_capi_last_error` - human-readable last error on a context.
 - `voicedetect_capi_embed_path` / `voicedetect_capi_embed_pcm` - L2-normalized embedding from a WAV file or in-memory mono float PCM (PCM is linearly resampled to 16 kHz if needed).
 - `voicedetect_capi_verify_paths` - cosine distance (`1 - cosine_similarity`) between two clips plus a same-speaker verdict against a threshold.
@@ -218,6 +219,25 @@ if (json) { voicedetect_capi_free_string(json); }
 voicedetect_capi_free(ctx);
 ```
 
+### Loading from memory
+
+A model can be loaded from a GGUF that is already in memory, with no file path and no temporary file. The same code runs on Linux, macOS and Windows.
+
+```c
+voicedetect_ctx* ctx = voicedetect_capi_load_from_memory(bytes, n_bytes);
+if (!ctx) fprintf(stderr, "%s\n", voicedetect_capi_last_load_error());
+free(bytes);   // allowed: the loader has already copied what it needs
+```
+
+- **Ownership.** The loader copies the tensor data into its own memory during the call. `data` is only read inside the call, so you may free or overwrite it as soon as the function returns, whether it succeeded or not. While the call runs, the buffer and the model are in memory together.
+- **Errors.** A NULL or empty buffer, a truncated buffer, and a corrupt buffer all return `NULL`. `voicedetect_capi_last_load_error()` returns the reason for the last failed load on the calling thread. No byte outside `[data, data + size)` is read. The same function also reports path-load errors. (`voicedetect_capi_load` used to return `NULL` without a reason.)
+- **Threads.** Loads are independent. Different threads may load at the same time, from different buffers or from the same read-only buffer. A context is used by one thread at a time, as before.
+- **C++.** `vd::Model::load_from_memory(data, size, &err)` and `vd::ModelLoader::load_from_memory(data, size)`. `vd::Model::load(path, &err)` gained the same optional error out-parameter.
+
+**Models inside a larger GGUF (bundles).** A GGUF can hold several models if every key and every tensor name of a model carries a prefix, for example `voice.`. `voicedetect_capi_load_from_memory_prefixed(data, size, "voice.")` reads such a component straight from the whole bundle. Only the tensors under the prefix are copied, so the caller does not need to build a standalone copy of the component first. Tensor names that are stored as string values in the model's metadata (the block manifests) are not prefixed: the loader uses them as they are. A prefix that matches no tensor is an error. An empty prefix behaves like `voicedetect_capi_load_from_memory`.
+
+Limits: the buffer must hold the complete GGUF (the loader does not stream). The tensors are copied once, so the peak memory is the buffer plus the model.
+
 ---
 
 ## Model coverage
@@ -243,6 +263,8 @@ export VOICEDETECT_TEST_AUDIO=tests/fixtures/clip.wav
 ctest --test-dir build --output-on-failure
 ```
 
+`test_load_memory` is model-independent: it builds GGUF files and bundles in memory and checks equivalence, truncated and corrupt buffers, and concurrent loads. `test_load_memory_embed` needs `VOICEDETECT_TEST_GGUF` and `VOICEDETECT_TEST_AUDIO` and checks that path, memory and prefixed loads give bitwise-identical embeddings.
+
 Tests labelled `model` return exit code 77 (ctest SKIP) when their required env vars or checkpoints are absent, so they never break a CI environment that has no model. `scripts/gpu_verify.sh` runs the parity gates plus the benchmark on a CUDA host (GPU only, not CI).
 
 ---

include/voicedetect_capi.h

@@ -1,6 +1,8 @@
 #ifndef VOICEDETECT_CAPI_H
 #define VOICEDETECT_CAPI_H
 
+#include <stddef.h>
+
 #ifdef __cplusplus
 extern "C" {
 #endif
@@ -23,8 +25,8 @@ extern "C" {
 typedef struct voicedetect_ctx voicedetect_ctx;
 
 // ABI version of this header/implementation. Bump on any breaking change to the
-// function signatures or semantics below. Additive changes (new functions) are
-// fine without a bump; LocalAI checks this integer for compatibility.
+// function signatures or semantics below. Additive changes (new functions, such
+// as the load_from_memory family) are fine without a bump; LocalAI checks this integer for compatibility.
 //
 // v1: initial flat C-API surface - load/free/last_error/free_string,
 //     embed_path / embed_pcm (+ free_vec), verify_paths, analyze_path_json.
@@ -35,6 +37,37 @@ int voicedetect_capi_abi_version(void);
 // voicedetect_capi_free.
 voicedetect_ctx* voicedetect_capi_load(const char* gguf_path);
 
+// Load a model from a GGUF held in memory (no file, no temporary file, no file
+// descriptor; works the same on every platform). `data` points to the complete
+// GGUF file of `size` bytes.
+//
+// OWNERSHIP: the loader copies the tensor data into its own memory during the
+// call. `data` is only read inside the call: the caller may free or overwrite it
+// as soon as the function returns, on success and on failure. While loading, the
+// buffer and the model are in memory together (peak = buffer + model).
+//
+// A NULL or empty buffer, a truncated buffer or a corrupt buffer gives NULL and
+// a message in voicedetect_capi_last_load_error(); no byte outside
+// [data, data+size) is read. Returns an owning context, released with
+// voicedetect_capi_free. Calls on different threads are independent.
+voicedetect_ctx* voicedetect_capi_load_from_memory(const void* data, size_t size);
+
+// As voicedetect_capi_load_from_memory, for a model that is one component of a
+// larger GGUF (a bundle): every metadata key and every tensor name of the model
+// is stored as `prefix` + name (for example "voice."). Pass the whole bundle and
+// the prefix: no standalone copy of the component is needed, and only the
+// tensors under the prefix are copied. Names stored as string values inside the
+// model's metadata (tensor manifests) are not prefixed. Same ownership and error
+// rules as above. A prefix that matches no tensor is an error. A NULL or empty
+// `prefix` behaves like voicedetect_capi_load_from_memory.
+voicedetect_ctx* voicedetect_capi_load_from_memory_prefixed(const void* data, size_t size,
+                                                            const char* prefix);
+
+// Reason for the last failed voicedetect_capi_load* call made on the CALLING
+// THREAD, or "" if the last such call succeeded. The pointer stays valid until
+// the next load call on the same thread. Never NULL.
+const char* voicedetect_capi_last_load_error(void);
+
 // Free a context obtained from voicedetect_capi_load. Safe on NULL.
 void voicedetect_capi_free(voicedetect_ctx* ctx);
 

src/gguf_check.cpp

@@ -0,0 +1,116 @@
+#include "gguf_check.hpp"
+
+#include "ggml.h"
+
+#include <cstdint>
+#include <cstring>
+
+namespace vd {
+
+namespace {
+
+struct Cursor {
+    const uint8_t* p;
+    size_t size;
+    size_t pos = 0;
+
+    bool have(uint64_t n) const { return n <= size - pos; }
+    bool skip(uint64_t n) {
+        if (!have(n)) return false;
+        pos += (size_t)n;
+        return true;
+    }
+    template <typename T>
+    bool read(T* out) {
+        if (!have(sizeof(T))) return false;
+        std::memcpy(out, p + pos, sizeof(T));
+        pos += sizeof(T);
+        return true;
+    }
+};
+
+// Size in bytes of a fixed-size GGUF value type, 0 for string/array/unknown.
+size_t scalar_size(uint32_t t) {
+    switch (t) {
+        case 0: case 1: case 7: return 1;   // u8, i8, bool
+        case 2: case 3: return 2;           // u16, i16
+        case 4: case 5: case 6: return 4;   // u32, i32, f32
+        case 10: case 11: case 12: return 8;  // u64, i64, f64
+        default: return 0;
+    }
+}
+
+constexpr uint32_t kString = 8, kArray = 9;
+constexpr uint64_t kMaxString = 1ull << 30;  // same cap as ggml's reader
+
+bool skip_string(Cursor& c) {
+    uint64_t n;
+    return c.read(&n) && n <= kMaxString && c.skip(n);
+}
+
+}  // namespace
+
+bool gguf_precheck(const void* data, size_t size, std::string* err) {
+    auto fail = [&](const char* m) {
+        if (err) *err = m;
+        return false;
+    };
+    Cursor c{static_cast<const uint8_t*>(data), size};
+    uint32_t version;
+    int64_t n_tensors, n_kv;
+    if (!c.have(4) || std::memcmp(c.p, "GGUF", 4) != 0) return fail("not a GGUF file (bad magic)");
+    c.pos = 4;
+    if (!c.read(&version) || (version != 2 && version != 3))
+        return fail("unsupported GGUF version");
+    if (!c.read(&n_tensors) || !c.read(&n_kv) || n_tensors < 0 || n_kv < 0)
+        return fail("GGUF header is truncated or has a negative count");
+    // Every key/value pair takes at least 13 bytes, so a count that cannot fit is
+    // corrupt (and must not drive a long loop).
+    if ((uint64_t)n_kv > (size - c.pos) / 13) return fail("GGUF metadata count exceeds the buffer size");
+
+    for (int64_t i = 0; i < n_kv; ++i) {
+        uint64_t klen;
+        if (!c.read(&klen) || klen > kMaxString || !c.have(klen))
+            return fail("GGUF metadata key is truncated");
+        if (klen == 0) return fail("GGUF metadata key has an empty name");
+        c.skip(klen);
+        uint32_t type;
+        if (!c.read(&type)) return fail("GGUF metadata is truncated");
+        if (type == kString) {
+            if (!skip_string(c)) return fail("GGUF string value is truncated");
+        } else if (type == kArray) {
+            uint32_t et;
+            uint64_t n;
+            if (!c.read(&et) || !c.read(&n)) return fail("GGUF array header is truncated");
+            if (et == kString) {
+                for (uint64_t j = 0; j < n; ++j)
+                    if (!skip_string(c)) return fail("GGUF string array is truncated");
+            } else {
+                const size_t es = scalar_size(et);
+                if (es == 0) return fail("GGUF array has an unsupported element type");
+                if (n > size / es || !c.skip(n * es)) return fail("GGUF array is truncated");
+            }
+        } else {
+            const size_t s = scalar_size(type);
+            if (s == 0) return fail("GGUF metadata has an unknown value type");
+            if (!c.skip(s)) return fail("GGUF metadata is truncated");
+        }
+    }
+
+    // Tensor table: a type id outside the ggml enum is read by ggml into an enum
+    // before it is validated (undefined behavior), so reject it here.
+    if ((uint64_t)n_tensors > (size - c.pos) / 24) return fail("GGUF tensor count exceeds the buffer size");
+    for (int64_t i = 0; i < n_tensors; ++i) {
+        uint64_t nlen;
+        uint32_t n_dims, type;
+        if (!c.read(&nlen) || nlen == 0 || nlen > kMaxString || !c.skip(nlen))
+            return fail("GGUF tensor name is truncated or empty");
+        if (!c.read(&n_dims) || n_dims == 0 || n_dims > GGML_MAX_DIMS)
+            return fail("GGUF tensor has an invalid number of dimensions");
+        if (!c.skip(8ull * n_dims) || !c.read(&type) || type >= (uint32_t)GGML_TYPE_COUNT || !c.skip(8))
+            return fail("GGUF tensor table is truncated or has an unknown tensor type");
+    }
+    return true;
+}
+
+}  // namespace vd

src/gguf_check.hpp

@@ -0,0 +1,21 @@
+#pragma once
+#include <cstddef>
+#include <string>
+
+namespace vd {
+
+// Cheap structural check of a GGUF held in memory, run BEFORE the buffer is
+// handed to ggml's reader. ggml validates sizes and bounds, but it still aborts
+// the whole process (GGML_ASSERT) on a few well-formed-looking inputs, such as a
+// metadata key with an empty name. A model buffer comes from the caller, so such
+// input must become an error instead.
+//
+// Walks the header, the metadata key/value section and the tensor table with
+// bounds checks on every read. Returns false and sets `err` on: a bad magic or
+// version, an empty or oversized key, an unknown value type, a nested array, a
+// tensor with an empty name, a bad dimension count or an unknown type id, or
+// anything that runs past the end of the buffer. Tensor offsets and sizes are
+// not checked here: the loaders check them against the buffer size.
+bool gguf_precheck(const void* data, size_t size, std::string* err);
+
+}  // namespace vd

src/model.cpp

@@ -11,14 +11,34 @@
 
 namespace vd {
 
-std::unique_ptr<Model> Model::load(const std::string& gguf_path) {
+std::unique_ptr<Model> Model::load(const std::string& gguf_path, std::string* err) {
     auto m = std::unique_ptr<Model>(new Model());
     if (!m->loader_.load(gguf_path)) {
+        if (err) *err = m->loader_.error();
         return nullptr;  // bad/missing GGUF, or required KV absent
     }
     return m;
 }
 
+std::unique_ptr<Model> Model::load_from_memory(const void* data, size_t size, std::string* err) {
+    auto m = std::unique_ptr<Model>(new Model());
+    if (!m->loader_.load_from_memory(data, size)) {
+        if (err) *err = m->loader_.error();
+        return nullptr;
+    }
+    return m;
+}
+
+std::unique_ptr<Model> Model::load_from_memory(const void* data, size_t size,
+                                               const std::string& prefix, std::string* err) {
+    auto m = std::unique_ptr<Model>(new Model());
+    if (!m->loader_.load_from_memory(data, size, prefix)) {
+        if (err) *err = m->loader_.error();
+        return nullptr;
+    }
+    return m;
+}
+
 std::vector<float> Model::embed_16k(const std::vector<float>& pcm16k) const {
     // Stage 1: Kaldi-compatible FBank front end -> feat-major [n_mels, T].
     FBank fbank(loader_.config());

src/model.hpp

@@ -18,7 +18,23 @@ namespace vd {
 class Model {
 public:
     // Loads the GGUF at `gguf_path`. Returns nullptr on failure (no throw).
-    static std::unique_ptr<Model> load(const std::string& gguf_path);
+    // On failure `*err` (if given) receives the reason.
+    static std::unique_ptr<Model> load(const std::string& gguf_path, std::string* err = nullptr);
+
+    // Loads a GGUF held in memory. The tensor data is copied during the call:
+    // `data` may be freed or overwritten as soon as this returns. Returns nullptr
+    // on a NULL/empty, truncated or corrupt buffer (reason in `*err`, no throw,
+    // no read outside [data, data+size)). Safe to call from several threads on
+    // different buffers (or the same read-only buffer).
+    static std::unique_ptr<Model> load_from_memory(const void* data, size_t size,
+                                                   std::string* err = nullptr);
+
+    // As load_from_memory for a model stored inside a larger GGUF (a bundle):
+    // every key and tensor name of the model carries `prefix` (for example
+    // "voice."). Only the tensors under the prefix are copied.
+    static std::unique_ptr<Model> load_from_memory(const void* data, size_t size,
+                                                   const std::string& prefix,
+                                                   std::string* err = nullptr);
 
     // Compute the L2-normalized speaker embedding from raw mono float PCM. If
     // `sample_rate != 16000` the audio is linearly resampled to 16 kHz first.