Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
dfa3fb5
gemma-metal
jakmro Jun 29, 2026
e6934b2
cleanup
jakmro Jul 1, 2026
5ccf7b0
cleanup v2
jakmro Jul 1, 2026
e81cc39
faster
jakmro Jul 1, 2026
57b3cf0
even faster
jakmro Jul 1, 2026
cba8a74
converter
jakmro Jul 1, 2026
49246f0
Fix C++ test build: expose vendored libs include (picojson) (#748)
rshemet Jul 1, 2026
648c28a
Trie-based grammar constraints for Gemma force_tools (#747)
rshemet Jul 1, 2026
2a48b6c
fixes + speedups
jakmro Jul 2, 2026
528ebe5
cq2/3
jakmro Jul 2, 2026
4a2efc4
cleanup
jakmro Jul 2, 2026
c7ae754
cleanup2
jakmro Jul 2, 2026
bc14911
Metal execution for vision and audio encoder towers
jakmro Jul 2, 2026
c64dcea
Generalize gemma decode into cactus-graph with automatic Metal fusion
jakmro Jul 2, 2026
c793e47
Metal pipeline optimization: fix flushed-command-buffer sync hazard, …
jakmro Jul 2, 2026
1e77c18
Metal encodes for the full elementwise, reduction, conv, norm, rope, …
jakmro Jul 2, 2026
0dc8631
Generic elementwise chain fusion for the Metal plan compiler
jakmro Jul 2, 2026
ed66391
Style and DRY pass over the Metal branch
jakmro Jul 2, 2026
6439dfd
Release Metal prefill dequant cache after each completion
jakmro Jul 2, 2026
159244c
Fix MatMul CQ staleness, pin retained outputs against fusion, PR revi…
jakmro Jul 2, 2026
25ab7ca
Flash attention kernel and kernel-level bandwidth pass for the towers
jakmro Jul 2, 2026
55d8986
Epilogue-style fusion: broadcast chain sides and fused residual-add r…
jakmro Jul 2, 2026
36a40e6
Review pass: close latent correctness holes and remove dead Metal paths
jakmro Jul 2, 2026
786f3cf
fixes
jakmro Jul 3, 2026
9a077bc
c
jakmro Jul 3, 2026
ac1af8c
fixes
jakmro Jul 4, 2026
240304f
fixes
jakmro Jul 4, 2026
82a1253
Merge origin/main into gemma-metal
jakmro Jul 4, 2026
5217de3
remove compilation warnings
jakmro Jul 4, 2026
73e47e1
fix
jakmro Jul 4, 2026
bcb5d0b
optimize gemma 4 26b
jakmro Jul 6, 2026
e76850d
more optim
jakmro Jul 6, 2026
81d57f4
fixes
jakmro Jul 6, 2026
3be1656
cleanup
jakmro Jul 6, 2026
39835a9
more kernels
jakmro Jul 6, 2026
7cc2660
ram optimizations
jakmro Jul 7, 2026
0b6feda
update default model IDs to match existing configurations
HenryNdubuaku Jul 7, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 16 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,19 +11,13 @@

A hybrid edge-cloud AI engine for mobile devices & wearables.

- **Fast & accurate:** fastest inference on ARM CPU, Cactus quants at 4-bit matches f16
- **Low RAM:** zero-copy memory mapping ensures 10x lower RAM use than other engines
- **Multimodal:** one engine for speech, vision, and language models
- **Cloud fallback:** automatically route requests to cloud models if needed
- **Model-Agnostic:** Custom PyTorch models can be exported to the Cactus runtime.

```
┌─────────────────┐
│ Cactus Engine │ ←── OpenAI-compatible APIs for text, speech, and vision.
└─────────────────┘
│
┌─────────────────┐
│ Cactus Graph │ ←── Zero-copy computation graph ensures 10x lower RAM
│ Cactus Graph │ ←── Zero-copy computation graph
└─────────────────┘
│
┌─────────────────┐
Expand Down Expand Up @@ -128,14 +122,14 @@ graph.hard_reset();

## Benchmarks

- LLM: Gemma-4-E2B-CQ4 (CPU, no speculative decode), 1k-prefill tps / 100-decode tps
- VLM: Gemma-4-E2B-CQ4 (NPU prefill, CPU decode), 256px input, latency / decode tps
- Transcribe: Parakeet-TDT-0.6B-CQ4 (NPU prefill, CPU decode), 20s audio, latency / decode tps
- LLM: Gemma-4-E2B-CQ4, 1k-prefill tps / 100-decode tps
- VLM: Gemma-4-E2B-CQ4 256px input, latency / decode tps
- Transcribe: Parakeet-TDT-0.6B-CQ4, 20s audio, latency / decode tps
- Missing latency == no NPU support for device

| Device | LLM | VLM | Transcribe | RAM |
| Device | LLM | VLM | Transcribe | 1k-Context RAM |
|--------|----------|------------|---------------|-----|
| Mac M4 Pro | 324 / 39 | 1.2s / 48 | 0.2s / 10.6M | 1385 MB |
| Mac M4 Pro | 2619tps / 139tps | 0.4s / 160tps | 0.1s / 11Mtps | 1327 MB |
| Mac M3 Pro | 390 / 26 | 2.76s / 28.06 | 0.32s / 2.29M | 1376 MB |
| iPhone 17 Pro | - | - | - | - |
| iPhone 13 Mini | - | - | - | - |
Expand Down Expand Up @@ -196,7 +190,8 @@ graph.hard_reset();
│ │
│ cactus run [model|path] run a model (downloads if needed) │
│ --bits 1|2|3|4 CQ quantization (default: 4) │
│ --platform auto|cpu|apple target platform (default: auto) │
│ --weights general|apple weights bundle variant (default: general) │
│ --backend auto|cpu|metal inference backend (default: auto) │
│ --image <path> image file for VLM inference │
│ --audio <path> audio file for audio chat │
│ --system <prompt> system prompt │
Expand All @@ -209,20 +204,20 @@ graph.hard_reset();
│ --file <audio.wav> audio file to transcribe (WAV) │
│ --language <code> language code (default: en) │
│ --bits 1|2|3|4 CQ quantization (default: 4) │
│ --platform auto|cpu|apple target platform (default: auto) │
│ --weights general|apple weights bundle variant (default: general) │
│ --token <token> HuggingFace token (gated models) │
│ --reconvert force local rebuild from source │
│ │
│ cactus download [model] get a bundle (prebuilt, else build) │
│ --bits 1|2|3|4 CQ quantization (default: 4) │
│ --platform auto|cpu|apple target platform (default: auto) │
│ --weights general|apple weights bundle variant (default: general) │
│ --token <token> HuggingFace token (gated models) │
│ --reconvert force local rebuild from source │
│ │
│ cactus convert <model> [dir] HuggingFace -> runnable cactus bundle │
│ (CQ weights + runtime graph) │
│ --bits 1|2|3|4 CQ quantization (default: 4) │
│ --platform auto|cpu|apple target platform (default: auto) │
│ --weights general|apple weights bundle variant (default: general) │
│ --token <token> HuggingFace token (gated models) │
│ --reconvert force local rebuild from source │
│ --lora <path> merge a LoRA adapter before converting │
Expand All @@ -233,7 +228,8 @@ graph.hard_reset();
│ --host <addr> bind address (default: 127.0.0.1) │
│ --port <port> port (default: 8080) │
│ --bits 1|2|3|4 CQ quantization (default: 4) │
│ --platform auto|cpu|apple target platform (default: auto) │
│ --weights general|apple weights bundle variant (default: general) │
│ --backend auto|cpu|metal inference backend (default: auto) │
│ --token <token> HuggingFace token (gated models) │
│ --reconvert force local rebuild from source │
│ --no-cloud-handoff disable automatic cloud handoff │
Expand All @@ -243,7 +239,8 @@ graph.hard_reset();
│ cactus code run the AI coding agent (TUI / print) │
│ --serve-model <id> auto-start a server with this model │
│ --bits 1|2|3|4 CQ quantization (default: 4) │
│ --platform auto|cpu|apple target platform (default: auto) │
│ --weights general|apple weights bundle variant (default: general) │
│ --backend auto|cpu|metal inference backend (default: auto) │
│ --token <token> HuggingFace token (gated models) │
│ --reconvert force local rebuild from source │
│ --host <addr> server address (default: 127.0.0.1) │
Expand All @@ -267,7 +264,7 @@ graph.hard_reset();
│ --model <hf-id> default: LiquidAI/LFM2-VL-450M │
│ --transcription-model <hf-id> default: openai/whisper-base │
│ --bits 1|2|3|4 CQ quantization (default: 4) │
│ --platform auto|cpu|apple target platform (default: auto) │
│ --weights general|apple weights bundle variant (default: general) │
│ --token <token> HuggingFace token (gated models) │
│ --reconvert force local rebuild of test models │
│ --suite <name> run a single test suite by name │
Expand Down
6 changes: 3 additions & 3 deletions apple/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
cmake_minimum_required(VERSION 3.10)
project(CactusApple LANGUAGES CXX)

set(CMAKE_OSX_DEPLOYMENT_TARGET "13.0" CACHE STRING "")
set(CMAKE_XCODE_ATTRIBUTE_IPHONEOS_DEPLOYMENT_TARGET "13.0" CACHE STRING "")
set(CMAKE_OSX_DEPLOYMENT_TARGET "13.3" CACHE STRING "")
set(CMAKE_XCODE_ATTRIBUTE_IPHONEOS_DEPLOYMENT_TARGET "16.4" CACHE STRING "")
set(CMAKE_BUILD_TYPE Release CACHE STRING "")
set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/lib)

Expand All @@ -27,7 +27,7 @@ if(BUILD_SHARED_LIBS)
)

target_link_libraries(cactus_framework PUBLIC
"-framework Accelerate" "-framework CoreML" "-framework Foundation"
"-framework Accelerate" "-framework CoreML" "-framework Foundation" "-framework Metal" "-framework MetalPerformanceShaders"
"-framework Security" "-framework SystemConfiguration" "-framework CFNetwork"
)

Expand Down
2 changes: 1 addition & 1 deletion apple/Info.plist
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,6 @@
<key>CFBundleVersion</key>
<string>1.0.0</string>
<key>MinimumOSVersion</key>
<string>13.0</string>
<string>13.3</string>
</dict>
</plist>
8 changes: 5 additions & 3 deletions apple/build.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
#!/bin/bash
IOS_MIN_VERSION=16.4
MACOS_MIN_VERSION=13.3
set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
Expand Down Expand Up @@ -59,7 +61,7 @@ function build_static_library() {

cmake -DCMAKE_SYSTEM_NAME=iOS \
-DCMAKE_OSX_ARCHITECTURES=arm64 \
-DCMAKE_OSX_DEPLOYMENT_TARGET=13.0 \
-DCMAKE_OSX_DEPLOYMENT_TARGET=$IOS_MIN_VERSION \
-DCMAKE_OSX_SYSROOT="$IOS_SDK_PATH" \
-DCMAKE_BUILD_TYPE="$CMAKE_BUILD_TYPE" \
-DBUILD_SHARED_LIBS=OFF \
Expand All @@ -86,7 +88,7 @@ function build_static_library() {

cmake -DCMAKE_SYSTEM_NAME=iOS \
-DCMAKE_OSX_ARCHITECTURES=arm64 \
-DCMAKE_OSX_DEPLOYMENT_TARGET=13.0 \
-DCMAKE_OSX_DEPLOYMENT_TARGET=$IOS_MIN_VERSION \
-DCMAKE_OSX_SYSROOT="$IOS_SIM_SDK_PATH" \
-DCMAKE_BUILD_TYPE="$CMAKE_BUILD_TYPE" \
-DBUILD_SHARED_LIBS=OFF \
Expand All @@ -110,7 +112,7 @@ function build_framework() {
-DCMAKE_SYSTEM_NAME=$1 \
-DCMAKE_OSX_ARCHITECTURES="$2" \
-DCMAKE_OSX_SYSROOT=$3 \
-DCMAKE_OSX_DEPLOYMENT_TARGET=13.0 \
-DCMAKE_OSX_DEPLOYMENT_TARGET=$([ "$1" = "iOS" ] && echo "$IOS_MIN_VERSION" || echo "$MACOS_MIN_VERSION") \
-DCMAKE_BUILD_TYPE="$CMAKE_BUILD_TYPE" \
-DBUILD_SHARED_LIBS=ON \
-DCACTUS_CURL_ROOT="$CACTUS_CURL_ROOT" \
Expand Down
4 changes: 2 additions & 2 deletions cactus-engine/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ target_link_libraries(cactus_engine PUBLIC cactus_graph)

if(APPLE)
target_link_libraries(cactus_engine PUBLIC
"-framework CoreML" "-framework Foundation"
"-framework CoreML" "-framework Foundation" "-framework Metal" "-framework MetalPerformanceShaders"
"-framework Security" "-framework SystemConfiguration" "-framework CFNetwork"
)

Expand Down Expand Up @@ -156,7 +156,7 @@ if(CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR)
target_link_libraries(cactus_engine_shared PRIVATE "${_curl}")
endif()
target_link_libraries(cactus_engine_shared PRIVATE
"-framework Accelerate" "-framework CoreML" "-framework Foundation"
"-framework Accelerate" "-framework CoreML" "-framework Foundation" "-framework Metal" "-framework MetalPerformanceShaders"
"-framework Security" "-framework SystemConfiguration" "-framework CFNetwork"
)
else()
Expand Down
2 changes: 2 additions & 0 deletions cactus-engine/cactus_engine.h
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ CACTUS_FFI_EXPORT void cactus_destroy(cactus_model_t model);
CACTUS_FFI_EXPORT void cactus_reset(cactus_model_t model);
CACTUS_FFI_EXPORT void cactus_stop(cactus_model_t model);

CACTUS_FFI_EXPORT int cactus_set_backend(const char* backend);

CACTUS_FFI_EXPORT int cactus_complete(
cactus_model_t model,
const char* messages_json,
Expand Down
8 changes: 8 additions & 0 deletions cactus-engine/src/complete.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
#include "chat_tools.h"
#include "telemetry.h"
#include "cactus_kernels.h"
#include "metal_backend.h"
#include "wav.h"
#include <algorithm>
#include <chrono>
Expand Down Expand Up @@ -796,6 +797,10 @@ int cactus_complete(
const uint8_t* pcm_buffer,
size_t pcm_buffer_size
) {
struct MetalTrimGuard {
~MetalTrimGuard() { cactus_metal_trim_prefill_cache(); }
} metal_trim_guard;

if (!model) {
std::string error_msg = last_error_message.empty() ?
"Model not initialized. Check model path and files." : last_error_message;
Expand Down Expand Up @@ -824,6 +829,9 @@ int cactus_complete(
}
auto* tokenizer = handle->model->get_tokenizer();
auto prompt = prepare_prompt(handle, messages_json, options_json, tools_json, true, true, pcm_buffer, pcm_buffer_size);
if (prompt.options.sample_seed != 0) {
handle->model->set_sample_seed(prompt.options.sample_seed);
}

CACTUS_LOG_DEBUG("complete", "Prompt tokens: " << prompt.tokens.size()
<< ", max_tokens: " << prompt.options.max_tokens);
Expand Down
9 changes: 6 additions & 3 deletions cactus-engine/src/constraints.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -124,11 +124,11 @@ void ToolCallConstrainer::mark_trie_bias(const TrieNode* node, char terminator,
}
if (!has_valid) return;

dense_bias_.assign(valid.size(), TRIE_BLOCK_BIAS);
for (size_t token_id = 0; token_id < valid.size(); ++token_id) {
if (!valid[token_id]) {
current_bias_[static_cast<uint32_t>(token_id)] = TRIE_BLOCK_BIAS;
}
if (valid[token_id]) dense_bias_[token_id] = 0.0f;
}
dense_ready_ = true;
}

void ToolCallConstrainer::reset_constraint_state() {
Expand Down Expand Up @@ -432,6 +432,7 @@ void ToolCallConstrainer::init(Config::ModelType model_type,
tokenizer_ = tokenizer;
generated_text_.clear();
current_bias_.clear();
dense_ready_ = false;
forced_tag_progress_ = 0;
const bool model_supported = is_needle() || Config::is_gemma_family(model_type_);
active_ = model_supported && !tool_specs_.empty() && tokenizer != nullptr;
Expand Down Expand Up @@ -520,6 +521,7 @@ void ToolCallConstrainer::update(uint32_t token_id, const std::string& decoded_t

void ToolCallConstrainer::compute_bias() {
current_bias_.clear();
dense_ready_ = false;

if (!active_) return;

Expand Down Expand Up @@ -615,6 +617,7 @@ void ToolCallConstrainer::compute_bias() {
void ToolCallConstrainer::reset() {
generated_text_.clear();
current_bias_.clear();
dense_ready_ = false;
forced_tag_progress_ = 0;
reset_constraint_state();

Expand Down
10 changes: 10 additions & 0 deletions cactus-engine/src/embed.cpp
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#include "../cactus_engine.h"
#include "utils.h"
#include "cactus_kernels.h"
#include "metal_backend.h"
#include "wav.h"
#include <cstring>
#include <algorithm>
Expand Down Expand Up @@ -37,6 +38,9 @@ int cactus_embed(
size_t* embedding_dim,
bool normalize
) {
struct MetalTrimGuard {
~MetalTrimGuard() { cactus_metal_trim_prefill_cache(); }
} metal_trim_guard;
if (!model || !text || !embeddings_buffer || buffer_size == 0) {
CACTUS_LOG_ERROR("embed", "Invalid parameters for text embedding");
return -1;
Expand Down Expand Up @@ -90,6 +94,9 @@ int cactus_image_embed(
size_t buffer_size,
size_t* embedding_dim
) {
struct MetalTrimGuard {
~MetalTrimGuard() { cactus_metal_trim_prefill_cache(); }
} metal_trim_guard;
if (!model || !image_path || !embeddings_buffer || buffer_size == 0) {
CACTUS_LOG_ERROR("image_embed", "Invalid parameters for image embedding");
return -1;
Expand Down Expand Up @@ -130,6 +137,9 @@ int cactus_audio_embed(
size_t buffer_size,
size_t* embedding_dim
) {
struct MetalTrimGuard {
~MetalTrimGuard() { cactus_metal_trim_prefill_cache(); }
} metal_trim_guard;
if (!model || !audio_path || !embeddings_buffer || buffer_size == 0) {
CACTUS_LOG_ERROR("audio_embed", "Invalid parameters for audio embedding");
return -1;
Expand Down
18 changes: 17 additions & 1 deletion cactus-engine/src/engine.h
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#pragma once

#include <vector>
#include <random>
#include <string>
#include <map>
#include <unordered_map>
Expand Down Expand Up @@ -522,6 +523,7 @@ class ToolCallConstrainer {
Tokenizer* tokenizer);

const std::unordered_map<uint32_t, float>& get_bias() const { return current_bias_; }
const std::vector<float>* get_dense_bias() const { return dense_ready_ ? &dense_bias_ : nullptr; }

void update(uint32_t token_id, const std::string& decoded_text);

Expand Down Expand Up @@ -561,6 +563,8 @@ class ToolCallConstrainer {
std::unordered_set<uint32_t> close_brace_tokens_;

std::unordered_map<uint32_t, float> current_bias_;
std::vector<float> dense_bias_;
bool dense_ready_ = false;

void compute_bias();
void tokenize_grammar_elements();
Expand Down Expand Up @@ -634,6 +638,7 @@ class Model {
uint32_t decode(const std::vector<uint32_t>& tokens, float temperature = -1.0f, float top_p = -1.0f,
size_t top_k = 0, const std::string& profile_file = "", float* out_entropy = nullptr,
float min_p = 0.15f, float repetition_penalty = 1.1f);
void set_sample_seed(uint64_t seed) { sample_rng_.seed(static_cast<std::mt19937::result_type>(seed)); }
bool prefill_and_sample_first_token(const std::vector<uint32_t>& tokens, uint32_t& out_token,
float* out_uncertainty = nullptr);

Expand Down Expand Up @@ -807,7 +812,7 @@ class Model {
bool load_component_graph(Component& comp);
void unload_component_graph(Component& comp);
bool bind_runtime_buffers(Component& comp);
void run_step(uint32_t token_id, size_t position, bool read_logits);
void run_step(uint32_t token_id, size_t position, bool read_logits, bool use_fused = true);
void run_step_batch(const std::vector<uint32_t>& token_ids, const std::vector<size_t>& positions);
void set_component_batch(Component& comp, size_t batch);
size_t decoder_cache_num_slots();
Expand Down Expand Up @@ -845,6 +850,9 @@ class Model {
size_t chunk_tokens, const std::vector<uint32_t>& tokens,
size_t processed, size_t start_position);
void run_full_context_text();
void prepare_sampling_context(float repetition_penalty);
uint32_t sample_component_logits(Component& comp, float temperature, float top_p, size_t top_k,
float min_p, bool greedy, float* out_uncertainty);
uint32_t argmax_component_logits(Component& comp, size_t logit_row = std::numeric_limits<size_t>::max(),
float* out_uncertainty = nullptr);
uint32_t argmax_logits_at(const BufferDesc& desc, void* ptr, size_t row_off, float* out_uncertainty);
Expand Down Expand Up @@ -942,6 +950,14 @@ class Model {

static constexpr size_t MAX_TOKEN_HISTORY = 128;
std::vector<uint32_t> token_history_;
std::vector<uint32_t> samp_recent_;
std::vector<float> samp_bias_dense_;
bool samp_has_bias_ = false;
float samp_penalty_ = 1.0f;
bool samp_ctx_active_ = false;
std::mt19937 sample_rng_{std::random_device{}()};
FusedEmbedCtx fused_embed_ctx_;
int ple_probe_state_ = 0;

ToolCallConstrainer tool_constrainer_;
std::unordered_map<uint32_t, float> vocab_bias_;
Expand Down
2 changes: 2 additions & 0 deletions cactus-engine/src/init.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -349,6 +349,8 @@ bool matches_stop_sequence(const std::vector<uint32_t>& generated_tokens,

extern "C" {

int cactus_set_backend(const char* backend) { return cactus_backend_select(backend); }

cactus_model_t cactus_init(const char* model_path, const char* corpus_dir, bool cache_index) {
constexpr size_t DEFAULT_CONTEXT_SIZE = 512; // matches default sliding window size

Expand Down
Loading
Loading