diff --git a/README.md b/README.md index 8edec0f..65ae30f 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **OpenArc** is an inference engine for Intel devices. -Serve LLMs, VLMs, Whisper, Kokoro-TTS, Qwen-TTS, Qwen-ASR, Embedding and Reranker models over OpenAI compatible endpoints, powered by OpenVINO on your device. Local, private, open source AI. +Serve LLMs, VLMs, Whisper, Kokoro-TTS, Qwen-TTS, Qwen-ASR, Embedding and Reranker models over OpenAI compatible endpoints, powered by OpenVINO on your device. Local, private, open source AI. OpenArc enables you to host speech to text, text to speech and an LLM on the same server, at the same time. OpenArc is a community-driven effort to make acceleration from OpenVINO easier to access, deploy and leverage for our usecases. @@ -31,6 +31,8 @@ Thanks to everyone on Discord for their continued support! ## Features + - + - Support for openvino genai `scheduler_config` - NEW! Containerization with Docker #60 by @meatposes - NEW! Speculative decoding support for LLMs #57 by @meatposes - NEW! Streaming cancellation support for LLMs and VLMs @@ -46,8 +48,7 @@ Thanks to everyone on Discord for their continued support! - `/v1/embeddings`: `qwen3-embedding` #33 by @mwrothbe - `/v1/rerank`: `qwen3-reranker` #39 by @mwrothbe - `jinja` templating with `AutoTokenizers` - - OpenAI Compatible tool calls with streaming and paralell - - tool call parser currently reads "name", "argument" + - OpenAI Compatible tool and reasoning parsing - Fully async multi engine, multi task architecture - Model concurrency: load and infer multiple models at once - Automatic unload on inference failure @@ -66,7 +67,7 @@ Thanks to everyone on Discord for their continued support! > [!NOTE] -> Interested in contributing? Please open an issue before submitting a PR! +> Interested in contributing? Please discuss with us on discord or open an issue before submitting a PR! ## Acknowledgments diff --git a/config.yaml b/config.yaml index 978a95c..7c16a87 100644 --- a/config.yaml +++ b/config.yaml @@ -11,7 +11,7 @@ startup_models : ${OPENRARC_AUTOLOAD_MODELS} models: qwen35-08b: engine: ovgenai - model_type: llm + model_type: vlm model_path: /mnt/Ironwolf-4TB/Models/OpenVINO/Qwen3.5/Qwen3.5-0.8B-int8_asym-ov/ device: CPU tool_call_parser: qwen35 @@ -107,10 +107,26 @@ models: stream_chunk_frames: stream_left_context: - qwen3_tts_oscar: model_type: qwen3_tts_voice_clone model_path: engine: openvino device: GPU.0 - qwen3_tts_voice_clone_options: \ No newline at end of file + qwen3_tts_voice_clone_options: + ref_text: + x_vector_only: + instruct: + qwen3_tts_options: + max_new_tokens: + do_sample: + top_k: + top_p: + temperature: + repetition_penalty: + subtalker_do_sample: + subtalker_top_k: + subtalker_top_p: + subtalker_temperature: + stream: + stream_chunk_frames: + stream_left_context: \ No newline at end of file diff --git a/docs/assets/tokens_as_straight_line_boundary_example.png b/docs/assets/tokens_as_straight_line_boundary_example.png new file mode 100644 index 0000000..d2d4056 Binary files /dev/null and b/docs/assets/tokens_as_straight_line_boundary_example.png differ diff --git a/docs/assets/tokens_as_straight_line_with_checkpoints.png b/docs/assets/tokens_as_straight_line_with_checkpoints.png new file mode 100644 index 0000000..7eda615 Binary files /dev/null and b/docs/assets/tokens_as_straight_line_with_checkpoints.png differ diff --git a/docs/assets/unnamed.png b/docs/assets/unnamed.png new file mode 100644 index 0000000..a7b648d Binary files /dev/null and b/docs/assets/unnamed.png differ diff --git a/docs/configure.md b/docs/configure.md index bf8d743..ca3b21a 100644 --- a/docs/configure.md +++ b/docs/configure.md @@ -1,4 +1,4 @@ -runtime_config +## runtime_config runtime_config is an OpenArc entrypoint to the *properties* way of configuring openvino runtime. These settings allow users to tune the behavior of openivno runtime without needing to change application logic and are meant to be "portable", requring no code changes. Since OpenArc diff --git a/docs/index.md b/docs/index.md index be151bb..f2e7890 100644 --- a/docs/index.md +++ b/docs/index.md @@ -13,7 +13,6 @@ icon: lucide/chess-rook Welcome to the OpenArc documentation! - ## Installation - [Linux](install.md#linux) @@ -32,6 +31,18 @@ OpenArc includes a command line tool for controlling the server. - [openarc bench](commands.md#bench) — Benchmarking tool for LLMs. - [openarc tool](commands.md#tool) — OpenVINO utilities. +## Configuration + +>Under construction! + +- [Advanced openvino properties](configure.md#runtime_config) + + +## Concepts + +- [Tool and Reasoning Parsing](tool_use.md#tool-and-reasoning-parsing) + + ## Models Models to get you started and where to find more! @@ -40,7 +51,7 @@ OpenArc is deeply integrated with the Huggingface Ecosytem and has been written We are working on improving this process with experimental GGUF support coming, as well as a new frontend application similar to LM-Studio! -Below are some models to get started which are known to work. My hugg +Below are some models to get started which are known to work. My huggingface has many - [Model Sources](models.md#sources) - [LLMs](models.md#llms) diff --git a/docs/models.md b/docs/models.md index 8c9e487..be57dfc 100644 --- a/docs/models.md +++ b/docs/models.md @@ -100,13 +100,15 @@ If you need help converting a particular model join Discord and we can help you! ## Model-Specific Instructions -### Qwen3.5/3.6 +### Tool and Reasoning Parsing -**Is Qwen3.5/3.6 supported?** -Qwen3.5 models has unofficial support. However, they do require you to build `openvino` and `openvino.genai` from source. You will also need to install the latest version of `optimum-intel`. +OpenArc now supports tool and reasoning parsing for several architectures. Our approach ensures correct openai compatible parsing using openvino genai by leveraging some useful facts about tokens. + +Since Autoregressive language models emit tokens in a continuous stream one by one, we can visualize it this way +- streaming decoded tokens to a buff + -To add a model, run the command `openarc add --model-name MODEL_NAME --model-path /path/to/model --model-type vlm --device GPU|CPU --runtime-config '{"ATTENTION_BACKEND": "SDPA"}'`. OpenArc resolves the VLM vision token from the model's `config.json`. Intel is currently working on adding support for Qwen3.5 to utilize the PA attention backend but it has not been merged yet. This currently appears to be much more performant. If you have built `openvino.genai` with the support included, you may change the runtime config parameter to use PA instead. **How do I control thinking?** diff --git a/docs/tool_use.md b/docs/tool_use.md new file mode 100644 index 0000000..8f8b529 --- /dev/null +++ b/docs/tool_use.md @@ -0,0 +1,49 @@ + +## Tool and Reasoning Parsing + +OpenArc now supports "proper" tool and reasoning parsing for several architectures. Our approach ensures correct openai compatible parsing using openvino genai by leveraging some useful facts about tokens and will be the strategy going forward so I figured it would be a good idea to write down how it works. + +These changes took a few months (of inconsistent work :D) to get right due to difficulties in the deepest undocumented parts of openvino and openvino genai. Plus, I just wasn't happy where every attempted implementation ended up. + +To motivate our approach lets talk about tokens to capture some intuition. + +### Tokens as a straight line + +Since Autoregressive language models emit tokens in a continuous stream one by one, it is safe to visualize them as a straight line. + + +Taking some liberties for readability we can visualize tokens as single words in a sequence starting at 1: + +![Tokens as a straight line](assets/tokens_as_straight_line.png) + +In most tokenizers `` is exactly one token, a fact we can use to check where the boundaries are in the raw token ids, before any algorithm touches them. A key thing is that the `<` can make it impossible to tokenize `` into exactly one id- and because tokens always occur in a sequence determined by probabilities, we cannot easily check where these critical tokens are along our straight line. So, if a model emits a think tag in its reasoning and our code checks text for think tags we are screwed, and need a better way to make sure programs who use the output of ai systems get content in a consumable format. + +If we know that the important tokens who always have a stable id can be used as boundaries to check raw token ids, we can imagine `` tags like a checkpoints along our sequence: + +![Tokens as a straight line with checkpoints](assets/tokens_as_straight_line_with_checkpoints.png) + +Now that we can be sure where the checkpoints we can write code that slices each section into its content, and safely tokenize each portion directly into buckets which are routed where they need to go. + +![Token stream sliced into buckets at boundaries](assets/tokens_as_straight_line_boundary_example.png) + + +So, for all models where this approach can be applied we will use it! The code lives under [src/engine/ov_genai/tool_parse](https://github.com/SearchSavior/OpenArc/tree/main/src/engine/ov_genai/tool_parse). Formally it follows a state machine pattern. + + + + + + + + + + + + + + + +Further reading: + +[How does tokenization work, and why do LLMs usually rely on subword tokenizers such as BPE?](https://sebastianraschka.com/faq/docs/tokenization-bpe.html) - great overview +