Skip to content

align c++ official Needle inference - #637

Merged
HenryNdubuaku merged 1 commit into
mainfrom
align_needle_inference_logic
May 13, 2026
Merged

HenryNdubuaku merged 1 commit into
mainfrom
align_needle_inference_logic

Conversation

@jakmro

@jakmro jakmro commented May 13, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

Copilot AI review requested due to automatic review settings May 13, 2026 02:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the FFI/tooling path and constraint logic for Config::ModelType::NEEDLE to better match “official Needle inference”, primarily by snake-casing tool names for Needle, reformatting Needle tool JSON, and updating Needle-specific token-bias constraints.

Changes:

  • Added Needle helpers to snake-case tool names and restore original tool names in parsed function calls.
  • Updated Needle tool serialization to emit Needle-oriented tool JSON and to apply snake-cased names to tool constraints.
  • Simplified Needle constraints in ToolCallConstrainer and adjusted blocking bias behavior for Needle.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File Description
cactus/ffi/needle_tools.h Adds to_snake_case and a helper to restore original tool names in function-call JSON.
cactus/ffi/cactus_complete.cpp Renames Needle tool names to snake_case during serialization/constraints and restores original names after parsing tool calls; adjusts Needle parameter serialization.
cactus/engine/engine.h Removes unused Needle constraint bookkeeping fields/method declarations from ToolCallConstrainer.
cactus/engine/engine_constraints.cpp Updates Needle constraint initialization and biasing logic (special token skipping, Needle-only block bias, simplified arg-key constraints).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 358 to 360
if (tools.empty()) {
return (raw_tools_json && std::strlen(raw_tools_json) > 0) ? raw_tools_json : "[]";
return "[]";
}
Comment on lines +369 to +372
std::string snake_name = needle::to_snake_case(tools[i].name);
name_map[snake_name] = tools[i].name;

oss << "{\"name\":\"" << escape_json_string(snake_name) << "\"";

if (properties.empty()) {
oss << schema_it->second;
oss << "{}";
Comment on lines +331 to +340
if (cp <= 0xFFFF) {
char buf[8];
std::snprintf(buf, sizeof(buf), "\\u%04x", cp);
compact += buf;
} else {
cp -= 0x10000;
char buf[14];
std::snprintf(buf, sizeof(buf), "\\u%04x\\u%04x",
0xD800 + (cp >> 10), 0xDC00 + (cp & 0x3FF));
compact += buf;
Comment on lines +241 to +244
constexpr uint32_t pad = 0;

for (uint32_t token_id = 0; token_id < vocab_size; ++token_id) {
if (token_id == pad || token_id == eos || token_id == bos || token_id == unk) {
@HenryNdubuaku
HenryNdubuaku merged commit 7276100 into main May 13, 2026
6 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants