Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 25 additions & 5 deletions docs/serving.md
Original file line number Diff line number Diff line change
Expand Up @@ -324,6 +324,7 @@ wire response contains typed `output` Items.
| `metadata` | at most 16 string pairs; keys at most 64 characters and values at most 512 |
| `client_metadata` | Codex client extension; an object or `null`, accepted as opaque tracing metadata with no generation effect |
| `reasoning.effort` | `none` disables thinking; `low`, `medium`, or `xhigh` selects an effort exposed by the loaded chat template; `minimal`, `high`, and `max` return `reasoning_effort_not_supported` for the registered templates |
| `reasoning.summary` | omitted, `null`, or any string; every string requests the same fixed protocol placeholder without changing model execution, and the original value is echoed in the response |
| `chat_template_kwargs.preserve_thinking` | optional boolean controlling whether closed-turn reasoning remains in reconstructed prompts |
| `preserve_thinking` | top-level alias for the same option; conflicting values are rejected |
| `text.format` | omitted or `{"type":"text"}` only |
Expand All @@ -335,7 +336,7 @@ wire response contains typed `output` Items.
| `top_logprobs` | omitted or `0` |
| `service_tier` | omitted, `auto`, or `default`; the response reports `default` |
| `background` | omitted or `false` |
| `include` | omitted or an empty array |
| `include` | omitted, empty, or `["reasoning.encrypted_content"]`; the supported value requests the local raw-reasoning mirror described below |
| `stream_options.include_obfuscation` | optional boolean; accepted as a transport hint, but this local server emits no padding |
| cache and client hints | `prompt_cache_key`, `prompt_cache_options`, `prompt_cache_retention`, and explicit breakpoints follow [OpenAI prompt caching](#openai-prompt-caching); `safety_identifier` and `user` are accepted as client hints |

Expand Down Expand Up @@ -436,10 +437,15 @@ invocation are also rejected because their semantics cannot be honored.
A terminal wire response has `object: "response"`, one of `completed`, `incomplete`, or
`cancelled` in `status`, and a typed `output` array. NInfer may emit:

- a `reasoning` Item containing raw `reasoning_text` and an empty summary;
- a `reasoning` item containing raw `reasoning_text`; it returns a placeholder summary if
`reasoning.summary` is requested;
- an assistant `message` containing an `output_text` part;
- one or more `function_call` Items.

When `include:["reasoning.encrypted_content"]` is requested, reasoning Items also
carries en `encrypted_content` equal to its raw `reasoning_text`. This field is **not**
**encrypted** and provides no confidentiality.

Ordinary model/string stops produce `completed`. Output-token or context-capacity exhaustion
produces `incomplete` with `incomplete_details.reason: "max_output_tokens"`. Errors accepted after
an SSE response has started produce `response.failed`; validation and preparation errors remain
Expand Down Expand Up @@ -483,6 +489,19 @@ The normal lifecycle is:
4. matching `*.done`, `response.content_part.done`, and `response.output_item.done` events;
5. exactly one `response.completed`, `response.incomplete`, or `response.failed` terminal event.

For a reasoning Item requested with any string-valued `reasoning.summary`, its
`response.output_item.added` and `.done` payloads carry the same placeholder summary. Immediately
after the Item is added, the stream emits `response.reasoning_summary_part.added`,
`response.reasoning_summary_text.delta`, `response.reasoning_summary_text.done`, and
`response.reasoning_summary_part.done` with `summary_index:0`, then continues with the raw
`reasoning_text` content lifecycle. Omitted or `null` summary requests emit none of these summary
events and retain an empty Item `summary` array.

For `include:["reasoning.encrypted_content"]`, the in-progress
`response.output_item.added` Item omits `encrypted_content` because the complete reasoning text is
not available yet. `response.output_item.done` and the terminal Response output contain the same
complete raw mirror. A response with no reasoning Item emits no encrypted placeholder.

Function arguments use `response.function_call_arguments.delta` and `.done`. IDs, output indices,
and content indices remain stable, and concatenated deltas equal the terminal Item. Responses SSE
does not emit the Chat Completions `[DONE]` sentinel. With tools enabled, ordinary answer text still
Expand Down Expand Up @@ -547,9 +566,10 @@ curl http://127.0.0.1:8080/v1/responses/input_tokens \
```

Unsupported Create fields include Conversations, prompt templates, context management, hosted
moderation, Structured Outputs/JSON mode, non-empty `include`, background execution, compaction,
files/audio, and OpenAI-hosted/MCP/custom tools. These are compatibility boundaries, not silently
accepted placeholders.
moderation, Structured Outputs/JSON mode, `include` values other than
`reasoning.encrypted_content`, background execution, compaction, files/audio, and
OpenAI-hosted/MCP/custom tools. Except for the two explicitly documented placeholders,
these are compatibility boundaries rather than silently accepted approximations.

## Anthropic Messages

Expand Down
8 changes: 5 additions & 3 deletions src/serve/openai_responses.h
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ struct OpenAIResponsesPromptRequest {
std::vector<nlohmann::json> input_items;
std::optional<std::string> instructions;
std::optional<std::string> previous_response_id;
std::optional<std::string> reasoning_summary;
};

struct OpenAIResponsesCreateRequest {
Expand All @@ -46,9 +47,10 @@ struct OpenAIResponsesCreateRequest {
std::unordered_map<std::string, OpenAIResponsesFunctionIdentity> tool_identities;
std::optional<int> requested_max_output_tokens;
std::optional<int> max_tool_calls;
bool parallel_tool_calls = true;
bool store = true;
bool stream = false;
bool parallel_tool_calls = true;
bool store = true;
bool stream = false;
bool include_reasoning_encrypted_content = false;
};

struct OpenAIResponsesResolvedPrompt {
Expand Down
22 changes: 16 additions & 6 deletions src/serve/openai_responses_request.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -920,7 +920,13 @@ void parse_reasoning(const Json& body, OpenAIResponsesPromptRequest& out) {
static const std::unordered_set<std::string> allowed = {"effort", "context", "summary",
"generate_summary", "mode"};
reject_nonnull_unknown_members(reasoning, allowed, "reasoning");
for (const char* key : {"context", "summary", "generate_summary", "mode"}) {
if (reasoning.contains("summary") && !reasoning.at("summary").is_null()) {
if (!reasoning.at("summary").is_string()) {
bad_request("reasoning.summary must be a string", "reasoning");
}
out.reasoning_summary = reasoning.at("summary").get<std::string>();
}
for (const char* key : {"context", "generate_summary", "mode"}) {
if (reasoning.contains(key) && !reasoning.at(key).is_null()) {
bad_request("reasoning." + std::string(key) +
" changes reasoning input or output and is not supported",
Expand Down Expand Up @@ -1192,12 +1198,16 @@ OpenAIResponsesCreateRequest parse_openai_responses_create_request(const Json& b
"background_not_supported");
}
}
if (body.contains("include") && !body.at("include").is_null()) {
if (body.contains("include")) {
if (!body.at("include").is_array()) { bad_request("include must be an array", "include"); }
if (!body.at("include").empty()) {
bad_request("the requested additional response fields have no available response "
"representation",
"include", "include_not_supported");
for (const Json& field : body.at("include")) {
if (!field.is_string()) { bad_request("include entries must be strings", "include"); }
const std::string value = field.get<std::string>();
if (value != "reasoning.encrypted_content") {
bad_request("additional response field '" + value + "' is not supported", "include",
"include_not_supported");
}
out.include_reasoning_encrypted_content = true;
}
}
if (body.contains("stream_options") && !body.at("stream_options").is_null()) {
Expand Down
114 changes: 82 additions & 32 deletions src/serve/openai_responses_response.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ namespace {

using Json = nlohmann::json;

constexpr char kReasoningSummaryPlaceholder[] = "Reasoning summary is not supported. (Ninfer: OpenAI Responses API)";

std::string response_status(ninfer::FinishReason reason) {
switch (reason) {
case ninfer::FinishReason::OutputLimit:
Expand Down Expand Up @@ -49,14 +51,31 @@ void add_wire_function_identity(Json& object, const OpenAIResponsesCreateRequest
if (position->second.wire_namespace) { object["namespace"] = *position->second.wire_namespace; }
}

// Build the display-only reasoning summary requested by the client.
Json reasoning_summary(const OpenAIResponsesCreateRequest& request) {
if (!request.prompt.reasoning_summary) { return Json::array(); }
return Json::array({Json{{"type", "summary_text"}, {"text", kReasoningSummaryPlaceholder}}});
}

// Mirror raw reasoning into the opaque field requested by Harness. This is deliberately not
// encryption; replayed prompt and cache semantics continue to come only from reasoning_text.
void add_reasoning_encrypted_content(Json& item, const OpenAIResponsesCreateRequest& request,
const std::string& reasoning) {
if (request.include_reasoning_encrypted_content) {
item["encrypted_content"] = reasoning;
}
}

Json response_common(const std::string& id, std::int64_t created_at,
const OpenAIResponsesCreateRequest& request,
const OpenAIResponsesRuntimeValues& runtime) {
const Json reasoning = {{"effort", request.prompt.generation.reasoning_effort
? Json(requested_reasoning_effort_name(
*request.prompt.generation.reasoning_effort))
: Json(nullptr)},
{"summary", nullptr}};
const Json reasoning = {
{"effort",
request.prompt.generation.reasoning_effort
? Json(requested_reasoning_effort_name(*request.prompt.generation.reasoning_effort))
: Json(nullptr)},
{"summary", request.prompt.reasoning_summary ? Json(*request.prompt.reasoning_summary)
: Json(nullptr)}};
return Json{
{"id", id},
{"object", "response"},
Expand Down Expand Up @@ -105,13 +124,15 @@ BuiltOpenAIResponse build_response(const std::string& id, std::int64_t created_a
const char* reasoning_status = (!outcome.text.empty() || !outcome.tool_calls.empty())
? "completed"
: item_status.c_str();
built.output_items.push_back(
Json{{"id", ids.reasoning},
{"type", "reasoning"},
{"status", reasoning_status},
{"summary", Json::array()},
{"content",
Json::array({Json{{"type", "reasoning_text"}, {"text", outcome.reasoning}}})}});
Json reasoning_item = {
{"id", ids.reasoning},
{"type", "reasoning"},
{"status", reasoning_status},
{"summary", reasoning_summary(request)},
{"content",
Json::array({Json{{"type", "reasoning_text"}, {"text", outcome.reasoning}}})}};
add_reasoning_encrypted_content(reasoning_item, request, outcome.reasoning);
built.output_items.push_back(std::move(reasoning_item));
}

if (needs_message_item(outcome, status)) {
Expand Down Expand Up @@ -233,21 +254,49 @@ class OpenAIResponsesEventStream::Impl {

std::vector<std::string> ensure_reasoning() {
if (reasoning_started) { return {}; }
reasoning_started = true;
ids.reasoning = new_openai_response_item_id("rs");
reasoning_index = next_output_index++;
const Json item = {{"id", ids.reasoning},
{"type", "reasoning"},
{"status", "in_progress"},
{"summary", Json::array()},
{"content", Json::array()}};
const Json part = {{"type", "reasoning_text"}, {"text", ""}};
return {sse(event("response.output_item.added",
Json{{"output_index", reasoning_index}, {"item", item}})),
sse(event("response.content_part.added", Json{{"item_id", ids.reasoning},
{"output_index", reasoning_index},
{"content_index", 0},
{"part", part}}))};
reasoning_started = true;
ids.reasoning = new_openai_response_item_id("rs");
reasoning_index = next_output_index++;
const Json summary = reasoning_summary(request);
const Json item = {{"id", ids.reasoning},
{"type", "reasoning"},
{"status", "in_progress"},
{"summary", summary},
{"content", Json::array()}};
const Json part = {{"type", "reasoning_text"}, {"text", ""}};
std::vector<std::string> events = {
sse(event("response.output_item.added",
Json{{"output_index", reasoning_index}, {"item", item}}))};
if (!summary.empty()) {
const Json added_summary_part = {{"type", "summary_text"}, {"text", ""}};
const Json& done_summary_part = summary.at(0);
events.push_back(sse(event("response.reasoning_summary_part.added",
Json{{"item_id", ids.reasoning},
{"output_index", reasoning_index},
{"summary_index", 0},
{"part", added_summary_part}})));
events.push_back(sse(event("response.reasoning_summary_text.delta",
Json{{"item_id", ids.reasoning},
{"output_index", reasoning_index},
{"summary_index", 0},
{"delta", kReasoningSummaryPlaceholder}})));
events.push_back(sse(event("response.reasoning_summary_text.done",
Json{{"item_id", ids.reasoning},
{"output_index", reasoning_index},
{"summary_index", 0},
{"text", kReasoningSummaryPlaceholder}})));
events.push_back(sse(event("response.reasoning_summary_part.done",
Json{{"item_id", ids.reasoning},
{"output_index", reasoning_index},
{"summary_index", 0},
{"part", done_summary_part}})));
}
events.push_back(
sse(event("response.content_part.added", Json{{"item_id", ids.reasoning},
{"output_index", reasoning_index},
{"content_index", 0},
{"part", part}})));
return events;
}

std::vector<std::string> close_reasoning(const std::string& final_text,
Expand All @@ -256,11 +305,12 @@ class OpenAIResponsesEventStream::Impl {
reasoning_done = true;
reasoning_text = final_text;
const Json part = {{"type", "reasoning_text"}, {"text", reasoning_text}};
const Json item = {{"id", ids.reasoning},
{"type", "reasoning"},
{"status", item_status},
{"summary", Json::array()},
{"content", Json::array({part})}};
Json item = {{"id", ids.reasoning},
{"type", "reasoning"},
{"status", item_status},
{"summary", reasoning_summary(request)},
{"content", Json::array({part})}};
add_reasoning_encrypted_content(item, request, reasoning_text);
return {sse(event("response.reasoning_text.done", Json{{"item_id", ids.reasoning},
{"output_index", reasoning_index},
{"content_index", 0},
Expand Down
Loading