feat(filter): add openai_responses_compact filter for context window management by crstrn13 · Pull Request #403 · praxis-proxy/ai

crstrn13 · 2026-07-17T12:08:15Z

Summary

Adds the openai_responses_compact filter (Filter 11, #30) that manages context window size by summarizing conversation history when token count exceeds a configurable threshold.

How it works

Token counting — uses tiktoken (cl100k_base / o200k_base) to estimate the token count of the full conversation text.
Threshold check — if the count exceeds the compact_threshold from the request's context_management array, compaction triggers.
Summarization callout — sends the conversation to a Chat Completions endpoint (configurable inference_url + default_model) and receives a summary.
Message replacement — replaces the full history with a single {"role": "system", "content": "<summary>"} item, preserving the current turn's input.

Runs after rehydrate (which populates messages) and before openai_tool_parse.

Performance notes

count_ordinary instead of encode_ordinary().len() — avoids materializing the full token vector
Cow<'_, str> in extract_content — zero-copy for plain string content, only allocates when joining array content parts
Single-buffer push_str pattern in build_conversation_text — one allocation instead of N+2
Conversation text built once and passed to both token counting and summarization

Files

apis/src/openai/responses/compact/ — filter implementation, config, unit tests (20 tests)
apis/src/openai/responses/config_validation.rs — shared callout settings extracted from web_search
server/src/lib.rs — filter registration
examples/configs/openai/responses/compact.yaml — example config
tests/integration/tests/suite/examples/compact.rs — integration tests (passthrough + multi-turn compaction)
docs/filters/openai_responses_compact.md — generated filter docs

Test plan

Unit tests: cargo test -p praxis-ai-apis -- compact (20 tests)
Integration tests: cargo test -p praxis-ai-integration -- compact (2 tests)
Full lint: make lint passes (clippy + rustfmt + filter docs + separator lint)
Example config builds and runs through the proxy pipeline

Closes #30

Signed-off-by: Alexander Cristurean <acristur@redhat.com> # Conflicts: # apis/src/openai/mod.rs # apis/src/openai/responses/mod.rs # examples/configs/openai/responses/full-flow.yaml

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

Signed-off-by: Alexander Cristurean <acristur@redhat.com> � Conflicts: � apis/src/openai/responses/mod.rs

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

… refactor Signed-off-by: Alexander Cristurean <acristur@redhat.com>

praxis-bot-app · 2026-07-22T07:30:27Z

PR too large: 1116 lines added (limit: 750, excludes Cargo files, tests, docs, examples, and benchmarks). Please split into smaller PRs. Add skip/pr-conventions label to override.

franciscojavierarceo

Thanks for putting this together—compaction is a substantial addition. I’m requesting changes on a few correctness and API-contract issues that I think should be addressed before merge:

The generated summary is persisted as a system message rather than canonical Responses compaction state.
Token estimation and summarization omit supported conversation items such as tool calls, tool outputs, and previous compaction state.
The documented failure-mode behavior is not currently reachable or applied consistently.
Missing or malformed thresholds trigger immediate compaction.
The integration test does not yet demonstrate that compaction actually occurred.

There is also a scope mismatch worth resolving: this PR says Closes #30, while #30 explicitly includes the standalone POST /v1/responses/compact flow, summary_prefix, a canonical OpenAIResponseCompaction item, and hidden chainable storage. If those pieces are intentionally deferred or handled through passthrough, I’m happy for this PR to be narrowed and for #30 to remain open; otherwise they should be completed here.

For context, I previously implemented and shipped analogous Responses compaction support in ogx-ai/ogx#5327, including the explicit compact endpoint, automatic context_management compaction, canonical compaction items, persistence/chaining, and Codex CLI end-to-end testing. I compared that lifecycle while reviewing this PR. The comments here are grounded in this PR’s linked issue and the Responses API contract rather than a request to reproduce OGX’s implementation exactly.

franciscojavierarceo · 2026-07-22T13:33:49Z

+///
+/// Returns: `{"role": "system", "content": "<summary>"}`
+fn build_compaction_item(summary: &str) -> Value {
+    serde_json::json!({


Could we represent this as a canonical Responses compaction item rather than a system message? The summary is model-generated from user-controlled conversation history, so persisting it as role: system raises its instruction priority. It also cannot round-trip as the OpenAIResponseCompaction state described in #30. A safer shape would be {type: "compaction", id, encrypted_content}, translated to an assistant-level message only at a Chat Completions backend boundary if necessary.

franciscojavierarceo · 2026-07-22T13:33:49Z

+///
+/// Returns `Cow::Borrowed` for plain strings (zero-copy) and
+/// `Cow::Owned` for array content that must be joined.
+fn extract_content(msg: &Value) -> Cow<'_, str> {


Could this serialization cover the complete canonical conversation state? Items such as function_call.arguments, function_call_output.output, and prior compaction encrypted_content currently produce an empty string and disappear from both the token estimate and the summary; tool definitions also contribute to rendered context outside these message contents. That can trigger compaction too late and leave the compacted conversation unable to continue correctly. Tests covering a tool round-trip and repeated compaction would help lock this down.

franciscojavierarceo · 2026-07-22T13:33:49Z

+
+    /// Failure mode for the inference callout.
+    #[serde(default)]
+    pub failure_mode: Option<FailureMode>,


I don’t think this setting can currently be populated from filter YAML: parse_filter_config removes failure_mode as a pipeline structural key before deserializing the filter-specific config, leaving this as None and defaulting the callout to closed mode. Could we use a non-reserved name such as callout_failure_mode, or explicitly pass the pipeline policy into the filter, with coverage for both open and closed behavior?

franciscojavierarceo · 2026-07-22T13:33:49Z

+        let request = build_summarization_request(conversation_text, instructions, model, &self.config.inference_url);
+
+        match self.callout_client.execute(request).await {
+            CalloutResult::Success(resp) => parse_summarization_response(&resp.body).map(Some).or_else(|e| {


Could parse failures follow the configured open/closed policy as well? A 2xx response with invalid JSON or without choices[0].message.content currently returns Ok(None) and silently skips compaction even in closed mode. Routing this through the same failure-outcome logic as the other callout failures, with a test for each mode, would make the behavior consistent.

franciscojavierarceo · 2026-07-22T13:33:49Z

+        if entry_type != "compaction" {
+            continue;
+        }
+        let compact_threshold = entry.get("compact_threshold").and_then(Value::as_u64).unwrap_or(0);


Could we distinguish an absent threshold from an invalid one here? Missing, null, negative, and incorrectly typed values all become zero, which unexpectedly triggers a summarization call for any non-empty history. It seems safer for absence to use an explicit operator default or skip compaction, while malformed values are rejected; an explicit numeric zero could still intentionally mean “compact immediately.”

franciscojavierarceo · 2026-07-22T13:33:49Z

+            r#"{"model":"gpt-4.1","input":"Compare with QUIC","previous_response_id":"resp_compact","context_management":[{"type":"compaction","compact_threshold":50}]}"#,
+        ),
+    );
+    let status2 = parse_status(&raw2);


Would it be possible for this test to assert the compaction behavior rather than only the final status? It currently passes even if compaction never runs because the mock returns a successful response for either call. A recording or sequenced backend could verify that the above-threshold case makes a summarization call followed by an inference call, and that the latter contains compacted state without the old history; the below-threshold case should make only the inference call.

franciscojavierarceo · 2026-07-22T13:33:49Z

+            return Ok(FilterAction::Release);
+        }
+        let streaming = is_streaming(ctx);
+        let Some(state) = ctx.extensions.get::<ResponsesState>() else {


Could you clarify whether auto-compaction is intentionally limited to requests rehydrated through previous_response_id or conversation? A direct full-input request with store: false and context_management has no ResponsesState here and is silently released. If that limitation is intentional, documenting and scoping it would be helpful; otherwise state likely needs to be initialized for every Responses request.

praxis-bot

Review Summary

Clean, well-structured filter that follows established patterns. The shared config_validation.rs extraction from web_search is a solid refactor. Unit and integration test coverage is thorough for the happy paths. Three issues to address around silent failure modes and documentation accuracy.

Severity	Count	Areas
Medium	3	Config validation, request parsing, documentation

Key concern: Two configuration/request-parsing paths fail silently in ways that are difficult to diagnose in production: an invalid tiktoken_encoding causes compaction to never trigger (no error, just a warning log), and a missing compact_threshold defaults to 0 causing compaction on every request.

praxis-bot · 2026-07-22T13:42:10Z

+    Ok(ValidatedConfig {
+        inference_url: raw.inference_url.clone(),
+        default_model: raw.default_model.clone(),
+        tiktoken_encoding: raw.tiktoken_encoding.clone(),


[Medium] tiktoken_encoding is passed through without validation. Only cl100k_base and o200k_base are supported at runtime (see get_token_count), but an invalid value like gpt4 silently disables compaction -- get_token_count returns None, should_compact returns None, and the filter does nothing with only a warn! at runtime.

Validate the encoding name in build_config against the supported set and return a FilterError at startup, matching how timeout_ms and status_on_error are validated here. This also makes the # Errors doc on this function accurate (it should mention invalid encoding).

praxis-bot · 2026-07-22T13:42:10Z

+        if entry_type != "compaction" {
+            continue;
+        }
+        let compact_threshold = entry.get("compact_threshold").and_then(Value::as_u64).unwrap_or(0);


[Medium] unwrap_or(0) means a compaction entry that omits compact_threshold (or sets it to a non-integer) silently defaults to a threshold of 0. Since any non-empty conversation has token_count > 0, compaction will trigger on every request, making a wasteful summarization callout each time.

Either return None when compact_threshold is missing/invalid (skip compaction -- the entry is malformed) or enforce a minimum and log a warning. Returning None is the safer choice since it matches how a missing type field is already handled.

praxis-bot · 2026-07-22T13:42:10Z

+//! this filter summarizes the conversation history via a sub-request
+//! to an inference backend, replacing it with a single compaction
+//! item. Runs after `rehydrate` (which populates messages and
+//! previous usage) and before `openai_tool_parse`.


[Medium] This doc says the filter runs "before openai_tool_parse", but the example config (examples/configs/openai/responses/compact.yaml) places openai_tool_parse at position 3 and openai_responses_compact at position 7 -- meaning tool_parse runs first. Update this doc comment (and the PR description) to match the actual example pipeline order, or reorder the example config if the doc is correct.

crstrn13 force-pushed the feat/compact-filter branch from 846d320 to d1d0ac1 Compare July 20, 2026 11:59

crstrn13 self-assigned this Jul 20, 2026

crstrn13 added 11 commits July 21, 2026 18:55

feat: initial scaffold for compact filter.

b855901

Signed-off-by: Alexander Cristurean <acristur@redhat.com> # Conflicts: # apis/src/openai/mod.rs # apis/src/openai/responses/mod.rs # examples/configs/openai/responses/full-flow.yaml

fix: CompactFilter scaffold.

901e0d5

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

fix: extracted common config with web search and config for compaction.

70b6187

Signed-off-by: Alexander Cristurean <acristur@redhat.com> � Conflicts: � apis/src/openai/responses/mod.rs

feat: added token counting with tiktoken.

68bc738

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

feat: wrapped up compaction filter.

94aa6d2

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

fix: cosmetic changes.

b0be589

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

fix: reduce allocations in compact filter hot path.

18c419e

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

test: added unit tests for functions used in compact filter.

672dfe7

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

test: added integration tests.

a68d199

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

fix: cosmetic changes.

9aa1733

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

fix: add CompactFilter re-export and fix web_search test after config…

2f05a1c

… refactor Signed-off-by: Alexander Cristurean <acristur@redhat.com>

crstrn13 force-pushed the feat/compact-filter branch from 416fe1f to 2f05a1c Compare July 21, 2026 17:15

crstrn13 changed the title ~~feat(filter): add filter for compacting conversations.~~ feat(filter): add openai_responses_compact filter for context window management Jul 21, 2026

crstrn13 marked this pull request as ready for review July 22, 2026 07:30

crstrn13 requested review from a team and aslakknutsen July 22, 2026 07:30

crstrn13 added the skip/pr-conventions label Jul 22, 2026

franciscojavierarceo requested changes Jul 22, 2026

View reviewed changes

praxis-bot reviewed Jul 22, 2026

View reviewed changes

Uh oh!

Conversation

crstrn13 commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How it works

Performance notes

Files

Test plan

Uh oh!

praxis-bot-app Bot commented Jul 22, 2026

Uh oh!

franciscojavierarceo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

praxis-bot left a comment

Choose a reason for hiding this comment

Review Summary

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

crstrn13 commented Jul 17, 2026 •

edited

Loading