Skip to content

docs: AI gateway documentation overhaul#328

Open
rikatz wants to merge 1 commit into
praxis-proxy:mainfrom
rikatz:docs/overhaul
Open

docs: AI gateway documentation overhaul#328
rikatz wants to merge 1 commit into
praxis-proxy:mainfrom
rikatz:docs/overhaul

Conversation

@rikatz

@rikatz rikatz commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Expand docs hub with guides (OpenAI Responses, token counting), architecture pages, and filter cross-links
  • Add ai-guardrails example with integration test; shrink lint-example-tests SKIP list

Test plan

  • Conventions CI (signoff, signed commits, no AI authorship)
  • make lint / cargo xtask lint-example-tests
  • Integration tests for new ai-guardrails example

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Overhauls the AI gateway documentation, generates filter-to-example references, and adds an integration-tested guardrails example.

Changes:

  • Adds OpenAI Responses, token-counting, and crate-architecture guides.
  • Expands generated filter documentation with related examples.
  • Adds the guardrails example and reduces the example-test allowlist.

Reviewed changes

Copilot reviewed 40 out of 40 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
xtask/src/lint_example_tests.rs Reduces skipped examples.
xtask/src/filter_docs.rs Enhances generated filter docs.
tests/integration/tests/suite/examples/mod.rs Registers guardrails tests.
tests/integration/tests/suite/examples/ai_guardrails.rs Tests guardrails example passthrough.
examples/README.md Documents routing and guardrails examples.
examples/configs/ai-guardrails.yaml Adds guardrails configuration.
docs/token-counting.md Adds token-counting guide.
docs/README.md Reorganizes documentation hub.
docs/quickstart.md Updates next-step links.
docs/proposals.md Corrects discussion URL.
docs/openai-responses.md Adds Responses API guide.
docs/filters/tool_parse.md Adds related examples.
docs/filters/token_usage_headers.md Updates generated reference.
docs/filters/token_count.md Updates generated reference.
docs/filters/responses_proxy.md Adds related examples.
docs/filters/README.md Updates filter overview.
docs/filters/prompt_enrich.md Adds related example.
docs/filters/openai_stream_events.md Adds related examples.
docs/filters/openai_responses_validate.md Adds related examples.
docs/filters/openai_responses_rehydrate.md Adds related examples.
docs/filters/openai_responses_model_rewrite.md Updates type formatting and examples.
docs/filters/openai_responses_format.md Adds related examples.
docs/filters/openai_response_store.md Updates generated reference.
docs/filters/openai_conversations.md Updates generated reference.
docs/filters/model_to_header.md Adds related example.
docs/filters/mcp.md Adds related examples.
docs/filters/extensions.md Updates dependency and generation guidance.
docs/filters/anthropic_validate.md Adds related example.
docs/filters/anthropic_to_openai.md Adds related example.
docs/filters/anthropic_stream_events.md Adds related example.
docs/filters/anthropic_messages_protocol.md Adds related example.
docs/filters/anthropic_messages_format.md Adds related examples.
docs/filters/ai_guardrails.md Adds related example.
docs/filters/a2a.md Updates type formatting and examples.
docs/features.md Refreshes feature descriptions.
docs/developing/getting-started.md Corrects binary name.
docs/architecture/response-store.md Corrects source paths.
docs/architecture/crate-layout.md Adds crate architecture guide.
docs/architecture/ai-inference.md Updates core ownership and paths.
docs/architecture/agentic-protocols.md Updates agentic module paths.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread xtask/src/filter_docs.rs
Comment thread xtask/src/filter_docs.rs Outdated
Comment thread docs/features.md Outdated
Comment thread examples/configs/ai-guardrails.yaml Outdated
Comment thread docs/token-counting.md Outdated
Comment thread docs/architecture/crate-layout.md Outdated
Copilot AI review requested due to automatic review settings July 9, 2026 23:14

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated 5 comments.

Comment thread xtask/src/filter_docs.rs Outdated
Comment thread docs/token-counting.md Outdated
Comment thread examples/configs/ai-guardrails.yaml Outdated
Comment thread docs/features.md Outdated
Comment thread xtask/src/filter_docs.rs Outdated
Copilot AI review requested due to automatic review settings July 9, 2026 23:19

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated 2 comments.

Comment thread xtask/src/filter_docs.rs
Comment thread xtask/src/lint_example_tests.rs
Copilot AI review requested due to automatic review settings July 9, 2026 23:47

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 42 out of 42 changed files in this pull request and generated 3 comments.

Comment thread xtask/src/lint_example_tests.rs
Comment thread xtask/src/filter_docs.rs
Comment thread docs/openai-responses.md Outdated
Copilot AI review requested due to automatic review settings July 10, 2026 00:00

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 42 out of 42 changed files in this pull request and generated 4 comments.

Comment thread docs/filters/extensions.md Outdated
Comment thread tests/integration/tests/suite/examples/ai_guardrails.rs
Comment thread examples/configs/ai-guardrails.yaml Outdated
Comment thread docs/features.md Outdated
Copilot AI review requested due to automatic review settings July 10, 2026 00:13

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 42 out of 42 changed files in this pull request and generated 2 comments.

Comment thread docs/filters/ai_guardrails.md Outdated
Comment thread docs/features.md Outdated
Copilot AI review requested due to automatic review settings July 10, 2026 00:18

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 44 out of 44 changed files in this pull request and generated no new comments.

@praxis-bot praxis-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation Overhaul Review

Solid docs overhaul: path corrections from the monolith-to-multi-crate migration, new guide pages for OpenAI Responses and token counting, automated related-example links via discover_example_paths, and the fix_broken_md_links utility for stripping broken intradoc references from generated docs. The new ai-guardrails example config and integration test follow established patterns.

Two items below.

Severity File Finding
Medium docs/filters/extensions.md "dot-prefix" wording is misleading
Medium xtask/src/filter_docs.rs Missing test for the already-linked skip path in fix_broken_md_links

Comment thread docs/filters/extensions.md
Comment thread xtask/src/filter_docs.rs
Copilot AI review requested due to automatic review settings July 13, 2026 14:04
@praxis-bot-app

Copy link
Copy Markdown

AI tool authorship detected:

  • 384c0d2: Co-authored-by: Cursor <cursoragent@cursor.com>

Sorry, this project does not accept commits authored by tools as valid.
Commits need to be authored by and signed-off by the human(s) responsible for the PR, with their name and contact.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 46 out of 46 changed files in this pull request and generated 3 comments.

Comment thread docs/overview.md Outdated
Comment thread docs/architecture/ai-inference.md Outdated
Comment thread docs/architecture/agentic-protocols.md Outdated

@praxis-bot praxis-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation Overhaul Review

Severity Count
Critical 0
Large 0
Medium 2

Solid documentation refresh: accurate path updates after the multi-crate split, good cross-linking between filter docs and example configs, and useful new guide pages (overview, openai-responses, token-counting, crate-layout). The fix_broken_md_links / intradoc_link_target machinery is well-designed and the filter_metadata mapping correctly threads through all rendering paths.

Two unresolved items from prior review remain actionable.

Comment thread xtask/src/filter_docs.rs Outdated
Comment thread tests/integration/tests/suite/examples/ai_guardrails.rs
Copilot AI review requested due to automatic review settings July 14, 2026 13:03

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 46 out of 46 changed files in this pull request and generated 1 comment.

Comment thread xtask/src/filter_docs.rs
Copilot AI review requested due to automatic review settings July 14, 2026 13:08

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI review requested due to automatic review settings July 14, 2026 13:23

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@praxis-bot praxis-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: docs: AI gateway documentation overhaul

Documentation restructuring that adds an AI Gateway overview page, OpenAI Responses and token counting guides, crate layout architecture doc, generated "Related examples" cross-links on every filter page, and an ai-guardrails example config with integration test. Also fixes broken intradoc links in generated filter docs and shrinks the lint-example-tests SKIP list from ~20 to 2.

Overall this is solid work. The xtask changes (intradoc link fixing, auto-discovered related examples) are well-tested. Two items to address:

Severity Count
Medium 2

Both findings are inline.

Comment thread xtask/src/filter_docs.rs Outdated
Comment thread docs/token-counting.md Outdated
@rikatz
rikatz force-pushed the docs/overhaul branch 2 times, most recently from 7e85c11 to 2cac414 Compare July 14, 2026 22:13

@praxis-bot praxis-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation Overhaul Review

Solid restructuring of the docs hub with useful new guides, cross-links, and the ai-guardrails example. The xtask improvements (intradoc link fixing, auto-discovery of related examples) are well-designed. Three documentation consistency issues below.

Severity Count
Medium 3
Total 3

Comment thread docs/token-counting.md Outdated
Comment thread docs/features.md
Comment thread xtask/src/filter_docs.rs

@praxis-bot praxis-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation Overhaul Review

Solid restructuring. New overview, OpenAI Responses guide, token-counting guide, and crate layout page give the docs hub real depth. Path updates, "Related examples" cross-links, and the SKIP-list shrink are welcome. The fix_broken_md_links xtask addition and ai-guardrails example + integration test round it out well.

Two findings in discover_example_paths; one test coverage gap.

Severity File Finding
Medium xtask/src/filter_docs.rs discover_example_paths substring match can false-positive
Medium xtask/src/filter_docs.rs discover_example_paths re-reads all YAML files per filter
Medium xtask/src/filter_docs.rs fix_broken_md_links missing test for skip-existing-link path

Comment thread xtask/src/filter_docs.rs Outdated
Comment thread xtask/src/filter_docs.rs

@praxis-bot praxis-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praxis-bot review

Documentation overhaul is well-structured: the new overview, OpenAI Responses guide, token-counting guide, and crate-layout page fill real gaps. Cross-linking filter docs with auto-generated "Related examples" sections and fixing broken intradoc links are solid improvements. The xtask/filter_docs.rs code changes are clean and well-tested. Shrinking the SKIP list from ~20 to 2 entries is a clear quality win.

Findings

Severity File Finding
Medium examples/configs/ai-guardrails.yaml Opening comment describes the filter as actively calling the provider, contradicting the implementation status note two lines later
Medium docs/filters/extensions.md praxis-proxy-filter version bumped from "0.3" to "0.4" in external-filter example Cargo.toml without verification that 0.4 is published

Comment thread examples/configs/ai-guardrails.yaml Outdated
Comment thread docs/filters/extensions.md

@praxis-bot praxis-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation Overhaul Review

Clean, well-structured documentation overhaul. The new overview, Responses API guide, token-counting guide, and crate-layout pages are thorough and correctly cross-linked. Auto-discovery of related examples in the xtask generator and the fix_broken_md_links intradoc-link cleanup are solid improvements. The ai-guardrails example config and integration test are properly wired, and the SKIP list reduction from ~20 to 2 entries reflects real test coverage progress.

One categorization inconsistency noted inline.

Severity Count
Critical 0
Large 0
Medium 1

Reviewed with praxis-bot.

Comment thread docs/token-counting.md
Add docs/overview.md as the canonical AI Gateway definition,
expand guides and generated filter reference, align examples
with integration tests, and match praxis framework-first
positioning. Align filter names with main (openai_responses_proxy,
openai_tool_parse), fix plane/wording consistency for token_count
and ai_guardrails, and tighten example discovery matching.

Signed-off-by: Ricardo Pchevuzinske Katz <katz@redhat.com>
Comment on lines +41 to +53
## Pipeline extensions

`ResponseStoreRegistry` implements `PipelineExtension` and is
injected when pipelines are built (`server/src/pipelines.rs`).
Filters such as `openai_response_store` and
`openai_responses_rehydrate` access stores through
`ctx.extensions`.

## Dynamic reload

Config hot-reload is inherited from Praxis core. Pipelines
swap atomically; `ResponseStoreRegistry` is recreated per
pipeline build.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is this relevant to the "crate layout"? This looks more like an implementation explanation.


## Related

- [AGENTS.md](../../AGENTS.md) - agent and contributor reference

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why link this?


## filter_metadata

Praxis attaches a per-request `filter_metadata` map to

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mentions the size limit?

[dependencies]
async-trait = "0.1"
praxis-proxy-filter = "0.3"
praxis-proxy-filter = "0.4"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
praxis-proxy-filter = "0.4"
praxis-proxy-filter = "0.4.1"

?

Comment thread docs/openai-responses.md
@@ -0,0 +1,267 @@
# OpenAI Responses API

Praxis AI supports the OpenAI Responses API (`/v1/responses`)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overstated claim, the support on vLLM is not complete.

Comment thread docs/README.md
## Developing

- [Getting started](developing/getting-started.md)
- [Conventions](developing/conventions.md)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this moved to CONTRIBUTING.md at the repo's root

@praxis-bot praxis-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation Overhaul Review

Purpose: Expand the docs hub with an AI Gateway overview page, guides (OpenAI Responses API, token counting), a crate layout architecture page, filter cross-links, and an ai-guardrails example config with integration test. Also improves xtask/filter_docs.rs with intradoc link fixing and auto-discovered related-example sections.

Assessment: Well-structured overhaul. Cross-references between docs are accurate, source paths correctly reflect the AI repo crate layout (apis/, filters/, server/), and the ai_guardrails implementation status claims are verified against the source (on_request_body does return Continue unconditionally). The fix_broken_md_links function in filter_docs.rs is well-tested with edge cases (mapped targets, already-linked refs). The SKIP list reduction from 20+ entries to 2 is a strong signal of integration test coverage. Two line-wrap convention issues below.

Severity Count
Critical 0
Large 0
Medium 2


Calls an external AI guardrail provider to evaluate request (and eventually response) bodies. The provider determines whether content should be passed, blocked, or redacted.

## Configuration Notes

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium: This line is ~200 characters. Prose in .md files should wrap at 80 characters per project conventions.

Suggested change
## Configuration Notes
**Implementation status:** Request-side evaluation is not
wired yet; bodies are buffered via `StreamBuffer` but
`on_request_body` returns `Continue` unconditionally.
Response-side evaluation is tracked in
<https://github.com/praxis-proxy/ai/issues/50>.

Comment thread docs/overview.md
| Plane | What it does | Examples |
| ----- | ------------ | -------- |
| **Route** | Classify format; pick upstream cluster | `openai_responses_format`, `anthropic_messages_format`, `model_to_header`, `router`, branch chains |
| **Manage** | Policy, limits, credentials, resilience | `rate_limit`, `credential_injection`, `guardrails` (core), `ai_guardrails`, `ip_acl`, health checks, circuit breaker |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium: The Manage row is ~175 chars, well over both the 80-char .md wrap convention and the 120-char code limit. Markdown tables cannot have mid-cell line breaks, but the Examples column could be shortened by listing a few representative filters and linking to the full set:

Suggested change
| **Manage** | Policy, limits, credentials, resilience | `rate_limit`, `credential_injection`, `guardrails` (core), `ai_guardrails`, `ip_acl`, health checks, circuit breaker |
| **Manage** | Policy, limits, credentials, resilience | `rate_limit`, `credential_injection`, `ai_guardrails`, ... ([full list](features.md)) |

@praxis-bot praxis-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praxis-bot review: docs: AI gateway documentation overhaul

Purpose

Expands the docs hub with an AI Gateway overview, OpenAI Responses guide, token counting guide, crate layout page, and improvements to filter doc generation (intradoc link normalization, related examples). Adds ai-guardrails example config with integration test.

Assessment

Strong documentation improvement. Prior review feedback from Copilot, rikatz, and leseb has been substantially addressed across several iterations. Three remaining medium issues found: an inline link exceeding the 80-char wrap, a YAML list prefix bug in example discovery, and a stale issue reference in source code.

Findings

Severity File Description
Medium docs/overview.md:34 Inline link ~110 chars, exceeding 80-char .md wrap
Medium xtask/src/filter_docs.rs:1657 discover_example_paths misses YAML list-item filter declarations
Medium filters/src/guardrails/filter.rs Stale #578 in expect(dead_code) reason (see below)

Non-inline finding

[Medium] filters/src/guardrails/filter.rs: The expect(dead_code, reason = "...#578") attribute on the provider field still references the old praxis-proxy/praxis issue #578. This PR updated the rustdoc above it (lines 28-31) to correctly reference praxis-proxy/ai#50, and also fixed #578 in the YAML example and docs. The expect reason string should use the same corrected reference for consistency.

Comment thread docs/overview.md
Unlike routing on URL and headers alone, an AI Gateway
classifies provider API formats from request bodies and
applies policy in the data plane. Content safety combines
core [`guardrails`](https://github.com/praxis-proxy/praxis/blob/main/docs/filters/http/security/guardrails.md)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] This inline link is ~110 characters, well over the 80-char .md wrap convention. Use a reference-style link to stay within the limit:

core [`guardrails`][core-guardrails]
(rule and PII matching) with

[core-guardrails]: https://github.com/praxis-proxy/praxis/blob/main/docs/filters/http/security/guardrails.md

Comment thread xtask/src/filter_docs.rs
let Ok(content) = fs::read_to_string(&path) else {
continue;
};
let matches_filter = content.lines().any(|line| {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] strip_prefix("filter:") never matches the primary filter declaration in YAML configs because list items start with - . For example, "- filter: ai_guardrails".strip_prefix("filter:") returns None.

The function only finds filters referenced in on_result.filter: branch chain conditions (which lack the - prefix). For filters like ai_guardrails that only appear in - filter: declarations, no Related examples section is generated even though ai-guardrails.yaml exists.

Fix:

let matches_filter = content.lines().any(|line| {
    line.trim()
        .trim_start_matches("- ")
        .strip_prefix("filter:")
        .is_some_and(|rest| rest.trim() == filter_name)
});

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants