docs: AI gateway documentation overhaul#328
Conversation
There was a problem hiding this comment.
Pull request overview
Overhauls the AI gateway documentation, generates filter-to-example references, and adds an integration-tested guardrails example.
Changes:
- Adds OpenAI Responses, token-counting, and crate-architecture guides.
- Expands generated filter documentation with related examples.
- Adds the guardrails example and reduces the example-test allowlist.
Reviewed changes
Copilot reviewed 40 out of 40 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
xtask/src/lint_example_tests.rs |
Reduces skipped examples. |
xtask/src/filter_docs.rs |
Enhances generated filter docs. |
tests/integration/tests/suite/examples/mod.rs |
Registers guardrails tests. |
tests/integration/tests/suite/examples/ai_guardrails.rs |
Tests guardrails example passthrough. |
examples/README.md |
Documents routing and guardrails examples. |
examples/configs/ai-guardrails.yaml |
Adds guardrails configuration. |
docs/token-counting.md |
Adds token-counting guide. |
docs/README.md |
Reorganizes documentation hub. |
docs/quickstart.md |
Updates next-step links. |
docs/proposals.md |
Corrects discussion URL. |
docs/openai-responses.md |
Adds Responses API guide. |
docs/filters/tool_parse.md |
Adds related examples. |
docs/filters/token_usage_headers.md |
Updates generated reference. |
docs/filters/token_count.md |
Updates generated reference. |
docs/filters/responses_proxy.md |
Adds related examples. |
docs/filters/README.md |
Updates filter overview. |
docs/filters/prompt_enrich.md |
Adds related example. |
docs/filters/openai_stream_events.md |
Adds related examples. |
docs/filters/openai_responses_validate.md |
Adds related examples. |
docs/filters/openai_responses_rehydrate.md |
Adds related examples. |
docs/filters/openai_responses_model_rewrite.md |
Updates type formatting and examples. |
docs/filters/openai_responses_format.md |
Adds related examples. |
docs/filters/openai_response_store.md |
Updates generated reference. |
docs/filters/openai_conversations.md |
Updates generated reference. |
docs/filters/model_to_header.md |
Adds related example. |
docs/filters/mcp.md |
Adds related examples. |
docs/filters/extensions.md |
Updates dependency and generation guidance. |
docs/filters/anthropic_validate.md |
Adds related example. |
docs/filters/anthropic_to_openai.md |
Adds related example. |
docs/filters/anthropic_stream_events.md |
Adds related example. |
docs/filters/anthropic_messages_protocol.md |
Adds related example. |
docs/filters/anthropic_messages_format.md |
Adds related examples. |
docs/filters/ai_guardrails.md |
Adds related example. |
docs/filters/a2a.md |
Updates type formatting and examples. |
docs/features.md |
Refreshes feature descriptions. |
docs/developing/getting-started.md |
Corrects binary name. |
docs/architecture/response-store.md |
Corrects source paths. |
docs/architecture/crate-layout.md |
Adds crate architecture guide. |
docs/architecture/ai-inference.md |
Updates core ownership and paths. |
docs/architecture/agentic-protocols.md |
Updates agentic module paths. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
praxis-bot
left a comment
There was a problem hiding this comment.
Documentation Overhaul Review
Solid docs overhaul: path corrections from the monolith-to-multi-crate migration, new guide pages for OpenAI Responses and token counting, automated related-example links via discover_example_paths, and the fix_broken_md_links utility for stripping broken intradoc references from generated docs. The new ai-guardrails example config and integration test follow established patterns.
Two items below.
| Severity | File | Finding |
|---|---|---|
| Medium | docs/filters/extensions.md |
"dot-prefix" wording is misleading |
| Medium | xtask/src/filter_docs.rs |
Missing test for the already-linked skip path in fix_broken_md_links |
|
AI tool authorship detected:
Sorry, this project does not accept commits authored by tools as valid. |
praxis-bot
left a comment
There was a problem hiding this comment.
Documentation Overhaul Review
| Severity | Count |
|---|---|
| Critical | 0 |
| Large | 0 |
| Medium | 2 |
Solid documentation refresh: accurate path updates after the multi-crate split, good cross-linking between filter docs and example configs, and useful new guide pages (overview, openai-responses, token-counting, crate-layout). The fix_broken_md_links / intradoc_link_target machinery is well-designed and the filter_metadata mapping correctly threads through all rendering paths.
Two unresolved items from prior review remain actionable.
praxis-bot
left a comment
There was a problem hiding this comment.
Review: docs: AI gateway documentation overhaul
Documentation restructuring that adds an AI Gateway overview page, OpenAI Responses and token counting guides, crate layout architecture doc, generated "Related examples" cross-links on every filter page, and an ai-guardrails example config with integration test. Also fixes broken intradoc links in generated filter docs and shrinks the lint-example-tests SKIP list from ~20 to 2.
Overall this is solid work. The xtask changes (intradoc link fixing, auto-discovered related examples) are well-tested. Two items to address:
| Severity | Count |
|---|---|
| Medium | 2 |
Both findings are inline.
7e85c11 to
2cac414
Compare
praxis-bot
left a comment
There was a problem hiding this comment.
Documentation Overhaul Review
Solid restructuring of the docs hub with useful new guides, cross-links, and the ai-guardrails example. The xtask improvements (intradoc link fixing, auto-discovery of related examples) are well-designed. Three documentation consistency issues below.
| Severity | Count |
|---|---|
| Medium | 3 |
| Total | 3 |
praxis-bot
left a comment
There was a problem hiding this comment.
Documentation Overhaul Review
Solid restructuring. New overview, OpenAI Responses guide, token-counting guide, and crate layout page give the docs hub real depth. Path updates, "Related examples" cross-links, and the SKIP-list shrink are welcome. The fix_broken_md_links xtask addition and ai-guardrails example + integration test round it out well.
Two findings in discover_example_paths; one test coverage gap.
| Severity | File | Finding |
|---|---|---|
| Medium | xtask/src/filter_docs.rs |
discover_example_paths substring match can false-positive |
| Medium | xtask/src/filter_docs.rs |
discover_example_paths re-reads all YAML files per filter |
| Medium | xtask/src/filter_docs.rs |
fix_broken_md_links missing test for skip-existing-link path |
praxis-bot
left a comment
There was a problem hiding this comment.
praxis-bot review
Documentation overhaul is well-structured: the new overview, OpenAI Responses guide, token-counting guide, and crate-layout page fill real gaps. Cross-linking filter docs with auto-generated "Related examples" sections and fixing broken intradoc links are solid improvements. The xtask/filter_docs.rs code changes are clean and well-tested. Shrinking the SKIP list from ~20 to 2 entries is a clear quality win.
Findings
| Severity | File | Finding |
|---|---|---|
| Medium | examples/configs/ai-guardrails.yaml |
Opening comment describes the filter as actively calling the provider, contradicting the implementation status note two lines later |
| Medium | docs/filters/extensions.md |
praxis-proxy-filter version bumped from "0.3" to "0.4" in external-filter example Cargo.toml without verification that 0.4 is published |
praxis-bot
left a comment
There was a problem hiding this comment.
Documentation Overhaul Review
Clean, well-structured documentation overhaul. The new overview, Responses API guide, token-counting guide, and crate-layout pages are thorough and correctly cross-linked. Auto-discovery of related examples in the xtask generator and the fix_broken_md_links intradoc-link cleanup are solid improvements. The ai-guardrails example config and integration test are properly wired, and the SKIP list reduction from ~20 to 2 entries reflects real test coverage progress.
One categorization inconsistency noted inline.
| Severity | Count |
|---|---|
| Critical | 0 |
| Large | 0 |
| Medium | 1 |
Reviewed with praxis-bot.
Add docs/overview.md as the canonical AI Gateway definition, expand guides and generated filter reference, align examples with integration tests, and match praxis framework-first positioning. Align filter names with main (openai_responses_proxy, openai_tool_parse), fix plane/wording consistency for token_count and ai_guardrails, and tighten example discovery matching. Signed-off-by: Ricardo Pchevuzinske Katz <katz@redhat.com>
| ## Pipeline extensions | ||
|
|
||
| `ResponseStoreRegistry` implements `PipelineExtension` and is | ||
| injected when pipelines are built (`server/src/pipelines.rs`). | ||
| Filters such as `openai_response_store` and | ||
| `openai_responses_rehydrate` access stores through | ||
| `ctx.extensions`. | ||
|
|
||
| ## Dynamic reload | ||
|
|
||
| Config hot-reload is inherited from Praxis core. Pipelines | ||
| swap atomically; `ResponseStoreRegistry` is recreated per | ||
| pipeline build. |
There was a problem hiding this comment.
How is this relevant to the "crate layout"? This looks more like an implementation explanation.
|
|
||
| ## Related | ||
|
|
||
| - [AGENTS.md](../../AGENTS.md) - agent and contributor reference |
|
|
||
| ## filter_metadata | ||
|
|
||
| Praxis attaches a per-request `filter_metadata` map to |
| [dependencies] | ||
| async-trait = "0.1" | ||
| praxis-proxy-filter = "0.3" | ||
| praxis-proxy-filter = "0.4" |
There was a problem hiding this comment.
| praxis-proxy-filter = "0.4" | |
| praxis-proxy-filter = "0.4.1" |
?
| @@ -0,0 +1,267 @@ | |||
| # OpenAI Responses API | |||
|
|
|||
| Praxis AI supports the OpenAI Responses API (`/v1/responses`) | |||
There was a problem hiding this comment.
Overstated claim, the support on vLLM is not complete.
| ## Developing | ||
|
|
||
| - [Getting started](developing/getting-started.md) | ||
| - [Conventions](developing/conventions.md) |
There was a problem hiding this comment.
this moved to CONTRIBUTING.md at the repo's root
praxis-bot
left a comment
There was a problem hiding this comment.
Documentation Overhaul Review
Purpose: Expand the docs hub with an AI Gateway overview page, guides (OpenAI Responses API, token counting), a crate layout architecture page, filter cross-links, and an ai-guardrails example config with integration test. Also improves xtask/filter_docs.rs with intradoc link fixing and auto-discovered related-example sections.
Assessment: Well-structured overhaul. Cross-references between docs are accurate, source paths correctly reflect the AI repo crate layout (apis/, filters/, server/), and the ai_guardrails implementation status claims are verified against the source (on_request_body does return Continue unconditionally). The fix_broken_md_links function in filter_docs.rs is well-tested with edge cases (mapped targets, already-linked refs). The SKIP list reduction from 20+ entries to 2 is a strong signal of integration test coverage. Two line-wrap convention issues below.
| Severity | Count |
|---|---|
| Critical | 0 |
| Large | 0 |
| Medium | 2 |
|
|
||
| Calls an external AI guardrail provider to evaluate request (and eventually response) bodies. The provider determines whether content should be passed, blocked, or redacted. | ||
|
|
||
| ## Configuration Notes |
There was a problem hiding this comment.
Medium: This line is ~200 characters. Prose in .md files should wrap at 80 characters per project conventions.
| ## Configuration Notes | |
| **Implementation status:** Request-side evaluation is not | |
| wired yet; bodies are buffered via `StreamBuffer` but | |
| `on_request_body` returns `Continue` unconditionally. | |
| Response-side evaluation is tracked in | |
| <https://github.com/praxis-proxy/ai/issues/50>. |
| | Plane | What it does | Examples | | ||
| | ----- | ------------ | -------- | | ||
| | **Route** | Classify format; pick upstream cluster | `openai_responses_format`, `anthropic_messages_format`, `model_to_header`, `router`, branch chains | | ||
| | **Manage** | Policy, limits, credentials, resilience | `rate_limit`, `credential_injection`, `guardrails` (core), `ai_guardrails`, `ip_acl`, health checks, circuit breaker | |
There was a problem hiding this comment.
Medium: The Manage row is ~175 chars, well over both the 80-char .md wrap convention and the 120-char code limit. Markdown tables cannot have mid-cell line breaks, but the Examples column could be shortened by listing a few representative filters and linking to the full set:
| | **Manage** | Policy, limits, credentials, resilience | `rate_limit`, `credential_injection`, `guardrails` (core), `ai_guardrails`, `ip_acl`, health checks, circuit breaker | | |
| | **Manage** | Policy, limits, credentials, resilience | `rate_limit`, `credential_injection`, `ai_guardrails`, ... ([full list](features.md)) | |
praxis-bot
left a comment
There was a problem hiding this comment.
praxis-bot review: docs: AI gateway documentation overhaul
Purpose
Expands the docs hub with an AI Gateway overview, OpenAI Responses guide, token counting guide, crate layout page, and improvements to filter doc generation (intradoc link normalization, related examples). Adds ai-guardrails example config with integration test.
Assessment
Strong documentation improvement. Prior review feedback from Copilot, rikatz, and leseb has been substantially addressed across several iterations. Three remaining medium issues found: an inline link exceeding the 80-char wrap, a YAML list prefix bug in example discovery, and a stale issue reference in source code.
Findings
| Severity | File | Description |
|---|---|---|
| Medium | docs/overview.md:34 |
Inline link ~110 chars, exceeding 80-char .md wrap |
| Medium | xtask/src/filter_docs.rs:1657 |
discover_example_paths misses YAML list-item filter declarations |
| Medium | filters/src/guardrails/filter.rs |
Stale #578 in expect(dead_code) reason (see below) |
Non-inline finding
[Medium] filters/src/guardrails/filter.rs: The expect(dead_code, reason = "...#578") attribute on the provider field still references the old praxis-proxy/praxis issue #578. This PR updated the rustdoc above it (lines 28-31) to correctly reference praxis-proxy/ai#50, and also fixed #578 in the YAML example and docs. The expect reason string should use the same corrected reference for consistency.
| Unlike routing on URL and headers alone, an AI Gateway | ||
| classifies provider API formats from request bodies and | ||
| applies policy in the data plane. Content safety combines | ||
| core [`guardrails`](https://github.com/praxis-proxy/praxis/blob/main/docs/filters/http/security/guardrails.md) |
There was a problem hiding this comment.
[Medium] This inline link is ~110 characters, well over the 80-char .md wrap convention. Use a reference-style link to stay within the limit:
core [`guardrails`][core-guardrails]
(rule and PII matching) with
[core-guardrails]: https://github.com/praxis-proxy/praxis/blob/main/docs/filters/http/security/guardrails.md| let Ok(content) = fs::read_to_string(&path) else { | ||
| continue; | ||
| }; | ||
| let matches_filter = content.lines().any(|line| { |
There was a problem hiding this comment.
[Medium] strip_prefix("filter:") never matches the primary filter declaration in YAML configs because list items start with - . For example, "- filter: ai_guardrails".strip_prefix("filter:") returns None.
The function only finds filters referenced in on_result.filter: branch chain conditions (which lack the - prefix). For filters like ai_guardrails that only appear in - filter: declarations, no Related examples section is generated even though ai-guardrails.yaml exists.
Fix:
let matches_filter = content.lines().any(|line| {
line.trim()
.trim_start_matches("- ")
.strip_prefix("filter:")
.is_some_and(|rest| rest.trim() == filter_name)
});
Summary
ai-guardrailsexample with integration test; shrinklint-example-testsSKIP listTest plan
make lint/cargo xtask lint-example-testsai-guardrailsexample