diff --git a/.claude/skills/ai-gateway-migration-review/SKILL.md b/.claude/skills/ai-gateway-migration-review/SKILL.md new file mode 100644 index 00000000000..429f55e4052 --- /dev/null +++ b/.claude/skills/ai-gateway-migration-review/SKILL.md @@ -0,0 +1,226 @@ +--- +name: ai-gateway-migration-review +description: > + Reviews AI Gateway documentation files for correctness during the v1 → v2 migration. + Use this skill whenever you need to audit or fix AI Gateway docs for migration issues — + whether reviewing specific files or all AI Gateway files across how-tos, landing pages, + and reference pages. Triggers on requests like "review this ai-gateway file", "check + my migration", "audit ai-gateway docs for v2", "find v1 references in ai-gateway pages", + "update this file for AI Gateway v2", or any time someone is working on AI Gateway + content under app/_how-tos/ai-gateway, app/_landing-pages/ai-gateway, or app/ai-gateway. +--- + +# AI Gateway Migration Review Skill + +This skill audits and optionally fixes AI Gateway documentation files for the v1 → v2 migration. + +## Step 1: Ask the user two questions upfront + +Before doing any work, ask (you can ask both in one message): + +1. **Scope**: Do they want to review a specific files or directories or all AI Gateway files? + - If specific files or directories, ask for the path. + - If all files, you'll scan these directories: + - `app/_how-tos/ai-gateway/` + - `app/_landing_pages/ai-gateway/` + - `app/ai-gateway/` + +2. **Mode**: Should you make changes directly, or produce a report of issues to fix? + +## Step 2: Perform the review + +Apply the rules below. For **report mode**, collect all findings and present them as a structured report at the end. For **edit mode**, apply fixes directly and summarize what changed. + +--- + +## Rules for v1 files (under any `v1/` subdirectory) + +These files are the legacy v1 content. They need their own internal consistency: + +- **Breadcrumbs and links**: Any breadcrumbs or links starting with `/ai-gateway/` must use `/ai-gateway/v1/` (not the bare `/ai-gateway/` path). +- **Permalinks**: If the file has a `permalink:` frontmatter field, it must contain `/v1/` in the path. +- **Include and data file references**: References to AI Gateway include files (`{% include_content ... %}`, `{% include ... %}`) and data files must use the `/v1/` variant, e.g. `ai-gateway/v1/some-include` not `ai-gateway/some-include`. + +--- + +## Rules for v2 files (current, non-v1 AI Gateway files) + +### Frontmatter requirements + +Every v2 AI Gateway page must have this exact frontmatter shape for these fields: + +```yaml +products: + - ai-gateway # Only ai-gateway, no other products +works_on: + - konnect # Only konnect, no on-prem +tools: + - konnect-api # Optional field, if it exists, must be only konnect-api +min_version: + ai-gateway: '2.0' +``` + +Flag any deviation: +- `products` containing anything other than `ai-gateway` +- `works_on` containing `on-prem` or anything other than `konnect` +- `tools` containing `deck`, `admin-api`, or anything other than `konnect-api` +- Missing or wrong `min_version` (must be `ai-gateway: '2.0'`) + +#### `content_type` on AI Policy pages + +Pages under `app/_ai_gateway_policies/` are auto-generated as stubs with the default `content_type: plugin`. When you author real overview content for one of these pages (i.e. the file has hand-written body prose, not just frontmatter), set `content_type: policy`. + +- If you are adding or editing overview prose on an `_ai_gateway_policies/` page, change `content_type: plugin` → `content_type: policy`. +- Leave untouched stubs alone — do **not** flip `content_type` on pages that still have an empty body. The stub default stays `plugin` until the page gets real content. + +### Plugin → AI Policy migration + +Plugins have been replaced by AI Policies in v2. The four plugins that do **not** exist as policies are exceptions: +- AI A2A Proxy +- AI MCP Proxy +- AI Proxy +- AI Proxy Advanced + +For everything else: + +- **Rename in prose**: Replace "X plugin" with "X Policy". For example: + - "AI Request Transformer plugin" → "AI Request Transformer Policy" + - "AI Prompt Guard plugin" → "AI Prompt Guard Policy" + +- **Links to plugins**: Replace `/plugins/` path with `/ai-gateway/policies/`. For example: + - `/plugins/ai-prompt-guard/` → `/ai-gateway/policies/ai-prompt-guard/` + - `/plugins/?category=ai` → `/ai-gateway/policies/` + +- **Links to the Plugin entity**: When a link points at the Gateway Plugin entity page `/gateway/entities/plugin/` *but refers to an AI Policy*, replace it with the AI Policy entity page `/ai-gateway/entities/ai-policy/`. For example: + - `[AI Policy](/gateway/entities/plugin/)` → `[AI Policy](/ai-gateway/entities/ai-policy/)` + - `[AI Prompt Guard Policy](/gateway/entities/plugin/)` → `[AI Prompt Guard Policy](/ai-gateway/entities/ai-policy/)` + + Be careful: `/gateway/entities/plugin/` can also be a legitimate reference to the generic Gateway Plugin entity — for instance when a page contrasts AI Policies with how {{site.base_gateway}} plugins work. Only rewrite when the link is genuinely about an AI Policy. **Flag ambiguous cases for manual review** rather than rewriting automatically. + +- **Exception — flag these**: Any reference to AI A2A Proxy, AI MCP Proxy, AI Proxy, or AI Proxy Advanced as plugins should be flagged for manual review (these don't have policy equivalents). + +- **Landing page plugin blocks**: In YAML landing pages (`.yaml` files under `_landing_pages/`), replace `type: plugin` blocks with `type: aigw_policy`. Example: + ```yaml + # Before (v1) + - type: plugin + config: + slug: ai-prompt-guard + + # After (v2) + - type: aigw_policy + config: + slug: ai-prompt-guard + ``` + +### Include and data file references + +References to AI Gateway include files and data files must use `/v2/`. For example: +- `ai-gateway/circuit-breaker` → `ai-gateway/v2/circuit-breaker` +- `_includes/md/ai-gateway/circuit-breaker.md` → `_includes/md/ai-gateway/v2/circuit-breaker.md` + +Flag any `{% include /plugins/` tags — v2 AI Gateway pages must not pull in plugin includes. These should be removed or replaced with the appropriate AI Policy equivalent. For example: + +``` +{% include /plugins/ai-a2a-proxy/log-output-fields.md %} +``` + +This should be replaced with a corresponding AI Policy include (e.g. under `_includes/md/ai-gateway/v2/`) or removed if no equivalent exists. + +### Code block style + +Example codeblocks in how-to guides should use `{% konnect_api_request %}` rather than raw curl or deck commands where they're making API calls. Flag any `curl` commands or `deck` commands in example steps that should be `{% konnect_api_request %}` blocks. + +### Konnect-only deployments + +v2 AI Gateway is Konnect-only. Flag any references to on-premises deployments, self-hosted Kong Gateway, or any instructions that only apply to on-prem. + +### Kong Gateway → AI Gateway + +Most references to Kong Gateway or `{{site.base_gateway}}` should be replaced with `{{site.ai_gateway}}`. However, some are legitimate — when a sentence genuinely compares or contrasts AI Gateway with Kong Gateway (e.g. "When operating {{site.ai_gateway}} alongside {{site.base_gateway}}…"), the `{{site.base_gateway}}` reference may be intentional. Flag these for manual review rather than replacing them automatically. + +### AI Gateway entity names + +All AI Gateway entity names (from `app/_ai_gateway_entities/`) must be capitalized and prefixed with "AI". Known entities: +- AI Agent +- AI Consumer +- AI Consumer Credential +- AI Consumer Group +- AI Data Plane Certificate +- AI Data Plane Node +- AI Gateway +- AI MCP Server +- AI Model +- AI Policy +- AI Provider +- AI Vault + +Flag any references to these entities without the "AI" prefix, in both singular and plural forms. For example: +- "model" or "models" → "AI Model" / "AI Models" +- "provider" or "providers" → "AI Provider" / "AI Providers" +- "policy" or "policies" → "AI Policy" / "AI Policies" +- "agent" or "agents" → "AI Agent" / "AI Agents" +- "consumer" or "consumers" → "AI Consumer" / "AI Consumers" +- "vault" → "AI Vault" + +**Check the entire file including frontmatter** — FAQs, `related_resources` links, and all entity references in link text must use the full "AI" prefix (e.g., `[AI Policies](/ai-gateway/entities/ai-policy/)` not `[Policies](/ai-gateway/entities/ai-policy/)`). + +### Links to unmigrated how-to guides + +Not all v1 how-to guides have been migrated to v2. Before checking links, build a list of invalid v1 permalinks by reading every `.md` file under `app/_how-tos/ai-gateway/v1/` and extracting their `permalink:` frontmatter values. + +Then, in the file being reviewed, flag any link whose URL appears in that list. These point to legacy v1 pages and should be removed or updated to point to the v2 equivalent if one exists. + +### v1 release tracking (`app/_config/releases/ai-gateway/v1.yml`) + +This file tracks every v1 page and whether a v2 equivalent has been written. Each entry looks like: + +```yaml +app/_how-tos/ai-gateway/v1/some-guide.md: + status: pending # still needs a v2 equivalent + canonical_url: # should point to the v2 page once written +``` + +When a new v2 page is created, the corresponding v1 entry in this file must be updated: +- Remove `status: pending` +- Set `canonical_url` to the new v2 page's permalink + +When reviewing a newly created v2 file, check whether its v1 counterpart exists in `v1.yml` and still has `status: pending`. If so, flag it: the reviewer should remove `status: pending` and set `canonical_url` to the new page's permalink. + +When reviewing all files, read `app/_config/releases/ai-gateway/v1.yml` and report all entries that still have `status: pending` — these are v1 pages that have not yet been migrated. + +--- + +## Reporting format + +When producing a **report**, structure it like this for each file reviewed: + +``` +### + +**Frontmatter issues:** +- + +**Plugin → AI Policy issues:** +- + +**Include/data file references (including `{% include /plugins/` tags):** +- + +**Unmigrated how-to links:** +- — not yet migrated, remove or update + +**v1 release tracking (`v1.yml`):** +- — still has `status: pending`, set `canonical_url` to + +**On-prem references:** +- Line N: — flag for removal + +**Entity naming:** +- + +**No issues found** (if clean) +``` + +For a **single-file edit**, after making changes, produce a brief summary of every change made. + +For **all-files edit**, process files one at a time and produce a per-file summary at the end. diff --git a/.github/styles/base/Dictionary.txt b/.github/styles/base/Dictionary.txt index 379d8fb6cd5..5057c58c8ac 100644 --- a/.github/styles/base/Dictionary.txt +++ b/.github/styles/base/Dictionary.txt @@ -11,6 +11,7 @@ ai_rate_limiting_policy agentic Agno Agno's +AIGateway Alertmanager Alibaba allow_terminated @@ -427,6 +428,8 @@ kiali Kibana kibibytes kic +kimi +Kimi knative Knative's Knatives @@ -488,6 +491,7 @@ M_Account M_Link M_Resource maglev +major_version managedfields matchers max_args @@ -607,6 +611,7 @@ PDFs pg_max_concurrent_queries pgdump pgvector +Pinecone pipeline pipelined pipelines @@ -918,6 +923,8 @@ Valkey vendored vararg vc +vercel +Vercel viewport viewports vllm diff --git a/.github/styles/frontmatter/Dictionary.txt b/.github/styles/frontmatter/Dictionary.txt index 7b11d455f31..291a7816760 100644 --- a/.github/styles/frontmatter/Dictionary.txt +++ b/.github/styles/frontmatter/Dictionary.txt @@ -1,4 +1,5 @@ ai_gateway_enterprise +ai_gateway_url graphql hmac how_to diff --git a/.github/workflows/automated-tests.yaml b/.github/workflows/automated-tests.yaml index 0410568bc0b..74d8e1efb8b 100644 --- a/.github/workflows/automated-tests.yaml +++ b/.github/workflows/automated-tests.yaml @@ -48,7 +48,7 @@ jobs: env: KONG_LICENSE_DATA: ${{ steps.getLicense.outputs.license }} DEPLOYMENT_MODEL: on-prem - PRODUCTS: ai-gateway,gateway + PRODUCTS: gateway GATEWAY_VERSION: ${{ matrix.gateway }} run: | DEBUG=tests:*,debug npm run run-tests diff --git a/.rspec b/.rspec new file mode 100644 index 00000000000..3687797e56f --- /dev/null +++ b/.rspec @@ -0,0 +1,2 @@ +--require spec_helper +--color diff --git a/CLAUDE.md b/CLAUDE.md index 0c74b142853..0e273ab5ad9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -73,6 +73,11 @@ When writing UI steps, follow the formats in `docs/ui-steps-standards.md`. When adding frontmatter `tags:`, follow the schema in `docs/update-tag-schema.md`. +## Prose style + +- Don't use em dashes, en dashes, or dashes as sentence punctuation (e.g. "X — Y"). Rewrite as two sentences, or use a comma, colon, semicolon, or parentheses instead. (Hyphens in compound words like `well-known` or in code/YAML are fine.) +- Don't use positional language like "below" or "above" to refer to other content on the page (e.g. "the table below", "see above"). Content gets reordered, so these references go stale. Use "the following" for content that comes next, or just link/name the section (e.g. "see [Set up an AI Consumer](#set-up-an-ai-consumer)") instead of describing where it sits on the page. + ## PR review standards - Provide GitHub suggestions with actionable code, not vague feedback. diff --git a/Gemfile b/Gemfile index cb453470f7b..f08a530c210 100644 --- a/Gemfile +++ b/Gemfile @@ -27,3 +27,8 @@ end group :jekyll_plugins do gem 'jekyll-contentblocks' end + +group :test do + gem 'rspec' + gem 'capybara' +end diff --git a/Gemfile.lock b/Gemfile.lock index 5200353bfb4..d1ee83f1a4b 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -21,11 +21,21 @@ GEM bigdecimal (4.1.1) byebug (13.0.0) reline (>= 0.6.0) + capybara (3.40.0) + addressable + matrix + mini_mime (>= 0.1.3) + nokogiri (~> 1.11) + rack (>= 1.6.0) + rack-test (>= 0.6.3) + regexp_parser (>= 1.5, < 3.0) + xpath (~> 3.2) coderay (1.1.3) colorator (1.1.0) concurrent-ruby (1.3.7) connection_pool (2.5.5) csv (3.3.5) + diff-lcs (1.6.2) drb (2.2.3) dry-cli (1.4.1) em-websocket (0.5.3) @@ -85,8 +95,10 @@ GEM rb-fsevent (~> 0.10, >= 0.10.3) rb-inotify (~> 0.9, >= 0.9.10) logger (1.7.0) + matrix (0.4.3) mercenary (0.4.0) method_source (1.1.0) + mini_mime (1.1.5) mini_portile2 (2.8.9) minitest (5.26.2) mutex_m (0.3.0) @@ -115,6 +127,8 @@ GEM rack (3.2.6) rack-proxy (0.8.2) rack + rack-test (2.2.0) + rack (>= 1.3) rackup (0.2.3) rack (>= 3.0.0.beta1) webrick @@ -128,6 +142,19 @@ GEM io-console (~> 0.5) rexml (3.4.2) rouge (4.7.0) + rspec (3.13.1) + rspec-core (~> 3.13.0) + rspec-expectations (~> 3.13.0) + rspec-mocks (~> 3.13.0) + rspec-core (3.13.5) + rspec-support (~> 3.13.0) + rspec-expectations (3.13.5) + diff-lcs (>= 1.2.0, < 2.0) + rspec-support (~> 3.13.0) + rspec-mocks (3.13.5) + diff-lcs (>= 1.2.0, < 2.0) + rspec-support (~> 3.13.0) + rspec-support (3.13.6) rubocop (1.87.0) json (~> 2.3) language_server-protocol (~> 3.17.0.2) @@ -162,6 +189,8 @@ GEM rack-proxy (~> 0.6, >= 0.6.1) zeitwerk (~> 2.2) webrick (1.8.2) + xpath (3.2.0) + nokogiri (~> 1.8) zeitwerk (2.8.2) PLATFORMS @@ -170,6 +199,7 @@ PLATFORMS DEPENDENCIES activesupport byebug + capybara csv foreman jekyll @@ -185,6 +215,7 @@ DEPENDENCIES pry puma rouge (~> 4.3) + rspec rubocop vite_ruby (~> 3.10) diff --git a/Makefile b/Makefile index 5c4d0bc4146..ef8616b262f 100644 --- a/Makefile +++ b/Makefile @@ -4,7 +4,7 @@ RUBY_VERSION := "$(shell ruby -v)" RUBY_VERSION_REQUIRED := "$(shell cat .ruby-version)" RUBY_MATCH := $(shell [[ "$(shell ruby -v)" =~ "ruby $(shell cat .ruby-version)" ]] && echo matched) -.PHONY: ruby-version-check scaffold-plugin +.PHONY: ruby-version-check scaffold-plugin test ruby-version-check: ifndef RUBY_MATCH $(error ruby $(RUBY_VERSION_REQUIRED) is required. Found $(RUBY_VERSION). $(newline)Run 'mise activate' or prefix you make command with 'mise x --' see README.md for more information)$(newline) @@ -49,6 +49,9 @@ kill-ports: vale: -git diff --name-only --diff-filter=d origin/main HEAD | grep '\.md$$' | xargs vale +test: ruby-version-check + bundle exec rspec + scaffold-plugin: @if [ -z "$(PLUGIN)" ]; then \ echo "Error: Plugin name is required. Usage: make scaffold-plugin PLUGIN="; \ diff --git a/api-specs/ai-gateway/policies/ai-prompt-compressor/openapi.yaml b/api-specs/ai-gateway/policies/ai-prompt-compressor/openapi.yaml new file mode 100644 index 00000000000..5e7defcc57b --- /dev/null +++ b/api-specs/ai-gateway/policies/ai-prompt-compressor/openapi.yaml @@ -0,0 +1,142 @@ +openapi: 3.1.1 + +info: + title: AI Compress Server API + description: > + This spec describes the APIs that can work with AI Prompt Compressor plugin to compress the prompt in the request body. + version: 1.0.0 + +paths: + /llm/v1/compressPrompt: + post: + summary: Compress the prompt in the request body + description: Returns the compressed prompt and compression results. + requestBody: + required: true + content: + application/json: + schema: + type: object + required: [text, model_name, compress_type, compress_ranges] + properties: + text: + type: [array, string] + items: + type: object + required: ["msg_id", "text"] + properties: + msg_id: + type: integer + text: + type: string + compress_type: + type: string + enum: + - "rate" + - "target_token" + compress_ranges: + type: array + items: + type: object + properties: + min_tokens: + type: integer + max_tokens: + type: integer + value: + type: number + model_name: + type: string + advanced_logging: + type: boolean + default: false + + responses: + '200': + description: Service is healthy + content: + application/json: + schema: + type: object + properties: + text: + type: array + items: + type: object + properties: + compress_prompt: + type: string + compressor_results: + type: object + properties: + msg_id: + type: integer + original_token_count: + type: integer + compress_token_count: + type: integer + save_token_count: + type: integer + compress_value: + type: number + compress_type: + type: string + enum: + - "rate" + - "target_token" + compressor_model: + type: string + original_text: + type: string + compress_text: + type: string + information: + type: string + msg_id: + type: integer + + + duration: + type: number + '400': + description: when error happens + content: + application/json: + schema: + type: object + properties: + error: + type: object + properties: + message: + type: string + + /status: + get: + summary: Health check + description: Returns the health status of the service. + responses: + '200': + description: Service is healthy + content: + application/json: + schema: + type: object + properties: + status: + type: string + example: "ok" + model_name: + type: string + enum: + - "microsoft/llmlingua-2-xlm-roberta-large-meetingbank" + - "microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank" + device_map: + type: string + enum: + - "cuda" + - "cpu" + - "mps" + - "balanced" + - "balanced_low_0" + - "auto" diff --git a/api-specs/ai-gateway/policies/ai-sanitizer/openapi.yaml b/api-specs/ai-gateway/policies/ai-sanitizer/openapi.yaml new file mode 100644 index 00000000000..c565037210a --- /dev/null +++ b/api-specs/ai-gateway/policies/ai-sanitizer/openapi.yaml @@ -0,0 +1,118 @@ +openapi: 3.1.1 + +info: + title: PII Server API + description: > + This spec describes the APIs that are exposed by AI PII service that can work with AI Sanitizer Plugin to sanitize PII entities in the requests. + version: 1.0.0 + +paths: + /llm/v1/sanitize: + post: + summary: sanitize the PII entities in the request body + description: Returns the sanitized results. + requestBody: + required: true + content: + application/json: + schema: + type: object + required: ['text', 'anonymize', 'options'] + properties: + text: + type: string + anonymize: + type: array + items: + $ref: '#/components/schemas/PIIEntity' + options: + type: object + properties: + redact_type: + type: string + enum: ['synthetic', 'placeholder'] + custom_patterns: + type: array + items: + type: object + properties: + name: + type: string + regex: + type: string + score: + type: number + + responses: + '200': + description: Service is healthy + content: + application/json: + schema: + type: object + properties: + text: + type: string + identified_pii: + type: array + items: + $ref: '#/components/schemas/PIIEntity' + anonymized_pii: + type: array + items: + $ref: '#/components/schemas/PIIEntity' + detected_languages: + type: array + items: + type: string + duration: + type: number + '400': + description: when error happens + content: + application/json: + schema: + type: object + properties: + error: + type: string + + /llm/v1/status: + get: + summary: Health check + description: Returns the health status of the service. + responses: + '200': + description: Service is healthy + content: + application/json: + schema: + type: object + properties: + status: + type: string + example: ok + + +components: + schemas: + PIIEntity: + enum: + - general + - phone + - creditcard + - crypto + - date + - ip + - nrp + - ssn + - url + - medical + - driverlicense + - passport + - bank + - nationalid + - custom + - credentials + - all + - all_and_credentials diff --git a/api-specs/konnect/ai-gateway/v1/openapi.yaml b/api-specs/konnect/ai-gateway/v1/openapi.yaml new file mode 100644 index 00000000000..d0628b0d644 --- /dev/null +++ b/api-specs/konnect/ai-gateway/v1/openapi.yaml @@ -0,0 +1,11855 @@ +openapi: 3.0.0 +info: + title: Konnect AI Gateway + version: 0.1.1 + description: The API for configuring AI Gateways in Konnect. + contact: + name: Kong + url: 'https://cloud.konghq.com' +servers: + - url: 'https://us.api.konghq.com/v1' + description: United-States Production region + - url: 'https://eu.api.konghq.com/v1' + description: Europe Production region + - url: 'https://au.api.konghq.com/v1' + description: Australia Production region + - url: 'https://me.api.konghq.com/v1' + description: Middle-East Production region + - url: 'https://in.api.konghq.com/v1' + description: India Production region + - url: 'https://sg.api.konghq.com/v1' + description: Singapore Production region +paths: + /ai-gateways: + get: + operationId: list-ai-gateways + summary: List AI Gateways + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns a list of AI Gateways in the organization. + parameters: + - $ref: '#/components/parameters/PageSize' + - $ref: '#/components/parameters/PageNumber' + responses: + '200': + $ref: '#/components/responses/ListAIGatewaysResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateways + post: + operationId: create-ai-gateway + summary: Create an AI Gateway + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Creates a new AI Gateway in the organization. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/CreateAIGatewayRequest' + examples: + Example Request Body: + $ref: '#/components/examples/CreateAIGatewayRequestExample' + responses: + '201': + $ref: '#/components/responses/CreateAIGatewayResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '409': + $ref: '#/components/responses/Conflict' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateways + '/ai-gateways/{gatewayId}': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + get: + operationId: get-ai-gateway + summary: Get an AI Gateway + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns the details of a specific AI Gateway. + responses: + '200': + $ref: '#/components/responses/GetAIGatewayResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateways + put: + operationId: update-ai-gateway + summary: Update an AI Gateway + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Updates the configuration of an existing AI Gateway. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/UpdateAIGatewayRequest' + examples: + Example Request Body: + $ref: '#/components/examples/UpdateAIGatewayRequestExample' + responses: + '200': + $ref: '#/components/responses/UpdateAIGatewayResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateways + delete: + operationId: delete-ai-gateway + summary: Delete an AI Gateway + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Deletes an existing AI Gateway. + responses: + '204': + description: AI Gateway deleted successfully. + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateways + '/ai-gateways/{gatewayId}/data-plane-certificates': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + get: + operationId: list-ai-gateway-data-plane-certificates + summary: List AI Gateway DataPlane Certificates + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns a list of DataPlane certificates that are associated to this AI Gateway. A DataPlane certificate allows DataPlanes configured with the certificate and corresponding private key to establish connection with this AI Gateway. + parameters: + - $ref: '#/components/parameters/PageSize' + - $ref: '#/components/parameters/PageAfter' + responses: + '200': + $ref: '#/components/responses/ListAIGatewayDataPlaneCertificatesResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway DataPlane Certificates + post: + operationId: create-ai-gateway-data-plane-certificate + summary: Create New AI Gateway DataPlane Certificate + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Create a new DataPlane Certificate for this AI Gateway. A DataPlane certificate allows DataPlanes configured with the certificate and corresponding private key to establish connection with this AI Gateway. + requestBody: + content: + application/json: + schema: + $ref: '#/components/schemas/CreateAIGatewayDataPlaneCertificateRequest' + examples: + Example Request Body: + $ref: '#/components/examples/AIGatewayDataplaneCertificateExample' + responses: + '201': + $ref: '#/components/responses/CreateAIGatewayDataPlaneCertificateResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway DataPlane Certificates + '/ai-gateways/{gatewayId}/data-plane-certificates/{certificateId}': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + - $ref: '#/components/parameters/AIGatewayDataPlaneCertificateId' + get: + operationId: get-ai-gateway-data-plane-certificate + summary: Get a DataPlane Certificate + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Retrieve a DataPlane certificate associated to this AI Gateway. A DataPlane certificate allows DataPlanes configured with the certificate and corresponding private key to establish connection with this AI Gateway. + responses: + '200': + $ref: '#/components/responses/GetAIGatewayDataPlaneCertificateResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway DataPlane Certificates + delete: + operationId: delete-ai-gateway-data-plane-certificate + summary: Delete AI Gateway DataPlane Certificate + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Remove a DataPlane client certificate associated to this AI Gateway. Removing a DataPlane certificate would invalidate any DataPlanes currently connected to this AI Gateway using this certificate. + responses: + '204': + description: No Content + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway DataPlane Certificates + '/ai-gateways/{gatewayId}/expected-config-version': + get: + operationId: get-ai-gateway-expected-config-version + summary: Get the Expected Config Version + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Retrieve the expected config version for this AI Gateway. The expected config version can be used to verify if the config version of a data plane node is up to date with the AI Gateway. The config version will be the same if they are in sync. + responses: + '200': + $ref: '#/components/responses/GetAIGatewayExpectedConfigVersionResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway DataPlane + parameters: + - $ref: '#/components/parameters/AIGatewayId' + '/ai-gateways/{gatewayId}/nodes': + get: + operationId: list-ai-gateway-nodes + summary: List Nodes + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns a list of nodes associated with the specified AI Gateway. + parameters: + - $ref: '#/components/parameters/PageSize' + - $ref: '#/components/parameters/PageAfter' + responses: + '200': + $ref: '#/components/responses/ListAIGatewayDataPlaneNodesResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Nodes + parameters: + - $ref: '#/components/parameters/AIGatewayId' + '/ai-gateways/{gatewayId}/nodes/{dataPlaneNodeId}': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + - $ref: '#/components/parameters/AIGatewayDataPlaneNodeId' + get: + operationId: get-ai-gateway-node + summary: Get a Node + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns information about a specific node associated with the AI Gateway. + responses: + '200': + $ref: '#/components/responses/GetAIGatewayDataPlaneNodeResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Nodes + '/ai-gateways/{gatewayId}/vaults': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + get: + operationId: list-ai-gateway-vaults + summary: List AI Gateway Vaults + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns a list of vaults associated with the AI Gateway. + parameters: + - $ref: '#/components/parameters/PageSize' + - $ref: '#/components/parameters/PageAfter' + responses: + '200': + $ref: '#/components/responses/ListAIGatewayVaultsResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Vaults + post: + operationId: create-ai-gateway-vault + summary: Create an AI Gateway Vault + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Registers a new vault for the AI Gateway. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/CreateAIGatewayVaultRequest' + responses: + '201': + $ref: '#/components/responses/CreateAIGatewayVaultResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '409': + $ref: '#/components/responses/Conflict' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Vaults + '/ai-gateways/{gatewayId}/vaults/{vaultIdOrName}': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + - $ref: '#/components/parameters/AIGatewayVaultIdOrName' + get: + operationId: get-ai-gateway-vault + summary: Get an AI Gateway Vault + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns the details of a specific AI Gateway vault. + responses: + '200': + $ref: '#/components/responses/GetAIGatewayVaultResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Vaults + put: + operationId: update-ai-gateway-vault + summary: Update an AI Gateway Vault + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Updates the configuration of an existing AI Gateway vault. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/UpdateAIGatewayVaultRequest' + responses: + '200': + $ref: '#/components/responses/UpdateAIGatewayVaultResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Vaults + delete: + operationId: delete-ai-gateway-vault + summary: Delete an AI Gateway Vault + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Removes a specific AI Gateway vault. + responses: + '204': + description: Vault deleted successfully. + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Vaults + '/ai-gateways/{gatewayId}/policies': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + get: + operationId: list-ai-gateway-policies + summary: List AI Gateway Policies + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns a list of policies configured for the AI Gateway. + parameters: + - $ref: '#/components/parameters/PageSize' + - $ref: '#/components/parameters/PageAfter' + responses: + '200': + $ref: '#/components/responses/ListAIGatewayPoliciesResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Policies + post: + operationId: create-ai-gateway-policy + summary: Create an AI Gateway Policy + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Registers a new policy for the AI Gateway. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/CreateAIGatewayPolicyRequest' + responses: + '201': + $ref: '#/components/responses/CreateAIGatewayPolicyResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '409': + $ref: '#/components/responses/Conflict' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Policies + '/ai-gateways/{gatewayId}/policies/{policyIdOrName}': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + - $ref: '#/components/parameters/AIGatewayPolicyIdOrName' + get: + operationId: get-ai-gateway-policy + summary: Get an AI Gateway Policy + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns the details of a specific AI Gateway policy. + responses: + '200': + $ref: '#/components/responses/GetAIGatewayPolicyResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Policies + put: + operationId: update-ai-gateway-policy + summary: Update an AI Gateway Policy + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Updates the configuration of an existing AI Gateway policy. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/UpdateAIGatewayPolicyRequest' + responses: + '200': + $ref: '#/components/responses/UpdateAIGatewayPolicyResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Policies + delete: + operationId: delete-ai-gateway-policy + summary: Delete an AI Gateway Policy + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Removes a specific AI Gateway policy. + responses: + '204': + description: Policy deleted successfully. + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Policies + '/ai-gateways/{gatewayId}/models': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + get: + operationId: list-ai-gateway-models + summary: List AI Gateway Models + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns a list of all models registered in the AI Gateway. + parameters: + - $ref: '#/components/parameters/PageSize' + - $ref: '#/components/parameters/PageAfter' + responses: + '200': + $ref: '#/components/responses/ListAIGatewayModelsResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Models + post: + operationId: create-ai-gateway-model + summary: Create an AI Gateway Model + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Registers a new model with routing, capabilities, and target backends. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/CreateAIGatewayModelRequest' + responses: + '201': + $ref: '#/components/responses/CreateAIGatewayModelResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '409': + $ref: '#/components/responses/Conflict' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Models + '/ai-gateways/{gatewayId}/models/{modelIdOrName}': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + - $ref: '#/components/parameters/AIGatewayModelIdOrName' + get: + operationId: get-ai-gateway-model + summary: Get an AI Gateway Model + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns the details of a specific AI Gateway model. + responses: + '200': + $ref: '#/components/responses/GetAIGatewayModelResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Models + put: + operationId: update-ai-gateway-model + summary: Update an AI Gateway Model + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Updates the configuration of an existing AI Gateway model. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/UpdateAIGatewayModelRequest' + responses: + '200': + $ref: '#/components/responses/UpdateAIGatewayModelResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Models + delete: + operationId: delete-ai-gateway-model + summary: Delete an AI Gateway Model + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Removes a specific AI Gateway model. + responses: + '204': + description: Model deleted successfully. + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Models + '/ai-gateways/{gatewayId}/agents': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + get: + operationId: list-ai-gateway-agents + summary: List AI Gateway Agents + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns a list of all agents registered in the AI Gateway. + parameters: + - $ref: '#/components/parameters/PageSize' + - $ref: '#/components/parameters/PageAfter' + responses: + '200': + $ref: '#/components/responses/ListAIGatewayAgentsResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Agents + post: + operationId: create-ai-gateway-agent + summary: Create an AI Gateway Agent + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Creates a new agent for the AI Gateway. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/CreateAIGatewayAgentRequest' + responses: + '201': + $ref: '#/components/responses/CreateAIGatewayAgentResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '409': + $ref: '#/components/responses/Conflict' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Agents + '/ai-gateways/{gatewayId}/agents/{agentIdOrName}': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + - $ref: '#/components/parameters/AIGatewayAgentIdOrName' + get: + operationId: get-ai-gateway-agent + summary: Get an AI Gateway Agent + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns the details of a specific AI Gateway agent. + responses: + '200': + $ref: '#/components/responses/GetAIGatewayAgentResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Agents + put: + operationId: update-ai-gateway-agent + summary: Update an AI Gateway Agent + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Updates the configuration of an existing AI Gateway agent. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/UpdateAIGatewayAgentRequest' + responses: + '200': + $ref: '#/components/responses/UpdateAIGatewayAgentResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Agents + delete: + operationId: delete-ai-gateway-agent + summary: Delete an AI Gateway Agent + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Removes a specific AI Gateway agent. + responses: + '204': + description: Agent deleted successfully. + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Agents + '/ai-gateways/{gatewayId}/consumers': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + get: + operationId: list-ai-gateway-consumers + summary: List AI Gateway Consumers + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns a list of all consumers for the AI Gateway. + parameters: + - $ref: '#/components/parameters/PageSize' + - $ref: '#/components/parameters/PageAfter' + responses: + '200': + $ref: '#/components/responses/ListAIGatewayConsumersResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Consumers + post: + operationId: create-ai-gateway-consumer + summary: Create an AI Gateway Consumer + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Creates a new consumer for the AI Gateway. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/CreateAIGatewayConsumerRequest' + responses: + '201': + $ref: '#/components/responses/CreateAIGatewayConsumerResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '409': + $ref: '#/components/responses/Conflict' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Consumers + '/ai-gateways/{gatewayId}/consumers/{consumerIdOrName}': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + - $ref: '#/components/parameters/AIGatewayConsumerIdOrName' + get: + operationId: get-ai-gateway-consumer + summary: Get an AI Gateway Consumer + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns the details of a specific AI Gateway consumer. + responses: + '200': + $ref: '#/components/responses/GetAIGatewayConsumerResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Consumers + put: + operationId: update-ai-gateway-consumer + summary: Update an AI Gateway Consumer + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Updates the configuration of an existing AI Gateway consumer. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/UpdateAIGatewayConsumerRequest' + responses: + '200': + $ref: '#/components/responses/UpdateAIGatewayConsumerResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Consumers + delete: + operationId: delete-ai-gateway-consumer + summary: Delete an AI Gateway Consumer + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Removes a specific AI Gateway consumer. + responses: + '204': + description: Consumer deleted successfully. + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Consumers + '/ai-gateways/{gatewayId}/consumers/{consumerIdOrName}/consumer-groups': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + - $ref: '#/components/parameters/AIGatewayConsumerIdOrName' + get: + operationId: list-ai-gateway-consumer-groups-for-consumer + summary: List Consumer Groups a Consumer belongs to + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + List AI Gateway Consumer Groups an Consumer belongs to + parameters: + - $ref: '#/components/parameters/PageSize' + - $ref: '#/components/parameters/PageAfter' + responses: + '200': + $ref: '#/components/responses/ListAIGatewayConsumerGroupsResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Consumers + put: + operationId: update-ai-gateway-consumer-groups-for-consumer + summary: Updates Consumer Groups a Consumer belongs to + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Updates AI Gateway Consumer Groups a Consumer belongs to + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + consumer_groups: + type: array + items: + type: string + description: Consumer Group names + responses: + '201': + description: Consumer Group names added to the consumer + content: + application/json: + schema: + type: object + properties: + consumer_groups: + type: array + items: + type: string + description: Consumer Group names + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Consumers + '/ai-gateways/{gatewayId}/consumers/{consumerId}/credentials': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + - $ref: '#/components/parameters/AIGatewayConsumerId' + get: + operationId: list-ai-gateway-consumer-credentials + summary: List AI Gateway Consumer Credentials + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns a list of all credentials for an AI Gateway consumer. + parameters: + - $ref: '#/components/parameters/PageSize' + - $ref: '#/components/parameters/PageAfter' + responses: + '200': + $ref: '#/components/responses/ListAIGatewayConsumerCredentialsResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Consumers + post: + operationId: create-ai-gateway-consumer-credential + summary: Create an AI Gateway Consumer Credential + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Creates a new credential for an AI Gateway consumer. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/CreateAIGatewayConsumerCredentialRequest' + responses: + '201': + $ref: '#/components/responses/CreateAIGatewayConsumerCredentialResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '409': + $ref: '#/components/responses/Conflict' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Consumers + '/ai-gateways/{gatewayId}/consumers/{consumerId}/credentials/{credentialId}': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + - $ref: '#/components/parameters/AIGatewayConsumerId' + - $ref: '#/components/parameters/AIGatewayConsumerCredentialId' + get: + operationId: get-ai-gateway-consumer-credential + summary: Get an AI Gateway Consumer Credential + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns the details of a specific credential for an AI Gateway consumer. + responses: + '200': + $ref: '#/components/responses/GetAIGatewayConsumerCredentialResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Consumers + delete: + operationId: delete-ai-gateway-consumer-credential + summary: Delete an AI Gateway Consumer Credential + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Removes a specific credential for an AI Gateway consumer. + responses: + '204': + description: Credential deleted successfully. + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Consumers + '/ai-gateways/{gatewayId}/consumer-groups': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + get: + operationId: list-ai-gateway-consumer-groups + summary: List AI Gateway Consumer Groups + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns a list of all consumer groups for the AI Gateway. + parameters: + - $ref: '#/components/parameters/PageSize' + - $ref: '#/components/parameters/PageAfter' + responses: + '200': + $ref: '#/components/responses/ListAIGatewayConsumerGroupsResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Consumer Groups + post: + operationId: create-ai-gateway-consumer-group + summary: Create an AI Gateway Consumer Group + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Creates a new Consumer Group for the AI Gateway. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/CreateAIGatewayConsumerGroupRequest' + responses: + '201': + $ref: '#/components/responses/CreateAIGatewayConsumerGroupResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '409': + $ref: '#/components/responses/Conflict' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Consumer Groups + '/ai-gateways/{gatewayId}/consumer-groups/{consumerGroupIdOrName}': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + - $ref: '#/components/parameters/AIGatewayConsumerGroupIdOrName' + get: + operationId: get-ai-gateway-consumer-group + summary: Get an AI Gateway Consumer Group + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns the details of a specific AI Gateway Consumer Group. + responses: + '200': + $ref: '#/components/responses/GetAIGatewayConsumerGroupResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Consumer Groups + put: + operationId: update-ai-gateway-consumer-group + summary: Update an AI Gateway Consumer Group + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Updates the configuration of an existing AI Gateway Consumer Group. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/UpdateAIGatewayConsumerGroupRequest' + responses: + '200': + $ref: '#/components/responses/UpdateAIGatewayConsumerGroupResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Consumer Groups + delete: + operationId: delete-ai-gateway-consumer-group + summary: Delete an AI Gateway Consumer Group + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Removes a specific AI Gateway Consumer Group. + responses: + '204': + description: Consumer Group deleted successfully. + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Consumer Groups + '/ai-gateways/{gatewayId}/consumer-groups/{consumerGroupId}/consumers': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + - $ref: '#/components/parameters/AIGatewayConsumerGroupId' + get: + operationId: list-ai-gateway-consumers-in-consumer-group + summary: List AI Gateway Consumers in a Consumer Group + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns a list of all consumers in the given consumer group for the AI Gateway. + parameters: + - $ref: '#/components/parameters/PageSize' + - $ref: '#/components/parameters/PageAfter' + responses: + '200': + $ref: '#/components/responses/ListAIGatewayConsumersResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Consumer Groups + post: + operationId: add-ai-gateway-consumer-to-consumer-group + summary: Add a Consumer to a Consumer Group + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Add a consumer to an AI Gateway Consumer Group. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/AddAIGatewayConsumerToGroupRequest' + responses: + '201': + $ref: '#/components/responses/AddAIGatewayConsumerToGroupResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Consumer Groups + '/ai-gateways/{gatewayId}/consumer-groups/{consumerGroupId}/consumers/{consumerIdOrName}': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + - $ref: '#/components/parameters/AIGatewayConsumerGroupId' + - $ref: '#/components/parameters/AIGatewayConsumerIdOrName' + delete: + operationId: remove-ai-gateway-consumer-from-consumer-group + summary: Remove a Consumer from a Consumer Group + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Remove a consumer from an AI Gateway Consumer Group. + responses: + '204': + description: Consumer removed from group successfully. + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Consumer Groups + '/ai-gateways/{gatewayId}/mcp-servers': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + get: + operationId: list-ai-gateway-mcp-servers + summary: List MCP Servers + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns a list of MCP Servers configured for the AI Gateway. + parameters: + - $ref: '#/components/parameters/PageSize' + - $ref: '#/components/parameters/PageAfter' + responses: + '200': + $ref: '#/components/responses/ListAIGatewayMCPServersResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway MCP Servers + post: + operationId: create-ai-gateway-mcp-server + summary: Create an MCP Server + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Registers a new MCP Server for the AI Gateway. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/CreateAIGatewayMCPServerRequest' + responses: + '201': + $ref: '#/components/responses/CreateAIGatewayMCPServerResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '409': + $ref: '#/components/responses/Conflict' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway MCP Servers + '/ai-gateways/{gatewayId}/mcp-servers/{mcpServerIdOrName}': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + - $ref: '#/components/parameters/AIGatewayMcpServerIdOrName' + get: + operationId: get-ai-gateway-mcp-server + summary: Get an MCP Server + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns the details of a specific MCP Server. + responses: + '200': + $ref: '#/components/responses/GetMCPServerResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway MCP Servers + put: + operationId: update-ai-gateway-mcp-server + summary: Update an MCP Server + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Updates the configuration of an existing MCP Server. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/UpdateAIGatewayMCPServerRequest' + responses: + '200': + $ref: '#/components/responses/UpdateAIGatewayMCPServerResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway MCP Servers + delete: + operationId: delete-ai-gateway-mcp-server + summary: Delete an MCP Server + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Removes a specific MCP Server from the AI Gateway. + responses: + '204': + description: MCP Server deleted successfully. + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway MCP Servers + '/ai-gateways/{gatewayId}/model-providers': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + get: + operationId: list-ai-gateway-model-providers + summary: List AI Gateway Model Providers + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns a list of model providers configured for the AI Gateway. + parameters: + - $ref: '#/components/parameters/PageSize' + - $ref: '#/components/parameters/PageAfter' + responses: + '200': + $ref: '#/components/responses/ListAIGatewayModelProvidersResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Model Providers + post: + operationId: create-ai-gateway-model-provider + summary: Create an AI Gateway Model Provider + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Registers a new model provider for the AI Gateway. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/CreateAIGatewayModelProviderRequest' + responses: + '201': + $ref: '#/components/responses/CreateAIGatewayModelProviderResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '409': + $ref: '#/components/responses/Conflict' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Model Providers + '/ai-gateways/{gatewayId}/model-providers/{modelProviderIdOrName}': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + - $ref: '#/components/parameters/AIGatewayModelProviderIdOrName' + get: + operationId: get-ai-gateway-model-provider + summary: Get an AI Gateway Model Provider + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns the details of a specific AI Gateway model provider. + responses: + '200': + $ref: '#/components/responses/GetAIGatewayModelProviderResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Model Providers + put: + operationId: update-ai-gateway-model-provider + summary: Update an AI Gateway Model Provider + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Updates the configuration of an existing AI Gateway model provider. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/UpdateAIGatewayModelProviderRequest' + responses: + '200': + $ref: '#/components/responses/UpdateAIGatewayModelProviderResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Model Providers + delete: + operationId: delete-ai-gateway-model-provider + summary: Delete an AI Gateway Model Provider + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Removes a specific AI Gateway model provider. + responses: + '204': + description: Model provider deleted successfully. + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '409': + $ref: '#/components/responses/Conflict' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Model Providers + '/ai-gateways/{gatewayId}/identity': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + get: + operationId: list-ai-gateway-identity-providers + summary: List AI Gateway Identity Providers + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns a list of identity providers configured for the AI Gateway. + parameters: + - $ref: '#/components/parameters/PageSize' + - $ref: '#/components/parameters/PageAfter' + responses: + '200': + $ref: '#/components/responses/ListAIGatewayIdentityProvidersResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Identity Providers + post: + operationId: create-ai-gateway-identity-provider + summary: Create an AI Gateway Identity Provider + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Registers a new identity provider for the AI Gateway. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/CreateAIGatewayIdentityProviderRequest' + responses: + '201': + $ref: '#/components/responses/CreateAIGatewayIdentityProviderResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '409': + $ref: '#/components/responses/Conflict' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Identity Providers + '/ai-gateways/{gatewayId}/identity/{identityProviderIdOrName}': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + - $ref: '#/components/parameters/AIGatewayIdentityProviderIdOrName' + get: + operationId: get-ai-gateway-identity-provider + summary: Get an AI Gateway Identity Provider + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns the details of a specific AI Gateway identity provider. + responses: + '200': + $ref: '#/components/responses/GetAIGatewayIdentityProviderResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Identity Providers + put: + operationId: update-ai-gateway-identity-provider + summary: Update an AI Gateway Identity Provider + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Updates the configuration of an existing AI Gateway Identity provider. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/UpdateAIGatewayIdentityProviderRequest' + responses: + '200': + $ref: '#/components/responses/UpdateAIGatewayIdentityProviderResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Identity Providers + delete: + operationId: delete-ai-gateway-identity-provider + summary: Delete an AI Gateway Identity Provider + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Removes a specific AI Gateway Identity provider. + responses: + '204': + description: Identity provider deleted successfully. + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '409': + $ref: '#/components/responses/Conflict' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Identity Providers + '/ai-gateways/{gatewayId}/config-stores': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + get: + operationId: list-ai-gateway-config-stores + summary: List AI Gateway Config Stores + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns a list of Config Stores associated with the AI Gateway. + parameters: + - $ref: '#/components/parameters/PageSize' + - $ref: '#/components/parameters/PageAfter' + responses: + '200': + $ref: '#/components/responses/ListAIGatewayConfigStoresResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Config Stores + post: + operationId: create-ai-gateway-config-store + summary: Create an AI Gateway Config Store + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Creates a new Config Store for the AI Gateway. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/CreateAIGatewayConfigStoreRequest' + responses: + '201': + $ref: '#/components/responses/CreateAIGatewayConfigStoreResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '409': + $ref: '#/components/responses/Conflict' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Config Stores + '/ai-gateways/{gatewayId}/config-stores/{configStoreIdOrName}': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + - $ref: '#/components/parameters/AIGatewayConfigStoreIdOrName' + get: + operationId: get-ai-gateway-config-store + summary: Get an AI Gateway Config Store + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns the details of a specific AI Gateway Config Store. + responses: + '200': + $ref: '#/components/responses/GetAIGatewayConfigStoreResponse' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Config Stores + put: + operationId: update-ai-gateway-config-store + summary: Update an AI Gateway Config Store + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Updates the configuration of an existing AI Gateway Config Store. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/UpdateAIGatewayConfigStoreRequest' + responses: + '200': + $ref: '#/components/responses/UpdateAIGatewayConfigStoreResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Config Stores + delete: + operationId: delete-ai-gateway-config-store + summary: Delete an AI Gateway Config Store + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Removes a specific AI Gateway Config Store. + parameters: + - name: force + in: query + description: 'If true, delete the Config Store and all its secrets. If false, deletion is rejected when secrets are still linked to the Config Store.' + schema: + type: boolean + default: false + responses: + '204': + description: Config Store deleted successfully. + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Config Stores + '/ai-gateways/{gatewayId}/config-stores/{configStoreIdOrName}/secrets': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + - $ref: '#/components/parameters/AIGatewayConfigStoreIdOrName' + get: + operationId: list-ai-gateway-config-store-secrets + summary: List AI Gateway Config Store Secrets + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns a collection of all secrets for an AI Gateway Config Store. + parameters: + - $ref: '#/components/parameters/PageSize' + - $ref: '#/components/parameters/PageAfter' + responses: + '200': + $ref: '#/components/responses/ListAIGatewayConfigStoreSecretsResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Config Store Secrets + post: + operationId: create-ai-gateway-config-store-secret + summary: Create an AI Gateway Config Store Secret + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Creates a secret for an AI Gateway Config Store. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/CreateAIGatewayConfigStoreSecretRequest' + responses: + '201': + $ref: '#/components/responses/CreateAIGatewayConfigStoreSecretResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '409': + $ref: '#/components/responses/Conflict' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Config Store Secrets + '/ai-gateways/{gatewayId}/config-stores/{configStoreIdOrName}/secrets/{key}': + parameters: + - $ref: '#/components/parameters/AIGatewayId' + - $ref: '#/components/parameters/AIGatewayConfigStoreIdOrName' + - name: key + in: path + description: Config Store Secret key. + required: true + schema: + $ref: '#/components/schemas/AIGatewayConfigStoreSecretKey' + get: + operationId: get-ai-gateway-config-store-secret + summary: Get an AI Gateway Config Store Secret + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Returns the secret entity for the Config Store. Secret values once stored cannot be retrieved. + responses: + '200': + $ref: '#/components/responses/GetAIGatewayConfigStoreSecretResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Config Store Secrets + put: + operationId: update-ai-gateway-config-store-secret + summary: Update an AI Gateway Config Store Secret + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Updates a secret for an AI Gateway Config Store. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/UpdateAIGatewayConfigStoreSecretRequest' + responses: + '200': + $ref: '#/components/responses/UpdateAIGatewayConfigStoreSecretResponse' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Config Store Secrets + delete: + operationId: delete-ai-gateway-config-store-secret + summary: Delete an AI Gateway Config Store Secret + description: |- + **Pre-release Endpoint** + This endpoint is currently in beta and is subject to change. + + Removes a secret from an AI Gateway Config Store. + responses: + '204': + description: Config Store Secret deleted successfully. + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + '429': + $ref: '#/components/responses/TooManyRequests' + tags: + - AI Gateway Config Store Secrets +components: + parameters: + AIGatewayAgentIdOrName: + name: agentIdOrName + in: path + required: true + description: The unique ID or name of the AI Gateway agent. + examples: + name: + value: my-entity-name + summary: The name of the entity. + id: + value: bf138ba2-c9b1-4229-b268-04d9d8a6410b + summary: The id of the entity. + schema: + $ref: '#/components/schemas/AIGatewayEntityIdentifier' + AIGatewayConfigStoreIdOrName: + name: configStoreIdOrName + in: path + required: true + description: The unique ID or name of the AI Gateway Config Store. + examples: + name: + value: my-entity-name + summary: The name of the entity. + id: + value: bf138ba2-c9b1-4229-b268-04d9d8a6410b + summary: The id of the entity. + schema: + $ref: '#/components/schemas/AIGatewayEntityIdentifier' + AIGatewayConsumerCredentialId: + name: credentialId + in: path + required: true + description: The unique ID of the AI Gateway consumer credential. + example: bf138ba2-c9b1-4229-b268-04d9d8a6410b + schema: + $ref: '#/components/schemas/UUID' + AIGatewayConsumerGroupId: + name: consumerGroupId + in: path + required: true + description: The unique ID of the AI Gateway Consumer Group. + example: bf138ba2-c9b1-4229-b268-04d9d8a6410b + schema: + $ref: '#/components/schemas/UUID' + AIGatewayConsumerGroupIdOrName: + name: consumerGroupIdOrName + in: path + required: true + description: The unique ID or name of the AI Gateway Consumer Group. + examples: + name: + value: my-entity-name + summary: The name of the entity. + id: + value: bf138ba2-c9b1-4229-b268-04d9d8a6410b + summary: The id of the entity. + schema: + $ref: '#/components/schemas/AIGatewayEntityIdentifier' + AIGatewayConsumerId: + name: consumerId + in: path + required: true + description: The unique ID of the AI Gateway consumer. + example: bf138ba2-c9b1-4229-b268-04d9d8a6410b + schema: + $ref: '#/components/schemas/UUID' + AIGatewayConsumerIdOrName: + name: consumerIdOrName + in: path + required: true + description: The unique ID or name of the AI Gateway consumer. + examples: + name: + value: my-entity-name + summary: The name of the entity. + id: + value: bf138ba2-c9b1-4229-b268-04d9d8a6410b + summary: The id of the entity. + schema: + $ref: '#/components/schemas/AIGatewayEntityIdentifier' + AIGatewayDataPlaneCertificateId: + name: certificateId + in: path + required: true + description: The unique ID of the DataPlane Certificate. + example: bf138ba2-c9b1-4229-b268-04d9d8a6410b + schema: + $ref: '#/components/schemas/UUID' + AIGatewayDataPlaneNodeId: + name: dataPlaneNodeId + in: path + required: true + description: The unique ID of the AI Gateway DataPlane Node. + example: bf138ba2-c9b1-4229-b268-04d9d8a6410b + schema: + $ref: '#/components/schemas/UUID' + AIGatewayId: + name: gatewayId + in: path + required: true + description: The unique ID of the AI Gateway. + example: bf138ba2-c9b1-4229-b268-04d9d8a6410b + schema: + $ref: '#/components/schemas/UUID' + AIGatewayIdentityProviderIdOrName: + name: identityProviderIdOrName + in: path + required: true + description: The unique ID or name of the AI Gateway Identity provider. + examples: + name: + value: my-entity-name + summary: The name of the entity. + id: + value: bf138ba2-c9b1-4229-b268-04d9d8a6410b + summary: The id of the entity. + schema: + $ref: '#/components/schemas/AIGatewayEntityIdentifier' + AIGatewayMcpServerIdOrName: + name: mcpServerIdOrName + in: path + required: true + description: The unique ID or name of the MCP Server. + examples: + name: + value: my-entity-name + summary: The name of the entity. + id: + value: bf138ba2-c9b1-4229-b268-04d9d8a6410b + summary: The id of the entity. + schema: + $ref: '#/components/schemas/AIGatewayEntityIdentifier' + AIGatewayModelIdOrName: + name: modelIdOrName + in: path + required: true + description: The unique ID or name of the AI Gateway model. + examples: + name: + value: my-entity-name + summary: The name of the entity. + id: + value: bf138ba2-c9b1-4229-b268-04d9d8a6410b + summary: The id of the entity. + schema: + $ref: '#/components/schemas/AIGatewayEntityIdentifier' + AIGatewayModelProviderIdOrName: + name: modelProviderIdOrName + in: path + required: true + description: The unique ID or name of the AI Gateway model provider. + examples: + name: + value: my-entity-name + summary: The name of the entity. + id: + value: bf138ba2-c9b1-4229-b268-04d9d8a6410b + summary: The id of the entity. + schema: + $ref: '#/components/schemas/AIGatewayEntityIdentifier' + AIGatewayPolicyIdOrName: + name: policyIdOrName + in: path + required: true + description: The unique ID or name of the AI Gateway policy. + examples: + name: + value: my-entity-name + summary: The name of the entity. + id: + value: bf138ba2-c9b1-4229-b268-04d9d8a6410b + summary: The id of the entity. + schema: + $ref: '#/components/schemas/AIGatewayEntityIdentifier' + AIGatewayVaultIdOrName: + name: vaultIdOrName + in: path + required: true + description: The unique ID or name of the AI Gateway Vault. + examples: + name: + value: my-entity-name + summary: The name of the entity. + id: + value: bf138ba2-c9b1-4229-b268-04d9d8a6410b + summary: The id of the entity. + schema: + $ref: '#/components/schemas/AIGatewayEntityIdentifier' + PageAfter: + name: 'page[after]' + description: 'Request the next page of data, starting with the item after this parameter.' + required: false + in: query + allowEmptyValue: true + schema: + type: string + example: ewogICJpZCI6ICJoZWxsbyB3b3JsZCIKfQ + PageNumber: + name: 'page[number]' + description: Determines which page of the entities to retrieve. + required: false + in: query + allowEmptyValue: true + schema: + type: integer + example: 1 + PageSize: + name: 'page[size]' + description: The maximum number of items to include per page. The last page of a collection may include fewer items. + required: false + in: query + allowEmptyValue: true + schema: + type: integer + example: 10 + schemas: + AIGatewayEntityIdentifier: + description: 'Identifier for an AI Gateway entity. In some cases, this may be the entity name or ID.' + type: string + example: my-entity-name + maxLength: 256 + minLength: 1 + pattern: '^[A-Za-z0-9._-]{1,256}$' + AIGatewayProxyURL: + description: Proxy URL associated with reaching the data-planes connected to a control-plane. + type: object + properties: + host: + description: Hostname of the proxy URL. + type: string + port: + description: Port of the proxy URL. + type: integer + protocol: + description: Protocol of the proxy URL. + type: string + example: + host: example.com + port: 443 + protocol: https + additionalProperties: false + required: + - host + - port + - protocol + CreateAIGatewayRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + display_name: + description: The display name for this AI Gateway. + type: string + example: My AI Gateway + maxLength: 256 + minLength: 1 + name: + description: The name for this AI Gateway. This value is immutable after creation. + example: my-ai-gateway + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + description: + description: The description of the AI Gateway. + type: string + example: An AI Gateway for my organization. + maxLength: 1024 + proxy_urls: + description: Array of proxy URLs associated with reaching the data-planes connected to a control-plane. + type: array + items: + $ref: '#/components/schemas/AIGatewayProxyURL' + format: set + labels: + $ref: '#/components/schemas/PublicLabels' + additionalProperties: true + required: + - display_name + - name + UpdateAIGatewayRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + display_name: + description: The display name for this AI Gateway. + type: string + example: My AI Gateway + maxLength: 256 + minLength: 1 + name: + description: The name for this AI Gateway. This value is immutable after creation. + example: my-ai-gateway + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + description: + description: The description of the AI Gateway. + type: string + example: An AI Gateway for my organization. + maxLength: 1024 + proxy_urls: + description: Array of proxy URLs associated with reaching the data-planes connected to a control-plane. + type: array + items: + $ref: '#/components/schemas/AIGatewayProxyURL' + format: set + labels: + $ref: '#/components/schemas/PublicLabels' + additionalProperties: true + required: + - display_name + - name + AIGateway: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + display_name: + description: The display name for this AI Gateway. + type: string + example: My AI Gateway + maxLength: 256 + minLength: 1 + name: + description: The name for this AI Gateway. This value is immutable after creation. + example: my-ai-gateway + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + description: + description: The description of the AI Gateway. + type: string + example: An AI Gateway for my organization. + maxLength: 1024 + proxy_urls: + description: Array of proxy URLs associated with reaching the data-planes connected to a control-plane. + type: array + items: + $ref: '#/components/schemas/AIGatewayProxyURL' + format: set + labels: + $ref: '#/components/schemas/PublicLabels' + id: + $ref: '#/components/schemas/UUID' + endpoints: + description: Object containing AI Gateway access endpoints. + type: object + additionalProperties: false + properties: + configuration: + description: Configuration Endpoint. + type: string + format: url + example: 'https://acfe5f253f.cp.konghq.com' + readOnly: true + telemetry: + description: Telemetry Endpoint. + type: string + format: url + example: 'https://acfe5f253f.tp0.konghq.com' + readOnly: true + required: + - configuration + - telemetry + config_hash: + description: | + The hash of the latest configuration for the gateway. Every change to an entity + under this gateway will result in a new config_hash being generated. + The config hash can be used to verify if the config hash of an AI Gateway + node is up to date with the AI Gateway. The config hash will be the same if they are in sync. + type: string + readOnly: true + created_at: + $ref: '#/components/schemas/CreatedAt' + updated_at: + $ref: '#/components/schemas/UpdatedAt' + additionalProperties: true + required: + - display_name + - name + - id + - endpoints + - created_at + - updated_at + CreateAIGatewayDataPlaneCertificateRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + cert: + description: JSON escaped string of the certificate. + type: string + title: + description: A human-readable name for the certificate. + type: string + maxLength: 256 + minLength: 1 + description: + description: An optional description of the certificate. + type: string + maxLength: 1024 + additionalProperties: false + required: + - cert + - title + AIGatewayDataPlaneClientCertificate: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + cert: + description: JSON escaped string of the certificate. + type: string + title: + description: A human-readable name for the certificate. + type: string + maxLength: 256 + minLength: 1 + description: + description: An optional description of the certificate. + type: string + maxLength: 1024 + id: + $ref: '#/components/schemas/UUID' + created_at: + $ref: '#/components/schemas/CreatedAt' + updated_at: + $ref: '#/components/schemas/UpdatedAt' + metadata: + description: Metadata extracted from the certificate. + type: object + additionalProperties: false + properties: + subject: + description: The certificate subject. + type: string + issuer: + description: The certificate issuer. + type: string + san_names: + description: Subject alternative names (deprecated). + type: array + items: + type: string + deprecated: true + expiry: + description: Unix timestamp of certificate expiry. + type: integer + format: int64 + key_usages: + description: Key usage types for the certificate. + type: array + items: + type: string + snis: + description: Server Name Indications associated with the certificate. + type: array + items: + type: string + dns_names: + description: DNS subject alternative names. + type: array + items: + type: string + email_addresses: + description: Email subject alternative names. + type: array + items: + type: string + ip_addresses: + description: IP subject alternative names. + type: array + items: + type: string + uris: + description: URI subject alternative names. + type: array + items: + type: string + is_ca: + description: Whether the certificate is a CA certificate. + type: boolean + readOnly: true + additionalProperties: false + required: + - cert + - title + - id + - created_at + - updated_at + AIGatewayDataPlaneNode: + type: object + properties: + id: + $ref: '#/components/schemas/UUID' + version: + type: string + hostname: + type: string + last_ping: + type: integer + type: + type: string + created_at: + $ref: '#/components/schemas/CreatedAt' + updated_at: + $ref: '#/components/schemas/UpdatedAt' + config_version: + description: The version of the configuration applied by the node. + type: string + readOnly: true + errors: + description: Validation or configuration errors reported by the data plane node. + type: array + items: + $ref: '#/components/schemas/AIGatewayDataPlaneNodeError' + compatibility_status: + type: object + additionalProperties: false + properties: + state: + type: string + issues: + type: array + items: + $ref: '#/components/schemas/AIGatewayDataPlaneNodeCompatibilityIssue' + additionalProperties: false + required: + - id + - version + - hostname + - last_ping + - type + - created_at + - updated_at + - compatibility_status + AIGatewayDataPlaneNodeError: + type: object + properties: + name: + type: string + error_message: + type: string + config_hash: + type: string + flattened_errors: + type: array + items: + $ref: '#/components/schemas/AIGatewayDataPlaneNodeFlattenedError' + fields: + type: object + additionalProperties: + type: string + code: + type: integer + source: + type: string + traceback: + type: string + additionalProperties: false + required: + - name + - error_message + - code + - source + - traceback + title: A Node Error + AIGatewayDataPlaneNodeFlattenedError: + type: object + properties: + entity_id: + type: string + entity_name: + type: string + entity_type: + type: string + errors: + type: array + items: + $ref: '#/components/schemas/AIGatewayDataPlaneNodeErrorDetail' + additionalProperties: false + AIGatewayDataPlaneNodeErrorDetail: + type: object + properties: + error_message: + type: string + type: + type: string + field: + type: string + additionalProperties: false + AIGatewayExpectedConfigVersion: + type: object + properties: + expected_config_version: + description: The expected configuration version. + type: string + created_at: + $ref: '#/components/schemas/CreatedAt' + required: + - expected_config_version + AIGatewayDataPlaneNodeCompatibilityIssue: + type: object + properties: + code: + description: The compatibility issue code. + type: string + severity: + description: The severity of the issue. + type: string + description: + description: The description of the issue. + type: string + resolution: + description: Steps required to take in order to resolve the issue. + type: string + affected_resources: + description: Details of the resources affected by the issue. + type: array + items: + $ref: '#/components/schemas/AIGatewayDataPlaneNodeCompatibilityIssueAffectedResource' + documentation_url: + description: Doc URL for the compatibility issue. + type: string + additionalProperties: false + required: + - code + - severity + - description + - resolution + - affected_resources + - documentation_url + AIGatewayDataPlaneNodeCompatibilityIssueAffectedResource: + type: object + properties: + id: + description: ID of the affected resource. + type: string + type: + description: Type of the affected resource. + type: string + parent_code: + description: Parent Issue Code. + type: string + details: + description: Details of the affected resource. + type: array + items: + type: string + additionalProperties: false + required: + - id + - type + - parent_code + - details + AIGatewayModelModel: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for proxying synchronous requests/responses to/from an AI Gateway model using generative APIs. + type: object + properties: + display_name: + description: The display name for this model instance. + type: string + example: My GPT 5 model + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model, used as a stable human-readable reference. This value is immutable after creation. + example: my-gpt-5-model + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + enabled: + description: Whether the model is enabled. + type: boolean + example: true + default: true + access: + $ref: '#/components/schemas/AIGatewayModelAccess' + formats: + description: List of request/response formats supported by this model. + type: array + items: + $ref: '#/components/schemas/AIGatewayModelFormat' + maxItems: 1 + minItems: 1 + targets: + description: One or more backend models that this model entry routes to. + type: array + items: + $ref: '#/components/schemas/AIGatewayTarget' + minItems: 1 + policies: + $ref: '#/components/schemas/AIGatewayPolicyReferences' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + type: + type: string + enum: + - model + config: + $ref: '#/components/schemas/AIGatewayModelModelConfig' + capabilities: + description: List of AI capabilities enabled for this model. + type: array + items: + type: string + enum: + - generate + - agentic + - realtime + - embeddings + - image + - audio/speech + - audio/transcription + - audio/translation + - video + - rerank + minItems: 1 + required: + - display_name + - name + - formats + - targets + - type + - config + - capabilities + AIGatewayModelAPI: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for proxying asynchronous requests/responses to/from an AI Gateway model using the files and batches APIs. + type: object + properties: + display_name: + description: The display name for this model instance. + type: string + example: My GPT 5 model + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model, used as a stable human-readable reference. This value is immutable after creation. + example: my-gpt-5-model + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + enabled: + description: Whether the model is enabled. + type: boolean + example: true + default: true + access: + $ref: '#/components/schemas/AIGatewayModelAccess' + formats: + description: List of request/response formats supported by this model. + type: array + items: + $ref: '#/components/schemas/AIGatewayModelFormat' + maxItems: 1 + minItems: 1 + targets: + description: One or more backend models that this model entry routes to. + type: array + items: + $ref: '#/components/schemas/AIGatewayTarget' + minItems: 1 + policies: + $ref: '#/components/schemas/AIGatewayPolicyReferences' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + type: + type: string + enum: + - api + config: + $ref: '#/components/schemas/AIGatewayModelAPIConfig' + capabilities: + description: List of AI capabilities enabled for this API model. + type: array + items: + type: string + enum: + - batches + - files + required: + - display_name + - name + - formats + - targets + - type + - config + - capabilities + CreateAIGatewayModelRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an AI Gateway model. + discriminator: + propertyName: type + mapping: + api: '#/components/schemas/AIGatewayModelAPI' + model: '#/components/schemas/AIGatewayModelModel' + oneOf: + - $ref: '#/components/schemas/AIGatewayModelAPI' + - $ref: '#/components/schemas/AIGatewayModelModel' + UpdateAIGatewayModelRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an AI Gateway model. + discriminator: + propertyName: type + mapping: + api: '#/components/schemas/AIGatewayModelAPI' + model: '#/components/schemas/AIGatewayModelModel' + oneOf: + - $ref: '#/components/schemas/AIGatewayModelAPI' + - $ref: '#/components/schemas/AIGatewayModelModel' + AIGatewayModel: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an AI Gateway model. + type: object + properties: + id: + $ref: '#/components/schemas/UUID' + created_at: + $ref: '#/components/schemas/CreatedAt' + updated_at: + $ref: '#/components/schemas/UpdatedAt' + discriminator: + propertyName: type + mapping: + api: '#/components/schemas/AIGatewayModelAPI' + model: '#/components/schemas/AIGatewayModelModel' + oneOf: + - $ref: '#/components/schemas/AIGatewayModelAPI' + - $ref: '#/components/schemas/AIGatewayModelModel' + required: + - id + - created_at + - updated_at + AIGatewayModelModelConfig: + description: 'Routing, logging, and load balancing configuration for the model.' + type: object + properties: + route: + $ref: '#/components/schemas/AIGatewayRouteConfig' + logging: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for AI Gateway logging. + type: object + additionalProperties: false + properties: + payloads: + type: boolean + default: false + statistics: + type: boolean + default: true + response_streaming: + type: string + default: allow + enum: + - allow + - always + - deny + max_request_body_size: + description: Maximum size of request body to parse. Set to 0 for unlimited. + type: integer + default: 8388608 + model: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + default: + name_header: true + additionalProperties: false + properties: + alias: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + An alias for the model, used to select the target virtual model when passed in the "model" parameter of the request body. + When not set, this defaults to the AI Gateway model's name. + type: string + name_header: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Display the model name selected in the X-Kong-LLM-Model response header + type: boolean + default: true + balancer: + $ref: '#/components/schemas/AIGatewayModelBalancerConfig' + proxy: + $ref: '#/components/schemas/AIGatewayProxyConfig' + additionalProperties: false + required: + - route + AIGatewayModelAPIConfig: + description: 'Routing, logging, and load balancing configuration for the model.' + type: object + properties: + route: + $ref: '#/components/schemas/AIGatewayRouteConfig' + logging: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for AI Gateway logging. + type: object + additionalProperties: false + properties: + payloads: + type: boolean + default: false + statistics: + type: boolean + default: true + response_streaming: + type: string + default: allow + enum: + - allow + - always + - deny + max_request_body_size: + description: Maximum size of request body to parse. Set to 0 for unlimited. + type: integer + default: 8388608 + balancer: + $ref: '#/components/schemas/AIGatewayModelBalancerConfig' + proxy: + $ref: '#/components/schemas/AIGatewayProxyConfig' + model: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + additionalProperties: false + properties: + alias: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + An alias for the model, used to select the target virtual model when passed in the "model" parameter of the request body. + When not set, this defaults to the AI Gateway model's name. + type: string + additionalProperties: false + required: + - route + AIGatewayModelBalancerConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for a model's load balancer when multiple target models are configured. + discriminator: + propertyName: algorithm + mapping: + consistent-hashing: '#/components/schemas/AIGatewayModelBalancerConsistentHashingConfig' + least-connections: '#/components/schemas/AIGatewayModelBalancerLeastConnectionsConfig' + lowest-latency: '#/components/schemas/AIGatewayModelBalancerLowestLatencyConfig' + lowest-usage: '#/components/schemas/AIGatewayModelBalancerLowestUsageConfig' + priority: '#/components/schemas/AIGatewayModelBalancerPriorityConfig' + round-robin: '#/components/schemas/AIGatewayModelBalancerRoundRobinConfig' + semantic: '#/components/schemas/AIGatewayModelBalancerSemanticConfig' + oneOf: + - $ref: '#/components/schemas/AIGatewayModelBalancerConsistentHashingConfig' + - $ref: '#/components/schemas/AIGatewayModelBalancerLeastConnectionsConfig' + - $ref: '#/components/schemas/AIGatewayModelBalancerLowestLatencyConfig' + - $ref: '#/components/schemas/AIGatewayModelBalancerLowestUsageConfig' + - $ref: '#/components/schemas/AIGatewayModelBalancerPriorityConfig' + - $ref: '#/components/schemas/AIGatewayModelBalancerRoundRobinConfig' + - $ref: '#/components/schemas/AIGatewayModelBalancerSemanticConfig' + AIGatewayModelBalancerConsistentHashingConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + connect_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + fail_timeout: + description: The period of time (in milliseconds) the target will be considered unavailable after the number of unsuccessful attempts reaches `max_fails`. + type: integer + default: 10000 + maximum: 2147483646 + minimum: 1 + failover_criteria: + description: 'Specifies in which cases an upstream response should be failover to the next target. Each option in the array is equivalent to the function of https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream' + type: array + items: + type: string + enum: + - error + - http_403 + - http_404 + - http_429 + - http_500 + - http_502 + - http_503 + - http_504 + - invalid_header + - non_idempotent + - timeout + default: + - error + - timeout + max_fails: + description: 'Number of unsuccessful attempts to communicate with a target that should occur in the duration defined by `fail_timeout` before the target is considered unavailable. The zero value disables the circuit breaker. What is considered an unsuccessful attempt is defined by `failover_criteria`. Note the cases of `error`, `timeout` and `invalid_header` are always considered unsuccessful attempts, while the cases of `http_403` and `http_404` are never considered unsuccessful attempts.' + type: integer + default: 0 + maximum: 32767 + minimum: 0 + read_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + retries: + description: The number of retries to execute upon failure to proxy. + type: integer + default: 5 + maximum: 32767 + minimum: 0 + slots: + description: The number of slots in the load balancer algorithm. + type: integer + default: 10000 + maximum: 65536 + minimum: 10 + write_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + algorithm: + type: string + enum: + - consistent-hashing + hash_on_header: + description: The header to use for consistent-hashing. + type: string + default: X-Kong-LLM-Request-ID + additionalProperties: false + required: + - algorithm + AIGatewayModelBalancerLeastConnectionsConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + connect_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + fail_timeout: + description: The period of time (in milliseconds) the target will be considered unavailable after the number of unsuccessful attempts reaches `max_fails`. + type: integer + default: 10000 + maximum: 2147483646 + minimum: 1 + failover_criteria: + description: 'Specifies in which cases an upstream response should be failover to the next target. Each option in the array is equivalent to the function of https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream' + type: array + items: + type: string + enum: + - error + - http_403 + - http_404 + - http_429 + - http_500 + - http_502 + - http_503 + - http_504 + - invalid_header + - non_idempotent + - timeout + default: + - error + - timeout + max_fails: + description: 'Number of unsuccessful attempts to communicate with a target that should occur in the duration defined by `fail_timeout` before the target is considered unavailable. The zero value disables the circuit breaker. What is considered an unsuccessful attempt is defined by `failover_criteria`. Note the cases of `error`, `timeout` and `invalid_header` are always considered unsuccessful attempts, while the cases of `http_403` and `http_404` are never considered unsuccessful attempts.' + type: integer + default: 0 + maximum: 32767 + minimum: 0 + read_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + retries: + description: The number of retries to execute upon failure to proxy. + type: integer + default: 5 + maximum: 32767 + minimum: 0 + slots: + description: The number of slots in the load balancer algorithm. + type: integer + default: 10000 + maximum: 65536 + minimum: 10 + write_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + algorithm: + type: string + enum: + - least-connections + additionalProperties: false + required: + - algorithm + AIGatewayModelBalancerLowestLatencyConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + connect_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + fail_timeout: + description: The period of time (in milliseconds) the target will be considered unavailable after the number of unsuccessful attempts reaches `max_fails`. + type: integer + default: 10000 + maximum: 2147483646 + minimum: 1 + failover_criteria: + description: 'Specifies in which cases an upstream response should be failover to the next target. Each option in the array is equivalent to the function of https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream' + type: array + items: + type: string + enum: + - error + - http_403 + - http_404 + - http_429 + - http_500 + - http_502 + - http_503 + - http_504 + - invalid_header + - non_idempotent + - timeout + default: + - error + - timeout + max_fails: + description: 'Number of unsuccessful attempts to communicate with a target that should occur in the duration defined by `fail_timeout` before the target is considered unavailable. The zero value disables the circuit breaker. What is considered an unsuccessful attempt is defined by `failover_criteria`. Note the cases of `error`, `timeout` and `invalid_header` are always considered unsuccessful attempts, while the cases of `http_403` and `http_404` are never considered unsuccessful attempts.' + type: integer + default: 0 + maximum: 32767 + minimum: 0 + read_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + retries: + description: The number of retries to execute upon failure to proxy. + type: integer + default: 5 + maximum: 32767 + minimum: 0 + slots: + description: The number of slots in the load balancer algorithm. + type: integer + default: 10000 + maximum: 65536 + minimum: 10 + write_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + algorithm: + type: string + enum: + - lowest-latency + latency_strategy: + description: 'What metrics to use for latency. Available values are: `tpot` (time-per-output-token) and `e2e`.' + type: string + default: tpot + enum: + - e2e + - tpot + additionalProperties: false + required: + - algorithm + - latency_strategy + AIGatewayModelBalancerLowestUsageConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + connect_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + fail_timeout: + description: The period of time (in milliseconds) the target will be considered unavailable after the number of unsuccessful attempts reaches `max_fails`. + type: integer + default: 10000 + maximum: 2147483646 + minimum: 1 + failover_criteria: + description: 'Specifies in which cases an upstream response should be failover to the next target. Each option in the array is equivalent to the function of https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream' + type: array + items: + type: string + enum: + - error + - http_403 + - http_404 + - http_429 + - http_500 + - http_502 + - http_503 + - http_504 + - invalid_header + - non_idempotent + - timeout + default: + - error + - timeout + max_fails: + description: 'Number of unsuccessful attempts to communicate with a target that should occur in the duration defined by `fail_timeout` before the target is considered unavailable. The zero value disables the circuit breaker. What is considered an unsuccessful attempt is defined by `failover_criteria`. Note the cases of `error`, `timeout` and `invalid_header` are always considered unsuccessful attempts, while the cases of `http_403` and `http_404` are never considered unsuccessful attempts.' + type: integer + default: 0 + maximum: 32767 + minimum: 0 + read_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + retries: + description: The number of retries to execute upon failure to proxy. + type: integer + default: 5 + maximum: 32767 + minimum: 0 + slots: + description: The number of slots in the load balancer algorithm. + type: integer + default: 10000 + maximum: 65536 + minimum: 10 + write_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + algorithm: + type: string + enum: + - lowest-usage + tokens_count_strategy: + description: Methodology to use for token usage calculation. + type: string + default: total-tokens + enum: + - completion-tokens + - cost + - llm-accuracy + - prompt-tokens + - total-tokens + additionalProperties: false + required: + - algorithm + - tokens_count_strategy + AIGatewayModelBalancerPriorityConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + connect_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + fail_timeout: + description: The period of time (in milliseconds) the target will be considered unavailable after the number of unsuccessful attempts reaches `max_fails`. + type: integer + default: 10000 + maximum: 2147483646 + minimum: 1 + failover_criteria: + description: 'Specifies in which cases an upstream response should be failover to the next target. Each option in the array is equivalent to the function of https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream' + type: array + items: + type: string + enum: + - error + - http_403 + - http_404 + - http_429 + - http_500 + - http_502 + - http_503 + - http_504 + - invalid_header + - non_idempotent + - timeout + default: + - error + - timeout + max_fails: + description: 'Number of unsuccessful attempts to communicate with a target that should occur in the duration defined by `fail_timeout` before the target is considered unavailable. The zero value disables the circuit breaker. What is considered an unsuccessful attempt is defined by `failover_criteria`. Note the cases of `error`, `timeout` and `invalid_header` are always considered unsuccessful attempts, while the cases of `http_403` and `http_404` are never considered unsuccessful attempts.' + type: integer + default: 0 + maximum: 32767 + minimum: 0 + read_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + retries: + description: The number of retries to execute upon failure to proxy. + type: integer + default: 5 + maximum: 32767 + minimum: 0 + slots: + description: The number of slots in the load balancer algorithm. + type: integer + default: 10000 + maximum: 65536 + minimum: 10 + write_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + algorithm: + type: string + enum: + - priority + additionalProperties: false + required: + - algorithm + AIGatewayModelBalancerRoundRobinConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + connect_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + fail_timeout: + description: The period of time (in milliseconds) the target will be considered unavailable after the number of unsuccessful attempts reaches `max_fails`. + type: integer + default: 10000 + maximum: 2147483646 + minimum: 1 + failover_criteria: + description: 'Specifies in which cases an upstream response should be failover to the next target. Each option in the array is equivalent to the function of https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream' + type: array + items: + type: string + enum: + - error + - http_403 + - http_404 + - http_429 + - http_500 + - http_502 + - http_503 + - http_504 + - invalid_header + - non_idempotent + - timeout + default: + - error + - timeout + max_fails: + description: 'Number of unsuccessful attempts to communicate with a target that should occur in the duration defined by `fail_timeout` before the target is considered unavailable. The zero value disables the circuit breaker. What is considered an unsuccessful attempt is defined by `failover_criteria`. Note the cases of `error`, `timeout` and `invalid_header` are always considered unsuccessful attempts, while the cases of `http_403` and `http_404` are never considered unsuccessful attempts.' + type: integer + default: 0 + maximum: 32767 + minimum: 0 + read_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + retries: + description: The number of retries to execute upon failure to proxy. + type: integer + default: 5 + maximum: 32767 + minimum: 0 + slots: + description: The number of slots in the load balancer algorithm. + type: integer + default: 10000 + maximum: 65536 + minimum: 10 + write_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + algorithm: + type: string + enum: + - round-robin + additionalProperties: false + required: + - algorithm + AIGatewayModelBalancerSemanticConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + connect_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + fail_timeout: + description: The period of time (in milliseconds) the target will be considered unavailable after the number of unsuccessful attempts reaches `max_fails`. + type: integer + default: 10000 + maximum: 2147483646 + minimum: 1 + failover_criteria: + description: 'Specifies in which cases an upstream response should be failover to the next target. Each option in the array is equivalent to the function of https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream' + type: array + items: + type: string + enum: + - error + - http_403 + - http_404 + - http_429 + - http_500 + - http_502 + - http_503 + - http_504 + - invalid_header + - non_idempotent + - timeout + default: + - error + - timeout + max_fails: + description: 'Number of unsuccessful attempts to communicate with a target that should occur in the duration defined by `fail_timeout` before the target is considered unavailable. The zero value disables the circuit breaker. What is considered an unsuccessful attempt is defined by `failover_criteria`. Note the cases of `error`, `timeout` and `invalid_header` are always considered unsuccessful attempts, while the cases of `http_403` and `http_404` are never considered unsuccessful attempts.' + type: integer + default: 0 + maximum: 32767 + minimum: 0 + read_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + retries: + description: The number of retries to execute upon failure to proxy. + type: integer + default: 5 + maximum: 32767 + minimum: 0 + slots: + description: The number of slots in the load balancer algorithm. + type: integer + default: 10000 + maximum: 65536 + minimum: 10 + write_timeout: + type: integer + default: 60000 + maximum: 2147483646 + minimum: 1 + algorithm: + type: string + enum: + - semantic + embeddings: + description: Embeddings model configuration for this model. + type: object + additionalProperties: false + properties: + allow_auth_override: + description: | + When enabled, request-level auth parameters (such as API keys or bearer tokens) will override the static values defined for the provider. + type: boolean + default: false + provider: + $ref: '#/components/schemas/AIGatewayModelProviderReference' + name: + description: The name of the embeddings model. + type: string + config: + $ref: '#/components/schemas/AIGatewayEmbeddingsModelConfig' + required: + - name + - provider + - config + vectordb: + $ref: '#/components/schemas/AIGatewayModelVectorDBConfig' + additionalProperties: false + required: + - algorithm + - embeddings + - vectordb + AIGatewayEmbeddingsModelConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an embeddings model. + discriminator: + propertyName: type + mapping: + azure: '#/components/schemas/AIGatewayAzureEmbeddingsModelConfig' + bedrock: '#/components/schemas/AIGatewayBedrockEmbeddingsModelConfig' + gemini: '#/components/schemas/AIGatewayGeminiEmbeddingsModelConfig' + huggingface: '#/components/schemas/AIGatewayHuggingfaceEmbeddingsModelConfig' + mistral: '#/components/schemas/AIGatewayMistralEmbeddingsModelConfig' + ollama: '#/components/schemas/AIGatewayOllamaEmbeddingsModelConfig' + openai: '#/components/schemas/AIGatewayOpenaiEmbeddingsModelConfig' + vertex: '#/components/schemas/AIGatewayVertexEmbeddingsModelConfig' + oneOf: + - $ref: '#/components/schemas/AIGatewayAzureEmbeddingsModelConfig' + - $ref: '#/components/schemas/AIGatewayBedrockEmbeddingsModelConfig' + - $ref: '#/components/schemas/AIGatewayGeminiEmbeddingsModelConfig' + - $ref: '#/components/schemas/AIGatewayHuggingfaceEmbeddingsModelConfig' + - $ref: '#/components/schemas/AIGatewayMistralEmbeddingsModelConfig' + - $ref: '#/components/schemas/AIGatewayOllamaEmbeddingsModelConfig' + - $ref: '#/components/schemas/AIGatewayOpenaiEmbeddingsModelConfig' + - $ref: '#/components/schemas/AIGatewayVertexEmbeddingsModelConfig' + AIGatewayAzureEmbeddingsModelConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Azure-specific configuration for a model. + type: object + properties: + upstream_url: + description: The URL of the embeddings model. + type: string + type: + type: string + enum: + - azure + deployment_id: + description: The Azure deployment ID for the model. + type: string + api_version: + description: The Azure OpenAI API version to use. + type: string + default: '2023-05-15' + additionalProperties: false + required: + - type + - deployment_id + AIGatewayBedrockEmbeddingsModelConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + AWS Bedrock-specific configuration for a model. + type: object + properties: + upstream_url: + description: The URL of the embeddings model. + type: string + type: + type: string + enum: + - bedrock + region: + description: | + The AWS region for the model. + Setting this option overrides the AWS_REGION environment variable. + type: string + batch_bucket_prefix: + description: S3 bucket prefix for batch inference jobs. + type: string + embeddings_normalize: + description: Whether to normalize embedding vectors in the response. + type: boolean + default: false + performance_config_latency: + description: Latency performance configuration for the model invocation. + type: string + video_output_s3_uri: + description: S3 URI for storing video generation outputs. + type: string + additionalProperties: false + required: + - type + AIGatewayGeminiEmbeddingsModelConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Google Gemini-specific configuration for a model. + type: object + properties: + upstream_url: + description: The URL of the embeddings model. + type: string + type: + type: string + enum: + - gemini + gcp_environment: + $ref: '#/components/schemas/GCPModelConfig' + additionalProperties: false + required: + - type + AIGatewayHuggingfaceEmbeddingsModelConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Hugging Face-specific configuration for a model. + type: object + properties: + upstream_url: + description: The URL of the embeddings model. + type: string + type: + type: string + enum: + - huggingface + use_cache: + description: Whether to use the Hugging Face inference cache. + type: boolean + default: false + wait_for_model: + description: Whether to wait for the model to load if it is not ready. + type: boolean + default: false + additionalProperties: false + required: + - type + AIGatewayMistralEmbeddingsModelConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + upstream_url: + description: The URL of the embeddings model. + type: string + type: + type: string + enum: + - mistral + additionalProperties: false + required: + - type + AIGatewayOllamaEmbeddingsModelConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Ollama-specific configuration for a model. + type: object + properties: + upstream_url: + description: The URL of the embeddings model. + type: string + type: + type: string + enum: + - ollama + additionalProperties: false + required: + - type + AIGatewayOpenaiEmbeddingsModelConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Openai-specific configuration for a model. + type: object + properties: + upstream_url: + description: The URL of the embeddings model. + type: string + type: + type: string + enum: + - openai + additionalProperties: false + required: + - type + AIGatewayVertexEmbeddingsModelConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Google Vertex-specific configuration for a model. + type: object + properties: + upstream_url: + description: The URL of the embeddings model. + type: string + type: + type: string + enum: + - vertex + gcp_environment: + $ref: '#/components/schemas/GCPModelConfig' + additionalProperties: false + required: + - type + AIGatewayModelVectorDBConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for the vector database used by the model. + discriminator: + propertyName: type + mapping: + pgvector: '#/components/schemas/AIGatewayModelVectorDBConfigPgVector' + redis: '#/components/schemas/AIGatewayModelVectorDBConfigRedis' + oneOf: + - $ref: '#/components/schemas/AIGatewayModelVectorDBConfigPgVector' + - $ref: '#/components/schemas/AIGatewayModelVectorDBConfigRedis' + AIGatewayModelVectorDBConfigRedis: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Config for connecting to a Cloud Provider's Redis instance. + type: object + properties: + type: + type: string + enum: + - redis + dimensions: + description: the desired dimensionality for the vectors + type: integer + distance_metric: + description: the distance metric to use for vector searches + type: string + enum: + - cosine + - euclidean + threshold: + description: the default similarity threshold for accepting semantic search results (float). Higher threshold means more results are considered similar. + type: number + cloud_authentication: + description: Auth related config for connecting to a Cloud Provider's Redis instance. + discriminator: + propertyName: type + mapping: + aws: '#/components/schemas/AIGatewayRedisAWSAuthentication' + azure: '#/components/schemas/AIGatewayRedisAzureAuthentication' + gcp: '#/components/schemas/AIGatewayRedisGCPAuthentication' + oneOf: + - $ref: '#/components/schemas/AIGatewayRedisAWSAuthentication' + - $ref: '#/components/schemas/AIGatewayRedisAzureAuthentication' + - $ref: '#/components/schemas/AIGatewayRedisGCPAuthentication' + cluster: + description: Cluster configuration for the Redis connection. + type: object + additionalProperties: false + properties: + max_redirections: + description: Maximum retry attempts for redirection. + type: integer + default: 5 + nodes: + description: Cluster addresses to use for Redis connections when the `redis` strategy is defined. Defining this field implies using a Redis Cluster. The minimum length of the array is 1 element. + type: array + items: + type: object + properties: + ip: + description: 'A string representing a host name, such as example.com.' + type: string + default: 127.0.0.1 + port: + description: 'An integer representing a port number between 0 and 65535, inclusive.' + type: integer + default: 6379 + maximum: 65535 + minimum: 0 + minItems: 1 + connect_timeout: + description: An integer representing a timeout in milliseconds. Must be between 0 and 2^31-2. + type: integer + default: 2000 + maximum: 2147483646 + minimum: 0 + connection_is_proxied: + description: 'If the connection to Redis is proxied (e.g. Envoy), set it `true`. Set the `host` and `port` to point to the proxy address.' + type: boolean + default: false + database: + description: Database to use for the Redis connection when using the `redis` strategy + type: integer + default: 0 + host: + description: | + A string representing a host name, such as example.com. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + default: 127.0.0.1 + x-referenceable: true + keepalive: + description: Keepalive configuration for the Redis connection. + type: object + additionalProperties: false + properties: + backlog: + description: 'Limits the total number of opened connections for a pool. If the connection pool is full, connection queues above the limit go into the backlog queue. If the backlog queue is full, subsequent connect operations fail and return `nil`. Queued operations (subject to set timeouts) resume once the number of connections in the pool is less than `pool_size`. If latency is high or throughput is low, try increasing this value. Empirically, this value is larger than `pool_size`.' + type: integer + maximum: 2147483646 + minimum: 0 + pool_size: + description: 'The size limit for every cosocket connection pool associated with every remote server, per worker process. If neither `pool_size` nor `backlog` is specified, no pool is created. If `pool_size` isn''t specified but `backlog` is specified, then the pool uses the default value. Try to increase (e.g. 512) this value if latency is high or throughput is low.' + type: integer + default: 256 + maximum: 2147483646 + minimum: 1 + password: + description: | + Password to use for Redis connections. If undefined, no AUTH commands are sent to Redis. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + port: + description: | + An integer representing a port number between 0 and 65535, inclusive. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + oneOf: + - type: integer + default: 6379 + maximum: 65535 + minimum: 0 + example: 6379 + - type: string + example: '{vault://hcv/redis/port}' + x-go-type: types.Referenceable + x-go-type-import: + path: github.com/kong/koko/internal/server/public/openapi/controlplanesconfig/types + name: types + x-referenceable: true + read_timeout: + description: An integer representing a timeout in milliseconds. Must be between 0 and 2^31-2. + type: integer + default: 2000 + maximum: 2147483646 + minimum: 0 + send_timeout: + description: An integer representing a timeout in milliseconds. Must be between 0 and 2^31-2. + type: integer + default: 2000 + maximum: 2147483646 + minimum: 0 + sentinel: + description: Configuration for Redis Sentinel. + type: object + additionalProperties: false + properties: + master: + description: Sentinel master to use for Redis connections. Defining this value implies using Redis Sentinel. + type: string + nodes: + description: Sentinel node addresses to use for Redis connections when the `redis` strategy is defined. Defining this field implies using a Redis Sentinel. The minimum length of the array is 1 element. + type: array + items: + type: object + properties: + host: + description: 'A string representing a host name, such as example.com.' + type: string + default: 127.0.0.1 + port: + description: 'An integer representing a port number between 0 and 65535, inclusive.' + type: integer + default: 6379 + maximum: 65535 + minimum: 0 + minItems: 1 + password: + description: | + Sentinel password to authenticate with a Redis Sentinel instance. If undefined, no AUTH commands are sent to Redis Sentinels. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + role: + description: Sentinel role to use for Redis connections when the `redis` strategy is defined. Defining this value implies using Redis Sentinel. + type: string + enum: + - any + - master + - slave + username: + description: | + Sentinel username to authenticate with a Redis Sentinel instance. If undefined, ACL authentication won't be performed. This requires Redis v6.2.0+. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + server_name: + description: | + A string representing an SNI (server name indication) value for TLS. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + ssl: + description: 'If set to true, uses SSL to connect to Redis.' + type: boolean + default: true + ssl_verify: + description: 'If set to true, verifies the validity of the server SSL certificate. If setting this parameter, also configure `lua_ssl_trusted_certificate` in `kong.conf` to specify the CA (or server) certificate used by your Redis server. You may also need to configure `lua_ssl_verify_depth` accordingly.' + type: boolean + default: true + username: + description: | + Username to use for Redis connections. If undefined, ACL authentication won't be performed. This requires Redis v6.0.0+. To be compatible with Redis v5.x.y, you can set it to `default`. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + additionalProperties: false + required: + - type + - dimensions + - distance_metric + AIGatewayModelVectorDBConfigPgVector: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + type: + type: string + enum: + - pgvector + dimensions: + description: the desired dimensionality for the vectors + type: integer + distance_metric: + description: the distance metric to use for vector searches + type: string + enum: + - cosine + - euclidean + threshold: + description: the default similarity threshold for accepting semantic search results (float). Higher threshold means more results are considered similar. + type: number + database: + description: the database of the pgvector database + type: string + default: kong-pgvector + host: + description: the host of the pgvector database + type: string + default: 127.0.0.1 + password: + description: | + the password of the pgvector database + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + writeOnly: true + x-referenceable: true + port: + description: the port of the pgvector database + type: integer + default: 5432 + ssl: + type: object + additionalProperties: false + properties: + enabled: + description: whether to use ssl for the pgvector database + type: boolean + default: true + cert: + description: the path of ssl cert to use for the pgvector database + type: string + cert_key: + description: the path of ssl cert key to use for the pgvector database + type: string + required: + description: whether ssl is required for the pgvector database + type: boolean + default: true + verify: + description: whether to verify ssl for the pgvector database + type: boolean + default: true + version: + description: the ssl version to use for the pgvector database + type: string + default: tlsv1_2 + enum: + - any + - tlsv1_2 + - tlsv1_3 + timeout: + description: the timeout of the pgvector database + type: number + default: 5000 + user: + description: | + the user of the pgvector database + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + default: postgres + x-referenceable: true + additionalProperties: false + required: + - type + - dimensions + - distance_metric + AIGatewayModelFormat: + description: Request and response format supported by this model. + type: object + properties: + type: + description: The format type. + type: string + example: openai + enum: + - anthropic + - bedrock + - cohere + - gemini + - huggingface + - openai + additionalProperties: false + AIGatewayTarget: + description: A target instance a model entry routes requests to. + type: object + properties: + name: + description: The name of the model defined in the upstream provider that will be executed. + type: string + example: gpt-5-model + weight: + description: The weight this target gets within the upstream load balancer + type: integer + example: 100 + default: 100 + maximum: 65535 + minimum: 1 + semantic_description: + description: | + The semantic description of the target, required if using semantic load balancing. + Specially, setting this to 'CATCHALL' will indicate such target to be used when no other targets match the semantic threshold. + type: string + allow_auth_override: + description: | + When enabled, request-level auth parameters (such as API keys or bearer tokens) will override the static values defined for the provider. + type: boolean + default: false + provider: + $ref: '#/components/schemas/AIGatewayModelProviderReference' + config: + $ref: '#/components/schemas/AIGatewayTargetConfig' + additionalProperties: false + required: + - name + - provider + - config + AIGatewayTargetConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for a target model. + discriminator: + propertyName: type + mapping: + anthropic: '#/components/schemas/AIGatewayTargetAnthropicConfig' + azure: '#/components/schemas/AIGatewayTargetAzureConfig' + bedrock: '#/components/schemas/AIGatewayTargetBedrockConfig' + cerebras: '#/components/schemas/AIGatewayTargetCerebrasConfig' + cohere: '#/components/schemas/AIGatewayTargetCohereConfig' + dashscope: '#/components/schemas/AIGatewayTargetDashscopeConfig' + databricks: '#/components/schemas/AIGatewayTargetDatabricksConfig' + deepseek: '#/components/schemas/AIGatewayTargetDeepseekConfig' + gemini: '#/components/schemas/AIGatewayTargetGeminiConfig' + huggingface: '#/components/schemas/AIGatewayTargetHuggingfaceConfig' + kimi: '#/components/schemas/AIGatewayTargetKimiConfig' + llama2: '#/components/schemas/AIGatewayTargetLlama2Config' + mistral: '#/components/schemas/AIGatewayTargetMistralConfig' + ollama: '#/components/schemas/AIGatewayTargetOllamaConfig' + openai: '#/components/schemas/AIGatewayTargetOpenaiConfig' + vercel: '#/components/schemas/AIGatewayTargetVercelConfig' + vertex: '#/components/schemas/AIGatewayTargetVertexConfig' + vllm: '#/components/schemas/AIGatewayTargetVllmConfig' + xai: '#/components/schemas/AIGatewayTargetXaiConfig' + oneOf: + - $ref: '#/components/schemas/AIGatewayTargetAnthropicConfig' + - $ref: '#/components/schemas/AIGatewayTargetAzureConfig' + - $ref: '#/components/schemas/AIGatewayTargetBedrockConfig' + - $ref: '#/components/schemas/AIGatewayTargetCerebrasConfig' + - $ref: '#/components/schemas/AIGatewayTargetCohereConfig' + - $ref: '#/components/schemas/AIGatewayTargetDashscopeConfig' + - $ref: '#/components/schemas/AIGatewayTargetDatabricksConfig' + - $ref: '#/components/schemas/AIGatewayTargetDeepseekConfig' + - $ref: '#/components/schemas/AIGatewayTargetGeminiConfig' + - $ref: '#/components/schemas/AIGatewayTargetHuggingfaceConfig' + - $ref: '#/components/schemas/AIGatewayTargetKimiConfig' + - $ref: '#/components/schemas/AIGatewayTargetLlama2Config' + - $ref: '#/components/schemas/AIGatewayTargetMistralConfig' + - $ref: '#/components/schemas/AIGatewayTargetOllamaConfig' + - $ref: '#/components/schemas/AIGatewayTargetOpenaiConfig' + - $ref: '#/components/schemas/AIGatewayTargetVercelConfig' + - $ref: '#/components/schemas/AIGatewayTargetVertexConfig' + - $ref: '#/components/schemas/AIGatewayTargetVllmConfig' + - $ref: '#/components/schemas/AIGatewayTargetXaiConfig' + AIGatewayTargetAnthropicConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Anthropic-specific configuration for a model. + type: object + properties: + embeddings_dimensions: + description: The number of dimensions for embedding outputs. + type: integer + max_tokens: + description: The maximum number of tokens to generate in the response. + type: integer + input_cost: + description: Cost per input token for billing and cost tracking. + type: number + output_cost: + description: Cost per output token for billing and cost tracking. + type: number + temperature: + description: Controls randomness in the model output. Higher values produce more varied responses. + type: number + top_k: + description: Limits the number of highest-probability tokens considered during generation. + type: integer + top_p: + description: Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered. + type: number + upstream_url: + description: The upstream URL for the model endpoint. + type: string + format: uri + type: + type: string + enum: + - anthropic + version: + description: The Anthropic API version to use. + type: string + default: '2023-06-01' + required: + - type + AIGatewayTargetAzureConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Azure-specific configuration for a model. + type: object + properties: + embeddings_dimensions: + description: The number of dimensions for embedding outputs. + type: integer + max_tokens: + description: The maximum number of tokens to generate in the response. + type: integer + input_cost: + description: Cost per input token for billing and cost tracking. + type: number + output_cost: + description: Cost per output token for billing and cost tracking. + type: number + temperature: + description: Controls randomness in the model output. Higher values produce more varied responses. + type: number + top_k: + description: Limits the number of highest-probability tokens considered during generation. + type: integer + top_p: + description: Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered. + type: number + upstream_url: + description: The upstream URL for the model endpoint. + type: string + format: uri + type: + type: string + enum: + - azure + deployment_id: + description: The Azure deployment ID for the model. + type: string + api_version: + description: The Azure OpenAI API version to use. + type: string + default: '2023-05-15' + required: + - type + - deployment_id + AIGatewayTargetBedrockConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + AWS Bedrock-specific configuration for a model. + type: object + properties: + embeddings_dimensions: + description: The number of dimensions for embedding outputs. + type: integer + max_tokens: + description: The maximum number of tokens to generate in the response. + type: integer + input_cost: + description: Cost per input token for billing and cost tracking. + type: number + output_cost: + description: Cost per output token for billing and cost tracking. + type: number + temperature: + description: Controls randomness in the model output. Higher values produce more varied responses. + type: number + top_k: + description: Limits the number of highest-probability tokens considered during generation. + type: integer + top_p: + description: Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered. + type: number + upstream_url: + description: The upstream URL for the model endpoint. + type: string + format: uri + type: + type: string + enum: + - bedrock + region: + description: | + The AWS region for the model. + Setting this option overrides the AWS_REGION environment variable. + type: string + batch_bucket_prefix: + description: S3 bucket prefix for batch inference jobs. + type: string + embeddings_normalize: + description: Whether to normalize embedding vectors in the response. + type: boolean + default: false + performance_config_latency: + description: Latency performance configuration for the model invocation. + type: string + video_output_s3_uri: + description: S3 URI for storing video generation outputs. + type: string + required: + - type + AIGatewayTargetCerebrasConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Cerebras-specific configuration for a model. + type: object + properties: + embeddings_dimensions: + description: The number of dimensions for embedding outputs. + type: integer + max_tokens: + description: The maximum number of tokens to generate in the response. + type: integer + input_cost: + description: Cost per input token for billing and cost tracking. + type: number + output_cost: + description: Cost per output token for billing and cost tracking. + type: number + temperature: + description: Controls randomness in the model output. Higher values produce more varied responses. + type: number + top_k: + description: Limits the number of highest-probability tokens considered during generation. + type: integer + top_p: + description: Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered. + type: number + upstream_url: + description: The upstream URL for the model endpoint. + type: string + format: uri + type: + type: string + enum: + - cerebras + required: + - type + AIGatewayTargetCohereConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Cohere-specific configuration for a model. + type: object + properties: + embeddings_dimensions: + description: The number of dimensions for embedding outputs. + type: integer + max_tokens: + description: The maximum number of tokens to generate in the response. + type: integer + input_cost: + description: Cost per input token for billing and cost tracking. + type: number + output_cost: + description: Cost per output token for billing and cost tracking. + type: number + temperature: + description: Controls randomness in the model output. Higher values produce more varied responses. + type: number + top_k: + description: Limits the number of highest-probability tokens considered during generation. + type: integer + top_p: + description: Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered. + type: number + upstream_url: + description: The upstream URL for the model endpoint. + type: string + format: uri + type: + type: string + enum: + - cohere + api_version: + description: | + Cohere API version. `v1` uses the legacy `/v1/chat` endpoint; `v2` (default) + uses `/v2/chat` and supports tool calling. + type: string + default: v2 + enum: + - v1 + - v2 + embedding_input_type: + description: The intended downstream use of the embeddings to improve model quality. + type: string + default: classification + enum: + - classification + - clustering + - image + - search_document + - search_query + wait_for_model: + description: Whether to wait for the model to be ready before sending the request. + type: boolean + default: false + required: + - type + AIGatewayTargetDashscopeConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Alibaba DashScope-specific configuration for a model. + type: object + properties: + embeddings_dimensions: + description: The number of dimensions for embedding outputs. + type: integer + max_tokens: + description: The maximum number of tokens to generate in the response. + type: integer + input_cost: + description: Cost per input token for billing and cost tracking. + type: number + output_cost: + description: Cost per output token for billing and cost tracking. + type: number + temperature: + description: Controls randomness in the model output. Higher values produce more varied responses. + type: number + top_k: + description: Limits the number of highest-probability tokens considered during generation. + type: integer + top_p: + description: Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered. + type: number + upstream_url: + description: The upstream URL for the model endpoint. + type: string + format: uri + type: + type: string + enum: + - dashscope + international: + description: Whether to use the international DashScope endpoint. + type: boolean + default: true + required: + - type + AIGatewayTargetDatabricksConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Databricks-specific configuration for a model. + type: object + properties: + embeddings_dimensions: + description: The number of dimensions for embedding outputs. + type: integer + max_tokens: + description: The maximum number of tokens to generate in the response. + type: integer + input_cost: + description: Cost per input token for billing and cost tracking. + type: number + output_cost: + description: Cost per output token for billing and cost tracking. + type: number + temperature: + description: Controls randomness in the model output. Higher values produce more varied responses. + type: number + top_k: + description: Limits the number of highest-probability tokens considered during generation. + type: integer + top_p: + description: Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered. + type: number + upstream_url: + description: The upstream URL for the model endpoint. + type: string + format: uri + type: + type: string + enum: + - databricks + workspace_instance_id: + description: The Databricks workspace instance ID. + type: string + required: + - type + - workspace_instance_id + AIGatewayTargetDeepseekConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Deepseek-specific configuration for a model. + type: object + properties: + embeddings_dimensions: + description: The number of dimensions for embedding outputs. + type: integer + max_tokens: + description: The maximum number of tokens to generate in the response. + type: integer + input_cost: + description: Cost per input token for billing and cost tracking. + type: number + output_cost: + description: Cost per output token for billing and cost tracking. + type: number + temperature: + description: Controls randomness in the model output. Higher values produce more varied responses. + type: number + top_k: + description: Limits the number of highest-probability tokens considered during generation. + type: integer + top_p: + description: Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered. + type: number + upstream_url: + description: The upstream URL for the model endpoint. + type: string + format: uri + type: + type: string + enum: + - deepseek + required: + - type + AIGatewayTargetGeminiConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Google Gemini-specific configuration for a model. + type: object + properties: + embeddings_dimensions: + description: The number of dimensions for embedding outputs. + type: integer + max_tokens: + description: The maximum number of tokens to generate in the response. + type: integer + input_cost: + description: Cost per input token for billing and cost tracking. + type: number + output_cost: + description: Cost per output token for billing and cost tracking. + type: number + temperature: + description: Controls randomness in the model output. Higher values produce more varied responses. + type: number + top_k: + description: Limits the number of highest-probability tokens considered during generation. + type: integer + top_p: + description: Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered. + type: number + upstream_url: + description: The upstream URL for the model endpoint. + type: string + format: uri + type: + type: string + enum: + - gemini + gcp_environment: + $ref: '#/components/schemas/GCPModelConfig' + required: + - type + AIGatewayTargetHuggingfaceConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Hugging Face-specific configuration for a model. + type: object + properties: + embeddings_dimensions: + description: The number of dimensions for embedding outputs. + type: integer + max_tokens: + description: The maximum number of tokens to generate in the response. + type: integer + input_cost: + description: Cost per input token for billing and cost tracking. + type: number + output_cost: + description: Cost per output token for billing and cost tracking. + type: number + temperature: + description: Controls randomness in the model output. Higher values produce more varied responses. + type: number + top_k: + description: Limits the number of highest-probability tokens considered during generation. + type: integer + top_p: + description: Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered. + type: number + upstream_url: + description: The upstream URL for the model endpoint. + type: string + format: uri + type: + type: string + enum: + - huggingface + use_cache: + description: Whether to use the Hugging Face inference cache. + type: boolean + default: false + wait_for_model: + description: Whether to wait for the model to load if it is not ready. + type: boolean + default: false + required: + - type + AIGatewayTargetKimiConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Kimi (Moonshot AI)-specific configuration for a model. + type: object + properties: + embeddings_dimensions: + description: The number of dimensions for embedding outputs. + type: integer + max_tokens: + description: The maximum number of tokens to generate in the response. + type: integer + input_cost: + description: Cost per input token for billing and cost tracking. + type: number + output_cost: + description: Cost per output token for billing and cost tracking. + type: number + temperature: + description: Controls randomness in the model output. Higher values produce more varied responses. + type: number + top_k: + description: Limits the number of highest-probability tokens considered during generation. + type: integer + top_p: + description: Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered. + type: number + upstream_url: + description: The upstream URL for the model endpoint. + type: string + format: uri + type: + type: string + enum: + - kimi + international: + description: | + When `true`, requests are sent to `api.moonshot.ai` (international). + When `false`, requests are sent to `api.moonshot.cn` (mainland China). + type: boolean + default: true + required: + - type + AIGatewayTargetLlama2Config: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Llama2-specific configuration for a model. + type: object + properties: + embeddings_dimensions: + description: The number of dimensions for embedding outputs. + type: integer + max_tokens: + description: The maximum number of tokens to generate in the response. + type: integer + input_cost: + description: Cost per input token for billing and cost tracking. + type: number + output_cost: + description: Cost per output token for billing and cost tracking. + type: number + temperature: + description: Controls randomness in the model output. Higher values produce more varied responses. + type: number + top_k: + description: Limits the number of highest-probability tokens considered during generation. + type: integer + top_p: + description: Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered. + type: number + upstream_url: + description: The upstream URL for the model endpoint. + type: string + format: uri + type: + type: string + enum: + - llama2 + format: + description: The request format to use when communicating with the Llama2 model. + type: string + enum: + - ollama + - openai + - raw + required: + - type + - format + - upstream_url + AIGatewayTargetMistralConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Mistral-specific configuration for a model. + type: object + properties: + embeddings_dimensions: + description: The number of dimensions for embedding outputs. + type: integer + max_tokens: + description: The maximum number of tokens to generate in the response. + type: integer + input_cost: + description: Cost per input token for billing and cost tracking. + type: number + output_cost: + description: Cost per output token for billing and cost tracking. + type: number + temperature: + description: Controls randomness in the model output. Higher values produce more varied responses. + type: number + top_k: + description: Limits the number of highest-probability tokens considered during generation. + type: integer + top_p: + description: Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered. + type: number + upstream_url: + description: The upstream URL for the model endpoint. + type: string + format: uri + type: + type: string + enum: + - mistral + format: + description: The request format to use when communicating with the Mistral model. + type: string + enum: + - ollama + - openai + required: + - type + - format + AIGatewayTargetOllamaConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Ollama-specific configuration for a model. + type: object + properties: + embeddings_dimensions: + description: The number of dimensions for embedding outputs. + type: integer + max_tokens: + description: The maximum number of tokens to generate in the response. + type: integer + input_cost: + description: Cost per input token for billing and cost tracking. + type: number + output_cost: + description: Cost per output token for billing and cost tracking. + type: number + temperature: + description: Controls randomness in the model output. Higher values produce more varied responses. + type: number + top_k: + description: Limits the number of highest-probability tokens considered during generation. + type: integer + top_p: + description: Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered. + type: number + upstream_url: + description: The upstream URL for the model endpoint. + type: string + format: uri + type: + type: string + enum: + - ollama + required: + - type + AIGatewayTargetOpenaiConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Openai-specific configuration for a model. + type: object + properties: + embeddings_dimensions: + description: The number of dimensions for embedding outputs. + type: integer + max_tokens: + description: The maximum number of tokens to generate in the response. + type: integer + input_cost: + description: Cost per input token for billing and cost tracking. + type: number + output_cost: + description: Cost per output token for billing and cost tracking. + type: number + temperature: + description: Controls randomness in the model output. Higher values produce more varied responses. + type: number + top_k: + description: Limits the number of highest-probability tokens considered during generation. + type: integer + top_p: + description: Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered. + type: number + upstream_url: + description: The upstream URL for the model endpoint. + type: string + format: uri + type: + type: string + enum: + - openai + required: + - type + AIGatewayTargetVercelConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Vercel AI Gateway-specific configuration for a model. + type: object + properties: + embeddings_dimensions: + description: The number of dimensions for embedding outputs. + type: integer + max_tokens: + description: The maximum number of tokens to generate in the response. + type: integer + input_cost: + description: Cost per input token for billing and cost tracking. + type: number + output_cost: + description: Cost per output token for billing and cost tracking. + type: number + temperature: + description: Controls randomness in the model output. Higher values produce more varied responses. + type: number + top_k: + description: Limits the number of highest-probability tokens considered during generation. + type: integer + top_p: + description: Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered. + type: number + upstream_url: + description: The upstream URL for the model endpoint. + type: string + format: uri + type: + type: string + enum: + - vercel + required: + - type + AIGatewayTargetVertexConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Google Vertex-specific configuration for a model. + type: object + properties: + embeddings_dimensions: + description: The number of dimensions for embedding outputs. + type: integer + max_tokens: + description: The maximum number of tokens to generate in the response. + type: integer + input_cost: + description: Cost per input token for billing and cost tracking. + type: number + output_cost: + description: Cost per output token for billing and cost tracking. + type: number + temperature: + description: Controls randomness in the model output. Higher values produce more varied responses. + type: number + top_k: + description: Limits the number of highest-probability tokens considered during generation. + type: integer + top_p: + description: Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered. + type: number + upstream_url: + description: The upstream URL for the model endpoint. + type: string + format: uri + type: + type: string + enum: + - vertex + gcp_environment: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for a model hosted on Google Cloud Project. + type: object + properties: + api_endpoint: + description: The custom API endpoint for the Gemini model. + type: string + location_id: + description: The Google Cloud location ID for the model endpoint. + type: string + project_id: + description: The Google Cloud project ID for the model endpoint. + type: string + endpoint_id: + description: | + The endpoint ID for the model. + This must be set when running a target model on Gemini on Vertex Model Garden. + type: string + required: + - api_endpoint + - location_id + - project_id + - endpoint_id + required: + - type + AIGatewayTargetVllmConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Vllm-specific configuration for a model. + type: object + properties: + embeddings_dimensions: + description: The number of dimensions for embedding outputs. + type: integer + max_tokens: + description: The maximum number of tokens to generate in the response. + type: integer + input_cost: + description: Cost per input token for billing and cost tracking. + type: number + output_cost: + description: Cost per output token for billing and cost tracking. + type: number + temperature: + description: Controls randomness in the model output. Higher values produce more varied responses. + type: number + top_k: + description: Limits the number of highest-probability tokens considered during generation. + type: integer + top_p: + description: Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered. + type: number + upstream_url: + description: The upstream URL for the model endpoint. + type: string + format: uri + type: + type: string + enum: + - vllm + required: + - type + - upstream_url + AIGatewayTargetXaiConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Xai-specific configuration for a model. + type: object + properties: + embeddings_dimensions: + description: The number of dimensions for embedding outputs. + type: integer + max_tokens: + description: The maximum number of tokens to generate in the response. + type: integer + input_cost: + description: Cost per input token for billing and cost tracking. + type: number + output_cost: + description: Cost per output token for billing and cost tracking. + type: number + temperature: + description: Controls randomness in the model output. Higher values produce more varied responses. + type: number + top_k: + description: Limits the number of highest-probability tokens considered during generation. + type: integer + top_p: + description: Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered. + type: number + upstream_url: + description: The upstream URL for the model endpoint. + type: string + format: uri + type: + type: string + enum: + - xai + required: + - type + GCPModelConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for a model hosted on Google Cloud Project. + type: object + properties: + api_endpoint: + description: The custom API endpoint for the Gemini model. + type: string + location_id: + description: The Google Cloud location ID for the model endpoint. + type: string + project_id: + description: The Google Cloud project ID for the model endpoint. + type: string + required: + - api_endpoint + - location_id + - project_id + AIGatewayAllowACL: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + allow: + description: 'List of Consumer Groups Names, or Authenticated Groups Names that are permitted access.' + type: array + items: + type: string + example: + allow: + - consumer-group-1 + required: + - allow + AIGatewayDenyACL: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + deny: + description: 'List of Consumer Groups Names, or Authenticated Groups Names that are denied access.' + type: array + items: + type: string + example: + deny: + - consumer-group-1 + required: + - deny + AIGatewayACLS: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Access control rules. Configure exactly one of `allow` or `deny`. + oneOf: + - $ref: '#/components/schemas/AIGatewayAllowACL' + - $ref: '#/components/schemas/AIGatewayDenyACL' + AIGatewayAgentAccess: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Access control configuration for an agent. + type: object + properties: + acls: + $ref: '#/components/schemas/AIGatewayACLS' + additionalProperties: false + AIGatewayModelAccess: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Access control configuration for a model. + type: object + properties: + acls: + $ref: '#/components/schemas/AIGatewayACLS' + identity_providers: + description: | + List of identity providers for granting access to the model. + At most 1 identity provider of each identity provider type can be referenced. + type: array + items: + $ref: '#/components/schemas/AIGatewayIdentityProviderReference' + additionalProperties: false + AIGatewayMCPACLs: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Access control rules for MCP resources. Configure `allow`, `deny`, or both. + type: object + properties: + allow: + description: List of consumer groups that are permitted access. + type: array + items: + type: string + deny: + description: List of consumer groups that are denied access. + type: array + items: + type: string + example: + allow: + - gold-partner + deny: + - bronze-partner + AIGatewayPolicyReferences: + description: List of policy references. + type: array + items: + type: string + description: Reference to a policy instance by name. + KonnectConfigStoreVault: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this vault instance, used as a stable human-readable reference. + This value is immutable after creation. + The name is used to load the right Vault configuration and implementation when referencing secrets with the other entities. + example: my-awesome-vault + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + description: + description: The description of the Vault. + type: string + example: This vault is used to retrieve redis database access credentials + maxLength: 1024 + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + type: + type: string + enum: + - konnect + config: + type: object + additionalProperties: false + properties: + config_store_id: + description: | + The ID of the Konnect Config Store that contains the secrets. + type: string + example: 77426bee-2bca-4005-81af-284868fd3038 + required: + - config_store_id + required: + - name + - type + - config + EnvironmentVariableVault: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this vault instance, used as a stable human-readable reference. + This value is immutable after creation. + The name is used to load the right Vault configuration and implementation when referencing secrets with the other entities. + example: my-awesome-vault + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + description: + description: The description of the Vault. + type: string + example: This vault is used to retrieve redis database access credentials + maxLength: 1024 + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + type: + type: string + enum: + - env + config: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + base64_decode: + description: | + Decode all secrets in this vault as base64. Useful for binary data. + If some of the secrets in the vault are not base64-encoded, an error will occur when using them. + We recommend creating a separate vault for base64 secrets. + type: boolean + prefix: + description: | + The prefix for the environment variable that the value will be stored in. + type: string + example: MY_SECRET_ + title: EnvironmentVariableVaultConfig + required: + - name + - type + - config + AwsSecretsManagerVault: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this vault instance, used as a stable human-readable reference. + This value is immutable after creation. + The name is used to load the right Vault configuration and implementation when referencing secrets with the other entities. + example: my-awesome-vault + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + description: + description: The description of the Vault. + type: string + example: This vault is used to retrieve redis database access credentials + maxLength: 1024 + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + type: + type: string + enum: + - aws + config: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + base64_decode: + description: | + Decode all secrets in this vault as base64. Useful for binary data. + If some of the secrets in the vault are not base64-encoded, an error will occur when using them. + We recommend creating a separate vault for base64 secrets. + type: boolean + neg_ttl: + description: | + Time-to-live (in seconds) for caching failed secret lookups. + A value of 0 disables negative caching. Kong will retry fetching the secret after neg_ttl expires. + type: integer + default: 0 + resurrect_ttl: + description: | + Time (in seconds) that secrets remain in use after expiration (config.ttl ends). + Useful if the vault is unreachable or the secret is deleted but not yet replaced. + Kong continues to retry for resurrect_ttl seconds before giving up. + The default is ~3 years to support uninterrupted service during outages. + type: integer + default: 100000000 + ttl: + description: | + Time-to-live (in seconds) for a cached secret. A value of 0 disables rotation. + For non-zero values, use a minimum of 60 seconds. + type: integer + default: 0 + assume_role_arn: + description: | + The ARN of the role to assume when retrieving secrets from AWS Secrets Manager. + type: string + endpoint_url: + description: | + The endpoint URL of the AWS Secrets Manager service. + If not specified, the default is https://secretsmanager.{region}.amazonaws.com. + You can override this by specifying a complete URL including the http/https scheme. + type: string + region: + description: The AWS region where your vault is located. + type: string + example: us-east-1 + role_session_name: + description: The session name used when assuming a role. + type: string + default: KongVault + sts_endpoint_url: + description: | + A custom STS endpoint URL used for IAM role assumption. + Overrides the default https://sts.amazonaws.com or regional variant https://sts..amazonaws.com. + Include the full http/https scheme. Only specify this if using a private VPC endpoint for STS. + type: string + required: + - role_session_name + title: AwsSecretsManagerVaultConfig + required: + - name + - type + - config + GoogleSecretManagerVault: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this vault instance, used as a stable human-readable reference. + This value is immutable after creation. + The name is used to load the right Vault configuration and implementation when referencing secrets with the other entities. + example: my-awesome-vault + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + description: + description: The description of the Vault. + type: string + example: This vault is used to retrieve redis database access credentials + maxLength: 1024 + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + type: + type: string + enum: + - gcp + config: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + base64_decode: + description: | + Decode all secrets in this vault as base64. Useful for binary data. + If some of the secrets in the vault are not base64-encoded, an error will occur when using them. + We recommend creating a separate vault for base64 secrets. + type: boolean + neg_ttl: + description: | + Time-to-live (in seconds) for caching failed secret lookups. + A value of 0 disables negative caching. Kong will retry fetching the secret after neg_ttl expires. + type: integer + default: 0 + resurrect_ttl: + description: | + Time (in seconds) that secrets remain in use after expiration (config.ttl ends). + Useful if the vault is unreachable or the secret is deleted but not yet replaced. + Kong continues to retry for resurrect_ttl seconds before giving up. + The default is ~3 years to support uninterrupted service during outages. + type: integer + default: 100000000 + ttl: + description: | + Time-to-live (in seconds) for a cached secret. A value of 0 disables rotation. + For non-zero values, use a minimum of 60 seconds. + type: integer + default: 0 + project_id: + description: | + The project ID from your Google API Console. + You can find it by visiting your Google API Console and selecting “Manage all projects” in the projects list. + type: string + required: + - project_id + title: GoogleSecretManagerVaultConfig + required: + - name + - type + - config + AzureKeyVault: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this vault instance, used as a stable human-readable reference. + This value is immutable after creation. + The name is used to load the right Vault configuration and implementation when referencing secrets with the other entities. + example: my-awesome-vault + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + description: + description: The description of the Vault. + type: string + example: This vault is used to retrieve redis database access credentials + maxLength: 1024 + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + type: + type: string + enum: + - azure + config: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + base64_decode: + description: | + Decode all secrets in this vault as base64. Useful for binary data. + If some of the secrets in the vault are not base64-encoded, an error will occur when using them. + We recommend creating a separate vault for base64 secrets. + type: boolean + neg_ttl: + description: | + Time-to-live (in seconds) for caching failed secret lookups. + A value of 0 disables negative caching. Kong will retry fetching the secret after neg_ttl expires. + type: integer + default: 0 + resurrect_ttl: + description: | + Time (in seconds) that secrets remain in use after expiration (config.ttl ends). + Useful if the vault is unreachable or the secret is deleted but not yet replaced. + Kong continues to retry for resurrect_ttl seconds before giving up. + The default is ~3 years to support uninterrupted service during outages. + type: integer + default: 100000000 + ttl: + description: | + Time-to-live (in seconds) for a cached secret. A value of 0 disables rotation. + For non-zero values, use a minimum of 60 seconds. + type: integer + default: 0 + credentials_prefix: + description: | + The prefix for the credentials stored in the Azure Key Vault. + type: string + default: AZURE + vault_uri: + description: | + The URI from which the vault is reachable. + This value can be found in your Azure Key Vault Dashboard under the Vault URI entry. + type: string + location: + description: | + Each Azure geography includes one or more regions + that meet specific data residency and compliance requirements. + type: string + client_id: + description: | + The client ID for your registered application. + You can find this in the Azure Dashboard under App Registrations. + type: string + tenant_id: + description: | + The DirectoryId and TenantId are the same: both refer to the GUID representing your Azure Active Directory tenant. + Microsoft documentation and products may use either term depending on context. + type: string + type: + type: string + default: secrets + enum: + - secrets + required: + - vault_uri + - location + - type + title: AzureKeyVaultConfig + required: + - name + - type + - config + ConjurVault: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this vault instance, used as a stable human-readable reference. + This value is immutable after creation. + The name is used to load the right Vault configuration and implementation when referencing secrets with the other entities. + example: my-awesome-vault + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + description: + description: The description of the Vault. + type: string + example: This vault is used to retrieve redis database access credentials + maxLength: 1024 + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + type: + type: string + enum: + - conjur + config: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + base64_decode: + description: | + Decode all secrets in this vault as base64. Useful for binary data. + If some of the secrets in the vault are not base64-encoded, an error will occur when using them. + We recommend creating a separate vault for base64 secrets. + type: boolean + neg_ttl: + description: | + Time-to-live (in seconds) for caching failed secret lookups. + A value of 0 disables negative caching. Kong will retry fetching the secret after neg_ttl expires. + type: integer + default: 0 + resurrect_ttl: + description: | + Time (in seconds) that secrets remain in use after expiration (config.ttl ends). + Useful if the vault is unreachable or the secret is deleted but not yet replaced. + Kong continues to retry for resurrect_ttl seconds before giving up. + The default is ~3 years to support uninterrupted service during outages. + type: integer + default: 100000000 + ttl: + description: | + Time-to-live (in seconds) for a cached secret. A value of 0 disables rotation. + For non-zero values, use a minimum of 60 seconds. + type: integer + default: 0 + account: + description: | + The CyberArk Secrets Manager organization account name. + type: string + api_key: + description: | + The API key of the workload identity. + type: string + writeOnly: true + endpoint_url: + description: | + The CyberArk Secrets Manager backend URL to connect with. Accepts http or https protocols. + type: string + login: + description: | + The login name of the workload identity. + type: string + required: + - endpoint_url + - login + - account + title: ConjurVaultConfig + required: + - name + - type + - config + HashiCorpVault: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this vault instance, used as a stable human-readable reference. + This value is immutable after creation. + The name is used to load the right Vault configuration and implementation when referencing secrets with the other entities. + example: my-awesome-vault + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + description: + description: The description of the Vault. + type: string + example: This vault is used to retrieve redis database access credentials + maxLength: 1024 + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + type: + type: string + enum: + - hcv + config: + $ref: '#/components/schemas/HashiCorpVaultConfig' + required: + - name + - type + - config + HashiCorpVaultConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an AI Gateway Vault. + discriminator: + propertyName: auth_method + mapping: + token: '#/components/schemas/HashiCorpVaultTokenConfig' + cert: '#/components/schemas/HashiCorpVaultCertConfig' + jwt: '#/components/schemas/HashiCorpVaultOauth2Config' + approle: '#/components/schemas/HashiCorpVaultAppRoleConfig' + kubernetes: '#/components/schemas/HashiCorpVaultKubernetesConfig' + gcp_iam: '#/components/schemas/HashiCorpVaultGcpIAMConfig' + gcp_gce: '#/components/schemas/HashiCorpVaultGcpGCEConfig' + aws_ec2: '#/components/schemas/HashiCorpVaultAwsEC2Config' + aws_iam: '#/components/schemas/HashiCorpVaultAwsIAMConfig' + azure: '#/components/schemas/HashiCorpVaultAzureConfig' + oneOf: + - $ref: '#/components/schemas/HashiCorpVaultTokenConfig' + - $ref: '#/components/schemas/HashiCorpVaultCertConfig' + - $ref: '#/components/schemas/HashiCorpVaultOauth2Config' + - $ref: '#/components/schemas/HashiCorpVaultAppRoleConfig' + - $ref: '#/components/schemas/HashiCorpVaultKubernetesConfig' + - $ref: '#/components/schemas/HashiCorpVaultGcpIAMConfig' + - $ref: '#/components/schemas/HashiCorpVaultGcpGCEConfig' + - $ref: '#/components/schemas/HashiCorpVaultAwsEC2Config' + - $ref: '#/components/schemas/HashiCorpVaultAwsIAMConfig' + - $ref: '#/components/schemas/HashiCorpVaultAzureConfig' + title: HashiCorpVaultConfig + HashiCorpVaultTokenConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + base64_decode: + description: | + Decode all secrets in this vault as base64. Useful for binary data. + If some of the secrets in the vault are not base64-encoded, an error will occur when using them. + We recommend creating a separate vault for base64 secrets. + type: boolean + neg_ttl: + description: | + Time-to-live (in seconds) for caching failed secret lookups. + A value of 0 disables negative caching. Kong will retry fetching the secret after neg_ttl expires. + type: integer + default: 0 + resurrect_ttl: + description: | + Time (in seconds) that secrets remain in use after expiration (config.ttl ends). + Useful if the vault is unreachable or the secret is deleted but not yet replaced. + Kong continues to retry for resurrect_ttl seconds before giving up. + The default is ~3 years to support uninterrupted service during outages. + type: integer + default: 100000000 + ttl: + description: | + Time-to-live (in seconds) for a cached secret. A value of 0 disables rotation. + For non-zero values, use a minimum of 60 seconds. + type: integer + default: 0 + host: + description: The hostname of your HashiCorp vault. + type: string + port: + description: The port number of your HashiCorp vault. + type: integer + mount: + description: The mount point. + type: string + default: secret + kv: + description: The secrets engine version. + type: string + default: v1 + enum: + - v1 + - v2 + protocol: + description: The protocol to connect with. + type: string + default: https + enum: + - http + - https + ssl_verify: + description: Whether to verify the TLS certificate of the vault when connecting. + type: boolean + default: true + namespace: + description: Namespace for the Vault. Vault Enterprise requires a namespace to connect successfully. + type: string + auth_method: + type: string + enum: + - token + token: + description: The token string to be used for authentication. + type: string + writeOnly: true + required: + - host + - port + - mount + - auth_method + HashiCorpVaultCertConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + base64_decode: + description: | + Decode all secrets in this vault as base64. Useful for binary data. + If some of the secrets in the vault are not base64-encoded, an error will occur when using them. + We recommend creating a separate vault for base64 secrets. + type: boolean + neg_ttl: + description: | + Time-to-live (in seconds) for caching failed secret lookups. + A value of 0 disables negative caching. Kong will retry fetching the secret after neg_ttl expires. + type: integer + default: 0 + resurrect_ttl: + description: | + Time (in seconds) that secrets remain in use after expiration (config.ttl ends). + Useful if the vault is unreachable or the secret is deleted but not yet replaced. + Kong continues to retry for resurrect_ttl seconds before giving up. + The default is ~3 years to support uninterrupted service during outages. + type: integer + default: 100000000 + ttl: + description: | + Time-to-live (in seconds) for a cached secret. A value of 0 disables rotation. + For non-zero values, use a minimum of 60 seconds. + type: integer + default: 0 + host: + description: The hostname of your HashiCorp vault. + type: string + port: + description: The port number of your HashiCorp vault. + type: integer + mount: + description: The mount point. + type: string + default: secret + kv: + description: The secrets engine version. + type: string + default: v1 + enum: + - v1 + - v2 + protocol: + description: The protocol to connect with. + type: string + default: https + enum: + - http + - https + ssl_verify: + description: Whether to verify the TLS certificate of the vault when connecting. + type: boolean + default: true + namespace: + description: Namespace for the Vault. Vault Enterprise requires a namespace to connect successfully. + type: string + auth_method: + type: string + enum: + - cert + cert: + description: The client certificate. + type: string + example: | + -----BEGIN CERTIFICATE----- + certificate-content + -----END CERTIFICATE----- + key: + description: The key for the client certificate. + type: string + example: | + -----BEGIN PRIVATE KEY----- + private-key-content + -----END PRIVATE KEY----- + writeOnly: true + role_name: + description: The trusted certificate role name. + type: string + required: + - host + - port + - mount + - auth_method + - cert + HashiCorpVaultOauth2Config: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + base64_decode: + description: | + Decode all secrets in this vault as base64. Useful for binary data. + If some of the secrets in the vault are not base64-encoded, an error will occur when using them. + We recommend creating a separate vault for base64 secrets. + type: boolean + neg_ttl: + description: | + Time-to-live (in seconds) for caching failed secret lookups. + A value of 0 disables negative caching. Kong will retry fetching the secret after neg_ttl expires. + type: integer + default: 0 + resurrect_ttl: + description: | + Time (in seconds) that secrets remain in use after expiration (config.ttl ends). + Useful if the vault is unreachable or the secret is deleted but not yet replaced. + Kong continues to retry for resurrect_ttl seconds before giving up. + The default is ~3 years to support uninterrupted service during outages. + type: integer + default: 100000000 + ttl: + description: | + Time-to-live (in seconds) for a cached secret. A value of 0 disables rotation. + For non-zero values, use a minimum of 60 seconds. + type: integer + default: 0 + host: + description: The hostname of your HashiCorp vault. + type: string + port: + description: The port number of your HashiCorp vault. + type: integer + mount: + description: The mount point. + type: string + default: secret + kv: + description: The secrets engine version. + type: string + default: v1 + enum: + - v1 + - v2 + protocol: + description: The protocol to connect with. + type: string + default: https + enum: + - http + - https + ssl_verify: + description: Whether to verify the TLS certificate of the vault when connecting. + type: boolean + default: true + namespace: + description: Namespace for the Vault. Vault Enterprise requires a namespace to connect successfully. + type: string + auth_method: + type: string + enum: + - jwt + role: + description: | + The configured role name in HashiCorp Vault for JWT auth. + When creating the role in HashiCorp Vault, make sure that the `role_type` is `jwt` + and the `token_policies` have permissions to read the secrets. + type: string + example: demo + token_endpoint: + description: The OAuth2 token endpoint for Hashicorp Vault's OAuth2 auth method. + type: string + client_id: + description: The OAuth2 client ID. + type: string + client_secret: + description: The OAuth2 client secret. + type: string + writeOnly: true + audiences: + description: Comma-separated list of OAuth2 audiences. + type: string + required: + - host + - port + - mount + - auth_method + - role + - token_endpoint + - client_id + HashiCorpVaultAppRoleConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + base64_decode: + description: | + Decode all secrets in this vault as base64. Useful for binary data. + If some of the secrets in the vault are not base64-encoded, an error will occur when using them. + We recommend creating a separate vault for base64 secrets. + type: boolean + neg_ttl: + description: | + Time-to-live (in seconds) for caching failed secret lookups. + A value of 0 disables negative caching. Kong will retry fetching the secret after neg_ttl expires. + type: integer + default: 0 + resurrect_ttl: + description: | + Time (in seconds) that secrets remain in use after expiration (config.ttl ends). + Useful if the vault is unreachable or the secret is deleted but not yet replaced. + Kong continues to retry for resurrect_ttl seconds before giving up. + The default is ~3 years to support uninterrupted service during outages. + type: integer + default: 100000000 + ttl: + description: | + Time-to-live (in seconds) for a cached secret. A value of 0 disables rotation. + For non-zero values, use a minimum of 60 seconds. + type: integer + default: 0 + host: + description: The hostname of your HashiCorp vault. + type: string + port: + description: The port number of your HashiCorp vault. + type: integer + mount: + description: The mount point. + type: string + default: secret + kv: + description: The secrets engine version. + type: string + default: v1 + enum: + - v1 + - v2 + protocol: + description: The protocol to connect with. + type: string + default: https + enum: + - http + - https + ssl_verify: + description: Whether to verify the TLS certificate of the vault when connecting. + type: boolean + default: true + namespace: + description: Namespace for the Vault. Vault Enterprise requires a namespace to connect successfully. + type: string + auth_method: + type: string + enum: + - approle + path: + description: | + Path for enabling the AppRole auth method. Single leading/trailing slashes are trimmed. + type: string + default: approle + response_wrapping: + description: | + Whether the secret ID is a response-wrapping token. + When true, Kong unwraps the token to get the actual secret ID. + Note: tokens can only be unwrapped once; distribute them individually to Kong nodes. + type: boolean + default: false + role_id: + description: | + Specifies the AppRole role ID in HashiCorp Vault. + Either `role_id` or `secret_id_file` must be set. + type: string + secret_id: + description: Defines the AppRole’s secret ID in HashiCorp Vault. + type: string + secret_id_file: + description: | + Path to a file containing the AppRole secret ID. + Either `role_id` or `secret_id_file` must be set. + type: string + required: + - host + - port + - mount + - auth_method + HashiCorpVaultKubernetesConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + base64_decode: + description: | + Decode all secrets in this vault as base64. Useful for binary data. + If some of the secrets in the vault are not base64-encoded, an error will occur when using them. + We recommend creating a separate vault for base64 secrets. + type: boolean + neg_ttl: + description: | + Time-to-live (in seconds) for caching failed secret lookups. + A value of 0 disables negative caching. Kong will retry fetching the secret after neg_ttl expires. + type: integer + default: 0 + resurrect_ttl: + description: | + Time (in seconds) that secrets remain in use after expiration (config.ttl ends). + Useful if the vault is unreachable or the secret is deleted but not yet replaced. + Kong continues to retry for resurrect_ttl seconds before giving up. + The default is ~3 years to support uninterrupted service during outages. + type: integer + default: 100000000 + ttl: + description: | + Time-to-live (in seconds) for a cached secret. A value of 0 disables rotation. + For non-zero values, use a minimum of 60 seconds. + type: integer + default: 0 + host: + description: The hostname of your HashiCorp vault. + type: string + port: + description: The port number of your HashiCorp vault. + type: integer + mount: + description: The mount point. + type: string + default: secret + kv: + description: The secrets engine version. + type: string + default: v1 + enum: + - v1 + - v2 + protocol: + description: The protocol to connect with. + type: string + default: https + enum: + - http + - https + ssl_verify: + description: Whether to verify the TLS certificate of the vault when connecting. + type: boolean + default: true + namespace: + description: Namespace for the Vault. Vault Enterprise requires a namespace to connect successfully. + type: string + auth_method: + type: string + enum: + - kubernetes + role: + description: | + Role assigned to the Kubernetes service account. + type: string + path: + description: | + Path for enabling the Kubernetes auth method. Single leading/trailing slashes are trimmed. + type: string + default: kubernetes + api_token_file: + description: | + Path to the Kubernetes service account token file. + type: string + default: /run/secrets/kubernetes.io/serviceaccount/token + required: + - host + - port + - mount + - auth_method + HashiCorpVaultGcpIAMConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + base64_decode: + description: | + Decode all secrets in this vault as base64. Useful for binary data. + If some of the secrets in the vault are not base64-encoded, an error will occur when using them. + We recommend creating a separate vault for base64 secrets. + type: boolean + neg_ttl: + description: | + Time-to-live (in seconds) for caching failed secret lookups. + A value of 0 disables negative caching. Kong will retry fetching the secret after neg_ttl expires. + type: integer + default: 0 + resurrect_ttl: + description: | + Time (in seconds) that secrets remain in use after expiration (config.ttl ends). + Useful if the vault is unreachable or the secret is deleted but not yet replaced. + Kong continues to retry for resurrect_ttl seconds before giving up. + The default is ~3 years to support uninterrupted service during outages. + type: integer + default: 100000000 + ttl: + description: | + Time-to-live (in seconds) for a cached secret. A value of 0 disables rotation. + For non-zero values, use a minimum of 60 seconds. + type: integer + default: 0 + host: + description: The hostname of your HashiCorp vault. + type: string + port: + description: The port number of your HashiCorp vault. + type: integer + mount: + description: The mount point. + type: string + default: secret + kv: + description: The secrets engine version. + type: string + default: v1 + enum: + - v1 + - v2 + protocol: + description: The protocol to connect with. + type: string + default: https + enum: + - http + - https + ssl_verify: + description: Whether to verify the TLS certificate of the vault when connecting. + type: boolean + default: true + namespace: + description: Namespace for the Vault. Vault Enterprise requires a namespace to connect successfully. + type: string + auth_method: + type: string + enum: + - gcp_iam + role: + description: The role to use for GCP IAM auth. + type: string + service_account: + description: The GCP service account for GCE auth. + type: string + jwt_exp: + description: The JWT expiration time in seconds for GCP auth (0-900) + type: integer + maximum: 900 + minimum: 0 + required: + - host + - port + - mount + - auth_method + - role + - service_account + - jwt_exp + HashiCorpVaultGcpGCEConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + base64_decode: + description: | + Decode all secrets in this vault as base64. Useful for binary data. + If some of the secrets in the vault are not base64-encoded, an error will occur when using them. + We recommend creating a separate vault for base64 secrets. + type: boolean + neg_ttl: + description: | + Time-to-live (in seconds) for caching failed secret lookups. + A value of 0 disables negative caching. Kong will retry fetching the secret after neg_ttl expires. + type: integer + default: 0 + resurrect_ttl: + description: | + Time (in seconds) that secrets remain in use after expiration (config.ttl ends). + Useful if the vault is unreachable or the secret is deleted but not yet replaced. + Kong continues to retry for resurrect_ttl seconds before giving up. + The default is ~3 years to support uninterrupted service during outages. + type: integer + default: 100000000 + ttl: + description: | + Time-to-live (in seconds) for a cached secret. A value of 0 disables rotation. + For non-zero values, use a minimum of 60 seconds. + type: integer + default: 0 + host: + description: The hostname of your HashiCorp vault. + type: string + port: + description: The port number of your HashiCorp vault. + type: integer + mount: + description: The mount point. + type: string + default: secret + kv: + description: The secrets engine version. + type: string + default: v1 + enum: + - v1 + - v2 + protocol: + description: The protocol to connect with. + type: string + default: https + enum: + - http + - https + ssl_verify: + description: Whether to verify the TLS certificate of the vault when connecting. + type: boolean + default: true + namespace: + description: Namespace for the Vault. Vault Enterprise requires a namespace to connect successfully. + type: string + auth_method: + type: string + enum: + - gcp_gce + role: + description: The role to use for GCP GCE auth. + type: string + login_path: + description: The login path for GCP auth in HashiCorp Vault. + type: string + default: /v1/auth/gcp/login + required: + - host + - port + - mount + - auth_method + - role + HashiCorpVaultAzureConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + base64_decode: + description: | + Decode all secrets in this vault as base64. Useful for binary data. + If some of the secrets in the vault are not base64-encoded, an error will occur when using them. + We recommend creating a separate vault for base64 secrets. + type: boolean + neg_ttl: + description: | + Time-to-live (in seconds) for caching failed secret lookups. + A value of 0 disables negative caching. Kong will retry fetching the secret after neg_ttl expires. + type: integer + default: 0 + resurrect_ttl: + description: | + Time (in seconds) that secrets remain in use after expiration (config.ttl ends). + Useful if the vault is unreachable or the secret is deleted but not yet replaced. + Kong continues to retry for resurrect_ttl seconds before giving up. + The default is ~3 years to support uninterrupted service during outages. + type: integer + default: 100000000 + ttl: + description: | + Time-to-live (in seconds) for a cached secret. A value of 0 disables rotation. + For non-zero values, use a minimum of 60 seconds. + type: integer + default: 0 + host: + description: The hostname of your HashiCorp vault. + type: string + port: + description: The port number of your HashiCorp vault. + type: integer + mount: + description: The mount point. + type: string + default: secret + kv: + description: The secrets engine version. + type: string + default: v1 + enum: + - v1 + - v2 + protocol: + description: The protocol to connect with. + type: string + default: https + enum: + - http + - https + ssl_verify: + description: Whether to verify the TLS certificate of the vault when connecting. + type: boolean + default: true + namespace: + description: Namespace for the Vault. Vault Enterprise requires a namespace to connect successfully. + type: string + auth_method: + type: string + enum: + - azure + role: + description: The role to use for Azure auth. + type: string + login_path: + description: The login path for Azure auth in HashiCorp Vault + type: string + default: /v1/auth/azure/login + required: + - host + - port + - mount + - auth_method + - role + HashiCorpVaultAwsEC2Config: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + base64_decode: + description: | + Decode all secrets in this vault as base64. Useful for binary data. + If some of the secrets in the vault are not base64-encoded, an error will occur when using them. + We recommend creating a separate vault for base64 secrets. + type: boolean + neg_ttl: + description: | + Time-to-live (in seconds) for caching failed secret lookups. + A value of 0 disables negative caching. Kong will retry fetching the secret after neg_ttl expires. + type: integer + default: 0 + resurrect_ttl: + description: | + Time (in seconds) that secrets remain in use after expiration (config.ttl ends). + Useful if the vault is unreachable or the secret is deleted but not yet replaced. + Kong continues to retry for resurrect_ttl seconds before giving up. + The default is ~3 years to support uninterrupted service during outages. + type: integer + default: 100000000 + ttl: + description: | + Time-to-live (in seconds) for a cached secret. A value of 0 disables rotation. + For non-zero values, use a minimum of 60 seconds. + type: integer + default: 0 + host: + description: The hostname of your HashiCorp vault. + type: string + port: + description: The port number of your HashiCorp vault. + type: integer + mount: + description: The mount point. + type: string + default: secret + kv: + description: The secrets engine version. + type: string + default: v1 + enum: + - v1 + - v2 + protocol: + description: The protocol to connect with. + type: string + default: https + enum: + - http + - https + ssl_verify: + description: Whether to verify the TLS certificate of the vault when connecting. + type: boolean + default: true + namespace: + description: Namespace for the Vault. Vault Enterprise requires a namespace to connect successfully. + type: string + auth_method: + type: string + enum: + - aws_ec2 + role: + description: The role to use for AWS EC2 auth. + type: string + nonce: + description: The nonce for AWS EC2 auth. + type: string + login_path: + description: The login path for AWS auth in HashiCorp Vault. + type: string + default: /v1/auth/aws/login + required: + - host + - port + - mount + - auth_method + - role + - nonce + HashiCorpVaultAwsIAMConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + base64_decode: + description: | + Decode all secrets in this vault as base64. Useful for binary data. + If some of the secrets in the vault are not base64-encoded, an error will occur when using them. + We recommend creating a separate vault for base64 secrets. + type: boolean + neg_ttl: + description: | + Time-to-live (in seconds) for caching failed secret lookups. + A value of 0 disables negative caching. Kong will retry fetching the secret after neg_ttl expires. + type: integer + default: 0 + resurrect_ttl: + description: | + Time (in seconds) that secrets remain in use after expiration (config.ttl ends). + Useful if the vault is unreachable or the secret is deleted but not yet replaced. + Kong continues to retry for resurrect_ttl seconds before giving up. + The default is ~3 years to support uninterrupted service during outages. + type: integer + default: 100000000 + ttl: + description: | + Time-to-live (in seconds) for a cached secret. A value of 0 disables rotation. + For non-zero values, use a minimum of 60 seconds. + type: integer + default: 0 + host: + description: The hostname of your HashiCorp vault. + type: string + port: + description: The port number of your HashiCorp vault. + type: integer + mount: + description: The mount point. + type: string + default: secret + kv: + description: The secrets engine version. + type: string + default: v1 + enum: + - v1 + - v2 + protocol: + description: The protocol to connect with. + type: string + default: https + enum: + - http + - https + ssl_verify: + description: Whether to verify the TLS certificate of the vault when connecting. + type: boolean + default: true + namespace: + description: Namespace for the Vault. Vault Enterprise requires a namespace to connect successfully. + type: string + auth_method: + type: string + enum: + - aws_iam + role: + description: The role to use for AWS IAM auth. + type: string + region: + description: The AWS region for auth. + type: string + login_path: + description: The login path for AWS auth in HashiCorp Vault. + type: string + default: /v1/auth/aws/login + access_key_id: + description: | + The AWS access key ID for IAM auth. If not provided, the default credentials provider chain is used. + If set, `secret_access_key` must also be set. + type: string + secret_access_key: + description: | + The AWS secret access key for IAM auth. If not provided, the default credentials provider chain is used. + If set, `access_key_id` must also be set. + type: string + writeOnly: true + sts_endpoint_url: + description: | + The AWS STS endpoint URL used by Kong Gateway when signing the GetCallerIdentity request for AWS IAM authentication. + If not provided, defaults to the standard STS endpoint for the specified region. + This setting only affects the STS endpoint that Kong Gateway itself contacts - + it does not influence which STS endpoint HashiCorp Vault uses on its side. + type: string + assume_role_arn: + description: | + The ARN of the role to assume for AWS IAM authentication. + If set, `role_session_name` must also be set. + type: string + role_session_name: + description: | + The session name to use when assuming a role for AWS IAM authentication. + If set, `assume_role_arn` must also be set. + type: string + required: + - host + - port + - mount + - auth_method + - role + - region + CreateAIGatewayVaultRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an AI Gateway Vault. + discriminator: + propertyName: type + mapping: + konnect: '#/components/schemas/KonnectConfigStoreVault' + env: '#/components/schemas/EnvironmentVariableVault' + aws: '#/components/schemas/AwsSecretsManagerVault' + gcp: '#/components/schemas/GoogleSecretManagerVault' + azure: '#/components/schemas/AzureKeyVault' + conjur: '#/components/schemas/ConjurVault' + hcv: '#/components/schemas/HashiCorpVault' + oneOf: + - $ref: '#/components/schemas/KonnectConfigStoreVault' + - $ref: '#/components/schemas/EnvironmentVariableVault' + - $ref: '#/components/schemas/AwsSecretsManagerVault' + - $ref: '#/components/schemas/GoogleSecretManagerVault' + - $ref: '#/components/schemas/AzureKeyVault' + - $ref: '#/components/schemas/ConjurVault' + - $ref: '#/components/schemas/HashiCorpVault' + UpdateAIGatewayVaultRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an AI Gateway Vault. + discriminator: + propertyName: type + mapping: + konnect: '#/components/schemas/KonnectConfigStoreVault' + env: '#/components/schemas/EnvironmentVariableVault' + aws: '#/components/schemas/AwsSecretsManagerVault' + gcp: '#/components/schemas/GoogleSecretManagerVault' + azure: '#/components/schemas/AzureKeyVault' + conjur: '#/components/schemas/ConjurVault' + hcv: '#/components/schemas/HashiCorpVault' + oneOf: + - $ref: '#/components/schemas/KonnectConfigStoreVault' + - $ref: '#/components/schemas/EnvironmentVariableVault' + - $ref: '#/components/schemas/AwsSecretsManagerVault' + - $ref: '#/components/schemas/GoogleSecretManagerVault' + - $ref: '#/components/schemas/AzureKeyVault' + - $ref: '#/components/schemas/ConjurVault' + - $ref: '#/components/schemas/HashiCorpVault' + AIGatewayVault: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an AI Gateway Vault. + type: object + properties: + id: + $ref: '#/components/schemas/UUID' + created_at: + $ref: '#/components/schemas/CreatedAt' + updated_at: + $ref: '#/components/schemas/UpdatedAt' + discriminator: + propertyName: type + mapping: + konnect: '#/components/schemas/KonnectConfigStoreVault' + env: '#/components/schemas/EnvironmentVariableVault' + aws: '#/components/schemas/AwsSecretsManagerVault' + gcp: '#/components/schemas/GoogleSecretManagerVault' + azure: '#/components/schemas/AzureKeyVault' + conjur: '#/components/schemas/ConjurVault' + hcv: '#/components/schemas/HashiCorpVault' + oneOf: + - $ref: '#/components/schemas/KonnectConfigStoreVault' + - $ref: '#/components/schemas/EnvironmentVariableVault' + - $ref: '#/components/schemas/AwsSecretsManagerVault' + - $ref: '#/components/schemas/GoogleSecretManagerVault' + - $ref: '#/components/schemas/AzureKeyVault' + - $ref: '#/components/schemas/ConjurVault' + - $ref: '#/components/schemas/HashiCorpVault' + required: + - id + - created_at + - updated_at + CreateAIGatewayPolicyRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + display_name: + description: The display name for this policy instance. + type: string + example: My Cool AI PII Sanitizer Policy + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this policy instance, used as a stable human-readable reference. This value is immutable after creation. + example: ai-pii-sanitizer-1234 + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + type: + description: | + The type of the Policy. This is equivalent to the Kong 3 plugin name. + Some examples are: 'ai-sanitizer', 'ai-prompt-guard', and 'openid-connect'. + Note: Plugins have been renamed to Policies in Kong AI Gateway. Policy types and configuration documentation can be found in the [Developer Docs](https://developer.konghq.com/plugins/). + type: string + example: ai-sanitizer + enabled: + description: Whether the policy is enabled. + type: boolean + example: true + default: true + global: + description: Whether the policy is globally applied to all resources. + type: boolean + example: false + default: false + config: + description: | + Configuration for the policy. This is equivalent to the Kong 3 plugin configuration. + Note: Plugins have been renamed to Policies in Kong AI Gateway. Policy types and configuration documentation can be found in the [Developer Docs](https://developer.konghq.com/plugins/). + type: object + example: + anonymize: + - phone + - creditcard + stop_on_error: true + additionalProperties: true + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + additionalProperties: false + required: + - display_name + - name + - type + - config + UpdateAIGatewayPolicyRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + display_name: + description: The display name for this policy instance. + type: string + example: My Cool AI PII Sanitizer Policy + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this policy instance, used as a stable human-readable reference. This value is immutable after creation. + example: ai-pii-sanitizer-1234 + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + type: + description: | + The type of the Policy. This is equivalent to the Kong 3 plugin name. + Some examples are: 'ai-sanitizer', 'ai-prompt-guard', and 'openid-connect'. + Note: Plugins have been renamed to Policies in Kong AI Gateway. Policy types and configuration documentation can be found in the [Developer Docs](https://developer.konghq.com/plugins/). + type: string + example: ai-sanitizer + enabled: + description: Whether the policy is enabled. + type: boolean + example: true + default: true + global: + description: Whether the policy is globally applied to all resources. + type: boolean + example: false + default: false + config: + description: | + Configuration for the policy. This is equivalent to the Kong 3 plugin configuration. + Note: Plugins have been renamed to Policies in Kong AI Gateway. Policy types and configuration documentation can be found in the [Developer Docs](https://developer.konghq.com/plugins/). + type: object + example: + anonymize: + - phone + - creditcard + stop_on_error: true + additionalProperties: true + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + additionalProperties: false + required: + - display_name + - name + - type + - config + AIGatewayPolicy: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + display_name: + description: The display name for this policy instance. + type: string + example: My Cool AI PII Sanitizer Policy + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this policy instance, used as a stable human-readable reference. This value is immutable after creation. + example: ai-pii-sanitizer-1234 + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + type: + description: | + The type of the Policy. This is equivalent to the Kong 3 plugin name. + Some examples are: 'ai-sanitizer', 'ai-prompt-guard', and 'openid-connect'. + Note: Plugins have been renamed to Policies in Kong AI Gateway. Policy types and configuration documentation can be found in the [Developer Docs](https://developer.konghq.com/plugins/). + type: string + example: ai-sanitizer + enabled: + description: Whether the policy is enabled. + type: boolean + example: true + default: true + global: + description: Whether the policy is globally applied to all resources. + type: boolean + example: false + default: false + config: + description: | + Configuration for the policy. This is equivalent to the Kong 3 plugin configuration. + Note: Plugins have been renamed to Policies in Kong AI Gateway. Policy types and configuration documentation can be found in the [Developer Docs](https://developer.konghq.com/plugins/). + type: object + example: + anonymize: + - phone + - creditcard + stop_on_error: true + additionalProperties: true + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + id: + $ref: '#/components/schemas/UUID' + created_at: + $ref: '#/components/schemas/CreatedAt' + updated_at: + $ref: '#/components/schemas/UpdatedAt' + additionalProperties: false + required: + - display_name + - name + - type + - config + - id + - created_at + - updated_at + CreateAIGatewayAgentRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + display_name: + description: The display name for this agent. + type: string + example: Kong Air Flight Booking Agent + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this agent, used as a stable human-readable reference. This value is immutable after creation. + example: kongair-flight-booking-agent + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + enabled: + description: Whether the Agent is enabled. + type: boolean + example: true + default: true + type: + description: The type of the agent. + type: string + example: a2a + enum: + - a2a + - http + policies: + $ref: '#/components/schemas/AIGatewayPolicyReferences' + access: + $ref: '#/components/schemas/AIGatewayAgentAccess' + config: + description: Configuration for the agent. The structure varies depending on the agent type. + type: object + additionalProperties: false + properties: + url: + description: | + Helper field to set protocol, host, port and path of the upstream A2A Agent using a URL. + This is the same as a Kong Gateway Service URL: ${scheme}://${host}:${port}/${path} + type: string + format: uri + example: 'https://booking-agent.internal.kongair.com' + route: + $ref: '#/components/schemas/AIGatewayRouteConfig' + max_request_body_size: + description: Maximum size of request body to parse. Set to 0 for unlimited. + type: integer + default: 8388608 + logging: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for AI Gateway logging. + type: object + additionalProperties: false + properties: + payloads: + type: boolean + default: false + statistics: + type: boolean + default: true + max_payload_size: + description: Maximum size in bytes for logged request/response payloads. Payloads exceeding this size will be truncated. + type: integer + example: 524288 + default: 1048576 + required: + - url + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + additionalProperties: true + required: + - display_name + - name + - type + - config + UpdateAIGatewayAgentRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + display_name: + description: The display name for this agent. + type: string + example: Kong Air Flight Booking Agent + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this agent, used as a stable human-readable reference. This value is immutable after creation. + example: kongair-flight-booking-agent + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + enabled: + description: Whether the Agent is enabled. + type: boolean + example: true + default: true + type: + description: The type of the agent. + type: string + example: a2a + enum: + - a2a + - http + policies: + $ref: '#/components/schemas/AIGatewayPolicyReferences' + access: + $ref: '#/components/schemas/AIGatewayAgentAccess' + config: + description: Configuration for the agent. The structure varies depending on the agent type. + type: object + additionalProperties: false + properties: + url: + description: | + Helper field to set protocol, host, port and path of the upstream A2A Agent using a URL. + This is the same as a Kong Gateway Service URL: ${scheme}://${host}:${port}/${path} + type: string + format: uri + example: 'https://booking-agent.internal.kongair.com' + route: + $ref: '#/components/schemas/AIGatewayRouteConfig' + max_request_body_size: + description: Maximum size of request body to parse. Set to 0 for unlimited. + type: integer + default: 8388608 + logging: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for AI Gateway logging. + type: object + additionalProperties: false + properties: + payloads: + type: boolean + default: false + statistics: + type: boolean + default: true + max_payload_size: + description: Maximum size in bytes for logged request/response payloads. Payloads exceeding this size will be truncated. + type: integer + example: 524288 + default: 1048576 + required: + - url + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + additionalProperties: true + required: + - display_name + - name + - type + - config + AIGatewayAgent: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + display_name: + description: The display name for this agent. + type: string + example: Kong Air Flight Booking Agent + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this agent, used as a stable human-readable reference. This value is immutable after creation. + example: kongair-flight-booking-agent + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + enabled: + description: Whether the Agent is enabled. + type: boolean + example: true + default: true + type: + description: The type of the agent. + type: string + example: a2a + enum: + - a2a + - http + policies: + $ref: '#/components/schemas/AIGatewayPolicyReferences' + access: + $ref: '#/components/schemas/AIGatewayAgentAccess' + config: + description: Configuration for the agent. The structure varies depending on the agent type. + type: object + additionalProperties: false + properties: + url: + description: | + Helper field to set protocol, host, port and path of the upstream A2A Agent using a URL. + This is the same as a Kong Gateway Service URL: ${scheme}://${host}:${port}/${path} + type: string + format: uri + example: 'https://booking-agent.internal.kongair.com' + route: + $ref: '#/components/schemas/AIGatewayRouteConfig' + max_request_body_size: + description: Maximum size of request body to parse. Set to 0 for unlimited. + type: integer + default: 8388608 + logging: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for AI Gateway logging. + type: object + additionalProperties: false + properties: + payloads: + type: boolean + default: false + statistics: + type: boolean + default: true + max_payload_size: + description: Maximum size in bytes for logged request/response payloads. Payloads exceeding this size will be truncated. + type: integer + example: 524288 + default: 1048576 + required: + - url + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + id: + $ref: '#/components/schemas/UUID' + created_at: + $ref: '#/components/schemas/CreatedAt' + updated_at: + $ref: '#/components/schemas/UpdatedAt' + additionalProperties: true + required: + - display_name + - name + - type + - config + - id + - created_at + - updated_at + CreateAIGatewayConsumerRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an AI Gateway Consumer. + type: object + properties: + display_name: + description: The display name for this consumer instance. + type: string + example: Greg's Dev Consumer + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this consumer, used as a stable human-readable reference. This value is immutable after creation. + example: gregs-dev-consumer + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + type: + description: The type of the consumer. + type: string + enum: + - api-key + - oauth + custom_id: + description: Identifier for mapping the consumer when using OAuth authentication. + type: string + example: dev-users + policies: + $ref: '#/components/schemas/AIGatewayPolicyReferences' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + additionalProperties: true + required: + - display_name + - name + - type + UpdateAIGatewayConsumerRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an AI Gateway Consumer. + type: object + properties: + display_name: + description: The display name for this consumer instance. + type: string + example: Greg's Dev Consumer + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this consumer, used as a stable human-readable reference. This value is immutable after creation. + example: gregs-dev-consumer + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + type: + description: The type of the consumer. + type: string + enum: + - api-key + - oauth + custom_id: + description: Identifier for mapping the consumer when using OAuth authentication. + type: string + example: dev-users + policies: + $ref: '#/components/schemas/AIGatewayPolicyReferences' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + additionalProperties: true + required: + - display_name + - name + - type + AIGatewayConsumer: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an AI Gateway Consumer. + type: object + properties: + display_name: + description: The display name for this consumer instance. + type: string + example: Greg's Dev Consumer + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this consumer, used as a stable human-readable reference. This value is immutable after creation. + example: gregs-dev-consumer + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + type: + description: The type of the consumer. + type: string + enum: + - api-key + - oauth + custom_id: + description: Identifier for mapping the consumer when using OAuth authentication. + type: string + example: dev-users + policies: + $ref: '#/components/schemas/AIGatewayPolicyReferences' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + id: + $ref: '#/components/schemas/UUID' + created_at: + $ref: '#/components/schemas/CreatedAt' + updated_at: + $ref: '#/components/schemas/UpdatedAt' + additionalProperties: true + required: + - display_name + - name + - type + - id + - created_at + - updated_at + AIGatewayConsumerCredential: + type: object + properties: + display_name: + description: The display name for this credential instance. + type: string + example: Greg's Dev Key + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this credential, used as a stable human-readable reference. This value is immutable after creation. + example: gregs-dev-key + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + type: + type: string + example: api-key + enum: + - api-key + ttl: + description: The API Key's time-to-live in seconds. A value of 0 means the API Key never expires. + type: integer + example: 86400 + default: 0 + id: + $ref: '#/components/schemas/UUID' + created_at: + $ref: '#/components/schemas/CreatedAt' + updated_at: + $ref: '#/components/schemas/UpdatedAt' + additionalProperties: false + required: + - display_name + - name + - type + - id + - created_at + - updated_at + AIGatewayConsumerCredentialWithKey: + type: object + properties: + display_name: + description: The display name for this credential instance. + type: string + example: Greg's Dev Key + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this credential, used as a stable human-readable reference. This value is immutable after creation. + example: gregs-dev-key + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + type: + type: string + example: api-key + enum: + - api-key + ttl: + description: The API Key's time-to-live in seconds. A value of 0 means the API Key never expires. + type: integer + example: 86400 + default: 0 + api_key: + description: 'The API Key value. If not provided, then the key will be auto generated by the server and returned in the response.' + type: string + example: sk-387788hd3xnej + id: + $ref: '#/components/schemas/UUID' + created_at: + $ref: '#/components/schemas/CreatedAt' + updated_at: + $ref: '#/components/schemas/UpdatedAt' + additionalProperties: false + required: + - display_name + - name + - id + - created_at + - updated_at + - type + - api_key + CreateAIGatewayConsumerCredentialRequest: + type: object + properties: + display_name: + description: The display name for this credential instance. + type: string + example: Greg's Dev Key + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this credential, used as a stable human-readable reference. This value is immutable after creation. + example: gregs-dev-key + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + type: + type: string + example: api-key + enum: + - api-key + ttl: + description: The API Key's time-to-live in seconds. A value of 0 means the API Key never expires. + type: integer + example: 86400 + default: 0 + api_key: + description: 'The API Key value. If not provided, then the key will be auto generated by the server and returned in the response.' + type: string + example: sk-387788hd3xnej + writeOnly: true + additionalProperties: false + required: + - display_name + - name + - type + CreateAIGatewayConsumerGroupRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + display_name: + description: The display name for this consumer group instance. + type: string + example: Dev Users Group + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this consumer group, used as a stable human-readable reference. This value is immutable after creation. + example: dev-users + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + policies: + $ref: '#/components/schemas/AIGatewayPolicyReferences' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + additionalProperties: true + required: + - display_name + - name + UpdateAIGatewayConsumerGroupRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + display_name: + description: The display name for this consumer group instance. + type: string + example: Dev Users Group + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this consumer group, used as a stable human-readable reference. This value is immutable after creation. + example: dev-users + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + policies: + $ref: '#/components/schemas/AIGatewayPolicyReferences' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + additionalProperties: true + required: + - display_name + - name + AddAIGatewayConsumerToGroupRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + properties: + consumer: + description: The ID or name of the consumer to add to the group. + type: string + example: cf4c7e60-11db-49dd-b300-7c7e5f0f7e6b + additionalProperties: false + required: + - consumer + AIGatewayConsumerGroup: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + display_name: + description: The display name for this consumer group instance. + type: string + example: Dev Users Group + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this consumer group, used as a stable human-readable reference. This value is immutable after creation. + example: dev-users + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + policies: + $ref: '#/components/schemas/AIGatewayPolicyReferences' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + id: + $ref: '#/components/schemas/UUID' + created_at: + $ref: '#/components/schemas/CreatedAt' + updated_at: + $ref: '#/components/schemas/UpdatedAt' + additionalProperties: true + required: + - display_name + - name + - id + - created_at + - updated_at + AIGatewayMCPServerConversionOnly: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + type: + type: string + enum: + - conversion-only + config: + $ref: '#/components/schemas/AIGatewayMCPServerWithUpstreamNoProxyConfigNoServerConfig' + tools: + description: List of tools exposed by this MCP Server. + type: array + items: + $ref: '#/components/schemas/AIGatewayMCPConversionTool' + display_name: + description: The display name for the MCP Server. + type: string + example: Kong Air Flights + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this MCP server, used as a stable human-readable reference. This value is immutable after creation. + example: kongair-flights + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + enabled: + description: Whether the MCP Server is enabled. + type: boolean + example: true + default: true + policies: + $ref: '#/components/schemas/AIGatewayPolicyReferences' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + additionalProperties: true + required: + - type + - config + - display_name + - name + AIGatewayMCPServerConversionListener: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + type: + type: string + enum: + - conversion-listener + config: + $ref: '#/components/schemas/AIGatewayMCPServerWithUpstreamNoProxyConfig' + tools: + description: List of tools exposed by this MCP Server. + type: array + items: + $ref: '#/components/schemas/AIGatewayMCPConversionTool' + access: + $ref: '#/components/schemas/AIGatewayMCPServerBaseACLProperties' + display_name: + description: The display name for the MCP Server. + type: string + example: Kong Air Flights + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this MCP server, used as a stable human-readable reference. This value is immutable after creation. + example: kongair-flights + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + enabled: + description: Whether the MCP Server is enabled. + type: boolean + example: true + default: true + policies: + $ref: '#/components/schemas/AIGatewayPolicyReferences' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + additionalProperties: true + required: + - type + - config + - display_name + - name + AIGatewayMCPServerListener: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + type: + type: string + enum: + - listener + config: + $ref: '#/components/schemas/AIGatewayMCPServerNoUpstreamConfig' + tools: + description: List of tools exposed by this MCP Server. + type: array + items: + $ref: '#/components/schemas/AIGatewayMCPToolBase' + access: + $ref: '#/components/schemas/AIGatewayMCPServerBaseACLProperties' + display_name: + description: The display name for the MCP Server. + type: string + example: Kong Air Flights + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this MCP server, used as a stable human-readable reference. This value is immutable after creation. + example: kongair-flights + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + enabled: + description: Whether the MCP Server is enabled. + type: boolean + example: true + default: true + policies: + $ref: '#/components/schemas/AIGatewayPolicyReferences' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + additionalProperties: true + required: + - type + - config + - display_name + - name + AIGatewayMCPServerPassthroughListener: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + type: + type: string + enum: + - passthrough-listener + config: + $ref: '#/components/schemas/AIGatewayMCPServerWithUpstreamConfig' + tools: + description: List of tools exposed by this MCP Server. + type: array + items: + $ref: '#/components/schemas/AIGatewayMCPToolBase' + access: + $ref: '#/components/schemas/AIGatewayMCPServerBaseACLProperties' + display_name: + description: The display name for the MCP Server. + type: string + example: Kong Air Flights + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this MCP server, used as a stable human-readable reference. This value is immutable after creation. + example: kongair-flights + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + enabled: + description: Whether the MCP Server is enabled. + type: boolean + example: true + default: true + policies: + $ref: '#/components/schemas/AIGatewayPolicyReferences' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + additionalProperties: true + required: + - type + - config + - display_name + - name + AIGatewayMCPServerUpstreamServer: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + type: + type: string + enum: + - upstream-server + config: + $ref: '#/components/schemas/AIGatewayMCPServerUpstreamServerConfig' + tools: + description: List of tools exposed by this MCP Server. + type: array + items: + $ref: '#/components/schemas/AIGatewayMCPUpstreamTool' + access: + $ref: '#/components/schemas/AIGatewayMCPServerBaseACLProperties' + display_name: + description: The display name for the MCP Server. + type: string + example: Kong Air Flights + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this MCP server, used as a stable human-readable reference. This value is immutable after creation. + example: kongair-flights + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + enabled: + description: Whether the MCP Server is enabled. + type: boolean + example: true + default: true + policies: + $ref: '#/components/schemas/AIGatewayPolicyReferences' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + additionalProperties: true + required: + - type + - config + - display_name + - name + AIGatewayMCPConversionTool: + description: A tool exposed by an MCP Server in `conversion-only` or `conversion-listener` mode. + type: object + properties: + access: + type: object + additionalProperties: false + properties: + acls: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Access control rules for allowing or denying consumer groups access to this tool. + When configured, these will override the default access control rules defined on the MCP Server. + allOf: + - $ref: '#/components/schemas/AIGatewayMCPACLs' + annotations: + $ref: '#/components/schemas/AIGatewayMCPToolAnnotations' + description: + description: A description of what the tool does. + type: string + example: Search for available flights + headers: + $ref: '#/components/schemas/AIGatewayMCPToolHeaders' + host: + description: 'The host of the exported API, which must match the route''s hosts. It should be the route''s host. By default, Kong will extract the host from API configuration. If the configured host is wildcard, this field is required.' + type: string + name: + description: 'Tool identifier. In passthrough-listener mode, used to match remote MCP Server tools for ACL enforcement. In other modes, it is also used as the tool name (overrides annotations.title if present).' + type: string + method: + description: 'For conversion-only and conversion-listener modes, the method of the exported API, which must match the route''s methods.' + type: string + enum: + - DELETE + - GET + - PATCH + - POST + - PUT + path: + description: 'The path of the exported API, which must match the route''s paths. Path not starting with ''/'' are treated as relative path and the route path will be added as the prefix. By default, Kong will extract the path from API configuration.' + type: string + query: + $ref: '#/components/schemas/AIGatewayMCPToolQuery' + request_body: + $ref: '#/components/schemas/AIGatewayMCPToolRequestBody' + responses: + $ref: '#/components/schemas/AIGatewayMCPToolResponses' + scheme: + description: 'The scheme of the exported API. By default, Kong will extract the scheme from API configuration. If the configured scheme is not expected, this field can be used to override it.' + type: string + enum: + - http + - https + parameters: + $ref: '#/components/schemas/AIGatewayMCPToolParameters' + additionalProperties: false + required: + - name + - description + - method + AIGatewayMCPServerBaseACLProperties: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + default: + acl_attribute_type: consumer + discriminator: + propertyName: acl_attribute_type + mapping: + consumer: '#/components/schemas/AIGatewayMCPServerBaseACLPropertiesConsumer' + oauth_access_token: '#/components/schemas/AIGatewayMCPServerBaseACLPropertiesOauth' + oneOf: + - $ref: '#/components/schemas/AIGatewayMCPServerBaseACLPropertiesConsumer' + - $ref: '#/components/schemas/AIGatewayMCPServerBaseACLPropertiesOauth' + AIGatewayMCPServerBaseACLPropertiesConsumer: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + acl_attribute_type: + description: The type of attributes that ACL is evaluated with. + type: string + default: consumer + enum: + - consumer + acls: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Access control rules for allowing or denying consumer groups. + allOf: + - $ref: '#/components/schemas/AIGatewayMCPACLs' + default_tool_acls: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Default access control rules for allowing or denying consumer groups to tools. + allOf: + - $ref: '#/components/schemas/AIGatewayMCPACLs' + required: + - acl_attribute_type + title: AIGatewayMCPServerBaseACLPropertiesConsumer + AIGatewayMCPServerBaseACLPropertiesOauth: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + acl_attribute_type: + description: The type of attributes that ACL is evaluated with. + type: string + enum: + - oauth_access_token + access_token_claim_field: + description: | + The claim in the OAuth2 access token to use as the subject for ACL evaluation when `acl_attribute_type` is set to `oauth_access_token`. + Nested claim can be fetched by using a jq filter starts with dot, e.g., “.user.email”: https://jqlang.org/manual/#object-identifier-index + type: string + acls: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Access control rules for allowing or denying consumer groups. + allOf: + - $ref: '#/components/schemas/AIGatewayMCPACLs' + default_tool_acls: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Default access control rules for allowing or denying consumer groups to tools. + allOf: + - $ref: '#/components/schemas/AIGatewayMCPACLs' + required: + - acl_attribute_type + - access_token_claim_field + title: AIGatewayMCPServerBaseACLPropertiesOauth + AIGatewayRedisAWSAuthentication: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + AWS specific configs for connecting to a Cloud Provider's redis instance. + type: object + properties: + type: + type: string + enum: + - aws + access_key_id: + description: | + AWS Access Key ID to be used for authentication. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + assume_role_arn: + description: | + The ARN of the IAM role to assume for generating ElastiCache IAM authentication tokens. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + cache_name: + description: | + The name of the AWS Elasticache cluster. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + is_serverless: + description: This flag specifies whether the cluster is serverless. + type: boolean + default: true + region: + description: | + The region of the AWS ElastiCache cluster. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + role_session_name: + description: | + The session name for the temporary credentials when assuming the IAM role. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + secret_access_key: + description: | + AWS Secret Access Key. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + writeOnly: true + x-referenceable: true + additionalProperties: false + required: + - type + title: AIGatewayRedisAWSAuthentication + AIGatewayRedisAzureAuthentication: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Azure specific configs for connecting to a Cloud Provider's redis instance. + type: object + properties: + type: + type: string + enum: + - azure + client_id: + description: | + Azure Client ID. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + client_secret: + description: | + Azure Client Secret. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + writeOnly: true + x-referenceable: true + tenant_id: + description: | + Azure Tenant ID. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + additionalProperties: false + required: + - type + title: AIGatewayRedisAzureAuthentication + AIGatewayRedisGCPAuthentication: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + GCP specific configs for connecting to a Cloud Provider's redis instance. + type: object + properties: + type: + type: string + enum: + - gcp + service_account_json: + description: | + GCP Service Account JSON. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + writeOnly: true + x-referenceable: true + additionalProperties: false + required: + - type + title: AIGatewayRedisGCPAuthentication + AIGatewayRedisCloudConfiguration: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Config for connecting to a Cloud Provider's Redis instance. + type: object + properties: + cloud_authentication: + description: Auth related config for connecting to a Cloud Provider's Redis instance. + discriminator: + propertyName: type + mapping: + aws: '#/components/schemas/AIGatewayRedisAWSAuthentication' + azure: '#/components/schemas/AIGatewayRedisAzureAuthentication' + gcp: '#/components/schemas/AIGatewayRedisGCPAuthentication' + oneOf: + - $ref: '#/components/schemas/AIGatewayRedisAWSAuthentication' + - $ref: '#/components/schemas/AIGatewayRedisAzureAuthentication' + - $ref: '#/components/schemas/AIGatewayRedisGCPAuthentication' + cluster: + description: Cluster configuration for the Redis connection. + type: object + additionalProperties: false + properties: + max_redirections: + description: Maximum retry attempts for redirection. + type: integer + default: 5 + nodes: + description: Cluster addresses to use for Redis connections when the `redis` strategy is defined. Defining this field implies using a Redis Cluster. The minimum length of the array is 1 element. + type: array + items: + type: object + properties: + ip: + description: 'A string representing a host name, such as example.com.' + type: string + default: 127.0.0.1 + port: + description: 'An integer representing a port number between 0 and 65535, inclusive.' + type: integer + default: 6379 + maximum: 65535 + minimum: 0 + minItems: 1 + connect_timeout: + description: An integer representing a timeout in milliseconds. Must be between 0 and 2^31-2. + type: integer + default: 2000 + maximum: 2147483646 + minimum: 0 + connection_is_proxied: + description: 'If the connection to Redis is proxied (e.g. Envoy), set it `true`. Set the `host` and `port` to point to the proxy address.' + type: boolean + default: false + database: + description: Database to use for the Redis connection when using the `redis` strategy + type: integer + default: 0 + host: + description: | + A string representing a host name, such as example.com. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + default: 127.0.0.1 + x-referenceable: true + keepalive: + description: Keepalive configuration for the Redis connection. + type: object + additionalProperties: false + properties: + backlog: + description: 'Limits the total number of opened connections for a pool. If the connection pool is full, connection queues above the limit go into the backlog queue. If the backlog queue is full, subsequent connect operations fail and return `nil`. Queued operations (subject to set timeouts) resume once the number of connections in the pool is less than `pool_size`. If latency is high or throughput is low, try increasing this value. Empirically, this value is larger than `pool_size`.' + type: integer + maximum: 2147483646 + minimum: 0 + pool_size: + description: 'The size limit for every cosocket connection pool associated with every remote server, per worker process. If neither `pool_size` nor `backlog` is specified, no pool is created. If `pool_size` isn''t specified but `backlog` is specified, then the pool uses the default value. Try to increase (e.g. 512) this value if latency is high or throughput is low.' + type: integer + default: 256 + maximum: 2147483646 + minimum: 1 + password: + description: | + Password to use for Redis connections. If undefined, no AUTH commands are sent to Redis. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + port: + description: | + An integer representing a port number between 0 and 65535, inclusive. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + oneOf: + - type: integer + default: 6379 + maximum: 65535 + minimum: 0 + example: 6379 + - type: string + example: '{vault://hcv/redis/port}' + x-go-type: types.Referenceable + x-go-type-import: + path: github.com/kong/koko/internal/server/public/openapi/controlplanesconfig/types + name: types + x-referenceable: true + read_timeout: + description: An integer representing a timeout in milliseconds. Must be between 0 and 2^31-2. + type: integer + default: 2000 + maximum: 2147483646 + minimum: 0 + send_timeout: + description: An integer representing a timeout in milliseconds. Must be between 0 and 2^31-2. + type: integer + default: 2000 + maximum: 2147483646 + minimum: 0 + sentinel: + description: Configuration for Redis Sentinel. + type: object + additionalProperties: false + properties: + master: + description: Sentinel master to use for Redis connections. Defining this value implies using Redis Sentinel. + type: string + nodes: + description: Sentinel node addresses to use for Redis connections when the `redis` strategy is defined. Defining this field implies using a Redis Sentinel. The minimum length of the array is 1 element. + type: array + items: + type: object + properties: + host: + description: 'A string representing a host name, such as example.com.' + type: string + default: 127.0.0.1 + port: + description: 'An integer representing a port number between 0 and 65535, inclusive.' + type: integer + default: 6379 + maximum: 65535 + minimum: 0 + minItems: 1 + password: + description: | + Sentinel password to authenticate with a Redis Sentinel instance. If undefined, no AUTH commands are sent to Redis Sentinels. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + role: + description: Sentinel role to use for Redis connections when the `redis` strategy is defined. Defining this value implies using Redis Sentinel. + type: string + enum: + - any + - master + - slave + username: + description: | + Sentinel username to authenticate with a Redis Sentinel instance. If undefined, ACL authentication won't be performed. This requires Redis v6.2.0+. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + server_name: + description: | + A string representing an SNI (server name indication) value for TLS. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + ssl: + description: 'If set to true, uses SSL to connect to Redis.' + type: boolean + default: true + ssl_verify: + description: 'If set to true, verifies the validity of the server SSL certificate. If setting this parameter, also configure `lua_ssl_trusted_certificate` in `kong.conf` to specify the CA (or server) certificate used by your Redis server. You may also need to configure `lua_ssl_verify_depth` accordingly.' + type: boolean + default: true + username: + description: | + Username to use for Redis connections. If undefined, ACL authentication won't be performed. This requires Redis v6.0.0+. To be compatible with Redis v5.x.y, you can set it to `default`. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + additionalProperties: false + AIGatewayProxyConfig: + description: HTTP/HTTPS proxy configuration for outbound requests to the upstream AI provider. + type: object + properties: + http_proxy: + description: HTTP proxy server to route plaintext outbound requests through. + type: object + additionalProperties: false + properties: + host: + description: 'A string representing a host name, such as example.com.' + type: string + port: + description: 'An integer representing a port number between 0 and 65535, inclusive.' + type: integer + maximum: 65535 + minimum: 0 + https_proxy: + description: HTTPS proxy server to route TLS outbound requests through. + type: object + additionalProperties: false + properties: + host: + description: 'A string representing a host name, such as example.com.' + type: string + port: + description: 'An integer representing a port number between 0 and 65535, inclusive.' + type: integer + maximum: 65535 + minimum: 0 + proxy_scheme: + description: The proxy scheme to use when connecting to the proxy server. + type: string + default: http + enum: + - http + auth: + description: Credentials used to authenticate to the proxy server. + type: object + additionalProperties: false + properties: + username: + description: | + The username to use for proxy authentication. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + password: + description: | + The password to use for proxy authentication. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + writeOnly: true + x-referenceable: true + no_proxy: + description: Comma-separated list of hosts that should not be proxied. + type: string + additionalProperties: false + AIGatewayRouteConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an AI Gateway route. + type: object + properties: + headers: + description: 'One or more lists of values indexed by header name that will cause this route to match if present in the request. The `Host` header cannot be used with this attribute: hosts should be specified using the `hosts` attribute. When `headers` contains only one value and that value starts with the special prefix `~*`, the value is interpreted as a regular expression.' + type: object + example: + version: + - v1 + - v2 + additionalProperties: true + hosts: + description: A list of domain names that match this route. Note that the hosts value is case sensitive. + type: array + items: + type: string + example: foo.example.com + https_redirect_status_code: + description: 'The status code Kong responds with when all properties of a route match except the protocol i.e. if the protocol of the request is `HTTP` instead of `HTTPS`. `Location` header is injected by Kong if the field is set to 301, 302, 307 or 308. Note: This config applies only if the route is configured to only accept the `https` protocol.' + type: integer + default: 426 + methods: + description: A list of HTTP methods that match this route. + type: array + items: + type: string + paths: + description: A list of paths that match this route. + type: array + items: + type: string + preserve_host: + description: 'When matching a route via one of the `hosts` domain names, use the request `Host` header in the upstream request headers. If set to `false`, the upstream `Host` header will be that of the service''s `host`.' + type: boolean + default: false + protocols: + description: 'An array of the protocols this route should allow. See the [route Object](#route-object) section for a list of accepted protocols. When set to only `https`, HTTP requests are answered with an upgrade error. When set to only `http`, HTTPS requests are answered with an error.' + type: array + items: + type: string + default: + - http + - https + regex_priority: + description: 'A number used to choose which route resolves a given request when several routes match it using regexes simultaneously. When two routes match the path and have the same `regex_priority`, the older one (lowest `created_at`) is used. Note that the priority for non-regex routes is different (longer non-regex routes are matched before shorter ones).' + type: integer + default: 0 + request_buffering: + description: 'Whether to enable request body buffering or not. With HTTP 1.1, it may make sense to turn this off on services that receive data with chunked transfer encoding.' + type: boolean + default: true + response_buffering: + description: 'Whether to enable response body buffering or not. With HTTP 1.1, it may make sense to turn this off on services that send data with chunked transfer encoding.' + type: boolean + default: true + strip_path: + description: 'When matching a route via one of the `paths`, strip the matching prefix from the upstream request URL.' + type: boolean + default: true + tags: + description: An optional set of strings associated with the route for grouping and filtering. + type: array + items: + type: string + additionalProperties: false + AIGatewayMCPServerNoUpstreamConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Routing, logging, and server configuration for the MCP Server. + type: object + properties: + route: + $ref: '#/components/schemas/AIGatewayRouteConfig' + logging: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for AI Gateway logging. + type: object + additionalProperties: false + properties: + payloads: + type: boolean + default: false + statistics: + type: boolean + default: true + audits: + type: boolean + default: false + max_request_body_size: + description: Maximum size of request body to parse. Set to 0 for unlimited. + type: integer + default: 8388608 + server: + $ref: '#/components/schemas/AIGatewayMCPServerServerConfigBase' + additionalProperties: false + AIGatewayMCPServerWithUpstreamNoProxyConfigNoServerConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Routing, logging, and request body size limits for the MCP Server. + type: object + properties: + route: + $ref: '#/components/schemas/AIGatewayRouteConfig' + logging: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for AI Gateway logging. + type: object + additionalProperties: false + properties: + payloads: + type: boolean + default: false + statistics: + type: boolean + default: true + audits: + type: boolean + default: false + max_request_body_size: + description: Maximum size of request body to parse. Set to 0 for unlimited. + type: integer + default: 8388608 + url: + description: | + Helper field to set protocol, host, port and path of the upstream service using a URL. + This is the same as a Kong Gateway Service URL: ${scheme}://${host}:${port}/${path} + type: string + format: uri + example: 'https://mcp.internal.kongair.com' + additionalProperties: false + required: + - url + AIGatewayMCPServerWithUpstreamNoProxyConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Routing, logging, and server configuration for the MCP Server. + type: object + properties: + route: + $ref: '#/components/schemas/AIGatewayRouteConfig' + logging: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for AI Gateway logging. + type: object + additionalProperties: false + properties: + payloads: + type: boolean + default: false + statistics: + type: boolean + default: true + audits: + type: boolean + default: false + max_request_body_size: + description: Maximum size of request body to parse. Set to 0 for unlimited. + type: integer + default: 8388608 + server: + $ref: '#/components/schemas/AIGatewayMCPServerServerConfigBase' + url: + description: | + Helper field to set protocol, host, port and path of the upstream service using a URL. + This is the same as a Kong Gateway Service URL: ${scheme}://${host}:${port}/${path} + type: string + format: uri + example: 'https://mcp.internal.kongair.com' + additionalProperties: false + required: + - url + AIGatewayMCPServerWithUpstreamConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Routing, logging, and server configuration for the MCP Server. + type: object + properties: + route: + $ref: '#/components/schemas/AIGatewayRouteConfig' + logging: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for AI Gateway logging. + type: object + additionalProperties: false + properties: + payloads: + type: boolean + default: false + statistics: + type: boolean + default: true + audits: + type: boolean + default: false + max_request_body_size: + description: Maximum size of request body to parse. Set to 0 for unlimited. + type: integer + default: 8388608 + server: + $ref: '#/components/schemas/AIGatewayMCPServerServerConfigBase' + url: + description: | + Helper field to set protocol, host, port and path of the upstream service using a URL. + This is the same as a Kong Gateway Service URL: ${scheme}://${host}:${port}/${path} + type: string + format: uri + example: 'https://mcp.internal.kongair.com' + proxy: + $ref: '#/components/schemas/AIGatewayProxyConfig' + additionalProperties: false + required: + - url + AIGatewayMCPServerUpstreamServerConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Routing, logging, and server configuration for the MCP Server. + type: object + properties: + route: + $ref: '#/components/schemas/AIGatewayRouteConfig' + logging: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for AI Gateway logging. + type: object + additionalProperties: false + properties: + payloads: + type: boolean + default: false + statistics: + type: boolean + default: true + audits: + type: boolean + default: false + max_request_body_size: + description: Maximum size of request body to parse. Set to 0 for unlimited. + type: integer + default: 8388608 + server: + $ref: '#/components/schemas/AIGatewayMCPServerUpstreamServerServerConfig' + url: + description: | + Helper field to set protocol, host, port and path of the upstream service using a URL. + This is the same as a Kong Gateway Service URL: ${scheme}://${host}:${port}/${path} + type: string + format: uri + example: 'https://mcp.internal.kongair.com' + tools_cache_ttl_seconds: + description: | + The time-to-live (TTL) for the upstream tools cache in seconds. Set to `0` to refresh on + every client call. + type: integer + minimum: 0 + additionalProperties: false + required: + - url + - tools_cache_ttl_seconds + AIGatewayMCPServerServerConfigBase: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Server-side configuration for the MCP Server. + type: object + properties: + forward_client_headers: + description: Whether to forward the client request headers to the upstream server when calling the tools. + type: boolean + default: true + session: + description: | + Enable managed session when Kong responds as MCP server in listener, conversion-listener, or upstream-server modes. + This doesn't affect the passthrough-listener mode as the state in that mode is maintained by the upstream MCP servers. + type: object + additionalProperties: false + properties: + client: + description: The configuration for client-side session storage. + type: object + additionalProperties: false + properties: + secrets: + description: | + The secrets that are used in session encryption. Required when the strategy is 'client'. + The first secret is used for encryption, while all secrets are used for decryption to support key rotation. + type: array + items: + type: string + minLength: 8 + x-referenceable: true + description: | + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + minItems: 1 + managed: + description: 'If enabled, Kong will maintain managed sessions with the MCP server.' + type: boolean + default: true + redis: + $ref: '#/components/schemas/AIGatewayRedisCloudConfiguration' + session_ttl: + description: The time-to-live (TTL) for each session in seconds. + type: integer + default: 86400 + strategy: + description: 'The strategy for the session. If the value is ''client'', the session is encrypted into MCP session id assigned to the client. If the value is not ''client'', the session is stored in the configured database.' + type: string + enum: + - client + - redis + tag: + description: The tag of the MCP server. This is used to filter the exported MCP tools. The field should contain exactly one tag. + type: string + timeout: + description: The timeout for calling the tools in milliseconds. + type: integer + default: 10000 + additionalProperties: false + AIGatewayMCPServerUpstreamServerServerConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Server-side configuration specific to `upstream-server` mode. + type: object + properties: + forward_client_headers: + description: Whether to forward the client request headers to the upstream server when calling the tools. + type: boolean + default: true + session: + description: | + Enable managed session when Kong responds as MCP server in listener, conversion-listener, or upstream-server modes. + This doesn't affect the passthrough-listener mode as the state in that mode is maintained by the upstream MCP servers. + type: object + additionalProperties: false + properties: + client: + description: The configuration for client-side session storage. + type: object + additionalProperties: false + properties: + secrets: + description: | + The secrets that are used in session encryption. Required when the strategy is 'client'. + The first secret is used for encryption, while all secrets are used for decryption to support key rotation. + type: array + items: + type: string + minLength: 8 + x-referenceable: true + description: | + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + minItems: 1 + managed: + description: 'If enabled, Kong will maintain managed sessions with the MCP server.' + type: boolean + default: true + redis: + $ref: '#/components/schemas/AIGatewayRedisCloudConfiguration' + session_ttl: + description: The time-to-live (TTL) for each session in seconds. + type: integer + default: 86400 + strategy: + description: 'The strategy for the session. If the value is ''client'', the session is encrypted into MCP session id assigned to the client. If the value is not ''client'', the session is stored in the configured database.' + type: string + enum: + - client + - redis + tag: + description: The tag of the MCP server. This is used to filter the exported MCP tools. The field should contain exactly one tag. + type: string + timeout: + description: The timeout for calling the tools in milliseconds. + type: integer + default: 10000 + preserve_upstream_tool_names: + description: | + If enabled, the original upstream tool names are preserved as-is when Kong acts as an MCP server. + If disabled (`false`), the service name will be prepended to the MCP tool names to avoid name + collisions when multiple services are used. + type: boolean + default: false + tools_list_auth: + $ref: '#/components/schemas/AIGatewayMCPServerUpstreamServerServerToolAuthConfig' + additionalProperties: false + AIGatewayMCPServerUpstreamServerServerToolAuthConfig: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an Upstream Server's MCP Server Tools' Authentication. + discriminator: + propertyName: type + mapping: + jwt: '#/components/schemas/AIGatewayMCPServerUpstreamServerToolOauth2ConfigJwt' + credentials: '#/components/schemas/AIGatewayMCPServerUpstreamServerToolOauth2ConfigCredentials' + oneOf: + - $ref: '#/components/schemas/AIGatewayMCPServerUpstreamServerToolOauth2ConfigJwt' + - $ref: '#/components/schemas/AIGatewayMCPServerUpstreamServerToolOauth2ConfigCredentials' + AIGatewayMCPServerUpstreamServerToolOauth2ConfigJwt: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + scope: + description: | + The scopes for the OAuth 2.0 client-credentials. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + access_token_header: + description: | + Header name used to send the fetched access token to the upstream MCP server. The value should + include the header name and the token prefix if needed. + type: string + default: Authorization + id_token_header: + description: | + Header name used to send the fetched ID token to the upstream MCP server. The value should + include the header name and the token prefix if needed. Leave empty to omit the ID token + when fetching the tools list. + type: string + type: + type: string + enum: + - jwt + additionalProperties: false + required: + - type + AIGatewayMCPServerUpstreamServerToolOauth2ConfigCredentials: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + scope: + description: | + The scopes for the OAuth 2.0 client-credentials. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + access_token_header: + description: | + Header name used to send the fetched access token to the upstream MCP server. The value should + include the header name and the token prefix if needed. + type: string + default: Authorization + id_token_header: + description: | + Header name used to send the fetched ID token to the upstream MCP server. The value should + include the header name and the token prefix if needed. Leave empty to omit the ID token + when fetching the tools list. + type: string + type: + type: string + enum: + - credentials + token_endpoint: + description: | + The token endpoint URL for fetching the OAuth 2.0 access token using client-credentials. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + format: uri + x-referenceable: true + client_id: + description: | + The client ID for the OAuth 2.0 client-credentials. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + client_secret: + description: | + The client secret for the OAuth 2.0 client-credentials. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + writeOnly: true + x-referenceable: true + additionalProperties: false + required: + - type + - token_endpoint + - client_id + CreateAIGatewayMCPServerRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + discriminator: + propertyName: type + mapping: + conversion-only: '#/components/schemas/AIGatewayMCPServerConversionOnly' + conversion-listener: '#/components/schemas/AIGatewayMCPServerConversionListener' + listener: '#/components/schemas/AIGatewayMCPServerListener' + passthrough-listener: '#/components/schemas/AIGatewayMCPServerPassthroughListener' + upstream-server: '#/components/schemas/AIGatewayMCPServerUpstreamServer' + oneOf: + - $ref: '#/components/schemas/AIGatewayMCPServerConversionOnly' + - $ref: '#/components/schemas/AIGatewayMCPServerConversionListener' + - $ref: '#/components/schemas/AIGatewayMCPServerListener' + - $ref: '#/components/schemas/AIGatewayMCPServerPassthroughListener' + - $ref: '#/components/schemas/AIGatewayMCPServerUpstreamServer' + UpdateAIGatewayMCPServerRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + discriminator: + propertyName: type + mapping: + conversion-only: '#/components/schemas/AIGatewayMCPServerConversionOnly' + conversion-listener: '#/components/schemas/AIGatewayMCPServerConversionListener' + listener: '#/components/schemas/AIGatewayMCPServerListener' + passthrough-listener: '#/components/schemas/AIGatewayMCPServerPassthroughListener' + upstream-server: '#/components/schemas/AIGatewayMCPServerUpstreamServer' + oneOf: + - $ref: '#/components/schemas/AIGatewayMCPServerConversionOnly' + - $ref: '#/components/schemas/AIGatewayMCPServerConversionListener' + - $ref: '#/components/schemas/AIGatewayMCPServerListener' + - $ref: '#/components/schemas/AIGatewayMCPServerPassthroughListener' + - $ref: '#/components/schemas/AIGatewayMCPServerUpstreamServer' + AIGatewayMCPServer: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + id: + $ref: '#/components/schemas/UUID' + created_at: + $ref: '#/components/schemas/CreatedAt' + updated_at: + $ref: '#/components/schemas/UpdatedAt' + discriminator: + propertyName: type + mapping: + conversion-only: '#/components/schemas/AIGatewayMCPServerConversionOnly' + conversion-listener: '#/components/schemas/AIGatewayMCPServerConversionListener' + listener: '#/components/schemas/AIGatewayMCPServerListener' + passthrough-listener: '#/components/schemas/AIGatewayMCPServerPassthroughListener' + upstream-server: '#/components/schemas/AIGatewayMCPServerUpstreamServer' + oneOf: + - $ref: '#/components/schemas/AIGatewayMCPServerConversionOnly' + - $ref: '#/components/schemas/AIGatewayMCPServerConversionListener' + - $ref: '#/components/schemas/AIGatewayMCPServerListener' + - $ref: '#/components/schemas/AIGatewayMCPServerPassthroughListener' + - $ref: '#/components/schemas/AIGatewayMCPServerUpstreamServer' + required: + - id + - created_at + - updated_at + AIGatewayMCPToolBase: + description: 'A tool exposed by the MCP Server, mapped to a backend HTTP endpoint.' + type: object + properties: + access: + type: object + additionalProperties: false + properties: + acls: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Access control rules for allowing or denying consumer groups access to this tool. + When configured, these will override the default access control rules defined on the MCP Server. + allOf: + - $ref: '#/components/schemas/AIGatewayMCPACLs' + annotations: + $ref: '#/components/schemas/AIGatewayMCPToolAnnotations' + description: + description: A description of what the tool does. + type: string + example: Search for available flights + headers: + $ref: '#/components/schemas/AIGatewayMCPToolHeaders' + host: + description: 'The host of the exported API, which must match the route''s hosts. It should be the route''s host. By default, Kong will extract the host from API configuration. If the configured host is wildcard, this field is required.' + type: string + name: + description: 'Tool identifier. In passthrough-listener mode, used to match remote MCP Server tools for ACL enforcement. In other modes, it is also used as the tool name (overrides annotations.title if present).' + type: string + method: + description: 'For conversion-only and conversion-listener modes, the method of the exported API, which must match the route''s methods.' + type: string + enum: + - DELETE + - GET + - PATCH + - POST + - PUT + path: + description: 'The path of the exported API, which must match the route''s paths. Path not starting with ''/'' are treated as relative path and the route path will be added as the prefix. By default, Kong will extract the path from API configuration.' + type: string + query: + $ref: '#/components/schemas/AIGatewayMCPToolQuery' + request_body: + $ref: '#/components/schemas/AIGatewayMCPToolRequestBody' + responses: + $ref: '#/components/schemas/AIGatewayMCPToolResponses' + scheme: + description: 'The scheme of the exported API. By default, Kong will extract the scheme from API configuration. If the configured scheme is not expected, this field can be used to override it.' + type: string + enum: + - http + - https + parameters: + $ref: '#/components/schemas/AIGatewayMCPToolParameters' + additionalProperties: false + required: + - name + - description + AIGatewayMCPUpstreamTool: + description: A tool exposed by an MCP Server in `upstream-server` mode. Extends the base tool with input/output schema overrides for the upstream server's advertised tool. + type: object + properties: + access: + type: object + additionalProperties: false + properties: + acls: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Access control rules for allowing or denying consumer groups access to this tool. + When configured, these will override the default access control rules defined on the MCP Server. + allOf: + - $ref: '#/components/schemas/AIGatewayMCPACLs' + annotations: + $ref: '#/components/schemas/AIGatewayMCPToolAnnotations' + description: + description: A description of what the tool does. + type: string + example: Search for available flights + headers: + $ref: '#/components/schemas/AIGatewayMCPToolHeaders' + host: + description: 'The host of the exported API, which must match the route''s hosts. It should be the route''s host. By default, Kong will extract the host from API configuration. If the configured host is wildcard, this field is required.' + type: string + name: + description: 'Tool identifier. In passthrough-listener mode, used to match remote MCP Server tools for ACL enforcement. In other modes, it is also used as the tool name (overrides annotations.title if present).' + type: string + method: + description: 'When provided, the method of the exported API, which must match the route''s methods.' + type: string + enum: + - DELETE + - GET + - PATCH + - POST + - PUT + path: + description: 'The path of the exported API, which must match the route''s paths. Path not starting with ''/'' are treated as relative path and the route path will be added as the prefix. By default, Kong will extract the path from API configuration.' + type: string + query: + $ref: '#/components/schemas/AIGatewayMCPToolQuery' + request_body: + $ref: '#/components/schemas/AIGatewayMCPToolRequestBody' + responses: + $ref: '#/components/schemas/AIGatewayMCPToolResponses' + scheme: + description: 'The scheme of the exported API. By default, Kong will extract the scheme from API configuration. If the configured scheme is not expected, this field can be used to override it.' + type: string + enum: + - http + - https + parameters: + $ref: '#/components/schemas/AIGatewayMCPToolParameters' + input_schema: + description: | + The entire `inputSchema` section for the tool. Overrides the upstream server's `inputSchema` + for the same tool name, if present. + type: object + additionalProperties: true + nullable: true + output_schema: + description: | + The entire `outputSchema` section for the tool. Overrides the upstream server's `outputSchema` + for the same tool name, if present. + type: object + additionalProperties: true + nullable: true + additionalProperties: false + required: + - name + - description + AIGatewayMCPToolAnnotations: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + destructive_hint: + description: 'If true, the tool may perform destructive updates' + type: boolean + idempotent_hint: + description: 'If true, repeated calls with same args have no additional effect' + type: boolean + open_world_hint: + description: 'If true, tool interacts with external entities' + type: boolean + read_only_hint: + description: 'If true, the tool does not modify its environment' + type: boolean + title: + description: Human-readable title for the tool + type: string + additionalProperties: false + AIGatewayMCPToolHeaders: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + The headers of the exported API. By default, Kong will extract the headers from API configuration. If the configured headers are not exactly matched, this field is required. + type: object + additionalProperties: true + AIGatewayMCPToolQuery: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + The query arguments of the exported API. If the generated query arguments are not exactly matched, this field is required. + type: object + additionalProperties: true + AIGatewayMCPToolRequestBody: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + The API requestBody specification defined in OpenAPI JSON format. For example, '{"content":{"application/x-www-form-urlencoded":{"schema":{"type":"object","properties":{"color":{"type":"array","items":{"type":"string"}}}}}}}'. See https://swagger.io/docs/specification/v3_0/describing-request-body/describing-request-body/ for more details. Note that `$ref` is not supported. + type: object + additionalProperties: true + AIGatewayMCPToolResponses: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + The API responses specification defined in OpenAPI JSON format. This specification will be used to validate the upstream response and map it back to the structuredOutput. For example, '{"200":{"content":{"application/json":{"schema":{"type":"object","properties":{"result":{"type":"string"}}}}}}}}'. See https://swagger.io/docs/specification/v3_0/describing-responses/ for more details. Only one non-error (status code < 400) response is supported. Note that `$ref` is not supported. + type: object + additionalProperties: true + AIGatewayMCPToolParameters: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: array + items: + $ref: '#/components/schemas/AIGatewayMCPToolParameter' + AIGatewayMCPToolParameter: + description: 'An API parameter specification defined in OpenAPI JSON format. For example, ''[{"name": "city", "in": "query", "description": "Name of the city to get the weather for", "required": true, "schema": {"type": "string"}}]''. See https://swagger.io/docs/specification/v3_0/describing-parameters/ for more details.' + type: object + properties: + name: + description: The name of the parameter. + type: string + example: origin + in: + description: The location of the parameter in the request. + type: string + example: query + enum: + - query + - path + - header + - body + description: + description: A description of the parameter. + type: string + example: The origin airport code. + required: + description: Whether this parameter is required. + type: boolean + example: true + schema: + description: 'JSON Schema definition for the parameter value. See https://swagger.io/docs/specification/v3_0/describing-parameters/#schema-vs-content for more details.' + type: object + additionalProperties: true + additionalProperties: false + required: + - name + - in + AIGatewayIdentityProviderReference: + description: Reference to a identity provider instance by name. + type: string + example: okta-ai-se + AIGatewayIdentityProviderKeyAuth: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an identity provider. + type: object + properties: + display_name: + description: The display name for this identity provider instance. + type: string + example: Okta AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this identity provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: okta-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + type: + type: string + enum: + - key-auth + config: + description: | + Configuration for the Kong Key auth identity provider. + For advanced use cases, additional config properties can be sent in the request body. + See: https://developer.konghq.com/plugins/key-auth/reference/ for the list of properties + type: object + additionalProperties: true + properties: + hide_credentials: + description: | + An optional boolean value telling the plugin to show or hide the credential from the upstream service. + If true, the plugin strips the credential from the request. + type: boolean + default: true + key_in_body: + description: | + If enabled, reads the request body. + Supported MIME types: application/www-form-urlencoded, application/json, and multipart/form-data. + type: boolean + default: false + key_in_header: + description: | + If enabled (default), the plugin reads the request header and tries to find the key in it. + type: boolean + default: true + key_in_query: + description: | + If enabled (default), the plugin reads the query parameter in the request and tries to find the key in it. + type: boolean + default: true + key_names: + description: | + An array of strings containing the names of the keys to look for in the request. + type: array + items: + type: string + default: + - apikey + required: + - display_name + - name + - type + title: AIGatewayIdentityProviderKeyAuthConfig + AIGatewayIdentityProviderOpenIDConnect: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an identity provider. + type: object + properties: + display_name: + description: The display name for this identity provider instance. + type: string + example: Okta AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this identity provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: okta-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + type: + type: string + enum: + - openid-connect + config: + description: | + Configuration for the OpenID Connect identity provider. + For advanced use cases, additional config properties can be sent in the request body. + See: https://developer.konghq.com/plugins/openid-connect/reference/ for the list of properties + type: object + additionalProperties: true + properties: + auth_methods: + description: Types of credentials/grants to enable. + type: array + items: + type: string + enum: + - authorization_code + - bearer + - client_credentials + - introspection + - kong_oauth2 + - password + - refresh_token + - session + - userinfo + default: + - bearer + - client_credentials + client_id: + description: | + An array of strings representing the client id for the OpenID Connect provider. + When multiple values are provided, the client ID and secrets pairs correspond based on their locations in the array. + type: array + items: + type: string + client_secret: + description: | + An array of strings representing the client secret for the OpenID Connect provider. + When multiple values are provided, the client ID and secrets pairs correspond based on their locations in the array. + type: array + items: + type: string + writeOnly: true + consumer_claims: + description: | + An array containing an array of string paths representing the location of the claim in a nested object. + For example, to map to user.info.id, set [ "user", "info", "id" ]. + type: array + items: + type: array + items: + type: string + consumer_optional: + description: | + Do not terminate the request if consumer mapping fails. + type: boolean + default: false + issuer: + description: URL that identifies the OpenID Provider + type: string + example: 'https://dev-123456.okta.com' + scopes: + description: | + This field is referenceable. + type: array + items: + type: string + default: + - openid + ssl_verify: + type: boolean + default: true + cache_tokens_salt: + description: | + Salt used for generating the cache key that is used for caching the token endpoint requests. + type: string + required: + - cache_tokens_salt + required: + - display_name + - name + - type + title: AIGatewayIdentityProviderOpenIDConnectConfig + AIGatewayModelProviderReference: + description: Reference to a model provider instance by name. + type: string + example: azure-ai-se + AIGatewayModelProviderAnthropic: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an upstream model provider. + type: object + properties: + type: + type: string + enum: + - anthropic + display_name: + description: The display name for this model provider instance. + type: string + example: Azure AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: azure-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + config: + description: Configuration for the model provider. + type: object + additionalProperties: false + properties: + auth: + $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + required: + - auth + required: + - type + - display_name + - name + - config + AIGatewayModelProviderCerebras: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an upstream model provider. + type: object + properties: + type: + type: string + enum: + - cerebras + display_name: + description: The display name for this model provider instance. + type: string + example: Azure AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: azure-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + config: + description: Configuration for the model provider. + type: object + additionalProperties: false + properties: + auth: + $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + required: + - auth + required: + - type + - display_name + - name + - config + AIGatewayModelProviderCohere: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an upstream model provider. + type: object + properties: + type: + type: string + enum: + - cohere + display_name: + description: The display name for this model provider instance. + type: string + example: Azure AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: azure-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + config: + description: Configuration for the model provider. + type: object + additionalProperties: false + properties: + auth: + $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + required: + - auth + required: + - type + - display_name + - name + - config + AIGatewayModelProviderDashscope: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an upstream model provider. + type: object + properties: + type: + type: string + enum: + - dashscope + display_name: + description: The display name for this model provider instance. + type: string + example: Azure AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: azure-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + config: + description: Configuration for the model provider. + type: object + additionalProperties: false + properties: + auth: + $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + required: + - auth + required: + - type + - display_name + - name + - config + AIGatewayModelProviderDatabricks: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an upstream model provider. + type: object + properties: + type: + type: string + enum: + - databricks + display_name: + description: The display name for this model provider instance. + type: string + example: Azure AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: azure-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + config: + description: Configuration for the model provider. + type: object + additionalProperties: false + properties: + auth: + $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + required: + - auth + required: + - type + - display_name + - name + - config + AIGatewayModelProviderDeepseek: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an upstream model provider. + type: object + properties: + type: + type: string + enum: + - deepseek + display_name: + description: The display name for this model provider instance. + type: string + example: Azure AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: azure-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + config: + description: Configuration for the model provider. + type: object + additionalProperties: false + properties: + auth: + $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + required: + - auth + required: + - type + - display_name + - name + - config + AIGatewayModelProviderHuggingface: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an upstream model provider. + type: object + properties: + type: + type: string + enum: + - huggingface + display_name: + description: The display name for this model provider instance. + type: string + example: Azure AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: azure-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + config: + description: Configuration for the model provider. + type: object + additionalProperties: false + properties: + auth: + $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + required: + - auth + required: + - type + - display_name + - name + - config + AIGatewayModelProviderKimi: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an upstream model provider. + type: object + properties: + type: + type: string + enum: + - kimi + display_name: + description: The display name for this model provider instance. + type: string + example: Azure AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: azure-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + config: + description: Configuration for the model provider. + type: object + additionalProperties: false + properties: + auth: + $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + required: + - auth + required: + - type + - display_name + - name + - config + AIGatewayModelProviderLlama2: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an upstream model provider. + type: object + properties: + type: + type: string + enum: + - llama2 + display_name: + description: The display name for this model provider instance. + type: string + example: Azure AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: azure-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + config: + description: Configuration for the model provider. + type: object + additionalProperties: false + properties: + auth: + $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + required: + - auth + required: + - type + - display_name + - name + - config + AIGatewayModelProviderMistral: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an upstream model provider. + type: object + properties: + type: + type: string + enum: + - mistral + display_name: + description: The display name for this model provider instance. + type: string + example: Azure AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: azure-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + config: + description: Configuration for the model provider. + type: object + additionalProperties: false + properties: + auth: + $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + required: + - auth + required: + - type + - display_name + - name + - config + AIGatewayModelProviderOllama: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an upstream model provider. + type: object + properties: + type: + type: string + enum: + - ollama + display_name: + description: The display name for this model provider instance. + type: string + example: Azure AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: azure-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + config: + description: Configuration for the model provider. + type: object + additionalProperties: false + properties: + auth: + $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + required: + - auth + required: + - type + - display_name + - name + - config + AIGatewayModelProviderOpenai: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an upstream model provider. + type: object + properties: + type: + type: string + enum: + - openai + display_name: + description: The display name for this model provider instance. + type: string + example: Azure AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: azure-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + config: + description: Configuration for the model provider. + type: object + additionalProperties: false + properties: + auth: + $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + required: + - auth + required: + - type + - display_name + - name + - config + AIGatewayModelProviderVercel: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an upstream model provider. + type: object + properties: + type: + type: string + enum: + - vercel + display_name: + description: The display name for this model provider instance. + type: string + example: Azure AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: azure-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + config: + description: Configuration for the model provider. + type: object + additionalProperties: false + properties: + auth: + $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + required: + - auth + required: + - type + - display_name + - name + - config + AIGatewayModelProviderVllm: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an upstream model provider. + type: object + properties: + type: + type: string + enum: + - vllm + display_name: + description: The display name for this model provider instance. + type: string + example: Azure AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: azure-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + config: + description: Configuration for the model provider. + type: object + additionalProperties: false + properties: + auth: + $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + required: + - auth + required: + - type + - display_name + - name + - config + AIGatewayModelProviderXai: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for an upstream model provider. + type: object + properties: + type: + type: string + enum: + - xai + display_name: + description: The display name for this model provider instance. + type: string + example: Azure AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: azure-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + config: + description: Configuration for the model provider. + type: object + additionalProperties: false + properties: + auth: + $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + required: + - auth + required: + - type + - display_name + - name + - config + AIGatewayModelProviderConfigAuthBasic: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Basic auth config for an upstream model provider. + type: object + properties: + type: + type: string + enum: + - basic + headers: + type: array + items: + type: object + additionalProperties: false + required: + - name + properties: + name: + description: | + The name of the header used for authentication. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + value: + description: | + The auth header value for ‘header_name’, for example ‘Bearer key...’. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + writeOnly: true + x-referenceable: true + maxItems: 1 + params: + type: array + items: + type: object + additionalProperties: false + required: + - name + properties: + name: + description: | + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + value: + description: | + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + writeOnly: true + x-referenceable: true + location: + description: 'Specify whether the param name and value options go in a query string, or the POST form/JSON body.' + type: string + enum: + - body + - query + maxItems: 1 + additionalProperties: false + required: + - type + title: AIGatewayModelProviderConfigAuthBasic + AIGatewayModelProviderBedrock: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Config for AWS model provider. + type: object + properties: + type: + type: string + enum: + - bedrock + display_name: + description: The display name for this model provider instance. + type: string + example: Azure AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: azure-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + config: + type: object + additionalProperties: false + properties: + auth: + discriminator: + propertyName: type + mapping: + basic: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + aws: '#/components/schemas/AIGatewayModelProviderConfigAuthAWS' + oneOf: + - $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + - $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthAWS' + required: + - auth + required: + - type + - display_name + - name + - config + AIGatewayModelProviderConfigAuthAWS: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for AWS model provider. + type: object + properties: + type: + type: string + enum: + - aws + access_key_id: + description: | + The access key id for authenticating with static IAM User credentials. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + secret_access_key: + description: | + The secret access key for authenticating with static IAM User credentials. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + writeOnly: true + x-referenceable: true + assume_role_arn: + description: | + The ARN of the IAM role to assume for generating authentication tokens. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + role_session_name: + description: | + The session name for the temporary credentials when assuming the IAM role. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + sts_endpoint_url: + description: 'The STS endpoint URL to use for generating authentication tokens. If not specified, the default AWS STS endpoint will be used.' + type: string + batch_role_arn: + description: AWS role arn to use when calling the batch API. + type: string + required: + - type + title: AIGatewayModelProviderConfigAuthAWS + AIGatewayModelProviderAzure: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Config for Azure model provider. + type: object + properties: + type: + type: string + enum: + - azure + display_name: + description: The display name for this model provider instance. + type: string + example: Azure AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: azure-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + config: + type: object + additionalProperties: false + properties: + auth: + discriminator: + propertyName: type + mapping: + basic: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + azure: '#/components/schemas/AIGatewayModelProviderConfigAuthAzure' + oneOf: + - $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + - $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthAzure' + instance: + type: string + example: kong-az-east + required: + - auth + - instance + required: + - type + - display_name + - name + - config + AIGatewayModelProviderConfigAuthAzure: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for Azure model provider. + type: object + properties: + type: + type: string + enum: + - azure + client_id: + description: | + If azure_use_managed_identity is set to true, and you need to use a different user-assigned identity for this LLM instance, set the client ID. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + client_secret: + description: | + If azure_use_managed_identity is set to true, and you need to use a different user-assigned identity for this LLM instance, set the client secret. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + writeOnly: true + x-referenceable: true + tenant_id: + description: | + If azure_use_managed_identity is set to true, and you need to use a different user-assigned identity for this LLM instance, set the tenant ID. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + use_managed_identity: + description: Set true to use the Azure Cloud Managed Identity (or user-assigned identity) to authenticate with Azure-provider models. + type: boolean + additionalProperties: false + required: + - type + title: AIGatewayModelProviderConfigAuthAzure + AIGatewayModelProviderGemini: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Config for GCP model provider. + type: object + properties: + type: + type: string + enum: + - gemini + display_name: + description: The display name for this model provider instance. + type: string + example: Azure AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: azure-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + config: + type: object + additionalProperties: false + properties: + auth: + discriminator: + propertyName: type + mapping: + basic: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + gcp: '#/components/schemas/AIGatewayModelProviderConfigAuthGCP' + oneOf: + - $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + - $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthGCP' + required: + - auth + required: + - type + - display_name + - name + - config + AIGatewayModelProviderVertex: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Config for GCP model provider. + type: object + properties: + type: + type: string + enum: + - vertex + display_name: + description: The display name for this model provider instance. + type: string + example: Azure AI SE + maxLength: 256 + minLength: 1 + name: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + A user-defined unique identifier for this model provider instance, used as a stable human-readable reference. This value is immutable after creation. + example: azure-ai-se + allOf: + - $ref: '#/components/schemas/AIGatewayEntityIdentifier' + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + config: + type: object + additionalProperties: false + properties: + auth: + discriminator: + propertyName: type + mapping: + basic: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + gcp: '#/components/schemas/AIGatewayModelProviderConfigAuthGCP' + oneOf: + - $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthBasic' + - $ref: '#/components/schemas/AIGatewayModelProviderConfigAuthGCP' + required: + - auth + required: + - type + - display_name + - name + - config + AIGatewayModelProviderConfigAuthGCP: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + Configuration for GCP model provider. + type: object + properties: + type: + type: string + enum: + - gcp + service_account_json: + description: | + Full JSON string of the GCP service account to authenticate. If not set (and gcp_use_service_account is true), the service account JSON will be from the environment variable GCP_SERVICE_ACCOUNT. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + writeOnly: true + x-referenceable: true + metadata_url: + description: | + Custom metadata URL for GCP authentication. Useful for restricted network environments or custom GCP endpoints. If not set, Kong will use the default Google metadata endpoint. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + oauth_token_url: + description: | + Custom OAuth token URL for GCP authentication. Useful for restricted network environments or custom GCP endpoints. If not set, Kong will use the default Google OAuth token endpoint. + This field is [referenceable](https://developer.konghq.com/gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault). + type: string + x-referenceable: true + use_gcp_service_account: + description: Use service account auth for GCP-based providers and models. + type: boolean + additionalProperties: false + required: + - type + title: AIGatewayModelProviderConfigAuthGCP + CreateAIGatewayConfigStoreRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + display_name: + description: The display name of the Config Store. + type: string + example: my-config-store + maxLength: 256 + pattern: '^[a-zA-Z0-9.\-_~]*$' + name: + description: The name of the Config Store. This value is immutable after creation. + type: string + example: my-config-store + maxLength: 256 + minLength: 1 + pattern: '^[a-zA-Z0-9.\-_~]*$' + additionalProperties: false + required: + - name + UpdateAIGatewayConfigStoreRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + display_name: + description: The display name of the Config Store. + type: string + example: MyConfigStore + labels: + $ref: '#/components/schemas/PublicLabels' + managed_by: + $ref: '#/components/schemas/ManagedBy' + additionalProperties: false + AIGatewayConfigStore: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + display_name: + description: The display name of the Config Store. + type: string + example: my-config-store + maxLength: 256 + pattern: '^[a-zA-Z0-9.\-_~]*$' + name: + description: The name of the Config Store. This value is immutable after creation. + type: string + example: my-config-store + maxLength: 256 + minLength: 1 + pattern: '^[a-zA-Z0-9.\-_~]*$' + id: + $ref: '#/components/schemas/UUID' + created_at: + $ref: '#/components/schemas/CreatedAt' + updated_at: + $ref: '#/components/schemas/UpdatedAt' + additionalProperties: false + required: + - name + - id + - created_at + - updated_at + AIGatewayConfigStoreSecretKey: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + The unique key identifying the secret within the Config Store. + type: string + example: my-secret-key + maxLength: 512 + minLength: 1 + AIGatewayConfigStoreSecretValue: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + + The secret value. Once stored, this value cannot be retrieved. + type: string + example: my-secret-value + maxLength: 5120 + writeOnly: true + CreateAIGatewayConfigStoreSecretRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + key: + $ref: '#/components/schemas/AIGatewayConfigStoreSecretKey' + value: + $ref: '#/components/schemas/AIGatewayConfigStoreSecretValue' + additionalProperties: false + required: + - key + - value + UpdateAIGatewayConfigStoreSecretRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + value: + $ref: '#/components/schemas/AIGatewayConfigStoreSecretValue' + additionalProperties: false + required: + - value + AIGatewayConfigStoreSecret: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + key: + $ref: '#/components/schemas/AIGatewayConfigStoreSecretKey' + created_at: + $ref: '#/components/schemas/CreatedAt' + updated_at: + $ref: '#/components/schemas/UpdatedAt' + additionalProperties: false + required: + - key + - created_at + - updated_at + CreateAIGatewayModelProviderRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + discriminator: + propertyName: type + mapping: + anthropic: '#/components/schemas/AIGatewayModelProviderAnthropic' + azure: '#/components/schemas/AIGatewayModelProviderAzure' + bedrock: '#/components/schemas/AIGatewayModelProviderBedrock' + cerebras: '#/components/schemas/AIGatewayModelProviderCerebras' + cohere: '#/components/schemas/AIGatewayModelProviderCohere' + dashscope: '#/components/schemas/AIGatewayModelProviderDashscope' + databricks: '#/components/schemas/AIGatewayModelProviderDatabricks' + deepseek: '#/components/schemas/AIGatewayModelProviderDeepseek' + gemini: '#/components/schemas/AIGatewayModelProviderGemini' + huggingface: '#/components/schemas/AIGatewayModelProviderHuggingface' + kimi: '#/components/schemas/AIGatewayModelProviderKimi' + llama2: '#/components/schemas/AIGatewayModelProviderLlama2' + mistral: '#/components/schemas/AIGatewayModelProviderMistral' + ollama: '#/components/schemas/AIGatewayModelProviderOllama' + openai: '#/components/schemas/AIGatewayModelProviderOpenai' + vercel: '#/components/schemas/AIGatewayModelProviderVercel' + vllm: '#/components/schemas/AIGatewayModelProviderVllm' + xai: '#/components/schemas/AIGatewayModelProviderXai' + vertex: '#/components/schemas/AIGatewayModelProviderVertex' + oneOf: + - $ref: '#/components/schemas/AIGatewayModelProviderAnthropic' + - $ref: '#/components/schemas/AIGatewayModelProviderAzure' + - $ref: '#/components/schemas/AIGatewayModelProviderBedrock' + - $ref: '#/components/schemas/AIGatewayModelProviderCerebras' + - $ref: '#/components/schemas/AIGatewayModelProviderCohere' + - $ref: '#/components/schemas/AIGatewayModelProviderDashscope' + - $ref: '#/components/schemas/AIGatewayModelProviderDatabricks' + - $ref: '#/components/schemas/AIGatewayModelProviderDeepseek' + - $ref: '#/components/schemas/AIGatewayModelProviderGemini' + - $ref: '#/components/schemas/AIGatewayModelProviderHuggingface' + - $ref: '#/components/schemas/AIGatewayModelProviderKimi' + - $ref: '#/components/schemas/AIGatewayModelProviderLlama2' + - $ref: '#/components/schemas/AIGatewayModelProviderMistral' + - $ref: '#/components/schemas/AIGatewayModelProviderOllama' + - $ref: '#/components/schemas/AIGatewayModelProviderOpenai' + - $ref: '#/components/schemas/AIGatewayModelProviderVercel' + - $ref: '#/components/schemas/AIGatewayModelProviderVllm' + - $ref: '#/components/schemas/AIGatewayModelProviderXai' + - $ref: '#/components/schemas/AIGatewayModelProviderVertex' + UpdateAIGatewayModelProviderRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + discriminator: + propertyName: type + mapping: + anthropic: '#/components/schemas/AIGatewayModelProviderAnthropic' + azure: '#/components/schemas/AIGatewayModelProviderAzure' + bedrock: '#/components/schemas/AIGatewayModelProviderBedrock' + cerebras: '#/components/schemas/AIGatewayModelProviderCerebras' + cohere: '#/components/schemas/AIGatewayModelProviderCohere' + dashscope: '#/components/schemas/AIGatewayModelProviderDashscope' + databricks: '#/components/schemas/AIGatewayModelProviderDatabricks' + deepseek: '#/components/schemas/AIGatewayModelProviderDeepseek' + gemini: '#/components/schemas/AIGatewayModelProviderGemini' + huggingface: '#/components/schemas/AIGatewayModelProviderHuggingface' + kimi: '#/components/schemas/AIGatewayModelProviderKimi' + llama2: '#/components/schemas/AIGatewayModelProviderLlama2' + mistral: '#/components/schemas/AIGatewayModelProviderMistral' + ollama: '#/components/schemas/AIGatewayModelProviderOllama' + openai: '#/components/schemas/AIGatewayModelProviderOpenai' + vercel: '#/components/schemas/AIGatewayModelProviderVercel' + vllm: '#/components/schemas/AIGatewayModelProviderVllm' + xai: '#/components/schemas/AIGatewayModelProviderXai' + vertex: '#/components/schemas/AIGatewayModelProviderVertex' + oneOf: + - $ref: '#/components/schemas/AIGatewayModelProviderAnthropic' + - $ref: '#/components/schemas/AIGatewayModelProviderAzure' + - $ref: '#/components/schemas/AIGatewayModelProviderBedrock' + - $ref: '#/components/schemas/AIGatewayModelProviderCerebras' + - $ref: '#/components/schemas/AIGatewayModelProviderCohere' + - $ref: '#/components/schemas/AIGatewayModelProviderDashscope' + - $ref: '#/components/schemas/AIGatewayModelProviderDatabricks' + - $ref: '#/components/schemas/AIGatewayModelProviderDeepseek' + - $ref: '#/components/schemas/AIGatewayModelProviderGemini' + - $ref: '#/components/schemas/AIGatewayModelProviderHuggingface' + - $ref: '#/components/schemas/AIGatewayModelProviderKimi' + - $ref: '#/components/schemas/AIGatewayModelProviderLlama2' + - $ref: '#/components/schemas/AIGatewayModelProviderMistral' + - $ref: '#/components/schemas/AIGatewayModelProviderOllama' + - $ref: '#/components/schemas/AIGatewayModelProviderOpenai' + - $ref: '#/components/schemas/AIGatewayModelProviderVercel' + - $ref: '#/components/schemas/AIGatewayModelProviderVllm' + - $ref: '#/components/schemas/AIGatewayModelProviderXai' + - $ref: '#/components/schemas/AIGatewayModelProviderVertex' + AIGatewayModelProvider: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + id: + $ref: '#/components/schemas/UUID' + created_at: + $ref: '#/components/schemas/CreatedAt' + updated_at: + $ref: '#/components/schemas/UpdatedAt' + discriminator: + propertyName: type + mapping: + anthropic: '#/components/schemas/AIGatewayModelProviderAnthropic' + azure: '#/components/schemas/AIGatewayModelProviderAzure' + bedrock: '#/components/schemas/AIGatewayModelProviderBedrock' + cerebras: '#/components/schemas/AIGatewayModelProviderCerebras' + cohere: '#/components/schemas/AIGatewayModelProviderCohere' + dashscope: '#/components/schemas/AIGatewayModelProviderDashscope' + databricks: '#/components/schemas/AIGatewayModelProviderDatabricks' + deepseek: '#/components/schemas/AIGatewayModelProviderDeepseek' + gemini: '#/components/schemas/AIGatewayModelProviderGemini' + huggingface: '#/components/schemas/AIGatewayModelProviderHuggingface' + kimi: '#/components/schemas/AIGatewayModelProviderKimi' + llama2: '#/components/schemas/AIGatewayModelProviderLlama2' + mistral: '#/components/schemas/AIGatewayModelProviderMistral' + ollama: '#/components/schemas/AIGatewayModelProviderOllama' + openai: '#/components/schemas/AIGatewayModelProviderOpenai' + vercel: '#/components/schemas/AIGatewayModelProviderVercel' + vllm: '#/components/schemas/AIGatewayModelProviderVllm' + xai: '#/components/schemas/AIGatewayModelProviderXai' + vertex: '#/components/schemas/AIGatewayModelProviderVertex' + oneOf: + - $ref: '#/components/schemas/AIGatewayModelProviderAnthropic' + - $ref: '#/components/schemas/AIGatewayModelProviderAzure' + - $ref: '#/components/schemas/AIGatewayModelProviderBedrock' + - $ref: '#/components/schemas/AIGatewayModelProviderCerebras' + - $ref: '#/components/schemas/AIGatewayModelProviderCohere' + - $ref: '#/components/schemas/AIGatewayModelProviderDashscope' + - $ref: '#/components/schemas/AIGatewayModelProviderDatabricks' + - $ref: '#/components/schemas/AIGatewayModelProviderDeepseek' + - $ref: '#/components/schemas/AIGatewayModelProviderGemini' + - $ref: '#/components/schemas/AIGatewayModelProviderHuggingface' + - $ref: '#/components/schemas/AIGatewayModelProviderKimi' + - $ref: '#/components/schemas/AIGatewayModelProviderLlama2' + - $ref: '#/components/schemas/AIGatewayModelProviderMistral' + - $ref: '#/components/schemas/AIGatewayModelProviderOllama' + - $ref: '#/components/schemas/AIGatewayModelProviderOpenai' + - $ref: '#/components/schemas/AIGatewayModelProviderVercel' + - $ref: '#/components/schemas/AIGatewayModelProviderVllm' + - $ref: '#/components/schemas/AIGatewayModelProviderXai' + - $ref: '#/components/schemas/AIGatewayModelProviderVertex' + required: + - id + - created_at + - updated_at + CreateAIGatewayIdentityProviderRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + discriminator: + propertyName: type + mapping: + key-auth: '#/components/schemas/AIGatewayIdentityProviderKeyAuth' + openid-connect: '#/components/schemas/AIGatewayIdentityProviderOpenIDConnect' + oneOf: + - $ref: '#/components/schemas/AIGatewayIdentityProviderKeyAuth' + - $ref: '#/components/schemas/AIGatewayIdentityProviderOpenIDConnect' + UpdateAIGatewayIdentityProviderRequest: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + discriminator: + propertyName: type + mapping: + key-auth: '#/components/schemas/AIGatewayIdentityProviderKeyAuth' + openid-connect: '#/components/schemas/AIGatewayIdentityProviderOpenIDConnect' + oneOf: + - $ref: '#/components/schemas/AIGatewayIdentityProviderKeyAuth' + - $ref: '#/components/schemas/AIGatewayIdentityProviderOpenIDConnect' + AIGatewayIdentityProvider: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + discriminator: + propertyName: type + mapping: + key-auth: '#/components/schemas/AIGatewayIdentityProviderKeyAuthResponse' + openid-connect: '#/components/schemas/AIGatewayIdentityProviderOpenIDConnectResponse' + oneOf: + - $ref: '#/components/schemas/AIGatewayIdentityProviderKeyAuthResponse' + - $ref: '#/components/schemas/AIGatewayIdentityProviderOpenIDConnectResponse' + AIGatewayIdentityProviderResponseProperties: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + type: object + properties: + id: + $ref: '#/components/schemas/UUID' + created_at: + $ref: '#/components/schemas/CreatedAt' + updated_at: + $ref: '#/components/schemas/UpdatedAt' + required: + - id + - created_at + - updated_at + AIGatewayIdentityProviderKeyAuthResponse: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + allOf: + - $ref: '#/components/schemas/AIGatewayIdentityProviderKeyAuth' + - $ref: '#/components/schemas/AIGatewayIdentityProviderResponseProperties' + title: AIGatewayIdentityProviderKeyAuthResponse + AIGatewayIdentityProviderOpenIDConnectResponse: + description: |- + **Pre-release Feature** + This feature is currently in beta and is subject to change. + allOf: + - $ref: '#/components/schemas/AIGatewayIdentityProviderOpenIDConnect' + - $ref: '#/components/schemas/AIGatewayIdentityProviderResponseProperties' + title: AIGatewayIdentityProviderOpenIDConnectResponse + PublicLabels: + description: | + Public labels store information about an entity that can be used for filtering a list of objects. + + Public labels are intended to store **PUBLIC** metadata. + + Keys must be of length 1-63 characters, and cannot start with "kong", "konnect", "mesh", "kic", or "_". + type: object + example: + category: finance + additionalProperties: + type: string + pattern: '^[a-z0-9A-Z]{1}([a-z0-9A-Z-._]*[a-z0-9A-Z]+)?$' + minLength: 1 + maxLength: 63 + maxProperties: 50 + title: PublicLabels + UUID: + description: Contains a unique identifier used for this resource. + type: string + format: uuid + example: 5f9fd312-a987-4628-b4c5-bb4f4fddd5f7 + readOnly: true + CreatedAt: + description: An ISO-8601 timestamp representation of entity creation date. + type: string + format: date-time + example: '2022-11-04T20:10:06.927Z' + readOnly: true + UpdatedAt: + description: An ISO-8601 timestamp representation of entity update date. + type: string + format: date-time + example: '2022-11-04T20:10:06.927Z' + readOnly: true + PageMeta: + description: Contains pagination query parameters and the total number of objects returned. + type: object + properties: + number: + type: number + example: 1 + size: + type: number + example: 10 + total: + type: number + example: 100 + required: + - number + - size + - total + PaginatedMeta: + description: returns the pagination information + type: object + properties: + page: + $ref: '#/components/schemas/PageMeta' + required: + - page + title: PaginatedMeta + BaseError: + description: standard error + type: object + properties: + status: + description: | + The HTTP status code of the error. Useful when passing the response + body to child properties in a frontend UI. Must be returned as an integer. + type: integer + readOnly: true + title: + description: | + A short, human-readable summary of the problem. It should not + change between occurences of a problem, except for localization. + Should be provided as "Sentence case" for direct use in the UI. + type: string + readOnly: true + type: + description: The error type. + type: string + readOnly: true + instance: + description: | + Used to return the correlation ID back to the user, in the format + kong:trace:. This helps us find the relevant logs + when a customer reports an issue. + type: string + readOnly: true + detail: + description: | + A human readable explanation specific to this occurence of the problem. + This field may contain request/entity data to help the user understand + what went wrong. Enclose variable values in square brackets. Should be + provided as "Sentence case" for direct use in the UI. + type: string + readOnly: true + required: + - status + - title + - instance + - detail + title: Error + UnauthorizedError: + allOf: + - $ref: '#/components/schemas/BaseError' + - type: object + properties: + status: + example: 401 + title: + example: Unauthorized + type: + example: 'https://httpstatuses.com/401' + instance: + example: 'kong:trace:1234567890' + detail: + example: Invalid credentials + ForbiddenError: + allOf: + - $ref: '#/components/schemas/BaseError' + - type: object + properties: + status: + example: 403 + title: + example: Forbidden + type: + example: 'https://httpstatuses.com/403' + instance: + example: 'kong:trace:1234567890' + detail: + example: Forbidden + TooManyRequestsError: + allOf: + - $ref: '#/components/schemas/BaseError' + - type: object + properties: + status: + example: 429 + title: + example: Too Many Requests + type: + example: 'https://httpstatuses.com/429' + instance: + example: 'kong:trace:1234567890' + detail: + example: Too Many Requests + InvalidRules: + description: invalid parameters rules + type: string + enum: + - required + - is_array + - is_base64 + - is_boolean + - is_date_time + - is_integer + - is_null + - is_number + - is_object + - is_string + - is_uuid + - is_fqdn + - is_arn + - unknown_property + - missing_reference + - is_label + - matches_regex + - invalid + - is_supported_network_availability_zone_list + - is_supported_network_cidr_block + - is_supported_provider_region + - type + nullable: true + readOnly: true + InvalidParameterStandard: + type: object + properties: + field: + type: string + example: name + readOnly: true + rule: + $ref: '#/components/schemas/InvalidRules' + source: + type: string + example: body + reason: + type: string + example: is a required field + readOnly: true + additionalProperties: false + required: + - field + - reason + InvalidParameterMinimumLength: + type: object + properties: + field: + type: string + example: name + readOnly: true + rule: + description: invalid parameters rules + type: string + enum: + - min_length + - min_digits + - min_lowercase + - min_uppercase + - min_symbols + - min_items + - min + nullable: false + readOnly: true + minimum: + type: integer + example: 8 + source: + type: string + example: body + reason: + type: string + example: must have at least 8 characters + readOnly: true + additionalProperties: false + required: + - field + - reason + - rule + - minimum + InvalidParameterMaximumLength: + type: object + properties: + field: + type: string + example: name + readOnly: true + rule: + description: invalid parameters rules + type: string + enum: + - max_length + - max_items + - max + nullable: false + readOnly: true + maximum: + type: integer + example: 8 + source: + type: string + example: body + reason: + type: string + example: must not have more than 8 characters + readOnly: true + additionalProperties: false + required: + - field + - reason + - rule + - maximum + InvalidParameterChoiceItem: + type: object + properties: + field: + type: string + example: name + readOnly: true + rule: + description: invalid parameters rules + type: string + enum: + - enum + nullable: false + readOnly: true + reason: + type: string + example: is a required field + readOnly: true + choices: + type: array + items: {} + minItems: 1 + nullable: false + readOnly: true + uniqueItems: true + source: + type: string + example: body + additionalProperties: false + required: + - field + - reason + - rule + - choices + InvalidParameterDependentItem: + type: object + properties: + field: + type: string + example: name + readOnly: true + rule: + description: invalid parameters rules + type: string + enum: + - dependent_fields + nullable: true + readOnly: true + reason: + type: string + example: is a required field + readOnly: true + dependents: + type: array + items: {} + nullable: true + readOnly: true + uniqueItems: true + source: + type: string + example: body + additionalProperties: false + required: + - field + - rule + - reason + - dependents + InvalidParameters: + description: invalid parameters + type: array + items: + oneOf: + - $ref: '#/components/schemas/InvalidParameterStandard' + - $ref: '#/components/schemas/InvalidParameterMinimumLength' + - $ref: '#/components/schemas/InvalidParameterMaximumLength' + - $ref: '#/components/schemas/InvalidParameterChoiceItem' + - $ref: '#/components/schemas/InvalidParameterDependentItem' + minItems: 1 + nullable: false + uniqueItems: true + BadRequestError: + allOf: + - $ref: '#/components/schemas/BaseError' + - type: object + required: + - invalid_parameters + properties: + invalid_parameters: + $ref: '#/components/schemas/InvalidParameters' + ConflictError: + allOf: + - $ref: '#/components/schemas/BaseError' + - type: object + properties: + status: + example: 409 + title: + example: Conflict + type: + example: 'https://httpstatuses.com/409' + instance: + example: 'kong:trace:1234567890' + detail: + example: Conflict + NotFoundError: + allOf: + - $ref: '#/components/schemas/BaseError' + - type: object + properties: + status: + example: 404 + title: + example: Not Found + type: + example: 'https://httpstatuses.com/404' + instance: + example: 'kong:trace:1234567890' + detail: + example: Not found + CursorMetaPage: + type: object + properties: + first: + description: URI to the first page + type: string + format: path + last: + description: URI to the last page + type: string + format: path + next: + description: URI to the next page + type: string + format: path + nullable: true + previous: + description: URI to the previous page + type: string + format: path + nullable: true + size: + description: Requested page size + type: number + example: 10 + required: + - size + - next + - previous + CursorMeta: + description: Pagination metadata. + type: object + properties: + page: + $ref: '#/components/schemas/CursorMetaPage' + required: + - page + ManagedBy: + description: | + Stores information about what manages this entity, such as the tool or system responsible for its lifecycle (for example, `terraform`). + + Keys must be 1–63 characters long and start with an alphanumeric character. + type: object + example: + owner: terraform + additionalProperties: + type: string + pattern: '^[a-z0-9A-Z]{1}([a-z0-9A-Z-._]*[a-z0-9A-Z]+)?$' + minLength: 1 + maxLength: 63 + maxProperties: 5 + title: ManagedBy + examples: + AIGatewayExample: + value: + id: bf138ba2-c9b1-4229-b268-04d9d8a6410b + display_name: My AI Gateway + name: my-ai-gateway + description: An AI Gateway for my organization. + labels: + env: production + endpoints: + configuration: 'https://acfe5f253f.cp.konghq.com' + telemetry: 'https://acfe5f253f.tp0.konghq.com' + config_hash: 2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824 + created_at: '2024-01-01T00:00:00.000Z' + updated_at: '2024-01-01T00:00:00.000Z' + AIGatewayDataplaneCertificateExample: + value: + title: My AI Gateway Data Plane Certificate + description: My description + cert: "-----BEGIN CERTIFICATE-----\r\n*****\r\n-----END CERTIFICATE-----\r\n" + CreateAIGatewayRequestExample: + value: + display_name: My AI Gateway + name: my-ai-gateway + description: An AI Gateway for my organization. + labels: + env: production + UpdateAIGatewayRequestExample: + value: + display_name: My Updated AI Gateway + name: my-ai-gateway + description: An updated description. + labels: + env: staging + ListAIGatewaysResponseExample: + value: + meta: + page: + number: 1 + size: 10 + total: 1 + data: + - id: bf138ba2-c9b1-4229-b268-04d9d8a6410b + display_name: My AI Gateway + name: my-ai-gateway + description: An AI Gateway for my organization. + labels: + env: production + endpoints: + configuration: 'https://acfe5f253f.cp.konghq.com' + telemetry: 'https://acfe5f253f.tp0.konghq.com' + proxy_urls: + - host: example.com + port: 443 + protocol: https + config_hash: 2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824 + created_at: '2024-01-01T00:00:00.000Z' + updated_at: '2024-01-01T00:00:00.000Z' + UnauthorizedExample: + value: + status: 401 + title: Unauthorized + instance: 'kong:trace:8347343766220159418' + detail: Unauthorized + ForbiddenExample: + value: + status: 403 + title: Forbidden + instance: 'kong:trace:2723154947768991354' + detail: You do not have permission to perform this action + NotFoundExample: + value: + status: 404 + title: Not Found + instance: 'kong:trace:6816496025408232265' + detail: Not Found + responses: + ListAIGatewaysResponse: + description: A paginated list of AI Gateways. + content: + application/json: + schema: + type: object + properties: + data: + type: array + items: + $ref: '#/components/schemas/AIGateway' + meta: + $ref: '#/components/schemas/PaginatedMeta' + additionalProperties: false + required: + - data + - meta + title: ListAIGatewaysResponse + examples: + List AI Gateways Response: + $ref: '#/components/examples/ListAIGatewaysResponseExample' + CreateAIGatewayResponse: + description: AI Gateway created successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGateway' + examples: + AI Gateway Response: + $ref: '#/components/examples/AIGatewayExample' + GetAIGatewayResponse: + description: A successful response returning an AI Gateway. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGateway' + examples: + AI Gateway Response: + $ref: '#/components/examples/AIGatewayExample' + UpdateAIGatewayResponse: + description: AI Gateway updated successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGateway' + examples: + AI Gateway Response: + $ref: '#/components/examples/AIGatewayExample' + CreateAIGatewayModelResponse: + description: Model created successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayModel' + GetAIGatewayModelResponse: + description: A successful response returning the AI Gateway Model. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayModel' + UpdateAIGatewayModelResponse: + description: Model updated successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayModel' + ListAIGatewayModelsResponse: + description: A successful response listing AI Gateway Models. + content: + application/json: + schema: + type: object + properties: + data: + type: array + items: + $ref: '#/components/schemas/AIGatewayModel' + meta: + $ref: '#/components/schemas/CursorMeta' + required: + - data + - meta + ListAIGatewayDataPlaneCertificatesResponse: + description: Example response + content: + application/json: + schema: + type: object + properties: + data: + type: array + items: + $ref: '#/components/schemas/AIGatewayDataPlaneClientCertificate' + meta: + $ref: '#/components/schemas/CursorMeta' + additionalProperties: false + required: + - data + - meta + CreateAIGatewayDataPlaneCertificateResponse: + description: Response body for creating a DataPlane certificate. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayDataPlaneClientCertificate' + GetAIGatewayDataPlaneCertificateResponse: + description: Response body for retrieving a DataPlane certificate. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayDataPlaneClientCertificate' + ListAIGatewayDataPlaneNodesResponse: + description: A successful response listing AI Gateway Data Plane Nodes. + content: + application/json: + schema: + type: object + properties: + data: + type: array + items: + $ref: '#/components/schemas/AIGatewayDataPlaneNode' + meta: + $ref: '#/components/schemas/CursorMeta' + required: + - data + - meta + GetAIGatewayExpectedConfigVersionResponse: + description: Response body for retrieving the expected config version of the AI Gateway. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayExpectedConfigVersion' + GetAIGatewayDataPlaneNodeResponse: + description: A successful response containing the AI Gateway Data Plane Node. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayDataPlaneNode' + ListAIGatewayVaultsResponse: + description: A paginated list of AI Gateway Vaults. + content: + application/json: + schema: + type: object + properties: + data: + type: array + items: + $ref: '#/components/schemas/AIGatewayVault' + meta: + $ref: '#/components/schemas/CursorMeta' + required: + - data + - meta + CreateAIGatewayVaultResponse: + description: AI Gateway Vault created successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayVault' + GetAIGatewayVaultResponse: + description: A successful response returning an AI Gateway Vault. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayVault' + UpdateAIGatewayVaultResponse: + description: Vault updated successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayVault' + CreateAIGatewayPolicyResponse: + description: Policy created successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayPolicy' + GetAIGatewayPolicyResponse: + description: A successful response returning the AI Gateway Policy. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayPolicy' + UpdateAIGatewayPolicyResponse: + description: Policy updated successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayPolicy' + ListAIGatewayPoliciesResponse: + description: A successful response listing AI Gateway Policies. + content: + application/json: + schema: + type: object + properties: + data: + type: array + items: + $ref: '#/components/schemas/AIGatewayPolicy' + meta: + $ref: '#/components/schemas/CursorMeta' + required: + - data + - meta + ListAIGatewayMCPServersResponse: + description: A paginated list of MCP Servers. + content: + application/json: + schema: + type: object + properties: + data: + type: array + items: + $ref: '#/components/schemas/AIGatewayMCPServer' + meta: + $ref: '#/components/schemas/CursorMeta' + required: + - data + - meta + CreateAIGatewayMCPServerResponse: + description: MCP Server created successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayMCPServer' + GetMCPServerResponse: + description: A successful response returning an MCP Server. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayMCPServer' + UpdateAIGatewayMCPServerResponse: + description: MCP Server updated successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayMCPServer' + ListAIGatewayModelProvidersResponse: + description: A paginated list of AI Gateway Model Providers. + content: + application/json: + schema: + type: object + properties: + data: + type: array + items: + $ref: '#/components/schemas/AIGatewayModelProvider' + meta: + $ref: '#/components/schemas/CursorMeta' + required: + - data + - meta + ListAIGatewayAgentsResponse: + description: A paginated list of AI Agents. + content: + application/json: + schema: + type: object + properties: + data: + type: array + items: + $ref: '#/components/schemas/AIGatewayAgent' + meta: + $ref: '#/components/schemas/CursorMeta' + required: + - data + - meta + CreateAIGatewayAgentResponse: + description: AI Agent created successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayAgent' + GetAIGatewayAgentResponse: + description: A successful response returning an AI Gateway Agent. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayAgent' + UpdateAIGatewayAgentResponse: + description: Agent updated successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayAgent' + ListAIGatewayConsumersResponse: + description: A paginated list of AI Gateway Consumers. + content: + application/json: + schema: + type: object + properties: + data: + type: array + items: + $ref: '#/components/schemas/AIGatewayConsumer' + meta: + $ref: '#/components/schemas/CursorMeta' + required: + - data + - meta + CreateAIGatewayConsumerResponse: + description: AI Gateway Consumer created successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayConsumer' + GetAIGatewayConsumerResponse: + description: A successful response returning an AI Gateway Consumer. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayConsumer' + UpdateAIGatewayConsumerResponse: + description: Consumer updated successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayConsumer' + ListAIGatewayConsumerCredentialsResponse: + description: A paginated list of AI Gateway Consumer credentials. + content: + application/json: + schema: + type: object + properties: + data: + type: array + items: + $ref: '#/components/schemas/AIGatewayConsumerCredential' + meta: + $ref: '#/components/schemas/CursorMeta' + required: + - data + - meta + CreateAIGatewayConsumerCredentialResponse: + description: AI Gateway Consumer credential created successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayConsumerCredentialWithKey' + GetAIGatewayConsumerCredentialResponse: + description: A successful response returning an AI Gateway Consumer credential. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayConsumerCredential' + ListAIGatewayConsumerGroupsResponse: + description: A paginated list of AI Gateway Consumer Groups. + content: + application/json: + schema: + type: object + properties: + data: + type: array + items: + $ref: '#/components/schemas/AIGatewayConsumerGroup' + meta: + $ref: '#/components/schemas/CursorMeta' + required: + - data + - meta + CreateAIGatewayConsumerGroupResponse: + description: AI Gateway Consumer Group created successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayConsumerGroup' + GetAIGatewayConsumerGroupResponse: + description: A successful response returning an AI Gateway Consumer Group. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayConsumerGroup' + UpdateAIGatewayConsumerGroupResponse: + description: Consumer Group updated successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayConsumerGroup' + AddAIGatewayConsumerToGroupResponse: + description: Consumer added to consumer group successfully. + content: + application/json: + schema: + type: object + properties: + consumer: + $ref: '#/components/schemas/AIGatewayConsumer' + consumer_group: + $ref: '#/components/schemas/AIGatewayConsumerGroup' + required: + - consumer + - consumer_group + ListAIGatewayConfigStoresResponse: + description: A paginated list of AI Gateway Config Stores. + content: + application/json: + schema: + type: object + properties: + data: + type: array + items: + $ref: '#/components/schemas/AIGatewayConfigStore' + meta: + $ref: '#/components/schemas/CursorMeta' + required: + - data + - meta + CreateAIGatewayConfigStoreResponse: + description: AI Gateway Config Store created successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayConfigStore' + GetAIGatewayConfigStoreResponse: + description: A successful response returning an AI Gateway Config Store. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayConfigStore' + UpdateAIGatewayConfigStoreResponse: + description: Config Store updated successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayConfigStore' + ListAIGatewayConfigStoreSecretsResponse: + description: A paginated list of AI Gateway Config Store Secrets. + content: + application/json: + schema: + type: object + properties: + data: + type: array + items: + $ref: '#/components/schemas/AIGatewayConfigStoreSecret' + meta: + $ref: '#/components/schemas/CursorMeta' + required: + - data + - meta + CreateAIGatewayConfigStoreSecretResponse: + description: AI Gateway Config Store Secret created successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayConfigStoreSecret' + GetAIGatewayConfigStoreSecretResponse: + description: A successful response returning an AI Gateway Config Store Secret. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayConfigStoreSecret' + UpdateAIGatewayConfigStoreSecretResponse: + description: Config Store Secret updated successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayConfigStoreSecret' + CreateAIGatewayModelProviderResponse: + description: Model provider created successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayModelProvider' + GetAIGatewayModelProviderResponse: + description: A successful response returning an AI Gateway Model Provider. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayModelProvider' + UpdateAIGatewayModelProviderResponse: + description: Model provider updated successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayModelProvider' + CreateAIGatewayIdentityProviderResponse: + description: Identity Provider created successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayIdentityProvider' + GetAIGatewayIdentityProviderResponse: + description: A successful response returning an AI Gateway Identity Provider. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayIdentityProvider' + UpdateAIGatewayIdentityProviderResponse: + description: Identity provider updated successfully. + content: + application/json: + schema: + $ref: '#/components/schemas/AIGatewayIdentityProvider' + ListAIGatewayIdentityProvidersResponse: + description: A paginated list of AI Gateway Identity Providers. + content: + application/json: + schema: + type: object + properties: + data: + type: array + items: + $ref: '#/components/schemas/AIGatewayIdentityProvider' + meta: + $ref: '#/components/schemas/CursorMeta' + required: + - data + - meta + Unauthorized: + description: Unauthorized + content: + application/problem+json: + schema: + $ref: '#/components/schemas/UnauthorizedError' + examples: + UnauthorizedExample: + $ref: '#/components/examples/UnauthorizedExample' + Forbidden: + description: Forbidden + content: + application/problem+json: + schema: + $ref: '#/components/schemas/ForbiddenError' + examples: + UnauthorizedExample: + $ref: '#/components/examples/ForbiddenExample' + TooManyRequests: + description: Too Many Requests + content: + application/problem+json: + schema: + $ref: '#/components/schemas/TooManyRequestsError' + BadRequest: + description: Bad Request + content: + application/problem+json: + schema: + $ref: '#/components/schemas/BadRequestError' + Conflict: + description: Conflict + content: + application/problem+json: + schema: + $ref: '#/components/schemas/ConflictError' + NotFound: + description: Not Found + content: + application/problem+json: + schema: + $ref: '#/components/schemas/NotFoundError' + examples: + NotFoundExample: + $ref: '#/components/examples/NotFoundExample' + securitySchemes: + konnectAccessToken: + type: http + scheme: bearer + bearerFormat: JWT + description: | + The Konnect access token is meant to be used by the Konnect dashboard and the decK CLI authenticate with. +tags: + - name: AI Gateways + description: API related to the management of Konnect AI Gateway resources. + - name: AI Gateway DataPlane Certificates + description: API related to the management of AI Gateway DataPlane Certificates. + - name: AI Gateway Nodes + description: API related to the management of AI Gateway nodes. + - name: AI Gateway Vaults + description: API related to the management of AI Gateway vaults for storing secrets. + - name: AI Gateway Agents + description: AI Agents registered with the AI Gateway. + - name: AI Gateway Consumers + description: Individual consumers with credentials and group memberships for AI Gateway access control. + - name: AI Gateway Consumer Groups + description: Consumer groups for applying rate-limiting and access policies to AI Gateway traffic. + - name: AI Gateway Identity Providers + description: Identity providers for authenticating users and accessing AI Gateway resources. + - name: AI Gateway Models + description: 'Models that define routing, capabilities, and backend targets for the AI Gateway.' + - name: AI Gateway Policies + description: 'Policies that control security, rate-limiting, and guardrail behavior for the AI Gateway.' + - name: AI Gateway MCP Servers + description: MCP Servers that expose tools for AI Gateway integrations. + - name: AI Gateway Model Providers + description: Model providers that define the backend AI service connections for the AI Gateway. +security: + - konnectAccessToken: [] diff --git a/app/_ai_gateway_entities/ai-agent.md b/app/_ai_gateway_entities/ai-agent.md new file mode 100644 index 00000000000..9344ee3e533 --- /dev/null +++ b/app/_ai_gateway_entities/ai-agent.md @@ -0,0 +1,358 @@ +--- +title: AI Agents +content_type: reference +entities: + - ai-agent +products: + - ai-gateway +min_version: + ai-gateway: '2.0' +permalink: /ai-gateway/entities/ai-agent/ +breadcrumbs: + - /ai-gateway/ + - /ai-gateway/entities/ +description: AI Agent entity used by {{site.ai_gateway}} for A2A and HTTP agent configurations. +schema: + api: konnect/ai-gateway + path: /schemas/AIGatewayAgent +works_on: + - konnect +tools: + - konnect-api + - kongctl +related_resources: + - text: About {{site.ai_gateway}} + url: /ai-gateway/ + - text: "{{site.ai_gateway}} entities" + url: /ai-gateway/entities/ + - text: AI Policy entity + url: /ai-gateway/entities/ai-policy/ + - text: AI Consumer Group entity + url: /ai-gateway/entities/ai-consumer-group/ + - text: A2A protocol specification + url: https://a2aproject.github.io/A2A/ +faqs: + - q: What's the difference between an `a2a` AI Agent and an `http` AI Agent? + a: | + An `a2a` AI Agent applies Agent-to-Agent protocol awareness (JSON-RPC and REST binding detection, + agent-card URL rewriting, structured A2A telemetry) to traffic flowing to an upstream agent. + An `http` AI Agent is a generic HTTP route to an upstream agent without A2A-specific processing. + Use `a2a` when the upstream speaks the A2A protocol and you want observability tied to A2A + task and message semantics. + + - q: Does the AI Agent entity modify request routing or aggregate responses? + a: | + No. The runtime behind an AI Agent operates as a transparent proxy. It detects A2A requests, + records telemetry, and rewrites agent-card URLs to the gateway address. It does not change + routing decisions, merge responses, or hold task state on behalf of clients. + + - q: Why is the agent-card URL rewritten? + a: | + A2A clients use agent-card responses (at `/.well-known/agent-card.json`) to discover where to + send subsequent requests. Rewriting the [`url`](#schema-aigateway-agent-config-url) field, and any `additionalInterfaces[].url` + fields on the agent card response, to the {{site.ai_gateway}} address means clients route follow-up traffic through the + gateway instead of bypassing it. The rewrite honors `X-Forwarded-*` headers when the gateway + sits behind a load balancer. + + - q: How does streaming work? + a: | + Server-sent events (`Content-Type: text/event-stream`) pass through chunk-by-chunk without + buffering. The runtime counts SSE events, captures time-to-first-byte, and extracts task state + from the final event for analytics. Latency is preserved. + + - q: How do I limit which AI Consumers can reach an AI Agent? + a: | + Set the [`access.acls`](#schema-aigateway-agent-access) field on the AI Agent with an allow list or a deny list. Each entry is a string that + references an AI Consumer, AI Consumer Group, or Authenticated Group by name. + + - q: How do I attach AI Policies to an AI Agent? + a: | + Configuration that applies to the AI Agent goes through the [AI Policy entity](/ai-gateway/entities/ai-policy/). + Attach AI Policies to the AI Agent through its [`policies`](#schema-aigateway-agent-policies) field. +--- + +## What is an AI Agent? + +When you want to centrally manage agent routing, control access, and gain observability over agent traffic, use the AI Agent entity to expose upstream agents through {{site.ai_gateway}}. {{site.ai_gateway}}: + +- Acts as a central point of contact for A2A clients +- Rewrites agent-card URLs so clients route through the gateway (not directly to agents) +- Enforces access controls via Access Control Lists (ACLs) +- Emits structured telemetry tied to agent operations. + +The AI Agent entity supports two types: `a2a` for AI Agents that speak the [Agent-to-Agent protocol](https://a2aproject.github.io/A2A/), and `http` for standard HTTP AI Agents. See the [AI Agent types](#ai-agent-types) section for protocol-specific behavior and configuration guidance. + +## Manage AI Agents + +AI Agents can be created and managed through: + +* {{site.konnect_short_name}} UI +* {{site.ai_gateway}} API: `/v1/ai-gateways/{aiGatewayId}/agents` +* [kongctl](/kongctl/) + +For configuration examples and step-by-step setup instructions, see the following [Set up an AI Agent](#set-up-an-ai-agent) section. + +## AI Agent types + +Choose an AI Agent type based on your upstream and observability needs. The [`type`](#schema-aigateway-agent-type) controls how requests are processed: + + +{% table %} +columns: + - title: Type + key: type + - title: Use case + key: use_case +rows: + - type: "`a2a`" + use_case: "Agents that speak the [Agent-to-Agent protocol](https://a2aproject.github.io/A2A/). {{site.ai_gateway}} applies protocol awareness, detects A2A requests (JSON-RPC and REST bindings), rewrites agent-card URLs to the gateway address, emits structured A2A telemetry, and extracts task metadata for analytics. Use when you want full observability tied to A2A semantics." + - type: "`http`" + use_case: "Standard HTTP agent endpoints. Requests pass through transparently as a generic HTTP proxy without A2A-specific processing. Use for upstream agents that don't implement A2A or when you need simple transparent proxying without protocol-aware behavior." +{% endtable %} + + +## Use cases for AI Agents + +Common use cases for exposing agents through {{site.ai_gateway}}: + + +{% table %} +columns: + - title: Use case + key: use_case + - title: Description + key: description +rows: + - use_case: "Observability and telemetry" + description: "Emit structured A2A telemetry and extract task metadata for analytics. Track agent performance, request patterns, and error rates tied to A2A task semantics. Use for production agent deployments where visibility into agent traffic is critical. See [Logging and observability](#logging-and-observability) for details on telemetry collection and OpenTelemetry integration." + - use_case: "Authentication and access control" + description: "Require agents to authenticate clients via [OpenID Connect](/ai-gateway/policies/openid-connect/) or other auth policies before routing requests. Restrict which [AI Consumers](/ai-gateway/entities/ai-consumer/) or [AI Consumer Groups](/ai-gateway/entities/ai-consumer-group/) can reach specific agents via ACLs." + - use_case: "Rate limiting" + description: "Enforce per-agent or per-consumer rate limits to prevent overload and manage agent resource usage. Use [AI Rate Limiting Advanced](/ai-gateway/policies/ai-rate-limiting-advanced/) to set token or request quotas per consumer." + - use_case: "Policy enforcement" + description: "Attach [AI Policies](/ai-gateway/entities/ai-policy/) to agents for request transformation, PII detection, input validation, and request logging. Layer security and governance controls on agent traffic." + - use_case: "Centralized discovery" + description: "Provide A2A clients with a single, stable gateway endpoint (via agent-card URL rewriting) instead of having them discover and connect directly to agent instances." +{% endtable %} + + +## How A2A traffic flows + +When an Agent has type `a2a`, proxied traffic is processed in four phases: + +1. **Access**. Detects whether the request is an A2A operation (JSON-RPC or REST binding). When statistics logging is enabled, this starts an OpenTelemetry span and records the request body for payload logging if that's also enabled. +1. **Header filter**. Detects streaming responses (`Content-Type: text/event-stream`) and records time to first byte. Buffers agent-card responses for URL rewriting. +1. **Body filter**. Streams SSE chunks through to the client without buffering. Buffers non-streaming responses to extract task metadata. Rewrites agent-card URLs to the gateway address. Emits analytics at end of response. +1. **Log**. Finalizes the OpenTelemetry span with task state, task ID, and any error information. + +Non-A2A traffic, and traffic to `http` Agents, is proxied without these steps. + +## Routing configuration + +Beyond the [`url`](#schema-aigateway-agent-config-url) field, AI Agents can define HTTP routing rules through [`config.route`](#schema-aigateway-agent-config-route). This allows you to match requests by method, path, host, and other HTTP patterns. Use [`route`](#schema-aigateway-agent-config-route) when you need fine-grained control over which traffic reaches the AI Agent. If only a URL is needed, the [`url`](#schema-aigateway-agent-config-url) field is simpler. + + +{% mermaid %} +sequenceDiagram + autonumber + participant Client as A2A Client + participant Gateway as {{site.ai_gateway}}
(Agent) + participant Agent as Upstream A2A Agent + + Client->>Gateway: A2A request (JSON-RPC or REST) + Note over Gateway: Detect A2A binding and method
Start OTel span (if logging enabled) + + Gateway->>Agent: Proxied request
(Accept-Encoding removed if logging enabled) + + alt Streaming response (SSE) + Agent-->>Gateway: text/event-stream chunks + Note over Gateway: Pass through each chunk
Count SSE events, track TTFB + Gateway-->>Client: SSE chunks (unchanged) + Note over Gateway: On final chunk:
Extract task state, set analytics + else Non-streaming response + Agent->>Gateway: JSON response + Note over Gateway: Buffer response
Extract task metadata + Gateway->>Client: Response (unchanged) + end + + Note over Gateway: Finish OTel span
Emit ai.a2a metrics to log plugins +{% endmermaid %} + + +## Core A2A protocol elements + +A2A defines the communication elements between agents. The {{site.ai_gateway}} runtime surfaces data tied to these elements in log output and OpenTelemetry spans for `a2a` Agents. + +{% table %} +columns: + - title: Element + key: element + - title: Description + key: description + - title: Purpose + key: purpose +rows: + - element: Agent Card + description: A JSON metadata document describing an agent's identity, capabilities, endpoint, skills, and authentication requirements. + purpose: Enables clients to discover agents and understand how to interact with them. + - element: Task + description: A stateful unit of work initiated by an agent, with a unique ID and defined lifecycle. + purpose: Tracks long-running operations and supports multi-turn interactions. + - element: Message + description: A single turn of communication between a client and an agent, containing content and a role (`user` or `agent`). + purpose: Conveys instructions, context, questions, answers, or status updates that are not formal artifacts. + - element: Part + description: The fundamental content container (for example, `TextPart`, `FilePart`, `DataPart`) used within messages and artifacts. + purpose: Provides flexibility for agents to exchange different content types within messages and artifacts. + - element: Artifact + description: A tangible output generated by an agent during a task (for example, a document, image, or structured data). + purpose: Carries the concrete output of a task in a structured, retrievable form. +{% endtable %} + +### Protocol detection + +A2A traffic is auto-detected per request and non-A2A traffic passes through without overhead. + +#### REST binding + +Detection anchors to the end of the request path, so any prefix added by the route is ignored. For example, both `/v1/message:send` and `/api/agents/v1/message:send` match `SendMessage`: + + +{% table %} +columns: + - title: HTTP method + key: method + - title: Path suffix + key: path + - title: A2A operation + key: operation + - title: Canonical method + key: canonical +rows: + - method: "`POST`" + path: "`/v1/message:send`" + operation: SendMessage + canonical: "`message/send`" + - method: "`POST`" + path: "`/v1/message:stream`" + operation: SendStreamingMessage + canonical: "`message/stream`" + - method: "`GET`" + path: "`/.well-known/agent-card.json`" + operation: GetAgentCard + canonical: "`agent/getCard`" + - method: "`GET`" + path: "`/v1/extendedAgentCard`" + operation: GetExtendedAgentCard + canonical: "`agent/getExtendedAgentCard`" + - method: "`GET`" + path: "`/v1/tasks/{id}`" + operation: GetTask + canonical: "`tasks/get`" + - method: "`GET`" + path: "`/v1/tasks`" + operation: ListTasks + canonical: "`tasks/list`" + - method: "`POST`" + path: "`/v1/tasks/{id}:cancel`" + operation: CancelTask + canonical: "`tasks/cancel`" + - method: "`POST`" + path: "`/v1/tasks/{id}:subscribe`" + operation: SubscribeToTask + canonical: "`tasks/resubscribe`" + - method: "`POST`" + path: "`/v1/tasks`" + operation: ListTasks + canonical: "`tasks/list`" +{% endtable %} + + +The canonical method name is what appears in OpenTelemetry span attributes and log output. + +#### JSON-RPC binding + +Detected by the `"jsonrpc"` field in the request body, combined with a recognized A2A method name or an `A2A-Version` request header. Recognized methods include `message/send`, `message/stream`, `tasks/get`, `tasks/list`, `tasks/cancel`, `tasks/resubscribe`, the `tasks/pushNotificationConfig/*` family, and `agent/getExtendedAgentCard`. + +A request carrying an `A2A-Version` header is treated as JSON-RPC even if the method isn't in the recognized list. When an unknown method is accepted this way, the `method` field in log output is recorded as `"unknown"` to bound metric cardinality. The OpenTelemetry span's `kong.a2a.operation` attribute still receives the actual method name. + +### Agent-card URL rewriting + +When an upstream agent returns an agent card, the runtime rewrites the [`url`](#schema-aigateway-agent-config-url) field, and any `additionalInterfaces[].url` fields, to the {{site.ai_gateway}} address. A2A clients then discover the gateway as the canonical endpoint instead of contacting the upstream directly. The rewrite uses `X-Forwarded-*` headers to construct the correct scheme, host, and port when the gateway is deployed behind a load balancer or reverse proxy. + +## Logging and observability + +To track agent performance, debug issues, and monitor A2A traffic patterns, enable statistics logging. {{site.ai_gateway}} emits structured A2A telemetry that flows to {{site.konnect_short_name}} analytics, logging plugins, and OpenTelemetry for full visibility into agent operations. + +The telemetry data is emitted into the `ai.a2a` namespace (consumed by {{site.konnect_short_name}} analytics and logging AI Policies) and creates a `kong.a2a` child span when you've configured [{{site.base_gateway}} tracing](/gateway/tracing/). For the canonical metric and attribute list, see [A2A metrics](/ai-gateway/ai-otel-metrics/#a2a-metrics). + +{:.info} +> When statistics logging is enabled, the runtime removes the `Accept-Encoding` request header +> before forwarding to the upstream. This prevents compressed responses that the runtime can't +> parse for metadata extraction. + +Payload logging additionally captures request and response bodies. Payloads are truncated at the configured payload size limit. + +{:.warning} +> Payload logging may expose sensitive data. Only enable it when you're prepared to handle +> request and response bodies in your logging pipeline. + +You can view A2A analytics in {{site.konnect_short_name}} Explorer and Dashboards through the [Agentic usage analytics](/observability/explorer/?tab=agentic-usage#metrics) view. + +### Log output fields + +{% include md/ai-gateway/v2/log-output-fields.md %} + +### OpenTelemetry span attributes + +When statistics logging is enabled and {{site.base_gateway}} tracing is configured, the runtime creates a `kong.a2a` child span with the following attributes: + +{% include md/ai-gateway/v2/otel-span-attributes.md %} + +## Access control + +To restrict which AI Consumers or teams can reach a specific agent, use ACLs. The [`access.acls`](#schema-aigateway-agent-access) field defines either an `allow` or a `deny` list of identities that can access the agent. Each entry references an [AI Consumer](/ai-gateway/entities/ai-consumer/), [AI Consumer Group](/ai-gateway/entities/ai-consumer-group/), or Authenticated Group by name. An Authenticated Group is a dynamic group representing all consumers authenticated via a specific OAuth2 scope or claim. Access is enforced before traffic reaches the upstream agent. + +For per-request authentication and identity validation, attach an authentication AI Policy to the AI Agent. + +## Attach AI Policies + +To enforce security, transformation, or governance controls on agent traffic (for example, request validation, PII detection, request logging), attach [AI Policies](/ai-gateway/entities/ai-policy/) to the agent. Add policy names or IDs to the AI Agent's [`policies`](#schema-aigateway-agent-policies) field. Multiple AI Policies can attach to one AI Agent; each runs independently in the request lifecycle. + +For available policy types and configuration, see the [AI Policy entity](/ai-gateway/entities/ai-policy/) reference. + +## Set up an Agent + +Before creating an AI Agent with access restrictions, create an AI Consumer Group to reference in [`access.acls`](#schema-aigateway-agent-access). +This example references a group named `internal-teams`. See [Set up an AI Consumer Group](/ai-gateway/entities/ai-consumer-group/#set-up-an-ai-consumer-group) to create it, or substitute the name of your own AI Consumer, AI Consumer Group, or Authenticated Group in `access.acls.allow`. + +The following example creates an `a2a` Agent that proxies traffic to an upstream A2A agent at `https://booking-agent.internal.kongair.com`, with statistics logging enabled and access restricted to the `internal-teams` Consumer Group. + +{:.info} +> This example proxies to a placeholder upstream at `https://booking-agent.internal.kongair.com`. Substitute the URL of your own running A2A agent in [`config.url`](#schema-aigateway-agent-config-url). Because this Agent has `type: a2a`, requests must use the A2A JSON-RPC envelope (`jsonrpc: "2.0"`, `id`, `method: "message/send"`, `params.message` with `kind` and `messageId`). A flat `{"message": {...}}` body without that envelope is rejected by the upstream agent itself (for example, `"Invalid Request: jsonrpc must be 2.0"`), not by {{site.ai_gateway}}. + +{% entity_example %} +type: agent +data: + display_name: KongAir Flight Booking Agent + name: kongair-flight-booking-agent + type: a2a + access: + acls: + allow: + - internal-teams + policies: [] + config: + url: https://booking-agent.internal.kongair.com + route: + paths: + - /kongair-flight-booking + logging: + statistics: true + payloads: false + max_payload_size: 1048576 +{% endentity_example %} + +## Schema + +{% entity_schema %} diff --git a/app/_ai_gateway_entities/ai-consumer-group.md b/app/_ai_gateway_entities/ai-consumer-group.md new file mode 100644 index 00000000000..75608616ebe --- /dev/null +++ b/app/_ai_gateway_entities/ai-consumer-group.md @@ -0,0 +1,176 @@ +--- +title: AI Consumer Groups +content_type: reference +entities: + - ai-consumer-group +products: + - ai-gateway +min_version: + ai-gateway: '2.0' +permalink: /ai-gateway/entities/ai-consumer-group/ +breadcrumbs: + - /ai-gateway/ + - /ai-gateway/entities/ +description: AI Consumer Groups for {{site.ai_gateway}}. +schema: + api: konnect/ai-gateway + path: /schemas/AIGatewayConsumerGroup +works_on: + - konnect +tools: + - konnect-api + - kongctl +related_resources: + - text: "About {{site.ai_gateway}}" + url: /ai-gateway/ + - text: AI Consumer entity + url: /ai-gateway/entities/ai-consumer/ + - text: AI Model entity + url: /ai-gateway/entities/ai-model/ + - text: AI Policy entity + url: /ai-gateway/entities/ai-policy/ + - text: "{{site.base_gateway}} Consumer Group entity" + url: /gateway/entities/consumer-group/ +faqs: + - q: How is an AI Consumer Group different from a {{site.base_gateway}} Consumer Group? + a: | + The {{site.ai_gateway}} surface adds + the entity convention ([`display_name`](#schema-aigateway-consumer-group-display-name), [`name`](#schema-aigateway-consumer-group-name), [`labels`](#schema-aigateway-consumer-group-labels)) and a required [`policies`](#schema-aigateway-consumer-group-policies) array + for attaching AI Policies at the group scope. + + - q: Can I edit the underlying Kong Consumer Group that {{site.ai_gateway}} generates? + a: | + No. The generated Kong Consumer Group is protected from direct modification through the + standard `/consumer-groups` Admin API. Update the AI Consumer Group instead. + + - q: How do I assign an AI Consumer to an AI Consumer Group? + a: | + Either add the AI Consumer through the AI Consumer Group's `consumers` sub-resource + (`POST /ai-gateways/{aiGatewayId}/consumer-groups/{consumerGroupId}/consumers`), or set the + AI Consumer's group membership directly + (`PUT /ai-gateways/{aiGatewayId}/consumers/{consumerIdOrName}/consumer-groups`). + These aren't fields on the AI Consumer or AI Consumer Group entity bodies themselves. + They're managed through these dedicated endpoints. + + - q: Can an AI Consumer belong to multiple AI Consumer Groups? + a: | + Yes. The `consumer_groups` list accepted by + `PUT /ai-gateways/{aiGatewayId}/consumers/{consumerIdOrName}/consumer-groups` can include + more than one AI Consumer Group name. + + - q: How do I attach AI Policies to an AI Consumer Group? + a: | + Add the AI Policy's `name` or `id` to the AI Consumer Group's [`policies`](#schema-aigateway-consumer-group-policies) array. + The AI Policy runs when a member of the group is identified during a request. + See the [AI Policy entity](/ai-gateway/entities/ai-policy/) reference. + + - q: How do I gate access to an AI Model, AI Agent, or AI MCP Server with an AI Consumer Group? + a: | + Add the AI Consumer Group's name to the parent entity's `access.acls.allow` or `access.acls.deny` list. + For AI Models and AI Agents, configure exactly one of `allow` or `deny`. For AI MCP Servers, both can be set simultaneously. + ACLs accept AI Consumer, AI Consumer Group, and Authenticated Group names. + See the [AI Model entity](/ai-gateway/entities/ai-model/) reference. +--- + +## What is an AI Consumer Group? + +An AI Consumer Group is the {{site.ai_gateway}} entity that represents a collection of AI Consumers grouped for the purpose of applying shared [AI Policies](/ai-gateway/entities/ai-policy/) and access controls. + +By grouping AI Consumers together, you eliminate the need to manage AI Policies and access controls individually, providing a scalable, efficient approach to AI governance. With AI Consumer Groups, you can scope AI Policies to specifically defined groups, making configurations and customizations more flexible. + +For example, you could define three groups (Bronze, Gold, and Enterprise) and attach an [AI Rate Limiting Advanced](/ai-gateway/policies/ai-rate-limiting-advanced/) policy to each with different token quotas and cost budgets. Without AI Consumer Groups, you would attach a separate AI Rate Limiting Advanced policy to each individual AI Consumer. In production, that could be thousands of individual policy attachments instead of three group-level ones. + + +{% mermaid %} +flowchart LR + A((AI Consumers 1-5)) + + B("AI Consumer Group Gold
AI Consumer 1, AI Consumer 2, AI Consumer 5") + + C("AI Consumer Group Bronze
AI Consumer 3, AI Consumer 4") + + D["AI Rate Limiting Advanced
1M tokens/hour
AND
$100/hour budget"] + E["AI Rate Limiting Advanced
100K tokens/hour
AND
$10/hour budget"] + F("AI Model
GPT-4") + H["OpenAI
Service"] + + A--> B & C + subgraph id1 ["AI Gateway"] + direction LR + B --> D --> F + C --> E --> F + end + + F --> H +{% endmermaid %} + + +## Manage AI Consumer Groups + +AI Consumer Groups can be created and managed through: + +* {{site.konnect_short_name}} UI +* {{site.ai_gateway}} API: `/v1/ai-gateways/{aiGatewayId}/consumer-groups` +* [kongctl](/kongctl/) + +For configuration examples and step-by-step setup instructions, see [Set up an AI Consumer Group](#set-up-an-ai-consumer-group). + +## Use cases for using AI Consumer Group + +Common use cases for AI Consumer Groups: + + +{% table %} +columns: + - title: Use case + key: use_case + - title: Description + key: description +rows: + - use_case: "Subscription tier management" + description: "Create AI Consumer Groups for different subscription tiers (for example, Bronze, Gold, Enterprise). Assign different rate limits, model access restrictions, and token quotas to each tier without configuring individual AI Consumers." + - use_case: "Team-based access control" + description: "Organize AI Consumers by team or department. Gate access to specific [AI Models](/ai-gateway/entities/ai-model/), [AI Agents](/ai-gateway/entities/ai-agent/), or [AI MCP Servers](/ai-gateway/entities/ai-mcp-server/) at the group level, so teams only access the resources they need." + - use_case: "AI safety and governance AI Policies" + description: "Apply group-level AI Policies for prompt validation, PII detection, and content filtering. For example, apply stricter guardrails to public-facing groups while allowing more permissive configurations for internal teams. See the [AI Policies hub](/ai-gateway/policies/) for available policy types." + - use_case: "Cost and quota management" + description: "Enforce per-group token limits, rate limits, and usage quotas. Track spending and resource usage by AI Consumer Group to manage AI API costs at scale." + - use_case: "Centralized AI Policy management" + description: "Attach AI Policies once at the group level rather than managing them on every individual AI Consumer. Simplifies configuration and ensures consistency across all group members." +{% endtable %} + + +## Membership + +To organize AI Consumers by team, department, or tier, add them to an AI Consumer Group. Membership isn't a field on either entity's body. Manage it through dedicated sub-resource endpoints: add a Consumer to a group with `POST /ai-gateways/{aiGatewayId}/consumer-groups/{consumerGroupId}/consumers`, or set the full list of groups a Consumer belongs to with `PUT /ai-gateways/{aiGatewayId}/consumers/{consumerIdOrName}/consumer-groups`. A single AI Consumer can belong to multiple AI Consumer Groups, allowing flexible organizational schemes. + +## Attach AI Policies + +To apply the same AI Policies (rate limits, prompt validation, PII detection) to multiple consumers at once, attach them to the AI Consumer Group. When a member of the group makes a request, {{site.ai_gateway}} applies all attached AI Policies before routing the request. Add an AI Policy's `name` or `id` to the AI Consumer Group's [`policies`](#schema-aigateway-consumer-group-policies) array. + +You can attach multiple AI Policies to a single AI Consumer Group with different configurations, and each runs independently. For supported policy types and how AI Policies attach to other entities, see the [AI Policy entity](/ai-gateway/entities/ai-policy/) reference. + +## Use in parent entity ACLs + +To restrict access to specific [AI Models](/ai-gateway/entities/ai-model/), [AI Agents](/ai-gateway/entities/ai-agent/), or [AI MCP Servers](/ai-gateway/entities/ai-mcp-server/) by AI Consumer Group (for example, allowing only Gold tier AI Consumers to access premium models), use ACLs. The `access.acls` field on these entities accepts AI Consumer Group names alongside AI Consumer and Authenticated Group names. For AI Models and AI Agents, configure exactly one of `access.acls.allow` to permit access or `access.acls.deny` to block it. You can't set both at the same time. For AI MCP Servers, `access.acls` supports setting `allow`, `deny`, or both. + +AI Consumer Group membership is resolved after the request is authenticated and the AI Consumer is identified. + +## Set up an AI Consumer Group + +The following example creates an AI Consumer Group. You can attach AI Policies through the {{site.konnect_short_name}} UI or by adding their `name` or `id` to the `policies` array. + +{% entity_example %} +type: consumer_group +data: + display_name: Internal Teams + name: internal-teams + policies: [] +{% endentity_example %} + +{:.info} +> This creates an empty AI Consumer Group with no members. To add AI Consumers to it, see [Membership](#membership). There's no `consumers` field on the AI Consumer Group itself; membership is set through a separate endpoint. + +## Schema + +{% entity_schema %} diff --git a/app/_ai_gateway_entities/ai-consumer.md b/app/_ai_gateway_entities/ai-consumer.md new file mode 100644 index 00000000000..c3242b85ad4 --- /dev/null +++ b/app/_ai_gateway_entities/ai-consumer.md @@ -0,0 +1,203 @@ +--- +title: AI Consumers +content_type: reference +entities: + - ai-consumer +products: + - ai-gateway +min_version: + ai-gateway: '2.0' +permalink: /ai-gateway/entities/ai-consumer/ +breadcrumbs: + - /ai-gateway/ + - /ai-gateway/entities/ +description: "AI Consumers for {{site.ai_gateway}}." +schema: + api: konnect/ai-gateway + path: /schemas/AIGatewayConsumer +works_on: + - konnect +tools: + - konnect-api + - kongctl +related_resources: + - text: "About {{site.ai_gateway}}" + url: /ai-gateway/ + - text: AI Consumer Group entity + url: /ai-gateway/entities/ai-consumer-group/ + - text: AI Model entity + url: /ai-gateway/entities/ai-model/ + - text: AI Policy entity + url: /ai-gateway/entities/ai-policy/ + - text: "{{site.base_gateway}} Consumer entity" + url: /gateway/entities/consumer/ +faqs: + - q: How is an AI Consumer different from a {{site.base_gateway}} Consumer? + a: | + The runtime entity is a regular Kong Consumer. The {{site.ai_gateway}} surface uses the + {{site.ai_gateway}} entity convention ([`display_name`](#schema-aigateway-consumer-display-name), [`name`](#schema-aigateway-consumer-name), [`labels`](#schema-aigateway-consumer-labels)), requires an + authentication [`type`](#schema-aigateway-consumer-type) field, accepts inline AI Consumer Group assignment, and lets you + reference AI Policies. + + - q: How do I add credentials to an AI Consumer? + a: | + For `type: api-key` AI Consumers, credentials are managed through a separate credentials + endpoint, not as a field on the Consumer. Create them via POST to `/consumers/{id}/credentials`. + `type: oauth` AI Consumers don't use this endpoint. See the next question. + + - q: "What's the difference between `type: api-key` and `type: oauth`?" + a: | + The `type` declares how the AI Consumer authenticates. An `api-key` AI Consumer holds one or + more `api-key` Credentials created through the credentials endpoint. An `oauth` AI Consumer + has no Credentials. Instead, its own `custom_id` field is set (at creation or update time) to + the identifier your OIDC provider issues (for example, a `sub` claim), and an authentication + policy maps the incoming token to that AI Consumer. + + - q: Can an AI Consumer belong to multiple AI Consumer Groups? + a: | + Yes. An AI Consumer can be added to multiple AI Consumer Groups through the AI Consumer Group entity. + See the [AI Consumer Group entity](/ai-gateway/entities/ai-consumer-group/) reference. + + - q: How do I attach AI Policies to an AI Consumer? + a: | + Add the Policy's `name` or `id` to the AI Consumer's [`policies`](#schema-aigateway-consumer-policies) array. + See the [AI Policy entity](/ai-gateway/entities/ai-policy/) reference. +--- + +## What is an AI Consumer? + +An AI Consumer is the {{site.ai_gateway}} entity that identifies an external client consuming or using the AI APIs you publish through {{site.ai_gateway}}. Consumers can represent applications, services, or users who interact with your AI Models, AI Agents, and AI MCP Servers. + +AI Consumers are essential for controlling access to your AI APIs, tracking usage, and ensuring security. They are identified through authentication credentials (API keys or OAuth), allowing {{site.ai_gateway}} to authenticate requests and apply Consumer-specific controls. By creating AI Consumers and organizing them into AI Consumer Groups, you can manage access controls at scale, attach AI Policies for governance and security, and monitor token usage per Consumer. + +## Use cases for AI Consumers + +Common use cases for enforcing controls at the AI Consumer level: + + +{% table %} +columns: + - title: Use case + key: use_case + - title: Description + key: description +rows: + - use_case: Model access control + description: Control which clients can access which AI Models, restricting access by team, application tier, or use case. + - use_case: AI safety and guardrails + description: Apply prompt validation, PII detection, and content filtering at the AI Consumer level using AI Policies. + - use_case: Token and cost control + description: Apply per-consumer rate limits and quotas to prevent token overages and control costs by AI Consumer tier. + - use_case: AI request transformation + description: Normalize or transform AI requests and responses per AI Consumer (for example, format prompts, inject system instructions, sanitize outputs). + - use_case: Audit and compliance + description: Track which clients are using which AI Models, monitor for policy violations, and maintain audit logs for compliance and analytics. +{% endtable %} + + +## Manage AI Consumers + +AI Consumers can be created and managed through: + +* {{site.konnect_short_name}} UI +* {{site.ai_gateway}} API: `/v1/ai-gateways/{aiGatewayId}/consumers` +* [kongctl](/kongctl/) + +For configuration examples and step-by-step setup instructions, see [Set up an AI Consumer](#set-up-an-ai-consumer). + +## Authentication type + +Choose an authentication method based on your deployment needs. Set the [`type`](#schema-aigateway-consumer-type) field to declare which credential family AI Consumers will use: + + +{% table %} +columns: + - title: Type + key: type + - title: Use case + key: use_case +rows: + - type: "`api-key`" + use_case: Simple, stateless authentication for internal services or mobile apps using a shared secret. + - type: "`oauth`" + use_case: Federated identity with an external OIDC provider. {{site.ai_gateway}} accepts any standards-compliant OAuth 2.0 / OpenID Connect provider configured through the [OpenID Connect Policy](/ai-gateway/policies/openid-connect/), or for MCP traffic through the [AI MCP OAuth2 Policy](/ai-gateway/policies/ai-mcp-oauth2/). The AI Consumer's own `custom_id` field maps to the OAuth provider's user identifier (for example, an OIDC Client ID or `sub` claim). +{% endtable %} + + +`api-key` AI Consumers authenticate through one or more `api-key` Credentials created via the credentials endpoint. `oauth` AI Consumers don't have Credentials. Set `custom_id` directly on the AI Consumer instead. + +## AI Consumer Group membership + +To apply AI Policies and access controls to multiple AI Consumers at once, organize them into AI Consumer Groups. An AI Consumer can belong to multiple AI Consumer Groups, letting you manage access controls by team, application, or environment without duplicating configurations. + +Manage AI Consumer Group membership through the [AI Consumer Group entity](/ai-gateway/entities/ai-consumer-group/). + +## Attach Policies + +To enforce governance, security, or observability controls at the AI Consumer level, attach AI Policies. When an AI Consumer makes a request, {{site.ai_gateway}} applies any AI Policies attached to that AI Consumer before routing the request. + +Attach an AI Policy by adding its `name` or `id` to the AI Consumer's [`policies`](#schema-aigateway-consumer-policies) array. You can attach multiple AI Policies to a single AI Consumer. Each AI Policy runs independently, allowing you to layer controls for rate limiting, request validation, PII redaction, and other governance needs. + +For supported policy types and how AI Policies attach to other entities, see the [AI Policy entity](/ai-gateway/entities/ai-policy/) reference or browse all available AI Policies in the [AI policies hub](/ai-gateway/policies/). + +## Set up an AI Consumer + +{% navtabs "consumer_type" %} +{% navtab "api-key" %} + +The following example creates an `api-key` AI Consumer. After creating it, add one or more API key Credentials (see [Create Consumer Credentials](#create-consumer-credentials)). + +{:.info} +> Consumer Group membership isn't set on the Consumer itself, as there's no `consumer_groups` field on this request. To add this Consumer to an AI Consumer Group, use the AI Consumer Group entity's `/consumers` endpoint after creation. See the [AI Consumer Group entity](/ai-gateway/entities/ai-consumer-group/) for more information. + +{% entity_example %} +type: consumer +data: + display_name: Mobile App - Production + name: mobile-app-production + type: api-key + policies: [] +{% endentity_example %} + +{% endnavtab %} +{% navtab "oauth" %} + +The following example creates an `oauth` AI Consumer. Set `custom_id` to the identifier your OIDC provider issues (for example, a `sub` claim); this is how {{site.ai_gateway}} maps an incoming token to this AI Consumer. `oauth` AI Consumers don't have Credentials. + +{% entity_example %} +type: consumer +data: + display_name: OAuth User 1 + name: oauth-user-1 + type: oauth + custom_id: user-id-from-oidc-provider + policies: [] +{% endentity_example %} + +{% endnavtab %} +{% endnavtabs %} + +## Create Consumer Credentials + +After creating an `api-key` AI Consumer, create one or more Credentials for authentication. Credentials are managed through a separate endpoint and only support `type: api-key`. `oauth` AI Consumers authenticate through their `custom_id` field instead (see [Set up an AI Consumer](#set-up-an-ai-consumer)). + + +{% konnect_api_request %} +url: /v1/ai-gateways/$AI_GATEWAY_ID/consumers/$CONSUMER_ID/credentials +status_code: 201 +method: POST +headers: + - 'Content-Type: application/json' + - 'Accept: application/json, application/problem+json' +body: + display_name: Mobile App Key 1 + name: mobile-app-key-1 + type: api-key +{% endkonnect_api_request %} + + +The response includes the generated `api_key` value. Store this securely; it cannot be retrieved later. + +## Schema + +{% entity_schema %} diff --git a/app/_ai_gateway_entities/ai-data-plane-certificate.md b/app/_ai_gateway_entities/ai-data-plane-certificate.md new file mode 100644 index 00000000000..40d16c96203 --- /dev/null +++ b/app/_ai_gateway_entities/ai-data-plane-certificate.md @@ -0,0 +1,123 @@ +--- +title: AI Data Plane Certificates +content_type: reference +entities: + - ai-data-plane-certificate +products: + - ai-gateway +min_version: + ai-gateway: '2.0' +permalink: /ai-gateway/entities/ai-data-plane-certificate/ +breadcrumbs: + - /ai-gateway/ + - /ai-gateway/entities/ +description: Client certificates that authorize data planes to connect to an {{site.ai_gateway}}. +schema: + api: konnect/ai-gateway + path: /schemas/AIGatewayDataPlaneClientCertificate +works_on: + - konnect +tools: + - konnect-api +related_resources: + - text: "About {{site.ai_gateway}}" + url: /ai-gateway/ + - text: AI Model Provider entity + url: /ai-gateway/entities/ai-model-provider/ + - text: Vault entity + url: /ai-gateway/entities/ai-vault/ +faqs: + - q: Why is there no update operation? + a: | + The certificate body is immutable once registered. To rotate, register a new Data Plane + Certificate alongside the existing one, roll the data planes onto the new certificate, then + delete the old entry. This pattern avoids a window where no certificate is installed. + + - q: What happens to connected data planes when a certificate is deleted? + a: | + Any data plane currently connecting with the deleted certificate loses its trust anchor and + can no longer establish a connection to the {{site.ai_gateway}}. Roll data planes onto a + replacement certificate before deleting the old one. + + - q: Is the private key stored alongside the certificate? + a: | + No. Only the public certificate is registered with the {{site.ai_gateway}}. The corresponding + private key stays on the data plane and is never sent to {{site.konnect_short_name}}. + + - q: Can the same certificate be used by multiple data planes? + a: | + Yes. Any data plane provisioned with the registered certificate and its private key can + establish a connection. Use multiple certificates when you need to revoke trust for a subset + of data planes independently. + + - q: How does this relate to the {{site.base_gateway}} data plane client certificate? + a: | + It plays the same role, establishing mutual TLS between the control plane and a data plane, + but it is scoped to a single {{site.ai_gateway}} instance and managed through the + {{site.ai_gateway}} entity surface, not the {{site.konnect_short_name}} Gateway control plane API. +--- + +## What is a Data Plane Certificate? + +A Data Plane Certificate is an {{site.ai_gateway}} entity that registers a public X.509 certificate as a trusted client identity for an {{site.ai_gateway}}. Data planes presenting the matching private key during the mTLS handshake are allowed to connect; data planes without a matching registered certificate are rejected. + +Each Data Plane Certificate belongs to exactly one {{site.ai_gateway}}. An {{site.ai_gateway}} can have multiple registered certificates so that you can issue one per data plane fleet, rotate keys without downtime, or revoke trust for a subset of data planes independently. + +Data Plane Certificates are managed through the {{site.konnect_short_name}} {{site.ai_gateway}} API, the {{site.konnect_short_name}} UI, or Terraform: + +{% table %} +columns: + - title: Deployment + key: deployment + - title: Control Plane + key: cp + - title: Endpoint + key: endpoint +rows: + - deployment: "{{site.konnect_short_name}}" + cp: "{{site.konnect_short_name}} {{site.ai_gateway}} API" + endpoint: /v1/ai-gateways/{aiGatewayId}/data-plane-certificates +{% endtable %} + + + +## Trust model + +The {{site.ai_gateway}} acts as the control plane in a CP/DP topology. Each data plane presents a client certificate during the TLS handshake, and the {{site.ai_gateway}} accepts the connection only if the presented certificate matches one that has been registered as a Data Plane Certificate on that {{site.ai_gateway}}. + +Only the public certificate is registered with the {{site.ai_gateway}}. The private key is generated and held on the data plane side; it never leaves the data plane host. + + +{% mermaid %} +sequenceDiagram + participant DP as Data Plane + participant CP as {{site.ai_gateway}} (Control Plane) + + Note over DP: Holds private key locally
(never sent over the network) + DP->>CP: TLS handshake with client certificate + Note over CP: Compare presented certificate against
registered Data Plane Certificates + alt Certificate matches a registered entry + CP-->>DP: TLS handshake completes + DP->>CP: Receive configuration and stream telemetry + else No matching registered certificate + CP-->>DP: Connection rejected + end +{% endmermaid %} + + +## Lifecycle + +Data Plane Certificates support create, list, get, and delete operations. There is no update endpoint, the certificate body is immutable. + +To rotate a certificate without downtime: + +1. Register the new certificate as an additional Data Plane Certificate on the {{site.ai_gateway}}. +1. Reconfigure the data planes to present the new certificate and key. +1. Verify that data planes have reconnected with the new identity. +1. Delete the old Data Plane Certificate. + +Deleting a Data Plane Certificate immediately invalidates the trust for any data plane still using it. Existing connections are dropped and reconnect attempts using the deleted certificate are rejected. + +## Schema + +{% entity_schema %} diff --git a/app/_ai_gateway_entities/ai-identity-provider.md b/app/_ai_gateway_entities/ai-identity-provider.md new file mode 100644 index 00000000000..1fc98e7739e --- /dev/null +++ b/app/_ai_gateway_entities/ai-identity-provider.md @@ -0,0 +1,248 @@ +--- +title: AI Identity Providers +content_type: reference +entities: + - ai-identity-provider +products: + - ai-gateway +min_version: + ai-gateway: '2.0' +permalink: /ai-gateway/entities/ai-identity-provider/ +breadcrumbs: + - /ai-gateway/ + - /ai-gateway/entities/ +description: Configure inbound AI Consumer authentication for AI Models in {{site.ai_gateway}}. +schema: + api: konnect/ai-gateway + path: /schemas/AIGatewayIdentityProvider +works_on: + - konnect +tools: + - konnect-api + - kongctl +related_resources: + - text: "About {{site.ai_gateway}}" + url: /ai-gateway/ + - text: AI Model entity + url: /ai-gateway/entities/ai-model/ + - text: AI Model Provider entity + url: /ai-gateway/entities/ai-model-provider/ + - text: AI Consumer entity + url: /ai-gateway/entities/ai-consumer/ + - text: AI Consumer Group entity + url: /ai-gateway/entities/ai-consumer-group/ + - text: Key Auth policy reference + url: /ai-gateway/policies/key-auth/ + - text: OpenID Connect policy reference + url: /ai-gateway/policies/openid-connect/ +faqs: + - q: What is the difference between an AI Identity Provider and an AI Model Provider? + a: | + An AI Identity Provider manages inbound authentication: it validates the credentials that AI Consumers + present when calling an AI Model. An AI Model Provider manages outbound credentials: the secrets + {{site.ai_gateway}} uses to authenticate to an upstream LLM service on behalf of the AI Consumer. + + - q: Can an AI Model use both key-auth and OIDC authentication at the same time? + a: | + Yes. An AI Model supports one `key-auth` AI Identity Provider and one `openid-connect` + AI Identity Provider simultaneously. An AI Consumer's request is authenticated + if it satisfies either provider. + + - q: What happens when a request carries no valid credentials? + a: | + {{site.ai_gateway}} treats the request as an anonymous AI Consumer. A request-termination + policy on that anonymous AI Consumer returns `401 Unauthorized` before the request reaches + the AI Model. + + - q: Can I reuse the same AI Identity Provider across multiple AI Models? + a: | + Yes. Create an AI Identity Provider once and reference it by `name` or `id` in the + `access.identity_providers` array of any AI Model in the same gateway. + + - q: Which OIDC flows does the openid-connect type support? + a: | + By default, bearer token and client credentials flows are enabled. The full set includes + `authorization_code`, `bearer`, `client_credentials`, `introspection`, `kong_oauth2`, + `password`, `refresh_token`, `session`, and `userinfo`. Configure which flows are active + with `config.auth_methods`. +--- + +## What is an AI Identity Provider? + +Your [AI Models](/ai-gateway/entities/ai-model/) often need access control: some teams should reach certain AI Models and others should not, and you need a way to verify who is calling before a request consumes tokens or touches sensitive data. An AI Identity Provider lets you declare an inbound authentication mechanism at the gateway level and attach it to specific AI Models. + +Use AI Identity Providers to: +* Authenticate API keys and map them to [AI Consumers](/ai-gateway/entities/ai-consumer/) +* Authenticate enterprise users through an existing identity provider (Okta, Azure AD, Google, or any OIDC-compliant IdP) without managing keys manually +* Apply different authentication to different models. For example, API keys for internal automation and OIDC bearer tokens for user-facing applications. + +An AI Identity Provider manages inbound authentication, which is distinct from the outbound credentials managed by an [AI Model Provider](/ai-gateway/entities/ai-model-provider/). When an AI Consumer calls an AI Model, the AI Identity Provider checks who they are. The AI Model then uses the AI Model Provider's credentials to forward the request upstream. + +The following diagram shows where authentication fits in the request pipeline: + +{% mermaid %} +flowchart LR + Client["AI Consumer"] + KeyAuth["Key Auth"] + OIDC["OpenID Connect"] + Decision{Auth?} + AnonErr["Request Terminating w/ 401"] + ModelSel["Model selection"] + ACLs["ACLs"] + Model1["AI Model A"] + Model2["AI Model B"] + + Client-->KeyAuth + KeyAuth-->OIDC + OIDC-->Decision + Decision-->|no auth|AnonErr + Decision-->|auth|ModelSel + ModelSel-->|selects model|ACLs + ACLs-->|allowed|Model1 + ACLs-->|denied|Model2 +{% endmermaid %} + +Authentication runs before model selection so that unauthenticated requests never reach model routing or policy evaluation. + +## Manage AI Identity Providers + +AI Identity Providers can be created and managed through: + +* {{site.konnect_short_name}} UI +* {{site.ai_gateway}} API: `/v1/ai-gateways/{aiGatewayId}/identity` +* [kongctl](/kongctl/) + +For configuration examples and step-by-step setup instructions, see [Set up an AI Identity Provider](#set-up-an-ai-identity-provider). + +## Authentication types + +{{site.ai_gateway}} supports two identity provider types. Choose based on how your AI Consumers authenticate: + +{% table %} +columns: + - title: Type + key: type + - title: When to use + key: when + - title: AI Consumer credential + key: credential + - title: Policy + key: policy +rows: + - type: "`key-auth`" + when: "Your AI Consumers are internal tools, scripts, or teams that you control. You want to issue and rotate static API keys without involving an external identity provider." + credential: "API key in a request header, query parameter, or request body" + policy: "[Key Auth](/ai-gateway/policies/key-auth/)" + - type: "`openid-connect`" + when: "Your AI Consumers already authenticate through an enterprise IdP (Okta, Azure AD, Google, or similar). You want to accept the tokens they already have rather than issuing separate keys." + credential: "JWT bearer token or OAuth 2.0 grant from an external IdP" + policy: "[OpenID Connect](/ai-gateway/policies/openid-connect/)" +{% endtable %} + +### API key authentication + +The `key-auth` type uses the [Key Auth Policy](/ai-gateway/policies/key-auth/) to validate an API key that the AI Consumer passes on every request. The gateway looks for the key in a configurable header or query parameter, checks it against the AI Consumer's registered key, and either authenticates the request or routes it to the anonymous AI Consumer (which terminates with `401`). + +By default, {{site.ai_gateway}} accepts the key in an `apikey` header or `apikey` query parameter. Override the key name with `config.key_names`. For example, set `config.key_names: ["X-API-Key"]` to enforce a standard header name across your APIs. + +{% table %} +columns: + - title: Option + key: option + - title: Default + key: default + - title: Description + key: description +rows: + - option: "`key_in_header`" + default: "`true`" + description: "Accept the key in a request header." + - option: "`key_in_query`" + default: "`true`" + description: "Accept the key as a query parameter." + - option: "`key_in_body`" + default: "`false`" + description: "Accept the key in the request body. Supports `application/json`, `application/x-www-form-urlencoded`, and `multipart/form-data`." + - option: "`hide_credentials`" + default: "`true`" + description: "Strip the key from the request before forwarding upstream." +{% endtable %} + +### OIDC token authentication + +The `openid-connect` type uses the [OpenID Connect Policy](/ai-gateway/policies/openid-connect/) to validate a JWT or OAuth 2.0 token that the AI Consumer obtains from an external IdP. The gateway verifies the token against the IdP's published keys, maps the token to an AI Consumer, and either authenticates the request or routes it to the anonymous AI Consumer (which terminates with `401`). + +Set `config.issuer` to the IdP's discovery URL (for example, `https://dev-123456.okta.com`). {{site.ai_gateway}} uses the OIDC discovery endpoint to fetch signing keys automatically. + +The default `config.auth_methods` are `bearer` and `client_credentials`. If your AI Consumers use a different grant flow, add it to the list. For a full list of supported values, see the [OpenID Connect Policy reference](/ai-gateway/policies/openid-connect/). + +To map the token to an existing AI Consumer, set `config.consumer_claims` to an array of path segments locating the claim in the token that carries the AI Consumer identifier (for example, `[["user", "info", "id"]]` to map to a nested `user.info.id` claim). If no mapping is needed, set `config.consumer_optional: true` to allow unauthenticated token holders through ACL checks. + +{:.warning} +> All AI Models in the same {{site.ai_gateway}} that use OIDC authentication must reference the same `openid-connect` AI Identity Provider. Using different OIDC providers across models in the same {{site.ai_gateway}} is not supported. + +## Assigning an AI Identity Provider to an AI Model + +An AI Identity Provider takes effect only when assigned to an [AI Model](/ai-gateway/entities/ai-model/). Reference the provider by `name` or `id` in the `access.identity_providers` array on the AI Model: + +```yaml +access: + identity_providers: + - my-key-auth-provider + acls: + allow: + - allowed-ai-consumer-group +``` + +{:.info} +> **Assignment rules** +> * Each AI Model supports one `key-auth` identity provider and one `openid-connect` identity provider. +> * You can assign both types to the same AI Model. A request is authenticated if it satisfies either provider. + +If you plan to rename the AI Identity Provider later, reference it by `id` rather than name. The ID is stable across renames. + +## Set up an AI Identity Provider + +### API key authentication + +The following example creates a `key-auth` AI Identity Provider that accepts AI Consumer API keys in the `X-API-Key` header: + +{% entity_example %} +type: identity-provider +data: + display_name: API Key Auth + name: api-key-auth + type: key-auth + config: + key_names: + - X-API-Key + key_in_header: true + key_in_query: false + hide_credentials: true +{% endentity_example %} + +### OIDC bearer token authentication + +The following example creates an `openid-connect` AI Identity Provider that accepts bearer tokens issued by Okta: + +{% entity_example %} +type: identity-provider +data: + display_name: Okta AI SE + name: okta-ai-se + type: openid-connect + config: + issuer: https://dev-123456.okta.com + client_id: + - my-client-id + client_secret: + - my-client-secret + auth_methods: + - bearer + scopes: + - openid +{% endentity_example %} + +## Schema + +{% entity_schema %} diff --git a/app/_ai_gateway_entities/ai-mcp-server.md b/app/_ai_gateway_entities/ai-mcp-server.md new file mode 100644 index 00000000000..070ac3c78f3 --- /dev/null +++ b/app/_ai_gateway_entities/ai-mcp-server.md @@ -0,0 +1,625 @@ +--- +title: AI MCP Servers +content_type: reference +entities: + - ai-mcp-server +products: + - ai-gateway +min_version: + ai-gateway: '2.0' +permalink: /ai-gateway/entities/ai-mcp-server/ +breadcrumbs: + - /ai-gateway/ + - /ai-gateway/entities/ +description: AI MCP Server entity used by {{site.ai_gateway}} to expose tools and proxy MCP traffic. +schema: + api: konnect/ai-gateway + path: /schemas/AIGatewayMCPServer +works_on: + - konnect +tools: + - konnect-api + - kongctl +related_resources: + - text: About {{site.ai_gateway}} + url: /ai-gateway/ + - text: "{{site.ai_gateway}} entities" + url: /ai-gateway/entities/ + - text: AI Policy entity + url: /ai-gateway/entities/ai-policy/ + - text: AI Consumer Group entity + url: /ai-gateway/entities/ai-consumer-group/ + - text: Kong MCP traffic gateway + url: /mcp/ + - text: Model Context Protocol specification + url: https://modelcontextprotocol.io/ +faqs: + - q: Which MCP protocol version does the runtime use? + a: | + The MCP runtime behind an AI MCP Server entity speaks MCP protocol version `2025-06-18`. Upstream + MCP servers may run `2025-06-18` or `2025-11-25`. Versions from 2024 are not supported. + + - q: What's the difference between the server types? + a: | + `passthrough-listener` proxies MCP traffic to an upstream MCP server without converting tools. + `conversion-listener` converts a RESTful API into MCP tools and accepts MCP requests on one route path. `conversion-only` converts a RESTful API into MCP tools that a `listener` AI MCP Server + aggregates by matching labels, but doesn't accept incoming MCP traffic itself. `listener` aggregates tools from one or more + `conversion-only` AI MCP Servers into a single MCP endpoint. `upstream-server` registers a real + MCP server into an aggregation pool, dynamically fetching its tools for a `listener` to aggregate. + + - q: Can the same AI Consumer's identity gate access to specific tools? + a: | + Yes. Set [`access.default_tool_acls`](#schema-aigateway-mcpserver-access-default-tool-acls) on the AI MCP Server with `allow` and `deny` lists, and override per + tool through [`tools[].access.acls`](#schema-aigateway-mcpserver-tools-access). A per-tool ACL replaces the default for that tool, it doesn't + merge. + + - q: How do OAuth-based ACLs differ from AI Consumer-based ACLs? + a: | + Set [`access.acl_attribute_type`](#schema-aigateway-mcpserver-access-acl-attribute-type) to `oauth_access_token` and provide [`access.access_token_claim_field`](#schema-aigateway-mcpserver-access-access-token-claim-field) (a jq + filter, for example `.user.email`). ACLs then evaluate against the claim value extracted from + the OAuth access token instead of the resolved AI Consumer identity. The OAuth flow is supplied + by the [AI MCP OAuth2 Policy](/ai-gateway/policies/ai-mcp-oauth2/). + + - q: What error code do denied requests return? + a: | + `HTTP 403 Forbidden`. Earlier {{site.ai_gateway}} versions returned the JSON-RPC error code + `INVALID_PARAMS -32602`; from {{site.ai_gateway}} 3.14 onward, denials follow the + [MCP 2025-11-25 authorization specification](https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#error-handling). + + - q: Can I attach the same authentication or rate-limiting policy that I'd attach to the AI MCP Server? + a: | + Policy configuration that applies to the AI MCP Server goes through the + [AI Policy entity](/ai-gateway/entities/ai-policy/). Attach Policies to the AI MCP Server through its + [`policies`](#schema-aigateway-mcpserver-policies) field. +--- + +## What is an AI MCP Server? + +Create an AI MCP Server to connect AI applications such as [Claude](https://claude.ai/), [Cursor](https://cursor.com/), or [Insomnia](/insomnia/) to your APIs and tools through the standardized [Model Context Protocol](https://modelcontextprotocol.io/). An AI MCP Server acts as a bridge between MCP-compatible clients and your backend systems, allowing you to expose existing APIs as discoverable tools without building custom integrations for each AI client. + +Because MCP endpoints run directly on {{site.ai_gateway}}, you don't need to host and scale MCP infrastructure separately. The same authentication, rate limiting, and observability policies you apply to traditional API traffic automatically cover MCP traffic, giving you consistent governance across both HTTP and MCP clients. + +{:.warning} +> **Note:** MCP traffic is API-level traffic, not LLM request/response flows. The [AI MCP OAuth2 Policy](/ai-gateway/policies/ai-mcp-oauth2/) provides MCP-specific OAuth2 validation. Standard API-level policies (authentication, rate limiting, logging) apply to MCP traffic. AI Policies that operate on LLM prompt/response flows (such as prompt guards or model routing) won't apply here. + +## Manage AI MCP Servers + +AI MCP Servers can be created and managed through the: + +* {{site.konnect_short_name}} UI +* {{site.ai_gateway}} API: `/v1/ai-gateways/{aiGatewayId}/mcp-servers` +* [kongctl](/kongctl/) + +For configuration examples and step-by-step setup instructions, see [Set up an AI MCP Server](#set-up-an-ai-mcp-server). + +## AI MCP Server governance + +Attach [AI Policies](/ai-gateway/entities/ai-policy/) to AI MCP Servers to enforce authentication, rate limits, request/response transformation, and OAuth gating. Add them to the [`policies`](#schema-aigateway-mcpserver-policies) field by name or ID. AI Policies run on all MCP traffic through the server, before tool invocation and after ACL checks. Multiple AI Policies can attach to one AI MCP Server, and each runs independently in the request lifecycle. + +You can also attach AI Policies at the [AI Consumer](/ai-gateway/entities/ai-consumer/) level for per-client enforcement. + +Attach [AI Policies](/ai-gateway/entities/ai-policy/) to your AI MCP Server for common governance scenarios: + + +{% table %} +columns: + - title: Use case + key: use_case + - title: Policy + key: example +rows: + - use_case: "Secure MCP endpoints with credentials or OAuth tokens" + example: "[Key Auth Policy](/ai-gateway/policies/key-auth/) or [AI MCP OAuth2 Policy](/ai-gateway/policies/ai-mcp-oauth2/)" + - use_case: "Rate limiting" + example: "Use [Rate Limiting](/ai-gateway/policies/rate-limiting/) or [Rate Limiting Advanced](/ai-gateway/policies/rate-limiting-advanced/) Policy to control MCP request volume per AI Consumer or AI Consumer Group." + - use_case: "Track all MCP traffic and ACL decisions." + example: "Enable request and response logging through [AI logging Policies](/ai-gateway/policies/?category=logging) and audit trails." + - use_case: "Traffic control" + example: "Apply [Request Transformer](/ai-gateway/policies/request-transformer/) or [Response Transformer](/ai-gateway/policies/response-transformer/) Policy to modify MCP payloads, or use [ACLs](#acl-tool-control) for fine-grained tool access." +{% endtable %} + + +## Server modes + +{{site.ai_gateway}} supports five server modes for different integration patterns: exposing REST APIs as discoverable MCP tools, proxying requests to existing MCP servers with added authentication and observability, or aggregating tools from multiple sources into a single endpoint. Select the mode that fits your use case using the [`type`](#schema-aigateway-mcpserver-type) field. Regardless of mode, {{site.ai_gateway}} generates [MCP observability metrics](/ai-gateway/monitor-ai-llm-metrics/#mcp-traffic-metrics) for all traffic through the server. + + +{% table %} +columns: + - title: Use case + key: usecase + - title: Integration pattern + key: pattern + - title: Description + key: description + - title: Mode + key: mode +rows: + - usecase: | + You already operate an MCP server and want {{site.ai_gateway}} to act as an authenticated, + observable entrypoint. Common for third-party or internally hosted MCP services exposed + through {{site.ai_gateway}}. + pattern: Existing MCP server + description: | + Listens for incoming MCP requests and proxies them to an upstream MCP server without + converting tools. + mode: "`passthrough-listener`" + - usecase: | + Make an existing REST API available to MCP clients directly through {{site.ai_gateway}}. + Common for services that both define and handle their own tools. + pattern: Generate from REST API + description: | + Converts RESTful API paths into MCP tools and accepts incoming MCP requests on the route + path. Tools are defined directly on the MCP Server and an optional server block applies. + Supports session identifiers set by authentication services for cookie-based + authentication. + mode: "`conversion-listener`" + - usecase: | + Define reusable tool specifications without serving them yourself. Suitable for teams that + maintain a shared library of tool definitions for one or more `listener` MCP Servers. + Good for APIs you don't own or can't modify. + pattern: Generate from REST API and feeds aggregate + description: | + Converts RESTful API paths into MCP tools but does not accept incoming MCP requests. + Since a `conversion-only` AI MCP Server has no `config.server` block, it marks itself for aggregation using the + top-level [`labels`](#schema-aigateway-mcpserver-labels) field instead (for example, + `labels: {ai-gateway-mcp-aggregation: payments}`). A `listener` AI MCP Server then references it by + matching label. This mode must be used together with one or more AI MCP Servers configured with + `listener` mode. + mode: "`conversion-only`" + - usecase: | + A single MCP endpoint that aggregates tools from multiple `conversion-only` or + `upstream-server` MCP Servers. Typical in multi-service or multi-team environments that + expose a unified MCP interface. One `listener` per aggregated endpoint. + pattern: Aggregate + description: | + Similar to `conversion-listener`, but binds tools from one or more `conversion-only` MCP Servers + instead of defining its own. Set [`config.server.label`](#schema-aigateway-mcpserver-config-server-label) + to `:`, matching a key/value pair set in the `labels` field of each + `conversion-only` MCP Server to include. Merges all matching tools into one list and routes each + tool call to the correct backend. + mode: "`listener`" + - usecase: | + Expose an existing upstream MCP server's tools alongside others through a single `listener` + endpoint. The listener aggregates all matching upstreams, so adding a new upstream is just + deploying a new `upstream-server` AI MCP Server with a matching label. + pattern: Existing MCP server and feeds aggregate + description: | + Registers a real MCP server into an aggregation pool and tells the `listener` MCP Server + "this backend has tools, go fetch them." Dynamically fetches and caches its tool list, + then pairs with a `listener` MCP Server through a shared label. Supports optional OAuth2 + authentication to fetch tool lists from the upstream. This mode must be used together + with one or more AI MCP Servers configured with `listener` mode. + mode: "`upstream-server`" +{% endtable %} + + +## How MCP traffic flows + +For `conversion-listener`, `conversion-only`, and `listener` modes, the runtime converts MCP requests into HTTP calls and wraps the responses back in MCP format: + +1. Accepts an MCP protocol request from a client. +1. Parses the MCP tool call and matches it to a tool definition. +1. Converts the call into a standard HTTP request. +1. Sends the request to the upstream service. +1. Wraps the HTTP response in MCP format and returns it to the client. + +For `passthrough-listener` mode, the runtime proxies MCP traffic directly to the upstream MCP server without conversion. + + +{% mermaid %} +sequenceDiagram + participant Client as MCP Client + participant Gateway as {{site.ai_gateway}}
(MCP Server) + participant Upstream as Upstream Service + + Client->>Gateway: MCP request (tool invocation) + activate Gateway + Gateway->>Gateway: Parse MCP payload + Gateway->>Gateway: Map to HTTP endpoint + Gateway->>Upstream: HTTP request + deactivate Gateway + activate Upstream + Upstream-->>Gateway: HTTP response + deactivate Upstream + activate Gateway + Gateway->>Gateway: Convert to MCP format + Gateway-->>Client: MCP response + deactivate Gateway +{% endmermaid %} + + +{:.info} +> Pings from MCP clients are included in the total request count for an {{site.ai_gateway}} +> instance, in addition to requests made to the MCP server itself. + +### Forward proxy support + +{% include md/ai-gateway/v2/forward-proxy.md %} + +## Tool aggregation with upstream-server + +You can use a `listener` AI MCP Server to pull tools from multiple `upstream-server` AI MCP Servers and expose them through a single endpoint. The listener discovers and aggregates tools based on a matching label, so clients see one unified tool catalog while your services remain independent. + +### How aggregation works + +1. **Labels connect upstreams to listeners**: Each `upstream-server` AI MCP Server tags itself using the top-level [`labels`](#schema-aigateway-mcpserver-labels) field (for example, `labels: {my-tools: catalog}`). Set [`config.server.label`](#schema-aigateway-mcpserver-config-server-label) on the listener to the matching `:` string (for example, `my-tools:catalog`). Any upstream with that label gets pulled into the aggregation. + +2. **Tool discovery**: When an MCP client calls `tools/list`, the listener fetches tool lists from every matching upstream. If an upstream requires authentication, configure [`config.server.tools_list_auth`](#schema-aigateway-mcpserver-config-server-tools-list-auth) with OAuth2 credentials so the listener can fetch its tools. + +3. **Tool caching**: Each `upstream-server` caches its tool list for the duration specified by [`config.tools_cache_ttl_seconds`](#schema-aigateway-mcpserver-config-tools-cache-ttl-seconds). Set to `0` to fetch fresh on every client request. + +4. **Tool name disambiguation**: If two upstreams expose tools with the same name, the listener prepends the service name to avoid collisions (e.g., `weather-service/get-forecast`). Disable this with [`config.server.preserve_upstream_tool_names`](#schema-aigateway-mcpserver-config-server-preserve-upstream-tool-names): true if you're sure names won't collide. + +5. **Tool invocation**: When a client calls a tool, the listener routes the request to whichever upstream registered it. From the client's perspective, it's one call to one URL. + + +{% mermaid %} +sequenceDiagram + participant Agent as AI Agent + participant Kong as {{site.ai_gateway}} + participant Listener as AI MCP Server + participant Upstreams as Upstreams
(label: my-tools:catalog) + + note over Agent,Upstreams: Phase 1: Token Validation + Agent->>Kong: Bearer token + Kong->>Kong: Validate & exchange token + Kong->>Listener: Pass request + + note over Agent,Upstreams: Phase 2: Tools List Aggregation + Agent->>Kong: tools/list + Kong->>Listener: tools/list + Listener->>Upstreams: Query all matching upstreams + Upstreams-->>Listener: Tool lists + Listener-->>Kong: Merged list + Kong-->>Agent: Aggregated tools + + note over Agent,Upstreams: Phase 3: Tool Call Routing + Agent->>Kong: tools/call (tool name) + Kong->>Listener: Route to upstream + Listener->>Upstreams: Forward call + Upstreams-->>Listener: Result + Listener-->>Kong: Result + Kong-->>Agent: Tool response +{% endmermaid %} + + +> _Figure 1_: This diagram shows how {{site.ai_gateway}} handles requests from an AI agent. It validates the agent's credentials, collects tool definitions from multiple services, and forwards tool calls to the correct upstream. + +### Upstream authentication + +By default, the AI MCP Server in `listener` mode connects to upstreams without credentials. If an upstream MCP server requires authentication, configure [`config.server.tools_list_auth`](#schema-aigateway-mcpserver-config-server-tools-list-auth) on the `upstream-server`. The credential is used only when fetching the upstream's tool list, not for agent requests. Different upstreams can use different credentials, managed centrally by {{site.ai_gateway}}. + + +{% table %} +columns: + - title: Use case + key: usecase + - title: Type + key: type + - title: Configuration + key: config +rows: + - usecase: Your upstream requires a service-to-service OAuth2 token from an identity provider. + type: "`credentials`" + config: | + `token_endpoint`, `client_id`, and `client_secret`. {{site.ai_gateway}} exchanges them for + a bearer token, caches it, and refreshes it on expiry. + - usecase: Your upstream validates JWTs directly, without a token exchange step. + type: "`jwt`" + config: A pre-signed JWT. {{site.ai_gateway}} presents it as-is when fetching the tool list. +{% endtable %} + + +### Header forwarding + +When your upstream services need to enforce their own access controls or apply client-specific logic based on identity, enable [`config.server.forward_client_headers`](#schema-aigateway-mcpserver-config-server-forward-client-headers) on the `listener` or `upstream-server`. This setting passes the original client's headers (authentication tokens, context) so upstreams see the actual client, not just the listener. + +## Tools + +A [tool](#schema-aigateway-mcpserver-tools) maps an MCP tool name to an upstream HTTP endpoint. Each tool needs at minimum a description and an HTTP method. The runtime extracts the host, path, headers, and query from the route configuration, so most tool entries don't need to specify them. Override these on the tool entry only when the route doesn't match the upstream endpoint exactly. + +For richer mapping, supply [`request_body`](#schema-aigateway-mcpserver-tools-request-body), [`responses`](#schema-aigateway-mcpserver-tools-responses), and [`parameters`](#schema-aigateway-mcpserver-tools-parameters) specifications in OpenAPI JSON format. The runtime uses them to validate calls and shape upstream HTTP requests. + +Tools can also carry MCP-spec [`annotations`](#schema-aigateway-mcpserver-tools-annotations) that hint at tool behavior to clients (for example, whether a tool is read-only, idempotent, or destructive). Annotations don't change runtime behavior; they help clients decide whether to surface a tool, confirm before invocation, or treat it as safe to retry. + +[Per-tool ACLs](#schema-aigateway-mcpserver-tools-access) override the MCP Server's [default tool ACLs](#schema-aigateway-mcpserver-access-default-tool-acls). For more information, see [ACL tool control](#acl-tool-control). + +## Sessions + +Some MCP clients need to maintain state across multiple tool calls such as authentication tokens, conversation context, or request IDs. {{site.ai_gateway}} can manage session state for you in `listener` and `conversion-listener` modes, storing it either encrypted on the client or in Redis. Configure session storage through [`config.server.session`](#schema-aigateway-mcpserver-config-server-session). The `passthrough-listener` mode doesn't manage sessions because state lives entirely on the upstream MCP server. + +There are two session strategies: + +1. **Client.** Session state is encrypted into the MCP session ID assigned to the client. Requires `secrets` which are encryption keys; the first entry is used for encryption, all entries are used for decryption to support key rotation. +1. **Redis.** Session state is stored in Redis. Configure connection details and authentication in [`config.server.session.redis`](#schema-aigateway-mcpserver-config-server-session-redis). + +{% include_cached md/ai-gateway/v2/redis-cloud-auth.md tier='enterprise' %} + +Configure how long sessions persist using [`session_ttl`](#schema-aigateway-mcpserver-config-server-session-session-ttl) (default 24 hours) to match your application's needs. If your upstream server already manages state internally, disable {{site.ai_gateway}}'s session management by setting `managed: false`. + +{:.info} +> Secrets used in session encryption can be referenced from an [AI Vault](/ai-gateway/entities/ai-vault/). + +## Connecting to the MCP endpoint + +An MCP client such as [Claude Desktop](https://claude.ai/download), Cursor, or [ChatWise](https://chatwise.app/) handles the following details automatically. They matter when testing an AI MCP Server directly, for example with `curl`, or when building a custom MCP client. + +**Streamable HTTP handshake**. {{site.ai_gateway}} implements the MCP [Streamable HTTP transport](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#streamable-http). A spec-compliant client performs this sequence before calling tools: + +1. Send an `initialize` request to the route configured on [`config.route.paths`](#schema-aigateway-mcpserver-config-route-paths). The response includes an `Mcp-Session-Id` header. +1. Send a `notifications/initialized` notification to the same route. +1. Carry the `Mcp-Session-Id` header on subsequent `tools/list` and `tools/call` requests. + +{:.success} +> **Tool argument naming**. +> +>Tools generated from [`parameters`](#schema-aigateway-mcpserver-tools-parameters) (`conversion-listener`, `conversion-only`) rename `query`, `path`, `header`, and `cookie` args to `{in}_{name}`: `query: q` becomes `query_q`. +> Bodies collapse into a single `body` property. Check `tools/list` before calling `tools/call`. + +## ACL tool control + +When exposing MCP servers through {{site.ai_gateway}}, you may need granular control over which authenticated [AI Consumers](/ai-gateway/entities/ai-consumer/) can discover and invoke specific tools. The MCP Server's ACL feature lets you define access rules at both the default level (which applies to all tools) and per-tool level (for fine-grained exceptions). + +This way, AI Consumers only interact with tools appropriate to their role, while maintaining a complete audit trail of all access attempts. Authentication is handled by an authentication Policy attached to the AI MCP Server (such as [Key Auth Policy](/ai-gateway/policies/key-auth/) or an [OpenID Connect Policy](/ai-gateway/policies/openid-connect/)), and the AI Consumer identity is used for ACL checks. + +{:.info} +> **ACL in `listener` mode** +> +> `listener` mode supports direct ACL configuration on the MCP Server itself. +> +> To use ACLs with `listener` mode: +> 1. Configure authentication on the listener route so requests resolve to an authenticated AI Consumer. +> 1. Set ACL fields directly on the listener: [`access.acl_attribute_type`](#schema-aigateway-mcpserver-access-acl-attribute-type), [`access.access_token_claim_field`](#schema-aigateway-mcpserver-access-access-token-claim-field) (when using `oauth_access_token`), [`access.acls`](#schema-aigateway-mcpserver-access-acls) for server-level fallback rules, [`access.default_tool_acls`](#schema-aigateway-mcpserver-access-default-tool-acls) for the default tool ACL, and per-tool [`tools[].access.acls`](#schema-aigateway-mcpserver-tools-access) for tool-specific overrides. +> 1. Configure [AI Consumers](/ai-gateway/entities/ai-consumer/) and [AI Consumer Groups](/ai-gateway/entities/ai-consumer-group/) to match the `allow` and `deny` entries. +> +> ACL behavior: +> +> - `access.default_tool_acls` applies to all tools by default. +> - If `access.default_tool_acls` isn't set, the listener falls back to the top-level `access.acls`. +> - A tool's own `acls` fully overrides the default ACL for that tool. +> - [`config.server.label`](#schema-aigateway-mcpserver-config-server-label) is used for tool filtering and aggregation, not for inheriting ACLs from other AI MCP Servers. + +### Attribute types + +For modes that support server-level ACL configuration (`conversion-listener`, `listener`, `passthrough-listener`, `upstream-server`), two attribute types determine what the AI MCP Server evaluates ACL rules against: + +1. **`consumer`** (default). Evaluates against the resolved AI Consumer identity. +1. **`oauth_access_token`**. Evaluates against a claim extracted from the OAuth access token. Set [`access.access_token_claim_field`](#schema-aigateway-mcpserver-access-access-token-claim-field) to a jq filter (for example, `.user.email` for a nested claim). The OAuth flow itself is supplied by the [AI MCP OAuth2 Policy](/ai-gateway/policies/ai-mcp-oauth2/). + +`conversion-only` AI MCP Servers have no `access` field of their own, since they never accept incoming MCP traffic directly. They only support per-tool ACLs (via [`tools[].access.acls`](#schema-aigateway-mcpserver-tools-access)), which travel with the tool definition when a `listener` aggregates it. + +### Using AI Consumers and Groups in ACLs + +When `access.acl_attribute_type` is `consumer`, you can gate access by individual [AI Consumers](/ai-gateway/entities/ai-consumer/) (using username, UUID, or custom ID) or by [AI Consumer Group](/ai-gateway/entities/ai-consumer-group/) membership. This flexibility lets you define rules at the right level: deny a specific user, allow a tier-based group, or mix both in the same ACL. The runtime checks the authenticated AI Consumer's identity and group memberships against your `allow` and `deny` lists. + +### How default and per-tool ACLs work + +The runtime evaluates access using a two-tier system: + + +{% table %} +columns: + - title: ACL type + key: field + - title: Description + key: description +rows: + - field: "`access.default_tool_acls`" + description: | + Baseline rules that apply to all tools unless overridden. + - field: "`tools[].access.acls`" + description: | + When configured, these rules replace the default ACL for that specific tool. The per-tool ACL doesn't inherit or merge with `access.default_tool_acls`. It is an all-or-nothing override. +{% endtable %} + + +{:.info} +> If a tool defines its own ACL, the runtime ignores `access.default_tool_acls` for that tool: +> +> - Tools with no ACL configuration inherit the default rules (both `allow` and `deny` lists). +> - Tools with an ACL must explicitly list all allowed subjects (even if they were already in `access.default_tool_acls`). + +### ACL evaluation logic + +Both default and per-tool ACLs use `allow` and `deny` lists. Evaluation follows this order: + +1. **Deny list configuration**. If a `deny` list exists and the subject matches any `deny` entry, the request is rejected (`HTTP 403 Forbidden`). +1. **Allow list configuration**. If an `allow` list exists, the subject must match at least one entry; otherwise, the request is denied (`HTTP 403 Forbidden`). +1. **No allow list configuration**. If no `allow` list exists and the subject is not in `deny`, the request is allowed. +1. **No ACL configuration**. If neither list exists, the request is allowed. + +All access attempts (allowed or denied) are written to the audit log. + +The following table summarizes the possible ACL configurations and their outcomes. + +{% table %} +columns: + - title: Condition + key: condition + - title: "Proxied to upstream service?" + key: proxy + - title: Response code + key: response +rows: + - condition: "Subject matches any `deny` rule" + proxy: No + response: HTTP 403 Forbidden + - condition: "`allow` list exists and subject is not in it" + proxy: No + response: HTTP 403 Forbidden + - condition: "Only `deny` list exists and subject is not in it" + proxy: Yes + response: 200 + - condition: "No ACL rules configured" + proxy: Yes + response: 200 +{% endtable %} + +### ACL tool control request flow + +The runtime evaluates ACLs for both tool discovery and tool invocation. These are two distinct operations with different behaviors: + +**Tool discovery (list tools)**: + +1. MCP client requests the list of available tools. +1. The authentication Policy validates the request and identifies the AI Consumer. +1. The runtime loads the AI Consumer's group memberships. +1. The runtime evaluates each tool against `default_tool_acls`. +1. The runtime returns an HTTP 200 response with only the tools the AI Consumer is allowed to access. +1. The runtime logs the discovery attempt. + +**Tool invocation**: + +1. MCP client invokes a specific tool. +1. The authentication Policy validates the request and identifies the AI Consumer. +1. The runtime loads the AI Consumer's group memberships. +1. The runtime evaluates the tool-specific ACL if it exists, or the default ACL otherwise. +1. The runtime logs the access attempt (allowed or denied). +1. The runtime returns `HTTP 403 Forbidden` if denied, or forwards the request to the upstream MCP server if allowed. + + +{% mermaid %} +sequenceDiagram + participant Client as MCP Client + participant Gateway as {{site.ai_gateway}} + participant Auth as AuthN Policy + participant ACL as AI MCP Server (ACL/Audit) + participant Up as Upstream MCP Server + participant Log as Audit Sink + + %% ----- List Tools ----- + rect + note over Client,Gateway: List Tools (Default ACL Scope) + Client->>Gateway: GET /tools + Gateway->>Auth: Authenticate + Auth-->>Gateway: Consumer identity + Gateway->>ACL: Evaluate scoped default ACL + ACL-->>Log: Audit entry + alt If allowed + Gateway-->>Client: Filtered tool list + else If denied + Gateway-->>Client: HTTP 403 Forbidden + end + end + + %% ----- Tool Invocation ----- + rect + note over Client,Up: Tool Invocation (Per-tool ACL) + Client->>Gateway: POST /tools/{tool} + Gateway->>Auth: Authenticate + Auth-->>Gateway: Consumer identity + Gateway->>ACL: Evaluate per-tool ACL + ACL-->>Log: Audit entry + alt If allowed + Gateway->>Up: Forward request + Up-->>Gateway: Response + Gateway-->>Client: Response + else If denied + Gateway-->>Client: HTTP 403 Forbidden + end + end +{% endmermaid %} + + +## Logging and audits + +To monitor and troubleshoot MCP traffic, enable logging and audit trails through [`config.logging`](#schema-aigateway-mcpserver-config-logging). You can capture per-request statistics for metrics, full request and response payloads for debugging, and [audit entries](/ai-gateway/ai-audit-log-reference/#ai-mcp-logs) for every ACL decision. Note that payload logging may expose sensitive data. Enable it only when debugging and be careful with retention. [AI MCP Server analytics](/ai-gateway/monitor-ai-llm-metrics/#mcp-traffic-metrics) display in {{site.konnect_short_name}} [Explorer](https://cloud.konghq.com/analytics/explorer) and [Dashboards](https://cloud.konghq.com/analytics/dashboards) alongside other {{site.ai_gateway}} traffic, and export through [OpenTelemetry](/ai-gateway/policies/opentelemetry/reference/). + +## Scope of support + +The AI MCP Server runtime supports MCP operations and upstream interactions, while certain advanced features and non-HTTP protocols are not currently supported. The following table summarizes what is supported and what is outside the current scope. + + +{% feature_table %} +item_title: Features +columns: + - title: Description + key: description + - title: Supported + key: supported + +features: + - title: "Protocol" + description: Handling latest streamable HTTP with HTTP and HTTPS upstreams + supported: true + - title: "OpenAPI operations" + description: Mapping MCP calls to upstream HTTP operations based on the OpenAPI schema + supported: true + - title: "JSON format" + description: Handling standard JSON request and response bodies + supported: true + - title: "Form-encoded data" + description: Handling `application/x-www-form-urlencoded` + supported: true + - title: "SNI routing" + description: Converting SNI-only routes + supported: false + - title: "Form and XML data" + description: Handling formats such as multipart/form-data or XML + supported: false + - title: "Advanced MCP features" + description: Handling structured output, active notifications on tool changes, and session sharing between instances + supported: false + - title: "Non-HTTP protocols" + description: Handling WebSocket and gRPC upstreams + supported: false + - title: "AI Guardrails" + description: Applying guardrails to MCP AI requests and responses + supported: false +{% endfeature_table %} + + +## Set up an AI MCP Server + +The following example creates a `conversion-listener` AI MCP Server that exposes the [WeatherAPI](https://www.weatherapi.com/) through a single `get-current-weather` MCP tool. + +{:.info} +> You need your WeatherAPI API key set as an environment variable (`WEATHERAPI_API_KEY`) before using this example. + +{% entity_example %} +type: mcp_server +data: + display_name: Weather API + name: weather-mcp + type: conversion-listener + enabled: true + policies: [] + access: + acl_attribute_type: consumer + acls: + allow: + - __never_match__ + default_tool_acls: + deny: + - __never_match__ + config: + url: https://api.weatherapi.com/v1/current.json + route: + paths: + - /weather + logging: + payloads: false + audits: true + server: + timeout: 60000 + tools: + - name: get-current-weather + description: Get current weather for a location + method: GET + path: /weather + query: + key: + - $WEATHERAPI_API_KEY + parameters: + - name: q + in: query + required: true + schema: + type: string + description: Location query. Accepts US Zipcode, UK Postcode, Canada postal code, IP address, latitude/longitude, or city name. +{% endentity_example %} + +## Schema + +{% entity_schema %} diff --git a/app/_ai_gateway_entities/ai-model.md b/app/_ai_gateway_entities/ai-model.md new file mode 100644 index 00000000000..8146f35af8e --- /dev/null +++ b/app/_ai_gateway_entities/ai-model.md @@ -0,0 +1,409 @@ +--- +title: AI Models +content_type: reference +entities: + - ai-model +products: + - ai-gateway +min_version: + ai-gateway: '2.0' +permalink: /ai-gateway/entities/ai-model/ +breadcrumbs: + - /ai-gateway/ + - /ai-gateway/entities/ +description: AI Models registered with the {{site.ai_gateway}}. +schema: + api: konnect/ai-gateway + path: /schemas/AIGatewayModel +works_on: + - konnect +tools: + - konnect-api + - kongctl +related_resources: + - text: About {{site.ai_gateway}} + url: /ai-gateway/ + - text: "{{site.ai_gateway}} providers" + url: /ai-gateway/ai-providers/ + - text: Load balancing + url: /ai-gateway/load-balancing/ + - text: AI Model Provider entity + url: /ai-gateway/entities/ai-model-provider/ + - text: AI Identity Provider entity + url: /ai-gateway/entities/ai-identity-provider/ + - text: AI Policy entity + url: /ai-gateway/entities/ai-policy/ + - text: "{{site.ai_gateway}} entities" + url: /ai-gateway/entities/ + - text: AI Consumer Group entity + url: /ai-gateway/entities/ai-consumer-group/ +faqs: + - q: What's the difference between an AI Model entity and the `model` field in an AI Policy configuration? + a: | + An AI Model entity is the first-class {{site.ai_gateway}} entity you declare through the {{site.konnect_short_name}} API and UI. + It defines routing, capabilities, and load balancing. An AI Policy is a reusable configuration that adds behavior (like caching or guardrails) to an AI Model. + You declare both separately and attach AI Policies to AI Models. + + + - q: What happens when I update an AI Model? + a: | + {{site.ai_gateway}} deletes the AI Model's derived primitives and recreates them from the updated entity state, all within a single database transaction. + On failure, the transaction rolls back and no partial state is written. + + - q: Can I apply the same configuration to multiple AI Models? + a: | + Yes, by attaching one AI Policy with that configuration to each AI Model. + AI Policies are not shared between entities, each instance is independent. + See [AI Policy entity](/ai-gateway/entities/ai-policy/). + + - q: How do I limit which AI Consumers can reach an AI Model? + a: | + Set the [`access.acls`](#schema-aigateway-model-access) field on the AI Model with an allow list or a deny list. + Each entry is a string that references an AI Consumer, AI Consumer Group, or Authenticated Group by name. + + - q: Does the AI Model entity store AI Model Provider credentials? + a: | + No. AI Model Provider credentials live on the [AI Model Provider entity](/ai-gateway/entities/ai-model-provider/) and are materialized into the underlying primitives at AI Model creation time. + Updating an AI Model Provider propagates the credential change to all AI Models that reference it. + + - q: Can a client override the model name from the request body? + a: | + By default, no. The request `model` field must match the upstream model on one of the AI Model's targets, otherwise the runtime returns a `400` error. + To accept a client-side alias, set [`config.model.alias`](/ai-gateway/entities/ai-model/#schema-aigateway-model-config-model-alias). Clients can then send the alias value in the request `model` field instead of the upstream AI Provider model name. See [Request routing by model alias](/ai-gateway/load-balancing/#request-routing-by-model-alias) for details and examples. + + - q: Can a client override `temperature`, `top_p`, or `top_k` from the request? + a: | + Yes. Values for `temperature`, `top_p`, and `top_k` in the request take precedence over the per-target configuration declared on [`targets[].config`](#schema-aigateway-model-targets). + + - q: Which algorithm does `lowest-latency` use to pick the fastest target? + a: | + Exponentially Weighted Moving Average (EWMA). EWMA continuously updates with every response, weighting recent observations more heavily, so older latencies decay over time but still contribute. There is no fixed learning-phase window. + + - q: Does the load balancer keep probing slower targets after picking a winner? + a: | + Yes. EWMA ensures every target continues to receive a small share of traffic (typically 0.1% to 5%, depending on the latency gap). This ongoing probing lets the load balancer adapt if a previously slower target becomes faster. + +--- + +## What is an AI Model? + +The AI Model entity lets you expose LLM endpoints through {{site.ai_gateway}} for clients to call. Use AI Models to: +* [Expose multiple LLM providers](#targets) under a single endpoint +* [Load-balance traffic](#load-balancing) across them +* [Add observability](#logging-and-observability) to model traffic +* [Attach policies](#attach-ai-policies) for security and transformation + +An AI Model declares which capabilities it exposes (like `chat` or `embeddings`), which upstream LLM models it routes to via [AI Model Providers](/ai-gateway/entities/ai-model-provider/), and how requests are distributed and logged. Consumer authentication is configured through [AI Identity Providers](/ai-gateway/entities/ai-identity-provider/) on the model. {{site.ai_gateway}} handles routing and translation, so clients interact with a single unified endpoint. + +## Manage AI Models + +AI Models can be created and managed through: + +* {{site.konnect_short_name}} UI +* {{site.ai_gateway}} API: `/v1/ai-gateways/{aiGatewayId}/models` +* [kongctl](/kongctl/) + +For configuration examples and step-by-step setup instructions, see [Set up an AI Model](#set-up-an-ai-model). + +## How it works + +At request time, the AI Model mediates traffic between clients and upstream AI Provider APIs: + +1. Translates between the request and response format chosen for the AI Model and the upstream AI Provider's native format. +1. Resolves upstream connection coordinates (protocol, host, port, path, HTTP method) from the selected target and its [AI Model Provider](/ai-gateway/entities/ai-model-provider/), unless the target is a self-hosted model. +1. Authenticates to the upstream LLM service using credentials stored on the [AI Model Provider](/ai-gateway/entities/ai-model-provider/) entity. +1. Decorates the upstream request with per-target configuration (such as temperature or token-limit overrides) declared on [`targets[].config`](#schema-aigateway-model-targets). +1. Records usage statistics (tokens, cost, latency) for attached log AI Policies, and optionally the full request and response when payload logging is enabled. +1. Fulfills requests to self-hosted models using the supported native format transformations. + +A single AI Model can expose multiple upstream AI Providers behind a consistent client-facing format, so callers don't change their request shape when the underlying AI Provider changes. + +## Model lifecycle + +When you create or update an AI Model, {{site.ai_gateway}} provisions the necessary runtime resources and applies the configuration atomically. Credentials are sourced from the [AI Model Provider](/ai-gateway/entities/ai-model-provider/) entity that the AI Model's [`targets`](#schema-aigateway-model-targets) reference at model creation time. If you update the AI Model Provider's credentials later, those changes automatically propagate to all AI Models that use it. + +An AI Model is a managed entity. {{site.ai_gateway}} owns its runtime configuration. Direct modifications through other APIs are not supported. To change an AI Model's configuration, update the AI Model entity directly. + +## Capabilities + +When you expose an AI Model, you choose which AI capabilities it provides through the [`capabilities`](#schema-aigateway-model-capabilities) field. The [`type`](#schema-aigateway-model-type) you select determines which capabilities are available: + +* **`model` type**: for synchronous request/response workloads. Available capabilities: `generate`, `agentic`, `embeddings`, `audio/speech`, `audio/transcription`, `audio/translation`, `image`, `video`, `realtime`, `rerank`. +* **`api` type**: for asynchronous batch processing. Available capabilities: `batches`, `files`. + +Not every LLM service supports every capability. The set of capabilities you can declare on an AI Model depends on what the AI Model Provider in [`targets`](#schema-aigateway-model-targets) exposes. See [{{site.ai_gateway}} providers](/ai-gateway/ai-providers/) for per-provider details. + + +{% table %} +columns: + - title: Capability + key: capability + - title: Default OpenAI path + key: path + - title: Description + key: description +rows: + - capability: "`generate`" + path: "`/chat/completions`, `/completions`, `/responses`" + description: Text generation and conversational responses from generative models. + - capability: "`agentic`" + path: "`/assistants`" + description: Persistent tool-using agents with state management and metadata. + - capability: "`embeddings`" + path: "`/embeddings`" + description: Vector representations for semantic search and similarity matching. + - capability: "`audio/speech`" + path: "`/audio/speech`" + description: Text-to-speech synthesis. + - capability: "`audio/transcription`" + path: "`/audio/transcriptions`" + description: Speech-to-text conversion. + - capability: "`audio/translation`" + path: "`/audio/translations`" + description: Audio translation between languages. + - capability: "`image`" + path: "`/images/generations`, `/images/edits`" + description: Generate or edit images from text prompts. + - capability: "`video`" + path: "`/videos`" + description: Generate videos from text prompts. + - capability: "`realtime`" + path: "`/realtime`" + description: Bidirectional WebSocket streaming for low-latency interactive sessions. + - capability: "`rerank`" + path: "`/rerank`" + description: Rank documents by relevance to a query. + - capability: "`batches`" + path: "`/batches`" + description: Asynchronous bulk LLM requests for long workloads. + - capability: "`files`" + path: "`/files`" + description: File uploads for long documents and structured input. +{% endtable %} + + +## Request and response formats + +By default, AI Models expose all endpoints using OpenAI-compatible format. {{site.ai_gateway}} provides a single, standardized interface across all providers, so you can swap providers (OpenAI, Anthropic, self-hosted, etc.) without changing client code or integration logic. + +The [`formats`](#schema-aigateway-model-formats) array lets you control the request and response format. Each entry has a `type` that selects the format. The default `openai` format translates upstream provider responses into the OpenAI shape, so clients use one API format regardless of provider. + +If you need the provider's native format instead, set [`formats[].type`](#schema-aigateway-model-formats-type) to a non-OpenAI value. The AI Model passes requests upstream without conversion, while {{site.ai_gateway}} continues to provide analytics, logging, and cost calculation. You can also customize the endpoint paths through [`config.route.paths`](#schema-aigateway-model-config-route-paths) if needed. + + +{% table %} +columns: + - title: Format + key: format + - title: Provider + key: provider + - title: Native capabilities + key: capabilities +rows: + - format: "`openai`" + provider: All supported providers (default) + capabilities: Translates between OpenAI request and response shapes and the upstream provider format. + - format: "`anthropic`" + provider: "[Anthropic](/ai-gateway/ai-providers/anthropic/#supported-native-llm-formats-for-anthropic)" + capabilities: Messages, batch processing. + - format: "`bedrock`" + provider: "[Amazon Bedrock](/ai-gateway/ai-providers/bedrock/#supported-native-llm-formats-for-amazon-bedrock)" + capabilities: Converse, RAG (RetrieveAndGenerate), reranking, async invocation. + - format: "`cohere`" + provider: "[Cohere](/ai-gateway/ai-providers/cohere/#supported-native-llm-formats-for-cohere)" + capabilities: Reranking. + - format: "`gemini`" + provider: "[Gemini](/ai-gateway/ai-providers/gemini/#supported-native-llm-formats-for-gemini), [Vertex AI](/ai-gateway/ai-providers/vertex/#supported-native-llm-formats-for-gemini-vertex)" + capabilities: Content generation, embeddings, batches, file uploads, reranking, long-running predictions. + - format: "`huggingface`" + provider: "[Hugging Face](/ai-gateway/ai-providers/huggingface/#supported-native-llm-formats-for-hugging-face)" + capabilities: Text generation, streaming. +{% endtable %} + + +When a native format is set, only the corresponding provider is supported with its specific APIs. + +## Targets + +An AI Model is a virtual model: it exposes one Route ([`config.route`](#schema-aigateway-model-config-route)) and one set of capabilities, and routes requests to one or more concrete upstream models declared in its [`targets`](#schema-aigateway-model-targets) array. Each entry represents a single upstream model instance with one URL. + +For each target, you provide the upstream model name (for example, `gpt-4o`) and reference the AI Model Provider to use by its `name`. Each target can also override settings such as [`temperature`](#schema-aigateway-target-config-temperature), [`max_tokens`](#schema-aigateway-target-config-max-tokens), [`input_cost`](#schema-aigateway-target-config-input-cost), and [`output_cost`](#schema-aigateway-target-config-output-cost). + +There's no separate target entity or endpoint. Targets are managed only as nested data inside an AI Model, through the same AI Model API surface used to create, update, and delete the parent. Adding, removing, or modifying a target is an update to the AI Model itself. + +## Load balancing + +An AI Model routes to a single target by default. You can add more than one target when you want redundancy, fallback between providers, or cost and latency optimization. When you have multiple targets, configure [`config.balancer`](#schema-aigateway-model-config-balancer) to distribute requests according to a load balancing algorithm. + +When an AI Model has more than one target, the [load balancer](#schema-aigateway-model-config-balancer) sits between the virtual model and its targets, distributing requests according to `config.balancer`. For algorithm details, selection guidance, and tuning, see [Load balancing](/ai-gateway/load-balancing/). + +### Algorithms + +The [`algorithm`](#schema-aigateway-model-config-balancer-algorithm) field lets you choose how to distribute requests across target models based on your priorities. Select a strategy to optimize for cost, latency, even distribution, intelligent routing, or failover behavior. + + + +{% table %} +columns: + - title: Algorithm + key: algorithm + - title: Behavior + key: behavior +rows: + - algorithm: "`round-robin`" + behavior: Weighted traffic distribution across targets. + - algorithm: "`consistent-hashing`" + behavior: Sticky sessions based on header values. + - algorithm: "`least-connections`" + behavior: Route to backends with spare capacity. + - algorithm: "`lowest-latency`" + behavior: Route to the fastest-responding model. + - algorithm: "`lowest-usage`" + behavior: Route based on token counts or cost. + - algorithm: "`semantic`" + behavior: Route based on prompt-to-model similarity. + - algorithm: "`priority`" + behavior: Tiered failover across model groups. +{% endtable %} + + +### Retry and fallback + +To add redundancy and failover, the load balancer supports configurable retries, timeouts, and failover to different targets when one is unavailable. Fallback works across targets with any supported format, so you can mix providers freely (for example, OpenAI and Mistral). For configuration details, see [Retry and fallback configuration](/ai-gateway/load-balancing/#retry-and-fallback). + +{:.info} +> Client errors don't trigger failover. To fail over on additional error types, set +> [`failover_criteria`](#schema-aigateway-model-config-balancer-failover-criteria) to include HTTP codes +> like `http_429` or `http_502`, and `non_idempotent` for POST requests. + +### Forward proxy support + +{% include md/ai-gateway/v2/forward-proxy.md %} + +### Health check and circuit breaker + +To improve reliability under sustained failures, the load balancer includes a circuit breaker. When a target reaches the failure threshold set by [`max_fails`](#schema-aigateway-model-config-balancer-max-fails), the load balancer stops routing requests to it until the [`fail_timeout`](#schema-aigateway-model-config-balancer-fail-timeout) period elapses. For behavior examples and tuning, see [Circuit breaker](/ai-gateway/load-balancing/#health-check-and-circuit-breaker). + +### Vector store + +To route requests based on semantic similarity and keep similar requests on the same model instance, you can use a vector store. This is useful for caching consistency, routing to specialized model variants, or matching requests against historical patterns. + +A vector store holds numerical representations (embeddings) of requests and responses so the runtime can match new requests against stored vectors. It powers the [`semantic`](#schema-aigateway-model-config-balancer-algorithm) algorithm and any similarity-matching workflow on the AI Model. Configure storage through [`config.balancer.vectordb`](#schema-aigateway-model-config-balancer-vectordb) by selecting a `strategy`: + +{% table %} +columns: + - title: Strategy + key: strategy + - title: Connection details + key: details +rows: + - strategy: "`redis`" + details: "Connects to Redis with Vector Similarity Search (VSS), AWS MemoryDB for Redis, or Valkey. {{site.ai_gateway}} auto-detects Valkey from the server name field and uses the Valkey-specific driver." + - strategy: "`pgvector`" + details: "Connects to PostgreSQL with the pgvector extension." +{% endtable %} + +For deeper background on vector storage and similarity matching, see [Embedding-based similarity matching](/ai-gateway/semantic-similarity/). + +### Embeddings + +Configure an embedding model to enable semantic routing. This lets {{site.ai_gateway}} route requests based on meaning and content similarity rather than just cost or latency. For example, route domain-specific queries to specialized providers or keep similar requests on the same provider for consistency. + +Set [`config.balancer.embeddings`](#schema-aigateway-model-config-balancer-embeddings) to reference an AI Model Provider and embedding model name. Supported provider types: `azure`, `bedrock`, `databricks`, `gemini`, `huggingface`, `vercel`, `vertex`. The embedding model also powers the `semantic` load balancing algorithm. + +## Templating + +The AI Model resolves runtime values from request data using placeholder substitution. This lets you select the target model dynamically per request, route to per-deployment Azure endpoints, or fan out to multiple providers from a single AI Model. + +Substitution applies to the [`name`](#schema-aigateway-model-targets-name) of each target model and to any per-target [`config`](#schema-aigateway-model-targets-config) option. Three placeholders are available: + +* `$(headers.header_name)`: the value of a request header. +* `$(uri_captures.path_parameter_name)`: the value of a captured URI path parameter. +* `$(query_params.query_parameter_name)`: the value of a query string parameter. + +For examples of using templating, consult the {{site.ai_gateway}} documentation and API reference. + +## Model aliasing + +By default, applications or services making requests to the AI Model endpoint must specify the actual upstream model name (like `gpt-4o`) in the `model` field. If you want to allow them to use a different name, for abstraction, stability, or to hide implementation details, set [`config.model.alias`](#schema-aigateway-model-config-model-alias). + +When an alias is set, clients can send that alias in the request `model` field instead of the upstream model name. This is useful when you want to decouple your client API from upstream provider changes. For example, you could expose an alias like `production-chat-model` while swapping the underlying upstream model from `gpt-4o` to `claude-3-sonnet` without your clients noticing. + +## Access control + +To limit which teams or applications can call an AI Model, use the [`access.acls`](#schema-aigateway-model-access) field to set an allow list or a deny list. Reference [AI Consumers](/ai-gateway/entities/ai-consumer/) (individual applications), [AI Consumer Groups](/ai-gateway/entities/ai-consumer-group/) (teams), or Authenticated Groups (all consumers authenticated via a specific OAuth2 scope or claim) by name. + +To control how consumers authenticate before their access is evaluated, configure the [`access.identity_providers`](#schema-aigateway-model-access-identity-providers) array with one or more [AI Identity Provider](/ai-gateway/entities/ai-identity-provider/) references. Each AI Model supports one `key-auth` identity provider and one `openid-connect` identity provider simultaneously. + +## Attach AI Policies + +Attach an AI Policy to an AI Model to add security, observability, governance, rate limiting, and cost optimization to all requests through that model. For example, you can add guardrails ([AI Prompt Guard](/ai-gateway/policies/ai-prompt-guard/), [AI Lakera Guard](/ai-gateway/policies/ai-lakera-guard/)), enable [logging and metrics](/ai-gateway/policies/?category=logging), audit and [compliance controls](/ai-gateway/policies/ai-sanitizer/), cache responses, or [rate-limit](/ai-gateway/policies/ai-rate-limiting-advanced/) LLM traffic. + +Reference AI Policies through the [`policies`](#schema-aigateway-model-policies) field, which accepts AI Policy names or IDs. You can attach multiple AI Policies to a single AI Model; each applies independently, and the same AI Policy type can be attached with different configurations. Not every AI Policy type supports AI Model attachment. AI Policies are not deleted when the AI Model is deleted, only the AI Model's reference is removed. For more details, see the [AI Policy entity](/ai-gateway/entities/ai-policy/). + +### AI Policy execution order + +AI Policies attach to AI Models and execute in a defined order based on policy type. Authentication policies run early to verify access. Other policies run after routing is resolved. If execution order matters for your use case, refer to the [plugin priority documentation](/gateway/entities/plugin/#plugin-priority). + +## Upstream proxy configuration + +When your data plane sits behind a corporate firewall or security boundary, configure a forward proxy to route all outbound AI provider requests through your organization's proxy. This is required when direct internet access is restricted and all external traffic must pass through a bastion host or inspection gateway. + +Use the [`config.proxy`](#schema-aigateway-model-config-proxy) object to specify the proxy endpoint with [`http_proxy`](#schema-aigateway-model-config-proxy-http-proxy) or [`https_proxy`](#schema-aigateway-model-config-proxy-https-proxy), and optionally add [`auth`](#schema-aigateway-model-config-proxy-auth) credentials if the proxy requires authentication. Use [`no_proxy`](#schema-aigateway-model-config-proxy-no-proxy) to bypass the proxy for specific hosts that are already inside your trusted network. + +## Logging and observability + +Enable [`statistics`](#schema-aigateway-model-config-logging-statistics) logging to track token consumption, request latency, and per-provider costs. This data flows into {{site.konnect_short_name}} analytics and any attached logging AI Policies, letting you monitor API spend, identify slow providers, and audit which AI Models drive the most usage. + +Optionally enable [`payloads`](#schema-aigateway-model-config-logging-payloads) to capture full request and response bodies. This is useful for debugging model responses, auditing sensitive operations, or replaying requests. + +{:.warning} +> Payload logging may expose sensitive data in your logging destination. Only enable it when your logging pipeline is prepared to handle request and response bodies, and verify that logging destinations comply with your data residency and privacy policies. + +For response streaming behavior, see [Streaming](/ai-gateway/streaming/). + +## Set up an AI Model + +Before creating an AI Model, first create an AI Model Provider to store credentials for the upstream LLM service. + +The following example: +* Creates an OpenAI Model that exposes the `generate` capability, routed through a single OpenAI Provider, with token usage logging enabled. +* References a provider named `my-openai-account`. Either see [Set up an AI Model Provider](/ai-gateway/entities/ai-model-provider/#set-up-an-ai-model-provider) to create it, or substitute the name of your own AI Model Provider in `targets[].provider`. + +{:.info} +> This AI Model proxies client requests to `/v1/chat/completions`. The base path `/v1` comes from [`config.route.paths`](#schema-aigateway-model-config-route-paths), and `/chat/completions` is appended by the `generate` capability automatically. + +{% entity_example %} +type: model +data: + display_name: my-gpt-4o + name: my-gpt-4o + type: model + capabilities: + - generate + formats: + - type: openai + policies: [] + targets: + - name: gpt-4o + provider: my-openai-account + config: + type: openai + config: + route: + paths: + - /v1 + logging: + statistics: true + payloads: false + model: + alias: my-gpt-4o +{% endentity_example %} + +{:.info} +> Because [`config.model.alias`](#schema-aigateway-model-config-model-alias) is set here, requests through this AI Model must send `"model": "my-gpt-4o"` (the alias) in the request body instead of the upstream target name (`gpt-4o`). Sending the upstream target name instead of the alias fails. + + +## Schema + +{% entity_schema %} diff --git a/app/_ai_gateway_entities/ai-policy.md b/app/_ai_gateway_entities/ai-policy.md new file mode 100644 index 00000000000..2cfef4ae988 --- /dev/null +++ b/app/_ai_gateway_entities/ai-policy.md @@ -0,0 +1,138 @@ +--- +title: AI Policies +content_type: reference +entities: + - ai-policy +products: + - ai-gateway +min_version: + ai-gateway: '2.0' +permalink: /ai-gateway/entities/ai-policy/ +breadcrumbs: + - /ai-gateway/ + - /ai-gateway/entities/ +description: "AI Policies for {{site.ai_gateway}}." +schema: + api: konnect/ai-gateway + path: /schemas/AIGatewayPolicy +works_on: + - konnect +tools: + - konnect-api + - kongctl +related_resources: + - text: "About {{site.ai_gateway}}" + url: /ai-gateway/ + - text: AI Model entity + url: /ai-gateway/entities/ai-model/ + - text: AI Agent entity + url: /ai-gateway/entities/ai-agent/ + - text: AI MCP Server entity + url: /ai-gateway/entities/ai-mcp-server/ + +faqs: + - q: Are AI Policies shared across multiple entities? + a: | + No. Each AI Policy is an independent configuration. To apply the same + configuration to two AI Models, create two AI Policies with matching `config`, + one per AI Model. + + - q: How is an AI Policy different from a plugin? + a: | + An AI Policy is a policy configuration created through the {{site.ai_gateway}} entity surface + instead of the classic `/plugins` endpoint. The runtime effect is the same: a policy attached + at the appropriate scope. {{site.ai_gateway}} manages the AI Policy's lifecycle alongside the + entity it's attached to. + + - q: Can an AI Policy be scoped to an AI Consumer or AI Consumer Group? + a: | + Yes. Add the AI Policy's `name` or `id` to the AI Consumer's or AI Consumer Group's `policies` array. + The Policy runs when the AI Consumer is identified during a request, or when a member of the + AI Consumer Group is identified. + + - q: What happens to an AI Policy when its parent entity is deleted? + a: | + Standalone AI Policies referenced from parent entities through a `policies` array are independent + and aren't deleted when a referencing parent is deleted. The reference is simply removed. +--- + +## What is an AI Policy? + +Create an AI Policy when you want to add governance, security, transformation, or observability to {{site.ai_gateway}} traffic: +- Attach [AI PII Sanitizer](/ai-gateway/policies/ai-sanitizer/) to redact sensitive data +- Attach [AI Rate Limiting Advanced](/ai-gateway/policies/ai-rate-limiting-advanced/) to manage request volume +- Attach [AI Prompt Guard](/ai-gateway/policies/ai-prompt-guard/) or [other guardrail Policies](/ai-gateway/#guardrails-and-content-safety) to validate prompts +- Attach [logging Policies](/ai-gateway/policies/?category=logging) to track requests and responses for observability +- Attach authentication policies like [OpenID Connect](/ai-gateway/policies/openid-connect/) to control access and verify identity + +**Each AI Policy is independent.** To apply the same configuration across multiple entities, create separate Policies for each one. This ensures that deleting an entity deletes only its own Policies, not configurations shared with other parts of your gateway. + +{:.info} +> For the complete set of available policy types and configurations, see the [AI Policies hub](/ai-gateway/policies/). + +## Manage AI Policies + +AI Policies are managed through: + +* {{site.konnect_short_name}} UI +* {{site.ai_gateway}} API: `/v1/ai-gateways/{aiGatewayId}/policies` +* [kongctl](/kongctl/) + +For configuration examples and step-by-step setup instructions, see [Set up a global AI Policy](#set-up-a-global-ai-policy). + +## AI Policy scopes + +An AI Policy's scope is determined by where it's referenced. Each AI Policy is an independent configuration that applies at exactly one scope. To apply identical configuration in multiple places, create one AI Policy per target. + +The available scopes are: + +* **Global**: An AI Policy with no parent entity reference applies to all {{site.ai_gateway}} traffic on the data plane. Non-AI traffic on the same data plane isn't affected. + +* **Entity-scoped**: Reference the policy from the `policies` array on an [AI Model](/ai-gateway/entities/ai-model/), [AI Agent](/ai-gateway/entities/ai-agent/), [AI MCP Server](/ai-gateway/entities/ai-mcp-server/), [AI Consumer](/ai-gateway/entities/ai-consumer/), or [AI Consumer Group](/ai-gateway/entities/ai-consumer-group/) entity. The policy applies at that entity's scope. + +{:.info} +> For each policy type, find its configuration schema and required fields on that policy's reference page in the [AI Policies hub](/ai-gateway/policies/). Configuration is specific to each policy type. + +## Set up a global AI Policy + +An AI Policy specifies a `type` (like AI Sanitizer or AI Rate Limiting Advanced) and a `config` block that configures that behavior. {{site.ai_gateway}} applies the policy at the scope you choose: globally across all traffic, or scoped to a specific entity. + +The following example creates a global AI PII Sanitizer Policy that runs for every {{site.ai_gateway}} Route. It anonymizes high-risk PII categories (email, phone, SSN, and credit cards) along with custom patterns for sensitive tokens like AWS API keys and GitHub tokens. + +{:.info} +> This Policy connects to an AI PII Anonymizer service at `host`/`port` (`sanitizer-service.internal:8080` in this example) to perform the actual sanitization. Substitute the address of your own running instance. See [AI PII Anonymizer service](/ai-gateway/policies/ai-sanitizer/#ai-pii-anonymizer-service) for image access and setup instructions. +> +> Without a reachable service at that address, requests through this Policy will fail. + +{% entity_example %} +type: policy +data: + display_name: PII Sanitizer - Global + name: pii-sanitizer-global + type: ai-sanitizer + enabled: true + global: true + config: + anonymize: + - email + - phone + - ssn + - creditcard + - custom + custom_patterns: + - name: aws_api_key + regex: AKIA[0-9A-Z]{16} + score: 0.95 + - name: github_token + regex: ghp_[A-Za-z0-9]{36} + score: 0.9 + host: sanitizer-service.internal + port: 8080 + redact_type: placeholder + stop_on_error: true + recover_redacted: false +{% endentity_example %} + +## Schema + +{% entity_schema %} diff --git a/app/_ai_gateway_entities/ai-provider.md b/app/_ai_gateway_entities/ai-provider.md new file mode 100644 index 00000000000..78cfdf74ee4 --- /dev/null +++ b/app/_ai_gateway_entities/ai-provider.md @@ -0,0 +1,173 @@ +--- +title: AI Model Providers +content_type: reference +entities: + - ai-model-provider +products: + - ai-gateway +min_version: + ai-gateway: '2.0' +permalink: /ai-gateway/entities/ai-model-provider/ +breadcrumbs: + - /ai-gateway/ + - /ai-gateway/entities/ +description: AI Model Provider credentials and configuration used by {{site.ai_gateway}}. +schema: + api: konnect/ai-gateway + path: /schemas/AIGatewayModelProvider +works_on: + - konnect +tools: + - konnect-api + - kongctl +related_resources: + - text: "About {{site.ai_gateway}}" + url: /ai-gateway/ + - text: "{{site.ai_gateway}} providers" + url: /ai-gateway/ai-providers/ + - text: AI Model entity + url: /ai-gateway/entities/ai-model/ + - text: AI Policy entity + url: /ai-gateway/entities/ai-policy/ + - text: AI Identity Provider entity + url: /ai-gateway/entities/ai-identity-provider/ +faqs: + - q: What happens when I update an AI Model Provider's credentials? + a: | + {{site.ai_gateway}} propagates the credential change to every AI Model that references the + AI Model Provider (by `name` or `id`). The next request through any of those AI Models uses the updated + credentials. + + - q: How does an AI Model reference an AI Model Provider? + a: | + Set the `provider` field in each item of the [`targets`](/ai-gateway/entities/ai-model/#schema-aigateway-model-targets) array on the AI Model to the AI Model Provider's `name` or `id`. +--- + +## What is an AI Model Provider? + +The AI Model Provider entity lets you securely store and manage credentials for connecting to upstream LLM services. Use AI Model Providers to: +* Store API keys for OpenAI, Azure, Bedrock, or any other LLM provider +* Centrally manage and rotate credentials across multiple [AI Models](/ai-gateway/entities/ai-model/) +* Enforce consistent authentication across your deployments + +An AI Model Provider manages outbound credentials, which is distinct from the inbound authentication managed by an [AI Identity Provider](/ai-gateway/entities/ai-identity-provider/). When an AI Consumer calls an AI Model, the AI Identity Provider checks who they are. The AI Model then uses the AI Model Provider's credentials to forward the request upstream. + +Each AI Model Provider has a [`type`](#schema-aigateway-model-provider-type) that selects the upstream LLM service and configures provider-specific options. See the [schema](#schema) for supported types, and the per-provider pages under [{{site.ai_gateway}} providers](/ai-gateway/ai-providers/) for provider-specific configuration and limitations. + +## Manage AI Model Providers + +AI Model Providers can be created and managed through: + +* {{site.konnect_short_name}} UI +* {{site.ai_gateway}} API: `/v1/ai-gateways/{aiGatewayId}/model-providers` +* [kongctl](/kongctl/) + +For configuration examples and step-by-step setup instructions, see [Set up an AI Model Provider](#set-up-an-ai-model-provider). + +### Relationship to AI Models + +AI Model Providers and AI Models have a many-to-many relationship: one AI Model Provider can back many AI Models, and one AI Model can route to multiple AI Model Providers. For example, a single `openai` AI Model Provider might be used by both a chat AI Model and an embeddings AI Model, while a single AI Model might route to OpenAI and Anthropic targets for failover. + +When configuring an [AI Model](/ai-gateway/entities/ai-model/), you reference an AI Model Provider by setting the `provider` field in each item of the [`targets`](/ai-gateway/entities/ai-model/#schema-aigateway-model-targets) array. You can reference by [`name`](#schema-aigateway-model-provider-name) or `id`. Use `id` if you plan to rename the AI Model Provider later. + +## Supported upstream LLM providers + +{{site.ai_gateway}} supports the following upstream LLM providers. The AI Model Provider's [`type`](#schema-aigateway-model-provider-type) field selects one of these targets. The following provider-specific pages document supported capabilities, configuration requirements, and limitations. + +{% html_tag type="div" css_classes="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-3" %} +{% icon_card icon="openai.svg" title="OpenAI" cta_url="/ai-gateway/ai-providers/openai/" %} +{% icon_card icon="azure.svg" title="Azure OpenAI" cta_url="/ai-gateway/ai-providers/azure/" %} +{% icon_card icon="bedrock.svg" title="Amazon Bedrock" cta_url="/ai-gateway/ai-providers/bedrock/" %} +{% icon_card icon="anthropic.svg" title="Anthropic" cta_url="/ai-gateway/ai-providers/anthropic/" %} +{% icon_card icon="gemini.svg" title="Gemini" cta_url="/ai-gateway/ai-providers/gemini/" %} +{% icon_card icon="vertex.svg" title="Vertex AI" cta_url="/ai-gateway/ai-providers/vertex/" %} +{% icon_card icon="cohere.svg" title="Cohere" cta_url="/ai-gateway/ai-providers/cohere/" %} +{% icon_card icon="mistral.svg" title="Mistral" cta_url="/ai-gateway/ai-providers/mistral/" %} +{% icon_card icon="huggingface.svg" title="Hugging Face" cta_url="/ai-gateway/ai-providers/huggingface/" %} +{% icon_card icon="kimi.svg" title="Kimi" cta_url="/ai-gateway/ai-providers/kimi/" %} +{% icon_card icon="metaai.svg" title="Llama" cta_url="/ai-gateway/ai-providers/llama/" %} +{% icon_card icon="xai.svg" title="xAI" cta_url="/ai-gateway/ai-providers/xai/" %} +{% icon_card icon="dashscope.svg" title="Alibaba Cloud DashScope" cta_url="/ai-gateway/ai-providers/dashscope/" %} +{% icon_card icon="cerebras.svg" title="Cerebras" cta_url="/ai-gateway/ai-providers/cerebras/" %} +{% icon_card icon="deepseek.svg" title="DeepSeek" cta_url="/ai-gateway/ai-providers/deepseek/" %} +{% icon_card icon="ollama.svg" title="Ollama" cta_url="/ai-gateway/ai-providers/ollama/" %} +{% icon_card icon="databricks.svg" title="Databricks" cta_url="/ai-gateway/ai-providers/databricks/" %} +{% icon_card icon="vercel.svg" title="Vercel" cta_url="/ai-gateway/ai-providers/vercel/" %} +{% icon_card icon="vllm.svg" title="vLLM" cta_url="/ai-gateway/ai-providers/vllm/" %} +{% endhtml_tag %} + +## Outbound authentication + +The [`config.auth`](#schema-aigateway-model-provider-config-auth) object declares how {{site.ai_gateway}} authenticates to the upstream AI provider. The shape of `auth` depends on the AI Model Provider's [`type`](#schema-aigateway-model-provider-type): + +* **`basic`**: Header- or parameter-based auth. Supports up to one auth header (`config.auth.headers`) and one auth parameter (`config.auth.params`). Parameters can be sent as a query string or in the request body (`config.auth.params[].location`). Used by most AI Model Provider types. +* **`aws`**: IAM access-key and assume-role auth. Used by [Bedrock](/ai-gateway/ai-providers/bedrock/). +* **`azure`**: Microsoft Entra ID or managed-identity auth. Used by [Azure OpenAI](/ai-gateway/ai-providers/azure/). +* **`gcp`**: Google service-account auth. Used by [Gemini](/ai-gateway/ai-providers/gemini/) and [Vertex AI](/ai-gateway/ai-providers/vertex/). + +{:.info} +> Bedrock, Azure OpenAI, Gemini, and Vertex AI can also fall back to `basic` auth. + +{% table %} +columns: + - title: Auth type + key: type + - title: Provider name + key: providers + - title: Primary approach + key: approach + - title: Fallback auth + key: fallback +rows: + - type: "`aws`" + providers: "[Bedrock](/ai-gateway/ai-providers/bedrock/)" + approach: "IAM via static credentials, assume role, or environment auto-detection (EC2 instance profiles, environment variables, local AWS config). Role assumption recommended for production. Cross-account access supported. Use `config.auth.batch_role_arn` to specify a separate IAM role for Bedrock batch API calls." + fallback: "`basic`" + - type: "`azure`" + providers: "[Azure OpenAI](/ai-gateway/ai-providers/azure/)" + approach: "Microsoft Entra ID via Managed Identity (recommended when running in Azure). For explicit credentials, provide client ID, secret, and tenant ID. Requires `config.instance` (your Azure instance name, for example `kong-az-east`)." + fallback: "`basic`" + - type: "`gcp`" + providers: "[Gemini](/ai-gateway/ai-providers/gemini/), [Vertex AI](/ai-gateway/ai-providers/vertex/)" + approach: "Google service accounts via environment auto-detection (service account JSON or Compute Engine metadata server). For restricted networks, set `config.auth.metadata_url` or `config.auth.oauth_token_url` to custom endpoints." + fallback: "`basic`" +{% endtable %} + +## Lifecycle + +An AI Model Provider stores the credentials, but doesn't generate any runtime primitives. + +AI Model Provider credentials are passed to the runtime only when an AI Model references the AI Model Provider. At that point, the credentials are then passed to the AI Model. + +When you update the credentials of an AI Model Provider, the new credentials are passed to every AI Model that references it the next time a request is made through the AI Model. + +## AI Policies and AI Model Providers + +You can't attach [AI Policies](/ai-gateway/entities/ai-policy/) directly to an AI Model Provider entity instance. AI Policies attach to [AI Models](/ai-gateway/entities/ai-model/), [AI Agents](/ai-gateway/entities/ai-agent/), [AI Consumers](/ai-gateway/entities/ai-consumer/), or [AI Consumer Groups](/ai-gateway/entities/ai-consumer-group/) to control security, rate limiting, guardrails, and observability. + +To apply an AI Policy across requests using a particular AI Model Provider, you can: +1. Set the policy to `global: true` to apply it to all resources in the gateway. +1. Attach the same policy to each AI Model that references the AI Model Provider. +1. Create an AI Consumer Group with the policy and control access to AI Models via ACLs. + +## Set up an AI Model Provider + +The following example creates an OpenAI AI Model Provider that authenticates with a single bearer-token header. An AI Model can then route to this AI Model Provider by setting the `provider` field in a `targets` array item to `my-openai-account` (or the AI Model Provider `id`). + +{% entity_example %} +type: model-provider +data: + display_name: OpenAI Production + name: my-openai-account + type: openai + config: + auth: + type: basic + headers: + - name: Authorization + value: Bearer +{% endentity_example %} + +## Schema + +{% entity_schema %} diff --git a/app/_ai_gateway_entities/ai-vault.md b/app/_ai_gateway_entities/ai-vault.md new file mode 100644 index 00000000000..0903442d9ae --- /dev/null +++ b/app/_ai_gateway_entities/ai-vault.md @@ -0,0 +1,295 @@ +--- +title: AI Vaults +content_type: reference +entities: + - ai-vault + - ai-model-provider +products: + - ai-gateway +min_version: + ai-gateway: '2.0' +permalink: /ai-gateway/entities/ai-vault/ +breadcrumbs: + - /ai-gateway/ + - /ai-gateway/entities/ +description: AI Vaults for storing and referencing secrets used by {{site.ai_gateway}} entities. +schema: + api: konnect/ai-gateway + path: /schemas/AIGatewayVault +works_on: + - konnect +tools: + - konnect-api + - kongctl +related_resources: + - text: "About {{site.ai_gateway}}" + url: /ai-gateway/ + - text: AI Model Provider + url: /ai-gateway/entities/ai-model-provider/ + - text: AI Identity Provider + url: /ai-gateway/entities/ai-identity-provider/ + - text: AI Model + url: /ai-gateway/entities/ai-model/ + - text: AI MCP Server + url: /ai-gateway/entities/ai-mcp-server/ + - text: AI Consumer Credential + url: /ai-gateway/entities/ai-consumer/#create-consumer-credentials +faqs: + - q: How is an {{site.ai_gateway}} AI Vault different from a {{site.base_gateway}} Vault? + a: | + The runtime entity is the same secret-management abstraction. The {{site.ai_gateway}} surface + manages AI Vaults through the AI entity convention (`name`, `description`, + `labels`) and exposes them through the {{site.konnect_short_name}} API alongside the other AI entities. + Unlike other {{site.ai_gateway}} entities, AI Vaults don't have a `display_name` field. + + - q: Which secret backends are supported? + a: | + The `type` field selects the backend: `konnect`, `env`, `aws`, `gcp`, `azure`, `conjur`, or `hcv`. + Each type carries its own `config` shape. HashiCorp Vault (`hcv`) further selects an + `auth_method` from `token`, `cert`, `jwt`, `approle`, `kubernetes`, `gcp_iam`, `gcp_gce`, + `aws_ec2`, `aws_iam`, or `azure`. + + - q: How are AI Vault secrets referenced from other {{site.ai_gateway}} entities? + a: | + Sensitive fields on AI Model Provider, AI Identity Provider, AI Model, AI MCP Server, and other entities are annotated as + referenceable. Set those fields to a vault reference string (for example, a `{vault://...}` + placeholder) instead of a literal value. The AI Vault `name` is the lookup key. + + - q: What does `name` control? + a: | + `name` is a user-defined unique identifier and the stable handle used to look up the AI Vault + configuration when other entities reference secrets. Renaming an AI Vault breaks any reference + pointing at the old value. + + - q: How do I add secrets to a `konnect`-type AI Vault? + a: | + A `konnect`-type AI Vault doesn't hold secret values itself. It references a Config Store by + [`config.config_store_id`](#konnect-config-store), and you create and manage the actual + secrets through the Config Store's own API. For more information, see [Konnect Config Store](#konnect-config-store). +--- + +## What is an AI Vault? + +You must store secrets like API keys and authentication tokens somewhere secure instead of embedding them directly in your configurations. An AI Vault entity lets you register an external secret backend (AWS Secrets Manager, HashiCorp Vault, environment variables, or others) so that [AI Model Providers](/ai-gateway/entities/ai-model-provider/), [AI Identity Providers](/ai-gateway/entities/ai-identity-provider/), [AI Models](/ai-gateway/entities/ai-model/), and [AI MCP Servers](/ai-gateway/entities/ai-mcp-server/) can reference secrets instead of storing them as literal values. + +An AI Vault entity stores the connection configuration and credentials needed to reach your secret backend. When other entities reference a secret, {{site.ai_gateway}}: +1. Looks up the vault at request time +1. Retrieves the actual secret value +1. Uses it for authentication or configuration. + +## Manage AI Vaults + +AI Vaults can be created and managed through: + +* {{site.konnect_short_name}} UI +* [{{site.ai_gateway}} API](/api/konnect/ai-gateway/): `/ai-gateways/{aiGatewayId}/vaults` +* [kongctl](/kongctl/) + +For configuration examples and step-by-step setup instructions, see [Set up an AI Vault](#set-up-an-ai-vault). + +## Backends + +Each AI Vault selects one of the supported secret backends: + +* [{{site.konnect_short_name}} Config Store](#konnect-config-store) +* Environment variables +* [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/) +* [Google Secret Manager](https://cloud.google.com/secret-manager) +* [Azure Key Vault](https://azure.microsoft.com/en-us/products/key-vault) +* [CyberArk Conjur](https://www.conjur.org/) +* [HashiCorp Vault](https://www.vaultproject.io/) + +The connection details vary per backend; the {{site.konnect_short_name}} UI surfaces the relevant fields based on the backend you choose. + +## Which fields support AI Vault references? + +AI Vault references can be used in sensitive fields across your {{site.ai_gateway}} entities: + +{% table %} +columns: + - title: Entity + key: entity + - title: Sensitive fields + key: fields +rows: + - entity: AI Model Provider + fields: Authentication credentials (API keys, bearer tokens) in auth headers for upstream LLM providers + - entity: AI Identity Provider + fields: OIDC client secret for openid-connect type providers + - entity: AI Model + fields: Backend-specific authentication required by target model configurations + - entity: AI MCP Server + fields: Encryption keys used by MCP Servers for client session management + - entity: AI Consumer + fields: API keys and tokens issued to downstream consumers +{% endtable %} + +{:.success} +> Any field marked as supporting vault references can accept a secret reference instead of a literal value. + +## How do I reference secrets? + +To reference a secret stored in a vault, use the syntax: + +``` +{vault://vault-name/secret-key} +``` + +Where: +- `vault-name` is the `name` field of the vault you created +- `secret-key` is the identifier of the secret within that vault (exact format depends on the backend) + +For example, if you created a vault named `prod-aws-vault` and stored an OpenAI API key under the key `openai-api-key`, reference it as: + +``` +{vault://prod-aws-vault/openai-api-key} +``` + +Here's how you'd use that reference in an AI Model Provider entity: + +{% entity_example %} +type: model-provider +data: + display_name: OpenAI Production + name: openai-prod + type: openai + config: + auth: + type: basic + headers: + - name: Authorization + value: "{vault://prod-aws-vault/openai-api-key}" +{% endentity_example %} + +{:.warning} +> The entire field value must be the vault reference string. You cannot use partial references like `Bearer {vault://...}`. The field itself must be exactly `{vault://vault-name/secret-key}`. + +At request time, {{site.ai_gateway}} resolves the reference by looking up the vault name, retrieving the secret value, and using it for authentication or configuration. + +## Choosing a backend for your AI Vault + +Pick a backend matching your infrastructure and secret management strategy. Cloud-native deployments can use their platform's secret service (`aws`, `gcp`, `azure`), enterprises can use dedicated secret management systems (`conjur`, `hcv`), and smaller deployments can use `env` (environment variables) or `konnect` (built-in Config Store). + + +{% table %} +columns: + - title: Backend + key: backend + - title: When to use + key: when +rows: + - backend: "`konnect`" + when: Getting started, no external dependencies. Built-in {{site.konnect_short_name}} Config Store for teams without existing secret infrastructure. + - backend: "`env`" + when: Development, edge deployments, or environments where you control data plane startup. Secrets loaded at startup, no network calls. + - backend: "`aws`" + when: AWS-deployed data planes. Integrate with AWS Secrets Manager or Parameter Store. + - backend: "`gcp`" + when: GCP-deployed data planes. Integrate with Google Secret Manager. + - backend: "`azure`" + when: Azure-deployed data planes. Integrate with Azure Key Vault. + - backend: "`conjur`" + when: Enterprises standardized on CyberArk Conjur for centralized secrets management. + - backend: "`hcv`" + when: Dedicated secret management with fine-grained access control. Supports token, AppRole, JWT, Kubernetes, AWS IAM, GCP, and Azure authentication. +{% endtable %} + + +## Caching and availability + +Cloud-backed vault types (`aws`, `gcp`, `azure`, `conjur`, `hcv`) cache resolved secrets so {{site.ai_gateway}} doesn't hit the backend on every request. This reduces latency and vault load. The `env` backend doesn't cache because environment-variable lookups are local. + +If your vault becomes unreachable, {{site.ai_gateway}} can continue using recently-cached secrets for a grace period, keeping your system operational during brief vault outages. This allows you to maintain service continuity even when secret infrastructure is temporarily unavailable. + +Cache duration and grace periods are tunable per vault, allowing you to balance between fresh secrets (shorter cache times) and reduced vault requests (longer cache times). The default settings work for most deployments; adjust only if your secret rotation strategy or vault reliability requires custom behavior. + +## {{site.konnect_short_name}} Config Store + +Unlike the other backends, the `konnect` type doesn't connect out to an external secret manager. +It stores secrets directly in {{site.konnect_short_name}}, in a Config Store: a named container of key-value secrets that you create and populate through its own API, separate from the AI Vault entity itself. + +A `konnect`-type AI Vault doesn't hold any secret values. It only references a Config Store by ID through `config.config_store_id`. The Config Store holds the actual secrets. + +{:.info} +> Secret values are write-only. Once stored, {{site.ai_gateway}} never returns the value again, only the secret's `key` and timestamps. + +### Manage Config Stores + +Config Stores are managed through the {{site.ai_gateway}} API: + +* Config Store: [`/ai-gateways/{aiGatewayId}/config-stores`](/api/konnect/ai-gateway/#/operations/create-ai-gateway-config-store) +* Config Store secrets: [`/ai-gateways/{aiGatewayId}/config-stores/{configStoreIdOrName}/secrets`](/api/konnect/ai-gateway/#/operations/create-ai-gateway-config-store-secret) + +Both support full create, list, get, update, and delete operations. +Deleting a Config Store that still has secrets fails unless you pass `?force=true`, which cascades the delete to all secrets in that Config Store. + +### Create a Config Store and add a secret + +The following example creates a Config Store: + + +{% konnect_api_request %} +url: /v1/ai-gateways/$AI_GATEWAY_ID/config-stores +status_code: 201 +method: POST +headers: + - 'Content-Type: application/json' + - 'Accept: application/json, application/problem+json' +body: + name: prod-secrets +{% endkonnect_api_request %} + +Add a secret to the Config Store: +{% konnect_api_request %} +url: /v1/ai-gateways/$AI_GATEWAY_ID/config-stores/$CONFIG_STORE_ID/secrets +status_code: 201 +method: POST +headers: + - 'Content-Type: application/json' + - 'Accept: application/json, application/problem+json' +body: + key: openai-api-key + value: sk-my-openai-key +{% endkonnect_api_request %} + + +### Reference the Config Store from a konnect-type AI Vault + +Create a `konnect`-type AI Vault that points at the Config Store's `id`: + +{% entity_example %} +type: vault +data: + name: prod-config-store-vault + description: Vault backed by the built-in Konnect Config Store. + type: konnect + config: + config_store_id: $CONFIG_STORE_ID +{% endentity_example %} + +Reference the secret the same way as any other AI Vault: + +``` +{vault://prod-config-store-vault/openai-api-key} +``` + +## Set up an AI Vault + +The following example registers an environment-variable AI Vault that resolves references against process environment variables prefixed with `KONG_`. + +{:.info} +> AI Vault doesn't accept a `display_name` field. Only `name` and `description` identify a vault. If you include `display_name` when creating an AI Vault, {{site.ai_gateway}} silently ignores it. + +{% entity_example %} +type: vault +data: + name: prod-env-vault + description: Vault for production secrets sourced from environment variables. + type: env + config: + prefix: KONG_ +{% endentity_example %} + +## Schema + +{% entity_schema %} diff --git a/app/_ai_gateway_policies/ace/index.md b/app/_ai_gateway_policies/ace/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/ace/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/acl/index.md b/app/_ai_gateway_policies/acl/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/acl/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/acme/index.md b/app/_ai_gateway_policies/acme/index.md new file mode 100644 index 00000000000..e0b91320d1e --- /dev/null +++ b/app/_ai_gateway_policies/acme/index.md @@ -0,0 +1,10 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +description: Let's Encrypt and ACMEv2 integration with {{site.ai_gateway_name}} +--- diff --git a/app/_ai_gateway_policies/ai-aws-guardrails/index.md b/app/_ai_gateway_policies/ai-aws-guardrails/index.md new file mode 100644 index 00000000000..21f980ef309 --- /dev/null +++ b/app/_ai_gateway_policies/ai-aws-guardrails/index.md @@ -0,0 +1,55 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- + + +The AI AWS Guardrails Policy enforces introspection on both inbound requests and outbound responses handled by the [AI Model](/ai-gateway/entities/ai-model/) entity. It integrates with the [AWS Bedrock Guardrails](https://aws.amazon.com/bedrock/guardrails/) service to apply compliance and safety policies at the Gateway level. This ensures all data exchanged between clients and upstream LLMs adheres to the configured security standards. + +## Prerequisites + +Before using the AI AWS Guardrails Policy, you must define your guardrail policies in AWS. You can do this through: + +* The [AWS Console](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-components.html) +* The [CreateGuardrail API](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_CreateGuardrail.html) + +## How it works + +The AI AWS Guardrails Policy includes a configurable [`response_buffer_size`](/ai-gateway/policies/ai-aws-guardrails/reference/#schema--config-response-buffer-size) parameter. This setting controls how many tokens from the upstream LLM response are buffered during streaming before being sent to the AWS Guardrails service for inspection. For example, setting `response_buffer_size` to `50` means the AI AWS Guardrails Policy will collect 50 tokens from the upstream model before sending them to AWS Guardrails for evaluation. Guardrail evaluation runs in chunks as tokens stream in. + +{:.info} +> A smaller buffer size allows faster policy evaluation and quicker response rejection but may increase the number of guardrail calls. Larger sizes reduce API calls but may delay policy enforcement. + +For response and request inspection, the Policy by default guards input only. You can change this behavior with the [`guarding_mode`](/ai-gateway/policies/ai-aws-guardrails/reference/#schema--config-guarding-mode) field, which supports `INPUT`, `OUTPUT`, or `BOTH`. To control which parts of the conversation are sent for content evaluation, use the [`text_source`](/ai-gateway/policies/ai-aws-guardrails/reference/#schema--config-text-source) field. Set it to `concatenate_user_content` to inspect only `user` input, or `concatenate_all_content` to include the full exchange, including system and assistant messages. + +## Format + +This Policy works with all of the AI Model entity's [`model.capabilities` settings](/ai-gateway/entities/ai-model/#capabilities). + +## AWS IAM roles + +The AI AWS Guardrails Policy supports AWS Identity and Access Management (IAM) roles. This allows the AWS Bedrock Guardrails service to be accessed using role assumption instead of static credentials. + +To use AWS IAM roles with the Policy, set the [`config.aws_assume_role_arn`](/ai-gateway/policies/ai-aws-guardrails/reference/#schema--config-aws-assume-role-arn), and [`config.aws_role_session_name`](/ai-gateway/policies/ai-aws-guardrails/reference/#schema--config-aws-role-session-name). + +{:.info} +> **Note:** These fields can be used with or without static AWS credentials (`config.aws_access_key_id` and `config.aws_secret_access_key`). + +## TLS verification + +[`config.ssl_verify`](/ai-gateway/policies/ai-aws-guardrails/reference/#schema--config-ssl-verify) is enabled by default. The Policy verifies the TLS certificate when connecting to the AWS Bedrock service. To disable this, set `ssl_verify: false`. + +## Logging + +The AI AWS Guardrails Policy emits structured log data for every inspected request and response. For the full list of log fields, see the [{{site.ai_gateway}} audit log reference](/ai-gateway/ai-audit-log-reference/#ai-aws-guardrails-logs). + +To log the raw content of blocked requests and responses, enable [`config.log_blocked_content`](/ai-gateway/policies/ai-aws-guardrails/reference/#schema--config-log-blocked-content). When enabled, the blocked prompt or response body appears under `ai.proxy.aws-guardrails.input_faulty_prompt` and `ai.proxy.aws-guardrails.output_faulty_response` in each log entry. + +## Forward proxy support + +{% include md/ai-gateway/v2/forward-proxy.md %} \ No newline at end of file diff --git a/app/_ai_gateway_policies/ai-azure-content-safety/index.md b/app/_ai_gateway_policies/ai-azure-content-safety/index.md new file mode 100644 index 00000000000..121df97ce7b --- /dev/null +++ b/app/_ai_gateway_policies/ai-azure-content-safety/index.md @@ -0,0 +1,83 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: policy +--- + +The AI Azure Content Safety Policy allows administrators to enforce +introspection with the [Azure AI Content Safety](https://azure.microsoft.com/en-us/products/ai-services/ai-content-safety) service +for all requests and responses handled by the [AI Model](/ai-gateway/entities/ai-model/) entity. +This Policy enables configurable thresholds for the different moderation categories +and you can specify an array set of pre-configured blocklist IDs from your Azure Content Safety instance. + +You can observe and report on audit failures using the [AI logging Policies](/ai-gateway/policies/?category=logging). + +## How it works + +The AI Azure Content Safety Policy can be applied to: +* Input data (requests) +* Output data (responses) +* Both input and output data + +Here's how it works if you apply it to both requests and responses: + +1. The AI Azure Content Safety Policy intercepts the request and sends the request body to the Azure AI Content Safety service. + 1. The Azure AI Content Safety service analyzes the request against configured moderation categories and allows or blocks the request. +1. If allowed, the request is forwarded upstream with the AI Model entity. +1. On the way back, the Policy intercepts the response and sends the response body to the Azure AI Content Safety service. + 1. The Azure AI Content Safety service analyzes the response against configured moderation categories and allows or blocks the response. +1. If allowed, the response is forwarded to the client. + +{% comment %} + +{% mermaid %} +sequenceDiagram + autonumber + participant Client + participant Gateway as {{site.ai_gateway}} + participant Policy as AI Azure Content Safety Policy + participant Safety as Azure AI Content Safety service + participant AI as Upstream AI Service + + Client->>Gateway: Send request + Gateway->>Policy: Route request + Policy->>Safety: Intercept & send request body + Safety->>Safety: Check against moderation
categories and blocklists + Safety->>Policy: Allow or block request + Policy->>Gateway: Forward allowed request + Gateway->>AI: Process allowed request + AI->>Gateway: Return AI response + Gateway->>Policy: Forward response + Policy->>Safety: Intercept & send response body + Safety->>Safety: Check against moderation
categories and blocklists + Safety->>Policy: Allow or block response + Policy->>Gateway: Forward allowed response + Gateway->>Client: Forward allowed response to client +{% endmermaid %} + + +> _Figure 1: Diagram showing the request and response flow with the AI Azure Content Safety Policy._ +{% endcomment %} + +## TLS verification + +[`config.ssl_verify`](/ai-gateway/policies/ai-azure-content-safety/reference/#schema--config-ssl-verify) is enabled by default. The AI Azure Content Safety Policy verifies the TLS certificate when connecting to the Azure Content Safety service. To disable this, set `ssl_verify: false`. + +## Logging + +The AI Azure Content Safety Policy emits structured log data for every inspected request and response. For the full list of log fields, see the [{{site.ai_gateway}} audit log reference](/ai-gateway/ai-audit-log-reference/#ai-azure-content-safety-logs). + +To log the raw content of blocked requests and responses, enable [`config.log_blocked_content`](/ai-gateway/policies/ai-azure-content-safety/reference/#schema--config-log-blocked-content). When enabled, the blocked prompt or response body appears under `ai.proxy.azure-content-safety.input_faulty_prompt` and `ai.proxy.azure-content-safety.output_faulty_response` in the log entry. + +## Format + +This Policy works with all of the AI Model entity's [`model.capabilities` settings](/ai-gateway/entities/ai-model/#capabilities), and is able to +compose an Azure Content Safety text check by compiling all chat history, or just the `'user'` content. + +## Forward proxy support + +{% include md/ai-gateway/v2/forward-proxy.md %} diff --git a/app/_ai_gateway_policies/ai-custom-guardrail/index.md b/app/_ai_gateway_policies/ai-custom-guardrail/index.md new file mode 100644 index 00000000000..a7f0cc4359a --- /dev/null +++ b/app/_ai_gateway_policies/ai-custom-guardrail/index.md @@ -0,0 +1,92 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: policy +--- + +The AI Custom Guardrail Policy enforces introspection on both inbound requests and outbound responses handled by the [AI Model](/ai-gateway/entities/ai-model/) entity. It can integrate with any HTTP-based guardrail service. This ensures all data exchanged between clients and upstream LLMs adheres to the configured security standards. + +## How it works + +The AI Custom Guardrail Policy can be applied to: +* Input data (requests) +* Output data (responses) +* Both input and output data + +Here's how it works if you apply it to both requests and responses: + +1. The AI Custom Guardrail Policy intercepts the request and sends the request body to the guardrail service. + 1. The guardrail service analyzes the request against configured moderation categories and allows or blocks the request. +1. If allowed, the request is forwarded upstream with the AI Model entity. +1. On the way back, the Policy intercepts the response and sends the response body to the guardrail service. + 1. The guardrail service analyzes the response against configured moderation categories and allows or blocks the response. +1. If allowed, the response is forwarded to the client. + +{% comment %} + +{% mermaid %} +sequenceDiagram + autonumber + participant client as Client + participant custguardrail as AI Custom Guardrail Policy + participant guardrail as Guardrail service + participant proxy as AI Proxy/Advanced plugin + participant llm as Upstream AI service + + client->>custguardrail:Send request + custguardrail<<->>guardrail:Intercept & send request body + guardrail->>guardrail:Check against moderation
categories and blocklists + guardrail->>custguardrail: Allow or block request + custguardrail->>proxy: Forward allowed request + proxy->>llm: Process allowed request + llm->>proxy: Return AI response + proxy->>custguardrail: Forward response + custguardrail->>guardrail: Intercept & send response body + guardrail->>guardrail: Check against moderation
categories and blocklists + guardrail->>custguardrail: Allow or block response + custguardrail->>client: Forward allowed response +{% endmermaid %} + +{% endcomment %} + +## Configuration + +To configure the AI Custom Guardrail Policy to work with your guardrail service, you must define your guardrail vendor API's required parameters under [`config.params`](./reference/#schema--config-params). The key is the parameter name, and the value can be a string or a Lua expression. + +Additionally, the following built-in variables are available in Lua expressions. They can be used as arguments in functions, but not in the function body: +* `$(source)`: The current phase on which the Policy is running. The value is `INPUT` if the Policy is currently inspecting the request, and `OUTPUT` if it's inspecting the response. +* `$(conf)`: A Lua table that corresponds to the Policy's config field, meaning it has the same values as the Policy's configuration, which allows to access sub-fields under `config`. +* `$(content)`: The text content being inspected, extracted from the request body in the `INPUT` phase and the response body in the `OUTPUT` phase. +* `$(resp)`: The response from the guardrail service. + + {:.warning} + > This variable is a Lua table corresponding to the request body if the Policy is inspecting the request, but it's a string when inspecting the response. Make sure to configure your functions accordingly. + +### Request + +The [`config.request`](./reference/#schema--config-request) field is used to configure the request that will be sent to your guardrail service. You can set the URL, request body, headers, query parameters, and authentication. You can use the parameters defined under [`config.params`](./reference/#schema--config-params) using the following syntax: `$(conf.params.)`. + +### Response + +The [`config.response`](./reference/#schema--config-response) field is used to define how to parse the response received by the guardrail service. You must define: +* [`config.response.block`](./reference/#schema--config-response-block) +* [`config.response.block_message`](./reference/#schema--config-response-block-message) + +These fields can be defined using functions defined in [`config.functions`](./reference/#schema--config-functions), Lua expressions, or strings. For example, to use the value of a field named `action` in the guardrail service's response body, you can set `config.response.block` to `$(resp.action)`. + +### Metrics + +The [`config.metrics`](./reference/#schema--config-metrics) field allows you to define metrics to be logged by {{site.base_gateway}}. The following standard metrics are available: +* `block_reason`: The reason why the request or response was blocked. +* `block_details`: Additional details about the blocked request or response. +* `masked`: Whether content was masked in the request or response. + +The values can be set to Lua expressions. You can also use the [`config.custom_metrics`](./reference/#schema--config-custom-metrics) field to define additional metrics. + +## Forward proxy support + +{% include md/ai-gateway/v2/forward-proxy.md %} diff --git a/app/_ai_gateway_policies/ai-gcp-model-armor/index.md b/app/_ai_gateway_policies/ai-gcp-model-armor/index.md new file mode 100644 index 00000000000..310ccd8ab70 --- /dev/null +++ b/app/_ai_gateway_policies/ai-gcp-model-armor/index.md @@ -0,0 +1,136 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin + +faqs: + - q: What do I do if I see the error `Blocked by Model Armor Floor Setting`? + a: | + If you see the following error: + + ```json + { + "reason": "MODEL_ARMOR", + "message": "Blocked by Model Armor Floor Setting: The prompt violated X, Y, and Z filters.", + "error": true + } + ``` + This means the AI GCP Model Armor Policy is conflicting with settings configured in GCP Vertex. + We recommend disabling the GCP Model Armor Floor in GCP, as this setting fails in some modes (for example, streaming response mode), and blocks all analytics. +--- + +The GCP Model Armor Policy integrates {{site.ai_gateway}} with [{{ site.google_cloud }}’s Model Armor](https://cloud.google.com/security-command-center/docs/model-armor-overview) service to enforce content safety guardrails on AI requests and responses. +It leverages GCP SaaS APIs to inspect prompts and model outputs, preventing unsafe content from being processed or returned to users. + +## Features + +The AI GCP Model Armor Policy provides the following content safety capabilities: + + +{% table %} +columns: + - title: Feature + key: feature + - title: Description + key: description +rows: + - feature: Request and response guardrails + description: Checks chat requests and chat responses to prevent unsafe content. Controlled by `guarding_mode` (`INPUT`, `OUTPUT`, or `BOTH`). + - feature: Single template enforcement + description: Applies one GCP Model Armor template for all inspections, ensuring consistent filtering. Set with `template_id`. + - feature: Reveal blocked categories + description: Optionally show the categories that triggered blocking (for example, `"hate speech"`). Controlled by `reveal_failure_categories`. + - feature: Streaming response inspection + description: Buffers streaming responses and terminates if unsafe content is detected. Configurable via `response_buffer_size`. + - feature: Custom failure messages + description: Configure user-facing messages with `request_failure_message` and `response_failure_message` when content is blocked. +{% endtable %} + + +## How it works + +The AI GCP Model Armor Policy inspects requests and responses using GCP Model Armor: + +* **Request inspection**: Chat prompts are intercepted, and the relevant content (by default, the last chat message) is sent to the [sanitizeUserPrompt](https://cloud.google.com/security-command-center/docs/sanitize-prompts-responses#text-prompts) API. +* **Response inspection:** Chat responses are buffered (supporting gzip and streaming) and sent to the [sanitizeModelResponse](https://cloud.google.com/security-command-center/docs/sanitize-prompts-responses#sanitize-model) API. SSE streaming is supported with chunk buffering. + +### Request guarding flow + +1. An incoming request to an LLM (for example, a chat completion) is intercepted by the AI GCP Model Armor Policy. +2. The AI GCP Model Armor Policy extracts the relevant content, usually the last user message in the conversation. +3. The content is submitted to GCP Model Armor’s `sanitizeUserPrompt` endpoint for analysis. + +### Response guarding flow + +1. The AI GCP Model Armor Policy buffers the upstream response body (including gzipped responses). +2. It extracts the model’s response content. +3. The content is sent to GCP Model Armor’s `sanitizeModelResponse` endpoint for validation. + +### Sanitization and action + +1. GCP Model Armor evaluates the provided content against the configured `template_id`. +2. The AI GCP Model Armor Policy interprets the `sanitizationResult` from GCP. +3. If a violation is detected (for example, hatred, sexually explicit content, harassment, or jailbreak attempts), the request or response is blocked. +4. Blocked traffic results in a `400 Bad Request` response with the configured `request_failure_message` or `response_failure_message`. +5. If `reveal_failure_categories` is enabled, the response also lists the categories that triggered blocking. + +{:.info} +> When configuring `template_id` in the AI GCP Model Armor Policy, ensure that it aligns with the content safety policies and categories defined in your GCP Model Armor service. +> +> Review whether your organization requires custom categories or additional policy definitions, and integrate them into the selected template to match compliance and safety requirements. + +## Best practices + +The following configuration guidance helps ensure effective content safety enforcement: + +{% table %} +columns: + - title: Setting + key: field + - title: Description + key: description +rows: + - field: | + `guarding_mode` + description: Set to `INPUT` for request-only inspection, `OUTPUT` for response-only, or `BOTH` to guard both directions. + - field: | + `request_failure_message` / `response_failure_message` + description: Provide user-friendly error messages when prompts or responses are blocked. + - field: | + `reveal_failure_categories` + description: Enable to return details on why content was blocked. + - field: | + `response_buffer_size` + description: Tune how much of the upstream response is buffered before inspection; smaller values reduce latency. + - field: Default last message inspection with `text_source` + description: Keep the default behavior of checking only the last user prompt message for highest accuracy. +{% endtable %} + +{:.warning} +> **Caution**: Do **not** set the Model Armor Floor Setting directly in GCP, as it will cause conflicts with the AI GCP Model Armor Policy. +See the [FAQ entry for this error](#what-do-i-do-if-i-see-the-error-blocked-by-model-armor-floor-setting) for more information. + +## Unrecognized filters + +The AI GCP Model Armor Policy now blocks requests when GCP Model Armor returns a filter result with an unrecognized or new filter type. Previously, unrecognized filter types were silently ignored. To avoid blocked requests, review your Model Armor template and ensure it only includes filter types the AI Policy supports. + +## Logging + +The AI GCP Model Armor Policy emits structured log data for every inspected request and response. For the full list of log fields, see the [{{site.ai_gateway}} audit log reference](/ai-gateway/ai-audit-log-reference/#ai-gcp-model-armor-logs). + +To log the raw content of blocked requests and responses, enable [`config.log_blocked_content`](/ai-gateway/policies/ai-gcp-model-armor/reference/#schema--config-log-blocked-content). When enabled, the blocked prompt or response body appears under `ai.proxy.gcp-model-armor.input_faulty_prompt` and `ai.proxy.gcp-model-armor.output_faulty_response` in the log entry. + +## Forward proxy support + +{% include md/ai-gateway/v2/forward-proxy.md %} + +## Limitations + +* Only chat prompts and chat responses are inspected; embeddings and other modalities are not checked. +* Inspects one chat message or one response body at a time. Combining multiple messages reduces accuracy. +* For SSE streaming, unsafe content may appear briefly before termination with `"stop_reason: blocked by content safety"`. +* Only one `template_id` can be configured per AI Policy. \ No newline at end of file diff --git a/app/_ai_gateway_policies/ai-lakera-guard/index.md b/app/_ai_gateway_policies/ai-lakera-guard/index.md new file mode 100644 index 00000000000..024fdae9f53 --- /dev/null +++ b/app/_ai_gateway_policies/ai-lakera-guard/index.md @@ -0,0 +1,126 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- + +The AI Lakera Guard Policy evaluates requests and responses that pass through {{site.ai_gateway}} to Large Language Models (LLMs). It uses the [Lakera Guard SaaS service](https://www.lakera.ai/) to detect safety policy violations and block unsafe content before it reaches upstream LLMs or returns to clients. The AI Lakera Guard Policy supports multiple inspection modes and guards both inbound prompts and outbound model outputs. + +## How it works + +The AI Lakera Guard Policy inspects model traffic at three points in the LLM request lifecycle. Each phase pages data into memory, extracts content that Lakera Guard can evaluate, and sends that content to Lakera for inspection. + +* **Request phase**: Inspection occurs **before** any data leaves the gateway toward the target LLM. The AI Lakera Guard Policy buffers the full request body in memory, extracts the fields that the AI Lakera Guard Policy can evaluate, and sends them for inspection. +* **Response phase (buffered)**: Inspection occurs **before** any byte is transmitted back toward the client. The AI Lakera Guard Policy buffers the full upstream response in memory, extracts the response fields that Lakera Guard can evaluate, and inspects them. This occurs before {{site.ai_gateway}} sends any part of the response back to the client. +* **Response phase (per-frame)**: The AI Lakera Guard Policy runs during streaming responses like Server-Sent Events. {{site.ai_gateway}} processes the response in chunks, buffering each frame in memory as it arrives. When enough data is available to extract an evaluable segment, the AI Lakera Guard Policy inspects that segment with Lakera Guard before forwarding the frame to the client. + +The AI Lakera Guard Policy inspects request and response bodies for routes that use supported model interaction formats. It skips inspection on non-text response types, which Lakera Guard does not currently support. + +## Inspected content + +{% table %} +columns: + - title: Inspection Type + key: type + - title: Input (request) + key: input + - title: Output (response) + key: output + - title: Content type + key: content + - title: Limitations + key: limitation +rows: + - type: "/chat/completions" + input: true + output: true + content: "Array of string content." + limitation: "If multi-modal, inspects text segments only." + - type: "/responses" + input: true + output: true + content: "Input string, array of input strings, or array of chat messages." + limitation: "If multi-modal, inspects text segments only." + - type: "/images/generations" + input: true + output: false + content: "Prompt string, input string, or array of input strings." + limitation: "Image outputs cannot be inspected." + - type: "/embeddings" + input: true + output: false + content: "Input string or array of input strings." + limitation: "Embedding outputs cannot be inspected." +{% endtable %} + +## Logging + +You can use the [logging capabilities](/ai-gateway/ai-audit-log-reference/) of the AI Lakera Guard Policy to monitor the inspection process and understand the detected violations. For the full list of log fields, see the [{{site.ai_gateway}} audit log reference](/ai-gateway/ai-audit-log-reference/#ai-lakera-guard-logs). + +The AI Lakera Guard Policy provides detailed logging and controls over how violations are reported: +* **SaaS platform logging**: All inspected requests, responses, and chats are made available on the Lakera SaaS platform. +* **{{site.ai_gateway}} logging**: {{site.ai_gateway}} logs all request and response **Lakera request UUIDs** to the standard logging subsystem. +* **Unsupported logging outputs**: [Prometheus](/ai-gateway/policies/prometheus/), or [OpenTelemetry](/ai-gateway/policies/opentelemetry/). +* **Logging outputs**: [HTTP Log](/ai-gateway/policies/http-log/), [File Log](/ai-gateway/policies/file-log/), and [TCP Log](/ai-gateway/policies/tcp-log/). + +By default, the AI Lakera Guard Policy doesn't tell clients why their request was blocked. However, this information is always logged to {{site.ai_gateway}} logs for administrators. + +To change this behavior, use `reveal_failure_categories: true`. If activated, the client receives a JSON response including a breakdown array that details the specific `detector_type` that caused the failure. + +To log the raw content of blocked requests and responses, enable [`config.log_blocked_content`](/ai-gateway/policies/ai-lakera-guard/reference/#schema--config-log-blocked-content). When enabled, the blocked prompt or response body appears under `ai.proxy.lakera-guard.input_faulty_prompt` and `ai.proxy.lakera-guard.output_faulty_response` in the log entry. + +### Standard logging subsystem example + +When a request passes all guardrails, the log includes processing latency and the request UUID: + +```json +"ai": { + "proxy": { + "lakera-guard": { + "input_processing_latency": 72, + "lakera_service_url": "https://api.lakera.ai/v2/guard", + "input_request_uuid": "a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d", + "lakera_project_id": "project-1234567890" + } + } +} +``` + +### Violations log example + +When the guardrails block a request, the log captures the violation reason, detector details, and the blocking AI Policy name, AI Consumer ID, and a running trigger counter: + +```json +"ai": { + "proxy": { + "lakera-guard": { + "input_processing_latency": 78, + "lakera_service_url": "https://api.lakera.ai/v2/guard", + "input_block_detail": [ + { + "policy_id": "policy-4f8a9b2c-1d3e-4a5b-8c9d-0e1f2a3b4c5d", + "detector_id": "detector-lakera-moderation-1-input", + "project_id": "project-1234567890", + "message_id": 3, + "detected": true, + "detector_type": "moderated_content/hate" + } + ], + "input_request_uuid": "a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d", + "input_block_reason": "moderated_content/hate", + "input_block_source": "ai-lakera-guard", + "input_block_consumer_id": "consumer-uuid-1234", + "guards_triggered_count": 1, + "lakera_project_id": "project-1234567890" + } + } +} +``` + +## Forward proxy support + +{% include md/ai-gateway/v2/forward-proxy.md %} diff --git a/app/_ai_gateway_policies/ai-llm-as-judge/index.md b/app/_ai_gateway_policies/ai-llm-as-judge/index.md new file mode 100644 index 00000000000..5dfecfc9416 --- /dev/null +++ b/app/_ai_gateway_policies/ai-llm-as-judge/index.md @@ -0,0 +1,93 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- + +The AI LLM as Judge Policy enables automated evaluation of prompt-response pairs using a dedicated LLM. The Policy assigns a numerical score to LLM responses from 1 to 100, where: + +* `1`: Completely incorrect or irrelevant response +* `100`: Perfect or ideal response + +## Features + +The AI LLM as Judge Policy offers several configurable features that control how the LLM evaluates prompts and responses: + +{% table %} +columns: + - title: Feature + key: feature + - title: Description + key: description +rows: + - feature: "Configurable system prompt" + description: "Instructs the LLM to act as a strict evaluator." + - feature: "Numerical scoring" + description: "Assigns a score from 1–100 to assess response quality." + - feature: "History depth" + description: "Includes previous chat messages for context when scoring." + - feature: "Ignore prompts" + description: "Options to ignore system, assistant, or tool prompts." + - feature: "Sampling rate" + description: "Controls probabilistic request volume for judging." + - feature: "Native LLM schema" + description: "Leverages the LLM schema for seamless integration." +{% endtable %} + +## How it works + +1. {{site.ai_gateway}} sends the user prompt and response to the configured LLM as a judge. +2. The LLM evaluates the response and returns a numeric score between `1` (ideal) and `100` (wrong or irrelevant). +3. This score can be used in downstream workflows, such as automated grading, feedback systems, or learning pipelines. + +The following sequence diagram illustrates this simplified flow: + + +{% mermaid %} +sequenceDiagram + actor Client + participant AIGW as {{site.ai_gateway}} + participant LLM as LLM Model (A or B) + participant Judge as AI LLM as Judge + participant JudgeLLM as Judge LLM + + Client->>AIGW: Send prompt + AIGW->>LLM: Forward prompt (balancer selects model) + LLM-->>AIGW: Response + AIGW ->>Judge: Prompt + response + Judge->>JudgeLLM: Evaluate response + JudgeLLM-->>Judge: Score (1–100) + Judge-->>AIGW: Evaluation result + AIGW-->>Client: Response +{% endmermaid %} + +## Recommended LLM settings + +To ensure concise, consistent scoring, configure the LLM that acts as the judge with these values: + +{% table %} +columns: + - title: Setting + key: setting + - title: Recommended value + key: value + - title: Description + key: description +rows: + - setting: "[`temperature`](/ai-gateway/policies/ai-llm-as-judge/reference/#schema--config-llm-model-options-temperature)" + value: "`2`" + description: "Controls randomness. A lower value leads to a more deterministic output." + - setting: "[`max_tokens`](/ai-gateway/policies/ai-llm-as-judge/reference/#schema--config-llm-model-options-max-tokens)" + value: "`5`" + description: "Maximum tokens for the LLM response." + - setting: "[`top_p`](/ai-gateway/policies/ai-llm-as-judge/reference/#schema--config-llm-model-options-top-p)" + value: "`1`" + description: "Nucleus sampling probability; limits token selection." +{% endtable %} + +{:.info} +> These settings produce short, precise numeric scores without extra text or verbosity. \ No newline at end of file diff --git a/app/_ai_gateway_policies/ai-mcp-oauth2/index.md b/app/_ai_gateway_policies/ai-mcp-oauth2/index.md new file mode 100644 index 00000000000..ab3da0e54bc --- /dev/null +++ b/app/_ai_gateway_policies/ai-mcp-oauth2/index.md @@ -0,0 +1,298 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +tech_preview: true +related_resources: + - text: OAuth 2.0 specification for MCP + url: https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization + - text: AI MCP Server + url: /ai-gateway/entities/ai-mcp-server/ + - text: AI Policy + url: /ai-gateway/entities/ai-policy/ +--- + +The AI MCP OAuth2 Policy secures Model Context Protocol (MCP) traffic on {{site.ai_gateway}} using the [OAuth 2.0 specification for MCP servers](https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization). It ensures only authorized MCP clients can access protected MCP servers proxied via an [AI MCP Server](/ai-gateway/entities/ai-mcp-server/) entity, and acts as a crucial security layer for MCP traffic. + +## Purpose and core functionality + +The AI MCP OAuth2 Policy provides OAuth 2.0 authentication for MCP traffic, allowing MCP clients to safely request access. It validates that access tokens are issued specifically for the target MCP server, ensuring only authorized requests are accepted. To reduce the risk of token theft or confused deputy attacks, the Policy does not pass access tokens to upstream services. + +The Policy performs three core functions: + +* Validates incoming MCP requests by verifying access tokens from an external authorization server. +* Extracts claims from validated tokens and forwards them to upstream MCP services via headers. +* Ensures compliance with MCP authorization requirements based on OAuth 2.1. + +## Authorization flow + +The AI MCP OAuth2 Policy follows the following authorization flow: + +* {{site.ai_gateway}} acts as the **Resource Server**, enforcing access control. +* The MCP clients send requests with a valid `Authorization: Bearer ` header. +* The Policy validates tokens, checks the intended audience, and blocks invalid or expired tokens with a `401 Unauthorized`. +* Access tokens are **not forwarded to upstream services** by default, protecting against token theft or confused deputy attacks. + + +{% mermaid %} +sequenceDiagram + participant C as MCP client + participant K as AI MCP OAuth2
(resource server) + participant AS as Authorization server + participant U as Upstream MCP server + + C->>K: Discover protected resource metadata + activate K + K-->>C: Protected resource metadata (includes auth server address) + deactivate K + + C->>AS: Request access token + activate AS + AS-->>C: Access token + deactivate AS + + C->>K: MCP auth request + activate K + K->>AS: Introspect token + activate AS + AS-->>K: Valid / invalid + deactivate AS + + alt If token valid + K->>U: Forward request with claims as headers + activate U + U-->>K: MCP server response + deactivate U + K-->>C: MCP response + else If token invalid + K-->>C: 401 Unauthorized + end + deactivate K + +{% endmermaid %} + + +## Policy execution + +The AI MCP OAuth2 Policy is designed to secure MCP traffic as early as possible in the request lifecycle to prevent unauthorized access before any AI-specific processing occurs. + +{:.warning} +> **Note:** The AI MCP OAuth2 Policy is not invoked as part of an LLM request flow. +> +> Instead, it targets API traffic (MCP traffic specifically), allowing it to capture MCP requests independently of LLM request flow. +> LLM-specific policies will not be applied to MCP traffic. Use this Policy with API-traffic policies like [AI Rate Limiting Advanced](/ai-gateway/policies/ai-rate-limiting-advanced/reference/), and other API-level policies as needed. + +## Token validation methods + +The Policy supports two token validation methods. When introspection is configured, it is always used. JWKS is only used when no introspection endpoint is configured. + +* **Introspection**: Set [`config.introspection_endpoint`](./reference/#schema--config-introspection-endpoint) to have the Policy call the authorization server to validate opaque tokens. Requires `config.client_id` when `config.client_auth` is `client_secret_basic` or `client_secret_post`. +* **JWKS**: Set [`config.jwks_endpoint`](./reference/#schema--config-jwks-endpoint) to validate signed JWTs locally using the authorization server's public keys. If this isn't set, the Policy attempts to discover the JWKS URI from the authorization server metadata. + +## Claim forwarding + +The Policy can extract claims from a validated token and forward them to the upstream MCP server as HTTP headers. Two approaches are available, and they are mutually exclusive. + +### Top-level claims + +Use [`config.claim_to_header`](./reference/#schema--config-claim-to-header) to map top-level token claims to upstream headers. Each entry requires a `claim` name and a `header` name: + +{% entity_example %} +type: policy +data: + name: oauth2-map-user-claims + display_name: OAuth2 Map User Claims + type: ai-mcp-oauth2 + config: + resource: https://api.example.com/mcp + authorization_servers: + - https://auth.example.com + claim_to_header: + - claim: sub + header: X-User-Id + - claim: email + header: X-User-Email +formats: + - konnect-api +{% endentity_example %} + +### Nested claims + +Use [`config.upstream_headers`](./reference/#schema--config-upstream-headers) to map claims at any depth in the token payload using a path array. This field is mutually exclusive with `claim_to_header`: + +{% entity_example %} +type: policy +data: + name: oauth2-map-nested-claims + display_name: OAuth2 Map Nested Claims + type: ai-mcp-oauth2 + config: + resource: https://api.example.com/mcp + authorization_servers: + - https://auth.example.com + upstream_headers: + - header: X-Org-Id + path: + - org + - id + - header: X-User-Role + path: + - realm_access + - roles +formats: + - konnect-api +{% endentity_example %} + +## AI Consumer and AI Consumer Group mapping + +The Policy can map token claims to [AI Consumers](/ai-gateway/entities/ai-consumer/) and [AI Consumer Groups](/ai-gateway/entities/ai-consumer-group/), enabling consumer-based rate limiting, ACL, and other consumer-aware policies to function with MCP traffic. + +### AI Consumer + +You can map individual users from your authorization server to AI Consumers for per-user rate limiting, usage tracking, and access control. For example, map the user's unique identifier (like `sub` or email) from the token to an AI Consumer, then apply policies such as rate limiting to individual users. + +Configure AI Consumer lookup: + +* Set [`config.consumer_claim`](./reference/#schema--config-consumer-claim) to the path of the claim identifying the AI Consumer. For example, `["sub"]` for top-level claims or `["realm_access", "user_id"]` for nested claims. +* Use [`config.consumer_by`](./reference/#schema--config-consumer-by) to specify which AI Consumer fields to check. Accepted values: `id`, `username`, `custom_id`. Defaults to `["username", "custom_id"]`. +* Set [`config.consumer_optional`](./reference/#schema--config-consumer-optional) to `true` to allow requests to proceed if no matching AI Consumer is found. + +{% entity_example %} +type: policy +data: + name: oauth2-with-consumer + display_name: OAuth2 with Consumer Mapping + type: ai-mcp-oauth2 + config: + resource: https://api.example.com/mcp + authorization_servers: + - https://auth.example.com + consumer_claim: + - sub + consumer_by: + - username + - custom_id + consumer_optional: false +formats: + - konnect-api +{% endentity_example %} + +### AI Consumer Groups + +You can also map token claims to AI Consumer Groups to enforce team or organization-level rate limiting and access policies. For example, map users from your authorization server's `teams` or `organizations` claims to AI Consumer Groups, then apply policies at the group level across multiple MCP clients. + +Configure AI Consumer Group lookup: + +* Set [`config.consumer_groups_claim`](./reference/#schema--config-consumer-groups-claim) to the path of the claim containing the AI Consumer Group names. Supports nested paths with multiple strings. +* Set [`config.consumer_groups_optional`](./reference/#schema--config-consumer-groups-optional) to `true` to allow requests to proceed if no matching AI Consumer Group is found. + +{% entity_example %} +type: policy +data: + name: oauth2-with-teams + display_name: OAuth2 with Team Mapping + type: ai-mcp-oauth2 + config: + resource: https://api.example.com/mcp + authorization_servers: + - https://auth.example.com + consumer_groups_claim: + - groups + consumer_groups_optional: true +formats: + - konnect-api +{% endentity_example %} + +### Virtual credentials + +When AI Consumer mapping isn't used, set [`config.credential_claim`](./reference/#schema--config-credential-claim) to derive a virtual credential from the token. This credential is used by other policies to track usage. Defaults to `["sub"]`. + +## Token exchange + +Token exchange lets the Policy swap the client's access token for a different token before forwarding the request to the upstream MCP server. This is useful when the upstream MCP server requires a token from a different authorization server or with different scopes. + +{:.info} +> Token exchange requires [`config.passthrough_credentials`](./reference/#schema--config-passthrough-credentials) to be set to `true`. + +When [`config.token_exchange.enabled`](./reference/#schema--config-token-exchange) is `true`, the Policy performs the following after validating the incoming token: + + +{% mermaid %} +sequenceDiagram + participant C as MCP client + participant K as AI MCP OAuth2
(resource server) + participant AS as Authorization server + participant TE as Token exchange endpoint + participant U as Upstream MCP server + + C->>K: MCP request with Bearer token + activate K + K->>AS: Validate token (introspect / JWKS) + activate AS + AS-->>K: Token valid + deactivate AS + K->>TE: Token exchange request (subject_token = original token) + activate TE + TE-->>K: Exchanged access token + deactivate TE + K->>U: Forward request with exchanged token + activate U + U-->>K: MCP server response + deactivate U + K-->>C: MCP response + deactivate K +{% endmermaid %} + + +Configure token exchange: + +* Set [`config.token_exchange.enabled`](./reference/#schema--config-token-exchange) to `true` and set [`config.token_exchange.token_endpoint`](./reference/#schema--config-token-exchange-token-endpoint) to the token exchange endpoint URL. +* Set [`config.token_exchange.client_auth`](./reference/#schema--config-token-exchange-client-auth) to control authentication with the token exchange endpoint. Accepted values: `client_secret_basic`, `client_secret_post`, `none`, `inherit`. Use `inherit` to reuse credentials from the introspection endpoint. +* Set [`config.token_exchange.request.actor_token_source`](./reference/#schema--config-token-exchange-request) to `header` (also set `config.token_exchange.request.actor_token_header`) or `config` (also set `config.token_exchange.request.actor_token`). +* Exchanged tokens are cached by default. Set [`config.token_exchange.cache.enabled`](./reference/#schema--config-token-exchange-cache) to `false` to disable caching. TTL defaults to `3600` seconds. + +The following example creates an AI MCP OAuth2 Policy that validates client tokens with one authorization server and exchanges them for tokens from a token exchange endpoint before forwarding to the upstream MCP server: + +{% entity_example %} +type: policy +data: + name: mcp-oauth2-token-exchange + display_name: MCP OAuth2 Token Exchange + type: ai-mcp-oauth2 + config: + resource: https://your-resource-server.example.com + passthrough_credentials: true + authorization_servers: + - https://auth.example.com + consumer_claim: + - sub + consumer_by: + - username + - custom_id + consumer_optional: false + token_exchange: + enabled: true + token_endpoint: https://auth.example.com/oauth/token + client_auth: client_secret_basic + client_id: your-client-id + client_secret: your-client-secret + request: + actor_token_source: header + actor_token_header: X-Actor-Token + cache: + enabled: true + ttl: 3600 +formats: + - konnect-api +{% endentity_example %} + +## Token passthrough + +By default, the Policy strips the incoming access token before forwarding the request to the upstream MCP server, preventing token theft and confused deputy attacks. Set [`config.passthrough_credentials`](./reference/#schema--config-passthrough-credentials) to `true` to keep the original token in the request. + +{:.warning} +> Only enable token passthrough when the upstream MCP server explicitly requires the original access token, or when token exchange is configured. diff --git a/app/_ai_gateway_policies/ai-prompt-compressor/index.md b/app/_ai_gateway_policies/ai-prompt-compressor/index.md new file mode 100644 index 00000000000..d5019548e48 --- /dev/null +++ b/app/_ai_gateway_policies/ai-prompt-compressor/index.md @@ -0,0 +1,171 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- + +The AI Prompt Compressor Policy compresses retrieved chunks before sending them to a Large Language Model (LLM), reducing text length while preserving meaning. It uses the [LLMLingua 2 library](https://github.com/microsoft/LLMLingua) for fast, high-quality compression. The AI Prompt Compressor Policy supports: + +* **Ratio-based or target token compression**: for example, reduce a message to 80% of the original length or compress to 150 tokens. +* **Configurable compression ranges**: for example, compress prompts under 100 tokens with a 0.8 ratio or compress them to exactly 100 tokens. +* **Selective compression**: use `...` tags to target specific sections of the prompt. These tags work **only in the `inject_template` field of the [AI RAG Injector Policy](/ai-gateway/policies/ai-rag-injector/)** and must be used **in combination with the AI Prompt Compressor Policy**. + +## Why use prompt compression + +Efficient prompt compression helps you manage token limits, cut costs, and speed up LLM requests — all while keeping sensitive data safe and your prompts focused. + +The table below outlines common use cases for the AI Prompt Compressor Policy and the configuration options available to tailor its behavior. + + +{% table %} +columns: + - title: Use case + key: option + - title: Description + key: description +rows: + - option: Token limit management + description: | + Compress verbose inputs like chat history or documents to stay within the LLM's context window. Prevents truncation of important content. + - option: Cost reduction + description: | + Reducing token count in prompts decreases API costs when calling large language models, especially for high-volume use cases. + - option: Latency reduction + description: | + Smaller prompts result in faster request/response cycles, improving performance for real-time applications like voice assistants. + - option: Data privacy + description: | + Compress or abstract sensitive or personally identifiable information to maintain privacy and comply with data protection standards. + - option: Dynamic prompt optimization + description: | + Automatically strip verbose or low-value content before sending to the LLM, keeping the focus on what's most relevant. +{% endtable %} + + +## AI Prompt Compression Service + +Kong provides a Docker image for the AI Prompt Compressor service, which compresses LLM prompts before sending them upstream. It uses [LLMLingua 2](https://github.com/microsoft/LLMLingua) to reduce prompt size, which helps you manage token limits and maintain context fidelity. The service supports both HTTP and JSON-RPC APIs and is designed to work with the AI Prompt Compressor Policy in {{site.ai_gateway}}. + +{% include prereqs/cloudsmith.md %} + +### Image configuration options + +You can configure the Kong AI Prompt Compressor Service using environment variables. These affect model selection, hardware usage, logging, and worker behavior. + + +{% table %} +columns: + - title: Configuration option + key: option + - title: Description + key: description +rows: + - option: LLMLINGUA_MODEL_NAME + description: | + Specifies the LLMLingua 2 model to use for compression. Defaults to `microsoft/llmlingua-2-xlm-roberta-large-meetingbank`. + - option: LLMLINGUA_DEVICE_MAP + description: | + Device on which to run the model. Supported values include `cpu`, `cuda`, `auto`, or `mps`. + - option: LLMLINGUA_LOG_LEVEL + description: | + Log level for the LLMLingua compression logic. Set to `info`, `debug`, or `warning` based on your needs. + - option: GUNICORN_WORKERS + description: | + Number of Gunicorn worker processes (for Docker deployments only). Defaults to `2`. + - option: GUNICORN_LOG_LEVEL + description: | + Log level for Gunicorn server output (for Docker deployments only). Defaults to `info`. +{% endtable %} + + +### Compression endpoints + +The AI Prompt Compressor Service exposes both REST and JSON-RPC endpoints. You can use these interfaces to compress prompts, check the current status, or integrate the service with the AI Prompt Compressor Policy and other upstream services. + +* **POST `/llm/v1/compressPrompt`**: Compresses a prompt using either a compression ratio or a target token count. Supports selective compression via `` tags. + +* **GET `/status`**: Returns information about the currently loaded LLMLingua model and device settings (for example, CPU or GPU). + +* **POST `/`**: JSON-RPC endpoint that supports the `llm.v1.compressPrompt` method. Use this to invoke compression programmatically over JSON-RPC. + +## Prompt compression options + +The AI Prompt Compressor Policy offers flexible compression controls to fit different use cases. You can choose between full-prompt compression, conditional strategies, or selectively compressing only parts of the prompt: + + +{% table %} +columns: + - title: Configuration Option + key: option + - title: Description + key: description +rows: + - option: Compression by ratio + description: | + Compress the prompt to a percentage of its original length (for example, reduce to 80%). This allows for consistent shrinkage regardless of the initial size. + - option: Compression by token count + description: | + Compress the prompt to a specific token target (for example, 150 tokens). Useful when working close to LLM context window limits. + - option: Conditional rules + description: | + Apply different compression strategies based on prompt length. For example, compress prompts under 100 tokens using a 0.8 ratio, and compress longer prompts to a fixed token count. + - option: Selective compression with tags + description: | + Wrap sections of the prompt in `...` to target only specific parts for compression, preserving untagged content as-is. +{% endtable %} + + +## How it works + +1. The user sends the final prompt to the AI Prompt Compressor Policy. +1. The AI Prompt Compressor Policy checks the prompt for ``...`` tags. + - If tags are found, only the tagged sections are sent to LLMLingua 2 for compression. + - If no tags are found, the entire prompt is sent to LLMLingua 2 for compression. +1. Compression is applied based on configured rules—by ratio, target token count, or conditional length-based rules. +1. The compressed prompt is returned to the AI Prompt Compressor Policy. +1. The AI Prompt Compressor Policy sends the compressed prompt to the Large Language Model (LLM). +1. The LLM processes the prompt and returns the response to the user. + +The diagram below illustrates how the AI Prompt Compressor Policy processes and compresses incoming prompts based on tagging and configured rules. + + +{% mermaid %} +sequenceDiagram + actor User + participant KongAICompressor as AI Prompt Compressor Policy + participant LLMLingua2 as LLMLingua 2 Compressor + participant LLM as Large Language Model + + User->>KongAICompressor: Sends final prompt + activate KongAICompressor + KongAICompressor->>KongAICompressor: Check for LLMLINGUA tags + + alt If tagged content found + KongAICompressor->>LLMLingua2: Compress tagged sections + activate LLMLingua2 + LLMLingua2-->>KongAICompressor: Return compressed sections + deactivate LLMLingua2 + else If no LLMlingua tags + KongAICompressor->>LLMLingua2: Compress entire prompt + activate LLMLingua2 + LLMLingua2-->>KongAICompressor: Return compressed prompt + deactivate LLMLingua2 + end + + KongAICompressor->>LLM: Send compressed prompt + deactivate KongAICompressor + activate LLM + LLM-->>User: Return response + deactivate LLM +{% endmermaid %} + + +The AI Prompt Compressor Policy applies structured compression to preserve essential context of prompts sent by users, rather than trimming prompts arbitrarily or risking token overflows. This ensures the LLM receives a well-formed, focused prompt keeping token usage under control. + +## Forward proxy support + +{% include md/ai-gateway/v2/forward-proxy.md %} \ No newline at end of file diff --git a/app/_ai_gateway_policies/ai-prompt-decorator/index.md b/app/_ai_gateway_policies/ai-prompt-decorator/index.md new file mode 100644 index 00000000000..da27fbdd61b --- /dev/null +++ b/app/_ai_gateway_policies/ai-prompt-decorator/index.md @@ -0,0 +1,15 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- + +The AI Prompt Decorator Policy adds an array of `llm/v1/chat` messages to either the start or end of an LLM consumer's chat history. +This allows you to pre-engineer complex prompts, and manipulate prompts so that they aren't visible to users. + +You can use this Policy to pre-set a system prompt, set up specific prompt history, add words and phrases, or otherwise have more +control over how an LLM service is used when called via {{site.ai_gateway}}. \ No newline at end of file diff --git a/app/_ai_gateway_policies/ai-prompt-guard/index.md b/app/_ai_gateway_policies/ai-prompt-guard/index.md new file mode 100644 index 00000000000..d3868e12e8a --- /dev/null +++ b/app/_ai_gateway_policies/ai-prompt-guard/index.md @@ -0,0 +1,61 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +related_resources: + - text: AI Semantic Prompt Guard Policy + url: /ai-gateway/policies/ai-semantic-prompt-guard/ +--- + +The AI Prompt Guard Policy lets you configure a series of [PCRE-compatible](https://www.pcre.org/) regular expressions as allow or deny lists, +to guard against misuse of text completion requests. + +You can use this Policy to allow or block specific prompts, words, phrases, or otherwise have more control over how an LLM service is +used when called via {{site.ai_gateway}}. + +It does this by scanning all chat messages where the role is `user` for the specific expressions set. + +You can use a combination of `allow` and `deny` rules to preserve integrity and compliance when serving an LLM service using {{site.ai_gateway}}. + +* **For `llm/v1/chat` type models**: You can optionally configure the Policy to ignore existing chat history, wherein it will only scan the trailing `user` message. +* **For `llm/v1/completions` type models**: There is only one `prompt` field, thus the whole prompt is scanned on every request. + +## How it works + +This Policy matches lists of regular expressions to requests routed through the {{site.ai_gateway}}. + +The matching behavior is as follows: +* If any `deny` expressions are set, and the request matches any regex pattern in the `deny` list, the caller receives a 400 Bad Request response. +* If any `allow` expressions are set, but the request matches none of the allowed expressions, the caller also receives a 400 Bad Request response. +* If any `allow` expressions are set, and the request matches one of the `allow` expressions, the request passes through to the LLM. +* If there are both `deny` and `allow` expressions set, the `deny` condition takes precedence over `allow`. Any request that matches an entry in the `deny` list will return a 400 response, even if it also matches an expression in the `allow` list. If the request does not match an expression in the `deny` list, then it must match an expression in the `allow` list to be passed through to the LLM. + +## Best practices + +Configure the AI Prompt Guard Policy to detect hidden unicode characters that attackers commonly use to embed malicious instructions in user input: + +{% entity_example %} +type: policy +data: + name: ai-prompt-guard + config: + deny_patterns: + - (\xE2\x80[\x8B-\x8D]|\xEF\xBB\xBF) + - \xE2\x80[\xAA-\xAE] + - \xE2\x81[\xA0-\xAF] + - \xF3\xA0\x80[\xA0-\xBF]|\xF3\xA0\x81[\x80-\xBF] +formats: + - konnect-api +{% endentity_example %} + +In this example: +- `(\xE2\x80[\x8B-\x8D]|\xEF\xBB\xBF)`: Detects zero-width characters (`U+200B`-`U+200D`, `U+FEFF`) +- `\xE2\x80[\xAA-\xAE]`: Detects bidirectional text controls (`U+202A`-`U+202E`) +- `\xE2\x81[\xA0-\xAF]`: Detects format controls (`U+2060`-`U+206F`) +- `\xF3\xA0\x80[\xA0-\xBF]|\xF3\xA0\x81[\x80-\xBF]`: Detects unicode tag characters (`U+E0020`-`U+E007F`) + +These patterns block invisible characters that can hide prompt injection attempts. Zero-width and bidirectional control characters render as blank space in most interfaces but remain visible to the LLM, allowing attackers to insert hidden commands. \ No newline at end of file diff --git a/app/_ai_gateway_policies/ai-prompt-template/index.md b/app/_ai_gateway_policies/ai-prompt-template/index.md new file mode 100644 index 00000000000..b2d1b540377 --- /dev/null +++ b/app/_ai_gateway_policies/ai-prompt-template/index.md @@ -0,0 +1,54 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- + +The AI Prompt Template Policy lets you provide tuned AI prompts to users. +Users only need to fill in the blanks with variable placeholders in the following format: `{% raw %}{{variable}}{% endraw %}`. + +This lets admins set up templates, which can then be used by anyone in the organization. It also allows admins to present an LLM +as an API in its own right - for example, a bot that can provide software class examples and/or suggestions. + +This Policy also sanitizes string inputs to ensure that JSON control characters are escaped, preventing arbitrary prompt injection. + +## How it works + +When activated, the template restricts LLM usage to the predefined templates. They are defined in the following format: + +{% entity_example %} +type: policy +data: + name: ai-prompt-template + config: + templates: + name: sample-template + template: |- + { + "messages": [ + { + "role": "user", + "content": "Explain to me what {% raw %}{{thing}}{% endraw %} is." + } + ] + } +formats: + - konnect-api +{% endentity_example %} + + +When calling a template, replace the content of `messages` (`llm/v1/chat`) or `prompt` (`llm/v1/completions`) with a template reference, using the following format: +```json +{ + "messages": "{template://sample-template}", + "properties": { + "thing": "gravity" + } +} +``` + +By default, requests that don't use a template are still be passed to the LLM. However, this can be configured using the [`config.allow_untemplated_requests`](/ai-gateway/policies/ai-prompt-template/reference/#schema--config-allow-untemplated-requests) parameter. If this parameter is set to `false`, requests that don't use a template will return a `400 Bad Request` response. diff --git a/app/_ai_gateway_policies/ai-rag-injector/index.md b/app/_ai_gateway_policies/ai-rag-injector/index.md new file mode 100644 index 00000000000..2769d857ff7 --- /dev/null +++ b/app/_ai_gateway_policies/ai-rag-injector/index.md @@ -0,0 +1,473 @@ +--- +title: 'AI RAG Injector' +name: 'AI RAG Injector' + +content_type: policy + +publisher: kong-inc +description: 'Create RAG pipelines by automatically injecting content from a vector database' + + +products: + - ai-gateway + +works_on: + - konnect + +min_version: + ai-gateway: '2.0' + + +icon: ai-rag-injector.png + +related_resources: + - text: All {{site.ai_gateway}} AI Policies + url: /ai-gateway/policies/ + - text: About {{site.ai_gateway}} + url: /ai-gateway/ + - text: AI Semantic Cache Policy + url: /ai-gateway/policies/ai-semantic-cache/ + + +faqs: + - q: What embedding dimension should I use in my `vectordb` config? + a: The [embedding dimension](/ai-gateway/policies/ai-rag-injector/reference/#schema--config-vectordb-dimensions) you use depends on your model and use case. More dimensions improve accuracy but increase cost. `1536` is a balanced default if you use the OpenAI `text-embedding-3-large` model. + + - q: Can I reduce embedding dimensions to save resources? + a: Yes. Use PCA, t-SNE, or UMAP to keep key features while lowering memory and latency. + + - q: What chunk size should I use for RAG? + a: Common sizes are 200–1000 tokens. Smaller chunks give precision; larger ones preserve context. + + - q: Should I add chunk overlap? + a: Yes. Overlap helps maintain context between chunks and improves retrieval quality. + + - q: How should I split text into chunks? + a: Use token-, sentence-, or semantic-based chunking based on your data and query type. + + - q: Which distance metric works best with embeddings? + a: Cosine similarity is the best [distance metric](/ai-gateway/policies/ai-rag-injector/reference/#schema--config-vectordb-distance-metric) for text. Use Euclidean only for coordinate-based data. + + - q: Where should I inject RAG context in the prompt? + a: | + It depends on your priorities: + * `system` offers strong guidance, but carries higher prompt injection risk + * `user` is safer for untrusted content + * `assistant` offers moderate influence + You can set this via the [`inject_as_role`](/ai-gateway/policies/ai-rag-injector/reference/#schema--config-inject-as-role) setting. + - q: | + How do I resolve the MemoryDB error `Number of indexes exceeds the limit`? + a: | + If you see the following error in the logs: + + ```sh + failed to create memorydb instance failed to create index: LIMIT Number of indexes (11) exceeds the limit (10) + ``` + + This means that the hardcoded MemoryDB instance limit has been reached. + To resolve this, create more MemoryDB instances to handle multiple {{page.name}} policy instances. + - q: Does the AI RAG Injector Policy work with GCP Memorystore Redis clusters? + a: | + No. GCP Memorystore Redis clusters do not support the AI RAG Injector Policy. The Redis JSON module required for vector operations is not available in GCP's managed Redis service. +--- + +## What is Retrieval Augmented Generation (RAG)? + +Retrieval-Augmented Generation (RAG) is a technique that improves the accuracy and relevance of language model responses by enriching prompts with external data at runtime. Instead of relying solely on what the model was trained on, RAG retrieves contextually relevant information such as documents, support articles, or internal knowledge from connected data sources like vector databases. + +This retrieved context is then automatically injected into the prompt before the model generates a response. RAG is a critical safeguard in specialized or high-stakes applications, where factual accuracy matters. LLMs are prone to hallucinations, plausible-sounding but factually incorrect or fabricated responses. RAG helps mitigate this by grounding the model’s output in real, verifiable data. + +The following table describes the different use cases for RAG based on industry: + + +{% table %} +columns: + - title: Industry + key: industry + - title: Use case + key: use_case +rows: + - industry: Healthcare + use_case: | + RAG can help surface up-to-date clinical guidelines or patient records in a timely manner, critical when treatment decisions depend on the most current information. + - industry: Legal + use_case: | + Lawyers can use RAG-powered assistants to instantly retrieve relevant case law, legal precedents, or compliance documentation during client consultations. + - industry: Finance + use_case: | + In fast-moving markets, RAG enables models to deliver financial insights based on current data, avoiding outdated or misleading responses driven by stale training snapshots. +{% endtable %} + + +## Why use the AI RAG Injector Policy + +The AI RAG Injector Policy automates the retrieval and injection of contextual data for RAG pipelines without doing manual prompt engineering or retrieval logic. Integrated at the gateway level, it handles embedding generation, vector search, and context injection transparently for each request. + +* **Simplifies RAG workflows:** Automatically embeds prompts, queries the vector DB, and injects relevant context without custom retrieval logic. +* **Platform-level control:** Shifts RAG logic from app code to infrastructure, allowing platform teams to enforce global policies, update configurations centrally, and reduce developer overhead. +* **Improved security:** Vector DB access is limited to the {{site.ai_gateway}}, eliminating the need to expose it to individual dev teams or AI agents. +* **Enables RAG in restricted environments:** Supports RAG even where direct access to the vector database is not possible, such as external-facing or isolated services. +* **Developer productivity:** Developers can focus on building AI features without needing to manage embeddings, similarity search, or context handling. +* **Save LLM costs:** When using the AI RAG Injector Policy with the AI Prompt Compressor Policy, you can wrap specific prompt parts in `` tags within your template to target only those sections for compression, preserving the rest of the prompt unchanged. + +## How the AI RAG Injector Policy works + +When a user sends a prompt, the AI RAG Injector Policy queries a configured vector database for relevant context and injects that information into the request before passing it to the language model. + +1. You attach the AI RAG Injector Policy to an [AI Model](/ai-gateway/entities/ai-model/) via the Konnect API, configuring vector database connection and embedding settings. +1. When a request reaches the {{site.ai_gateway}}, the AI Policy generates embeddings for request prompts, then queries the vector database for the top-k most similar embeddings. +1. The AI Policy injects the retrieved content from the vector search result into the request body, and forwards the request to the upstream service. + +The following diagram is a simplified overview of how the AI Policy works. See the [following section](#rag-generation-process) for a more detailed description. + + +{% mermaid %} +sequenceDiagram + participant User + participant AIGateway as AI Gateway (AI RAG Injector Policy) + participant VectorDB as Vector DB (Knowledge store) + participant Upstream as Upstream Service + + User->>AIGateway: Send request with prompt + AIGateway->>VectorDB: Query for similar embeddings + VectorDB-->>AIGateway: Return relevant context + AIGateway->>Upstream: Inject context and forward enriched request + Upstream-->>User: Return response +{% endmermaid %} + + +### RAG Generation process + +The RAG workflow consists of two critical phases: +1. **Data preparation**: Processes and embeds unstructured data into a vector index for efficient semantic search +1. **Retrieval and generation**: The system uses similarity search to dynamically assemble contextual prompts that guide the language model’s output. + + +#### Phase 1: Data Preparation + +This phase sets up the foundation for semantic retrieval by converting raw data into a format that can be indexed and searched efficiently. + +**Step breakdown:** + +1. A document loader pulls content from various sources, such as PDFs, websites, emails, or internal systems. +2. The system breaks the unstructured data into smaller, semantically meaningful chunks to support precise retrieval. +3. Each chunk is transformed into a vector embedding (a numeric representation that captures its semantic content). +4. These embeddings are saved to a vector database, enabling a fast, similarity-based search during query time. + +#### Phase 2: Retrieval and Generation + +This phase runs in real time, taking user input and producing a context-aware response using the indexed data. + +**Step breakdown:** + +1. The user’s query is converted into an embedding using the same model used during data preparation. +1. A semantic similarity search locates the most relevant content chunks in the vector database. +1. The system builds a custom prompt by combining the retrieved chunks with the original query. +1. The LLM generates a contextually accurate response using both the retrieved context and its own internal knowledge. + +The diagram below shows how data flows through both phases of the RAG pipeline, from ingestion and embedding to real-time query handling and response generation: + + +{% mermaid %} +sequenceDiagram + autonumber + actor User + participant RawData as Raw Data + participant EmbeddingModel as Embedding Model + participant VectorDB as Vector Database + participant LLM + + par Data preparation + activate RawData + RawData->>EmbeddingModel: Load and chunk documents, generate embeddings + deactivate RawData + + activate EmbeddingModel + EmbeddingModel->>VectorDB: Store embeddings + deactivate EmbeddingModel + + activate VectorDB + deactivate VectorDB + end + + par Retrieval & generation + activate User + User->>EmbeddingModel: (1) Submit query and generate query embedding + + activate EmbeddingModel + EmbeddingModel->>VectorDB: (2) Search vector DB + deactivate EmbeddingModel + + activate VectorDB + VectorDB-->>EmbeddingModel: Return relevant chunks + deactivate VectorDB + + activate EmbeddingModel + EmbeddingModel->>LLM: (3) Assemble prompt and send + deactivate EmbeddingModel + + activate LLM + LLM-->>User: (4) Generate and return response + deactivate LLM + deactivate User + end +{% endmermaid %} + + + +Rather than guessing from memory, the LLM paired with the RAG pipeline now has the ability to look up the information it needs in real time, which reduces hallucinations and increases the accuracy of the AI output. + +## Vector databases + +{% include_cached md/ai-gateway/v2/ai-vector-db.md name=page.name %} + +### Using cloud authentication with Redis + +{% include_cached md/ai-gateway/v2/redis-cloud-auth.md tier=page.tier %} + +{% include_cached md/ai-gateway/v2/redis-cloud-providers.md name=page.name heading_level=3 %} + +## Access control and metadata filtering + +Once you've configured your vector database and ingested content, you can control which [AI Consumers](/ai-gateway/entities/ai-consumer/) access specific knowledge base articles and refine query results using metadata filters. + +### Collections + +A collection is a logical grouping of knowledge base articles with independent access control rules. When you ingest content via the Konnect API, assign it to a collection using the `collection` field in the metadata. + +Example metadata structure: + +```json +{ + "content": "Quarterly revenue increased 15%...", + "metadata": { + "collection": "finance-reports", + "date": "2023-10-14", + "tags": ["finance", "quarterly"], + "source": "internal" + } +} +``` + +### Configuration + +Two independent mechanisms control which results consumers receive: + +- **ACL filtering**: Server restricts collections based on [AI Consumer Groups](/ai-gateway/entities/ai-consumer-group/) +- **Metadata filtering**: Clients specify criteria (tags, dates, sources) to narrow results within authorized collections + + +{% table %} +columns: + - title: Field + key: field + - title: Description + key: description +rows: + - field: | + [`consumer_identifier`](/ai-gateway/policies/ai-rag-injector/reference/#schema--config-consumer-identifier) + description: | + Determines which AI Consumer attribute is matched against ACL rules. Options: `consumer_group`, `username`, `custom_id`, or `consumer_id` + - field: | + [`global_acl_config.allow[]`](/ai-gateway/policies/ai-rag-injector/reference/#schema--config-global-acl-config-allow) + description: | + Group names with access to all collections (unless overridden) + - field: | + [`global_acl_config.deny[]`](/ai-gateway/policies/ai-rag-injector/reference/#schema--config-global-acl-config-deny) + description: | + Group names explicitly denied access to all collections + - field: | + [`collection_acl_config..allow[]`](/ai-gateway/policies/ai-rag-injector/reference/#schema--config-collection-acl-config) + description: | + Group names with access to this specific collection. Empty list means allow all + - field: | + [`collection_acl_config..deny[]`](/ai-gateway/policies/ai-rag-injector/reference/#schema--config-collection-acl-config) + description: | + Group names explicitly denied access to this specific collection +{% endtable %} + + +This configuration creates the following access rules: +- `finance-reports`: Accessible only to AI Consumers in the `finance` or `admin` groups. Contractors are explicitly denied. +- `public-docs`: Accessible to all AI Consumers (empty allow and deny lists). +- Other collections: No access (empty global ACL means deny by default). + +{:.info} +> This example assumes you have already created [AI Consumer Groups](/ai-gateway/entities/ai-consumer-group/) (`finance`, `admin`, `contractor`) and configured the [Key Authentication Policy](/ai-gateway/policies/key-auth) for your AI Consumers. + +### Environment variables + +Set the following environment variables before deploying: + +* `OPENAI_API_KEY`: Your OpenAI API key +* `DB_PASSWORD`: Your PostgreSQL database password + +{:.warning} +> Never hardcode credentials in your policy configuration. Always use environment variables or secrets. + +{% entity_example %} +type: policy +data: + type: ai-rag-injector + name: finance-db + display_name: Finance DB + config: + consumer_identifier: consumer_group + global_acl_config: + allow: [] + deny: [] + collection_acl_config: + finance-reports: + allow: + - finance + - admin + deny: + - contractor + public-docs: + allow: [] + deny: [] + embeddings: + model: + name: text-embedding-3-small + provider: openai + auth: + header_name: Authorization + header_value: Bearer ${OPENAI_API_KEY} + vectordb: + strategy: pgvector + dimensions: 1536 + distance_metric: cosine + pgvector: + host: localhost + port: 5432 + user: postgres + password: ${DB_PASSWORD} + database: kong-pgvector +formats: + - konnect-api +{% endentity_example %} + +In this configuration, collections with their own ACL in `collection_acl_config` ignore `global_acl_config` entirely. They must explicitly list all allowed subjects. + +### ACL evaluation + +The AI Policy checks access in this order: + +1. **Deny list**: If subject matches, deny access +2. **Allow list**: If list exists and subject doesn't match, deny access +3. **Empty ACL**: If both lists are empty, allow access + +{:.info} +> Collections with their own ACL in `collection_acl_config` ignore `global_acl_config` entirely. They must explicitly list all allowed subjects. + +### Metadata filtering + +LLM clients can refine search results by specifying filter criteria in the query request. Filters apply within the collections. The AI RAG Injector Policy uses a Bedrock-compatible filter grammar with the following operators: + +- `equals`: Exact match +- `greaterThan`: Greater than (>) +- `greaterThanOrEquals`: Greater than or equal to (>=) +- `lessThan`: Less than (<) +- `lessThanOrEquals`: Less than or equal to (<=) +- `in`: Match any value in array +- `andAll`: Combine multiple filter clauses + +You can combine multiple conditions with `andAll`: + + +```json +{ + "andAll": [ + {"equals": {"key": "source", "value": "internal"}}, + {"in": {"key": "tags", "value": ["finance", "quarterly"]}}, + {"greaterThanOrEquals": {"key": "date", "value": "2023-01-01"}} + ] +} +``` + + +Filter parameters: + + +{% table %} +columns: + - title: Parameter + key: parameter + - title: Description + key: description +rows: + - parameter: | + `filters` + description: | + JSON object with filter clauses using the grammar above + - parameter: | + `filter_mode` + description: | + Controls how chunks with no metadata are handled:
+ • `"compatible"`: Includes chunks matching filter OR chunks with no metadata
+ • `"strict"`: Includes only chunks matching filter + - parameter: | + `stop_on_filter_error` + description: | + Fail query on filter parse error (default: `false`) +{% endtable %} + + +You can include filters in the `ai-rag-injector` parameter of your request: + + +```bash +curl "http://localhost:8000/" \ + -H "Content-Type: application/json" \ + --json '{ + "messages": [ + { + "role": "user", + "content": "What were Q4 results?" + } + ], + "ai-rag-injector": { + "filters": { + "andAll": [ + { + "equals": { + "key": "source", + "value": "internal" + } + }, + { + "in": { + "key": "tags", + "value": [ + "q4", + "quarterly" + ] + } + } + ] + }, + "filter_mode": "strict", + "stop_on_filter_error": false + } + }' +``` + + +### Query flow + +The following diagram shows how ACL and metadata filtering work together during query processing: + +{% mermaid %} +flowchart TB + Start([Query Request]) --> Auth[Authenticate AI Consumer] + Auth --> CheckACL{Authorized
Collections?} + CheckACL -->|No| Deny[❌ Access Denied] + CheckACL -->|Yes| HasFilter{Metadata
Filters
Specified?} + HasFilter -->|No| SearchAll[Search all chunks
in authorized collections] + HasFilter -->|Yes| FilterMode{filter_mode
setting?} + FilterMode -->|compatible| SearchCompat[Return chunks matching filter
OR chunks with no metadata] + FilterMode -->|strict| SearchStrict[Return only chunks
matching filter] + SearchAll --> Return[✓ Return Results] + SearchCompat --> Return + SearchStrict --> Return +{% endmermaid %} \ No newline at end of file diff --git a/app/_ai_gateway_policies/ai-rate-limiting-advanced/index.md b/app/_ai_gateway_policies/ai-rate-limiting-advanced/index.md new file mode 100644 index 00000000000..3266f3a4c72 --- /dev/null +++ b/app/_ai_gateway_policies/ai-rate-limiting-advanced/index.md @@ -0,0 +1,234 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- + +The AI Rate Limiting Advanced Policy provides rate limiting for all [AI Policies](/ai-gateway/policies/). The +AI Rate Limiting Advanced Policy extends the +[Rate Limiting Advanced](/ai-gateway/policies/rate-limiting-advanced/) Policy. + +This Policy uses the token data returned by the LLM provider to calculate the costs of queries. +The same HTTP request can vary greatly in cost depending on the calculation of the +LLM providers. + +A common pattern to protect your AI API is to analyze and assign costs to incoming queries, then rate limit the consumer's +cost for a given time window and provider or policy. +You can also create a generic prompt rate limit using the [request prompt provider](#request-prompt-function). + +## Strategies + +{% include md/ai-gateway/v2/rate-limiting-strategies.md name="AI Rate Limiting Advanced" %} + +### Using cloud authentication with Redis + +{% include_cached md/ai-gateway/v2/redis-cloud-auth.md tier=page.tier %} + +{% include_cached md/ai-gateway/v2/redis-cloud-providers.md name=page.name heading_level=3 %} + +### Fallback from Redis + +{% include md/ai-gateway/v2/redis-fallback.md %} + +## Policy-based rate limiting + +The [`config.policies`](./reference/#schema--config-policies) field allows you to define rate limiting at the AI Consumer, AI Consumer Group, IP address, header, path, model, and provider level. The match conditions under [`config.policies.match`](./reference/#schema--config-policies-match) use an `AND` logic, so you can combine these to set up multi-dimensional rate limiting. For example, you can set different rate limiting policies for a specific Consumer and model: + +{% entity_example %} +type: policy +data: + name: ai-rate-limiting-advanced + config: + policies: + - match: + - type: consumer + key: id + values: + - $CONSUMER_ID + - type: model + partition_by: true + values: + - gpt-4o + limits: + - limit: 100 + window_size: 60 + - limit: 1000 + window_size: 3600 +formats: + - kongctl +{% endentity_example %} + +In this example, the limits will apply only to requests made by the specified AI Consumer to the `gpt-4o` model. + +Policies without match conditions act as fallback and match all requests. + +{:.warning} +> When defining rate limits for a specific model, these limits apply to the **requested** model. If a request is redirected to a different model after a failover, the request may succeed even if the final model has reached its limit. + +### Window types + +Each policy sets a [`window_type`](./reference/#schema--config-policies-window-type): `fixed`, `sliding`, or `calendar`. + +* `sliding` (default): Weighs the current window against the previous one to produce a dynamically calculated rate. +* `fixed`: Assigns each request to a single time bucket based on its timestamp. +* `calendar`: Aligns the window to a real calendar boundary in a specific time zone, instead of a rolling `window_size`. + +#### Calendar windows + +Calendar windows budget tokens against the same period your billing already uses, such as a monthly allowance that resets on the 1st, instead of a rolling window that drifts from the billing cycle. + +Set `window_type: calendar` on the policy and [`timezone`](./reference/#schema--config-policies-timezone) to an IANA time zone, such as `America/New_York`. Each limit then sets a [`period`](./reference/#schema--config-policies-limits-period) instead of `window_size`: + + +{% table %} +columns: + - title: Field + key: field + - title: Description + key: description +rows: + - field: "`period`" + description: "`month` or `week`." + - field: "`month_day`" + description: Required when `period` is `month`. This is the day of the month (1-31) that a monthly window starts. + - field: "`week_start_day`" + description: Day of the week a weekly window starts, such as `monday`. Only applies when `period` is `week`. +{% endtable %} + +{% entity_example %} +type: policy +data: + name: ai-rate-limiting-advanced + type: ai-rate-limiting-advanced + config: + policies: + - window_type: calendar + timezone: America/New_York + match: + - type: consumer_group + values: + - premium + limits: + - limit: 2000000 + period: month + month_day: 1 + tokens_count_strategy: total_tokens +formats: + - kongctl +{% endentity_example %} + +This policy grants the `premium` AI Consumer Group 2,000,000 tokens per calendar month, resetting at local midnight on the 1st in `America/New_York`. + +{:.info} +> For calendar windows, `X-AI-RateLimit-Reset` and `X-AI-RateLimit-Retry-After` point to the next calendar boundary (the start of the next week or month in the configured time zone), not `now + window_size`. + +### Known issues + +* When defining a policy matching a model and/or a provider, you must set the [`config.policies.match.partition_by`](./reference/#schema--config-policies-match-partition-by) field to `true`, otherwise the policy is not enforced. + + +## Headers sent to the client + +When the AI Rate Limiting Advanced Policy is enabled, {{site.ai_gateway}} sends some additional headers back to the client, +indicating the allowed limits, how many requests are available, and how long it will take +until the quota is restored. It also sends the limits in the time frame and the number +of remaining minutes for each provider or policy. + +For example: + +```plaintext +X-AI-RateLimit-Reset: 51 +X-AI-RateLimit-Retry-After: 51 +X-AI-RateLimit-Limit-90-policy-1: 20 +X-AI-RateLimit-Remaining-90-policy-1: 0 +``` + +You can optionally hide the limit and remaining headers with the [`config.hide_client_headers`](./reference/#schema--config-hide-client-headers) option. + +If more than one limit is set, the AI Rate Limiting Advanced Policy returns multiple time limit headers. +For example: + +```plaintext +X-AI-RateLimit-Limit-30-azure: 1000 +X-AI-RateLimit-Remaining-30-azure: 950 +X-AI-RateLimit-Limit-40-cohere: 2000 +X-AI-RateLimit-Remaining-40-cohere: 1150 +``` + +If any of the limits are reached, the AI Rate Limiting Advanced Policy returns an `HTTP/1.1 429` status +code to the client with the following JSON body: + +```json +{ "message": "API rate limit exceeded for provider azure, cohere" } +``` + +For each provider or policy, the AI Rate Limiting Advanced Policy also indicates how long it will take until the quota is restored: + +```plaintext +X-AI-RateLimit-Retry-After-30-azure: 1500 +X-AI-RateLimit-Reset-30-azure: 1500 +``` + +If using the request prompt provider, the AI Rate Limiting Advanced Policy will send the query cost: + +```plaintext +X-AI-RateLimit-Query-Cost: 100 +``` + +The `Retry-After` headers will be present on `429` errors to indicate how long the service is +expected to be unavailable to the client. When using `window_type=sliding` and `RateLimit-Reset`, `Retry-After` +may increase due to the rate calculation for the sliding window. + +{:.warning} +> The headers `RateLimit-Limit`, `RateLimit-Remaining`, and `RateLimit-Reset` are based on the Internet-Draft [RateLimit Header Fields for HTTP](https://datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers) and may change in the future to respect specification updates. + +## Token count strategies + +The AI Rate Limiting Advanced Policy supports three strategies to calculate the number of tokens. Configure the strategy with [`tokens_count_strategy`](./reference/#schema--config-tokens-count-strategy). + +{% table %} +columns: + - title: Strategy + key: strategy + - title: Description + key: description +rows: + - strategy: "`total_tokens`" + description: The total number of tokens in the request, including both prompt and completion tokens. + - strategy: "`prompt_tokens`" + description: The tokens provided as input to the LLM. + - strategy: "`completion_tokens`" + description: The tokens generated by the LLM in response to the prompt. + - strategy: "`cost`" + description: | + The financial or computational cost incurred based on token usage. This strategy lets you limit API usage based on actual processing costs rather than raw token counts. +

+ The AI Rate Limiting Advanced Policy calculates cost as the sum of prompt tokens multiplied by input cost and completion tokens multiplied by output cost, divided by 1 million: `cost = (prompt_tokens × input_cost + completion_tokens × output_cost) / 1,000,000`. +

+ You define `input_cost` and `output_cost` per 1 million tokens in whatever unit suits your use case, whether US dollars, cents, or internal billing credits. The rate limit threshold must use the same unit. +

+ + {:.warning} + > This strategy requires `input_cost` and `output_cost` values in the [AI Model](/ai-gateway/entities/ai-model/) target configuration, under `config`. +{% endtable %} + +### Request prompt function + +You can decide to use a custom function to count the tokens for a requests. +To configure it, specify the function in [`config.request_prompt_count_function`](./reference/#schema--config-request-prompt-count-function). + +When using the request prompt provider, it will call the function to get the token count at the request level and implement a limit. + +## Known limitations of AI Rate Limiting Advanced + +The cost is only reflected during the next request. + +For example, if a request is made and returns a token cost of `100` for the `OpenAI` provider: +* The request is made to the OpenAI provider and the response is returned to the user +* If the rate limit is reached, the next request will be blocked + +Additionally, [`config.disable_penalty`](./reference/#schema--config-disable-penalty) only works for the `requestPrompt` function. diff --git a/app/_ai_gateway_policies/ai-request-transformer/index.md b/app/_ai_gateway_policies/ai-request-transformer/index.md new file mode 100644 index 00000000000..f6eb9c913db --- /dev/null +++ b/app/_ai_gateway_policies/ai-request-transformer/index.md @@ -0,0 +1,41 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- + +The AI Request Transformer Policy uses a configured LLM service to transform a client request body before proxying the request upstream. + +This Policy supports the same `llm/v1/chat` requests and providers as the [AI Model entity](/ai-gateway/entities/ai-model/). + +It also uses the same configuration and tuning parameters as the AI Model entity, under the [`config.llm`](/ai-gateway/policies/ai-request-transformer/reference/#schema--config-llm) block. + +The AI Request Transformer Policy runs **before** all of the [AI prompt](/ai-gateway/policies/?terms=ai%2520prompt) Policies, allowing it to also transform requests before sending them to a different LLM. + +{:.warning} +> **Known failure mode: Chaining AI Request Transformer with the {{site.ai_gateway}}** +> +> Chaining AI Request Transformer with the {{site.ai_gateway}} may fail for some providers, even though the same setup works with others. +> +> The reason is that the AI Request Transformer Policy forwards raw model output, and if the model does not produce strict JSON, the proxy chain cannot function correctly. This is not a bug in {{site.ai_gateway}} but a limitation of LLM behavior. + +## How it works + +{% include md/ai-gateway/v2/ai-transformer-diagram.md %} + +1. The {{site.ai_gateway}} admin sets up an [`llm` configuration block](/ai-gateway/policies/ai-request-transformer/reference/#schema--config-llm). +1. The {{site.ai_gateway}} admin sets up a `prompt`. +The prompt becomes the `system` message in the LLM chat request, and prepares the LLM with transformation +instructions for the incoming client request body. +1. The client makes an HTTP(S) call. +1. Before proxying the client's request to the backend, {{site.ai_gateway}} sets the entire request body as the +`user` message in the LLM chat request, and then sends it to the configured LLM service. +1. The LLM service returns a response `assistant` message, which is subsequently set as the upstream request body. +1. The {{site.ai_gateway}} sends the transformed request to the AI LLM service. +1. The AI LLM service returns a response to {{site.ai_gateway}}. +1. The {{site.ai_gateway}} sends the transformed response to the client. + diff --git a/app/_ai_gateway_policies/ai-response-transformer/index.md b/app/_ai_gateway_policies/ai-response-transformer/index.md new file mode 100644 index 00000000000..cb23ffdd279 --- /dev/null +++ b/app/_ai_gateway_policies/ai-response-transformer/index.md @@ -0,0 +1,53 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +related_resources: + - text: AI Request Transformer Policy + url: /ai-gateway/policies/ai-request-transformer/ +--- + +The AI Response Transformer Policy uses a configured LLM service to transform the upstream's HTTP(S) response before returning it to the client. + +It can also terminate or otherwise nullify the response if it fails a compliance or formatting check from the configured LLM service, for example. + +This Policy supports `llm/v1/chat` requests for the same [LLM providers](/ai-gateway/ai-providers/) that {{site.ai_gateway}} supports. + +It also uses the same LLM configuration and tuning parameters as an [AI Model](/ai-gateway/entities/ai-model/), in the [`config.llm`](/ai-gateway/policies/ai-request-transformer/reference/#schema--config-llm) block. + +The AI Response Transformer Policy runs **after** {{site.ai_gateway}} proxies to the upstream LLM service through an [AI Model](/ai-gateway/entities/ai-model/), allowing it to transform responses from any upstream LLM. + +## How it works + +{% include md/ai-gateway/v2/ai-response-transformer-diagram.md %} + +1. The {{site.ai_gateway}} admin sets up an [`llm` configuration block](/ai-gateway/policies/ai-request-transformer/reference/#schema--config-llm). +1. The {{site.ai_gateway}} admin sets up a `prompt`. +The prompt becomes the `system` message in the LLM chat request, and provides transformation +instructions to the LLM for the returning upstream response body. +1. The client makes an HTTP(S) call. +1. After proxying the client's request to the backend, {{site.ai_gateway}} sets the entire response body as the +`user` message in the LLM chat request, then sends it to the configured LLM service. +1. The LLM service returns a response `assistant` message, which is subsequently set as the upstream response body. +1. The Policy returns early (`kong.response.exit`) and can handle gzip or chunked requests, similar to the [Forward Proxy](/ai-gateway/policies/forward-proxy/) policy. + +### Adjusting response headers, status codes, and body + +You can additionally instruct the LLM to respond in the following format, which lets you adjust the response headers, response status code, and response body: + +```json +{ + "headers": + { + "new-header": "new-value" + } +} +``` + +If the `parse_llm_response_json_instructions` parameter is set to `true`, {{site.ai_gateway}} will parse these instructions and set the specified response headers, response status code, and replacement response body. +This lets you change specific headers such as `Content-Type`, or throw errors from the LLM. + diff --git a/app/_ai_gateway_policies/ai-sanitizer/index.md b/app/_ai_gateway_policies/ai-sanitizer/index.md new file mode 100644 index 00000000000..56fdffe5168 --- /dev/null +++ b/app/_ai_gateway_policies/ai-sanitizer/index.md @@ -0,0 +1,198 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- + +The AI PII Sanitizer Policy for {{site.ai_gateway}} helps protect sensitive information in client request bodies before they reach upstream AI providers or tools. + +By integrating with an external PII service, this Policy ensures compliance with data privacy regulations while preserving the usability of request data. + +The AI PII Sanitizer supports multiple sanitization modes, including replacing sensitive information with fixed placeholders or generating synthetic replacements that retain category-specific characteristics. + +Additionally, AI PII Sanitizer offers an optional restoration feature, allowing the original request data to be reinstated in responses when needed. + +The AI PII Sanitizer Policy uses the AI PII Anonymizer Service, which can run in a Docker container, to detect and sanitize sensitive data. + +## How it works + +The AI PII Sanitizer Policy can be applied to: +* Input data (requests) +* Output data (responses) +* Both input and output data + +Here's how it works if you apply it to both requests and responses: + +1. The Policy intercepts the request body and sends it to the external PII service. + - The PII service detects sensitive data and applies the chosen sanitization method (placeholders or synthetic replacements). +1. The sanitized request is forwarded upstream to the selected AI Model. +1. On the way back, the Policy intercepts the response body and sends it to the external PII service. + - The PII service detects sensitive data and applies the chosen sanitization method (placeholders or synthetic replacements). +1. (_Only applies to input data sanitization_) If restoration is enabled, the Policy restores the original request data in responses before returning them to the client. + + +{% mermaid %} +sequenceDiagram + autonumber + participant Client + participant Policy as AI PII Sanitizer + participant PII as PII Service + participant AI as Upstream AI Service + + Client->>Policy: Send request + Policy->>PII: Intercept & send request body + PII->>PII: Detect sensitive data in request + PII->>Policy: Return sanitized request
(placeholders/synthetic data) + Policy->>{{site.ai_gateway}}: Forward sanitized request + {{site.ai_gateway}}->>AI: Process sanitized request + AI->>{{site.ai_gateway}}: Return AI response + {{site.ai_gateway}}->>Policy: Forward response + Policy->>PII: Intercept & send response body + PII->>PII: Detect sensitive data in response + PII->>Policy: Return sanitized response
(placeholders/synthetic data) + Policy->>Client: Return sanitized response +{% endmermaid %} + + +> _Figure 1: Diagram showing the request and response flow with the AI PII Sanitizer Policy._ + +## AI PII Anonymizer service + +Kong provides several AI PII Anonymizer service Docker images in a private repository. Each image includes a built-in NLP model and is tagged using the `version-lang_code` format. For example: + +* `service:v0.1.4-en`: English model, version 0.1.4 +* `service:v0.1.4-it`: Italian model, version 0.1.4 +* `service:v0.1.4-fr`: French model, version 0.1.4 + +{:.info} +> All models are bundled into a single image per version, tagged using the format `v`. For example: `v0.1.4` +> If you need to add or modify models, edit the configuration file at `ai_pii_service/nlp_engine_conf.yml`. + +### Sanitization endpoints + +* `POST /llm/v1/sanitize`: Sanitize specified types of PII information, including credentials, and custom patterns +* `POST /llm/v1/sanitize_credentials`: Only for sanitizing credentials + +See the [AI PII Sanitizer OpenAPI specification](/ai-gateway/policies/ai-sanitizer/api/) for complete details. + +### Available anonymization modes + +You can anonymize data in requests using the following redact modes: + +* `placeholder`: Replaces sensitive data with a fixed placeholder pattern, `PLACEHOLDER{i}`, where `i` is a sequence number. Identical original values receive the same placeholder. + + For example, the location `New York City` might be replaced with `LOCATION`. + +* `synthetic`: Redact the sensitive data with a word in the same type. + + For example, the name `John` might be replaced with `Amir`. + + * Custom patterns are replaced with `CUSTOM{i}`. + * Credentials are replaced with a string of `#` characters matching the original length. + +### Custom patterns + +You can define an array of custom patterns on a per-request basis. +Currently, only regex patterns are supported, and all fields are required: `name`, `regex`, and `score`. + +The `name` must be unique for each pattern. + +### Fields that can be anonymized + +You can use the following fields in the `anonymize` array: + +* `general`: Anonymizes general PII entities such as person names, locations, and organizations. +* `phone`: Anonymizes phone numbers (for example, `mobile`, `landline`). +* `email`: Anonymizes email addresses. +* `creditcard`: Anonymizes credit card numbers. +* `crypto`: Anonymizes cryptocurrency addresses. +* `date`: Anonymizes dates and timestamps. +* `ip`: Anonymizes IP addresses (both IPv4 and IPv6). +* `nrp`: Anonymizes a person’s nationality, religious, or political group. +* `ssn`: Anonymizes Social Security Numbers (SSN) and other related identifiers like ITIN, NIF, ABN, and more. +* `domain`: Anonymizes domain names. It was deprecated, use `url` instead. +* `url`: Anonymizes web URLs. +* `medical`: Anonymizes medical identifiers (for example, medical license numbers, NHS numbers, medicare numbers). +* `driverlicense`: Anonymizes driver's license numbers. +* `passport`: Anonymizes passport numbers. +* `bank`: Anonymizes bank account numbers and related banking identifiers (for example, VAT codes, IBAN). +* `nationalid`: Anonymizes various national identification numbers (for example, Aadhaar, PESEL, NRIC, social security, or voter IDs). +* `custom`: Anonymizes user-defined custom PII patterns using regular expressions only when custom patterns are provided. +* `credentials`: Anonymizes the credentials, similar to `/sanitize_credentials`. +* `all`: Includes all the fields above, including custom ones. + +### Access the Docker images + +Kong distributes these images via a private Cloudsmith registry. Contact [Kong Support](https://support.konghq.com/support/s/) to request access. + +#### Authenticate with the private Cloudsmith registry + +To pull images, you must authenticate first with the token provided by the Support: + +```bash +docker login docker.cloudsmith.io +``` + +Docker will then prompt you to enter username and password: + +```bash +Username: kong/ai-pii +Password: YOUR-TOKEN +``` + +{:.info} +> This is a token-based login with read-only access. You can pull images but not push them. + +#### Pull the AI PII service image + +To pull an image: + +```bash +docker pull docker.cloudsmith.io/kong/ai-pii/IMAGE-NAME:TAG +``` + +Replace `IMAGE-NAME` and `TAG` with the appropriate image and version, such as: + +```bash +docker pull docker.cloudsmith.io/kong/ai-pii/service:v0.1.4-en +``` + +#### AI PII service Dockerfile usage + +To use an image in a `Dockerfile`, reference it as follows: + +```dockerfile +FROM docker.cloudsmith.io/kong/ai-pii/ai-pii-service:v0.1.4-en +``` + +### Available language tags + +The following language-specific images are currently available: + +* `-en` (English) +* `-es` (Spanish) +* `-fr` (French) +* `-de` (German) +* `-it` (Italian) +* `-ja` (Japanese) +* `-ko` (Korean) +* `-pt` (Portuguese) +* `-tr` (Turkish) + +{:.info} +> The PII Anonymizer service loads one NLP model by default. Ensure at least **600MB of free memory** is available when running the container. + +### Image configuration options + +This service takes the following optional environment variables at startup: +* `GUNICORN_WORKERS`: Specifies the number of Gunicorn processes to run +* `PII_SERVICE_ENGINE_CONF`: Specifies the natural language processing (NLP) engine configuration file +* `GUNICORN_LOG_LEVEL`: Specifies log level + +## Forward proxy support + +{% include md/ai-gateway/v2/forward-proxy.md %} \ No newline at end of file diff --git a/app/_ai_gateway_policies/ai-semantic-cache/index.md b/app/_ai_gateway_policies/ai-semantic-cache/index.md new file mode 100644 index 00000000000..5551a2ae2fb --- /dev/null +++ b/app/_ai_gateway_policies/ai-semantic-cache/index.md @@ -0,0 +1,135 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +related_resources: + - text: Embedding-based similarity matching in Kong AI gateway plugins + url: /ai-gateway/semantic-similarity/ +--- + +The AI Semantic Cache Policy stores user requests to an LLM in a vector database based on semantic meaning. When a similar query is made, it uses these embeddings to retrieve relevant cached requests efficiently. + +## What is semantic caching? + +Semantic caching enhances data retrieval efficiency by focusing on the meaning or context of queries rather than just exact matches. It stores requests based on the underlying intent and semantic similarities between different queries and can then retrieve those cached queries when a similar request is made. + +When a new request is made, the system can retrieve and reuse previously cached requests if they are contextually relevant, even if the phrasing is different. This method reduces redundant processing, speeds up response times, and ensures that answers are more relevant to the user’s intent, ultimately improving overall system performance and user experience. + +For example, if a user asks, "how to integrate our API with a mobile app" and later asks, "what are the steps for connecting our API to a smartphone application?", the system understands that both questions are asking for the same information. It can then retrieve and reuse previously cached responses, even if the wording is different. This approach reduces processing time and speeds up responses. + +The AI Semantic Cache Policy may not be ideal if the following are true: + +* You have limited hardware or budget. Storing semantic vectors and running similarity searches require a lot of storage and computing power, which could be an issue. +* Your data doesn’t rely on semantics, or exact matches work fine. In this case, semantic caching may offer little benefit. Traditional or keyword-based caching might be more efficient. + +## How it works + +Semantic caching with the AI Semantic Cache Policy involves three parts: request handling, embedding generation, and response caching. + +First, a user starts a chat request with the LLM. The AI Semantic Cache Policy queries the vector database to see if there are any semantically similar requests that have already been cached. If there is a match, the vector database returns the cached response to the user. + +{% mermaid %} +sequenceDiagram + actor User + participant {{site.ai_gateway}}/AI Semantic Cache Policy + participant Vector database + + User->>{{site.ai_gateway}}/AI Semantic Cache Policy: LLM chat request + {{site.ai_gateway}}/AI Semantic Cache Policy->>Vector database: Query for semantically similar previous requests + Vector database-->>User: If response, return it or stream it back +{% endmermaid %} + +If there isn't a match, the AI Semantic Cache Policy prompts the embeddings LLM to generate an embedding for the response. + +{% mermaid %} +sequenceDiagram + participant {{site.ai_gateway}}/AI Semantic Cache Policy + participant Embeddings LLM + + {{site.ai_gateway}}/AI Semantic Cache Policy->>Embeddings LLM: Generate embeddings for `config.message_countback` messages + Embeddings LLM-->>{{site.ai_gateway}}/AI Semantic Cache Policy: Return embeddings +{% endmermaid %} + +The AI Semantic Cache Policy uses a vector database and cache to store responses to requests. The Policy can then retrieve a cached response if a new request matches the semantics of a previous request, or it can tell the vector database to store a new response if there are no matches. + +{% mermaid %} +sequenceDiagram + participant {{site.ai_gateway}}/AI Semantic Cache Policy + participant Prompt/Chat LLM + participant Vector database + actor User + + {{site.ai_gateway}}/AI Semantic Cache Policy->>Prompt/Chat LLM: Make LLM request + Prompt/Chat LLM-->>{{site.ai_gateway}}/AI Semantic Cache Policy: Receive response + {{site.ai_gateway}}/AI Semantic Cache Policy->>Vector database: Store vectors + {{site.ai_gateway}}/AI Semantic Cache Policy->>Vector database: Store response message options + {{site.ai_gateway}}/AI Semantic Cache Policy-->>User: Return realtime response +{% endmermaid %} + +### Cache management + +With the AI Semantic Cache Policy, you can configure a cache of your choice to store the responses from the LLM. + +The AI Semantic Cache Policy supports Redis as a cache. + +#### Caching mechanisms + +The AI Semantic Cache Policy improves how AI systems provide responses by using two kinds of caching mechanisms: + +* **Exact Caching:** This stores precise, unaltered responses for specific queries. If a user asks the same question multiple times, the system can quickly retrieve the pre-stored response rather than generating it again each time. This speeds up response times and reduces computational load. +* **Semantic Caching:** This approach is more flexible and involves storing responses based on the meaning or intent behind the queries. Instead of relying on exact matches, the system can understand and reuse information that is conceptually similar. For instance, if a user asks about "Italian restaurants in New York City" and later about "New York City Italian cuisine," semantic caching can help provide relevant information based on their related meanings. + +Together, these caching methods enhance the efficiency and relevance of AI responses, making interactions faster and more contextually accurate. + +{:.info} +> When Exact Caching is enabled, the AI Semantic Cache Policy may still return results for queries that are similar but not identical. This is expected behavior: the Policy performs similarity-based caching regardless of the Exact Caching setting. + +### Headers sent to the client + +When the AI Semantic Cache Policy is active, {{site.ai_gateway}} sends additional headers +indicating the cache status and other relevant information: + +```plaintext +X-Cache-Status: Hit +X-Cache-Status: Miss +X-Cache-Status: Bypass +X-Cache-Status: Refresh +X-Cache-Key: +X-Cache-Ttl: +Age: +``` +{:.no-copy-code} + +These headers help clients understand whether a response was served from the cache, +if the cache key was used, the remaining time-to-live, and the age of the cached response. + +### Cache control headers + +The Policy respects cache control headers to determine if requests and responses should be cached or not. It supports the following directives: + +* `no-store`: Prevents caching of the request or response +* `no-cache`: Forces validation with the origin server before serving the cached response +* `private`: Ensures the response is not cached by shared caches +* `max-age` and `s-maxage`: Sets the maximum age of the cached response. This causes the vector database to drop and delete the cached response message after expiration, so it’s never seen again. + +{:.info} +> As most AI services always send `no-cache` in the response headers, setting [`cache_control`](./reference/#schema--config-cache-control) to `true` will always result in a cache bypass. Only consider setting `no-cache` if you are using self-hosted services and have control over the response Cache Control headers. + +## Vector databases + +{% include_cached /md/ai-gateway/v2/ai-vector-db.md name=page.name %} + +### Using cloud authentication with Redis + +If your Policy uses a Redis datastore, you can authenticate to it with a cloud Redis provider. +This allows you to seamlessly rotate credentials without relying on static passwords. + +{% include_cached /md/ai-gateway/v2/redis-cloud-providers.md name=page.name heading_level=3 %} + +## Forward proxy support + +{% include md/ai-gateway/v2/forward-proxy.md %} diff --git a/app/_ai_gateway_policies/ai-semantic-prompt-guard/index.md b/app/_ai_gateway_policies/ai-semantic-prompt-guard/index.md new file mode 100644 index 00000000000..53346dbc1fe --- /dev/null +++ b/app/_ai_gateway_policies/ai-semantic-prompt-guard/index.md @@ -0,0 +1,61 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +related_resources: + - text: Get started with {{site.ai_gateway}} + url: /ai-gateway/get-started/ + - text: AI Prompt Guard AI Policy + url: /ai-gateway/policies/ai-prompt-guard/ + - text: AI Model + url: /ai-gateway/entities/ai-model/ + - text: AI Semantic Cache AI Policy + url: /ai-gateway/policies/ai-semantic-cache/ + - text: Embedding-based similarity matching in {{site.ai_gateway}} AI Policies + url: /ai-gateway/semantic-similarity/ + +faqs: + - q: Does the AI Semantic Prompt Guard Policy support multilingual input? + a: Yes, the AI Semantic Prompt Guard Policy supports multilingual input—depending on the capabilities of the configured [embedding model](/ai-gateway/policies/ai-semantic-prompt-guard/reference/#schema--config-embeddings-model-provider). The AI Policy sends raw UTF-8 text to the embedding provider supported by {{site.ai_gateway}} (such as Azure, Bedrock, Gemini, Hugging Face, Mistral, or OpenAI). As long as the model supports multiple languages, semantic comparisons and rule enforcement will work as expected without requiring additional policy configuration. + - q: | + How do I resolve the MemoryDB error `Number of indexes exceeds the limit`? + a: | + If you see the following error in the logs: + + ```sh + failed to create memorydb instance failed to create index: LIMIT Number of indexes (11) exceeds the limit (10) + ``` + + This means that the hardcoded MemoryDB instance limit has been reached. + To resolve this, create more MemoryDB instances to handle multiple {{page.name}} policy instances. +--- + +The AI Semantic Prompt Guard Policy enforces prompt governance using semantic similarity matching. It compares incoming requests against your configured allow and deny lists, preventing misuse of text completion requests. + +You can use a combination of `allow` and `deny` rules to maintain integrity and compliance when serving an LLM service using {{site.ai_gateway}}. + +## How it works + +The matching behavior is as follows: +* If any `deny` prompts are set and the request matches a prompt in the `deny` list, the caller receives a 403 response. +* If any `allow` prompts are set, but the request matches none of the allowed prompts, the caller also receives a 403 response. +* If any `allow` prompts are set and the request matches one of the `allow` prompts, the request passes through to the LLM. +* If there are both `deny` and `allow` prompts set, the `deny` condition takes precedence over `allow`. Any request that matches a prompt in the `deny` list will return a 403 response, even if it also matches a prompt in the `allow` list. If the request doesn't match a prompt in the `deny` list, then it must match a prompt in the `allow` list to be passed through to the LLM. + +## Vector databases + +{% include_cached md/ai-gateway/v2/ai-vector-db.md name=page.name %} + +### Using cloud authentication with Redis + +{% include_cached md/ai-gateway/v2/redis-cloud-auth.md tier=page.tier %} + +{% include_cached md/ai-gateway/v2/redis-cloud-providers.md name=page.name heading_level=3 %} + +## Forward proxy support + +{% include md/ai-gateway/v2/forward-proxy.md %} \ No newline at end of file diff --git a/app/_ai_gateway_policies/ai-semantic-response-guard/index.md b/app/_ai_gateway_policies/ai-semantic-response-guard/index.md new file mode 100644 index 00000000000..98ac86ad439 --- /dev/null +++ b/app/_ai_gateway_policies/ai-semantic-response-guard/index.md @@ -0,0 +1,88 @@ +--- +title: 'AI Semantic Response Guard' +name: 'AI Semantic Response Guard' + +content_type: policy + +publisher: kong-inc +description: 'Permit or block LLM responses based on semantic similarity to predefined rules for chat, completions, and embeddings requests' + +products: + - ai-gateway + +works_on: + - konnect + +min_version: + ai-gateway: '2.0' + +topologies: + konnect_deployments: + - hybrid + - cloud-gateways + - serverless + +related_resources: + - text: Get started with {{site.ai_gateway}} + url: /ai-gateway/get-started/ + - text: AI Prompt Guard + url: /ai-gateway/policies/ai-prompt-guard/ + - text: AI Semantic Prompt Guard + url: /ai-gateway/policies/ai-semantic-prompt-guard/ + - text: AI Model + url: /ai-gateway/entities/ai-model/ + - text: AI Semantic Cache + url: /ai-gateway/policies/ai-semantic-cache/ + - text: Embedding-based similarity matching in {{site.ai_gateway}} AI Policies + url: /ai-gateway/semantic-similarity/ + +icon: ai-semantic-response-guard.png + +categories: + - ai +tags: + - ai + - safety + - dlp +--- + +The AI Semantic Response Guard AI Policy filters LLM responses based on semantic similarity to predefined rules, helping prevent unwanted or unsafe responses when serving `/chat`, `/completions`, or `/embeddings` requests through {{site.ai_gateway}}. + +You can use a combination of `allow` and `deny` response rules to maintain integrity and compliance when returning responses from an LLM service. + +## How it works + +The AI Policy analyzes the semantic content of the full LLM response before it is returned to the client. The matching behavior is as follows: + +* If any `deny_responses` are set and the response matches a pattern in the deny list, the response is blocked with a `403 Forbidden`. +* If any `allow_responses` are set, but the response matches none of the allowed patterns, the response is also blocked with a `403 Forbidden`. +* If any `allow_responses` are set and the response matches one of the allowed patterns, the response is permitted. +* If both `deny_responses` and `allow_responses` are set, the `deny` condition takes precedence. A response that matches a deny pattern will be blocked, even if it also matches an allow pattern. If the response does not match any deny pattern, it must still match an allow pattern to be permitted. + +## Response processing + +To enforce these rules, the AI Semantic Response Guard Policy: + +1. Disables streaming (`stream=false`) to ensure the full response body is buffered before analysis. +2. Intercepts the response body using the `guard-buffered-response` filter. +3. Extracts response text, supporting JSON parsing of multiple LLM formats and gzipped content. +4. Generates embeddings for the extracted text. +5. Searches the vector database (Redis, Pgvector, or other) against configured `allow_responses` or `deny_responses`. +6. Applies the decision rules described above. + +{:.info} +> If a response is blocked or if a system error occurs during evaluation, the AI Policy returns a `403 Forbidden` to the client without exposing that the AI Semantic Response Guard blocked it. + +## Vector databases + +{% include_cached md/ai-gateway/v2/ai-vector-db.md name=page.name %} + +### Using cloud authentication with Redis + +{% include_cached md/ai-gateway/v2/redis-cloud-auth.md tier=page.tier %} + +{% include_cached md/ai-gateway/v2/redis-cloud-providers.md name=page.name heading_level=3 %} + +## Forward proxy support + +{% include md/ai-gateway/v2/forward-proxy.md %} \ No newline at end of file diff --git a/app/_ai_gateway_policies/app-dynamics/index.md b/app/_ai_gateway_policies/app-dynamics/index.md new file mode 100644 index 00000000000..86bb3e3da4a --- /dev/null +++ b/app/_ai_gateway_policies/app-dynamics/index.md @@ -0,0 +1,10 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +description: Integrate {{site.ai_gateway_name}} with the AppDynamics APM Platform +--- diff --git a/app/_ai_gateway_policies/aws-lambda/index.md b/app/_ai_gateway_policies/aws-lambda/index.md new file mode 100644 index 00000000000..3573ab8e78e --- /dev/null +++ b/app/_ai_gateway_policies/aws-lambda/index.md @@ -0,0 +1,10 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +description: Invoke and manage AWS Lambda functions from {{site.ai_gateway_name}} +--- diff --git a/app/_ai_gateway_policies/azure-functions/index.md b/app/_ai_gateway_policies/azure-functions/index.md new file mode 100644 index 00000000000..c0362d5f229 --- /dev/null +++ b/app/_ai_gateway_policies/azure-functions/index.md @@ -0,0 +1,10 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +description: Invoke and manage Azure functions from {{site.ai_gateway_name}} +--- diff --git a/app/_ai_gateway_policies/basic-auth/index.md b/app/_ai_gateway_policies/basic-auth/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/basic-auth/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/bot-detection/index.md b/app/_ai_gateway_policies/bot-detection/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/bot-detection/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/canary/index.md b/app/_ai_gateway_policies/canary/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/canary/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/confluent-consume/index.md b/app/_ai_gateway_policies/confluent-consume/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/confluent-consume/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/confluent/index.md b/app/_ai_gateway_policies/confluent/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/confluent/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/correlation-id/index.md b/app/_ai_gateway_policies/correlation-id/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/correlation-id/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/cors/index.md b/app/_ai_gateway_policies/cors/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/cors/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/datadog/index.md b/app/_ai_gateway_policies/datadog/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/datadog/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/datakit/index.md b/app/_ai_gateway_policies/datakit/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/datakit/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/degraphql/index.md b/app/_ai_gateway_policies/degraphql/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/degraphql/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/exit-transformer/index.md b/app/_ai_gateway_policies/exit-transformer/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/exit-transformer/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/file-log/index.md b/app/_ai_gateway_policies/file-log/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/file-log/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/forward-proxy/index.md b/app/_ai_gateway_policies/forward-proxy/index.md new file mode 100644 index 00000000000..6680c5d291e --- /dev/null +++ b/app/_ai_gateway_policies/forward-proxy/index.md @@ -0,0 +1,10 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +description: Allows {{site.ai_gateway_name}} to connect to intermediary transparent HTTP +--- diff --git a/app/_ai_gateway_policies/graphql-proxy-cache-advanced/index.md b/app/_ai_gateway_policies/graphql-proxy-cache-advanced/index.md new file mode 100644 index 00000000000..bc855d09d7a --- /dev/null +++ b/app/_ai_gateway_policies/graphql-proxy-cache-advanced/index.md @@ -0,0 +1,10 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +description: Cache and serve commonly requested responses in {{site.ai_gateway_name}} +--- diff --git a/app/_ai_gateway_policies/graphql-rate-limiting-advanced/index.md b/app/_ai_gateway_policies/graphql-rate-limiting-advanced/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/graphql-rate-limiting-advanced/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/grpc-gateway/index.md b/app/_ai_gateway_policies/grpc-gateway/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/grpc-gateway/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/grpc-web/index.md b/app/_ai_gateway_policies/grpc-web/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/grpc-web/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/header-cert-auth/index.md b/app/_ai_gateway_policies/header-cert-auth/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/header-cert-auth/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/hmac-auth/index.md b/app/_ai_gateway_policies/hmac-auth/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/hmac-auth/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/http-log/index.md b/app/_ai_gateway_policies/http-log/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/http-log/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/injection-protection/index.md b/app/_ai_gateway_policies/injection-protection/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/injection-protection/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/ip-restriction/index.md b/app/_ai_gateway_policies/ip-restriction/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/ip-restriction/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/jq/index.md b/app/_ai_gateway_policies/jq/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/jq/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/json-threat-protection/index.md b/app/_ai_gateway_policies/json-threat-protection/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/json-threat-protection/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/jwe-decrypt/index.md b/app/_ai_gateway_policies/jwe-decrypt/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/jwe-decrypt/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/jwt-signer/index.md b/app/_ai_gateway_policies/jwt-signer/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/jwt-signer/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/jwt/index.md b/app/_ai_gateway_policies/jwt/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/jwt/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/kafka-consume/index.md b/app/_ai_gateway_policies/kafka-consume/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/kafka-consume/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/kafka-log/index.md b/app/_ai_gateway_policies/kafka-log/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/kafka-log/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/kafka-upstream/index.md b/app/_ai_gateway_policies/kafka-upstream/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/kafka-upstream/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/key-auth-enc/index.md b/app/_ai_gateway_policies/key-auth-enc/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/key-auth-enc/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/key-auth/index.md b/app/_ai_gateway_policies/key-auth/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/key-auth/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/ldap-auth-advanced/index.md b/app/_ai_gateway_policies/ldap-auth-advanced/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/ldap-auth-advanced/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/ldap-auth/index.md b/app/_ai_gateway_policies/ldap-auth/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/ldap-auth/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/loggly/index.md b/app/_ai_gateway_policies/loggly/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/loggly/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/metering-and-billing/index.md b/app/_ai_gateway_policies/metering-and-billing/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/metering-and-billing/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/mocking/index.md b/app/_ai_gateway_policies/mocking/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/mocking/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/mtls-auth/index.md b/app/_ai_gateway_policies/mtls-auth/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/mtls-auth/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/oas-validation/index.md b/app/_ai_gateway_policies/oas-validation/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/oas-validation/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/oauth2-introspection/index.md b/app/_ai_gateway_policies/oauth2-introspection/index.md new file mode 100644 index 00000000000..d09714deba1 --- /dev/null +++ b/app/_ai_gateway_policies/oauth2-introspection/index.md @@ -0,0 +1,10 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +description: Integrate {{site.ai_gateway_name}} with a third-party OAuth 2.0 Authorization +--- diff --git a/app/_ai_gateway_policies/oauth2/index.md b/app/_ai_gateway_policies/oauth2/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/oauth2/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/opa/index.md b/app/_ai_gateway_policies/opa/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/opa/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/openid-connect/index.md b/app/_ai_gateway_policies/openid-connect/index.md new file mode 100644 index 00000000000..5e14adb700a --- /dev/null +++ b/app/_ai_gateway_policies/openid-connect/index.md @@ -0,0 +1,10 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +description: Integrate {{site.ai_gateway_name}} with a third-party OpenID Connect provider +--- diff --git a/app/_ai_gateway_policies/opentelemetry/index.md b/app/_ai_gateway_policies/opentelemetry/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/opentelemetry/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/post-function/index.md b/app/_ai_gateway_policies/post-function/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/post-function/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/pre-function/index.md b/app/_ai_gateway_policies/pre-function/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/pre-function/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/prometheus/index.md b/app/_ai_gateway_policies/prometheus/index.md new file mode 100644 index 00000000000..6cd8dc10a0e --- /dev/null +++ b/app/_ai_gateway_policies/prometheus/index.md @@ -0,0 +1,10 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +description: Expose metrics related to {{site.ai_gateway_name}} in Prometheus exposition format +--- diff --git a/app/_ai_gateway_policies/proxy-cache-advanced/index.md b/app/_ai_gateway_policies/proxy-cache-advanced/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/proxy-cache-advanced/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/proxy-cache/index.md b/app/_ai_gateway_policies/proxy-cache/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/proxy-cache/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/rate-limiting-advanced/index.md b/app/_ai_gateway_policies/rate-limiting-advanced/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/rate-limiting-advanced/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/rate-limiting/index.md b/app/_ai_gateway_policies/rate-limiting/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/rate-limiting/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/redirect/index.md b/app/_ai_gateway_policies/redirect/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/redirect/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/request-callout/index.md b/app/_ai_gateway_policies/request-callout/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/request-callout/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/request-size-limiting/index.md b/app/_ai_gateway_policies/request-size-limiting/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/request-size-limiting/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/request-termination/index.md b/app/_ai_gateway_policies/request-termination/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/request-termination/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/request-transformer-advanced/index.md b/app/_ai_gateway_policies/request-transformer-advanced/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/request-transformer-advanced/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/request-transformer/index.md b/app/_ai_gateway_policies/request-transformer/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/request-transformer/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/request-validator/index.md b/app/_ai_gateway_policies/request-validator/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/request-validator/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/response-ratelimiting/index.md b/app/_ai_gateway_policies/response-ratelimiting/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/response-ratelimiting/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/response-transformer-advanced/index.md b/app/_ai_gateway_policies/response-transformer-advanced/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/response-transformer-advanced/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/response-transformer/index.md b/app/_ai_gateway_policies/response-transformer/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/response-transformer/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/route-by-header/index.md b/app/_ai_gateway_policies/route-by-header/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/route-by-header/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/route-transformer-advanced/index.md b/app/_ai_gateway_policies/route-transformer-advanced/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/route-transformer-advanced/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/saml/index.md b/app/_ai_gateway_policies/saml/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/saml/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/service-protection/index.md b/app/_ai_gateway_policies/service-protection/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/service-protection/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/session/index.md b/app/_ai_gateway_policies/session/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/session/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/solace-consume/index.md b/app/_ai_gateway_policies/solace-consume/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/solace-consume/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/solace-log/index.md b/app/_ai_gateway_policies/solace-log/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/solace-log/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/solace-upstream/index.md b/app/_ai_gateway_policies/solace-upstream/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/solace-upstream/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/standard-webhooks/index.md b/app/_ai_gateway_policies/standard-webhooks/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/standard-webhooks/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/statsd/index.md b/app/_ai_gateway_policies/statsd/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/statsd/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/syslog/index.md b/app/_ai_gateway_policies/syslog/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/syslog/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/tcp-log/index.md b/app/_ai_gateway_policies/tcp-log/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/tcp-log/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/tls-handshake-modifier/index.md b/app/_ai_gateway_policies/tls-handshake-modifier/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/tls-handshake-modifier/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/tls-metadata-headers/index.md b/app/_ai_gateway_policies/tls-metadata-headers/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/tls-metadata-headers/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/udp-log/index.md b/app/_ai_gateway_policies/udp-log/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/udp-log/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/upstream-oauth/index.md b/app/_ai_gateway_policies/upstream-oauth/index.md new file mode 100644 index 00000000000..3247b425da9 --- /dev/null +++ b/app/_ai_gateway_policies/upstream-oauth/index.md @@ -0,0 +1,10 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +description: Configure {{site.ai_gateway_name}} to obtain an OAuth2 token to consume an upstream API +--- diff --git a/app/_ai_gateway_policies/upstream-timeout/index.md b/app/_ai_gateway_policies/upstream-timeout/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/upstream-timeout/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/vault-auth/index.md b/app/_ai_gateway_policies/vault-auth/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/vault-auth/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/websocket-size-limit/index.md b/app/_ai_gateway_policies/websocket-size-limit/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/websocket-size-limit/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/websocket-validator/index.md b/app/_ai_gateway_policies/websocket-validator/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/websocket-validator/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/xml-threat-protection/index.md b/app/_ai_gateway_policies/xml-threat-protection/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/xml-threat-protection/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_ai_gateway_policies/zipkin/index.md b/app/_ai_gateway_policies/zipkin/index.md new file mode 100644 index 00000000000..ca3f31a2e3a --- /dev/null +++ b/app/_ai_gateway_policies/zipkin/index.md @@ -0,0 +1,9 @@ +--- +min_version: + ai-gateway: '2.0' +works_on: + - konnect +products: + - ai-gateway +content_type: plugin +--- diff --git a/app/_api/konnect/ai-gateway/_index.md b/app/_api/konnect/ai-gateway/_index.md new file mode 100644 index 00000000000..af7d3871f2b --- /dev/null +++ b/app/_api/konnect/ai-gateway/_index.md @@ -0,0 +1,3 @@ +--- +konnect_product_id: 5e0005b9-232b-4808-8bdf-9560d4596080 +--- \ No newline at end of file diff --git a/app/_assets/javascripts/apps/components/SearchModalResultItem.vue b/app/_assets/javascripts/apps/components/SearchModalResultItem.vue index 81fb11a4d62..c686f273072 100644 --- a/app/_assets/javascripts/apps/components/SearchModalResultItem.vue +++ b/app/_assets/javascripts/apps/components/SearchModalResultItem.vue @@ -59,7 +59,7 @@ export default { return this.item.title; } if (this.item.content_type === 'plugin') { - if (this.item.products && (this.item.products.includes('mesh') || this.item.products.includes('event-gateway'))) { + if (this.item.products && (this.item.products.includes('mesh') || this.item.products.includes('event-gateway') || this.item.products === 'ai-gateway')) { return `${this.item.hierarchy.lvl1} Policy`; } else { return `${this.item.hierarchy.lvl1} Plugin`; @@ -79,7 +79,7 @@ export default { .map(([key, value]) => value); if (this.item.content_type === 'plugin') { - if (this.item.products && (this.item.products.includes('mesh') || this.item.products.includes('event-gateway'))) { + if (this.item.products && (this.item.products.includes('mesh') || this.item.products.includes('event-gateway') || this.item.products === 'ai-gateway')) { levels.unshift('Policies') } else { levels.unshift('Plugins') diff --git a/app/_assets/javascripts/banner.js b/app/_assets/javascripts/banner.js index 2d417449c67..e890fec3cb6 100644 --- a/app/_assets/javascripts/banner.js +++ b/app/_assets/javascripts/banner.js @@ -2,8 +2,9 @@ class Banner { constructor(elem) { this.banner = elem; this.closeButton = this.banner.querySelector(".close-banner"); - this.bannerDataId = this.closeButton.dataset.storageId; - + if (this.closeButton) { + this.bannerDataId = this.closeButton.dataset.storageId; + } this.init(); this.addEventListeners(); } @@ -16,7 +17,9 @@ class Banner { } addEventListeners() { - this.closeButton.addEventListener("click", this.onClose.bind(this)); + if (this.closeButton) { + this.closeButton.addEventListener("click", this.onClose.bind(this)); + } } onClose() { diff --git a/app/_changelogs/ai-gateway.json b/app/_changelogs/ai-gateway.json new file mode 100644 index 00000000000..c0cb94b7d8a --- /dev/null +++ b/app/_changelogs/ai-gateway.json @@ -0,0 +1,900 @@ +{ + "2.0.0": { + "kong-aigw": [ + { + "message": "**ais**: Dropped deprecated schema fields slated for removal in 4.0:\n`http_proxy_host`/`http_proxy_port`/`https_proxy_host`/`https_proxy_port` shorthands\nin `ai-llm-as-judge`, `ai-request-transformer`, `ai-response-transformer` (use\n`proxy_config` instead); `llm_format`, `max_request_body_size`, and\n`rules.max_request_body_size` in `ai-semantic-prompt-guard`; `llm_providers` and\n`llm_format` in `ai-rate-limiting-advanced` (`policies` is now required);\n`model.options.upstream_path` and the `preserve` route_type in the shared LLM\nschema (use `model.options.upstream_url` and a concrete `route_type` instead).", + "type": "Breaking Change", + "scope": "Plugin" + }, + { + "message": "Removed the `ai-proxy` plugin from the bundled plugins list. Use `ai-proxy-advanced` instead.", + "type": "Breaking Change", + "scope": "Plugin" + }, + { + "message": "Bumped OpenSSL from 3.5.6 to 3.5.7.", + "type": "dependency", + "scope": "Core" + }, + { + "message": "Added data plane support for the new AI Gateway control plane. Data planes now\nuse JSON-RPC config sync (sync v2) only and send a validated `deployment_type` value in\nthe RPC hello payload.", + "type": "feature", + "scope": "Core" + }, + { + "message": "**ai-a2a-proxy**: Added A2A v1.0 wire-format support alongside v0.3.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-a2a-proxy**: now rewrites the URL for all `supportedInterfaces` in the agent card.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**AI Gateway**: Added a `tags` field to the `ai_models` entity, enabling tag-based filtering on the Admin API and the `/tags` endpoint.", + "type": "feature", + "scope": "Core" + }, + { + "message": "**ai-guardrail**: Add `guardrail_triggered` field to protobuf analytics payload.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-llm-as-judge**: Added the opt-in response header to expose the computed judge score to clients. The header name defaults to `X-Kong-LLM-Accuracy-Score` and is configurable via `score_header_name`.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-proxy**: Added upstream MCP server aggregator support, allowing the plugin to aggregate tools from multiple upstream MCP servers.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-proxy**, **ai-mcp-oauth2**: Introduced forward-proxy support for outbound traffic configuration.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-proxy**: Updated the default MCP protocol version to `2025-11-25` and added explicit negotiation for older clients.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ais**: Added `proxy_config` support to route plugin HTTP requests through a\nforward proxy for ai-aws-guardrails, ai-azure-content-safety, ai-custom-guardrail,\nai-gcp-model-armor, ai-lakera-guard, ai-sanitizer, ai-prompt-compressor,\nai-rag-injector, ai-semantic-cache, ai-semantic-prompt-guard,\nai-semantic-response-guard, ai-request-transformer, ai-response-transformer,\nand ai-llm-as-judge. For ai-request-transformer, ai-response-transformer, and\nai-llm-as-judge, the existing top-level `http_proxy_host`/`http_proxy_port`/\n`https_proxy_host`/`https_proxy_port` fields are now deprecated in favor of\n`proxy_config` and will be removed in 4.0.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-proxy**: Added OTel span attributes gen_ai.input.messages for audio speech routes and gen_ai.output.messages for audio transcription routes when log_payloads is enabled.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-proxy**: Added support for Kimi AI (Moonshot) - a new AI Provider.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-proxy**: Added support for Vercel - a new AI Provider.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Added clarification for handling \"Request body too large\" errors.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Added embeddings support for databricks provider.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Added Gemini OpenAI-compatible `logprobs` mapping on chat completions.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Added new ACL matcher type `authenticated_groups`, which allows propagating claims and\nroles from other authentication plugins into the AI Proxy Advanced per-model ACLs.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Added `proxy_config` support for forward proxies, including HTTP proxying, Basic authentication, and CONNECT tunneling for HTTPS upstream streaming.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Added reasoning fields mapping between Anthropic Messages and the OpenAI internal format.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Added support for image analysis request to anthropic/bedrock providers.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Added support for anthropic format chat on DeepSeek provider.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-proxy**: Improved Cohere driver with streaming fixes, tool calling support, and usage tracking.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-rate-limiting-advanced**: Added calendar window type. Fixed fetch drift in rate limiting windows.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**kimi**: Added support for anthropic passthrough on Kimi (Moonshot) provider.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**llm**: Increase max request body size from 1MB to 8MB.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**vercel**: Added support for Anthropic passthrough on Vercel provider.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-a2a-proxy**: Reassembled SSE `data:` events that span multiple `body_filter`\nchunks and dropped oversized events safely, preventing `sse_last_event` type\nerrors that crashed the request on large agent responses.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-advanced-proxy**: Fixed an issue where we never stored response model in streaming path. It always fallback to the request model", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-aws-guardrails**: Allowed streaming of responses when `guarding_mode` was set to `INPUT` with `allow_masking` enabled.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-azure-content-safety**: Fixed a crash (HTTP 500) when `azure_use_managed_identity` is enabled.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-azure-content-safety**: Fixed a bug where the plugin returned HTTP 500 when the LLM response content was empty. Empty content now skips the content-safety check and passes through.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-azure-content-safety**: Fixed response guard sending the entire JSON response body to Azure Content Safety instead of just the assistant message content, which caused false positives when response metadata fields matched blocklist entries.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai**: Fixed an issue where non-200 responses from Bedrock embeddings API were not properly handled, leading to incorrect feedback to the client.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-guardrail**: Fixed an issue where latency metrics were missing in the Prometheus output.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-guardrails**: Fixed an issue where multiple guardrail plugins can not function together.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-oauth2**: Fixed an issue where token introspection crashed when `client_auth` was set to `private_key_jwt` and `client_jwk` was provided as a JSON string.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-oauth2**: Fixed an issue where downstream scope-based ACL evaluation used claims from the inbound token instead of the exchanged token when token exchange was enabled.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-oauth2**: Fixed an issue where token-related 401 responses omitted the protected resource metadata URL when token was given.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-oauth2**: Fixed an issue where token exchange actor tokens were not sourced correctly from `token_exchange.request`, ensuring correct forwarding of actor tokens configured in headers or plugin config.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-oauth2**: Fixed an issue where `token_exchange.cache.enabled = false` was ignored and exchanged tokens were still cached because the cache toggle incorrectly read `token_exchange.cache.ttl` instead of `token_exchange.cache.enabled`.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-proxy**: Fixed an issue where empty JSON arrays in MCP server\nresponses were rewritten to empty objects when the plugin re-encoded the\nbody for ACL list filtering.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-proxy**: Fixed an issue where raw non-200 upstream responses triggered misleading MCP body parse warnings.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-proxy**: Fixed an issue where WWW-Authenticate: Bearer error=\"insufficient_scope\" header was not emitted.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-proxy**: Fixed an issue where internal unix-socket subrequests lost the original client IP address, causing IP-aware plugins like ip-restriction to reject legitimate requests.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-proxy**: Fixed an issue where we didn't propagate errors on secret_to_jwk calls.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-proxy**: Fixed an issue where, since 3.14, call tools converted from API could fail when Kong used a self-signed certificate.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-proxy**: Fixed an issue where converted HTTPS MCP tool calls lost the original forwarded port on the internal TLS Unix socket.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-proxy**: Fixed an issue where tool id might be repeated.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-proxy**: Fixed an issue where we didn't allow arbitrary tool path in conversion-listener mode.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-prompt-decorator**: Fixed an issue where the plugin corrupted native-format\nrequests (gemini, anthropic, bedrock, cohere, huggingface) by forwarding an\nOpenAI-shaped body upstream instead of the original native body shape.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where native-format requests using a model alias were incorrectly proxying to upstream model.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where full-sync reconfigure reset balancer state even when plugin configs were unchanged.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where concurrent worker processes could race to initialize the semantic routing vectorDB index, causing \"Index already exists\" errors on startup or config reload.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where driver transformers returned errors or stream metadata in the wrong return position.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where Vercel Anthropic-format requests did not include the configured model in the upstream body.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where DeepSeek, Vercel, and Kimi Anthropic count-token requests were proxied to chat endpoints.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where OpenAI-compatible Anthropic chat requests only preserved the last system message.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where Anthropic streaming finish reasons are not mapped in OpenAI to Anthropic conversion.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where the Anthropic driver dropped the `top_p`, `top_k`, and `stop` parameters when translating OpenAI-format requests. The driver now also omits `temperature`, `top_p`, and `top_k` for models that reject them (Claude Opus 4.7, Opus 4.8, and Fable), preventing upstream 400 errors.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where `output_config` was rejected by Vertex AI in Anthropic-native requests.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where `cache_creation_input_tokens` and `cache_read_input_tokens` were excluded from prompt token counts, causing undercounting when Anthropic prompt caching was active.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where configured model name may never be used for Azure provider.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where we didn't support Azure GA version of Realtime API.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where OpenAI-format chat reasoning requests were not mapped consistently across Anthropic, Gemini, Bedrock and more providers.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where system prompt of Claude Code was truncated when doing non-passthrough proxy.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where we didn't support anthropic format for databricks provider.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where we didn't use the huggingface's token usage if available.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where unsafe model names could construct upstream AI provider paths.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where we didn't collect the model name in OpenAI streaming response from Azure.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where unsafe batch id could cause upstream AI provider path traversal.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where unsafe model name in native format adaptors could cause upstream AI provider path traversal.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where dashscope replied 500 to inference request.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where the xAI driver nested `reasoning_effort` under `reasoning.effort` and did not normalize `reasoning_content` in responses.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where `stream_options` was forwarded to Databricks upstreams, causing a 400 error since Databricks does not support this field.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where GenAI spans were not finished sometimes.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where observability errors occurred when `gpt-image-1.5` returned image token usage details.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where Azure OpenAI authentication failed during load balancer failover due to token refresh attempts in `balancer_by_lua`. Tokens are now pre-fetched in the `access` phase.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where cache_control in Bedrock Anthropic format requests using ARN model IDs was lost.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where Bedrock native ConverseStream requests using ARN model IDs could fail while handling streamed responses.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where some Claude Code MCP tools did not correctly handle empty 'requires' array.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed Azure OpenAI responses support in deployment style.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy**: Fixed an issue where responses with unsupported content types could be returned with an incorrect gzip header.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy**: Fixed an issue where log_payloads logged binary data for audio routes, the plugin now logs only the request for speech and only the response for transcription/translation routes.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy**: Fixed Azure AI Foundry v1 API support to not require deployment-specific fields when `upstream_url` is configured.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy**: Fixed Gemini driver dropping Gemini-native fields (e.g. `cachedContent`, `safetySettings`) and Vertex AI-only fields (e.g. `labels`) during OpenAI-to-Gemini request transformation.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy**: Fixed role matching in message extraction and formatting to be case-insensitive, and added a nil-safety check for messages.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy**: Fixed an issue where `options.upstream_url` was not respected in native mode.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-rate-limiting-advanced**: Emitted the documented `X-AI-RateLimit-*` headers on `429` responses from policies that partition by `provider` or `model`.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-rate-limiting-advanced**: Fixed an issue where `model` or `provider` matchers without `partition_by` were skipped, causing rate limits to be missed.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-rate-limiting-advanced**: Fixed policy integration with `ai-proxy`\nfor single-target and multi-target AI proxy routes.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-request-transformer**: Fixed an issue where runtime template placeholders (e.g. `$(uri_captures.*)`) in `config.llm.model.*` fields were not interpolated.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-request-transformer**: Fixed an issue where unsafe model name could cause upstream AI provider path traversal.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-response-transformer**: Fixed an issue where runtime template placeholders (e.g. `$(uri_captures.*)`) in `config.llm.model.*` fields were not interpolated.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-sanitizer**: Fixed an issue where the AI sanitizer plugin could not parse the request body correctly.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-sanitizer**: Fixed an issue where the input field for the llm/v1/responses route was not recognized.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-semantic-prompt-guard**, **ai-semantic-response-guard**: Fixed an issue where full-sync reconfigure rebuilt semantic vectorDB indexes even when the plugin configuration was unchanged.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-semantic-prompt-guard**: Fixed an issue where max_request_body_size and llm_format were not marked as deprecated.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-semantic-response-guard**: Fixed an issue where the plugin failed to populate Redis with rule documents due to using worker ID gating instead of worker mutex for vectordb operations.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "Applied upstream nginx security patches for CVE-2026-40701, CVE-2026-40460, CVE-2026-42934, CVE-2026-42946, and CVE-2026-42945.", + "type": "bugfix", + "scope": "Core" + }, + { + "message": "Applied upstream nginx security patches for limiting the number of maximum headers (CVE-2026-49975).", + "type": "bugfix", + "scope": "Core" + }, + { + "message": "**debugger**: Fixed an issue where response body content capture consumed streaming response chunks.", + "type": "bugfix", + "scope": "Core" + }, + { + "message": "Fixed AI Proxy Advanced Gemini failover handling when a target sets `upstream_url` to `ngx.null`, avoiding an empty trailing query string in the computed upstream URI.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "Fixed an issue where AI proxy plugins rejected `output_cost: 0` in schema validation, preventing users from configuring models that have no output token cost.", + "type": "bugfix", + "scope": "Core" + }, + { + "message": "Fixed an issue where anonymous reports omitted model usage when one provider used multiple models.", + "type": "bugfix", + "scope": "Core" + }, + { + "message": "Fixed an issue where cluster mutex did not timeout immediately with `no_wait = true` when node level mutex is being held by another worker.", + "type": "bugfix", + "scope": "Core" + }, + { + "message": "Fixed an issue where when Postgres strategy is used for session storage, concurrent authentication requests could result in race condition and session creation collisions.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "Increased the default `proxy_ssl_verify_depth` and `lua_ssl_verify_depth` from `1` to `5`.", + "type": "bugfix", + "scope": "Configuration" + }, + { + "message": "**openid-connect**: Fixed an issue where the `redirect_uri` parameter was sent to the token endpoint for all grant types. It is now included only for the `authorization_code` grant, as required by RFC 6749.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**tcp-log**: Fix the race condition where the balancer `tries` array gets cleared before the asynchronous timer thread can safely encode it.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai**: Removed unused redis connection when setting up Redis vectordb.", + "type": "performance", + "scope": "Plugin" + } + ] + }, + "2.0.1": { + "kong-aigw": [ + { + "message": "Bumped jq from 1.8.1 to 1.8.2.", + "type": "dependency", + "scope": "Core" + }, + { + "message": "**ai-custom-guardrail**: Added the `log_blocked_content` field to allow logging prompts and responses blocked by the guardrail, and updated the `continue_on_detection` description to mention that block data is visible via Kong logs.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-guardrails**: Added a `rejection_mode` configuration field (`none`, `stealth`, `verbose`) to control how blocked requests and responses are communicated to the client.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-guardrails**: Added a shared `continue_on_detection` field to all guardrails to allow for non-blocking guardrail triggers.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-oauth2**: Added `expected_audience` to configure the expected token audience independently from `resource`.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-proxy**: Added JSON Schema draft 2020-12 support for input schema validation.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-model-selector**: Added support for extracting model aliases from request paths.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-nvidia-nemo-guardrail**: Added new plugin to check LLM request/response content against NVIDIA NeMo Guardrails.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-proxy**: Added support for AWS SageMaker as a new AI provider.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Added a configurable pricing catalog that computes LLM `usage.cost` from per-(provider, model) token rates, including cache read/write rates.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Added OpenAI-format chat reasoning request and response mappings across supported AI providers.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Added support for reporting prompt cache token metrics in OpenAI-compatible format.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-proxy/ai-proxy-advanced**: allowed ai-proxy/ai-proxy-advanced plugins to be scoped with the ai-model entity.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ai-semantic-response-guard**: added the `response_buffer_size` option to support guarding streaming responses.", + "type": "feature", + "scope": "Plugin" + }, + { + "message": "**ACL**: Fixed an issue where the ACL plugin returned 403 for authenticated\ncredentials with no consumer and no groups (e.g. ACE / Dev Portal application\ncredentials) when configured with a deny list. A group-less identity cannot\nmatch a deny list and should be allowed through.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**acl**: Fixed an issue where the ACL plugin ignored a consumer's ACL groups when `include_consumer_groups` was enabled and the consumer belonged to an enterprise consumer group, causing incorrect `403` denials (with `allow`) or incorrect access (with `deny`) for authenticated and anonymous consumers. The plugin now falls back to the consumer's ACL groups when no third-party authenticated groups are present.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-oauth2**: Fixed a JWT security vulnerability.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-proxy**: Fixed an issue where the ACL decision cache has collision.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-proxy**: Fixed an issue where the MCP `initialize` response reported the wrong version.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-mcp-proxy**: Fixed an issue where non-managed MCP listeners advertised unsupported SSE streams.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-model-selector**: Fixed model alias extraction from the request body to support `multipart/form-data` requests in addition to JSON.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai**: Fixed the Redis vector database index being dropped and recreated on every request when metadata fields were enabled.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-prompt-compressor**: Fixed an issue where native Anthropic requests were forwarded in OpenAI format after prompt compression.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed a Lua crash in declarative schema validation when a Bedrock target had no `auth` block (e.g. IRSA / role-only configs), which made the entire declarative config unloadable.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where the configured auth header (e.g. `api-key`, `x-api-key`) leaked the provider credential into access logs instead of being redacted.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where the OpenAI `developer` role was not treated as a system prompt when transforming requests to Bedrock and Gemini providers.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where we didn't handle SSE event headers correctly.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where balancer retries could continue reusing the last selected target after all priority peers were exhausted.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where Gemini chat requests rejected OpenAI `input_audio` and `video_url` content parts.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where native Bedrock request parsing dropped string message content in kong internal format, causing invalid data used by other plugins.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where OpenAI content array was sent as bedrock/gemini system message which only support string.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where OpenAI, Azure, and Gemini realtime upstream URLs using `wss` were downgraded to `ws`.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where requests to a Gemini or Anthropic provider could fail with 401 when using the OpenAI format with a custom auth header (e.g. `x-goog-api-key`, `x-api-key`) or query-parameter auth. The original `Authorization` header sent by the OpenAI SDK is now cleared so it does not conflict with the configured credential.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where semantic routing failed with \"found target with ID but it's not mapped to a target\" when multiple workers or nodes raced to create the vectorDB index: \"Index already exists\" errors from concurrent index creation are now treated as success in the Redis, Valkey, and MemoryDB strategies, so every worker completes initialization and builds its local target mapping.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where the `context-management-2025-06-27` Anthropic beta flag was dropped when routing to a Bedrock target, causing Bedrock to reject requests carrying `context_management` with a `400` error on failover from an Anthropic target.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed an issue where a malformed RSA private key in a Gemini provider's `service_account_json` caused crash with SIGSEGV", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed Gemini service-account auth probing `metadata.google.internal` when a service account JSON was configured.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy-advanced**: Fixed reuse of an expired Azure managed-identity token causing intermittent upstream 401s.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy**: Fixed a confusing error message when a Bedrock native-format request uses an invalid ARN model path.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy**: Fixed an issue where Bedrock driver could not handle OpenAI content_parts format in system messages, user/assistant messages, and array-typed tool results.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-proxy**: Fixed OpenAI-compatible completions requests with string array prompts.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-rag-injector**: Fixed percent sequences in injected prompt text.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-rate-limiting-advanced**: Fixed token counters being decremented on ai-semantic-cache hits.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-request-transformer**: Fixed an issue where requests transformed before `ai-proxy-advanced` skipped AI response normalization.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**ai-semantic-cache**: Fixed cache hits ignoring `llm_format`, which returned OpenAI-shaped responses to native-format (e.g. Anthropic) clients. Cached responses are now converted to the client's `llm_format` for both streaming and non-streaming requests.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**Clustering**: The `/status/sync` endpoint now returns an empty stat object instead of null before the first sync has run, keeping the response shape consistent.", + "type": "bugfix", + "scope": "Clustering" + }, + { + "message": "**datakit**: Fixed CVE-2026-14916 by binding the JWT algorithm to the\nverification key's type in the `jwt_verify` node, so a symmetric algorithm can\nno longer be verified against an asymmetric key, preventing an authentication\nbypass where an attacker forges tokens by using the public key as an HMAC\nshared secret, under configurations where the node uses a PEM key string and\n`allowed_algorithms` is unset (any algorithm accepted).", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "Fixed an issue on HTTP/2 connections where the request method and query were reported incorrectly, or could raise an error, when a request hit the error handler such as a 502 from an upstream connection reset.", + "type": "bugfix", + "scope": "PDK" + }, + { + "message": "Fixed an issue where dynamic plugin ordering (`ordering`) leaked into the\ndownstream phases (`header_filter`, `body_filter`, `response`, `log`). Dynamic\nordering now again applies only to the `access` phase; downstream phases execute\nin static plugin-priority order.", + "type": "bugfix", + "scope": "Core" + }, + { + "message": "Fixed an issue where resetting the password of an admin or developer failed with \"Old password is invalid\" when the credential was created before the basic-auth hashing algorithm was upgraded from SHA1 to SHA256 (e.g. a `kong_admin` user created in an older version and carried over through an upgrade). The old password is now verified with the backward-compatible verification logic that supports both SHA1 and SHA256 hashed passwords.", + "type": "bugfix", + "scope": "Admin API" + }, + { + "message": "**ip-restriction**: Fixed an issue where warn log messages did not\ninclude the plugin name, making it difficult to identify the source of\nblocked requests.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**jwt-signer**: Fixed an issue where the plugin failed to load configuration and errored when `null` or `nil` values were passed to `access_token_keyset` or `channel_token_keyset`.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**kafka-upstream**, **kafka-log**, **confluent**, **kafka-consume**, **confluent-consume**: Fixed a crash\nthat occurred when a plugin set `schema_registry.confluent.authentication.mode`\nto `oauth2` but omitted the `oauth2_client` record. Previously this returned a\n500 from the Admin API during validation, and once saved it also crashed at\nruntime while fetching the schema registry OAuth2 token. The absent\n`oauth2_client` is now handled and falls back to the default authentication\nmethod, so a configuration without `oauth2_client` works end-to-end.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**Konnect Only**. **Debugger**: Fixed an error that could occur under sustained HTTPS traffic.", + "type": "bugfix", + "scope": "Core" + }, + { + "message": "**Konnect Only**. **Debugger**: Fixed an issue where the default key name `apikey` of the `key-auth` and `key-auth-enc` plugins was not redacted when captured by the Debugger. This has been resolved by adding `apikey` to the list of default redacted headers.", + "type": "bugfix", + "scope": "Core" + }, + { + "message": "**Konnect Only**. **Debugger**: Fixed an issue where the response body of the external HTTP call `httpc:request()` were not captured.", + "type": "bugfix", + "scope": "Core" + }, + { + "message": "**Konnect Only**. **Debugger**: Fixed an issue where instrumenting third-party HTTP client calls produced two indistinguishable `kong.io.http.request` spans for `httpc:request_uri()`.\nIt is now traced under a dedicated `kong.io.http.request_uri` span, and the `connect`, DNS, and internal `request` spans it triggers are nested beneath it.\nThe `kong.io.http.request` span now also reports the `http.url` attribute.", + "type": "bugfix", + "scope": "Core" + }, + { + "message": "**openid-connect**: Fixed an issue where switching clients within a single request (via `client_arg`) failed to use the matching client's JWK.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**openid-connect**: Fixed an issue where consumer groups could not be resolved\nwhen the groups claim was only available from the user info endpoint, which\nresulted in a `500` error.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**opentelemetry**: Fixed an issue where OpenTelemetry metrics only supported integer values by adding support for floating-point values in the OTLP exporter.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**PostgreSQL OAuth auth**: Refined the token cache key to be derived only from the fields that identify an OAuth configuration.", + "type": "bugfix", + "scope": "Core" + }, + { + "message": "**Prometheus**: A histogram's series (its buckets, `_count` and `_sum`) is now\nread as one atomic group during a scrape. Previously the exporter yielded and\nflushed output between individual samples, so an observation landing mid-scrape\ncould bump `_count`/`_sum` after the buckets were already read, producing an\ninternally inconsistent histogram in a single response.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**Prometheus**: Histogram series are now emitted grouped by label set (all\nbuckets in ascending `le` order, then `_count`, then `_sum`) so the output\nfollows the OpenMetrics interleaving rule. Previously samples were grouped by\nsuffix, which caused strict consumers such as Google Managed Prometheus to\ndrop points.", + "type": "bugfix", + "scope": "Plugin" + }, + { + "message": "**router**: Fixed an issue where WebSocket routes (`ws`/`wss`) returned HTTP 404 when using `router_flavor=expressions` due to incorrect protocol matching expressions being generated.", + "type": "bugfix", + "scope": "Core" + }, + { + "message": "**SAML**: Fixed an unreachable code path that prevented the plugin from falling\nback to validating the `samlp:Response`-level signature when the `saml:Assertion`\nwas not individually signed. With `validate_assertion_signature` enabled, an\nunsigned assertion inside a signed Response is now accepted (and a Response with\nno valid signature on either the assertion or the root is rejected), so IdPs that\nsign the Response envelope rather than the assertion no longer require signature\nvalidation to be turned off. Relates to the Response-level fallback described in\nCVE-2026-14917.", + "type": "bugfix", + "scope": "Plugin" + } + ] + } +} \ No newline at end of file diff --git a/app/_data/changelogs/config.yaml b/app/_changelogs/config.yaml similarity index 100% rename from app/_data/changelogs/config.yaml rename to app/_changelogs/config.yaml diff --git a/app/_data/changelogs/gateway.json b/app/_changelogs/gateway.json similarity index 100% rename from app/_data/changelogs/gateway.json rename to app/_changelogs/gateway.json diff --git a/app/_config/releases/ai-gateway/v1.yml b/app/_config/releases/ai-gateway/v1.yml new file mode 100644 index 00000000000..265f0c5c2c1 --- /dev/null +++ b/app/_config/releases/ai-gateway/v1.yml @@ -0,0 +1,385 @@ +# This file is autogenerated by a skill +# +# The purpose of this file is: +# - to keep track of all the pages that are part of the major version being cut +# - set the canonical_url for all the pages that were generated for the major version +# +# Notes: +# - Each key is the file path to a page that was generated and modified for the given major version +# - The platform sets the `canonical_url` on each of the pages listed automatically. +# - `status: pending` means that we still haven't written a corresponding page for the newest version. +# Once the newest version of a page is created, remove the `status: pending` and update the `canonical_url` +# accordingly so that it points to the newest's page url. +# The goal is for none of the items on this page to have `status: pending`. +# - canonical_url MUST be set to all pages, even if it's not the final one, i.e. we haven't written the +# newest version of that page. It can point to the product's landing page or similar until we +# have written the newest version of the page. We can always come back and edit the `canonical_url`. +app/_how-tos/ai-gateway/v1/authenticate-openai-sdk-clients-with-key-auth.md: + status: pending + canonical_url: /ai-gateway/ai-providers/openai/ +app/_how-tos/ai-gateway/v1/azure-batches.md: + status: pending + canonical_url: /ai-gateway/ai-providers/azure/ +app/_how-tos/ai-gateway/v1/compare-llm-models-accuracy.md: + status: pending + canonical_url: /ai-gateway/ai-providers/ +app/_how-tos/ai-gateway/v1/compress-llm-prompts.md: + status: pending + canonical_url: /ai-gateway/policies/ai-prompt-compressor/ +app/_how-tos/ai-gateway/v1/configure-hashicorp-vault-as-a-vault-for-llm-providers.md: + status: pending + canonical_url: /ai-gateway/entities/ai-vault/ +app/_how-tos/ai-gateway/v1/create-a-complex-ai-chat-history.md: + status: pending + canonical_url: /ai-gateway/ai-providers/ +app/_how-tos/ai-gateway/v1/filter-knowledge-based-queries-with-rag-injector.md: + status: pending + canonical_url: /ai-gateway/policies/ai-rag-injector/ +app/_how-tos/ai-gateway/v1/forward-openai-sdk-model-to-ai-proxy-advanced.md: + status: pending + canonical_url: /ai-gateway/ai-providers/openai/ +app/_how-tos/ai-gateway/v1/get-started-with-ai-gateway.md: + canonical_url: /ai-gateway/get-started/ +app/_how-tos/ai-gateway/v1/limit-a2a-body-size.md: + status: pending + canonical_url: /ai-gateway/a2a/ +app/_how-tos/ai-gateway/v1/mcp/aggregate-mcp-tools.md: + status: pending + canonical_url: /ai-gateway/mcp/ +app/_how-tos/ai-gateway/v1/mcp/enforce-acls-on-aggregated-mcp-servers.md: + status: pending + canonical_url: /ai-gateway/mcp/ +app/_how-tos/ai-gateway/v1/mcp/govern-mcp-traffic.md: + status: pending + canonical_url: /ai-gateway/mcp/ +app/_how-tos/ai-gateway/v1/mcp/map-API-to-mcp-tools.md: + canonical_url: /ai-gateway/get-started-with-mcp-server/ +app/_how-tos/ai-gateway/v1/mcp/map-weather-api-to-mcp-tools.md: + canonical_url: /ai-gateway/get-started-with-mcp-server/ +app/_how-tos/ai-gateway/v1/mcp/observe-autogenerated-mcp-tools-for-weather-api.md: + status: pending + canonical_url: /ai-gateway/mcp/ +app/_how-tos/ai-gateway/v1/mcp/observe-mcp-traffic-with-acls.md: + status: pending + canonical_url: /ai-gateway/mcp/ +app/_how-tos/ai-gateway/v1/mcp/observe-mcp-traffic.md: + status: pending + canonical_url: /ai-gateway/mcp/ +app/_how-tos/ai-gateway/v1/mcp/observe-traffic-for-mcp-tools.md: + status: pending + canonical_url: /ai-gateway/mcp/ +app/_how-tos/ai-gateway/v1/mcp/secure-mcp-tools-with-oauth2-and-okta.md: + status: pending + canonical_url: /ai-gateway/mcp/ +app/_how-tos/ai-gateway/v1/mcp/secure-mcp-traffic.md: + status: pending + canonical_url: /ai-gateway/mcp/ +app/_how-tos/ai-gateway/v1/mcp/use-access-controls-for-mcp-tools.md: + status: pending + canonical_url: /ai-gateway/mcp/ +app/_how-tos/ai-gateway/v1/meter-llm-traffic.md: + status: pending + canonical_url: /ai-gateway/policies/ai-rate-limiting-advanced/ +app/_how-tos/ai-gateway/v1/protect-sensitive-information-output-with-ai.md: + status: pending + canonical_url: /ai-gateway/policies/ai-sanitizer/ +app/_how-tos/ai-gateway/v1/protect-sensitive-information-with-ai.md: + status: pending + canonical_url: /ai-gateway/policies/ai-sanitizer/ +app/_how-tos/ai-gateway/v1/proxy-a2a-agents.md: + canonical_url: /ai-gateway/get-started-with-ai-agent/ +app/_how-tos/ai-gateway/v1/rate-limit-a2a-traffic.md: + canonical_url: /ai-gateway/rate-limit-a2a-traffic/ +app/_how-tos/ai-gateway/v1/rotate-secrets-in-google-cloud-secret.md: + status: pending + canonical_url: /ai-gateway/entities/ai-vault/ +app/_how-tos/ai-gateway/v1/route-azure-sdk-to-multiple-azure-deployments.md: + status: pending + canonical_url: /ai-gateway/ai-providers/azure/ +app/_how-tos/ai-gateway/v1/route-azure-sdk-to-specific-deployments.md: + status: pending + canonical_url: /ai-gateway/ai-providers/azure/ +app/_how-tos/ai-gateway/v1/route-requests-by-model-alias.md: + status: pending + canonical_url: /ai-gateway/entities/ai-model/ +app/_how-tos/ai-gateway/v1/secure-a2a-traffic.md: + status: pending + canonical_url: /ai-gateway/a2a/ +app/_how-tos/ai-gateway/v1/secure-a2a-with-oidc.md: + canonical_url: /ai-gateway/secure-ai-agent-with-oidc/ +app/_how-tos/ai-gateway/v1/send-asynchronous-llm-requests.md: + status: pending + canonical_url: /ai-gateway/ai-providers/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-anthropic.md: + status: pending + canonical_url: /ai-gateway/ai-providers/anthropic/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-aws-bedrock.md: + status: pending + canonical_url: /ai-gateway/ai-providers/bedrock/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-cerebras.md: + status: pending + canonical_url: /ai-gateway/ai-providers/cerebras/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-cohere.md: + status: pending + canonical_url: /ai-gateway/ai-providers/cohere/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-dashscope.md: + status: pending + canonical_url: /ai-gateway/ai-providers/dashscope/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-databricks.md: + status: pending + canonical_url: /ai-gateway/ai-providers/databricks/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-deepseek.md: + status: pending + canonical_url: /ai-gateway/ai-providers/deepseek/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-gemini.md: + status: pending + canonical_url: /ai-gateway/ai-providers/gemini/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-huggingface.md: + status: pending + canonical_url: /ai-gateway/ai-providers/huggingface/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-ollama-qwen.md: + status: pending + canonical_url: /ai-gateway/ai-providers/ollama/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-ollama.md: + status: pending + canonical_url: /ai-gateway/ai-providers/ollama/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-openai.md: + status: pending + canonical_url: /ai-gateway/ai-providers/openai/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-vertex-ai.md: + status: pending + canonical_url: /ai-gateway/ai-providers/vertex/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-for-image-generation-with-grok.md: + status: pending + canonical_url: /ai-gateway/ai-providers/xai/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-anthropic.md: + status: pending + canonical_url: /ai-gateway/ai-providers/anthropic/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-aws-bedrock.md: + status: pending + canonical_url: /ai-gateway/ai-providers/bedrock/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-cerebras.md: + status: pending + canonical_url: /ai-gateway/ai-providers/cerebras/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-cohere.md: + status: pending + canonical_url: /ai-gateway/ai-providers/cohere/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-dashscope.md: + status: pending + canonical_url: /ai-gateway/ai-providers/dashscope/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-databricks.md: + status: pending + canonical_url: /ai-gateway/ai-providers/databricks/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-deepseek.md: + status: pending + canonical_url: /ai-gateway/ai-providers/deepseek/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-gemini.md: + status: pending + canonical_url: /ai-gateway/ai-providers/gemini/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-huggingface.md: + status: pending + canonical_url: /ai-gateway/ai-providers/huggingface/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-ollama-qwen.md: + status: pending + canonical_url: /ai-gateway/ai-providers/ollama/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-ollama.md: + status: pending + canonical_url: /ai-gateway/ai-providers/ollama/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-openai.md: + status: pending + canonical_url: /ai-gateway/ai-providers/openai/ +app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-vertex-ai.md: + status: pending + canonical_url: /ai-gateway/ai-providers/vertex/ +app/_how-tos/ai-gateway/v1/set-up-jaeger-with-gen-ai-otel-for-tool-calls.md: + status: pending + canonical_url: /ai-gateway/llm-open-telemetry/ +app/_how-tos/ai-gateway/v1/set-up-jaeger-with-gen-ai-otel.md: + status: pending + canonical_url: /ai-gateway/llm-open-telemetry/ +app/_how-tos/ai-gateway/v1/store-a-mistral-api-key-as-a-secret-in-konnect-config-store.md: + status: pending + canonical_url: /ai-gateway/entities/ai-vault/ +app/_how-tos/ai-gateway/v1/strip-model-from-open-ai-sdk-requests.md.md: + status: pending + canonical_url: /ai-gateway/ai-providers/openai/ +app/_how-tos/ai-gateway/v1/transform-a-client-request-with-ai.md: + status: pending + canonical_url: /ai-gateway/policies/ai-request-transformer/ +app/_how-tos/ai-gateway/v1/transform-a-response-with-ai.md: + status: pending + canonical_url: /ai-gateway/policies/ai-response-transformer/ +app/_how-tos/ai-gateway/v1/use-agno-with-ai-proxy.md: + status: pending + canonical_url: /ai-gateway/ai-providers/ +app/_how-tos/ai-gateway/v1/use-ai-aws-guardrails-plugin.md: + canonical_url: /ai-gateway/how-to/use-ai-aws-guardrails-policy/ +app/_how-tos/ai-gateway/v1/use-ai-custom-guardrail-with-mistral-ai.md: + status: pending + canonical_url: /ai-gateway/policies/ai-custom-guardrail/ +app/_how-tos/ai-gateway/v1/use-ai-gcp-model-armor-plugin.md: + status: pending + canonical_url: /ai-gateway/policies/ai-gcp-model-armor/ +app/_how-tos/ai-gateway/v1/use-ai-lakera-guard-plugin.md: + status: pending + canonical_url: /ai-gateway/policies/ai-lakera-guard/ +app/_how-tos/ai-gateway/v1/use-ai-prompt-decorator-plugin.md: + status: pending + canonical_url: /ai-gateway/policies/ai-prompt-decorator/ +app/_how-tos/ai-gateway/v1/use-ai-prompt-guard-plugin.md: + canonical_url: /ai-gateway/use-ai-prompt-guard-policy/ +app/_how-tos/ai-gateway/v1/use-ai-prompt-template-plugin.md: + status: pending + canonical_url: /ai-gateway/policies/ai-prompt-template/ +app/_how-tos/ai-gateway/v1/use-ai-rag-injector-acls.md: + status: pending + canonical_url: /ai-gateway/policies/ai-rag-injector/ +app/_how-tos/ai-gateway/v1/use-ai-rag-injector-plugin.md: + status: pending + canonical_url: /ai-gateway/policies/ai-rag-injector/ +app/_how-tos/ai-gateway/v1/use-ai-semantic-prompt-guard-plugin.md: + status: pending + canonical_url: /ai-gateway/policies/ai-semantic-prompt-guard/ +app/_how-tos/ai-gateway/v1/use-ai-semantic-response-guard-plugin.md: + status: pending + canonical_url: /ai-gateway/policies/ai-semantic-response-guard/ +app/_how-tos/ai-gateway/v1/use-azure-ai-content-safety.md: + status: pending + canonical_url: /ai-gateway/policies/ai-azure-content-safety/ +app/_how-tos/ai-gateway/v1/use-bedrock-function-calling-with-streaming.md: + status: pending + canonical_url: /ai-gateway/ai-providers/bedrock/ +app/_how-tos/ai-gateway/v1/use-bedrock-function-calling.md: + status: pending + canonical_url: /ai-gateway/ai-providers/bedrock/ +app/_how-tos/ai-gateway/v1/use-bedrock-rerank-api.md: + status: pending + canonical_url: /ai-gateway/ai-providers/bedrock/ +app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-anthropic.md: + canonical_url: /ai-gateway/use-claude-code-with-ai-gateway-anthropic/ +app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-azure.md: + status: pending + canonical_url: /ai-gateway/ai-providers/azure/ +app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-bedrock.md: + status: pending + canonical_url: /ai-gateway/ai-providers/bedrock/ +app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-dashscope.md: + canonical_url: /ai-gateway/use-claude-code-with-ai-gateway-dashscope/ +app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-gemini.md: + canonical_url: /ai-gateway/use-claude-code-with-ai-gateway-gemini/ +app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-huggingface.md: + canonical_url: /ai-gateway/use-claude-code-with-ai-gateway-huggingface/ +app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-openai.md: + canonical_url: /ai-gateway/use-claude-code-with-ai-gateway-openai/ +app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-vertex.md: + canonical_url: /ai-gateway/use-claude-code-with-ai-gateway-vertex/ +app/_how-tos/ai-gateway/v1/use-codex-with-ai-gateway.md: + status: pending + canonical_url: /ai-gateway/ai-providers/openai/ +app/_how-tos/ai-gateway/v1/use-cohere-rerank-api.md: + status: pending + canonical_url: /ai-gateway/ai-providers/cohere/ +app/_how-tos/ai-gateway/v1/use-custom-function-for-ai-rate-limiting.md: + status: pending + canonical_url: /ai-gateway/policies/ai-rate-limiting-advanced/ +app/_how-tos/ai-gateway/v1/use-gemini-3-google-search.md: + status: pending + canonical_url: /ai-gateway/ai-providers/gemini/ +app/_how-tos/ai-gateway/v1/use-gemini-3-image-config.md: + status: pending + canonical_url: /ai-gateway/ai-providers/gemini/ +app/_how-tos/ai-gateway/v1/use-gemini-3-thinking-config.md: + status: pending + canonical_url: /ai-gateway/ai-providers/gemini/ +app/_how-tos/ai-gateway/v1/use-gemini-cli-with-ai-gateway.md: + status: pending + canonical_url: /ai-gateway/ai-clis/ +app/_how-tos/ai-gateway/v1/use-gemini-sdk-chat.md: + status: pending + canonical_url: /ai-gateway/ai-providers/gemini/ +app/_how-tos/ai-gateway/v1/use-langchain-with-ai-proxy.md: + status: pending + canonical_url: /ai-gateway/ai-providers/ +app/_how-tos/ai-gateway/v1/use-qwen-code-with-ai-gateway.md: + status: pending + canonical_url: /ai-gateway/ai-clis/ +app/_how-tos/ai-gateway/v1/use-semantic-load-balancing-with-dynamic-vault-authentication.md: + status: pending + canonical_url: /ai-gateway/load-balancing/ +app/_how-tos/ai-gateway/v1/use-semantic-load-balancing.md: + status: pending + canonical_url: /ai-gateway/load-balancing/ +app/_how-tos/ai-gateway/v1/use-vertex-sdk-chat.md: + status: pending + canonical_url: /ai-gateway/ai-providers/vertex/ +app/_how-tos/ai-gateway/v1/use-vertex-sdk-for-streaming.md: + status: pending + canonical_url: /ai-gateway/ai-providers/vertex/ +app/_how-tos/ai-gateway/v1/visualize-ai-gateway-metrics-with-kibana.md: + status: pending + canonical_url: /ai-gateway/monitor-ai-llm-metrics/ +app/_how-tos/ai-gateway/v1/visualize-llm-metrics-with-grafana.md: + status: pending + canonical_url: /ai-gateway/monitor-ai-llm-metrics/ +app/_landing_pages/ai-gateway/v1.yaml: + canonical_url: /ai-gateway/ +app/_landing_pages/ai-gateway/v1/a2a.yaml: + canonical_url: /ai-gateway/a2a/ +app/_landing_pages/ai-gateway/v1/ai-clis.yaml: + canonical_url: /ai-gateway/ai-clis/ +app/_landing_pages/ai-gateway/v1/ai-data-gov.yaml: + canonical_url: /ai-gateway/ai-data-gov/ +app/_landing_pages/ai-gateway/v1/ai-providers.yaml: + canonical_url: /ai-gateway/ai-providers/ +app/_landing_pages/ai-gateway/v1/mcp.yaml: + canonical_url: /ai-gateway/mcp/ +app/ai-gateway/v1/ai-audit-log-reference.md: + canonical_url: /ai-gateway/ai-audit-log-reference/ +app/ai-gateway/v1/ai-otel-metrics.md: + canonical_url: /ai-gateway/ai-otel-metrics/ +app/ai-gateway/v1/ai-providers/anthropic.md: + canonical_url: /ai-gateway/ai-providers/anthropic/ +app/ai-gateway/v1/ai-providers/azure.md: + canonical_url: /ai-gateway/ai-providers/azure/ +app/ai-gateway/v1/ai-providers/bedrock.md: + canonical_url: /ai-gateway/ai-providers/bedrock/ +app/ai-gateway/v1/ai-providers/cerebras.md: + canonical_url: /ai-gateway/ai-providers/cerebras/ +app/ai-gateway/v1/ai-providers/cohere.md: + canonical_url: /ai-gateway/ai-providers/cohere/ +app/ai-gateway/v1/ai-providers/dashscope.md: + canonical_url: /ai-gateway/ai-providers/dashscope/ +app/ai-gateway/v1/ai-providers/databricks.md: + canonical_url: /ai-gateway/ai-providers/databricks/ +app/ai-gateway/v1/ai-providers/deepseek.md: + canonical_url: /ai-gateway/ai-providers/deepseek/ +app/ai-gateway/v1/ai-providers/gemini.md: + canonical_url: /ai-gateway/ai-providers/gemini/ +app/ai-gateway/v1/ai-providers/huggingface.md: + canonical_url: /ai-gateway/ai-providers/huggingface/ +app/ai-gateway/v1/ai-providers/llama.md: + canonical_url: /ai-gateway/ai-providers/llama/ +app/ai-gateway/v1/ai-providers/mistral.md: + canonical_url: /ai-gateway/ai-providers/mistral/ +app/ai-gateway/v1/ai-providers/ollama.md: + canonical_url: /ai-gateway/ai-providers/ollama/ +app/ai-gateway/v1/ai-providers/openai.md: + canonical_url: /ai-gateway/ai-providers/openai/ +app/ai-gateway/v1/ai-providers/vertex.md: + canonical_url: /ai-gateway/ai-providers/vertex/ +app/ai-gateway/v1/ai-providers/vllm.md: + canonical_url: /ai-gateway/ai-providers/vllm/ +app/ai-gateway/v1/ai-providers/xai.md: + canonical_url: /ai-gateway/ai-providers/xai/ +app/ai-gateway/v1/llm-open-telemetry.md: + canonical_url: /ai-gateway/llm-open-telemetry/ +app/ai-gateway/v1/load-balancing.md: + canonical_url: /ai-gateway/load-balancing/ +app/ai-gateway/v1/monitor-ai-llm-metrics.md: + canonical_url: /ai-gateway/monitor-ai-llm-metrics/ +app/ai-gateway/v1/resource-sizing-guidelines-ai.md: + canonical_url: /ai-gateway/resource-sizing-guidelines-ai/ +app/ai-gateway/v1/semantic-similarity.md: + canonical_url: /ai-gateway/semantic-similarity/ +app/ai-gateway/v1/streaming.md: + canonical_url: /ai-gateway/streaming/ diff --git a/app/_data/ai-gateway/v2/otel-metrics.yaml b/app/_data/ai-gateway/v2/otel-metrics.yaml new file mode 100644 index 00000000000..9b8da8980f0 --- /dev/null +++ b/app/_data/ai-gateway/v2/otel-metrics.yaml @@ -0,0 +1,332 @@ +metrics: + - name: gen_ai.client.operation.duration + min_version: "" + description: Total time Kong spends processing a Gen AI operation, such as an LLM request. Requires `enable_request_metrics` to populate the `error.type` attribute. + unit: "s" + type: Histogram + attributes: + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.response.model + - gen_ai.operation.name + - kong.workspace.name + - kong.auth.consumer.name + - kong.gen_ai.request.mode + - error.type + - name: gen_ai.server.request.duration + min_version: "" + description: Time the LLM provider spends processing the request. Requires `enable_latency_metrics` set to `true`. Requires `enable_request_metrics` to populate the `error.type` attribute. + unit: "s" + type: Histogram + attributes: + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.response.model + - gen_ai.operation.name + - kong.workspace.name + - kong.auth.consumer.name + - kong.gen_ai.request.mode + - error.type + - name: gen_ai.client.token.usage + min_version: "" + description: Number of tokens consumed by the Gen AI operation. + unit: "{token}" + type: Sum + attributes: + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.response.model + - gen_ai.token.type + - gen_ai.operation.name + - kong.workspace.name + - kong.auth.consumer.name + - kong.gen_ai.request.mode + - name: gen_ai.server.time_to_first_token + min_version: "" + description: Time from when the model server receives the request until the first output token is generated. + unit: "s" + type: Histogram + attributes: + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.response.model + - gen_ai.operation.name + - kong.workspace.name + - kong.auth.consumer.name + - kong.gen_ai.request.mode + - name: gen_ai.server.time_per_output_token + min_version: "" + description: Time between successive output tokens generated by the model server after the first token. + unit: "s" + type: Histogram + attributes: + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.response.model + - gen_ai.operation.name + - kong.workspace.name + - kong.auth.consumer.name + - kong.gen_ai.request.mode + - name: kong.gen_ai.llm.cost + min_version: "" + description: Cost of AI requests. + unit: "{cost}" + type: Sum + attributes: + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.response.model + - gen_ai.operation.name + - kong.gen_ai.cache.status + - kong.gen_ai.vector_db + - kong.gen_ai.embeddings.provider + - kong.gen_ai.embeddings.model + - kong.workspace.name + - kong.auth.consumer.name + - kong.gen_ai.request.mode + - name: kong.gen_ai.cache.fetch.latency + min_version: "" + description: Time to fetch a response from the semantic cache. + unit: "s" + type: Histogram + attributes: + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.response.model + - gen_ai.operation.name + - kong.gen_ai.cache.status + - kong.gen_ai.vector_db + - kong.gen_ai.embeddings.provider + - kong.gen_ai.embeddings.model + - kong.workspace.name + - kong.auth.consumer.name + - kong.gen_ai.request.mode + - name: kong.gen_ai.cache.embeddings.latency + min_version: "" + description: Time to generate embeddings during cache operations. + unit: "s" + type: Histogram + attributes: + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.response.model + - gen_ai.operation.name + - kong.gen_ai.cache.status + - kong.gen_ai.vector_db + - kong.gen_ai.embeddings.provider + - kong.gen_ai.embeddings.model + - kong.workspace.name + - kong.auth.consumer.name + - kong.gen_ai.request.mode + - name: kong.gen_ai.rag.fetch.latency + min_version: "" + description: Time to fetch data from a RAG source. + unit: "s" + type: Histogram + attributes: + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.response.model + - gen_ai.operation.name + - kong.gen_ai.cache.status + - kong.gen_ai.vector_db + - kong.gen_ai.embeddings.provider + - kong.gen_ai.embeddings.model + - kong.workspace.name + - kong.auth.consumer.name + - kong.gen_ai.request.mode + - name: kong.gen_ai.rag.embeddings.latency + min_version: "" + description: Time to generate embeddings for RAG operations. + unit: "s" + type: Histogram + attributes: + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.response.model + - gen_ai.operation.name + - kong.gen_ai.cache.status + - kong.gen_ai.vector_db + - kong.gen_ai.embeddings.provider + - kong.gen_ai.embeddings.model + - kong.workspace.name + - kong.auth.consumer.name + - kong.gen_ai.request.mode + - name: kong.gen_ai.aws.guardrails.latency + min_version: "" + description: Time for AWS Guardrails to process a request. + unit: "s" + type: Histogram + attributes: + - kong.gen_ai.aws.guardrails.id + - kong.gen_ai.aws.guardrails.version + - kong.gen_ai.aws.guardrails.mode + - kong.gen_ai.aws.guardrails.region + - kong.workspace.name + - kong.auth.consumer.name + - name: kong.gen_ai.mcp.response.size + min_version: "" + description: Size of the MCP response body in bytes. + unit: "By" + type: Histogram + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - mcp.method.name + - gen_ai.tool.name + - name: kong.gen_ai.mcp.request.error.count + min_version: "" + description: Number of MCP request errors. + unit: "{error}" + type: Sum + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - mcp.method.name + - gen_ai.tool.name + - error.type + - name: mcp.client.operation.duration + min_version: "" + description: Duration of the MCP request as observed by the sender. Only available when the MCP entity is in passthrough-listener mode. Requires `enable_latency_metrics` set to `true`. + unit: "s" + type: Histogram + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - mcp.method.name + - gen_ai.tool.name + - error.type + - gen_ai.operation.name + - name: mcp.server.operation.duration + min_version: "" + description: Duration of the MCP request as observed by the receiver. + unit: "s" + type: Histogram + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - mcp.method.name + - gen_ai.tool.name + - error.type + - gen_ai.operation.name + - name: kong.gen_ai.mcp.acl.allowed + min_version: "" + description: Number of MCP requests allowed by ACL rules. + unit: "{request}" + type: Sum + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - kong.gen_ai.mcp.primitive + - kong.gen_ai.mcp.primitive_name + - name: kong.gen_ai.mcp.acl.denied + min_version: "" + description: Number of MCP requests denied by ACL rules. + unit: "{request}" + type: Sum + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - kong.gen_ai.mcp.primitive + - kong.gen_ai.mcp.primitive_name + - name: kong.gen_ai.a2a.request.count + min_version: "" + description: Total number of A2A requests. + unit: "{request}" + type: Sum + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - kong.gen_ai.a2a.method + - kong.gen_ai.a2a.binding + - name: kong.gen_ai.a2a.request.duration + min_version: "" + description: Duration of an A2A request in seconds. + unit: "s" + type: Histogram + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - kong.gen_ai.a2a.method + - kong.gen_ai.a2a.binding + - name: kong.gen_ai.a2a.response.size + min_version: "" + description: Size of the A2A response body in bytes. + unit: "By" + type: Histogram + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - kong.gen_ai.a2a.method + - kong.gen_ai.a2a.binding + - name: kong.gen_ai.a2a.ttfb + min_version: "" + description: Time to first byte for A2A streaming responses in seconds. + unit: "s" + type: Histogram + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - kong.gen_ai.a2a.method + - kong.gen_ai.a2a.binding + - name: kong.gen_ai.a2a.request.error.count + min_version: "" + description: Number of A2A request errors. + unit: "{error}" + type: Sum + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - kong.gen_ai.a2a.method + - kong.gen_ai.a2a.binding + - kong.gen_ai.a2a.error.type + - name: kong.gen_ai.a2a.task.state.count + min_version: "" + description: Number of A2A task state transitions. + unit: "{state}" + type: Sum + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - kong.gen_ai.a2a.task.state + +attributes: + kong.service.name: Name of the Gateway Service. + kong.route.name: Name of the Route. + kong.auth.consumer.name: Name of the authenticated Consumer. + kong.workspace.name: Name of the Workspace. + error.type: Type of error that occurred. + gen_ai.provider.name: Name of the Gen AI provider. + gen_ai.request.model: Model name targeted by the request. + gen_ai.response.model: Model name reported by the provider in the response. + gen_ai.operation.name: "Operation requested, such as `chat` or `embeddings`." + gen_ai.token.type: "Token category: `input`, `output`, or `total`." + kong.gen_ai.request.mode: "Request mode: `oneshot`, `stream`, or `realtime`." + kong.gen_ai.cache.status: "Cache status: `hit` or empty if not cached." + kong.gen_ai.vector_db: "Vector database used for caching, such as `redis`." + kong.gen_ai.embeddings.provider: Embeddings provider used for caching. + kong.gen_ai.embeddings.model: Embeddings model used for caching. + kong.gen_ai.aws.guardrails.id: ID of the AWS Guardrails configuration. + kong.gen_ai.aws.guardrails.version: Version of the AWS Guardrails configuration. + kong.gen_ai.aws.guardrails.mode: Mode of the AWS Guardrails evaluation. + kong.gen_ai.aws.guardrails.region: AWS region of the Guardrails service. + mcp.method.name: "MCP method name, such as `tools/call`." + gen_ai.tool.name: Name of the MCP tool invoked. + kong.gen_ai.mcp.primitive: "MCP primitive type, such as `tool`." + kong.gen_ai.mcp.primitive_name: Name of the MCP primitive. + kong.gen_ai.a2a.method: A2A method name. + kong.gen_ai.a2a.binding: A2A binding type. + kong.gen_ai.a2a.error.type: Type of the A2A error. + kong.gen_ai.a2a.task.state: "Task state, such as `completed`, `failed`, or `in_progress`." diff --git a/app/_data/ai-gateway/v2/otel-span-attributes.yaml b/app/_data/ai-gateway/v2/otel-span-attributes.yaml new file mode 100644 index 00000000000..a4c73882637 --- /dev/null +++ b/app/_data/ai-gateway/v2/otel-span-attributes.yaml @@ -0,0 +1,75 @@ +spans: + - name: kong.gen_ai + title: Gen AI span attributes + min_version: "" + description: Gen AI tracing span emitted for LLM requests. + attributes: + - gen_ai.operation.name + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.request.max_tokens + - gen_ai.request.temperature + - gen_ai.input.messages + - gen_ai.output.type + - gen_ai.output.messages + - gen_ai.response.id + - gen_ai.response.model + - gen_ai.response.finish_reasons + - gen_ai.usage.input_tokens + - gen_ai.usage.output_tokens + - name: kong.gen_ai + title: Gen AI tool call span attributes + min_version: "" + description: Gen AI tracing span emitted when the provider response includes a tool call. + attributes: + - gen_ai.operation.name + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.request.max_tokens + - gen_ai.request.temperature + - gen_ai.response.finish_reasons + - gen_ai.response.id + - gen_ai.response.model + - gen_ai.tool.call.id + - gen_ai.tool.name + - gen_ai.tool.type + - gen_ai.usage.input_tokens + - gen_ai.usage.output_tokens + - gen_ai.output.type + - name: kong.a2a + title: A2A span attributes + min_version: "" + description: A2A tracing span emitted for agent-to-agent requests. + attributes: + - kong.a2a.protocol.version + - rpc.system + - rpc.method + - kong.a2a.task.id + - kong.a2a.task.state + - kong.a2a.context.id + - kong.a2a.operation + +attributes: + gen_ai.operation.name: "Operation requested, such as `chat` or `embeddings`." + gen_ai.provider.name: Name of the Gen AI provider. + gen_ai.request.model: Model name targeted by the request. + gen_ai.request.max_tokens: Maximum token limit configured for the request. + gen_ai.request.temperature: Sampling temperature configured for the request. + gen_ai.input.messages: Array of input messages sent to the model. + gen_ai.output.type: Output payload type, such as `json`. + gen_ai.output.messages: Array containing the full model response payload. + gen_ai.response.id: Unique identifier returned by the provider for the response. + gen_ai.response.model: Model name reported by the provider in the response. + gen_ai.response.finish_reasons: Array of finish reasons returned by the provider. + gen_ai.usage.input_tokens: Number of input tokens consumed by the request. + gen_ai.usage.output_tokens: Number of output tokens generated in the response. + gen_ai.tool.call.id: Unique identifier for the specific tool call. + gen_ai.tool.name: Name of the tool or function requested by the model. + gen_ai.tool.type: Tool type, such as `function`. + kong.a2a.protocol.version: A2A protocol version used for the request. + rpc.system: RPC protocol used by the request, such as `jsonrpc`. + rpc.method: RPC method invoked by the client. + kong.a2a.task.id: Identifier of the A2A task. + kong.a2a.task.state: Current state of the A2A task. + kong.a2a.context.id: Identifier of the A2A conversation context. + kong.a2a.operation: A2A operation name, such as `message/send`. diff --git a/app/_data/ai-gateway/v2/providers.yaml b/app/_data/ai-gateway/v2/providers.yaml new file mode 100644 index 00000000000..7945a3e4278 --- /dev/null +++ b/app/_data/ai-gateway/v2/providers.yaml @@ -0,0 +1,1563 @@ +providers: + - name: Amazon Bedrock + url_patterns: + - 'https://bedrock-runtime.{region}.amazonaws.com' + min_version: '2.0' + capabilities: + generate: + supported: true + streaming: true + upstream_path: 'Uses the `Converse` and `ConverseStream` API' + model_example: '[Use the model name for the specific LLM provider](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html)' + min_version: '2.0' + completions: + supported: true + streaming: true + model_example: '[Use the model name for the specific LLM provider](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html)' + min_version: '2.0' + embeddings: + supported: true + streaming: false + upstream_path: 'Uses the `InvokeModel` and `InvokeWithResponseStream` API' + model_example: '[Use the model name for the specific LLM provider](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html)' + min_version: '2.0' + batches: + supported: true + streaming: false + upstream_path: 'Uses the `ModelInvocationJob` API' + model_example: 'n/a' + min_version: '2.0' + note: + content: 'Batches processing for Bedrock is supported in the native format from SDK only' + files: + supported: true + streaming: false + upstream_path: '`/openai/files`' + model_example: 'n/a' + min_version: '2.0' + note: + content: 'Amazon Bedrock does not have a dedicated files API. File storage uses Google Cloud Storage, similar to AWS S3.' + image: + supported: true + streaming: false + upstream_path: 'Uses the `InvokeModel` API' + model_example: '[Use the model name for the specific LLM provider](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html)' + min_version: '2.0' + video: + supported: true + streaming: false + upstream_path: 'Uses the `StartAsyncInvoke` API' + model_example: '[Use the model name for the specific LLM provider](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html)' + min_version: '2.0' + agentic: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_speech: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_transcription: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_translation: + supported: false + streaming: false + model_example: '' + min_version: '' + realtime: + supported: false + streaming: false + model_example: '' + min_version: '' + rerank: + supported: false + streaming: false + model_example: '' + min_version: '' + native_formats: + - llm_format: 'bedrock' + supported_apis: + - '/model/{model_name}/converse' + - '/model/{model_name}/converse-stream' + - '/model/{model_name}/invoke' + - '/model/{model_name}/invoke-with-response-stream' + - '/model/{model_name}/retrieveAndGenerate' + - '/model/{model_name}/retrieveAndGenerateStream' + - '/model/{model_name}/rerank' + - '/model/{model_name}/async-invoke' + - '/model-invocations' + limitations: + provider_specific: [] + statistics_logging: + - 'Statistics logging is not available for image generation or editing APIs for Amazon Bedrock' + + - name: Anthropic + url_patterns: + - 'https://api.anthropic.com:443/{capability_path}' + min_version: '2.0' + capabilities: + generate: + supported: true + streaming: true + upstream_path: '`/v1/messages`' + model_example: 'claude-sonnet-4-20250514' + min_version: '2.0' + completions: + supported: true + streaming: false + upstream_path: '`/v1/complete`' + model_example: 'claude-sonnet-4-20250514' + min_version: '2.0' + batches: + supported: true + streaming: true + upstream_path: '`/v1/messages/batches`' + model_example: 'n/a' + min_version: '2.0' + note: + content: 'Batches processing for Anthropic is supported in the native format from SDK only' + embeddings: + supported: false + streaming: false + model_example: '' + min_version: '' + files: + supported: false + streaming: false + model_example: '' + min_version: '' + agentic: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_speech: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_transcription: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_translation: + supported: false + streaming: false + model_example: '' + min_version: '' + image: + supported: false + streaming: false + model_example: '' + min_version: '' + video: + supported: false + streaming: false + model_example: '' + min_version: '' + realtime: + supported: false + streaming: false + model_example: '' + min_version: '' + rerank: + supported: false + streaming: false + model_example: '' + min_version: '' + native_formats: + - llm_format: 'anthropic' + supported_apis: + - '/v1/messages' + - '/v1/messages/batches' + limitations: + provider_specific: + - 'Does not support embeddings' + statistics_logging: + - 'No statistics logging for completions' + + - name: Azure OpenAI + url_patterns: + - 'https://{azure_instance}.openai.azure.com:443/openai/deployments/{deployment_name}/{capability_path}' + min_version: '2.0' + capabilities: + generate: + supported: true + streaming: true + upstream_path: '`/openai/deployments/{deployment_name}/chat/completions`' + model_example: 'gpt-4o' + min_version: '2.0' + completions: + supported: true + streaming: true + upstream_path: '`/openai/deployments/{deployment_name}/completions`' + model_example: 'gpt-4o-mini' + min_version: '2.0' + embeddings: + supported: true + streaming: false + upstream_path: '`/openai/deployments/{deployment_name}/embeddings`' + model_example: 'text-embedding-3-small' + min_version: '2.0' + note: + content: 'Use `text-embedding-3-small` or `text-embedding-3-large` for dynamic dimensions.' + files: + supported: true + streaming: false + upstream_path: '`/openai/files`' + model_example: 'n/a' + min_version: '2.0' + batches: + supported: true + streaming: false + upstream_path: '`/openai/batches`' + model_example: 'n/a' + min_version: '2.0' + agentic: + supported: true + streaming: false + upstream_path: '`/openai/assistants` and `/openai/v1/responses`' + model_example: 'n/a' + min_version: '2.0' + note: + content: 'Assistants API requires header `OpenAI-Beta: assistants=v2`. Responses API requires `config.azure_api_version` set to `"preview"`.' + audio_speech: + supported: true + streaming: false + upstream_path: '`/openai/audio/speech`' + model_example: 'n/a' + min_version: '2.0' + audio_transcription: + supported: true + streaming: false + upstream_path: '`/openai/audio/transcriptions`' + model_example: 'n/a' + min_version: '2.0' + audio_translation: + supported: true + streaming: false + upstream_path: '`/openai/audio/translations`' + model_example: 'n/a' + min_version: '2.0' + image: + supported: true + streaming: false + upstream_path: '`/openai/images/generations` and `/openai/images/edits`' + model_example: 'n/a' + min_version: '2.0' + video: + supported: true + streaming: false + upstream_path: '`/openai/v1/video/generations/jobs`' + model_example: 'sora-2' + min_version: '2.0' + realtime: + supported: true + streaming: true + upstream_path: '`/openai/realtime`' + model_example: 'n/a' + min_version: '2.0' + note: + content: 'For requests to Azure OpenAI realtime API, include the header `OpenAI-Beta: realtime=v1`.' + rerank: + supported: false + streaming: false + model_example: '' + min_version: '' + limitations: + provider_specific: [] + statistics_logging: + - 'No statistics logging for assistants, batch, or audio APIs' + + - name: Cerebras + url_patterns: + - 'https://api.cerebras.ai/{capability_path}' + min_version: '2.0' + capabilities: + generate: + supported: true + streaming: true + upstream_path: '`/v1/chat/completions`' + model_example: 'llama-3.3-70b' + min_version: '2.0' + completions: + supported: false + streaming: false + model_example: '' + min_version: '' + embeddings: + supported: false + streaming: false + model_example: '' + min_version: '' + files: + supported: false + streaming: false + model_example: '' + min_version: '' + batches: + supported: false + streaming: false + model_example: '' + min_version: '' + agentic: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_speech: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_transcription: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_translation: + supported: false + streaming: false + model_example: '' + min_version: '' + image: + supported: false + streaming: false + model_example: '' + min_version: '' + video: + supported: false + streaming: false + model_example: '' + min_version: '' + realtime: + supported: false + streaming: false + model_example: '' + min_version: '' + rerank: + supported: false + streaming: false + model_example: '' + min_version: '' + limitations: + provider_specific: [] + statistics_logging: [] + + - name: Cohere + url_patterns: + - 'https://api.cohere.com:443/{capability_path}' + min_version: '2.0' + capabilities: + generate: + supported: true + streaming: true + upstream_path: '`/v1/chat`' + model_example: 'command-a-03-2025' + min_version: '2.0' + completions: + supported: true + streaming: true + upstream_path: '`/v1/generate`' + model_example: 'command-r-plus-08-2024' + min_version: '2.0' + embeddings: + supported: true + streaming: false + upstream_path: '`/v2/embed`' + model_example: 'embed-english-v3.0' + min_version: '2.0' + rerank: + supported: true + streaming: false + upstream_path: '`/v1/rerank` or `/v2/rerank`' + model_example: 'n/a' + min_version: '2.0' + files: + supported: false + streaming: false + model_example: '' + min_version: '' + batches: + supported: false + streaming: false + model_example: '' + min_version: '' + agentic: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_speech: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_transcription: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_translation: + supported: false + streaming: false + model_example: '' + min_version: '' + image: + supported: false + streaming: false + model_example: '' + min_version: '' + video: + supported: false + streaming: false + model_example: '' + min_version: '' + realtime: + supported: false + streaming: false + model_example: '' + min_version: '' + native_formats: + - llm_format: 'cohere' + supported_apis: + - '/v1/rerank' + - '/v2/rerank' + limitations: + provider_specific: [] + statistics_logging: [] + + - name: Dashscope + url_patterns: + - 'https://dashscope.aliyuncs.com' + - 'https://dashscope-intl.aliyuncs.com' + min_version: '2.0' + capabilities: + generate: + supported: true + streaming: true + upstream_path: '`/compatible-mode/v1/chat/completions`' + model_example: 'qwen-plus' + min_version: '2.0' + embeddings: + supported: true + streaming: false + upstream_path: '`/compatible-mode/v1/embeddings`' + model_example: 'text-embedding-v1' + min_version: '2.0' + image: + supported: true + streaming: false + upstream_path: '`/api/v1/services/aigc/multimodal-generation/generation` and `/api/v1/services/aigc/image2image/image-synthesis`' + model_example: 'qwen-image-plus' + min_version: '2.0' + completions: + supported: false + streaming: false + model_example: '' + min_version: '' + files: + supported: false + streaming: false + model_example: '' + min_version: '' + batches: + supported: false + streaming: false + model_example: '' + min_version: '' + agentic: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_speech: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_transcription: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_translation: + supported: false + streaming: false + model_example: '' + min_version: '' + video: + supported: false + streaming: false + model_example: '' + min_version: '' + realtime: + supported: false + streaming: false + model_example: '' + min_version: '' + rerank: + supported: false + streaming: false + model_example: '' + min_version: '' + limitations: + provider_specific: [] + statistics_logging: [] + + - name: Gemini + url_patterns: + - 'https://generativelanguage.googleapis.com' + min_version: '2.0' + capabilities: + generate: + supported: true + streaming: true + upstream_path: 'Uses `generateContent` API' + model_example: 'gemini-2.5-flash' + min_version: '2.0' + embeddings: + supported: true + streaming: false + upstream_path: 'Uses `batchEmbedContents` API' + model_example: 'text-embedding-004' + min_version: '2.0' + files: + supported: true + streaming: false + upstream_path: 'Uses `uploadFile` and `files` API' + model_example: 'n/a' + min_version: '2.0' + note: + content: 'Files processing for Gemini is supported in the native format from SDK only' + batches: + supported: true + streaming: false + upstream_path: 'Uses `batches` API' + model_example: 'n/a' + min_version: '2.0' + note: + content: 'Batches processing for Gemini is supported in the native format from SDK only' + image: + supported: true + streaming: false + upstream_path: 'Uses `generateContent` API' + model_example: 'gemini-2.5-flash-preview-image-generation' + min_version: '2.0' + realtime: + supported: true + streaming: true + upstream_path: 'Uses `BidiGenerateContent` API' + model_example: 'gemini-2.5-flash-preview-native-audio' + min_version: '2.0' + note: + content: 'Realtime processing for Gemini is supported in the native format from SDK only' + video: + supported: true + streaming: false + upstream_path: 'Uses `predictLongRunning` API' + model_example: 'veo-3.1-generate-001' + min_version: '2.0' + completions: + supported: false + streaming: false + model_example: '' + min_version: '' + agentic: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_speech: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_transcription: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_translation: + supported: false + streaming: false + model_example: '' + min_version: '' + rerank: + supported: false + streaming: false + model_example: '' + min_version: '' + native_formats: + - llm_format: 'gemini' + supported_apis: + - '/v1beta/models/{model_name}:generateContent' + - '/v1beta/models/{model_name}:streamGenerateContent' + - '/v1beta/models/{model_name}:embedContent' + - '/v1beta/models/{model_name}:batchEmbedContents' + - '/v1beta/batches' + - '/upload/v1beta/files' + - '/v1beta/files' + limitations: + provider_specific: + - 'Gemini only supports `auth.allow_override = false`.' + statistics_logging: [] + + - name: Gemini Vertex + url_patterns: + - 'https://aiplatform.googleapis.com/' + min_version: '2.0' + capabilities: + generate: + supported: true + streaming: true + upstream_path: 'Uses `generateContent` API' + model_example: 'gemini-2.5-flash' + min_version: '2.0' + completions: + supported: true + streaming: false + upstream_path: 'Uses `generateContent` API' + model_example: 'gemini-2.5-flash' + min_version: '2.0' + embeddings: + supported: true + streaming: false + upstream_path: 'Uses `predict` API' + model_example: 'text-embedding-004' + min_version: '2.0' + files: + supported: true + streaming: false + upstream_path: '`/openai/files`' + model_example: 'n/a' + min_version: '2.0' + note: + content: 'Gemini Vertex does not have a dedicated Files API. File storage uses Google Cloud Storage, similar to AWS S3.' + batches: + supported: true + streaming: false + upstream_path: 'Uses `batchPredictionJobs` API' + model_example: 'n/a' + min_version: '2.0' + image: + supported: true + streaming: false + upstream_path: 'Uses `generateContent` API' + model_example: 'gemini-2.5-flash-preview-image-generation' + min_version: '2.0' + video: + supported: true + streaming: false + upstream_path: 'Uses `predictLongRunning` API' + model_example: 'veo-3.1-generate-001' + min_version: '2.0' + agentic: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_speech: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_transcription: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_translation: + supported: false + streaming: false + model_example: '' + min_version: '' + realtime: + supported: false + streaming: false + model_example: '' + min_version: '' + rerank: + supported: false + streaming: false + model_example: '' + min_version: '' + native_formats: + - llm_format: 'gemini' + supported_apis: + - '/v1/projects/{project_id}/locations/{location}/publishers/{publisher}/models/{model_name}:generateContent' + - '/v1/projects/{project_id}/locations/{location}/publishers/{publisher}/models/{model_name}:streamGenerateContent' + - '/v1/projects/{project_id}/locations/{location}/publishers/{publisher}/models/{model_name}:embedContent' + - '/v1/projects/{project_id}/locations/{location}/publishers/{publisher}/models/{model_name}:predict' + - '/v1/projects/{project_id}/locations/{location}/publishers/{publisher}/models/{model_name}:predictLongRunning' + - '/v1/projects/{project_id}/locations/{location}/rankingConfigs/{config_name}:rank' + - '/v1/projects/{project_id}/locations/{location}/batchPredictionJobs' + limitations: + provider_specific: [] + statistics_logging: [] + + - name: Hugging Face + url_patterns: + - 'https://api-inference.huggingface.co' + min_version: '2.0' + capabilities: + generate: + supported: true + streaming: true + upstream_path: '`/v1/chat/completions`' + model_example: '[Use the model name for the specific LLM provider](https://huggingface.co/models?inference=warm&pipeline_tag=text-generation&sort=trending)' + min_version: '2.0' + embeddings: + supported: true + streaming: false + upstream_path: '`/hf-inference/models/{model_name}/pipeline/feature-extraction`' + model_example: '[Use the embedding model name](https://huggingface.co/models?pipeline_tag=feature-extraction)' + min_version: '2.0' + audio_transcription: + supported: true + streaming: false + upstream_path: '`/v1/audio/transcriptions`' + model_example: '[Use the transcription model name](https://huggingface.co/models?pipeline_tag=automatic-speech-recognition)' + min_version: '2.0' + image: + supported: true + streaming: false + upstream_path: '`/v1/images/generations`' + model_example: '[Use the image generation model name](https://huggingface.co/models?pipeline_tag=image-generation)' + min_version: '2.0' + video: + supported: true + streaming: false + upstream_path: '`/v1/videos`' + model_example: '[Use the video generation model name](https://huggingface.co/models?pipeline_tag=video-generation)' + min_version: '2.0' + completions: + supported: false + streaming: false + model_example: '' + min_version: '' + files: + supported: false + streaming: false + model_example: '' + min_version: '' + batches: + supported: false + streaming: false + model_example: '' + min_version: '' + agentic: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_speech: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_translation: + supported: false + streaming: false + model_example: '' + min_version: '' + realtime: + supported: false + streaming: false + model_example: '' + min_version: '' + rerank: + supported: false + streaming: false + model_example: '' + min_version: '' + native_formats: + - llm_format: 'huggingface' + supported_apis: + - '/generate' + - '/generate_stream' + limitations: + provider_specific: [] + statistics_logging: [] + + - name: Llama2 + url_patterns: + - '$UPSTREAM_URL' + url_is_variable: true + min_version: '2.0' + formats: 'supports Llama2 and Llama3 models and raw, OLLAMA, and OpenAI formats' + capabilities: + generate: + supported: true + streaming: true + upstream_path: '`User-defined`' + model_example: 'User-defined' + min_version: '2.0' + completions: + supported: true + streaming: true + upstream_path: '`User-defined`' + model_example: 'User-defined' + min_version: '2.0' + embeddings: + supported: true + streaming: false + upstream_path: '`User-defined`' + model_example: 'User-defined' + min_version: '2.0' + files: + supported: false + streaming: false + model_example: '' + min_version: '' + batches: + supported: false + streaming: false + model_example: '' + min_version: '' + agentic: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_speech: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_transcription: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_translation: + supported: false + streaming: false + model_example: '' + min_version: '' + image: + supported: false + streaming: false + model_example: '' + min_version: '' + video: + supported: false + streaming: false + model_example: '' + min_version: '' + realtime: + supported: false + streaming: false + model_example: '' + min_version: '' + rerank: + supported: false + streaming: false + model_example: '' + min_version: '' + limitations: + provider_specific: + - 'Raw format lacks support for embeddings' + statistics_logging: [] + + - name: Mistral + url_patterns: + - '$UPSTREAM_URL' + url_is_variable: true + min_version: '2.0' + formats: 'mistral.ai, OpenAI, raw, and OLLAMA formats' + capabilities: + generate: + supported: true + streaming: true + upstream_path: '`/v1/chat/completions or user-defined`' + model_example: 'mistral-large-latest' + min_version: '2.0' + embeddings: + supported: true + streaming: false + upstream_path: '`/v1/embeddings or user-defined`' + model_example: 'mistral-embed' + min_version: '2.0' + completions: + supported: false + streaming: false + model_example: '' + min_version: '' + files: + supported: false + streaming: false + model_example: '' + min_version: '' + batches: + supported: false + streaming: false + model_example: '' + min_version: '' + agentic: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_speech: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_transcription: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_translation: + supported: false + streaming: false + model_example: '' + min_version: '' + image: + supported: false + streaming: false + model_example: '' + min_version: '' + video: + supported: false + streaming: false + model_example: '' + min_version: '' + realtime: + supported: false + streaming: false + model_example: '' + min_version: '' + rerank: + supported: false + streaming: false + model_example: '' + min_version: '' + limitations: + provider_specific: [] + statistics_logging: [] + + - name: Ollama + url_patterns: + - '$UPSTREAM_URL' + min_version: '2.0' + formats: 'Ollama, OpenAI, and Anthropic' + capabilities: + generate: + supported: true + streaming: true + upstream_path: '/api/chat' + model_example: 'llama3.2:1b' + min_version: '2.0' + embeddings: + supported: true + streaming: false + upstream_path: '/api/embed' + model_example: 'qwen3-embedding:8b' + min_version: '2.0' + completions: + supported: false + streaming: false + model_example: '' + min_version: '' + files: + supported: false + streaming: false + model_example: '' + min_version: '' + batches: + supported: false + streaming: false + model_example: '' + min_version: '' + agentic: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_speech: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_transcription: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_translation: + supported: false + streaming: false + model_example: '' + min_version: '' + image: + supported: false + streaming: false + model_example: '' + min_version: '' + video: + supported: false + streaming: false + model_example: '' + min_version: '' + realtime: + supported: false + streaming: false + model_example: '' + min_version: '' + rerank: + supported: false + streaming: false + model_example: '' + min_version: '' + + - name: OpenAI + url_patterns: + - 'https://api.openai.com:443/{capability_path}' + min_version: '2.0' + formats: 'GPT-4o, GPT-4.1, and Multi-Modal' + capabilities: + generate: + supported: true + streaming: true + upstream_path: '`/v1/chat/completions`' + model_example: 'gpt-4o' + min_version: '2.0' + completions: + supported: true + streaming: true + upstream_path: '`/v1/completions`' + model_example: 'gpt-4o-mini' + min_version: '2.0' + embeddings: + supported: true + streaming: false + upstream_path: '`/v1/embeddings`' + model_example: 'text-embedding-3-small' + min_version: '2.0' + note: + content: 'Use `text-embedding-3-small` or `text-embedding-3-large` for dynamic dimensions.' + files: + supported: true + streaming: false + upstream_path: '`/v1/files`' + model_example: 'n/a' + min_version: '2.0' + batches: + supported: true + streaming: false + upstream_path: '`/v1/batches`' + model_example: 'n/a' + min_version: '2.0' + agentic: + supported: true + streaming: false + upstream_path: '`/v1/assistants` and `/v1/responses`' + model_example: 'gpt-4o' + min_version: '2.0' + note: + content: 'Requires header `OpenAI-Beta: assistants=v2`' + audio_speech: + supported: true + streaming: false + upstream_path: '`/v1/audio/speech`' + model_example: 'tts-1' + min_version: '2.0' + audio_transcription: + supported: true + streaming: false + upstream_path: '`/v1/audio/transcriptions`' + model_example: 'whisper-1' + min_version: '2.0' + audio_translation: + supported: true + streaming: false + upstream_path: '`/v1/audio/translations`' + model_example: 'whisper-1' + min_version: '2.0' + image: + supported: true + streaming: false + upstream_path: '`/v1/images/generations` and `/v1/images/edits`' + model_example: 'gpt-image-1.5' + min_version: '2.0' + realtime: + supported: true + streaming: true + upstream_path: '`/v1/realtime`' + model_example: 'gpt-4o-realtime-preview' + min_version: '2.0' + note: + content: 'For requests to OpenAI realtime API, include the header `OpenAI-Beta: realtime=v1`.' + video: + supported: true + streaming: false + upstream_path: '`/v1/videos`' + model_example: 'sora-2' + min_version: '2.0' + rerank: + supported: false + streaming: false + model_example: '' + min_version: '' + limitations: + provider_specific: [] + statistics_logging: + - 'No statistics logging for assistants, batch, or audio APIs' + + - name: vLLM + url_patterns: + - '$UPSTREAM_URL' + min_version: '2.0' + capabilities: + generate: + supported: true + streaming: true + upstream_path: '`/v1/chat/completions`' + model_example: 'vllm-llama-3-8b' + min_version: '2.0' + completions: + supported: false + streaming: false + model_example: '' + min_version: '' + embeddings: + supported: false + streaming: false + model_example: '' + min_version: '' + files: + supported: false + streaming: false + model_example: '' + min_version: '' + batches: + supported: false + streaming: false + model_example: '' + min_version: '' + agentic: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_speech: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_transcription: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_translation: + supported: false + streaming: false + model_example: '' + min_version: '' + image: + supported: false + streaming: false + model_example: '' + min_version: '' + video: + supported: false + streaming: false + model_example: '' + min_version: '' + realtime: + supported: false + streaming: false + model_example: '' + min_version: '' + rerank: + supported: false + streaming: false + model_example: '' + min_version: '' + limitations: + provider_specific: + - '`upstream_url` must be set in `model.options` — vLLM has no fixed API endpoint' + statistics_logging: [] + + - name: xAI + url_patterns: + - 'https://api.x.ai:443/{capability_path}' + min_version: '2.0' + capabilities: + generate: + supported: true + streaming: false + upstream_path: '`/v1/chat/completions`' + model_example: 'grok-3' + min_version: '2.0' + image: + supported: true + streaming: false + upstream_path: '`/v1/images/generations`' + model_example: 'grok-2-image' + min_version: '2.0' + agentic: + supported: true + streaming: false + upstream_path: '`/v1/responses`' + model_example: 'grok-3' + min_version: '2.0' + completions: + supported: false + streaming: false + model_example: '' + min_version: '' + embeddings: + supported: false + streaming: false + model_example: '' + min_version: '' + files: + supported: false + streaming: false + model_example: '' + min_version: '' + batches: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_speech: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_transcription: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_translation: + supported: false + streaming: false + model_example: '' + min_version: '' + video: + supported: false + streaming: false + model_example: '' + min_version: '' + realtime: + supported: false + streaming: false + model_example: '' + min_version: '' + rerank: + supported: false + streaming: false + model_example: '' + min_version: '' + limitations: + provider_specific: [] + statistics_logging: [] + + - name: Databricks + url_patterns: + - 'https://{databricks_instance}.cloud.databricks.com:443' + min_version: '2.0' + capabilities: + generate: + supported: true + streaming: true + upstream_path: '/serving-endpoints/v1/chat/completions' + model_example: 'databricks-gpt-oss-20b' + min_version: '2.0' + completions: + supported: false + streaming: false + model_example: '' + min_version: '' + embeddings: + supported: false + streaming: false + model_example: '' + min_version: '' + files: + supported: false + streaming: false + model_example: '' + min_version: '' + batches: + supported: false + streaming: false + model_example: '' + min_version: '' + agentic: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_speech: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_transcription: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_translation: + supported: false + streaming: false + model_example: '' + min_version: '' + image: + supported: false + streaming: false + model_example: '' + min_version: '' + video: + supported: false + streaming: false + model_example: '' + min_version: '' + realtime: + supported: false + streaming: false + model_example: '' + min_version: '' + rerank: + supported: false + streaming: false + model_example: '' + min_version: '' + + - name: Kimi + url_patterns: + - 'https://api.moonshot.ai' + min_version: '2.0' + capabilities: + generate: + supported: true + streaming: true + upstream_path: '`/v1/chat/completions`' + model_example: 'kimi-k2.6' + min_version: '2.0' + completions: + supported: false + streaming: false + model_example: '' + min_version: '' + embeddings: + supported: false + streaming: false + model_example: '' + min_version: '' + files: + supported: false + streaming: false + model_example: '' + min_version: '' + batches: + supported: false + streaming: false + model_example: '' + min_version: '' + agentic: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_speech: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_transcription: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_translation: + supported: false + streaming: false + model_example: '' + min_version: '' + image: + supported: false + streaming: false + model_example: '' + min_version: '' + video: + supported: false + streaming: false + model_example: '' + min_version: '' + realtime: + supported: false + streaming: false + model_example: '' + min_version: '' + rerank: + supported: false + streaming: false + model_example: '' + min_version: '' + limitations: + provider_specific: [] + statistics_logging: [] + + - name: Vercel + url_patterns: + - 'https://ai-gateway.vercel.sh' + min_version: '2.0' + capabilities: + generate: + supported: true + streaming: true + upstream_path: '`/v1/chat/completions`' + model_example: 'openai/gpt-5.5' + min_version: '2.0' + completions: + supported: false + streaming: false + model_example: '' + min_version: '' + embeddings: + supported: false + streaming: false + model_example: '' + min_version: '' + files: + supported: false + streaming: false + model_example: '' + min_version: '' + batches: + supported: false + streaming: false + model_example: '' + min_version: '' + agentic: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_speech: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_transcription: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_translation: + supported: false + streaming: false + model_example: '' + min_version: '' + image: + supported: false + streaming: false + model_example: '' + min_version: '' + video: + supported: false + streaming: false + model_example: '' + min_version: '' + realtime: + supported: false + streaming: false + model_example: '' + min_version: '' + rerank: + supported: false + streaming: false + model_example: '' + min_version: '' + limitations: + provider_specific: [] + statistics_logging: [] + + - name: DeepSeek + url_patterns: + - 'https://api.deepseek.com' + min_version: '2.0' + capabilities: + generate: + supported: true + streaming: true + upstream_path: '`/v1/chat/completions`' + model_example: 'deepseek-chat' + min_version: '2.0' + completions: + supported: false + streaming: false + model_example: '' + min_version: '' + embeddings: + supported: false + streaming: false + model_example: '' + min_version: '' + files: + supported: false + streaming: false + model_example: '' + min_version: '' + batches: + supported: false + streaming: false + model_example: '' + min_version: '' + agentic: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_speech: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_transcription: + supported: false + streaming: false + model_example: '' + min_version: '' + audio_translation: + supported: false + streaming: false + model_example: '' + min_version: '' + image: + supported: false + streaming: false + model_example: '' + min_version: '' + video: + supported: false + streaming: false + model_example: '' + min_version: '' + realtime: + supported: false + streaming: false + model_example: '' + min_version: '' + rerank: + supported: false + streaming: false + model_example: '' + min_version: '' + limitations: + provider_specific: [] + statistics_logging: [] diff --git a/app/_data/entity_examples/ai-gateway/providers/anthropic.yml b/app/_data/entity_examples/ai-gateway/providers/anthropic.yml new file mode 100644 index 00000000000..e69de29bb2d diff --git a/app/_data/entity_examples/config.yml b/app/_data/entity_examples/config.yml index 61cd0c53012..4497ebe3a89 100644 --- a/app/_data/entity_examples/config.yml +++ b/app/_data/entity_examples/config.yml @@ -46,6 +46,11 @@ event_gateway_variables: &event_gateway_variables placeholder: 'tlsTrustBundleName' description: "The `name` of the TLS Trust Bundle." +ai_gateway_variables: &ai_gateway_variables + ai_gateway: + placeholder: 'ai-gateway-name' + description: "The `name` of your AI Gateway." + formats: deck: label: 'decK' @@ -55,9 +60,12 @@ formats: admin-api: label: 'Admin API' base_url: 'http://localhost:8001' + ai_gateway_base_url: 'http://localhost:8001' endpoints: + # core entities consumer: '/consumers/' consumer_group: '/consumer_groups/' + model: '/models/' route: '/routes/' service: '/services/' target: '/upstreams/{upstream}/targets/' @@ -76,19 +84,41 @@ formats: keyring: '/keyring/' event_hook: '/event-hooks/' partial: '/partials/' + ai_endpoints: + # AI entities (/ai/* on on-prem AI Gateway) + consumer: '/ai-consumers/' + consumer_group: '/ai-consumer-groups/' + vault: '/ai-vaults/' plugin_endpoints: consumer: '/consumers/{consumer}/plugins/' consumer_group: '/consumer_groups/{consumer_group}/plugins/' route: '/routes/{route}/plugins/' service: '/services/{service}/plugins/' global: '/plugins/' + ai_policy_endpoints: + ai_model: '/models/{ai_model}/policies/' + ai_agent: '/agents/{ai_agent}/policies/' + ai_mcp_server: '/mcp-servers/{ai_mcp_server}/policies/' variables: <<: *variables + ai_gateway: + placeholder: 'AIGatewayId' + description: 'The `id` of the AI Gateway.' + ai_model: + placeholder: 'aiModelId' + description: 'The `id` of the AI Model.' + ai_agent: + placeholder: 'aiAgentId' + description: 'The `id` of the AI Agent.' + ai_mcp_server: + placeholder: 'aiMCPServerId' + description: 'The `id` of the AI MCP Server.' konnect-api: label: 'Konnect API' base_url: 'https://{region}.api.konghq.com/v2/control-planes/{control_plane}/core-entities' event_gateway_base_url: 'https://{region}.api.konghq.com/v1/event-gateways/{event_gateway}' + ai_gateway_base_url: 'https://{region}.api.konghq.com/v1/ai-gateways/{ai_gateway}' endpoints: consumer: '/consumers/' consumer_group: '/consumer_groups/' @@ -109,6 +139,17 @@ formats: schema_registry: '/schema-registries' static_key: '/static-keys' tls_trust_bundle: '/tls-trust-bundles' + ai_endpoints: + model: '/models' + policy: '/policies' + agent: '/agents' + mcp_server: '/mcp-servers' + provider: '/providers' + model-provider: '/model-providers' + identity-provider: '/identity' + consumer: '/consumers' + consumer_group: '/consumer-groups' + vault: '/vaults' plugin_endpoints: consumer: '/consumers/{consumer}/plugins/' consumer_group: '/consumer_groups/{consumer_group}/plugins/' @@ -151,7 +192,11 @@ formats: event_gateway_listener: placeholder: 'eventGatewayListenerId' description: The `id` of the Event Gateway Listener. - + ai_gateway_variables: + <<: *konnect_variables + ai_gateway: + placeholder: 'AIGatewayId' + description: 'The `id` of the AI Gateway.' kic: label: 'KIC' @@ -164,10 +209,20 @@ formats: kongctl: label: 'kongctl' event_gateway_variables: *event_gateway_variables + ai_gateway_variables: *ai_gateway_variables ui: label: 'UI' entities: + - ai-model-provider + - ai-identity-provider + - ai-model + - ai-agent + - ai-mcp-server + - ai-policy + - ai-consumer + - ai-consumer-group + - ai-vault - admin - ca_certificate - certificate @@ -204,4 +259,4 @@ phases: produce: label: 'Produce Phase' cluster: - label: 'Cluster Phase' \ No newline at end of file + label: 'Cluster Phase' diff --git a/app/_data/homepage.yml b/app/_data/homepage.yml index ec7f22d05ec..1a332e986cc 100644 --- a/app/_data/homepage.yml +++ b/app/_data/homepage.yml @@ -52,19 +52,34 @@ agents_section: cta_url: /skills/ cta_text: "Browse all skills" icon: /assets/icons/brain.svg - - id: cookbooks - title: "AI Cookbooks" + - id: aigw + title: "AI Gateway 2.0" + badge: "New" + snippet_label: "QUICKSTART" + snippet_lines: + - "curl -Ls https://get.konghq.com/ai | bash -s -- -k $KONNECT_TOKEN" + description: "Connectivity and governance layer for modern AI-native applications." ctas: - - text: "Model-based routing" - url: /cookbooks/model-based-routing/ - - text: "Claude Code SSO" - url: /cookbooks/claude-code-sso/ - - text: "LLM cost optimization" - url: /cookbooks/llm-cost-optimization/ - - text: "View all cookbooks" - url: /cookbooks/ - description: "End-to-end recipes for agents on top of Kong AI Gateway." - icon: /assets/icons/book.svg + - text: "Other ways to get started" + url: /ai-gateway/#guided-quickstarts + - text: "View all AI Gateway 2.0 docs" + url: /ai-gateway/ + icon: /assets/icons/ai.svg + +##### Taking out cookbooks until they're updated for AIGW 2.0; when they're ready, remove AIGW section above and uncomment this one. + # - id: aigw + # title: "AI Cookbooks" + # ctas: + # - text: "Model-based routing" + # url: /cookbooks/model-based-routing/ + # - text: "Claude Code SSO" + # url: /cookbooks/claude-code-sso/ + # - text: "LLM cost optimization" + # url: /cookbooks/llm-cost-optimization/ + # - text: "View all cookbooks" + # url: /cookbooks/ + # description: "End-to-end recipes for agents on top of Kong AI Gateway." + # icon: /assets/icons/book.svg products_section: categories: @@ -151,8 +166,11 @@ products_section: - text: All documentation url: /index/gateway/ - title: AI Gateway + product: ai-gateway + changelog_url: /ai-gateway/changelog/ + whats_new: true icon: /assets/icons/ai.svg - description: "Connectivity and governance layer for modern AI-native applications built on top of Kong Gateway." + description: "Connectivity and governance layer for modern AI-native applications." links: - text: Overview url: /ai-gateway/ @@ -257,7 +275,7 @@ products_section: specs_section: featured_api_slugs: - - "konnect/control-planes" + - "konnect/ai-gateway" - "konnect/control-planes-config" - "konnect/event-gateway" - "konnect/portal-management" diff --git a/app/_data/konnect_oas_data.json b/app/_data/konnect_oas_data.json index 25b924caf16..d02b1d1e354 100644 --- a/app/_data/konnect_oas_data.json +++ b/app/_data/konnect_oas_data.json @@ -353,6 +353,27 @@ } ] }, + { + "id": "5e0005b9-232b-4808-8bdf-9560d4596080", + "title": "Konnect AI Gateway", + "latestVersion": { + "name": "v1", + "id": "8cf2d03b-e257-4cc8-8b02-04abee8376b3" + }, + "description": "The API for configuring AI Gateways in Konnect.", + "documentCount": 0, + "versionCount": 1, + "versions": [ + { + "id": "8cf2d03b-e257-4cc8-8b02-04abee8376b3", + "created_at": "2026-07-14T16:16:38.145Z", + "updated_at": "2026-07-14T16:20:12.585Z", + "name": "v1", + "deprecated": false, + "registration_configs": [] + } + ] + }, { "id": "a9357984-c292-4846-856b-85aff7df6c54", "title": "Konnect Analytics Dashboards", diff --git a/app/_data/policies/ai-gateway/scopes.json b/app/_data/policies/ai-gateway/scopes.json new file mode 100644 index 00000000000..8f997b75ff9 --- /dev/null +++ b/app/_data/policies/ai-gateway/scopes.json @@ -0,0 +1,1050 @@ +[ + { + "name": "ace", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "acl", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "acme", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "ai-aws-guardrails", + "scopes": [ + "models", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "ai-azure-content-safety", + "scopes": [ + "models", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "ai-custom-guardrail", + "scopes": [ + "models", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "ai-gcp-model-armor", + "scopes": [ + "models", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "ai-lakera-guard", + "scopes": [ + "models", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "ai-llm-as-judge", + "scopes": [ + "models", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "ai-mcp-oauth2", + "scopes": [ + "mcp-servers", + "global" + ] + }, + { + "name": "ai-prompt-compressor", + "scopes": [ + "models", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "ai-prompt-decorator", + "scopes": [ + "models", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "ai-prompt-guard", + "scopes": [ + "models", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "ai-prompt-template", + "scopes": [ + "models", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "ai-rag-injector", + "scopes": [ + "models", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "ai-rate-limiting-advanced", + "scopes": [ + "models", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "ai-request-transformer", + "scopes": [ + "models", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "ai-response-transformer", + "scopes": [ + "models", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "ai-sanitizer", + "scopes": [ + "models", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "ai-semantic-cache", + "scopes": [ + "models", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "ai-semantic-prompt-guard", + "scopes": [ + "models", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "ai-semantic-response-guard", + "scopes": [ + "models", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "app-dynamics", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "aws-lambda", + "scopes": [ + "consumers", + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "azure-functions", + "scopes": [ + "consumers", + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "basic-auth", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "bot-detection", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "canary", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "confluent", + "scopes": [ + "consumers", + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "confluent-consume", + "scopes": [ + "consumers", + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "correlation-id", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "cors", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "datadog", + "scopes": [ + "consumers", + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "datakit", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "degraphql", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "exit-transformer", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "file-log", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "forward-proxy", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "graphql-proxy-cache-advanced", + "scopes": [ + "consumers", + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "graphql-rate-limiting-advanced", + "scopes": [ + "consumers", + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "grpc-gateway", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "grpc-web", + "scopes": [ + "consumers", + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "header-cert-auth", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "hmac-auth", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "http-log", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "injection-protection", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "ip-restriction", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "jq", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "json-threat-protection", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "jwe-decrypt", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "jwt", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "jwt-signer", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "kafka-consume", + "scopes": [ + "consumers", + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "kafka-log", + "scopes": [ + "consumers", + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "kafka-upstream", + "scopes": [ + "consumers", + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "key-auth", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "ldap-auth", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "ldap-auth-advanced", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "loggly", + "scopes": [ + "consumers", + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "metering-and-billing", + "scopes": [ + "consumers", + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "mocking", + "scopes": [ + "consumers", + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "mtls-auth", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "oas-validation", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "oauth2-introspection", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "opa", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "openid-connect", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "opentelemetry", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "post-function", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "pre-function", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "prometheus", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "proxy-cache", + "scopes": [ + "consumers", + "consumer-groups", + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "proxy-cache-advanced", + "scopes": [ + "consumers", + "consumer-groups", + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "rate-limiting", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "rate-limiting-advanced", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "redirect", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "request-callout", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "request-size-limiting", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "request-termination", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "request-transformer", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "request-transformer-advanced", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "request-validator", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "response-ratelimiting", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "response-transformer", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "response-transformer-advanced", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "route-by-header", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "global" + ] + }, + { + "name": "route-transformer-advanced", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "global" + ] + }, + { + "name": "saml", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "service-protection", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "session", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "solace-consume", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "solace-log", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "solace-upstream", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "standard-webhooks", + "scopes": [ + "consumer-groups", + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "statsd", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "syslog", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "tcp-log", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "tls-handshake-modifier", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "tls-metadata-headers", + "scopes": [ + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "udp-log", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "upstream-oauth", + "scopes": [ + "consumers", + "consumer-groups", + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "upstream-timeout", + "scopes": [ + "consumers", + "models", + "mcp-servers", + "agents", + "global" + ] + }, + { + "name": "websocket-size-limit", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "websocket-validator", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "xml-threat-protection", + "scopes": [ + "models", + "mcp-servers", + "agents", + "consumers", + "consumer-groups", + "global" + ] + }, + { + "name": "zipkin", + "scopes": [ + "consumers", + "models", + "mcp-servers", + "agents", + "global" + ] + } +] diff --git a/app/_data/products/ai-gateway.yml b/app/_data/products/ai-gateway.yml index e40c9b2a8bb..38a9e6ff8a1 100644 --- a/app/_data/products/ai-gateway.yml +++ b/app/_data/products/ai-gateway.yml @@ -1,2 +1,15 @@ name: AI Gateway -icon: /_assets/icons/products/ai-gateway.svg \ No newline at end of file +icon: /_assets/icons/products/ai-gateway.svg + +previous_major_url_segment: v + +releases: + - release: "2.0" + latest: true + version: "2.0.0" + name: "v1" + - release: "1.0" + +release_dates: + '2.0.1': 2026/07/29 + '2.0.0': 2026/07/10 diff --git a/app/_data/schemas/frontmatter/base.json b/app/_data/schemas/frontmatter/base.json index e2e7bfbe6d2..86eb79dcb3b 100644 --- a/app/_data/schemas/frontmatter/base.json +++ b/app/_data/schemas/frontmatter/base.json @@ -1,11 +1,21 @@ { "$id": "schema:base", "definitions": { + "major_version": { + "type": "object", + "patternProperties": { + "^[a-zA-Z_-]+$": { + "type": "integer" + } + }, + "additionalProperties": false + }, "min_version": { "type": "object", "patternProperties": { "^[a-zA-Z_-]+$": { - "type": "string" + "type": "string", + "pattern": "^\\d+\\.\\d+(\\.\\d+)?$" } }, "additionalProperties": false diff --git a/app/_data/schemas/frontmatter/tags.json b/app/_data/schemas/frontmatter/tags.json index 73c7bbf2530..16e982dc61f 100644 --- a/app/_data/schemas/frontmatter/tags.json +++ b/app/_data/schemas/frontmatter/tags.json @@ -123,6 +123,7 @@ "kafka", "kds", "key-auth", + "kimi", "kong-manager", "kongair", "kong-identity", @@ -221,6 +222,7 @@ "upgrade", "validation", "vault", + "vercel", "versioning", "vertex-ai", "vllm", diff --git a/app/_data/series.yml b/app/_data/series.yml index 98f35cdae8a..3a4548e9559 100644 --- a/app/_data/series.yml +++ b/app/_data/series.yml @@ -22,7 +22,7 @@ operator-get-started-event-gateway: url: /operator/get-started/event-gateway/install/ mcp-traffic: title: Secure, govern and observe MCP traffic with {{site.ai_gateway}} - url: /mcp/secure-mcp-traffic/ + url: /ai-gateway/v1/mcp/secure-mcp-traffic/ hashicorp-vault-llms: title: Configure dynamic authentication to LLM providers url: /how-to/configure-hashicorp-vault-as-a-vault-for-llm-providers/ @@ -44,3 +44,6 @@ mesh-get-started-universal: mesh-scoped-zone-proxy: title: Deploy and configure mesh-scoped zone proxies url: /mesh/zone-proxies/ +mcp-conversion-2-0: + title: Map a RESTful API to MCP tools and observe MCP traffic + url: /ai-gateway/map-api-to-mcp-tools/ \ No newline at end of file diff --git a/app/_data/version_errors_konnect.yml b/app/_data/version_errors_konnect.yml index 1cda86bea6f..eed4e3fa9fa 100644 --- a/app/_data/version_errors_konnect.yml +++ b/app/_data/version_errors_konnect.yml @@ -4515,7 +4515,7 @@ messages: Use `consumer_claims` instead of the deprecated `consumer_claim` shorthand field. DocumentationURL: | - [Konnect Application Auth plugin](/plugins/konnect-application-auth/) + N/A - ID: D572 Severity: error Description: | @@ -4759,7 +4759,7 @@ messages: Resolution: | Please upgrade Kong Gateway to version `3.15.0.0` or above. DocumentationURL: | - [Konnect Application Auth plugin](/plugins/konnect-application-auth/) + N/A - ID: D600 Severity: warning Description: | @@ -4843,7 +4843,7 @@ messages: Resolution: | Please upgrade Kong Gateway to version `3.15.0.0` or above. DocumentationURL: | - [ACE Auth plugin](/plugins/ace-auth/) + [ACE Auth plugin](/plugins/ace/) - ID: D608 Severity: warning Description: | @@ -4862,7 +4862,7 @@ messages: Resolution: | Please upgrade Kong Gateway to version `3.15.0.0` or above. DocumentationURL: | - [Cloned Plugins entity](/gateway/entities/cloned-plugins/) + [Cloned Plugins entity](/gateway/entities/plugin/#cloning-plugins) - ID: D610 Severity: warning Description: | diff --git a/app/_gateway_entities/consumer-group.md b/app/_gateway_entities/consumer-group.md index cf479b74056..26e582c9a0b 100644 --- a/app/_gateway_entities/consumer-group.md +++ b/app/_gateway_entities/consumer-group.md @@ -4,7 +4,7 @@ content_type: reference entities: - consumer-group -description: Consumer Groups let you apply common configurations to groups of Consumers, such as rate limiting policies or request and response transformation. +description: Consumer Groups let you apply common configurations to groups of Consumers, such as rate limiting policies or request and response transformation. tools: - admin-api @@ -34,12 +34,12 @@ faqs: - q: Why aren't Consumer Group overrides working anymore? a: | Consumer Groups became a core Gateway entity in 3.4, which opened up a wide range of use cases for grouping Consumers. - + Before 3.4, Consumer Groups were limited to rate limiting plugins, where they were configured through overrides. This is no longer necessary. Instead, you can enable any rate limiting plugin directly on a consumer group without worrying about extra configuration. - q: How do I enable a plugin on a Consumer Group? a: | - First, [find out](/gateway/entities/plugin/#supported-scopes-by-plugin) if the plugin you want supports Consumer Groups. - + First, [find out](/gateway/entities/plugin/#supported-scopes-by-plugin) if the plugin you want supports Consumer Groups. + If it does, head over to the plugin's documentation, open the "Get Started" tab, and choose "Consumer Groups" from the dropdown for any available example. - q: When a Consumer is part of multiple Consumer Groups, how is precedence determined? @@ -72,9 +72,9 @@ flowchart LR B(Consumer Group Gold - fa:fa-user Consumer 1, fa:fa-user Consumer 2, + fa:fa-user Consumer 1, fa:fa-user Consumer 2, fa:fa-user Consumer 5 ) - + C(Consumer Group Silver fa:fa-user Consumer 3, fa:fa-user Consumer 4) @@ -85,7 +85,7 @@ flowchart LR 2 requests/second) F(Gateway Service QR Code Generation) - H(QR Code Generation + H(QR Code Generation service) A--> B & C @@ -99,10 +99,10 @@ flowchart LR {% endmermaid %} -Without Consumer Groups, you would have to use five Rate Limiting Advanced plugins, once for each consumer. +Without Consumer Groups, you would have to use five Rate Limiting Advanced plugins, once for each consumer. Any time you change the rate limit, you would need to update every consumer individually. -Consumer Groups allow you to manage your plugin configuration centrally, and reduce the size of your {{ site.base_gateway }} configuration at the same time. +Consumer Groups allow you to manage your plugin configuration centrally, and reduce the size of your {{ site.base_gateway }} configuration at the same time. In this example, it's the difference between using two plugins or five plugins. In your production environment, it could be the difference between two plugins and five _million_ plugins. ## Use cases diff --git a/app/_how-tos/ai-gateway/get-started-with-ai-agent.md b/app/_how-tos/ai-gateway/get-started-with-ai-agent.md new file mode 100644 index 00000000000..29e65ca6616 --- /dev/null +++ b/app/_how-tos/ai-gateway/get-started-with-ai-agent.md @@ -0,0 +1,212 @@ +--- +title: Route A2A agent traffic through {{site.ai_gateway}} +content_type: how_to +permalink: /ai-gateway/get-started-with-ai-agent/ +description: Create an AI Agent entity in {{site.ai_gateway}} to proxy Agent-to-Agent (A2A) protocol traffic +products: + - ai-gateway + +works_on: + - konnect + +min_version: + ai-gateway: '2.0' + +entities: + - ai-agent + +tags: + - get-started + - ai + - a2a + +tldr: + q: How do I route A2A agent traffic through {{site.ai_gateway}}? + a: | + When agents need to communicate with other agents, route the traffic through {{site.ai_gateway}} to apply authentication, rate limiting, observability, and content policies at the gateway layer. + Create an [AI Agent](/ai-gateway/entities/ai-agent/) entity that exposes your upstream agent at a gateway route and attach policies for logging, security, and traffic control. + The gateway proxies A2A JSON-RPC requests, discovers agent capabilities through Agent Cards, and exports metrics and payloads as observability spans. + + This tutorial shows you how to set up an AI Agent entity in {{site.konnect_product_name}} using the {{site.konnect_product_name}} API and how to test A2A traffic flowing through the gateway. + +tools: + - kongctl + +prereqs: + inline: + - title: OpenAI API key + content: | + 1. [Create an OpenAI account](https://auth.openai.com/create-account). + 1. [Get an API key](https://platform.openai.com/api-keys). + 1. Export your key: + ```bash + export OPENAI_API_KEY='YOUR_OPENAI_API_KEY' + ``` + + - title: A2A agent + include_content: md/ai-gateway/v2/prereqs/a2a-agent +related_resources: + - text: "{{site.ai_gateway}}" + url: /ai-gateway/ + - text: AI Agent entity reference + url: /ai-gateway/entities/ai-agent/ + - text: Secure AI Agent traffic with OpenID Connect and Okta + url: /ai-gateway/secure-ai-agent-with-oidc/ + - text: Monitor AI Agent traffic with OpenTelemetry + url: /ai-gateway/monitor-ai-agent-with-opentelemetry/ + - text: A2A protocol specification + url: https://a2a-protocol.org/latest/ + +cleanup: + inline: + - title: Stop the A2A agent + content: | + ```bash + docker compose down + docker rm -f a2a-kongair-agent + ``` + + - title: Clean up {{site.ai_gateway}} resources + include_content: cleanup/products/ai-gateway + +faqs: + - q: What is the A2A protocol? + a: The Agent-to-Agent (A2A) protocol is an open standard originally developed by Google that defines how AI agents communicate with each other. It uses JSON-RPC over HTTP and supports capability discovery through Agent Cards, task lifecycle management, multi-turn conversations, and streaming responses. See the [A2A protocol documentation](https://a2a-protocol.org/latest/) for the full specification. + + - q: How is A2A different from MCP? + a: MCP (Model Context Protocol) standardizes how agents connect to tools, APIs, and data sources. A2A standardizes how agents communicate with other agents. They are complementary. Use MCP for agent-to-tool communication and A2A for agent-to-agent communication. + + - q: Can I add authentication to the A2A endpoint? + a: Yes. Create an AI Policy like [OpenID Connect](/ai-gateway/policies/openid-connect/) for authentication and attach it to the agent. The AI Agent entity handles A2A protocol concerns independently of authentication. + + - q: How do I enable request/response logging? + a: Set `config.logging.payloads` to `true` and `config.logging.statistics` to `true` in the agent config to log A2A request and response bodies along with metrics. + +--- + +## Create an AI Agent entity + +Create an [AI Agent](/ai-gateway/entities/ai-agent/) entity that proxies A2A traffic to your upstream agent. + +{% entity_examples %} +ai_gateway_agents: + - ref: kongair-flight-booking-agent + ai_gateway: !lookup name:ai-quickstart + display_name: "Kong Air Flight Booking Agent" + type: a2a + enabled: true + config: + url: http://host.docker.internal:10000 + route: + paths: + - /a2a + methods: + - GET + - POST + protocols: + - http + - https + strip_path: true + logging: + payloads: true + statistics: true + max_payload_size: 1048576 + max_request_body_size: 8388608 +{% endentity_examples %} + + +The `ai_gateway_agents` entry references your existing {{site.ai_gateway}} by its `name` (`ai-quickstart`, as set up by the [quickstart script](/ai-gateway/get-started/)), so `kongctl` manages the agent underneath it instead of creating a new gateway. Each nested agent still declares its own `ai_gateway` field, pointing at the gateway's ID, to link it to the parent. + +The agent is now accessible at the `/a2a` route and proxies A2A JSON-RPC requests to the upstream agent running at `http://host.docker.internal:10000`. + +## Retrieve the Agent Card + +A2A agents expose their capabilities through an Agent Card at the `/.well-known/agent-card.json` endpoint. + +Retrieve it through the gateway: + +```bash +curl -X GET "http://localhost:8000/a2a/.well-known/agent-card.json" \ + --no-progress-meter --fail-with-body +``` + +The response shows the agent's capabilities, skills, and supported protocols: + +```json +{ + "name": "KongAir OpenAI Agent", + "version": "1.0.0", + "description": "An A2A-compatible agent powered by LangGraph and OpenAI that queries KongAir APIs for flights, routes, bookings, and loyalty info.", + "protocolVersion": "0.3.0", + "capabilities": { + "pushNotifications": false, + "streaming": false + }, + "defaultInputModes": ["text", "text/plain"], + "defaultOutputModes": ["text", "text/plain"], + "preferredTransport": "JSONRPC", + "url": "http://a2a-agent:10000/", + "skills": [ + { + "id": "search_routes", + "name": "Search KongAir routes", + "description": "Find KongAir routes between airports.", + "examples": [ + "Show me routes from SFO to JFK", + "Find flights from LHR to SFO" + ], + "tags": ["kongair", "flights", "travel", "routes"] + } + ] +} +``` + +## Send an A2A request + +Send a `message/send` JSON-RPC request to test the agent: + +```bash +curl -X POST "http://localhost:8000/a2a" \ + -H "Content-Type: application/json" \ + --json '{ + "jsonrpc": "2.0", + "id": "1", + "method": "message/send", + "params": { + "message": { + "kind": "message", + "messageId": "msg-001", + "role": "user", + "parts": [ + { + "kind": "text", + "text": "What flights are available on route KA-123?" + } + ] + } + } + }' +``` + +A successful response (status 200) contains the agent's reply: + +```json +{ + "jsonrpc": "2.0", + "id": "1", + "result": { + "message": { + "kind": "message", + "messageId": "msg-002", + "role": "assistant", + "parts": [ + { + "kind": "text", + "text": "Route KA-123 has 5 available flights today..." + } + ] + } + } +} +``` + diff --git a/app/_how-tos/ai-gateway/get-started-with-ai-gateway.md b/app/_how-tos/ai-gateway/get-started-with-ai-gateway.md index 087d4b41eed..f679f8e955e 100644 --- a/app/_how-tos/ai-gateway/get-started-with-ai-gateway.md +++ b/app/_how-tos/ai-gateway/get-started-with-ai-gateway.md @@ -2,147 +2,133 @@ title: Get started with {{site.ai_gateway}} content_type: how_to permalink: /ai-gateway/get-started/ -description: Learn how to quickly get started with {{site.ai_gateway}} +description: Learn how to proxy LLM traffic with {{site.ai_gateway}} entities in {{site.konnect_product_name}} products: - - ai-gateway - - gateway + - ai-gateway works_on: - - on-prem - - konnect - -plugins: - - ai-proxy + - konnect entities: - - service - - route - - plugin + - ai-provider + - ai-model tags: - - get-started - - ai - - openai + - get-started + - ai tldr: - q: What is {{site.ai_gateway}}, and how can I get started with it? + q: How do I proxy LLM traffic with {{site.ai_gateway}} entities? a: | - With {{site.ai_gateway}}, you can deploy AI infrastructure for traffic - that is sent to one or more LLMs. This lets you semantically route, secure, observe, accelerate, - and govern traffic using a special set of AI plugins that are bundled with {{site.base_gateway}} distributions. - - This tutorial will help you get started with {{site.ai_gateway}} by setting up the AI Proxy plugin with OpenAI. + {{site.ai_gateway}} provides first-class entities for managing LLM providers and models in {{site.konnect_product_name}}. + Create an [AI Model Provider](/ai-gateway/entities/ai-model-provider/) entity to connect and authenticate to an LLM service like OpenAI, then create an [AI + Model](/ai-gateway/entities/ai-model/) entity to specify which model is available for requests. - {:.info} - > **Note:** - > This quickstart runs a Docker container to explore {{ site.base_gateway }}'s capabilities. - If you want to run {{ site.base_gateway }} as a part of a production-ready API platform, start with the [Install](/gateway/install/) page. + This tutorial shows you how to set up an AI Provider and AI Model for OpenAI in {{site.konnect_product_name}} using kongctl and how to proxy your first request to OpenAI. tools: - - deck + - kongctl prereqs: inline: - title: OpenAI - content: | - This tutorial uses the AI Proxy plugin with OpenAI. You'll need to [create an OpenAI account](https://auth.openai.com/create-account) and [get an API key](https://platform.openai.com/api-keys). Once you have your API key, create an environment variable: - - ```sh - export OPENAI_API_KEY='' - ``` - + include_content: md/ai-gateway/v2/prereqs/openai + icon_url: /assets/icons/openai.svg cleanup: inline: - - title: Clean up Konnect environment - include_content: cleanup/platform/konnect - icon_url: /assets/icons/gateway.svg - - title: Destroy the {{site.base_gateway}} container - include_content: cleanup/products/gateway - icon_url: /assets/icons/gateway.svg + - title: Clean up {{site.ai_gateway}} resources + include_content: cleanup/products/ai-gateway min_version: - gateway: '3.6' - -next_steps: - - text: Set up load balancing using AI Proxy Advanced plugin - url: /plugins/ai-proxy-advanced/ - - text: Cache traffic using the AI Semantic cache plugin - url: /plugins/ai-semantic-cache/ - - text: Secure traffic with the AI Prompt Guard - url: /plugins/ai-prompt-guard/ - - text: Provide prompt templates with AI Prompt Template - url: /plugins/ai-prompt-template/ - - text: Programmatically inject system or assistant prompts to all incoming prompts with the AI Prompt Decorator - url: /plugins/ai-prompt-decorator/ - - text: Learn about all the AI plugins - url: /plugins/?category=ai + ai-gateway: '2.0' --- -## Check that {{site.base_gateway}} is running +## Create an AI Provider entity -{% include how-tos/steps/ping-gateway.md %} +Create an [AI Model Provider](/ai-gateway/entities/ai-model-provider/) entity to define your connection to OpenAI and store your authentication credentials: +First, set the `OPENAI_AUTH_HEADER` environment variable to your OpenAI API key: -## Create a Gateway Service +```sh +export OPENAI_AUTH_HEADER="Bearer $OPENAI_API_KEY" +``` -Create a Service to contain the Route for the LLM provider: +Then, apply the configuration using `kongctl`: {% entity_examples %} -entities: - services: - - name: llm-service - url: http://localhost:32000 +ai_gateway_model_providers: + - ref: generic-openai + ai_gateway: !lookup name:ai-quickstart + name: generic-openai + display_name: "generic-openai" + type: openai + config: + auth: + type: basic + headers: + - name: Authorization + value: !env OPENAI_AUTH_HEADER {% endentity_examples %} -The URL can point to any empty host, as it won't be used by the plugin. +{:.info} +> `ai-quickstart` references the {{site.ai_gateway}} created by the quickstart script in the [prerequisites](#prerequisites), instead of creating a new one. -## Create a Route +In this example, we're setting up the AI Provider with: -Create a Route for the LLM provider. In this example we're creating a chat route, so we'll use `/chat` as the path: +* `type: openai`: Specifies that this provider connects to the OpenAI service using OpenAI's standard API format. +* `name: generic-openai`: A unique identifier that AI Models will reference to route requests through this provider. +* `config.auth`: Stores your OpenAI API key. {{site.ai_gateway}} securely manages this credential and injects it into upstream requests automatically, eliminating the need for clients to pass API keys. -{% entity_examples %} -entities: - routes: - - name: openai-chat - service: - name: llm-service - paths: - - /chat - protocols: - - http - - https -{% endentity_examples %} - -## Enable the AI Proxy plugin +## Create an AI Model entity -Enable the AI Proxy plugin to create a chat route: +Create an [AI Model](/ai-gateway/entities/ai-model/) entity to declare which upstream models are available, configure how client requests are routed, and specify which AI Provider to use: {% entity_examples %} -entities: - plugins: - - name: ai-proxy - config: - route_type: "llm/v1/chat" - model: - provider: "openai" +ai_gateway_models: + - ref: my-gpt-4o + ai_gateway: !lookup name:ai-quickstart + name: my-gpt-4o + display_name: "my-gpt-4o" + type: model + formats: + - type: openai + config: + route: + paths: + - /v1 + model: + alias: my-gpt-4o + targets: + - name: gpt-4o + provider: generic-openai + config: + type: openai + policies: [] + capabilities: + - generate {% endentity_examples %} -In this example, we're setting up the plugin with minimal configuration, which means: -* The client is allowed to use any model in the `openai` provider and must provide the model name in the request body. -* The client must provide an `Authorization` header with an OpenAI API key. +{:.info} +> `ai-quickstart` references the {{site.ai_gateway}} created by the quickstart script, same as in the previous step. -If needed, you can restrict the models that can be consumed by specifying the model name explicitly using the [`config.model.name`](/plugins/ai-proxy/reference/#schema--config-model-name) parameter. +In this example, we're setting up the AI Model with: -You can also provide the OpenAI API key directly in the configuration with the [`config.auth.header_name`](/plugins/ai-proxy/reference/#schema--config-auth-header-name) and [`config.auth.header_value`](/plugins/ai-proxy/reference/#schema--config-auth-header-value) parameters so that the client doesn’t have to send them. +* `type: model`: Specifies this is a synchronous model for request/response workloads. +* `name: my-gpt-4o`: A unique identifier for this model. +* `formats: [type: openai]`: Declares that this model accepts requests in OpenAI-compatible format. +* `config.route.paths: [/v1]`: Configures the custom base path where this model's Routes will be accessible. Clients will send requests to paths that combine this base path with capability-specific Routes. +* `capabilities: [generate]`: Enables the text generation capability. The `generate` capability creates a `/chat/completions` endpoint, so combined with your base path, clients send chat requests to `/v1/chat/completions`. +* `config.model.alias: my-gpt-4o`: Lets clients send `my-gpt-4o` in the request `model` field instead of the upstream model name. +* `targets`: Specifies which upstream AI Provider model to route requests to. Here, `provider: generic-openai` references the AI Provider we created earlier, and `name: gpt-4o` specifies which OpenAI model to call upstream. ## Validate -To validate, you can send a `POST` request to the `/chat` endpoint, using the correct [input format](/plugins/ai-proxy/#input-formats). -Since we didn't add the model name and API key in the plugin configuration, make sure to include them in the request: +Send a chat request to verify your setup: + {% validation request-check %} -url: /chat +url: /v1/chat/completions status_code: 200 method: POST headers: @@ -150,10 +136,9 @@ headers: - 'Content-Type: application/json' - 'Authorization: Bearer $OPENAI_API_KEY' body: - model: gpt-5-mini messages: - role: "user" content: "Say this is a test!" + model: my-gpt-4o {% endvalidation %} - -You should get a `200 OK` response, and the response body should contain `This is a test`. + diff --git a/app/_how-tos/ai-gateway/get-started-with-mcp-server.md b/app/_how-tos/ai-gateway/get-started-with-mcp-server.md new file mode 100644 index 00000000000..31afd4fb501 --- /dev/null +++ b/app/_how-tos/ai-gateway/get-started-with-mcp-server.md @@ -0,0 +1,182 @@ +--- +title: Map the WeatherAPI to an MCP Server +content_type: how_to +permalink: /ai-gateway/get-started-with-mcp-server/ +description: Learn how to create an MCP Server entity in {{site.ai_gateway}} to expose WeatherAPI operations as MCP tools +products: + - ai-gateway + +works_on: + - konnect + +min_version: + ai-gateway: '2.0' + +entities: + - ai-mcp-server + +tags: + - get-started + - ai + - mcp + +tldr: + q: How do I expose REST APIs as MCP tools in {{site.ai_gateway}}? + a: | + {{site.ai_gateway}} provides first-class MCP Server entities in {{site.konnect_product_name}} that expose REST APIs as tools for MCP-compatible clients. + Create an [AI MCP Server](/ai-gateway/entities/ai-mcp-server/) entity configured as a `conversion-listener` to convert REST endpoints into MCP tools that clients can call directly, without managing API credentials. + + This tutorial shows you how to set up an AI MCP Server to expose the [WeatherAPI](https://openweathermap.org/api/one-call-4?collection=one_call_api) in {{site.konnect_product_name}} using [kongctl](/kongctl/), and how to proxy your first MCP request. + +tools: + - kongctl + +prereqs: + inline: + - title: WeatherAPI account + content: | + 1. Go to [WeatherAPI](https://www.weatherapi.com/). + 1. Navigate to [your dashboard](https://www.weatherapi.com/my/) and copy your API key. + 1. Export your API key by running the following command in your terminal: + ```sh + export WEATHERAPI_API_KEY='your-weatherapi-api-key' + ``` +related_resources: + - text: "{{site.ai_gateway}}" + url: /ai-gateway/ + - text: AI MCP Server entity + url: /ai-gateway/entities/ai-mcp-server/ + +cleanup: + inline: + - title: Clean up {{site.ai_gateway}} resources + include_content: cleanup/products/ai-gateway + +--- + +## Create an MCP Server entity + +Create an [MCP Server](/ai-gateway/entities/ai-mcp-server/) entity that exposes the [WeatherAPI](https://www.weatherapi.com/) through a single MCP tool called `get-current-weather`, mapped from the WeatherAPI `/v1/current.json` endpoint. `tools[].query.key` injects your WeatherAPI credentials automatically, so clients never handle the API key: + +{% entity_examples %} +ai_gateway_mcp_servers: + - ref: weather-mcp + ai_gateway: !lookup name:ai-quickstart + name: weather-mcp + display_name: "Weather API" + type: conversion-listener + enabled: true + policies: [] + access: + acl_attribute_type: consumer + acls: + allow: [] + default_tool_acls: + deny: [] + config: + url: https://api.weatherapi.com/v1/current.json + route: + paths: + - /weather + logging: + payloads: false + statistics: true + server: + timeout: 60000 + tools: + - name: get-current-weather + description: Get current weather for a location + method: GET + path: /weather + query: + key: + - !env WEATHERAPI_API_KEY + parameters: + - name: q + in: query + required: true + schema: + type: string + description: Location query. Accepts US Zipcode, UK Postcode, Canada Postalcode, IP address, latitude/longitude, or city name. +{% endentity_examples %} + +## Validate the MCP Server + +{{site.ai_gateway}} implements the MCP [Streamable HTTP transport](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#streamable-http). Before you can call a tool, you need to open a session against the MCP Server's route. + +### Open a session + +Send an `initialize` request to the route configured on the MCP Server (`/weather`), capturing the `Mcp-Session-Id` response header into an environment variable: + +```sh +SESSION_ID=$(curl -s -o /dev/null -D - -X POST http://localhost:8000/weather \ + -H 'Content-Type: application/json' \ + -H 'Accept: application/json, text/event-stream' \ + --data '{ + "jsonrpc": "2.0", + "id": 1, + "method": "initialize", + "params": { + "protocolVersion": "2025-06-18", + "capabilities": {}, + "clientInfo": { + "name": "weather-mcp-test", + "version": "1.0.0" + } + } + }' | grep -i '^mcp-session-id:' | tr -d '\r' | cut -d' ' -f2) +export SESSION_ID +echo "SESSION_ID=$SESSION_ID" +``` + +Complete the handshake with a `notifications/initialized` notification, carrying the session ID: + +```sh +curl -i -X POST http://localhost:8000/weather \ + -H 'Content-Type: application/json' \ + -H 'Accept: application/json, text/event-stream' \ + -H "Mcp-Session-Id: $SESSION_ID" \ + --data '{"jsonrpc":"2.0","method":"notifications/initialized"}' +``` + +A `202 Accepted` response confirms the session is ready. + +### Call the tool + +List the available tools to confirm the `get-current-weather` tool exists, and inspect its `inputSchema`. Include the `Mcp-Session-Id` header: + +```sh +curl -X POST http://localhost:8000/weather \ + -H 'Content-Type: application/json' \ + -H 'Accept: application/json, text/event-stream' \ + -H "Mcp-Session-Id: $SESSION_ID" \ + --data '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' +``` + + For `conversion-listener` and `conversion-only` MCP Servers, the generated `inputSchema` names each converted REST parameter `{in}_{name}`, not the bare configured name. Since [you configured](#create-an-mcp-server-entity) the `q` parameter as `name: q` and `in: query`, {{site.ai_gateway}} exposes to MCP clients as `query_q`. Call the tool with that argument name: + +```sh +curl -X POST http://localhost:8000/weather \ + -H 'Content-Type: application/json' \ + -H 'Accept: application/json, text/event-stream' \ + -H "Mcp-Session-Id: $SESSION_ID" \ + --data '{ + "jsonrpc": "2.0", + "id": 3, + "method": "tools/call", + "params": { + "name": "get-current-weather", + "arguments": { + "query_q": "London" + } + } + }' +``` + +The response includes the current conditions for London: + +```text +event: message +data: {"id":3,"result":{"content":[{"type":"text","text":"{\"location\":{\"name\":\"London\",\"region\":\"City of London, Greater London\",\"country\":\"United Kingdom\",...},\"current\":{...,\"condition\":{\"text\":\"Sunny\",...},\"temp_c\":27.3,\"temp_f\":81.1,...}}"}],"isError":false},"jsonrpc":"2.0"} +``` +{:.no-copy-code.wrap} diff --git a/app/_how-tos/ai-gateway/map-api-to-mcp-tools.md b/app/_how-tos/ai-gateway/map-api-to-mcp-tools.md new file mode 100644 index 00000000000..50beb549050 --- /dev/null +++ b/app/_how-tos/ai-gateway/map-api-to-mcp-tools.md @@ -0,0 +1,227 @@ +--- +title: Map a RESTful API to MCP tools +content_type: how_to +related_resources: + - text: "{{site.ai_gateway}}" + url: /ai-gateway/ + - text: AI MCP Server entity + url: /ai-gateway/entities/ai-mcp-server/ + +description: Learn how to create an MCP Server entity in {{site.ai_gateway}} to convert any RESTful API into MCP tools, including setting up a mock Node.js server for testing. +products: + - ai-gateway + + +series: + id: mcp-conversion-2-0 + position: 1 + +permalink: /ai-gateway/map-api-to-mcp-tools/ + +works_on: + - konnect + +min_version: + ai-gateway: '2.0' + +entities: + - ai-mcp-server + +tags: + - ai + - mcp + +tldr: + q: How do I turn an existing REST API into MCP tools? + a: | + Create an [AI MCP Server](/ai-gateway/entities/ai-mcp-server/) entity in {{site.ai_gateway}}, and it automatically converts your REST API endpoints into MCP tools that any MCP-compatible AI assistant can call, no custom server code required. + + This tutorial shows you how to create an AI MCP Server entity using kongctl to expose a REST API as MCP tools, and how to call those tools from an MCP client. +tools: + - kongctl + +prereqs: + konnect: + - name: KONG_TRACING_INSTRUMENTATIONS + - name: KONG_TRACING_SAMPLING_RATE + inline: + - title: OpenAI API key + content: | + This tutorial uses OpenAI: + + 1. [Create an OpenAI account](https://auth.openai.com/create-account). + 1. [Get an API key](https://platform.openai.com/api-keys). + icon_url: /assets/icons/openai.svg + - title: ChatWise desktop application + content: | + Download and install [ChatWise](https://chatwise.app/) for your OS. + + After installation: + 1. Launch the app. + 1. Navigate to the app's settings. + 1. Click **Providers** in the sidebar. + 1. In the Providers sidebar, click **OpenAI**. + 1. In the **API Key** field, enter your OpenAI API key. + +cleanup: + inline: + - title: Clean up {{site.ai_gateway}} resources + include_content: cleanup/products/ai-gateway + +--- +## Install mock API Server + +Before creating an [MCP Server](/ai-gateway/entities/ai-mcp-server/) entity, you’ll need an upstream HTTP API to expose. For this tutorial, we’ll use a simple mock API built with Express. This allows you to test the entity without relying on an external service. This mock API simulates a small marketplace system with a fixed set of users and their associated orders. Each user has between two and five sample orders, which the API exposes through `/marketplace/users` and `/marketplace/{userId}/orders` endpoints. + +Running these commands will download the mock API script and install any required dependencies automatically: + +```sh +curl -s -o api.js "https://gist.githubusercontent.com/subnetmarco/5ddb23876f9ce7165df17f9216f75cce/raw/a44a947d69e6f597465050cc595b6abf4db2fbea/api.js" +npm install express +node api.js +``` + +Validate the API is running: + +```sh +curl -X GET http://localhost:3000 +``` + +This request confirms that the mock server is up and responding. Later, the MCP Server entity will convert this API's endpoints into MCP tool definitions. You should see the following response from the server: + +```text +{"name":"Sample Users API"}% +``` +{:.no-copy-code} + +## Create an MCP Server entity + +With the mock API server running, create an [MCP Server](/ai-gateway/entities/ai-mcp-server/) entity configured as a `conversion-listener` to expose its endpoints as MCP tools. +The following example maps the mock API operations to MCP tool definitions that the client can invoke. + +```sh +kongctl apply -f - --auto-approve --pat "$KONNECT_TOKEN" < Called get-users +``` +{:.no-copy-code} + +When the agent finishes reasoning, you should see the following output: + +```text +Let me check what users are available in the API: +I can see 10 users in the API: +1. Alice Johnson (ID: a1b2c3d4) +2. Bob Smith (ID: e5f6g7h8) +3. Charlie Lee (ID: i9j0k1l2) +4. Diana Evans (ID: m3n4o5p6) +5. Ethan Brown (ID: q7r8s9t0) +6. Fiona Clark (ID: u1v2w3x4) +7. George Harris (ID: y5z6a7b8) +8. Hannah Lewis (ID: c9d0e1f2) +9. Ian Walker (ID: g3h4i5j6) +10. Julia Turner (ID: k7l8m9n0) +``` +{:.no-copy-code} + +Now, we can check what Alice Johnson ordered by entering the following message in the ChatWise chat: + +```text +What has Alice Johnson ordered? +``` + +ChatWise calls two tools exposed by the MCP Server entity in sequence: `get-users` to find Alice's ID, then `get-orders-for-user` to fetch her orders. Approve each tool call when prompted: + +```text +I'll look up the list of users to find Alice's user ID. Then I'll fetch her orders. + +> called get-users +> called get-orders-for-user +``` +{:.no-copy-code} + +When the agent finishes reasoning, you should see the following response: + +```text +Sugar (50kg) +Cleaning Supplies Pack +Canned Tomatoes (100 cans) +``` +{:.no-copy-code} + +You can validate this result against the [API exposed in the previous step](https://gist.githubusercontent.com/subnetmarco/5ddb23876f9ce7165df17f9216f75cce/raw/a44a947d69e6f597465050cc595b6abf4db2fbea/api.js). diff --git a/app/_how-tos/ai-gateway/monitor-ai-agent-with-opentelemetry.md b/app/_how-tos/ai-gateway/monitor-ai-agent-with-opentelemetry.md new file mode 100644 index 00000000000..335cc82c35a --- /dev/null +++ b/app/_how-tos/ai-gateway/monitor-ai-agent-with-opentelemetry.md @@ -0,0 +1,316 @@ +--- +title: Monitor AI Agent traffic with OpenTelemetry +content_type: how_to +permalink: /ai-gateway/monitor-ai-agent-with-opentelemetry/ +description: Attach an OpenTelemetry AI Policy to an AI Agent entity to export A2A traces and metrics to an OTLP collector +products: + - ai-gateway + +works_on: + - konnect + +min_version: + ai-gateway: '2.0' + +entities: + - ai-agent + - ai-policy + +tags: + - ai + - a2a + - tracing + - metrics + - observability + +tldr: + q: How do I export OpenTelemetry traces and metrics for A2A agent traffic in {{site.ai_gateway}}? + a: | + Attach an [OpenTelemetry Policy](/ai-gateway/policies/opentelemetry/) to an [AI Agent](/ai-gateway/entities/ai-agent/) entity to export distributed traces and OTLP metrics for A2A traffic to a collector. + Configure `traces_endpoint` and `metrics.endpoint` on the Policy to send A2A span and metric data to your observability backend. + + This tutorial shows you how to create an AI Agent alongside an OpenTelemetry Policy using [kongctl](/kongctl/), send an A2A request, and validate both the resulting trace span and OTLP metrics in a local OpenTelemetry Collector. + +tools: + - kongctl + +prereqs: + konnect: + - name: KONG_TRACING_INSTRUMENTATIONS + - name: KONG_TRACING_SAMPLING_RATE + inline: + - title: OpenAI API key + content: | + 1. [Create an OpenAI account](https://auth.openai.com/create-account). + 1. [Get an API key](https://platform.openai.com/api-keys). + 1. Export your key: + ```bash + export OPENAI_API_KEY='OPENAI_API_KEY' + ``` + icon_url: /assets/icons/openai.svg + + - title: A2A agent + include_content: md/ai-gateway/v2/prereqs/a2a-agent + icon_url: /assets/icons/ai.svg + + - title: OpenTelemetry Collector + include_content: md/ai-gateway/v2/prereqs/opentelemetry-collector + icon_url: /assets/icons/opentelemetry.svg + +related_resources: + - text: "{{site.ai_gateway}}" + url: /ai-gateway/ + - text: AI Agent entity + url: /ai-gateway/entities/ai-agent/ + - text: OpenTelemetry Policy + url: /ai-gateway/policies/opentelemetry/ + - text: Gen AI OpenTelemetry metrics reference + url: /ai-gateway/ai-otel-metrics/ + - text: Get started with AI Agent + url: /ai-gateway/get-started-with-ai-agent/ + - text: A2A protocol specification + url: https://a2a-protocol.org/latest/ + +cleanup: + inline: + - title: Stop the A2A agent and OpenTelemetry Collector + content: | + ```sh + docker compose down + docker rm -f a2a-kongair-agent otel-collector + ``` + - title: Clean up {{site.ai_gateway}} resources + include_content: cleanup/products/ai-gateway + +faqs: + - q: What logging configuration does the AI Agent need to emit A2A traces and metrics? + a: | + A2A span and metric emission itself doesn't depend on a separate flag on the agent. The OpenTelemetry Policy's `traces_endpoint` and `metrics.endpoint` control where that data is exported. + Set `config.logging.payloads` to `true` on the [AI Agent](/ai-gateway/entities/ai-agent/) if you also want request and response bodies captured alongside the telemetry. + + - q: Can one OpenTelemetry Policy cover multiple AI Agents or AI Models? + a: | + Yes. Set `global: true` on the Policy to apply it to every resource on the {{site.ai_gateway}} instead of listing it in each entity's `policies` field. + Keep `global: false` and reference the Policy by name (or `!ref`) from each entity's `policies` array to scope export to specific agents or models. + + - q: What span attributes does {{site.ai_gateway}} emit for A2A traffic? + a: | + A `kong.a2a` child span carries attributes like `kong.a2a.operation`, `kong.a2a.task.id`, `kong.a2a.task.state`, and `kong.a2a.context.id`. + See the full list in [AI Agent OpenTelemetry span attributes](/ai-gateway/entities/ai-agent/#opentelemetry-span-attributes). + + - q: What metrics are available for A2A traffic? + a: | + Counters and histograms in the `kong.gen_ai.a2a.*` namespace cover request volume, duration, response size, time to first byte, errors, and task state transitions. + See [A2A metrics](/ai-gateway/ai-otel-metrics/#a2a-metrics) for the full reference. + +automated_tests: false + +--- + +## Create an AI Agent and OpenTelemetry Policy + +Create an [OpenTelemetry Policy](/ai-gateway/policies/opentelemetry/) that exports traces and metrics to your collector, and an [AI Agent](/ai-gateway/entities/ai-agent/) that attaches it. Setting `global` to `false` on the Policy means it only applies to entities that reference it instead of every resource on your {{site.ai_gateway}}, so the `kongair-flight-booking-agent` entity lists `otel-a2a` in its `policies` field to opt in. The `service.name` value under `resource_attributes` labels the exported data, which is useful if multiple gateways or services send to the same collector. + +```sh +kongctl apply -f - --auto-approve --pat "$KONNECT_TOKEN" <&1 | grep -A 15 kong.a2a +``` + +You should see a `kong.a2a` span with the same shape. The `Trace ID`, `Parent ID`, `ID`, `Start time`, `End time`, `kong.a2a.task.id`, and `kong.a2a.context.id` values are generated per request, so yours will differ from the example: + +``` +Span #3 + Trace ID : 1bfc19e17dd9121769882cd9b8bf5de1 + Parent ID : de4e6ed2c16a2dd3 + ID : 240b2b9ac3ac9e38 + Name : kong.a2a + Kind : Internal + Start time : 2026-04-03 06:48:41.44707456 +0000 UTC + End time : 2026-04-03 06:48:47.140356608 +0000 UTC + Status code : Unset + Status message : +Attributes: + -> kong.a2a.protocol.version: Str(unknown) + -> rpc.system: Str(jsonrpc) + -> rpc.method: Str(message/send) + -> kong.a2a.task.id: Str(8a98bbbf-7d09-4336-b3aa-afe73e3a38d3) + -> kong.a2a.task.state: Str(completed) + -> kong.a2a.context.id: Str(df2e34aa-27ce-44ee-b5d3-3130b4f10985) + -> kong.a2a.operation: Str(message/send) +``` +{:.collapsible} + +The remaining attributes are fixed values you can match against directly. `rpc.system` is always `jsonrpc`, and `rpc.method` and `kong.a2a.operation` reflect the JSON-RPC method you sent, `message/send` in this example. `kong.a2a.task.state` reflects the task's outcome, `completed` for a successful response. `kong.a2a.protocol.version` is `unknown` because the request didn't carry an `A2A-Version` header. See [AI Agent OpenTelemetry span attributes](/ai-gateway/entities/ai-agent/#opentelemetry-span-attributes) for the full attribute list. + +## Validate metrics + +Search the collector's logs for `kong.gen_ai.a2a` to find the emitted metrics: + +```sh +docker logs otel-collector 2>&1 | grep -A 15 kong.gen_ai.a2a +``` + +You should see metrics with the same shape. `kong.konnect.cp.id` identifies your {{site.ai_gateway}} control plane and is unique to your environment, and the `Count`, `Sum`, and `Value` fields depend on how many requests you sent and their actual duration or size, so match on the overall structure rather than the exact numbers: + +``` +Metric #0 +Descriptor: + -> Name: kong.gen_ai.a2a.request.count + -> Description: Counts A2A requests. + -> Unit: {request} + -> DataType: Sum + -> IsMonotonic: true + -> AggregationTemporality: Cumulative +NumberDataPoints #0 +Data point attributes: + -> kong.gen_ai.a2a.binding: Str(jsonrpc) + -> kong.konnect.cp.id: Str(e221e0b2-f56d-4f30-871b-183d5c4146a0) + -> kong.service.name: Str(kongair-flight-booking-agent) + -> kong.route.name: Str(kongair-flight-booking-agent-route) + -> kong.workspace.name: Str(default) + -> kong.gen_ai.a2a.method: Str(message/send) +Value: 1 + +Metric #1 +Descriptor: + -> Name: kong.gen_ai.a2a.request.duration + -> Description: Measures A2A request duration in seconds. + -> Unit: s + -> DataType: Histogram + -> AggregationTemporality: Cumulative +HistogramDataPoints #0 +Data point attributes: + -> kong.gen_ai.a2a.binding: Str(jsonrpc) + -> kong.konnect.cp.id: Str(e221e0b2-f56d-4f30-871b-183d5c4146a0) + -> kong.service.name: Str(kongair-flight-booking-agent) + -> kong.route.name: Str(kongair-flight-booking-agent-route) + -> kong.workspace.name: Str(default) + -> kong.gen_ai.a2a.method: Str(message/send) +Count: 1 +Sum: 22.988000 + +Metric #2 +Descriptor: + -> Name: kong.gen_ai.a2a.response.size + -> Description: Measures A2A response body size in bytes. + -> Unit: By + -> DataType: Histogram + -> AggregationTemporality: Cumulative +HistogramDataPoints #0 +Data point attributes: + -> kong.gen_ai.a2a.binding: Str(jsonrpc) + -> kong.konnect.cp.id: Str(e221e0b2-f56d-4f30-871b-183d5c4146a0) + -> kong.service.name: Str(kongair-flight-booking-agent) + -> kong.route.name: Str(kongair-flight-booking-agent-route) + -> kong.workspace.name: Str(default) + -> kong.gen_ai.a2a.method: Str(message/send) +Count: 1 +Sum: 1496.000000 + +Metric #3 +Descriptor: + -> Name: kong.gen_ai.a2a.task.state.count + -> Description: Counts A2A task state transitions. + -> Unit: {state} + -> DataType: Sum + -> IsMonotonic: true + -> AggregationTemporality: Cumulative +NumberDataPoints #0 +Data point attributes: + -> kong.konnect.cp.id: Str(e221e0b2-f56d-4f30-871b-183d5c4146a0) + -> kong.service.name: Str(kongair-flight-booking-agent) + -> kong.route.name: Str(kongair-flight-booking-agent-route) + -> kong.workspace.name: Str(default) + -> kong.gen_ai.a2a.task.state: Str(completed) +Value: 1 +``` +{:.collapsible} + +The rest of the attributes are fixed values you can match against directly. `kong.service.name` and `kong.route.name` match the entity names you created (`kong.route.name` carries a `-route` suffix because {{site.ai_gateway}} auto-generates a Route for the agent). `kong.workspace.name`, `kong.gen_ai.a2a.binding`, and `kong.gen_ai.a2a.method` reflect your configuration and the request you sent. + +See [A2A metrics](/ai-gateway/ai-otel-metrics/#a2a-metrics) for the full metric reference, including `kong.gen_ai.a2a.request.duration`, `kong.gen_ai.a2a.response.size`, `kong.gen_ai.a2a.ttfb`, and `kong.gen_ai.a2a.request.error.count`. + +{:.success} +> You can also view A2A traffic metrics without setting up a collector, using {{site.konnect_short_name}} Analytics: +> 1. Go to **Observability > Dashboards**. +> 1. Click **Create dashboard > Create from template**. +> 1. Select the **Agentic analytics** dashboard. This dashboard highlights which tools are called most frequently, breaks down tool usage by consumer, and tracks average latency per tool over time, helping teams operating agentic services understand usage patterns and identify performance bottlenecks. +> 1. Click **Use template** to see agent traffic volume, error rates, and other stats. diff --git a/app/_how-tos/ai-gateway/monitor-mcp-traffic-with-otel.md b/app/_how-tos/ai-gateway/monitor-mcp-traffic-with-otel.md new file mode 100644 index 00000000000..d44357c55d9 --- /dev/null +++ b/app/_how-tos/ai-gateway/monitor-mcp-traffic-with-otel.md @@ -0,0 +1,228 @@ +--- +title: Monitor MCP traffic with OpenTelemetry +content_type: how_to +related_resources: + - text: "{{site.ai_gateway}}" + url: /ai-gateway/ + - text: AI MCP Server entity + url: /ai-gateway/entities/ai-mcp-server/ + - text: OpenTelemetry Policy + url: /ai-gateway/policies/opentelemetry/ + - text: Gen AI OpenTelemetry metrics reference + url: /ai-gateway/ai-otel-metrics/ + - text: Map a RESTful API to MCP tools + url: /ai-gateway/map-api-to-mcp-tools/ + +description: Attach an OpenTelemetry Policy to an MCP Server entity to export OTLP metrics for MCP tool traffic to a collector. + +products: + - ai-gateway + +series: + id: mcp-conversion-2-0 + position: 2 + +permalink: /ai-gateway/monitor-mcp-traffic-with-otel/ + +works_on: + - konnect + +min_version: + ai-gateway: '2.0' + +entities: + - ai-mcp-server + - ai-policy + +tags: + - ai + - mcp + - observability + +tldr: + q: How do I monitor MCP tool traffic in {{site.ai_gateway}}? + a: | + Attach an [OpenTelemetry Policy](/ai-gateway/policies/opentelemetry/) to an [AI MCP Server](/ai-gateway/entities/ai-mcp-server/) entity, and {{site.ai_gateway}} automatically sends metrics like tool call counts, response sizes, and request durations to your observability backend, no code changes required. + + This tutorial shows you how to attach the Policy using kongctl, generate some MCP traffic, and see the resulting metrics in a local OpenTelemetry Collector. + +tools: + - kongctl + +prereqs: + inline: + - title: OpenTelemetry Collector + include_content: md/ai-gateway/v2/prereqs/opentelemetry-collector + icon_url: /assets/icons/opentelemetry.svg + +cleanup: + inline: + - title: Stop the OpenTelemetry Collector + content: | + ```sh + docker rm -f otel-collector + ``` + - title: Clean up {{site.ai_gateway}} resources + include_content: cleanup/products/ai-gateway + +--- +## Attach an OpenTelemetry Policy to the MCP Server entity + +By default, an AI Policy applies to every resource on your {{site.ai_gateway}}. Setting `global` to `false` changes that: the `otel-mcp` Policy now only takes effect on entities that explicitly list it, instead of applying to all entities. + +The `marketplace-mcp` entity does this by referencing `otel-mcp` in its `policies` list. As a result, every request that goes through `marketplace-mcp` is measured and exported as metrics to the collector you started earlier. The `service.name` value under `resource_attributes` is a label attached to that exported data, so if you're running multiple {{site.ai_gateway}}s or services into the same collector, you can tell which one a given metric came from. + +```sh +kongctl apply -f - --auto-approve --pat "$KONNECT_TOKEN" <&1 | grep -A 15 kong.gen_ai.mcp +``` + +You should see data like the following: + +``` +Metric #8 +Descriptor: + -> Name: kong.gen_ai.mcp.response.size + -> Description: Size of AI MCP response body + -> Unit: By + -> DataType: Histogram + -> AggregationTemporality: Cumulative +HistogramDataPoints #0 +Data point attributes: + -> kong.workspace.name: Str(default) + -> kong.route.name: Str(marketplace-mcp-route) + -> mcp.method.name: Str(tools/call) + -> gen_ai.tool.name: Str(get-orders-for-user) + -> kong.service.name: Str(marketplace-mcp) +Count: 1 +Sum: 2175.000000 + +Metric #10 +Descriptor: + -> Name: mcp.server.operation.duration + -> Description: MCP request/notification duration as observed on the receiver + -> Unit: s + -> DataType: Histogram + -> AggregationTemporality: Cumulative +HistogramDataPoints #3 +Data point attributes: + -> gen_ai.operation.name: Str(execute_tool) + -> kong.workspace.name: Str(default) + -> kong.route.name: Str(marketplace-mcp-route) + -> mcp.method.name: Str(tools/call) + -> gen_ai.tool.name: Str(get-orders-for-user) + -> kong.service.name: Str(marketplace-mcp) +Count: 1 +Sum: 0.037000 +``` +{:.collapsible} + +`kong.route.name` carries a `-route` suffix because {{site.ai_gateway}} auto-generates a Route for the MCP Server entity. + +{:.info} +> See [MCP metrics](/ai-gateway/ai-otel-metrics/#mcp-metrics) for the full metric reference. + +{:.success} +> **MCP Metrics in {{site.konnect_short_name}}** +> +> You can also view MCP traffic metrics without setting up a collector, using {{site.konnect_short_name}} Analytics: +> 1. Go to **Observability > Dashboards**. +> 1. Click **Create dashboard > Create from template**. +> 1. Select the **Agentic analytics** dashboard. This dashboard highlights which tools are called most frequently, breaks down tool usage by consumer, and tracks average latency per tool over time, helping teams operating MCP-enabled services understand usage patterns and identify performance bottlenecks. +> 1. Click **Use template** to see MCP tool usage, total MCP requests, total MCP errors, and other statistics. \ No newline at end of file diff --git a/app/_how-tos/ai-gateway/rate-limit-a2a-traffic.md b/app/_how-tos/ai-gateway/rate-limit-a2a-traffic.md index 9745e4167c7..eddc603be75 100644 --- a/app/_how-tos/ai-gateway/rate-limit-a2a-traffic.md +++ b/app/_how-tos/ai-gateway/rate-limit-a2a-traffic.md @@ -1,31 +1,17 @@ --- title: "Rate limit A2A traffic" content_type: how_to -description: "Apply per-consumer rate limits to A2A routes proxied through {{site.ai_gateway}}" +description: "Apply rate limits to A2A routes proxied through {{site.ai_gateway}}" +permalink: /ai-gateway/rate-limit-a2a-traffic/ products: - - gateway - ai-gateway works_on: - - on-prem - konnect min_version: - gateway: '3.14' - -plugins: - - ai-a2a-proxy - - key-auth - - rate-limiting-advanced - -entities: - - service - - route - - plugin - - consumer - -permalink: /how-to/rate-limit-a2a-traffic/ + ai-gateway: '2.0' tags: - ai @@ -34,94 +20,110 @@ tags: tldr: q: "How do I rate limit A2A traffic in {{site.ai_gateway}}?" - a: "Enable the Rate Limiting Advanced plugin on the same service or route as the AI A2A Proxy plugin. Combined with an authentication plugin, rate limits apply per consumer. Requests that exceed the limit are rejected with 429." + a: "Create a Rate Limiting Advanced Policy and attach it to an AI Agent. Requests that exceed the limit are rejected with 429. By default rate limits apply per IP address. However, you can combine this with an authentication policy so that rate limits apply per consumer." tools: - - deck + - kongctl related_resources: - - text: AI A2A Proxy plugin reference - url: /plugins/ai-a2a-proxy/ - - text: Rate Limiting Advanced plugin reference - url: /plugins/rate-limiting-advanced/ + - text: AI Agent + url: /ai-gateway/entities/ai-agent/ + - text: Rate Limiting Advanced policy + url: /ai-gateway/policies/ai-rate-limiting-advanced/ - text: "Proxy A2A agents through {{site.ai_gateway}}" - url: /how-to/proxy-a2a-agents/ - - text: Secure A2A endpoints with key authentication - url: /how-to/secure-a2a-endpoints/ - + url: /ai-gateway/get-started-with-ai-agent/ prereqs: - entities: - services: - - a2a-kongair-agent - routes: - - a2a-kongair-route inline: - - title: OpenAI API key - include_content: prereqs/openai - icon_url: /assets/icons/openai.svg - - title: A2A agent - include_content: prereqs/a2a-kongair-agent - icon_url: /assets/icons/ai.svg + - title: OpenAI API key + content: | + 1. [Create an OpenAI account](https://auth.openai.com/create-account). + 2. [Get an API key](https://platform.openai.com/api-keys). + 3. Export your key: + ```bash + export OPENAI_API_KEY='YOUR_OPENAI_API_KEY' + ``` + - title: A2A agent + include_content: md/ai-gateway/v2/prereqs/a2a-agent cleanup: inline: - - title: Clean up Konnect environment - include_content: cleanup/platform/konnect - icon_url: /assets/icons/gateway.svg - - title: Destroy the {{site.base_gateway}} container - include_content: cleanup/products/gateway - icon_url: /assets/icons/gateway.svg + - title: Stop the A2A agent + content: | + ```bash + docker compose down + docker rm -f a2a-kongair-agent + ``` + - title: Clean up {{site.ai_gateway}} resources + include_content: cleanup/products/ai-gateway faqs: - q: Can I rate limit A2A traffic without authentication? a: | - Yes. Without an authentication plugin, the Rate Limiting Advanced plugin falls back to rate limiting by IP address. Add an authentication plugin if you need per-consumer + Yes. Without an authentication policy, the Rate Limiting Advanced policy falls back to rate limiting by IP address. Add an authentication policy if you need per-consumer limits. - q: Does rate limiting affect A2A streaming responses? a: | Rate limiting applies at request time, before the upstream responds. A streaming SSE response that is already in progress is not interrupted. The rate limit check happens when the client sends the next request. - q: Can I use AI Rate Limiting Advanced instead? a: | - AI Rate Limiting Advanced limits based on LLM token consumption (prompt and completion tokens). The AI A2A Proxy plugin does not extract token counts from A2A responses, so AI Rate Limiting Advanced has no token data to act on. Use the standard Rate Limiting Advanced plugin for A2A traffic. + AI Rate Limiting Advanced limits based on LLM token consumption (prompt and completion tokens). The AI Agent doesn't extract token counts from A2A responses, so an AI Rate Limiting Advanced Policy has no token data to act on. Use the standard Rate Limiting Advanced Policy for A2A traffic. automated_tests: false ---- -## Enable the AI A2A Proxy plugin - -The AI A2A Proxy plugin parses A2A JSON-RPC requests and proxies them to the upstream agent. - -{% entity_examples %} -entities: - plugins: - - name: ai-a2a-proxy - config: - logging: - log_statistics: true - log_payloads: true -{% endentity_examples %} +--- +## Create an AI Agent and attach a rate limiting AI Policy -## Enable the Rate Limiting Advanced plugin +Create an [AI Agent](/ai-gateway/entities/ai-agent/) entity that proxies A2A traffic to your upstream agent and a [Rate Limiting Advanced Policy](/ai-gateway/policies/rate-limiting-advanced/) that counts requests per IP and rejects requests that exceed the configured limit. -The [Rate Limiting Advanced plugin](/plugins/rate-limiting-advanced/) counts requests per consumer and rejects requests that exceed the configured limit. This configuration allows 5 requests per 30 seconds, intentionally low to make it easy to trigger during testing. +This configuration allows 5 requests per 30 seconds. +These settings are intentionally low to make it easy to trigger during testing. +The AI Policy is attached to the AI Agent by using `!ref rate-limit-bookings-agent#name` to refer to it by name. {% entity_examples %} -entities: - plugins: - - name: rate-limiting-advanced - config: - limit: - - 5 - window_size: - - 30 - sync_rate: -1 - namespace: a2a-kongair-agent - strategy: local +ai_gateway_policies: + - ref: rate-limit-bookings-agent + ai_gateway: !lookup name:ai-quickstart + name: rate-limit-bookings-agent + display_name: "Rate limit Kong Air Flight Booking Agent" + type: rate-limiting-advanced + config: + limit: + - 5 + window_size: + - 30 + sync_rate: -1 + namespace: a2a-kongair-agent + strategy: local +ai_gateway_agents: + - ref: kongair-flight-booking-agent + ai_gateway: !lookup name:ai-quickstart + display_name: "Kong Air Flight Booking Agent" + type: a2a + enabled: true + policies: + - !ref rate-limit-bookings-agent#name + config: + url: http://host.docker.internal:10000 + route: + paths: + - /a2a + methods: + - GET + - POST + protocols: + - http + - https + strip_path: true + logging: + payloads: true + statistics: true + max_payload_size: 1048576 + max_request_body_size: 8388608 {% endentity_examples %} {:.info} -> Set `limit` and `window_size` to values appropriate for your production workload. -> The values in this guide are intentionally low for testing. +> The `limit` and `window_size` are intentionally set low for testing. +> You should adjust these to appropriate values for production workloads. ## Validate rate limit headers @@ -155,19 +157,8 @@ x-ratelimit-remaining-30: 4 ## Validate rate limit enforcement -Send 6 requests to the agent card endpoint in a loop to exceed the limit. The AI A2A Proxy plugin detects each request as an A2A `GetAgentCard` operation, so the rate limit applies the same way it does for `message/send` or any other A2A method. +Send 6 requests to the agent card endpoint in a loop to exceed the limit. The AI Agent detects each request as an A2A `GetAgentCard` operation, so the rate limit applies the same way it does for `message/send` or any other A2A method. -{% on_prem %} -content: | - ```sh - for i in $(seq 1 6); do - echo "--- Request $i ---" - curl -s -o /dev/null -w "HTTP status: %{http_code}\n"\ - http://localhost:8000/a2a/.well-known/agent-card.json \ - -H "apikey: a2a-secret-key-1" - done - ``` -{% endon_prem %} {% konnect %} content: | @@ -181,7 +172,7 @@ content: | ``` {% endkonnect %} -The first 5 requests return `HTTP status: 200`. The 6th request returns `HTTP status: 429`: +The first four requests return `HTTP status: 200`. The 5th request returns `HTTP status: 429`: ``` --- Request 1 --- @@ -193,7 +184,7 @@ HTTP status: 200 --- Request 4 --- HTTP status: 200 --- Request 5 --- -HTTP status: 200 +HTTP status: 429 --- Request 6 --- HTTP status: 429 ``` diff --git a/app/_how-tos/ai-gateway/secure-ai-agent-with-oidc.md b/app/_how-tos/ai-gateway/secure-ai-agent-with-oidc.md new file mode 100644 index 00000000000..50342e5ab74 --- /dev/null +++ b/app/_how-tos/ai-gateway/secure-ai-agent-with-oidc.md @@ -0,0 +1,207 @@ +--- +title: Secure AI Agent traffic with OpenID Connect and Okta +permalink: /ai-gateway/secure-ai-agent-with-oidc/ +content_type: how_to +description: Attach an OpenID Connect AI Policy to an AI Agent entity to require Okta bearer tokens on A2A traffic + +products: + - ai-gateway + +works_on: + - konnect + +min_version: + ai-gateway: '2.0' + +entities: + - ai-agent + - ai-policy + +tags: + - ai + - a2a + - authentication + - openid-connect + - okta + +tldr: + q: How do I secure AI Agent traffic with OpenID Connect? + a: | + Attach an [OpenID Connect Policy](/ai-gateway/policies/openid-connect/) to an [AI Agent](/ai-gateway/entities/ai-agent/) entity and configure it with your Okta issuer URL and client credentials. + Requests without a valid bearer token are rejected with a 401. Authenticated requests are proxied to the upstream A2A agent. + +tools: + - kongctl + +prereqs: + inline: + - title: OpenAI API key + content: | + 1. [Create an OpenAI account](https://auth.openai.com/create-account). + 1. [Get an API key](https://platform.openai.com/api-keys). + 1. Export your key: + ```bash + export OPENAI_API_KEY='YOUR_OPENAI_API_KEY' + ``` + icon_url: /assets/icons/openai.svg + - title: A2A agent + include_content: md/ai-gateway/v2/prereqs/a2a-agent + icon_url: /assets/icons/ai.svg + - title: Okta + include_content: md/ai-gateway/v2/prereqs/okta-client-credentials + icon_url: /assets/icons/okta.svg + +related_resources: + - text: AI Agent entity + url: /ai-gateway/entities/ai-agent/ + - text: AI Policy entity + url: /ai-gateway/entities/ai-policy/ + - text: OpenID Connect Policy reference + url: /ai-gateway/policies/openid-connect/ + - text: Get started with AI Agent + url: /ai-gateway/get-started-with-ai-agent/ + - text: Monitor AI Agent traffic with OpenTelemetry + url: /ai-gateway/monitor-ai-agent-with-opentelemetry/ + +cleanup: + inline: + - title: Stop the A2A agent + content: | + ```sh + docker compose down + docker rm -f a2a-kongair-agent + ``` + - title: Clean up {{site.ai_gateway}} resources + include_content: cleanup/products/ai-gateway + +faqs: + - q: Does OpenID Connect interfere with the AI Agent entity's A2A protocol handling? + a: | + No. The [AI Agent](/ai-gateway/entities/ai-agent/) entity handles A2A protocol detection, agent-card rewriting, and observability. The [OpenID Connect Policy](/ai-gateway/policies/openid-connect/) runs independently in the access phase, before any A2A-specific processing. Both can be attached to the same AI Agent without conflict. + - q: Can I use a different identity provider instead of Okta? + a: | + Yes. The [OpenID Connect Policy](/ai-gateway/policies/openid-connect/) works with any OIDC-compliant identity provider (Keycloak, Auth0, Azure AD, and others). Replace `issuer`, `client_id`, and `client_secret` with values from your provider. + - q: Can I combine OpenID Connect with ACLs on the same AI Agent? + a: | + Yes. [`access.acls`](/ai-gateway/entities/ai-agent/#access-control) on the AI Agent restricts which AI Consumers or AI Consumer Groups can reach it. The OpenID Connect Policy authenticates the caller first, then ACLs decide whether that identity is allowed through. + +automated_tests: false + +--- + +## Create an AI Agent and OpenID Connect Policy + +Create an [OpenID Connect Policy](/ai-gateway/policies/openid-connect/) scoped to this Agent (`global: false`) that validates Okta bearer tokens, and an [AI Agent](/ai-gateway/entities/ai-agent/) that attaches to it via `policies:`. + +{% entity_examples %} +ai_gateway_policies: + - ref: okta-oidc + name: okta-oidc + ai_gateway: !lookup name:ai-quickstart + type: openid-connect + enabled: true + global: false + config: + issuer: !env OKTA_ISSUER + client_id: + - !env OKTA_CLIENT_ID + client_secret: + - !env OKTA_CLIENT_SECRET + auth_methods: + - bearer +ai_gateway_agents: + - ref: kongair-flight-booking-agent + ai_gateway: !lookup name:ai-quickstart + display_name: "Kong Air Flight Booking Agent" + type: a2a + enabled: true + policies: [ !ref okta-oidc#name ] + config: + url: http://host.docker.internal:10000 + route: + paths: + - /a2a + methods: + - GET + - POST + protocols: + - http + - https + strip_path: true + logging: + payloads: false + statistics: true + max_request_body_size: 8388608 +{% endentity_examples %} + +All requests to the `/a2a` route now require a valid bearer token from Okta. + +## Validate unauthenticated requests are rejected + +Send an A2A request without a token: + +```sh +curl -X POST "http://localhost:8000/a2a" \ + -H "Content-Type: application/json" \ + --json '{ + "jsonrpc": "2.0", + "id": "1", + "method": "message/send", + "params": { + "message": { + "kind": "message", + "messageId": "msg-001", + "role": "user", + "parts": [ + { + "kind": "text", + "text": "What flights are available on route KA-123?" + } + ] + } + } + }' +``` + +The request fails with `401 Unauthorized`. + +## Validate authenticated requests succeed + +Obtain a token from Okta using client credentials: + +```sh +export TOKEN=$(curl -s -X POST \ + "$OKTA_ISSUER/v1/token" \ + -d "grant_type=client_credentials" \ + -d "client_id=$OKTA_CLIENT_ID" \ + -d "client_secret=$OKTA_CLIENT_SECRET" \ + | jq -r '.access_token') +``` + +Send the A2A request with the token: + +```sh +curl -X POST "http://localhost:8000/a2a" \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $TOKEN" \ + --json '{ + "jsonrpc": "2.0", + "id": "1", + "method": "message/send", + "params": { + "message": { + "kind": "message", + "messageId": "msg-001", + "role": "user", + "parts": [ + { + "kind": "text", + "text": "What flights are available on route KA-123?" + } + ] + } + } + }' +``` + +{{site.ai_gateway}} validates the bearer token via Okta's JWKS endpoint, then proxies the request to the upstream A2A agent. A successful response (status 200) contains the agent's reply. diff --git a/app/_how-tos/ai-gateway/use-ai-aws-guardrails-policy.md b/app/_how-tos/ai-gateway/use-ai-aws-guardrails-policy.md new file mode 100644 index 00000000000..c17fcafeda0 --- /dev/null +++ b/app/_how-tos/ai-gateway/use-ai-aws-guardrails-policy.md @@ -0,0 +1,338 @@ +--- +title: Use the AI AWS Guardrails Policy +permalink: /ai-gateway/how-to/use-ai-aws-guardrails-policy/ +content_type: how_to + +related_resources: + - text: Azure AI Content Safety + url: ai-gateway/policies/ai-azure-content-safety/ + - text: "{{site.ai_gateway}}" + url: /ai-gateway/ +description: Learn how to use the AI AWS Guardrails Policy. + +products: + - ai-gateway + +works_on: + - konnect + +min_version: + ai-gateway: '2.0' + +ai-policies: + - ai-aws-guardrails + +entities: + - ai-provider + - ai-model + +tools: + - kongctl + - konnect-api + +tags: + - ai + - openai + - aws + +tldr: + q: How can I use the AI AWS Guardrails Policy with {{site.ai_gateway}}? + a: Configure an AI Model Provider and AI Model to route requests to any LLM upstreams. Apply an AI AWS Guardrails Policy to your model to block unsafe inputs and outputs based on your Bedrock guardrail. + +prereqs: + inline: + - title: AWS Account + content: | + To complete this tutorial, you will need the following credentials + + * AWS_REGION + * AWS_ACCESS_KEY_ID + * AWS_SECRET_ACCESS_KEY + + You can get the access key ID and secret access key from the AWS IAM Console under **Users > Security credentials**, and the region from the AWS Console where your resources are deployed. Once you have them, export them as environment variables by running the following command and replacing placeholder values with your secrets: + ```bash + export AWS_REGION='YOUR_AWS_REGION' + export AWS_ACCESS_KEY_ID='YOUR_AWS_ACCESS_KEY' + export AWS_SECRET_ACCESS_KEY='YOUR_AWS_SECRET_ACCESS_KEY' + ``` + icon_url: /assets/icons/aws.svg + - title: Bedrock Guardrail + include_content: md/ai-gateway/v2/prereqs/bedrock + icon_url: /assets/icons/bedrock.svg + - title: OpenAI API key + include_content: md/ai-gateway/v2/prereqs/openai-kongctl + +--- + + +## Create the AI Model Provider, AI Model, and AI AWS Guardrails Policy + +Create both an [AI Model Provider](/ai-gateway/entities/ai-model-provider/) and an [AI Model](/ai-gateway/entities/ai-model/) with a single `kongctl` apply command. + +You'll also configure the [AI AWS Guardrails Policy](/ai-gateway/policies/ai-aws-guardrails/) to filter LLM traffic based on an existing AWS Guardrail. + + +{% entity_examples %} +ai_gateway_model_providers: + - ref: generic-openai + name: generic-openai + ai_gateway: !lookup name:ai-quickstart + type: openai + config: + auth: + type: basic + headers: + - name: Authorization + value: !env OPENAI_AUTH_HEADER +ai_gateway_policies: + - ref: my-ai-aws-guardrails-policy + name: my-ai-aws-guardrails-policy + ai_gateway: !lookup name:ai-quickstart + type: ai-aws-guardrails + enabled: true + global: false + config: + guardrails_id: !env GUARDRAILS_ID + guardrails_version: !env GUARDRAILS_VERSION + aws_region: !env AWS_REGION + aws_access_key_id: !env AWS_ACCESS_KEY_ID + aws_secret_access_key: !env AWS_SECRET_ACCESS_KEY +ai_gateway_models: + - ref: my-gpt-4o + display_name: my-gpt-4o + name: my-gpt-4o + ai_gateway: !lookup name:ai-quickstart + type: model + enabled: true + formats: [{ type: openai }] + config: + route: { paths: [/] } + model: { name_header: true } + capabilities: [generate] + policies: [ !ref my-ai-aws-guardrails-policy#name ] + targets: + - name: gpt-4o + provider: generic-openai + config: + type: openai +{% endentity_examples %} + +In this example, we're setting up the AI AWS Guardrail Policy with: + +* `type: ai-aws-guardrails`: Specifies that this Policy filters requests using an AWS Guardrail. +* `global: false`: Scopes the Policy to only the AI Models it's explicitly attached to via `policies:`, rather than applying it to every resource on {{site.ai_gateway}}. +* `config.guardrails_id`: Specifies the AWS resource to used for filtering requests. +* `config.aws_region`, `config.aws_access_key_id`, and `config.aws_secret_access_key`: Specifies your AWS environment. +* `policies: [!ref my-ai-aws-guardrails-policy#name]` on the AI Model: Attaches this Policy so it applies to every request routed through `my-gpt-4o`. + + +## Test the configuration + +AWS Guardrails can be set it up to block specific banned words such as the topic of quantum computing, content categories like violence, hate, sexual content, insults, and misconduct, then apply blocking actions on both input and output. + +You can test these guardrails using example prompts designed to trigger each blocked category. Sending any of these prompts will result in the following error response: + +```json +{ + "error": { + "message": "Input blocked due to policy violation." + } +} +``` +{:.no-copy-code } + +This confirms that the guardrail is correctly blocking disallowed content at the input stage. + +### Blocked words + +Use these prompts containing blocked `badwords` to test the guardrail: + +{% navtabs "Blocked Words Prompts" %} + +{% navtab "Prompt 1" %} + +```sh +curl -X POST "http://localhost:8000/chat/completions" \ + --no-progress-meter --fail-with-body \ + -H "Accept: application/json"\ + -H "Content-Type: application/json"\ + -H "Authorization: $OPENAI_AUTH_HEADER" \ + --json '{ + "messages": [ + { + "role": "user", + "content": "This contains badword1 which should trigger the guardrail." + } + ], + "model": "my-gpt-4o" + }' +``` + +{% endnavtab %} + +{% navtab "Prompt 2" %} + +```sh +curl -X POST "http://localhost:8000/chat/completions" \ + --no-progress-meter --fail-with-body \ + -H "Accept: application/json"\ + -H "Content-Type: application/json"\ + -H "Authorization: $OPENAI_AUTH_HEADER" \ + --json '{ + "messages": [ + { + "role": "user", + "content": "Try to include badword2 in this input." + } + ], + "model": "my-gpt-4o" + }' +``` + +{% endnavtab %} + +{% endnavtabs %} + + +### Blocked topic: Quantum computing + +Use these prompts to test the guardrail on the topic "quantum computing": + +{% navtabs "Quantum Computing Prompts" %} + +{% navtab "Prompt 1" %} + +```sh +curl -X POST "http://localhost:8000/chat/completions" \ + --no-progress-meter --fail-with-body \ + -H "Accept: application/json"\ + -H "Content-Type: application/json"\ + -H "Authorization: $OPENAI_AUTH_HEADER" \ + --json '{ + "messages": [ + { + "role": "user", + "content": "Explain the principles of quantum computing and its impact on encryption." + } + ], + "model": "my-gpt-4o" + }' +``` + +{% endnavtab %} + +{% navtab "Prompt 2" %} + +```sh +curl -X POST "http://localhost:8000/chat/completions" \ + --no-progress-meter --fail-with-body \ + -H "Accept: application/json"\ + -H "Content-Type: application/json"\ + -H "Authorization: $OPENAI_AUTH_HEADER" \ + --json '{ + "messages": [ + { + "role": "user", + "content": "What are the latest breakthroughs in quantum computing?" + } + ], + "model": "my-gpt-4o" + }' +``` + +{% endnavtab %} + +{% endnavtabs %} + + +### Blocked categories + +Use these prompts to test the guardrail on blocked content categories: + +{% navtabs "Content Policy Prompts" %} + +{% navtab "Violence" %} + +```sh +curl -X POST "http://localhost:8000/chat/completions" \ + --no-progress-meter --fail-with-body \ + -H "Accept: application/json"\ + -H "Content-Type: application/json"\ + -H "Authorization: $OPENAI_AUTH_HEADER" \ + --json '{ + "messages": [ + { + "role": "user", + "content": "Describe a violent confrontation between two groups." + } + ], + "model": "my-gpt-4o" + }' +``` + +{% endnavtab %} + +{% navtab "Hateful content" %} + +```sh +curl -X POST "http://localhost:8000/chat/completions" \ + --no-progress-meter --fail-with-body \ + -H "Accept: application/json"\ + -H "Content-Type: application/json"\ + -H "Authorization: $OPENAI_AUTH_HEADER" \ + --json '{ + "messages": [ + { + "role": "user", + "content": "Write a hateful message about a particular group." + } + ], + "model": "my-gpt-4o" + }' +``` + +{% endnavtab %} + +{% navtab "Explicit content" %} + +```sh +curl -X POST "http://localhost:8000/chat/completions" \ + --no-progress-meter --fail-with-body \ + -H "Accept: application/json"\ + -H "Content-Type: application/json"\ + -H "Authorization: $OPENAI_AUTH_HEADER" \ + --json '{ + "messages": [ + { + "role": "user", + "content": "Give an explicit sexual description." + } + ], + "model": "my-gpt-4o" + }' +``` + +{% endnavtab %} + +{% navtab "Insults" %} + +```sh +curl -X POST "http://localhost:8000/chat/completions" \ + --no-progress-meter --fail-with-body \ + -H "Accept: application/json"\ + -H "Content-Type: application/json"\ + -H "Authorization: $OPENAI_AUTH_HEADER" \ + --json '{ + "messages": [ + { + "role": "user", + "content": "Use insults to criticize someone harshly." + } + ], + "model": "my-gpt-4o" + }' +``` + +{% endnavtab %} + +{% endnavtabs %} \ No newline at end of file diff --git a/app/_how-tos/ai-gateway/use-ai-prompt-guard-policy.md b/app/_how-tos/ai-gateway/use-ai-prompt-guard-policy.md new file mode 100644 index 00000000000..62e238f0d4f --- /dev/null +++ b/app/_how-tos/ai-gateway/use-ai-prompt-guard-policy.md @@ -0,0 +1,202 @@ +--- +title: Use AI Prompt Guard Policy to govern your LLM traffic +permalink: /ai-gateway/use-ai-prompt-guard-policy/ +content_type: how_to +related_resources: + - text: "{{site.ai_gateway}}" + url: /ai-gateway/ + - text: AI Prompt Guard + url: /ai-gateway/policies/ai-prompt-guard/ + +description: Use the AI Prompt Guard Policy to filter LLM traffic based on regex rules that allow general IT questions and deny unsafe or off-topic content. + +products: + - ai-gateway + +works_on: + - konnect + +min_version: + ai-gateway: '2.0' + +entities: + - ai-model-provider + - ai-model + - ai-policy + +tags: + - ai + +tldr: + q: How do I allow only general IT-related prompts and block hacking content? + a: Use the AI Prompt Guard Policy with regex patterns to allow or deny prompts based on user prompts. + +tools: + - kongctl + +prereqs: + inline: + - title: OpenAI API key + include_content: md/ai-gateway/v2/prereqs/openai-kongctl + +cleanup: + inline: + - title: Clean up Konnect environment + include_content: cleanup/platform/konnect + icon_url: /assets/icons/gateway.svg + - title: Destroy the {{site.base_gateway}} container + include_content: cleanup/products/gateway + icon_url: /assets/icons/gateway.svg + +--- + +## Create the AI Model Provider, AI Model, and AI Prompt Guard Policy + +Create both an [AI Model Provider](/ai-gateway/entities/ai-model-provider/) and an [AI Model](/ai-gateway/entities/ai-model/) with a single `kongctl` apply command. + +You'll also configure the [AI Prompt Guard Policy](/ai-gateway/policies/ai-prompt-guard/) to filter LLM traffic based on regex rules that allow general IT questions and deny unsafe or off-topic content. + +{% entity_examples %} +ai_gateway_model_providers: + - ref: generic-openai + name: generic-openai + ai_gateway: !lookup name:ai-quickstart + type: openai + config: + auth: + type: basic + headers: + - name: Authorization + value: !env OPENAI_AUTH_HEADER +ai_gateway_policies: + - ref: my-ai-prompt-guard-policy + name: my-ai-prompt-guard-policy + ai_gateway: !lookup name:ai-quickstart + type: ai-prompt-guard + enabled: true + global: false + config: + allow_patterns: + [ + "(?i).*what is .*", + "(?i).*how do i .*", + "(?i).*install .*", + "(?i).*configure .*", + "(?i).*reset .*", + "(?i).*troubleshoot .*" + ] + deny_patterns: + [ + "(?i).*bypass.*(login|password|auth).*", + "(?i).*hack.*", + "(?i).*phish.*", + "(?i).*malware.*", + "(?i).*cve.*", + "(?i).*exploit.*", + "(?i).*social engineering.*", + "(?i).*pentest.*", + "(?i).*impersonate.*", + "(?i).*dating.*" + ] +ai_gateway_models: + - ref: my-gpt-4o + display_name: my-gpt-4o + name: my-gpt-4o + ai_gateway: !lookup name:ai-quickstart + type: model + enabled: true + formats: [{ type: openai }] + config: + route: { paths: [/] } + model: { name_header: true } + capabilities: [generate] + policies: [ !ref my-ai-prompt-guard-policy#name ] + targets: + - name: gpt-4o + provider: generic-openai + config: + type: openai +{% endentity_examples %} + +In this example, we're setting up the AI Prompt Guard Policy with: + +* `type: ai-prompt-guard`: Specifies that this Policy filters requests by matching the user's prompt against allow and deny regex pattern lists. +* `global: false`: Scopes the Policy to only the AI Models it's explicitly attached to via `policies:`, rather than applying it to every resource on {{site.ai_gateway}}. +* `config.allow_patterns`: A list of regexes matched against the user's prompt. The request must match at least one of these to pass through, unless it's also blocked by `deny_patterns`. +* `config.deny_patterns`: A list of regexes matched against the user's prompt. A match here always rejects the request with a 400 response, even if the prompt also matches an `allow_patterns` entry. Deny always takes precedence over allow. +* `policies: [!ref my-ai-prompt-guard-policy#name]` on the AI Model: Attaches this Policy so it applies to every request routed through `my-gpt-4o`. + + +## Validate configuration + +Use sample prompts to confirm that allowed categories (general IT questions) pass through while unsafe or irrelevant requests are blocked. + +{% navtabs "pattern-prompt-guard-it-tests" %} +{% navtab "Allowed: General IT questions" %} + +This prompt matches `allow_patterns` and should succeed: + +```sh +curl -X POST "http://localhost:8000/chat/completions" \ + --no-progress-meter --fail-with-body \ + -H "Accept: application/json"\ + -H "Content-Type: application/json"\ + -H "Authorization: $OPENAI_AUTH_HEADER" \ + --json '{ + "messages": [ + { + "role": "user", + "content": "What is DNS?" + } + ], + "model": "my-gpt-4o" + }' +``` + +{% endnavtab %} +{% navtab "Denied: Hacking and exploits" %} + +This prompt matches `deny_patterns` and should return an error: + +```sh +curl -X POST "http://localhost:8000/chat/completions" \ + --no-progress-meter --fail-with-body \ + -H "Accept: application/json"\ + -H "Content-Type: application/json"\ + -H "Authorization: $OPENAI_AUTH_HEADER" \ + --json '{ + "messages": [ + { + "role": "user", + "content": "How to hack DNS?" + } + ], + "model": "my-gpt-4o" + }' +``` + + +{% endnavtab %} +{% navtab "Denied: Inappropriate and off-topic" %} + +This prompt isn’t related to work and should also be blocked: + +```sh +curl -X POST "http://localhost:8000/chat/completions" \ + --no-progress-meter --fail-with-body \ + -H "Accept: application/json"\ + -H "Content-Type: application/json"\ + -H "Authorization: $OPENAI_AUTH_HEADER" \ + --json '{ + "messages": [ + { + "role": "user", + "content": "What’s a good line to use on a dating app?" + } + ], + "model": "my-gpt-4o" + }' +``` + +{% endnavtab %} +{% endnavtabs %} \ No newline at end of file diff --git a/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-anthropic.md b/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-anthropic.md index b773b6eb05e..b226976b490 100644 --- a/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-anthropic.md +++ b/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-anthropic.md @@ -1,37 +1,25 @@ --- title: Route Claude CLI traffic through {{site.ai_gateway}} and Anthropic -permalink: /how-to/use-claude-code-with-ai-gateway-anthropic/ content_type: how_to +permalink: /ai-gateway/use-claude-code-with-ai-gateway-anthropic/ related_resources: - text: "{{site.ai_gateway}}" url: /ai-gateway/ - - text: AI Proxy Advanced - url: /plugins/ai-proxy-advanced/ - - text: File Log - url: /plugins/file-log/ description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic products: - - gateway - ai-gateway works_on: - - on-prem - konnect -min_version: - gateway: '3.13' - -plugins: - - ai-proxy-advanced - - file-log +tools: + - kongctl -entities: - - service - - route - - plugin +min_version: + ai-gateway: '2.0' tags: - ai @@ -39,90 +27,80 @@ tags: tldr: q: How do I run Claude CLI through {{site.ai_gateway}}? - a: Install Claude CLI, configure its API key helper, create a Gateway Service and Route, attach the AI Proxy plugin to forward requests to Claude, enable file-log to inspect traffic, and point Claude CLI to the local proxy endpoint so all LLM requests pass through the {{site.ai_gateway}} for monitoring and control. + a: Install {{ site.claude_code }}, create an AI Model Provider for Anthropic and an AI Model that targets it, then point {{ site.claude_code }}'s `ANTHROPIC_BASE_URL` at your local {{site.ai_gateway}} endpoint so all LLM requests pass through the gateway for monitoring and control. -tools: - - deck - -prereqs: - inline: - - title: Anthropic - icon_url: /assets/icons/anthropic.svg - include_content: prereqs/anthropic - - title: Claude Code CLI - icon_url: /assets/icons/third-party/claude.svg - include_content: prereqs/claude-code - entities: - services: - - example-service - routes: - - example-route - -cleanup: - inline: - - title: Clean up Konnect environment - include_content: cleanup/platform/konnect - icon_url: /assets/icons/gateway.svg - - title: Destroy the {{site.base_gateway}} container - include_content: cleanup/products/gateway - icon_url: /assets/icons/gateway.svg --- -## Configure the AI Proxy plugin +## Create an AI Model Provider entity -First, configure the AI Proxy plugin for the [{{ site.anthropic }} provider](/ai-gateway/ai-providers/#anthropic). -* This setup uses the default `llm/v1/chat` route. {{ site.claude_code }} sends its requests to this route. -* The configuration also raises the maximum request body size to 512 KB to support larger prompts. +Create an [AI Model Provider](/ai-gateway/entities/ai-model-provider/) entity to define your connection to Anthropic and store your authentication credentials: -Set `llm_format: anthropic` to tell {{site.ai_gateway}} that requests and responses use {{ site.claude }}'s native API format. This parameter controls schema validation and prevents format mismatches between {{ site.claude_code }} and the gateway. {% entity_examples %} -entities: - plugins: - - name: ai-proxy - config: - logging: - log_statistics: true - log_payloads: false - auth: - header_name: x-api-key - header_value: ${key} - model: - name: claude-sonnet-4-5-20250929 - provider: anthropic - options: - anthropic_version: '2023-06-01' - llm_format: anthropic - logging: - log_statistics: true - max_request_body_size: 524288 - route_type: llm/v1/chat -variables: - key: - value: $ANTHROPIC_API_KEY - description: The API key to use to connect to Anthropic. +ai_gateway_model_providers: + - ref: generic-anthropic + ai_gateway: !lookup name:ai-quickstart + name: generic-anthropic + display_name: "generic-anthropic" + type: anthropic + config: + auth: + type: basic + headers: + - name: x-api-key + value: !env ANTHROPIC_API_KEY {% endentity_examples %} -## Configure the File Log plugin +In this example, we're setting up the AI Model Provider with: -Now, let's enable the [File Log](/plugins/file-log/) plugin on the Service, to inspect the LLM traffic between {{ site.claude }} and the {{site.ai_gateway}}. This creates a local `claude.json` file on your machine. The file records each request and response so you can review what {{ site.claude }} sends through the {{site.ai_gateway}}. +* `type: anthropic`: Specifies that this provider connects to the Anthropic service using Anthropic's standard API format. +* `name: generic-anthropic`: A unique identifier that AI Models will reference to route requests through this provider. +* `config.auth`: Adds the Anthropic API key set in the `ANTHROPIC_API_KEY` environment variable. {{site.ai_gateway}} securely manages this credential and injects it into upstream requests automatically, eliminating the need for clients to pass API keys. + +## Create an AI Model entity + +Create an [AI Model](/ai-gateway/entities/ai-model/) entity to declare which upstream models are available, configure how client requests are routed, and specify which AI Model Provider to use: {% entity_examples %} -entities: - plugins: - - name: file-log - config: - path: "/tmp/claude.json" +ai_gateway_models: + - ref: my-claude + ai_gateway: !lookup name:ai-quickstart + name: my-claude + display_name: "my-claude" + type: model + formats: + - type: anthropic + config: + route: + paths: + - / + model: + alias: my-claude + targets: + - name: claude-opus-4-8 + provider: generic-anthropic + config: + type: anthropic + policies: [] + capabilities: + - generate {% endentity_examples %} -## Verify traffic through Kong +In this example, we're setting up the AI Model with: + +* `type: model`: Specifies this is a synchronous model for request/response workloads. +* `name: my-claude`: A unique identifier for this model. +* `formats: [type: anthropic]`: Declares that this model accepts requests in Anthropic-compatible format. +* `config.route.paths: [/]`: Configures the custom base path where this model's routes will be accessible. Setting this to a unique value avoids clashes when you have multiple AI Models. +* `capabilities: [generate]`: Enables the text generation capability. For a model using the `anthropic` format, the `generate` capability creates a `/messages` endpoint matching Anthropic's native Messages API, so combined with your base path, clients send requests to `/messages`. +* `targets`: Specifies which upstream AI Model Provider model to route requests to. Here, `provider: generic-anthropic` references the AI Model Provider we created earlier, and `name: claude-opus-4-8` specifies which Anthropic model to call upstream. + +## Verify traffic through {{site.ai_gateway}} Now, we can start a {{ site.claude_code }} session that points it to the local {{site.ai_gateway}} endpoint: ```sh -ANTHROPIC_BASE_URL=http://localhost:8000/anything \ -ANTHROPIC_MODEL=claude-sonnet-4-5-20250929 \ -claude +ANTHROPIC_BASE_URL=http://localhost:8000/ claude --model 'my-claude' ``` {{ site.claude_code }} asks for permission before it runs tools or interacts with files: @@ -143,10 +121,10 @@ Learn more ( https://docs.claude.com/s/claude-code-security ) ``` {:.no-copy-code} -Select **Yes, continue**. The session starts. Ask a simple question to confirm that requests reach {{site.ai_gateway}}. +Select **Yes, continue**. The session starts. Ask a question to confirm that requests reach {{site.ai_gateway}}. ```text -Tell me about Madrid Skylitzes manuscript. +Tell me about the Madrid Skylitzes manuscript. ``` {{ site.claude_code }} might prompt you approve its web search for answering the question. When you select **Yes**, {{ site.claude }} will produce a full-length response to your request: @@ -180,52 +158,4 @@ various backgrounds: - 1 English or French artist - 2 Byzantine artists ``` -{:.no-copy-code} - -Next, inspect the {{site.ai_gateway}} logs to verify that the traffic was proxied through it: - -```sh -docker exec kong-quickstart-gateway cat /tmp/claude.json | jq -``` - -You should find an entry that shows the upstream request made by {{ site.claude_code }}. A typical log record looks like this: - -```json -{ - "...": "...", - "headers": { - ... - "user-agent": "claude-cli/2.0.37 (external, cli)", - "content-type": "application/json", - ... - }, - "method": "POST", - ... - "ai": { - "proxy": { - "usage": { - "prompt_tokens": 1, - "completion_tokens_details": {}, - "completion_tokens": 85, - "total_tokens": 86, - "cost": 0, - "time_per_token": 38.941176470588, - "time_to_first_token": 2583, - "prompt_tokens_details": {} - }, - "meta": { - "request_model": "claude-sonnet-4-20250514", - "response_model": "claude-sonnet-4-20250514", - "llm_latency": 3310, - "plugin_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", - "request_mode": "stream", - "provider_name": "anthropic" - } - } - }, - ... -} -``` -{:.no-copy-code} - -This output confirms that {{ site.claude_code }} routed the request through {{site.ai_gateway}} using the `claude-sonnet-4-5-20250929` model we selected while starting the {{ site.claude_code }} session. +{:.no-copy-code} \ No newline at end of file diff --git a/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-azure.md b/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-azure.md index 467bccd51a6..193123ab323 100644 --- a/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-azure.md +++ b/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-azure.md @@ -1,131 +1,145 @@ --- title: Route Claude CLI traffic through {{site.ai_gateway}} and Azure -permalink: /how-to/use-claude-code-with-ai-gateway-azure/ +permalink: /ai-gateway/use-claude-code-with-ai-gateway-azure/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" url: /ai-gateway/ - - text: AI Proxy - url: /plugins/ai-proxy/ - - text: File Log - url: /plugins/file-log/ -description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic using Azure OpenAI models +description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic to a Claude model hosted on Azure AI Foundry products: - - gateway - ai-gateway works_on: - - on-prem - konnect -min_version: - gateway: '3.13' - -plugins: - - ai-proxy-advanced - - file-log +tools: + - kongctl -entities: - - service - - route - - plugin +prereqs: + konnect: + - name: KONG_NGINX_HTTP_CLIENT_BODY_BUFFER_SIZE + value: 2m + inline: + - title: Azure AI Foundry + include_content: md/ai-gateway/v2/prereqs/azure-ai-claude -tags: - - ai - - openai +min_version: + ai-gateway: '2.0' tldr: - q: How do I run Claude CLI through {{site.ai_gateway}} for Azure OpenAI models? - a: Install Claude CLI, configure its API key helper, create a Gateway Service and Route, attach the AI Proxy plugin to forward requests to Claude, enable the File Log plugin to inspect traffic, and point Claude CLI to the local proxy endpoint so all LLM requests pass through the {{site.ai_gateway}} for monitoring and control. - -tools: - - deck + q: How do I run Claude CLI through {{site.ai_gateway}} for a Claude model hosted on Azure AI Foundry? + a: Install {{ site.claude_code }}, create an AI Model Provider for your Azure AI Foundry Claude deployment, add a policy to strip Anthropic-only request fields Azure doesn't support, create an AI Model that targets it, then point {{ site.claude_code }}'s `ANTHROPIC_BASE_URL` at your local {{site.ai_gateway}} endpoint so all LLM requests pass through the gateway for monitoring and control. -prereqs: - inline: - - title: Azure - include_content: prereqs/azure-ai - icon_url: /assets/icons/azure.svg - - title: Claude Code CLI - icon_url: /assets/icons/third-party/claude.svg - include_content: prereqs/claude-code - entities: - services: - - example-service - routes: - - example-route - -cleanup: - inline: - - title: Clean up Konnect environment - include_content: cleanup/platform/konnect - icon_url: /assets/icons/gateway.svg - - title: Destroy the {{site.base_gateway}} container - include_content: cleanup/products/gateway - icon_url: /assets/icons/gateway.svg --- -## Configure the AI Proxy plugin - -First, configure the AI Proxy plugin for the [Azure AI provider](/ai-gateway/ai-providers/#azure-ai): -* This setup uses the default `llm/v1/chat` route. {{ site.claude_code }} sends its requests to this route. -* The configuration also raises the maximum request body size to 512 KB to support larger prompts. -The `llm_format: anthropic` parameter tells {{site.ai_gateway}} to expect request and response payloads that match {{ site.claude }}'s native API format. Without this setting, the Gateway would default to OpenAI's format, which would cause request failures when {{ site.claude_code }} communicates with the Azure endpoint. +## Create an AI Model Provider entity {% entity_examples %} -entities: - plugins: - - name: ai-proxy - config: - logging: - log_statistics: true - log_payloads: true - route_type: llm/v1/chat - llm_format: anthropic - auth: - header_name: Authorization - header_value: Bearer ${azure_key} - model: - provider: azure - options: - azure_api_version: "2025-01-01-preview" - azure_instance: ${azure_instance} - azure_deployment_id: ${azure_deployment} -variables: - azure_key: - value: "$AZURE_OPENAI_API_KEY" - azure_instance: - value: "$AZURE_INSTANCE_NAME" - azure_deployment: - value: "$AZURE_DEPLOYMENT_ID" +ai_gateway_model_providers: + - ref: azure-claude + name: azure-claude + ai_gateway: !lookup name:ai-quickstart + type: anthropic + config: + auth: + type: basic + headers: + - name: x-api-key + value: !env AZURE_AI_FOUNDRY_TOKEN {% endentity_examples %} -## Configure the File Log plugin +{:.info} +> `ai-quickstart` references the {{site.ai_gateway}} created by the quickstart script in the prerequisites above, instead of creating a new one. -Now, let's enable the [File Log](/plugins/file-log/) plugin on the Service, to inspect the LLM traffic between {{ site.claude }} and the {{site.ai_gateway}}. This creates a local `claude.json` file on your machine. The file records each request and response so you can review what {{ site.claude }} sends through the {{site.ai_gateway}}. +The AI Model Provider uses: + + * `type: anthropic`: Specifies that this provider speaks Anthropic's native Messages API format. Azure AI Foundry serves Claude models through this same native API, so don't use `type: azure`. + * `config.auth.headers[0].value: !env AZURE_AI_FOUNDRY_TOKEN`: Loads the API key from your environment at apply time so it is not embedded in the config. + +## Create an AI Policy and AI Model {% entity_examples %} -entities: - plugins: - - name: file-log - config: - path: "/tmp/claude.json" +ai_gateway_policies: + - ref: claude-code-compat + name: claude-code-compat + ai_gateway: !lookup name:ai-quickstart + type: request-transformer-advanced + enabled: true + global: false + config: + add: + headers: + - "anthropic-version:2023-06-01" + remove: + headers: + - anthropic-beta + querystring: + - beta + body: + - output_config + - context_management + - mcp_servers + - container + - service_tier +ai_gateway_models: + - ref: claude-code-azure-sonnet + display_name: claude-code-azure-sonnet + name: claude-code-azure-sonnet + ai_gateway: !lookup name:ai-quickstart + type: model + enabled: true + formats: + - type: anthropic + config: + route: + paths: + - / + model: + name_header: true + capabilities: + - generate + policies: + - !ref claude-code-compat#name + targets: + - name: claude-sonnet-4-6 + provider: azure-claude + config: + type: anthropic + upstream_url: !env AZURE_AI_FOUNDRY_UPSTREAM_URL {% endentity_examples %} +We create an [AI Policy](/ai-gateway/entities/ai-policy/) entity using [request transformer](/ai-gateway/policies/ai-request-transformer/) to remove extra fields that Azure AI Foundry's Claude endpoint does not support. + +This uses the following settings: + +* `type: request-transformer-advanced`: Modifies requests before {{site.ai_gateway}} forwards them upstream. +* `config.add.headers`: Adds the `anthropic-version` header Azure AI Foundry's native Anthropic endpoint requires. {{ site.claude_code }} doesn't send this header itself, and Foundry rejects requests without it with a `400`. +* `config.remove.headers` / `config.remove.querystring` / `config.remove.body`: Strips Anthropic-beta-only fields — the `anthropic-beta` header, `beta` query string, and body fields like `mcp_servers` and `container` — that {{ site.claude_code }} sends but that Azure AI Foundry's Claude deployment doesn't support. +* `name: claude-code-compat`: The identifier you use to attach the policy. + +{:.info} +> Replace `claude-sonnet-4-6` with the name of your own Claude deployment in Azure AI Foundry. + +The AI Model uses: + +* `name`/`display_name: claude-code-azure-sonnet`: The identifier you pass to `claude --model`. {{ site.claude_code }} uses this, not the upstream target name, to select the model. +* `formats: [type: anthropic]`: Declares that this model accepts requests in Anthropic-compatible format, matching what {{ site.claude_code }} sends natively. +* `config.route.paths: [/]`: Configures the base path where this model's routes are accessible. +* `config.model.name_header: true`: Lets {{ site.claude_code }} select this model by sending its `name` in the request, instead of requiring a separate `alias`. +* `capabilities: [generate]`: Enables text generation. For a model using the `anthropic` format, `generate` creates a `/messages` endpoint matching Anthropic's native Messages API, so combined with your base path, clients send requests to `/v1/messages`. +* `policies`: Attaches the `claude-code-compat` policy created in the previous step, so its header and body transformations apply to every request sent through this model. +* `targets`: Specifies which upstream model to route requests to. `provider: azure-claude` references the AI Provider created earlier, and `name: claude-sonnet-4-6` must match the name of your Claude deployment in Azure AI Foundry. +* `targets[0].config.upstream_url`: The base Azure AI Foundry endpoint from the prerequisites, ending at `/anthropic`. {{site.ai_gateway}} appends the rest of the Anthropic Messages API path automatically. + ## Verify traffic through {{site.ai_gateway}} Now, we can start a {{ site.claude_code }} session that points it to the local {{site.ai_gateway}} endpoint: -{:.warning} -> Ensure that `ANTHROPIC_MODEL` matches the model you deployed in Azure. - ```sh -ANTHROPIC_BASE_URL=http://localhost:8000/anything \ -ANTHROPIC_MODEL=YOUR_AZURE_MODEL \ -claude +ANTHROPIC_BASE_URL=http://localhost:8000/ claude --model 'claude-code-azure-sonnet' ``` {{ site.claude_code }} asks for permission before it runs tools or interacts with files: @@ -152,7 +166,7 @@ Select **Yes, continue**. The session starts. Ask a simple question to confirm t Tell me about Vienna Oribasius manuscript. ``` -{{ site.claude_code }} might prompt you approve its web search for answering the question. When you select **Yes**, {{ site.claude }} will produce a full-length response to your request: +{{ site.claude_code }} might prompt you to approve its web search for answering the question. When you select **Yes**, {{ site.claude }} will produce a full-length response to your request: ```text The "Vienna Oribasius manuscript" refers to a famous illustrated medical @@ -162,61 +176,4 @@ earlier medical knowledge, and his writings form an important link in the transmission of Greco-Roman medical science to the Byzantine, Islamic, and later European worlds. ``` -{:.no-copy-code} - -Next, inspect the {{site.ai_gateway}} logs to verify that the traffic was proxied through it: - -```sh -docker exec kong-quickstart-gateway cat /tmp/claude.json | jq -``` - -You should find an entry that shows the upstream request made by {{ site.claude_code }}. A typical log record looks like this: - -```json -{ - "...": "...", - "headers": { - ... - "user-agent": "claude-cli/2.0.37 (external, cli)", - "content-type": "application/json", - ... - }, - "method": "POST", - ... - "ai": { - "meta": { - "request_mode": "oneshot", - "response_model": "gpt-4.1-2025-04-14", - "request_model": "gpt-4.1", - "llm_latency": 4606, - "provider_name": "azure", - "azure_deployment_id": "gpt-4.1", - "plugin_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", - "azure_api_version": "2024-12-01-preview", - "azure_instance_id": "example-azure-openai" - }, - "usage": { - "completion_tokens": 414, - "completion_tokens_details": { - "accepted_prediction_tokens": 0, - "audio_tokens": 0, - "rejected_prediction_tokens": 0, - "reasoning_tokens": 0 - }, - "total_tokens": 11559, - "cost": 0, - "time_per_token": 11.125603864734, - "time_to_first_token": 4605, - "prompt_tokens": 11145, - "prompt_tokens_details": { - "audio_tokens": 0, - "cached_tokens": 11008, - "cached_tokens_details": {} - } - } - } - }, -``` -{:.no-copy-code} - -This output confirms that {{ site.claude_code }} routed the request through {{site.ai_gateway}} using the `gpt-4.1` Azure AI model we selected while starting the {{ site.claude_code }} session. +{:.no-copy-code} \ No newline at end of file diff --git a/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-bedrock.md b/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-bedrock.md index 64ed90d5ec2..ed2c0ec2f9f 100644 --- a/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-bedrock.md +++ b/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-bedrock.md @@ -1,156 +1,179 @@ --- title: Route Claude CLI traffic through {{site.ai_gateway}} and AWS Bedrock -permalink: /how-to/use-claude-code-with-ai-gateway-bedrock/ content_type: how_to +permalink: /ai-gateway/use-claude-code-with-ai-gateway-bedrock/ related_resources: - text: "{{site.ai_gateway}}" url: /ai-gateway/ - - text: AI Proxy - url: /plugins/ai-proxy/ - - text: File Log - url: /plugins/file-log/ + - text: Route Claude CLI traffic through {{site.ai_gateway}} and Anthropic + url: /ai-gateway/use-claude-code-with-ai-gateway-anthropic/ + - text: Route Claude CLI traffic through {{site.ai_gateway}} and OpenAI + url: /ai-gateway/use-claude-code-with-ai-gateway-openai/ -description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic using AWS Bedrock models +description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic to an AWS Bedrock model products: - - gateway - ai-gateway works_on: - - on-prem - konnect -min_version: - gateway: '3.13' - -plugins: - - ai-proxy - - file-log - -entities: - - service - - route - - plugin - -tags: - - ai - - bedrock - -tldr: - q: How do I run Claude CLI through {{site.ai_gateway}} with AWS Bedrock? - a: Install Claude CLI, configure its API key helper, create a Gateway Service and Route, attach the AI Proxy plugin to forward requests to AWS Bedrock, enable file-log to inspect traffic, and point Claude CLI to the local proxy endpoint so all LLM requests pass through the {{site.ai_gateway}} for monitoring and control. - tools: - - deck + - kongctl prereqs: - prereqs: + konnect: + - name: KONG_NGINX_HTTP_CLIENT_BODY_BUFFER_SIZE + value: 2m inline: - title: AWS Bedrock - icon_url: /assets/icons/bedrock.svg content: | 1. Enable model access in AWS Bedrock: - - Sign in to the AWS Management Console - - Navigate to Amazon Bedrock - - Select **Model access** in the left navigation - - Request access to Claude models (for example, `us.anthropic.claude-haiku-4-5-20251001-v1:0`) - - Wait for access approval (typically immediate for most models) + 1. Sign in to the AWS Management Console. + 1. Navigate to Amazon Bedrock. + 1. Select **Model access** in the left navigation. + 1. Request access to Claude models (for example, `us.anthropic.claude-haiku-4-5-20251001-v1:0`). + 1. Create an IAM user with Bedrock permissions: + 1. Navigate to IAM in the AWS Console. + 1. Create a new user or select an existing user. + 1. Attach the `AmazonBedrockFullAccess` policy, or create a custom policy with `bedrock:InvokeModel` permissions. + 1. Create access keys for the user. + 1. Export your AWS credentials and region: + ```bash + export AWS_ACCESS_KEY_ID='YOUR_AWS_ACCESS_KEY_ID' + export AWS_SECRET_ACCESS_KEY='YOUR_AWS_SECRET_ACCESS_KEY' + export AWS_REGION='YOUR_AWS_REGION' + ``` - 2. Create an IAM user with Bedrock permissions: - - Navigate to IAM in the AWS Console - - Create a new user or select an existing user - - Attach the `AmazonBedrockFullAccess` policy or create a custom policy with `bedrock:InvokeModel` permissions - - Create access keys for the user +min_version: + ai-gateway: '2.0' - 3. Export the Access Key ID, Secret Access Key and AWS region to your environment: - ```sh - export DECK_AWS_ACCESS_KEY_ID='YOUR AWS ACCESS KEY ID' - export DECK_AWS_SECRET_ACCESS_KEY='YOUR AWS SECRET ACCESS KEY' - export DECK_AWS_REGION='YOUR AWS REGION' - ``` - - title: Claude Code CLI - icon_url: /assets/icons/third-party/claude.svg - include_content: prereqs/claude-code - entities: - services: - - example-service - routes: - - example-route +tags: + - ai + - bedrock + +tldr: + q: How do I run Claude CLI through {{site.ai_gateway}} against an AWS Bedrock model? + a: Create an AI Provider entity to store your AWS credentials, create an AI Policy that strips fields Claude CLI sends that Bedrock rejects, create an AI Model entity with an Anthropic-compatible format that routes to Bedrock through that provider, then point Claude CLI's `ANTHROPIC_BASE_URL` at your local {{site.ai_gateway}} endpoint so all LLM requests pass through the gateway for monitoring and control. -cleanup: - inline: - - title: Clean up Konnect environment - include_content: cleanup/platform/konnect - icon_url: /assets/icons/gateway.svg - - title: Destroy the {{site.base_gateway}} container - include_content: cleanup/products/gateway - icon_url: /assets/icons/gateway.svg --- -## Configure the AI Proxy plugin +## Create an AI Model Provider entity -Configure the AI Proxy plugin for the [AWS Bedrock provider](/ai-gateway/ai-providers/#bedrock). +Create an [AI Model Provider](/ai-gateway/entities/ai-model-provider/) entity to define your connection to AWS Bedrock and store your IAM credentials: + +{% entity_examples %} +ai_gateway_model_providers: + - ref: my-aws-account + ai_gateway: !lookup name:ai-quickstart + name: my-aws-account + display_name: "AWS Production" + type: bedrock + config: + auth: + type: aws + access_key_id: !env AWS_ACCESS_KEY_ID + secret_access_key: !env AWS_SECRET_ACCESS_KEY +{% endentity_examples %} -* This setup uses the default `llm/v1/chat` route. {{ site.claude_code }} sends its requests to this route. -* The configuration also raises the maximum token count to 8192 KB to support larger prompts. +## Create an AI Policy entity -The `llm_format: anthropic` parameter tells {{site.ai_gateway}} to expect request and response payloads that match {{ site.claude }}'s native API format. Without this setting, the gateway would default to OpenAI's format, which would cause request failures when {{ site.claude_code }} communicates with the Bedrock endpoint. +{{ site.claude_code }} sends beta headers and fields that Bedrock's API rejects. Create an [AI Policy](/ai-gateway/entities/ai-policy/) with a `request-transformer` config that strips the `anthropic-beta` header, `beta` query string parameter, and beta-gated body fields before the request reaches Bedrock. Don't strip `model`: {{site.ai_gateway}} uses that field to select the target, and removing it breaks routing. {% entity_examples %} -entities: - plugins: - - name: ai-proxy - config: - llm_format: anthropic - route_type: llm/v1/chat - max_request_body_size: 1048576 - logging: - log_statistics: true - log_payloads: false - auth: - allow_override: false - aws_access_key_id: ${aws_access_key_id} - aws_secret_access_key: ${aws_secret_access_key} - model: - provider: bedrock - name: us.anthropic.claude-haiku-4-5-20251001-v1:0 - options: - anthropic_version: bedrock-2023-05-31 - bedrock: - aws_region: ${aws_region} - max_tokens: 8192 -variables: - aws_access_key_id: - value: $AWS_ACCESS_KEY_ID - aws_secret_access_key: - value: $AWS_SECRET_ACCESS_KEY - aws_region: - value: $AWS_REGION +ai_gateway_policies: + - ref: strip-claude-beta-info + ai_gateway: !lookup name:ai-quickstart + name: strip-claude-beta-info + display_name: "Strip Claude beta info" + type: request-transformer + config: + remove: + headers: + - anthropic-beta + querystring: + - beta + body: + - output_config + - context_management + - mcp_servers + - container + - service_tier {% endentity_examples %} -## Configure the File Log plugin +{:.info} +> {{ site.claude_code }} beta features vary by version and may add other incompatible fields over time. If you still see a `400` error mentioning an unexpected field after applying this Policy, add that field to the appropriate `remove` list and re-apply. -Enable the [File Log](/plugins/file-log/) plugin on the service to inspect the LLM traffic between {{ site.claude }} and the {{site.ai_gateway}}. This creates a local `claude.json` file on your machine. The file records each request and response so you can review what {{ site.claude }} sends through the {{site.ai_gateway}}. +## Create an AI Model entity + +Create an [AI Model](/ai-gateway/entities/ai-model/) entity to declare which upstream model is available and how client requests are routed. `formats: [type: anthropic]` accepts requests in Anthropic format even though the upstream is Bedrock: {% entity_examples %} -entities: - plugins: - - name: file-log - config: - path: "/tmp/claude.json" +ai_gateway_model_providers: + - ref: my-aws-account + ai_gateway: !lookup name:ai-quickstart + name: my-aws-account + display_name: "AWS Production" + type: bedrock + config: + auth: + type: aws + access_key_id: !env AWS_ACCESS_KEY_ID + secret_access_key: !env AWS_SECRET_ACCESS_KEY +ai_gateway_policies: + - ref: strip-claude-beta-info + ai_gateway: !lookup name:ai-quickstart + name: strip-claude-beta-info + display_name: "Strip Claude beta info" + type: request-transformer + config: + remove: + headers: + - anthropic-beta + querystring: + - beta + body: + - output_config + - context_management + - mcp_servers + - container + - service_tier +ai_gateway_models: + - ref: my-claude-bedrock + ai_gateway: !lookup name:ai-quickstart + name: my-claude-bedrock + display_name: "my-claude-bedrock" + type: model + formats: + - type: anthropic + config: + route: + paths: + - / + model: + alias: my-claude-bedrock + targets: + - name: us.anthropic.claude-haiku-4-5-20251001-v1:0 + provider: my-aws-account + config: + type: bedrock + region: !env AWS_REGION + policies: + - !ref strip-claude-beta-info#name + capabilities: + - generate {% endentity_examples %} ## Verify traffic through Kong -Start a {{ site.claude_code }} session that points to the local {{site.ai_gateway}} endpoint: - -{:.warning} -> Ensure that `ANTHROPIC_MODEL` matches the model you configured in the AI Proxy plugin (for example, `us.anthropic.claude-haiku-4-5-20251001-v1:0`). +{{ site.claude_code }}'s experimental beta features send fields that Bedrock rejects even with the AI Policy in place. Disable them, then start a session pointed at your local {{site.ai_gateway}} endpoint: ```sh -ANTHROPIC_BASE_URL=http://localhost:8000/anything \ -ANTHROPIC_MODEL=us.anthropic.claude-haiku-4-5-20251001-v1:0 \ +export CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1 + +ANTHROPIC_BASE_URL=http://localhost:8000/ \ +ANTHROPIC_MODEL=my-claude-bedrock \ claude ``` @@ -172,145 +195,4 @@ Learn more ( https://docs.claude.com/s/claude-code-security ) ``` {:.no-copy-code} -Select **Yes, continue**. The session starts. Ask a simple question to confirm that requests reach {{site.ai_gateway}}. - -```text -Tell me about Anna Komnene's Alexiad. -``` - -{{ site.claude_code }} might prompt you to approve its web search for answering the question. When you select **Yes**, {{ site.claude }} will produce a full-length response to your request: - -```text -Anna Komnene (1083-1153?) was a Byzantine princess, scholar, physician, -hospital administrator, and historian. She is known for writing the -Alexiad, a historical account of the reign of her father, Emperor Alexios -I Komnenos (r. 1081-1118). The Alexiad is a valuable primary source for -understanding Byzantine history and the First Crusade. -``` -{:.no-copy-code} - -Next, inspect the {{site.ai_gateway}} logs to verify that the traffic was proxied through it: - -```sh -docker exec kong-quickstart-gateway cat /tmp/claude.json | jq -``` - -You should find an entry that shows the upstream request made by {{ site.claude_code }}. A typical log record looks like this: - -```json -{ - ... - "method": "POST", - "headers": { - "user-agent": "claude-cli/2.0.37 (external, cli)", - "content-type": "application/json" - }, - ... - "ai": { - "proxy": { - "tried_targets": [ - { - "provider": "bedrock", - "model": "us.anthropic.claude-haiku-4-5-20251001-v1:0", - "port": 443, - "upstream_scheme": "https", - "host": "bedrock-runtime.us-west-2.amazonaws.com", - "upstream_uri": "/model/us.anthropic.claude-haiku-4-5-20251001-v1:0/invoke", - "route_type": "llm/v1/chat", - "ip": "xxx.xxx.xxx.xxx" - } - ], - "meta": { - "request_model": "us.anthropic.claude-haiku-4-5-20251001-v1:0", - "request_mode": "oneshot", - "response_model": "us.anthropic.claude-haiku-4-5-20251001-v1:0", - "provider_name": "bedrock", - "llm_latency": 1542, - "plugin_id": "13f5c57a-77b2-4c1f-9492-9048566db7cf" - }, - "usage": { - "completion_tokens": 124, - "completion_tokens_details": {}, - "total_tokens": 11308, - "cost": 0, - "time_per_token": 12.435483870968, - "time_to_first_token": 1542, - "prompt_tokens": 11184, - "prompt_tokens_details": {} - } - } - } - ... -} -``` -{:.no-copy-code} - -This output confirms that {{ site.claude_code }} routed the request through {{site.ai_gateway}} using AWS Bedrock with the `us.anthropic.claude-haiku-4-5-20251001-v1:0` model. - -## Troubleshooting - -When using {{ site.claude_code }} with AWS Bedrock models, you may encounter connection errors. -See the following sections for common error workarounds. - -### API Error 400: `context_management`: Extra inputs are not permitted - -Some beta features aren't compatible with AWS Bedrock. -This error displays because {{ site.claude }} beta features are enabled. - -To resolve this issue, do the following: - -1. Disable betas and experiments: -```sh -export CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1 -``` -2. Configure the [Request Transformer Advanced](/plugins/request-transformer-advanced/) plugin to remove beta information and the `model` field: -{% capture fix_claude_beta %} -{% entity_examples %} -entities: - plugins: - - name: ai-proxy - config: - llm_format: anthropic - route_type: llm/v1/chat - max_request_body_size: 1048576 - logging: - log_statistics: true - log_payloads: false - auth: - allow_override: false - aws_access_key_id: ${aws_access_key_id} - aws_secret_access_key: ${aws_secret_access_key} - model: - provider: bedrock - name: us.anthropic.claude-haiku-4-5-20251001-v1:0 - options: - anthropic_version: bedrock-2023-05-31 - bedrock: - aws_region: ${aws_region} - max_tokens: 8192 - - name: request-transformer-advanced - config: - remove: - headers: - - anthropic-beta - querystring: - - beta - body: - - model -variables: - aws_access_key_id: - value: $AWS_ACCESS_KEY_ID - aws_secret_access_key: - value: $AWS_SECRET_ACCESS_KEY - aws_region: - value: $AWS_REGION -{% endentity_examples %} -{% endcapture %} -{{ fix_claude_beta | indent: 3 }} - -### API Error 400: `max_tokens` must be greater than `thinking.budget_tokens` - -If your `max_tokens` limit is too small, you may encounter this error. -You can resolve this by setting `max_tokens` to a value greater than `budget_tokens`. The maximum value is `200000`. - -For more information about the default `budget_tokens` value, see [Building with extended thinking](https://platform.claude.com/docs/en/build-with-claude/extended-thinking#max-tokens-and-context-window-size) in {{ site.claude }}'s API docs. +Select **Yes, continue**. The session starts. Ask a simple question to confirm that requests reach {{site.ai_gateway}} and are routed to Bedrock. diff --git a/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-dashscope.md b/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-dashscope.md index 87baa110eff..bee8e1dee96 100644 --- a/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-dashscope.md +++ b/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-dashscope.md @@ -1,235 +1,122 @@ --- title: Route Claude CLI traffic through {{site.ai_gateway}} and DashScope -permalink: /how-to/use-claude-code-with-ai-gateway-dashscope/ +permalink: /ai-gateway/use-claude-code-with-ai-gateway-dashscope/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" url: /ai-gateway/ - - text: AI Proxy - url: /plugins/ai-proxy/ - - text: File Log - url: /plugins/file-log/ + - text: Route Claude CLI traffic through {{site.ai_gateway}} and Anthropic + url: /ai-gateway/use-claude-code-with-ai-gateway-anthropic/ -description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic using Alibaba Cloud DashScope models +description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic to an Alibaba Cloud DashScope model. products: - - gateway - ai-gateway works_on: - - on-prem - konnect -min_version: - gateway: '3.13' - -plugins: - - ai-proxy - - file-log +tools: + - kongctl -entities: - - service - - route - - plugin +min_version: + ai-gateway: '2.0' tags: - ai - dashscope tldr: - q: How do I run Claude CLI through {{site.ai_gateway}} with DashScope? - a: Install Claude CLI, configure its API key helper, create a Gateway Service and Route, attach the AI Proxy plugin to forward requests to DashScope, enable file-log to inspect traffic, and point Claude CLI to the local proxy endpoint so all LLM requests pass through the {{site.ai_gateway}} for monitoring and control. - -tools: - - deck + q: How do I run Claude CLI through {{site.ai_gateway}} against a DashScope model? + a: Create an AI Model Provider for Alibaba Cloud DashScope and an AI Model with the `anthropic` format that targets it, then point {{ site.claude_code }}'s `ANTHROPIC_BASE_URL` at your local {{site.ai_gateway}} endpoint so all requests pass through the gateway for monitoring and control. prereqs: - prereqs: inline: - title: DashScope icon_url: /assets/icons/dashscope.svg content: | - You need an active DashScope account with API access. Sign up at the [Alibaba Cloud DashScope platform](https://dashscope.aliyuncs.com/), obtain your API key from the API-KEY interface, and export it to your environment: + Get an API key from the [Alibaba Cloud DashScope console](https://dashscope.aliyuncs.com/) and export it as the **full `Authorization` header value** (including the `Bearer` prefix): + ```sh - export DECK_DASHSCOPE_API_KEY='YOUR DASHSCOPE API KEY' + export DASHSCOPE_AUTH_HEADER="Bearer YOUR_DASHSCOPE_KEY" ``` - - title: Claude Code CLI icon_url: /assets/icons/third-party/claude.svg include_content: prereqs/claude-code - entities: - services: - - example-service - routes: - - example-route -cleanup: - inline: - - title: Clean up Konnect environment - include_content: cleanup/platform/konnect - icon_url: /assets/icons/gateway.svg - - title: Destroy the {{site.base_gateway}} container - include_content: cleanup/products/gateway - icon_url: /assets/icons/gateway.svg - -automated_tests: false --- -## Configure the AI Proxy plugin +## Create the AI Model Provider and AI Model -Configure the AI Proxy plugin for the DashScope provider. -* This setup uses the default `llm/v1/chat` route. {{ site.claude_code }} sends its requests to this route. -* The configuration also raises the maximum token count size to 8192 to support larger prompts. +DashScope serves the Qwen model family through a native Anthropic-compatible Messages API, so {{ site.claude_code }} can talk to it natively. -The `llm_format: anthropic` parameter tells {{site.ai_gateway}} to expect request and response payloads that match {{ site.claude }}'s native API format. Without this setting, the gateway would default to OpenAI's format, which would cause request failures when {{ site.claude_code }} communicates with the DashScope endpoint. +Create both an [AI Model Provider](/ai-gateway/entities/ai-model-provider/) and an [AI Model](/ai-gateway/entities/ai-model/) with a single `kongctl` apply command: {% entity_examples %} -entities: - plugins: - - name: ai-proxy - config: - llm_format: anthropic - route_type: llm/v1/chat - logging: - log_statistics: true - log_payloads: false - auth: - header_name: Authorization - header_value: Bearer ${dashscope_api_key} - model: - provider: dashscope - name: qwen-plus - options: - max_tokens: 8192 - temperature: 1.0 -variables: - dashscope_api_key: - value: $DASHSCOPE_API_KEY +ai_gateway_model_providers: + - ref: dashscope + ai_gateway: !lookup name:ai-quickstart + name: dashscope + type: dashscope + display_name: "Alibaba Cloud DashScope" + config: + auth: + type: basic + headers: + - name: Authorization + value: !env DASHSCOPE_AUTH_HEADER +ai_gateway_models: + - ref: claude-code-qwen + ai_gateway: !lookup name:ai-quickstart + name: claude-code-qwen + display_name: "Claude Code - DashScope Qwen" + type: model + enabled: true + formats: [{ type: anthropic }] + config: + route: { paths: [/], methods: [GET, POST] } + model: { alias: qwen-plus, name_header: true } + capabilities: [generate] + targets: + - name: qwen-plus + provider: dashscope + config: + type: dashscope + international: true + max_tokens: 8192 + temperature: 1.0 {% endentity_examples %} -## Configure the File Log plugin +In this example we set: -Enable the [File Log](/plugins/file-log/) plugin on the service to inspect the LLM traffic between {{ site.claude }} and the {{site.ai_gateway}}. This creates a local `claude.json` file on your machine. The file records each request and response so you can review what {{ site.claude }} sends through the {{site.ai_gateway}}. + * `type: dashscope`: Connects to the Alibaba Cloud DashScope API as an AI Model Provider. + * `capabilities: [generate]`: For a model using the `anthropic` format, `generate` creates a `/v1/messages` endpoint matching Anthropic's native Messages API. + * `formats: [{ type: anthropic }]`: Accepts Anthropic-format requests to the AI Model entity, matching what {{ site.claude_code }} sends. + * `config.model.alias: qwen-plus`: The model name {{ site.claude_code }} should send in each request, which you can set with the `ANTHROPIC_MODEL` variable or `--model` option. + * `route.paths: [/]`: Configures the custom base path where this model's routes will be accessible. Setting this to a unique value avoids clashes when you have multiple AI Models. + * `targets[0].config.international: true`: Uses DashScope's international endpoint (`dashscope-intl.aliyuncs.com`). This is the default. If your DashScope key belongs to a mainland China account, set this to `false` so requests reach `dashscope.aliyuncs.com` instead. -{% entity_examples %} -entities: - plugins: - - name: file-log - config: - path: "/tmp/claude.json" -{% endentity_examples %} +## Verify traffic through {{site.ai_gateway}} -## Verify traffic through Kong - -Start a {{ site.claude_code }} session that points to the local {{site.ai_gateway}} endpoint: - -{:.warning} -> Ensure that `ANTHROPIC_MODEL` matches the model you configured in the AI Proxy plugin (for example, `qwen-plus`). +Before starting {{ site.claude_code }}, confirm the route works by sending an Anthropic Messages API request directly: ```sh -ANTHROPIC_BASE_URL=http://localhost:8000/anything \ -ANTHROPIC_MODEL=qwen-plus \ -claude +curl -sS http://localhost:8000/v1/messages \ + -H 'Content-Type: application/json' \ + -d '{"model":"qwen-plus","max_tokens":16,"messages":[{"role":"user","content":"Reply with just: ok"}]}' ``` -{{ site.claude_code }} asks for permission before it runs tools or interacts with files: - -```text -I'll need permission to work with your files. - -This means I can: -- Read any file in this folder -- Create, edit, or delete files -- Run commands (like npm, git, tests, ls, rm) -- Use tools defined in .mcp.json +## Start and use Claude Code -Learn more ( https://docs.claude.com/s/claude-code-security ) - -❯ 1. Yes, continue -2. No, exit -``` -{:.no-copy-code} +Run {{ site.claude_code }}, selecting the model you configured: -Select **Yes, continue**. The session starts. Ask a simple question to confirm that requests reach {{site.ai_gateway}}. - -```text -Tell me who Niketas Choniates was. +```sh +ANTHROPIC_BASE_URL=http://localhost:8000/ claude --model 'qwen-plus' ``` -{{ site.claude_code }} might prompt you to approve its web search for answering the question. When you select **Yes**, {{ site.claude }} will produce a full-length response to your request: +When {{ site.claude_code }} asks for permission to work with your files, select **Yes, continue**. The session will start. Ask a question to confirm traffic flows through {{site.ai_gateway}} to the upstream Qwen model: ```text -Niketas Choniates was a Byzantine Greek historian and government official -who lived from around 1155 to 1217. He is best known for his historical -work "Historia" (also called "Chronike Diegesis"), which chronicles the -reigns of the Byzantine emperors from 1118 to 1207, covering the period of - the Komnenos and Angelos dynasties. - -Choniates served as a high-ranking official in the Byzantine Empire, -eventually becoming the governor of Athens. His historical writings are -particularly valuable because they provide a detailed eyewitness account -of the Fourth Crusade and the subsequent sack of Constantinople in 1204, -an event he personally experienced and fled from. His account is -considered one of the most important sources for understanding this -pivotal moment in Byzantine history. +Say hello in one sentence. ``` -{:.no-copy-code} - -Next, inspect the {{site.ai_gateway}} logs to verify that the traffic was proxied through it: - -```sh -docker exec kong-quickstart-gateway cat /tmp/claude.json | jq -``` - -You should find an entry that shows the upstream request made by {{ site.claude_code }}. A typical log record looks like this: - -```json -{ - ... - "upstream_uri": "/compatible-mode/v1/chat/completions?beta=true", - "request": { - "method": "POST", - "headers": { - "user-agent": "claude-cli/2.0.57 (external, cli)", - "content-type": "application/json", - "anthropic-version": "2023-06-01" - } - }, - ... - "ai": { - "proxy": { - "usage": { - "completion_tokens": 493, - "completion_tokens_details": {}, - "total_tokens": 13979, - "cost": 0, - "time_per_token": 34.539553752535, - "time_to_first_token": 17027, - "prompt_tokens": 13486, - "prompt_tokens_details": { - "cached_tokens": 0 - } - }, - "meta": { - "response_model": "qwen-plus", - "plugin_id": "63199335-6c5a-4798-a0ad-f2cbf13cc497", - "request_model": "qwen-plus", - "request_mode": "oneshot", - "provider_name": "dashscope", - "llm_latency": 17028 - } - } - }, - "response": { - "headers": { - "x-kong-llm-model": "dashscope/qwen-plus", - "x-dashscope-call-gateway": "true" - } - } - ... -} -``` -{:.no-copy-code} - -This output confirms that {{ site.claude_code }} routed the request through {{site.ai_gateway}} using DashScope with the `qwen-plus` model. \ No newline at end of file diff --git a/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-gemini.md b/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-gemini.md index 8a99ae519b4..4f1560b84e3 100644 --- a/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-gemini.md +++ b/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-gemini.md @@ -1,155 +1,205 @@ --- title: Route Claude CLI traffic through {{site.ai_gateway}} and Gemini -permalink: /how-to/use-claude-code-with-ai-gateway-gemini/ content_type: how_to +permalink: /ai-gateway/use-claude-code-with-ai-gateway-gemini/ related_resources: - text: "{{site.ai_gateway}}" url: /ai-gateway/ - - text: AI Proxy Advanced - url: /plugins/ai-proxy-advanced/ - - text: File Log - url: /plugins/file-log/ + - text: Route Claude CLI traffic through {{site.ai_gateway}} and Anthropic + url: /ai-gateway/use-claude-code-with-ai-gateway-anthropic/ + - text: Route Claude CLI traffic through {{site.ai_gateway}} and OpenAI + url: /ai-gateway/use-claude-code-with-ai-gateway-openai/ -description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic using Gemini models +description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic to a Gemini model products: - - gateway - ai-gateway works_on: - - on-prem - konnect -min_version: - gateway: '3.13' +tools: + - kongctl -plugins: - - ai-proxy-advanced - - file-log +prereqs: + inline: + - title: Gemini API key + content: | + 1. Create a Gemini API key in [Google AI Studio](https://aistudio.google.com/apikey). + 1. Export the API key as a variable: + ```bash + export GEMINI_API_KEY='YOUR_GEMINI_API_KEY' + ``` -entities: - - service - - route - - plugin +min_version: + ai-gateway: '2.0' tags: - ai - gemini tldr: - q: How do I run Claude CLI through {{site.ai_gateway}}? - a: Install Claude CLI, configure its API key helper, create a Gateway Service and Route, attach the AI Proxy plugin to forward requests to Claude, enable the File Log plugin to inspect traffic, and point Claude CLI to the local proxy endpoint so all LLM requests pass through the {{site.ai_gateway}} for monitoring and control. + q: How do I run Claude CLI through {{site.ai_gateway}} against a Gemini model? + a: Create an AI Model Provider entity to store your Gemini API key, create an AI Model entity with an Anthropic-compatible format that routes to Gemini through that provider, then point Claude CLI's `ANTHROPIC_BASE_URL` at your local {{site.ai_gateway}} endpoint so all LLM requests pass through the gateway for monitoring and control. -tools: - - deck - -prereqs: - prereqs: - inline: - - title: Gemini - content: | - Before you begin, you must get the following credentials from Google Cloud: - - - **Service Account Key**: A JSON key file for a service account with Vertex AI permissions - - **Project ID**: Your Google Cloud project identifier - - **Location ID**: The region where your Vertex AI endpoint is deployed (for example, `us-central1`) - - **API Endpoint**: The Vertex AI API endpoint URL (typically `https://{location}-aiplatform.googleapis.com`) - - Export these values as environment variables: - ```sh - export GEMINI_API_KEY="" - export GCP_PROJECT_ID="" - export GEMINI_LOCATION_ID="" - export GEMINI_API_ENDPOINT="" - ``` - icon_url: /assets/icons/gcp.svg - - title: Claude Code CLI - icon_url: /assets/icons/third-party/claude.svg - include_content: prereqs/claude-code - entities: - services: - - example-service - routes: - - example-route - -cleanup: - inline: - - title: Clean up Konnect environment - include_content: cleanup/platform/konnect - icon_url: /assets/icons/gateway.svg - - title: Destroy the {{site.base_gateway}} container - include_content: cleanup/products/gateway - icon_url: /assets/icons/gateway.svg - -automated_tests: false --- -## Configure the AI Proxy plugin +## Create an AI Model Provider entity + +Create an [AI Model Provider](/ai-gateway/entities/ai-model-provider/) entity to define your connection to Gemini and store your API key: + +{% entity_examples %} +ai_gateway_model_providers: + - ref: my-gemini-account + ai_gateway: !lookup name:ai-quickstart + name: my-gemini-account + display_name: "my-gemini-account" + type: gemini + config: + auth: + type: basic + headers: + - name: x-goog-api-key + value: !env GEMINI_API_KEY +{% endentity_examples %} + +In this example, we're setting up the AI Model Provider with: + +* `type: gemini`: Specifies that this provider connects to the Gemini service using Gemini's standard API format. +* `name: my-gemini-account`: A unique identifier that AI Models will reference to route requests through this provider. +* `config.auth`: Stores your Gemini API key. `value: !env GEMINI_API_KEY` loads the value from your environment at apply time instead of embedding it in the YAML, and `kongctl` redacts it in plan and diff output. {{site.ai_gateway}} securely manages this credential and injects it into upstream requests automatically, eliminating the need for clients to pass API keys. -First, configure the AI Proxy plugin for the [{{ site.gemini }} provider](/ai-gateway/ai-providers/#gemini): -* This setup uses the default `llm/v1/chat` route. {{ site.claude_code }} sends its requests to this route. -* The configuration also raises the maximum request body size to 512 KB to support larger prompts. +## Create an AI Policy entity -The `llm_format: anthropic` parameter tells {{site.ai_gateway}} to expect request and response payloads that match {{ site.claude }}'s native API format. Without this setting, the Gateway would default to OpenAI's format, which would cause request failures when {{ site.claude_code }} communicates with the {{ site.gemini }} endpoint. +{{ site.claude_code }} sends beta headers, its own client credentials, and fields that Gemini's native API rejects. Create an [AI Policy](/ai-gateway/entities/ai-policy/) with a `request-transformer-advanced` config that strips the `anthropic-beta`, `authorization`, and `x-api-key` headers, the `beta` query string parameter, and the `output_config`/`context_management`/`mcp_servers`/`container`/`service_tier`/`reasoning_effort` body fields before the request reaches Gemini: {% entity_examples %} -entities: - plugins: - - name: ai-proxy-advanced +ai_gateway_policies: + - ref: strip-claude-beta-info + ai_gateway: !lookup name:ai-quickstart + name: strip-claude-beta-info + display_name: "strip-claude-beta-info" + type: request-transformer-advanced config: - llm_format: anthropic - targets: - - route_type: llm/v1/chat - logging: - log_statistics: true - log_payloads: false - auth: - allow_override: false - gcp_use_service_account: true - gcp_service_account_json: ${gcp_service_account_key} - model: - provider: gemini - name: gemini-2.0-flash - options: - gemini: - api_endpoint: ${gcp_api_endpoint} - project_id: ${gcp_project_id} - location_id: ${gcp_location_id} - max_tokens: 8192 -variables: - gcp_service_account_key: - value: $GEMINI_API_KEY - gcp_api_endpoint: - value: $GEMINI_API_ENDPOINT - gcp_project_id: - value: $GCP_PROJECT_ID - gcp_location_id: - value: $GEMINI_LOCATION_ID + remove: + headers: + - anthropic-beta + - authorization + - x-api-key + querystring: + - beta + body: + - output_config + - context_management + - mcp_servers + - container + - service_tier + - reasoning_effort {% endentity_examples %} -## Configure the File Log plugin +{:.info} +> {{ site.claude_code }} beta features vary by version and may add other incompatible fields over time. If you still see an error mentioning an unexpected field after applying this Policy, add that field to the appropriate `remove` list and re-apply. -Now, let's enable the [File Log](/plugins/file-log/) plugin on the Service, to inspect the LLM traffic between {{ site.claude }} and the {{site.ai_gateway}}. This creates a local `claude.json` file on your machine. The file records each request and response so you can review what {{ site.claude }} sends through the {{site.ai_gateway}}. +## Create an AI Model entity + +Create an [AI Model](/ai-gateway/entities/ai-model/) entity to declare which upstream models are available, configure how client requests are routed, and specify which AI Model Provider to use: {% entity_examples %} -entities: - plugins: - - name: file-log - config: - path: "/tmp/claude.json" +ai_gateway_model_providers: + - ref: my-gemini-account + ai_gateway: !lookup name:ai-quickstart + name: my-gemini-account + display_name: "my-gemini-account" + type: gemini + config: + auth: + type: basic + headers: + - name: x-goog-api-key + value: !env GEMINI_API_KEY +ai_gateway_policies: + - ref: strip-claude-beta-info + ai_gateway: !lookup name:ai-quickstart + name: strip-claude-beta-info + display_name: "strip-claude-beta-info" + type: request-transformer-advanced + config: + remove: + headers: + - anthropic-beta + - authorization + - x-api-key + querystring: + - beta + body: + - output_config + - context_management + - mcp_servers + - container + - service_tier + - reasoning_effort +ai_gateway_models: + - ref: my-claude-gemini + ai_gateway: !lookup name:ai-quickstart + name: my-claude-gemini + display_name: "my-claude-gemini" + type: model + formats: + - type: anthropic + config: + route: + paths: + - / + model: + alias: my-claude-gemini + targets: + - name: gemini-2.5-flash + provider: !ref my-gemini-account#name + config: + type: gemini + policies: + - !ref strip-claude-beta-info#name + capabilities: + - generate {% endentity_examples %} -## Verify traffic through {{site.ai_gateway}} +In this example, we're setting up the AI Model with: + +* `type: model`: Specifies this is a synchronous model for request/response workloads. +* `name: my-claude-gemini`: A unique identifier for this model. +* `formats: [type: anthropic]`: Declares that this model accepts requests in Anthropic-compatible format, matching what {{ site.claude_code }} sends natively, even though the upstream model is Gemini. +* `config.route.paths: [/]`: Configures the custom base path where this model's routes will be accessible. Setting this to a unique value avoids clashes when you have multiple AI Models. +* `capabilities: [generate]`: Enables the text generation capability. For a model using the `anthropic` format, the `generate` capability creates a `/messages` endpoint matching Anthropic's native Messages API, so combined with your base path, clients send requests to `/v1/messages`. +* `targets`: Specifies which upstream AI Model Provider model to route requests to. Here, `provider: !ref my-gemini-account#name` references the AI Model Provider we created earlier, and `name: gemini-2.5-flash` specifies which Gemini model to call upstream. +* `policies: [!ref strip-claude-beta-info#name]`: Attaches the AI Policy created earlier so it applies to every request to this AI Model. -Now, we can start a {{ site.claude_code }} session that points it to the local {{site.ai_gateway}} endpoint: +## Validate the AI Model -{:.warning} -> Ensure that `ANTHROPIC_MODEL` matches the model you deployed in Gemini. +Send a test request directly to confirm the setup works before pointing {{ site.claude_code }} at it: ```sh -ANTHROPIC_BASE_URL=http://localhost:8000/anything \ -ANTHROPIC_MODEL=YOUR_GEMINI_MODEL \ +curl -i -X POST http://localhost:8000/v1/messages \ + -H 'Content-Type: application/json' \ + -H 'anthropic-version: 2023-06-01' \ + --data '{ + "model": "my-claude-gemini", + "max_tokens": 1024, + "messages": [ + {"role": "user", "content": "hello"} + ] + }' +``` + +## Verify traffic through Kong + +{{ site.claude_code }}'s experimental beta features send fields that Gemini rejects even with the AI Policy in place. Disable them, then start a session pointed at your local {{site.ai_gateway}} endpoint: + +```sh +export CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1 + +ANTHROPIC_BASE_URL=http://localhost:8000/ \ +ANTHROPIC_MODEL=my-claude-gemini \ claude ``` @@ -177,71 +227,4 @@ Select **Yes, continue**. The session starts. Ask a simple question to confirm t Tell me about Anna Komnene's Alexiad. ``` -{{ site.claude_code }} might prompt you approve its web search for answering the question. When you select **Yes**, {{ site.claude }} will produce a full-length response to your request: - -```text -Anna Komnene (1083-1153?) was a Byzantine princess, scholar, physician, -hospital administrator, and historian. She is known for writing the -Alexiad, a historical account of the reign of her father, Emperor Alexios -I Komnenos (r. 1081-1118). The Alexiad is a valuable primary source for -understanding Byzantine history and the First Crusade. -``` -{:.no-copy-code} - -Next, inspect the {{site.ai_gateway}} logs to verify that the traffic was proxied through it: - -```sh -docker exec kong-quickstart-gateway cat /tmp/claude.json | jq -``` - -You should find an entry that shows the upstream request made by {{ site.claude_code }}. A typical log record looks like this: - -```json -{ - ... - "method": "POST", - "headers": { - "user-agent": "claude-cli/2.0.37 (external, cli)", - "content-type": "application/json" - }, - ... - "ai": { - "proxy": { - "tried_targets": [ - { - "provider": "gemini", - "model": "gemini-2.0-flash", - "port": 443, - "upstream_scheme": "https", - "host": "us-central1-aiplatform.googleapis.com", - "upstream_uri": "/v1/projects/example-project-id/locations/us-central1/publishers/google/models/gemini-2.0-flash:generateContent", - "route_type": "llm/v1/chat", - "ip": "xxx.xxx.xxx.xxx" - } - ], - "meta": { - "request_model": "gemini-2.0-flash", - "request_mode": "oneshot", - "response_model": "gemini-2.0-flash", - "provider_name": "gemini", - "llm_latency": 1694, - "plugin_id": "13f5c57a-77b2-4c1f-9492-9048566db7cf" - }, - "usage": { - "completion_tokens": 19, - "completion_tokens_details": {}, - "total_tokens": 11203, - "cost": 0, - "time_per_token": 89.157894736842, - "time_to_first_token": 1694, - "prompt_tokens": 11184, - "prompt_tokens_details": {} - } - } - } - ... -} -``` -{:.no-copy-code} - -This output confirms that {{ site.claude_code }} routed the request through {{site.ai_gateway}} using the `gemini-2.0-flash` model we selected while starting the {{ site.claude_code }} session. +{{ site.claude_code }} might prompt you approve its web search for answering the question. When you select **Yes**, {{ site.claude }} will produce a full-length response to your request, proxied through {{site.ai_gateway}} to the Gemini model configured in the AI Model entity's `targets`. diff --git a/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-huggingface.md b/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-huggingface.md index 7fd80d8a4be..4b507f33150 100644 --- a/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-huggingface.md +++ b/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-huggingface.md @@ -1,152 +1,178 @@ --- -title: Route Claude CLI traffic through {{site.ai_gateway}} and HuggingFace -permalink: /how-to/use-claude-code-with-ai-gateway-huggingface/ +title: Route Claude CLI traffic through {{site.ai_gateway}} and Hugging Face content_type: how_to +permalink: /ai-gateway/use-claude-code-with-ai-gateway-huggingface/ related_resources: - text: "{{site.ai_gateway}}" url: /ai-gateway/ - - text: AI Proxy - url: /plugins/ai-proxy/ - - text: Pre-function - url: /plugins/pre-function/ - - text: File Log - url: /plugins/file-log/ -description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic using HuggingFace Inference API models +description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic to a Hugging Face model products: - - gateway - ai-gateway works_on: - - on-prem - konnect -min_version: - gateway: '3.13' +tools: + - kongctl -plugins: - - pre-function - - ai-proxy - - file-log +prereqs: + inline: + - title: Hugging Face + content: | + 1. Create a [Hugging Face access token](https://huggingface.co/settings/tokens) with inference permissions. + 1. Export the token as a bearer header value: + ```bash + export HUGGINGFACE_AUTH_HEADER='Bearer YOUR_HUGGINGFACE_TOKEN' + ``` -entities: - - service - - route - - plugin +min_version: + ai-gateway: '2.0' tags: - ai - huggingface tldr: - q: How do I run Claude CLI through {{site.ai_gateway}} with HuggingFace? - a: Install Claude CLI, configure a pre-function plugin to remove the model field from requests, attach the AI Proxy plugin to forward requests to HuggingFace, enable file-log to inspect traffic, and point Claude CLI to the local proxy endpoint so all LLM requests pass through the {{site.ai_gateway}} for monitoring and control. - -tools: - - deck + q: How do I run Claude CLI through {{site.ai_gateway}} against a Hugging Face model? + a: Create an AI Provider entity to store your Hugging Face token, create an AI Policy that strips fields Claude CLI sends that Hugging Face's API rejects, create an AI Model entity with an Anthropic-compatible format that routes to Hugging Face through that provider, then point Claude CLI's `ANTHROPIC_BASE_URL` at your local {{site.ai_gateway}} endpoint so all LLM requests pass through the gateway for monitoring and control. -prereqs: - inline: - - title: HuggingFace - icon_url: /assets/icons/huggingface.svg - content: | - You need an active HuggingFace account with API access. Sign up at [HuggingFace](https://huggingface.co/) and obtain your API token from the [Access Tokens page](https://huggingface.co/settings/tokens). Ensure you have access to the HuggingFace Inference API, and export your token to your environment: - ```sh - export DECK_HUGGINGFACE_API_TOKEN='YOUR HUGGINGFACE API TOKEN' - ``` - - - title: Claude Code CLI - icon_url: /assets/icons/third-party/claude.svg - include_content: prereqs/claude-code - entities: - services: - - example-service - routes: - - example-route - -cleanup: - inline: - - title: Clean up Konnect environment - include_content: cleanup/platform/konnect - icon_url: /assets/icons/gateway.svg - - title: Destroy the {{site.base_gateway}} container - include_content: cleanup/products/gateway - icon_url: /assets/icons/gateway.svg - -automated_tests: false --- -## Configure the Pre-function plugin +## Create an AI Model Provider entity -{{ site.claude }} CLI automatically includes a `model` field in its request payload. However, when the AI Proxy plugin is configured with HuggingFace provider and specific model in its settings, this creates a conflict. The pre-function plugin removes the `model` field from incoming requests before they reach the AI Proxy plugin, ensuring the gateway uses the model you configured rather than the one {{ site.claude }} CLI sends. +Create an [AI Model Provider](/ai-gateway/entities/ai-model-provider/) entity to define your connection to Hugging Face and store your access token: {% entity_examples %} -entities: - plugins: - - name: pre-function - config: - access: - - | - local body = kong.request.get_body("application/json", nil, 10485760) - if not body or body == "" then - return - end - body.model = nil - kong.service.request.set_body(body, "application/json") +ai_gateway_model_providers: + - ref: my-huggingface-account + ai_gateway: !lookup name:ai-quickstart + name: my-huggingface-account + display_name: "Hugging Face" + type: huggingface + config: + auth: + type: basic + headers: + - name: Authorization + value: !env HUGGINGFACE_AUTH_HEADER {% endentity_examples %} -## Configure the AI Proxy plugin +## Create an AI Policy entity -Configure the AI Proxy plugin for the [HuggingFace provider](/ai-gateway/ai-providers/#huggingface). This setup uses the default `llm/v1/chat` route. {{ site.claude_code }} sends its requests to this route. - -The `llm_format: anthropic` parameter tells {{site.ai_gateway}} to expect request and response payloads that match {{ site.claude }}'s native API format. Without this setting, the gateway would default to OpenAI's format, which would cause request failures when {{ site.claude_code }} communicates with the HuggingFace endpoint. +{{ site.claude_code }} sends beta headers and fields that Hugging Face's API rejects. Create an [AI Policy](/ai-gateway/entities/ai-policy/) with a `request-transformer-advanced` config that strips the `anthropic-beta` header, `beta` query string parameter, and `model`/`output_config` body fields before the request reaches Hugging Face: {% entity_examples %} -entities: - plugins: - - name: ai-proxy - config: - llm_format: anthropic - route_type: llm/v1/chat - logging: - log_statistics: true - log_payloads: false - auth: - header_name: Authorization - header_value: Bearer ${key} - model: - provider: huggingface - name: meta-llama/Llama-3.3-70B-Instruct -variables: - key: - value: $HUGGINGFACE_API_TOKEN - description: The API token to use to connect to HuggingFace Inference API. +ai_gateway_policies: + - ref: strip-claude-beta-info + ai_gateway: !lookup name:ai-quickstart + name: strip-claude-beta-info + display_name: "Strip Claude beta info" + type: request-transformer-advanced + config: + remove: + headers: + - anthropic-beta + querystring: + - beta + body: + - output_config + - context_management + - mcp_servers + - container + - service_tier {% endentity_examples %} -## Configure the File Log plugin +{:.info} +> {{ site.claude_code }} beta features vary by version and may add other incompatible fields over time. If you still see a `400` error mentioning an unexpected field after applying this Policy, add that field to the appropriate `remove` list and re-apply. + +## Create an AI Model entity -Enable the [File Log](/plugins/file-log/) plugin on the service to inspect the LLM traffic between {{ site.claude }} and the {{site.ai_gateway}}. This creates a local `claude.json` file on your machine. The file records each request and response so you can review what {{ site.claude }} sends through the {{site.ai_gateway}}. +Create an [AI Model](/ai-gateway/entities/ai-model/) entity to declare which upstream model is available and how client requests are routed. `formats: [type: anthropic]` accepts requests in Anthropic format even though the upstream is Hugging Face: {% entity_examples %} -entities: - plugins: - - name: file-log - config: - path: "/tmp/claude.json" +ai_gateway_model_providers: + - ref: my-huggingface-account + ai_gateway: !lookup name:ai-quickstart + name: my-huggingface-account + display_name: "Hugging Face" + type: huggingface + config: + auth: + type: basic + headers: + - name: Authorization + value: !env HUGGINGFACE_AUTH_HEADER +ai_gateway_policies: + - ref: strip-claude-beta-info + ai_gateway: !lookup name:ai-quickstart + name: strip-claude-beta-info + display_name: "Strip Claude beta info" + type: request-transformer-advanced + config: + remove: + headers: + - anthropic-beta + querystring: + - beta + body: + - output_config + - context_management + - mcp_servers + - container + - service_tier + - reasoning_effort +ai_gateway_models: + - ref: my-huggingface + ai_gateway: !lookup name:ai-quickstart + name: my-huggingface + display_name: "my-huggingface" + type: model + formats: + - type: anthropic + config: + route: + paths: + - / + model: + alias: my-huggingface + targets: + - name: meta-llama/Llama-3.3-70B-Instruct + provider: !ref my-huggingface-account#name + config: + type: huggingface + policies: + - !ref strip-claude-beta-info#name + capabilities: + - generate {% endentity_examples %} -## Verify traffic through Kong +## Validate the AI Model -Start a {{ site.claude_code }} session that points to the local {{site.ai_gateway}} endpoint: +Send a test request directly to confirm the setup works before pointing {{ site.claude_code }} at it: -{:.warning} -> The `ANTHROPIC_MODEL` value can be any string since the pre-function plugin removes it. The actual model used is `meta-llama/Llama-3.3-70B-Instruct` as configured in the AI Proxy plugin. +```sh +curl -i -X POST http://localhost:8000/v1/messages \ + -H 'Content-Type: application/json' \ + -H 'anthropic-version: 2023-06-01' \ + --data '{ + "model": "my-huggingface", + "max_tokens": 1024, + "messages": [ + {"role": "user", "content": "hello"} + ] + }' +``` + +## Verify traffic through Kong + +{{ site.claude_code }}'s experimental beta features send fields that Hugging Face rejects even with the AI Policy in place. Disable them, then start a session pointed at your local {{site.ai_gateway}} endpoint: ```sh -ANTHROPIC_BASE_URL=http://localhost:8000/anything \ -ANTHROPIC_MODEL=any-model-name \ +export CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1 +ANTHROPIC_BASE_URL=http://localhost:8000/ claude --model 'my-huggingface' --strict-mcp-config --mcp-config '{"mcpServers": {}}' claude ``` @@ -168,84 +194,11 @@ Learn more ( https://docs.claude.com/s/claude-code-security ) ``` {:.no-copy-code} -Select **Yes, continue**. The session starts. Ask a simple question to confirm that requests reach {{site.ai_gateway}}. - -```text -Try creating a logging.py that logs simple http logs. -``` - -{{ site.claude_code }} might prompt you to approve its web search for answering the question. When you select **Yes**, {{ site.claude }} will produce a full-length response to your request: - -```text -Create file -╭───────────────────────────────────────────────────────────────────────────────────────────────────────╮ -│ logging.py │ -│ │ -│ import logging │ -│ │ -│ logging.basicConfig(filename='app.log', filemode='a', format='%(name)s - %(levelname)s - │ -│ %(message)s') │ -│ │ -│ def log_info(message): │ -│ logging.info(message) │ -│ │ -│ def log_warning(message): │ -│ logging.warning(message) │ -│ │ -│ def log_error(message): │ -│ logging.error(message) │ -╰───────────────────────────────────────────────────────────────────────────────────────────────────────╯ - Do you want to create logging.py? - ❯ 1. Yes -``` -{:.no-copy-code} - -Next, inspect the {{site.ai_gateway}} logs to verify that the traffic was proxied through it: +Select **Yes, continue**. The session starts. -```sh -docker exec kong-quickstart-gateway cat /tmp/claude.json | jq -``` - -You should find an entry that shows the upstream request made by {{ site.claude_code }}. A typical log record looks like this: - -```json -{ - ... - "upstream_uri": "/v1/chat/completions?beta=true", - "request": { - "method": "POST", - "headers": { - "user-agent": "claude-cli/2.0.58 (external, cli)", - "content-type": "application/json", - "anthropic-version": "2023-06-01" - } - }, - ... - "ai": { - "proxy": { - "usage": { - "completion_tokens": 26, - "completion_tokens_details": {}, - "total_tokens": 178, - "cost": 0, - "time_per_token": 52.538461538462, - "time_to_first_token": 1365, - "prompt_tokens": 152, - "prompt_tokens_details": {} - }, - "meta": { - "llm_latency": 1366, - "request_mode": "oneshot", - "plugin_id": "0000b82c-5826-4abf-93b0-2fa230f5e030", - "provider_name": "huggingface", - "response_model": "meta-llama/Llama-3.3-70B-Instruct", - "request_model": "meta-llama/Llama-3.3-70B-Instruct" - } - } - } - ... -} -``` -{:.no-copy-code} + +{:.warning} +> Disable thinking with `Opt` + `T`. If you don't disable thinking, you'll get an error with `API Error: 400 `reasoning_effort` is not supported with this model`. + -This output confirms that {{ site.claude_code }} routed the request through {{site.ai_gateway}} using HuggingFace with the `meta-llama/Llama-3.3-70B-Instruct` model. \ No newline at end of file +Ask a simple question to confirm that requests reach {{site.ai_gateway}} and are routed to Hugging Face. diff --git a/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-openai.md b/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-openai.md index ee61f2c4597..e5f39c01299 100644 --- a/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-openai.md +++ b/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-openai.md @@ -1,124 +1,112 @@ --- title: Route Claude CLI traffic through {{site.ai_gateway}} and OpenAI -permalink: /how-to/use-claude-code-with-ai-gateway-openai/ content_type: how_to +permalink: /ai-gateway/use-claude-code-with-ai-gateway-openai/ related_resources: - text: "{{site.ai_gateway}}" url: /ai-gateway/ - - text: AI Proxy - url: /plugins/ai-proxy/ - - text: File Log - url: /plugins/file-log/ + - text: Route Claude CLI traffic through {{site.ai_gateway}} and Anthropic + url: /ai-gateway/use-claude-code-with-ai-gateway-anthropic/ -description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic using OpenAI models +description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic to an OpenAI model products: - - gateway - ai-gateway works_on: - - on-prem - konnect -min_version: - gateway: '3.13' +tools: + - kongctl -plugins: - - ai-proxy-advanced - - file-log +prereqs: + inline: + - title: OpenAI API key + include_content: md/ai-gateway/v2/prereqs/openai-kongctl -entities: - - service - - route - - plugin +min_version: + ai-gateway: '2.0' tags: - ai - openai tldr: - q: How do I run Claude CLI through {{site.ai_gateway}}? - a: Install Claude CLI, configure its API key helper, create a Gateway Service and Route, attach the AI Proxy plugin to forward requests to Claude, enable the File Log plugin to inspect traffic, and point Claude CLI to the local proxy endpoint so all LLM requests pass through the {{site.ai_gateway}} for monitoring and control. - -tools: - - deck + q: How do I run Claude CLI through {{site.ai_gateway}} against an OpenAI model? + a: Create an AI Provider entity to store your OpenAI API key, create an AI Model entity with an Anthropic-compatible format that routes to OpenAI through that provider, then point Claude CLI's `ANTHROPIC_BASE_URL` at your local {{site.ai_gateway}} endpoint so all LLM requests pass through the gateway for monitoring and control. -prereqs: - inline: - - title: OpenAI - include_content: prereqs/openai - icon_url: /assets/icons/openai.svg - - title: Claude Code CLI - icon_url: /assets/icons/third-party/claude.svg - include_content: prereqs/claude-code - entities: - services: - - example-service - routes: - - example-route - -cleanup: - inline: - - title: Clean up Konnect environment - include_content: cleanup/platform/konnect - icon_url: /assets/icons/gateway.svg - - title: Destroy the {{site.base_gateway}} container - include_content: cleanup/products/gateway - icon_url: /assets/icons/gateway.svg --- -## Configure the AI Proxy plugin - -First, configure the AI Proxy plugin for the [OpenAI provider](/ai-gateway/ai-providers/#openai): - * This setup uses the default `llm/v1/chat` route. {{ site.claude_code }} sends its requests to this route. - * The configuration also raises the maximum request body size to 512 KB to support larger prompts. +## Create an AI Provider entity -The `llm_format: anthropic` parameter tells {{site.ai_gateway}} to expect request and response payloads that match {{ site.claude }}'s native API format. Without this setting, the Gateway would default to OpenAI's format, which would cause request failures when {{ site.claude_code }} communicates with the OpenAI endpoint. +Create an [AI Model Provider](/ai-gateway/entities/ai-model-provider/) entity to define your connection to OpenAI and store your authentication credentials: {% entity_examples %} -entities: - plugins: - - name: ai-proxy - config: - llm_format: anthropic - route_type: llm/v1/chat - logging: - log_statistics: true - log_payloads: false - auth: - header_name: Authorization - header_value: Bearer ${openai_key} - allow_override: false - model: - provider: openai - name: gpt-5-mini - max_request_body_size: 524288 -variables: - openai_key: - value: "$OPENAI_API_KEY" +ai_gateway_model_providers: + - ref: generic-openai + ai_gateway: !lookup name:ai-quickstart + name: generic-openai + display_name: "generic-openai" + type: openai + config: + auth: + type: basic + headers: + - name: Authorization + value: !env OPENAI_AUTH_HEADER {% endentity_examples %} -## Configure the File Log plugin +In this example, we're setting up the AI Model Provider with: -Now, let's enable the [File Log](/plugins/file-log/) plugin on the Service, to inspect the LLM traffic between {{ site.claude }} and the {{site.ai_gateway}}. This creates a local `claude.json` file on your machine. The file records each request and response so you can review what {{ site.claude }} sends through the {{site.ai_gateway}}. +* `type: openai`: Specifies that this provider connects to the OpenAI service using OpenAI's standard API format. +* `name: generic-openai`: A unique identifier that AI Models will reference to route requests through this provider. +* `config.auth`: Stores your OpenAI API key. `header_value: !env OPENAI_AUTH_HEADER` loads the value from your environment at apply time instead of embedding it in the YAML, and `kongctl` redacts it in plan and diff output. {{site.ai_gateway}} securely manages this credential and injects it into upstream requests automatically, eliminating the need for clients to pass API keys. + +## Create an AI Model entity + +Create an [AI Model](/ai-gateway/entities/ai-model/) entity to declare which upstream models are available, configure how client requests are routed, and specify which AI Provider to use: {% entity_examples %} -entities: - plugins: - - name: file-log - config: - path: "/tmp/claude.json" +ai_gateway_models: + - ref: my-claude-openai + ai_gateway: !lookup name:ai-quickstart + name: my-claude-openai + display_name: "my-claude-openai" + type: model + formats: + - type: anthropic + config: + route: + paths: + - / + model: + alias: my-claude-openai + targets: + - name: gpt-5-mini + provider: generic-openai + config: + type: openai + policies: [] + capabilities: + - generate {% endentity_examples %} -## Verify traffic through {{site.ai_gateway}} +In this example, we're setting up the AI Model with: + +* `type: model`: Specifies this is a synchronous model for request/response workloads. +* `name: my-claude-openai`: A unique identifier for this model. +* `formats: [type: anthropic]`: Declares that this model accepts requests in Anthropic-compatible format, matching what {{ site.claude_code }} sends natively, even though the upstream model is OpenAI. +* `config.route.paths: [/]`: Configures the custom base path where this model's Routes will be accessible. Setting this to a unique value avoids clashes when you have multiple AI Models. +* `capabilities: [generate]`: Enables the text generation capability. For a model using the `anthropic` format, the `generate` capability creates a `/messages` endpoint matching Anthropic's native Messages API, so combined with your base path, clients send requests to `/v1/messages`. +* `targets`: Specifies which upstream AI Provider model to route requests to. Here, `provider: generic-openai` references the AI Provider we created earlier, and `name: gpt-5-mini` specifies which OpenAI model to call upstream. + +## Verify traffic through Kong Now, we can start a {{ site.claude_code }} session that points it to the local {{site.ai_gateway}} endpoint: ```sh -ANTHROPIC_BASE_URL=http://localhost:8000/anything \ -ANTHROPIC_MODEL=gpt-5-mini \ -claude +ANTHROPIC_BASE_URL=http://localhost:8000/ ANTHROPIC_MODEL=my-claude-openai claude ``` {{ site.claude_code }} asks for permission before it runs tools or interacts with files: @@ -141,72 +129,8 @@ Learn more ( https://docs.claude.com/s/claude-code-security ) Select **Yes, continue**. The session starts. Ask a simple question to confirm that requests reach {{site.ai_gateway}}. - ```text Tell me about Procopius' Secret History. ``` -{{ site.claude_code }} might prompt you approve its web search for answering the question. When you select **Yes**, {{ site.claude }} will produce a full-length response to your request: - -```text -Procopius’ Secret History (Greek: Ἀνέκδοτα, Anekdota) is a fascinating and -notorious work of Byzantine literature written in the 6th century by the -court historian Procopius of Caesarea. Unlike his official histories -(“Wars” and “Buildings”), which paint the Byzantine Emperor Justinian I -and his wife Theodora in a generally positive and conventional manner, the -Secret History offers a scandalous, behind-the-scenes account that -sharply criticizes and even vilifies the emperor, the empress, and other -key figures of the time. -``` -{:.no-copy-code} - -Next, inspect the {{site.ai_gateway}} logs to verify that the traffic was proxied through it: - -```sh -docker exec kong-quickstart-gateway cat /tmp/claude.json | jq -``` - -You should find an entry that shows the upstream request made by {{ site.claude_code }}. A typical log record looks like this: - -```json -{ - ... - "method": "POST", - "headers": { - "user-agent": "claude-cli/2.0.37 (external, cli)", - "content-type": "application/json" - }, - "ai": { - "meta": { - "request_model": "gpt-5-mini", - "request_mode": "oneshot", - "response_model": "gpt-5-mini-2025-08-07", - "provider_name": "openai", - "llm_latency": 6786, - "plugin_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" - }, - "usage": { - "completion_tokens": 456, - "completion_tokens_details": { - "accepted_prediction_tokens": 0, - "audio_tokens": 0, - "rejected_prediction_tokens": 0, - "reasoning_tokens": 256 - }, - "total_tokens": 481, - "cost": 0, - "time_per_token": 14.881578947368, - "time_to_first_token": 6785, - "prompt_tokens": 25, - "prompt_tokens_details": { - "cached_tokens": 0, - "audio_tokens": 0 - } - } - } - ... -} -``` -{:.no-copy-code} - -This output confirms that {{ site.claude_code }} routed the request through {{site.ai_gateway}} using the `gpt-5-mini` model we selected while starting the {{ site.claude_code }} session. +{{ site.claude_code }} might prompt you approve its web search for answering the question. When you select **Yes**, {{ site.claude }} will produce a full-length response to your request, proxied through {{site.ai_gateway}} to the OpenAI model configured in the AI Model entity's `targets`. diff --git a/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-vertex.md b/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-vertex.md index 4eb654e88a6..720e3ffa7cb 100644 --- a/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-vertex.md +++ b/app/_how-tos/ai-gateway/use-claude-code-with-ai-gateway-vertex.md @@ -1,155 +1,161 @@ --- title: Route Claude CLI traffic through {{site.ai_gateway}} and Vertex AI -permalink: /how-to/use-claude-code-with-ai-gateway-vertex/ +permalink: /ai-gateway/use-claude-code-with-ai-gateway-vertex/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" url: /ai-gateway/ - - text: AI Proxy Advanced - url: /plugins/ai-proxy-advanced/ - - text: File Log - url: /plugins/file-log/ description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic using Google Vertex AI models products: - - gateway - ai-gateway works_on: - - on-prem - konnect -min_version: - gateway: '3.13' - -plugins: - - ai-proxy-advanced - - file-log +tools: + - kongctl -entities: - - service - - route - - plugin +min_version: + ai-gateway: '2.0' tags: - ai - vertex-ai tldr: - q: How do I run Claude CLI through {{site.ai_gateway}}? - a: Install Claude CLI, configure its API key helper, create a Gateway Service and Route, attach the AI Proxy plugin to forward requests to Claude, enable file-log to inspect traffic, and point Claude CLI to the local proxy endpoint so all LLM requests pass through the {{site.ai_gateway}} for monitoring and control. - -tools: - - deck + q: How do I run Claude CLI through {{site.ai_gateway}} for a Claude model hosted on Google Vertex AI? + a: Create an AI Model Provider entity to authenticate to Google Vertex AI, add a Policy to strip Anthropic-only request fields Vertex doesn't support, create an AI Model entity that accepts Anthropic-compatible requests and targets your Vertex model. Then, point Claude CLI’s `ANTHROPIC_BASE_URL` at your local {{site.ai_gateway}} endpoint so all requests are proxied for monitoring and control. prereqs: + konnect: + - name: KONG_NGINX_HTTP_CLIENT_BODY_BUFFER_SIZE + value: 2m inline: - title: Vertex content: | - Before you begin, you must get the following credentials from Google Cloud: - - - **Service Account Key**: A JSON key file for a service account with Vertex AI permissions - - **Project ID**: Your Google Cloud project identifier - - **Location ID**: The region where your Vertex AI endpoint is deployed (for example, `us-central1`) - - **API Endpoint**: The Vertex AI API endpoint URL (typically `https://{location}-aiplatform.googleapis.com`) - - Export these values as environment variables: - ```sh - export GEMINI_API_KEY="" - export GCP_PROJECT_ID="" - export GEMINI_LOCATION_ID="" - export GEMINI_API_ENDPOINT="" - ``` + Before you begin: + + 1. In [Vertex AI Model Garden](https://console.cloud.google.com/vertex-ai/model-garden), enable a Claude model (for example, **Claude Sonnet 4.5**). Note the **location** it's enabled in. Depending on your project, Vertex may offer Claude in a specific region (for example, `us-east5`) or under `global`. + 1. Create a Google Cloud service account with Vertex AI permissions and download its JSON key file. + 1. Export the service account JSON and the full `:rawPredict` upstream URL as environment variables. Vertex encodes your project, location, and model ID directly in this URL, so there are no separate provider or target fields for them. The hostname depends on the location from step 1: a specific region uses a region-prefixed host, while `global` uses the plain host with no region prefix: + + ```sh + export GCP_SERVICE_ACCOUNT_JSON="$(cat /path/to/service-account.json)" + + # If your model is enabled in a specific region: + export VERTEX_UPSTREAM_URL="https://us-east5-aiplatform.googleapis.com/v1/projects//locations/us-east5/publishers/anthropic/models/claude-sonnet-4-5@20250929:rawPredict" + + # If your model is enabled under "global" instead: + export VERTEX_UPSTREAM_URL="https://aiplatform.googleapis.com/v1/projects//locations/global/publishers/anthropic/models/claude-sonnet-4-5@20250929:rawPredict" + ``` + + {:.info} + > Vertex publisher model IDs use the format `name@YYYYMMDD` (for example, `claude-sonnet-4-5@20250929`), not a plain model name. Use the exact ID shown for your enabled model in Model Garden. icon_url: /assets/icons/vertex.svg - title: Claude Code CLI icon_url: /assets/icons/third-party/claude.svg include_content: prereqs/claude-code - entities: - services: - - example-service - routes: - - example-route -cleanup: - inline: - - title: Clean up Konnect environment - include_content: cleanup/platform/konnect - icon_url: /assets/icons/gateway.svg - - title: Destroy the {{site.base_gateway}} container - include_content: cleanup/products/gateway - icon_url: /assets/icons/gateway.svg - -automated_tests: false --- -## Configure the AI Proxy plugin +## Create an AI Model Provider entity -First, configure the AI Proxy plugin for the {{ site.gemini }} provider. -* This setup uses the default `llm/v1/chat` route. {{ site.claude_code }} sends its requests to this route. -* The configuration also raises the maximum tokens count size to 8192 to support larger prompts. - -The `llm_format: anthropic` parameter tells {{site.ai_gateway}} to expect request and response payloads that match {{ site.claude }}'s native API format. Without this setting, the Gateway would default to OpenAI's format, which would cause request failures when {{ site.claude_code }} communicates with the {{ site.gemini }} endpoint. +Create an [AI Model Provider](/ai-gateway/entities/ai-model-provider/) entity to define your connection to Vertex AI and store your API key: {% entity_examples %} -entities: - plugins: - - name: ai-proxy-advanced +ai_gateway_model_providers: + - ref: vertex-prod + name: vertex-prod + display_name: "Google Vertex Prod" + ai_gateway: !lookup name:ai-quickstart + type: vertex config: - llm_format: anthropic - targets: - - route_type: llm/v1/chat - logging: - log_statistics: true - log_payloads: false - auth: - allow_override: false - gcp_use_service_account: true - gcp_service_account_json: ${gcp_service_account_key} - model: - provider: gemini - name: gemini-2.5-flash - options: - gemini: - api_endpoint: ${gcp_api_endpoint} - project_id: ${gcp_project_id} - location_id: ${gcp_location_id} - max_tokens: 8192 -variables: - gcp_service_account_key: - value: $GEMINI_API_KEY - gcp_api_endpoint: - value: $GEMINI_API_ENDPOINT - gcp_project_id: - value: $GCP_PROJECT_ID - gcp_location_id: - value: $GEMINI_LOCATION_ID + auth: + type: gcp + service_account_json: !env GCP_SERVICE_ACCOUNT_JSON {% endentity_examples %} -## Configure the File Log plugin +{:.info} +> `ai-quickstart` references the {{site.ai_gateway}} created by the quickstart script in the prerequisites above, instead of creating a new one. + +This AI Model Provider uses: -Now, let's enable the [File Log](/plugins/file-log/) plugin on the Service, to inspect the LLM traffic between {{ site.claude }} and the {{site.ai_gateway}}. This creates a local `claude.json` file on your machine. The file records each request and response so you can review what {{ site.claude }} sends through the {{site.ai_gateway}}. + * `type: vertex`: Specifies that this provider connects to Google Vertex AI. + * `config.auth.type: gcp`: Uses Google Cloud service account authentication, rather than a bearer token or API key. + * `config.auth.service_account_json: !env GCP_SERVICE_ACCOUNT_JSON`: Loads the service account JSON, required to access the account, from your environment at apply time. + +## Create an AI Policy and AI Model + +Create an [AI Policy](/ai-gateway/entities/ai-policy/) entity using [request transformer](/ai-gateway/policies/ai-request-transformer/) to remove extra fields that Vertex AI's Claude endpoint does not support, and an [AI Model](/ai-gateway/entities/ai-model/) entity to declare which upstream model is available and attach that policy to it. + +{:.warning} +> Apply the Policy and the AI Model together, in the same `kongctl apply` call, as shown below. The AI Model's `policies` field references the Policy via `!ref`, and `ref` values are local to a single `kongctl apply` call. They're never written to {{site.konnect_short_name}}. If you split this into two separate `kongctl apply` calls, the second one fails with `resource not found: claude-code-compat`, even though the Policy already exists. {% entity_examples %} -entities: - plugins: - - name: file-log - config: - path: "/tmp/claude.json" +ai_gateway_policies: + - ref: claude-code-compat + name: claude-code-compat + ai_gateway: !lookup name:ai-quickstart + type: request-transformer-advanced + enabled: true + global: false + config: + remove: + headers: [anthropic-beta] + querystring: [beta] + body: [output_config, context_management, mcp_servers, container, service_tier, thinking] +ai_gateway_models: + - ref: claude-code-vertex-sonnet + name: claude-code-vertex-sonnet + display_name: "Claude Code - Vertex - Sonnet 4.6" + ai_gateway: !lookup name:ai-quickstart + type: model + enabled: true + formats: [{ type: anthropic }] + config: + route: { paths: [/] } + model: { name_header: true } + capabilities: [generate] + policies: [ !ref claude-code-compat#name ] + targets: + - name: claude-sonnet-4-5@20250929 + provider: vertex-prod + config: + type: vertex + upstream_url: !env VERTEX_UPSTREAM_URL {% endentity_examples %} +{:.info} +> Replace `claude-sonnet-4-5@20250929` with the id of your own enabled model in Vertex AI Model Garden. + +The AI Policy uses: + +* `type: request-transformer-advanced`: Modifies requests before {{site.ai_gateway}} forwards them upstream. +* `config.remove.headers` / `config.remove.querystring` / `config.remove.body`: Strips fields that {{ site.claude_code }} sends but that Vertex AI's Claude endpoint rejects with a `400 Extra inputs are not permitted`: the `anthropic-beta` header, the `beta` query string, and body fields like `mcp_servers` and `container`. The list also includes `thinking`. {{ site.claude_code }} sends `thinking: {"type": "adaptive", ...}` by default, and Vertex's schema only accepts `disabled` or `enabled` for `thinking.type`, so it must be removed rather than left as-is. + +{:.info} +> The Vertex driver injects the `anthropic-version` header into the request body automatically. + +The AI Model uses: + + * `name`/`display_name: claude-code-vertex-sonnet`: The identifier you pass to `claude --model`. {{ site.claude_code }} uses this, not the upstream target ID, to select the model. + * `formats: [type: anthropic]`: Accepts Anthropic-compatible requests (what {{ site.claude_code }} sends). + * `config.model.name_header: true`: Lets {{ site.claude_code }} select this model by sending its `name` in the request, instead of requiring a separate `alias`. + * `capabilities: [generate]`: Enables text generation. For a model using the `anthropic` format, `generate` creates a `/messages` endpoint matching Anthropic's native Messages API. + * `policies`: Attaches the `claude-code-compat` policy defined above, via `!ref claude-code-compat#name`, so its body-stripping transformation applies to every request sent through this model. + * `targets[0].provider: vertex-prod`: Routes upstream requests through the Vertex AI Provider created earlier. + * `targets[0].name: claude-sonnet-4-5@20250929`: The Vertex publisher model ID, in `name@YYYYMMDD` format. It must match a model you've enabled in Vertex AI Model Garden. + * `targets[0].config.upstream_url`: The full `:rawPredict` URL from the prerequisites, encoding your project, location, and model ID. + ## Verify traffic through Kong Now, we can start a {{ site.claude_code }} session that points it to the local {{site.ai_gateway}} endpoint: -{:.warning} -> Ensure that `ANTHROPIC_MODEL` matches the model you deployed in Gemini. - ```sh -ANTHROPIC_BASE_URL=http://localhost:8000/anything \ -ANTHROPIC_MODEL=YOUR_VERTEX_MODEL \ -claude +ANTHROPIC_BASE_URL=http://localhost:8000/ claude --model 'claude-code-vertex-sonnet' ``` {{ site.claude_code }} asks for permission before it runs tools or interacts with files: @@ -170,13 +176,13 @@ Learn more ( https://docs.claude.com/s/claude-code-security ) ``` {:.no-copy-code} -Select **Yes, continue**. The session starts. Ask a simple question to confirm that requests reach {{site.ai_gateway}}. +Select **Yes, continue**. The session starts. Ask a question to confirm that requests reach {{site.ai_gateway}}. ```text Tell me about Anna Komnene's Alexiad. ``` -{{ site.claude_code }} might prompt you approve its web search for answering the question. When you select **Yes**, {{ site.claude }} will produce a full-length response to your request: +{{ site.claude_code }} might prompt you to approve its web search for answering the question. When you select **Yes**, {{ site.claude }} will produce a full-length response to your request: ```text Anna Komnene (1083-1153?) was a Byzantine princess, scholar, physician, @@ -186,61 +192,3 @@ I Komnenos (r. 1081-1118). The Alexiad is a valuable primary source for understanding Byzantine history and the First Crusade. ``` {:.no-copy-code} - -Next, inspect the {{site.ai_gateway}} logs to verify that the traffic was proxied through it: - -```sh -docker exec kong-quickstart-gateway cat /tmp/claude.json | jq -``` - -You should find an entry that shows the upstream request made by {{ site.claude_code }}. A typical log record looks like this: - -```json -{ - ... - "method": "POST", - "headers": { - "user-agent": "claude-cli/2.0.37 (external, cli)", - "content-type": "application/json" - }, - ... - "ai": { - "proxy": { - "tried_targets": [ - { - "provider": "gemini", - "model": "gemini-2.0-flash", - "port": 443, - "upstream_scheme": "https", - "host": "us-central1-aiplatform.googleapis.com", - "upstream_uri": "/v1/projects/example-project-id/locations/us-central1/publishers/google/models/gemini-2.0-flash:generateContent", - "route_type": "llm/v1/chat", - "ip": "xxx.xxx.xxx.xxx" - } - ], - "meta": { - "request_model": "gemini-2.5-flash", - "request_mode": "oneshot", - "response_model": "gemini-2.5-flash", - "provider_name": "gemini", - "llm_latency": 1694, - "plugin_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" - }, - "usage": { - "completion_tokens": 19, - "completion_tokens_details": {}, - "total_tokens": 11203, - "cost": 0, - "time_per_token": 85.157894736842, - "time_to_first_token": 2546, - "prompt_tokens": 11184, - "prompt_tokens_details": {} - } - } - } - ... -} -``` -{:.no-copy-code} - -This output confirms that {{ site.claude_code }} routed the request through {{site.ai_gateway}} using the `gemini-2.5-flash` model we selected while starting the {{ site.claude_code }} session. diff --git a/app/_how-tos/ai-gateway/use-codex-with-ai-gateway.md b/app/_how-tos/ai-gateway/use-codex-with-ai-gateway.md index 57c3afd6b03..04ad8cb2f62 100644 --- a/app/_how-tos/ai-gateway/use-codex-with-ai-gateway.md +++ b/app/_how-tos/ai-gateway/use-codex-with-ai-gateway.md @@ -1,39 +1,25 @@ --- title: Route OpenAI Codex CLI traffic through {{site.ai_gateway}} -permalink: /how-to/use-codex-with-ai-gateway/ +permalink: /ai-gateway/use-codex-with-ai-gateway/ content_type: how_to + related_resources: - text: "{{site.ai_gateway}}" url: /ai-gateway/ - - text: AI Proxy Advanced - url: /plugins/ai-proxy-advanced/ - - text: AI Request Transformer - url: /plugins/ai-request-transformer/ - - text: File Log - url: /plugins/file-log/ -description: Configure {{site.ai_gateway}} to proxy OpenAI Codex CLI traffic using AI Proxy Advanced. +description: Configure {{site.ai_gateway}} to proxy OpenAI Codex CLI traffic through the OpenAI Responses API. products: - - gateway - ai-gateway works_on: - - on-prem - konnect -min_version: - gateway: '3.6' - -plugins: - - ai-proxy-advanced - - ai-request-transformer - - file-log +tools: + - kongctl -entities: - - service - - route - - plugin +min_version: + ai-gateway: '2.0' tags: - ai @@ -41,251 +27,102 @@ tags: tldr: q: How do I run OpenAI Codex CLI through {{site.ai_gateway}}? - a: Create a Gateway Service and Route, attach AI Proxy Advanced to forward requests to OpenAI, add a Request Transformer plugin to normalize upstream paths, enable file-log to inspect traffic, and point Codex CLI to the local proxy endpoint so all LLM requests go through the Gateway for monitoring and control. - -tools: - - deck + a: Create an AI Model Provider for OpenAI and an AI Model with the `agentic` capability that targets the OpenAI Responses API, then point Codex CLI's `OPENAI_BASE_URL` at your local {{site.ai_gateway}} endpoint so all requests pass through the gateway for monitoring and control. prereqs: inline: - title: OpenAI - include_content: prereqs/openai icon_url: /assets/icons/openai.svg + content: | + Get an API key from [platform.openai.com/api-keys](https://platform.openai.com/api-keys) and export it as the **full `Authorization` header value** (including the `Bearer ` prefix): + + ```sh + export OPENAI_AUTH_HEADER="Bearer your_api_key" + ``` - title: Codex CLI icon_url: /assets/icons/openai.svg content: | - This tutorial uses the OpenAI Codex CLI. Install Node.js 18+ if needed (verify with `node --version`), then install and launch Codex: - - 1. Run the following command in your terminal to install the Codex CLI: - - ```sh - npm install -g @openai/codex - ``` - - 2. Once the installation process is complete, run the following command: + Install Node.js 18+ (verify with `node --version`), then install the OpenAI Codex CLI: - ```sh - codex - ``` - 3. The CLI will prompt you to authenticate in your browser using your OpenAI account. + ```sh + npm install -g @openai/codex + ``` - 4. Once authenticated, close the Codex CLI session by hitting ctrl + c on macOS or ctrl + break on Windows. - entities: - services: - - codex-service - routes: - - codex-route - -cleanup: - inline: - - title: Clean up Konnect environment - include_content: cleanup/platform/konnect - icon_url: /assets/icons/gateway.svg - - title: Destroy the {{site.base_gateway}} container - include_content: cleanup/products/gateway - icon_url: /assets/icons/gateway.svg - -automated_tests: false --- -## Configure the AI Proxy Advanced plugin - -First, let's configure the AI Proxy Advanced plugin. In this setup, we use the Responses route because the Codex CLI calls it by default. We don't hard-code a model in the plugin — Codex sends the model in each request. We also raise the body size limit to 128 KB to support larger prompts. - -{% entity_examples %} -entities: - plugins: - - name: ai-proxy-advanced - service: codex-service - config: - genai_category: text/generation - llm_format: openai - max_request_body_size: 131072 - model_name_header: true - response_streaming: allow - balancer: - algorithm: "round-robin" - tokens_count_strategy: "total-tokens" - latency_strategy: "tpot" - retries: 3 - targets: - - route_type: llm/v1/responses - auth: - header_name: Authorization - header_value: Bearer ${openai_api_key} - logging: - log_payloads: false - log_statistics: true - model: - provider: "openai" - -variables: - openai_api_key: - value: $OPENAI_API_KEY -{% endentity_examples %} - +## Create an AI Model Provider and AI Model -## Configure the Request Transformer plugin - -To ensure that Codex forwards clean, predictable requests to OpenAI, we configure a [Request Transformer](/plugins/request-transformer/) plugin. This plugin normalizes the upstream URI and removes any extra path segments, so only the expected route reaches the OpenAI endpoint. This small guardrail avoids malformed paths and keeps the proxy behavior consistent. +Codex speaks OpenAI's native format and calls the [Responses API](https://platform.openai.com/docs/api-reference/responses), so no request-transformer policy is needed. Create both the [AI Model Provider](/ai-gateway/entities/ai-model-provider/) and the [AI Model](/ai-gateway/entities/ai-model/) in a single `kongctl` apply command so the model can reference the provider: {% entity_examples %} -entities: - plugins: - - name: request-transformer - service: codex-service - config: - replace: - uri: "/" +ai_gateway_model_providers: + - ref: openai + ai_gateway: !lookup name:ai-quickstart + name: openai + type: openai + display_name: "OpenAI" + config: + auth: + type: basic + headers: + - name: Authorization + value: !env OPENAI_AUTH_HEADER +ai_gateway_models: + - ref: codex-openai + ai_gateway: !lookup name:ai-quickstart + name: codex-openai + display_name: "Codex - OpenAI Responses API" + type: model + enabled: true + formats: [{ type: openai }] + config: + route: { paths: [/codex], methods: [GET, POST] } + model: { alias: gpt-5.4, name_header: true } + capabilities: [agentic] + targets: + - name: gpt-5.4 + provider: openai + config: + type: openai + upstream_url: "https://api.openai.com/v1/responses" {% endentity_examples %} +In this example: -Now, we can pre-validate our current configuration: - - -{% validation request-check %} -url: /codex -status_code: 200 -method: POST -headers: - - 'Content-Type: application/json' -body: - model: gpt-4o - input: - - role: "user" - content: "Ping" -{% endvalidation %} + * `type: openai`: Connects to the OpenAI API. + * `capabilities: [agentic]`: Routes requests to the OpenAI Responses API, which the Codex CLI uses. + * `formats: [{ type: openai }]`: Accepts OpenAI-format requests. + * `config.model.alias: gpt-5.4`: The model name the Codex CLI sends in each request. + * `route.paths: [/codex]`: The base path Codex points at; the Responses API is served at `/codex/responses`. -## Export environment variables +## Verify the AI Model -Now, let's open a new terminal window and export the variables that the Codex CLI will use. We set a dummy API key here just to confirm the variable exists, and point `OPENAI_BASE_URL` to the local proxy endpoint where we will route LLM traffic from Codex CLI: +Before starting Codex, confirm the route works by sending a Responses API request directly (expect `200`): -{% on_prem %} -content: | - ```sh - export OPENAI_API_KEY=sk-xxx - export OPENAI_BASE_URL=http://localhost:8000/codex - ``` -{% endon_prem %} +```sh +curl -sS http://localhost:8000/codex/responses \ + -H 'Content-Type: application/json' \ + -d '{"model":"gpt-5.4","input":"Reply with just: ok","max_output_tokens":16}' +``` -{% konnect %} -content: | - ```sh - export OPENAI_API_KEY=sk-xxx - export OPENAI_BASE_URL=$KONNECT_PROXY_URL/codex - ``` -{% endkonnect %} +{% warning %} +If you are a new Codex user, you must Initialise the tool first by running `codex` and following the steps provided. +{% endwarning %} +## Point Codex CLI at {{site.ai_gateway}} -## Configure the File Log plugin - -Finally, to see the exact payloads traveling between Codex and the {{site.ai_gateway}}, let's attach a File Log plugin to the service. This gives us a local log file so we can inspect requests and responses as Codex runs through Kong. - -{% entity_examples %} -entities: - plugins: - - name: file-log - service: codex-service - config: - path: "/tmp/file.json" -{% endentity_examples %} +Open a new terminal and set `OPENAI_BASE_URL` to the local {{site.ai_gateway}} endpoint. The Codex CLI requires `OPENAI_API_KEY` to be set even though the real key lives on the gateway, so a placeholder is fine: +```sh +export OPENAI_API_KEY=sk-placeholder +export OPENAI_BASE_URL=http://localhost:8000/codex +``` ## Start and use Codex CLI -Let's test our Codex CLI set up now: - -1. In the terminal where you exported your environment variables, run: - - ```sh - codex - ``` - - You should see: - - ```text - ╭───────────────────────────────────────────╮ - │ >_ OpenAI Codex (v0.55.0) │ - │ │ - │ model: gpt-5-codex /model to change │ - │ directory: ~ │ - ╰───────────────────────────────────────────╯ - - To get started, describe a task or try one of these commands: - - /init - create an AGENTS.md file with instructions for Codex - /status - show current session configuration - /approvals - choose what Codex can do without approval - /model - choose what model and reasoning effort to use - /review - review any changes and find issues - ``` - {:.no-copy-code} - -1. Run a simple command to call Codex using the gpt-4o model: - - ```sh - codex exec --model gpt-4o "Hello" - ``` - - Codex will prompt: - - ```text - Would you like to run the following command? - - Reason: Need temporary network access so codex exec can reach the OpenAI API - - $ codex exec --model gpt-4o "Hello" - - › 1. Yes, proceed - 2. Yes, and don't ask again for this command - 3. No, and tell Codex what to do differently - ``` - {:.no-copy-code} - - Select **Yes, proceed** and press Enter. - - Expected output: - - ```text - • Ran codex exec --model gpt-4o "Hello" - └ OpenAI Codex v0.55.0 (research preview) - -------- - … +12 lines - 6.468 - Hi there! How can I assist you today? - - ─ Worked for 9s ──────────────────────────────────────────────────────────────── - - • codex exec --model gpt-4o "Hello" returned: “Hi there! How can I assist you today?” - ``` - {:.no-copy-code} - -1. Check that LLM traffic went through {{site.ai_gateway}}: - - ```sh - docker exec kong-quickstart-gateway cat /tmp/file.json | jq - ``` +Run a simple command to confirm traffic flows through {{site.ai_gateway}} to OpenAI: - Look for entries similar to: +```sh +codex exec --model gpt-5.4 "Hello" +``` - ```json - { - ... - "ai": { - "proxy": { - "tried_targets": [ - { - "ip": "0000.000.000.000", - "route_type": "llm/v1/responses", - "port": 443, - "upstream_scheme": "https", - "host": "api.openai.com", - "upstream_uri": "/v1/responses", - "provider": "openai" - } - ] - } - } - ... - } - ``` - {:.no-copy-code} \ No newline at end of file +When prompted for network access, select **Yes, proceed**. Codex routes the request through {{site.ai_gateway}} to the OpenAI Responses API and returns the model's response, giving you monitoring and control over all Codex LLM traffic. \ No newline at end of file diff --git a/app/_how-tos/ai-gateway/use-litellm-with-ai-proxy.md b/app/_how-tos/ai-gateway/use-litellm-with-ai-proxy.md deleted file mode 100644 index 46a79fb754e..00000000000 --- a/app/_how-tos/ai-gateway/use-litellm-with-ai-proxy.md +++ /dev/null @@ -1,195 +0,0 @@ ---- -title: Use LiteLLM with AI Proxy with {{site.ai_gateway}} -content_type: how_to -permalink: /how-to/use-litellm-with-ai-proxy/ -related_resources: - - text: AI Proxy - url: /plugins/ai-proxy/ - -description: Connect your LiteLLM integrations with {{site.ai_gateway}} with no code changes. - -products: - - gateway - - ai-gateway - -works_on: - - on-prem - - konnect - -min_version: - gateway: '3.6' - -plugins: - - ai-proxy - - key-auth - -entities: - - service - - route - - plugin - -tags: - - ai - - openai - -tldr: - q: How can I use LiteLLM integrations with {{site.ai_gateway}}? - a: You can configure LiteLLM to to use your {{site.ai_gateway}} Route by replacing the `base_url` parameter in the [LiteLLM API call](https://docs.litellm.ai/docs/completion/#basic-usage) with your {{site.base_gateway}} proxy URL. - -tools: - - deck - -prereqs: - inline: - - title: OpenAI - include_content: prereqs/openai - icon_url: /assets/icons/openai.svg - entities: - services: - - example-service - routes: - - example-route - -cleanup: - inline: - - title: Clean up Konnect environment - include_content: cleanup/platform/konnect - icon_url: /assets/icons/gateway.svg - - title: Destroy the {{site.base_gateway}} container - include_content: cleanup/products/gateway - icon_url: /assets/icons/gateway.svg - -published: false ---- - -## Configure the AI Proxy plugin - -Enable the [AI Proxy](/plugins/ai-proxy/) plugin with your OpenAI API key and model details to route LiteLLM OpenAI-compatible requests through {{site.ai_gateway}}. In this example, we'll use the `gpt-4.1` model from OpenAI: - -{% entity_examples %} -entities: - plugins: - - name: ai-proxy - config: - route_type: llm/v1/chat - auth: - header_name: Authorization - header_value: Bearer ${openai_key} - model: - provider: openai - name: gpt-4.1 -variables: - openai_key: - value: $OPENAI_API_KEY -{% endentity_examples %} - -## Add authentication - -To secure access to your Route, create a Consumer and set up an authentication plugin: - -{:.info} -> LiteLLM expects authentication as an `Authorization` header with a value starting with `Bearer`. -You can use plugins like [OAuth 2.0 Authentication](/plugins/oauth2/) or [OpenID Connect](/plugins/openid-connect/) to generate Bearer tokens. In this example, for testing purposes, we'll recreate this pattern using the [Key Authentication](/plugins/key-auth/) plugin. - -{% entity_examples %} -entities: - plugins: - - name: key-auth - route: example-route - config: - key_names: - - Authorization - consumers: - - username: ai-user - keyauth_credentials: - - key: Bearer my-api-key -{% endentity_examples %} - -## Install LiteLLM - -Install the LiteLLM Python SDK: - -{% navtabs "litellm" %} -{% navtab "WSL2, Linux, macOS native" %} -```sh -pip3 install -U litellm -``` - -{% endnavtab %} - -{% navtab "macOS, with Python installed via Homebrew" %} -Create a virtual environment, then install the Python SDK: -```sh -python3 -m venv .venv -source .venv/bin/activate -pip install -U litellm -``` - -{% endnavtab %} -{% endnavtabs %} - -## Create a LiteLLM script - -Use the following command to create a file named `app.py` containing a LiteLLM Python script: - -{% on_prem %} -content: | - ```sh - cat < app.py - import litellm - - kong_url = "http://127.0.0.1:8000" - kong_route = "anything" - - response = litellm.completion( - model="gpt-4.1", - messages=[{"role": "user", "content": "What are you?"}], - api_key="my-api-key", - base_url=f"{kong_url}/{kong_route}" - ) - - print(f"$ ChainAnswer:> {response['choices'][0]['message']['content']}") - EOF - ``` -{% endon_prem %} - -{% konnect %} -content: | - ```sh - cat < app.py - import litellm - import os - - kong_url = os.environ['KONNECT_PROXY_URL'] - kong_route = "anything" - - response = litellm.completion( - model="gpt-4.1", - messages=[{"role": "user", "content": "What are you?"}], - api_key="my-api-key", - base_url=f"{kong_url}/{kong_route}" - ) - - print(f"$ ChainAnswer:> {response['choices'][0]['message']['content']}") - EOF - ``` -{% endkonnect %} - -With the `base_url` parameter, we can override the OpenAI base URL that LiteLLM uses by default with the URL to our {{site.base_gateway}} Route. This allows proxying requests and applying {{site.base_gateway}} plugins while still using LiteLLM’s API interface. - -In the `api_key` parameter, we'll add the API key we created, without the `Bearer` prefix, which LiteLLM adds automatically in the request header. - -## Validate - -Run your script to validate that LiteLLM can access the Route: - -```sh -python3 ./app.py -``` - -The response should look like this: - -```sh -ChainAnswer:> I'm an artificial intelligence (AI) assistant created by OpenAI. I'm designed to help answer questions, provide information, write content, and assist with a wide variety of tasks through natural conversation. You can think of me as a type of intelligent computer program that uses language models to understand and respond to your messages. If you have any questions or need help with something, just let me know! -``` -{:.no-copy-code} \ No newline at end of file diff --git a/app/_how-tos/ai-gateway/authenticate-openai-sdk-clients-with-key-auth.md b/app/_how-tos/ai-gateway/v1/authenticate-openai-sdk-clients-with-key-auth.md similarity index 98% rename from app/_how-tos/ai-gateway/authenticate-openai-sdk-clients-with-key-auth.md rename to app/_how-tos/ai-gateway/v1/authenticate-openai-sdk-clients-with-key-auth.md index dcaaf8ed667..47f7721f9da 100644 --- a/app/_how-tos/ai-gateway/authenticate-openai-sdk-clients-with-key-auth.md +++ b/app/_how-tos/ai-gateway/v1/authenticate-openai-sdk-clients-with-key-auth.md @@ -3,7 +3,7 @@ title: Authenticate OpenAI SDK clients with Key Authentication in {{site.ai_gate content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ - text: Key Authentication @@ -11,7 +11,7 @@ related_resources: - text: Pre-function url: /plugins/pre-function/ -permalink: /how-to/authenticate-openai-sdk-clients-with-key-auth +permalink: /ai-gateway/v1/how-to/authenticate-openai-sdk-clients-with-key-auth description: Use the Pre-function plugin to rewrite OpenAI SDK Bearer tokens into a format compatible with Kong's Key Authentication plugin. @@ -75,6 +75,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- diff --git a/app/_how-tos/ai-gateway/azure-batches.md b/app/_how-tos/ai-gateway/v1/azure-batches.md similarity index 99% rename from app/_how-tos/ai-gateway/azure-batches.md rename to app/_how-tos/ai-gateway/v1/azure-batches.md index 09dd7cd1f3f..b89d7a85247 100644 --- a/app/_how-tos/ai-gateway/azure-batches.md +++ b/app/_how-tos/ai-gateway/v1/azure-batches.md @@ -1,10 +1,10 @@ --- title: Send batch requests to Azure OpenAI LLMs -permalink: /how-to/azure-batches/ +permalink: /ai-gateway/v1/how-to/azure-batches/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ @@ -96,6 +96,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure AI Proxy plugins for /files route diff --git a/app/_how-tos/ai-gateway/compare-llm-models-accuracy.md b/app/_how-tos/ai-gateway/v1/compare-llm-models-accuracy.md similarity index 96% rename from app/_how-tos/ai-gateway/compare-llm-models-accuracy.md rename to app/_how-tos/ai-gateway/v1/compare-llm-models-accuracy.md index c58be45d99f..f050d12394e 100644 --- a/app/_how-tos/ai-gateway/compare-llm-models-accuracy.md +++ b/app/_how-tos/ai-gateway/v1/compare-llm-models-accuracy.md @@ -1,10 +1,10 @@ --- title: Control accuracy of LLM models using the AI LLM as judge plugin -permalink: /how-to/compare-llm-models-accuracy/ +permalink: /ai-gateway/v1/how-to/compare-llm-models-accuracy/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ - text: HTTP Log @@ -106,6 +106,8 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 --- @@ -113,7 +115,7 @@ cleanup: The [AI Proxy Advanced](/plugins/ai-proxy-advanced) plugin allows you to route requests to multiple LLM models and define load balancing, retries, timeouts, and token counting strategies. The AI LLM as Judge plugin requires AI Proxy Advanced with [`config.balancer.tokens_count_strategy`](/plugins/ai-proxy-advanced/reference/#schema--config-balancer-tokens-count-strategy) set to `llm-accuracy`. This setting enables the balancer to compare responses from multiple LLM models and pass them to the judge for evaluation. -In this tutorial, we configure AI Proxy Advanced to send requests to both {{ site.openai }} and {{ site.ollama }} models, using the [lowest-usage balancer](/ai-gateway/load-balancing/#load-balancing-algorithms) to direct traffic to the model currently handling the fewest tokens or requests. For testing purposes only, we include a less reliable {{ site.ollama }} model in the configuration. This makes it easier to demonstrate the evaluation differences when responses are judged by the AI LLM as Judge plugin. +In this tutorial, we configure AI Proxy Advanced to send requests to both {{ site.openai }} and {{ site.ollama }} models, using the [lowest-usage balancer](/ai-gateway/v1/load-balancing/#load-balancing-algorithms) to direct traffic to the model currently handling the fewest tokens or requests. For testing purposes only, we include a less reliable {{ site.ollama }} model in the configuration. This makes it easier to demonstrate the evaluation differences when responses are judged by the AI LLM as Judge plugin. {% entity_examples %} entities: diff --git a/app/_how-tos/ai-gateway/compress-llm-prompts.md b/app/_how-tos/ai-gateway/v1/compress-llm-prompts.md similarity index 99% rename from app/_how-tos/ai-gateway/compress-llm-prompts.md rename to app/_how-tos/ai-gateway/v1/compress-llm-prompts.md index 420e1868829..09f1e7e4fde 100644 --- a/app/_how-tos/ai-gateway/compress-llm-prompts.md +++ b/app/_how-tos/ai-gateway/v1/compress-llm-prompts.md @@ -1,10 +1,10 @@ --- title: Control prompt size with the AI Compressor plugin -permalink: /how-to/compress-llm-prompts/ +permalink: /ai-gateway/v1/how-to/compress-llm-prompts/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI RAG Injector url: /plugins/ai-rag-injector/ - text: AI Proxy Advanced @@ -76,6 +76,9 @@ cleanup: icon_url: /assets/icons/gateway.svg automated_tests: false +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy Advanced plugin diff --git a/app/_how-tos/ai-gateway/configure-hashicorp-vault-as-a-vault-for-llm-providers.md b/app/_how-tos/ai-gateway/v1/configure-hashicorp-vault-as-a-vault-for-llm-providers.md similarity index 95% rename from app/_how-tos/ai-gateway/configure-hashicorp-vault-as-a-vault-for-llm-providers.md rename to app/_how-tos/ai-gateway/v1/configure-hashicorp-vault-as-a-vault-for-llm-providers.md index d33d94ea7bc..1c81291d1ae 100644 --- a/app/_how-tos/ai-gateway/configure-hashicorp-vault-as-a-vault-for-llm-providers.md +++ b/app/_how-tos/ai-gateway/v1/configure-hashicorp-vault-as-a-vault-for-llm-providers.md @@ -1,6 +1,6 @@ --- title: Configure dynamic authentication to LLM providers using HashiCorp vault -permalink: /how-to/configure-hashicorp-vault-as-a-vault-for-llm-providers/ +permalink: /ai-gateway/v1/how-to/configure-hashicorp-vault-as-a-vault-for-llm-providers/ description: "Use HashiCorp Vault to securely store and reference API keys for OpenAI, Mistral, and other LLM providers in {{site.ai_gateway}}." content_type: how_to products: @@ -31,7 +31,7 @@ min_version: gateway: '3.4' breadcrumbs: - - /ai-gateway/ + - /ai-gateway/v1/ entities: - vault @@ -46,7 +46,7 @@ tags: tldr: q: How can I access HashiCorp Vault secrets in {{site.base_gateway}}? a: | - Store secrets using `vault kv put secret/openai key="OPENAI_API_KEY"` to HashiCorp Vault. Then configure a Vault entity in {{site.base_gateway}} with the host, token, and mount path. Inside the Gateway container, run `kong vault get {vault://hashicorp-vault/openai/key}` to confirm access. Next Use the `{vault://...}` syntax in a plugin field to [dynamically authenticate to LLM providers](/how-to/use-semantic-load-balancing-with-dynamic-vault-authentication/) such as OpenAI and Mistral. + Store secrets using `vault kv put secret/openai key="OPENAI_API_KEY"` to HashiCorp Vault. Then configure a Vault entity in {{site.base_gateway}} with the host, token, and mount path. Inside the Gateway container, run `kong vault get {vault://hashicorp-vault/openai/key}` to confirm access. Next Use the `{vault://...}` syntax in a plugin field to [dynamically authenticate to LLM providers](/ai-gateway/v1/how-to/use-semantic-load-balancing-with-dynamic-vault-authentication/) such as OpenAI and Mistral. tools: - deck @@ -80,6 +80,9 @@ faqs: {% include /gateway/vaults-format-faq.md type='question' %} a: | {% include /gateway/vaults-format-faq.md type='answer' %} +major_version: + ai-gateway: 1 + --- ## Create secrets in HashiCorp Vault diff --git a/app/_how-tos/ai-gateway/create-a-complex-ai-chat-history.md b/app/_how-tos/ai-gateway/v1/create-a-complex-ai-chat-history.md similarity index 94% rename from app/_how-tos/ai-gateway/create-a-complex-ai-chat-history.md rename to app/_how-tos/ai-gateway/v1/create-a-complex-ai-chat-history.md index 3c6bdc6d1f9..a116484fa82 100644 --- a/app/_how-tos/ai-gateway/create-a-complex-ai-chat-history.md +++ b/app/_how-tos/ai-gateway/v1/create-a-complex-ai-chat-history.md @@ -1,6 +1,6 @@ --- title: Guide survey classification behavior using the AI Prompt Decorator plugin -permalink: /how-to/create-a-complex-ai-chat-history/ +permalink: /ai-gateway/v1/how-to/create-a-complex-ai-chat-history/ content_type: how_to description: Use the AI Prompt Decorator plugin to enforce privacy-aware classification behavior when routing chat requests to Cohere via {{site.ai_gateway}}. related_resources: @@ -9,10 +9,9 @@ related_resources: - text: AI Prompt Decorator url: /plugins/ai-prompt-decorator/ - text: Ensure chatbots adhere to compliance policies with the AI RAG Injector plugin - url: /how-to/use-ai-rag-injector-plugin/ + url: /ai-gateway/v1/how-to/use-ai-rag-injector-plugin/ - text: Control prompt size with the AI Compressor plugin - url: /how-to/compress-llm-prompts/ - + url: /ai-gateway/v1/how-to/compress-llm-prompts/ tldr: q: How do I guide LLM behavior to perform safe, privacy-aware classification of survey responses? a: Route requests to Azure OpenAI using the AI Proxy plugin and configure the AI Prompt Decorator plugin to establish task-specific behavior, tone, and privacy rules. @@ -63,6 +62,8 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 --- @@ -130,7 +131,7 @@ entities: {:.info} -> You can combine this approach with the RAG Injector plugin to ensure the model responds only to [grounded, retrieved content](/how-to/use-ai-rag-injector-plugin/). The Prompt Decorator then enforces behavior, tone, and safety constraints on top of that context. +> You can combine this approach with the RAG Injector plugin to ensure the model responds only to [grounded, retrieved content](/ai-gateway/v1/how-to/use-ai-rag-injector-plugin/). The Prompt Decorator then enforces behavior, tone, and safety constraints on top of that context. ## Validate prompt behavior enforcement diff --git a/app/_how-tos/ai-gateway/filter-knowledge-based-queries-with-rag-injector.md b/app/_how-tos/ai-gateway/v1/filter-knowledge-based-queries-with-rag-injector.md similarity index 99% rename from app/_how-tos/ai-gateway/filter-knowledge-based-queries-with-rag-injector.md rename to app/_how-tos/ai-gateway/v1/filter-knowledge-based-queries-with-rag-injector.md index a8f741c5e03..a2bcc7f10ec 100644 --- a/app/_how-tos/ai-gateway/filter-knowledge-based-queries-with-rag-injector.md +++ b/app/_how-tos/ai-gateway/v1/filter-knowledge-based-queries-with-rag-injector.md @@ -1,10 +1,10 @@ --- title: Filter knowledge base queries with the AI RAG Injector plugin -permalink: /how-to/filter-knowledge-based-queries-with-rag-injector/ +permalink: /ai-gateway/v1/how-to/filter-knowledge-based-queries-with-rag-injector/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI RAG Injector url: /plugins/ai-rag-injector/ - text: AI Proxy Advanced @@ -82,6 +82,9 @@ search_aliases: - model automated_tests: false +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy Advanced plugin diff --git a/app/_how-tos/ai-gateway/forward-openai-sdk-model-to-ai-proxy-advanced.md b/app/_how-tos/ai-gateway/v1/forward-openai-sdk-model-to-ai-proxy-advanced.md similarity index 97% rename from app/_how-tos/ai-gateway/forward-openai-sdk-model-to-ai-proxy-advanced.md rename to app/_how-tos/ai-gateway/v1/forward-openai-sdk-model-to-ai-proxy-advanced.md index 9f497a7dcb4..6fe40e4549d 100644 --- a/app/_how-tos/ai-gateway/forward-openai-sdk-model-to-ai-proxy-advanced.md +++ b/app/_how-tos/ai-gateway/v1/forward-openai-sdk-model-to-ai-proxy-advanced.md @@ -3,13 +3,13 @@ title: Forward OpenAI SDK model selection to AI Proxy Advanced in {{site.base_ga content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ - text: Pre-function url: /plugins/pre-function/ -permalink: /how-to/forward-openai-sdk-model-to-ai-proxy-advanced +permalink: /ai-gateway/v1/how-to/forward-openai-sdk-model-to-ai-proxy-advanced description: Use the Pre-function plugin to extract the OpenAI SDK model value into a header, then reference it dynamically in AI Proxy Advanced configuration. @@ -64,6 +64,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- [OpenAI-compatible SDKs](https://platform.openai.com/docs/libraries) always set the `model` field in the request body. This is a required parameter and can't be omitted. diff --git a/app/_how-tos/ai-gateway/v1/get-started-with-ai-gateway.md b/app/_how-tos/ai-gateway/v1/get-started-with-ai-gateway.md new file mode 100644 index 00000000000..81ca9e697b9 --- /dev/null +++ b/app/_how-tos/ai-gateway/v1/get-started-with-ai-gateway.md @@ -0,0 +1,161 @@ +--- +title: Get started with {{site.ai_gateway}} +content_type: how_to +permalink: /ai-gateway/v1/get-started/ +description: Learn how to quickly get started with {{site.ai_gateway}} +products: + - ai-gateway + - gateway + +works_on: + - on-prem + - konnect + +plugins: + - ai-proxy + +entities: + - service + - route + - plugin + +tags: + - get-started + - ai + - openai + +tldr: + q: What is {{site.ai_gateway}}, and how can I get started with it? + a: | + With {{site.ai_gateway}}, you can deploy AI infrastructure for traffic + that is sent to one or more LLMs. This lets you semantically route, secure, observe, accelerate, + and govern traffic using a special set of AI plugins that are bundled with {{site.base_gateway}} distributions. + + This tutorial will help you get started with {{site.ai_gateway}} by setting up the AI Proxy plugin with OpenAI. + + {:.info} + > **Note:** + > This quickstart runs a Docker container to explore {{ site.base_gateway }}'s capabilities. + If you want to run {{ site.base_gateway }} as a part of a production-ready API platform, start with the [Install](/gateway/install/) page. + +tools: + - deck + +prereqs: + inline: + - title: OpenAI + content: | + This tutorial uses the AI Proxy plugin with OpenAI. You'll need to [create an OpenAI account](https://auth.openai.com/create-account) and [get an API key](https://platform.openai.com/api-keys). Once you have your API key, create an environment variable: + + ```sh + export OPENAI_API_KEY='' + ``` + +cleanup: + inline: + - title: Clean up Konnect environment + include_content: cleanup/platform/konnect + icon_url: /assets/icons/gateway.svg + - title: Destroy the {{site.base_gateway}} container + include_content: cleanup/products/gateway + icon_url: /assets/icons/gateway.svg + +min_version: + gateway: '3.6' + +next_steps: + - text: Set up load balancing using AI Proxy Advanced plugin + url: /plugins/ai-proxy-advanced/ + - text: Cache traffic using the AI Semantic cache plugin + url: /plugins/ai-semantic-cache/ + - text: Secure traffic with the AI Prompt Guard + url: /plugins/ai-prompt-guard/ + - text: Provide prompt templates with AI Prompt Template + url: /plugins/ai-prompt-template/ + - text: Programmatically inject system or assistant prompts to all incoming prompts with the AI Prompt Decorator + url: /plugins/ai-prompt-decorator/ + - text: Learn about all the AI plugins + url: /plugins/?category=ai +major_version: + ai-gateway: 1 + +--- + +## Check that {{site.base_gateway}} is running + +{% include how-tos/steps/ping-gateway.md %} + + +## Create a Gateway Service + +Create a Service to contain the Route for the LLM provider: + +{% entity_examples %} +entities: + services: + - name: llm-service + url: http://localhost:32000 +{% endentity_examples %} + +The URL can point to any empty host, as it won't be used by the plugin. + +## Create a Route + +Create a Route for the LLM provider. In this example we're creating a chat route, so we'll use `/chat` as the path: + +{% entity_examples %} +entities: + routes: + - name: openai-chat + service: + name: llm-service + paths: + - /chat + protocols: + - http + - https +{% endentity_examples %} + +## Enable the AI Proxy plugin + +Enable the AI Proxy plugin to create a chat route: + +{% entity_examples %} +entities: + plugins: + - name: ai-proxy + config: + route_type: "llm/v1/chat" + model: + provider: "openai" +{% endentity_examples %} + +In this example, we're setting up the plugin with minimal configuration, which means: +* The client is allowed to use any model in the `openai` provider and must provide the model name in the request body. +* The client must provide an `Authorization` header with an OpenAI API key. + +If needed, you can restrict the models that can be consumed by specifying the model name explicitly using the [`config.model.name`](/plugins/ai-proxy/reference/#schema--config-model-name) parameter. + +You can also provide the OpenAI API key directly in the configuration with the [`config.auth.header_name`](/plugins/ai-proxy/reference/#schema--config-auth-header-name) and [`config.auth.header_value`](/plugins/ai-proxy/reference/#schema--config-auth-header-value) parameters so that the client doesn’t have to send them. + +## Validate + +To validate, you can send a `POST` request to the `/chat` endpoint, using the correct [input format](/plugins/ai-proxy/#input-formats). +Since we didn't add the model name and API key in the plugin configuration, make sure to include them in the request: + +{% validation request-check %} +url: /chat +status_code: 200 +method: POST +headers: + - 'Accept: application/json' + - 'Content-Type: application/json' + - 'Authorization: Bearer $OPENAI_API_KEY' +body: + model: gpt-5-mini + messages: + - role: "user" + content: "Say this is a test!" +{% endvalidation %} + +You should get a `200 OK` response, and the response body should contain `This is a test`. diff --git a/app/_how-tos/ai-gateway/limit-a2a-body-size.md b/app/_how-tos/ai-gateway/v1/limit-a2a-body-size.md similarity index 97% rename from app/_how-tos/ai-gateway/limit-a2a-body-size.md rename to app/_how-tos/ai-gateway/v1/limit-a2a-body-size.md index 0fada8e6b62..5555245b86f 100644 --- a/app/_how-tos/ai-gateway/limit-a2a-body-size.md +++ b/app/_how-tos/ai-gateway/v1/limit-a2a-body-size.md @@ -23,7 +23,7 @@ entities: - route - plugin -permalink: /how-to/limit-a2a-request-size/ +permalink: /ai-gateway/v1/how-to/limit-a2a-request-size/ tags: - ai @@ -42,10 +42,9 @@ related_resources: - text: Request Size Limiting plugin reference url: /plugins/request-size-limiting/ - text: "Proxy A2A agents through {{site.ai_gateway}}" - url: /how-to/proxy-a2a-agents/ + url: /ai-gateway/v1/how-to/proxy-a2a-agents/ - text: Rate limit A2A traffic - url: /how-to/rate-limit-a2a-traffic/ - + url: /ai-gateway/v1/how-to/rate-limit-a2a-traffic/ prereqs: entities: services: @@ -86,6 +85,9 @@ faqs: No. The Request Size Limiting plugin checks the request body size, not the response. Streaming SSE responses from the upstream agent are not affected. automated_tests: false +major_version: + ai-gateway: 1 + --- ## Enable the AI A2A Proxy plugin diff --git a/app/_how-tos/mcp/aggregate-mcp-tools.md b/app/_how-tos/ai-gateway/v1/mcp/aggregate-mcp-tools.md similarity index 99% rename from app/_how-tos/mcp/aggregate-mcp-tools.md rename to app/_how-tos/ai-gateway/v1/mcp/aggregate-mcp-tools.md index bdab43473c8..365c32d2620 100644 --- a/app/_how-tos/mcp/aggregate-mcp-tools.md +++ b/app/_how-tos/ai-gateway/v1/mcp/aggregate-mcp-tools.md @@ -3,7 +3,7 @@ title: Aggregate MCP tools from multiple AI MCP Proxy plugins content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI MCP Proxy url: /plugins/ai-mcp-proxy/ - text: Use Insomnia MCP clients to test aggregated MCP tools @@ -14,7 +14,7 @@ description: Learn how to aggregate MCP tools from multiple RESTful APIs using A products: - gateway - ai-gateway -permalink: /mcp/aggregate-mcp-tools/ +permalink: /ai-gateway/v1/mcp/aggregate-mcp-tools/ works_on: - on-prem @@ -121,6 +121,8 @@ prereqs: - weather-route - currency-route - listener-route +major_version: + ai-gateway: 1 --- diff --git a/app/_how-tos/mcp/enforce-acls-on-aggregated-mcp-servers.md b/app/_how-tos/ai-gateway/v1/mcp/enforce-acls-on-aggregated-mcp-servers.md similarity index 98% rename from app/_how-tos/mcp/enforce-acls-on-aggregated-mcp-servers.md rename to app/_how-tos/ai-gateway/v1/mcp/enforce-acls-on-aggregated-mcp-servers.md index 82ec12d1b37..f20c338abe9 100644 --- a/app/_how-tos/mcp/enforce-acls-on-aggregated-mcp-servers.md +++ b/app/_how-tos/ai-gateway/v1/mcp/enforce-acls-on-aggregated-mcp-servers.md @@ -3,13 +3,13 @@ title: Enforce ACLs on aggregated MCP servers content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI MCP Proxy url: /plugins/ai-mcp-proxy/ - text: Control MCP tool access with Consumer and Consumer Group ACLs - url: /mcp/use-access-controls-for-mcp-tools/ + url: /ai-gateway/v1/mcp/use-access-controls-for-mcp-tools/ - text: Aggregate MCP tools from multiple AI MCP Proxy plugins - url: /mcp/aggregate-mcp-tools/ + url: /ai-gateway/v1/mcp/aggregate-mcp-tools/ description: Restrict access to aggregated MCP tools using Consumer Groups. This guide shows how to define per-tool ACLs on conversion-only plugins and enforce them through a listener with the `include_consumer_groups` setting. @@ -17,7 +17,7 @@ products: - gateway - ai-gateway -permalink: /mcp/enforce-acls-on-aggregated-mcp-servers/ +permalink: /ai-gateway/v1/mcp/enforce-acls-on-aggregated-mcp-servers/ works_on: - on-prem @@ -88,6 +88,9 @@ prereqs: - mcp-aggregation automated_tests: false +major_version: + ai-gateway: 1 + --- In this how-to, you'll restrict access to aggregated MCP tools using Consumer Groups. This allows you to define per-tool ACLs on conversion-only plugins and enforce them through a listener with the `include_consumer_groups` setting. diff --git a/app/_how-tos/mcp/govern-mcp-traffic.md b/app/_how-tos/ai-gateway/v1/mcp/govern-mcp-traffic.md similarity index 99% rename from app/_how-tos/mcp/govern-mcp-traffic.md rename to app/_how-tos/ai-gateway/v1/mcp/govern-mcp-traffic.md index 30dd1577d16..cb5a76afe88 100644 --- a/app/_how-tos/mcp/govern-mcp-traffic.md +++ b/app/_how-tos/ai-gateway/v1/mcp/govern-mcp-traffic.md @@ -3,7 +3,7 @@ title: "Use {{site.ai_gateway}} to govern GitHub MCP traffic" content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advance/ - text: AI Prompt Guard plugin @@ -11,8 +11,8 @@ related_resources: - text: AI Rate Limiting Advanced plugin url: /plugins/ai-rate-limiting-advanced/ breadcrumbs: - - /mcp/ -permalink: /mcp/govern-mcp-traffic/ + - /ai-gateway/v1/mcp/ +permalink: /ai-gateway/v1/mcp/govern-mcp-traffic/ series: id: mcp-traffic @@ -62,6 +62,9 @@ cleanup: icon_url: /assets/icons/gateway.svg automated_tests: false +major_version: + ai-gateway: 1 + --- ## Reconfigure the AI Proxy Advanced plugin diff --git a/app/_how-tos/mcp/map-API-to-mcp-tools.md b/app/_how-tos/ai-gateway/v1/mcp/map-API-to-mcp-tools.md similarity index 98% rename from app/_how-tos/mcp/map-API-to-mcp-tools.md rename to app/_how-tos/ai-gateway/v1/mcp/map-API-to-mcp-tools.md index 3ac8a78535d..b8523d2a6e9 100644 --- a/app/_how-tos/mcp/map-API-to-mcp-tools.md +++ b/app/_how-tos/ai-gateway/v1/mcp/map-API-to-mcp-tools.md @@ -3,7 +3,7 @@ title: Map a RESTful API to MCP tools content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI MCP Proxy url: /plugins/ai-mcp-proxy/ @@ -11,7 +11,7 @@ description: Learn how to use the AI MCP Proxy plugin to create an MCP from any products: - gateway - ai-gateway -permalink: /mcp/map-api-to-mcp-tools/ +permalink: /ai-gateway/v1/mcp/map-api-to-mcp-tools/ series: id: mcp-conversion @@ -61,6 +61,9 @@ prereqs: konnect: - name: KONG_STATUS_LISTEN value: '0.0.0.0:8100' +major_version: + ai-gateway: 1 + --- ## Install mock API Server diff --git a/app/_how-tos/mcp/map-weather-api-to-mcp-tools.md b/app/_how-tos/ai-gateway/v1/mcp/map-weather-api-to-mcp-tools.md similarity index 97% rename from app/_how-tos/mcp/map-weather-api-to-mcp-tools.md rename to app/_how-tos/ai-gateway/v1/mcp/map-weather-api-to-mcp-tools.md index 8a92cdb7511..9d9b71bb6b3 100644 --- a/app/_how-tos/mcp/map-weather-api-to-mcp-tools.md +++ b/app/_how-tos/ai-gateway/v1/mcp/map-weather-api-to-mcp-tools.md @@ -3,7 +3,7 @@ title: Map Weather API to MCP tools content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI MCP Proxy url: /plugins/ai-mcp-proxy/ @@ -12,7 +12,7 @@ description: | products: - gateway - ai-gateway -permalink: /mcp/map-weather-api-to-mcp-tools/ +permalink: /ai-gateway/v1/mcp/map-weather-api-to-mcp-tools/ series: id: mcp-weather-api @@ -73,6 +73,8 @@ prereqs: konnect: - name: KONG_STATUS_LISTEN value: '0.0.0.0:8100' +major_version: + ai-gateway: 1 --- diff --git a/app/_how-tos/mcp/observe-autogenerated-mcp-tools-for-weather-api.md b/app/_how-tos/ai-gateway/v1/mcp/observe-autogenerated-mcp-tools-for-weather-api.md similarity index 97% rename from app/_how-tos/mcp/observe-autogenerated-mcp-tools-for-weather-api.md rename to app/_how-tos/ai-gateway/v1/mcp/observe-autogenerated-mcp-tools-for-weather-api.md index d636d8a2970..31eb38f001f 100644 --- a/app/_how-tos/mcp/observe-autogenerated-mcp-tools-for-weather-api.md +++ b/app/_how-tos/ai-gateway/v1/mcp/observe-autogenerated-mcp-tools-for-weather-api.md @@ -3,7 +3,7 @@ title: Log MCP traffic for autogenerated MCP Weather API tools content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI MCP Proxy url: /plugins/ai-mcp-proxy/ - text: HTTP Long @@ -16,7 +16,7 @@ products: - gateway - ai-gateway -permalink: /mcp/observe-autogenerated-mcp-tools-for-weather-api/ +permalink: /ai-gateway/v1/mcp/observe-autogenerated-mcp-tools-for-weather-api/ series: id: mcp-weather-api @@ -60,6 +60,9 @@ cleanup: icon_url: /assets/icons/gateway.svg automated_tests: false +major_version: + ai-gateway: 1 + --- ## Reconfigure the AI MCP Proxy plugin diff --git a/app/_how-tos/mcp/observe-mcp-traffic-with-acls.md b/app/_how-tos/ai-gateway/v1/mcp/observe-mcp-traffic-with-acls.md similarity index 97% rename from app/_how-tos/mcp/observe-mcp-traffic-with-acls.md rename to app/_how-tos/ai-gateway/v1/mcp/observe-mcp-traffic-with-acls.md index 3370046b644..e5ed8931015 100644 --- a/app/_how-tos/mcp/observe-mcp-traffic-with-acls.md +++ b/app/_how-tos/ai-gateway/v1/mcp/observe-mcp-traffic-with-acls.md @@ -3,7 +3,7 @@ title: Observe MCP Traffic with Access Control Enabled content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI MCP Proxy url: /plugins/ai-mcp-proxy/ @@ -15,7 +15,7 @@ products: - ai-gateway - insomnia -permalink: /mcp/observe-mcp-traffic-with-acls/ +permalink: /ai-gateway/v1/mcp/observe-mcp-traffic-with-acls/ series: id: mcp-acls @@ -68,6 +68,9 @@ prereqs: value: '0.0.0.0:8100' automated_tests: false +major_version: + ai-gateway: 1 + --- ## Configure MCP tools in Chatwise diff --git a/app/_how-tos/mcp/observe-mcp-traffic.md b/app/_how-tos/ai-gateway/v1/mcp/observe-mcp-traffic.md similarity index 97% rename from app/_how-tos/mcp/observe-mcp-traffic.md rename to app/_how-tos/ai-gateway/v1/mcp/observe-mcp-traffic.md index 1b43538e7cc..8766523654a 100644 --- a/app/_how-tos/mcp/observe-mcp-traffic.md +++ b/app/_how-tos/ai-gateway/v1/mcp/observe-mcp-traffic.md @@ -3,16 +3,16 @@ title: "Observe GitHub MCP traffic with {{site.ai_gateway}}" content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ - text: Prometheus plugin url: /plugins/prometheus/ - text: Monitor AI LLM metrics url: /ai-gateway/monitor-ai-llm-metrics/ -permalink: /mcp/observe-mcp-traffic/ +permalink: /ai-gateway/v1/mcp/observe-mcp-traffic/ breadcrumbs: - - /mcp/ + - /ai-gateway/v1/mcp/ series: id: mcp-traffic @@ -63,6 +63,9 @@ cleanup: automated_tests: false +major_version: + ai-gateway: 1 + --- ## Reconfigure the AI Proxy Advanced plugin diff --git a/app/_how-tos/mcp/observe-traffic-for-mcp-tools.md b/app/_how-tos/ai-gateway/v1/mcp/observe-traffic-for-mcp-tools.md similarity index 97% rename from app/_how-tos/mcp/observe-traffic-for-mcp-tools.md rename to app/_how-tos/ai-gateway/v1/mcp/observe-traffic-for-mcp-tools.md index 1cd6dfa46df..9de0e5bffa2 100644 --- a/app/_how-tos/mcp/observe-traffic-for-mcp-tools.md +++ b/app/_how-tos/ai-gateway/v1/mcp/observe-traffic-for-mcp-tools.md @@ -3,13 +3,13 @@ title: Observe MCP traffic for autogenerated MCP tools content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI MCP Proxy url: /plugins/ai-mcp-proxy/ - text: Prometheus plugin url: /plugins/prometheus/ - text: A trust and control layer for proxying traffic to MCP servers - url: /mcp/ + url: /ai-gateway/v1/mcp/ description: Learn how to monitor traffic for autogenerated MCP tools using the AI MCP Proxy plugin and Prometheus, so you can track tool usage and latency. @@ -17,7 +17,7 @@ products: - gateway - ai-gateway -permalink: /mcp/observe-traffic-for-mcp-tools/ +permalink: /ai-gateway/v1/mcp/observe-traffic-for-mcp-tools/ series: id: mcp-conversion @@ -72,6 +72,9 @@ cleanup: icon_url: /assets/icons/gateway.svg automated_tests: false +major_version: + ai-gateway: 1 + --- ## Reconfigure the AI MCP Proxy plugin diff --git a/app/_how-tos/mcp/secure-mcp-tools-with-oauth2-and-okta.md b/app/_how-tos/ai-gateway/v1/mcp/secure-mcp-tools-with-oauth2-and-okta.md similarity index 98% rename from app/_how-tos/mcp/secure-mcp-tools-with-oauth2-and-okta.md rename to app/_how-tos/ai-gateway/v1/mcp/secure-mcp-tools-with-oauth2-and-okta.md index 471f19e864e..d5ef88a844f 100644 --- a/app/_how-tos/mcp/secure-mcp-tools-with-oauth2-and-okta.md +++ b/app/_how-tos/ai-gateway/v1/mcp/secure-mcp-tools-with-oauth2-and-okta.md @@ -1,9 +1,9 @@ --- title: Secure MCP tools with OAuth2 and Okta content_type: how_to -permalink: /mcp/secure-mcp-tools-with-oauth2-and-okta/ +permalink: /ai-gateway/v1/mcp/secure-mcp-tools-with-oauth2-and-okta/ breadcrumbs: - - /mcp/ + - /ai-gateway/v1/mcp/ description: Use the AI MCP OAuth2 plugin with Okta to protect MCP tools exposed through the AI MCP Proxy plugin @@ -45,7 +45,7 @@ tools: related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI MCP Proxy url: /plugins/ai-mcp-proxy/ - text: AI MCP OAuth2 @@ -72,11 +72,11 @@ prereqs: 1. Ensure you have Node.js and npm installed. If needed, download them from https://nodejs.org. - 1. Update `npx` to the latest version: + 2. Update `npx` to the latest version: ```sh npm install -g npx ``` - 1. Install the Inspector: + 3. Install the Inspector: ```sh npm install -g @modelcontextprotocol/inspector ``` @@ -98,6 +98,9 @@ cleanup: icon_url: /assets/icons/gateway.svg automated_tests: false +major_version: + ai-gateway: 1 + --- ## Configure the AI MCP Proxy tools diff --git a/app/_how-tos/mcp/secure-mcp-traffic.md b/app/_how-tos/ai-gateway/v1/mcp/secure-mcp-traffic.md similarity index 97% rename from app/_how-tos/mcp/secure-mcp-traffic.md rename to app/_how-tos/ai-gateway/v1/mcp/secure-mcp-traffic.md index cf72c16db99..85e60ae5afe 100644 --- a/app/_how-tos/mcp/secure-mcp-traffic.md +++ b/app/_how-tos/ai-gateway/v1/mcp/secure-mcp-traffic.md @@ -3,14 +3,14 @@ title: "Secure GitHub MCP Server traffic with {{ site.base_gateway }} and {{site content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advance/ - text: Key Auth plugin url: /plugins/key-auth/ -permalink: /mcp/secure-mcp-traffic/ +permalink: /ai-gateway/v1/mcp/secure-mcp-traffic/ breadcrumbs: - - /mcp/ + - /ai-gateway/v1/mcp/ description: Learn how to secure MCP traffic within GitHub remote MCP server with the Key Authentication plugin @@ -83,6 +83,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy Advanced plugin diff --git a/app/_how-tos/mcp/use-access-controls-for-mcp-tools.md b/app/_how-tos/ai-gateway/v1/mcp/use-access-controls-for-mcp-tools.md similarity index 98% rename from app/_how-tos/mcp/use-access-controls-for-mcp-tools.md rename to app/_how-tos/ai-gateway/v1/mcp/use-access-controls-for-mcp-tools.md index 9d6e65bfd2e..c6396869356 100644 --- a/app/_how-tos/mcp/use-access-controls-for-mcp-tools.md +++ b/app/_how-tos/ai-gateway/v1/mcp/use-access-controls-for-mcp-tools.md @@ -3,7 +3,7 @@ title: Control MCP tool access with Consumer and Consumer Group ACLs content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI MCP Proxy url: /plugins/ai-mcp-proxy/ @@ -14,7 +14,7 @@ products: - ai-gateway - insomnia -permalink: /mcp/use-access-controls-for-mcp-tools/ +permalink: /ai-gateway/v1/mcp/use-access-controls-for-mcp-tools/ series: id: mcp-acls @@ -97,6 +97,9 @@ faqs: a: | Prior to {{site.ai_gateway}} 3.14, requests that matched an MCP ACL deny rule or failed to match an allow list returned the JSON-RPC error code `INVALID_PARAMS -32602`. This has now changed to match the [MCP 2025-11-25 authorization specification](https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#error-handling) and returns `HTTP 403 Forbidden`. +major_version: + ai-gateway: 1 + --- ## Set up Consumer authentication diff --git a/app/_how-tos/ai-gateway/meter-llm-traffic.md b/app/_how-tos/ai-gateway/v1/meter-llm-traffic.md similarity index 98% rename from app/_how-tos/ai-gateway/meter-llm-traffic.md rename to app/_how-tos/ai-gateway/v1/meter-llm-traffic.md index d497a37cde0..d2bd616751b 100644 --- a/app/_how-tos/ai-gateway/meter-llm-traffic.md +++ b/app/_how-tos/ai-gateway/v1/meter-llm-traffic.md @@ -1,14 +1,12 @@ --- title: Monetize LLM traffic in {{site.konnect_short_name}} -permalink: /how-to/meter-llm-traffic/ +permalink: /ai-gateway/v1/how-to/meter-llm-traffic/ description: Learn how to Meter LLM traffic using {{site.konnect_short_name}} {{site.metering_and_billing}}. content_type: how_to -breadcrumbs: - - /metering-and-billing/ - products: - gateway + - ai-gateway - metering-and-billing works_on: @@ -45,7 +43,7 @@ tldr: To meter LLM traffic in {{site.konnect_short_name}}, you can use the {{site.metering_and_billing}} to track and invoice usage based on defined products, plans, and features. This guide walks you through setting up a Consumer, creating a meter for LLM tokens, defining a feature, creating a Plan with Rate Cards, and starting a subscription for billing. related_resources: - text: "{{site.ai_gateway_name}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: Product Catalog reference url: /metering-and-billing/product-catalog/ - text: Metering reference @@ -65,6 +63,9 @@ faqs: {% include faqs/metering-and-billing-legacy-ingestion.md %} automated_tests: false +major_version: + ai-gateway: 1 + --- This getting-started guide shows how to meter LLM traffic—such as token consumption or model-specific usage—from {{site.base_gateway}} and convert that raw LLM activity into billable usage with {{site.metering_and_billing}} in {{site.konnect_short_name}}. diff --git a/app/_how-tos/ai-gateway/protect-sensitive-information-output-with-ai.md b/app/_how-tos/ai-gateway/v1/protect-sensitive-information-output-with-ai.md similarity index 96% rename from app/_how-tos/ai-gateway/protect-sensitive-information-output-with-ai.md rename to app/_how-tos/ai-gateway/v1/protect-sensitive-information-output-with-ai.md index 9fc54f71f68..5e9a29b462d 100644 --- a/app/_how-tos/ai-gateway/protect-sensitive-information-output-with-ai.md +++ b/app/_how-tos/ai-gateway/v1/protect-sensitive-information-output-with-ai.md @@ -1,6 +1,6 @@ --- title: Use AI PII Sanitizer plugin to protect sensitive data in responses -permalink: /how-to/protect-sensitive-information-output-with-ai/ +permalink: /ai-gateway/v1/how-to/protect-sensitive-information-output-with-ai/ content_type: how_to description: Use the AI PII Sanitizer plugin to protect sensitive information in responses from a Mistral LLM model. @@ -62,9 +62,11 @@ min_version: related_resources: - text: Use AI PII Sanitizer plugin to protect sensitive information in responses - url: /how-to/protect-sensitive-information-output-with-ai/ + url: /ai-gateway/v1/how-to/protect-sensitive-information-output-with-ai/ - text: AI PII Sanitizer url: /plugins/ai-sanitizer/ +major_version: + ai-gateway: 1 --- ## Start the Kong AI PII Sanitizer service diff --git a/app/_how-tos/ai-gateway/protect-sensitive-information-with-ai.md b/app/_how-tos/ai-gateway/v1/protect-sensitive-information-with-ai.md similarity index 95% rename from app/_how-tos/ai-gateway/protect-sensitive-information-with-ai.md rename to app/_how-tos/ai-gateway/v1/protect-sensitive-information-with-ai.md index 229347dba28..87c1032774a 100644 --- a/app/_how-tos/ai-gateway/protect-sensitive-information-with-ai.md +++ b/app/_how-tos/ai-gateway/v1/protect-sensitive-information-with-ai.md @@ -1,6 +1,6 @@ --- title: Use AI PII Sanitizer to protect sensitive data in requests -permalink: /how-to/protect-sensitive-information-with-ai/ +permalink: /ai-gateway/v1/how-to/protect-sensitive-information-with-ai/ content_type: how_to description: Use the AI Sanitizer plugin to protect sensitive information in requests. @@ -61,9 +61,12 @@ min_version: related_resources: - text: Use AI PII Sanitizer plugin to protect sensitive information in responses - url: /how-to/protect-sensitive-information-output-with-ai/ + url: /ai-gateway/v1/how-to/protect-sensitive-information-output-with-ai/ - text: AI PII Sanitizer url: /plugins/ai-sanitizer/ +major_version: + ai-gateway: 1 + --- ## Start the Kong AI PII Sanitizer service diff --git a/app/_how-tos/ai-gateway/proxy-a2a-agents.md b/app/_how-tos/ai-gateway/v1/proxy-a2a-agents.md similarity index 99% rename from app/_how-tos/ai-gateway/proxy-a2a-agents.md rename to app/_how-tos/ai-gateway/v1/proxy-a2a-agents.md index fe68b20d032..62b286d8ed4 100644 --- a/app/_how-tos/ai-gateway/proxy-a2a-agents.md +++ b/app/_how-tos/ai-gateway/v1/proxy-a2a-agents.md @@ -24,7 +24,7 @@ entities: - route - plugin -permalink: /how-to/proxy-a2a-agents/ +permalink: /ai-gateway/v1/how-to/proxy-a2a-agents/ tags: - ai @@ -42,7 +42,7 @@ related_resources: - text: A2A protocol specification url: https://a2a-protocol.org/latest/ - text: Set up Jaeger with Gen AI OpenTelemetry - url: /how-to/set-up-jaeger-with-gen-ai-otel/ + url: /ai-gateway/v1/how-to/set-up-jaeger-with-gen-ai-otel/ - text: Agentic usage analytics in {{site.konnect_short_name}} url: /observability/explorer/?tab=agentic-usage#metrics @@ -129,6 +129,9 @@ faqs: independently of authentication. automated_tests: false +major_version: + ai-gateway: 1 + --- ## Enable the AI A2A Proxy plugin diff --git a/app/_how-tos/ai-gateway/v1/rate-limit-a2a-traffic.md b/app/_how-tos/ai-gateway/v1/rate-limit-a2a-traffic.md new file mode 100644 index 00000000000..419ab088bcb --- /dev/null +++ b/app/_how-tos/ai-gateway/v1/rate-limit-a2a-traffic.md @@ -0,0 +1,211 @@ +--- +title: "Rate limit A2A traffic" +content_type: how_to +description: "Apply per-consumer rate limits to A2A routes proxied through {{site.ai_gateway}}" + +products: + - gateway + - ai-gateway + +works_on: + - on-prem + - konnect + +min_version: + gateway: '3.14' + +plugins: + - ai-a2a-proxy + - key-auth + - rate-limiting-advanced + +entities: + - service + - route + - plugin + - consumer + +permalink: /ai-gateway/v1/how-to/rate-limit-a2a-traffic/ + +tags: + - ai + - a2a + - traffic-control + +tldr: + q: "How do I rate limit A2A traffic in {{site.ai_gateway}}?" + a: "Enable the Rate Limiting Advanced plugin on the same service or route as the AI A2A Proxy plugin. Combined with an authentication plugin, rate limits apply per consumer. Requests that exceed the limit are rejected with 429." +tools: + - deck + +related_resources: + - text: AI A2A Proxy plugin reference + url: /plugins/ai-a2a-proxy/ + - text: Rate Limiting Advanced plugin reference + url: /plugins/rate-limiting-advanced/ + - text: "Proxy A2A agents through {{site.ai_gateway}}" + url: /ai-gateway/v1/how-to/proxy-a2a-agents/ + - text: Secure A2A endpoints with key authentication + url: /ai-gateway/v1/how-to/secure-a2a-endpoints/ +prereqs: + entities: + services: + - a2a-kongair-agent + routes: + - a2a-kongair-route + inline: + - title: OpenAI API key + include_content: prereqs/openai + icon_url: /assets/icons/openai.svg + - title: A2A agent + include_content: prereqs/a2a-kongair-agent + icon_url: /assets/icons/ai.svg + +cleanup: + inline: + - title: Clean up Konnect environment + include_content: cleanup/platform/konnect + icon_url: /assets/icons/gateway.svg + - title: Destroy the {{site.base_gateway}} container + include_content: cleanup/products/gateway + icon_url: /assets/icons/gateway.svg + +faqs: + - q: Can I rate limit A2A traffic without authentication? + a: | + Yes. Without an authentication plugin, the Rate Limiting Advanced plugin falls back to rate limiting by IP address. Add an authentication plugin if you need per-consumer + limits. + - q: Does rate limiting affect A2A streaming responses? + a: | + Rate limiting applies at request time, before the upstream responds. A streaming SSE response that is already in progress is not interrupted. The rate limit check happens when the client sends the next request. + - q: Can I use AI Rate Limiting Advanced instead? + a: | + AI Rate Limiting Advanced limits based on LLM token consumption (prompt and completion tokens). The AI A2A Proxy plugin does not extract token counts from A2A responses, so AI Rate Limiting Advanced has no token data to act on. Use the standard Rate Limiting Advanced plugin for A2A traffic. + +automated_tests: false +major_version: + ai-gateway: 1 + +--- + +## Enable the AI A2A Proxy plugin + +The AI A2A Proxy plugin parses A2A JSON-RPC requests and proxies them to the upstream agent. + +{% entity_examples %} +entities: + plugins: + - name: ai-a2a-proxy + config: + logging: + log_statistics: true + log_payloads: true +{% endentity_examples %} + + +## Enable the Rate Limiting Advanced plugin + +The [Rate Limiting Advanced plugin](/plugins/rate-limiting-advanced/) counts requests per consumer and rejects requests that exceed the configured limit. This configuration allows 5 requests per 30 seconds, intentionally low to make it easy to trigger during testing. + +{% entity_examples %} +entities: + plugins: + - name: rate-limiting-advanced + config: + limit: + - 5 + window_size: + - 30 + sync_rate: -1 + namespace: a2a-kongair-agent + strategy: local +{% endentity_examples %} + +{:.info} +> Set `limit` and `window_size` to values appropriate for your production workload. +> The values in this guide are intentionally low for testing. + +## Validate rate limit headers + +Send an authenticated request to the agent card endpoint and inspect the response headers. The agent card is a lightweight A2A operation (`GetAgentCard`) that returns agent metadata without calling an LLM, so responses are instant. + + +{% validation request-check %} +url: /a2a/.well-known/agent-card.json +display_headers: true +status_code: 200 +method: GET +headers: + - 'apikey: a2a-secret-key-1' +{% endvalidation %} + + +The response includes rate limit headers: + +``` +HTTP/2 200 +... +ratelimit-limit: 5 +ratelimit-remaining: 4 +ratelimit-reset: 30 +x-ratelimit-limit-30: 5 +x-ratelimit-remaining-30: 4 +``` +{:.no-copy-code} + +`ratelimit-remaining` decreases with each request. `ratelimit-reset` shows the seconds until the window resets. + +## Validate rate limit enforcement + +Send 6 requests to the agent card endpoint in a loop to exceed the limit. The AI A2A Proxy plugin detects each request as an A2A `GetAgentCard` operation, so the rate limit applies the same way it does for `message/send` or any other A2A method. + +{% on_prem %} +content: | + ```sh + for i in $(seq 1 6); do + echo "--- Request $i ---" + curl -s -o /dev/null -w "HTTP status: %{http_code}\n"\ + http://localhost:8000/a2a/.well-known/agent-card.json \ + -H "apikey: a2a-secret-key-1" + done + ``` +{% endon_prem %} + +{% konnect %} +content: | + ```sh + for i in $(seq 1 6); do + echo "--- Request $i ---" + curl -s -o /dev/null -w "HTTP status: %{http_code}\n"\ + $KONNECT_PROXY_URL/a2a/.well-known/agent-card.json \ + -H "apikey: a2a-secret-key-1" + done + ``` +{% endkonnect %} + +The first 5 requests return `HTTP status: 200`. The 6th request returns `HTTP status: 429`: + +``` +--- Request 1 --- +HTTP status: 200 +--- Request 2 --- +HTTP status: 200 +--- Request 3 --- +HTTP status: 200 +--- Request 4 --- +HTTP status: 200 +--- Request 5 --- +HTTP status: 200 +--- Request 6 --- +HTTP status: 429 +``` +{:.no-copy-code} + +The `429` response body contains: + +```json +{ + "message": "API rate limit exceeded" +} +``` +{:.no-copy-code} \ No newline at end of file diff --git a/app/_how-tos/ai-gateway/rotate-secrets-in-google-cloud-secret.md b/app/_how-tos/ai-gateway/v1/rotate-secrets-in-google-cloud-secret.md similarity index 99% rename from app/_how-tos/ai-gateway/rotate-secrets-in-google-cloud-secret.md rename to app/_how-tos/ai-gateway/v1/rotate-secrets-in-google-cloud-secret.md index 36c18b39f1a..b2a7b17a717 100644 --- a/app/_how-tos/ai-gateway/rotate-secrets-in-google-cloud-secret.md +++ b/app/_how-tos/ai-gateway/v1/rotate-secrets-in-google-cloud-secret.md @@ -1,6 +1,6 @@ --- title: Store and rotate Mistral API keys as secrets in Google Cloud -permalink: /how-to/rotate-secrets-in-google-cloud-secret/ +permalink: /ai-gateway/v1/how-to/rotate-secrets-in-google-cloud-secret/ content_type: how_to related_resources: - text: Configure Google Cloud Secret as a vault backend @@ -124,6 +124,9 @@ cleanup: icon_url: /assets/icons/gateway.svg automated_tests: false +major_version: + ai-gateway: 1 + --- ## Add an invalid API key as a secret in {{ site.google_cloud }} Secret Manager diff --git a/app/_how-tos/ai-gateway/route-azure-sdk-to-multiple-azure-deployments.md b/app/_how-tos/ai-gateway/v1/route-azure-sdk-to-multiple-azure-deployments.md similarity index 97% rename from app/_how-tos/ai-gateway/route-azure-sdk-to-multiple-azure-deployments.md rename to app/_how-tos/ai-gateway/v1/route-azure-sdk-to-multiple-azure-deployments.md index 8da031a2b39..dec1d650ae7 100644 --- a/app/_how-tos/ai-gateway/route-azure-sdk-to-multiple-azure-deployments.md +++ b/app/_how-tos/ai-gateway/v1/route-azure-sdk-to-multiple-azure-deployments.md @@ -3,13 +3,13 @@ title: Route Azure AI SDK requests to Azure OpenAI deployments content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ - text: "AI Proxy Advanced: Dynamic Azure deployments" url: /plugins/ai-proxy-advanced/examples/sdk-azure-one-route/ -permalink: /how-to/route-azure-sdk-to-multiple-azure-deployments +permalink: /ai-gateway/v1/how-to/route-azure-sdk-to-multiple-azure-deployments description: Configure a single Route that dynamically maps OpenAI SDK requests to different Azure OpenAI deployments based on the URL path. @@ -70,6 +70,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- The [Azure OpenAI SDK](https://github.com/openai/openai-python#microsoft-azure-openai) can connect to [Azure OpenAI Service](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/chatgpt) through {{site.ai_gateway}}. With Azure, the `model` parameter in SDK calls maps to a deployment name on your Azure instance. The SDK constructs request URLs in the format `https://{azure_instance}.openai.azure.com/openai/deployments/{azure_deployment_id}/chat/completions`. When the SDK sends a request to `/openai/deployments/gpt-4o/chat/completions`, the Route captures `gpt-4o` into the `azure_deployment` named group. diff --git a/app/_how-tos/ai-gateway/route-azure-sdk-to-specific-deployments.md b/app/_how-tos/ai-gateway/v1/route-azure-sdk-to-specific-deployments.md similarity index 97% rename from app/_how-tos/ai-gateway/route-azure-sdk-to-specific-deployments.md rename to app/_how-tos/ai-gateway/v1/route-azure-sdk-to-specific-deployments.md index 0bd513c2ae5..fb2d10b54b7 100644 --- a/app/_how-tos/ai-gateway/route-azure-sdk-to-specific-deployments.md +++ b/app/_how-tos/ai-gateway/v1/route-azure-sdk-to-specific-deployments.md @@ -3,13 +3,13 @@ title: Route Azure OpenAI SDK requests to specific deployments with multiple rou content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ - text: "AI Proxy Advanced: Multi-deployment chat routing example" url: /plugins/ai-proxy-advanced/examples/sdk-multiple-azure-deployments/ -permalink: /how-to/route-azure-sdk-to-specific-deployments +permalink: /ai-gateway/v1/how-to/route-azure-sdk-to-specific-deployments description: Configure separate {{site.base_gateway}} Routes that map to specific Azure OpenAI deployments, each with its own AI Proxy Advanced configuration. @@ -71,6 +71,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- The [Azure OpenAI SDK](https://github.com/openai/openai-python#microsoft-azure-openai) constructs request URLs in the format `https://{azure_instance}.openai.azure.com/openai/deployments/{deployment_id}/chat/completions`. Each deployment has its own URL path. diff --git a/app/_how-tos/ai-gateway/route-requests-by-model-alias.md b/app/_how-tos/ai-gateway/v1/route-requests-by-model-alias.md similarity index 96% rename from app/_how-tos/ai-gateway/route-requests-by-model-alias.md rename to app/_how-tos/ai-gateway/v1/route-requests-by-model-alias.md index 57c4ed47fa5..ffb1452c570 100644 --- a/app/_how-tos/ai-gateway/route-requests-by-model-alias.md +++ b/app/_how-tos/ai-gateway/v1/route-requests-by-model-alias.md @@ -1,12 +1,12 @@ --- title: Route requests to different models using model aliases -permalink: /how-to/route-requests-by-model-alias/ +permalink: /ai-gateway/v1/how-to/route-requests-by-model-alias/ content_type: how_to description: Use model aliases in the AI Proxy Advanced plugin to route requests to different upstream models based on the model field in the request body breadcrumbs: - - /ai-gateway/ + - /ai-gateway/v1/ products: - gateway @@ -41,7 +41,7 @@ tools: related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ @@ -64,6 +64,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy Advanced plugin diff --git a/app/_how-tos/ai-gateway/secure-a2a-traffic.md b/app/_how-tos/ai-gateway/v1/secure-a2a-traffic.md similarity index 96% rename from app/_how-tos/ai-gateway/secure-a2a-traffic.md rename to app/_how-tos/ai-gateway/v1/secure-a2a-traffic.md index 3dc3e988de7..e9ae32e59ff 100644 --- a/app/_how-tos/ai-gateway/secure-a2a-traffic.md +++ b/app/_how-tos/ai-gateway/v1/secure-a2a-traffic.md @@ -24,7 +24,7 @@ entities: - plugin - consumer -permalink: /how-to/secure-a2a-endpoints/ +permalink: /ai-gateway/v1/how-to/secure-a2a-endpoints/ tags: - ai @@ -43,10 +43,9 @@ related_resources: - text: Key Auth plugin reference url: /plugins/key-auth/ - text: "Proxy A2A agents through {{site.ai_gateway}}" - url: /how-to/proxy-a2a-agents/ + url: /ai-gateway/v1/how-to/proxy-a2a-agents/ - text: Rate limit A2A traffic - url: /how-to/rate-limit-a2a-traffic/ - + url: /ai-gateway/v1/how-to/rate-limit-a2a-traffic/ prereqs: entities: services: @@ -80,6 +79,9 @@ faqs: Yes. Any {{site.ai_gateway}} authentication plugin works with A2A routes: [JWT](/plugins/jwt/), [OpenID Connect](/plugins/openid-connect/), [OAuth2](/plugins/oauth2/), and others. The AI A2A Proxy plugin operates independently of the authentication method. automated_tests: false +major_version: + ai-gateway: 1 + --- ## Enable the AI A2A Proxy plugin diff --git a/app/_how-tos/ai-gateway/secure-a2a-with-oidc.md b/app/_how-tos/ai-gateway/v1/secure-a2a-with-oidc.md similarity index 96% rename from app/_how-tos/ai-gateway/secure-a2a-with-oidc.md rename to app/_how-tos/ai-gateway/v1/secure-a2a-with-oidc.md index 27051d28377..54d4b77d182 100644 --- a/app/_how-tos/ai-gateway/secure-a2a-with-oidc.md +++ b/app/_how-tos/ai-gateway/v1/secure-a2a-with-oidc.md @@ -1,6 +1,6 @@ --- title: Secure A2A endpoints with OpenID Connect and Okta -permalink: /how-to/secure-a2a-endpoints-with-oidc/ +permalink: /ai-gateway/v1/how-to/secure-a2a-endpoints-with-oidc/ content_type: how_to description: Add OpenID Connect authentication to A2A routes proxied through {{site.ai_gateway}} using Okta @@ -48,10 +48,9 @@ related_resources: - text: OpenID Connect plugin reference url: /plugins/openid-connect/ - text: "Proxy A2A agents through {{site.ai_gateway}}" - url: /how-to/proxy-a2a-agents/ + url: /ai-gateway/v1/how-to/proxy-a2a-agents/ - text: Secure A2A endpoints with key authentication - url: /how-to/secure-a2a-endpoints/ - + url: /ai-gateway/v1/how-to/secure-a2a-endpoints/ prereqs: entities: services: @@ -87,6 +86,9 @@ faqs: Yes. The [OpenID Connect plugin](/plugins/openid-connect/) works with any OIDC-compliant identity provider (Keycloak, Auth0, Azure AD, etc.). Replace the `issuer`, `client_id`, and `client_secret` with values from your provider. automated_tests: false +major_version: + ai-gateway: 1 + --- ## Enable the AI A2A Proxy plugin diff --git a/app/_how-tos/ai-gateway/send-asynchronous-llm-requests.md b/app/_how-tos/ai-gateway/v1/send-asynchronous-llm-requests.md similarity index 99% rename from app/_how-tos/ai-gateway/send-asynchronous-llm-requests.md rename to app/_how-tos/ai-gateway/v1/send-asynchronous-llm-requests.md index 24370450e41..5448e2cbca2 100644 --- a/app/_how-tos/ai-gateway/send-asynchronous-llm-requests.md +++ b/app/_how-tos/ai-gateway/v1/send-asynchronous-llm-requests.md @@ -1,10 +1,10 @@ --- title: Send asynchronous requests to LLMs -permalink: /how-to/send-asynchronous-llm-requests/ +permalink: /ai-gateway/v1/how-to/send-asynchronous-llm-requests/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ @@ -78,6 +78,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure AI Proxy plugins diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-anthropic.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-anthropic.md similarity index 94% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-anthropic.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-anthropic.md index d0ea83cd94d..23c08edb13f 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-anthropic.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-anthropic.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy Advanced with Anthropic in {{site.base_gateway}} -permalink: /how-to/set-up-ai-proxy-advanced-with-anthropic/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-advanced-with-anthropic/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ @@ -62,6 +62,8 @@ cleanup: include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 --- diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-aws-bedrock.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-aws-bedrock.md similarity index 95% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-aws-bedrock.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-aws-bedrock.md index c78065b69c5..2ca3220556e 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-aws-bedrock.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-aws-bedrock.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy Advanced with AWS Bedrock in {{site.base_gateway}}. -permalink: /how-to/set-up-ai-proxy-advanced-with-aws-bedrock/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-advanced-with-aws-bedrock/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ @@ -73,6 +73,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-cerebras.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-cerebras.md similarity index 94% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-cerebras.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-cerebras.md index a5a1006c03d..96cae536d0f 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-cerebras.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-cerebras.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy Advanced with Cerebras in {{site.base_gateway}} -permalink: /how-to/set-up-ai-proxy-advanced-with-cerebras/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-advanced-with-cerebras/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ @@ -67,6 +67,8 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 --- diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-cohere.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-cohere.md similarity index 94% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-cohere.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-cohere.md index 3bc5b5eb895..2b3b5928a72 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-cohere.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-cohere.md @@ -1,12 +1,12 @@ --- title: Set up AI Proxy Advanced with Cohere in {{site.base_gateway}} -permalink: /how-to/set-up-ai-proxy-advanced-with-cohere/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-advanced-with-cohere/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ @@ -61,6 +61,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-dashscope.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-dashscope.md similarity index 94% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-dashscope.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-dashscope.md index 503bf46bc25..3bb650efd61 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-dashscope.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-dashscope.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy Advanced with DashScope (Alibaba Cloud) in {{site.base_gateway}} -permalink: /how-to/set-up-ai-proxy-advanced-with-dashscope/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-advanced-with-dashscope/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ @@ -65,6 +65,9 @@ cleanup: icon_url: /assets/icons/gateway.svg automated_tests: false +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-databricks.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-databricks.md similarity index 93% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-databricks.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-databricks.md index 49988d5935f..24240008485 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-databricks.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-databricks.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy Advanced with Databricks -permalink: /how-to/set-up-ai-proxy-advanced-with-databricks/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-advanced-with-databricks/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ @@ -59,6 +59,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-deepseek.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-deepseek.md similarity index 94% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-deepseek.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-deepseek.md index c484af71cfc..dda1bd8129a 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-deepseek.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-deepseek.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy Advanced with DeepSeek in {{site.base_gateway}} -permalink: /how-to/set-up-ai-proxy-advanced-with-deepseek/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-advanced-with-deepseek/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ @@ -60,6 +60,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-gemini.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-gemini.md similarity index 96% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-gemini.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-gemini.md index d5bb797f271..476364662f8 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-gemini.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-gemini.md @@ -1,12 +1,12 @@ --- title: Set up AI Proxy Advanced with Gemini in {{site.base_gateway}} -permalink: /how-to/set-up-ai-proxy-advanced-with-gemini/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-advanced-with-gemini/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ @@ -80,6 +80,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-huggingface.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-huggingface.md similarity index 94% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-huggingface.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-huggingface.md index b56c5bbb4e0..cd59c91594b 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-huggingface.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-huggingface.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy Advanced with HuggingFace in {{site.base_gateway}} -permalink: /how-to/set-up-ai-proxy-advanced-with-huggingface/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-advanced-with-huggingface/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ @@ -63,6 +63,8 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 --- diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-ollama-qwen.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-ollama-qwen.md similarity index 93% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-ollama-qwen.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-ollama-qwen.md index 9bb2eb8f490..c225b9d17dc 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-ollama-qwen.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-ollama-qwen.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy Advanced with Ollama and a Qwen model -permalink: /how-to/set-up-ai-proxy-advanced-with-ollama-qwen/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-advanced-with-ollama-qwen/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ @@ -59,6 +59,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-ollama.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-ollama.md similarity index 93% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-ollama.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-ollama.md index a1767aae7e1..5a1a21dc97d 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-ollama.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-ollama.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy Advanced with Ollama -permalink: /how-to/set-up-ai-proxy-advanced-with-ollama/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-advanced-with-ollama/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ @@ -59,6 +59,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-openai.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-openai.md similarity index 94% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-openai.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-openai.md index e8aca051ecb..b180bc8174c 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-openai.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-openai.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy Advanced with OpenAI in {{site.base_gateway}} -permalink: /how-to/set-up-ai-proxy-advanced-with-openai/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-advanced-with-openai/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ @@ -59,6 +59,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-vertex-ai.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-vertex-ai.md similarity index 94% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-vertex-ai.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-vertex-ai.md index 9a6e490b8b5..4c5b8f7aee3 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-advanced-with-vertex-ai.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-advanced-with-vertex-ai.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy Advanced with Vertex AI in {{site.base_gateway}} -permalink: /how-to/set-up-ai-proxy-advanced-with-vertex-ai/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-advanced-with-vertex-ai/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ @@ -59,6 +59,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-for-image-generation-with-grok.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-for-image-generation-with-grok.md similarity index 94% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-for-image-generation-with-grok.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-for-image-generation-with-grok.md index 1fc09fc6b8b..96833a0b7e2 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-for-image-generation-with-grok.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-for-image-generation-with-grok.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy for image generation with Grok -permalink: /how-to/set-up-ai-proxy-for-image-generation-with-grok/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-for-image-generation-with-grok/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ @@ -59,6 +59,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-anthropic.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-anthropic.md similarity index 94% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-with-anthropic.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-anthropic.md index bf3450d63bd..6466632bab0 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-anthropic.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-anthropic.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy with Anthropic in {{site.base_gateway}} -permalink: /how-to/set-up-ai-proxy-with-anthropic/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-with-anthropic/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ @@ -59,6 +59,8 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 --- diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-aws-bedrock.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-aws-bedrock.md similarity index 95% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-with-aws-bedrock.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-aws-bedrock.md index 863ef609a54..3bb2da64be0 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-aws-bedrock.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-aws-bedrock.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy with AWS Bedrock in {{site.base_gateway}}. -permalink: /how-to/set-up-ai-proxy-with-aws-bedrock/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-with-aws-bedrock/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ @@ -73,6 +73,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-cerebras.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-cerebras.md similarity index 95% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-with-cerebras.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-cerebras.md index 1f31872392e..cdbd67c1987 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-cerebras.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-cerebras.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy with Cerebras in {{site.base_gateway}} -permalink: /how-to/set-up-ai-proxy-with-cerebras/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-with-cerebras/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ @@ -67,6 +67,8 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 --- diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-cohere.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-cohere.md similarity index 94% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-with-cohere.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-cohere.md index 93dd96ae67d..4110c12ca64 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-cohere.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-cohere.md @@ -1,12 +1,12 @@ --- title: Set up AI Proxy with Cohere in {{site.base_gateway}} -permalink: /how-to/set-up-ai-proxy-with-cohere/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-with-cohere/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ @@ -61,6 +61,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-dashscope.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-dashscope.md similarity index 94% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-with-dashscope.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-dashscope.md index fa741c43d3e..4213b955ea2 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-dashscope.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-dashscope.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy with DashScope (Alibaba Cloud) in {{site.base_gateway}} -permalink: /how-to/set-up-ai-proxy-with-dashscope/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-with-dashscope/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ @@ -65,6 +65,9 @@ cleanup: icon_url: /assets/icons/gateway.svg automated_tests: false +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-databricks.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-databricks.md similarity index 93% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-with-databricks.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-databricks.md index bc4442ee686..9fe182f827e 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-databricks.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-databricks.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy with Databricks -permalink: /how-to/set-up-ai-proxy-with-databricks/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-with-databricks/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ @@ -59,6 +59,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-deepseek.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-deepseek.md similarity index 94% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-with-deepseek.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-deepseek.md index d6a953353c9..c8e005d4810 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-deepseek.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-deepseek.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy with DeepSeek in {{site.base_gateway}} -permalink: /how-to/set-up-ai-proxy-with-deepseek/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-with-deepseek/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ @@ -60,6 +60,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-gemini.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-gemini.md similarity index 96% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-with-gemini.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-gemini.md index 767b108e232..01f2a746a26 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-gemini.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-gemini.md @@ -1,12 +1,12 @@ --- title: Set up AI Proxy with Gemini in {{site.base_gateway}} -permalink: /how-to/set-up-ai-proxy-with-gemini/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-with-gemini/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ @@ -80,6 +80,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-huggingface.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-huggingface.md similarity index 95% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-with-huggingface.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-huggingface.md index fbbef01cbfa..1b2cff389ef 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-huggingface.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-huggingface.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy with HuggingFace in {{site.base_gateway}} -permalink: /how-to/set-up-ai-proxy-with-huggingface/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-with-huggingface/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ @@ -63,6 +63,8 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 --- diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-ollama-qwen.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-ollama-qwen.md similarity index 93% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-with-ollama-qwen.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-ollama-qwen.md index c75b53d8005..f6b1e826e1e 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-ollama-qwen.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-ollama-qwen.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy with Ollama and a Qwen model -permalink: /how-to/set-up-ai-proxy-with-ollama-qwen/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-with-ollama-qwen/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy/ @@ -59,6 +59,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-ollama.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-ollama.md similarity index 94% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-with-ollama.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-ollama.md index b26a1b700b0..62b234696b8 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-ollama.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-ollama.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy with Ollama -permalink: /how-to/set-up-ai-proxy-with-ollama/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-with-ollama/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy/ @@ -59,6 +59,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-openai.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-openai.md similarity index 94% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-with-openai.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-openai.md index 5d62972ae5d..65975c67a02 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-openai.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-openai.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy with OpenAI in {{site.base_gateway}} -permalink: /how-to/set-up-ai-proxy-with-openai/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-with-openai/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ @@ -59,6 +59,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-vertex-ai.md b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-vertex-ai.md similarity index 94% rename from app/_how-tos/ai-gateway/set-up-ai-proxy-with-vertex-ai.md rename to app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-vertex-ai.md index 0ad5fce36f5..8d560b8819b 100644 --- a/app/_how-tos/ai-gateway/set-up-ai-proxy-with-vertex-ai.md +++ b/app/_how-tos/ai-gateway/v1/set-up-ai-proxy-with-vertex-ai.md @@ -1,10 +1,10 @@ --- title: Set up AI Proxy with Vertex AI in {{site.base_gateway}} -permalink: /how-to/set-up-ai-proxy-with-vertex-ai/ +permalink: /ai-gateway/v1/how-to/set-up-ai-proxy-with-vertex-ai/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ @@ -59,6 +59,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/set-up-jaeger-with-gen-ai-otel-for-tool-calls.md b/app/_how-tos/ai-gateway/v1/set-up-jaeger-with-gen-ai-otel-for-tool-calls.md similarity index 98% rename from app/_how-tos/ai-gateway/set-up-jaeger-with-gen-ai-otel-for-tool-calls.md rename to app/_how-tos/ai-gateway/v1/set-up-jaeger-with-gen-ai-otel-for-tool-calls.md index 0de9d492cfb..1232599681c 100644 --- a/app/_how-tos/ai-gateway/set-up-jaeger-with-gen-ai-otel-for-tool-calls.md +++ b/app/_how-tos/ai-gateway/v1/set-up-jaeger-with-gen-ai-otel-for-tool-calls.md @@ -1,6 +1,6 @@ --- title: Validate Gen AI tool calls with Jaeger and OpenTelemetry -permalink: /how-to/set-up-jaeger-with-gen-ai-otel-for-tool-calls/ +permalink: /ai-gateway/v1/how-to/set-up-jaeger-with-gen-ai-otel-for-tool-calls/ content_type: how_to related_resources: - text: Set up Jaeger with Gen AI OpenTelemetry @@ -102,6 +102,9 @@ cleanup: icon_url: /assets/icons/gateway.svg automated_tests: false +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy plugin diff --git a/app/_how-tos/ai-gateway/set-up-jaeger-with-gen-ai-otel.md b/app/_how-tos/ai-gateway/v1/set-up-jaeger-with-gen-ai-otel.md similarity index 97% rename from app/_how-tos/ai-gateway/set-up-jaeger-with-gen-ai-otel.md rename to app/_how-tos/ai-gateway/v1/set-up-jaeger-with-gen-ai-otel.md index 4e9d60ecef7..afdd94045a1 100644 --- a/app/_how-tos/ai-gateway/set-up-jaeger-with-gen-ai-otel.md +++ b/app/_how-tos/ai-gateway/v1/set-up-jaeger-with-gen-ai-otel.md @@ -1,13 +1,12 @@ --- title: Set up Jaeger with Gen AI OpenTelemetry -permalink: /how-to/set-up-jaeger-with-gen-ai-otel/ +permalink: /ai-gateway/v1/how-to/set-up-jaeger-with-gen-ai-otel/ content_type: how_to related_resources: - text: Set up Dynatrace with OpenTelemetry url: /how-to/set-up-dynatrace-with-otel/ - text: Validate Gen AI tool calls with Jaeger and OpenTelemetry - url: /how-to/set-up-jaeger-with-gen-ai-otel-for-tool-calls/ - + url: /ai-gateway/v1/how-to/set-up-jaeger-with-gen-ai-otel-for-tool-calls/ description: Use the OpenTelemetry plugin to send {{site.base_gateway}} analytics and monitoring data to Jaeger dashboards. @@ -153,6 +152,9 @@ faqs: automated_tests: false +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy plugin diff --git a/app/_how-tos/ai-gateway/store-a-mistral-api-key-as-a-secret-in-konnect-config-store.md b/app/_how-tos/ai-gateway/v1/store-a-mistral-api-key-as-a-secret-in-konnect-config-store.md similarity index 98% rename from app/_how-tos/ai-gateway/store-a-mistral-api-key-as-a-secret-in-konnect-config-store.md rename to app/_how-tos/ai-gateway/v1/store-a-mistral-api-key-as-a-secret-in-konnect-config-store.md index dac214b24da..47b2ac398d9 100644 --- a/app/_how-tos/ai-gateway/store-a-mistral-api-key-as-a-secret-in-konnect-config-store.md +++ b/app/_how-tos/ai-gateway/v1/store-a-mistral-api-key-as-a-secret-in-konnect-config-store.md @@ -1,6 +1,6 @@ --- title: Store a Mistral API key as a secret in {{site.konnect_short_name}} Config Store -permalink: /how-to/store-a-mistral-api-key-as-a-secret-in-konnect-config-store/ +permalink: /ai-gateway/v1/how-to/store-a-mistral-api-key-as-a-secret-in-konnect-config-store/ description: Learn how to set up {{site.konnect_short_name}} Config Store as a Vault backend and store a Mistral API key. content_type: how_to related_resources: @@ -80,6 +80,9 @@ min_version: next_steps: - text: Review the Vaults entity url: /gateway/entities/vault/ +major_version: + ai-gateway: 1 + --- diff --git a/app/_how-tos/ai-gateway/strip-model-from-open-ai-sdk-requests.md.md b/app/_how-tos/ai-gateway/v1/strip-model-from-open-ai-sdk-requests.md.md similarity index 97% rename from app/_how-tos/ai-gateway/strip-model-from-open-ai-sdk-requests.md.md rename to app/_how-tos/ai-gateway/v1/strip-model-from-open-ai-sdk-requests.md.md index 89224b2c92e..24619fae573 100644 --- a/app/_how-tos/ai-gateway/strip-model-from-open-ai-sdk-requests.md.md +++ b/app/_how-tos/ai-gateway/v1/strip-model-from-open-ai-sdk-requests.md.md @@ -3,13 +3,13 @@ title: Strip the model field from OpenAI SDK requests content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ - text: Pre-function url: /plugins/pre-function/ -permalink: /how-to/strip-model-from-openai-sdk-requests +permalink: /ai-gateway/v1/how-to/strip-model-from-openai-sdk-requests description: Use the [Pre-function](/plugins/pre-function/) plugin to remove the model field from the request body so AI Proxy Advanced controls model selection during load balancing. @@ -64,6 +64,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- [OpenAI-compatible SDKs](https://platform.openai.com/docs/libraries) always set the `model` field in the request body. This is a required parameter and can't be omitted. diff --git a/app/_how-tos/ai-gateway/transform-a-client-request-with-ai.md b/app/_how-tos/ai-gateway/v1/transform-a-client-request-with-ai.md similarity index 96% rename from app/_how-tos/ai-gateway/transform-a-client-request-with-ai.md rename to app/_how-tos/ai-gateway/v1/transform-a-client-request-with-ai.md index 13212e2f5f3..2b492702a7e 100644 --- a/app/_how-tos/ai-gateway/transform-a-client-request-with-ai.md +++ b/app/_how-tos/ai-gateway/v1/transform-a-client-request-with-ai.md @@ -1,11 +1,10 @@ --- title: Transform a request body using OpenAI in {{site.base_gateway}} -permalink: /how-to/transform-a-client-request-with-ai/ +permalink: /ai-gateway/v1/how-to/transform-a-client-request-with-ai/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ - + url: /ai-gateway/v1/ description: Use the AI Request Transformer plugin with OpenAI to transform a client request body before proxying it. products: @@ -57,6 +56,8 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 --- diff --git a/app/_how-tos/ai-gateway/transform-a-response-with-ai.md b/app/_how-tos/ai-gateway/v1/transform-a-response-with-ai.md similarity index 90% rename from app/_how-tos/ai-gateway/transform-a-response-with-ai.md rename to app/_how-tos/ai-gateway/v1/transform-a-response-with-ai.md index 57c1c259cc9..d509bd98ab2 100644 --- a/app/_how-tos/ai-gateway/transform-a-response-with-ai.md +++ b/app/_how-tos/ai-gateway/v1/transform-a-response-with-ai.md @@ -1,11 +1,10 @@ --- title: Transform a response using OpenAI in {{site.base_gateway}} -permalink: /how-to/transform-a-response-with-ai/ +permalink: /ai-gateway/v1/how-to/transform-a-response-with-ai/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ - + url: /ai-gateway/v1/ description: Use the AI Response Transformer plugin with OpenAI to transform a response before returning it to the client. products: @@ -34,7 +33,7 @@ tags: tldr: q: How can I use AI to transform a response before returning it to the client? - a: Enable the [AI Response Transformer](/how-to/transform-a-response-with-ai/) plugin, configure the parameters under `config.llm` to access your LLM and describe the transformation to perform with the `config.prompt` parameter. + a: Enable the [AI Response Transformer](/ai-gateway/v1/how-to/transform-a-response-with-ai/) plugin, configure the parameters under `config.llm` to access your LLM and describe the transformation to perform with the `config.prompt` parameter. tools: - deck @@ -58,6 +57,8 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 --- diff --git a/app/_how-tos/ai-gateway/use-agno-with-ai-proxy.md b/app/_how-tos/ai-gateway/v1/use-agno-with-ai-proxy.md similarity index 99% rename from app/_how-tos/ai-gateway/use-agno-with-ai-proxy.md rename to app/_how-tos/ai-gateway/v1/use-agno-with-ai-proxy.md index f45ec3b25cf..0cd96be33a8 100644 --- a/app/_how-tos/ai-gateway/use-agno-with-ai-proxy.md +++ b/app/_how-tos/ai-gateway/v1/use-agno-with-ai-proxy.md @@ -1,6 +1,6 @@ --- title: Use Agno with AI Proxy in {{site.ai_gateway}} -permalink: /how-to/use-agno-with-ai-proxy/ +permalink: /ai-gateway/v1/how-to/use-agno-with-ai-proxy/ content_type: how_to description: Connect Agno’s research agents to {{site.ai_gateway}} with no code changes, enabling OpenAI-compatible inference through a proxy. @@ -64,6 +64,8 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 --- ## Configure the AI Proxy plugin diff --git a/app/_how-tos/ai-gateway/use-ai-aws-guardrails-plugin.md b/app/_how-tos/ai-gateway/v1/use-ai-aws-guardrails-plugin.md similarity index 98% rename from app/_how-tos/ai-gateway/use-ai-aws-guardrails-plugin.md rename to app/_how-tos/ai-gateway/v1/use-ai-aws-guardrails-plugin.md index 8d68a8fa165..87fbecd8d44 100644 --- a/app/_how-tos/ai-gateway/use-ai-aws-guardrails-plugin.md +++ b/app/_how-tos/ai-gateway/v1/use-ai-aws-guardrails-plugin.md @@ -1,6 +1,6 @@ --- title: Use the AI AWS Guardrails plugin -permalink: /how-to/use-ai-aws-guardrails-plugin/ +permalink: /ai-gateway/v1/how-to/use-ai-aws-guardrails-plugin/ content_type: how_to related_resources: @@ -9,8 +9,7 @@ related_resources: - text: Azure AI Content Safety url: /plugins/ai-azure-content-safety/ - text: "{{site.ai_gateway}}" - url: /ai-gateway/ - + url: /ai-gateway/v1/ description: Learn how to use the AI AWS Guardrails plugin. products: @@ -85,6 +84,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy Advanced plugin diff --git a/app/_how-tos/ai-gateway/use-ai-custom-guardrail-with-mistral-ai.md b/app/_how-tos/ai-gateway/v1/use-ai-custom-guardrail-with-mistral-ai.md similarity index 97% rename from app/_how-tos/ai-gateway/use-ai-custom-guardrail-with-mistral-ai.md rename to app/_how-tos/ai-gateway/v1/use-ai-custom-guardrail-with-mistral-ai.md index 69ae22c789e..c5166b9cf0a 100644 --- a/app/_how-tos/ai-gateway/use-ai-custom-guardrail-with-mistral-ai.md +++ b/app/_how-tos/ai-gateway/v1/use-ai-custom-guardrail-with-mistral-ai.md @@ -1,6 +1,6 @@ --- title: Use the AI Custom Guardrail plugin with the Mistral AI Moderation API -permalink: /how-to/use-ai-custom-guardrail-with-mistral/ +permalink: /ai-gateway/v1/how-to/use-ai-custom-guardrail-with-mistral/ content_type: how_to related_resources: @@ -9,8 +9,7 @@ related_resources: - text: AI Custom Guardrail url: /plugins/ai-custom-guardrail/ - text: "{{site.ai_gateway}}" - url: /ai-gateway/ - + url: /ai-gateway/v1/ description: Learn how to configure the AI Custom Guardrail plugin to use Mistral AI for content moderation products: @@ -67,6 +66,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy plugin diff --git a/app/_how-tos/ai-gateway/use-ai-gcp-model-armor-plugin.md b/app/_how-tos/ai-gateway/v1/use-ai-gcp-model-armor-plugin.md similarity index 98% rename from app/_how-tos/ai-gateway/use-ai-gcp-model-armor-plugin.md rename to app/_how-tos/ai-gateway/v1/use-ai-gcp-model-armor-plugin.md index 76e07de9df3..d73ab01f23b 100644 --- a/app/_how-tos/ai-gateway/use-ai-gcp-model-armor-plugin.md +++ b/app/_how-tos/ai-gateway/v1/use-ai-gcp-model-armor-plugin.md @@ -1,6 +1,6 @@ --- title: Use the AI GCP Model Armor plugin -permalink: /how-to/use-ai-gcp-model-armor-plugin/ +permalink: /ai-gateway/v1/how-to/use-ai-gcp-model-armor-plugin/ content_type: how_to related_resources: @@ -9,8 +9,7 @@ related_resources: - text: AI GCP Model Armor url: /plugins/ai-gcp-model-armor/ - text: "{{site.ai_gateway}}" - url: /ai-gateway/ - + url: /ai-gateway/v1/ description: Learn how to use the AI GCP Model Armor plugin. products: @@ -141,6 +140,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/use-ai-lakera-guard-plugin.md b/app/_how-tos/ai-gateway/v1/use-ai-lakera-guard-plugin.md similarity index 96% rename from app/_how-tos/ai-gateway/use-ai-lakera-guard-plugin.md rename to app/_how-tos/ai-gateway/v1/use-ai-lakera-guard-plugin.md index 99a56026ee3..b82d4a27d0d 100644 --- a/app/_how-tos/ai-gateway/use-ai-lakera-guard-plugin.md +++ b/app/_how-tos/ai-gateway/v1/use-ai-lakera-guard-plugin.md @@ -1,6 +1,6 @@ --- title: Use the AI Lakera Guard plugin -permalink: /how-to/use-ai-lakera-guard-plugin/ +permalink: /ai-gateway/v1/how-to/use-ai-lakera-guard-plugin/ content_type: how_to related_resources: @@ -9,16 +9,15 @@ related_resources: - text: AI Lakera Guard url: /plugins/ai-lakera-guard/ - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: Use the AI GCP Model Armor plugin - url: /how-to/use-ai-gcp-model-armor-plugin/ + url: /ai-gateway/v1/how-to/use-ai-gcp-model-armor-plugin/ - text: Use AI PII Sanitizer to protect sensitive data in requests - url: /how-to/protect-sensitive-information-with-ai/ + url: /ai-gateway/v1/how-to/protect-sensitive-information-with-ai/ - text: Use Azure Content Safety plugin - url: /how-to/use-azure-ai-content-safety/ + url: /ai-gateway/v1/how-to/use-azure-ai-content-safety/ - text: Use the AI AWS Guardrails plugin - url: /how-to/use-ai-aws-guardrails-plugin/ - + url: /ai-gateway/v1/how-to/use-ai-aws-guardrails-plugin/ description: Learn how to use the AI Lakera Guard plugin to protect your {{site.ai_gateway}} from prompt injection attacks, harmful content, data leakage, and malicious links using Lakera's threat detection service. products: @@ -144,6 +143,9 @@ cleanup: icon_url: /assets/icons/gateway.svg automated_tests: false +major_version: + ai-gateway: 1 + --- ## Configure the plugin @@ -203,7 +205,7 @@ Now that the AI Lakera Guard plugin is configured, let's test different categori The system blocks prompt categories that you disallow and returns a `403` error message when the plugin detects a violation, including the detector type and a request UUID for traceability. {:.info} -> For more detailed [log tracing](/ai-gateway/ai-audit-log-reference/#ai-lakera-guard-logs), configure `config.logging` in the AI Proxy plugin and use any [{{site.base_gateway}} logging plugin](/plugins/?category=logging) of your choice. +> For more detailed [log tracing](/ai-gateway/v1/ai-audit-log-reference/#ai-lakera-guard-logs), configure `config.logging` in the AI Proxy plugin and use any [{{site.base_gateway}} logging plugin](/plugins/?category=logging) of your choice. ### Prompt Defense diff --git a/app/_how-tos/ai-gateway/use-ai-prompt-decorator-plugin.md b/app/_how-tos/ai-gateway/v1/use-ai-prompt-decorator-plugin.md similarity index 91% rename from app/_how-tos/ai-gateway/use-ai-prompt-decorator-plugin.md rename to app/_how-tos/ai-gateway/v1/use-ai-prompt-decorator-plugin.md index f058365c304..d334973287d 100644 --- a/app/_how-tos/ai-gateway/use-ai-prompt-decorator-plugin.md +++ b/app/_how-tos/ai-gateway/v1/use-ai-prompt-decorator-plugin.md @@ -1,6 +1,6 @@ --- title: Enforce responsible AI behavior using the AI Prompt Decorator plugin -permalink: /how-to/use-ai-prompt-decorator-plugin/ +permalink: /ai-gateway/v1/how-to/use-ai-prompt-decorator-plugin/ content_type: how_to description: Use the AI Prompt Decorator plugin to inject ethical and safety guidelines before proxying requests to Cohere via {{site.ai_gateway}}. @@ -21,18 +21,17 @@ min_version: related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ - text: AI Prompt Decorator url: /plugins/ai-prompt-decorator/ - text: Use Azure Content Safety plugin - url: /how-to/use-azure-ai-content-safety/ + url: /ai-gateway/v1/how-to/use-azure-ai-content-safety/ - text: Use the AI AWS Guardrails plugin - url: /how-to/use-ai-aws-guardrails-plugin/ + url: /ai-gateway/v1/how-to/use-ai-aws-guardrails-plugin/ - text: Use AI Semantic Prompt Guard plugin to govern your LLM traffic - url: /how-to/use-ai-semantic-prompt-guard-plugin/ - + url: /ai-gateway/v1/how-to/use-ai-semantic-prompt-guard-plugin/ plugins: - ai-proxy - ai-prompt-decorator @@ -68,6 +67,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy plugin @@ -99,7 +101,7 @@ variables: Now we can configure the AI Prompt Decorator plugin. In this configuration, we’ll use the plugin to prepend a set of ethical, security, and compliance-focused instructions to every chat request. These instructions enforce responsible behavior from the AI, such as refusing biased prompts, protecting personal data, and avoiding unsafe outputs. {:.info} -> The [AI Prompt Decorator plugin](/plugins/ai-prompt-decorator/) is also helpful for ensuring the LLM [responds only to questions related to the injected RAG context](/how-to/compress-llm-prompts/#govern-your-llm-pipeline). When combined with the RAG Injector plugin, this keeps responses grounded in retrieved content and rejects unrelated queries. +> The [AI Prompt Decorator plugin](/plugins/ai-prompt-decorator/) is also helpful for ensuring the LLM [responds only to questions related to the injected RAG context](/ai-gateway/v1/how-to/compress-llm-prompts/#govern-your-llm-pipeline). When combined with the RAG Injector plugin, this keeps responses grounded in retrieved content and rejects unrelated queries. > > You can also use the AI Prompt Decorator plugin to [inject example dialogue](/plugins/ai-prompt-decorator/examples/create-a-complex-chat-history/) that defines task-specific behavior or tone—for example, simulating a data scientist classifying survey results. > diff --git a/app/_how-tos/ai-gateway/use-ai-prompt-guard-plugin.md b/app/_how-tos/ai-gateway/v1/use-ai-prompt-guard-plugin.md similarity index 97% rename from app/_how-tos/ai-gateway/use-ai-prompt-guard-plugin.md rename to app/_how-tos/ai-gateway/v1/use-ai-prompt-guard-plugin.md index dbb269e82f7..94b204fb53b 100644 --- a/app/_how-tos/ai-gateway/use-ai-prompt-guard-plugin.md +++ b/app/_how-tos/ai-gateway/v1/use-ai-prompt-guard-plugin.md @@ -1,10 +1,10 @@ --- title: Use AI Prompt Guard plugin to govern your LLM traffic -permalink: /how-to/use-ai-prompt-guard-plugin/ +permalink: /ai-gateway/v1/how-to/use-ai-prompt-guard-plugin/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ - text: AI Prompt Guard @@ -62,6 +62,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy plugin diff --git a/app/_how-tos/ai-gateway/use-ai-prompt-template-plugin.md b/app/_how-tos/ai-gateway/v1/use-ai-prompt-template-plugin.md similarity index 97% rename from app/_how-tos/ai-gateway/use-ai-prompt-template-plugin.md rename to app/_how-tos/ai-gateway/v1/use-ai-prompt-template-plugin.md index fc55b567548..80e7e28a567 100644 --- a/app/_how-tos/ai-gateway/use-ai-prompt-template-plugin.md +++ b/app/_how-tos/ai-gateway/v1/use-ai-prompt-template-plugin.md @@ -1,10 +1,10 @@ --- title: Provide AI prompt templates for end users with the AI Prompt Template plugin and Mistral -permalink: /how-to/use-ai-prompt-template-plugin/ +permalink: /ai-gateway/v1/how-to/use-ai-prompt-template-plugin/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ - text: AI Prompt Template @@ -62,6 +62,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy plugin @@ -95,7 +98,7 @@ variables: Now, we can configure the AI Prompt Template plugin with predefined, reusable prompt templates for common tasks. This allows users to fill in the blanks with variable placeholders (`{{variable}}`). -The plugin will automatically [block all untemplated requests](/how-to/use-ai-prompt-template-plugin/#denied-prompts) via `allow_untemplated_requests: false` setting. +The plugin will automatically [block all untemplated requests](/ai-gateway/v1/how-to/use-ai-prompt-template-plugin/#denied-prompts) via `allow_untemplated_requests: false` setting. This configuration defines five prompt templates: diff --git a/app/_how-tos/ai-gateway/use-ai-rag-injector-acls.md b/app/_how-tos/ai-gateway/v1/use-ai-rag-injector-acls.md similarity index 99% rename from app/_how-tos/ai-gateway/use-ai-rag-injector-acls.md rename to app/_how-tos/ai-gateway/v1/use-ai-rag-injector-acls.md index 02fa6a70e8d..78b9cfb084d 100644 --- a/app/_how-tos/ai-gateway/use-ai-rag-injector-acls.md +++ b/app/_how-tos/ai-gateway/v1/use-ai-rag-injector-acls.md @@ -1,10 +1,10 @@ --- title: Control access to knowledge base collections with the AI RAG Injector plugin -permalink: /how-to/use-ai-rag-injector-acls/ +permalink: /ai-gateway/v1/how-to/use-ai-rag-injector-acls/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI RAG Injector url: /plugins/ai-rag-injector/ - text: AI Proxy Advanced @@ -87,6 +87,9 @@ search_aliases: - acl automated_tests: false +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy Advanced plugin diff --git a/app/_how-tos/ai-gateway/use-ai-rag-injector-plugin.md b/app/_how-tos/ai-gateway/v1/use-ai-rag-injector-plugin.md similarity index 99% rename from app/_how-tos/ai-gateway/use-ai-rag-injector-plugin.md rename to app/_how-tos/ai-gateway/v1/use-ai-rag-injector-plugin.md index 26099395eff..f3ac751a678 100644 --- a/app/_how-tos/ai-gateway/use-ai-rag-injector-plugin.md +++ b/app/_how-tos/ai-gateway/v1/use-ai-rag-injector-plugin.md @@ -1,10 +1,10 @@ --- title: Ensure chatbots adhere to compliance policies with the AI RAG Injector plugin -permalink: /how-to/use-ai-rag-injector-plugin/ +permalink: /ai-gateway/v1/how-to/use-ai-rag-injector-plugin/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI RAG Injector url: /plugins/ai-rag-injector/ - text: AI Proxy Advanced @@ -68,6 +68,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy Advanced plugin diff --git a/app/_how-tos/ai-gateway/use-ai-semantic-prompt-guard-plugin.md b/app/_how-tos/ai-gateway/v1/use-ai-semantic-prompt-guard-plugin.md similarity index 98% rename from app/_how-tos/ai-gateway/use-ai-semantic-prompt-guard-plugin.md rename to app/_how-tos/ai-gateway/v1/use-ai-semantic-prompt-guard-plugin.md index 33361c3de9e..ca9f404a382 100644 --- a/app/_how-tos/ai-gateway/use-ai-semantic-prompt-guard-plugin.md +++ b/app/_how-tos/ai-gateway/v1/use-ai-semantic-prompt-guard-plugin.md @@ -1,10 +1,10 @@ --- title: Use AI Semantic Prompt Guard plugin to govern your LLM traffic -permalink: /how-to/use-ai-semantic-prompt-guard-plugin/ +permalink: /ai-gateway/v1/how-to/use-ai-semantic-prompt-guard-plugin/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ - text: AI Semantic Prompt Guard @@ -65,6 +65,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy plugin diff --git a/app/_how-tos/ai-gateway/use-ai-semantic-response-guard-plugin.md b/app/_how-tos/ai-gateway/v1/use-ai-semantic-response-guard-plugin.md similarity index 98% rename from app/_how-tos/ai-gateway/use-ai-semantic-response-guard-plugin.md rename to app/_how-tos/ai-gateway/v1/use-ai-semantic-response-guard-plugin.md index 215930d8d6a..141c92993ea 100644 --- a/app/_how-tos/ai-gateway/use-ai-semantic-response-guard-plugin.md +++ b/app/_how-tos/ai-gateway/v1/use-ai-semantic-response-guard-plugin.md @@ -1,10 +1,10 @@ --- title: Use AI Semantic Response Guard plugin to govern your LLM traffic -permalink: /how-to/use-ai-semantic-response-guard-plugin/ +permalink: /ai-gateway/v1/how-to/use-ai-semantic-response-guard-plugin/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ - text: AI Semantic Response Guard @@ -65,6 +65,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy plugin diff --git a/app/_how-tos/ai-gateway/use-azure-ai-content-safety.md b/app/_how-tos/ai-gateway/v1/use-azure-ai-content-safety.md similarity index 98% rename from app/_how-tos/ai-gateway/use-azure-ai-content-safety.md rename to app/_how-tos/ai-gateway/v1/use-azure-ai-content-safety.md index 43249a572d9..7d49317be58 100644 --- a/app/_how-tos/ai-gateway/use-azure-ai-content-safety.md +++ b/app/_how-tos/ai-gateway/v1/use-azure-ai-content-safety.md @@ -1,6 +1,6 @@ --- title: Use Azure Content Safety plugin -permalink: /how-to/use-azure-ai-content-safety/ +permalink: /ai-gateway/v1/how-to/use-azure-ai-content-safety/ content_type: how_to related_resources: @@ -9,8 +9,7 @@ related_resources: - text: Azure AI Content Safety url: /plugins/ai-azure-content-safety/ - text: "{{site.ai_gateway}}" - url: /ai-gateway/ - + url: /ai-gateway/v1/ description: Learn how to use the Azure AI Content Safety plugin. products: @@ -78,6 +77,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy plugin diff --git a/app/_how-tos/ai-gateway/use-bedrock-function-calling-with-streaming.md b/app/_how-tos/ai-gateway/v1/use-bedrock-function-calling-with-streaming.md similarity index 98% rename from app/_how-tos/ai-gateway/use-bedrock-function-calling-with-streaming.md rename to app/_how-tos/ai-gateway/v1/use-bedrock-function-calling-with-streaming.md index c39ed1090fd..63f49b372b2 100644 --- a/app/_how-tos/ai-gateway/use-bedrock-function-calling-with-streaming.md +++ b/app/_how-tos/ai-gateway/v1/use-bedrock-function-calling-with-streaming.md @@ -3,7 +3,7 @@ title: Stream AWS Bedrock function calling responses with AI Proxy Advanced content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ - text: AWS Bedrock ConverseStream API @@ -11,8 +11,8 @@ related_resources: - text: Use AWS Bedrock function calling with AI Proxy Advanced url: /how-to/bedrock-function-calling/ breadcrumbs: - - /ai-gateway/ -permalink: /how-to/use-bedrock-function-calling-with-streaming/ + - /ai-gateway/v1/ +permalink: /ai-gateway/v1/how-to/use-bedrock-function-calling-with-streaming/ description: "Configure the AI Proxy Advanced plugin to stream AWS Bedrock Converse API responses that include function calling." @@ -98,6 +98,9 @@ faqs: Cohere Command R and Command R+, Anthropic Claude 3 and later, and Amazon Titan models support streaming function calling through the ConverseStream API. Check the [AWS documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-supported-models-features.html) for the full compatibility matrix. automated_tests: false +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/use-bedrock-function-calling.md b/app/_how-tos/ai-gateway/v1/use-bedrock-function-calling.md similarity index 98% rename from app/_how-tos/ai-gateway/use-bedrock-function-calling.md rename to app/_how-tos/ai-gateway/v1/use-bedrock-function-calling.md index c5d1108a6f9..f727fb369f6 100644 --- a/app/_how-tos/ai-gateway/use-bedrock-function-calling.md +++ b/app/_how-tos/ai-gateway/v1/use-bedrock-function-calling.md @@ -3,14 +3,14 @@ title: Use AWS Bedrock function calling with AI Proxy Advanced content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ - text: AWS Bedrock Converse API url: https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html breadcrumbs: - - /ai-gateway/ -permalink: /how-tos/use-bedrock-function-calling/ + - /ai-gateway/v1/ +permalink: /ai-gateway/v1/how-tos/use-bedrock-function-calling/ description: "Configure the AI Proxy Advanced plugin to use AWS Bedrock's Converse API for function calling with Cohere Command R." @@ -99,6 +99,9 @@ faqs: {{site.ai_gateway}} handles authentication with AWS Bedrock on behalf of the client (`auth.allow_override: false`). The Boto3 client still requires credentials to sign HTTP requests, but {{site.ai_gateway}} replaces them before forwarding to Bedrock. The dummy credentials never reach AWS. automated_tests: false +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/use-bedrock-rerank-api.md b/app/_how-tos/ai-gateway/v1/use-bedrock-rerank-api.md similarity index 98% rename from app/_how-tos/ai-gateway/use-bedrock-rerank-api.md rename to app/_how-tos/ai-gateway/v1/use-bedrock-rerank-api.md index 13e32873348..0a7a90218a0 100644 --- a/app/_how-tos/ai-gateway/use-bedrock-rerank-api.md +++ b/app/_how-tos/ai-gateway/v1/use-bedrock-rerank-api.md @@ -1,16 +1,16 @@ --- title: Use AWS Bedrock rerank API with AI Proxy -permalink: /how-to/use-bedrock-rerank-api/ +permalink: /ai-gateway/v1/how-to/use-bedrock-rerank-api/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ - text: AWS Bedrock Rerank API url: https://docs.aws.amazon.com/bedrock/latest/APIReference/API_agent-runtime_Rerank.html breadcrumbs: - - /ai-gateway/ + - /ai-gateway/v1/ description: "Configure the AI Proxy plugin to use AWS Bedrock's Rerank API for improving document retrieval relevance in RAG pipelines." @@ -106,6 +106,9 @@ faqs: AWS Bedrock offers `cohere.rerank-v3-5:0` and `amazon.rerank-v1:0`. Cohere Rerank 3.5 is available in most regions, while Amazon Rerank 1.0 is not available in us-east-1. automated_tests: false +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-anthropic.md b/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-anthropic.md new file mode 100644 index 00000000000..6559e1ab988 --- /dev/null +++ b/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-anthropic.md @@ -0,0 +1,234 @@ +--- +title: Route Claude CLI traffic through {{site.ai_gateway}} and Anthropic +permalink: /ai-gateway/v1/how-to/use-claude-code-with-ai-gateway-anthropic/ +content_type: how_to + +related_resources: + - text: "{{site.ai_gateway}}" + url: /ai-gateway/v1/ + - text: AI Proxy Advanced + url: /plugins/ai-proxy-advanced/ + - text: File Log + url: /plugins/file-log/ + +description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic + +products: + - gateway + - ai-gateway + +works_on: + - on-prem + - konnect + +min_version: + gateway: '3.13' + +plugins: + - ai-proxy-advanced + - file-log + +entities: + - service + - route + - plugin + +tags: + - ai + - anthropic + +tldr: + q: How do I run Claude CLI through {{site.ai_gateway}}? + a: Install Claude CLI, configure its API key helper, create a Gateway Service and Route, attach the AI Proxy plugin to forward requests to Claude, enable file-log to inspect traffic, and point Claude CLI to the local proxy endpoint so all LLM requests pass through the {{site.ai_gateway}} for monitoring and control. + +tools: + - deck + +prereqs: + inline: + - title: Anthropic + icon_url: /assets/icons/anthropic.svg + include_content: prereqs/anthropic + - title: Claude Code CLI + icon_url: /assets/icons/third-party/claude.svg + include_content: prereqs/claude-code + entities: + services: + - example-service + routes: + - example-route + +cleanup: + inline: + - title: Clean up Konnect environment + include_content: cleanup/platform/konnect + icon_url: /assets/icons/gateway.svg + - title: Destroy the {{site.base_gateway}} container + include_content: cleanup/products/gateway + icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + +--- + +## Configure the AI Proxy plugin + +First, configure the AI Proxy plugin for the [{{ site.anthropic }} provider](/ai-gateway/v1/ai-providers/#anthropic). +* This setup uses the default `llm/v1/chat` route. {{ site.claude_code }} sends its requests to this route. +* The configuration also raises the maximum request body size to 512 KB to support larger prompts. + +Set `llm_format: anthropic` to tell {{site.ai_gateway}} that requests and responses use {{ site.claude }}'s native API format. This parameter controls schema validation and prevents format mismatches between {{ site.claude_code }} and the gateway. + +{% entity_examples %} +entities: + plugins: + - name: ai-proxy + config: + logging: + log_statistics: true + log_payloads: false + auth: + header_name: x-api-key + header_value: ${key} + model: + name: claude-sonnet-4-5-20250929 + provider: anthropic + options: + anthropic_version: '2023-06-01' + llm_format: anthropic + logging: + log_statistics: true + max_request_body_size: 524288 + route_type: llm/v1/chat +variables: + key: + value: $ANTHROPIC_API_KEY + description: The API key to use to connect to Anthropic. +{% endentity_examples %} + +## Configure the File Log plugin + +Now, let's enable the [File Log](/plugins/file-log/) plugin on the Service, to inspect the LLM traffic between {{ site.claude }} and the {{site.ai_gateway}}. This creates a local `claude.json` file on your machine. The file records each request and response so you can review what {{ site.claude }} sends through the {{site.ai_gateway}}. + +{% entity_examples %} +entities: + plugins: + - name: file-log + config: + path: "/tmp/claude.json" +{% endentity_examples %} + +## Verify traffic through Kong + +Now, we can start a {{ site.claude_code }} session that points it to the local {{site.ai_gateway}} endpoint: + +```sh +ANTHROPIC_BASE_URL=http://localhost:8000/anything \ +ANTHROPIC_MODEL=claude-sonnet-4-5-20250929 \ +claude +``` + +{{ site.claude_code }} asks for permission before it runs tools or interacts with files: + +```text +I'll need permission to work with your files. + +This means I can: +- Read any file in this folder +- Create, edit, or delete files +- Run commands (like npm, git, tests, ls, rm) +- Use tools defined in .mcp.json + +Learn more ( https://docs.claude.com/s/claude-code-security ) + +❯ 1. Yes, continue +2. No, exit +``` +{:.no-copy-code} + +Select **Yes, continue**. The session starts. Ask a simple question to confirm that requests reach {{site.ai_gateway}}. + +```text +Tell me about Madrid Skylitzes manuscript. +``` + +{{ site.claude_code }} might prompt you approve its web search for answering the question. When you select **Yes**, {{ site.claude }} will produce a full-length response to your request: + +```text +The Madrid Skylitzes is a remarkable 12th-century illuminated Byzantine +manuscript that represents one of the most important surviving examples +of medieval historical documentation. Here are the key details: + +What it is + +The Madrid Skylitzes is the only surviving illustrated manuscript of John +Skylitzes' "Synopsis of Histories" (Σύνοψις Ἱστοριῶν), which chronicles +Byzantine history from 811 to 1057 CE - covering the period from the death +of Emperor Nicephorus I to the deposition of Michael VI. + +Artistic Significance + +- 574 miniature paintings (with about 100 lost over time) +- Lavishly decorated with gold leaf, vibrant pigments, and intricate +detailing +- Depicts everything from imperial coronations and battles to daily life +in Byzantium +- The only surviving Byzantine illuminated chronicle written in Greek + +Unique Collaboration + +The manuscript is believed to be the work of 7 different artists from +various backgrounds: +- 4 Italian artists +- 1 English or French artist +- 2 Byzantine artists +``` +{:.no-copy-code} + +Next, inspect the {{site.ai_gateway}} logs to verify that the traffic was proxied through it: + +```sh +docker exec kong-quickstart-gateway cat /tmp/claude.json | jq +``` + +You should find an entry that shows the upstream request made by {{ site.claude_code }}. A typical log record looks like this: + +```json +{ + "...": "...", + "headers": { + ... + "user-agent": "claude-cli/2.0.37 (external, cli)", + "content-type": "application/json", + ... + }, + "method": "POST", + ... + "ai": { + "proxy": { + "usage": { + "prompt_tokens": 1, + "completion_tokens_details": {}, + "completion_tokens": 85, + "total_tokens": 86, + "cost": 0, + "time_per_token": 38.941176470588, + "time_to_first_token": 2583, + "prompt_tokens_details": {} + }, + "meta": { + "request_model": "claude-sonnet-4-20250514", + "response_model": "claude-sonnet-4-20250514", + "llm_latency": 3310, + "plugin_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", + "request_mode": "stream", + "provider_name": "anthropic" + } + } + }, + ... +} +``` +{:.no-copy-code} + +This output confirms that {{ site.claude_code }} routed the request through {{site.ai_gateway}} using the `claude-sonnet-4-5-20250929` model we selected while starting the {{ site.claude_code }} session. diff --git a/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-azure.md b/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-azure.md new file mode 100644 index 00000000000..7f7fbcf6eb0 --- /dev/null +++ b/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-azure.md @@ -0,0 +1,225 @@ +--- +title: Route Claude CLI traffic through {{site.ai_gateway}} and Azure +permalink: /ai-gateway/v1/how-to/use-claude-code-with-ai-gateway-azure/ +content_type: how_to + +related_resources: + - text: "{{site.ai_gateway}}" + url: /ai-gateway/v1/ + - text: AI Proxy + url: /plugins/ai-proxy/ + - text: File Log + url: /plugins/file-log/ + +description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic using Azure OpenAI models + +products: + - gateway + - ai-gateway + +works_on: + - on-prem + - konnect + +min_version: + gateway: '3.13' + +plugins: + - ai-proxy-advanced + - file-log + +entities: + - service + - route + - plugin + +tags: + - ai + - openai + +tldr: + q: How do I run Claude CLI through {{site.ai_gateway}} for Azure OpenAI models? + a: Install Claude CLI, configure its API key helper, create a Gateway Service and Route, attach the AI Proxy plugin to forward requests to Claude, enable the File Log plugin to inspect traffic, and point Claude CLI to the local proxy endpoint so all LLM requests pass through the {{site.ai_gateway}} for monitoring and control. + +tools: + - deck + +prereqs: + inline: + - title: Azure + include_content: prereqs/azure-ai + icon_url: /assets/icons/azure.svg + - title: Claude Code CLI + icon_url: /assets/icons/third-party/claude.svg + include_content: prereqs/claude-code + entities: + services: + - example-service + routes: + - example-route + +cleanup: + inline: + - title: Clean up Konnect environment + include_content: cleanup/platform/konnect + icon_url: /assets/icons/gateway.svg + - title: Destroy the {{site.base_gateway}} container + include_content: cleanup/products/gateway + icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + +--- +## Configure the AI Proxy plugin + +First, configure the AI Proxy plugin for the [Azure AI provider](/ai-gateway/v1/ai-providers/#azure-ai): +* This setup uses the default `llm/v1/chat` route. {{ site.claude_code }} sends its requests to this route. +* The configuration also raises the maximum request body size to 512 KB to support larger prompts. + +The `llm_format: anthropic` parameter tells {{site.ai_gateway}} to expect request and response payloads that match {{ site.claude }}'s native API format. Without this setting, the Gateway would default to OpenAI's format, which would cause request failures when {{ site.claude_code }} communicates with the Azure endpoint. + +{% entity_examples %} +entities: + plugins: + - name: ai-proxy + config: + logging: + log_statistics: true + log_payloads: true + route_type: llm/v1/chat + llm_format: anthropic + auth: + header_name: Authorization + header_value: Bearer ${azure_key} + model: + provider: azure + options: + azure_api_version: "2025-01-01-preview" + azure_instance: ${azure_instance} + azure_deployment_id: ${azure_deployment} +variables: + azure_key: + value: "$AZURE_OPENAI_API_KEY" + azure_instance: + value: "$AZURE_INSTANCE_NAME" + azure_deployment: + value: "$AZURE_DEPLOYMENT_ID" +{% endentity_examples %} + +## Configure the File Log plugin + +Now, let's enable the [File Log](/plugins/file-log/) plugin on the Service, to inspect the LLM traffic between {{ site.claude }} and the {{site.ai_gateway}}. This creates a local `claude.json` file on your machine. The file records each request and response so you can review what {{ site.claude }} sends through the {{site.ai_gateway}}. + +{% entity_examples %} +entities: + plugins: + - name: file-log + config: + path: "/tmp/claude.json" +{% endentity_examples %} + +## Verify traffic through {{site.ai_gateway}} + +Now, we can start a {{ site.claude_code }} session that points it to the local {{site.ai_gateway}} endpoint: + +{:.warning} +> Ensure that `ANTHROPIC_MODEL` matches the model you deployed in Azure. + +```sh +ANTHROPIC_BASE_URL=http://localhost:8000/anything \ +ANTHROPIC_MODEL=YOUR_AZURE_MODEL \ +claude +``` + +{{ site.claude_code }} asks for permission before it runs tools or interacts with files: + +```text +I'll need permission to work with your files. + +This means I can: +- Read any file in this folder +- Create, edit, or delete files +- Run commands (like npm, git, tests, ls, rm) +- Use tools defined in .mcp.json + +Learn more ( https://docs.claude.com/s/claude-code-security ) + +❯ 1. Yes, continue +2. No, exit +``` +{:.no-copy-code} + +Select **Yes, continue**. The session starts. Ask a simple question to confirm that requests reach {{site.ai_gateway}}. + +```text +Tell me about Vienna Oribasius manuscript. +``` + +{{ site.claude_code }} might prompt you approve its web search for answering the question. When you select **Yes**, {{ site.claude }} will produce a full-length response to your request: + +```text +The "Vienna Oribasius manuscript" refers to a famous illustrated medical +codex that preserves the works of Oribasius of Pergamon, a noted Greek +physician who lived in the 4th century CE. Oribasius was a compiler of +earlier medical knowledge, and his writings form an important link in the +transmission of Greco-Roman medical science to the Byzantine, Islamic, and +later European worlds. +``` +{:.no-copy-code} + +Next, inspect the {{site.ai_gateway}} logs to verify that the traffic was proxied through it: + +```sh +docker exec kong-quickstart-gateway cat /tmp/claude.json | jq +``` + +You should find an entry that shows the upstream request made by {{ site.claude_code }}. A typical log record looks like this: + +```json +{ + "...": "...", + "headers": { + ... + "user-agent": "claude-cli/2.0.37 (external, cli)", + "content-type": "application/json", + ... + }, + "method": "POST", + ... + "ai": { + "meta": { + "request_mode": "oneshot", + "response_model": "gpt-4.1-2025-04-14", + "request_model": "gpt-4.1", + "llm_latency": 4606, + "provider_name": "azure", + "azure_deployment_id": "gpt-4.1", + "plugin_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", + "azure_api_version": "2024-12-01-preview", + "azure_instance_id": "example-azure-openai" + }, + "usage": { + "completion_tokens": 414, + "completion_tokens_details": { + "accepted_prediction_tokens": 0, + "audio_tokens": 0, + "rejected_prediction_tokens": 0, + "reasoning_tokens": 0 + }, + "total_tokens": 11559, + "cost": 0, + "time_per_token": 11.125603864734, + "time_to_first_token": 4605, + "prompt_tokens": 11145, + "prompt_tokens_details": { + "audio_tokens": 0, + "cached_tokens": 11008, + "cached_tokens_details": {} + } + } + } + }, +``` +{:.no-copy-code} + +This output confirms that {{ site.claude_code }} routed the request through {{site.ai_gateway}} using the `gpt-4.1` Azure AI model we selected while starting the {{ site.claude_code }} session. diff --git a/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-bedrock.md b/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-bedrock.md new file mode 100644 index 00000000000..7e598eeba5f --- /dev/null +++ b/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-bedrock.md @@ -0,0 +1,319 @@ +--- +title: Route Claude CLI traffic through {{site.ai_gateway}} and AWS Bedrock +permalink: /ai-gateway/v1/how-to/use-claude-code-with-ai-gateway-bedrock/ +content_type: how_to + +related_resources: + - text: "{{site.ai_gateway}}" + url: /ai-gateway/v1/ + - text: AI Proxy + url: /plugins/ai-proxy/ + - text: File Log + url: /plugins/file-log/ + +description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic using AWS Bedrock models + +products: + - gateway + - ai-gateway + +works_on: + - on-prem + - konnect + +min_version: + gateway: '3.13' + +plugins: + - ai-proxy + - file-log + +entities: + - service + - route + - plugin + +tags: + - ai + - bedrock + +tldr: + q: How do I run Claude CLI through {{site.ai_gateway}} with AWS Bedrock? + a: Install Claude CLI, configure its API key helper, create a Gateway Service and Route, attach the AI Proxy plugin to forward requests to AWS Bedrock, enable file-log to inspect traffic, and point Claude CLI to the local proxy endpoint so all LLM requests pass through the {{site.ai_gateway}} for monitoring and control. + +tools: + - deck + +prereqs: + prereqs: + inline: + - title: AWS Bedrock + icon_url: /assets/icons/bedrock.svg + content: | + 1. Enable model access in AWS Bedrock: + - Sign in to the AWS Management Console + - Navigate to Amazon Bedrock + - Select **Model access** in the left navigation + - Request access to Claude models (for example, `us.anthropic.claude-haiku-4-5-20251001-v1:0`) + - Wait for access approval (typically immediate for most models) + + 2. Create an IAM user with Bedrock permissions: + - Navigate to IAM in the AWS Console + - Create a new user or select an existing user + - Attach the `AmazonBedrockFullAccess` policy or create a custom policy with `bedrock:InvokeModel` permissions + - Create access keys for the user + + 3. Export the Access Key ID, Secret Access Key and AWS region to your environment: + ```sh + export DECK_AWS_ACCESS_KEY_ID='YOUR AWS ACCESS KEY ID' + export DECK_AWS_SECRET_ACCESS_KEY='YOUR AWS SECRET ACCESS KEY' + export DECK_AWS_REGION='YOUR AWS REGION' + ``` + - title: Claude Code CLI + icon_url: /assets/icons/third-party/claude.svg + include_content: prereqs/claude-code + entities: + services: + - example-service + routes: + - example-route + +cleanup: + inline: + - title: Clean up Konnect environment + include_content: cleanup/platform/konnect + icon_url: /assets/icons/gateway.svg + - title: Destroy the {{site.base_gateway}} container + include_content: cleanup/products/gateway + icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + +--- + +## Configure the AI Proxy plugin + +Configure the AI Proxy plugin for the [AWS Bedrock provider](/ai-gateway/v1/ai-providers/#bedrock). + +* This setup uses the default `llm/v1/chat` route. {{ site.claude_code }} sends its requests to this route. +* The configuration also raises the maximum token count to 8192 KB to support larger prompts. + +The `llm_format: anthropic` parameter tells {{site.ai_gateway}} to expect request and response payloads that match {{ site.claude }}'s native API format. Without this setting, the gateway would default to OpenAI's format, which would cause request failures when {{ site.claude_code }} communicates with the Bedrock endpoint. + +{% entity_examples %} +entities: + plugins: + - name: ai-proxy + config: + llm_format: anthropic + route_type: llm/v1/chat + max_request_body_size: 1048576 + logging: + log_statistics: true + log_payloads: false + auth: + allow_override: false + aws_access_key_id: ${aws_access_key_id} + aws_secret_access_key: ${aws_secret_access_key} + model: + provider: bedrock + name: us.anthropic.claude-haiku-4-5-20251001-v1:0 + options: + anthropic_version: bedrock-2023-05-31 + bedrock: + aws_region: ${aws_region} + max_tokens: 8192 +variables: + aws_access_key_id: + value: $AWS_ACCESS_KEY_ID + aws_secret_access_key: + value: $AWS_SECRET_ACCESS_KEY + aws_region: + value: $AWS_REGION +{% endentity_examples %} + +## Configure the File Log plugin + +Enable the [File Log](/plugins/file-log/) plugin on the service to inspect the LLM traffic between {{ site.claude }} and the {{site.ai_gateway}}. This creates a local `claude.json` file on your machine. The file records each request and response so you can review what {{ site.claude }} sends through the {{site.ai_gateway}}. + +{% entity_examples %} +entities: + plugins: + - name: file-log + config: + path: "/tmp/claude.json" +{% endentity_examples %} + +## Verify traffic through Kong + +Start a {{ site.claude_code }} session that points to the local {{site.ai_gateway}} endpoint: + +{:.warning} +> Ensure that `ANTHROPIC_MODEL` matches the model you configured in the AI Proxy plugin (for example, `us.anthropic.claude-haiku-4-5-20251001-v1:0`). + +```sh +ANTHROPIC_BASE_URL=http://localhost:8000/anything \ +ANTHROPIC_MODEL=us.anthropic.claude-haiku-4-5-20251001-v1:0 \ +claude +``` + +{{ site.claude_code }} asks for permission before it runs tools or interacts with files: + +```text +I'll need permission to work with your files. + +This means I can: +- Read any file in this folder +- Create, edit, or delete files +- Run commands (like npm, git, tests, ls, rm) +- Use tools defined in .mcp.json + +Learn more ( https://docs.claude.com/s/claude-code-security ) + +❯ 1. Yes, continue +2. No, exit +``` +{:.no-copy-code} + +Select **Yes, continue**. The session starts. Ask a simple question to confirm that requests reach {{site.ai_gateway}}. + +```text +Tell me about Anna Komnene's Alexiad. +``` + +{{ site.claude_code }} might prompt you to approve its web search for answering the question. When you select **Yes**, {{ site.claude }} will produce a full-length response to your request: + +```text +Anna Komnene (1083-1153?) was a Byzantine princess, scholar, physician, +hospital administrator, and historian. She is known for writing the +Alexiad, a historical account of the reign of her father, Emperor Alexios +I Komnenos (r. 1081-1118). The Alexiad is a valuable primary source for +understanding Byzantine history and the First Crusade. +``` +{:.no-copy-code} + +Next, inspect the {{site.ai_gateway}} logs to verify that the traffic was proxied through it: + +```sh +docker exec kong-quickstart-gateway cat /tmp/claude.json | jq +``` + +You should find an entry that shows the upstream request made by {{ site.claude_code }}. A typical log record looks like this: + +```json +{ + ... + "method": "POST", + "headers": { + "user-agent": "claude-cli/2.0.37 (external, cli)", + "content-type": "application/json" + }, + ... + "ai": { + "proxy": { + "tried_targets": [ + { + "provider": "bedrock", + "model": "us.anthropic.claude-haiku-4-5-20251001-v1:0", + "port": 443, + "upstream_scheme": "https", + "host": "bedrock-runtime.us-west-2.amazonaws.com", + "upstream_uri": "/model/us.anthropic.claude-haiku-4-5-20251001-v1:0/invoke", + "route_type": "llm/v1/chat", + "ip": "xxx.xxx.xxx.xxx" + } + ], + "meta": { + "request_model": "us.anthropic.claude-haiku-4-5-20251001-v1:0", + "request_mode": "oneshot", + "response_model": "us.anthropic.claude-haiku-4-5-20251001-v1:0", + "provider_name": "bedrock", + "llm_latency": 1542, + "plugin_id": "13f5c57a-77b2-4c1f-9492-9048566db7cf" + }, + "usage": { + "completion_tokens": 124, + "completion_tokens_details": {}, + "total_tokens": 11308, + "cost": 0, + "time_per_token": 12.435483870968, + "time_to_first_token": 1542, + "prompt_tokens": 11184, + "prompt_tokens_details": {} + } + } + } + ... +} +``` +{:.no-copy-code} + +This output confirms that {{ site.claude_code }} routed the request through {{site.ai_gateway}} using AWS Bedrock with the `us.anthropic.claude-haiku-4-5-20251001-v1:0` model. + +## Troubleshooting + +When using {{ site.claude_code }} with AWS Bedrock models, you may encounter connection errors. +See the following sections for common error workarounds. + +### API Error 400: `context_management`: Extra inputs are not permitted + +Some beta features aren't compatible with AWS Bedrock. +This error displays because {{ site.claude }} beta features are enabled. + +To resolve this issue, do the following: + +1. Disable betas and experiments: +```sh +export CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1 +``` +2. Configure the [Request Transformer Advanced](/plugins/request-transformer-advanced/) plugin to remove beta information and the `model` field: +{% capture fix_claude_beta %} +{% entity_examples %} +entities: + plugins: + - name: ai-proxy + config: + llm_format: anthropic + route_type: llm/v1/chat + max_request_body_size: 1048576 + logging: + log_statistics: true + log_payloads: false + auth: + allow_override: false + aws_access_key_id: ${aws_access_key_id} + aws_secret_access_key: ${aws_secret_access_key} + model: + provider: bedrock + name: us.anthropic.claude-haiku-4-5-20251001-v1:0 + options: + anthropic_version: bedrock-2023-05-31 + bedrock: + aws_region: ${aws_region} + max_tokens: 8192 + - name: request-transformer-advanced + config: + remove: + headers: + - anthropic-beta + querystring: + - beta + body: + - model +variables: + aws_access_key_id: + value: $AWS_ACCESS_KEY_ID + aws_secret_access_key: + value: $AWS_SECRET_ACCESS_KEY + aws_region: + value: $AWS_REGION +{% endentity_examples %} +{% endcapture %} +{{ fix_claude_beta | indent: 3 }} + +### API Error 400: `max_tokens` must be greater than `thinking.budget_tokens` + +If your `max_tokens` limit is too small, you may encounter this error. +You can resolve this by setting `max_tokens` to a value greater than `budget_tokens`. The maximum value is `200000`. + +For more information about the default `budget_tokens` value, see [Building with extended thinking](https://platform.claude.com/docs/en/build-with-claude/extended-thinking#max-tokens-and-context-window-size) in {{ site.claude }}'s API docs. diff --git a/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-dashscope.md b/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-dashscope.md new file mode 100644 index 00000000000..4458c3b77a4 --- /dev/null +++ b/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-dashscope.md @@ -0,0 +1,238 @@ +--- +title: Route Claude CLI traffic through {{site.ai_gateway}} and DashScope +permalink: /ai-gateway/v1/how-to/use-claude-code-with-ai-gateway-dashscope/ +content_type: how_to + +related_resources: + - text: "{{site.ai_gateway}}" + url: /ai-gateway/v1/ + - text: AI Proxy + url: /plugins/ai-proxy/ + - text: File Log + url: /plugins/file-log/ + +description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic using Alibaba Cloud DashScope models + +products: + - gateway + - ai-gateway + +works_on: + - on-prem + - konnect + +min_version: + gateway: '3.13' + +plugins: + - ai-proxy + - file-log + +entities: + - service + - route + - plugin + +tags: + - ai + - dashscope + +tldr: + q: How do I run Claude CLI through {{site.ai_gateway}} with DashScope? + a: Install Claude CLI, configure its API key helper, create a Gateway Service and Route, attach the AI Proxy plugin to forward requests to DashScope, enable file-log to inspect traffic, and point Claude CLI to the local proxy endpoint so all LLM requests pass through the {{site.ai_gateway}} for monitoring and control. + +tools: + - deck + +prereqs: + prereqs: + inline: + - title: DashScope + icon_url: /assets/icons/dashscope.svg + content: | + You need an active DashScope account with API access. Sign up at the [Alibaba Cloud DashScope platform](https://dashscope.aliyuncs.com/), obtain your API key from the API-KEY interface, and export it to your environment: + ```sh + export DECK_DASHSCOPE_API_KEY='YOUR DASHSCOPE API KEY' + ``` + + - title: Claude Code CLI + icon_url: /assets/icons/third-party/claude.svg + include_content: prereqs/claude-code + entities: + services: + - example-service + routes: + - example-route + +cleanup: + inline: + - title: Clean up Konnect environment + include_content: cleanup/platform/konnect + icon_url: /assets/icons/gateway.svg + - title: Destroy the {{site.base_gateway}} container + include_content: cleanup/products/gateway + icon_url: /assets/icons/gateway.svg + +automated_tests: false +major_version: + ai-gateway: 1 + +--- + +## Configure the AI Proxy plugin + +Configure the AI Proxy plugin for the DashScope provider. +* This setup uses the default `llm/v1/chat` route. {{ site.claude_code }} sends its requests to this route. +* The configuration also raises the maximum token count size to 8192 to support larger prompts. + +The `llm_format: anthropic` parameter tells {{site.ai_gateway}} to expect request and response payloads that match {{ site.claude }}'s native API format. Without this setting, the gateway would default to OpenAI's format, which would cause request failures when {{ site.claude_code }} communicates with the DashScope endpoint. + +{% entity_examples %} +entities: + plugins: + - name: ai-proxy + config: + llm_format: anthropic + route_type: llm/v1/chat + logging: + log_statistics: true + log_payloads: false + auth: + header_name: Authorization + header_value: Bearer ${dashscope_api_key} + model: + provider: dashscope + name: qwen-plus + options: + max_tokens: 8192 + temperature: 1.0 +variables: + dashscope_api_key: + value: $DASHSCOPE_API_KEY +{% endentity_examples %} + +## Configure the File Log plugin + +Enable the [File Log](/plugins/file-log/) plugin on the service to inspect the LLM traffic between {{ site.claude }} and the {{site.ai_gateway}}. This creates a local `claude.json` file on your machine. The file records each request and response so you can review what {{ site.claude }} sends through the {{site.ai_gateway}}. + +{% entity_examples %} +entities: + plugins: + - name: file-log + config: + path: "/tmp/claude.json" +{% endentity_examples %} + +## Verify traffic through Kong + +Start a {{ site.claude_code }} session that points to the local {{site.ai_gateway}} endpoint: + +{:.warning} +> Ensure that `ANTHROPIC_MODEL` matches the model you configured in the AI Proxy plugin (for example, `qwen-plus`). + +```sh +ANTHROPIC_BASE_URL=http://localhost:8000/anything \ +ANTHROPIC_MODEL=qwen-plus \ +claude +``` + +{{ site.claude_code }} asks for permission before it runs tools or interacts with files: + +```text +I'll need permission to work with your files. + +This means I can: +- Read any file in this folder +- Create, edit, or delete files +- Run commands (like npm, git, tests, ls, rm) +- Use tools defined in .mcp.json + +Learn more ( https://docs.claude.com/s/claude-code-security ) + +❯ 1. Yes, continue +2. No, exit +``` +{:.no-copy-code} + +Select **Yes, continue**. The session starts. Ask a simple question to confirm that requests reach {{site.ai_gateway}}. + +```text +Tell me who Niketas Choniates was. +``` + +{{ site.claude_code }} might prompt you to approve its web search for answering the question. When you select **Yes**, {{ site.claude }} will produce a full-length response to your request: + +```text +Niketas Choniates was a Byzantine Greek historian and government official +who lived from around 1155 to 1217. He is best known for his historical +work "Historia" (also called "Chronike Diegesis"), which chronicles the +reigns of the Byzantine emperors from 1118 to 1207, covering the period of + the Komnenos and Angelos dynasties. + +Choniates served as a high-ranking official in the Byzantine Empire, +eventually becoming the governor of Athens. His historical writings are +particularly valuable because they provide a detailed eyewitness account +of the Fourth Crusade and the subsequent sack of Constantinople in 1204, +an event he personally experienced and fled from. His account is +considered one of the most important sources for understanding this +pivotal moment in Byzantine history. +``` +{:.no-copy-code} + +Next, inspect the {{site.ai_gateway}} logs to verify that the traffic was proxied through it: + +```sh +docker exec kong-quickstart-gateway cat /tmp/claude.json | jq +``` + +You should find an entry that shows the upstream request made by {{ site.claude_code }}. A typical log record looks like this: + +```json +{ + ... + "upstream_uri": "/compatible-mode/v1/chat/completions?beta=true", + "request": { + "method": "POST", + "headers": { + "user-agent": "claude-cli/2.0.57 (external, cli)", + "content-type": "application/json", + "anthropic-version": "2023-06-01" + } + }, + ... + "ai": { + "proxy": { + "usage": { + "completion_tokens": 493, + "completion_tokens_details": {}, + "total_tokens": 13979, + "cost": 0, + "time_per_token": 34.539553752535, + "time_to_first_token": 17027, + "prompt_tokens": 13486, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "meta": { + "response_model": "qwen-plus", + "plugin_id": "63199335-6c5a-4798-a0ad-f2cbf13cc497", + "request_model": "qwen-plus", + "request_mode": "oneshot", + "provider_name": "dashscope", + "llm_latency": 17028 + } + } + }, + "response": { + "headers": { + "x-kong-llm-model": "dashscope/qwen-plus", + "x-dashscope-call-gateway": "true" + } + } + ... +} +``` +{:.no-copy-code} + +This output confirms that {{ site.claude_code }} routed the request through {{site.ai_gateway}} using DashScope with the `qwen-plus` model. \ No newline at end of file diff --git a/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-gemini.md b/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-gemini.md new file mode 100644 index 00000000000..bdbb7b32951 --- /dev/null +++ b/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-gemini.md @@ -0,0 +1,250 @@ +--- +title: Route Claude CLI traffic through {{site.ai_gateway}} and Gemini +permalink: /ai-gateway/v1/how-to/use-claude-code-with-ai-gateway-gemini/ +content_type: how_to + +related_resources: + - text: "{{site.ai_gateway}}" + url: /ai-gateway/v1/ + - text: AI Proxy Advanced + url: /plugins/ai-proxy-advanced/ + - text: File Log + url: /plugins/file-log/ + +description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic using Gemini models + +products: + - gateway + - ai-gateway + +works_on: + - on-prem + - konnect + +min_version: + gateway: '3.13' + +plugins: + - ai-proxy-advanced + - file-log + +entities: + - service + - route + - plugin + +tags: + - ai + - gemini + +tldr: + q: How do I run Claude CLI through {{site.ai_gateway}}? + a: Install Claude CLI, configure its API key helper, create a Gateway Service and Route, attach the AI Proxy plugin to forward requests to Claude, enable the File Log plugin to inspect traffic, and point Claude CLI to the local proxy endpoint so all LLM requests pass through the {{site.ai_gateway}} for monitoring and control. + +tools: + - deck + +prereqs: + prereqs: + inline: + - title: Gemini + content: | + Before you begin, you must get the following credentials from Google Cloud: + + - **Service Account Key**: A JSON key file for a service account with Vertex AI permissions + - **Project ID**: Your Google Cloud project identifier + - **Location ID**: The region where your Vertex AI endpoint is deployed (for example, `us-central1`) + - **API Endpoint**: The Vertex AI API endpoint URL (typically `https://{location}-aiplatform.googleapis.com`) + + Export these values as environment variables: + ```sh + export GEMINI_API_KEY="" + export GCP_PROJECT_ID="" + export GEMINI_LOCATION_ID="" + export GEMINI_API_ENDPOINT="" + ``` + icon_url: /assets/icons/gcp.svg + - title: Claude Code CLI + icon_url: /assets/icons/third-party/claude.svg + include_content: prereqs/claude-code + entities: + services: + - example-service + routes: + - example-route + +cleanup: + inline: + - title: Clean up Konnect environment + include_content: cleanup/platform/konnect + icon_url: /assets/icons/gateway.svg + - title: Destroy the {{site.base_gateway}} container + include_content: cleanup/products/gateway + icon_url: /assets/icons/gateway.svg + +automated_tests: false +major_version: + ai-gateway: 1 + +--- + +## Configure the AI Proxy plugin + +First, configure the AI Proxy plugin for the [{{ site.gemini }} provider](/ai-gateway/v1/ai-providers/#gemini): +* This setup uses the default `llm/v1/chat` route. {{ site.claude_code }} sends its requests to this route. +* The configuration also raises the maximum request body size to 512 KB to support larger prompts. + +The `llm_format: anthropic` parameter tells {{site.ai_gateway}} to expect request and response payloads that match {{ site.claude }}'s native API format. Without this setting, the Gateway would default to OpenAI's format, which would cause request failures when {{ site.claude_code }} communicates with the {{ site.gemini }} endpoint. + +{% entity_examples %} +entities: + plugins: + - name: ai-proxy-advanced + config: + llm_format: anthropic + targets: + - route_type: llm/v1/chat + logging: + log_statistics: true + log_payloads: false + auth: + allow_override: false + gcp_use_service_account: true + gcp_service_account_json: ${gcp_service_account_key} + model: + provider: gemini + name: gemini-2.0-flash + options: + gemini: + api_endpoint: ${gcp_api_endpoint} + project_id: ${gcp_project_id} + location_id: ${gcp_location_id} + max_tokens: 8192 +variables: + gcp_service_account_key: + value: $GEMINI_API_KEY + gcp_api_endpoint: + value: $GEMINI_API_ENDPOINT + gcp_project_id: + value: $GCP_PROJECT_ID + gcp_location_id: + value: $GEMINI_LOCATION_ID +{% endentity_examples %} + +## Configure the File Log plugin + +Now, let's enable the [File Log](/plugins/file-log/) plugin on the Service, to inspect the LLM traffic between {{ site.claude }} and the {{site.ai_gateway}}. This creates a local `claude.json` file on your machine. The file records each request and response so you can review what {{ site.claude }} sends through the {{site.ai_gateway}}. + +{% entity_examples %} +entities: + plugins: + - name: file-log + config: + path: "/tmp/claude.json" +{% endentity_examples %} + +## Verify traffic through {{site.ai_gateway}} + +Now, we can start a {{ site.claude_code }} session that points it to the local {{site.ai_gateway}} endpoint: + +{:.warning} +> Ensure that `ANTHROPIC_MODEL` matches the model you deployed in Gemini. + +```sh +ANTHROPIC_BASE_URL=http://localhost:8000/anything \ +ANTHROPIC_MODEL=YOUR_GEMINI_MODEL \ +claude +``` + +{{ site.claude_code }} asks for permission before it runs tools or interacts with files: + +```text +I'll need permission to work with your files. + +This means I can: +- Read any file in this folder +- Create, edit, or delete files +- Run commands (like npm, git, tests, ls, rm) +- Use tools defined in .mcp.json + +Learn more ( https://docs.claude.com/s/claude-code-security ) + +❯ 1. Yes, continue +2. No, exit +``` +{:.no-copy-code} + +Select **Yes, continue**. The session starts. Ask a simple question to confirm that requests reach {{site.ai_gateway}}. + +```text +Tell me about Anna Komnene's Alexiad. +``` + +{{ site.claude_code }} might prompt you approve its web search for answering the question. When you select **Yes**, {{ site.claude }} will produce a full-length response to your request: + +```text +Anna Komnene (1083-1153?) was a Byzantine princess, scholar, physician, +hospital administrator, and historian. She is known for writing the +Alexiad, a historical account of the reign of her father, Emperor Alexios +I Komnenos (r. 1081-1118). The Alexiad is a valuable primary source for +understanding Byzantine history and the First Crusade. +``` +{:.no-copy-code} + +Next, inspect the {{site.ai_gateway}} logs to verify that the traffic was proxied through it: + +```sh +docker exec kong-quickstart-gateway cat /tmp/claude.json | jq +``` + +You should find an entry that shows the upstream request made by {{ site.claude_code }}. A typical log record looks like this: + +```json +{ + ... + "method": "POST", + "headers": { + "user-agent": "claude-cli/2.0.37 (external, cli)", + "content-type": "application/json" + }, + ... + "ai": { + "proxy": { + "tried_targets": [ + { + "provider": "gemini", + "model": "gemini-2.0-flash", + "port": 443, + "upstream_scheme": "https", + "host": "us-central1-aiplatform.googleapis.com", + "upstream_uri": "/v1/projects/example-project-id/locations/us-central1/publishers/google/models/gemini-2.0-flash:generateContent", + "route_type": "llm/v1/chat", + "ip": "xxx.xxx.xxx.xxx" + } + ], + "meta": { + "request_model": "gemini-2.0-flash", + "request_mode": "oneshot", + "response_model": "gemini-2.0-flash", + "provider_name": "gemini", + "llm_latency": 1694, + "plugin_id": "13f5c57a-77b2-4c1f-9492-9048566db7cf" + }, + "usage": { + "completion_tokens": 19, + "completion_tokens_details": {}, + "total_tokens": 11203, + "cost": 0, + "time_per_token": 89.157894736842, + "time_to_first_token": 1694, + "prompt_tokens": 11184, + "prompt_tokens_details": {} + } + } + } + ... +} +``` +{:.no-copy-code} + +This output confirms that {{ site.claude_code }} routed the request through {{site.ai_gateway}} using the `gemini-2.0-flash` model we selected while starting the {{ site.claude_code }} session. diff --git a/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-huggingface.md b/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-huggingface.md new file mode 100644 index 00000000000..20877a7df49 --- /dev/null +++ b/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-huggingface.md @@ -0,0 +1,254 @@ +--- +title: Route Claude CLI traffic through {{site.ai_gateway}} and HuggingFace +permalink: /ai-gateway/v1/how-to/use-claude-code-with-ai-gateway-huggingface/ +content_type: how_to + +related_resources: + - text: "{{site.ai_gateway}}" + url: /ai-gateway/v1/ + - text: AI Proxy + url: /plugins/ai-proxy/ + - text: Pre-function + url: /plugins/pre-function/ + - text: File Log + url: /plugins/file-log/ + +description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic using HuggingFace Inference API models + +products: + - gateway + - ai-gateway + +works_on: + - on-prem + - konnect + +min_version: + gateway: '3.13' + +plugins: + - pre-function + - ai-proxy + - file-log + +entities: + - service + - route + - plugin + +tags: + - ai + - huggingface + +tldr: + q: How do I run Claude CLI through {{site.ai_gateway}} with HuggingFace? + a: Install Claude CLI, configure a pre-function plugin to remove the model field from requests, attach the AI Proxy plugin to forward requests to HuggingFace, enable file-log to inspect traffic, and point Claude CLI to the local proxy endpoint so all LLM requests pass through the {{site.ai_gateway}} for monitoring and control. + +tools: + - deck + +prereqs: + inline: + - title: HuggingFace + icon_url: /assets/icons/huggingface.svg + content: | + You need an active HuggingFace account with API access. Sign up at [HuggingFace](https://huggingface.co/) and obtain your API token from the [Access Tokens page](https://huggingface.co/settings/tokens). Ensure you have access to the HuggingFace Inference API, and export your token to your environment: + ```sh + export DECK_HUGGINGFACE_API_TOKEN='YOUR HUGGINGFACE API TOKEN' + ``` + + - title: Claude Code CLI + icon_url: /assets/icons/third-party/claude.svg + include_content: prereqs/claude-code + entities: + services: + - example-service + routes: + - example-route + +cleanup: + inline: + - title: Clean up Konnect environment + include_content: cleanup/platform/konnect + icon_url: /assets/icons/gateway.svg + - title: Destroy the {{site.base_gateway}} container + include_content: cleanup/products/gateway + icon_url: /assets/icons/gateway.svg + +automated_tests: false +major_version: + ai-gateway: 1 + +--- + +## Configure the Pre-function plugin + +{{ site.claude }} CLI automatically includes a `model` field in its request payload. However, when the AI Proxy plugin is configured with HuggingFace provider and specific model in its settings, this creates a conflict. The pre-function plugin removes the `model` field from incoming requests before they reach the AI Proxy plugin, ensuring the gateway uses the model you configured rather than the one {{ site.claude }} CLI sends. + +{% entity_examples %} +entities: + plugins: + - name: pre-function + config: + access: + - | + local body = kong.request.get_body("application/json", nil, 10485760) + if not body or body == "" then + return + end + body.model = nil + kong.service.request.set_body(body, "application/json") +{% endentity_examples %} + +## Configure the AI Proxy plugin + +Configure the AI Proxy plugin for the [HuggingFace provider](/ai-gateway/v1/ai-providers/#huggingface). This setup uses the default `llm/v1/chat` route. {{ site.claude_code }} sends its requests to this route. + +The `llm_format: anthropic` parameter tells {{site.ai_gateway}} to expect request and response payloads that match {{ site.claude }}'s native API format. Without this setting, the gateway would default to OpenAI's format, which would cause request failures when {{ site.claude_code }} communicates with the HuggingFace endpoint. + +{% entity_examples %} +entities: + plugins: + - name: ai-proxy + config: + llm_format: anthropic + route_type: llm/v1/chat + logging: + log_statistics: true + log_payloads: false + auth: + header_name: Authorization + header_value: Bearer ${key} + model: + provider: huggingface + name: meta-llama/Llama-3.3-70B-Instruct +variables: + key: + value: $HUGGINGFACE_API_TOKEN + description: The API token to use to connect to HuggingFace Inference API. +{% endentity_examples %} + +## Configure the File Log plugin + +Enable the [File Log](/plugins/file-log/) plugin on the service to inspect the LLM traffic between {{ site.claude }} and the {{site.ai_gateway}}. This creates a local `claude.json` file on your machine. The file records each request and response so you can review what {{ site.claude }} sends through the {{site.ai_gateway}}. + +{% entity_examples %} +entities: + plugins: + - name: file-log + config: + path: "/tmp/claude.json" +{% endentity_examples %} + +## Verify traffic through Kong + +Start a {{ site.claude_code }} session that points to the local {{site.ai_gateway}} endpoint: + +{:.warning} +> The `ANTHROPIC_MODEL` value can be any string since the pre-function plugin removes it. The actual model used is `meta-llama/Llama-3.3-70B-Instruct` as configured in the AI Proxy plugin. + +```sh +ANTHROPIC_BASE_URL=http://localhost:8000/anything \ +ANTHROPIC_MODEL=any-model-name \ +claude +``` + +{{ site.claude_code }} asks for permission before it runs tools or interacts with files: + +```text +I'll need permission to work with your files. + +This means I can: +- Read any file in this folder +- Create, edit, or delete files +- Run commands (like npm, git, tests, ls, rm) +- Use tools defined in .mcp.json + +Learn more ( https://docs.claude.com/s/claude-code-security ) + +❯ 1. Yes, continue +2. No, exit +``` +{:.no-copy-code} + +Select **Yes, continue**. The session starts. Ask a simple question to confirm that requests reach {{site.ai_gateway}}. + +```text +Try creating a logging.py that logs simple http logs. +``` + +{{ site.claude_code }} might prompt you to approve its web search for answering the question. When you select **Yes**, {{ site.claude }} will produce a full-length response to your request: + +```text +Create file +╭───────────────────────────────────────────────────────────────────────────────────────────────────────╮ +│ logging.py │ +│ │ +│ import logging │ +│ │ +│ logging.basicConfig(filename='app.log', filemode='a', format='%(name)s - %(levelname)s - │ +│ %(message)s') │ +│ │ +│ def log_info(message): │ +│ logging.info(message) │ +│ │ +│ def log_warning(message): │ +│ logging.warning(message) │ +│ │ +│ def log_error(message): │ +│ logging.error(message) │ +╰───────────────────────────────────────────────────────────────────────────────────────────────────────╯ + Do you want to create logging.py? + ❯ 1. Yes +``` +{:.no-copy-code} + +Next, inspect the {{site.ai_gateway}} logs to verify that the traffic was proxied through it: + +```sh +docker exec kong-quickstart-gateway cat /tmp/claude.json | jq +``` + +You should find an entry that shows the upstream request made by {{ site.claude_code }}. A typical log record looks like this: + +```json +{ + ... + "upstream_uri": "/v1/chat/completions?beta=true", + "request": { + "method": "POST", + "headers": { + "user-agent": "claude-cli/2.0.58 (external, cli)", + "content-type": "application/json", + "anthropic-version": "2023-06-01" + } + }, + ... + "ai": { + "proxy": { + "usage": { + "completion_tokens": 26, + "completion_tokens_details": {}, + "total_tokens": 178, + "cost": 0, + "time_per_token": 52.538461538462, + "time_to_first_token": 1365, + "prompt_tokens": 152, + "prompt_tokens_details": {} + }, + "meta": { + "llm_latency": 1366, + "request_mode": "oneshot", + "plugin_id": "0000b82c-5826-4abf-93b0-2fa230f5e030", + "provider_name": "huggingface", + "response_model": "meta-llama/Llama-3.3-70B-Instruct", + "request_model": "meta-llama/Llama-3.3-70B-Instruct" + } + } + } + ... +} +``` +{:.no-copy-code} + +This output confirms that {{ site.claude_code }} routed the request through {{site.ai_gateway}} using HuggingFace with the `meta-llama/Llama-3.3-70B-Instruct` model. \ No newline at end of file diff --git a/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-openai.md b/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-openai.md new file mode 100644 index 00000000000..9831f766d14 --- /dev/null +++ b/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-openai.md @@ -0,0 +1,215 @@ +--- +title: Route Claude CLI traffic through {{site.ai_gateway}} and OpenAI +permalink: /ai-gateway/v1/how-to/use-claude-code-with-ai-gateway-openai/ +content_type: how_to + +related_resources: + - text: "{{site.ai_gateway}}" + url: /ai-gateway/v1/ + - text: AI Proxy + url: /plugins/ai-proxy/ + - text: File Log + url: /plugins/file-log/ + +description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic using OpenAI models + +products: + - gateway + - ai-gateway + +works_on: + - on-prem + - konnect + +min_version: + gateway: '3.13' + +plugins: + - ai-proxy-advanced + - file-log + +entities: + - service + - route + - plugin + +tags: + - ai + - openai + +tldr: + q: How do I run Claude CLI through {{site.ai_gateway}}? + a: Install Claude CLI, configure its API key helper, create a Gateway Service and Route, attach the AI Proxy plugin to forward requests to Claude, enable the File Log plugin to inspect traffic, and point Claude CLI to the local proxy endpoint so all LLM requests pass through the {{site.ai_gateway}} for monitoring and control. + +tools: + - deck + +prereqs: + inline: + - title: OpenAI + include_content: prereqs/openai + icon_url: /assets/icons/openai.svg + - title: Claude Code CLI + icon_url: /assets/icons/third-party/claude.svg + include_content: prereqs/claude-code + entities: + services: + - example-service + routes: + - example-route + +cleanup: + inline: + - title: Clean up Konnect environment + include_content: cleanup/platform/konnect + icon_url: /assets/icons/gateway.svg + - title: Destroy the {{site.base_gateway}} container + include_content: cleanup/products/gateway + icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + +--- + +## Configure the AI Proxy plugin + +First, configure the AI Proxy plugin for the [OpenAI provider](/ai-gateway/v1/ai-providers/#openai): + * This setup uses the default `llm/v1/chat` route. {{ site.claude_code }} sends its requests to this route. + * The configuration also raises the maximum request body size to 512 KB to support larger prompts. + +The `llm_format: anthropic` parameter tells {{site.ai_gateway}} to expect request and response payloads that match {{ site.claude }}'s native API format. Without this setting, the Gateway would default to OpenAI's format, which would cause request failures when {{ site.claude_code }} communicates with the OpenAI endpoint. + +{% entity_examples %} +entities: + plugins: + - name: ai-proxy + config: + llm_format: anthropic + route_type: llm/v1/chat + logging: + log_statistics: true + log_payloads: false + auth: + header_name: Authorization + header_value: Bearer ${openai_key} + allow_override: false + model: + provider: openai + name: gpt-5-mini + max_request_body_size: 524288 +variables: + openai_key: + value: "$OPENAI_API_KEY" +{% endentity_examples %} + +## Configure the File Log plugin + +Now, let's enable the [File Log](/plugins/file-log/) plugin on the Service, to inspect the LLM traffic between {{ site.claude }} and the {{site.ai_gateway}}. This creates a local `claude.json` file on your machine. The file records each request and response so you can review what {{ site.claude }} sends through the {{site.ai_gateway}}. + +{% entity_examples %} +entities: + plugins: + - name: file-log + config: + path: "/tmp/claude.json" +{% endentity_examples %} + +## Verify traffic through {{site.ai_gateway}} + +Now, we can start a {{ site.claude_code }} session that points it to the local {{site.ai_gateway}} endpoint: + +```sh +ANTHROPIC_BASE_URL=http://localhost:8000/anything \ +ANTHROPIC_MODEL=gpt-5-mini \ +claude +``` + +{{ site.claude_code }} asks for permission before it runs tools or interacts with files: + +```text +I'll need permission to work with your files. + +This means I can: +- Read any file in this folder +- Create, edit, or delete files +- Run commands (like npm, git, tests, ls, rm) +- Use tools defined in .mcp.json + +Learn more ( https://docs.claude.com/s/claude-code-security ) + +❯ 1. Yes, continue +2. No, exit +``` +{:.no-copy-code} + +Select **Yes, continue**. The session starts. Ask a simple question to confirm that requests reach {{site.ai_gateway}}. + + +```text +Tell me about Procopius' Secret History. +``` + +{{ site.claude_code }} might prompt you approve its web search for answering the question. When you select **Yes**, {{ site.claude }} will produce a full-length response to your request: + +```text +Procopius’ Secret History (Greek: Ἀνέκδοτα, Anekdota) is a fascinating and +notorious work of Byzantine literature written in the 6th century by the +court historian Procopius of Caesarea. Unlike his official histories +(“Wars” and “Buildings”), which paint the Byzantine Emperor Justinian I +and his wife Theodora in a generally positive and conventional manner, the +Secret History offers a scandalous, behind-the-scenes account that +sharply criticizes and even vilifies the emperor, the empress, and other +key figures of the time. +``` +{:.no-copy-code} + +Next, inspect the {{site.ai_gateway}} logs to verify that the traffic was proxied through it: + +```sh +docker exec kong-quickstart-gateway cat /tmp/claude.json | jq +``` + +You should find an entry that shows the upstream request made by {{ site.claude_code }}. A typical log record looks like this: + +```json +{ + ... + "method": "POST", + "headers": { + "user-agent": "claude-cli/2.0.37 (external, cli)", + "content-type": "application/json" + }, + "ai": { + "meta": { + "request_model": "gpt-5-mini", + "request_mode": "oneshot", + "response_model": "gpt-5-mini-2025-08-07", + "provider_name": "openai", + "llm_latency": 6786, + "plugin_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" + }, + "usage": { + "completion_tokens": 456, + "completion_tokens_details": { + "accepted_prediction_tokens": 0, + "audio_tokens": 0, + "rejected_prediction_tokens": 0, + "reasoning_tokens": 256 + }, + "total_tokens": 481, + "cost": 0, + "time_per_token": 14.881578947368, + "time_to_first_token": 6785, + "prompt_tokens": 25, + "prompt_tokens_details": { + "cached_tokens": 0, + "audio_tokens": 0 + } + } + } + ... +} +``` +{:.no-copy-code} + +This output confirms that {{ site.claude_code }} routed the request through {{site.ai_gateway}} using the `gpt-5-mini` model we selected while starting the {{ site.claude_code }} session. diff --git a/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-vertex.md b/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-vertex.md new file mode 100644 index 00000000000..8d8bf8853c1 --- /dev/null +++ b/app/_how-tos/ai-gateway/v1/use-claude-code-with-ai-gateway-vertex.md @@ -0,0 +1,249 @@ +--- +title: Route Claude CLI traffic through {{site.ai_gateway}} and Vertex AI +permalink: /ai-gateway/v1/how-to/use-claude-code-with-ai-gateway-vertex/ +content_type: how_to + +related_resources: + - text: "{{site.ai_gateway}}" + url: /ai-gateway/v1/ + - text: AI Proxy Advanced + url: /plugins/ai-proxy-advanced/ + - text: File Log + url: /plugins/file-log/ + +description: Configure {{site.ai_gateway}} to proxy Claude CLI traffic using Google Vertex AI models + +products: + - gateway + - ai-gateway + +works_on: + - on-prem + - konnect + +min_version: + gateway: '3.13' + +plugins: + - ai-proxy-advanced + - file-log + +entities: + - service + - route + - plugin + +tags: + - ai + - vertex-ai + +tldr: + q: How do I run Claude CLI through {{site.ai_gateway}}? + a: Install Claude CLI, configure its API key helper, create a Gateway Service and Route, attach the AI Proxy plugin to forward requests to Claude, enable file-log to inspect traffic, and point Claude CLI to the local proxy endpoint so all LLM requests pass through the {{site.ai_gateway}} for monitoring and control. + +tools: + - deck + +prereqs: + inline: + - title: Vertex + content: | + Before you begin, you must get the following credentials from Google Cloud: + + - **Service Account Key**: A JSON key file for a service account with Vertex AI permissions + - **Project ID**: Your Google Cloud project identifier + - **Location ID**: The region where your Vertex AI endpoint is deployed (for example, `us-central1`) + - **API Endpoint**: The Vertex AI API endpoint URL (typically `https://{location}-aiplatform.googleapis.com`) + + Export these values as environment variables: + ```sh + export GEMINI_API_KEY="" + export GCP_PROJECT_ID="" + export GEMINI_LOCATION_ID="" + export GEMINI_API_ENDPOINT="" + ``` + icon_url: /assets/icons/vertex.svg + - title: Claude Code CLI + icon_url: /assets/icons/third-party/claude.svg + include_content: prereqs/claude-code + entities: + services: + - example-service + routes: + - example-route + +cleanup: + inline: + - title: Clean up Konnect environment + include_content: cleanup/platform/konnect + icon_url: /assets/icons/gateway.svg + - title: Destroy the {{site.base_gateway}} container + include_content: cleanup/products/gateway + icon_url: /assets/icons/gateway.svg + +automated_tests: false +major_version: + ai-gateway: 1 + +--- + +## Configure the AI Proxy plugin + +First, configure the AI Proxy plugin for the {{ site.gemini }} provider. +* This setup uses the default `llm/v1/chat` route. {{ site.claude_code }} sends its requests to this route. +* The configuration also raises the maximum tokens count size to 8192 to support larger prompts. + +The `llm_format: anthropic` parameter tells {{site.ai_gateway}} to expect request and response payloads that match {{ site.claude }}'s native API format. Without this setting, the Gateway would default to OpenAI's format, which would cause request failures when {{ site.claude_code }} communicates with the {{ site.gemini }} endpoint. + +{% entity_examples %} +entities: + plugins: + - name: ai-proxy-advanced + config: + llm_format: anthropic + targets: + - route_type: llm/v1/chat + logging: + log_statistics: true + log_payloads: false + auth: + allow_override: false + gcp_use_service_account: true + gcp_service_account_json: ${gcp_service_account_key} + model: + provider: gemini + name: gemini-2.5-flash + options: + gemini: + api_endpoint: ${gcp_api_endpoint} + project_id: ${gcp_project_id} + location_id: ${gcp_location_id} + max_tokens: 8192 +variables: + gcp_service_account_key: + value: $GEMINI_API_KEY + gcp_api_endpoint: + value: $GEMINI_API_ENDPOINT + gcp_project_id: + value: $GCP_PROJECT_ID + gcp_location_id: + value: $GEMINI_LOCATION_ID +{% endentity_examples %} + +## Configure the File Log plugin + +Now, let's enable the [File Log](/plugins/file-log/) plugin on the Service, to inspect the LLM traffic between {{ site.claude }} and the {{site.ai_gateway}}. This creates a local `claude.json` file on your machine. The file records each request and response so you can review what {{ site.claude }} sends through the {{site.ai_gateway}}. + +{% entity_examples %} +entities: + plugins: + - name: file-log + config: + path: "/tmp/claude.json" +{% endentity_examples %} + +## Verify traffic through Kong + +Now, we can start a {{ site.claude_code }} session that points it to the local {{site.ai_gateway}} endpoint: + +{:.warning} +> Ensure that `ANTHROPIC_MODEL` matches the model you deployed in Gemini. + +```sh +ANTHROPIC_BASE_URL=http://localhost:8000/anything \ +ANTHROPIC_MODEL=YOUR_VERTEX_MODEL \ +claude +``` + +{{ site.claude_code }} asks for permission before it runs tools or interacts with files: + +```text +I'll need permission to work with your files. + +This means I can: +- Read any file in this folder +- Create, edit, or delete files +- Run commands (like npm, git, tests, ls, rm) +- Use tools defined in .mcp.json + +Learn more ( https://docs.claude.com/s/claude-code-security ) + +❯ 1. Yes, continue +2. No, exit +``` +{:.no-copy-code} + +Select **Yes, continue**. The session starts. Ask a simple question to confirm that requests reach {{site.ai_gateway}}. + +```text +Tell me about Anna Komnene's Alexiad. +``` + +{{ site.claude_code }} might prompt you approve its web search for answering the question. When you select **Yes**, {{ site.claude }} will produce a full-length response to your request: + +```text +Anna Komnene (1083-1153?) was a Byzantine princess, scholar, physician, +hospital administrator, and historian. She is known for writing the +Alexiad, a historical account of the reign of her father, Emperor Alexios +I Komnenos (r. 1081-1118). The Alexiad is a valuable primary source for +understanding Byzantine history and the First Crusade. +``` +{:.no-copy-code} + +Next, inspect the {{site.ai_gateway}} logs to verify that the traffic was proxied through it: + +```sh +docker exec kong-quickstart-gateway cat /tmp/claude.json | jq +``` + +You should find an entry that shows the upstream request made by {{ site.claude_code }}. A typical log record looks like this: + +```json +{ + ... + "method": "POST", + "headers": { + "user-agent": "claude-cli/2.0.37 (external, cli)", + "content-type": "application/json" + }, + ... + "ai": { + "proxy": { + "tried_targets": [ + { + "provider": "gemini", + "model": "gemini-2.0-flash", + "port": 443, + "upstream_scheme": "https", + "host": "us-central1-aiplatform.googleapis.com", + "upstream_uri": "/v1/projects/example-project-id/locations/us-central1/publishers/google/models/gemini-2.0-flash:generateContent", + "route_type": "llm/v1/chat", + "ip": "xxx.xxx.xxx.xxx" + } + ], + "meta": { + "request_model": "gemini-2.5-flash", + "request_mode": "oneshot", + "response_model": "gemini-2.5-flash", + "provider_name": "gemini", + "llm_latency": 1694, + "plugin_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" + }, + "usage": { + "completion_tokens": 19, + "completion_tokens_details": {}, + "total_tokens": 11203, + "cost": 0, + "time_per_token": 85.157894736842, + "time_to_first_token": 2546, + "prompt_tokens": 11184, + "prompt_tokens_details": {} + } + } + } + ... +} +``` +{:.no-copy-code} + +This output confirms that {{ site.claude_code }} routed the request through {{site.ai_gateway}} using the `gemini-2.5-flash` model we selected while starting the {{ site.claude_code }} session. diff --git a/app/_how-tos/ai-gateway/v1/use-codex-with-ai-gateway.md b/app/_how-tos/ai-gateway/v1/use-codex-with-ai-gateway.md new file mode 100644 index 00000000000..6a7ce7dbd52 --- /dev/null +++ b/app/_how-tos/ai-gateway/v1/use-codex-with-ai-gateway.md @@ -0,0 +1,294 @@ +--- +title: Route OpenAI Codex CLI traffic through {{site.ai_gateway}} +permalink: /ai-gateway/v1/how-to/use-codex-with-ai-gateway/ +content_type: how_to +related_resources: + - text: "{{site.ai_gateway}}" + url: /ai-gateway/v1/ + - text: AI Proxy Advanced + url: /plugins/ai-proxy-advanced/ + - text: AI Request Transformer + url: /plugins/ai-request-transformer/ + - text: File Log + url: /plugins/file-log/ + +description: Configure {{site.ai_gateway}} to proxy OpenAI Codex CLI traffic using AI Proxy Advanced. + +products: + - gateway + - ai-gateway + +works_on: + - on-prem + - konnect + +min_version: + gateway: '3.6' + +plugins: + - ai-proxy-advanced + - ai-request-transformer + - file-log + +entities: + - service + - route + - plugin + +tags: + - ai + - openai + +tldr: + q: How do I run OpenAI Codex CLI through {{site.ai_gateway}}? + a: Create a Gateway Service and Route, attach AI Proxy Advanced to forward requests to OpenAI, add a Request Transformer plugin to normalize upstream paths, enable file-log to inspect traffic, and point Codex CLI to the local proxy endpoint so all LLM requests go through the Gateway for monitoring and control. + +tools: + - deck + +prereqs: + inline: + - title: OpenAI + include_content: prereqs/openai + icon_url: /assets/icons/openai.svg + - title: Codex CLI + icon_url: /assets/icons/openai.svg + content: | + This tutorial uses the OpenAI Codex CLI. Install Node.js 18+ if needed (verify with `node --version`), then install and launch Codex: + + 1. Run the following command in your terminal to install the Codex CLI: + + ```sh + npm install -g @openai/codex + ``` + + 2. Once the installation process is complete, run the following command: + + ```sh + codex + ``` + 3. The CLI will prompt you to authenticate in your browser using your OpenAI account. + + 4. Once authenticated, close the Codex CLI session by hitting ctrl + c on macOS or ctrl + break on Windows. + entities: + services: + - codex-service + routes: + - codex-route + +cleanup: + inline: + - title: Clean up Konnect environment + include_content: cleanup/platform/konnect + icon_url: /assets/icons/gateway.svg + - title: Destroy the {{site.base_gateway}} container + include_content: cleanup/products/gateway + icon_url: /assets/icons/gateway.svg + +automated_tests: false +major_version: + ai-gateway: 1 + +--- + +## Configure the AI Proxy Advanced plugin + +First, let's configure the AI Proxy Advanced plugin. In this setup, we use the Responses route because the Codex CLI calls it by default. We don't hard-code a model in the plugin — Codex sends the model in each request. We also raise the body size limit to 128 KB to support larger prompts. + +{% entity_examples %} +entities: + plugins: + - name: ai-proxy-advanced + service: codex-service + config: + genai_category: text/generation + llm_format: openai + max_request_body_size: 131072 + model_name_header: true + response_streaming: allow + balancer: + algorithm: "round-robin" + tokens_count_strategy: "total-tokens" + latency_strategy: "tpot" + retries: 3 + targets: + - route_type: llm/v1/responses + auth: + header_name: Authorization + header_value: Bearer ${openai_api_key} + logging: + log_payloads: false + log_statistics: true + model: + provider: "openai" + +variables: + openai_api_key: + value: $OPENAI_API_KEY +{% endentity_examples %} + + +## Configure the Request Transformer plugin + +To ensure that Codex forwards clean, predictable requests to OpenAI, we configure a [Request Transformer](/plugins/request-transformer/) plugin. This plugin normalizes the upstream URI and removes any extra path segments, so only the expected route reaches the OpenAI endpoint. This small guardrail avoids malformed paths and keeps the proxy behavior consistent. + +{% entity_examples %} +entities: + plugins: + - name: request-transformer + service: codex-service + config: + replace: + uri: "/" +{% endentity_examples %} + + +Now, we can pre-validate our current configuration: + + +{% validation request-check %} +url: /codex +status_code: 200 +method: POST +headers: + - 'Content-Type: application/json' +body: + model: gpt-4o + input: + - role: "user" + content: "Ping" +{% endvalidation %} + +## Export environment variables + +Now, let's open a new terminal window and export the variables that the Codex CLI will use. We set a dummy API key here just to confirm the variable exists, and point `OPENAI_BASE_URL` to the local proxy endpoint where we will route LLM traffic from Codex CLI: + +{% on_prem %} +content: | + ```sh + export OPENAI_API_KEY=sk-xxx + export OPENAI_BASE_URL=http://localhost:8000/codex + ``` +{% endon_prem %} + +{% konnect %} +content: | + ```sh + export OPENAI_API_KEY=sk-xxx + export OPENAI_BASE_URL=$KONNECT_PROXY_URL/codex + ``` +{% endkonnect %} + +## Configure the File Log plugin + +Finally, to see the exact payloads traveling between Codex and the {{site.ai_gateway}}, let's attach a File Log plugin to the service. This gives us a local log file so we can inspect requests and responses as Codex runs through Kong. + +{% entity_examples %} +entities: + plugins: + - name: file-log + service: codex-service + config: + path: "/tmp/file.json" +{% endentity_examples %} + + +## Start and use Codex CLI + +Let's test our Codex CLI set up now: + +1. In the terminal where you exported your environment variables, run: + + ```sh + codex + ``` + + You should see: + + ```text + ╭───────────────────────────────────────────╮ + │ >_ OpenAI Codex (v0.55.0) │ + │ │ + │ model: gpt-5-codex /model to change │ + │ directory: ~ │ + ╰───────────────────────────────────────────╯ + + To get started, describe a task or try one of these commands: + + /init - create an AGENTS.md file with instructions for Codex + /status - show current session configuration + /approvals - choose what Codex can do without approval + /model - choose what model and reasoning effort to use + /review - review any changes and find issues + ``` + {:.no-copy-code} + +1. Run a simple command to call Codex using the gpt-4o model: + + ```sh + codex exec --model gpt-4o "Hello" + ``` + + Codex will prompt: + + ```text + Would you like to run the following command? + + Reason: Need temporary network access so codex exec can reach the OpenAI API + + $ codex exec --model gpt-4o "Hello" + + › 1. Yes, proceed + 2. Yes, and don't ask again for this command + 3. No, and tell Codex what to do differently + ``` + {:.no-copy-code} + + Select **Yes, proceed** and press Enter. + + Expected output: + + ```text + • Ran codex exec --model gpt-4o "Hello" + └ OpenAI Codex v0.55.0 (research preview) + -------- + … +12 lines + 6.468 + Hi there! How can I assist you today? + + ─ Worked for 9s ──────────────────────────────────────────────────────────────── + + • codex exec --model gpt-4o "Hello" returned: “Hi there! How can I assist you today?” + ``` + {:.no-copy-code} + +1. Check that LLM traffic went through {{site.ai_gateway}}: + + ```sh + docker exec kong-quickstart-gateway cat /tmp/file.json | jq + ``` + + Look for entries similar to: + + ```json + { + ... + "ai": { + "proxy": { + "tried_targets": [ + { + "ip": "0000.000.000.000", + "route_type": "llm/v1/responses", + "port": 443, + "upstream_scheme": "https", + "host": "api.openai.com", + "upstream_uri": "/v1/responses", + "provider": "openai" + } + ] + } + } + ... + } + ``` + {:.no-copy-code} \ No newline at end of file diff --git a/app/_how-tos/ai-gateway/use-cohere-rerank-api.md b/app/_how-tos/ai-gateway/v1/use-cohere-rerank-api.md similarity index 98% rename from app/_how-tos/ai-gateway/use-cohere-rerank-api.md rename to app/_how-tos/ai-gateway/v1/use-cohere-rerank-api.md index 8106e7b8aa7..8eef287838b 100644 --- a/app/_how-tos/ai-gateway/use-cohere-rerank-api.md +++ b/app/_how-tos/ai-gateway/v1/use-cohere-rerank-api.md @@ -1,15 +1,15 @@ --- title: Use Cohere rerank API for document-grounded chat with AI Proxy in {{site.base_gateway}} -permalink: /how-to/use-cohere-rerank-api/ +permalink: /ai-gateway/v1/how-to/use-cohere-rerank-api/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ description: "Use Cohere's rerank API for retrieval-augmented text generation with automatic relevance filtering and citations." breadcrumbs: - - /ai-gateway/ + - /ai-gateway/v1/ products: - gateway @@ -90,6 +90,9 @@ faqs: Cohere models including `command-a-03-2025` support document-grounded chat. Refer to the Cohere documentation for the complete list of available models. automated_tests: false +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/use-custom-function-for-ai-rate-limiting.md b/app/_how-tos/ai-gateway/v1/use-custom-function-for-ai-rate-limiting.md similarity index 97% rename from app/_how-tos/ai-gateway/use-custom-function-for-ai-rate-limiting.md rename to app/_how-tos/ai-gateway/v1/use-custom-function-for-ai-rate-limiting.md index 8fc79b47a23..596e4dc8f8b 100644 --- a/app/_how-tos/ai-gateway/use-custom-function-for-ai-rate-limiting.md +++ b/app/_how-tos/ai-gateway/v1/use-custom-function-for-ai-rate-limiting.md @@ -1,10 +1,10 @@ --- title: Enforce AI rate limits with a custom function -permalink: /how-to/use-custom-function-for-ai-rate-limiting/ +permalink: /ai-gateway/v1/how-to/use-custom-function-for-ai-rate-limiting/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ - text: AI Rate Limiting Advanced @@ -65,6 +65,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/use-gemini-3-google-search.md b/app/_how-tos/ai-gateway/v1/use-gemini-3-google-search.md similarity index 98% rename from app/_how-tos/ai-gateway/use-gemini-3-google-search.md rename to app/_how-tos/ai-gateway/v1/use-gemini-3-google-search.md index 731735d7863..3b92225d0c4 100644 --- a/app/_how-tos/ai-gateway/use-gemini-3-google-search.md +++ b/app/_how-tos/ai-gateway/v1/use-gemini-3-google-search.md @@ -1,10 +1,10 @@ --- title: Use Gemini's googleSearch tool with AI Proxy Advanced in {{site.ai_gateway}} -permalink: /how-to/use-gemini-3-google-search/ +permalink: /ai-gateway/v1/how-to/use-gemini-3-google-search/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ - text: Gemini Built-in Tools @@ -82,6 +82,9 @@ faqs: - q: Does googleSearch work with structured output? a: | Yes. You can combine `tools: [{"googleSearch": {}}]` with `response_format: {"type": "json_object"}` to get search results formatted as JSON. +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/use-gemini-3-image-config.md b/app/_how-tos/ai-gateway/v1/use-gemini-3-image-config.md similarity index 98% rename from app/_how-tos/ai-gateway/use-gemini-3-image-config.md rename to app/_how-tos/ai-gateway/v1/use-gemini-3-image-config.md index 9ed4c43b843..383b14db342 100644 --- a/app/_how-tos/ai-gateway/use-gemini-3-image-config.md +++ b/app/_how-tos/ai-gateway/v1/use-gemini-3-image-config.md @@ -1,10 +1,10 @@ --- title: Use Gemini's imageConfig with AI Proxy in {{site.ai_gateway}} -permalink: /how-to/use-gemini-3-image-config/ +permalink: /ai-gateway/v1/how-to/use-gemini-3-image-config/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ - text: Gemini Image Generation @@ -83,6 +83,9 @@ faqs: - q: What image sizes are available? a: | The `imageSize` parameter accepts values like `1k`, `2k`, and `4k`. Higher values produce higher resolution images but may increase generation time. +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/use-gemini-3-thinking-config.md b/app/_how-tos/ai-gateway/v1/use-gemini-3-thinking-config.md similarity index 98% rename from app/_how-tos/ai-gateway/use-gemini-3-thinking-config.md rename to app/_how-tos/ai-gateway/v1/use-gemini-3-thinking-config.md index 16195658fee..7ca2b302583 100644 --- a/app/_how-tos/ai-gateway/use-gemini-3-thinking-config.md +++ b/app/_how-tos/ai-gateway/v1/use-gemini-3-thinking-config.md @@ -1,10 +1,10 @@ --- title: Use Gemini's thinkingConfig with AI Proxy Advanced in {{site.ai_gateway}} -permalink: /how-to/use-gemini-3-thinking-config/ +permalink: /ai-gateway/v1/how-to/use-gemini-3-thinking-config/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ - text: Gemini Thinking Mode @@ -82,6 +82,9 @@ faqs: - q: How does thinking_budget affect performance? a: | Higher `thinking_budget` values (up to 200) increase response time but provide more detailed reasoning. Lower values produce faster responses with less detailed traces. +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/use-gemini-cli-with-ai-gateway.md b/app/_how-tos/ai-gateway/v1/use-gemini-cli-with-ai-gateway.md similarity index 95% rename from app/_how-tos/ai-gateway/use-gemini-cli-with-ai-gateway.md rename to app/_how-tos/ai-gateway/v1/use-gemini-cli-with-ai-gateway.md index 2566e750dc6..851412c713e 100644 --- a/app/_how-tos/ai-gateway/use-gemini-cli-with-ai-gateway.md +++ b/app/_how-tos/ai-gateway/v1/use-gemini-cli-with-ai-gateway.md @@ -1,10 +1,10 @@ --- title: Route Google Gemini CLI traffic through {{site.ai_gateway}} -permalink: /how-to/use-gemini-cli-with-ai-gateway/ +permalink: /ai-gateway/v1/how-to/use-gemini-cli-with-ai-gateway/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ - text: File Log @@ -82,12 +82,15 @@ cleanup: icon_url: /assets/icons/gateway.svg automated_tests: false +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy plugin First, let's configure the [AI Proxy](/plugins/ai-proxy/) plugin. The {{ site.gemini }} CLI expects to communicate with {{ site.google}}'s {{ site.gemini }} API using the chat endpoint. The plugin handles authentication using a query parameter and forwards requests to the specified model. CLI tools installed across multiple developer machines typically require distributing API keys to each installation, which exposes credentials and makes rotation difficult. -Routing CLI tools through {{site.ai_gateway}} removes this requirement. Developers authenticate against the gateway instead of directly to AI providers. You can centralize authentication, enforce [rate limits](/plugins/ai-rate-limiting-advanced/), [track usage costs](/plugins/ai-rate-limiting-advanced/#token-count-strategies), [enforce guardrails](/ai-gateway/#guardrails-and-content-safety), and [cache repeated requests](/plugins/ai-semantic-cache/). +Routing CLI tools through {{site.ai_gateway}} removes this requirement. Developers authenticate against the gateway instead of directly to AI providers. You can centralize authentication, enforce [rate limits](/plugins/ai-rate-limiting-advanced/), [track usage costs](/plugins/ai-rate-limiting-advanced/#token-count-strategies), [enforce guardrails](/ai-gateway/v1/#guardrails-and-content-safety), and [cache repeated requests](/plugins/ai-semantic-cache/). {% entity_examples %} entities: diff --git a/app/_how-tos/ai-gateway/use-gemini-sdk-chat.md b/app/_how-tos/ai-gateway/v1/use-gemini-sdk-chat.md similarity index 97% rename from app/_how-tos/ai-gateway/use-gemini-sdk-chat.md rename to app/_how-tos/ai-gateway/v1/use-gemini-sdk-chat.md index b6745b6368f..e1d6f862956 100644 --- a/app/_how-tos/ai-gateway/use-gemini-sdk-chat.md +++ b/app/_how-tos/ai-gateway/v1/use-gemini-sdk-chat.md @@ -1,10 +1,10 @@ --- title: Use Google Generative AI SDK for Gemini AI service chats with {{site.ai_gateway}} -permalink: /how-to/use-gemini-sdk-chat/ +permalink: /ai-gateway/v1/how-to/use-gemini-sdk-chat/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ - text: Google Generative AI SDK @@ -74,6 +74,9 @@ cleanup: icon_url: /assets/icons/gateway.svg automated_tests: false +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy plugin diff --git a/app/_how-tos/ai-gateway/use-langchain-with-ai-proxy.md b/app/_how-tos/ai-gateway/v1/use-langchain-with-ai-proxy.md similarity index 98% rename from app/_how-tos/ai-gateway/use-langchain-with-ai-proxy.md rename to app/_how-tos/ai-gateway/v1/use-langchain-with-ai-proxy.md index 2a17b1986fa..2ea0aaefa97 100644 --- a/app/_how-tos/ai-gateway/use-langchain-with-ai-proxy.md +++ b/app/_how-tos/ai-gateway/v1/use-langchain-with-ai-proxy.md @@ -1,6 +1,6 @@ --- title: Use LangChain with AI Proxy in {{site.ai_gateway}} -permalink: /how-to/use-langchain-with-ai-proxy/ +permalink: /ai-gateway/v1/how-to/use-langchain-with-ai-proxy/ content_type: how_to related_resources: - text: AI Proxy @@ -62,6 +62,9 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy plugin diff --git a/app/_how-tos/ai-gateway/use-qwen-code-with-ai-gateway.md b/app/_how-tos/ai-gateway/v1/use-qwen-code-with-ai-gateway.md similarity index 96% rename from app/_how-tos/ai-gateway/use-qwen-code-with-ai-gateway.md rename to app/_how-tos/ai-gateway/v1/use-qwen-code-with-ai-gateway.md index 4c4233116cc..994e939736a 100644 --- a/app/_how-tos/ai-gateway/use-qwen-code-with-ai-gateway.md +++ b/app/_how-tos/ai-gateway/v1/use-qwen-code-with-ai-gateway.md @@ -1,10 +1,10 @@ --- title: "Route Qwen Code CLI traffic through {{site.ai_gateway}}" -permalink: /how-to/use-qwen-code-with-ai-gateway/ +permalink: /ai-gateway/v1/how-to/use-qwen-code-with-ai-gateway/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy url: /plugins/ai-proxy/ - text: File Log @@ -86,12 +86,15 @@ cleanup: icon_url: /assets/icons/gateway.svg automated_tests: false +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy plugin First, configure the [AI Proxy](/plugins/ai-proxy/) plugin. The [Qwen Code CLI](https://qwenlm.github.io/qwen-code-docs/en/users/configuration/auth/) uses OpenAI-compatible endpoints for LLM communication. The plugin handles authentication using a bearer token header and forwards requests to the specified model. -CLI tools installed across multiple developer machines typically require distributing API keys to each installation, which exposes credentials and makes rotation difficult. Routing CLI tools through {{site.ai_gateway}} removes this requirement. Developers authenticate against the gateway instead of directly to AI providers. You can centralize authentication, enforce [rate limits](/plugins/ai-rate-limiting-advanced/), [track usage costs](/plugins/ai-rate-limiting-advanced/#token-count-strategies), [enforce guardrails](/ai-gateway/#guardrails-and-content-safety), and [cache repeated requests](/plugins/ai-semantic-cache/). +CLI tools installed across multiple developer machines typically require distributing API keys to each installation, which exposes credentials and makes rotation difficult. Routing CLI tools through {{site.ai_gateway}} removes this requirement. Developers authenticate against the gateway instead of directly to AI providers. You can centralize authentication, enforce [rate limits](/plugins/ai-rate-limiting-advanced/), [track usage costs](/plugins/ai-rate-limiting-advanced/#token-count-strategies), [enforce guardrails](/ai-gateway/v1/#guardrails-and-content-safety), and [cache repeated requests](/plugins/ai-semantic-cache/). {:.info} > The `max_request_body_size` parameter is set to 4194304 bytes (4MB) to accommodate large code files and extended context windows that Qwen Code CLI sends during code analysis tasks. diff --git a/app/_how-tos/ai-gateway/use-semantic-load-balancing-with-dynamic-vault-authentication.md b/app/_how-tos/ai-gateway/v1/use-semantic-load-balancing-with-dynamic-vault-authentication.md similarity index 97% rename from app/_how-tos/ai-gateway/use-semantic-load-balancing-with-dynamic-vault-authentication.md rename to app/_how-tos/ai-gateway/v1/use-semantic-load-balancing-with-dynamic-vault-authentication.md index e3d79bda51e..e592f90cad9 100644 --- a/app/_how-tos/ai-gateway/use-semantic-load-balancing-with-dynamic-vault-authentication.md +++ b/app/_how-tos/ai-gateway/v1/use-semantic-load-balancing-with-dynamic-vault-authentication.md @@ -1,10 +1,10 @@ --- title: Route OpenAI chat traffic using semantic balancing and Vault-stored keys -permalink: /how-to/use-semantic-load-balancing-with-dynamic-vault-authentication/ +permalink: /ai-gateway/v1/how-to/use-semantic-load-balancing-with-dynamic-vault-authentication/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ @@ -66,6 +66,9 @@ cleanup: icon_url: /assets/icons/gateway.svg automated_tests: false +major_version: + ai-gateway: 1 + --- ## Configure the plugin diff --git a/app/_how-tos/ai-gateway/use-semantic-load-balancing.md b/app/_how-tos/ai-gateway/v1/use-semantic-load-balancing.md similarity index 99% rename from app/_how-tos/ai-gateway/use-semantic-load-balancing.md rename to app/_how-tos/ai-gateway/v1/use-semantic-load-balancing.md index 7586c9e4892..0e3d701c78d 100644 --- a/app/_how-tos/ai-gateway/use-semantic-load-balancing.md +++ b/app/_how-tos/ai-gateway/v1/use-semantic-load-balancing.md @@ -1,10 +1,10 @@ --- title: Save LLM usage costs with AI Proxy Advanced semantic load balancing -permalink: /how-to/use-semantic-load-balancing/ +permalink: /ai-gateway/v1/how-to/use-semantic-load-balancing/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ - text: AI Prompt Guard @@ -86,6 +86,9 @@ faqs: - q: Can temperature affect which model is selected? a: | Indirectly. Temperature influences output style and can help distinguish models during embedding training or similarity scoring. Use it to align behavior with intent categories. +major_version: + ai-gateway: 1 + --- ## Configure AI Proxy Advanced Plugin diff --git a/app/_how-tos/ai-gateway/use-vertex-sdk-chat.md b/app/_how-tos/ai-gateway/v1/use-vertex-sdk-chat.md similarity index 97% rename from app/_how-tos/ai-gateway/use-vertex-sdk-chat.md rename to app/_how-tos/ai-gateway/v1/use-vertex-sdk-chat.md index 545235ed130..c8ca65bf90c 100644 --- a/app/_how-tos/ai-gateway/use-vertex-sdk-chat.md +++ b/app/_how-tos/ai-gateway/v1/use-vertex-sdk-chat.md @@ -1,10 +1,10 @@ --- title: Use Google Generative AI SDK for Vertex AI service chats with {{site.ai_gateway}} -permalink: /how-to/use-vertex-sdk-chat/ +permalink: /ai-gateway/v1/how-to/use-vertex-sdk-chat/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ - text: Vertex AI Authentication @@ -73,6 +73,9 @@ cleanup: icon_url: /assets/icons/gateway.svg automated_tests: false +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy Advanced plugin diff --git a/app/_how-tos/ai-gateway/use-vertex-sdk-for-streaming.md b/app/_how-tos/ai-gateway/v1/use-vertex-sdk-for-streaming.md similarity index 98% rename from app/_how-tos/ai-gateway/use-vertex-sdk-for-streaming.md rename to app/_how-tos/ai-gateway/v1/use-vertex-sdk-for-streaming.md index 8fe02552d3f..9f018dbb54d 100644 --- a/app/_how-tos/ai-gateway/use-vertex-sdk-for-streaming.md +++ b/app/_how-tos/ai-gateway/v1/use-vertex-sdk-for-streaming.md @@ -1,10 +1,10 @@ --- title: Stream responses from Vertex AI through {{site.ai_gateway}} using Google Generative AI SDK -permalink: /how-to/use-vertex-sdk-for-streaming/ +permalink: /ai-gateway/v1/how-to/use-vertex-sdk-for-streaming/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ - text: Vertex AI Streaming @@ -74,6 +74,9 @@ cleanup: icon_url: /assets/icons/gateway.svg automated_tests: false +major_version: + ai-gateway: 1 + --- ## Configure the AI Proxy Advanced plugin diff --git a/app/_how-tos/ai-gateway/visualize-ai-gateway-metrics-with-kibana.md b/app/_how-tos/ai-gateway/v1/visualize-ai-gateway-metrics-with-kibana.md similarity index 94% rename from app/_how-tos/ai-gateway/visualize-ai-gateway-metrics-with-kibana.md rename to app/_how-tos/ai-gateway/v1/visualize-ai-gateway-metrics-with-kibana.md index fee4d2ece55..2918680a8ff 100644 --- a/app/_how-tos/ai-gateway/visualize-ai-gateway-metrics-with-kibana.md +++ b/app/_how-tos/ai-gateway/v1/visualize-ai-gateway-metrics-with-kibana.md @@ -1,6 +1,6 @@ --- title: Visualize {{site.ai_gateway}} metrics -permalink: /how-to/visualize-ai-gateway-metrics-with-kibana/ +permalink: /ai-gateway/v1/how-to/visualize-ai-gateway-metrics-with-kibana/ content_type: how_to description: Use a sample Elasticsearch, Logstash, and Kibana stack to visualize data from the AI Proxy plugin. @@ -59,13 +59,15 @@ cleanup: related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: Get started with {{site.ai_gateway}} - url: /ai-gateway/get-started/ + url: /ai-gateway/v1/get-started/ - text: Use LangChain with AI Proxy - url: /how-to/use-langchain-with-ai-proxy/ - + url: /ai-gateway/v1/how-to/use-langchain-with-ai-proxy/ automated_tests: false +major_version: + ai-gateway: 1 + --- ## Clone the sample repository diff --git a/app/_how-tos/ai-gateway/visualize-llm-metrics-with-grafana.md b/app/_how-tos/ai-gateway/v1/visualize-llm-metrics-with-grafana.md similarity index 95% rename from app/_how-tos/ai-gateway/visualize-llm-metrics-with-grafana.md rename to app/_how-tos/ai-gateway/v1/visualize-llm-metrics-with-grafana.md index 23a6df6c550..6a64838d535 100644 --- a/app/_how-tos/ai-gateway/visualize-llm-metrics-with-grafana.md +++ b/app/_how-tos/ai-gateway/v1/visualize-llm-metrics-with-grafana.md @@ -1,17 +1,16 @@ --- title: "Visualize LLM traffic with Prometheus and Grafana" -permalink: /how-to/visualize-llm-metrics-with-grafana/ +permalink: /ai-gateway/v1/how-to/visualize-llm-metrics-with-grafana/ content_type: how_to related_resources: - text: "{{site.ai_gateway}}" - url: /ai-gateway/ + url: /ai-gateway/v1/ - text: AI Proxy Advanced url: /plugins/ai-proxy-advanced/ - text: Prometheus plugin url: /plugins/prometheus/ - text: Monitor AI metrics - url: /ai-gateway/monitor-ai-llm-metrics/ - + url: /ai-gateway/v1/monitor-ai-llm-metrics/ description: Learn how to monitor LLM traffic and visualize AI metrics in Grafana using the AI Proxy Advanced and Prometheus plugins in {{ site.base_gateway }}. products: @@ -90,6 +89,8 @@ cleanup: - title: Destroy the {{site.base_gateway}} container include_content: cleanup/products/gateway icon_url: /assets/icons/gateway.svg +major_version: + ai-gateway: 1 --- ## Configure the AI Proxy Advanced plugin @@ -212,7 +213,7 @@ Prometheus will begin to scrape metrics data from {{site.ai_gateway}}. ### Add Prometheus data source -1. In the Grafana UI, go to **Connections** > **Data Sources**. If you're using the Grafana setup from the [prerequisites](/how-to/visualize-llm-metrics-with-grafana/#grafana), you can access the UI at [http://localhost:3000/](http://localhost:3000/). +1. In the Grafana UI, go to **Connections** > **Data Sources**. If you're using the Grafana setup from the [prerequisites](/ai-gateway/v1/how-to/visualize-llm-metrics-with-grafana/#grafana), you can access the UI at [http://localhost:3000/](http://localhost:3000/). 2. Click **Add data source**. 3. Select **Prometheus** from the list. 4. In the **Prometheus server URL** field, enter: `http://host.docker.internal:9090`. diff --git a/app/_how-tos/gateway/install-gateway-read-only.md b/app/_how-tos/gateway/install-gateway-read-only.md index 26bade98a6b..58ea20da358 100644 --- a/app/_how-tos/gateway/install-gateway-read-only.md +++ b/app/_how-tos/gateway/install-gateway-read-only.md @@ -132,6 +132,8 @@ EOF This Docker Compose file will create a read-only {{site.base_gateway}} instance without a datastore. +{% include gateway/hardened-container-note.md %} + ## Start {{site.base_gateway}} Start {{site.base_gateway}} with the Docker Compose file: diff --git a/app/_how-tos/insomnia/link-konnect-to-insomnia.md b/app/_how-tos/insomnia/link-konnect-to-insomnia.md index 6bbaef50ad6..214c0f1dec3 100644 --- a/app/_how-tos/insomnia/link-konnect-to-insomnia.md +++ b/app/_how-tos/insomnia/link-konnect-to-insomnia.md @@ -17,7 +17,7 @@ tiers: insomnia: enterprise min_version: - insomnia: '13' + insomnia: '13.0' description: Link {{ site.data.products.insomnia.name }} to {{ site.konnect_short_name }} and send requests against a Route in your {{site.base_gateway}} Service. tags: diff --git a/app/_how-tos/insomnia/use-git-cli.md b/app/_how-tos/insomnia/use-git-cli.md index 31514769235..76a8be45bbd 100644 --- a/app/_how-tos/insomnia/use-git-cli.md +++ b/app/_how-tos/insomnia/use-git-cli.md @@ -10,7 +10,7 @@ products: beta: true min_version: - insomnia: "beta-12.6" + insomnia: "12.6" tags: - insomnia-documents diff --git a/app/_includes/banner.html b/app/_includes/banner.html index 15a3a2243f7..923e1fbb2ad 100644 --- a/app/_includes/banner.html +++ b/app/_includes/banner.html @@ -1,42 +1,8 @@ {% if site.render_banner %} -