Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions docs/cli_reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,25 @@ Validate a deployment, then start the proxy:
--host 127.0.0.1 --port 4000
```

### Error Responses

OpenAI Chat Completions, OpenAI Responses, and unknown URL paths return this
error body:

```json
{"error": {"message": "...", "type": "...", "code": "..."}}
```

Anthropic Messages returns the Anthropic error body:

```json
{"type": "error", "error": {"type": "...", "message": "..."}}
```

Common OpenAI-compatible `code` values include `invalid_body`, `empty_messages`,
`model_not_found`, `endpoint_not_found`, `upstream_error`,
`internal_chain_error`, and `context_length_exceeded`.
Comment on lines +90 to +105

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Inspect standalone-server error handling and error-code definitions.
fd -t f . crates/switchyard-server
rg -n -C 3 \
  'endpoint_not_found|invalid_body|empty_messages|context_length_exceeded|Not Found' \
  crates/switchyard-server || true

Repository: NVIDIA-NeMo/Switchyard

Length of output: 4971


🏁 Script executed:

#!/bin/bash
# Inspect the Rust error-envelope helpers, route handlers, and focused tests.
ast-grep outline crates/switchyard-server/src/lib.rs --match 'fn error_response' --view expanded
sed -n '780,970p' crates/switchyard-server/src/lib.rs
sed -n '1020,1060p' crates/switchyard-server/src/lib.rs
rg -n -C 4 'empty_messages|model_not_found|upstream_error|internal_chain_error|context_length_exceeded|error_response|render_error_response' crates/switchyard-server/src/lib.rs crates/switchyard-server/tests/server.rs
sed -n '70,115p' docs/cli_reference.md

Repository: NVIDIA-NeMo/Switchyard

Length of output: 26197


🏁 Script executed:

#!/bin/bash
# Determine whether every documented code exists in the Rust server and inspect
# the request-validation paths that could emit empty_messages or internal_chain_error.
rg -n -C 3 \
  'empty_messages|internal_chain_error|invalid_body|model_not_found|endpoint_not_found|upstream_error|context_length_exceeded' \
  . ':!target' || true

printf '\n-- focused Rust request and response tests --\n'
sed -n '1588,1695p' crates/switchyard-server/tests/server.rs
sed -n '560,645p' crates/switchyard-server/src/lib.rs
sed -n '470,520p' crates/switchyard-server/src/lib.rs

printf '\n-- exact error_response code arguments in Rust --\n'
python3 - <<'PY'
from pathlib import Path
import re

path = Path("crates/switchyard-server/src/lib.rs")
text = path.read_text()
for match in re.finditer(r'error_response\((.*?)\)', text, re.S):
    args = match.group(1)
    strings = re.findall(r'"([^"]+)"', args)
    if strings:
        print(strings[-1])
PY

Repository: NVIDIA-NeMo/Switchyard

Length of output: 50380


🏁 Script executed:

#!/bin/bash
# Extract only the Rust server's documented-code occurrences and the request
# validation/dispatch paths. Avoid the broad repository output.
printf '%s\n' '-- Rust documented codes --'
for code in invalid_body empty_messages model_not_found endpoint_not_found upstream_error internal_chain_error context_length_exceeded; do
  printf '%-25s ' "$code"
  rg -n --count-matches "\"$code\"" crates/switchyard-server/src crates/switchyard-server/tests || true
done

printf '\n%s\n' '-- Rust request validation and dispatch --'
sed -n '560,640p' crates/switchyard-server/src/lib.rs
sed -n '1595,1660p' crates/switchyard-server/tests/server.rs

printf '\n%s\n' '-- Rust error-envelope tests --'
rg -n -C 8 'wire_format|AnthropicMessages|OpenAiResponses|invalid_body|endpoint_not_found' \
  crates/switchyard-server/tests/server.rs crates/switchyard-server/src/lib.rs

Repository: NVIDIA-NeMo/Switchyard

Length of output: 28725


Scope the error-code list to the correct server.

The Rust server uses both documented error envelopes, but it does not define empty_messages or internal_chain_error. Remove these codes from the standalone Rust section, or label the list as project-wide and document the Rust-specific codes separately.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/cli_reference.md` around lines 90 - 105, Update the error-code
documentation in the OpenAI-compatible section of the CLI reference to avoid
attributing empty_messages and internal_chain_error to the Rust server. Remove
those codes from the standalone Rust list, or clearly label the list as
project-wide and provide a separate list of Rust-specific codes.


## Removed Setup Commands

`switchyard configure` and `switchyard verify` are not available. Export the
Expand Down
48 changes: 38 additions & 10 deletions switchyard/server/switchyard_app.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,31 +9,33 @@
translation internally.
"""

from __future__ import annotations

import inspect
from collections.abc import AsyncIterator, Callable, Iterable
from contextlib import asynccontextmanager
from typing import TYPE_CHECKING, cast
from typing import cast

from fastapi import FastAPI, Request
from fastapi.exception_handlers import request_validation_exception_handler
from fastapi.exception_handlers import http_exception_handler, request_validation_exception_handler
from fastapi.exceptions import RequestValidationError
from fastapi.responses import Response
from fastapi.responses import JSONResponse, Response
from starlette.exceptions import HTTPException as StarletteHTTPException

from switchyard.lib.endpoints import outcome_metrics
from switchyard.lib.endpoints.anthropic_messages_endpoint import (
AnthropicMessagesEndpoint,
)
from switchyard.lib.endpoints.base import Endpoint
from switchyard.lib.endpoints.dispatch import invalid_request_response
from switchyard.lib.endpoints.error_envelope import ERROR_SOURCE_HEADER, error_response
from switchyard.lib.endpoints.models_endpoint import ModelsEndpoint
from switchyard.lib.endpoints.openai_chat_endpoint import (
OpenAIChatEndpoint,
)
from switchyard.lib.endpoints.responses_endpoint import (
ResponsesEndpoint,
)
from switchyard.lib.proxy_context import ERROR_SOURCE_SWITCHYARD
from switchyard.lib.route_table import SwitchyardApp
from switchyard_rust.core import SwitchyardInvalidRequestError

#: Inbound LLM-serving paths whose response status codes feed the
Expand All @@ -46,8 +48,21 @@
"/v1/responses",
})

if TYPE_CHECKING:
from switchyard.lib.route_table import SwitchyardApp

def _request_error_response(request: Request, message: str, code: str) -> Response:
if request.url.path == "/v1/messages":
return JSONResponse(
status_code=400,
content={
"type": "error",
"error": {
"type": "invalid_request_error",
"message": message,
},
},
headers={ERROR_SOURCE_HEADER: ERROR_SOURCE_SWITCHYARD},
)
return invalid_request_response(message, code=code)


async def _run_lifecycle_method(component: object, method_name: str) -> None:
Expand Down Expand Up @@ -113,6 +128,19 @@ async def _lifespan(_app: FastAPI) -> AsyncIterator[None]:

app = FastAPI(title="Switchyard", lifespan=_lifespan)

@app.exception_handler(StarletteHTTPException)
async def _http_error_handler(
request: Request, exc: StarletteHTTPException
) -> Response:
if exc.status_code == 404 and "endpoint" not in request.scope:
return error_response(
404,
"Not Found",
error_type="not_found",
code="endpoint_not_found",
)
return await http_exception_handler(request, exc)

@app.exception_handler(RequestValidationError)
async def _request_validation_error_handler(
request: Request, exc: RequestValidationError
Expand All @@ -133,12 +161,12 @@ async def _request_validation_error_handler(
if is_json_parse
else "Request body must be a JSON object"
)
return invalid_request_response(message, code="invalid_body")
return _request_error_response(request, message, "invalid_body")
return await request_validation_exception_handler(request, exc)

@app.exception_handler(SwitchyardInvalidRequestError)
async def _invalid_request_handler(
_request: Request, exc: SwitchyardInvalidRequestError
request: Request, exc: SwitchyardInvalidRequestError
) -> Response:
"""Map request validation failures to the 400 envelope.

Expand All @@ -148,7 +176,7 @@ async def _invalid_request_handler(
``messages`` array, so the envelope uses ``code="empty_messages"``;
revisit if more validations start sharing this error.
"""
return invalid_request_response(str(exc), code="empty_messages")
return _request_error_response(request, str(exc), "empty_messages")

app.state.switchyard = switchyard

Expand Down
69 changes: 50 additions & 19 deletions tests/test_build_and_serve.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,6 @@
fails one of these tests.
"""

from __future__ import annotations

import argparse
from collections.abc import AsyncIterator
from typing import Any
Expand Down Expand Up @@ -100,27 +98,55 @@ async def _sentinel() -> dict[str, str]:
# ---------------------------------------------------------------------------


def _ns(**overrides: Any) -> argparse.Namespace:
def _ns(
host: str = "127.0.0.1",
port: int | None = 4000,
reload: bool = False,
workers: int = 1,
) -> argparse.Namespace:
"""Build the argparse namespace ``build_and_serve`` expects."""
defaults = {"host": "127.0.0.1", "port": 4000, "reload": False, "workers": 1}
defaults.update(overrides)
return argparse.Namespace(**defaults)
return argparse.Namespace(host=host, port=port, reload=reload, workers=workers)


def _capture_uvicorn(monkeypatch: pytest.MonkeyPatch) -> dict[str, Any]:
"""Patch ``uvicorn.run`` to capture its kwargs without starting a server."""
captured: dict[str, Any] = {}

def _fake_run(app: FastAPI, **kwargs: Any) -> None:
def _fake_run(
app: FastAPI,
host: str,
port: int,
reload: bool,
workers: int,
) -> None:
captured["app"] = app
captured["kwargs"] = kwargs
captured["kwargs"] = {
"host": host,
"port": port,
"reload": reload,
"workers": workers,
}

import uvicorn

monkeypatch.setattr(uvicorn, "run", _fake_run)
return captured


def _assert_invalid_request_response(
response: httpx.Response,
path: str,
code: str,
) -> None:
body = response.json()
assert body["error"]["type"] == "invalid_request_error"
if path == "/v1/messages":
assert body["type"] == "error"
assert set(body["error"]) == {"type", "message"}
else:
assert body["error"]["code"] == code


def _switchyard() -> Switchyard:
return Switchyard(backend=_StubBackend(), translator=TranslationEngine())

Expand Down Expand Up @@ -266,9 +292,7 @@ async def test_malformed_json_returns_400(
)
assert resp.status_code == 400
assert resp.headers["content-type"].startswith("application/json")
body = resp.json()
assert body["error"]["type"] == "invalid_request_error"
assert body["error"]["code"] == "invalid_body"
_assert_invalid_request_response(resp, path, "invalid_body")

@pytest.mark.parametrize("path", _ENDPOINTS)
async def test_json_array_body_returns_400(
Expand All @@ -281,9 +305,7 @@ async def test_json_array_body_returns_400(
)
assert resp.status_code == 400
assert resp.headers["content-type"].startswith("application/json")
body = resp.json()
assert body["error"]["type"] == "invalid_request_error"
assert body["error"]["code"] == "invalid_body"
_assert_invalid_request_response(resp, path, "invalid_body")

async def test_server_stays_healthy_after_bad_request(
self, served_client: httpx.AsyncClient
Expand All @@ -304,9 +326,20 @@ async def counting_client(
"""Like served_client but exposes a call-counting backend for short-circuit checks."""
captured: dict[str, Any] = {}

def _fake_run(app: FastAPI, **kwargs: Any) -> None:
def _fake_run(
app: FastAPI,
host: str,
port: int,
reload: bool,
workers: int,
) -> None:
captured["app"] = app
captured["kwargs"] = kwargs
captured["kwargs"] = {
"host": host,
"port": port,
"reload": reload,
"workers": workers,
}

import uvicorn

Expand Down Expand Up @@ -357,9 +390,7 @@ async def test_anthropic_messages_empty_messages_returns_400(
)
assert resp.status_code == 400
assert resp.headers["content-type"].startswith("application/json")
body = resp.json()
assert body["error"]["type"] == "invalid_request_error"
assert body["error"]["code"] == "empty_messages"
_assert_invalid_request_response(resp, "/v1/messages", "empty_messages")
assert backend.call_count == 0, "backend must not be invoked for empty messages"

async def test_non_empty_messages_still_succeed(
Expand Down
19 changes: 16 additions & 3 deletions tests/test_switchyard_app_factory.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,6 @@

"""Tests for the FastAPI app factory wiring."""

from __future__ import annotations

from typing import Protocol

from fastapi import FastAPI
Expand All @@ -25,7 +23,6 @@ def __init__(self) -> None:
async def call(
self,
request: _RequestWithBody,
*,
ctx: object | None = None,
) -> dict[str, object]:
self.requests.append(request)
Expand Down Expand Up @@ -86,3 +83,19 @@ def test_app_registers_component_contributed_endpoints() -> None:

assert response.status_code == 200
assert response.json() == {"status": "ok"}


def test_unknown_path_uses_switchyard_error_envelope() -> None:
app = build_switchyard_app(_RecordingSwitchyard()) # type: ignore[arg-type]

with TestClient(app, raise_server_exceptions=False) as client:
response = client.get("/this/does/not/exist")

assert response.status_code == 404
assert response.json() == {
"error": {
"message": "Not Found",
"type": "not_found",
"code": "endpoint_not_found",
}
}
Loading