[feature] Ship a container image, Helm chart and Kubernetes deployment docs

[ansjindal]

Problem

Switchyard has no production container image and no Kubernetes packaging, so
every team deploying switchyard-server builds both from scratch. The two
Dockerfiles under benchmark/ build the Python launcher for benchmark
harnesses and are not deployment artifacts — they carry the full Rust
toolchain, run as root, and have no release build of the proxy binary.

This hits anyone running Switchyard as a service rather than as a local
launcher, which is the Server Path the README documents. It is a
one-time-per-team cost, but it is duplicated work, and the details that are
easy to get wrong are the ones that matter in production:

• terminationGracePeriodSeconds must exceed --shutdown-timeout, or
  Kubernetes kills streaming completions the server was deliberately draining.
• api_key_env names environment variables, so credentials belong in a Secret
  loaded with envFrom, not in the TOML.
• The server exits non-zero when a named api_key_env is absent, which is
  good, but surfaces as a crash-looping pod unless the chart wires the Secret
  correctly.
• .cargo/config.toml builds x86_64 with -C target-cpu=x86-64-v3, so the
  image needs an AVX2-class node and that constraint has to be documented
  somewhere.

Separately, .dockerignore does not list target, so any docker build from
a developer checkout uploads a multi-gigabyte build context.

Proposed solution

Four additions, no change to any Rust or Python source:

1. Root Dockerfile. Multi-stage; builds only
cargo build --locked --release -p switchyard-server and ships it on
debian:bookworm-slim. 40 MB, UID 65532, read-only-root friendly. The builder
needs cmake and clang because rustls pulls in aws-lc-rs.

docker build -t switchyard-server:0.2.0 .
docker run --rm -p 4000:4000 \
  -v "$PWD/routes.toml:/etc/switchyard/routes.toml:ro" \
  -e NVIDIA_API_KEY switchyard-server:0.2.0 --config /etc/switchyard/routes.toml

2. deploy/helm/switchyard/. The user-facing surface is the values file;
the deployment TOML is passed through verbatim, so the chart adds no second
schema to learn:

image:
  repository: ghcr.io/nvidia-nemo/switchyard/switchyard-server
  tag: "0.2.0"

apiKeySecret:
  name: switchyard-keys        # keys become env vars named by api_key_env

config:
  routes: |                    # crates/switchyard-server/README.md schema
    schema_version = 1
    [llm_clients.nvidia]
    ...

Pods carry a checksum/config annotation so editing the TOML rolls the
Deployment. Optional ServiceMonitor, PDB, HPA, TLS and routing-log volumes.

3. examples/kubernetes/. Envoy AI Gateway integration in both
directions, because they have different security properties and neither is
universally right:

	Envoy AI Gateway in front	Switchyard in front
Chain	client → Envoy → Switchyard → provider	client → Switchyard → Envoy → provider
Provider credentials	Switchyard pod env	BackendSecurityPolicy, never in the pod
Client identity	SecurityPolicy on the Gateway	nothing by default

That last row is worth stating plainly in docs: Switchyard authenticates no
one. It serves every request reaching its port. That is correct behaviour
for a component behind a gateway, and a real exposure when it is the front
door.

4. A release workflow publishing the image and chart to GHCR on
vMAJOR.MINOR.PATCH tags, mirroring the split already in publish.yml.

Alternatives considered

Extend benchmark/switchyard-rust-server.Dockerfile. Rejected: it exists
to serve benchmark harnesses, and overloading it would couple release
packaging to benchmark needs. It also builds without the release profile the
proxy wants.

Kustomize instead of Helm. Rejected: the deployment TOML is the one thing
users must template per environment, and Helm values express that directly. A
kustomize base can be generated from helm template if anyone wants one.

Document a kubectl apply manifest set with no chart. Simpler to review,
but pushes config-checksum rollout, probe tuning and Secret wiring onto every
user — exactly the duplicated work this is meant to remove.

Leave packaging downstream entirely. Defensible, and the status quo. The
argument against is that the correctness details listed under Problem are
properties of switchyard-server's own behaviour, so the project is best
placed to encode them once.

Scope notes

• Not a new role or component. Nothing in the
  RequestProcessor / LLMBackend / ResponseProcessor / ResponseTranslator chain
  changes. This is packaging and documentation only.
• No public API impact. Nothing added to or removed from
  switchyard/__init__.py.__all__; no Rust crate surface changes.
• Backward compatibility: additive. The only edit to an existing file is
  one line in .dockerignore (target), which strictly shrinks build context
  and cannot change build output.
• Open question on paths: deploy/helm/switchyard/ vs charts/, and whether
  the published image should live at ghcr.io/nvidia-nemo/switchyard/….

Additional context

Branches, deployed and exercised end to end before proposing:

• feature/kubernetes-deployment-assets — items 1–3 feat(deploy): add container image, Helm chart and Envoy AI Gateway, litellm examples #319
• feature/container-publish-workflow — item 4, stacked on the above

Validated on single-node k3s v1.34.10 with Envoy Gateway v1.8.3 and Envoy AI
Gateway v1.0.0, against nemotron-3-nano-30b-a3b, -super-v3 and -ultra:
both Gateways Programmed=True, all AI Gateway resources Accepted, real
completions through both chains, and client auth returning 401/401/200 for
missing, wrong and valid keys.

One interoperability finding, which may be a Switchyard-side bug

Envoy preserves the client's original Host in x-forwarded-host when it
rewrites Host for the backend. Switchyard forwards inbound request headers to
the provider it selects, so that header travels all the way upstream. The
NVIDIA inference endpoint rejects any request carrying it, returning a
model-group 404 even when the header holds that endpoint's own hostname.

Isolated by bisecting the forwarded header set: x-forwarded-host alone
reproduces the failure, every other forwarded header passes, and an otherwise
identical direct request succeeds.

AIServiceBackend.headerMutation does not fix it, because the AI Gateway's
ext-proc mutation runs before Envoy sets the header; the examples ship an
EnvoyPatchPolicy that removes it during routing instead.

Should Switchyard forward inbound x-forwarded-* and x-envoy-* headers to
third-party providers at all? Doing so leaks internal topology and, as here,
can break upstream routing. An allowlist, or stripping hop-by-hop and proxy
headers before the upstream call, may belong in Switchyard rather than in
every deployment's gateway config. Happy to split this into its own issue.