Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions infra/gemma/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -391,6 +391,24 @@ endpoint with short requests until it answers, then send the inference to a warm
it returns in seconds. Many brief calls instead of one long one — the same shape as `202 +
poll`, applied one layer down.

### The detector's address must be https

`screening-gemma` has `allowInsecure: false`, so its ingress does not serve plain
HTTP — it answers with a redirect to HTTPS. That produces two different failures
depending on what the caller does with the redirect, and neither names its cause:

| Caller | Behaviour | Symptom |
|---|---|---|
| Does not follow redirects | Sees a non-200 forever | Endpoint reported unreachable, and the redirect alone does not start an app scaled to zero, so it never becomes reachable |
| Follows a 301 | The redirect rewrites POST as GET | `405 Method Not Allowed` from an endpoint that only accepts POST |

The second is the more misleading: the URL is right, the client is working, the
request arrives — and the verb has been changed underneath it by the transport.

Use `https://` in `SCREENING_LLM_GUARDRAIL_BASE_URL` and neither arises. Internal
ingress terminates TLS inside the environment, so this costs nothing and the
traffic still never leaves it.

### Cost scales with wake-ups, not with screenings

The GPU bills for its cold start whether or not the caller survives it, so the unit of cost is
Expand Down
4 changes: 4 additions & 0 deletions infra/worker-job.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,10 @@ properties:
- name: SCREENING_PORTKEY_VIRTUAL_KEY
value: screening-openrouter
## The detector's address, resolvable only inside this environment.
## MUST be https. The detector's ingress sets allowInsecure false, so
## a plain HTTP request is answered with a redirect, and a client
## following a 301 turns a POST into a GET -- the endpoint then
## rejects it with 405 rather than doing the work.
- name: SCREENING_LLM_GUARDRAIL_BASE_URL
value: ${GEMMA_URL}
## Names which identity DefaultAzureCredential should use, since the
Expand Down
Loading