Skip to content

WIP: Add Gemini and Vertex AI inference providers #41

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
yangcao77 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

WIP: Add Gemini and Vertex AI inference providers #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions compose/compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ services:
- ../lightspeed-core-configs/rhdh-profile.py:/app-root/rhdh-profile.py:Z
- ../llama-stack-configs/config.yaml:/app-root/config.yaml:Z
- ../rag-content:/rag-content:Z
# Only used when the vertexai provider is configured. Falls back to a

@qodo-code-review qodo-code-review Bot Jul 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

2. Invalid gcp creds placeholder 🐞 Bug ☼ Reliability

compose/compose.yaml defaults the credentials bind-mount to an {} placeholder file, while
default-values.env points GOOGLE_APPLICATION_CREDENTIALS at that mounted path. When Vertex AI is
used, Google auth will read an invalid credentials JSON, causing authentication/configuration
failures unless GOOGLE_APPLICATION_CREDENTIALS_HOST_PATH is explicitly set to a real credentials
file.
Agent Prompt 
## Issue description
The compose setup always mounts a file at `/app-root/gcp-credentials.json`, defaulting to `env/gcp-credentials-placeholder.json` which contains `{}`. At the same time, `env/default-values.env` sets `GOOGLE_APPLICATION_CREDENTIALS=/app-root/gcp-credentials.json`, forcing Google auth to read the placeholder when users haven’t configured real credentials.

## Issue Context
This creates a confusing failure mode: Vertex AI usage will fail due to an invalid credentials JSON even though the environment variable is set and a file exists.

## Fix Focus Areas
- compose/compose.yaml[21-33]
- env/default-values.env[21-31]
- env/gcp-credentials-placeholder.json[1-1]

## Suggested fix
Avoid setting `GOOGLE_APPLICATION_CREDENTIALS` by default when the host credentials path is unset. Options:
1) Remove `GOOGLE_APPLICATION_CREDENTIALS` from `env/default-values.env` (leave it empty), and set it only in `env/values.env` when using Vertex.
2) Alternatively, set it in `compose/compose.yaml` using conditional interpolation (only set when `GOOGLE_APPLICATION_CREDENTIALS_HOST_PATH` is non-empty), and keep the placeholder mount purely to prevent mount errors.
Also consider updating docs wording to reflect whichever behavior you choose.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

# harmless empty placeholder so the mount doesn't fail otherwise.
- ${GOOGLE_APPLICATION_CREDENTIALS_HOST_PATH:-../env/gcp-credentials-placeholder.json}:/app-root/gcp-credentials.json:Z,ro
environment:
- SERVICE_HOST=0.0.0.0
env_file:
Expand Down
30 changes: 30 additions & 0 deletions docs/PROVIDERS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,26 @@ You must pass the environment variable `OPENAI_API_KEY` to the Lightspeed Core c

Get your API key from [platform.openai.com](https://platform.openai.com/settings/organization/api-keys).

## Gemini

There is no dedicated `gemini` provider type in Lightspeed Core's inference schema. Gemini exposes an OpenAI-compatible endpoint, so configure it as an `openai`-type provider with a custom `base_url` instead:

```yaml
inference:
providers:
- type: openai
id: <your-unique-id>
api_key_env: GEMINI_API_KEY
extra:
base_url: https://generativelanguage.googleapis.com/v1beta/openai/
```

You must pass the environment variable `GEMINI_API_KEY` to the Lightspeed Core container.

Get your API key from [Google AI Studio](https://aistudio.google.com/apikey).

This is the consumer Gemini API (`generativelanguage.googleapis.com`), authenticated with a simple API key. If you need enterprise features (VPC-SC, IAM, billing tied to a GCP project, etc.), use Vertex AI below instead.

## Vertex AI (Gemini)

To add the `vertexai` inference provider, include the following in [lightspeed-stack.yaml](../lightspeed-core-configs/lightspeed-stack.yaml):
Expand All @@ -85,6 +105,16 @@ inference:

Additionally, you need to ensure your Google Application Credentials are mounted to the Lightspeed Core container and the `GOOGLE_APPLICATION_CREDENTIALS` environment variable is the path to the mount location.

To set this up with the provided `compose/compose.yaml`, set `GOOGLE_APPLICATION_CREDENTIALS_HOST_PATH` to the path on your host machine of a GCP service account JSON key (or your `gcloud auth application-default login` credentials file). The compose file mounts that file into the container and points `GOOGLE_APPLICATION_CREDENTIALS` at the mounted path for you — do not set `GOOGLE_APPLICATION_CREDENTIALS` to a host path yourself, since that path won't exist inside the container.

```env
VERTEX_AI_PROJECT=
VERTEX_AI_LOCATION=
GOOGLE_APPLICATION_CREDENTIALS_HOST_PATH=<path-on-your-host-to-a-gcp-service-account-json-key>
```

The service account (or `gcloud auth application-default login` credentials) needs the `Vertex AI User` role, and the Vertex AI API must be enabled on `VERTEX_AI_PROJECT`.

Provider details: [Llama Stack (OGX) Vertex AI docs](https://ogx-ai.github.io/docs/providers/inference/remote_vertexai).

## Restricting Models (`allowed_models`)
Expand Down
11 changes: 10 additions & 1 deletion env/default-values.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,19 @@ VLLM_TLS_VERIFY=
# OpenAI Inference Settings
OPENAI_API_KEY=

# Gemini Inference Settings
GEMINI_API_KEY=

# Vertex AI Inference Settings
VERTEX_AI_PROJECT=
VERTEX_AI_LOCATION=
GOOGLE_APPLICATION_CREDENTIALS=
## Host path to your GCP service account JSON key. The compose file mounts
## this into the container for you; do not set GOOGLE_APPLICATION_CREDENTIALS
## to a host path yourself.
GOOGLE_APPLICATION_CREDENTIALS_HOST_PATH=
## In-container path where the credentials above are mounted. Leave as-is
## unless you also change the volume mount in compose/compose.yaml.
GOOGLE_APPLICATION_CREDENTIALS=/app-root/gcp-credentials.json

# Ollama Inference Settings
OLLAMA_URL=
Expand Down
1 change: 1 addition & 0 deletions env/gcp-credentials-placeholder.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{}
10 changes: 10 additions & 0 deletions lightspeed-core-configs/lightspeed-stack.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,16 @@ llama_stack:
inference:
providers:
- type: sentence_transformers
- type: openai

@qodo-code-review qodo-code-review Bot Jul 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Gitops generator leaks providers 🐞 Bug ≡ Correctness

scripts/generate-gitops-manifests.sh only replaces the single - type: sentence_transformers line,
so adding Gemini and Vertex blocks immediately after it in lightspeed-stack.yaml causes the
generated GitOps ConfigMap to include those providers unintentionally. This can make generated
deployments expose/initialize Gemini/Vertex providers and fail at startup or request-time when the
new credentials/env vars are not configured.
Agent Prompt 
## Issue description
`scripts/generate-gitops-manifests.sh` injects providers by matching the `sentence_transformers` line, but it does not remove subsequent providers from the source file. Since `lightspeed-core-configs/lightspeed-stack.yaml` now includes `gemini` and `vertexai` immediately after `sentence_transformers`, the generated GitOps manifest will accidentally include those providers.

## Issue Context
The generator’s awk block triggers on `- type: sentence_transformers`, prints vLLM/OpenAI providers, then continues printing the rest of `lightspeed-stack.yaml` unchanged. With the new provider blocks placed directly after the trigger line, they will be emitted into the generated ConfigMap.

## Fix Focus Areas
- scripts/generate-gitops-manifests.sh[56-92]
- lightspeed-core-configs/lightspeed-stack.yaml[28-40]

## Suggested fix
Update the generator so it replaces the *entire* `inference.providers` section (or explicitly skips the original providers following the match) rather than only skipping the one `sentence_transformers` line. For example, have awk enter a “skip providers” mode after the match until it reaches the next top-level key (e.g., `user_data_collection:`), and print only the injected providers for GitOps output.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

id: gemini
api_key_env: GEMINI_API_KEY
extra:
base_url: https://generativelanguage.googleapis.com/v1beta/openai/
- type: vertexai
id: vertexai
extra:
project: ${env.VERTEX_AI_PROJECT:=}
location: ${env.VERTEX_AI_LOCATION:=global}
user_data_collection:
feedback_enabled: true
feedback_storage: '/tmp/data/feedback'
Expand Down
Loading