WIP: Add Gemini and Vertex AI inference providers#41
Open
yangcao77 wants to merge 1 commit into
Open
Conversation
Signed-off-by: Stephanie <yangcao@redhat.com>
PR Summary by QodoAdd Gemini (OpenAI-compatible) and Vertex AI inference providers
AI Description
Diagram
High-Level Assessment
Files changed (5)
|
Code Review by Qodo
1. GitOps generator leaks providers
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
| inference: | ||
| providers: | ||
| - type: sentence_transformers | ||
| - type: openai |
There was a problem hiding this comment.
1. Gitops generator leaks providers 🐞 Bug ≡ Correctness
scripts/generate-gitops-manifests.sh only replaces the single - type: sentence_transformers line, so adding Gemini and Vertex blocks immediately after it in lightspeed-stack.yaml causes the generated GitOps ConfigMap to include those providers unintentionally. This can make generated deployments expose/initialize Gemini/Vertex providers and fail at startup or request-time when the new credentials/env vars are not configured.
Agent Prompt
## Issue description
`scripts/generate-gitops-manifests.sh` injects providers by matching the `sentence_transformers` line, but it does not remove subsequent providers from the source file. Since `lightspeed-core-configs/lightspeed-stack.yaml` now includes `gemini` and `vertexai` immediately after `sentence_transformers`, the generated GitOps manifest will accidentally include those providers.
## Issue Context
The generator’s awk block triggers on `- type: sentence_transformers`, prints vLLM/OpenAI providers, then continues printing the rest of `lightspeed-stack.yaml` unchanged. With the new provider blocks placed directly after the trigger line, they will be emitted into the generated ConfigMap.
## Fix Focus Areas
- scripts/generate-gitops-manifests.sh[56-92]
- lightspeed-core-configs/lightspeed-stack.yaml[28-40]
## Suggested fix
Update the generator so it replaces the *entire* `inference.providers` section (or explicitly skips the original providers following the match) rather than only skipping the one `sentence_transformers` line. For example, have awk enter a “skip providers” mode after the match until it reaches the next top-level key (e.g., `user_data_collection:`), and print only the injected providers for GitOps output.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| - ../lightspeed-core-configs/rhdh-profile.py:/app-root/rhdh-profile.py:Z | ||
| - ../llama-stack-configs/config.yaml:/app-root/config.yaml:Z | ||
| - ../rag-content:/rag-content:Z | ||
| # Only used when the vertexai provider is configured. Falls back to a |
There was a problem hiding this comment.
2. Invalid gcp creds placeholder 🐞 Bug ☼ Reliability
compose/compose.yaml defaults the credentials bind-mount to an {} placeholder file, while
default-values.env points GOOGLE_APPLICATION_CREDENTIALS at that mounted path. When Vertex AI is
used, Google auth will read an invalid credentials JSON, causing authentication/configuration
failures unless GOOGLE_APPLICATION_CREDENTIALS_HOST_PATH is explicitly set to a real credentials
file.
Agent Prompt
## Issue description
The compose setup always mounts a file at `/app-root/gcp-credentials.json`, defaulting to `env/gcp-credentials-placeholder.json` which contains `{}`. At the same time, `env/default-values.env` sets `GOOGLE_APPLICATION_CREDENTIALS=/app-root/gcp-credentials.json`, forcing Google auth to read the placeholder when users haven’t configured real credentials.
## Issue Context
This creates a confusing failure mode: Vertex AI usage will fail due to an invalid credentials JSON even though the environment variable is set and a file exists.
## Fix Focus Areas
- compose/compose.yaml[21-33]
- env/default-values.env[21-31]
- env/gcp-credentials-placeholder.json[1-1]
## Suggested fix
Avoid setting `GOOGLE_APPLICATION_CREDENTIALS` by default when the host credentials path is unset. Options:
1) Remove `GOOGLE_APPLICATION_CREDENTIALS` from `env/default-values.env` (leave it empty), and set it only in `env/values.env` when using Vertex.
2) Alternatively, set it in `compose/compose.yaml` using conditional interpolation (only set when `GOOGLE_APPLICATION_CREDENTIALS_HOST_PATH` is non-empty), and keep the placeholder mount purely to prevent mount errors.
Also consider updating docs wording to reflect whichever behavior you choose.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
yangcao77
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
vertexaiandgeminiinference providers tolightspeed-core-configs/lightspeed-stack.yaml. Since this schema has no dedicatedgeminiprovider type, Gemini is configured as anopenai-type provider pointed at Gemini's OpenAI-compatible endpoint (https://generativelanguage.googleapis.com/v1beta/openai/).GOOGLE_APPLICATION_CREDENTIALS_HOST_PATHvolume mount incompose/compose.yaml(with a harmless empty placeholder fallback via the newenv/gcp-credentials-placeholder.jsonso the mount never fails when unset), and updatesenv/default-values.envaccordingly.docs/PROVIDERS.md, including theopenai-type workaround for Gemini and expanded Vertex AI credential setup instructions.Test plan
make local-up) againstquay.io/lightspeed-core/lightspeed-stack:dev-20260710-2c3e52e; container reachedhealthy.GET /v1/modelsreturns models for both new providers (gemini: 55 models,vertexai: 124 models via dynamic discovery), alongsidesentence-transformers.POST /v1/queryreturns a valid response usingprovider: gemini.POST /v1/queryreturns a valid response usingprovider: vertexai.