feat: improve VLA template AI assistant

test-env/README.md

@@ -35,12 +35,18 @@ RabbitMQ.
 The template assistant is optional. To enable it, provide these variables before starting the stack:
 
 ```console
-export VLA_MANAGER_AI_URL=https://your-provider.example/v1
+export VLA_MANAGER_AI_PROVIDER=gemini
 export VLA_MANAGER_AI_API_KEY=your-key
-export VLA_MANAGER_AI_MODEL=your-model
 docker compose up -d --build
 ```
 
+Gemini uses Google's OpenAI-compatible endpoint and defaults to the low-cost
+`gemini-3.1-flash-lite` model. For OpenAI-compatible services, set
+`VLA_MANAGER_AI_PROVIDER=openai`, `VLA_MANAGER_AI_URL`, and `VLA_MANAGER_AI_MODEL`.
+For Anthropic, set `VLA_MANAGER_AI_PROVIDER=anthropic`, `VLA_MANAGER_AI_URL`, and
+`VLA_MANAGER_AI_MODEL`; its native Messages API is used automatically. `VLA_MANAGER_AI_URL`
+may be a provider base URL or the complete endpoint URL.
+
 The key is passed only to the VLA Manager API container. It is never included in the frontend bundle. Without these variables, the VLA Manager remains available and the assistant reports that it is not configured.
 
 To inspect a service while reviewing a failure:

test-env/compose.yml

@@ -85,6 +85,7 @@ services:
       VLA_MANAGER_DB_URL: postgresql://postgres:postgres@postgres-vla-manager:5432/dva
       # Rendered requirements are validated by asking processing to compile them.
       VLA_MANAGER_API_PROCESSING_URL: http://dva-processing-vla-manager:5000
+      VLA_MANAGER_AI_PROVIDER: ${VLA_MANAGER_AI_PROVIDER:-openai}
       VLA_MANAGER_AI_URL: ${VLA_MANAGER_AI_URL:-}
       VLA_MANAGER_AI_API_KEY: ${VLA_MANAGER_AI_API_KEY:-}
       VLA_MANAGER_AI_MODEL: ${VLA_MANAGER_AI_MODEL:-}

vla-manager-api/README.md

@@ -77,11 +77,32 @@ spec is still served and `/swagger/components.yaml` answers `404`.
 | `VLA_MANAGER_API_PORT` | `8000` | Listen port |
 | `VLA_MANAGER_API_LOG_LEVEL` | `info` | One of `critical`, `error`, `warning`, `info`, `debug` |
 | `VLA_MANAGER_API_PROCESSING_URL` | `http://localhost:5000` | URL to a DVA processing instance |
-| `VLA_MANAGER_AI_URL` | *(empty)* | Base URL or full `/chat/completions` URL of an OpenAI-compatible assistant service. Empty disables the assistant. |
+| `VLA_MANAGER_AI_PROVIDER` | `openai` | `openai` for OpenAI-compatible services, `gemini` for Gemini's compatibility endpoint, or `anthropic` for the native Messages API |
+| `VLA_MANAGER_AI_URL` | *(empty)* | Provider base URL or complete endpoint URL. Defaults to the provider's public endpoint. Empty credentials disable the assistant. |
 | `VLA_MANAGER_AI_API_KEY` | *(empty)* | API key used only by the VLA Manager API. |
-| `VLA_MANAGER_AI_MODEL` | *(empty)* | Model name sent to the assistant service. |
+| `VLA_MANAGER_AI_MODEL` | *(empty)* | Model name sent to the assistant service. Gemini defaults to `gemini-3.1-flash-lite`; other providers require an explicit model. |
 | `VLA_MANAGER_AI_TIMEOUT_SECONDS` | `30` | Maximum assistant request duration. |
 
 The assistant returns a structured draft and never saves a template. The UI must show the
 draft for review and use the normal template validation and save actions afterward. Do not
 place the API key in frontend environment variables or browser code.
+
+Example provider settings:
+
+```console
+# Gemini
+VLA_MANAGER_AI_PROVIDER=gemini
+VLA_MANAGER_AI_API_KEY=your-gemini-key
+
+# OpenAI or another OpenAI-compatible gateway
+VLA_MANAGER_AI_PROVIDER=openai
+VLA_MANAGER_AI_URL=https://api.openai.com/v1
+VLA_MANAGER_AI_API_KEY=your-openai-key
+VLA_MANAGER_AI_MODEL=your-model
+
+# Anthropic
+VLA_MANAGER_AI_PROVIDER=anthropic
+VLA_MANAGER_AI_URL=https://api.anthropic.com/v1
+VLA_MANAGER_AI_API_KEY=your-anthropic-key
+VLA_MANAGER_AI_MODEL=your-model
+```

vla-manager-api/src/vla_manager_api/assistant.py

@@ -11,6 +11,13 @@
 from .config import cfg
 from .models import TemplateNew
 
+_DEFAULT_URLS = {
+    "openai": "https://api.openai.com/v1",
+    "gemini": "https://generativelanguage.googleapis.com/v1beta/openai/",
+    "anthropic": "https://api.anthropic.com/v1",
+}
+_DEFAULT_MODELS = {"gemini": "gemini-3.1-flash-lite"}
+
 
 class AssistantUnavailable(RuntimeError):
     """The configured assistant service cannot be reached or is disabled."""
@@ -41,9 +48,25 @@ def build_assistant_messages(
         "VLA Manager template fields name, description, criterionType, "
         "targetAspect, and evaluationMethod. evaluationMethod must contain "
         "engine, variableSchema, and implementationTemplate. Supported "
-        "engines are SCHEMA, JQ, and GREAT_EXPECTATIONS. Never save data, "
-        "invent unsupported engines, or claim that a generated proposal is "
-        "validated. examples may contain passing and failing JSON samples. "
+        "engines are SCHEMA, JQ, and GREAT_EXPECTATIONS. criterionType must "
+        "be one of VALID_INVALID, IN_RANGE, GREATER_THAN, LESS_THAN. "
+        "targetAspect must be one of SYNTAX, TIMELINESS, ACCURACY, "
+        "COMPLETENESS, CONSISTENCY. Never save data, invent unsupported "
+        "engines, or claim that a generated proposal is validated. examples "
+        'must be an object with "passing" and "failing" values. Each value '
+        "should contain at least two representative examples when possible; "
+        "use an array for multiple examples, never a top-level array. Return "
+        "plain JSON without markdown fences. The implementationTemplate is "
+        "executed directly by the selected evaluator, so it must be code, "
+        "not an explanation. SCHEMA implementationTemplate must be a string "
+        "containing valid JSON Schema JSON, for example "
+        '\'{"type":"object","properties":{}}\'. '
+        "JQ implementationTemplate must be the executable jq expression "
+        "that returns an object with a boolean success field and optional "
+        "details string. GREAT_EXPECTATIONS implementationTemplate must be "
+        "a string containing valid YAML expectation configuration with type, "
+        "kwargs, and optional meta fields. Never put prose such as 'check "
+        "that...' in implementationTemplate. "
         f"Available templates: {json.dumps(template_context)}. "
         f"Current draft: {json.dumps(current_template or {})}."
     )
@@ -53,32 +76,45 @@ def build_assistant_messages(
     return messages
 
 
-def _completion_url() -> str:
-    base = cfg.ai_url.rstrip("/")
+def _provider_config() -> tuple[str, str, str, str]:
+    provider = cfg.ai_provider
+    if provider not in _DEFAULT_URLS:
+        raise AssistantUnavailable(
+            "The configured assistant provider is not supported."
+        )
+
+    api_key = cfg.ai_api_key.strip()
+    base_url = cfg.ai_url.strip() or _DEFAULT_URLS[provider]
+    model = cfg.ai_model.strip() or _DEFAULT_MODELS.get(provider, "")
+    if not api_key or not model:
+        raise AssistantUnavailable("The template assistant is not configured.")
+    return provider, base_url, model, api_key
+
+
+def _completion_url(base_url: str) -> str:
+    base = base_url.rstrip("/")
     if base.endswith("/chat/completions"):
         return base
     return f"{base}/chat/completions"
 
 
-def _complete_sync(messages: list[dict[str, str]]) -> str:
-    if not cfg.ai_url or not cfg.ai_api_key or not cfg.ai_model:
-        raise AssistantUnavailable("The template assistant is not configured.")
+def _messages_url(base_url: str) -> str:
+    base = base_url.rstrip("/")
+    if base.endswith("/messages"):
+        return base
+    return f"{base}/messages"
 
-    body = json.dumps(
-        {
-            "model": cfg.ai_model,
-            "messages": messages,
-            "temperature": 0.2,
-            "response_format": {"type": "json_object"},
-        }
-    ).encode()
+
+def _post_json(
+    url: str,
+    body: dict[str, Any],
+    headers: dict[str, str],
+) -> dict[str, Any]:
+    headers = {**headers, "Content-Type": "application/json"}
     request = Request(
-        _completion_url(),
-        data=body,
-        headers={
-            "Authorization": f"Bearer {cfg.ai_api_key}",
-            "Content-Type": "application/json",
-        },
+        url,
+        data=json.dumps(body).encode(),
+        headers=headers,
         method="POST",
     )
     try:
@@ -88,6 +124,34 @@ def _complete_sync(messages: list[dict[str, str]]) -> str:
         raise AssistantUnavailable(
             "The template assistant could not be reached."
         ) from exc
+    except (json.JSONDecodeError, TypeError, UnicodeDecodeError) as exc:
+        raise AssistantResponseError("The assistant returned invalid JSON.") from exc
+
+    if not isinstance(payload, dict):
+        raise AssistantResponseError("The assistant response had an unexpected shape.")
+    return payload
+
+
+def _complete_openai_compatible(
+    messages: list[dict[str, str]],
+    base_url: str,
+    model: str,
+    api_key: str,
+    provider: str,
+) -> str:
+    headers = {"Authorization": f"Bearer {api_key}"}
+    if provider == "gemini":
+        headers["x-goog-api-client"] = "prometheus-x-data-veracity/0.1"
+    payload = _post_json(
+        _completion_url(base_url),
+        {
+            "model": model,
+            "messages": messages,
+            "temperature": 0.2,
+            "response_format": {"type": "json_object"},
+        },
+        headers,
+    )
 
     try:
         content = payload["choices"][0]["message"]["content"]
@@ -100,6 +164,49 @@ def _complete_sync(messages: list[dict[str, str]]) -> str:
     return content
 
 
+def _complete_anthropic(
+    messages: list[dict[str, str]],
+    base_url: str,
+    model: str,
+    api_key: str,
+) -> str:
+    system_messages = [item["content"] for item in messages if item["role"] == "system"]
+    chat_messages = [item for item in messages if item["role"] in {"user", "assistant"}]
+    body: dict[str, Any] = {
+        "model": model,
+        "max_tokens": 2048,
+        "temperature": 0.2,
+        "messages": chat_messages,
+    }
+    if system_messages:
+        body["system"] = "\n\n".join(system_messages)
+    payload = _post_json(
+        _messages_url(base_url),
+        body,
+        {"x-api-key": api_key, "anthropic-version": "2023-06-01"},
+    )
+    blocks = payload.get("content")
+    if not isinstance(blocks, list):
+        raise AssistantResponseError("The assistant response had an unexpected shape.")
+    content = "".join(
+        block["text"]
+        for block in blocks
+        if isinstance(block, dict)
+        and block.get("type") == "text"
+        and isinstance(block.get("text"), str)
+    )
+    if not content:
+        raise AssistantResponseError("The assistant response was not text.")
+    return content
+
+
+def _complete_sync(messages: list[dict[str, str]]) -> str:
+    provider, base_url, model, api_key = _provider_config()
+    if provider == "anthropic":
+        return _complete_anthropic(messages, base_url, model, api_key)
+    return _complete_openai_compatible(messages, base_url, model, api_key, provider)
+
+
 async def complete_assistant(messages: list[dict[str, str]]) -> str:
     """Call the configured service without blocking the API event loop."""
     return await asyncio.to_thread(_complete_sync, messages)
@@ -117,6 +224,7 @@ def parse_assistant_response(content: str) -> dict[str, Any]:
     if proposal is not None and not isinstance(proposal, dict):
         raise AssistantResponseError("The assistant proposal is not an object.")
     if proposal is not None:
+        _validate_implementation_template(proposal)
         try:
             proposal = TemplateNew.model_validate(proposal).model_dump(
                 by_alias=True, exclude_none=True
@@ -129,3 +237,34 @@ def parse_assistant_response(content: str) -> dict[str, Any]:
     if examples is not None and not isinstance(examples, dict):
         raise AssistantResponseError("The assistant examples are not an object.")
     return {"message": response["message"], "proposal": proposal, "examples": examples}
+
+
+def _validate_implementation_template(proposal: dict[str, Any]) -> None:
+    """Reject assistant prose before it reaches the template editor.
+
+    Templates may contain Mustache placeholders, so static validation is only
+    applied when the implementation is self-contained. The selected evaluator
+    will validate rendered templates again when they are used.
+    """
+    evaluation_method = proposal.get("evaluationMethod")
+    if not isinstance(evaluation_method, dict):
+        return
+
+    engine = evaluation_method.get("engine")
+    implementation = evaluation_method.get("implementationTemplate")
+    if not isinstance(implementation, str) or not implementation.strip():
+        return
+    if "{{" in implementation or "}}" in implementation:
+        return
+
+    if engine == "SCHEMA":
+        try:
+            parsed = json.loads(implementation)
+        except json.JSONDecodeError as exc:
+            raise AssistantResponseError(
+                "The assistant returned a SCHEMA implementation that is not valid JSON Schema JSON."
+            ) from exc
+        if not isinstance(parsed, (dict, bool)):
+            raise AssistantResponseError(
+                "The assistant returned a SCHEMA implementation that is not a JSON Schema object."
+            )

vla-manager-api/src/vla_manager_api/config.py

@@ -8,6 +8,7 @@
 from pydantic_settings import BaseSettings, SettingsConfigDict
 
 LogLevel = Literal["critical", "error", "warning", "info", "debug"]
+AIProvider = Literal["openai", "gemini", "anthropic"]
 
 
 class Config(BaseSettings):
@@ -41,8 +42,11 @@ class Config(BaseSettings):
         "/app/openapi.yaml", validation_alias="VLA_MANAGER_OPENAPI_FILE"
     )
 
-    # Optional OpenAI-compatible chat completion service used by the
-    # template assistant. Empty URL disables the assistant cleanly.
+    # Optional chat completion service used by the template assistant.
+    # Empty credentials disable the assistant cleanly.
+    ai_provider: AIProvider = Field(
+        "openai", validation_alias="VLA_MANAGER_AI_PROVIDER"
+    )
     ai_url: str = Field("", validation_alias="VLA_MANAGER_AI_URL")
     ai_api_key: str = Field("", validation_alias="VLA_MANAGER_AI_API_KEY")
     ai_model: str = Field("", validation_alias="VLA_MANAGER_AI_MODEL")