Skip to content

feat: add native xAI image and video generation support - #584

Merged
chenyme merged 3 commits into
devfrom
grok
Aug 7, 2026
Merged

feat: add native xAI image and video generation support#584
chenyme merged 3 commits into
devfrom
grok

Conversation

@chenyme

@chenyme chenyme commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Add native xAI/Grok media generation support across provider routing, backend adapters, model capabilities, and chat configuration.

This change:

  • Adds native xai_video protocol support for:
    • POST /v1/videos/generations
    • GET /v1/videos/{request_id}
    • Text-to-video and single-image-to-video requests
    • Asynchronous pending, done, and failed result polling
  • Aligns xAI image generation and image editing with the official API:
    • Supports aspect_ratio, resolution, n, and response_format
    • Uses the correct image/images payload fields for image editing
    • Parses URL, base64, and file_output.public_url responses
    • Preserves upstream MIME types when available
  • Registers Grok image and video models in model-kind inference, provider routing, endpoint selection, and admin protocol options.
  • Adds xAI capability presets for image generation, image editing, and video generation.
  • Adds chat controls for image aspect ratio, resolution, output format, image count, video duration, and video resolution.
  • Restricts video duration to integer values from 1 to 15 seconds and preserves numeric values when selected in the UI.
  • Sanitizes unsupported or invalid billable video parameters before routing and billing.
  • Prevents hot polling when Retry-After is missing, invalid, zero, or negative:
    • Minimum polling interval: 1 second
    • Maximum polling interval: 10 seconds
  • Fixes protected generated-video downloads by forwarding the provider bearer token only to same-origin artifact URLs.
  • Removes provider credentials on cross-origin redirects to prevent credential leakage.
  • Migrates the previous default model-option allowlist to include xai_video without overwriting administrator-customized allowlists.

Actual upstream-result duration billing is intentionally outside the scope of this pull request.

Change type

  • Bug fix
  • Feature
  • Documentation
  • Refactor
  • Configuration / deployment
  • Security hardening
  • Other

Affected areas

  • Frontend / UI
  • Backend / API
  • Authentication / authorization
  • Conversations / streaming
  • Files / RAG / extraction
  • Model routing / providers
  • MCP / tools
  • Billing / payments
  • Admin console
  • Deployment / Docker / configuration
  • Documentation

Verification

  • go test ./internal/infra/llm -run XAIVideo
  • go test ./...
  • pnpm check
  • git diff --check
  • Backend adapter, routing, option-policy, settings-migration, and media-artifact tests passed.
  • Frontend Biome and TypeScript checks passed.

No browser testing or paid xAI generation requests were performed.

Screenshots, API examples, or logs

No screenshots are included. The new controls reuse the existing model capability and chat configuration components.

Example video generation request:

{
  "model": "grok-imagine-video",
  "prompt": "A cinematic view of waves at sunset",
  "aspect_ratio": "16:9",
  "duration": 8,
  "resolution": "720p"
}

Example asynchronous submission response:

{
  "request_id": "request-id"
}

The adapter polls:

GET /v1/videos/{request_id}

until the result is done or failed.

Supported video controls:

  • Aspect ratio: 1:1, 16:9, 9:16, 4:3, 3:4, 3:2, 2:3
  • Duration: integer values from 1 to 15 seconds
  • Resolution: 480p, 720p, 1080p

Supported image controls:

  • Official xAI aspect ratios, including auto
  • Resolution: 1k, 2k
  • Image count: 1–10
  • Response format: url, b64_json

Configuration, migration, and compatibility notes

  • Adds the internal xai_video protocol and video_generations endpoint identifier.
  • Adds aspect_ratio, duration, and resolution to the default xai_video model-option allowlist.
  • Existing installations using the previous unmodified default allowlist are migrated automatically.
  • Administrator-customized allowlists are preserved. Administrators using a custom allowlist must explicitly add the xai_video paths if needed.
  • No database schema migration is required.
  • No environment variables, Docker configuration, public URLs, or CORS rules are changed.
  • Existing provider protocols remain unchanged.
  • Provider credentials are sent to generated artifact URLs only when the artifact and configured provider endpoint have the same HTTP origin.
  • Provider credentials are removed before following a cross-origin redirect.
  • Billing continues to use sanitized requested video duration. Accounting from the actual upstream result duration is not changed in this PR.

Documentation

  • Documentation is not needed for this change.
  • Documentation was updated.
  • Documentation still needs to be updated.

The supported options and constraints are exposed through the existing admin capability presets and chat configuration UI.

Security and privacy

  • No secrets, tokens, credentials, local config, or personal data are included.
  • User data access remains scoped by authenticated user context unless an admin-only path explicitly requires broader access.
  • Security-sensitive behavior was reviewed, including authentication, authorization, provider routing, generated-media downloads, billing inputs, and admin APIs.
  • Provider bearer tokens are restricted to same-origin artifact requests and stripped from cross-origin redirects.
  • Debug payloads do not include image data or provider credentials.

Checklist

  • I searched existing issues and pull requests.
  • Changes are focused and do not include unrelated refactors.
  • Tests or static verification were run where practical.
  • User-facing behavior, API behavior, compatibility notes, and configuration changes are documented.
  • Generated artifacts are included only when this project explicitly requires them.
  • Caches, build output, .pyc files, .env files, and local storage data are not committed.

Comment thread backend/internal/infra/llm/xai_videos.go Fixed
@chenyme
chenyme merged commit 5f6932c into dev Aug 7, 2026
11 checks passed
@chenyme
chenyme deleted the grok branch August 7, 2026 02:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: grok图片编辑 错误:图片编辑仅支持 application/json [Bug]: 无法使用grok-imagine-video,报错route_not_found

2 participants