Skip to content

Fix full inference E2E model contention - #427

Open
ajroetker wants to merge 4 commits into
mainfrom
codex/fix-e2e-full-inference-ci
Open

Fix full inference E2E model contention#427
ajroetker wants to merge 4 commits into
mainfrom
codex/fix-e2e-full-inference-ci

Conversation

@ajroetker

Copy link
Copy Markdown
Contributor

Summary

  • use ggml-org/gemma-4-e2b-it-gguf as the single default for ordinary, tool-calling, and multimodal generation
  • bootstrap explicitly configured large models and pin the Gemma GGUF variant
  • keep ReleaseSafe assertions enabled while extending request and job budgets for CPU inference
  • accept both current and legacy ClipClap manifest shapes, and replace the brittle synthetic-color semantic assertion with embedding-space invariants

Root cause

The failed full E2E job loaded separate generator defaults under ReleaseSafe. Slow requests timed out after 30 seconds, retained competing model sessions, then cascaded through MODEL_RESOURCE_BUSY into server termination. The job also exposed two independent ClipClap test assumptions: the generated manifest no longer always has a files array, and solid-color images do not provide a stable semantic ranking signal.

Failed job: https://github.com/antflydb/antfly/actions/runs/30694586941/job/91355247573

Validation

  • 12 model-helper tests passed
  • ClipClap golden-contract test passed
  • live ClipClap embedding-space test passed
  • all 132 inference E2E tests collect successfully
  • workflow YAML parses successfully
  • git diff --check passes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant