CLIP embedder gets stuck with pool_size=1, causing infinite CPU spin

[dovinmu]

Summary

With pool_size=1, CLIP embedding requests can get permanently stuck, blocking all subsequent requests and causing infinite CPU spin.

Environment

• Model: openai/clip-vit-base-patch32
• Backend: GoMLX (Go)
• Config: pool_size: 1

Symptoms

1. Backfill processes some embeddings (e.g., 5 or 55), then stops
2. Subsequent embedding requests timeout with: "acquiring pipeline slot: context canceled"
3. CPU spins indefinitely (one core at 100%)
4. Restarting Antfly doesn't help - slot gets stuck again on first embedding request

Logs

lvl=info msg="Successfully created pooled embedder" poolSize=1 backend=go hasVisual=true hasAudio=false
...
lvl=error msg="failed to generate embeddings" model=openai/clip-vit-base-patch32 error="acquiring pipeline slot: context canceled"

Observations

• Text-only models work fine: BGE (BAAI/bge-small-en-v1.5) returns embeddings instantly
• CLIP specifically gets stuck: Multimodal embedding appears to acquire the pipeline slot and never release it
• Workaround: Setting pool_size: 2 or higher allows other requests to proceed while one slot is stuck

Possible Causes

1. Infinite loop in GoMLX backend when processing certain images
2. Deadlock in pipeline slot acquisition/release
3. Memory issue causing silent hang

Reproduction

1. Configure pool_size: 1 in Termite config
2. Create an Antfly table with CLIP index using base64 data URIs
3. Insert multiple documents with images
4. Observe backfill stalls and CPU spins indefinitely

Workaround

Set pool_size: 2 or higher in config.