feat(server): serve Florence-2 through a seq2seq worker loop #1073

Problem / Background

Epic #850 landed Florence-2 support, and it works end to end through the CLI:

mlxcel generate -m models/Florence-2-base-ft-bf16 --image <img> -p '<OD>'

mlxcel-server does not serve it. Startup refuses the checkpoint with a named error, added by #856 (PR #1071) in src/server/startup.rs:

Florence-2 is an encoder-decoder (seq2seq) VLM that mlxcel-server cannot serve yet. Run it through the CLI instead...

The reason is architectural, not incidental. mlxcel's generation engine is decoder-only everywhere except Whisper's dedicated ASR pipeline. Florence-2 is BART-style seq2seq: it runs an encoder pass over the fused vision-plus-prompt sequence, then decodes autoregressively with cross-attention against the cached encoder output. #856 wired this into the CLI as a pre-loop early exit in run_generate_once (src/commands/generate.rs dispatching to src/commands/generate_florence2.rs), following the DiffusionGemma and LLaDA-2 pattern, deliberately not routing through compute_vlm_embeddings, which cannot express an encoder pass plus a cross-attention decode. The server's worker loop is decoder-only and would serve garbage from the trait-completeness forward, so the startup refusal is the correct interim behavior rather than a bug.

Proposed Solution

Give mlxcel-server a seq2seq worker path so Florence-2 becomes reachable over HTTP, then remove the startup refusal.

Shape of the work:

• A seq2seq worker variant that runs the encoder pass once per request, caches the encoder output, and drives the cross-attention decode loop, parallel to how the decoder-only worker loop is structured today. Whisper's ASR pipeline is the closest existing precedent for a non-decoder-only serving path.
• Request handling that maps an OpenAI-compatible chat/completions request carrying an image plus a Florence-2 task prompt onto the same parse_task_prompt / Florence2Task path the CLI uses, and returns the post-processed Florence2TaskResult (text, boxes, polygons, OCR regions) in a stable response shape.
• Removal of the Florence2VLM bail in src/server/startup.rs once the path is live.

Security requirements carried forward from #855

Both items were explicitly handed forward and must be honored by any server surface.

1. Decompression-bomb defense is not inside the processor. preprocess_with_sizes (src/vision/processors/florence2.rs) takes an already-decoded DynamicImage, so it cannot bound the decode. Every request path must route decoding through decode_request_image_with_limits / ImageInputLimits (src/server/model_worker.rs, src/server/media.rs). The CLI path added by feat(models): Florence-2 end-to-end integration and real-checkpoint validation (sub of #850) #856 already does this via decode_image_payloads_with_limits (src/lib.rs).
2. Task prompt input text is untrusted on a server surface. Florence2Task::expand (src/models/florence2/tasks.rs) interpolates caller-supplied text into the encoder prompt for 7 of the 15 task modes (CaptionToPhraseGrounding, ReferringExpressionSegmentation, RegionToSegmentation, OpenVocabularyDetection, RegionToCategory, RegionToDescription, RegionToOcr). The CLI sidesteps the question by rejecting anything that is not a recognized task marker; a server accepting free-form input needs explicit validation (length bound, character/format constraints, and for the region tasks a check that the coordinate form is well formed).

Acceptance Criteria

• mlxcel-server accepts a Florence-2 checkpoint at startup and reaches a serving state.
• An OpenAI-compatible request carrying an image and a task prompt returns the same answer the CLI produces for the same model, image, and task marker (verified against a real checkpoint, for example Florence-2-base-ft-bf16, on at least the pure-text <CAPTION> and the box-producing <OD> paths).
• Server-side image decoding for this family is bounded by ImageInputLimits through decode_request_image_with_limits; an oversized or bomb payload is rejected before decode rather than after.
• Task-prompt input text reaching Florence2Task::expand is validated at the request boundary for the 7 input-taking task modes, with a rejection test for malformed and oversized input.
• The Florence2VLM startup refusal in src/server/startup.rs is removed, and no code path can hand a Florence-2 checkpoint to a decoder-only worker loop.
• Tests cover the seq2seq worker loop (encoder cache reuse across decode steps) and the request-to-response mapping.
• docs/supported-models.md updated to state that Florence-2 is servable, not CLI-only.

Technical Considerations

• The encoder output is per-request state, not per-token state. It must live alongside the decode KV cache for the request's lifetime and be dropped with it; a shared or leaked encoder cache across requests would cross-contaminate answers.
• Batching semantics are open. Florence-2's encoder pass has a different cost profile from the decode loop, so a naive drop into the existing continuous-batching admission logic may not be appropriate. Serving requests one at a time on the seq2seq path is an acceptable first landing if the limitation is documented and measured.
• Response shape for the structured task outputs (boxes, polygons, OCR regions) is a design decision this issue must settle, since OpenAI's chat schema has no native place for them. Options include serializing into the message content or an mlxcel-specific response field.