Skip to content

Fix full E2E promotion and inference flakes - #439

Open
ajroetker wants to merge 4 commits into
mainfrom
codex/fix-postmerge-full-e2e-flakes
Open

Fix full E2E promotion and inference flakes#439
ajroetker wants to merge 4 commits into
mainfrom
codex/fix-postmerge-full-e2e-flakes

Conversation

@ajroetker

Copy link
Copy Markdown
Contributor

Summary\n\n- make ordinary, tool-calling, and multimodal E2E generation share ggml-org/gemma-4-e2b-it-gguf\n- make promotion reconciliation retry pending work across leadership changes that do not emit a worker wake\n- fix Florence decoder tensor lifetime ownership and apply the borrowed-view contract to every encoder-decoder pipeline\n- honor --max-loaded-models in the inference CLI and make image extraction release models between GLiNER and Florence stages\n- reject incompatible cached reader artifacts during automatic reader selection\n- surface the terminating signal and server output tail when a local inference child exits\n\n## Root causes\n\nThe Antfly E2E timeout left promotion at target 3/applied 2 because source leadership is a live predicate, not an event source. A leadership transition could therefore leave pending work asleep forever.\n\nThe inference server aborted because Florence shallow-copied an encoder tensor whose output-admission lifetime hook was then released more than once. Once that was fixed, the test also exposed two masked routing issues: the CLI ignored the one-model limit, and image extraction could reinterpret the GLiNER directory as a reader while pinning GLiNER across the Florence stage.\n\n## Validation\n\n- make generate\n- 648 DB tests: 643 passed, 5 skipped\n- 2,056 inference tests passed in partitioned runs; loopback download fixtures rerun outside the sandbox\n- focused GLiNER to Florence to GLiNER E2E with max_loaded_models=1: 2 passed\n- repeated resolution race harness: 15/15 passed\n- native full-edition ReleaseSafe stripped build\n- Python model helper tests: 13 passed\n- Ruff and git diff checks

Use an interruptible std.Io futex generation for promotion retries, avoid redundant extraction model loads, cache lightweight OCR reader selection with bounded refresh, and move the residency CLI parser into the normal inference test graph.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant