Run model-dependent E2E tests against a small LLM inside GitHub Actions #3764

Problem

The model-dependent end-to-end tests currently require third-party model credentials. Contributors and PR checks should be able to exercise real model behavior, including tool calls, using a small model hosted on the runner instead.

Approach

• Evaluate a current open-weight small model released by September 26, 2026. Start with Qwen3.5-2B (released March 2, 2026); test Qwen3.5-4B if 2B cannot reliably complete the existing tasks. Run quantized weights through a local OpenAI-compatible server such as Ollama, using an established GitHub Actions CPU-runner installation pattern.
• Prove locally that the model works through the actual LLxprt integration test rig, not merely a direct model API call: verify a shell tool invocation and the real read/replace operation with exact file-content assertion. Measure repeated runs and capture failures, model digest, context usage, runtime, memory and disk use.
• Adapt the relevant E2E workflow/test-runner configuration so a GitHub-hosted runner can download and serve the model without a third-party API key, run the model-dependent tests and fail on genuine missing/incorrect tool calls. Handle host versus Docker networking explicitly. Preserve the existing E2E assertions, budget guard and required-check enforcement. Leave credentialed provider-specific coverage available for behavior the local model cannot represent.
• Validate the runner job in GitHub Actions and document reproduction and resource requirements. Do not present a locally passing Apple Silicon run as proof of CPU-only x64 runner feasibility.

Acceptance criteria

• A reproducible GitHub Actions E2E path exercises real LLxprt model calls without a remote inference service or secret API key; the test fails if expected tool calls or replacement results are wrong.
• Existing shell and replace canaries pass repeatedly with the selected model using the real test rig, and the selected model runs within hosted-runner CPU, memory, disk and job limits. Report measured pass counts and any retries.
• Model release, model artifact/digest, runtime version and Actions install pattern are documented. All changes to required CI coverage and sandbox behavior are explicit and justified, with no weakened tests or checks.

References

• Qwen's release list dates Qwen3.5 0.8B/2B/4B/9B to 2026-03-02: https://github.com/QwenLM/Qwen3.5
• Ollama's quantized Qwen3.5 variants: https://ollama.com/library/qwen3.5
• Established hosted-runner Ollama setup example (older model; deployment pattern only): https://stackoverflow.com/questions/78486506/ollama-running-with-github-actions
• Runner capacities: https://docs.github.com/en/actions/reference/runners/github-hosted-runners