Skip to content

Run Linux model E2E on pinned local Gemma (Refs #3764) - #3765

Merged
acoliver merged 21 commits into
mainfrom
issue3764
Sep 28, 2026
Merged

acoliver merged 21 commits into
mainfrom
issue3764

Conversation

@acoliver

@acoliver acoliver commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

Runner-local E2E cleanup (Refs #3764)

At e3a442dbe, the existing ordinary Linux E2E job/check name is preserved as the single runner-local Gemma route. The optional pilot job and pilot_local_model dispatch input are removed. Manual workflow_dispatch with branch_ref runs the same real host and Docker E2E legs; pushes to main run the host leg only. The renamed local profile, timeout flag and labels describe this one route without changing ordinary provider settings.

The latest ordinary PR E2E run 36377286727 passed on both host (job 108785708956) and Docker (job 108785708975). Both runners were AMD. Each leg covered 31/31 selected files and three selected shell cases, including the real echo hello-world tool and exact bar content replacement. Host replacement passed on its second file attempt; Docker passed on its first. The unchanged 4/4 request budget counts distinct tests, not total transport retries.

CI run 36377286242 passed Lint, Test, CodeQL and Actions lint. It has no optional pilot job or skipped matrix label. The earlier pilot run 36342130687 passed Intel Docker with a Haswell pin at an older SHA; it does not prove Intel repeatability at this SHA.

Ollama 0.31.1 is pinned to archive SHA-256 d297381efc136451f6fabb9dd644a67f70fe51c16815a0c4a95ff0e327a3afb4; Gemma is pinned to digest 07ea59a474013479c8b6b802bef095c40e964a1d776ba02f264c0e30e1aede0c. Historical todo-continuation and shell-case exclusions remain unchanged. Credentialed provider-specific E2E remains nightly, and normal PR runs receive no model-provider key. Fork PR heads remain outside the privileged target path.

Local verification at e3a442dbe passed: tests, lint, typecheck, build, formatting, actionlint and test audit. The PR remains draft. The bot Run LLxprt review check is still running after a previous instance timed out; no approval or completion is claimed.

@coderabbitai

coderabbitai Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository: vybestack/llxprt-code/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: e255eadb-8571-4dc2-9c74-06a5754ef46b

📥 Commits

Reviewing files that changed from the base of the PR and between fde30e4 and e3a442d.

⛔ Files ignored due to path filters (1)
  • dev-docs/integration-tests.md is excluded by !dev-docs/**
📒 Files selected for processing (20)
  • .github/workflows/e2e.yml
  • integration-tests/replace.test.ts
  • integration-tests/run_shell_command.test.ts
  • integration-tests/sandbox-git-ceiling.real.test.ts
  • packages/cli/src/utils/sandbox-containers.ts
  • packages/cli/src/utils/sandbox-git-ceiling.test.ts
  • packages/providers/src/openai/OpenAIClientFactory.test.ts
  • packages/providers/src/openai/OpenAIClientFactory.ts
  • packages/providers/src/openai/OpenAIProvider.ts
  • packages/test-utils/src/cli-args.test.ts
  • packages/test-utils/src/cli-args.ts
  • packages/test-utils/src/test-rig-setup.ts
  • packages/test-utils/src/test-rig.test.ts
  • packages/test-utils/src/test-rig.ts
  • scripts/bun-test-roots.ts
  • scripts/tests/bun-test-roots.bun.test.ts
  • scripts/tests/e2e-local-model.test.ts
  • scripts/tests/e2e-platform-cadence.bun.test.ts
  • scripts/tests/pr-mergeability-workflow-wiring-b.test.ts
  • scripts/tests/workflow-quota-selection.test.ts
 _______________________________________________
< Now streaming live: defusing your code bombs. >
 -----------------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the maintainer:e2e:ok Trusted contributor; maintainer-approved E2E run label Sep 26, 2026
@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

OpenCodeReview — automatic reviews suspended

Automatic OCR reviews are suspended for this PR after 2 of 2 automatic reviews.

To get more reviews you can:

  • Check the box below to re-enable automatic reviews (resets the counter), or

  • Comment /review, /ocr, or /open-code-review to request a single review on demand.

  • Re-enable automatic reviews

@github-actions

Copy link
Copy Markdown
Contributor

LLxprt PR Review unavailable

The walkthrough pipeline could not complete. Please inspect the workflow logs.

@acoliver

Copy link
Copy Markdown
Collaborator Author

Hosted pilot run 36265295571, head 2e734877e, failed both legs. Ollama 0.31.1 archive SHA-256 verified, Qwen3.5:2b digest matched 324d162be6ca5629ae4517c8710434d0bd2d665bc94dbad46e9af8fbf8a2f0df, and the model loaded on both CPU runners at 262,144 context (6,817,579,005 bytes, size_vram:0). Install, download and Docker build completed. In both legs, replace timed out at 240 seconds on all three file-level attempts, so neither demonstrated the required exact bar content result. sandbox:none shell timed out once and then failed the strict successful-literal-echo trace check twice despite displaying hello-world: 0/2 test files passed. sandbox:docker shell timed out once and passed on its second attempt after an actual hello-world tool invocation: 1/2 files passed. This establishes Docker-to-host connectivity and a real shell tool call, but not reliable completion of both canaries on hosted CPU.

A separate diagnostics bug prevented deeper attribution: upload-artifact rejects local-qwen35-pilot-sandbox:none and local-qwen35-pilot-sandbox:docker because of :. Thus the Ollama server log and model ledger were not uploaded, and the kept TestRig outputs were not in the artifact paths. The logs alone do not prove whether the repeated replace timeouts came from slow CPU inference, context processing, tool selection, or a stalled request; they do show no OOM or exhausted disk at the final resource snapshot (about 8 GiB available RAM; 78-81 GiB free disk). The budget step was skipped after the canary failures. Keep the pilot draft/non-required. Before another measurement, make artifact names filesystem-safe and collect the Ollama server log, ledger, and kept TestRig tool traces; then measure tokens and per-request latency for replace on the hosted runner, testing context/model choices without relaxing the tool-call or exact-content assertions.

@acoliver

Copy link
Copy Markdown
Collaborator Author

Hosted pilot run 36278157447 at 8260cd94f completed with both jobs failing. Host job: shell 1/1 in 218.142s; replace 0/3 (attempts 1 and 3 hit the 300,000 ms first-response watchdog; attempt 2 hit the 1,200,000 ms TestRig deadline after taking unrelated tools, including ast_edit). Docker job: shell 1/1 in 149.448s; replace 0/3 (attempt 1 hit the 300,000 ms watchdog, attempt 2 hit the 1,200,000 ms TestRig deadline, attempt 3 failed the strict unrelated-tool assertion after invalid file paths, unrelated tool calls, and Ollama XML tool-call parsing warnings). No exact replace success.

The intended pilot-only 600,000 ms watchdog did not take effect for the attempts reporting 300,000 ms. Compared with run 36276039788, shell still passes in both legs; the longer rig deadline now exposes wrong-tool behavior alongside the unchanged first-response timeouts rather than making replace pass. Both diagnostic artifacts uploaded: host and Docker, with Ollama logs, model ledger, and TestRig telemetry. Observed replace prompts reached about 18,064 tokens, below the configured 32,768 context; Ollama reported no truncation. Final snapshots: host 10 GiB RAM available / 81 GiB disk free; Docker 10 GiB / 78 GiB; neither job approached the 90-minute job cap. Qwen3.5:2b Q8_0 digest 324d162be6ca5629ae4517c8710434d0bd2d665bc94dbad46e9af8fbf8a2f0df loaded CPU-only. Budget checks were skipped after canary failures, not disabled. PR description updated; keep this pilot draft and #3764 open. Next: verify pilot watchdog wiring at runtime, then address model tool-selection/path errors and measure repeated hosted shell plus exact replace successes before promoting a credential-free required check.

@acoliver acoliver changed the title Pilot runner-local Qwen3.5 E2E (Refs #3764) Pilot runner-local Gemma 4 E2B QAT E2E (Refs #3764) Sep 27, 2026
@acoliver

Copy link
Copy Markdown
Collaborator Author

Hosted Gemma pilot: five-minute timeout traced to the HTTP transport

I inspected the downloaded host and Docker artifacts for run 36283740686, the installed OpenAI/Undici sources and the pilot/test-rig settings. No code, assertions or required checks changed in this investigation; HEAD remains 41b7d9613d926722db6dd29e30bf88e381af46d0.

  • Host replace: telemetry records an API request at 00:57:03.416Z and Request timed out. at 01:02:03.442Z (300,022 ms). The Ollama log records a /v1/chat/completions HTTP 500 at 01:02:03 after exactly 5m0s, followed by cancellation of a still-running 15,201-token prompt at 13,312 tokens. Two further attempts show the same pattern (01:07:05 and 01:12:08). Docker repeats it at 01:01:23, 01:06:26 and 01:11:30; on its first replace request only about 9,216 of 15,200 prompt tokens had been processed at cancellation. These were active CPU prompt evaluations, not an idle slot, completed model error, context truncation or runner job timeout. The model was already loaded and the server remained available afterward.
  • The OpenAI SDK is the source of the message Request timed out.: node_modules/openai/src/client.ts maps fetch timeouts to APIConnectionTimeoutError and src/core/error.ts supplies that text. Its configured default request timeout is 600,000 ms, and OpenAIClientFactory.ts does not reduce it. The pilot's first-response setting is also 600,000 ms (test-rig-setup.ts); the CLI watchdog would emit a distinct stream-watchdog error. Ollama's OLLAMA_LOAD_TIMEOUT=5m applies to loading, but the prompt was already being evaluated. GitHub permits 90 minutes for this job. The installed Undici lib/dispatcher/client.js sets headersTimeout=300e3 by default; LLxprt's OpenAI factory passes no dispatcher or fetch headers-timeout override. A controlled local delayed-headers probe with a short Undici deadline emitted UND_ERR_HEADERS_TIMEOUT, and the OpenAI SDK converted that failure to the exact hosted Request timed out. text. This points to Node/Undici's five-minute response-headers deadline as the initiating limit; Ollama's HTTP 500 follows cancellation by the client. The artifacts do not retain the nested fetch cause, so a future probe capturing that code would make the attribution conclusive at the individual hosted request level.
  • CPU feasibility is not demonstrated. On host, the 15.2k-token replace prompt reached 13.3k tokens by 300s, with throughput falling to ~44 tokens/s. In Docker the same prompt reached only ~9.2k by 300s, falling to ~31 tokens/s. The Docker shell completion itself took 237.189s of model time, and the request returned at roughly 237.8s, leaving too little time for the tool round trip under the pilot's unchanged 240,000ms shell TestRig limit. That is a real pilot deadline mismatch, not a failed tool-content assertion. The Docker job's 90-minute limit and memory/disk snapshots were not the observed failure.

Recommended test-first change, not applied: Add a Bun integration-style test against a delayed-header local HTTP server that exercises the real OpenAI client factory and verifies that a scoped, explicitly configured transport headers deadline longer than the delay succeeds while an unconfigured client keeps its usual behavior. Carry that opt-in transport setting through the pilot profile, and set its headers deadline above the 600,000ms first-response watchdog; do not alter the global Undici dispatcher or loosen normal provider defaults. Separately, give only the pilot shell TestRig a CPU-justified deadline (at least enough for the observed ~238s response plus the tool round trip), with a Bun test proving non-pilot deadlines remain unchanged. Re-run local canaries and targeted tests before one further opt-in hosted pilot, keeping exact bar content, strict shell trace, budget guard and required checks. Raising the watchdog or retrying the hosted job alone cannot fix Undici's current 300s headers limit; raising all limits alone does not prove this model can finish both tool round trips reliably on two CPU threads. No further pilot dispatched without a tested transport change.

@acoliver

Copy link
Copy Markdown
Collaborator Author

Hosted Gemma pilot run 36289820501 at 39997cf0c is complete. Host passed shell 1/1 (206.967 s), exact foo content → bar content replace 1/1 (567.999 s), and budget 4/4 distinct-test API requests (ceiling 4). Docker passed shell 1/1 (250.038 s), but replace failed 0/3: each attempt hit the 600,000 ms first-response watchdog at ~602 s, before a replace tool call. Docker budget was skipped following failure. Job wall times: host 15m50s; Docker 39m27s.

Host diagnostics confirm HTTP 200 for replace after 541.375 s, a successful replace tool call and exact bar content. Docker diagnostics show each 15,200-token replace prompt almost finished evaluation (15,196 tokens after ~581 s), then client cancellation before a response or tool call. The former five-minute HTTP 500 / Request timed out. failure did not recur: the pilot-scoped extended transport deadline let the host complete after 300 s and Docker requests run to the separate 600 s watchdog. No context truncation; CPU-only model. Final available memory/free disk: host 9.4 GiB/80 GiB; Docker 9.0 GiB/77 GiB. Neither job neared the 90-minute cap.

This is a hosted host-leg success, not the repeated two-leg result required for promotion. Next bounded experiment: test a pilot-only first-response watchdog increase to 750,000 ms (below the existing scoped 900,000 ms transport deadline), leave the strict shell/replace assertions and budget unchanged, then run one opt-in hosted host/Docker pilot. Keep #3764 open, this PR draft and the pilot non-required.

@acoliver

acoliver commented Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator Author

Same-SHA hosted repeat and E2E parity, #3764

The repeat opt-in run 36295275993 succeeded on 522090b6b544d7c3c4c003beda24a070c04d283a, with pilot_local_model=true. Both jobs completed on their first workflow attempt, each with two passing canary files, zero test failures, no file retries in the logs and a passing 4/4 real-model budget check (two declared requests per canary). The previous same-SHA run 36294170871 also passed both legs without observed retries. Across these two runs, host shell and exact-content replace passed 2/2 each; Docker shell and replace passed 2/2 each. These four per-sandbox successes do not establish the full E2E suite's parity.

Repeat leg Shell tool canary Exact foo content to bar content replace End-of-job resources Result
Host 1/1, 243.449 s 1/1, 648.469 s 15 GiB RAM total, 9.4 GiB available; 80 GiB disk free; 4.1 GiB model storage Success, 04:46:24–05:04:01 UTC
Docker 1/1, 246.696 s 1/1, 647.149 s 15 GiB RAM total, 9.3 GiB available; 77 GiB disk free; 4.1 GiB model storage Success, 04:46:24–05:06:15 UTC

Both used verified Ollama 0.31.1 and Gemma 4 gemma4:e2b-it-qat digest 07ea59a474013479c8b6b802bef095c40e964a1d776ba02f264c0e30e1aede0c, 32,768 context and CPU only (size_vram:0). Diagnostics uploaded as host artifact and Docker artifact. Both shell runs logged the existing advisory that the response did not include exit code 0; the strict actual-tool, literal-echo and output assertions passed. No failed test attempts or model-call flakes appeared in this repeat.

Parity before any full migration: .github/workflows/e2e.yml currently runs credentialed E2E Test (Linux) - sandbox:none and E2E Test (Linux) - sandbox:docker on internal PRs, merge groups, manual normal dispatches and main pushes (main push excludes Docker). The maintainer:e2e:ok internal PR label path requires a successful mergeability gate. The doc-only/duplicate filters, fork/secret isolation, 60-minute timeout, Node 24, npm ci, build and conditional Docker setup are part of this path. Its first test invocation selects 31 of the 33 discovered integration-test files: it excludes todo-continuation.e2e.test.ts and run_shell_command.test.ts. Its second invocation selects three shell tests from run_shell_command.test.ts: real-model shell selection plus fixture-backed stdin shell and platform listing. The first invocation includes the real-model replace.test.ts canary and all other selected fixture-backed, zero-provider-request, sandbox, and CI-skipped cases. The pilot selects only shell and replace, one case each, so it has not run the remaining 30 first-invocation files or two additional shell cases. Both lanes run scripts/check-e2e-model-budget.ts --ledger after successful tests: baseline 9, ceiling 4 per distinct test; a retry is reported separately. Lint also validates the budget declaration. The full nightly matrix (nightly.yml) separately runs credentialed E2E on Linux host/Docker and macOS/Windows host.

The credentialed path selects the provider/model/base URL via vars.LLXPRT_DEFAULT_PROVIDER, vars.LLXPRT_DEFAULT_MODEL, vars.OPENAI_BASE_URL, and a key chosen from secrets[vars.KEY_VAR_NAME] / secrets[vars.KEY_VAR_NAME_2] by the trusted-base quota selector before PR code checkout. It validates nonempty provider configuration. The pilot uses no provider secret: a placeholder key, local OpenAI-compatible Ollama at 127.0.0.1:12644/v1, and SANDBOX_FLAGS=--network host only for its Linux Docker leg to reach the host's loopback server. Pilot-specific profile/timeouts, Git ceiling and diagnostic steps are opt-in; credentialed provider settings are unchanged.

A safe migration would provision and pin Ollama in the full host/Docker matrix, retain the exact 31-file invocation plus three shell cases, existing assertions, budget guard, doc-only/fork/mergeability logic, check context names and required-check enforcement, and prove this larger suite works within the chosen job budget. Do not replace the full suite with the two pilot canaries or silently apply --network host to secret-bearing provider jobs. Keep credentialed provider-specific authentication, quota, endpoint, model and remote-service behavior in a separately named, scoped lane (and preserve nightly cross-platform coverage); explicitly decide and validate any check-name/ruleset transition rather than counting a non-required manual pilot as a required PR check. The effective main ruleset currently names Lint and Test as required status contexts, while this PR's ordinary credentialed E2E host/Docker checks are still failing due to provider HTTP 429 insufficient balance. The pilot result does not make PR #3765 green. Keep the PR draft, #3764 open, and do not merge or promote yet. No source/workflow or enforcement settings changed in this measurement.

@acoliver

Copy link
Copy Markdown
Collaborator Author

Opt-in full-selected run 36314643244: both legs pass, Intel backend still unmeasured

At 3965d14c786deb0e47a3aa84584cf88c820ceb39, workflow run 36314643244 completed success on its first workflow attempt. I read both job logs and both uploaded diagnostics: host job, Docker job, host artifact, Docker artifact.

CPU/backend: The Docker Ollama server identifies AMD EPYC 9V74 80-Core Processor, with AVX2 = 1 and AVX512 = 1. Host Ollama identifies AMD EPYC 7763 64-Core Processor, with AVX2 = 1 and no AVX512 entry. The Docker selection step printed Docker pilot host is not Intel; leaving Ollama CPU variants untouched; the warm-up verification step printed Docker pilot host is not Intel; CPU backend A/B not exercised. At this SHA the workflow's Intel-only pre-test guard would issue /api/generate, read the live llama-server process's /proc/$pid/maps, and require the only mapped libggml-cpu-*.so to be libggml-cpu-haswell.so. That guard exited before preflight/maps on this AMD runner. There is no positive live-maps/haswell evidence from this run. In prior Intel run 36310416392, warm-up succeeded but the old system_info assertion stopped before maps or Docker tests; it likewise cannot supply positive mapped-library proof. This run is an AMD full-selected pass, not an Intel force-haswell pass.

Selected tests and tools: Each leg passed 31/31 isolated selected files (zero failed files), then 3/3 selected shell cases (12 other shell cases filtered out). Docker replace passed on its first file attempt in 414.257 s, host in 648.571 s. Both artifact TestRig diagnostics show exact bar content after changing foo content; each replace telemetry records one successful native replace tool call. Docker's real shell test passed in 142.111 s, host in 246.053 s. Each shell telemetry records one successful native run_shell_command invocation with command: "echo hello-world"; tool output has Stdout: hello-world and Exit Code: 0. Stdin and listing cases were fixture-backed. In both shell logs the model's final prose omitted exit code 0, producing a non-fatal warning, while the strict actual-tool checks passed.

Budget/retries/resources: Both budget steps passed: 4/4 distinct-test model API requests, ceiling 4, historical baseline 9, split two replace/two real shell per leg. Each ledger has one replace test record and one real-shell test record: no selected canary file retry is observed. Each also has four zero-request extension install test entries from one shared setup directory; the budget report calls these retried: 4 runs, but they are not four model-canary retries. End-of-job host: 9.3 GiB RAM available, 3.0 GiB swap free, 80 GiB disk free. Docker: 9.2 GiB RAM available, 3.0 GiB swap free, 77 GiB disk free; model storage 4.1 GiB in each. No resource exhaustion is evident in these snapshots.

Promotion decision: Keep this PR draft and pilot opt-in/non-required. On an actual Intel Docker runner, first require the pre-test warm-up and positive live /proc/$pid/maps haswell-only proof, then pass the same 31/31 selected files, 3/3 shell cases, exact replace/real shell calls and 4/4 budget without canary retries; repeat to establish reliability across Intel samples. Separately decide how to cover excluded todo-continuation.e2e.test.ts and the 12 filtered shell cases, and retain provider-specific/credentialed E2E unless replaced by equivalent verified coverage. The existing budget guard and strict tool/content checks must remain. At this SHA the separate PR credentialed E2E host and Docker checks are failing, while Lint and Test pass, so the PR is not green for promotion. No source edit, review, OCR or merge was performed for this measurement.

@acoliver

Copy link
Copy Markdown
Collaborator Author

Implementation update for #3764: commit e705829877da67e932d61ab6bec4d738ebb2b262 is pushed to issue3764. It applies Intel CPU-library selection and warm-up verification to both host and Docker pilot legs, reads proc maps without relying on stat size, and captures cmdline, PPid, parent cmdline, and maps evidence. The focused Bun test file passes 13/13, focused actionlint passes, and formatting passes.\n\nHosted run 36320244031 completed both selected suites successfully, but it ran against prior SHA 3965d14c, before this commit was pushed. Both AMD runners skipped Intel diagnostics; Docker reports AMD EPYC 7763. Both selected suite legs completed 31/31 isolated test files plus 1/1 selected shell-test file and the unchanged budget guard: 4 requests against ceiling 4, baseline 9. That run does not prove the new workflow changes.\n\nThe one permitted repeat, 36321643875, is running at the pushed commit. I will add its final results when both legs finish. PR remains draft; no merge.

@acoliver
acoliver force-pushed the issue3764 branch 5 times, most recently from 9899392 to c78779b Compare September 27, 2026 13:38
@acoliver

Copy link
Copy Markdown
Collaborator Author

Bounded backend correction: hosted outcome

The Intel diagnostic correction was pushed to issue3764 as e705829877da67e932d61ab6bec4d738ebb2b262. The -s /proc/$pid/maps check was removed; the Intel-only guard now reads a maps snapshot, requires the mapped haswell library and rejects other CPU libraries. It captures the matched process cmdline, PID/PPid and parent cmdline. The haswell-only library selection and warm-up guard apply to both opt-in Linux pilot legs; AMD exits without changing libraries. A test-only follow-up at 68a0346ad0db6e78f8653c155939ada046360ec6 adds executed Bun coverage for a zero-size readable FIFO, a Linux-only live /proc case, vanished PID, spoofed cmdline, absent/alternative/additional AVX512 and AMX mappings, and Intel host/Docker versus AMD selection. The workflow contents are unchanged in the test-only commit.

The first dispatch after the implementation push, 36320244031, resolved to the old SHA 3965d14c and therefore is not proof of the fix. It passed both legs on the old revision, but Docker reported non-Intel. The one corrected-SHA dispatch, 36321643875, finished successfully on e70582987: host and Docker each passed 31/31 selected isolated files and the separate 1/1 selected shell-test file (three selected shell cases), exact replacement yielded bar content, and the unchanged budget guard passed at 4 distinct-test requests of ceiling 4 (baseline 9), two replacement and two shell requests. Both jobs explicitly logged Pilot host is not Intel in both the selection and verification steps. These are AMD-path results by the workflow's vendor check; the job logs do not identify a specific AMD model. Neither job selected or mapped haswell, so this run does not establish that Intel host warm-up avoids SIGSEGV, nor that Intel Docker maps haswell and excludes AVX512/AMX. No additional dispatch was made after the corrected run.

Focused Bun tests: 17 passed, one Linux-only test skipped on macOS, zero failed; focused workflow actionlint passed. Typecheck, build and test audit passed in the initial full cycle. The full local test suite did not pass: two runs each failed one unrelated CLI file (sandbox-launch-release.test.ts with kill() EPERM, then ProfileSaveStep.test.tsx with an asynchronous duplicate-name assertion). Full actionlint reports existing ShellCheck findings in ci.yml (SC2129, SC2016) and release.yml (SC2016); edited e2e.yml passes focused actionlint. The smoke request reached Zai but returned HTTP 429 insufficient balance. Do not treat these checks as green. Required/provider-specific E2E, correctness assertions, retries and the 4/4 request budget are unchanged. PR remains draft; no review/OCR or merge was run. The Intel mapped-backend and Intel selected-suite proof remain outstanding.

Comment thread scripts/tests/e2e-local-model.test.ts Fixed
@acoliver

Copy link
Copy Markdown
Collaborator Author

Final status for issue #3764 at fc81a74bde4f7a299b9897107afce1203e905c03

The branch is pushed and clean. The workflow change remains limited to the opt-in local-model pilot. On Intel, both matrix legs retain only libggml-cpu-haswell.so and run the warm-up/maps guard. AMD selection leaves its CPU variants alone. Required E2E checks, selected suites, retries, correctness assertions, and the 4-request budget were left unchanged.

The test additions execute the workflow's extracted shell scripts. They cover Intel selection in host and Docker matrix legs, AMD selection, absent and vanished PIDs, wrong cmdline, missing maps backend, AVX512 and AMX alternate/additional mappings, and cmdline/PPid/parent evidence. A Linux-only test starts a helper llama-server process that maps a haswell-named shared library, confirms /proc/$pid/maps has stat size zero, and runs the extracted guard against that live procfs file. Focused local Bun tests pass 17/17 on macOS with the Linux-only test skipped. On final revision fc81a74, the hosted Linux scripts job 108634517060 passed all 18 tests, including the live stat-zero procfs case. The final revision's JavaScript lint job also passed.

The permitted pilot repeat, run 36321643875, completed successfully at e705829877da67e932d61ab6bec4d738ebb2b262. Host and Docker each passed 31/31 selected files and 3/3 selected shell cases. The exact foo content to bar content replace passed on its first file attempt in each leg. The real echo hello-world shell case passed in each leg. Both unchanged budget checks passed at 4 requests against a ceiling of 4 and baseline 9. No canary file retries appear in the logs. The ledger's extension install test entry has four zero-request setup runs; those do not count toward the model request budget.

Both pilot legs reported a non-Intel result and left the installed CPU variants untouched. The warm-up step printed Pilot host is not Intel; CPU backend A/B not exercised in both jobs. The Ollama logs report AVX2, but neither the job log nor uploaded artifact records an AMD vendor ID or exact CPU model. Run 363216 therefore provides no live haswell mapping evidence and does not exercise the Intel backend path. The final test's live Linux procfs evidence comes from the separate hosted scripts job linked above.

Local verification: focused actionlint .github/workflows/e2e.yml, npx eslint scripts/tests/e2e-local-model.test.ts, full npm run lint, npm run typecheck, npm run format, and npm run build passed. The full test-audit scan completed with 3,014 files and zero parse errors. Full actionlint exits 1 on three existing ShellCheck findings in ci.yml and release.yml, outside the changed workflow. Full npm run test exits 1 on sandbox-podman-diagnostics.test.ts: it expected Bad remote forwarding specification for credential socket and received Credential proxy bridge timed out waiting for TCP tunnel in Podman VM. That four-test file passes 4/4 when run alone. The zai-glm-flash smoke failed after six transport attempts with HTTP 429: Insufficient balance or no resource package. Please recharge.

The separate credentialed PR E2E run 36324371266 failed both legs on the same Zai 429 insufficient-balance error. PR #3765 remains open and draft. No merge was performed.

@acoliver

Copy link
Copy Markdown
Collaborator Author

Final verification correction

The final pushed SHA is fc81a74bde4f7a299b9897107afce1203e905c03. Required Lint and Test checks pass, including Linux scripts job 108634517060, with 18/18 workflow-shell tests and the live stat-zero /proc/$pid/maps case.

The final local full npm run test rerun completed with exit code 1 because packages/tools/src/tools/direct-web-fetch.test.ts, DirectWebFetchTool > cancels observed 5 MiB overflow and returns no partial body, timed out at 180000 ms. The CLI shard passed 764/764 files and 9811 tests. This differs from the earlier transient Podman full-suite failure; that prior Podman file passed 4/4 in isolation.

Full local lint, typecheck, format, build, focused workflow actionlint, and test-audit passed. Full repository actionlint still reports three findings in unchanged ci.yml and release.yml. Zai smoke and credentialed E2E checks are blocked by HTTP 429 insufficient balance.

The permitted pilot repeat run 36321643875 passed both legs at earlier SHA e705829. Ollama artifacts identify AMD EPYC 7763 for both legs. Both selection and warmup steps reported non-Intel, so there is no Intel haswell mapping evidence. Both legs passed 31/31 selected files, 3/3 selected shell cases, first-attempt replace and shell correctness, and the 4/4 request budget. No third pilot was dispatched.

PR #3765 remains draft and is not ready to claim complete Intel hardware proof.

@acoliver

Copy link
Copy Markdown
Collaborator Author

The single opt-in x64 pilot at 2bbc746b9 completed successfully. This is the first full-selected Intel Haswell Docker result, but it is one Intel sample, so the draft pilot remains non-required. The normal credentialed PR E2E checks are unchanged and still fail on provider HTTP 429 insufficient balance.

  • Docker job: GenuineIntel, x86_64; the CPU-variant directory was reduced to libggml-cpu-haswell.so, and the live llama-server /proc/$pid/maps snapshot reported only that mapped CPU backend. Model inference passed with size_vram: 0. The selected run passed 31/31 files and 3/3 separate shell cases. The real replace produced exact bar content; the real shell invoked echo hello-world. The unchanged budget guard passed 4 distinct-test requests against ceiling 4 (baseline 9), two each for replace and shell. No selected-file retry was reported.
  • Host job: AuthenticAMD, x86_64; CPU variants were left untouched, and Ollama natively mapped libggml-cpu-haswell.so on this AMD runner. This is not a second Intel sample. Inference reported size_vram: 0. The first replace-file attempt failed the exact check (foo content remained); its second attempt produced exact bar content. The selected suite then passed 31/31 files and 3/3 shell cases, including the real echo hello-world call. Budget passed 4/4 distinct-test requests, with the replace record labeled retried: 2 runs.

Local focused tests (22 passed, one Linux-only skip), full npm run test, lint, typecheck, build, workflow actionlint, targeted Prettier and the changed-test audit passed. The local startup smoke reached Zai but received HTTP 429 insufficient balance. Current PR Lint and Test checks pass; the credentialed E2E host and Docker checks fail on the same provider balance, and LLxprt review remains pending. No required E2E, fork check, secret, budget, or correctness assertion was changed. Keep PR #3765 draft and issue #3764 open. Intel Haswell has one proven Docker job here, not a repeated Intel result or coverage equivalence for the excluded tests.

@acoliver acoliver changed the title Pilot runner-local Gemma 4 E2B QAT E2E (Refs #3764) Run Linux model E2E on pinned local Gemma (Refs #3764) Sep 27, 2026
@acoliver
acoliver marked this pull request as ready for review September 28, 2026 18:07
@acoliver
acoliver merged commit a302483 into main Sep 28, 2026
45 of 49 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintainer:e2e:ok Trusted contributor; maintainer-approved E2E run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants