Skip to content

[sentry-diagnostic:7851254696] FOUNDRY_LOCAL_HOST_ERROR on atomic-chat-core@0.23.4 (dist c21d7796d3b2) is raised only after the hidden foundry-host child exits before the "ready":true line. `hostEr - #119

Merged
Ooooze merged 3 commits into
mainfrom
codex/sentry-diagnostic/7851254696
Nov 18, 2026
Merged

Ooooze merged 3 commits into
mainfrom
codex/sentry-diagnostic/7851254696

Conversation

@Ooooze

@Ooooze Ooooze commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Diagnostic instrumentation only. The original issue is NOT fixed or closed.

FOUNDRY_LOCAL_HOST_ERROR on atomic-chat-core@0.23.4 (dist c21d779) is raised only after the hidden foundry-host child exits before the "ready":true line. hostError in src/runtime/foundry-local/runtime.ts collapses every such outcome into one of two AtomicCoreError messages, and loadFailureReport records neither the host exit code nor its stderr, so the event carries failure_reason=unknown, cause_code=FOUNDRY_LOCAL_HOST_ERROR and a generic exception value runtime load: unknown. The competing explanations (exit 3: the native Foundry Local library cannot load on this Windows machine; exit 4: the library loaded but a call failed; a signal death during load; exit 2: a core-side argv bug) are indistinguishable from the current telemetry. Release 0.23.4 and base a66aad0 differ only in src/diffusion/args*, src/cli/owner.test.ts, src/runtime/fastflowlm/runtime.test.ts and src/version.ts, so the foundry-local path is byte-identical to base and there is no existing fix. This phase implemented the planned bounded diagnostic candidate: the controller-generated observer now carries the Foundry Local sibling pair of the existing GenieX codes, and the regression test src/telemetry/sentry-agent-diagnostics.test.ts asserts both codes through the existing diagnosticTags/classifyFailure boundary, plus privacy, cyclic-cause and unchanged-reporting cases; it passes on the generated observer and fails on the original base where the rules are absent. No TypeScript defect is claimed and no repair is proposed; the original failure remains unexplained until released events carry the codes.

Evidence:
Event ATOMIC-CHAT-CORE-CM (ced119d9c8434e5e9a41fa0e30013202): type FOUNDRY_LOCAL_HOST_ERROR, value runtime load: unknown, facts provider=foundry-local, component=runtime, operation=load, source=model_load, failure_category=unknown, failure_reason=unknown, diagnostic_version=core-v1, active_backend=win-cuda-13.3-x64, gpu_model=NVIDIA GeForce RTX 5060 Ti, context_length=49152, gpu_layers=100, release atomic-chat-core@0.23.4 dist c21d779.
Stack: start (src/runtime/foundry-local/runtime.ts:303) → startHost (:364) → spawnAndAwaitReady (src/runtime/shared/process.ts:305) → hostError (src/runtime/foundry-local/runtime.ts:135) — the innermost frame is hostError, i.e. the child exited before readiness.
src/runtime/foundry-local/runtime.ts hostError(exitCode, stderr): exit 5 → FOUNDRY_LOCAL_ACCELERATOR_NOT_PREPARED; exit 3 → FOUNDRY_LOCAL_HOST_ERROR with message 'The Foundry Local library does not load on this computer.'; every other code → FOUNDRY_LOCAL_HOST_ERROR with message 'The Foundry Local host stopped before the model was ready.'; details = last 5 stderr lines.
src/runtime/foundry-local/host/run.ts documents the host exit codes: 0 done, 2 usage (a core bug), 3 the library does not load or speaks another C API, 4 a library call failed, 5 accelerator not prepared. Only 3 and 5 are distinguishable to a reader of the message.
src/runtime/shared/process.ts spawnAndAwaitReady routes an early exit, a clean exit without the ready marker, and a signal death to classifyExit; a signal death passes code: null, so a native crash of the host also becomes FOUNDRY_LOCAL_HOST_ERROR with no signal recorded.
src/telemetry/reports.ts loadFailureReport tags only provider, error_code, model_id, quant, backend, context_length, gpu_layers, is_embedding and oom_subtype; the host exit code and stderr never reach the report, matching the observed absence of exit_code/signal tags.
src/telemetry/diagnostics.ts classifyFailure has no FOUNDRY_LOCAL_HOST_ERROR entry in EXPECTED or OPERATIONAL, so the category and reason stay 'unknown'; the event's exception value runtime load: unknown is that component/operation/reason triple.
Release diff for 0.23.4#c21d7796d3b2 → base a66aad0 touches only src/cli/owner.test.ts, src/diffusion/args.ts, src/diffusion/args.test.ts, src/diffusion/compat.ts, src/diffusion/compat.test.ts, src/diffusion/server-process.test.ts, src/runtime/fastflowlm/runtime.test.ts and src/version.ts; src/runtime/foundry-local/ is unchanged, so the defect is present in base and is not an existing fix.
src/telemetry/sentry-agent-diagnostics.ts (generated, base + this candidate) now holds 14 rules: the 12 pre-existing sd.cpp / GenieX / mlx-server rules unchanged, plus 'The Foundry Local library does not load on this computer.' → FOUNDRY_LOCAL_LIBRARY_NOT_LOADED and 'The Foundry Local host stopped before the model was ready.' → FOUNDRY_LOCAL_HOST_STOPPED_EARLY, both sourced from src/runtime/foundry-local/runtime.ts, with include_cause_codes=true.
src/telemetry/sentry-agent-diagnostics.ts reads own-property message and code up to five causes, and AtomicCoreError declares both as own properties (src/contracts/errors.ts:318), so exact-message rules match without exposing details or paths.
Regression test src/telemetry/sentry-agent-diagnostics.test.ts imports the existing boundary (classifyFailure, diagnosticTags, failureObserver from ./diagnostics.js), asserts both Foundry Local codes, asserts the untagged FOUNDRY_LOCAL_ACCELERATOR_NOT_PREPARED sibling, cyclic causes and unchanged AtomicCoreError.toJSON()/failureObserver capture. check run: 9/9 tests passed on the generated observer; on the original base the 'tags Foundry Local host load failures with finite diagnostic codes' case fails because diagnostic_code stays undefined.
diagnostic_policy (this session, base a66aad0): status available, existing_rules 12, remaining_rule_capacity 52, cause_codes_enabled true; the proposed two-rule plan returned status accepted.

Question: When a Foundry Local model load fails with FOUNDRY_LOCAL_HOST_ERROR, was the hidden host unable to load Microsoft's native library (exit 3) or did the library or host fail after loading (exit 4/2) or exit without readiness?
Expected observation: Existing FOUNDRY_LOCAL_HOST_ERROR events from atomic-chat-core@0.23.4 gain diagnostic_code=FOUNDRY_LOCAL_LIBRARY_NOT_LOADED for the exit-3 message and diagnostic_code=FOUNDRY_LOCAL_HOST_STOPPED_EARLY for the other exit paths, so the two explanations separate on the same events without any new report, severity or exit-code change.
Removal condition: Remove the two rules once the library-not-loaded versus stopped-early split is established from released events, or once the host exit code and signal are carried in the report tags directly instead of being reduced to a fixed message.
Bounds: two finite tags on an existing event, five Error causes; no new events, free-form messages, user content or reporting-policy changes.

Regression tests: src/telemetry/sentry-agent-diagnostics.test.ts

Full CI and human review are required.

Report: http://127.0.0.1:8080/tenants/707d0855-80ab-4e1f-a156-f1c4546cbf52/runs/85d5e810-05b5-45d5-b39b-74dc5d8036b4

Sentry: https://atomic-chat.sentry.io/issues/7851254696/

@Ooooze
Ooooze marked this pull request as ready for review November 18, 2026 11:39
@Ooooze
Ooooze merged commit f6b59c0 into main Nov 18, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant