feat(providers): probe streaming native-tool contract at setup by plombeer31 · Pull Request #315 · AtomicBot-ai/atomic-agent

plombeer31 · 2026-09-02T10:42:20Z

The problem

A successful /v1/models request proves an OpenAI-compatible endpoint is
reachable. The pre-save key check (verifyProviderKey) goes further and proves
the credential is live and funded — but it sends one non-streaming completion
with no tools.

Neither exercises what an Atomic turn actually is: a streamed chat completion
carrying a tools payload, from which a native tool call has to come back
whole. Routes that pass both checks and then fail a real turn are exactly what
issue #110 reports from the field — STREAM_EARLY_EOF, HTTP 400 that appears
only once tools is in the body, forced tool choice refused, incomplete SSE
tool-call deltas — and until now the operator's first message was the test.

The fix

A conformance probe next to the existing key check, extending
src/llm/provider/verify/ rather than starting a parallel subsystem. It reuses
classifyVerifyResponse for auth/quota/model refusals (so the two checks cannot
disagree about a 402 or a Gemini 400), parseOpenAiSseEvent + mergeToolName
for reading the stream (so the probe and a real turn cannot disagree about what a
provider sent), and openAiFetch for transport.

The ladder

Three rungs, each reached only when the one above left the question open, so a
healthy route costs exactly one request:

forced named tool, streaming, tools payload. A complete native tool call
here is the whole answer → tools_supported.
tool_choice: auto, same payload. This is the request a real turn makes,
so it is what settles anything rung 1 left open — a refusal that was not
about the key, the quota or the model, or an acceptance the route then
ignored. A tool call here after a refusal means only the forcing was
unacceptable (forced_tool_choice_rejected); after an ignored forcing it
means the route emits tool calls in the mode Atomic uses (tools_supported).
no tools at all, same model, same streaming transport — reached only when
both tool requests were refused. If this answers, the route works and it is
specifically tools it rejects → tools_payload_rejected; if it fails too,
the failure was never about tools → provider_error. Skipped when rung 1
streamed: tools in the body is already demonstrably fine, and the control
could only mislead.

Rung 3 is deliberately an experiment, not a regex over the error body.
Provider wording for "tools unsupported" is not stable enough to hang a verdict
on; "refuses with tools, answers without them" is unambiguous, and it is the same
evidence a human would gather by hand. That is why no tools-related wording
matching exists anywhere in the classifier.

The request is the request a turn makes

Same streamed transport, same tools payload shape, parallel_tool_calls resolved
the way step-executor resolves it (agent.maxParallelToolCalls > 1), and the
same max_tokens cap — buildOpenAiChatBody sets that on every turn and has no
max_completion_tokens fallback, so a model that rejects the field fails every
real turn. A probe that omitted it could pass a route whose every message 400s,
which is the exact failure #110 is about; a retry_token_field refusal is
therefore reported (token_cap_rejected) rather than retried in a shape Atomic
never sends. The cap (1024) is far above a one-field tool call, and
finish_reason: "length" is read as inconclusive so it can never be reported as
a route defect either.

The one thing a turn sends that the probe cannot is a forced tool choice —
because a turn never sends one. tool_choice: "auto" is deliberate in
step-executor, with production-observed reasons written down beside it (the
Alibaba/Qwen-thinking gate answers 400 InvalidParameter to a forced choice at
all). The forced rung is the probe's instrument, not a requirement: it is the
only mode in which "no tool call" says something about the route rather than
about the model's mood.

Statuses distinguished

tools_supported, inconclusive_no_tool_call, forced_tool_choice_ignored
(forced choice accepted and ignored, and nothing called under auto either),
forced_tool_choice_rejected, tools_payload_rejected, token_cap_rejected,
model_unavailable, endpoint_auth_failed, quota_or_routing_failed,
stream_early_eof, malformed_tool_call, plus unreachable / timeout /
cancelled / provider_error. Each gets a sentence naming the next move
(describe-contract-probe.ts), not HTTP N.

stream_early_eof is judged first and unconditionally among accepted
streams: a tool call assembled out of a body that never announced its end may be
missing argument fragments still in flight, which is the mistake
applyToolCallTerminationSafety already exists to prevent on the real path. It
is reached only for a body that actually stopped — our own deadline and our own
abort are reported as timeout and cancelled, never as a route defect.

Safety properties

Never dispatched. The synthetic function is a diagnostic fixture. Nothing
registers it, nothing dispatches it, the probe does not import the registry,
and its name (atomic_contract_probe) sits outside the dotted namespace every
built-in tool uses so it cannot shadow one. A test pins that against
DEFAULT_TOOL_DESCRIPTORS.
Inconclusive stays inconclusive. A text answer under auto is a legal
model choice; it is reported as unproven, never as incompatible.
Nothing silently marked compatible. reportModelConfigured requires a
clean key check and a probe that proved the route. A skip (local server,
CLI-backed provider, no key yet) reports as a skip with its own reason, never
as a pass.
Nothing warned about that a turn cannot hit. A route that refuses a forced
choice and streams a call under auto runs every real turn, so it counts as
proven and produces no warning.
Never blocks a save. Some providers block synthetic probes; a custom
endpoint the operator knows works must stay configurable, so the probe is
advisory and the warning is the override.
Redaction. Details are bounded (300 chars, never the full body) and
credential-scrubbed: the exact key by match plus key-shaped strings
(sk-…, AIza…, Bearer …) by pattern. That redaction is now shared with
verifyProviderKey, which previously only removed the exact key.
Never per turn. Two callers only: a provider save, and /llm check.

Wiring

Providers wizard save and first-run cloud onboarding run it once, after the key
gate passes and while Esc can still abandon the whole submit (nothing has
reached disk yet). 12s budget, and the abort reaches the open stream, not just
the connect.
/llm check runs it on demand against a saved provider, described to the probe
through the same configure wizard state the panel builds for c, so endpoint
/ key / model resolution is literally the wizard's rather than a second
implementation that could drift. It prints the redacted provider detail when
the route did not pass.

Review round (commits 2–3)

An adversarial review of the first commit found five high-severity problems.
All are addressed:

The headline wiring had no test. Both save paths and the
reportModelConfigured gate could be deleted and every test still passed.
completeWizard and first-run onboarding are now driven end to end with only
the disk writes stubbed, in both shapes (proven → saved and reported;
early-EOF → saved, warned, not reported).
The redaction test was vacuous — 400 filler characters meant the length
cap satisfied it with redaction removed. Fixture fixed, and
redact-provider-detail.test.ts pins each rule on short strings.
The probe reported its own deadline as stream_early_eof as soon as one
byte had arrived (an OpenRouter queue keepalive is enough). Now timeout.
Forced-choice verdicts warned about a request Atomic never makes.
forced_tool_choice_rejected is proven; an ignored forcing escalates to the
auto rung instead of stopping there.
The probe omitted max_tokens, which every turn sends and some models
reject — a pass could precede a turn failure of exactly the class Provider setup should probe streaming native-tool compatibility #110 is
about. Now sent, with token_cap_rejected for the refusal.

Also from that review: a mid-stream abort is cancelled rather than
stream_early_eof; the dead contractProbeFoundDefect (whose doc described a
policy the code contradicted) is gone; parallel_tool_calls is resolved instead
of hardcoded; keyless-listing presets are skipped rather than told their absent
key "was rejected"; the unreachable MAX_PROBE_REQUESTS guard and the second,
unused timeout constant are gone; /llm check is in the /llm menu description;
and the redacted detail now has a consumer.

Test evidence

$ npm run lint                       # tsc -p tsconfig.json --noEmit
(clean)

$ npx vitest run src/llm/provider/verify
 Test Files  7 passed (7)      Tests  78 passed (78)

$ npx vitest run src/tui/providers src/tui/commands src/tui/submit-handler.test.ts src/tui/menu
 Test Files  23 passed (23)    Tests  386 passed (386)

$ npx vitest run src/tui/components/cloud-provider-onboarding.test.tsx \
    src/tui/components/cloud-provider-onboarding-mouse.test.tsx \
    src/tui/hooks/use-onboarding-lifecycle.test.tsx
 Test Files  3 passed (3)      Tests  15 passed (15)

$ npx vitest run src/llm src/tui          # broad regression sweep
 Test Files  305 passed (305)  Tests  3300 passed (3300)

(The first version of this PR quoted "27 files / 415 tests" for the third
command; that number was wrong and did not reproduce. The numbers above are from
runs on the current head, Node v25.7.0.)

The broad sweep reports one pre-existing unhandled error unrelated to this
change: spawn .../llama-server EACCES from
src/tui/local-models/local-models-orchestrator-auto-update.test.ts, an
environment permission issue on this machine. No test failed.

New test files: run-contract-probe.test.ts (23),
classify-contract-probe.test.ts (17), probe-wizard-contract.test.ts (9),
accumulate-probe-stream.test.ts (7), redact-provider-detail.test.ts (6),
plus new cases in providers-orchestrator.test.ts,
cloud-provider-onboarding.test.tsx and slash-command-handler.test.ts.

Acceptance case	Test
valid streamed tool call	`run-contract-probe.test.ts` "proves support from one forced, streamed native tool call" (also asserts exactly one request)
forced-tool rejection	"reports a forced tool choice the route refuses but tools it accepts"
forced choice ignored	"settles a route that ignores forcing by asking the way a turn asks" / "reports an ignored forcing when auto declines as well"
auto-mode text response	"calls a plain text answer under auto inconclusive, not unsupported"
endpoint success with no tools	"proves the tools payload is the problem by answering without it" — asserts the control body carries no `tools`/`tool_choice`
early EOF	"reports a stream that ended early"
our deadline / our abort	"calls its own deadline a timeout, even once bytes have arrived" / "reports an abort that lands mid-stream as cancelled"
token cap refused	"reports a rejected token cap instead of retrying with the other field"
quota failure	"stops at a quota refusal instead of climbing the ladder"
malformed deltas	"reports malformed tool-call deltas", plus nameless-call and wrong-name cases in `classify-contract-probe.test.ts`
redaction	`redact-provider-detail.test.ts` (per rule) + "keeps credentials and whole response bodies out of the detail"
save wiring + gate	`providers-orchestrator.test.ts` "probes the route on save and reports a backend only once it is proven" / "saves but reports no working backend when the probe fails"; `cloud-provider-onboarding.test.tsx` "exercises the streaming tool contract before finishing first-run" / "carries the probe's verdict into the finish note without blocking the save"

Each of these mutations now fails at least one test: redaction disabled; the
KEY_SHAPED loop deleted; the probe removed from either save path; the gate
weakened to gate.warning === null; the old "timed out with zero bytes" rule
restored; the abort dropped from the read race; max_tokens removed from the
probe body; forced_tool_choice_rejected dropped from the proven set.

Deliberately left out

No override UI control. The issue asks that custom endpoints stay savable
"with an explicit warning/override". This ships the warning half: the probe
cannot refuse a save, so every endpoint remains savable and the operator is
told what was not proven. A persisted "I know, stop telling me" acknowledgement
is a config-shape decision (where it lives, whether it survives a model change)
I did not want to make unilaterally.
No per-provider result stored or shown in the LLM panel. The verdict is
emitted as providers_status + runtime_info and not persisted, so a stale
verdict can never outlive the route it described. A badge on the provider row
would need a staleness policy first.
No max_completion_tokens fallback in the probe. Adding one would make the
probe pass a route the turn path cannot use; the fix for such a route belongs
in buildOpenAiChatBody, as its own change.
Not wired to openai-provider.ts's health(). That method is called on
paths where spending a generation would be wrong; the probe stays setup-time
and on-demand as the issue requires.
Gemini's native surface is untouched. The probe speaks the
OpenAI-compatible contract, which is what Atomic uses for every cloud kind
including Gemini's /v1beta/openai.

Fixes #110

A live API key proves the account, not the route. `/v1/models` proves reachability, and the pre-save key check proves one non-streaming completion with no tools — neither exercises what an Atomic turn actually is: a streamed chat completion carrying a `tools` payload from which a native tool call must come back whole. Routes that pass both and then fail a turn are common (STREAM_EARLY_EOF, HTTP 400 only once `tools` is in the body, forced tool choice refused, incomplete tool-call deltas), and until now the operator's first message was the test. Adds a conformance probe next to the existing key check, built as a three-rung ladder so a healthy route costs exactly one request: 1. forced named tool, streaming, tools payload — a complete tool call here is the whole answer; 2. `tool_choice: auto` with the same payload, reached only when rung 1 was refused for a reason that is not the key, the quota or the model — a tool call now means it was the forcing the route refused; 3. the same streamed completion with no tools at all, reached only when both tool requests were refused — if this answers, the route works and it is specifically `tools` it rejects. Rung 3 is an experiment rather than a regex over error wording, which is not stable across providers; "refuses with tools, answers without them" is. A text answer under `auto` stays inconclusive and is never reported as "tools unsupported". The synthetic function is a diagnostic fixture: nothing registers it, nothing dispatches it, and its name sits outside the dotted namespace every built-in tool uses so it cannot shadow one. Details are bounded and credential-scrubbed — the exact key by match, key-shaped strings by pattern — and that redaction is now shared with the key check. Wired into the two setup paths (Providers wizard save, first-run cloud onboarding) as advisory only: it can never refuse a save, so a custom endpoint that blocks synthetic probes stays configurable, but a route it could not prove no longer reports as a working backend. `/llm check` runs it on demand against a provider that is already saved. It never runs on a turn path.

Review of the contract probe found several verdicts that describe something Atomic did, not something the route did. Each one reaches an operator as a sentence about their provider, so each is a bug. * **Our deadline was reported as `stream_early_eof`.** The timeout flag was discarded the moment any byte arrived, so a queued OpenRouter request (it sends `: OPENROUTER PROCESSING` while it waits) or a slow first token was classified as "closed the stream before finishing the answer. Turns will end mid-tool-call on this route" — and blocked from `reportModelConfigured`. The flag now survives: only a stream that announced its own end before the timer fired is classified at all, anything else is `timeout`, which is what it always was. * **An abort mid-stream was reported the same way.** Once headers had arrived the reader's rejection was swallowed and the partial body classified, so cancelling produced an invented route defect. The read now races the caller's signal, which also means Esc gets the screen back at once instead of at the deadline. * **Forced-tool-choice verdicts warned about a request Atomic never makes.** `step-executor` sends `tool_choice: "auto"` on every turn, deliberately, with production-observed reasons written down beside it. A route that refuses a forced choice and streams a complete call under `auto` therefore runs every real turn — it is now `proven`, not a warning that suppresses "this install has a working backend". A route that *accepts* forcing and ignores it no longer stops the ladder either: rung 2 asks the way a turn asks, and only if that also produces nothing is `forced_tool_choice_ignored` reported. * **The probe sent no `max_tokens`, which every turn sends.** `buildOpenAiChatBody` sets it unconditionally and has no `max_completion_tokens` fallback, so a model that rejects the field fails every turn — exactly the class of failure this probe exists to catch, and one it passed. The cap is now in the body, generous enough (1024) that it cannot truncate a one-field tool call, a `retry_token_field` refusal is reported as the new `token_cap_rejected` instead of being retried in a shape Atomic never uses, and `finish_reason: "length"` is read as inconclusive so our own cap can never be reported as a route defect either. Smaller corrections from the same review: * `parallel_tool_calls` is no longer hardcoded: the wizard resolves what a turn would send (`agent.maxParallelToolCalls > 1`). * Keyless-listing presets (Nous, Novita, Ollama Cloud, SambaNova, Sarvam) are skipped rather than probed with an empty credential and told their absent key "was rejected" — that flag is about listing models, not about completions. * `contractProbeFoundDefect` is gone. It had no caller and its doc described a warning policy the shipped code contradicts. * One timeout constant instead of two: the 20s default no caller passed described a budget that never existed. * The unreachable `MAX_PROBE_REQUESTS` guard is gone; the ladder is three fixed rungs and the deadline is the real bound. * `/llm check` is now in the `/llm` menu description, and prints the redacted provider detail when the route did not pass — until now that string was computed, scrubbed and never shown to anyone.

Three claims the feature is sold on had no test that could fail. Both save paths could be deleted wholesale and every test still passed: the wizard's `completeWizard` tests all stop at a refused key, so nothing reached the probe call, the gate, or the note the operator gets. `completeWizard` now runs end to end with only the disk writes stubbed — real key check, real probe, real gate — in two shapes: a route that streams a native tool call is saved *and* reported as a working backend, and a route whose stream ends early is saved, warned about, and not reported. First-run onboarding gets the same pair. Deleting either call site, or weakening the gate to `gate.warning === null`, now fails four tests. The redaction test was vacuous: 400 filler characters sat in front of the credentials, so the 300-character cap satisfied both assertions on its own and the test passed with `redactProviderDetail` reduced to a plain slice. The fixture now puts both keys inside the cap, and `redact-provider-detail.test.ts` pins each rule on strings short enough that truncation cannot do the work — including the pattern pass, the one capability this adds over the key check's exact-key removal, which nothing in the repo detected the loss of. Verified by mutation, each of these now fails at least one test: redaction disabled; the `KEY_SHAPED` loop deleted; the probe removed from either save path; the gate weakened; the old "timed out with zero bytes" rule restored; the abort dropped from the read race; `max_tokens` removed from the probe body; `forced_tool_choice_rejected` dropped from the proven set.

Conflict resolutions against the context-usage work already on the rc branch: - runtime/bootstrap.ts: one save at the end of a turn now stamps both the turn's context usage (AtomicBot-ai#320) and the provider/model it ran on (AtomicBot-ai#321). Two PRs rewrote the same choke point; keeping either alone silently dropped the other's stamp. - session/index.ts: both barrels re-exported. - bootstrap.test.ts: the two near-identical turn-stamp tests folded into one that asserts both stamps, returned and reloaded. Also fixes merge-caused breakage between AtomicBot-ai#309 and AtomicBot-ai#315: AtomicBot-ai#309 made `OpenAiToolCallDelta.index` optional, so the contract probe's accumulator no longer type-checks and would have keyed every index-less call into slot 0. It now resolves call identity the way the stream consumer does — index > id > the call currently open.

plombeer31 added 3 commits September 2, 2026 13:41

plombeer31 mentioned this pull request Sep 3, 2026

Release candidate 2026-09-03: PRs #309-#324 #325

Merged

plombeer31 merged commit 4e80015 into main Sep 3, 2026
2 checks passed

Navigation Menu

Uh oh!

feat(providers): probe streaming native-tool contract at setup - #315

feat(providers): probe streaming native-tool contract at setup#315
plombeer31 merged 3 commits into
mainfrom
fix/issue-110-provider-contract-probe

plombeer31 commented Sep 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

plombeer31 commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The problem

The fix

The ladder

The request is the request a turn makes

Statuses distinguished

Safety properties

Wiring

Review round (commits 2–3)

Test evidence

Deliberately left out

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

plombeer31 commented Sep 2, 2026 •

edited

Loading