Explore local open-source model (Qwen3.5 via Ollama) for on-device task interpretation

[hayesdavis]

Summary

Explore (not commit to) running a small local open-source model — the Qwen3.5 family via Ollama or llama.cpp — as an alternative provider for task interpretation. Like the Apple Foundation Models idea (#226), this would be free, offline, and private, and it fits the shared LLM layer (src/llm/) as a keyless OpenAI-compatible provider reusing the existing openai-compatible.ts transport — one descriptor + one registry entry, no config.ts/secrets.ts edits. Unlike the Apple path, the Qwen small models have very large context windows, so there is no 4096-token blocker.

This issue is exploratory. A smoke test (below) shows the concept is plausible on speed but not viable at the 2B size as tested due to output-reliability problems. Filing so the findings aren't lost and so a follow-up can test larger sizes.

The models (grounded, current as of mid-2026)

Qwen3.5 (Feb–Mar 2026) small dense models: 0.8B, 2B, 4B, 9B. Native 262K context, Apache 2.0, 201 languages, non-thinking mode by default (thinking opt-in), hybrid Gated-DeltaNet + sparse-MoE architecture. Pullable via Ollama at roughly 1.0 GB (0.8b), 2.7 GB (2b), 3.4 GB (4b), 6.6 GB (9b); MLX variants exist for Apple Silicon. (A newer Qwen3.6 series exists but so far only in larger sizes.)

Sources: Qwen3.5 blog, Qwen3.5-0.8B model card, Ollama qwen3.5 library.

How it would fit Argus

Every local runtime (Ollama, llama.cpp llama-server, LM Studio, vLLM) exposes an OpenAI-compatible endpoint, which our openai-compatible.ts already targets (comment there literally names these runtimes). The clean integration is a small keyless descriptor:

export const localProvider: ProviderDescriptor = {
  name: "local-openai",              // or "ollama"
  defaultModel: "qwen3.5:4b",
  configFields: ["model", "baseUrl", "maxTokens"],   // no apiKeyEnv, no requiresApiKey
  complete: (call) => openaiCompatibleComplete(call, {
    baseUrl: call.baseUrl || "http://localhost:11434/v1",
    tokenParam: "max_tokens",
  }),
};

The real cost is not the code — it is shipping/managing a runtime + weights (~1–3 GB): BYO runtime → download-on-first-use → full bundle in the desktop app.

Smoke test results

Setup: one real local session (a substantive multi-task session, ~22 filtered user messages) run through the exact production pass-1 task-extraction prompt (buildTaskExtractionPrompt), compared head-to-head between the current default (claude -p --model haiku) and qwen3.5:2b via Ollama. Validity was checked against Argus's real parseTaskExtractionOutput. Hardware: Apple Silicon laptop (M2 Pro), 16 GB. This is a smoke test (n=1 session, a few runs), not a benchmark.

Path	Wall-clock	Output accepted by Argus's parser
claude -p --model haiku	~20–42 s	2/2 (clean task lists, 7–8 tasks)
qwen3.5:2b (bare ollama run)	~4–33 s	0/3 (invalid/truncated/prose)
qwen3.5:2b (API, forced JSON schema)	~22–24 s	0/2 (markdown prose, no JSON)

Qwen internals (warm): prefill ~4,200 tok/s, decode ~61 tok/s, model load ~0.3 s. Wall-clock is dominated by how many tokens it generates, not prefill.

Findings:

• Speed is not the blocker. qwen3.5:2b was competitive-to-faster than claude -p. Notably, claude -p is slow (~20–42 s) because it boots the full Claude Code agent harness per call, not because Haiku is slow — a raw Haiku call via the claude-api provider would beat both. (Separate actionable takeaway: moving interpretation off claude -p to claude-api would speed the current default regardless of this issue.)
• Reliability at 2B is the blocker. qwen3.5:2b produced parser-acceptable output in 0/5 attempts. It is an instruction-following failure, not a formatting near-miss: when the transcript content itself contains agentic instructions, the small model gets hijacked and responds as though it were the agent described in the session (proposing next steps, asking clarifying questions) instead of standing outside the transcript to extract tasks. Ollama's structured-output format schema did not enforce JSON on this setup — output came back as markdown prose.

Open questions / next steps

• Re-run the same harness with qwen3.5:4b (and possibly 9B), plus the target schema embedded in the prompt (a few-shot example of the exact shape). The failure at 2B is upstream of JSON coercion, so the real question is whether a larger model holds the meta-instruction. This is the decision point for viability.
• If 4B is reliable: decide distribution strategy (BYO runtime vs download-on-first-use vs full bundle) and how to detect/degrade when no runtime is present.
• Confirm whether structured-output enforcement (format / GBNF grammars) works with a newer Ollama build; it did not take effect as tested.
• Guard against prompt-injection-style hijacking from transcript content regardless of model (stronger role separation, system prompt, output validation + retry).

References

• Related: Add Apple on-device Foundation Models as an LLM provider (on-device task interpretation) #226 (Apple on-device Foundation Models as a provider)
• In-repo analogs: src/llm/providers/openai-compatible.ts, src/llm/providers/openrouter.ts, src/indexing/interpret/task-extraction.ts
• Docs: docs/internals/llm-providers.md, docs/internals/task-interpretation.md