Local engine model management: core, gateway, and protocol

[rexlunae]

What & why

Local inference engines (Joshua, llama.cpp, Ollama, LM Studio, exo) now have real model-management backing for the UI: pickers show what is actually available on disk, the engine layer knows which servers are already running on the host, and engine parameters are typed config instead of opaque extra_args.

Changes

Engine config & parameters (rustyclaw-core/src/engines)

• EngineConfig gains typed, optional parameters: context_length, device, huge_pages, mmap, lazy_weights, max_output_tokens, max_concurrency (all defaulted — existing configs parse unchanged). Joshua maps them to its serve flags (--n-ctx, --device, --huge-pages, --mmap, --lazy-weights, --max-output-tokens, --max-concurrency); llama.cpp maps context_length → --ctx-size. Raw extra_args remain the escape hatch and still win on the command line.
• joshua_serve_flags() centralises the flag mapping for both the spawn path and the auto-start service definitions.

Running-server detection

• running_server_cmdlines() / parse_server_cmdlines() (Linux, best-effort) find joshua serve and llama-server processes already on the host — including ones started outside RustyClaw — so status() reports them as running and list_models() marks the models they serve as loaded. This is what lets the UI say which local models are running.
• stop() now receives the engine config and scopes its kill to the engine's own port (no more pkill -f 'joshua serve' nuking every server on the machine). load/unload/start are honest about externally-run servers: report which port they're on, refuse to kill them, and don't spawn a duplicate.

Provider model-list fallback

• provider_models_with_local_fallback() (and a detailed variant) merges the live provider API list with the engine's own model list — for the file-based engines that is a scan of the models directory, so pickers show on-disk models even when the server isn't running — and reports which models are loaded. llama.cpp's list_models gained the same on-disk scan.

Honest failures

• Joshua's spawn now returns a real error when the server never answers ("did not answer on … within 10s — run joshua serve --model … manually") instead of a forever "may still be loading", and reports when joshua isn't installed.

Configurable model-call deadline (rustyclaw-gateway/src/dispatch.rs)

• The per-turn model call used a hard-coded 180s cap that local engines on a loaded machine can exceed while prefilling a large prompt. RUSTYCLAW_MODEL_TIMEOUT_SECS overrides it (default stays 180).

Wire / protocol

• EngineConfigSet client command (persists an engine's full config to config.toml).
• EngineInfoDto carries the engine's full EngineConfig so clients can round-trip it.
• ProviderModelListResult carries loaded model ids (appended last, positional-bincode safe).

Testing

• cargo check --workspace, clippy clean on touched crates.
• Unit tests: engine fallback, cmdline parsing for running servers, Joshua flag mapping (23 engine tests).
• Verified live against a preview gateway + desktop client: on-disk scans, running-server detection (real joshua/llama-server processes on the host), parameters save/round-trip, honest Load/Unload errors, and a real chat reply through a managed joshua server.

---

[rexlunae]

All four Devin review findings are addressed in f0c96a8:

1. llama.cpp remove phantom success — LlamaCppEngine::remove now resolves the model name back to the scanned on-disk path (file stem, possibly inside a per-repo subdirectory) before deleting, and errors with the available list when nothing matches, so the UI can no longer report success while the file stays on disk.
2. Joshua slow-load false failure — spawn_server now checks whether the spawned process is still alive after the health-probe window: a live process reports "still loading", and only an exited process produces the error. Switching to a large model no longer shows a scary error while it loads fine moments later.
3. Missing public-field docs — ProviderModels::models and ::error now carry doc comments per the style guide.
4. Shell injection via device/huge_pages — joshua_serve_flags only emits --device/--huge-pages for the values Joshua actually accepts (auto/cpu/metal/cuda, transparent/2mb/1gb/huge), dropping anything else instead of interpolating it into the shell command. Added a regression test for hostile values.

Verified: cargo check --workspace clean, clippy clean on the touched crates, engine tests 24/24 (incl. the new hostile-value test).

[rexlunae]

Second-round Devin findings addressed in 6e3e115 (all threads resolved):

• Non-Linux start/stop false failures — the 'process exited' hard failure is now Linux-only (process inspection exists there); on macOS/Windows a slow load reports 'may still be loading' instead of a failure, and llama.cpp stop falls back to stopping every llama-server rather than claiming success while one keeps running.
• llama.cpp auto-start stop — the service definition now always passes the resolved --port, so the port-scoped stop can identify auto-started servers.
• Non-Linux warning — running_server_cmdlines no longer emits an unused-variable warning off Linux.
• models_dir quoting — llama.cpp start interpolates the configured models directory via sh_quote.
• Joshua 'already loaded' — Load only claims already-loaded when the configured endpoint actually serves the model; a joshua on another port no longer produces a phantom success.

All review threads across #486/#487/#488 are resolved. CI is re-running (Lint already green).

[devin-ai-integration]

Devin Review found 2 new potential issues.

View 6 additional findings in Devin Review.

[devin-ai-integration]

🔴 Starting a slow-loading local model always reports a bogus failure

The check for whether the just-started server is still alive scans for the port text and then inspects the first character of the match itself instead of the character following it (line.match_indices(&needle) at crates/rustyclaw-core/src/engines/joshua.rs:223), so the check can never succeed and a model that is merely slow to load is declared crashed.

Impact: On Linux, starting a large local model that needs more than ten seconds to become responsive always ends with an error telling the user the process exited, even though the server is running fine and finishes loading moments later.

Why the alive-check can never return true

String::match_indices yields (start_index, matched_str) pairs — the index is the start of the match, not its end. The closure binds it as end and evaluates line[end..].chars().next(), which is therefore the first character of the needle itself, i.e. '1' of "127.0.0.1:<port>". Since '1'.is_ascii_digit() is true, is_none_or(|c| !c.is_ascii_digit()) is false for every match, so server_process_alive returns false unconditionally.

In spawn_server (crates/rustyclaw-core/src/engines/joshua.rs:186-201) the 20×500 ms health-probe loop falls through to the Linux block, server_process_alive(port) reports false, and the function bails with "…its process exited — the model file may be invalid…" even though the process is alive and still prefilling. This defeats the purpose of the new branch, which exists precisely so slow loads are not reported as failures.

The intended index is start + needle.len().

Suggested change

	async fn server_process_alive(port: u16) -> bool {
	let needle = format!("127.0.0.1:{}", port);
	crate::engines::running_server_cmdlines("joshua serve")
	.await
	.iter()
	.any(|line| {
	line.match_indices(&needle).any(|(end, _)| {
	line[end..]
	.chars()
	.next()
	.is_none_or(|c| !c.is_ascii_digit())
	})
	})
	}
	async fn server_process_alive(port: u16) -> bool {
	let needle = format!("127.0.0.1:{}", port);
	crate::engines::running_server_cmdlines("joshua serve")
	.await
	.iter()
	.any(|line| {
	line.match_indices(&needle).any(|(start, m)| {
	line[start + m.len()..]
	.chars()
	.next()
	.is_none_or(|c| !c.is_ascii_digit())
	})
	})
	}

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🟨 Client-supplied engine port is interpolated into pkill/pgrep shell patterns

EngineConfigSet lets a connected client set an arbitrary engine port, which is later interpolated into sh -c command lines used for process matching (pkill -f 'joshua serve .*127.0.0.1:{port}( |$)' at crates/rustyclaw-core/src/engines/joshua.rs:443-446, and the llama.cpp equivalent at crates/rustyclaw-core/src/engines/llamacpp.rs:253-267). The value is a u16, so no shell metacharacters can reach the command line, and validate_engine_config (crates/rustyclaw-core/src/engines/mod.rs:131-136) additionally rejects 0. The residual risk is behavioural rather than injective: the pattern is regex-matched against every process command line, so an attacker-chosen port could be crafted to match unrelated processes and have them killed by the pkill on the stop path.

---

Was this helpful? React with 👍 or 👎 to provide feedback.