feat(lm): sglang LM block — preflight the hardware, repair the runtime, serve from a named venv by se-jo-ma · Pull Request #199 · KrakenNet/stargraph

se-jo-ma · 2026-08-18T14:01:04Z

A graph can now carry its own LM endpoint. examples/sglang-qa.yaml is the
whole feature in twelve lines: an lm: block naming an LFM model, and a
kind: dspy node bound to it. No --lm-url/--lm-model, no operator setup
beyond a GPU.

lm:
  provider: sglang
  model: LiquidAI/LFM2.5-1.2B-Instruct
  port: 30000
  startup_timeout_s: 600

stargraph run examples/sglang-qa.yaml --inputs question="what routes stargraph?"

What it does before the server starts

lm/hardware.py preflights the spawn the way LM Studio does — detect, then
match the artifacts to what was detected:

Hardware from the vendor tools (nvidia-smi / rocm-smi / xpu-smi /
npu-smi), never from torch. A CPU-only wheel on a two-GPU box reports no
CUDA, so asking torch what hardware exists gets the wrong answer exactly when
it matters.
Runtime probed inside the interpreter that would be spawned, not this one.
Weights fetched before the startup clock starts, so a 600s timeout isn't
spent on a download.
Model format validated (GGUF refused with the reason), never rewritten.

Three rules hold throughout: nothing installs without --install-runtime,
kernel drivers are only ever reported, and the attach branch preflights nothing
— that server isn't ours.

The bug that motivated most of this

--install-runtime installed sglang, then failed with:

runtime install completed but the runtime still cannot serve
(torch 2.11.0+cpu (cpu build), 0 device(s) visible)

2.11.0+cpu satisfies torch==2.11.0 — PEP 440 local versions are ignored in
that comparison — so installing sglang over a CPU torch leaves the CPU torch
exactly where it is, and the resolver is right to. Repair now runs in rounds:
sglang first, then a --force-reinstall of torch off the CUDA index at the pin
sglang's own metadata resolved to (not a number copied from the docs, which
goes stale). A round that changes nothing stops instead of looping.

Platforms sglang doesn't ship as a plain wheel — ROCm, XPU, Ascend, Apple Metal
— get their doc page rather than a pip command that would fail.

`--sglang-python`

Points the whole spawn (preflight, weight fetch, launch) at another
interpreter, so the venv running stargraph never has to become the venv serving
the model. A venv directory is accepted and mapped to its bin/python.

Deliberately not a YAML key: an interpreter is argv into a subprocess, the
same class of operator-only value as args and a non-loopback host, all of
which _check_graph_declared still refuses from a graph.

The interpreter is used without .resolve() — a venv's bin/python is a
symlink into the base install, and following it lands in the base install's
bin/, silently missing every console script the venv has. That is not
theoretical: flashinfer JIT-compiles attention kernels at startup and shells
out to ninja, which lives in the venv's bin/. Without it the server died
with code -9 and a FileNotFoundError buried in the child log. _child_env
now gives the child the PATH an activation would have.

Run counters

The summary's "llm calls" was fed by ToolCallEvent, so it counted tool calls
and read 0 llm calls for a graph whose only work was an LM call. It now comes
from the DSPy client's history.

DSPy's disk cache stays on — it is what makes a re-run cheap — but a cached run
reached no server and must not read like one that did:

✓ done in 72ms  (0 steps, 1 llm calls, 1 cached)

That line is from a run pointed at a dead port. Before this change the same run
printed 0 llm calls, indistinguishable from a GPU-served one — which is how a
cached answer got mistaken for a live one while verifying this very example.
--summary-json carries llm_call_count, llm_cache_hits and tool_call_count.

Also here

chore: regenerate uv.lock for the release version bump — release-please bumps
pyproject.toml but not uv.lock, so the lock's stargraph entry sat two
releases behind. A stale project version leaves the lock unresolved-clean, so
every uv run rewrote it and pre-commit failed on the modified lockfile while
ruff reported no findings. No package moved; uv lock --check is clean.

Verification

Run end-to-end on an L40S, not just in tests: preflight → torch repair →
weights → server up → answer. The hermetic path is an OpenAI-shaped stub server
whose hit log is asserted, so a cache hit can't pass for a served answer.
Full suite green: 2075 passed, 39 skipped, 2 xfailed.

…block `stargraph run` now resolves an LM endpoint before the first node executes: it probes `/v1/models` and attaches when a server already serves the requested model (leaving it running — it is not ours), otherwise it spawns `python -m sglang.launch_server`, waits for the endpoint to answer, and terminates it at run end. It never imports sglang and never kills a server it did not start. The endpoint is expressible two ways: `--sglang-model/--sglang-port/ --sglang-arg` flags, or a top-level `lm:` block in the authoring format (lowered to the `SGLangServer` IR model). The block is not part of the structural graph hash — like `--lm-url`, an endpoint is an environment binding, not graph topology. A graph-declared block is partly untrusted: `_check_graph_declared` refuses YAML-supplied `args` (argv into a subprocess) and a non-loopback `host` (it would receive `--lm-key` and every prompt) unless the operator re-states them as flags. This is the only path by which a graph reaches `subprocess` — every process-spawning std tool sits behind the default-deny capability gate. Adds `LMServerError` for the attach/spawn failure modes: a port serving a different model, a launch that exits before answering, and a startup that exceeds `startup_timeout_s`. Signed-off-by: Sean Mauk <seanmauk@krakennetworks.com>

Adds the runnable example for the `lm:` block: the graph names the model it wants and `stargraph run` resolves the endpoint before the first node, instead of the caller supplying --lm-url/--lm-model. Writing it surfaced a regression in the previous commit. `_configure_lm` used to run before `build_node_registry`; folding it into the `_lm_endpoint` context manager moved it *after*, and `kind: dspy` validates that an LM is configured while the node is being constructed, not when it runs. Every `kind: dspy` graph was therefore unrunnable -- with a declared `lm:` block *and* with --lm-url/--lm-model, which is a regression of shipped behaviour. The endpoint context now opens before the registry is built and still spans the whole run, so a spawned server is torn down only once the loop is done or has raised. The existing sglang tests missed it because none of them build a registry containing a dspy node; the graphs are all `kind: echo`. `test_the_lm_is_configured_before_the_node_registry_is_built` pins the call order directly rather than relying on the one example that happens to use a dspy node. The example's golden test starts a loopback OpenAI-compatible stub server and re-points the declared block with `--sglang-port`, which drives the production *attach* branch -- a server already serving the requested model is used as-is. SGLang is GPU-only, so the spawn branch stays covered by tests/integration/test_lm_sglang_spawn.py; no engine code is monkeypatched here, and the assertion on the stub's exact answer proves the resolved base URL plus model really configured the LM the node ran against. Signed-off-by: Sean Mauk <seanmauk@krakennetworks.com>

Spawning an SGLang server only works when three separate things line up: the machine has an accelerator, the interpreter we spawn has an sglang build *for that accelerator*, and the weights are on disk. Left to itself sglang reports all three as the same opaque subprocess death, part-way through a startup timeout. `stargraph.lm.hardware` checks each one before the spawn: * Hardware comes from the vendor tools (`nvidia-smi`, `rocm-smi`, `xpu-smi`, `npu-smi`), never from torch. `torch.cuda.is_available()` answers "was this torch built with CUDA", not "does this box have a GPU" -- a CPU-only wheel on a two-L40S machine says False, which is exactly the case that needs an install command. * The runtime is probed inside the interpreter that would be spawned, and a mismatch is reported with the command for that platform. Install commands are copied verbatim from the SGLang install docs, including the CUDA 12.9 wheel set for drivers older than r580. `--install-runtime` runs them; without it, nothing is installed and the run stops. Kernel drivers are only ever reported. ROCm, XPU, Ascend NPU and Apple Metal ship as platform builds rather than plain wheels, so those get their doc page instead of a pip line that would fail. * Weights are fetched before the server starts, so `startup_timeout_s` measures server boot rather than racing a multi-gigabyte download. * The model format is validated, not rewritten: a GGUF repo is refused with the servable repo named (that is llama.cpp's format), and an FP8 checkpoint on pre-sm_89 hardware warns. A graph runs the weights it declares -- replay depends on it. The attach branch preflights nothing: that server is already up, and it is not ours to diagnose. Also switches the example to LiquidAI/LFM2.5-1.2B-Instruct. Signed-off-by: Sean Mauk <seanmauk@krakennetworks.com>

…over it `--install-runtime` installed sglang and then failed its own verify with "runtime install completed but the runtime still cannot serve (torch 2.11.0+cpu)". The plan was wrong, not the check. Installing sglang cannot fix a CPU torch: `2.11.0+cpu` satisfies sglang's `torch==2.11.0` pin, because PEP 440 ignores the local version segment. The resolver is satisfied, the CPU wheel stays, and the box's GPUs stay invisible to that interpreter -- which is exactly the state a venv gets into when something pinned torch to the PyTorch CPU index (this repo does, deliberately, to keep CI wheels small). The plan now distinguishes the two failures: * sglang missing -> install sglang, and stop there. Its torch pin is not readable until it lands. * sglang present, torch built for the wrong accelerator -> `--force-reinstall` torch/torchaudio/torchvision off the CUDA index, pinned to the versions the *installed* sglang requires (read from its metadata, not hard-coded here -- sglang moves that pin every release). * sglang present, CUDA torch, zero devices, current driver -> not an install problem at all. Reported, with the container/driver cause named, instead of reinstalling wheels that are already correct. Repair therefore runs in rounds (install, re-probe, re-plan) with a no-progress guard: a round whose plan repeats stops the loop and hands over the commands that did not take, rather than retrying them. Index selection follows the driver as before: cu130 on r580+, the cu129 wheel set plus sglang-kernel/sgl-deep-gemm below it. Signed-off-by: Sean Mauk <seanmauk@krakennetworks.com>

stargraph spawns `python -m sglang.launch_server` from its own interpreter, which is the wrong one whenever sglang lives somewhere else -- the common case, since a venv that runs stargraph may deliberately pin a CPU torch (this repo does) while the venv that serves has the CUDA stack. `--sglang-python` moves the whole spawn: preflight, weight fetch and launch all target the named interpreter, so the environment stargraph runs in never has to become the environment sglang serves from. A venv directory is accepted and resolved to its `bin/python`, and a path that is not an executable interpreter is refused at the flag rather than as an ENOENT from a subprocess after the graph has compiled. Flag only, deliberately: naming the interpreter to execute is argv into a subprocess, the same operator-only class as `lm.args` and a non-loopback `lm.host` that `_check_graph_declared` already refuses. A graph that could choose the interpreter would be choosing what code runs. Signed-off-by: Sean Mauk <seanmauk@krakennetworks.com>

A launched sglang died mid-startup with `sglang exited with code -9`, which reads like an OOM kill and is not one. The child's log had the real cause: FileNotFoundError: [Errno 2] No such file or directory: 'ninja' ... flashinfer/jit/cpp_ext.py, in run_ninja Launching an interpreter by absolute path runs its packages but not its scripts -- nothing puts that venv's `bin/` on PATH, which activation (or `uv run`) would have done. sglang depends on `ninja` and flashinfer shells out to it to JIT-compile attention kernels during startup, so the console script is installed and unreachable at the same time. The failure surfaces as a signal from a grandchild process, minutes into a run. The child now inherits PATH with the interpreter's directory prepended, which is what activation does and nothing more. Already-present entries are not duplicated. Found by running the example end-to-end in a throwaway venv on a real GPU box; caught by a test that spawns a child and reads back the PATH it saw, because asserting on the helper alone left `env=` free to be deleted. Signed-off-by: Sean Mauk <seanmauk@krakennetworks.com>

… symlink The previous commit prepended the wrong directory. A venv's `bin/python` is a symlink into the base install, so `.resolve().parent` lands in `~/.local/share/uv/python/.../bin` -- where none of the venv's console scripts are. The end-to-end run failed identically after the "fix": ninja sat in the venv's bin, unreachable. Paths are now made absolute and never resolved, in both places that touch an interpreter path: the spawn's PATH and `--sglang-python` (which would otherwise hand sglang an interpreter outside the venv the operator named, with none of its packages). The tests missed this because both sides of the assertion resolved. They now build a venv-shaped symlink and assert the venv's own bin, which fails against `.resolve()`. Signed-off-by: Sean Mauk <seanmauk@krakennetworks.com>

… out of it The attach test asserted only that the stub's answer landed in state. DSPy caches completions on disk across processes (`~/.dspy_cache`, keyed by model id + prompt + params), so that assertion could be satisfied with no server involved at all -- and worse, the test writes an entry that a later *real* run of the same example, with the same model and question, is served instead of calling the GPU. That is exactly what happened while verifying this branch: a run against a live sglang server on two L40S returned the stub's sentence and reported `0 llm calls`. The stub now appends to a hits file on every completion request, the test asserts it was reached, and the fixture disables DSPy's disk and memory caches so the run under test cannot be answered from -- or poison -- a cache shared with the developer's own runs. Signed-off-by: Sean Mauk <seanmauk@krakennetworks.com>

The run summary's "llm calls" number was fed by ToolCallEvent, so it counted tool calls and reported zero for a graph whose only work was an LM call. A `kind: dspy` node calls its LM directly and publishes nothing on the bus, so the count cannot come from there: it now comes from the DSPy client's own history, read at summary time. DSPy's disk cache stays enabled -- it is what makes a re-run cheap -- but a cached run reached no server and must not read like one that did. The history says which completions came back with `cache_hit`, so the summary names them: done in 72ms (0 steps, 1 llm calls, 1 cached) That line is from a run whose endpoint was a dead port. Before this change the same run printed "0 llm calls" and was indistinguishable from a GPU-served one -- which is exactly how a cached answer was mistaken for a live one while verifying the sglang example. The tool counter keeps its meaning under its own name (`tool_call_count`, JSON only); `llm_call_count` and the new `llm_cache_hits` join it in --summary-json. Signed-off-by: Sean Mauk <seanmauk@krakennetworks.com>

release-please bumps `pyproject.toml`, `__init__.py`, `openapi.json` and the manifest, but not `uv.lock`, so the lock's own `stargraph` entry sat two releases behind (0.5.3 against a 0.7.0 project). A stale project version leaves the lock unresolved-clean, so every `uv run` re-resolved and rewrote it -- and since the pre-commit ruff and pyright hooks shell through `uv run`, pre-commit failed on the modified lockfile while ruff itself reported no findings. `UV_FROZEN=1` worked around it one commit at a time. Regenerated with plain `uv lock`: no package moved. The diff is the project version plus the marker annotations current uv writes for dependencies it can already resolve (`more-itertools` under keyring, the torch/torchvision split); no url or hash changed. `uv lock --check` is clean and `uv sync --locked` audits the existing environment unchanged. The next release bump will drift it again until release-please regenerates the lock itself. Signed-off-by: Sean Mauk <seanmauk@krakennetworks.com>

+    with pytest.raises(RuntimeError, match="boom"), sg.sglang_server(_spec()):
+        raise RuntimeError("boom")
+
+    assert terminated == ["proc"]


+    directly, so a future reshuffle of this function fails here rather than in
+    the one example that happens to use a dspy node.
+    """
+    import stargraph.cli.run as run_mod


+import tempfile
+import time
+from pathlib import Path
+from typing import TYPE_CHECKING, Any, cast


changelog-check guards src/stargraph/ir/ and src/stargraph/schemas/; this branch adds SGLangServer + IRDocument.lm and regenerates both schemas, so the gate wants the entry that says what changed and what it means for the hash. Signed-off-by: Sean Mauk <seanmauk@krakennetworks.com>

krakennetworks added 10 commits August 18, 2026 01:22

github-code-quality Bot found potential problems Aug 18, 2026

View reviewed changes

krakennetworks approved these changes Aug 18, 2026

View reviewed changes

krakennetworks enabled auto-merge (squash) August 18, 2026 14:07

krakennetworks merged commit 92eaaf0 into main Aug 18, 2026
17 checks passed

se-jo-ma mentioned this pull request Aug 18, 2026

chore(main): release 0.8.0 #200

Merged

Navigation Menu

Uh oh!

feat(lm): sglang LM block — preflight the hardware, repair the runtime, serve from a named venv - #199

feat(lm): sglang LM block — preflight the hardware, repair the runtime, serve from a named venv#199
krakennetworks merged 11 commits into
mainfrom
feat/sglang-lm-block

se-jo-ma commented Aug 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

se-jo-ma commented Aug 18, 2026

What it does before the server starts

The bug that motivated most of this

--sglang-python

Run counters

Also here

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

`--sglang-python`