[zhanghaohit]

What

A new endpoint on the sidecar's existing port:

$ curl -s http://<pod-ip>:9090/engine
{"engine_id":"pst-1791943641.29","started_at":"2026-10-14T02:07:21.29Z","source":"process_start_time",
 "restarts":1,"watcher_started_at":"2026-10-14T01:52:10.48Z"}

restarts counts the engine restarts this sidecar has seen since watcher_started_at (its own start); it resets with the sidecar, so engine_id remains the field a client should compare.

503 until the engine has been identified, so "not known yet" is never read as an id.

Why

A client that keeps state tied to one engine process -- a prefill/decode router holding transfer sessions with its workers -- needs to know when that process was replaced. The kubelet restarts the engine container in place: the pod, its IP and its endpoints stay the same, so there is nothing to watch from outside. The sidecar already scrapes the engine every poll, so it can say which instance it is looking at.

Where the identity comes from

source	taken from	started_at	id survives a sidecar restart
process_start_time	vLLM's process_start_time_seconds	the engine's own start time	yes
startup_fingerprint	sglang's startup_time_seconds / startup_cuda_graph_time_seconds (>= 0.5.19)	when the sidecar first saw the instance	yes
observed	neither available; minted by the sidecar, renewed on counter regression	when the sidecar first saw the instance	no

sglang publishes no start timestamp, only per-phase startup durations. They are set once per process and measured to the nanosecond, so they work as a fingerprint. They appear over several scrapes during startup, so new series joining is the same process; only a changed value or a vanished series is a new one. Consumers should compare engine_id, not started_at.

Reading the engine process directly (the way ps does) is not possible: containers in a pod have separate pid namespaces. shareProcessNamespace would allow it but makes the pause container pid 1, which breaks charts whose preStop signals pid 1 and kills everything else it can see.

Behaviour change

A restart seen this way resets the verdict state machine (fresh baseline, warm-up grace period re-armed). Counter regression, the only restart signal so far, misses an engine that was idle (counters 0 before and after) or whose new process outran the old count within one poll.

Testing

• Unit tests with metric lines taken from a real vLLM (v0.29 nightly) and sglang (0.5.19): restart without counter regression (with a control showing the counters alone miss it), idle restart, fingerprint growing across scrapes without a restart, id stable across sidecar restarts, exact response shape.
• smoke_engine_test.sh (added to CI): the real binary against a fake engine, for both sources -- 503 before the first scrape, id changes on engine restart, same id after a sidecar restart.
• Existing tests and smoke tests unchanged and passing.

[zhanghaohit]

Tested 0.1.27-rc1 on a test cluster with real engines, deployed from the published charts (vllm 0.8.12 pd-colocate: prefill + decode + router in one pod; sglang 0.9.14 single engine; A100, a 27B FP8 model). All checks passed:

	vLLM prefill (:9090)	vLLM decode (:9091)	sglang
source	process_start_time	process_start_time	startup_fingerprint
started_at vs kubelet startedAt	-0.74s	-0.44s	+406s (first seen after model load, as documented)
same answer from another pod via pod IP	yes	yes	yes
engine container restarted (SIGTERM to pid 1)	new id, started_at within 0.22s of the new startedAt	new id, within 0.02s	new id
the other engine in the pod during that restart	id unchanged	id unchanged	--
hang-watcher container restarted	id unchanged	id unchanged	id unchanged (started_at moved, as documented for this source)

While a restarted engine is still loading, /engine keeps returning the previous id; the new one appears once the new process serves /metrics.

Script: real engines on the test cluster, run by hand (not part of CI).

[zhanghaohit]

Re-tested with 0.1.27-rc2 (adds restarts and watcher_started_at) on the same real-engine setup. All checks passed again, plus:

	vLLM prefill	vLLM decode	sglang
restarts after one engine container restart	1	1	1
restarts after the sidecar container restarted	0	0	0
watcher_started_at vs kubelet startedAt of the sidecar container	+0.73s	+0.74s	+0.13s
started_at vs kubelet startedAt of the engine container (before / after restart)	+0.04s / -0.85s	-0.44s / -0.53s	first seen after load, as documented

The latest commit (802aab3, review fixes: missing startup series no longer read as a restart, a warning when the identity falls back to observed) is covered by unit and smoke tests; it is not in rc2.

[remotego]

lgtm