Serve the engine's identity and start time on GET /engine

.github/workflows/ci.yml

@@ -37,6 +37,7 @@ jobs:
       # /healthz flip. No Kubernetes needed.
       - run: bash smoke_test.sh
       - run: bash smoke_probe_test.sh
+      - run: bash smoke_engine_test.sh
       - run: docker build -t hang-watcher:ci .
 
   gate:

CHANGELOG.md

@@ -9,6 +9,23 @@ of; the commit history for them was carried over, but the release tags were not.
 ## [Unreleased]
 
 ### Added
+- **`GET /engine`: which engine instance is running and when it started**, for
+  clients that share state with the engine and must notice a restart -- a
+  prefill/decode router, for one. The kubelet restarts the engine container in
+  place, keeping the pod IP, so watching pods or endpoints shows nothing.
+  Returns `{"engine_id", "started_at", "source", "restarts",
+  "watcher_started_at"}` on the `/healthz` port (`restarts` counts the engine
+  restarts this sidecar has seen since it started);
+  `503` before the engine has been identified. The identity is taken from
+  vLLM's `process_start_time_seconds` (exact start time), sglang's `startup_*`
+  gauges (a fingerprint that survives sidecar restarts; `started_at` is then
+  when the sidecar first saw the instance), or, with neither, an id minted by
+  the sidecar that does not survive its restart.
+- A restart detected this way puts the watcher back into its startup state: a
+  fresh baseline and warm-up grace period from the new instance's first
+  scrape. Counter regression, the only signal before, misses an engine that was
+  idle (counters stay 0) or whose new process outran the old count within one
+  poll.
 - CI on every pull request: `gofmt`, `go vet`, `go test`, both smoke tests and
   a docker build, plus a gate for the license text and committed credentials.
 - Dependabot for the GitHub Actions, which are pinned to commit SHAs.

README.md

@@ -200,6 +200,69 @@ Two things matter:
 One sidecar and one probe cover both; LWS only adds the switch that turns a
 single-pod restart into a group recreation.
 
+## Engine identity: `GET /engine`
+
+A client that keeps state tied to one engine process -- a prefill/decode router
+holding KV-transfer sessions with its workers, for example -- needs to know when
+that process was replaced. Watching the pod does not tell it: the kubelet
+restarts the engine container in place, and the pod, its IP and its endpoints
+all stay the same. The sidecar already scrapes the engine every poll, so it
+reports which instance it is looking at, on the same port as `/healthz`:
+
+```console
+$ curl -s http://<pod-ip>:9090/engine
+{"engine_id":"pst-1791943641.29","started_at":"2026-10-14T02:07:21.29Z","source":"process_start_time",
+ "restarts":1,"watcher_started_at":"2026-10-14T01:52:10.48Z"}
+```
+
+Before the engine has been identified it answers `503`, so "not known yet" can
+never be mistaken for an id.
+
+- `engine_id` changes exactly when the engine process is replaced. **Compare
+  this**, not `started_at`.
+- `started_at` is the engine's start time (UTC), as well as it can be known.
+- `restarts` is how many times this sidecar has seen the engine replaced since
+  `watcher_started_at`, the sidecar's own start. It starts again from 0 when the
+  sidecar restarts, so it reads well for a person; a client deciding whether
+  to act should compare `engine_id`.
+- `source` says where `engine_id` and `started_at` came from, and so how far
+  each can be trusted:
+
+| `source` | taken from | `started_at` | `engine_id` survives a sidecar restart |
+|---|---|---|---|
+| `process_start_time` | `process_start_time_seconds` (vLLM) | the engine's own start time | yes |
+| `startup_fingerprint` | sglang's `startup_time_seconds` / `startup_cuda_graph_time_seconds` (sglang >= 0.5.19) | when this sidecar first saw the instance | yes |
+| `observed` | neither available; minted by the sidecar, renewed when token counters go backwards | when this sidecar first saw the instance | **no** |
+
+sglang publishes no start timestamp, only how long each startup phase took.
+Those durations are set once per process and measured to the nanosecond, so two
+processes practically never report the same set, and together they serve as a
+fingerprint -- but they cannot say *when* the process started. A scrape need
+not carry all of them (some phases are reported late, and a body cut off at the
+read limit loses the tail), so only a series seen before with a different value
+counts as a new process; series joining or missing do not.
+
+With `observed`, a restarted sidecar reports a new id for the same engine, and
+an engine restarted while idle (counters at 0 before and after) goes unnoticed;
+treat it as advisory. The sidecar logs a warning when it falls back to it.
+
+**vLLM with `--api-server-count` > 1 ends up on `observed`.** vLLM then runs
+its metrics in multiprocess mode (it sets `PROMETHEUS_MULTIPROC_DIR` itself,
+and so does any deployment that sets it by hand), and that registry has no
+process collector, so `process_start_time_seconds` is not exported.
+
+Why not read the engine process the way `ps` does: containers in a pod have
+separate pid namespaces, so the sidecar's `/proc` holds only its own processes.
+`shareProcessNamespace` would change that, but it makes the pause container
+pid 1, and engine charts commonly rely on the engine being pid 1 of its
+container -- a preStop hook that signals pid 1 and then kills every other
+process it can see would take down the whole pod.
+
+A restart detected this way puts the watcher back into its startup state (a
+fresh baseline and warm-up grace period from the new instance's first scrape).
+Counter regression, the only signal before, misses an engine that was idle or
+whose new process outran the old count within one poll.
+
 ## Build and test
 
 ```bash
@@ -208,7 +271,7 @@ CGO_ENABLED=0 go build -o hang-watcher .
 docker build -t hang-watcher:dev .
 ```
 
-`smoke_test.sh` and `smoke_probe_test.sh` run the real binary against a fake
+`smoke_test.sh`, `smoke_probe_test.sh` and `smoke_engine_test.sh` run the real binary against a fake
 engine and assert end-to-end behaviour, including the case a unit test cannot
 reach: that a *single* probe failure does not flip `/healthz` to 503.
 

engine.go

@@ -0,0 +1,236 @@
+// Engine instance identity: which engine process is the sidecar looking at,
+// and when did it start. Served on GET /engine so that something sharing state
+// with the engine -- a prefill/decode router, for one -- can tell that the
+// engine behind it was replaced and act on it.
+//
+// Why the sidecar and not the engine: the kubelet restarts the engine
+// container in place, keeping the pod and its IP, so a client that only
+// watches pod events or endpoints sees nothing. And the engines do not agree
+// on how to say it: vLLM exposes process_start_time_seconds, sglang does not.
+// One endpoint with one meaning saves every consumer from learning both.
+//
+// Why /metrics and not the process itself: containers in a pod have separate
+// pid namespaces, so the sidecar cannot see the engine's processes (its /proc
+// holds only its own). shareProcessNamespace would change that, but it makes
+// the pause container pid 1, and engine charts commonly rely on the engine
+// being pid 1 of its container (a preStop that signals pid 1 and then kills
+// every other process it can see would take down the whole pod).
+//
+// The identity comes from the engine's own /metrics, strongest first:
+//
+//  1. process_start_time_seconds (vLLM, and anything using the default
+//     Prometheus process collector) -- an exact start time, so the id is
+//     derived from it and started_at is exact. Not there when vLLM runs its
+//     metrics in multiprocess mode (PROMETHEUS_MULTIPROC_DIR, which it sets
+//     itself for --api-server-count > 1): that registry has no process
+//     collector, and the identity falls through to observed.
+//  2. sglang's startup_* gauges (startup_time_seconds,
+//     startup_cuda_graph_time_seconds, sglang >= 0.5.19). They are durations,
+//     not timestamps, but they are set once per process and measured to the
+//     nanosecond, so two processes practically never report the same set.
+//     That makes them a fingerprint: the id is stable across sidecar restarts,
+//     but started_at is only the time this sidecar first saw the instance.
+//  3. Neither present: the id is minted by the sidecar on first sight and
+//     re-minted when the token counters go backwards. This one does NOT
+//     survive a sidecar restart -- a restarted sidecar reports a new id for
+//     the same engine -- so source tells consumers how far to trust it.
+//
+// Consumers should compare engine_id, not started_at: for sources 2 and 3
+// started_at is an observation time and moves when the sidecar restarts.
+package main
+
+import (
+	"encoding/json"
+	"hash/fnv"
+	"math"
+	"net/http"
+	"sort"
+	"strconv"
+	"time"
+)
+
+const processStartMetric = "process_start_time_seconds"
+
+// startupMetrics: sglang gauges that are written once, at startup.
+var startupMetrics = map[string]bool{
+	"sglang:startup_time_seconds":            true,
+	"sglang:startup_cuda_graph_time_seconds": true,
+}
+
+const (
+	idProcessStart = "process_start_time"
+	idStartup      = "startup_fingerprint"
+	idObserved     = "observed"
+)
+
+type engineState struct {
+	id        string
+	idSource  string
+	startedAt time.Time
+	restarts  int // restarts seen by this sidecar since it started
+
+	// The startup fingerprint is kept as series -> value rather than as a hash,
+	// because a scrape need not carry every series: during startup some phases
+	// have not been reported yet, and a /metrics body cut off at the read limit
+	// loses whatever came last. So only a series seen before with a DIFFERENT
+	// value is a new process; series joining or missing are not.
+	startup map[string]float64
+}
+
+// noteEngine folds one successful scrape into the identity and reports whether
+// the engine was replaced since the last one. The first identity ever seen is
+// not a restart, and neither is moving to a stronger source (an sglang engine
+// whose startup gauges only show up after its first scrape).
+func (w *watcher) noteEngine(now time.Time, m metricsSnap, metricsOK bool) bool {
+	w.mu.Lock()
+	defer w.mu.Unlock()
+	e := &w.eng
+	if !metricsOK {
+		return false
+	}
+	switch {
+	case m.startTime > 0:
+		id := "pst-" + strconv.FormatFloat(m.startTime, 'f', -1, 64)
+		if e.idSource == idProcessStart && e.id == id {
+			return false
+		}
+		restarted := e.idSource == idProcessStart
+		e.id, e.idSource, e.startup = id, idProcessStart, nil
+		// to the millisecond: the float carries no more, and finer digits are noise
+		e.startedAt = time.UnixMilli(int64(math.Round(m.startTime * 1000))).UTC()
+		if restarted {
+			e.restarts++
+		}
+		return restarted
+	case len(m.startup) > 0:
+		if e.idSource == idStartup && sameProcess(e.startup, m.startup) {
+			for k, v := range m.startup {
+				e.startup[k] = v // phases reported late join the set; the id stays put
+			}
+			return false
+		}
+		restarted := e.idSource == idStartup
+		e.id, e.idSource, e.startup = "sfp-"+fingerprint(m.startup), idStartup, map[string]float64{}
+		for k, v := range m.startup {
+			e.startup[k] = v
+		}
+		if restarted || e.startedAt.IsZero() {
+			e.startedAt = now
+		}
+		if restarted {
+			e.restarts++
+		}
+		return restarted
+	default:
+		if e.id == "" {
+			e.id, e.idSource, e.startedAt = observedID(now, 0), idObserved, now
+		}
+		return false
+	}
+}
+
+// noteRegress handles token counters going backwards. That is only the
+// identity signal when the engine offers nothing better; with a real identity
+// noteEngine has already decided, and a regression without an identity change
+// is not a second restart.
+func (w *watcher) noteRegress(now time.Time) {
+	w.mu.Lock()
+	defer w.mu.Unlock()
+	e := &w.eng
+	if e.idSource != idObserved {
+		return
+	}
+	e.restarts++
+	e.id, e.startedAt = observedID(now, e.restarts), now
+}
+
+// sameProcess: no series present in both has changed value. A series missing
+// from cur says nothing (not reported yet, or cut off) and is not a restart.
+func sameProcess(old, cur map[string]float64) bool {
+	for k, v := range cur {
+		if ov, ok := old[k]; ok && ov != v {
+			return false
+		}
+	}
+	return true
+}
+
+// fingerprint hashes the startup series. Computed when the identity is first
+// taken, so a series joining later does not change the id.
+func fingerprint(m map[string]float64) string {
+	keys := make([]string, 0, len(m))
+	for k := range m {
+		keys = append(keys, k)
+	}
+	sort.Strings(keys)
+	h := fnv.New64a()
+	for _, k := range keys {
+		_, _ = h.Write([]byte(k + "=" + strconv.FormatFloat(m[k], 'g', -1, 64) + "\n"))
+	}
+	return strconv.FormatUint(h.Sum64(), 16)
+}
+
+func observedID(now time.Time, n int) string {
+	return "obs-" + strconv.FormatInt(now.UnixNano(), 36) + "-" + strconv.Itoa(n)
+}
+
+// engineInfo is the /engine response body. source says how far started_at can
+// be trusted: process_start_time is the engine's real start;
+// startup_fingerprint and observed are fallbacks, where started_at is when this
+// sidecar first saw the instance.
+//
+// restarts counts engine restarts this sidecar has seen since
+// watcher_started_at. It starts from 0 whenever the sidecar does, so it is a
+// convenience for people reading the endpoint; a client deciding whether the
+// engine was replaced should still compare engine_id.
+type engineInfo struct {
+	EngineID         string `json:"engine_id"`
+	StartedAt        string `json:"started_at"`
+	Source           string `json:"source"`
+	Restarts         int    `json:"restarts"`
+	WatcherStartedAt string `json:"watcher_started_at"`
+}
+
+func (w *watcher) Engine() (engineInfo, bool) {
+	w.mu.RLock()
+	defer w.mu.RUnlock()
+	e := w.eng
+	return engineInfo{
+		EngineID: e.id, StartedAt: rfc(e.startedAt), Source: e.idSource,
+		Restarts: e.restarts, WatcherStartedAt: rfc(w.bornAt),
+	}, e.id != ""
+}
+
+// engineRestarts: restarts seen by this sidecar since it started.
+func (w *watcher) engineRestarts() int {
+	w.mu.RLock()
+	defer w.mu.RUnlock()
+	return w.eng.restarts
+}
+
+func rfc(t time.Time) string {
+	if t.IsZero() {
+		return ""
+	}
+	return t.UTC().Format(time.RFC3339Nano)
+}
+
+// engineHandler serves GET /engine: 200 with the identity once one is known,
+// 503 before the first sighting, so "not known yet" is never read as an id.
+func engineHandler(w *watcher) http.HandlerFunc {
+	return func(rw http.ResponseWriter, r *http.Request) {
+		if r.Method != http.MethodGet && r.Method != http.MethodHead {
+			rw.Header().Set("Allow", "GET, HEAD")
+			http.Error(rw, "method not allowed", http.StatusMethodNotAllowed)
+			return
+		}
+		info, known := w.Engine()
+		rw.Header().Set("Cache-Control", "no-store")
+		if !known {
+			http.Error(rw, "engine not identified yet", http.StatusServiceUnavailable)
+			return
+		}
+		rw.Header().Set("Content-Type", "application/json")
+		_ = json.NewEncoder(rw).Encode(info)
+	}
+}