feat: support instance incarnation tracking and lease-lost failover.#48
Merged
Merged
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
此拉取请求通过引入实例世代跟踪和租约丢失故障转移机制，显著增强了 xllm-service 的容错能力。核心改动包括在实例和请求中引入 incarnation_id，以及实现 ACTIVE -> LEASE_LOST -> SUSPECT 运行时状态机。这些改动确保了在实例重启、替换或 etcd 租约丢失场景下，系统能够准确区分不同世代的实例，避免旧事件误伤新实例上的请求。健康探测和状态协调线程的引入进一步完善了故障恢复流程。整体而言，代码质量高，逻辑清晰，并发处理得当，有效提升了服务的健壮性。
JimHsiung
force-pushed
the
feat/instance_failover
branch
from
2900f50 to
e47a6c5
Compare
JimHsiung
force-pushed
the
feat/instance_failover
branch
from
e47a6c5 to
3de0c6a
Compare
RobbieLeung
approved these changes
Mar 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
此次改动主要是增加 xllm-service 在“实例重启/替换”和“etcd lease 丢失”场景下的容错链路。核心思路是给实例和请求都引入 incarnation_id，把同名实例的不同世代区分开，避免旧 heartbeat、旧 delete 事件或失败清理误伤新实例上的请求。
实现上，InstanceMgr 新增了 ACTIVE -> LEASE_LOST -> SUSPECT 运行时状态机：收到 etcd delete 后不再立刻摘实例，而是先做一次 /health 探测；探测成功则进入 LEASE_LOST，实例在持续 heartbeat 的情况下仍可继续参与调度；如果后续心跳超时，再转成 SUSPECT，窗口到期后才真正注销。
同时，调度阶段会把请求绑定到具体的 prefill/decode incarnation_id，后续实例失效时按 instance + incarnation 精确清理请求；调度和负载均衡也会跳过不可调度实例。RPC 协议和 master 配置项也同步补了 incarnation_id、探测超时和 LEASE_LOST 心跳超时参数。