Skip to content

feat(server): expose Canary relationships without write guards - #39

Merged
aceforeverd merged 12 commits into
modelsphere:masterfrom
primowenjoy:feat/canary-arm-ownership
Nov 5, 2026
Merged

aceforeverd merged 12 commits into
modelsphere:masterfrom
primowenjoy:feat/canary-arm-ownership

Conversation

@primowenjoy

@primowenjoy primowenjoy commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Swiss reports the live Kubernetes relationship of each LLMService while service operations follow normal backend rules. Console owns Canary-specific action protection.

Behavior

  • List/status exposes canaryState: standalone, referenced, or owned, alongside the service UID and managedBy / assignedTo parent identities.
  • Derive relationships from Canary ownership labels / ownerReferences and the assignment annotation. Ownership takes precedence; incomplete or malformed metadata remains associated with verified: false rather than becoming standalone. Releasing an assignment returns the service to standalone without deleting it.
  • Remove Canary-specific guards from install, apply, rollback, scaling, uninstall and SLO access, including adaptive concurrency field restrictions. Existing Helm fallback behavior is preserved.
  • reuseSource: true with fromRelease and namespace composes from a live LLMService plan without resolving a catalog. It accepts arbitrary form overrides and edit values, retaining source, chart and the other layers. This is general plan composition with no Canary assignment, baseline or adaptive field whitelist.
  • Preserve ordinary backend checks, locking, UID/resourceVersion delete preconditions, conflict recovery and state restoration for rejected deletes.
  • Describe the API in docs/openapi.yaml; no README, architecture or separate design-document changes.

Validation

  • Go 1.26.0 full go test -p 2 ./... and go vet -p 2 ./... passed using the CI-pinned unit-test catalog.
  • Regression tests cover live standalone/referenced/owned transitions, ownership precedence, incomplete metadata, unrestricted non-adaptive live-plan overrides, null values, ordinary writes and deletion conflict recovery.
  • OpenAPI route/reference tests, Redocly 2.72.1 lint and page build passed.
  • Public Console: typecheck, 507 tests, production and standalone Swiss builds passed.
  • Integrated Console/plugin: typecheck, 620 tests and production build passed. Canary plugin Go tests and vet passed.
  • Latest GitHub e2e, Go CI OpenAPI build and publish passed for 7be48b9.
  • This revision does not change the shared test deployment.

Coordinate Console #108 and Canary plugin #5. UI adaptive upgrades now use reuseSource; backend and frontend should be updated together. An independently enabled Kubernetes admission webhook still applies.

Release note

Swiss exposes standalone, referenced and owned Canary relationships and accepts normal service operations without Canary-specific or adaptive concurrency guards. General live-plan composition uses reuseSource; Console controls allowed experiment-service actions.

@huskcass huskcass left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blockers. go test -p 2 ./... passes locally at 30a42c7; go and gate are green, e2e was still pending when I looked.

Medium:

  • Uninstall (backend_llm.go, DeleteObserved): the resourceVersion precondition fails whenever the operator writes status between the Get and the Delete, which happens often on a service that's scaling or rolling. That conflict then goes through finishUninstallFailed, so the plan is marked failed and a failed uninstall is recorded even though nothing was deleted. On a conflict, restore the plan (as the NotFound branch does) and either re-Get, re-check ownership and retry once, or keep only the UID precondition and rely on the ownership re-check.
  • Failing closed now applies to helm too: a discovery error (cached for crdServedTTL, 30s) or a Forbidden on LLMServices blocks every install, apply, rollback, uninstall and SLO call, even for helm releases on clusters that never run Canary. That reverses the old "invisible CRD behaves like unserved" rule. The bundled chart always grants llmservices, so this mostly affects custom RBAC and API blips, but it should be a deliberate choice, noted in the PR or the README.

Low:

  • handleSLO runs refuseCanaryWrite for GET as well, so SLO reads for helm releases now 502 when LLMService access is forbidden. That contradicts "preserving read fallback". Only refuse arm reads when the object is actually seen, and skip the fail-closed branch for GET.
  • The ownership check in rollback, uninstall and SLO runs outside the apply lock, so an adoption between the check and the write isn't caught (apply is fine because it checks inside applyLocked). This is narrow, but worth a comment.
  • If the label-only fallback finds no canary-namespace label, it returns Namespace: "", and the 409 message reads LLMCanary /demo. Fall back to the object's namespace.

Verdict: the approach looks right. Fix the uninstall conflict handling before merge, and confirm the fail-closed behaviour for helm is intended.

— Review Golden

@primowenjoy primowenjoy changed the title feat(server): protect Canary-managed model services feat(server): protect Canary-owned and assigned model services Nov 2, 2026
@primowenjoy primowenjoy changed the title feat(server): protect Canary-owned and assigned model services feat(server): protect Canary services and allow adaptive-only upgrades Nov 4, 2026
@primowenjoy

Copy link
Copy Markdown
Contributor Author

让 Swiss 识别并保护 Canary 实验服务,同时允许实验中的已有服务单独调整自适应并发参数

为什么干?

从 PR 描述和代码看,要解决两个问题:

  • 避免破坏实验配置。 Canary 会创建实验服务,也会把已有服务指定为实验组。如果 Swiss 仍按普通服务执行升级、扩缩容、回滚或卸载,就可能改变实验条件,甚至删除实验服务。
  • 给并发调参留一个受控入口。 全面禁止修改又太严格:希望在保留模型、镜像、Chart、资源等配置的前提下,调整 adaptive concurrency,而不必重建实验。

干了什么?

  1. 识别归属,保持可见。 列表和状态接口增加 Kubernetes UID、managedBy、assignedTo,区分 Canary 创建的子服务与被实验征用的已有服务。归属信息缺失、损坏或不一致时,仍按受保护对象处理;合法的 production 角色关联保持正常操作。
  2. 在服务端拦截危险操作。 对受保护服务,安装覆盖、普通升级、回滚、扩缩容、SLO 修改和卸载等操作会被拒绝,并返回实验归属。即使走 Helm 后端,也需要检查;因权限或 API 故障无法确认归属时,拒绝写入,保留读取回退行为。
  3. 新增“只改自适应并发”的升级路径。 POST /api/plans 支持 adaptiveOnly: true,只允许已验证的 existing-service arm 修改 modelRoute.nginx.values 下的 adaptive_cc、adaptive_cc_*。直接基于服务当前配置生成计划,不重新解析 catalog,其他配置保持固定。实际 apply 时再次检查归属、比较配置,并要求 Canary controller 已建立基线,防止绕过限制。升级约定 (https://github.com/modelsphere/swiss/blob/11e4a87cdef25310f80e09a31b389fb044103337/docs/design-adaptive-canary-upgrades.md)
  4. 补齐卸载并发保护和回归测试。 删除携带 UID/resourceVersion 条件;发生冲突时重新读取、检查身份和归属,再重试一次。拒绝删除时恢复计划状态,避免误记卸载失败。

这套能力需要配合 Console、canaryd 和 Operator 的关联 PR;旧 controller 没有基线时仍会拒绝调参。

@primowenjoy primowenjoy changed the title feat(server): protect Canary services and allow adaptive-only upgrades feat(server): expose Canary metadata and pinned adaptive plans Nov 5, 2026
@primowenjoy primowenjoy changed the title feat(server): expose Canary metadata and pinned adaptive plans feat(server): expose Canary relationships without write guards Nov 5, 2026
@aceforeverd
aceforeverd merged commit 21f4188 into modelsphere:master Nov 5, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants