feat(k8s): retry transient k8s API failures in auth preflight; bump runner to 2.338.0 by KadenZhang3321 · Pull Request #64 · opensourceways/runner-container-hooks

KadenZhang3321 · 2026-10-15T09:03:37Z

说明：同一分支此前误开的空描述 PR 为 #63（已关闭），本 PR 为该分支合入 main 的正式提交；同分支合入 release-2.337.0-4e9113c 的 PR 为 #61。

背景

resolved https://github.com/opensourceways/backlog/issues/2467
vllm-ascend / sglang 等社区 CI 频发 Executing the custom container implementation failed（job 测试本身全绿，收尾/上传阶段连坐），排查结论见 ascend-gha-runners/docs#274。

定位到的机理：runner pod 经节点本地 ClusterIP 规则访问 in-cluster API（10.247.0.1），托管面 apiserver 后端每次轮换，该转发链路都会开出秒级窗口，三种形态随机：

形态	表现
规则瞬间无后端	`connect ECONNREFUSED 10.247.0.1:443`
规则指向过期后端	请求挂死（实测 6s+ 无响应）
打到未预热实例	`403 selfsubjectaccessreviews ... is forbidden`（重试即恢复）

而 hook 入口 isAuthPermissionsOK() 在每个 step 执行前都要跑 13 个并行、无 timeout、无重试的 SelfSubjectAccessReview，任一请求失败即 process.exit(1)。一个 job 上百次 API 触点，窗口命中变成必然。实测同一 token/同一请求在窗口内 200/挂死/403 随机，直连 apiserver 后端 IP 则始终正常——排除权限配置问题，纯瞬时窗口。

变更

retryK8sApiCall（utils 新增，可复用）：有限次重试 + 指数退避（3s/6s/12s×4 次）+ 单次请求 timeout（默认 15s，覆盖挂死形态）。仅重试瞬态错误：连接层错误码（ECONNREFUSED/ECONNRESET/ETIMEDOUT/UND_* 等）、消息签名、以及 HTTP 401/403/408/429/5xx（未预热实例的瞬时 forbidden 正是要跨过去的窗口）；400/404/409 等确定性错误立即抛出，行为与原逻辑一致。
isAuthPermissionsOK 接入重试；顺带修复原实现所有 13 个请求共享同一个可变 SAR 对象的问题——不修复的话，重试闭包重新序列化时会拿到外层循环最后一次赋值的 body（提交错误的 verb 组合）。
Dockerfile：actions-runner 2.337.0 → 2.338.0。

效果

秒级窗口从「step 判死」降级为「warning + 多等数秒」。默认预算 4 次尝试 / 最长约 21s 退避，覆盖并远超实测到的窗口时长（1–6s）。

验证

npx jest tests/retry-utils-test.ts：9/9 通过（分类器 4 组 + 重试行为 5 组，含"超时弃挂起重试后成功"用例）
npm run build --prefix packages/k8s：tsc + ncc 通过，产物含新逻辑
eslint / prettier：新增代码 0 告警（仓库存量 7 个历史 lint error 与本次无关）

重跑窗口截图证据（故障窗口内 ClusterIP 路径）：c200/c403/6s挂死随机 → 窗口外同请求 240/240 全绿。

后续建议（不在本 PR）

预检结果按 job 缓存（state 传递），免去每 step 重问 13 次
重试逐步推广到 createJobPod / execPodStep / cleanup 的 API 调用

resolve ascend-gha-runners/docs#274

opensourceways-bot · 2026-10-15T09:03:42Z

Welcome To opensourceways Community

Hey @KadenZhang3321 , thanks for your contribution to the community.

Bot Usage Manual

I'm the Bot here serving you. You can find the instructions on how to interact with me at Here . That means you can comment below every pull request or issue to trigger Bot Commands.

Contact Guide

If you have any questions, please contact the SIG: infratructure ,
and any of the maintainers: @GeorgeCao-hw, @TangJia025, @pkking, @zhongjun2 ,
and any of the committers: @GeorgeCao-hw, @Goalina, @Hourunze1997, @JavaPythonAIForBAT, @KadenZhang3321, @LiYanghang00, @ccijunk, @drizzlezyk, @pkking, @rosecoffe, @tfhddd, @yao-xiaobai, @zhongjun2, @zkhzkhz .

opensourceways-bot · 2026-10-15T09:03:50Z

Linking Issue Notice

@KadenZhang3321 , the pull request must be linked to at least one issue.
If an issue has already been linked, but the needs-issue label remains, you can remove the label by commenting /check-issue .

opensourceways-bot · 2026-10-15T09:03:50Z

CLA Signature Guide

@KadenZhang3321 , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit	Reason
ea25fd1b Change command to remove sudo to...	the email used in the commit is not linked to a signed CLA! please verify that it matches the email you used when signing the CLA.
2934de33 Sort 'find' output before hashin...	the email used in the commit is an invalid email! please check all commits.
ad9cb43c feat: check if required binaries...	the email used in the commit is not linked to a signed CLA! please verify that it matches the email you used when signing the CLA.
15e80893 Allow non-root container (#264) ...	the email used in the commit is not linked to a signed CLA! please verify that it matches the email you used when signing the CLA.
0951cc73 Improve validation checks after ...	the email used in the commit is not linked to a signed CLA! please verify that it matches the email you used when signing the CLA.
a8f7515b feat(k8s): ride through transien...	the email used in the commit is an invalid email! please check all commits.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

opensourceways-bot · 2026-10-15T09:40:58Z

检查项	状态
敏感信息扫描	✅
安全编码扫描	✅
漏洞扫描	❌
开源license合规扫描	✅
UT测试覆盖率	❌
开发阶段设计文档检查	❌
流水线链接	点击跳转查看日志

opensourceways-bot · 2026-10-15T09:41:22Z

CLA Signature Guide

@KadenZhang3321 , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit	Reason
ea25fd1b Change command to remove sudo to...	the email used in the commit is not linked to a signed CLA! please verify that it matches the email you used when signing the CLA.
2934de33 Sort 'find' output before hashin...	the email used in the commit is an invalid email! please check all commits.
ad9cb43c feat: check if required binaries...	the email used in the commit is not linked to a signed CLA! please verify that it matches the email you used when signing the CLA.
15e80893 Allow non-root container (#264) ...	the email used in the commit is not linked to a signed CLA! please verify that it matches the email you used when signing the CLA.
0951cc73 Improve validation checks after ...	the email used in the commit is not linked to a signed CLA! please verify that it matches the email you used when signing the CLA.
a8f7515b feat(k8s): ride through transien...	the email used in the commit is an invalid email! please check all commits.
26b0a6d8 Merge origin/main into feat/retr...	the email used in the commit is not linked to a signed CLA! please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

KadenZhang3321 · 2026-10-15T09:46:43Z

/check-cla

opensourceways-bot · 2026-10-15T09:46:52Z

CLA Signature Pass

KadenZhang3321, thanks for your pull request. All authors of the commits have signed the CLA. 👍

opensourceways-bot · 2026-10-15T09:46:53Z

CLA Signature Pass

KadenZhang3321, thanks for your pull request. All authors of the commits have signed the CLA. 👍

Goalina

lgtm

opensourceways-bot · 2026-10-15T10:12:56Z

检查项	状态
敏感信息扫描	✅
安全编码扫描	✅
漏洞扫描	❌
开源license合规扫描	✅
UT测试覆盖率	❌
开发阶段设计文档检查	❌
流水线链接	点击跳转查看日志

opensourceways-bot · 2026-10-15T10:27:57Z

检查项	状态
敏感信息扫描	✅
安全编码扫描	✅
漏洞扫描	❌
开源license合规扫描	✅
UT测试覆盖率	❌
开发阶段设计文档检查	❌
流水线链接	点击跳转查看日志

…unner to 2.338.0 Runner pods reach the in-cluster API (10.247.0.1) through node-local ClusterIP rules; every control-plane endpoint rotation opens a sub-second window with three random shapes: ECONNREFUSED (no backend), request hang (stale backend), and transient 403 on unwarmed apiserver replicas. The hook entrypoint isAuthPermissionsOK() runs 13 parallel, untimed, un-retried SelfSubjectAccessReviews before each step; any single failure aborts the whole step (process.exit(1)). - Add retryK8sApiCall: bounded retries with exponential backoff (3s/6s/12s, 4 attempts) + per-request timeout (15s); retries only transient errors (connection-layer codes, hang signatures, HTTP 401/403/408/429/5xx); deterministic errors throw immediately - Wrap the SAR preflight checks with retryK8sApiCall, and fix the shared mutable V1SelfSubjectAccessReview instance (now a fresh SAR per request) - Register actions-runner 2.338.0 in .github/runner-versions.yaml - Add tests/retry-utils-test.ts (classifier + retry behavior)

opensourceways-bot · 2026-10-15T10:42:20Z

CLA Signature Pass

KadenZhang3321, thanks for your pull request. All authors of the commits have signed the CLA. 👍

KadenZhang3321 · 2026-10-15T10:42:22Z

/check-cla

opensourceways-bot · 2026-10-15T10:42:29Z

CLA Signature Pass

KadenZhang3321, thanks for your pull request. All authors of the commits have signed the CLA. 👍

socket-security · 2026-10-15T10:42:32Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	js-yaml@4.3.1 ⏵ 4.3.2	⁺¹	⁺¹⁶	⁺¹

View full report

opensourceways-bot · 2026-10-15T10:56:10Z

检查项	状态
敏感信息扫描	✅
安全编码扫描	✅
漏洞扫描	✅
开源license合规扫描	✅
UT测试覆盖率	❌
开发阶段设计文档检查	✅
流水线链接	点击跳转查看日志

opensourceways-bot · 2026-10-15T11:21:54Z

检查项	状态
敏感信息扫描	✅
安全编码扫描	✅
漏洞扫描	✅
开源license合规扫描	✅
UT测试覆盖率	❌
开发阶段设计文档检查	✅
流水线链接	点击跳转查看日志

opensourceways-bot added the sig/infratructure label Oct 15, 2026

opensourceways-bot added stat/needs-squash needs-issue opensourceways-cla/no labels Oct 15, 2026

KadenZhang3321 force-pushed the feat/retry-k8s-api-calls-2.338 branch from edcb2c9 to 0dab1c9 Compare October 15, 2026 09:46

opensourceways-bot added opensourceways-cla/yes and removed opensourceways-cla/no stat/needs-squash labels Oct 15, 2026

Goalina approved these changes Oct 15, 2026

View reviewed changes

opensourceways-bot removed the needs-issue label Oct 15, 2026

KadenZhang3321 force-pushed the feat/retry-k8s-api-calls-2.338 branch from 0dab1c9 to 25f6887 Compare October 15, 2026 10:42

KadenZhang3321 merged commit 23f0a51 into main Oct 15, 2026
8 of 9 checks passed

KadenZhang3321 mentioned this pull request Oct 16, 2026

chore(release): bump version to 0.8.0-ascend.2 #66

Open

KadenZhang3321 mentioned this pull request Oct 27, 2026

feat(k8s): ride through control-plane stalls in the auth preflight (cache + jitter + wider budget) #68

Draft

Conversation

KadenZhang3321 commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

背景

变更

效果

验证

后续建议（不在本 PR）

Uh oh!

opensourceways-bot commented Oct 15, 2026

Welcome To opensourceways Community

Bot Usage Manual

Contact Guide

Uh oh!

opensourceways-bot commented Oct 15, 2026

Linking Issue Notice

Uh oh!

opensourceways-bot commented Oct 15, 2026

CLA Signature Guide

Uh oh!

opensourceways-bot commented Oct 15, 2026

Uh oh!

opensourceways-bot commented Oct 15, 2026

CLA Signature Guide

Uh oh!

KadenZhang3321 commented Oct 15, 2026

Uh oh!

opensourceways-bot commented Oct 15, 2026

CLA Signature Pass

Uh oh!

opensourceways-bot commented Oct 15, 2026

CLA Signature Pass

Uh oh!

Goalina left a comment

Choose a reason for hiding this comment

Uh oh!

opensourceways-bot commented Oct 15, 2026

Uh oh!

opensourceways-bot commented Oct 15, 2026

Uh oh!

opensourceways-bot commented Oct 15, 2026

CLA Signature Pass

Uh oh!

KadenZhang3321 commented Oct 15, 2026

Uh oh!

opensourceways-bot commented Oct 15, 2026

CLA Signature Pass

Uh oh!

socket-security Bot commented Oct 15, 2026

Uh oh!

opensourceways-bot commented Oct 15, 2026

Uh oh!

opensourceways-bot commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

KadenZhang3321 commented Oct 15, 2026 •

edited

Loading