Repository navigation
feat(k8s): ride through control-plane stalls in the auth preflight (cache + jitter + wider budget) - #68
feat(k8s): ride through control-plane stalls in the auth preflight (cache + jitter + wider budget)#68KadenZhang3321 wants to merge 1 commit into
Conversation
Follow-up to #64. The 2026-10-21 nightly incident showed the retry wrapper added there was not enough: a 25 s all-replica apiserver stall exceeded its ~21 s default budget (attempts=4, delays 3/6/12 s, no jitter), and with no cache each step boundary paid 15 fresh SelfSubjectAccessReview POSTs — measured at ~1.58k/min during one nightly fan-out — re-exposing every step to the next blip. - retryK8sApiCall: equal-jitter delays by default, so hundreds of hook processes stopped stampeding a recovering API server in lockstep (observed: 148 review POSTs in the single second it came back). Jitter opt-out + injectable RNG for tests; default behavior of non-retry errors unchanged, existing retry tests pass as-is. - New per-job on-disk cache (RUNNER_TEMP, falls back to os.tmpdir()): a successful verdict is written once and every later hook call of the same job answers from the file with zero API calls; TTL 1 h, and past the TTL a recent verdict is still reused for up to 24 h when the API is unreachable — degrading to a warning instead of killing an in-flight step. The file lives in the job pod, so cross-job staleness is impossible; RBAC mid-job changes remain out of scope (real pod calls still enforce RBAC), same as #64. - Auth preflight retry budget widened (6 attempts, 1.5-8 s capped backoff) so single-review coverage is ~60 s of stall. - Accurate semantics: a review that exhausts its transient-error budget is 'inconclusive', not 'denied' — it throws a distinct 'NOT a permissions denial' error (or serves the cache) instead of misreporting an unreachable authorization decision chain as a missing grant. Definitive allowed:false answers keep the legacy false return, uncached. - Fresh SAR object per probe and per retry is preserved (fixed in #64) and now covered by a test asserting retried probes re-submit their own verb. New tests: packages/k8s/tests/auth-permissions-test.ts (9 cases). tsc/jest/eslint/prettier clean locally; kind e2e unchanged.
Welcome To opensourceways CommunityHey @KadenZhang3321 , thanks for your contribution to the community. Bot Usage ManualI'm the Bot here serving you. You can find the instructions on how to interact with me at Here . That means you can comment below every pull request or issue to trigger Bot Commands. Contact GuideIf you have any questions, please contact the SIG: infratructure , |
CLA Signature Guide@KadenZhang3321 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
Linking Issue Notice@KadenZhang3321 , the pull request must be linked to at least one issue. |
|
|||||||||||||||||
Follow-up to #64, based on the full post-mortem of the 2026-10-21 nightly incident (run #15 of vllm-project/vllm-ascend).
Why #64 was not enough
That night the cluster's control plane had a 25-second hole (02:15:26–02:15:51 UTC): all three kube-apiserver replicas simultaneously unreachable, in-cluster clients got
connect ECONNREFUSEDon the kubernetes service; on recovery, the authorization decision chain briefly answered403 selfsubjectaccessreviews is forbiddenfor SAs whose permissions were intact (18 confirmed by audit, across four runner namespaces). #64's wrapper couldn't ride that out:exit 1with a message implying missing RBAC.Changes
retryK8sApiCall: equal-jitter delays by default (jitter?: boolean, injectablerandom); non-retry error semantics untouched, existing tests pass as-is.packages/k8s/src/k8s/auth-cache.ts: per-job on-disk cache (RUNNER_TEMP, fallbackos.tmpdir()). First hook call of a job checks and persists; all later calls answer from the file with zero API calls. TTL 1 h; past the TTL, a successful verdict is still reused for up to 24 h only when reviews are undeliverable (warning, not death). Cache lives in the job pod → no cross-job staleness by construction.isAuthPermissionsOK: per-review budget widened to 6 attempts / 1.5–8 s capped backoff (~60 s coverage of fast-refused stalls); three-way outcome — definitiveallowed:falsekeeps the legacyreturn false(uncached), transient exhaustion is inconclusive (serve cache, else throw an explicit "This is NOT a permissions denial"), success caches. Fresh SAR per probe/retry preserved from feat(k8s): retry transient k8s API failures in auth preflight; bump runner to 2.338.0 #64, now with a test that a retried probe re-submits its own verb.packages/k8s/tests/auth-permissions-test.ts— 9 cases (cache write/skip/grace/namespace-isolation, denial vs inconclusive, recovery-mid-retry, error propagation, jitter-disabled determinism paths).Validation
tsc --noEmit,jest(auth 9/9, retry 9/9, utils spec 31/31, wait-for-pod-phases 72/72, constants 16/16, unit suites green),npm run lint,npm run format-check,npm run build(ncc) all pass locally. Kind-based e2e untouched.Rollout
Per current plan, merge together with the runner 2.339 image bump (same train as the version matrix bump); validate with the standard chaos check on a test runner:
iptables -A OUTPUT -p tcp -d 10.247.0.1 --dport 443 -j REJECT; sleep 30; delete→ jobs should finish with at most::warning::reusing this job's successful check.A parallel generalization of the caching idea was also proposed upstream (actions#458); if it lands eventually, this implementation can be swapped for it with no caller changes (
isAuthPermissionsOKkeeps its exported shape).