Skip to content

fix(vllm): recover hybrid KV loads through request-level outcomes - #382

Merged
ketor merged 3 commits into
dingodb:mainfrom
ketor:fix/vllm-request-load-failures
Sep 17, 2026
Merged

ketor merged 3 commits into
dingodb:mainfrom
ketor:fix/vllm-request-load-failures

Conversation

@ketor

@ketor ketor commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Release candidate v2.27.1: migrate hybrid/multi-group loads to native request-level completion/failure outcomes; never emit ambiguous block errors for those layouts.
  • Separate scheduler waiting from I/O execution. Serialized loads retain explicit owner-device pre/post fences, including runners that defer the load hook until after forward.
  • Enforce the request-level completion fence even with the legacy DFKV_GPU_LOAD_FENCE=0 override.
  • Preserve cancellation/ownership and all-rank completion, prevent repeated submission, and quarantine failed external hits until request completion.
  • Remove the obsolete scheduler invalid-block patch. Native ABI, wire format, storage keys and functional C++ code are unchanged.

Verification

  • Pinned real vLLM runtime: 8b1d188046034b42024d22ee52fa6c3948b4acbd ([Bugfix][Mooncake] Report request-level KV load failures under HMA vllm-project/vllm#56855).
  • Connector suite: 158 passed plus 24 subtests; optional live-storage case independently passed with actual GPUDirect RDMA GPU byte comparison.
  • Real multi-rank model qualification: async/serialized I/O × recompute/fail policy; healthy external loads, persistent genuine store NotFound after lookup, injected native-completion exception, delayed peer completion, correct subsequent requests.
  • Cancellation during a pending load retained ownership until completion; subsequent request and prefix-cache reset succeeded.
  • Unrelated cold prefill remained usable during a delayed failed load; asynchronous execution made independent progress.
  • Real dual-device completion regression includes the legacy fence opt-out; pre-fix test failed and corrected contract passed.
  • Three v2.27.1 wheels built and imported in the pinned engine environment; release-contract tests passed.
  • Two independent static reviews plus final inline fence review. All eight checks on final head 02bb44d are green.

Compatibility

Requires the complete native get_transfer_results / failed_recving protocol in the vLLM runner, executor and scheduler. Upgrade older engines lacking that API together with this connector; no compatibility shim or scheduler patch is shipped.

@ketor
ketor marked this pull request as ready for review September 17, 2026 07:40
@ketor
ketor added this pull request to the merge queue Sep 17, 2026
Merged via the queue into dingodb:main with commit b30870c Sep 17, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant