Skip to content

adaptation: avoid holding lock across runtime update callback - #301

Merged
mikebrow merged 1 commit into
containerd:mainfrom
ningmingxiao:fix_deadlock
Jul 18, 2026
Merged

adaptation: avoid holding lock across runtime update callback#301
mikebrow merged 1 commit into
containerd:mainfrom
ningmingxiao:fix_deadlock

Conversation

@ningmingxiao

@ningmingxiao ningmingxiao commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to address the deadlock reported in containerd/containerd#13733 by adjusting lock behavior in the NRI runtime adaptation, introducing an optional external mutex intended to control lock ordering between the runtime and the adaptation layer.

Changes:

  • Added adaptMu to Adaptation and a new option WithAdaptLock to configure an external mutex.
  • Updated updateContainers to acquire the optional external mutex before acquiring the internal Adaptation mutex.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread pkg/adaptation/adaptation.go Outdated
Comment thread pkg/adaptation/adaptation.go
@fuweid fuweid changed the title nri:fix deadlock adaptation: avoid holding lock across runtime update callback Jul 16, 2026
@fuweid

fuweid commented Jul 16, 2026

Copy link
Copy Markdown
Member

@ningmingxiao could you please update the git commit message with new title and this information? Thanks

  updateContainers currently holds the Adaptation mutex while calling the
  runtime-provided updateFn. containerd's updateFn acquires its local NRI
  mutex, while CRI lifecycle paths acquire the same locks in the opposite
  order:

      lifecycle: local mutex -> Adaptation mutex
      update:    Adaptation mutex -> local mutex

  Concurrent lifecycle and plugin-initiated update requests can therefore
  deadlock.

  updateContainers does not access mutable Adaptation state. Invoke updateFn
  without holding the Adaptation mutex and leave synchronization of
  runtime-owned state to the runtime.

  Fixes https://github.com/containerd/containerd/issues/13733

@ningmingxiao

ningmingxiao commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

@fuweid done thanks

@ningmingxiao
ningmingxiao force-pushed the fix_deadlock branch 3 times, most recently from a9aabad to 39a71d5 Compare July 17, 2026 00:43
updateContainers currently holds the Adaptation mutex while calling the
runtime-provided updateFn. containerd's updateFn acquires its local NRI
mutex, while CRI lifecycle paths acquire the same locks in the opposite
order:

  lifecycle: local mutex -> Adaptation mutex
  update:    Adaptation mutex -> local mutex

Concurrent lifecycle and plugin-initiated update requests can therefore
deadlock.

updateContainers does not access mutable Adaptation state. Invoke updateFn
without holding the Adaptation mutex and leave synchronization of
runtime-owned state to the runtime.

Fixes containerd/containerd#13733

Signed-off-by: ningmingxiao <ning.mingxiao@zte.com.cn>

@fuweid fuweid left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Thanks

@mikebrow mikebrow left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mikebrow
mikebrow merged commit 84180b6 into containerd:main Jul 18, 2026
16 checks passed
@ningmingxiao
ningmingxiao deleted the fix_deadlock branch July 20, 2026 01:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NRI deadlock between plugin-initiated Runtime.UpdateContainers and CRI lifecycle/update events

4 participants