Skip to content

A datapath upgrade blacks out every VM on a node #774

Closed

Description

@privateip

Summary

When an upgrade changes the layout of one eBPF map, the node recreates every map empty, and compute nodes then drop all traffic to their VMs until each VM is recreated.

This hit both staging compute nodes on 2026-10-07 at 02:02 UTC, when one map's key grew.

The next release carries that change and would do the same to every production compute node.

Acceptance criteria

  • An upgrade that changes one map's layout leaves every other map's contents in place
  • VMs keep receiving traffic through a datapath upgrade that changes a map's layout

Activity

  1. added theissue type on Oct 7, 2026
  2. privateip commented on Oct 8, 2026

    @privateip
    CollaboratorAuthor

    Fixed by #775, #776 and #777, released in v0.5.4. A map layout change now recreates only that map, existing workloads move to the new egress program at startup, and missing attachment rows are rebuilt without recreating workloads. One case the repair misses is tracked in #806.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinggalactic-cniRelated to galactic-cni

    Type

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions