Enforce OS-level CPU pinning for pinned workloads (cpuset cgroup)

[t0gun]

Enforce OS-level CPU pinning for pinned workloads

Summary

The capacity ledger already models a "pinned" CPU mode (CPU_MODE_PINNED) where workloads get dedicated threads with no overcommit. However, this is currently accounting only — there is no OS-level enforcement. Pinned workloads are scheduled by the Linux CFS scheduler identically to shared workloads. This issue tracks implementing actual cpuset cgroup pinning so that pinned workloads get real physical core isolation.

Current state

The capacity ledger in auction_linux.go correctly separates pinned and shared pools:

• IsPinned: true workloads are counted 1:1 against the sellable thread pool (no overcommit)
• IsPinned: false workloads are counted against an overcommitted shared pool (4:1)

But at the OS level, nothing enforces this distinction:

• No cpuset cgroup is written for the Firecracker jailer process
• No sched_setaffinity is called on the Firecracker VMM process
• No vCPU-to-pCPU mapping exists in the Firecracker machine configuration
• The MachineConfiguration only sets VcpuCount, MemSizeMib, and Smt — no CPU template, no affinity

This means a "pinned" workload gets a capacity guarantee in the ledger (its threads are reserved and not overcommitted), but the Linux scheduler can place its vCPU threads on any core. Under contention, a pinned workload can still be displaced from its "dedicated" cores by other workloads or host processes.

What needs to happen

When a workload with CPU_MODE_PINNED is launched, the edge daemon must:

1. Allocate specific physical core pairs from the sellable pool (not just decrement a counter)
2. Write cpuset.cpus on the Firecracker jailer cgroup to restrict the VMM process to those cores
3. Track which core pairs are assigned to which VM so they can be returned to the free pool on stop
4. Ensure core scheduling cookies remain per-tenant so pinned workloads from the same tenant can still share sibling threads safely

Current capacity model (accounting only)

// auction_linux.go — Capacity struct
type Capacity struct {
    // ...
    sellableThreads     uint32
    usedPinnedCores     uint32    // just a counter, no core IDs tracked
    reservedPinnedCores uint32    // just a counter, no core IDs tracked
    // ...
}

Reserve and Commit just increment and decrement counters. No record of which specific cores belong to which VM.

Required capacity model (with core tracking)

The capacity ledger needs to track actual core pair assignments:

type Capacity struct {
    // ...
    sellableThreads  uint32
    // Map of assigned core pair index → microvm ID
    pinnedCoreAssignments map[uint32]string
    // ...
}

When a pinned workload is reserved:

• Find N free core pairs in pinnedCoreAssignments
• Assign them to the microvm ID
• Return the list of CPU IDs (e.g. 0,1 for core pair 0, 2,3 for core pair 1)

When a pinned workload stops:

• Look up the microvm ID in pinnedCoreAssignments
• Remove all entries for that ID
• Return the core pairs to the free pool

Files that need changes

File	Change
scaled/workload/auction_linux.go	Replace counter-based pinned accounting with core pair tracking. Reserve and Commit for pinned workloads must return the assigned CPU IDs.
scaled/workload/auction_linux.go	Add PinnedCoreIDs() []uint32 method or extend HardwareSpec/reservation to carry assigned CPU IDs
scaled/workload/launch_linux.go	Pass assigned CPU IDs through to the launcher when building LaunchRequest
scaled/workload/microvm/launcher_linux.go	After Firecracker starts, write cpuset.cpus on the jailer cgroup with the assigned CPU IDs
scaled/workload/microvm/firecracker_linux.go	Optionally pass CPU IDs to Firecracker via CpuTemplate or jailer cgroup config
scaled/workload/microvm/launcher_linux.go	On Stop, return the CPU IDs to the capacity ledger

Design considerations

Core pair allocation

With SMT enabled and core scheduling active, pinned workloads should be allocated in core pairs (both sibling threads). This ensures:

• The workload owns the entire physical core
• Core scheduling cookies are respected (both siblings have the same tenant's cookie)
• No cross-tenant sharing on the same physical core

A 4-vCPU pinned workload gets 2 core pairs (4 threads). A 2-vCPU pinned workload gets 1 core pair.

Cgroup integration

The Firecracker jailer already creates a cgroupv2 hierarchy for the VMM process (CgroupVersion: "2" in firecracker_linux.go:83). After the jailer starts, the cgroup path is deterministic based on the jailer ID (microvm ID). The launcher would write to:

/sys/fs/cgroup/<jailer-cgroup-path>/cpuset.cpus

With the assigned CPU IDs (e.g. 0,1 for one core pair, 0,1,4,5 for two).

Interaction with core scheduling

Pinned workloads still get a core scheduling cookie (from issue #51). The cookie ensures no cross-tenant sibling sharing. The cpuset restriction ensures the VMM can only run on its assigned cores. Both mechanisms work together:

• cpuset.cpus restricts WHICH cores the VMM can use
• Core scheduling cookie restricts WHO can share sibling threads with the VMM

Shared workloads remain unpinned

Shared workloads (CPU_MODE_SHARED) continue to use the overcommitted pool with no cpuset restriction. They run on any available sellable thread that isn't assigned to a pinned workload. Core scheduling still applies — shared workloads from the same tenant can share sibling threads.

NUMA awareness (future)

On multi-socket nodes, core pair allocation should prefer cores on the same NUMA node as the VM's memory. This is a future optimization and can be layered on top of the core pair assignment logic.

What is NOT changing

• Host reservation stays as a soft capacity reservation (no host cpuset pinning)
• Shared workloads remain unpinned with 4:1 overcommit
• Core scheduling cookies remain per-tenant for all workloads
• The auction/bidding flow stays the same — the bidder still checks freePinnedCoresLocked() before bidding

Acceptance criteria

• Pinned workloads get specific core pairs assigned at launch time
• The Firecracker VMM process is restricted to those cores via cpuset.cpus
• Stopping a pinned workload returns its core pairs to the free pool
• Two pinned workloads are never assigned overlapping core pairs
• Shared workloads cannot run on cores assigned to pinned workloads
• Tests cover core pair allocation, assignment, and release