Repository navigation
feat(server): serve Memory, Processors, Storage and EthernetInterfaces - #25
TomerKarniol wants to merge 8 commits into
Conversation
|
Hi @TomerKarniol. Thanks for your PR. PRs from untrusted users cannot be marked as trusted with I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
| return memoryGB, nil | ||
| } | ||
| // vmMemoryBytes returns the VM memory in bytes, or 0 when the spec declares none. | ||
| func vmMemoryBytes(vm *kubevirtv1.VirtualMachine) int64 { |
There was a problem hiding this comment.
Is this accounting for cpu and memory hotplug OR at least for staged changes? In other words, should it return the current values from VMI or the staged values from VM?
| return topo, nil | ||
| } | ||
| topo.Architecture = vm.Spec.Template.Spec.Architecture | ||
| cpu := vm.Spec.Template.Spec.Domain.CPU |
There was a problem hiding this comment.
Same comment about staged and hotplug values. Should it return data from VM or VMI?
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| * | ||
| * Copyright 2025 KubeVirt Redfish project and its authors. |
| return | ||
| } | ||
|
|
||
| // Drop a trailing empty element ("/Storage/" -> "/Storage"). |
There was a problem hiding this comment.
It might be better to show the whole path as an example, not just the trailing element to explain the constant 5 in the code.
|
|
||
| // redfishArchitecture maps a KubeVirt architecture to the Redfish | ||
| // ProcessorArchitecture and InstructionSet values. Unknown architectures map to "". | ||
| func redfishArchitecture(kubevirtArch string) (arch, instructionSet string) { |
There was a problem hiding this comment.
Should ppc be supported? Is this mapping necessary? It will require contant maintenance in case new platform designations apear?
There was a problem hiding this comment.
The mapping is needed because Redfish only accepts its own architecture names (like x86 and ARM), so KubeVirt's amd64 and arm64 can't be passed through. This matters because sushy, the Python Redfish client that OpenStack Ironic uses to manage machines, reads these fields to work out each system's CPU architecture. I didn't add ppc: the KubeVirt API marks ppc64le as "no longer supported", so it could never be reached. It's a two-case switch, and an unknown architecture is simply left out, so a new platform gives a missing value, never a wrong one.
| func newDevicesTestServerWith(t *testing.T, arch string, vmi *kubevirtv1.VirtualMachineInstance, k8s *fake.Clientset) *Server { | ||
| t.Helper() | ||
| mock := kubevirt.NewMockDynamicClient() | ||
| vm := &kubevirtv1.VirtualMachine{ |
There was a problem hiding this comment.
This is a bit weird, isn't it? You get the VMI from outside and then create a VM with constant device specification. How does it make sure the VM and VMI form a reasonably consistent pair?
| } | ||
|
|
||
| // When the cluster cannot be read for a reason other than "not found", the | ||
| // endpoints answer 500 instead of a plausible but wrong 200. |
There was a problem hiding this comment.
Why not 403 Forbidden directly?
There was a problem hiding this comment.
A 403 would tell the client that they aren't allowed to do this. Here the caller has already passed the access check. What is forbidden is the controller's own service account, which can't read the PVC. That is a server misconfiguration the client can't fix, so 500 is the right status. 403 would also be misleading, because a client might retry with other credentials or stop asking.
| return | ||
| } | ||
|
|
||
| // Log all incoming system requests for monitoring |
There was a problem hiding this comment.
I wonder if it might be a good idea to extract the access check and do it here instead of using it in every single sub-handler.
There was a problem hiding this comment.
Good idea, done. The access check for the hardware sub-resources (Memory, Processors, Storage, EthernetInterfaces) now happens once in handleSystem, right after the method check, and handleSystemDevices receives the resolved namespace and VM name.
I kept the other handlers (power, boot, virtual media, the main GET) as they are. They call the check themselves and have many direct unit tests. Moving them changes handler signatures and the order of body validation versus access check, which seems out of scope for a PR about read-only hardware data. I'm happy to do that as a follow-up PR, or open an issue for it if you prefer.
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
…ge-nics) into fork main Conflicts: the mock client keeps the fork's notFoundError helper; the CPU/memory lookups drop the unused GetVMCPU. Adapted to the fork: the new hardware resources use setNoCacheHeaders and read VMs/VMIs through the informer cache (getVMForRead/getVMIForRead).
…resources) (fork-only)
|
Hi @MarSik, friendly ping on this one. I addressed your last comment by |
b4277dd to
05be86f
Compare
|
Rebased onto main after #34. The only code change: the hardware resources now use
|
…eVirt.NodeName
An inventory that holds both the VMs behind this API and the nodes they
run on cannot link the two: nothing in the ComputerSystem says where
the VM runs. Tooling that wants the physical CPU, the rack or the
health of the host behind a VM has to go to the Kubernetes API with
its own credentials, which a Redfish client usually does not have.
GET /redfish/v1/Systems/{id}, and the response to a boot-device PATCH,
now carry Oem.KubeVirt.NodeName: the running VMI's status.nodeName. It
is omitted when the VM is not running (no VMI, or a VMI in a final
phase) and when the VMI cannot be read; the System still answers 200.
Redfish has no standard property for the host of a virtual
ComputerSystem, so it sits next to the existing Oem.KubeVirt.Processors.
Oem.KubeVirt.Processors becomes a pointer (OemProcessors), so an Oem
that carries only the node name does not report a 0x0x0 vCPU layout.
Its JSON is unchanged whenever the topology is read.
The node name is visible to every Redfish user who can see the VM's
chassis. It names a Kubernetes node, not an address or a credential.
Depends on kubevirt#25 (getRunningVMI, processorFields, Oem.KubeVirt.Processors).
Signed-off-by: itayherzberg <itayherzberg@gmail.com>
…e ComputerSystem
A Redfish client that inventories machines recognises the same machine
across reads by its serial number and UUID, and shows its maker and
model. The ComputerSystem had none of them, so a VM could only be
recognised by its System Id, which changes when the VM is renamed or
when system_id_convention changes.
GET /redfish/v1/Systems/{id}, and the response to a boot-device PATCH,
now carry:
- Manufacturer: always "KubeVirt".
- SerialNumber: spec.domain.firmware.serial.
- UUID: spec.domain.firmware.uuid, else the VM's metadata.uid, so a
stopped VM still has a stable UUID.
- Model: spec.domain.machine.type (the QEMU machine type).
Each is read from the running VMI first and the VM template second
(GetVMIdentity, built like GetVMCPUTopology on getRunningVMI and
effectiveSpec). If the VM or the VMI cannot be read, the three read
properties are left out and the System still answers 200; a VMI read
error does not fall back to the template, whose staged values may not
be what is running.
KubeVirt writes a random firmware serial and uuid and the default
machine type into the VM template within seconds of creation, Halted
VMs included, so on a current cluster every VM has all four.
Both ComputerSystem literals in server.go (GET and the boot PATCH
response) are replaced by one buildComputerSystem.
Depends on kubevirt#25 (getRunningVMI, effectiveSpec, processorFields).
Signed-off-by: itayherzberg <itayherzberg@gmail.com>
|
Two follow-up PRs now build on this one, so a heads-up in case it affects how you'd like to land it:
Both use this PR's |
|
I apologize for yet another merge conflict, I merged the VM/VMI informers PR first. |
…e ComputerSystem
A Redfish client that inventories machines recognises the same machine
across reads by its serial number and UUID, and shows its maker and
model. The ComputerSystem had none of them, so a VM could only be
recognised by its System Id, which changes when the VM is renamed or
when system_id_convention changes.
GET /redfish/v1/Systems/{id}, and the response to a boot-device PATCH,
now carry:
- Manufacturer: always "KubeVirt".
- SerialNumber: spec.domain.firmware.serial.
- UUID: spec.domain.firmware.uuid, else the VM's metadata.uid, so a
stopped VM still has a stable UUID.
- Model: spec.domain.machine.type (the QEMU machine type).
Each is read from the running VMI first and the VM template second
(GetVMIdentity, built like GetVMCPUTopology on getRunningVMI and
effectiveSpec). If the VM or the VMI cannot be read, the three read
properties are left out and the System still answers 200; a VMI read
error does not fall back to the template, whose staged values may not
be what is running.
KubeVirt writes a random firmware serial and uuid and the default
machine type into the VM template within seconds of creation, Halted
VMs included, so on a current cluster every VM has all four.
Both ComputerSystem literals in server.go (GET and the boot PATCH
response) are replaced by one buildComputerSystem.
Depends on kubevirt#25 (getRunningVMI, effectiveSpec, processorFields).
Signed-off-by: itayherzberg <itayherzberg@gmail.com>
…eVirt.NodeName
An inventory that holds both the VMs behind this API and the nodes they
run on cannot link the two: nothing in the ComputerSystem says where
the VM runs. Tooling that wants the physical CPU, the rack or the
health of the host behind a VM has to go to the Kubernetes API with
its own credentials, which a Redfish client usually does not have.
GET /redfish/v1/Systems/{id}, and the response to a boot-device PATCH,
now carry Oem.KubeVirt.NodeName: the running VMI's status.nodeName. It
is omitted when the VM is not running (no VMI, or a VMI in a final
phase) and when the VMI cannot be read; the System still answers 200.
Redfish has no standard property for the host of a virtual
ComputerSystem, so it sits next to the existing Oem.KubeVirt.Processors.
Oem.KubeVirt.Processors becomes a pointer (OemProcessors), so an Oem
that carries only the node name does not report a 0x0x0 vCPU layout.
Its JSON is unchanged whenever the topology is read.
The node name is visible to every Redfish user who can see the VM's
chassis. It names a Kubernetes node, not an address or a credential.
Depends on kubevirt#25 (getRunningVMI, processorFields, Oem.KubeVirt.Processors).
Signed-off-by: itayherzberg <itayherzberg@gmail.com>
ComputerSystem advertised Storage and EthernetInterfaces links, but requests to them fell through to the system handler and returned the parent resource. Memory had the same problem. Add collection and member resources for Memory, Processors, Storage (with drives) and EthernetInterfaces, built from the VM spec. Compute ProcessorSummary from sockets x cores x threads instead of cores only, and stop reporting made-up defaults when the VM does not declare memory. Note: ComputerSystem.Memory is now a link and the memory total moves to MemorySummary, matching the Redfish schema. Signed-off-by: Tomer Karniol <tomer.karniol@gmail.com>
Processors had no architecture, so clients such as sushy/Ironic reported the architecture as unknown. Map spec.architecture (unset means amd64, the KubeVirt default) to ProcessorArchitecture and InstructionSet: amd64 -> x86 / x86-64 and arm64 -> ARM / ARM-A64. Other architectures are omitted rather than guessed. Signed-off-by: Tomer Karniol <tomer.karniol@gmail.com>
When a VM declares no spec.architecture, leave ProcessorArchitecture and InstructionSet out instead of assuming amd64. KubeVirt's real default is the cluster's configured architecture, so a hardcoded amd64 would be wrong on an arm64 cluster. KubeVirt normally fills the field in on creation, so the common case is unchanged. Signed-off-by: Tomer Karniol <tomer.karniol@gmail.com>
…hardware Follow-up fixes found while testing the Memory, Processors, Storage and EthernetInterfaces resources on OpenShift Virtualization and reviewing the change. Cost: - Read the size of one drive per request instead of one PVC per disk. GetVMDisks now returns names and bus only; GetVMDisk resolves a single size. A 24-disk VM went from about 4.7s to about 0.5s per request. Errors that were hidden: - A PVC that cannot be read for a reason other than NotFound (forbidden, timeout) is now an error (HTTP 500) instead of a size taken from the DataVolume template. - A VMI that cannot be read for a reason other than NotFound is now an error instead of reporting NoLink for a VM that may be running. The test mock returns a typed NotFound like the real API. - If the CPU topology cannot be read, ProcessorSummary and Oem are left out instead of showing a guessed 1/1/1. GetVMCPUTopology returns the zero value on error, and the now unused GetVMCPU is removed. Schema: - Declare versions that define what is emitted, checked with the DMTF Redfish Service Validator: ComputerSystem v1_15_0, Memory v1_1_0, EthernetInterface v1_1_0 and VirtualMedia v1_2_0. - ProcessorSummary has no Oem property in any schema version, so the vCPU layout moves to the system-level Oem. - Add IPv6Addresses next to IPv4Addresses. - A zero-sized emptyDisk reports no capacity. Remove the unused Drive.MediaType and EthernetInterface.Description fields. Tests cover forbidden/timeout/not-found for PVC and VMI reads, the PVC lookup count for a 24-disk VM, users without access to the chassis, a running NIC with IPv4 and IPv6, empty VMs, and the schema versions. Signed-off-by: Tomer Karniol <tomer.karniol@gmail.com>
Update the single-system example to the current ComputerSystem shape (MemorySummary and Memory link, Processors link, ProcessorSummary counts, system-level Oem) and add a section with real responses for Processors, Memory, a Drive and an EthernetInterface. Signed-off-by: Tomer Karniol <tomer.karniol@gmail.com>
The VM spec holds the staged values; the VMI is what the guest is running. Reviewing the Processors and Memory resources raised the question whether hotplug and not-yet-applied edits are accounted for. They were not: every value came from the VM spec. A running VM now reports its VMI, a stopped VM its VM spec: - CPU: status.currentCPUTopology, then the VMI spec, then the VM spec. - Memory: status.memory.guestCurrent, then the VMI spec, then the VM spec. - Architecture: the VMI spec, then the VM spec. - Disks and NICs: the VMI spec, then the VM spec. A volume hot-plugged without --persist is listed, one only staged in the VM is not. - NICs report the operational linkState (up/down) the VMI gives. Only "VMI not found" and a VMI in a final phase (Succeeded/Failed, which KubeVirt can leave behind after a shutdown) count as stopped. Any other failure to read the VMI is returned, so a running VM is never shown with its staged values. GetVMNICs shares the same helper. Tests cover VMI over VM for each value, hotplug status, nil status fields, final-phase VMIs, hot-plugged disks, link state and VMI read errors. Signed-off-by: Tomer Karniol <tomer.karniol@gmail.com>
Routing:
- handleSystem sent every PATCH under /Systems/{id}/... to the boot update
handler before looking at the path, so PATCH /Systems/vm-1/Memory/1 was
treated as a boot update and answered 200 instead of 405. The hardware
resources (Memory, Processors, Storage, EthernetInterfaces) are now
routed first.
- The Reset action was matched with strings.Contains, so POST
/Systems/vm-1/Memory/1/Actions/ComputerSystem.Reset reached the power
handler. Only /Systems/{id}/Actions/ComputerSystem.Reset matches now.
NICs:
- A NIC the VMI reports as linkState "down" is NoLink. A running NIC with
no reported state is still LinkUp.
Tests and review follow-ups:
- 405 with Allow: GET for POST/PATCH/PUT/DELETE (with a boot body) on every
hardware path, and the Reset action under them.
- A VMI that cannot be read gives 500 on every hardware endpoint.
- Asymmetric topologies (3x2x1, 1x3x4), hotplug status over the VMI spec,
VMI over VM for storage and NICs, final-phase VMIs, odd member paths.
- Test helper comments explain that the VMI is not derived from the VM and
why a controller RBAC failure is a 500 and not a 403.
- Document the VMI-first rule and the link state.
Signed-off-by: Tomer Karniol <tomer.karniol@gmail.com>
handleSystem now validates the method and resolves the VM with its access check for Memory, Processors, Storage and EthernetInterfaces, then hands the resolved namespace and VM name to handleSystemDevices. The handler no longer repeats the check itself. The method check still comes first, so non-GET requests keep answering 405 with Allow: GET. The power, boot and virtual media handlers are unchanged. Signed-off-by: Tomer Karniol <tomer.karniol@gmail.com>
05be86f to
078cde4
Compare
|
Rebased onto current main to resolve the merge conflict (upstream's informer-cache change in #… touched the same client.go and mock code). |
What this PR does / why we need it:
ComputerSystemalready links to/Memory,/Storageand/EthernetInterfaces, but the router had no handlers for them. Every one of those URLs fell through and returned the plain ComputerSystem body with HTTP 200, so Redfish clients that walk those links (Ironic/sushy, inventory tooling) got no hardware data.This PR implements them read-only:
/Systems/{id}/Processors[/CPU{n}]Socket,TotalCores,TotalThreads,Model,ProcessorArchitecture,InstructionSet/Systems/{id}/Memory[/1]CapacityMiB/Systems/{id}/Storage[/1],.../Drives/{disk}CapacityByteswhere declared/Systems/{id}/EthernetInterfaces[/{nic}]MACAddress,LinkStatus(LinkUpwhile the VM runs and the link is up,NoLinkwhen it is stopped or KubeVirt reports the link as down), IPv4 and IPv6 addressesUnknown members return 404 and non-GET requests get 405 with
Allow: GET. That includesPATCHand theResetaction path below these resources: both used to be caught earlier inhandleSystem(everyPATCHunder a system was treated as a boot update, and theResetcheck matched any path containing/Actions/ComputerSystem.Reset), soPATCH /Systems/{id}/Memory/1answered 200 andPOST /Systems/{id}/Memory/1/Actions/ComputerSystem.Resetreached the power handler. The hardware resources are now routed first and theResetaction only matches/Systems/{id}/Actions/ComputerSystem.Reset. The existing access check (resolveSystemVMandCheckAccess) is reused. If the cluster cannot be read (for example the controller is not allowed to read PVCs), the endpoint answers 500 and logs the reason, instead of returning a made-up answer.It also fixes inaccuracies in the existing ComputerSystem output:
cores: 2, sockets: 2reportedCount: 2.ProcessorSummarynow follows the Redfish definitions:Count= sockets,CoreCount= sockets x cores,LogicalProcessorCount= sockets x cores x threads (the vCPU count), plusThreadingEnabledand the configuredModel(domain.cpu.model, omitted when unset). The exact layout is underOem.KubeVirt.Processors. If the topology cannot be read, these fields are left out instead of showing a guessed 1/1/1. An existing test that expected sockets to be ignored is updated.domain.memory.guestwas read, so a VM that setsresources.requests.memorygot 2 GiB. It now reads guest, then requests, then limits (see "Running and stopped VMs" below for which spec). If none is set the field is omitted instead of guessed, and a failed VM lookup returns an error instead of being swallowed.Memorywas a non-standardMemorySummarycontaining an@odata.id.Memoryis now a link to the collection and the total moved to the standardMemorySummary.@odata.typeversions were older than the properties they carry. Everything wasv1_0_0. The versions are nowComputerSystem.v1_15_0,Memory.v1_1_0,EthernetInterface.v1_1_0andVirtualMedia.v1_2_0(needed forInsertMedia/EjectMedia), checked with the DMTF Redfish Service Validator.Processor,StorageandDrivestay atv1_0_0, which already defines everything they emit.Notes and things to know:
CapacityBytes/CapacityMiBare the only capacity fields the Redfish schemas define and clients read them. A human-readable GiB copy is added underOem.KubeVirt.CapacityGiBfor drives and memory.status.capacity, thenspec.requests), then the DataVolume template, oremptyDisk/hostDisk. AcontainerDiskor cloud-init disk has no declared size, so none is shown.SpeedMbpsis intentionally omitted: virtual NICs have no link speed. A NIC the VMI reports aslinkState: downisNoLink; a running NIC with no reported state isLinkUp.Modelis the configured virtual CPU model, not a hardware model; KubeVirt does not expose the node's physical CPU here.ProcessorArchitecture/InstructionSetcome fromspec.architectureof the VMI, or of the VM when it is stopped (amd64->x86/x86-64,arm64->ARM/ARM-A64). The Redfish values are fixed enums, so the mapping is needed; an unknown architecture (for examples390x, which has no Redfish value) is omitted, so a new platform never produces a wrong value. If no architecture is declared (KubeVirt normally fills it in on creation), both fields are omitted rather than guessed. sushy builds its summary architecture from these fields.status.currentCPUTopology, then the VMI spec, then the VM spec.status.memory.guestCurrent, then the VMI spec, then the VM spec.--persistis listed; one only staged in the VM is not.RestartRequired) and a hotplug in progress do not show up before the guest has them. A VMI in a final phase (Succeeded/Failed, which KubeVirt can leave behind after a shutdown) counts as stopped.OembecauseProcessorSummaryhas noOemproperty in any schema version.pkg/kubevirt/devices.goandpkg/server/system_devices.goto avoid growingclient.go/server.gofurther.GetVMCPUis removed (nothing used it any more) and the test mock now returns a typed NotFound like the real API.Memoryin the ComputerSystem body changes from{@odata.id, TotalSystemMemoryGiB}to a plain link; the total moves toMemorySummary.TotalSystemMemoryGiB.ProcessorSummary.Countnow means the number of sockets (it used to be the number of cores). A 1-socket, 4-core VM goes fromCount: 4toCount: 1withCoreCount: 4andLogicalProcessorCount: 4.Testing:
Unit tests and checks:
make fmt,make validateandmake testpass. New tests cover CPU topology, model and architecture, memory sources, disk capacity (PVC, DataVolume template, emptyDisk including size 0, containerDisk, LUN), the number of PVC lookups for a 24-disk VM, forbidden / timeout / server error / not found for PVC and VMI reads, users of another chassis and unauthenticated requests on every new path, a running NIC with IPv4 and IPv6, VMs that declare nothing (empty collections are[], notnull), the schema versions, and every endpoint including 404s and method checks.golangci-lintreports no new issues. I used a v2-format equivalent of the config: the repo's.golangci.ymlis in the v1 format and does not load in current golangci-lint, which is probably worth a separate fix.go test -racepasses forpkg/kubevirt,pkg/redfishand the new tests.pkg/serverstill fails under-race: data races inconnection_pool.go,memory_monitor.go,job_scheduler.goand the logger, andTestServerShutdowntimes out. The same tests fail the same way on currentmain(2e38e46), so they are not caused by this PR.Manual tests, kind (KubeVirt v1.9.0, software emulation):
LinkUp, the KubeVirt-assigned MAC and an IPv4 address; stopped VMs reportedNoLink.Manual tests, OpenShift 4.20 with OpenShift Virtualization 4.20 (software emulation), controller run locally against the cluster:
host-modeland a fixed MAC, LUN / Filesystem-mode / Block-mode PVCs with a CD-ROM (left out ofStorage), 24 PVC-backed disks, and a VM that declares no disks, NICs, CPU or memory.threads: 2set on a running VM and restarted through the RedfishForceRestart:ThreadingEnabled: true,LogicalProcessorCount: 8and each socket 2 cores / 4 threads.enhancedsystem ID convention (namespace.vm) on every new path.vm_selectorby labels and by names behaves the same way.members_identitieskey in the VirtualMedia collection that is already onmain. The warnings are the deprecated/Systems/{id}/Storage/{id}/Drives/...URI.//, dot segments, encoded and very long names, odd methods): all answered, no panics and no 500s.geton PVCs removed from the role, PVC-backed drives answered 500 with the forbidden PVC in the log, while the drive list andemptyDiskdrives still worked. Everything was removed afterwards.Manual tests of the running/stopped rule and the routing, same OpenShift cluster, controller built from this branch and installed with the Helm chart:
RestartRequiredis set and the endpoints keep the VMI values; after a stop they show the VM spec; after a restart the new values.guestCurrent, so it stays at the old value until the guest has the memory.runStrategy: Oncewhose QEMU process was killed: the VMI stays inFailed, the VM spec is edited, and the endpoints show the VM spec andNoLink.PATCH,POST,PUTandDELETEon every hardware path answer 405;POSTof theResetaction below a hardware path answers 405, and the real action path still reaches the power handler.CPU-1,CPU0x1,CPU00, encoded dots and NULs, a 5000-character name, query strings) all answer 404 or 400, never 500; every link in the resource tree of three VMs resolves; deleting the VMI, halting and deleting the VM while polling gave 200 and then 404, with no 500.