Repository navigation
feat(kubevirt): serve read-only VM/VMI lookups from an informer cache #32
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,281 @@ | ||
| /* | ||
| * This file is part of the KubeVirt Redfish project | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| * | ||
| * Copyright 2025 KubeVirt Redfish project and its authors. | ||
| * | ||
| */ | ||
|
|
||
| package kubevirt | ||
|
|
||
| // A read-only cache of VirtualMachine and VirtualMachineInstance objects. | ||
| // | ||
| // Every Redfish request fans out to several Kubernetes API calls, and nearly | ||
| // all of them are reads of the same VM/VMI objects (a GET System is five GETs). | ||
| // With many BareMetalHosts polling at once, those reads are what saturate the | ||
| // client-side rate limiter. For every configured namespace the client keeps | ||
| // one client-go informer for VMs and one for VMIs, so the read-only Redfish | ||
| // paths are served from memory. | ||
| // | ||
| // Two rules keep cached reads correct: | ||
| // | ||
| // - Only read-only inventory lookups use the cache (getVMForRead and its | ||
| // callers). Every path that reads an object in order to write it, and the | ||
| // handler that reads an object back right after writing it, reads from | ||
| // the API server (GetVM, GetVMBootOptionsLive). A write is therefore | ||
| // never computed from, or reported from, a copy the watch has not caught | ||
| // up with. | ||
| // - A cache that has not finished its initial list is not consulted, and | ||
| // once it has, its answer is final, including "not found". Falling back | ||
| // to a GET on every miss would cost a GET on every power poll of a | ||
| // stopped VM, which has no VMI, and that is the dominant state of a BMH | ||
| // during inspection and provisioning. | ||
| // | ||
| // A write by anyone, this process included, reaches the cache with its watch | ||
| // event, usually within milliseconds. Until then a cached read returns the | ||
| // previous version. No write path reads from the cache, so a stale read can | ||
| // make a response out of date but can never be written back over a change. | ||
|
|
||
| import ( | ||
| "fmt" | ||
| "sort" | ||
| "time" | ||
|
|
||
| apierrors "k8s.io/apimachinery/pkg/api/errors" | ||
| "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" | ||
| "k8s.io/apimachinery/pkg/labels" | ||
| "k8s.io/apimachinery/pkg/runtime/schema" | ||
| "k8s.io/client-go/dynamic" | ||
| "k8s.io/client-go/dynamic/dynamicinformer" | ||
| "k8s.io/client-go/informers" | ||
| kubevirtv1 "kubevirt.io/api/core/v1" | ||
|
|
||
| "github.com/kubevirt/redfish-controller/pkg/logger" | ||
| ) | ||
|
|
||
| var ( | ||
| vmGVR = schema.GroupVersionResource{Group: "kubevirt.io", Version: "v1", Resource: "virtualmachines"} | ||
| vmiGVR = schema.GroupVersionResource{Group: "kubevirt.io", Version: "v1", Resource: "virtualmachineinstances"} | ||
| ) | ||
|
|
||
| // vmCache holds one VM informer and one VMI informer per configured namespace. | ||
| type vmCache struct { | ||
| stopCh chan struct{} | ||
| factories []dynamicinformer.DynamicSharedInformerFactory | ||
| vms map[string]informers.GenericInformer | ||
| vmis map[string]informers.GenericInformer | ||
| } | ||
|
|
||
| // newVMCache starts informers for every distinct, non-empty namespace. It does | ||
| // not wait for them to sync; readers check HasSynced per call. | ||
| func newVMCache(client dynamic.Interface, namespaces []string) *vmCache { | ||
| vc := &vmCache{ | ||
| stopCh: make(chan struct{}), | ||
| vms: make(map[string]informers.GenericInformer), | ||
| vmis: make(map[string]informers.GenericInformer), | ||
| } | ||
| seen := make(map[string]bool) | ||
| for _, namespace := range namespaces { | ||
| if namespace == "" || seen[namespace] { | ||
| continue | ||
| } | ||
| seen[namespace] = true | ||
|
|
||
| factory := dynamicinformer.NewFilteredDynamicSharedInformerFactory(client, 0, namespace, nil) | ||
| vmInformer := factory.ForResource(vmGVR) | ||
| vmiInformer := factory.ForResource(vmiGVR) | ||
| // Transforms must be set before the factory starts. | ||
| for _, inf := range []informers.GenericInformer{vmInformer, vmiInformer} { | ||
| if err := inf.Informer().SetTransform(stripManagedFields); err != nil { | ||
| logger.Error("Failed to set informer transform for namespace %s: %v", namespace, err) | ||
| } | ||
| } | ||
| vc.vms[namespace] = vmInformer | ||
| vc.vmis[namespace] = vmiInformer | ||
| vc.factories = append(vc.factories, factory) | ||
| factory.Start(vc.stopCh) | ||
| } | ||
| return vc | ||
| } | ||
|
|
||
| // stop closes the stop channel first so the informer goroutines exit, then | ||
| // waits for them. The other order deadlocks. | ||
| func (vc *vmCache) stop() { | ||
| if vc == nil { | ||
| return | ||
| } | ||
| close(vc.stopCh) | ||
| for _, factory := range vc.factories { | ||
| factory.Shutdown() | ||
| } | ||
| } | ||
|
|
||
| // informer returns the synced informer for gvr in namespace, or false when | ||
| // there is none or it has not finished its initial list. | ||
| func (vc *vmCache) informer(gvr schema.GroupVersionResource, namespace string) (informers.GenericInformer, bool) { | ||
| if vc == nil { | ||
| return nil, false | ||
| } | ||
| m := vc.vms | ||
| if gvr == vmiGVR { | ||
| m = vc.vmis | ||
| } | ||
| inf, ok := m[namespace] | ||
| if !ok || !inf.Informer().HasSynced() { | ||
| return nil, false | ||
| } | ||
| return inf, true | ||
| } | ||
|
|
||
| // stripManagedFields drops the managedFields block before an object enters | ||
| // the store; it is the largest part of a VM object and nothing here reads it. | ||
| func stripManagedFields(obj interface{}) (interface{}, error) { | ||
| if u, ok := obj.(*unstructured.Unstructured); ok { | ||
| u.SetManagedFields(nil) | ||
| } | ||
| return obj, nil | ||
| } | ||
|
|
||
| // restartVMCache replaces the running cache with one covering namespaces. | ||
| // Called with the watchers on startup and on config reload. | ||
| func (c *Client) restartVMCache(namespaces []string) { | ||
| c.cacheMu.Lock() | ||
| defer c.cacheMu.Unlock() | ||
|
|
||
| if c.vmCache != nil { | ||
| c.vmCache.stop() | ||
| c.vmCache = nil | ||
| } | ||
| if c.dynamicClient == nil { | ||
| return | ||
| } | ||
| c.vmCache = newVMCache(c.dynamicClient, namespaces) | ||
| logger.Info("VM cache started for namespaces %v", namespaces) | ||
| } | ||
|
|
||
| func (c *Client) stopVMCache() { | ||
| c.cacheMu.Lock() | ||
| defer c.cacheMu.Unlock() | ||
| if c.vmCache != nil { | ||
| c.vmCache.stop() | ||
| c.vmCache = nil | ||
| } | ||
| } | ||
|
|
||
| func (c *Client) currentVMCache() *vmCache { | ||
| c.cacheMu.RLock() | ||
| defer c.cacheMu.RUnlock() | ||
| return c.vmCache | ||
| } | ||
|
|
||
| // GetVMForRead returns a VM for a read-only purpose. It is served from the | ||
| // informer cache when the cache is synced for the namespace; otherwise it is | ||
| // a GetVM. Callers that go on to patch the VM must use GetVM. | ||
| func (c *Client) GetVMForRead(namespace, name string) (*kubevirtv1.VirtualMachine, error) { | ||
| return c.getVMForRead(namespace, name) | ||
| } | ||
|
|
||
| func (c *Client) getVMForRead(namespace, name string) (*kubevirtv1.VirtualMachine, error) { | ||
| start := time.Now() | ||
| if u, served, err := c.cachedObject(vmGVR, namespace, name); served { | ||
| c.trackOperation("GetVM.cache", time.Since(start)) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to get VM: %w", err) | ||
| } | ||
| return unstructuredToVM(u) | ||
| } | ||
| c.trackOperation("GetVM.direct", time.Since(start)) | ||
| return c.GetVM(namespace, name) | ||
| } | ||
|
|
||
| // getVMIForRead is GetVMI served from the cache under the same rules as | ||
| // getVMForRead. A synced cache's "not found" means the VM has no running | ||
| // instance, which is what callers use to report power state Off. | ||
| func (c *Client) getVMIForRead(namespace, name string) (*kubevirtv1.VirtualMachineInstance, error) { | ||
| start := time.Now() | ||
| if u, served, err := c.cachedObject(vmiGVR, namespace, name); served { | ||
| c.trackOperation("GetVMI.cache", time.Since(start)) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to get VMI %s: %w", name, err) | ||
| } | ||
| return unstructuredToVMI(u) | ||
| } | ||
| c.trackOperation("GetVMI.direct", time.Since(start)) | ||
| return c.GetVMI(namespace, name) | ||
| } | ||
|
|
||
| // cachedObject looks the object up in the cache. served is false when the | ||
| // caller must ask the API server instead: the cache is not synced for the | ||
| // namespace or the lister failed. When served, err is a NotFound error if the | ||
| // object is absent, and the returned object is a copy the caller may modify. | ||
| func (c *Client) cachedObject(gvr schema.GroupVersionResource, namespace, name string) (*unstructured.Unstructured, bool, error) { | ||
| inf, ok := c.currentVMCache().informer(gvr, namespace) | ||
| if !ok { | ||
| return nil, false, nil | ||
| } | ||
| obj, err := inf.Lister().ByNamespace(namespace).Get(name) | ||
| if apierrors.IsNotFound(err) { | ||
| return nil, true, err | ||
| } | ||
| if err != nil { | ||
| return nil, false, nil | ||
| } | ||
| u, ok := obj.(*unstructured.Unstructured) | ||
| if !ok { | ||
| return nil, false, nil | ||
| } | ||
| return u.DeepCopy(), true, nil | ||
| } | ||
|
|
||
| // listVMNamesFromCache returns the names of the VMs in namespace matching | ||
| // selector, or false when the cache is not synced for that namespace. | ||
| func (c *Client) listVMNamesFromCache(namespace string, selector labels.Selector) ([]string, bool) { | ||
| inf, ok := c.currentVMCache().informer(vmGVR, namespace) | ||
| if !ok { | ||
| return nil, false | ||
| } | ||
| objs, err := inf.Lister().ByNamespace(namespace).List(selector) | ||
| if err != nil { | ||
| return nil, false | ||
| } | ||
| names := make([]string, 0, len(objs)) | ||
| for _, obj := range objs { | ||
| if u, ok := obj.(*unstructured.Unstructured); ok { | ||
| names = append(names, u.GetName()) | ||
| } | ||
| } | ||
| c.trackOperation("ListVMs.cache", 0) | ||
| return names, true | ||
| } | ||
|
|
||
| // filterVMNames applies the selector's explicit name list, if any, and sorts | ||
| // the result so collections render in a stable order regardless of source. | ||
| func filterVMNames(names []string, vmSelector *VMSelectorConfig) []string { | ||
| var out []string | ||
| if vmSelector != nil && len(vmSelector.Names) > 0 { | ||
| allowed := make(map[string]bool, len(vmSelector.Names)) | ||
| for _, n := range vmSelector.Names { | ||
| allowed[n] = true | ||
| } | ||
| for _, n := range names { | ||
| if allowed[n] { | ||
| out = append(out, n) | ||
| } | ||
| } | ||
| } else { | ||
| out = append(out, names...) | ||
| } | ||
| sort.Strings(out) | ||
| return out | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this reimplements too much. The client-go project already has informer implementation available and reimplementing it all is not a good idea maintenance wise.
See for example https://oneuptime.com/blog/post/2026-02-09-client-go-informers-watch-resources/view (random google result).
Also I think we can make the use of informers mandatory, saving api server requests is a good idea.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or maybe not quite "reimplements", it uses the informer tooling, but it does that too complicately. I think it would be enough to just use informers for few specific CRDs and not construct a complicated dynamic cache.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, both points taken. I pushed the change as a second commit rather than amending, so the delta since your review is visible.
Mandatory cache. The
cache_enabledsetting is gone, together with its config, helm and test plumbing. Informers always run for the configured namespaces. The only fallback to a direct GET is while an informer has not finished its initial list.The complexity. What made
cache.go600 lines was not the informers but a write-tracking layer around them: reads of an object bypassed the cache until the informer had stored the resourceVersion our own write returned. It existed for one real case.handleBootUpdatewrites the VM and then reads it back through the cache to build the PATCH response, so with plain lister reads that body can show the pre-write boot options to Ironic whenever the watch event lags the write. I replaced the tracker with a live read at that one point (GetVMBootOptionsLive, one GET per PATCH) and checked that no other write path reads through a cached getter. What that trades away is the next request landing in the few milliseconds before the watch event arrives, which is the same window the PR already accepts for writes byocor virt-controller.cache.gois now the textbook shape, about 280 lines with comments: a per-namespacedynamicinformerfactory with one informer for VMs and one for VMIs,SetTransformto dropmanagedFields, listers behindHasSynced, and the restart/stop hooks tied to the existing watchers."Informers for a few specific CRDs." I read that as exactly these two, VM and VMI. They are dynamic informers because the repo has no
kubevirt.io/client-godependency and uses the dynamic client throughout. If you meant typed KubeVirt informers instead, tell me and I will switch; that would addkubevirt.io/client-gotogo.mod.