[itayherzberg]

Problem

Background jobs (virtual-media insert/eject and the other async Redfish tasks) never run in parallel. Eight InsertMedia requests sent at the same moment are processed one after another by a single worker while the other three sit idle, and the pool size (4) is hardcoded.

Cause

TaskManager.getAvailableWorker scans the pool from the first worker and returns the first one whose job channel has capacity. Each channel holds 10 jobs, so worker 1 always has room and receives every job; workers 2–4 never see one.

Unit test added by this PR, run against current main (4 workers, 8 picks):

--- FAIL: TestGetAvailableWorkerRoundRobin (0.00s)
    task_manager_test.go:43: worker 2 was picked 0 times, want 2 (picks per worker: map[1:8])
    task_manager_test.go:43: worker 3 was picked 0 times, want 2 (picks per worker: map[1:8])
    task_manager_test.go:43: worker 4 was picked 0 times, want 2 (picks per worker: map[1:8])

On a cluster (upstream b40c637, OpenShift 4.22 / OpenShift Virtualization 4.22.12), 8 InsertMedia POSTs sent at once: all 8 jobs ran on Worker 1, completing one at a time 1.2–1.9 s apart; the batch took 10.3 s and the import pods finished 21–24 s after the POSTs.

Fix

• Round-robin start. An atomic cursor picks a different starting worker for each job; the scan then moves forward to the first worker with capacity, so a full queue still falls through to the next worker. One sync/atomic counter, no new locking.
• Configurable pool. server.workers (KUBEVIRT_REDFISH_SERVER_WORKERS) replaces the hardcoded 4: default 8, allowed 1–64, zero means default. Read once at startup; NewServer logs Task manager started with N workers.
• Chart: server.workers value (default 8), rendered into the config when set.

Behaviour change: the default pool grows from 4 to 8 workers. Each job is a short burst of Kubernetes API calls (create the PVC and the copy pod; the ISO copy itself runs inside that pod), so the pool size bounds how many inserts start at once, not how many ISOs copy at once.

Verification on the cluster

This branch, defaults, 8 InsertMedia POSTs sent at once:

Started 8 workers
Task manager started with 8 workers
16:38:31.220  Worker 4 processing job job-…205856137
16:38:31.220  Worker 2 processing job job-…186810467
16:38:31.220  Worker 3 processing job job-…187717787
16:38:33.635  Worker 5 processing job job-…579434721
16:38:34.037  Worker 6 processing job job-…979878074
16:38:34.239  Worker 7 processing job job-…178478314
16:38:34.440  Worker 8 processing job job-…379058911
16:38:34.641  Worker 1 processing job job-…579453827

8 distinct workers, jobs overlapping (workers 2, 3 and 4 started in the same millisecond). All 8 drives reported Inserted: true within 20 s of the POSTs; ejecting all 8 in parallel returned 200 each, Inserted: false on every drive, and no PVCs or pods were left behind.

KUBEVIRT_REDFISH_SERVER_WORKERS=3 → Started 3 workers / Task manager started with 3 workers.
KUBEVIRT_REDFISH_SERVER_WORKERS=65 → Failed to load configuration: … Invalid worker count (server.workers must be between 1 and 64, got 65).

Tests

• pkg/server/task_manager_test.go: TestGetAvailableWorkerRoundRobin (fails on current main as shown above, passes with the fix).
• pkg/config/config_test.go: default 8; value from the file and from the environment; out-of-range rejected; zero means default.
• go build ./..., go vet ./..., go test ./..., gofmt, go mod tidy (no change) and helm lint pass.

Related

One of four independent PRs from running many concurrent BMHs against kubevirt-redfish; each stands on its own and they can merge in any order: #30 (client-go rate limit), #32 (VM/VMI informer cache), #29 (shutdown hang). The three config PRs add the same two test helpers (minimalConfig, writeConfig) to pkg/config/config_test.go; whichever lands later drops its copy on rebase.

[kubevirt-prow]

Hi @itayherzberg. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[itayherzberg]

@MarSik Gentle ping on this when you have a chance! Let me know if you need any additional context

[MarSik]

/lgtm
/approved

[MarSik]

/approve

[kubevirt-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: MarSik

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [MarSik]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment