Agent-less chaos engineering for KVM/libvirt virtual machines.
kvmchaos injects reversible faults into VMs from the hypervisor, using
libvirt, tc netem and cgroup v2. Nothing is installed inside the guest. Each
fault is injected, verified, held for a set duration and then reverted
automatically, and every run leaves a JSON audit record.
Use it to test how your services handle a paused database, a lossy network, a full disk or a clock that jumped an hour, on the same KVM hosts they run on.
Warning
kvmchaos deliberately breaks VMs. Use it in labs and pre-production, set up
an allowlist, and always try --dry-run first.
- 12 faults across CPU, memory, network, disk and clock. See the fault catalogue.
- Nothing to install in the guest. Faults are applied at the hypervisor
(tap devices, cgroups, libvirt domain APIs). The one exception is
clock.skew, which needs the standardqemu-guest-agent. - Automatic revert. Reverts run on normal exit, on Ctrl-C and on
SIGTERM, and are idempotent. - Declarative experiments. Multi-step TOML scenarios, with fan-out across several VMs and optional parallel execution.
- Safety rails. VM allowlists, rate limits, confirmation prompts,
--dry-runand anabort-allemergency stop. - Observability. Structured JSON events on stderr or in a JSONL file, webhook notifications, per-run JSON records and a self-contained HTML report.
- Built for RHEL 9. Ships as a self-contained RPM for air-gapped hosts.
Every fault implements the same three-method interface, and the runner drives it through a fixed lifecycle:
flowchart LR
A[preflight<br/>allowlist · rate limit · confirm] --> B[inject]
B --> C[verify]
C --> D[hold<br/>--duration N]
D --> E[revert]
E --> F[(run record<br/>JSON)]
B -. failure / Ctrl-C / SIGTERM .-> E
Faults act on the VM's own resources on the host:
| Layer | Mechanism |
|---|---|
| CPU / lifecycle | libvirt suspend / destroy / scheduler vCPU quota |
| Memory | virtio-balloon target via libvirt |
| Network | tc netem qdisc on the VM's host tap device (no br_netfilter) |
| Disk I/O | cgroup v2 io.max on the QEMU process |
| Disk space | fill file placed next to the VM image |
| Clock | libvirt setTime, backed by qemu-guest-agent |
Each release includes a self-contained x86_64 RPM (a PyInstaller binary with no Python runtime needed), which suits air-gapped hosts:
sudo dnf install ./kvmchaos-*.el9.x86_64.rpm
kvmchaos --versionThe RPM also installs bash completion.
Prerequisites: a Linux KVM host with libvirtd running, Python 3.11+ and
uv.
sudo dnf install libvirt-devel pkg-config iproute # Fedora / RHEL
sudo usermod -aG libvirt $USER # then log out and back in
git clone https://github.com/hervad/kvmchaos.git && cd kvmchaos
uv sync
uv run kvmchaos doctorkvmchaos doctor checks for tc, the cgroup v2 mount, libvirt group
membership, libvirtd reachability and write access to the runs directory.
It exits 0 when everything passes, 1 on warnings and 2 on failures.
kvmchaos runs on the hypervisor host itself. Remote
qemu+ssh://URIs are not supported.net.*anddisk.latencyneed root (orCAP_NET_ADMIN); see docs/sudo-setup.md for a sudoers drop-in that keeps run records in your own home directory.
kvmchaos list-vms # running domains
kvmchaos list-faults # available faults
# Preview a fault: prints the plan and changes nothing
kvmchaos inject vm.pause web1 --dry-run
# Freeze a VM for 10 seconds, then resume it
kvmchaos inject vm.pause web1 --yes --duration 10
# Add 200 ms of latency to a VM's NIC for 30 seconds
sudo kvmchaos inject net.latency web1 --yes --duration 30
# Review what happened
kvmchaos runs list
kvmchaos report --output report.html| Fault | Effect | Tuning flag (default) | Root |
|---|---|---|---|
vm.pause |
Freeze all vCPUs (libvirt suspend) | — | |
vm.kill |
Hard power-off; restart on revert | — | |
vm.freeze |
Cap vCPU time to 5% via scheduler quota | — | |
vm.starve |
Shrink guest RAM via virtio-balloon | — | |
net.latency |
Add 200 ms one-way delay | — | ✓ |
net.packet-loss |
Drop a percentage of packets | --loss % (10) |
✓ |
net.bandwidth |
Cap throughput | --rate kbps (1000) |
✓ |
net.corrupt |
Corrupt a percentage of packets | --corrupt % (1) |
✓ |
net.partition |
Drop 100% of traffic | — | ✓ |
disk.latency |
Throttle disk read/write bandwidth | --bandwidth MB/s (1) |
✓ |
disk.fill |
Consume host disk space next to the image | --size MiB (1024) |
✓¹ |
clock.skew |
Shift the guest clock (negative values go backward) | --skew s (3600) |
¹ Only needs write access to the VM image directory. That usually means root,
because the default /var/lib/libvirt/images is owned by root.
Common inject flags: --duration/-d N (hold time, default 20 s),
--dry-run/-n, --yes/-y, --force (bypass safety rails) and
--config PATH.
Describe a multi-step scenario in TOML and run it with kvmchaos run:
name = "database-failover-drill"
description = "Partition the primary, then pause both replicas together."
[[step]]
fault = "net.partition"
vm = "db-primary"
duration = 60
[[step]]
fault = "vm.pause"
vms = ["db-replica-1", "db-replica-2"] # fan-out: one fault per VM
parallel = true # run fan-out concurrently
duration = 15
continue_on_failure = truekvmchaos run drill.toml --dry-run
sudo kvmchaos run drill.toml --yes --max-workers 4Steps run in order. A failed step stops the experiment unless it sets
continue_on_failure. Consecutive parallel = true steps run as one
concurrent batch. On Ctrl-C, every worker stops its hold and reverts. See
docs/examples/experiment.toml for the full
schema.
kvmchaos reads the first config file it finds: the path given with --config,
then $XDG_CONFIG_HOME/kvmchaos/config.toml, then
~/.config/kvmchaos/config.toml, then /etc/kvmchaos/config.toml. Without a
config file, no safety rails are applied:
[allowlist] # only these VMs can be targeted
vms = ["server1", "server2"]
patterns = ["staging-*"]
[rate_limit]
injects_per_hour = 20
min_interval_between_destructive_seconds = 300Targeting a VM that isn't on the allowlist exits with code 2; --force
overrides the check. Every inject asks for confirmation unless you pass
--yes.
Emergency stop. If kvmchaos was killed before it could revert (for example,
with SIGKILL), this command clears every active netem qdisc on the host:
sudo kvmchaos abort-all --yes| Output | How |
|---|---|
Structured events on stderr (JSON; works with journald and jq) |
Always on. --verbose adds DEBUG detail |
| JSONL event file | kvmchaos --json-log events.jsonl inject … |
| Webhook (Slack relay, alerting, CI) | [notifier] in config.toml |
| Per-run JSON audit record | ~/.local/state/kvmchaos/runs/ |
| HTML report | kvmchaos report --since 2026-01-01 --fault net.latency |
Events: inject.start, inject.success, inject.error, revert.success,
revert.error, experiment.start, experiment.end.
[notifier]
webhook_url = "https://hooks.example.com/kvmchaos"
auth_header = "Bearer <token>" # optional
timeout_s = 3
events = ["inject.error", "revert.error"] # omit for all eventskvmchaos runs list and kvmchaos report both accept --since, --fault,
--outcome and --vm filters. kvmchaos runs show <id-prefix> prints a
single run record.
docs/recipes/ contains worked scenarios, each with prerequisites, commands and the expected outcome:
| RHEL 9 | RHEL 8 | Notes | |
|---|---|---|---|
vm.*, net.* |
✅ | ✅ | net.* uses tc netem; no nftables or br_netfilter needed |
disk.latency |
✅ | Requires cgroup v2. RHEL 8 needs systemd.unified_cgroup_hierarchy=1 |
|
disk.fill |
✅ | ✅ | Needs write access to the image directory |
clock.skew |
✅ | ✅ | Needs qemu-guest-agent in the guest |
Primary target is RHEL 9. Development and lab validation are done on Fedora 43 with RHEL 9.x guests.
Start with kvmchaos doctor, which catches most setup problems.
| Symptom | Cause | Fix |
|---|---|---|
Operation not permitted on net.* / disk.latency |
Needs root / CAP_NET_ADMIN |
Run with sudo (see sudo setup) |
failed to connect to the hypervisor |
User not in libvirt group |
sudo usermod -aG libvirt $USER, then log out and back in |
tc: command not found |
iproute2 missing | sudo dnf install iproute |
no network interface found for domain |
VM not running or has no vNIC | virsh start <vm> / virsh domiflist <vm> |
cgroups v2 hierarchy not found |
Host on cgroup v1 | Enable cgroup v2, or skip disk.latency |
clock.skew fails |
Guest agent missing | Install and start qemu-guest-agent in the guest |
Run records end up in /root |
SUDO_USER not resolved |
Set XDG_STATE_HOME (see sudo setup) |
kvmchaos: command not found under sudo |
$PATH not preserved |
Use the full path, e.g. sudo ~/kvmchaos/.venv/bin/kvmchaos |
netem qdisc left behind after a crash |
Revert never ran | sudo kvmchaos abort-all --yes |
uv sync
uv run pytest # tests + coverage report
uv run ruff check && uv run ruff format --check- Tests run against libvirt's built-in
test:///defaultdriver, which provides realvirDomainobjects and needs neither root nor QEMU. 450+ tests, about 96% coverage. - Every fault is also lab-validated on a real KVM host before release.
- CI runs lint, format and tests on every push. Tagged releases build the sdist, wheel and RHEL 9 RPM.
- Runtime dependencies are deliberately limited to
libvirt-pythonandtyper.
Design notes and the roadmap are in PLAN.md and docs/superpowers/specs/. Release history is in CHANGELOG.md.
MIT © Vadym Herman