KVM Testbed Setup

This document describes the steps to setup a virtual switch based testbed, deploy a T0 topology to it, and run a quick test to verify it is working as expected.

Prepare testbed host

First, we need to prepare the host where we will be configuring the virtual testbed and running the tests.

Install Ubuntu AMD64 on your host or VM
- To setup a T0 topology, the server needs to have at least 20GB of memory free
- If the testbed host is a VM, then it must support nested virtualization
  - Instructions for Hyper-V based VMs
Prepare your environment on Ubuntu 22.04 or Ubuntu 24.04, and make sure that Python and pip are installed:
```
sudo apt update
sudo apt install -y python3 python3-pip openssh-server
```
Run the host setup script to install required packages and initialize the management bridge network

git clone https://github.com/sonic-net/sonic-mgmt
cd sonic-mgmt/ansible
sudo -H ./setup-management-network.sh

Install Docker CE. Be sure to follow the post-install instructions so that you don't need sudo privileges to run docker commands.

Download a VM image

We currently support EOS-based or SONiC VMs to simulate neighboring devices in the virtual testbed, much like we do for physical testbeds. To do so, we need to download the image to our testbed host.

Prepare folder for image files on testbed host

The location for storing image files on the testbed host is specified by the root_path variable in the ansible/group_vars/vm_host/main.yml file. Please update this variable to reflect the location you have planned for the testbed host. You can use either an absolute path or a relative path. If you choose a relative path, it will be relative to the home directory of the user accessing the testbed host.

For example, if root_path is set to veos-vm, the image location would be /home/$USER/veos-vm/images/. If root_path is set to /data/veos-vm, the image location would be /data/veos-vm/images.

As you may have noticed, image files are usually stored under subfolder images of location determined by root_path.

Example 1:

root_path: veos-vm

Example 2:

root_path: /data/veos-vm

Create a subfolder called images inside the root_path directory defined in ansible/group_vars/vm_host/main.yml file. For instance, if root_path is set to veos-vm, you should run the following command:

mkdir -p ~/veos-vm/images

Option 1: vEOS (KVM-based) image

Download the vEOS image from Arista
Place below image files in the images subfolder located within the directory specified by the root_path variable in the ansible/group_vars/vm_host/main.yml file.
- Aboot-veos-serial-8.0.0.iso
- vEOS-lab-4.20.15M.vmdk
Update ansible/group_vars/vm_host/veos.yml if you decided to use different veos image files from above.

Option 2: cEOS (container-based) image (recommended)

Option 2.1: Manually download cEOS image

Obtain the cEOS image from Arista's software download page. You can choose later cEOS versions, but they are not guaranteed to work (the latest 4.35.0F does not).

Note: You may need to register an Arista guest account to access the download resources.

Ensure that the cEOS version you download matches the version specified in ansible/group_vars/vm_host/ceos.yml. For example, the following steps use cEOS64-lab-4.29.3M as a reference.
Unxz it with unxz cEOS64-lab-4.29.3M.tar.xz.
Place the image file in the images subfolder located within the directory specified by the root_path variable in the ansible/group_vars/vm_host/main.yml file.

Assuming you set root_path to veos-vm, you should run the following command:
```
cp cEOS64-lab-4.29.3M.tar ~/veos-vm/images/
```
The Ansible playbook for deploying testbed topology will automatically use the manually prepared image file from this location.
Update ansible/group_vars/vm_host/ceos.yml if you decided to use different ceos image files from above.

Option 2.2: Host the cEOS image file on an HTTP server

If you need to deploy VS setup on multiple testbed hosts, this option is more recommended.

Download the cEOS Image

Obtain the cEOS image from Arista's software download page.
Host the cEOS Image

Host the cEOS image file on an HTTP server. Ensure that the image file is accessible via HTTP from the sonic-mgmt container running the testbed deployment code. For example, the URL might look like http://192.168.1.10/cEOS64-lab-4.29.3M.tar.
Update the Ansible Configuration

Update the ceos_image_url variable in ansible/group_vars/all/ceos.yml with the URL of the cEOS image. This variable can be a single string for one URL or a list of strings for multiple URLs.

The Ansible playbook will attempt to download the image from each URL in the list until it succeeds. The downloaded file is stored in the images subfolder of the location determined by the root_path variable in ansible/group_vars/vm_host/main.yml. For example, if root_path is /data/veos-vm, then the downloaded image file is placed in /data/veos-vm/images

Variable skip_ceos_image_downloading in ansible/group_vars/all/ceos.yml also must be set to false if you wish the Ansible playbook to automatically try downloading the cEOS image file. For example:
```
ceos_image_url: http://192.168.1.10/cEOS64-lab-4.29.3M.tar
skip_ceos_image_downloading: false
```
Or:
```
ceos_image_url:
   - http://192.168.1.10/cEOS64-lab-4.29.3M.tar
skip_ceos_image_downloading: false
```
Update ansible/group_vars/vm_host/ceos.yml if you decided to use different ceos image files from above.

Option 3: Use SONiC image as neighboring devices

You need to create a valid SONiC image named sonic-vs.img in the ~/veos-vm/images/ directory. Currently, we don't support downloading a pre-built SONiC image. However, for testing purposes, you can refer to the Download the sonic-vs image section to obtain an available image and place it in the ~/veos-vm/images/ directory.

Download the sonic-vs image

1. To run the tests with a virtual SONiC device, we need a virtual SONiC image.

Option 1: Download sonic-vs image

The simplest way to do so is to download the latest successful build. Download the sonic-vs image from here

wget "https://sonic-build.azurewebsites.net/api/sonic/artifacts?branchName=master&platform=vs&target=target/sonic-vs.img.gz" -O sonic-vs.img.gz

Option 2: Build sonic-vpp image

Follow the instructions from sonic-platform-vpp and build a KVM VM image.

2. Unzip the image and copy it into `~/sonic-vm/images/` and `~/veos-vm/images/`

vs image

gzip -d sonic-vs.img.gz
mkdir -p ~/sonic-vm/images
cp sonic-vs.img ~/sonic-vm/images
mkdir -p ~/veos-vm/images
mv sonic-vs.img ~/veos-vm/images

vpp image

gzip -d sonic-vpp.img.gz
mkdir -p ~/sonic-vm/images
cp sonic-vpp.img ~/sonic-vm/images
mkdir -p ~/veos-vm/images
mv sonic-vpp.img ~/veos-vm/images

Setup sonic-mgmt Docker

All testbed configuration steps and tests are run from a sonic-mgmt Docker container. This container has all the necessary packages and tools for SONiC testing so that test behavior is consistent between different developers and lab setups.

Run the setup-container.sh in the root directory of the sonic-mgmt repository:

cd sonic-mgmt
./setup-container.sh -n <container name> -d /data

(Required for IPv6 test cases): Follow the steps in IPv6 for docker default bridge to enable IPv6 for the container. For example, edit the Docker daemon configuration file located at /etc/docker/daemon.json with the following parameters to use a ULA address if there is no special requirement. Then restart the Docker daemon by running sudo systemctl restart docker for the change to take effect.

{
    "ipv6": true,
    "fixed-cidr-v6": "fd00:1::1/64",
    "experimental": true,
    "ip6tables": true
}

From now on, all steps run inside the sonic-mgmt Docker container, unless otherwise specified.

You can enter your sonic-mgmt container with the following command:

docker exec --user $USER -it <container name> bash

You will find your sonic-mgmt directory mounted at /data/sonic-mgmt:

$ ls /data/sonic-mgmt/
LICENSE  README.md  __pycache__  ansible  docs	lgtm.yml  setup-container.sh  spytest  test_reporting  tests

Setup host public key in sonic-mgmt Docker

In order to configure the testbed on your host automatically, Ansible needs to be able to SSH into it without a password prompt. The setup-container script from the previous step will setup all the necessary SSH keys for you, but there are a few more modifications needed to make Ansible work:

Modify /data/sonic-mgmt/ansible/veos_vtb to use the user name (e.g. foo) you want to use to login to the host machine (this can be your username on the host)

     STR-ACS-VSERV-01:
       ansible_host: 172.17.0.1
       ansible_user: foo
       vm_host_user: use_own_value  // you can leave it as is

Modify /data/sonic-mgmt/ansible/group_vars/vm_host/creds.yml to use the username (e.g. foo) and password (e.g. foo123) you want to use to login to the host machine. This can be your username and sudo password on the host, and you might not need to set the password if your host machine is accessed with an SSH key and needs no further password for sudo. For more information about credentials variables, see: credentials management configuration.

vm_host_user: foo
vm_host_password: foo123
vm_host_become_password: foo123

Note: If the above two modifications are done correctly, use the git diff command and it will show output similar to the following:

foo@sonic:/data/sonic-mgmt/ansible$ git diff
diff --git a/ansible/group_vars/vm_host/creds.yml b/ansible/group_vars/vm_host/creds.yml
index 029ab9a6..e00d3852 100644
--- a/ansible/group_vars/vm_host/creds.yml
+++ b/ansible/group_vars/vm_host/creds.yml
@@ -1,4 +1,4 @@
-vm_host_user: use_own_value
-vm_host_password: use_own_value
-vm_host_become_password: use_own_value
+vm_host_user: foo
+vm_host_password: foo123
+vm_host_become_password: foo123

diff --git a/ansible/veos_vtb b/ansible/veos_vtb
index 99727bcf3..2a9c36006 100644
--- a/ansible/veos_vtb
+++ b/ansible/veos_vtb
@@ -274,7 +274,7 @@ vm_host_1:
   hosts:
     STR-ACS-VSERV-01:
       ansible_host: 172.17.0.1
-      ansible_user: use_own_value
+      ansible_user: foo
       vm_host_user: use_own_value

 vms_1:

Create a dummy password.txt file under /data/sonic-mgmt/ansible
- Note: Here, password.txt is the Ansible Vault password file. Ansible allows users to use Ansible Vault to encrypt password files.
  
  By default, the testbed scripts require a password file. If you are not using Ansible Vault, you can create a file with a dummy password (e.g. abc) and pass the filename to the command line. The file name and location are created and maintained by the user.
On the host, run sudo visudo and add the following line at the end:

foo ALL=(ALL) NOPASSWD:ALL

Verify that you can log in to the host (e.g. ssh foo@172.17.0.1, if the default Docker bridge IP is 172.18.0.1/16, follow https://docs.docker.com/network/bridge/#configure-the-default-bridge-network to change it to 172.17.0.1/16, delete the current sonic-mgmt Docker container using the command docker rm -f <sonic-mgmt_container_name>, then start over from step 1 of the Setup sonic-mgmt Docker section) from the sonic-mgmt container without a password prompt.
(Required for IPv6 test cases) Verify that you can log in to the host via IPv6 (e.g. ssh foo@fd00:1::1 if the default Docker bridge is fd00:1::1/64) from the sonic-mgmt container without a password prompt.
Verify that you can use sudo without a password prompt inside the host (e.g. sudo bash).
On the host, verify that your home directory has the correct permissions (755) by running:
```
sudo chmod 755 /home/<username>
```
Also verify that image files and the folder containing them have the correct permissions.

Setup VMs on the server

(Skip this step if you are using cEOS - the containers will be automatically setup in a later step.)

Now we need to spin up some VMs on the host to act as neighboring devices to our virtual SONiC switch.

Start the VMs:

./testbed-cli.sh -m veos_vtb -n 4 -k veos start-vms server_1 password.txt

If you use a SONiC image as the neighbor devices (Not DUT), you need to add the extra parameter -k vsonic so that this command is ./testbed-cli.sh -m veos_vtb -n 4 -k vsonic start-vms server_1 password.txt. If you want to stop VMs, you also need to append this parameter to the original command.

Reminder: By default, this shell script requires a password file. If you are not using Ansible Vault, just create a file with a dummy password and pass the filename to the command line.

Check that all VMs are up and running. For the EOS-based VMs Note: The password is 123456.

$ ansible -m ping -i veos_vtb server_1 -u root -k
VM0102 | SUCCESS => {
        "changed": false,
                "ping": "pong"
}
VM0101 | SUCCESS => {
        "changed": false,
                "ping": "pong"
}
STR-ACS-VSERV-01 | SUCCESS => {
        "changed": false,
                "ping": "pong"
}
VM0103 | SUCCESS => {
        "changed": false,
                "ping": "pong"
}
VM0100 | SUCCESS => {
        "changed": false,
                "ping": "pong"
}

For the SONiC VMs Note: The password is password.

$ ansible -m ping -i veos_vtb server_1 -u admin -k
VM0102 | SUCCESS => {
        "changed": false,
                "ping": "pong"
}
VM0101 | SUCCESS => {
        "changed": false,
                "ping": "pong"
}
STR-ACS-VSERV-01 | SUCCESS => {
        "changed": false,
                "ping": "pong"
}
VM0103 | SUCCESS => {
        "changed": false,
                "ping": "pong"
}
VM0100 | SUCCESS => {
        "changed": false,
                "ping": "pong"
}

SONiC-VPP testbeds

If you are setting up a SONiC-VPP testbed, follow Running SONiC-mgmt with SONiC-VPP as DUT instead of the T0 deployment steps below.

Deploy T0 topology

Now we're finally ready to deploy the topology for our testbed! Run the following command, depending on what type of EOS image you are using for your setup:

vEOS

cd /data/sonic-mgmt/ansible
./testbed-cli.sh -t vtestbed.yaml -m veos_vtb -k veos add-topo vms-kvm-t0 password.txt

cEOS

cd /data/sonic-mgmt/ansible
./testbed-cli.sh -t vtestbed.yaml -m veos_vtb add-topo vms-kvm-t0 password.txt

Command add-topo above defaults to ceos if you do not provide -k.

If you see something like cached_topologies_path file content is empty, that does not mean the add-topo is not successful.

Verify that the cEOS neighbors were created properly:

$ docker ps
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED              STATUS              PORTS               NAMES
575064498cbc        ceosimage:4.23.2F                                     "/sbin/init systemd.…"   About a minute ago   Up About a minute                       ceos_vms6-1_VM0103
d71b8970bcbb        debian:jessie                                         "bash"                   About a minute ago   Up About a minute                       net_vms6-1_VM0103
3d2e5ecdd472        ceosimage:4.23.2F                                     "/sbin/init systemd.…"   About a minute ago   Up About a minute                       ceos_vms6-1_VM0102
28d64c74fa54        debian:jessie                                         "bash"                   About a minute ago   Up About a minute                       net_vms6-1_VM0102
0fa067a47c7f        ceosimage:4.23.2F                                     "/sbin/init systemd.…"   About a minute ago   Up About a minute                       ceos_vms6-1_VM0101
47066451fa4c        debian:jessie                                         "bash"                   About a minute ago   Up About a minute                       net_vms6-1_VM0101
e07bd0245bd9        ceosimage:4.23.2F                                     "/sbin/init systemd.…"   About a minute ago   Up About a minute                       ceos_vms6-1_VM0100
4584820bf368        debian:jessie                                         "bash"                   7 minutes ago        Up 7 minutes                            net_vms6-1_VM0100
c929c622232a        sonicdev-microsoft.azurecr.io:443/docker-ptf:latest   "/usr/local/bin/supe…"   7 minutes ago        Up 7 minutes                            ptf_vms6-1

vSONiC

cd /data/sonic-mgmt/ansible
./testbed-cli.sh -t vtestbed.yaml -m veos_vtb -k vsonic add-topo vms-kvm-t0 password.txt

For vtestbed, add-topo/remove-topo also takes care of creating/removing the KVM DUT. The old KVM DUT will be removed and recreated if it was present prior to add-topo.

Deploy minigraph on the DUT

Once the topology has been created, we need to give the DUT an initial configuration.

(Optional) Connectivity to the public internet is necessary during the setup. If the lab environment of your organization requires an HTTP/HTTPS proxy server to reach out to the internet, you need to configure the proxy server. It will automatically be leveraged in the required steps (e.g. Docker daemon config for image pulling, APT configuration for installing packages). You can configure it in ansible/group_vars/all/env.yml

Deploy the minigraph.xml to the DUT and save the configuration:

./testbed-cli.sh -t vtestbed.yaml -m veos_vtb deploy-mg vms-kvm-t0 veos_vtb password.txt

IPv6-Only Management Network (Optional)

If you want to configure the DUT with IPv6-only management (no IPv4 on the management interface), use the --ipv6-only-mgmt flag:

Set up the local NTP server (required for IPv6-only management):
```
./setup-ntp-server.sh start
```
This deploys a Chrony NTP server container on the management network that DUTs can reach via IPv6.

Generate minigraph with IPv6-only management:

./testbed-cli.sh -t vtestbed.yaml -m veos_vtb gen-mg vms-kvm-t0 veos_vtb password.txt --ipv6-only-mgmt

Deploy minigraph with IPv6-only management:

./testbed-cli.sh -t vtestbed.yaml -m veos_vtb deploy-mg vms-kvm-t0 veos_vtb password.txt --ipv6-only-mgmt

Access the DUT via IPv6:
```
ssh admin@fec0::ffff:afa:1
```
The IPv6 address is defined as ansible_hostv6 in the inventory file (e.g., veos_vtb).

For detailed information about IPv6-only management setup, including configuration options, NTP server setup, and troubleshooting, see IPv6 Management Setup Guide.

Verify the DUT is created successfully. On your host, run:

~$ virsh list
 Id   Name      State
-------------------------
 3    vlab-01   running

It's good if you can see it, but if you don't, you can further verify if there is a qemu process running.

Then you can try to login to your DUT through the command and get logged in as shown below. For more information about how to get the DUT IP address, please refer to doc testbed.Example#access-the-dut

~$ ssh admin@10.250.0.101
admin@10.250.0.101's password:
Linux vlab-01 4.19.0-12-2-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64
You are on
 ____   ___  _   _ _  ____
/ ___| / _ \| \ | (_)/ ___|
\___ \| | | |  \| | | |
 ___) | |_| | |\  | | |___
|____/ \___/|_| \_|_|\____|

-- Software for Open Networking in the Cloud --

Unauthorized access and/or use are prohibited.
All access and/or use are subject to monitoring.

Help:    http://azure.github.io/SONiC/

Last login: Thu Jul 29 03:55:53 2021 from 10.250.0.1
admin@vlab-01:~$ exit

Verify that you can login to the SONiC KVM using Mgmt IP = 10.250.0.101 and admin:password.

ssh admin@10.250.0.101
admin@10.250.0.101's password: password

After logging in to the SONiC KVM, you should be able to see BGP sessions with:

show ip bgp sum

If neighbor devices are EOS:

admin@vlab-01:~$ show ip bgp sum

IPv4 Unicast Summary:
BGP router identifier 10.1.0.32, local AS number 65100 vrf-id 0
BGP table version 6405
RIB entries 12807, using 2458944 bytes of memory
Peers 4, using 87264 KiB of memory
Peer groups 4, using 256 bytes of memory


Neighbhor      V     AS    MsgRcvd    MsgSent    TblVer    InQ    OutQ  Up/Down      State/PfxRcd  NeighborName
-----------  ---  -----  ---------  ---------  --------  -----  ------  ---------  --------------  --------------
10.0.0.57      4  64600       3792       3792         0      0       0  00:29:24             6400  ARISTA01T1
10.0.0.59      4  64600       3792       3795         0      0       0  00:29:24             6400  ARISTA02T1
10.0.0.61      4  64600       3792       3792         0      0       0  00:29:24             6400  ARISTA03T1
10.0.0.63      4  64600       3795       3796         0      0       0  00:29:24             6400  ARISTA04T1

Total number of neighbors 4

If neighbor devices are SONiC:

admin@vlab-01:~$ show ip bgp sum

IPv4 Unicast Summary:
BGP router identifier 10.1.0.32, local AS number 65100 vrf-id 0
BGP table version 3
RIB entries 5, using 920 bytes of memory
Peers 4, using 83680 KiB of memory
Peer groups 4, using 256 bytes of memory


Neighbhor      V     AS    MsgRcvd    MsgSent    TblVer    InQ    OutQ  Up/Down    State/PfxRcd    NeighborName
-----------  ---  -----  ---------  ---------  --------  -----  ------  ---------  --------------  --------------
10.0.0.57      4  64600          8          8         0      0       0  00:00:10   3               ARISTA01T1
10.0.0.59      4  64600          0          0         0      0       0  00:00:10   3               ARISTA02T1
10.0.0.61      4  64600          0          0         0      0       0  00:00:11   3               ARISTA03T1
10.0.0.63      4  64600          0          0         0      0       0  00:00:11   3               ARISTA04T1

Run a pytest

Now that the testbed has been fully setup and configured, let's run a simple test to make sure everything is functioning as expected.

Switch over to the tests directory:

cd sonic-mgmt/tests

Run the following command to execute the bgp_fact test (including the pre/post setup steps):

If neighbor devices are EOS

./run_tests.sh -n vms-kvm-t0 -d vlab-01 -c bgp/test_bgp_fact.py -f vtestbed.yaml -i ../ansible/veos_vtb

If neighbor devices are SONiC

./run_tests.sh -n vms-kvm-t0 -d vlab-01 -c bgp/test_bgp_fact.py -f vtestbed.yaml -i ../ansible/veos_vtb -e "--neighbor_type=sonic"

You should see three sets of tests run and pass. You're now set up and ready to use the KVM testbed!

Running Tests with IPv6-Only Management

If you deployed the DUT with --ipv6-only-mgmt (see IPv6-Only Management Network), you must run tests with the -6 flag:

./run_tests.sh -6 -n vms-kvm-t0 -d vlab-01 -c bgp/test_bgp_fact.py -f vtestbed.yaml -i ../ansible/veos_vtb

The -6 flag tells the test framework to:

Use the IPv6 address (ansible_hostv6) as the DUT management IP
Skip IPv4 management connectivity checks
Use ping6 for reachability tests

Alternatively, you can use pytest directly with the --ipv6_only_mgmt option:

pytest --ipv6_only_mgmt --testbed vms-kvm-t0 --testbed_file vtestbed.yaml --inventory ../ansible/veos_vtb --host-pattern vlab-01 bgp/test_bgp_fact.py

For more details, see IPv6 Management Setup Guide.

Restore/Remove the testing environment

If you want to clear your testing environment, you can log in to the sonic-mgmt Docker container that you created in the Setup sonic-mgmt Docker section.

Then run the command:

./testbed-cli.sh -t vtestbed.yaml -m veos_vtb -k ceos remove-topo vms-kvm-t0 password.txt

This will clean up the PTF container, cEOS container, and KVM DUT. The -k option defaults to ceos, but you can provide veos or vsonic.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KVM Testbed Setup

Prepare testbed host

Download a VM image

Option 1: vEOS (KVM-based) image

Option 2: cEOS (container-based) image (recommended)

Option 2.1: Manually download cEOS image

Option 2.2: Host the cEOS image file on an HTTP server

Option 3: Use SONiC image as neighboring devices

Download the sonic-vs image

1. To run the tests with a virtual SONiC device, we need a virtual SONiC image.

Option 1: Download sonic-vs image

Option 2: Build sonic-vpp image

2. Unzip the image and copy it into `~/sonic-vm/images/` and `~/veos-vm/images/`

Setup sonic-mgmt Docker

Setup host public key in sonic-mgmt Docker

Setup VMs on the server

SONiC-VPP testbeds

Deploy T0 topology

vEOS

cEOS

vSONiC

Deploy minigraph on the DUT

IPv6-Only Management Network (Optional)

Run a pytest

Running Tests with IPv6-Only Management

Restore/Remove the testing environment

FilesExpand file tree

README.testbed.VsSetup.md

Latest commit

History

README.testbed.VsSetup.md

File metadata and controls

KVM Testbed Setup

Prepare testbed host

Download a VM image

Option 1: vEOS (KVM-based) image

Option 2: cEOS (container-based) image (recommended)

Option 2.1: Manually download cEOS image

Option 2.2: Host the cEOS image file on an HTTP server

Option 3: Use SONiC image as neighboring devices

Download the sonic-vs image

1. To run the tests with a virtual SONiC device, we need a virtual SONiC image.

Option 1: Download sonic-vs image

Option 2: Build sonic-vpp image

2. Unzip the image and copy it into ~/sonic-vm/images/ and ~/veos-vm/images/

Setup sonic-mgmt Docker

Setup host public key in sonic-mgmt Docker

Setup VMs on the server

SONiC-VPP testbeds

Deploy T0 topology

vEOS

cEOS

vSONiC

Deploy minigraph on the DUT

IPv6-Only Management Network (Optional)

Run a pytest

Running Tests with IPv6-Only Management

Restore/Remove the testing environment

2. Unzip the image and copy it into `~/sonic-vm/images/` and `~/veos-vm/images/`