XEN and KVM in

INFN production systems and a comparison between them

Riccardo Veraldi Andrea Chierici INFN - CNAF HEPiX Spring 2009

Outline
xen kvm Test description Benchmarks Conclusions

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

What we have now: Xen

Open source industry standard for virtualization Included in all recent linux distributions Supports para and hvm approaches Widely used in HEP community Wont go into details during presentation

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

What could be the future: KVM

Kernel-based Virtual Machine Open source included in latest linux kernels: implemented as a module
A

user space program uses /dev/kvm interface to set up VMs (qemu-kvm)

Supports hvm approach Rather new to HEP community Qumranet now owned by Red Hat
Foreseen

boost in development
4

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

KVM virtio

Direct access to hardware

Hard

disk (/dev/vdx) Ethernet interface

Support from kernel 2.6.25 Supported since RHEL 5.3 (virtio backport)
kernel

2.6.18-128.1.1 Virtio drivers must be on the VM, not on the host EGEE sw cant currently benefit from this

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

KVM seen by system admin

rpms: basically the kernel module and a modified version of qemu (qemu-kvm) Network configuration to be done by hand in order to get public IP

bridge-utils and tunctl rpms were used to set-up tap interface

Script developed to automate the process

A modified init.d script to configure software bridge has been developed at CNAF

No VM configuration file direct support

VMs are launched via a standard UNIX command + command line options libvirt

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

KVM seen by system admin

No direct interface to manage VMs by command line (e.g. xm)

Libvirt

support

Virt-manager working Monitor interface

Very

powerful, lots of options available

qemu already well documented, support available on-line

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 7

KVM introduced at CNAF

We use quattor to configure and install grid nodes and virtual machines Profiles describe machine configuration Network boot for installation working Keeps booting from network, need to restart with -boot c No need for any modification, just like xenhvm
Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 8

Qualitative test

CNAF is running LHCb tier2 site entirely on xen VMs (2 CEs, 1 SE) Changed one Computing Element with a KVM machine

Has been working for more that 3 weeks flawlessly Quattor machine profile unmodified, no effort for sysadmin

CMS secondary squid server installed on the same host with same result (2 weeks ago) Running a KVM in production since 2 months for INFN-CA backups. Works flawlessly KVM executed (and live migrated!) a win7 VM Hardware used: 1 node, dual E5420, 16GB ram, sata disks via Areca controller

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

Quantitative test: description

Need some measures to understand what is the best solution Tested 3 classic parameters
CPU

hep-spec06 (v1.1), PI test for cpu distribution Network iperf (v2.0.4) Disk access bonnie++ (v1.94)

Compared Xen (para-virtualized and hvm) with KVM, using non virtualized machine as a baseline
Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 10

Quantitative test: description

Hardware used: 1 blade, dual E5420, 16GB ram, 10k sas disk via LSI logic raid controller (raid0) Xen-para VM specs: 1 vcpu, 2 GB ram, disk on a file Xen-hvm VM specs: 1 vcpu, 2GB ram, disk on a file, netfront network
driver emulation (non-virtio tests)

KVM VM specs: 1 vcpu, 2GB ram, disk on a file, e1000 network driver Host OS: SL 5.2 x86_64, kernel 2.6.18-92.1.22.el5 VM OS: SLC 4.5 i386, kernel 2.6.9-67.0.15.EL.cern VM OS: SLC 5.3 x86_64, kernel 2.6.18-128.1.1 for kvm virtio tests KVM version: 83 Xen version: 3.2.1

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

11

Benchmarks: HEP-Spec06
XENvs.KVMondualIntelE5420,singleperformancemeasure
11 10.5

10

HEPSpec06

9.5

8.5

7.5 1 4 7 8

NumberofconcurrentVMs
Xenhvm Xenpara KVM E5420

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

12

Benchmarks: HEP-Spec06
VMsvs.CPU
70 60 50

HEPSpec06

40 30 20 10 0 1vm kvm 4vm xenhvm xenpara 7vm E5420 8vm

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

13

Benchmarks: HEP-Spec06
8VMsaggregatevs.CPUs
70 60 50

HEPSpec06

40 30 20 10 0 E5410 E5420 E5420kvm E5420xenhvm E5420xenpara E5430

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

14

Benchmarks: HEP-Spec06
Virtualization Technology E5420kvm E5420xen-hvm E5420xen-para E5410 vs. E5420 % loss from non emulated CPU (E5420, 8vm) 3,42 4,55 2,02 4,07
15

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

Benchmarks: PI test

System tools to measure CPU usage are not reliable enough Calculating PI with n arbitrary ciphers
Algorithm based on Numerical Recipes Written in C Compiled with options

gcc -O3 -ffast-math -finline-limit=1000

Returns the number of CPU cycles

An external function included into the code calculates the number of CPU cycles, using rdtsc IA32 registry

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

16

Benchmarks: PI test
KVMhostvs.guest
3.1E+12 3E+12 2.9E+12 2.8E+12

CPUcycles

2.7E+12 2.6E+12 2.5E+12 2.4E+12 2.3E+12 2.2E+12 2.1E+12

difference < 2%

host

guest

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

17

Benchmarks: PI test
8concurrentguest
3.1E+12 3E+12 2.9E+12 2.8E+12

CPUcycles

2.7E+12 2.6E+12 2.5E+12 2.4E+12 2.3E+12 2.2E+12 2.1E+12 guest singolo guest1 guest2 guest3 guest4 guest5 guest6 guest7 guest8

8 core Balanced Performance Every guest uses one core at 100%

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 18

Benchmarks: PI test
17concurrentguests
7E+12 6E+12 5E+12 CPUcycles 4E+12 3E+12 2E+12 1E+12

Good balancing on guest Max difference 14%, average ~5%

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 19

Benchmarks: Iperf
KVMNetwork Performance,e1000
1000 900 800 700

Mbits/sec

600 500 400 300 200 100 0 1vm 2vm 4vm 8vm host

in out

iperfw256kP5t900

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

20

Benchmarks: Iperf
KVMvsXEN,1vm,networkperformance
1000 900 800 700

Mbits/sec

600 500 400 300 200 100 0 kvme1000 kvmvirtio xenpara xenhvmpv in out

iperfw256kP5t900

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

21

Benchmarks: bonnie++, file vs lvm

2GBRam,4GBdataset,1vmcomparison
160000 140000 120000 100000

K/sec

80000 60000 40000 20000 0 Host 1kvmfile 1kvmlvm 1xenparafile 1xenparalvm 1xenhvmfile 1xenhvmlvm

bonnie++onasingleVM
seqoutblock seqoutrewrite seqinblock

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

22

Benchmarks: bonnie++
2GBram,4GBdataset,single
120000 100000

80000

60000

40000

20000

0 1kvmvm 4kvmvm 8kvmvm 1xenpara 4xenpara seqoutrewrite 8xenpara seqinblock 1xenhvm 4xenhvm 8xenhvm

seqoutblock

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

23

Benchmarks: bonnie++, kvm virtio

32GBdataset,lvmpartition,single
140000 120000 100000 80000 60000 40000 20000 0 Host hdb vdb

K/sec

Bonnie++onasinglevm
seqoutblock seqoutrewrite seqinblock

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

24

Benchmarks: bonnie++
2GBram,4GBdataset,8vmaggregate
120000 100000

80000 K/sec

60000

40000

20000

seqoutputblock

seqoutputrewrite

seqinputblock

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

25

Conclusions

KVM proved good stability and reliability

No problems on running production machines for more than 4 weeks CPU performances are extremely good

Modern CPU virtualization technologies help!

Network performances are fair with e1000, good with virtio Disk I/O seems the most problematic aspect

Other solutions have problems too

Requires sysadmins only a small effort

Even if looking promising, right now xen is the most performing solution
Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 26

Future work

qemu snapshot features

Backup

and cloning of nodes

high-level VM managers
Ovirt,

enomalism, ganeti

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

27

Bibliography and links

Xen

Xen repository Kvm repository

KVM

Quattor HEP-SPEC Numerical Recipes: second edition in C Computer Systems: A Programmer's Perspective

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

28