Professional Documents
Culture Documents
Xen Clustering
Xen Clustering
I have tried to run both Debian Etch and Ubuntu 8.04 Server on the cluster nodes, in Dom0. I started my tests with Debian, but I had some issues with slow samba performance in one VM that I couldn't fix so I decided to try Ubuntu Server, for the first time. Both installation went OK, the main difference was that I used mainly source code in Debian, but only packages in Ubuntu. I actually ran into more problems with Ubuntu due to some early bugs in the 8.04 release, will describe them below as I go along.
2011-02-27: The setup in Ubuntu was very stable and I had initially very few problems. I ran into some trouble later on when it turned out that my heartbeat in combination with XEN was negatively effected by some system updates. Never had time to resolve it so I eventually switch off heartbeat as it was just causing issues. Instead I have started with a new install consisting of Debian Squeeze and XEN version 4. I'm migrating the data upgrading one node at the time. Documenting the migration phase and will publish it together with a fresh install guide. Installation How to Index: Linux - Ubuntu 8.04 Server Ubuntu - Base configuration Installing XEN Configure LVM Configure XEN-tools Using XEN-tools to install DomU Install and configure DRBD Configure DomU to use your DRBD device Configure DomU on your other Dom0 node Configure Live Migration
Install and Configure Heartbeat Bug in default xendomains script Troubleshooting Linux - Ubuntu 8.04 Server
I am not going to go through each and every step of the Ubuntu installation as that is really straight forward. I downloaded the 8.04 Server version from http://www.ubuntu.com/getubuntu/download and started the installation one on of the nodes.
During disk partitioning, select manual mode and delete any existing partitions (if there are any). Create one partition for your main system and make it 5GB. This partition will be used for your Dom0 and 5GB should be more than enough. Create a second partition for your swap for Dom0. Make this 512MB as we later will configure Dom0 to use 512MB of RAM. Create a third partition with all your remaining space.
At the packages selection step do not select any packages. Leave everything deselected.
A few minutes later you should have a basic installation of Ubuntu 8.04 Server running. Ubuntu - Base configuration
A few things need to be done to the freshly installed Ubuntu Server before we start installing XEN.
Most of the commands need to be executed with super user rights, so we start by typing
sudo su
I installed SSH early on so I didn't need to run everything locally on the console
nano /etc/networking/interfaces
# The primary network interface auto eth0 iface eth0 inet static address 10.1.1.100 netmask 255.255.255.0 network 10.1.1.0 broadcast 10.1.1.255 gateway 10.1.1.1
auto eth1 iface eth1 inet static address 192.168.1.100 netmask 255.255.255.0 network 192.168.1.0 broadcast 192.168.1.255
This setup will use eth0 to connect to my network and eth1 as the link between the two nodes. For node ha2 use ip address 10.1.1.101 and 192.168.1.101.
nano /etc/hosts
127.0.0.1 localhost 127.0.1.1 ha1.domain.local ha1 10.1.1.100 ha1.domain.local ha1 10.1.1.101 ha2.domain.local ha2 192.168.1.100 ha1X 192.168.1.101 ha2X
for node ha2 change the second line to 127.0.1.1 ha2.domain.local ha2
Edit the /etc/hostname file to match your Fully Qualified Domain Name
(FQDN)
nano /etc/hostname
ha1.domain.local
reboot
You should now be able to connect to your static IP address and your hostname should be your FQDN. Run:
hostname hostname -f
Install NTP to have your time synchronized. This is important for your VMs, especially when migrating a VM to the other node:
Now we are done with the basic configuration of the system and can move on to the fun part, to install Xen, DRBD & Heartbeat Installing XEN
I will use only packages during this installation, and installing XEN on Ubuntu is then a quick and hassle free process. I will only cover the important parts and make comments where needed, if you want to look at a more comprehensive guide for installing XEN on Ubuntu then please check this link: http://howtoforge.com/ubuntu-8.04-server-install-xen-from-ubunturepositories
Even though you might not use loop devices in your XEN setup it can be a good idea to extend the number of allowed loop devices so you don't run into trouble later if you plan to use them. Edit /etc/module and modify the line with "loop" as below:
nano /etc/modules
loop max_loop=64
Now it is time to reboot your system so it will boot with the new xen kernel:
reboot
uname -r
to confirm your system is using the new xen kernel. It should look like this:
Configure LVM
We will now configure a Volume Group of the third partition created during installation. On my system this is /dev/sda3. You can run:
fdisk -l
to confirm this.
vgdisplay
reboot
Within this volume group we will later create logical volumes that will be use by our VMs. You can create the logical volumes manually when installing a new VM, but below I will use xen-tools which will do all that for us. Configure XEN-tools
If you have a cluster with two machines like me it is not necessary to configure xen-tools on both of them. You can decide that you always will use one when adding a new DomU. For flexibility I have my both machines configured the same way so I can run xen-tools on both.
Xen-tools is installed automatically as part of the Xen installation above. So now we just need to configure it. The configuration for xen-tools is stored in /etc/xen-tools/xen-tools.conf
Run:
nano /etc/xen-tools/xen-tools.conf
We first uncomment the line with LVM and specify the volume group we created above:
lvm = vg
Configure the "Disk and Sizing options" section. Mine looks like this at the moment:
size = 5Gb # Disk image size. memory = 384Mb # Memory size swap = 384Mb # Swap size # noswap = 1 # Don't use swap at all for the new system. fs = ext3 # use the EXT3 filesystem for the disk image. dist = hardy # Default distribution to install. image = sparse # Specify sparse vs. full disk images.
the disk size and the swap size will be used to create Logical Volumes (LVs) in your VG specified above.
we will specify the IP address of the VM and its hostname when creating the VM with xen-tools.
Uncomment the line with passwd to always be asked for the password for your VM. Should look like:
passwd = 1
As I have a 64bit AMD CPU I set the following value for my architecture:
arch=amd64
Next and last we change the mirrors used when installing Debian and Ubuntu. This is how mine looks like using mirrors in The Netherlands:
# The default mirror for debootstrap to install Debian-derived distributions # mirror = http://ftp.nl.debian.org/debian/
# # A mirror suitable for use when installing the Dapper release of Ubuntu. # mirror = http://nl.archive.ubuntu.com/ubuntu/
# # If you like you could use per-distribution mirrors, which will # be more useful if you're working in an environment where you want # to regularly use multiple distributions: # mirror_sid=http://ftp.nl.debian.org/debian
for Debian I had to enable the per-distribution mirrors as shown above. Otherwise installing sid, sarge or etch would fail.
The default setting in xen-tools.conf is using the debootstrap installation method. For installing Fedora and CentOS you need to use rinse. If you want to try that later on you should install rinse by running:
Perform this step on only one of your machines, either HA1 or HA2, to create your first DomU.
Time to install our first DomU. The application we use for this part of Xentools is called: xen-create-image. The only two parameters we supply when using xen-create-image is the IP address and hostname of the DomU. All other settings will be taken from xen-tools.conf. But you can override all these settings at command line. For example, to change the distribution you would write --dist=etch to install Debian etch instead of Ubuntu Hardy.
Now, run the following to create a DomU with IP: 10.1.1.50 and with hostname: test:
After a while the process hopefully completed successfully and your DomU is ready. For me it took about 5 minutes to complete.
xen-create-image has now automatically created two LVs in your VG. If you used the hostname "test", you will have one LV called "test-disk" and another one called "test-swap". Full path to these are: "/dev/vg/test-disk" and "/dev/vg/test-swap"
Run:
lvdisplay
You should now be able to start your DomU called: test. The config files for your DomUs are stored in /etc/xen/. We first change to that folder:
root@ha1:/# cd /etc/xen
In /etc/xen/ you will find test.cfg which is the config file for your newly created DomU.
you can also add -c to the line above, that will automatically bring you to the console of the DomU. If you didn't, we can check the status of the running DomU with:
xm list
xm console test
if everything went fine you will now see the prompt to login to your DomU test. To leave console mode in your DomU you need to press Ctrl and ]:
Ctrl + ]
Perfect! You now have your first virtual machine running. NEXT
We will soon go ahead and configure DRBD and Heartbeat to allow for live migration and high availability. This is the fun part! But before we do that we need to duplicate the LV disk setup on the other machine.
So on HA2 we need to create one LV with 5GB and another one with 384MB.
Run:
to create the "test-disk" with 5GB of space. We use the same name here for simplicity, but the names doesn't need to match as we will specify that later in the DRBD configuration.
Run:
The configuration file for DRBD is located in /etc/drbb.conf. We will now configure it to use the LVs we create above and later we will change the Xen configuration of our test DomU to use the DRBD device instead of directly the LV. Edit /etc/drbd.conf:
Below is a copy of my settings with all default comments removed from the file:
resource test-disk { protocol C; startup { wfc-timeout 120; ## 2 min degr-wfc-timeout 120; ## 2 minutes. } disk { on-io-error detach; } net { allow-two-primaries; after-sb-0pri discard-zero-changes; after-sb-1pri discard-secondary;
timeout 60;
startup { wfc-timeout 120; ## 2 min degr-wfc-timeout 120; ## 2 minutes. } disk { on-io-error detach; } net { allow-two-primaries; after-sb-0pri discard-zero-changes; after-sb-1pri discard-secondary;
timeout 60; connect-int 10; ping-int 10; max-buffers 2048; max-epoch-size 2048; } syncer { }
I have given the DRBD resource the same name as its corresponding LV. So DRBD resource: test-disk is using LV: test-disk.
Next we will create a separate volume where we store DRBD's meta data. Meta data is used by DRBD to store information about the device. This can either be internal or external. Internal mode is easier to setup for a new devices but requires resizing operations when using an already formatted device. Read further about this on: http://www.drbd.org/users-guide/chinternals.html
As we already have data on our LVs, created by XEN tools, we will use external meta data. So we create another LV with 1GB of space:
lvcreate -L 1G -n meta vg
If drbd.conf, above, you will see that we specify the line with meta-disk as using /dev/vg/meta[0] and /dev/vg/meta[1]. Same device can be used to store meta-data for several DRBD resources, that is done with adding [X] after the device name.
Redo the DRBD configuration above for the other node if you haven't done already.
/etc/init.d/drbd start
Before the replication of data will begin we have to make one node the primary node for each DRBD resource. Run the following on the node you installed your DomU above to replicate the data to the other node (Do NOT run it on the other node):
If everything is OK after data is replicated you should see something like this for each DRBD device when running /etc/init.d/drbd status. It should state Primary/Secondary and UpToDate/UpToDate:
/etc/init.d/drbd status
Now we are done setting up the DRBD device for our LV. Next is to configure or DomU to use DRBD.
The configuration files for your DomUs are stored in /etc/xen/. So we first change to that directory:
cd /etc/xen
disk = [
'phy:/dev/vg/test-swap,xvda1,w', 'phy:/dev/vg/test-disk,xvda2,w', ]
Done! That is all, now your DomU is ready to be started using the DRBD device.
xm shutdown test
xm create test.cfg -c
Hopefully everything went fine. Smile Login and then shutdown your DomU:
shutdown -h now
When you are back to your Dom0 prompt check the DRBD status again:
/etc/init.d/drbd status
You will now see the status of the DRBD devices as Secondary/Secondary:
The good thing with this is that Xen takes care of your DRBD devices and will automatically bring them up and down as needed. Same goes when you start a DomU, Xen will first make sure the DRBD devices are in primary mode before starting the DomU.
Now we need to prepare the other Dom0 node to be able to run the DomU called test. This is simply done by copying the /etc/xen/test.cfg from ha1 to ha2. Copy the whole file or the content of the file, what ever is easier for you. When the file is copied we will try to start the DomU on ha2.
root@ha1:/# xm list
If everything is fine you should see DomU test running. Check with xm list:
root@ha2:/# xm list
Perfect! You can now start the DomU on both nodes. (Note: Don't try to start the DomU on both nodes at the same time.)
Next we will start with the real cool things! We will go ahead and configure live migration so we can move a DomU between the two Dom0 nodes with hardly any downtime. Cool
By default XEN does not allow live migration, we have to enable this is /etc/xen/xend-config.sxp. Make sure the following line is commented, it should look like this:
and that the following line is not commented, it should look like this:
(xend-relocation-port 8002)
Restart xend, reload has no effect, but restart will not kill running DomUs. Run:
/etc/init.d/xend restart
Make sure to do this two changes above on both nodes, both ha1 & ha2.
If everything is in order your DomU called test should be running on your ha2 node. Please confirm this with xm list:
root@ha2:/# xm list
Name ID Mem VCPUs State Time(s) Domain-0 0 500 2 r----- 10268.0 test 7 384 1 -b---- 4694.7
So first we write xm migrate. Then the name of the DomU we want to migrate, in this case test. Then the hostname or IP of the other node, in this case ha1x (remember that we specified ha1x and ha2x in the hosts file on both nodes and mapped it to the IP of the interface with the cross over connection between ha1 and ha2). And we end the command with --live, that
If everything went fine your DomU test should be running on ha1. Run:
root@ha1:/# xm list
Name ID Mem VCPUs State Time(s) Domain-0 0 500 2 r----- 10260.1 test 11 384 1 -b---- 2.2
Try the migration a few times between the two nodes. Try accessing the DomU during a live migration, ping it from another machine in your network, open a SSH session and see how incredible fast it migrates. I have not timed it myself, but running a normal ping from a Windows machine I lose no packets or maximum 1 packet during a migration. Smile
We will use Heartbeat, part of the Linux High Availability project (http://www.linux-ha.org/), to monitor our XEN resources and provide failover between our two nodes. I currently use version 2 of hearbeat but utilizing versions 1's configuration files, so that is what I will describe below. Read more in the link above about the difference in configuration files.
To install Heartbeat we will use package in the Ubuntu repository. Run this on both nodes:
Configuration files for Heartbeat is stored in /etc/ha.d/. We need to configure the following files: authkeys, haresources and ha.cf.
We will start with configuring authkeys and ha.cf as they are the easiest to explain. Authkeys is used to configure authentication between the cluster nodes. Configure it to look something like this:
This will tell heartbeat to use the method sha1 with the supplied key. Note: Make sure to copy the exact same copy to your ha2 node.
Lets continue with ha.cf. This file contains the configuration for heartbeat: nodes in the cluster, how they communicate and timer settings. What I will use is the version 1 configuration format. Configure the file to look something like this:
ucast eth0 10.1.1.100 ucast eth0 10.1.1.101 auto_failback on watchdog /dev/watchdog debugfile /var/log/ha-debug node ha1.domain.local node ha2.domain.local
The two lines above that begin with "ucast eth0 " configures the heartbeat communication. The reason I have put both node's IP addresses is so the file can be identical on both nodes, heartbeat will ignore the IP of the local machine so this is perfectly fine. Note: Make sure to copy the exact same copy to your ha2 node.
Now we will continue with haresources file. The file itself is actually very easy but we need to setup the resources used by heartbeat which requires some explanation. We begin with the file, configure it to look like this:
Looks very simple, doesn't it? Smile What this means is that the resource(script that will control our DomUs) xendomainsHA1 will default to node HA1 and xendomainsHA2 to node HA2. These two scripts are copies of the /etc/init.d/xendomains script and modified for our two node cluster. We need to do this to be able to differentiate between DomUs on HA1 resp. HA2.
Now we edit both files to change two lines so it looks like below:
LOCKFILE=/var/lock/xendomainsHA1 XENDOM_CONFIG=/etc/default/xendomainsHA1
LOCKFILE=/var/lock/xendomainsHA2 XENDOM_CONFIG=/etc/default/xendomainsHA2
Please make sure that all the Heartbeat configuration above is exactly the same on node HA2. Below the configuration will differ slightly.
As you noticed above we specified different configuration files for the resources. There is a default configuration file already located in /etc/default, called xendomains. We will copy it as below:
We copy only xendomainsHA1 to begin with. We will modify it and then later use it for xendomainsHA2.
XENDOMAINS_MIGRATE="ha2X --live"
This will allow for live migration to HA2 when node is shutdown.
XENDOMAINS_SAVE=
XENDOMAINS_SHUTDOWN_ALL=
Disable this to prevent ALL DomUs to be shutdown even them not controlled by this script
XENDOMAINS_RESTORE=false
XENDOMAINS_AUTO=/etc/xen/auto/HA1
Point to location for DomU configuration files that will be controlled by this script
XENDOMAINS_AUTO_ONLY=true
And we modify xendomainsHA2 to point to correct folder for DomU configuration files:
XENDOMAINS_AUTO=/etc/xen/auto/HA2
Now we can copy /etc/default/xendomainsHA1 and /etc/default/xendomainsHA2 to node HA2. Do that by any means you want, either file transfer or copy and paste.
XENDOMAINS_MIGRATE="ha1X --live"
XENDOMAINS_MIGRATE="ha1X --live"
Next we need to create the two folders /etc/xen/auto/HA1 and /etc/xen/auto/HA2 as referred above. Do this on both node HA1 & HA2.
Create a symlink on both nodes in /etc/xen/auto/HA1 pointing to our test.cfg file in /etc/xen/
ln -s /etc/xen/test.cfg /etc/xen/auto/HA1/test
Whenever creating a new DomU you need to decide if you want it by default to run on HA1 or HA2, the location of the symlink will decide that.
Remove the default xendomains script to start automatically, heartbeat will now control this for us. Do this on both nodes:
xm shutdown test
/etc/init.d/heartbeat start
Hopefully everything is fine. Try to reboot one node at a time to see that DomU test is migrated between the two nodes. Bug in default xendomains script
There is a bug in the default xendomains script that breaks the script if there are more than one domain in the AUTO directory. This is only true for non standard configurations like mine above when XENDOMAINS_AUTO_ONLY=true in /etc/defaults/xendomains. By default this setting is false hence I guess no one picked up on it. Been too lazy to create a patch for it but I will try to get around and do that.
Solution:
rdnames() { NAMES= if ! contains_something "$XENDOMAINS_AUTO" then return fi for dom in $XENDOMAINS_AUTO/*; do rdname $dom if test -z "$NAMES"; then NAMES=$NM; else NAMES="$NAMES $NM"; fi done }
stop() { # Collect list of domains to shut down if test "$XENDOMAINS_AUTO_ONLY" = "true"; then rdnames fi echo -n "Shutting down Xen domains:" while read LN; do parseln "$LN" if test "$id" = "0"; then continue; fi echo -n " $name" found="0" if test "$XENDOMAINS_AUTO_ONLY" = "true"; then for i in ${NAMES[@]} do if test $found="0"; then if test $i = $name; then found=1 fi fi done if test $found = "0"; then echo -n "(skip)"
This script shall now work for more than one DomU. Smile Troubleshooting
Here is a string I use in my Dom0 to check relevant logs of the system, xen and heartbeat. Keep this running in a separate shell when testing migration and heartbeat failover: