Openvswitch - Brezular's Technical Blog

Openvswitch | Brezular's Technical Blog
http://brezular.wordpress.com/category/openvswitch/
Brezular's Technical Blog

My worthless personal blog about networking
Part4 OPENVSWITCH Playing with Bonding on Openvswitch

DECEMBER 4, 2011 15 COMMENTS (HTTP://BREZULAR.WORDPRESS.COM/2011/12 /04/PART4-OPENVSWITCH-PLAYING-WITH-BONDING-ON-OPENVSWITCH/#COMMENTS)
i 1 Vote Bonding (h,p://en.wikipedia.org/wiki/Channel_bonding) is aggregation multiple links to single link in order to increase throughput and achieve redundancy in case one of links fails. At least, ocials say it so it must be true. Now, let me ask the question. How can we congure bonding? From what I know about bonding, they are two possibilities to congure it. The rst option (h,p://www.linuxfoundation.org/collaborate/workgroups/networking/bonding) is to load bonding module to kernel and use ifenslave (h,p://linux.die.net/man/8/ifenslave) utility for its conguration. Conguration steps for Microcore are described here. h,p://brezular.wordpress.com/2011/01/20/how-to-setup-linux-microcore-3-x-router-qemu-imagein-fedora-linux-part2/ (h,p://brezular.wordpress.com/2011/01/20/how-to-setup-linux-microcore3-x-router-qemu-image-in-fedora-linux-part2/) If you want to have fun as I had with bonding conguration, check this lab for reference. h,p://brezular.wordpress.com/2011/03/10/etherchannelvrrp-dhcp-ospf-conguration-cisco-vya,amicrocore/ (h,p://brezular.wordpress.com/2011/03/10/etherchannelvrrp-dhcp-ospf-conguration-ciscovya,a-microcore/) As the tutorial should be dedicated to bonding on Openvswitch (h,p://openvswitch.org/) I should focus on this topic. This is the second option how can we congure bonding without loading bonding module (h,p://www.linuxfoundation.org/collaborate/workgroups/networking/bonding) to Linux kernel. Lets say we have two Microcore Linux (h,p://distro.ibiblio.org/tinycorelinux/welcome.html) boxes
1 de 48 02/10/2013 08:01 a.m.
connected with two links. As usual, our topology is created using GNS3 (h,p://www.gns3.net/) and Qemu (h,p://en.wikipedia.org/wiki/QEMU) images of Microcore with installed Openvswitch 1.2.2 and Quagga (h,p://www.quagga.net/) 0.99.20 on the top of Microcore.
(h,p://brezular.les.wordpress.com/2011/12/topology.png)There is nothing in the world what could prevent you to download my Openvswitch Microcore Qemu image available here. h,p://sourceforge.net/projects/gns-3/les/Qemu%20Appliances/linux-microcore-4.0-openvswitch-1.2.2quagga-0.99.20.img/download (h,p://sourceforge.net/projects/gns-3/les/Qemu%20Appliances/linuxmicrocore-4.0-openvswitch-1.2.2-quagga-0.99.20.img/download) As for Ethernet card I suggest you to use emulated i82557b Ethernet card. Hint check GNS3 Qemu se,ing. According to Bill (h,p://en.wikipedia.org/wiki/Bill_Gates), 640 kB is enough for anyone but it is be,er to assign at least 128 MB RAM for Microcore. Here is a snapshot of Qemu se,ing.
2 de 48
02/10/2013 08:01 a.m.
(h,p://brezular.les.wordpress.com/2011/12/qemu-se,ings.jpeg)
Part1 Basic Microcore, Openvswitch and Quagga Configuration

Before we start to congure Openvswitch we should be aware of initial conguration requirement for Openvswitch and Quagga. Frankly to say, Openvswitch is not Cisco IOS (h,p://en.wikipedia.org /wiki/Cisco_IOS) which you can congure right after device is booted. First you need (h,p://openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob_plain;f=INSTALL.Linux;hb=HEAD) to create database, conguration les, start services etc. My Qemu Microcore image with Openvswitch and Quaggae is precongured for such things so feel free to use it. 1. Switch 1 conguration hostname, trunks, vlan interfaces tc@box:~$ echo "hostname Switch1" >> /opt/bootlocal.sh tc@box:~$ sudo hostname Switch1 tc@Switch1:~$ sudo su root@Switch1:~# ovs-vsctl add-br br0 root@Switch1:~# ovs-vsctl add-port br0 vlan10 tag=10 -- set interface vlan10 type=internal root@Switch1:~# ovs-vsctl add-port br0 vlan20 tag=20 -- set interface vlan20 type=internal Note: I usually specify list of VLANs to be allowed on trunk link root@Switch1:~# ovs-vsctl add-port br0 eth0 trunks=10,20 root@Switch1:~# ovs-vsctl add-port br0 eth1 trunks=10,20 Unfortunately if we do it now, we cannot create bond0 interface, later in Part2. In this case a warning message informs us that eth0 and eth1 already exists on bridge br0. Therefore we skip a trunk denition here and we will congure the trunk later together with bonding conguration. Start Quagga vtysh (h,p://linux.die.net/man/1/vtysh) shell. root@Switch1:~# vtysh Hello, this is Quagga (version 0.99.20). Copyright 1996-2005 Kunihiro Ishiguro, et al. Switch1# conf t
3 de 48 02/10/2013 08:01 a.m.
Switch1(cong)# interface vlan10 Switch1(cong-if)# ip address 192.168.10.1/24 Switch1(cong-if)# no shutdown Switch1(cong-if)# Switch1(cong-if)# interface vlan20 Switch1(cong-if)# ip address 192.168.20.1/24 Switch1(cong-if)# no shutdown Switch1(cong-if)# do wr Switch1(cong-if)# exit Switch1(cong)# exit Switch1# exit. Save Microcore conguration. root@Switch1:~# /usr/bin/filetool.sh -b Ctrl+S 2. Switch 2 conguration hostname, trunks, vlan interfaces tc@box:~$ echo "hostname Switch2" >> /opt/bootlocal.sh tc@box:~$ sudo hostname Switch2 tc@Switch1:~$ sudo su root@Switch2:~# ovs-vsctl add-br br0 root@Switch2:~# ovs-vsctl add-port br0 vlan10 tag=10 -- set interface vlan10 type=internal root@Switch2:~# ovs-vsctl add-port br0 vlan20 tag=20 -- set interface vlan20 type=internal Start Quagga vtysh shell. root@Switch2:~# vtysh Hello, this is Quagga (version 0.99.20). Copyright 1996-2005 Kunihiro Ishiguro, et al. Switch2# conf t Switch2(cong)# interface vlan10 Switch2(cong-if)# ip address 192.168.10.2/24 Switch2(cong-if)# no shutdown Switch2(cong-if)# Switch2(cong-if)# interface vlan20 Switch2(cong-if)# ip address 192.168.20.2/24 Switch2(cong-if)# no shutdown Switch2(cong-if)# do wr Switch2(cong-if)# exit Switch2(cong)# exit
4 de 48 02/10/2013 08:01 a.m.
Switch2# exit root@Switch2:~# Save Microcore conguration. root@Switch2:~# /usr/bin/filetool.sh -b Ctrl+S
Part2 Bonding
Now, it is time to start playing game called bonding. They are not any rules in our game but it is not bad idea to read documentation rst. Page number ten could answer many future question. h,p://openvswitch.org/ovs-vswitchd.conf.db.5.pdf (h,p://openvswitch.org/ovs-vswitchd.conf.db.5.pdf) As you can see, they are four bond modes with balance-slb as default mode. Modes are described on page 11. 1. Switch1 and Switch2 conguration for static bonding root@Switch1:~# ovs-vsctl add-bond br0 bond0 eth0 eth1 trunks=10,20 root@Switch2:~# ovs-vsctl add-bond br0 bond0 eth0 eth1 trunks=10,20 2. Static Bonding Testing Check bond interface bond0 on Switch1. root@Switch1:~# ovs-appctl bond/show bond0 bond_mode: balance-slb bond-hash-algorithm: balance-slb bond-hash-basis: 0 updelay: 0 ms downdelay: 0 ms next rebalance: 9476 ms lacp_negotiated: false slave eth1: enabled may_enable: true slave eth0: enabled active slave may_enable: true Now do the same for bond0 on Switch2. root@Switch2:~# ovs-appctl bond/show bond0
5 de 48 02/10/2013 08:01 a.m.
bond_mode: balance-slb bond-hash-algorithm: balance-slb bond-hash-basis: 0 updelay: 0 ms downdelay: 0 ms next rebalance: 7420 ms lacp_negotiated: false slave eth1: enabled may_enable: true slave eth0: enabled active slave may_enable: true Now, start pinging IP address 192.168.10.2 from Switch1. Ping is supposed to be working. While pinging, start tcpdump to listen on interface ethernet1 of Switch2 root@Switch2:~# tcpdump -i eth1 tcpdump: WARNING: eth1: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 68 bytes 11:47:07.214753 IP 192.168.10.2 > 192.168.10.1: ICMP echo reply, id 36360, seq 15, length 64 11:47:08.215483 IP 192.168.10.2 > 192.168.10.1: ICMP echo reply, id 36360, seq 16, length 64 11:47:09.215823 IP 192.168.10.2 > 192.168.10.1: ICMP echo reply, id 36360, seq 17, length 64 11:47:10.216846 IP 192.168.10.2 > 192.168.10.1: ICMP echo reply, id 36360, seq 18, length 64 11:47:11.217868 IP 192.168.10.2 > 192.168.10.1: ICMP echo reply, id 36360, seq 19, length 64 <output truncated> 15 packets captured 15 packets received by lter 0 packets dropped by kernel Interface Ethernet1 sends icmp echo reply messages back to Switch1. As they are not any ICMP echo requests listed in the output, only the interface Ethernet0 receives these ICMP requests coming from Switch1. My question is, what happens if we shutdown Ethernet0 on Switch2? Will be ICMP requests interrupted and possibly re-forwarded via interface Ethernet0 of Switch1? If yes, how long does it take? root@Switch2:~# ifconfig eth0 down Check the bond interface bond0 on Switch2 again. As soon as Switch2 detects failure of its interface, Openvswitch puts slave eth0 do disabled state root@Switch2:~# ovs-appctl bond/show bond0 bond_mode: balance-slb
02/10/2013 08:01 a.m.
6 de 48
bond-hash-algorithm: balance-slb bond-hash-basis: 0 updelay: 0 ms downdelay: 0 ms next rebalance: 7612 ms lacp_negotiated: false slave eth1: enabled active slave may_enable: true hash 52: 0 kB load slave eth0: disabled may_enable: false Switch1 cannot detect failure of far end interface Ethernet0. It continues to send ICMP requests to Switch2 via its Ethernet0 interface. Stop ping command and check bond interface bond0 on Switch1. Obviously, Ethernet0 is still enabled in bundle root@Switch1:~# ovs-appctl bond/show bond0 bond_mode: balance-slb bond-hash-algorithm: balance-slb bond-hash-basis: 0 updelay: 0 ms downdelay: 0 ms next rebalance: 4007 ms lacp_negotiated: false slave eth1: enabled may_enable: true slave eth0: enabled active slave may_enable: true hash 133: 0 kB load This behaviour we cannot tolerate. We need such as mechanism which helps to detect far end failure. Therefore LACP must be deployed in your conguration. According to wiki (h,p://en.wikipedia.org /wiki/Link_Aggregation_Control_Protocol#Link_Aggregation_Control_Protocol), LACP works by sending frames (LACPDUs) down all links that have the protocol enabled. These are two main advantages comparing to static bonding. 1. Failover when a link fails when the peer will not see the link down. With static link aggregation the peer would continue sending trac down the link causing it to be lost. 2. The device can conrm that the conguration at the other end can handle link aggregation. With Static link aggregation a cabling or conguration mistake could go undetected and cause undesirable network behaviour.
7 de 48 02/10/2013 08:01 a.m.
Uhm, it looks like exactly what we want. So do not hesitate and lets go congure bonding with LACP. 3. Switch1 and Switch2 conguration for bonding with LACP Delete existing bond0 port. root@Switch1:~# ovs-vsctl del-port br0 bond0 root@Switch2:~# ovs-vsctl del-port br0 bond0 Create a new bond interface. root@Switch1:~# ovs-vsctl add-bond br0 bond0 eth0 eth1 lacp=active trunks=10,20 root@Switch2:~# ovs-vsctl add-bond br0 bond0 eth0 eth1 lacp=active trunks=10,20 Check bonding on Switch1. root@Switch1:~# ovs-appctl bond/show bond0 bond_mode: balance-slb bond-hash-algorithm: balance-slb bond-hash-basis: 0 updelay: 0 ms downdelay: 0 ms next rebalance: 7763 ms lacp_negotiated: true slave eth0: disabled may_enable: false slave eth1: enabled active slave may_enable: true Notice state of Ethernet0 interface. Remember, we shut-downed Ethernet0 on Switch2. Thanks to LACP, Ethernet0 is disabled in bundle on Switch1. It is awesome! Note Even an interface Ethernet0 of Switch1 is disabled in bundle, it remains in up state root@Switch1:~# ifconfig eth0 eth0 Link encap:Ethernet HWaddr 00:AA:00:D8:E6:00 inet6 addr: fe80::2aa::fed8:e600/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:112 errors:0 dropped:0 overruns:0 frame:0 TX packets:2030 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:36156 (35.3 KiB) TX bytes:336741 (328.8 KiB) 4. LACP bonding testing The last point we want to test is how much time is required for transition for interface in bundle to disabled state, if far end fails. Logically it should depend how often are LACP messages exchanged
8 de 48 02/10/2013 08:01 a.m.
between peers. Start tcpdump listening on eth0 of Switch1. Then bring interface eth0 up on Switch2. Right after LACP messages are exchanged via eth0 between Switch1 and Switch2, Ethernet0 interface is brought to enabled state in bundle. See the output below root@Switch1:~# tcpdump -i eth0 -v tcpdump: WARNING: eth0: no IPv4 address assigned tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 68 bytes < DHCP requests> 14:00:14.555758 IP (tos 00, ,l 64, id 0, oset 0, ags [none], proto UDP (17), length 317) 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request [|bootp] <IPV6 messages coming from Switch1> 14:00:18.457096 00:aa:00:85:34:00 (oui Unknown) > 33:33:00:00:00:16 (oui Unknown), ethertype IPv6 (0x86dd), length 110: 00000: 6000 0000 0038 0001 0000 0000 0000 0000 `.8. 00010: 0000 0000 0000 0000 02 0000 0000 0000 . 00020: 0000 0000 0000 0016 3a00 0502 ..: 14:00:18.957535 00:aa:00:85:34:00 (oui Unknown) > 33:33::85:34:00 (oui Unknown), ethertype IPv6 (0x86dd), length 78: 00000: 6000 0000 0018 3a 0000 0000 0000 0000 `..: 00010: 0000 0000 0000 0000 02 0000 0000 0000 . 00020: 0000 0001 85 3400 8700 1173 4.s <LACP message coming from Switch1. Notice System 00:00:00:00:00:00. It means that Switch2 has not known about partners system yet> 14:00:19.217584 LACPv1, length: 110 Actor Information TLV (001), length: 20 System 00:aa:00:85:34:00 (oui Unknown), System Priority 65534, Key 5, Port 6, Port Priority 65535 State Flags [Activity, Aggregation, Default] Partner Information TLV (002), length: 20 System 00:00:00:00:00:00 (oui Ethernet), System Priority 0, Key 0, Port 0, Port Priority 0 State Flags [none] Collector Information TLV (003), length: 16 packet exceeded snapshot <LACP message sent to Switch2. As Switch1 knows partner ID 00:aa:00:85:34:00 MAC address of interface eth0 of Switch2> 14:00:19.217914 LACPv1, length: 110 Actor Information TLV (001), length: 20 System 00:aa:00:d8:e6:00 (oui Unknown), System Priority 65534, Key 5, Port 6, Port Priority 65535 State Flags [Activity, Aggregation, Synchronization, Collecting, Distributing] Partner Information TLV (002), length: 20 System 00:aa:00:85:34:00 (oui Unknown), System Priority 65534, Key 5, Port 6, Port Priority 65535
9 de 48 02/10/2013 08:01 a.m.
State Flags [Activity, Aggregation, Default] Collector Information TLV (003), length: 16 packet exceeded snapshot <LACP message sent to Switch1. Now, Switch1 knows partners ID 00:aa:00:d8:e6:00 MAC address of interface eth0 of Switch1> 14:00:19.220189 LACPv1, length: 110 Actor Information TLV (001), length: 20 System 00:aa:00:85:34:00 (oui Unknown), System Priority 65534, Key 5, Port 6, Port Priority 65535 State Flags [Activity, Aggregation, Synchronization, Collecting, Distributing] Partner Information TLV (002), length: 20 System 00:aa:00:d8:e6:00 (oui Unknown), System Priority 65534, Key 5, Port 6, Port Priority 65535 State Flags [Activity, Aggregation, Synchronization, Collecting, Distributing] Collector Information TLV (003), length: 16 packet exceeded snapshot Explanation 00:aa:00:85:34:00 MAC address of eth0 interface of Switch2 00:aa:00:d8:e6:00 MAC address of eth0 interface of Switch1: Nice! We can see structure of LACP messages but still do not know anything about time required for transition. It is time to check it. Start pinging 192.168.10.2 from Switch1 and with the help of tcpdump nd an interface where packets enter Switch2. In my case it is Ethernet1. Shutdown this interface and cheek the output of ping command. You see, ping is interrupted and it took approximately 50 seconds for ping to be recovered. It means that for 50 seconds interface Ethernet1 of Switch1 had been in enabled state. It is too much. It must be a way to short this time to acceptable level. Issue the command below on Switch1. LACP statistic can be found there. root@Switch1:~# ovs-appctl lacp/show bond0 lacp: bond0 status: active negotiated sys_id: 00:aa:00:d8:e6:00 sys_priority: 65534 aggregation key: 5 lacp_time: slow slave: eth0: current a,ached port_id: 6 port_priority: 65535 actor sys_id: 00:aa:00:d8:e6:00 actor sys_priority: 65534 actor port_id: 6 actor port_priority: 65535 actor key: 5
10 de 48 02/10/2013 08:01 a.m.
actor state: activity aggregation synchronized collecting distributing partner sys_id: 00:aa:00:85:34:00 partner sys_priority: 65534 partner port_id: 6 partner port_priority: 65535 partner key: 5 partner state: activity aggregation synchronized collecting distributing slave: eth1: current a,ached port_id: 5 port_priority: 65535 actor sys_id: 00:aa:00:d8:e6:00 actor sys_priority: 65534 actor port_id: 5 actor port_priority: 65535 actor key: 5 actor state: activity aggregation synchronized collecting distributing partner sys_id: 00:aa:00:85:34:00 partner sys_priority: 65534 partner port_id: 5 partner port_priority: 65535 partner key: 5 partner state: activity aggregation synchronized collecting distributing Focus on lacp-time parameter which is set to slow. According to documentation (h,p://openvswitch.org /ovs-vswitchd.conf.db.5.pdf) it is: The LACP timing used on this Port. Possible values are fast, slow and a positive number of milliseconds. By default slow is used. When congured to be fast LACP heartbeats are requested at a rate of once per second causing connectivity problems to be detected more quickly. In slow mode, heartbeats are requested at a rate of once every 30 seconds. That is exactly what we need. Changing lacp_time from slow to fast should help us to detect failure of peers interface more quickly. 5. Switch1 and Switch2 conguration for bonding with LACP and lacp time set to fast Delete existing bond interface bond0. root@Switch1:~# ovs-vsctl del-port br0 bond0 root@Switch2:~# ovs-vsctl del-port br0 bond0 Create a new bond interface. root@Switch1:~# ovs-vsctl add-bond br0 bond0 eth0 eth1 lacp=active other_config:lacptime=fast trunks=10,20 root@Switch2:~# ovs-vsctl add-bond br0 bond0 eth0 eth1 lacp=active other_config:lacp_time=fast trunks=10,20
11 de 48 02/10/2013 08:01 a.m.
Save conguration. root@Switch1:~# /usr/bin/filetool.sh -b root@Switch2:~# /usr/bin/filetool.sh -b Ctrl+S If you want to check interval of sending LACP messages, start tcpdump on Switch2 while pinging IP address of VLAN interface of Switch2 from Switch1. Now, LACP message are sent every 1 second root@Switch2:~# tcpdump -i eth0 tcpdump: WARNING: eth0: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 68 bytes 21:42:25.525007 LACPv1, length: 110 21:42:26.529846 LACPv1, length: 110 21:42:27.532092 LACPv1, length: 110 21:42:28.534532 LACPv1, length: 110 21:42:29.543080 LACPv1, length: 110 21:42:30.545620 LACPv1, length: 110 21:42:31.547848 LACPv1, length: 110 21:42:32.550717 LACPv1, length: 110 21:42:33.551271 LACPv1, length: 110 21:42:34.553382 LACPv1, length: 110 10 packets captured 10 packets received by lter 0 packets dropped by kernel 6. LACP bonding testing with enabled lacp_time=fast option Start pinging 192.168.10.2 from Switch1 and nd interface receiving ICMP echo requests on Switch2. In my case it is interface eth0. While pinging, shutdown this interface and check ping statistic root@Switch1:~# ping 192.168.10.2 PING 192.168.10.2 (192.168.10.2): 56 data bytes 64 bytes from 192.168.10.2: seq=0 ,l=64 time=1.810 ms 64 bytes from 192.168.10.2: seq=1 ,l=64 time=0.940 ms 64 bytes from 192.168.10.2: seq=2 ,l=64 time=1.610 ms 64 bytes from 192.168.10.2: seq=3 ,l=64 time=0.755 ms <output truncated> 192.168.10.2 ping statistics 27 packets transmi,ed, 24 packets received, 11% packet loss round-trip min/avg/max = 0.755/3.378/48.484 ms You see, three packet are lost. That is what we can call link redundancy if one of links fails.
12 de 48
02/10/2013 08:01 a.m.
Following commands helpful during troubleshooting. Show MAC address table. root@Switch1:~# ovs-appctl fdb/show br0 List ports. root@Switch1:~# ovs-vsctl list port END. FILED UNDER OPENVSWITCH TAGGED WITH BONDING, CORE, LINUX TEAMING NIC, LLINUX TEAMING NETWORK, MICROCORE, TEAMING
Part3 OPENVSWICH Campus Model with Layer2 Access, Built with Open-Source Applications
NOVEMBER 23, 2011 1 COMMENT (HTTP://BREZULAR.WORDPRESS.COM/2011/11/23/PART3OPENVSWICH-CAMPUS-MODEL-WITH-LAYER2-ACCESS-BUILT-WITH-OPEN-SOURCEAPPLICATIONS/#COMMENTS)
i 2 Votes In part one (h,p://brezular.wordpress.com/2011/09/03/part1-openvswich-creating-and-submi,ingopenvswitch-extension-to-microcore-upstream/) we showed how to create Openvswitch (h,p://openvswitch.org/) extension and submit it to Microcore (h,p://distro.ibiblio.org/tinycorelinux /welcome.html) repository (h,p://distro.ibiblio.org/tinycorelinux/4.x/x86/tcz/). There were also presented after-install steps for Openvswitch, adapted for specic Microcore needs. h,p://brezular.wordpress.com/2011/09/03/part1-openvswich-creating-and-submi,ing-openvswitchextension-to-microcore-upstream/ (h,p://brezular.wordpress.com/2011/09/03/part1-openvswich-creatingand-submi,ing-openvswitch-extension-to-microcore-upstream/) In part two (h,p://brezular.wordpress.com/2011/06/25/part2-openvswich-vlans-trunks-l3-vlan-interfaceintervlan-routing-conguration-and-testing/) we did several tests in order to test feature such as vlans (h,p://en.wikipedia.org/wiki/Virtual_LAN), 8021q trunks (h,p://en.wikipedia.org/wiki/IEEE_802.1Q) and VLAN interfaces (h,p://www.itsyourip.com/cisco/how-to-create-vlan-interfaces-for-intervlan-routingin-cisco-ios/) widely used in typical multilayer switches (h,p://en.wikipedia.org/wiki/Multilayer_switch).
13 de 48 02/10/2013 08:01 a.m.
h,p://brezular.wordpress.com/2011/06/25/part2-openvswich-vlans-trunks-l3-vlan-interface-intervlanrouting-conguration-and-testing/ (h,p://brezular.wordpress.com/2011/06/25/part2-openvswich-vlanstrunks-l3-vlan-interface-intervlan-routing-conguration-and-testing/) In part three we are going to create Campus Model (h,p://www.mcmcse.com/cisco/guides /hierarchical_model.shtml) using Linux Microcore with installed Openvswitch extension. As we need to enable routing between Distribution and Core layer (h,p://en.wikipedia.org /wiki/Cisco%27s_3_Layered_Model) to follow reccomendations of design guide, we have to install Quagga (h,p://www.quagga.net/) open-source routing suite. Keepalived (h,p://www.keepalived.org/) extension helps us to achieve default gateways redundancy using VRRP (h,p://en.wikipedia.org/wiki/Virtual_Router_Redundancy_Protocol) protocol, in case of failure of default gateway. Other extensions such as iproute2 (h,p://en.wikipedia.org/wiki/Iproute2), tcpdump (h,p://en.wikipedia.org/wiki/Tcpdump) are not neccessary but useful and they are installed in our Qemu image (h,p://en.wikibooks.org/wiki/QEMU/Images). Here is my own linux-microcore-3.0.3 Qemu image with installed Openvswitch 1.2.2, Quagga 0.99.20 and Keepalived 1.2.2 as extensions. h,p://sourceforge.net/projects/gns-3/les/Qemu%20Appliances/linux-microcore-4.0-openvswitch-1.2.2quagga-0.99.20.img/download (h,p://sourceforge.net/projects/gns-3/les/Qemu%20Appliances/linuxmicrocore-4.0-openvswitch-1.2.2-quagga-0.99.20.img/download) h,p://www.4shared.com/le/CAi_zNuO/linux-microcore-40-openvswitch.html (h,p://www.4shared.com /le/CAi_zNuO/linux-microcore-40-openvswitch.html) I had to recompile the kernel because a default Microcore kernel is not compiled with Multipath option. Read more here. h,p://brezular.wordpress.com/2011/11/13/part1-recompiling-linux-microcore-4-0-2-for-equal-cost-routes/ (h,p://brezular.wordpress.com/2011/11/13/part1-recompiling-linux-microcore-4-0-2-for-equalcost-routes/) The image also contains tcpdump and iproute2 command. I recommend you to use my Qemu image otherwise additional after install conguration for Quagga and Openvswitch is required. At least you should put following commands to /opt/bootlocal.sh # put other system startup commands here #Load modules to kernel modprobe openvswitch_mod modprobe ipv6 #Start ovsdb-server ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,manager_options --private-key=db:SSL,private_key --certificate=db:SSL,certificate --bootstrap-ca-cert=db:SSL,ca_cert --pidfile --detach #Initialize database ovs-vsctl no-wait init #Start vswitchd daemon
02/10/2013 08:01 a.m.
14 de 48
ovs-vswitchd pidle detach #Enable forwarding between interfaces sysctl -w net.ipv4.ip_forward=1 sysctl -w net.ipv6.conf.all.forwarding=1 #Start Quagga routing daemons /usr/local/sbin/zebra -u root -d -f /usr/local/etc/quagga/zebra.conf /usr/local/sbin/ripd -u root -d -f /usr/local/etc/quagga/ripd.conf /usr/local/sbin/ripngd -u root -d -f /usr/local/etc/quagga/ripngd.conf /usr/local/sbin/ospfd -u root -d -f /usr/local/etc/quagga/ospfd.conf /usr/local/sbin/ospf6d -u root -d -f /usr/local/etc/quagga/ospf6d.conf /usr/local/sbin/bgpd -u root -d -f /usr/local/etc/quagga/bgpd.conf /usr/local/sbin/isisd -u root -d -f /usr/local/etc/quagga/isisd.conf Note: 8021q module does not have to be loaded to get trunk working with Openvswitch. It has to be loaded when creating sub-interfaces using vcong (h,p://linux.about.com/library /cmd/blcmdl8_vcong.htm) command is needed. On the other hand, choosing the right emulated network card in GNS3 Qemu se,ings is crucial to get trunk working. You cannot miss with i82557b ethernet card. Note: Iproute2 extension is needed to show multiple equal-cost routes presented in Linux kernel. The topic is described more in detail here. h,p://brezular.wordpress.com/2011/11/14/part2-testing-equal-cost-routes-in-linux-microcore-4-0-2/ (h,p://brezular.wordpress.com/2011/11/14/part2-testing-equal-cost-routes-in-linux-microcore-4-0-2/)
15 de 48
02/10/2013 08:01 a.m.
(h,p://brezular.les.wordpress.com/2011/11/campus_network.jpg)
Part1 Core Layer

Core layer consists from two Multilayer Core switches Core1 nad Core2. They are connected with pointto-point layer3 links. The full-mesh topology (h,p://en.wikipedia.org/wiki/Network_topology) between Core and Distribution layer provides path even in case of failure of two links. For example, if both interfaces eth4 and eth1 fail on switch Distrib1 there is still path to Core Layer via interface eth0. Quagga routing daemon is running on all Core and Distribution switches. It oers routing capabilities using OSPF routing protoco (h,p://en.wikipedia.org/wiki/Open_Shortest_Path_First)l. Use vtysh (h,p://linux.die.net/man/1/vtysh) Quagga shell to congure Core switches as following. 1. Command for accessing Quagga CLI tc@box:~$ sudo /usr/local/bin/vtysh Hello, this is Quagga (version 0.99.20). Copyright 1996-2005 Kunihiro Ishiguro, et al. 2. Congure hostname, interfaces and OSPF routing protocol for Core1 box# congure terminal box(cong)# hostname Core1 Core1(cong)# interface eth0 Core1(cong-if)# description Link to Core2 Core1(cong-if)# ip address 10.10.10.5/30 Core1(cong-if)# no shutdown Core1(cong-if)# exit Core1(cong)# interface eth2 Core1(cong-if)# description Link to Distrib2 Core1(cong-if)# ip address 10.10.10.18/30 Core1(cong-if)# no shutdown Core1(cong-if)# exit
02/10/2013 08:01 a.m.
16 de 48
Core1(cong)# interface eth1 Core1(cong-if)# description Link to Distrib1 Core1(cong-if)# ip address 10.10.10.10/30 Core1(cong-if)# no shutdown Core1(cong-if)# exit Core1(cong)# router ospf Core1(cong-router)# network 10.10.10.4/30 area 0 Core1(cong-router)# network 10.10.10.16/30 area 0 Core1(cong-router)# network 10.10.10.8/30 area 0 Core1(cong-router)# do write Building Conguration Conguration saved to /usr/local/etc/quagga/zebra.conf Conguration saved to /usr/local/etc/quagga/ripd.conf Conguration saved to /usr/local/etc/quagga/ripngd.conf Conguration saved to /usr/local/etc/quagga/ospfd.conf Conguration saved to /usr/local/etc/quagga/ospf6d.conf Conguration saved to /usr/local/etc/quagga/bgpd.conf Conguration saved to /usr/local/etc/quagga/isisd.conf [OK] Now, exit from vtysh. As Quagga had not saved hostname Core1 to /usr/local/etc/quagga/zebra.conf, congure it from Microcore CLI.. tc@box:~$ echo "hostname Core1" >> /opt/bootlocal.sh tc@box:~$ sudo hostname Core1 Force Microcore to save Quaggas conguration les in /usr/locac/etc/quagga/ and other les in /opt/.letool.lst. tc@Core1:~$ /usr/bin/letool.sh -b Save GNS3. Go to File-> Save or use Ctrl + s. 3. Congure hostname, interfaces and OSPF routing protocol for Core2 Start Core2 and congure router according to topology. Check if Core switches can see themselves as OSPF neighbours. Core2# show ip ospf neighbor Neighbor ID
17 de 48
Pri State
Dead Time Address
Interface
RXmtL RqstL DBsmL

02/10/2013 08:01 a.m.
10.10.10.18
1 Full/DR
35.785s 10.10.10.5
eth0:10.10.10.6
Check if routes are properly propagated in the routing table. Core2# show ip route ospf Codes: K kernel route, C connected, S static, R RIP, O OSPF, I ISIS, B BGP, > selected route, * FIB route O 10.10.10.4/30 [110/10] is directly connected, eth0, 00:08:20 O>* 10.10.10.8/30 [110/20] via 10.10.10.5, eth0, 00:08:10 O 10.10.10.12/30 [110/10] is directly connected, eth1, 00:00:27 O>* 10.10.10.16/30 [110/20] via 10.10.10.5, eth0, 00:08:10 O 10.10.10.20/30 [110/10] is directly connected, eth2, 00:08:20
Part2 Distribution Layer

Distribution layer consists from two Multilayer Distribution switches Distrib1 and Distrib2. The main job for Distribution switches is routing between dierent vlans subnet that are terminated here. Any trac ltering rules should be congured on Distribution swiches. Uplink interfaces connecting Distribution switches to Core switches are layer3 interfaces. They participate in OSPF messages forwarding. Downlink interfaces connecting Distribution switches to Access switches are Layer2 interfaces. They are trunks capable of carrying trac belongig to multiple VLANs. The next congurations steps are shown for Distrib1 switch only. Continue and make similiar conguration for Distrib2 switch. 1. Congure Layer3 interfaces and OSPF routing protocol on Distrib1 switch tc@box:~$ echo "hostname Distrib1" >> /opt/bootlocal.sh tc@box:~$ /usr/bin/letool.sh -b tc@box:~$ sudo hostname Distrib1 tc@Distrib1:~$ sudo /usr/local/bin/vtysh Distrib1# conf t Distrib1(cong)# hostname Distrib1 Distrib1(cong)# interface eth1 Distrib1(cong-if)# description Link to Core1 Distrib1(cong-if)# ip address 10.10.10.9/30
18 de 48
02/10/2013 08:01 a.m.
Distrib1(cong-if)# no shutdown Distrib1(cong-if)# exit Distrib1(cong)# int eth2 Distrib1(cong-if)# description Link to Core2 Distrib1(cong-if)# ip address 10.10.10.21/30 Distrib1(cong-if)# no shutdown Distrib1(cong-if)# exit Distrib1(cong)# int eth0 Distrib1(cong-if)# description Link to Distrib2 Distrib1(cong-if)# ip address 10.10.10.1/30 Distrib1(cong-if)# no shutdown Distrib1(cong-if)# exit Distrib1(cong)# router ospf Distrib1(cong-router)# network 10.10.10.0/30 area 0 Distrib1(cong-router)# network 10.10.10.20/30 area 0 Distrib1(cong-router)# network 10.10.10.8/30 area 0 Distrib1(cong-router)# exit Distrib1(cong)# do write Exit from vtysh and save content of /usr/local/etc/quagga/ directory. tc@box:~$ /usr/bin/letool.sh -b Ctrl + s 2. Congure Layer3 interfaces and OSPF routing protocol on Distrib2 switch Congure Distrib1 switch according to the topology. Check if Distrib2 can see all three OSPF neighbours. Distrib2# show ip ospf neighbor Neighbor ID Pri State 10.10.10.21 1 Full/DR 10.10.10.22 1 Full/DR 10.10.10.18 1 Full/DR Dead Time Address 38.389s 10.10.10.1 38.927s 10.10.10.14 33.153s 10.10.10.18 Interface RXmtL RqstL DBsmL eth0:10.10.10.2 0 0 0 eth1:10.10.10.13 0 0 0 eth2:10.10.10.17 0 0 0
Check if routes are properly propagated in the routing table of Quagga.

19 de 48 02/10/2013 08:01 a.m.
Distrib2# show ip route ospf Codes: K kernel route, C connected, S static, R RIP, O OSPF, I ISIS, B BGP, > selected route, * FIB route O 10.10.10.0/30 [110/10] is directly connected, eth0, 00:01:47 O>* 10.10.10.4/30 [110/20] via 10.10.10.14, eth1, 00:01:17 * via 10.10.10.18, eth2, 00:01:17 O>* 10.10.10.8/30 [110/20] via 10.10.10.1, eth0, 00:01:17 * via 10.10.10.18, eth2, 00:01:17 O 10.10.10.12/30 [110/10] is directly connected, eth1, 00:01:37 O 10.10.10.16/30 [110/10] is directly connected, eth2, 00:01:24 O>* 10.10.10.20/30 [110/20] via 10.10.10.1, eth0, 00:01:31 * via 10.10.10.14, eth1, 00:01:31 Exit from Quagga vtysh shell and check routing table of Linux Microcore. tc@Distrib2:~$ ip route show 10.10.10.0/30 dev eth0 proto kernel scope link src 10.10.10.2 10.10.10.4/30 proto zebra metric 20 nexthop via 10.10.10.14 dev eth1 weight 1 nexthop via 10.10.10.18 dev eth2 weight 1 10.10.10.8/30 proto zebra metric 20 nexthop via 10.10.10.1 dev eth0 weight 1 nexthop via 10.10.10.18 dev eth2 weight 1 10.10.10.12/30 dev eth1 proto kernel scope link src 10.10.10.13 10.10.10.16/30 dev eth2 proto kernel scope link src 10.10.10.17 10.10.10.20/30 proto zebra metric 20 nexthop via 10.10.10.1 dev eth0 weight 1 nexthop via 10.10.10.14 dev eth1 weight 1 127.0.0.1 dev lo scope link They are two euqal-cost paths for each of network 10.10.10.4/30, 10.10.10.8/30, 10.10.10.20/30 presented in kernel routing table of Distrib2. 3. Openvswitch Conguration congure Layer2 trunk ports, Layer3 VLAN interfaces on Distrib1 switch Openvswitch does not have separate CLI for its conguration thus all the conguration must be done from Microcore CLI. a) Congure eth3 and to become trunk port to allow carry trac from VLAN 10 and VLAN 20 tc@Distrib1:~$ sudo ovs-vsctl add-br br0 tc@Distrib1:~$ sudo ovs-vsctl add-port br0 eth3 trunks=10,20 b) Congure eth4 to become trunk port to allow carry trac from VLAN 30 and 40 tc@Distrib1:~$ sudo ovs-vsctl add-port br0 eth4 trunks=30,40
02/10/2013 08:01 a.m.
20 de 48
c) Create VLAN interfaces tc@Distrib1:~$ sudo ovs-vsctl add-port br0 vlan10 tag=10 -- set interface vlan10 type=internal tc@Distrib1:~$ sudo ovs-vsctl add-port br0 vlan20 tag=20 -- set interface vlan20 type=internal tc@Distrib1:~$ sudo ovs-vsctl add-port br0 vlan30 tag=30 -- set interface vlan30 type=internal tc@Distrib1:~$ sudo ovs-vsctl add-port br0 vlan40 tag=40 -- set interface vlan40 type=internal d) Check Openvswitch conguration tc@Distrib1:~$ sudo ovs-vsctl show a66779-0224-40ef-89f1-0deb21b939dBridge br0 Port eth3 trunks: [10, 20] Interface eth3 Port eth4 trunks: [30, 40] Interface eth4 Port vlan20 tag: 20 Interface vlan20 type: internal Port vlan10 tag: 10 Interface vlan10 type: internal Port br0 Interface br0 type: internal Port vlan30 tag: 30 Interface vlan30 type: internal Port vlan40 tag: 40 Interface vlan40 type: internal e) Congure IP addresses of VLAN interfaces and OSPF routing protocol on Distrib1 switch We have two options here either to use Linux kernel command ifcong (h,p://en.wikipedia.org /wiki/Ifcong) or Quagga vtysh shell. I chose vtysh no need to put commands to /opt/bootlocal.sh
21 de 48
02/10/2013 08:01 a.m.
tc@Distrib1:~$ sudo /usr/local/bin/vtysh Distrib1# conf t Distrib1(cong)# interface vlan10 Distrib1(cong-if)# ip address 192.168.10.2/24 Distrib1(cong-if)# no shutdown Distrib1(cong-if)# interface vlan20 Distrib1(cong-if)# ip address 192.168.20.2/24 Distrib1(cong-if)# no shutdown Distrib1(cong-if)# interface vlan30 Distrib1(cong-if)# ip address 192.168.30.2/24 Distrib1(cong-if)# no shutdown Distrib1(cong-if)# interface vlan40 Distrib1(cong-if)# ip address 192.168.40.2/24 Distrib1(cong-if)# no shutdown Distrib1(cong-if)# router ospf Distrib1(cong-router)# network 192.168.10.0/24 area 0 Distrib1(cong-router)# network 192.168.20.0/24 area 0 Distrib1(cong-router)# network 192.168.30.0/24 area 0 Distrib1(cong-router)# network 192.168.40.0/24 area 0 Distrib1(cong)# do write Exit from vtysh shell and save conguration. tc@Distrib1:~$ /usr/bin/letool.sh -b Ctrl + s. Do similar conguration for Distrib2 switch.
22 de 48
02/10/2013 08:01 a.m.
Part3 Access Layer

Access Layer consists of two Layer2 switches Access1 and Access2. VLANs are created here and switchports are assigned to VLANS. As each VLAN is restricted to the local Access switch we call them local VLANs. Primary task of Access layer is to provide switchports for end users and forward trac. Since Layer2 Access switches do not route between dierent VLANs, trac is sent to Distribution layer via 8021q trunks when routing is required. The design guides recommend a campus model with local VLANs when 20 percent of users trac stays in Campus and 80 percent of trac is forwarded outside campus. 1. Congure hostname, access ports for VLAN 10 and 20 and 8021q trunk ports on Access1 switch tc@box:~$ echo "hostname Access1" >> /opt/bootlocal.sh tc@box:~$ sudo hostname Access1 tc@Access1:~$ sudo ovs-vsctl add-br br0 tc@Access1:~$ sudo ovs-vsctl add-port br0 eth0 tag=10 tc@Access1:~$ sudo ovs-vsctl add-port br0 eth1 tag=20 tc@Access1:~$ sudo ovs-vsctl add-port br0 eth3 trunks=10,20 tc@Access1:~$ sudo ovs-vsctl add-port br0 eth4 trunks=10,20 2. Check Openvswitch conguration tc@Access1:~$ sudo ovs-vsctl show a66779-0224-40ef-89f1-0deb21b939db Bridge br0 Port br0 Interface br0 type: internal Port eth1 tag: 20 Interface eth1 Port eth4 trunks: [10, 20] Interface eth4 Port eth0 tag: 10 Interface eth0 Port eth3 trunks: [10, 20] Interface eth3
23 de 48 02/10/2013 08:01 a.m.
Save conguration. tc@Access1:~$ /usr/bin/letool.sh -b Ctrl + s. 3. Congure access ports for VLAN 30 and 40 and 8021q trunk ports on Access2 switch Congure Access2 switch according to topology. 4. Congure hostname and IP seetings for all hosts, PC1 is an example tc@box:~$ echo "hostname PC1" >> /opt/bootlocal.sh tc@box:~$ sudo hostname PC1 tc@PC1:~$ echo "ifconfig /opt/bootlocal.sh eth0 192.168.10.10 netmask 255.255.255.0 up" >>
tc@PC1:~$ sudo ifcong eth0 192.168.10.10 netmask 255.255.255.0 up tc@PC1:~$ echo "route add default gw 192.168.10.1" >> /opt/bootlocal.sh tc@PC1:~$ sudo route add default gw 192.168.10.1 tc@PC1:~$ /usr/bin/letool.sh -b 5. Test connectivity inside the same VLAN, PC1 is an example PC1 should ping IP address of vlan10 interface on both Distribution switches. tc@PC1:~$ ping 192.168.10.2 PING 192.168.10.2 (192.168.10.2): 56 data bytes 64 bytes from 192.168.10.2: seq=0 ,l=64 time=1.949 ms 64 bytes from 192.168.10.2: seq=1 ,l=64 time=2.342 ms 64 bytes from 192.168.10.2: seq=2 ,l=64 time=3.016 ms ^C 192.168.10.2 ping statistics 3 packets transmi,ed, 3 packets received, 0% packet loss round-trip min/avg/max = 1.949/2.435/3.016 ms tc@PC1:~$ ping 192.168.10.3 PING 192.168.10.3 (192.168.10.3): 56 data bytes 64 bytes from 192.168.10.3: seq=0 ,l=64 time=2.297 ms 64 bytes from 192.168.10.3: seq=1 ,l=64 time=2.470 ms 64 bytes from 192.168.10.3: seq=2 ,l=64 time=2.815 ms ^C 192.168.10.3 ping statistics 3 packets transmi,ed, 3 packets received, 0% packet loss round-trip min/avg/max = 2.297/2.527/2.815 ms2
24 de 48 02/10/2013 08:01 a.m.
Part4 Virtual Router Redundancy Protocol /VRRP/ Configuration

Things look good now. We can ping a virtual interfaces on Distribution switches from host residing in the same VLAN. But we still cannot ping any IP address residing in dierent subnet than host. It is because both Distribution switches have not been congured to act as default gateway yet. You might have noticed that all the hosts were congured with IP address of default gateway 192.168.x.1. This is a virtual IP address created by Keepalived extension. Keepalived oers default gateways redundancy using VRRP protocol. In case of failure one of the Distribution switches, the other Distrib switch takes responsibility and continues to forward packets. Switch with higher priority is called Master and forwards packet. At least it is always one Master switch in one VRRP group. For example, in VRRP group 10 it is Distrib1 with priority 150. Switch with lower priority is called Backup and as it was said, it forwards packets only if Master switch fails. The Backup switch in VRRP group 10 is Distrib2 with priority 100. The higher is priority, more likely switch becomes Master switch. Obviously, it must be communication between switches to let known each other about its existence. By default every one second Advertisiment is sent from Master switch to each memember of VRRP group to multicast IP address 224.0.0.18. After three missing Advertisiment plus screw time, Backup switch knowns that Master is down and transition from Backup to Master state occurs. It forwards packets now. Note: Virtual IP address is tied with virtual MAC address 00-00-5E-00-01-XX. The last byte of the address (XX) is the Virtual Router Identier (VRID), which is dierent for each VRRP instance in the network. This address is used by only one physical router at a time, and it will reply with this MAC address when an ARP request is sent for the virtual routers IP address. If Master fails, the new Master will broadcast Gratious ARP containing the virtual router MAC address for the associated IP address. If I understand it correctly, nothing has been change in host conguration but Access switches had changed their CAM table frames with destination MAC address 00-00-5E-00-01-XX will be send via new path to the new MASTER router. Keepalived works slightly dierently it uses real MAC address of interface instead of Virtual MAC address. Check ARP cache of hosts in VRRP testing section it seems that the current Keepalived/VRRP does not support Virtual MAC addresses. 1. VRRP conguration on switch Distrib1 tc@Distrib1:~$ sudo su root@Distrib1:~# mkdir /usr/local/etc/keepalived/ root@Distrib1:~# echo "/usr/local/etc/keepalived/" >> /opt/.filetool.lst
25 de 48
02/10/2013 08:01 a.m.
root@Distrib1:~# vi /usr/local/etc/keepalived/keepalived.conf vrrp_instance 10 { state MASTER interface vlan10 virtual_router_id 10 priority 150 authentication { auth_type PASS auth_pass Campus123 } # VRRP instance declaration # Start-up default state # Binding interface # VRRP VRID # VRRP PRI # Simple Passwd or IPSEC AH # Password string
virtual_ipaddress { # VRRP IP address block 192.168.10.1/24 brd 192.168.10.255 dev vlan10 } } vrrp_instance 20 { state BACKUP interface vlan20 virtual_router_id 20 priority 100 authentication { auth_type PASS auth_pass Campus123 } virtual_ipaddress { 192.168.20.1/24 brd 192.168.20.255 dev vlan20 } } vrrp_instance 30 { state BACKUP interface vlan30 virtual_router_id 30 priority 100 authentication { auth_type PASS auth_pass Campus123 } virtual_ipaddress { 192.168.30.1/24 brd 192.168.30.255 dev vlan30 } } vrrp_instance 40 {
26 de 48 02/10/2013 08:01 a.m.
state MASTER interface vlan40 virtual_router_id 40 priority 150 authentication { auth_type PASS auth_pass Campus123 } virtual_ipaddress { 192.168.40.1/24 brd 192.168.40.255 dev vlan40 } } Start Keepalived daemon. As Distrib2 is not congured for VRRP, all the VRRP instances should transit to MASTER state. root@Distrib1:~# /usr/local/sbin/keepalived -P -f /usr/local/etc/keepalived/keepalived.conf Make Keepalived daemon to be started after boot of Microcore. root@Distrib1:~# echo "/usr/local/sbin/keepalived -P -f /usr/local/etc/keepalived /keepalived.conf" >> /opt/bootlocal.sh Save Keepalived conguration le. root@Distrib1:~# /usr/bin/letool.sh -b Note: After each change in keepalived.conf le you have to restart keepalived daemon to accept changes. 2. VRRP conguration on switch Distrib2 tc@Distrib2:~$ sudo su root@Distrib2:~# mkdir /usr/local/etc/keepalived/ root@Distrib2:~# echo "/usr/local/etc/keepalived/" >> /opt/.filetool.lst root@Distrib2:~# vi /usr/local/etc/keepalived/keepalived.conf vrrp_instance 10 { state BACKUP interface vlan10 virtual_router_id 10 priority 100 authentication { auth_type PASS auth_pass Campus123 } # VRRP instance declaration # Start-up default state # Binding interface # VRRP VRID # VRRP PRI # Simple Passwd or IPSEC AH # Password string
27 de 48
02/10/2013 08:01 a.m.
virtual_ipaddress { # VRRP IP address block 192.168.10.1/24 brd 192.168.10.255 dev vlan10 } } vrrp_instance 20 { state MASTER interface vlan20 virtual_router_id 20 priority 150 authentication { auth_type PASS auth_pass Campus123 } virtual_ipaddress { 192.168.20.1/24 brd 192.168.20.255 dev vlan20 } } vrrp_instance 30 { state MASTER interface vlan30 virtual_router_id 30 priority 150 authentication { auth_type PASS auth_pass Campus123 } virtual_ipaddress { 192.168.30.1/24 brd 192.168.30.255 dev vlan30 } } vrrp_instance 40 { state BACKUP interface vlan40 virtual_router_id 40 priority 100 authentication { auth_type PASS auth_pass Campus123 } virtual_ipaddress { 192.168.40.1/24 brd 192.168.40.255 dev vlan40
28 de 48
02/10/2013 08:01 a.m.
} } Start keepalived daeomon. root@Distrib2:~# /usr/local/sbin/keepalived -P -f /usr/local/etc/keepalived/keepalived.conf Distrib1 should become BACKUP router in VRRP group 20 and 30 and stays in MASTER state in VRRP 10 and 40. Similarly, DOSTRIB2 is BACKUP router in VRRP group 10 and 40 and MASTER in VRRP group 20 and 30. Make keepalived daeomon to be started after boot of Microcore. root@Distrib2:~# echo "/usr/local/sbin/keepalived -P -f /usr/local/etc/keepalived /keepalived.conf" >> /opt/bootlocal.sh Save keepalived conguration le. root@Distrib2:~# /usr/bin/letool.sh -b
Part5 Testing
Connectivity should be established between any two nodes in our Campus network and we are going to test it. 1. Test routing between VLANs Issue ping from host PC1 in VLAN10 to host P2, PC3, PC4 and check if intervlan routing is working. tc@PC1:~$ ping 192.168.20.1 PING 192.168.20.10 (192.168.20.10): 56 data bytes 64 bytes from 192.168.20.10: seq=0 ,l=63 time=14.443 ms 64 bytes from 192.168.20.10: seq=1 ,l=63 time=4.257 ms 64 bytes from 192.168.20.10: seq=2 ,l=63 time=4.962 ms ^C 192.168.20.10 ping statistics 3 packets transmi,ed, 3 packets received, 0% packet loss round-trip min/avg/max = 4.257/7.887/14.443 ms tc@PC1:~$ ping 192.168.30.10 PING 192.168.30.10 (192.168.30.10): 56 data bytes 64 bytes from 192.168.30.10: seq=0 ,l=63 time=9.720 ms 64 bytes from 192.168.30.10: seq=1 ,l=63 time=4.125 ms 64 bytes from 192.168.30.10: seq=2 ,l=63 time=4.920 ms ^C
29 de 48
02/10/2013 08:01 a.m.
192.168.30.10 ping statistics 3 packets transmi,ed, 3 packets received, 0% packet loss round-trip min/avg/max = 4.125/6.255/9.720 ms tc@PC1:~$ ping 192.168.40.10 PING 192.168.40.10 (192.168.40.10): 56 data bytes 64 bytes from 192.168.40.10: seq=0 ,l=63 time=8.404 ms 64 bytes from 192.168.40.10: seq=1 ,l=63 time=7.604 ms 64 bytes from 192.168.40.10: seq=2 ,l=63 time=4.798 ms ^C 192.168.40.10 ping statistics 3 packets transmi,ed, 3 packets received, 0% packet loss round-trip min/avg/max = 4.798/6.935/8.404 ms0 2. Check a routing table of Core switch Check if redundant routes are presented in kernel routing table of Core1. Non-redundant routes such as 10.10.10.4/30, 10.10.10.8/30, 10.10.10.16/30 and 127.0.0.1 are directly connected networks. Two equal-cost path should exist to other network learned by OSPF routing protocol. tc@Core1:~$ ip route show 10.10.10.0/30 proto zebra metric 20 nexthop via 10.10.10.9 dev eth1 weight 1 nexthop via 10.10.10.17 dev eth2 weight 1 10.10.10.4/30 dev eth0 proto kernel scope link src 10.10.10.5 10.10.10.8/30 dev eth1 proto kernel scope link src 10.10.10.10 10.10.10.12/30 proto zebra metric 20 nexthop via 10.10.10.6 dev eth0 weight 1 nexthop via 10.10.10.17 dev eth2 weight 1 10.10.10.16/30 dev eth2 proto kernel scope link src 10.10.10.18 10.10.10.20/30 proto zebra metric 20 nexthop via 10.10.10.6 dev eth0 weight 1 nexthop via 10.10.10.9 dev eth1 weight 1 127.0.0.1 dev lo scope link 192.168.10.0/24 proto zebra metric 20 nexthop via 10.10.10.9 dev eth1 weight 1 nexthop via 10.10.10.17 dev eth2 weight 1 192.168.20.0/24 proto zebra metric 20 nexthop via 10.10.10.9 dev eth1 weight 1 nexthop via 10.10.10.17 dev eth2 weight 1 192.168.30.0/24 proto zebra metric 20 nexthop via 10.10.10.9 dev eth1 weight 1 nexthop via 10.10.10.17 dev eth2 weight 1 192.168.40.0/24 proto zebra metric 20 nexthop via 10.10.10.9 dev eth1 weight 1 nexthop via 10.10.10.17 dev eth2 weight 1 3. Check connectivity between Access and Core layer
30 de 48 02/10/2013 08:01 a.m.
Issue ping from host PC1 in VLAN10 to eth0 interface of Core2. tc@PC1:~$ ping 10.10.10.6 PING 10.10.10.6 (10.10.10.6): 56 data bytes 64 bytes from 10.10.10.6: seq=0 ,l=63 time=7.052 ms 64 bytes from 10.10.10.6: seq=1 ,l=63 time=4.383 ms 64 bytes from 10.10.10.6: seq=2 ,l=63 time=5.246 ms ^C 10.10.10.6 ping statistics 3 packets transmi,ed, 3 packets received, 0% packet loss round-trip min/avg/max = 4.383/5.560/7.052 ms6 4. VRRP testing Issue ping from PC1 to IP address of VLAN10 interfaces 192.168.10.2 and 192.168.10.3 on both Distribution switches. Stop ping sequence and ping VRRP Virtual IP address 192.168.10.1. Now check ARP cache of PC1. tc@PC1:~$ arp ? (192.168.10.2) at 00:23:20:bc:57:67 [ether] on eth0 ? (192.168.10.1) at 00:23:20:bc:57:67 [ether] on eth0 ? (192.168.10.3) at 00:23:20:8c:1d:cf [ether] on eth Virtual IP address 192.168.10.1 has assigned MAC address of VLAN10 interface 00:23:20:bc:57:67 on Distrib1 switch. Packets destined outside VLAN10 will leave subnet 192.168.10.0/24 via vlan10 interface on Distrib1. We have expected it because Distrib1 is MASTER router for VRRP group 10. Now start to ping IP address of eth0 interface of Core2 10.10.10.6/30 from PC1. We are going to kill keepalived proccess on Distrib1 switch to check if Disrib2 will transit to MASTER state for VRRP group 10 and 40. List the keepalived proccess on Distrib1 switch root@Distrib1:~# ps -ef | grep keepalived 2549 root 2550 root 4937 root /usr/local/sbin/keepalived -P -f /usr/local/etc/keepalived/keepalived.conf /usr/local/sbin/keepalived -P -f /usr/local/etc/keepalived/keepalived.conf grep keepalived
Now, kill Keepalived process. root@Distrib1:~# kill 2549 Keepalived: Terminating on signal Keepalived: Stopping Keepalived v1.2.2 (06/28,2011) Keepalived_vrrp: Terminating VRRP child process on signal Check the output of console of Distrib2 switch. Information messages tell us about transition of Distrib2 switch to MASTER state for VRRP group 10 and 40.
31 de 48 02/10/2013 08:01 a.m.
Keepalived_vrrp: VRRP_Instance(10) Transition to MASTER STATE Keepalived_vrrp: VRRP_Instance(40) Transition to MASTER STATE Keepalived_vrrp: VRRP_Instance(10) Entering MASTER STATE Keepalived_vrrp: VRRP_Instance(40) Entering MASTER STATE Check ARP cache PC1 again. It is not big surprise that Virtual IP address 192.168.10.1 has assigned MAC address 00:23:20:8c:1d:cf of VLAN10 interface of Distrib2 switch. tc@PC1:~$ arp ? (192.168.10.2) at 00:23:20:bc:57:67 [ether] on eth0 ? (192.168.10.1) at 00:23:20:8c:1d:cf [ether] on eth0 ? (192.168.10.3) at 00:23:20:8c:1d:cf [ether] on eth0p It seems that series of ping requests from PC1 to 10.10.10.6 has not been broken by transition. 10.10.10.6 ping statistics 206 packets transmi,ed, 206 packets received, 0% packet loss round-trip min/avg/max = 1.664/4.343/15.437 ms. Start keepalived daemon on Distrib1 switch again. Distrib1 is going immediately to state MASTER for VRRP group 10 and 40 and to BACKUP for VRRP group 20 and 30. root@Distrib1:~# /usr/local/sbin/keepalived -P -f /usr/local/etc/keepalived/keepalived.conf Keepalived: Starting VRRP child process, pid=5134 Keepalived_vrrp: Registering Kernel netlink reector Keepalived_vrrp: Registering Kernel netlink command channel Keepalived_vrrp: Registering gratutious ARP shared channel Keepalived_vrrp: Opening le /usr/local/etc/keepalived/keepalived.conf. Keepalived_vrrp: Conguration is using : 45592 Bytes Keepalived_vrrp: Using LinkWatch kernel netlink reector Keepalived_vrrp: VRRP_Instance(20) Entering BACKUP STATE Keepalived_vrrp: VRRP_Instance(30) Entering BACKUP STATE Keepalived_vrrp: VRRP_Instance(10) Transition to MASTER STATE Keepalived_vrrp: VRRP_Instance(40) Transition to MASTER STATE Keepalived_vrrp: VRRP_Instance(10) Entering MASTER STATE Keepalived_vrrp: VRRP_Instance(40) Entering MASTER STATE Distrib2 received VRRP Advertisiment with priority higher than its own and transit to BACKUP state for VRRP instance 10 and 40. Keepalived_vrrp: VRRP_Instance(10) Received higher prio advert Keepalived_vrrp: VRRP_Instance(10) Entering BACKUP STATE Keepalived_vrrp: VRRP_Instance(40) Received higher prio advert Keepalived_vrrp: VRRP_Instance(40) Entering BACKUP STATE. Also in this case ping started on PC1 was not interrupted by transition back to MASTER state on Distrib1 for VRRP group 10.
32 de 48
02/10/2013 08:01 a.m.
END. h,p://en.wikipedia.org/wiki/Virtual_Router_Redundancy_Protocol /wiki/Virtual_Router_Redundancy_Protocol) h,p://www.estoile.com/links/vrrp.htm (h,p://www.estoile.com/links/vrrp.htm) h,p://osdir.com/ml/linux.keepalived.devel/2005-01/msg00000.html /ml/linux.keepalived.devel/2005-01/msg00000.html) (h,p://en.wikipedia.org
(h,p://osdir.com
FILED UNDER OPENVSWITCH TAGGED WITH ACCESS LAYER, CAMPUS MODEL, CORE LAYER, CORE LINUX, DISTRIBUTION LAYER, KEEPALIVED, MICROCORE, OPENVSWITCH, QEMU, QUAGGA
Part1 OPENVSWICH Creating and Submitting Openvswitch Extension To Tinycore Upstream

SEPTEMBER 3, 2011 8 COMMENTS (HTTP://BREZULAR.WORDPRESS.COM/2011/09/03/PART1OPENVSWICH-CREATING-AND-SUBMITTING-OPENVSWITCH-EXTENSION-TO-MICROCOREUPSTREAM/#COMMENTS)
i 2 Votes In May 2011, I read a request (h,p://www.gns3.net/phpBB /topic3549.html?sid=9da8d8dc809d413978e92277b9205d90) for installation Openvswitch (h,p://openvswitch.org/) on Qemu image. I started to play with Openvswitch and nally became fan of this project. I realized how powerful can be Openvswitch, oering many features listed here (h,p://openvswitch.org/). Spending my time playing with Microcore and Openvswitch after several days I was able to create Microcore Qemu image with 8021q (h,p://brezular.wordpress.com/2011/06/06/linuxmicrocore-kernel-compilation-for-802-1q-support/) support and install Openvswitch on the top of it. In the tutorial I would like to show how to create Openvswitch extension (h,p://wiki.tinycorelinux.net /wiki:creating_extensions) and make it ready for submit to the Tinycore upstream. I chose Core Linux, because I am familiar with this minimal Linux distribution and the Core is incredibly small. The following steps describe installation Openvswitch on Qemu image with pre-installed Microcore Linux. I also created three labs using Openvswitch. I tested how can Openvswitch works with VLANs, 802.1q trunk ports, if it was capable of creating L3 VLAN interfaces and Inter VLAN routing was working. Labs are available here (h,p://brezular.wordpress.com/2011/06/25/part2-openvswich-vlans-trunks-l3-vlan-
33 de 48
02/10/2013 08:01 a.m.
interface-intervlan-routing-conguration-and-testing/): 1. Start Qemu Microcore image Assuming your Microcore Qemu image with console support (h,p://brezular.wordpress.com/2011/01 /26/how-to-setup-linux-microcore-3-x-router-qemu-image-in-fedora-linux-part1/) has been created and supports 8021q VLAN tagging (h,p://brezular.wordpress.com/2011/06/06/linux-microcore-kernelcompilation-for-802-1q-support/), start the image: qemu-kvm /home/brezular/linux-microcore-4.7.7.img -m 512 2. Download Openvswitch and Extract it wget hAp://openvswitch.org/releases/openvswitch-1.11.0.tar.gz (hAp://openvswitch.org/releases /openvswitch-1.11.0.tar.gz) tar zxvf ./openvswitch-1.11.0.tar.gz cd ./openvswitch-1.1.0 3. Install Necessary Dependences for Openvswitch Compilation tce-load -w -i compiletc.tcz python.tcz perl5.tcz openssl-1.0.0-dev.tcz tunctl.tcz bridge-utils.tcz linuxheaders-3.0.21-tinycore.tcz To compile openvswitch with the ovsdmonitor tool, the machine must also have the following installed software: tce-load -w -i python_twisted-2.7.tcz python-simplejson.tcz python_zope.interface-2.7 pyqt.tcz Note: The kernel source will be saved in /lib/modules/`uname -r`/build It is recommended to check the list of necessary applications here (h,p://git.openvswitch.org/cgi-bin /gitweb.cgi?p=openvswitch;a=blob_plain;f=INSTALL;hb=HEAD): 4. Openvswitch Installation It is recommended to start here (h,p://wiki.tinycorelinux.net/wiki:creating_extensions) and continue here (h,p://git.openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob_plain;f=INSTALL;hb=HEAD) :-) After that you can type following: export CFLAGS="-march=i486 -mtune=i686 -Os -pipe" export CXXFLAGS="-march=i486 -mtune=i686 -Os -pipe" export LDFLAGS="-Wl,-O1" cd ./openvswitch-1.11.0 ./configure --prefix=/usr/local --with-linux=/lib/modules/`uname -r`/build make -j2 sudo su
34 de 48
02/10/2013 08:01 a.m.
cd /home/tc/openvswitch-1.11.0/ make DESTDIR=/tmp/openvswitch install 5. BACKUP and LOAD Module openvswitch.ko a) BACKUP module openvswitch.ko After building, a kernel module openvswitch.ko will be saved in a ./datapath/linux/ directory. Weve built both userspace and kernel module as well. The performance is be,er when a kernel module is used. Create a directory in a structure of the openvswitch extension where the kernel module openvswitch.ko will be saved. sudo mkdir -p /tmp/openvswitch/usr/local/lib/modules/3.0.21-tinycore/kernel/openvswitch/ Copy the kernel module to the directory. sudo cp /home/tc/openvswitch-1.11.0/datapath/linux/openvswitch.ko /tmp/openvswitch/usr/local /lib/modules/3.0.21-tinycore/kernel/openvswitch/ b) Check if module can be loaded correctly sudo mkdir -p /usr/local/lib/modules/3.0.21-tinycore/kernel/openvswitch/ sudo cp /home/tc/openvswitch-1.11.0/datapath/linux/openvswitch.ko /usr/local/lib/modules/3.0.21tinycore/kernel/openvswitch/ depmod -a sudo modprobe openvswitch Check if module openvswitch_mod.ko is loaded to the kernel with lsmod command. lsmod | grep openvswitch openvswitch 49152 0 6. Backup /home/tc/openvswitch-1.11.0/vswitchd/vswitch.ovsschema In order to initialize the conguration database using ovsdb-tool later, the le /home/tc/openvswitch1.11.0/vswitchd/vswitch.ovsschema is needed. You need to copy it to /tmp/openvswitch/usr/local /etc/openvswitch/vswitchd/ to become part of the extension. sudo mkdir -p /tmp/openvswitch/usr/local/etc/openvswitch/vswitchd/ sudo cp /home/tc/openvswitch-1.11.0/vswitchd/vswitch.ovsschema /tmp/openvswitch/usr/local /etc/openvswitch/vswitchd/ 7. Create Openvswitch Extension a) Remove unneccessary les sudo rm -rf /tmp/openvswitch/usr/local/share/man/
35 de 48
02/10/2013 08:01 a.m.
b) Install squashfs and create openvswitch-3.0.21-tinycore.tcz extension tce-load -w -i squashfs-tools-4.x sudo su cd /tmp/ mksquashfs openvswitch/ openvswitch-3.0.21-tinycore.tcz c) Create a list of les presented in extension sudo su cd /tmp/openvswitch/ nd usr -not -type d > ../openvswitch-3.0.21-tinycore.tcz.list cd .. d) Create md5 check sum of openvswitch-3.0.21-tinycore.tcz md5sum openvswitch-3.0.21-tinycore.tcz > openvswitch-3.0.21-tinycore.tcz.md5.txt e) Create openvswitch-3.0.21-tinycore.tcz.info An info le describing its contents (.tcz.info) this content is standardized. Check repository (h,p://distro.ibiblio.org/tinycorelinux/4.x/x86/tcz/) for examples. f) Create openvswitch-3.0.21-tinycore.tcz.build-dep le Additional build instructions in a plain text le for future reference, mentioning such things as which extensions are required to build the package and what compile ags were used. g) Create the dependency list openvswitch-3.0.21-tinycore.tcz.dep List of the extensions that have to be presented to run openvswitch extension correctly. h) Install openssh.tcz an extension and copy all openvswitch-3.0.21-tinycore.tcz.* les and a source le to the guest system tce-load -wi openssh cp -rv /home/tc/openvswitch-1.11.0.tar.gz /tmp scp -rv /tmp/openvswitch-3.0.21-tinycore.* brezular@10.0.2.2:/home/brezular/ scp -rv /tmp/openvswitch-1.11.0.tar.gz brezular@10.0.2.2:/home/brezular/ 8. Test and Submit Openvswitch Extension a) Create the new clean Microcore Qemu image without any extensions installed Assuming the image is ready to start, run it: qemu-kvm /home/brezular/linux-microcore-4.7.7-clean.img -m 512 b) Install submitqc.tcz and openssh.tcz
36 de 48
02/10/2013 08:01 a.m.
tce-load -wi submitqc4.tcz openssh.tcz c) Copy the openvswitch les you have created in steps 1 7 (.tcz, .list, .md5.txt, .info, .dep, build-dep) from old Qemu image to the clean Qemu image, to the directory /tmp/All. mkdir /tmp/All cd /tmp/All scp -rv brezular@10.0.2.2:/home/brezular/linux-core-nemaz/extensions/openvswitch/* . Run an extension testing script: sudo submitqc4 The script checks all openvswitch les in /tmp/All and creates the directory /tmp/submitqc/ with log les. d) Send extension sudo mv /tmp/submitqc /tmp/All cd /tmp/All sudo tar zcvf openvswitch-1.11.0-extension.tar.gz * Send openvswitch-1.11.0-extension.tar.gz to tcesubmit@gmail.com e) Copy new created les to the guest sudo scp -rv submitqc/ brezular@10.0.2.2:/home/brezular/ sudo scp -rv openvswitch-3.0.21-tinycore.tcz.zsync brezular@10.0.2.2:/home/brezular/ sudo scp -rv openvswitch-1.11.0-extension.tar.gz brezular@10.0.2.2:/home/brezular/ 9. Openvswitch After-Install Conguration a) Make openvswitch, 8021q, ipv6 modules to be loaded to a kernel during boot of Microcore echo "modprobe openvswitch" >> /opt/bootlocal.sh echo "modprobe 8021q" >> /opt/bootlocal.sh echo "modprobe ipv6" >> /opt/bootlocal.sh sudo modprobe openvswitch_mod sudo modprobe 8021q sudo modprobe ipv6 b) Initialize the conguration database using ovsdb-tool Check if a directory /usr/local/etc/openvswitch/ exists, if not create it. sudo mkdir -p /usr/local/etc/openvswitch/ Create conf.db conguration le.
37 de 48
02/10/2013 08:01 a.m.
sudo ovsdb-tool create /usr/local/etc/openvswitch/conf.db /usr/local/etc/openvswitch/vswitchd /vswitch.ovsschema Add /usr/local/etc/openvswitch/ to the list of after restart kept les. echo "/usr/local/etc/openvswitch/" >> /opt/.filetool.lst c) Make ovsdb-server to be started after start of the Microcore vi /opt/bootlocal.sh /usr/local/sbin/ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \ --remote=db:Open_vSwitch,Open_vSwitch,manager_options \ --private-key=db:Open_vSwitch,SSL,private_key \ --certificate=db:Open_vSwitch,SSL,certificate \ --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert \ --pidfile --detach :wq! Note: If you are not familiar with vi editor, use the reference below, please: h,p://nemesis.lonestar.org/reference/docs/vi.html (h,p://nemesis.lonestar.org/reference/docs/vi.html) d) Make the database initialialization using ovs-vsctl This is only necessary the rst time after you create the database with ovsdb-tool (but running it at any time is harmless). echo "/usr/local/bin/ovs-vsctl --no-wait init" >> /opt/bootlocal.sh e) Make the main Open vSwitch daemon being started, telling it to connect to the same Unix domain socket echo "/usr/local/sbin/ovs-vswitchd --pidfile --detach" >> /opt/bootlocal.sh f) Enable IPv4 and IPV6 packets forwarding between interfaces Although not directly connected with Openvswitch conguration we need to enable ipv4 and ipv6 packets forwarding between interfaces for Microcore. This option is disabled in kernel by default. sudo sysctl -w net.ipv4.ip_forward=1 sudo sysctl -w net.ipv6.conf.all.forwarding=1 echo "sysctl -w net.ipv4.ip_forward=1" >> /opt/bootlocal.sh echo "sysctl -w net.ipv6.conf.all.forwarding=1" >> /opt/bootlocal.sh g) Run commands you have entered into /opt/bootlocal.sh and make them persistent after restart of the Microcore sudo /opt/bootlocal.sh
38 de 48
02/10/2013 08:01 a.m.
Optionally: It is recommended to delete the history of used commands. echo "" > /home/tc/.ash_history Save changes made for les and directories which are listed in /opt/.letool.lst /usr/bin/letool.sh -b 10. Conguration example Now you may use ovs-vsctl to set up bridges and other Open vSwitch features. For example, to create a bridge named br0 and add ports eth0, eth1 and eth2 to it: sudo ovs-vsctl add-br br0 sudo ovs-vsctl add-port br0 eth0 sudo ovs-vsctl add-port br0 eth1 sudo ovs-vsctl add-port br0 eth2 Before shutdown you always force Core to save conguration changes in the openvswitch database le /usr/local/etc/openvswitch/conf.db. Use the command: /usr/bin/letool.sh -b Reference h,p://openvswitch.org/ (h,p://openvswitch.org/) h,p://openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob_plain;f=INSTALL.userspace;hb=HEAD (h,p://openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob_plain;f=INSTALL.userspace;hb=HEAD) h,p://openvswitch.org/?page_id=14 (h,p://openvswitch.org/?page_id=14) h,p://openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob_plain;f=INSTALL.Linux;hb=HEAD (h,p://openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob_plain;f=INSTALL.Linux;hb=HEAD) FILED UNDER OPENVSWITCH MICROCORE TAGGED WITH CORE LINUX, EXTENSION, LINUX SWITCH,
Part2 OPENVSWICH VLANs, Trunks, L3 VLAN interface, InterVLAN Routing Configuration And Testing
JUNE 25, 2011 17 COMMENTS (HTTP://BREZULAR.WORDPRESS.COM/2011/06/25/PART2OPENVSWICH-VLANS-TRUNKS-L3-VLAN-INTERFACE-INTERVLAN-ROUTINGCONFIGURATION-AND-TESTING/#COMMENTS)
39 de 48
02/10/2013 08:01 a.m.
i 1 Vote In a previous tutorial we showed how to install Openvswitch on Qemu image with Microcore Linux. At the end of tutorial we created Openvswitch extension (h,p://brezular.wordpress.com/2011/06/25/part1openvswich-creating-and-submi,ing-extension-to-microcore-upstream/) and submi,ed it to Microcore upstream. Assuming that Openvswitch is congured and functional, we are ready to make three labs which helps us to test features such as VLANs, trunks and interVLAN routing.
LAB 1 Access VLAN Configuration And Testing

In this lab we are going to test connectivity between PCs which reside in same VLAN. PC1 and PC2 are assigned to VLAN 10 and PC3 and PC4 reside in VLAN 20. According to theory, each VLAN should have its own separate subnet. In our case, we are going to test not only connectivity between computers in the same VLANs but also to test connectivity between PCs placed in dierent VLAN. For this reason I chose one subnet 192.168.1.0/24 for all the computers so L3 connectivity exists between e.g. PC1 (VLAN 10) and PC3 (VLAN20). Of course, the computers assigned to dierent VLANs are separated on Layer 2 so we do not expect any connectivity between those computers. (h,p://brezular.les.wordpress.com/2011/06/access_vlans.jpeg)
(h,p://brezular.les.wordpress.com/2011/06/access_vlan1.jpeg) Figure 1 Openvswitch with congured VLANs on its access ports click image to enlarge 1) Openvswitch conguration Login is tc without password set.
40 de 48
02/10/2013 08:01 a.m.
a) Lets create bridge br0 sudo ovs-vsctl add-br br0 b) Add access port eth0 to the bridge br0 and assign it to VLAN 10 sudo ovs-vsctl add-port br0 br0 eth0 tag=10 c) Assign remaining ports to the br0 sudo ovs-vsctl add-port br0 eth2 tag=10 sudo ovs-vsctl add-port br0 eth1 tag=20 sudo ovs-vsctl add-port br0 eth3 tag=20 d) Assign hostname to Microcore sudo hostname openvswitch echo "hostname openvswitch" >> /opt/bootlocal.sh e) Save conf.db le to keep it persistent after the next reboot of Microcore Linux /usr/bin/letool.sh -b 2) PC1 conguration Login is tc without password set. Assign IP address 192.168.1.1/24 to eth0 and make it persistent after next reboot of Microcore sudo hostname PC1 sudo ifcong eth0 192.168.1.1 netmask 255.255.255.0 echo "hostname PC1" >> /opt/bootlocal.sh echo "ifconfig eth0 192.168.1.1 netmask 255.255.255.0" >> /opt/bootlocal.sh /usr/bin/letool.sh -b 3) PC2 to PC4 conguration Congure all remaining computers with correct IP paramters, similary as it was done in point 2).. 4) Access VLAN connectivity test a) Connectivity test between PC which are residing in the same VLAN Issue ping from PC1 (VLAN10) to PC2 (VLAN10). tc@bPC1:~$ ping 192.168.1.2 PING 192.168.1.2 (192.168.1.2): 56 data bytes 64 bytes from 192.168.1.2: seq=0 ,l=64 time=26.666 ms 64 bytes from 192.168.1.2: seq=1 ,l=64 time=0.000 ms 64 bytes from 192.168.1.2: seq=2 ,l=64 time=3.334 ms 64 bytes from 192.168.1.2: seq=3 ,l=64 time=0.000 ms
41 de 48 02/10/2013 08:01 a.m.
192.168.1.2 ping statistics 4 packets transmi,ed, 4 packets received, 0% packet loss round-trip min/avg/max = 0.000/6.000/26.666 ms tc@PC1:~$ Issue ping from PC3 (VLAN20) to PC4 (VLAN20). tc@PC3:~$ ping 192.168.1.4 PING 192.168.1.4 (192.168.1.4): 56 data bytes 64 bytes from 192.168.1.4: seq=0 ,l=64 time=3.334 ms 64 bytes from 192.168.1.4: seq=1 ,l=64 time=3.333 ms 64 bytes from 192.168.1.4: seq=2 ,l=64 time=3.333 ms 64 bytes from 192.168.1.4: seq=3 ,l=64 time=0.000 ms 192.168.1.4 ping statistics 4 packets transmi,ed, 4 packets received, 0% packet loss round-trip min/avg/max = 0.000/1.666/3.334 ms tc@PC3:~$ As we have expected ping should have worked in both cases. b) Test connectivity between computers placed to dierent VLANs Issue ping from PC1 (VLAN10) to PC3 (VLAN20). tc@PC1:~$ ping 192.168.1.4 PING 192.168.1.4 (192.168.1.4): 56 data bytes 192.168.1.4 ping statistics 11 packets transmi,ed, 0 packets received, 100% packet loss We can see that ping is not working between PC1 and PC3. We conrm that there is not connectivity between computers placed in to dierent VLANs and openvswitch is working correctly.
LAB 2 VLAN Trunk Configuration And Testing

In previous LAB we proved, that openvswitch only forwards trac between computers which reside in the same VLAN. In this LAB we are going to add additional switch to the topology openvswitch2 and connect switches with link congured as 8021.1q trunk. The access VLAN 10, 20, 30 is added to openvswitch2 and VLAN 30 is added to openvswitch. There is only VLAN 10 and 30 allowed on both sides of the trunk. Our goal is to show that only trac from VLAN 10 and VLAN 30 is forwarded between switches.
42 de 48
02/10/2013 08:01 a.m.
(h,p://brezular.les.wordpress.com/2011/06/vlan_trunk2.jpeg) Figure 2 Two openvswitches connected with a link congured as 8021.q trunk click image to enlarge 1) Openvswitch conguration Assuming that the previous conguration of openvswitch has been stored, we only add following commands. a) Add access port eth4 to the bridge br0 and assign it to VLAN 30 sudo ovs-vsctl add-port br0 eth4 tag=30 b) Add trunk port eth5 to bridge br0 and allow VLAN 10 and 30 on the trunk sudo ovs-vsctl add-port br0 eth5 trunks=10,30 c) Save conguration of Microcore /usr/bin/letool.sh -b 2) Openvswitch2 conguration As it was explained every single step of conguration of openvswitch in previous examples I am going to skip it now. sudo hostname openvswitch2 echo "hostname openvswitch2" >> /opt/bootlocal.sh sudo ovs-vsctl add-br br0 sudo ovs-vsctl add-port br0 eth5 trunks=10,30 sudo ovs-vsctl add-port br0 eth0 tag=30 sudo ovs-vsctl add-port br0 eth1 tag=20 sudo ovs-vsctl add-port br0 eth2 tag=10 /usr/bin/letool.sh -b Similarity, congure IP parameters for the PC5, PC6 to PC8 according to the topology. 3) Trunk VLAN test
43 de 48
02/10/2013 08:01 a.m.
a) Issue the ping from PC1 (VLAN10) to PC8 (VLAN10) tc@PC1:~$ ping 192.168.1.8 PING 192.168.1.8 (192.168.1.8): 56 data bytes 64 bytes from 192.168.1.8: seq=0 ,l=64 time=6.667 ms 64 bytes from 192.168.1.8: seq=1 ,l=64 time=3.333 ms 64 bytes from 192.168.1.8: seq=2 ,l=64 time=0.000 ms 64 bytes from 192.168.1.8: seq=3 ,l=64 time=3.334 ms ^C 192.168.1.8 ping statistics 4 packets transmi,ed, 4 packets received, 0% packet loss round-trip min/avg/max = 0.000/3.333/6.667 ms tc@PC1:~$ We can see that trac in VLAN 10 is succesfully transferred through trunk. If the same VLAN exists on two switches we call it end-to-end VLAN. b) Issue the ping from PC5 (VLAN30) to PC6 (VLAN30) tc@PC5:~$ ping 192.168.1.6 PING 192.168.1.6 (192.168.1.6): 56 data bytes 64 bytes from 192.168.1.6: seq=0 ,l=64 time=6.667 ms 64 bytes from 192.168.1.6: seq=1 ,l=64 time=3.334 ms 64 bytes from 192.168.1.6: seq=2 ,l=64 time=3.333 ms 64 bytes from 192.168.1.6: seq=3 ,l=64 time=3.333 ms ^C 192.168.1.6 ping statistics 4 packets transmi,ed, 4 packets received, 0% packet loss round-trip min/avg/max = 3.333/4.166/6.667 ms tc@PC5:~$ From what we see, trunk is successfully carrying the trac marked with tag 30 between switches. c) Issue the ping from PC3 (VLAN20) to PC7 (VLAN20) tc@PC3:~$ ping 192.168.1.7 PING 192.168.1.7 (192.168.1.7): 56 data bytes 192.168.1.7 ping statistics 6 packets transmi,ed, 0 packets received, 100% packet loss tc@PC3:~$ Trac from vlan 20 is not forwarded between switches. It is because VLAN 20 is not allowed on the trunk. If there would be need to allow all the VLANs on the trunk, only parameter trunks (without specic VLAN) should be presented.
LAB 3 L3 VLAN interfaces, InterVLAN routing Configuration
44 de 48
02/10/2013 08:01 a.m.
And Testing
In this last LAB we are going to create L3 VLAN interface for VLAN 10, 20, 30. After that we will check if Microcore can forward trac between dierent VLANs. We assume that particular ports were added to VLANs in previous LABs and only additional conguration will be shown in this LAB. I have also changed a topology to make it more simple and create the new IP plan for the devices.
(h,p://brezular.les.wordpress.com/2011/06/l3_routing.jpeg) Figure 3 Topology for InterVLAN routing testing click image to enlarge 1) Openvswitch conguration a) Add an internal port vlan10 to bridge br0 as a VLAN access port for VLAN 10 sudo ovs-vsctl add-port br0 vlan10 tag=10 -- set interface vlan10 type=internal b) Create vlan20 and vlan30 interfaces sudo ovs-vsctl add-port br0 vlan20 tag=20 -- set interface vlan20 type=internal sudo ovs-vsctl add-port br0 vlan30 tag=30 -- set interface vlan30 type=internal c) Add access port eth4 and eth5 to the bridge br0 and assigned them to VLAN 30 First, We need to delete conguration for eth4 and eth5 that was done for LAB1 and LAB2. sudo ovs-vsctl del-port br0 eth4 sudo ovs-vsctl del-port br0 eth5 Now we can assign ports to the VLAN 30. sudo ovs-vsctl add-port br0 eth4 tag=30 sudo ovs-vsctl add-port br0 eth5 tag=30 d) Assign IP adresses to the VLAN interfaces sudo ifcong vlan10 192.168.10.254 netmask 255.255.255.0
02/10/2013 08:01 a.m.
45 de 48
sudo ifcong vlan20 192.168.20.254 netmask 255.255.255.0 sudo ifcong vlan30 192.168.30.254 netmask 255.255.255.0 echo "ifconfig vlan10 192.168.10.254 netmask 255.255.255.0" >> /opt/bootlocal.sh echo "ifconfig vlan20 192.168.20.254 netmask 255.255.255.0" >> /opt/bootlocal.sh echo "ifconfig vlan30 192.168.30.254 netmask 255.255.255.0" >> /opt/bootlocal.sh /usr/bin/letool.sh -b e) Enable IPv4 and IPV6 packets forwarding between interfaces Forwarding between network interfaces is disabled by default. To activate ipv4 and ipv6 forwarding for Microcore you need to enable it: sudo sysctl -w net.ipv4.ip_forward=1 sudo sysctl -w net.ipv6.conf.all.forwarding=1 echo "sysctl -w net.ipv4.ip_forward=1" >> /opt/bootlocal.sh echo "sysctl -w net.ipv6.conf.all.forwarding=1" >> /opt/bootlocal.sh /usr/bin/letool.sh -b 2) PC1 to PC6 IP address, netmask, default gateway conguration a) Delete old IP addresses from the computers We should split IP address space in such way when each VLAN would have assigned separate subnet. First, for each computer delete a row with old IP address from /opt/bootlocal.sh which was previously assigned in LAB1 (use dd command for delete of particular row). b) Assign IP address and default GW to PC1 to PC6 Example for PC1: sudo ifcong eth0 192.168.10.1 netmask 255.255.255.0 sudo route add default gw 192.168.10.254 echo "ifconfig eth0 192.168.10.1 netmask 255.255.255.0" >> /opt/bootlocal.sh echo "route add default gw 192.168.10.254" >> /opt/bootlocal.sh /usr/bin/letool.sh -b Similiary, do it for all the PCs, according to topology and IP address plan. 3) InterVLAN routing testing As long as PCs have had correct IP address, netmask and default gateway assigned, we can either ping PC in the same VLAN or any other vlan interface. a) Issue ping from PC1 (VLAN10) to PC2 (VLAN10) tc@PC1:~$ ping 192.168.10.2 PING 192.168.10.2 (192.168.10.2): 56 data bytes
46 de 48
02/10/2013 08:01 a.m.
64 bytes from 192.168.10.2: seq=0 ,l=64 time=3.333 ms 64 bytes from 192.168.10.2: seq=1 ,l=64 time=3.333 ms 64 bytes from 192.168.10.2: seq=2 ,l=64 time=3.334 ms 64 bytes from 192.168.10.2: seq=3 ,l=64 time=3.333 ms 192.168.10.2 ping statistics 4 packets transmi,ed, 4 packets received, 0% packet loss round-trip min/avg/max = 3.333/3.333/3.334 ms tc@PC1:~$ b) Issue ping from PC1 (VLAN10) to interface vlan20 tc@PC1:~$ ping 192.168.20.254 PING 192.168.20.254 (192.168.20.254): 56 data bytes 64 bytes from 192.168.20.254: seq=0 ,l=64 time=3.333 ms 64 bytes from 192.168.20.254: seq=1 ,l=64 time=0.000 ms 64 bytes from 192.168.20.254: seq=2 ,l=64 time=0.000 ms 64 bytes from 192.168.20.254: seq=3 ,l=64 time=3.333 ms 192.168.20.254 ping statistics 4 packets transmi,ed, 4 packets received, 0% packet loss round-trip min/avg/max = 0.000/1.666/3.333 ms tc@PC1:~$ We can see that in both cases, ping was succesfull. c) Issue the ping from PC1 (VLAN10) to PC3 (VLAN20) tc@PC1:~$ ping 192.168.20.1 PING 192.168.20.1 (192.168.20.1): 56 data bytes 64 bytes from 192.168.20.1: seq=0 ,l=63 time=0.000 ms 64 bytes from 192.168.20.1: seq=1 ,l=63 time=3.333 ms 64 bytes from 192.168.20.1: seq=2 ,l=63 time=3.333 ms 64 bytes from 192.168.20.1: seq=3 ,l=63 time=3.334 ms 192.168.20.1 ping statistics 4 packets transmi,ed, 4 packets received, 0% packet loss round-trip min/avg/max = 0.000/2.500/3.334 ms tc@PC1:~$ Thanks to enabled forwarding between interfaces in kernel, ping is working between dierent VLANS. Microcore with Openvswitch is acting as the Layer3 switch. End. FILED UNDER OPENVSWITCH TAGGED WITH 8021Q LINUX, CORE LINUX, GNS3, LINUX ROUTER, LINUX SWITCH, MICROCORE, OPENVSWITCH, TRUNKS LINUX, VLAN LINUX
Blog at WordPress.com.
47 de 48
02/10/2013 08:01 a.m.
The Enterprise Theme. Follow
Follow Brezular's Technical Blog

Powered by WordPress.com
48 de 48
02/10/2013 08:01 a.m.

Openvswitch - Brezular's Technical Blog

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Openvswitch - Brezular's Technical Blog

Uploaded by

Copyright:

Available Formats

Openvswitch | Brezular's Technical Blog

Brezular's Technical Blog

Part4 OPENVSWITCH Playing with Bonding on Openvswitch