2023 6th International Conference on Information Systems and Computer Networks (ISCON)
GLA University, Mathura, India. Mar 3-4, 2023
Implementation and Performance Evaluation of
Load Balanced Routing in SDN based Fat Tree
Data Center
2023 6th International Conference on Information Systems and Computer Networks (ISCON) | 979-8-3503-4696-1/23/$31.00 ©2023 IEEE | DOI: 10.1109/ISCON57294.2023.10112200
Dhanya Raj P.
Department of Computer Science and Engineering
Government Engineering College, Wayanad
APJ Abdul Kalam Technological University
Kerala, India
dhanyaraj@gecwyd.ac.in
Abstract—As the number of data intensive and distributed
applications increase, data centers are highly utilized for their
storage, processing and accessing. These applications cause
throughput sensitive incast traffic in the data center.
Traditional shortest path routing algorithms cause traffic
hotspots in the shortest path links and lead to congestion.
Therefore, load balanced routing strategies need to be
employed to spread the traffic across multiple paths. This work
employs the idea of multi path routing to achieve load
balancing in SDN based Fat tree Data center. It implements
Equal Cost Multi Path routing algorithm on Fat tree Data
Center Network in Ryu SDN controller and evaluates its load
balancing performance against traditional Shortest Path
Routing algorithm under different data center benchmark
traffic scenarios. We have conducted an exhaustive study about
the load balancing effect of ECMP on Ryu based SDN Fat tree
data center and found that ECMP increases Average network
throughput by 68% and Average Link utilization by 60%. Also
packet drop is decreased by 44% with ECMP.
Keywords—Data Center, Multi path routing, Software
Defined Networking, Load Balancing, Fat Tree
I. INTRODUCTION
Data Centers Networks(DCN) host thousands of
networked servers to provide storage and computing to
business organizations. They are used to store, process and
access data intensive and distributed applications like Map
Reduce applications, social networking platforms, large scale
web search etc. These applications create high volume of
throughput intensive east west traffic in the DCN. To provide
network efficiency and to satisfy the service requirement of
diverse flows in the data center, congestion possibility should
be avoided by sharing the traffic load across available
alternative links. Traditional shortest path routing in a heavy
traffic scenario results in the creation of congestion hot spots
due to single best path selection. This demands for load
balanced routing techniques in the data center.
Inefficiency of traditional network technologies in
handling the large network of thousands of servers in the
data center network introduced Software Defined
Networking into it. SDN uses centralized software
controllers to connect with switches in the network and shift
the routing logic implementation from the switch to the
controller. Also, in SDN paradigm, switching/forwarding
element can store flow state in flow table as flow entries.
Thus each forwarding element can store flow state without
any additional overhead. This allows next hops
979-8-3503-4696-1/23/$31.00 ©2023 IEEE
Authorized licensed use limited to: UNIVERSITAS GADJAH MADA. Downloaded on December 11,2023 at 03:39:45 UTC from IEEE Xplore. Restrictions apply.
Anitha V. S.
Department of Computer Science and Engineering
Government Engineering College, Wayanad
APJ Abdul Kalam Technological University
Kerala, India
anithavs@gecwyd.ac.in
corresponding to precomputed paths to be stored in the
forwarding element at the time of flow creation and avoids
per packet computation [2]. So selection of a path from
multiple available paths can be done proactively.
POX [4], Beacon [6], Floodlight [5], and Ryu [7] are the
major controller platforms for implementing Software
Defined Networking. None of these controllers provide
inbuilt module for multi path support. They provide
Spanning Tree Protocol(STP) to deal with ARP broadcast
streams occurring in the topologies having loops. But STP
result in the selection of a single path and load balancing
along multiple available paths cannot be performed. This
paper implements a load balanced routing strategy for Fat
tree based data center that works on software defined
network architecture. Here, Mod-N based Equal Cost Multi
Path routing algorithm is implemented on top of the Ryu
SDN controller. Performance of this algorithm in terms of
average network throughput, average link utilization, and
packet drop are compared with the implementation of
traditional Shortest Path Routing algorithm under different
benchmark traffic patterns to observe its load balancing
effect in the data center. This work utilizes parallel TCP
flows to conduct an exhaustive study of the load balancing
effect of ECMP Mod-N strategy in Ryu SDN controller
based Fat tree data canter.
The rest of the paper is organized as follows: Section II
explains the background topics and Section III gives related
works. The proposed ECMP routing framework for SDN
based Fat tree DCN is given in section IV. Section V
describes the implementation details and section VI gives
results and a discussion on it. Paper concludes in section VII.
II. BACKGROUND
Data Centers are highly utilized for their storage,
processing and accessing by different applications.
Traditional TCP mechanisms are not suitable for data center
traffic and fails to prevent the occurrence of congestion [15].
Also commonly used shortest path routing strategy results in
the selection of the same least cost path in all the
communications between a source-destination pair. Thus
Data centers need to avoid the creation of traffic hot spots
through appropriate load balance based network layer or
transport layer mechanisms.
Fat tree is a symmetric, non-blocking, fault tolerant, and
switch-centric data center topology. Fat tree connect the
1
network elements with three layers. A k-ary fat tree consist
of 5k2/4 k-port switches and k3/4 servers. A pod in the fat
tree is a group of (k/2)2 servers, k/2 lower layer Edge
switches, and k/2 middle layer Aggregation switches. Each
Edge switch connects with k/2 servers and k/2 Aggregation
switches. Each Aggregation switch connect with k/2 core
switches to complete the connectivity. There exists (k)2 /4
equal cost paths between any two hosts on different pods.
Fig. 1. Fat Tree Topology
Software Defined Networking(SDN) is a new network
paradigm that makes network more flexible and controllable
through their global network visibility and programmable
nature. SDN architecture comprises of three layers namely,
Data plane, Control plane, and Application plane [1].
Switches in the control plane communicate with the
controller using OpenFlow protocol. The interface between
control and data plane is known as Southbound interface.
OpenFlow is the standard south bound interface. The
interface between control and application plane is known as
Northbound interface. There is no standard North bound
interface. Controllers provide their own API to write
application programs.
Multi Path routing is applied where multiple equal cost
paths are available between a source-destination pair. As the
selection of a path is from a set of equal cost paths, this type
of routing is known as Equal Cost Multi Path routing
(ECMP). ECMP implementation is defined in RFC 2991.
Both flow based and packet based implementations of ECMP
are possible. In the case of packet based ECMP, next hop is
selected without regard to other packets in the flow.
Generally flow based implementations of ECMP routing are
preferred over packet based implementations as latter cause
packet reordering at the receiving side [3]. Also in SDN
architecture, OpenFlow switches support flow matching and
hence per flow state can be maintained in it. Fat tree based
data center has many equal cost paths between a given
source destination pair. This makes it suitable for applying
ECMP based routing for load balancing.
Multi path load balancing problem on fat tree data
centers can be represented using a directed graph, where
nodes correspond to hosts and switches. Edges correspond to
host to switch links and switch to switch links. Then for an
incoming flow, routing solutions that provide assignment of
a path from the set of equal cost paths, which keeps the
average link utilization and throughput at optimum values
are required.
III. RELATED WORK
The term load balancing in network by default refers to
server load balancing. But here we discuss about balancing
traffic load among alternative available equal cost paths.
Both static and dynamic load balancing mechanisms exists in
2
Authorized licensed use limited to: UNIVERSITAS GADJAH MADA. Downloaded on December 11,2023 at 03:39:45 UTC from IEEE Xplore. Restrictions apply.
the literature in this context. Static mechanisms select a path
from the alternative choices by some fixed rules. They do not
consider network load or network element status while
making the decision whereas dynamic mechanism collect
network statistics, measure network load and choose path
according to the load. Simplicity of selection policy and low
overhead of implementation are the characteristics of static
mechanisms.
This work [10] implements ECMP Mod-N Hash on a Fat
tree topology on a POX controller. It runs three TCP and
UDP background traffic between two selected hosts in the
network and measures network performance against shortest
path routing algorithm. Nikolaosc et al in the work [14]
analyze multi path routing algorithms on a Spine-leaf data
center and argues for packet-level multi pathing. In this work
[9] Eric Jo et al model load balanced multipath routing
problem as a linear program and implements ECMP and
their own dynamic load balanced multi path routing. This
work [16] implements ECMP in a tree topology over Ryu
SDN controller and performs UDP traffic test between fixed
hosts. Here in this work [17] ECMP and a weighted-ECMP
are implemented and compared. X. Shi et al in [18] give
different treatment for elephant flows in path selection and
tests their strategy on a custom topology with six hosts and
switches over a Pox controller.
Our work is limited to static load balanced routing
method and it implements Equal Cost Multi Path routing on
SDN based fat tree data center using the next hop selection
method, Mod-N with hash over packet header fields. Here,
we measure the throughput of the network, packet drop and
average link utilization of the ECMP routing algorithm along
with that of Shortest Path Routing algorithm under different
data center benchmark traffic patterns to analyze their load
balancing effect.
IV. PROPOSED ECMP ROUTING FRAMEWORK ON
SDN FAT TREE DATA CENTER
Figure 2 shows the architecture of the proposed ECMP
routing framework. Arrival of a flow with no matching flow
rule installed in the switch causes Packet-In message to the
SDN controller. Controller invokes Packet-In handler in
order to find the port of the sending switch through which
Fig. 2. Load Balanced Routing Framework in Ryu based Fat tree DCN
this packet is to be forwarded, so that it will reach the
destination. Source to destination path computation is done at
this point to find multiple equal cost shortest paths between
them.
Selection of one path from these multiple equal cost
paths are done using Hash based Mod-N operation.
Algorithm 1 describes it. Here G indicates the graph
representation of the network, H is the set of hosts, S is the
set of switches and L denotes the set of links.
In Mod-N hash based selection, flow is identified at the
micro level using source and destination ip addresses, source
and destination port numbers and protocol fields. A unique
hash value is generated for each such flow. A mod-N
operation on this hash value, where N is the number of equal
cost paths available between the source and the destination,
is performed to get the choice number for the selection of
path. Path corresponding to this choice number is selected
from the list of all paths.
Major steps in the load balanced routing framework
design are listed below:
1) Collect network topology information into a graph G
2) Start network status monitoring to collect performance
evaluation parameters
3) Initiate parallel network traffic between source
destination pairs, that can test the effectiveness of load
balanced routing strategy
4) For an incoming flow with no matches in the flow
table of a switch, find the list of alternative paths between the
source-destination of that flow
5) Select a path based on Modulo-N Hash based ECMP
load balancing route selection strategy, as explained in the
Algorithm 1
6) Install flow entry on all switches along the selected
path
ECMP Hash Mod-N algorithm assigns each flow to a
particular path between the source destination pair. So it do
not require packet reordering at the receiver side.
V. IMPLEMENTATION
We implement SDN based fat tree Data center using Ryu
SDN controller and Mininet emulation tool. Mininet[11]
creates realistic virtual network on a single machine using
Linux network name spaces. The virtual network consist of
virtual hosts, switches, controllers and connection links.
Ryu[7] is a Software defined networking controller, written
in Python. It provides APIs to write network management
applications. A Ryu application can write decorator functions
3
Authorized licensed use limited to: UNIVERSITAS GADJAH MADA. Downloaded on December 11,2023 at 03:39:45 UTC from IEEE Xplore. Restrictions apply.
for the interested events and can handle it through its own
instruction.
We emulate a 4-ary Fat tree that has 16 hosts and 20
switches to make Data center topology. Emulation
parameters are given in the Table I. Each link is given 1GB
bandwidth. The Data center topology is connected with Ryu
to implement SDN based architecture. An application is
written to compute equal cost shortest paths among a
particular source destination pair. Path selection module
selects a path based on the ECMP Hash Mod-N algorithm
discussed in the design section. Newly selected path
information is installed among the switches along the path.
TABLE I. EXPERIMENT SET UP
Experiment Set up Values
Number of hosts 16
Number of switches 20
Link Bandwidth 1 GB
Switch to controller protocol Open Flow V1.3
Two types of traffic can be seen in the data centers, east-
west traffic and north-south traffic. East-west traffic is the
traffic between server racks in a data center and north-south
traffic is the communication coming and going from internet
to the data center. Studies[12] have shown that east-west
traffic contributes about seventy seven percentage of total
data center traffic. Due to the difficulty in getting real data
center traffic traces, we have generated data center
benchmark communication patterns as per[13] namely
stride(i), staggered(Pe,Pp), and random. Stride pattern with
argument value 1, 2, 4 and 8 generates east-west traffic
between the server hosts. When 1 is used as the argument,
each server sends to the next neighbour in the fat tree
topology and the traffic pattern is h1 to h2, h2 to h3...., h15
to h16 and h16 to h1. With arguments 2, 4 and 8
communication distance between the hosts increases. Stride
pattern generates both intra pod and inter pod
communications.
With staggered pattern a server sends traffic to a server
within the same edge switch with a probability of Pe and to
the server in the same pod with a probability Pp and to the
remaining servers in the network with 1 − Pe − Pp. In
Random pattern, for each source host, the destination host is
selected randomly. Iperf tool [8] is used to generate TCP
traffic flows between hosts in the network. Parallel flows are
applied to create multiple flow between a source destination
pair. All the traffic patterns are implemented and applied
over the hosts in the fat tree data center multiple times to take
average measurements.
VI. RESULTS AND DISCUSSION
We measured the average throughput of the network,
average link utilization and packet drop under the above
mentioned traffic patterns for traditional Shortest Path
Routing and ECMP hash based Mod-N. Received bytes from
each host in the network is collected during fixed time
intervals. This data is used to compute average network
throughput. The values of average network throughput on
running the ECMP Mod-N path selection algorithm on
different traffic patterns are plotted in the Figure 5. ECMP
Mod-N give high throughput compared to single shortest
path routing in all the patterns.

Fig. 4. Throughput comparison of the ECMP algorithm on different traffic

patterns with Time

Fig. 3. Throughput comparison of the SPR algorithm on different traffic

patterns with Time TABLE II. PERFORMANCE ANALYSIS
Performance Improvement
Figures 3 and 4 shows the throughput performance Parameter
SPR ECMP Mod-N Difference %
comparison of the different patterns over time with SPR and Throughput(Gbps) 5.51 9.23 3.72 68
ECMP Mod-N algorithms. On closely analyzing the Link Utilization(%) 34.27 54.87 20.61 60
performance, we can see that even though both the
Packet Drop(x1000) 153.47 86.71 66.76 44
algorithms performs equally on stride(1) pattern that has very
less number of inter pod traffic, on stride(4) and stride(8)
patterns in which inter pod communication is high and core average of link utilization of all links in the network is
switch is involved in the path from source to destination computed under different traffic patterns for the two
route, ECMP-Mod N clearly gives high throughput. Also on algorithms. Figure 6 clearly shows that average link
Staggered(0.2,0.3) pattern that has fifty percentage inter pod utilization is high for ECMP Mod-N compared to SPR in all
communication, ECMP Mod-N algorithm over perform the traffic patterns. Overall increase in link utilization is
single Shortest path routing. Average increase in throughput 60%. This clearly shows the spreading of flows along
of all patterns, in ECMP Mod-N is 68%, as shown in Table multiple available paths by ECMP Mod-N strategy.
II.
Packet drop in the network is measured and plotted in the
Plotting average link utilization caused by different Figure 7. ECMP Mod-N has reduced packet drop in all the
routing algorithms under same network load provides a clear traffic patterns. Overall reduction in packet drop with ECMP
indication of their load balancing effect. Here, link utilization over SPR is 44%. High network throughput, link utilization
is calculated from the switches with the number of bytes and reduced packet drop clearly shows that ECMP transmits
received information on each switch port. Threads are the traffic successfully to the destination and spreads it
created to collect port statistics from all switches in the fat evenly.
tree data center topology. Port status request messages are
sent from the controller and a reply handler is written to save
relevant information and to calculate link utilization. The

Fig. 5. Average Network Throughput comparison of SPR and ECMP Mod-N algorithms on different traffic patterns in SDN Fat tree Data Center

4
Authorized licensed use limited to: UNIVERSITAS GADJAH MADA. Downloaded on December 11,2023 at 03:39:45 UTC from IEEE Xplore. Restrictions apply.
 Fig. 6. Average Link Utilization comparison of SPR and ECMP Mod-N algorithms on different traffic patterns in SDN Fat tree Data Center
Fig. 7. Packet Drop comparison of SPR and ECMP Mod-N algorithms on different traffic patterns in SDN Fat tree Data Center
VII. CONCLUSION
This paper presents a load balanced routing framework
for a Ryu based SDN Fat tree data center. ECMP Mod-N
static path selection strategy is implemented in Ryu SDN
controller to select a path from available multiple paths in the
Fat tree data center topology. The performance of this
algorithm along with traditional Shortest Path Routing
algorithm, in terms of average network throughput, average
link utilization and packet drop are measured for different
data center benchmark traffic patterns, generated with
parallel flows. ECMP Mod-N increased the throughput by
68%, link utilization by 60% and reduced the packet drop by
44%. The performance of ECMP Mod-N shows that
distributing traffic across multiple paths can increase average
network utilization and throughput and decrease packet drop.
This inspires to conduct further studies that can utilize the
multi path routing for further improvement in Fat tree Data
center load balancing.
The implementation of ECMP Mod-N on Ryu based Fat
tree data center with the benchmark communication pattern
can be used as a test bed for future investigations towards
load balancing routing. Further research is needed to find
enhanced techniques to spread the traffic more uniformly
across the available multiple paths and thereby improving the
load balancing performance of the routing strategies in the
data center.
REFERENCES
[1] Goransson, P. Software Defined Networks (Second Edition), A
Comprehensive Approach (2017).
[2] D. Thaler and C. Hopps., ”Rfc2991: Multipath issues in unicast and
multicast next-hop selection.”, 2000
5
Authorized licensed use limited to: UNIVERSITAS GADJAH MADA. Downloaded on December 11,2023 at 03:39:45 UTC from IEEE Xplore. Restrictions apply.
[3] C. Hopps, ”Analysis of an equal-cost multi-path algorithm. RFC
2992, Internet Engineering Task Force”, 2000.
[4] Pox. https://noxrepo.github.io/pox-doc
[5] Floodlight.
https://floodlight.atlassian.net/wiki/spaces/floodlightcontroller
[6] D. Erickson, ”The beacon openflow controller.”, Proceedings of the
second ACM SIGCOMM workshop on Hot topics in software defined
networking, 2013.
[7] Ryu. https://ryu-sdn.org/
[8] Iperf. https://iperf.fr
[9] E. Jo, D. Pan, J. Liu, L Butler, ”A simulation and emulation study of
SDN-based multipath routing for fat-tree data center networks.”
Proceedings of the Winter Simulation Conference 2014. IEEE, 2014.
[10] F. Rhamdani, N.A. Suwastika, and M.A. Nugroho, ”Equal-cost
multipath routing in data center network based on software defined
network.” 2018 6th International Conference on Information and
Communication Technology (ICoICT). IEEE, 2018.
[11] Mininet. http://mininet.org/
[12] Index, Cisco Global Cloud. ”Forecast and methodology, 2015-2020
white paper.” Retrieved 1st June (2016): 15.
[13] M. Al-Fares, S. Radhakrishnan, B. Raghavan, N. Huang, and A.
Vahdat, ”Hedera: dynamic flow scheduling for data center networks.”
Nsdi. Vol. 10. No. 8. 2010.
[14] N. Chrysos, et al. ”High performance multipath routing for
datacenters.” 2014 IEEE 15th International Conference on High
Performance Switching and Routing (HPSR). IEEE, 2014.
[15] ] T. Hafeez, N. Ahmed, B. Ahmed, and A.W. Malik, ”Detection and
mitigation of congestion in SDN enabled data center networks: A
survey.” IEEE Access 6 (2017): 1730-1740.
[16] H.E. Wahanani, M. Idhom, and E.P. Mandyartha, ”Equal cost
multipath ryu controller analysis in software-defined networking.”
2020 6th Information Technology International Seminar (ITIS). IEEE,
2020.
[17] E. Nepolo, and G.A.L. Zodi, ”A Predictive ECMP Routing Protocol
for Fat-Tree Enabled Data Centre Networks.” 2021 15th International
Conference on Ubiquitous Information Management and
Communication (IMCOM). IEEE, 2021.
[18] X. Shi, L. Yangyang, X. Haiyong, Y. Tengfei, Z. Linchao, L. Panyu,
Z. Heng, and L. Zhiyao, ”An openflow-based load balancing strategy
in SDN.” Comput. Mater. Contin 62, no. 1 (2020): 385-398.

6
Authorized licensed use limited to: UNIVERSITAS GADJAH MADA. Downloaded on December 11,2023 at 03:39:45 UTC from IEEE Xplore. Restrictions apply.