2/10/2021 How to make MySQL highly available | by Vivek Puri | DevOps Enthusiast | Medium

How to make MySQL highly available

Vivek Puri Follow
Feb 22, 2020 · 5 min read

Firstly, I am writing a short description of the problem, I am going to solve through this
article. In the world of large scale applications in terms of user base or the criticality, the
risk of downtime of even a single minute can cost you millions of dollars. As we know,
our applications are hosted on servers and servers have a lifespan. If you have hosted
your application on the cloud, then you don’t even have the control of the actual
machine. In our case, we use AWS and many times, we have faced StatusCheckFailure
which can lead to complete machine failure. In order to make it running again, it has to
be stopped and then started which can take 2–5 minutes. Similar is the case with any
server hosted locally. If it fails, it has to be replaced with the new one and that is going to
cost more in terms of time. Now, let’s see, how we can make MySQL highly available.

https://medium.com/devops-enthusiast/how-to-make-mysql-highly-available-a383111f19ad 1/6
2/10/2021 How to make MySQL highly available | by Vivek Puri | DevOps Enthusiast | Medium

DB1 as MASTER

As shown above, we are considering DB1 as our primary database server and DB2 is a
passive database server. DB1 and DB2 are in sync and are master-master replicated. DB1
is having two IPs, Primary IP 172.31.1.1 and a Secondary IP 172.31.1.23. DB2 is only
having one IP 172.31.1.2. On AWS, you can easily attach secondary private IP to a server
from the console. Please consider this secondary IP important as we are going to play
around it now. Application servers are connected to DB1 using its secondary IP address.
Let’s get introduced to a service called Keepalived.

Keepalived is a routing service that works on VRRP protocol and helps in load-balancing
and high-availability to Linux system and Linux based infrastructures. You can read more
about keepalived here.

Install keepalived service on both database servers.

For Ubuntu, install keepalived using the below command.

sudo apt-get install keepalived

For CentOS, install keepalived using the below command.

sudo yum install keepalived

Installation created a directory /etc/keepalived. In this directory, a default config

keepalived.conf is also created. As we are working on AWS, we also need awscli
installed. Let’s install awscli using pip and then configure using AWS access credentials.
You can ignore this part if you are not using AWS.

https://medium.com/devops-enthusiast/how-to-make-mysql-highly-available-a383111f19ad 2/6
2/10/2021 How to make MySQL highly available | by Vivek Puri | DevOps Enthusiast | Medium

pip install awscli --upgrade --user

aws configure
AWS Access Key ID [****************ABCD]: <AccessKeyGoesHere>
AWS Secret Access Key [****************WXYZ]: <SecretKeyGoesHere>
Default region name [ap-south-1]: <RegionOfMySQLHere>
Default output format [None]: <LeaveItBlank>

Now we are ready to configure keepalived.

replace default keepalived.conf on DB1 in /etc/keepalived with below config

vrrp_script chk_mysql {
script "/usr/bin/mysqladmin ping -h 127.0.0.1 -P 3306 -u dbuser -
pdbpass > /dev/null 2>&1"
interval 2
timeout 1
fall 3
rise 3
}
vrrp_instance VI_1 {
debug 2
interface eth0 # interface to monitor
state MASTER
virtual_router_id 51 # Assign one ID for this route
priority 101 # 101 on master, 100 on backup
unicast_src_ip 172.31.1.1 # Primary IP address
unicast_peer {
172.31.1.2 # Primary IP address of DB2
}
track_script {
chk_mysql
}

notify_master /etc/keepalived/master.sh
}

create master.sh on DB1 in /etc/keepalived directory and paste below config

#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/us
r/games:/usr/local/games

secondaryIp="172.31.1.23" # Secondary IP
https://medium.com/devops-enthusiast/how-to-make-mysql-highly-available-a383111f19ad 3/6
2/10/2021 How to make MySQL highly available | by Vivek Puri | DevOps Enthusiast | Medium

instanceIdA="<InstanceIDOfDB1>"
instanceIdB="<InstanceIDOfDB2>"

interfaceIdA=`aws ec2 describe-network-interfaces --filters

Name=attachment.instance-id,Values=$instanceIdA | jq
".NetworkInterfaces[].NetworkInterfaceId"`

interfaceIdA=`echo $interfaceIdA | sed 's|"||g'`

interfaceIdB=`aws ec2 describe-network-interfaces --filters

Name=attachment.instance-id,Values=$instanceIdB | jq
".NetworkInterfaces[].NetworkInterfaceId"`

interfaceIdB=`echo $interfaceIdB | sed 's|"||g'`

aws ec2 unassign-private-ip-addresses --network-interface-id

$interfaceIdB --private-ip-addresses $secondaryIp

aws ec2 assign-private-ip-addresses --network-interface-id

$interfaceIdA --private-ip-addresses $secondaryIp

sleep 5
service network restart # On CentOS

notify_master section of keepalived.conf executes master.sh which is detaching the

secondary IP from the secondary server and attaching that IP to itself. Also, it is
restarting the network as a new IP is now attached to the server. Note: A few more
changes are required on ubuntu for fetching the new IP attached to the server. This
script is executed in case of state transition by keepalived. So initially, this script is not
executed on DB1 as the secondary IP is already attached on DB1 and keepalived is
running as MASTER on DB1. Let's configure keepalived on DB2 now.

vrrp_script chk_mysql {
script "/usr/bin/mysqladmin ping -h 127.0.0.1 -P 3306 -u dbuser -
pdbpass > /dev/null 2>&1"
interval 2
timeout 1
fall 3
rise 3
}
vrrp_instance VI_1 {
debug 2
interface eth0 # interface to monitor
state BACKUP
virtual_router_id 51 # Assign one ID for this route

https://medium.com/devops-enthusiast/how-to-make-mysql-highly-available-a383111f19ad 4/6
2/10/2021 How to make MySQL highly available | by Vivek Puri | DevOps Enthusiast | Medium

priority 100 # 100 on backup, 101 on master

unicast_src_ip 172.31.1.2 # Primary IP address
unicast_peer {
172.31.1.1 # Primary IP address of DB1
}
track_script {
chk_mysql
}

notify_master /etc/keepalived/master.sh
}

master.sh on DB2 is going to be the same as of DB1 except one change which is instance-id.

instanceIdA="<InstanceIDOfDB2>"
instanceIdB="<InstanceIDOfDB1>"

Restart keepalived on both servers.

service keepalived restart

Keepalived is configured on DB1 as MASTER and as BACKUP on DB2. Initially, we have

attached a secondary IP to DB1. We are checking the health of MySQL using mysqladmin
ping command. If MySQL is stopped/crashed/not responding, keepalived is going to change
the state on DB2 to master. master.sh is going to execute on DB2 when this state change
happens which results in detaching the secondary IP from DB1 and attaching it to DB2. All
application servers are still connected to secondary IP which is now attached to DB2.

To test the failover, stop MySQL on DB1. In a span of 3 seconds, DB2 will become
MASTER. Secondary IP 172.31.1.23 will be detached from DB1 and will be attached to
DB2 now using the script master.sh on DB2. In the next 5 seconds, network service will
be restarted and the failover will happen in a total of 8 seconds. In case DB1 comes up
again, it will be re-elected as MASTER and again secondary IP will be attached to DB1.

https://medium.com/devops-enthusiast/how-to-make-mysql-highly-available-a383111f19ad 5/6
2/10/2021 How to make MySQL highly available | by Vivek Puri | DevOps Enthusiast | Medium

DB2 as MASTER

DevOps AWS MySQL High Availability Database

About Help Legal

Get the Medium app

https://medium.com/devops-enthusiast/how-to-make-mysql-highly-available-a383111f19ad 6/6