04/07/2024, 14:46 MySQL InnoDB Cluster Setup Step by Step – MySQL DataBase Administration

MySQL
MySQL DataBase
DataBase Administration
Administration <<
https://deepakmysqldba.wordpress.com/>
https://deepakmysqldba.wordpress.com/>
( Beginner to Expert )

MySQL InnoDB Cluster Setup Step by Step

MySQL InnoDB Cluster has proven with the first GA release its potential as the native out-of-the-box HA
solution for MySQL. With the second GA release, usability, stability and flexibility were greatly improved as
described in the previous blog post.

This tutorial aims to show how does a typical InnoDB cluster setup look like and how that setup even became
simpler and more effective as before. It’s a great starting point for those not familiar with InnoDB cluster and
shall help you to get started.

For that purpose, we will use the new MySQL Shell < https://dev.mysql.com/doc/refman/8.0/en/mysql-
shell.html> to manage 3 MySQL Server < https://dev.mysql.com/doc/refman/8.0/en/> instances and create a 3
member InnoDB cluster (running Group Replication < https://dev.mysql.com/doc/refman/8.0/en/group-
replication.html> ). Then we will set up the new MySQL Router < https://dev.mysql.com/doc/mysql-
router/8.0/en/> as to hide the multiple MySQL instances behind a single TCP port. Client applications can then
connect to the ports MySQL Router provides, without any need to be aware of the InnoDB cluster topology. In
the event of an unexpected failure, the InnoDB cluster adjusts itself automatically and MySQL Router detects
the change. This removes the need for your client application to handle failover.

Nowadays, it’s very important to have a high availability solution that gives you serenity and security for your
data. The task of ensuring your data will always be available is a challenging one, that not everyone wants to
do because it demands a lot of knowledge about tools, configuration, and technologies.

MySQL InnoDB cluster

The good news is that MySQL provides a complete suite of products to set up a reliable high availability
configuration for your application’s data, while not giving up the familiar and powerful features of the InnoDB
storage engine: InnoDB cluster.

Existing MySQL administration knowledge is not required. Whether you’re an advanced user who wants to
automate and improve tooling around replication maintenance, or a novice user interested in trying out high

https://deepakmysqldba.wordpress.com/2020/08/19/mysql-innodb-cluster-setup-step-by-step/ 1/10
04/07/2024, 14:46 MySQL InnoDB Cluster Setup Step by Step – MySQL DataBase Administration

availability solutions, the new tools make the task much simpler than previously possible.

The two main tools required to accomplish an easy and fast setup are:

MySQL Shell
Configure and do all the backstage work
Create a cluster
Add and remove instances to the cluster
Monitor the cluster
Cluster management
Besides doing queries, updates and administration operations to the instances

MySQL Router

Create a proxy for the cluster

Route database traffic

In a production environment, a cluster should obviously be created in separate hosts in the network. So if one
instance goes down because of a hardware or system problem, the other instances can take over its place. But
to make testing or just trying things out easier, MySQL Shell has a built in functionality that let you create a
cluster with multiple instances on the same host, a sandbox cluster with sandbox instances of MySQL.

The following diagram illustrates how a typical production setup of an InnoDB cluster looks like:

Requirements
To start using InnoDB cluster you need to install the following:

MySQL Server 5.7.17 or higher

MySQL Shell 1.0.8 or higher
MySQL Router 2.1.2 or higher

The host needs to meet the following requisites:

Python 2.7.x
Host mapping already configured for the hosts that will compound the InnoDB cluster

As you can see the requirements are minimum, and it’s highly possible that your hosts already meets them.

MySQL InnoDB clusters are a free and open source solution for high availability, with minimum requirements,
simple installation and ease of user with big benefits. Start working with InnoDB cluster and discover how it
can change your data management. Take a look at the previous blog posts and the official documentation. Start
learning and testing, then you will have a better view how you can incorporate InnoDB in your work.

Below a typical InnoDB cluster setup:

Downloading and Installing Required Packages

In addition to MySQL Server 8.0.11 GA < https://dev.mysql.com/downloads/mysql/> , you will need to download
the two other components of MySQL InnoDB Cluster:

MySQL Shell 8.0.11 GA with X AdminAPI < https://dev.mysql.com/downloads/shell/>

MySQL Router 8.0.11 GA < https://dev.mysql.com/downloads/router/>

https://deepakmysqldba.wordpress.com/2020/08/19/mysql-innodb-cluster-setup-step-by-step/ 2/10
04/07/2024, 14:46 MySQL InnoDB Cluster Setup Step by Step – MySQL DataBase Administration

Download and install the packages, through the usual platform-specific methods:

MySQL Server < https://dev.mysql.com/doc/refman/8.0/en/installing.html>

MySQL Shell < https://dev.mysql.com/doc/refman/8.0/en/document-store-shell-install.html>
MySQL Router < https://dev.mysql.com/doc/mysql-router/8.0/en/mysql-router-installation.html>

Traditionally, setting up high availability (HA) in MySQL has been a challenging task, especially for people
without advanced knowledge of MySQL. From understanding concepts and technologies, to the tooling, specific
commands and files to execute and edit, there’s a lot of things you need to know even when planning a test
deployment (the Quick Start Guide < http://mysqlhighavailability.com/mysql-group-replication-a-quick-start-
guide/> for Group Replication should give you an idea). So many people end up procrastinating setting up HA
until disaster strikes and downtime happens.

The MySQL team at Oracle has now released a preview of a solution that aims to make it easy to combine
multiple instances of MySQL together in order to provide redundancy for High Availability of the MySQL
databases used by your applications.

In this tutorial, we will use the new MySQL Shell < http://mysqlserverteam.com/mysql-5-7-12-part-4-a-new-
mysql-command-line-shell/> to create 3 sandbox instances, create a 3 member Group Replication <
http://mysqlhighavailability.com/tag/mysql-group-replication/> cluster with them and setup MySQL Router <
http://www.mysql.com/products/enterprise/router.html> as a proxy to hide the multiple MySQL instances
behind a single TCP port. Group Replication provides synchronous replication for InnoDB and built-in automatic
failover, while the Router lets applications connect to an appropriate member of the cluster with no changes to
their code, other than an update to the MySQL connection parameters.

Create a user that can connect to each instance

Run this on every instance :

set sql_log_bin=0;

GRANT ALL PRIVILEGES ON *.* TO ‘mysqldba’@’localhost’ identified by ‘Password@123’ with grant option;

GRANT ALL PRIVILEGES ON *.* TO ‘mysqldba’@’%’ identified by ‘Password@123′ with grant option;

change master to master_user=’repluser’, master_password=’Password@123′ for channel

‘group_replication_recovery’;

flush privileges;

set sql_log_bin =1;

reset master;

Deploying Instances

The first step is to deploy MySQL Server instances, so that we can play around and experiment safely, without
affecting any existing MySQL databases.

Start MySQL Shell (as your ordinary, non-root OS user):

https://deepakmysqldba.wordpress.com/2020/08/19/mysql-innodb-cluster-setup-step-by-step/ 3/10
04/07/2024, 14:46 MySQL InnoDB Cluster Setup Step by Step – MySQL DataBase Administration

$ mysqlsh

And then execute:

mysql-js> dba.deployLocalInstance(3306)

You will be prompted to pick a MySQL root account password for that instance, this is the password that you
will use to connect to that instance in the future.

Note: use the same password for all your instance in the same cluster.

Repeat the above command two more times, using different port numbers each time. This will allow us
to create an InnoDB cluster that is tolerant to up to one failure.

Initializing the InnoDB Cluster

Next, we connect the shell to one of the instances we just created, which will be the seedinstance. The seed
instance is the one that would hold the initial state of the database, which will be replicated to the other
instances as they’re added to the cluster. In this case, we’re working with empty sandbox instances, so any of
them will do.

1. Connect to the first node

MySQL JS > shell.connect(‘mysqldba@10.172.52.60:3306’)

2. Configure the local instance

MySQL 10.172.52.60:3306 ssl JS > dba.configureLocalInstance(‘mysqldba@10.172.52.60:3306’)

3. checkInstanceConfiguration

MySQL 10.172.52.60:3306 ssl JS > dba.checkInstanceConfiguration(‘mysqldba@10.172.52.60:3306’)

4. Next, create the InnoDB cluster:

MySQL 10.172.52.60:3306 ssl JS > var cluster = dba.createCluster(‘InnodbPOCcluster’)

The createCluster() command takes one parameter, which is a symbolic name you give to this InnoDB cluster.
This will, among other things:

Deploy the metadata schema in that instance (mysql_innodb_cluster_metadata)

Verify that its configuration is correct for Group Replication, making changes if necessary
Register it as the seed instance of the new cluster
Create necessary internal administrative accounts

https://deepakmysqldba.wordpress.com/2020/08/19/mysql-innodb-cluster-setup-step-by-step/ 4/10
04/07/2024, 14:46 MySQL InnoDB Cluster Setup Step by Step – MySQL DataBase Administration

Start Group Replication

You will prompted for a master administrative key. This key is needed to manage the InnoDB cluster and you
will also need it later to bootstrap MySQL Router instances for use with the cluster. This allows a secure by
default deployment without sacrificing ease-of-use. The system will create and manage single use accounts
with minimal privileges internally for you.

5. After this configure the instance of other 2 node

MySQL 10.172.52.60:3306 ssl JS > dba.configureLocalInstance(‘mysqldba@10.172.52.69:3306’)

MySQL 10.172.52.60:3306 ssl JS > dba.configureLocalInstance(‘mysqldba@10.172.52.70:3306’)

Remember, by default we’re in single primary mode, i.e. there will only be 1 RW server and the other 2, the ones
we just added, will be in SUPER READ ONLY mode, i.e. no writes will be allowed. If you want to go to multi
master mode, then that’s a different createCluster option.

6. Add Instances to InnoDB Cluster

Now, you need to add replicas to the InnoDB cluster. We use the blank sandbox instances that we created
earlier. Usually, whenever a new instance is added to a replica set in a cluster, they will be completely empty
and need to catch up to the current state of the seed instance. If the amount of pre-existing data in the seed
instance is very large, you may want to clone it or copy that data through a fast method beforehand. The MySQL
Shell will soon also provide a mechanism to perform that step with minimal effort. Otherwise, Group
Replication will perform a sync automatically, re-executing all transactions from the seed, as long as
they’re in the MySQL binary log. Since the seed instance in this example has little to no data (ie. just the
metadata schema and internal accounts) and have binary logging enabled from the beginning, there’s very
little that new replicas need to catch up with. Any transactions that were executed in the seed instance will be
re-executed in each added replica.

Add two blank instances to the InnoDB cluster:

MySQL 10.172.52.60:3306 ssl JS > cluster.addInstance(‘mysqldba@10.172.52.69:3306’)

MySQL 10.172.52.60:3306 ssl JS > cluster.addInstance(‘mysqldba@10.172.52.70:3306’)

7. And now, we check the current status of the cluster:

MySQL 10.172.52.60:3306 ssl JS > cluster.status()

This command queries the current status of the InnoDB cluster and produces a short report. The status field
for each instance should show either ONLINE or RECOVERING. RECOVERING means that the instance is receiving
updates from the seed instance and should eventually switch to ONLINE.

Another point to note is that one of the instances (the PRIMARY) is marked R/W (read/writable), while the other
two are marked R/O (read only). Only the instance marked R/W can execute transactions that update the
database. If that instance becomes unreachable for any reason (like a system crash), one of the remaining two
instances automatically takes over its place and becomes the new PRIMARY.

https://deepakmysqldba.wordpress.com/2020/08/19/mysql-innodb-cluster-setup-step-by-step/ 5/10
04/07/2024, 14:46 MySQL InnoDB Cluster Setup Step by Step – MySQL DataBase Administration

Deploy MySQL Router

In order for applications to handle failover, they need to be aware of the topology of the InnoDB cluster. They
also need to know, at any time, which of the instances is the PRIMARY. While it’s possible for applications to
implement that logic by themselves, MySQL Router can do that for you, with minimal work and no code changes
in applications.

The recommended deployment of MySQL Router is on the same host as the application. In our tutorial, we’re
running everything on the same host, so we’ll do that for MySQL Router too. You need the MASTER key of the
InnoDB cluster to auto-configure MySQL Router.

Assuming MySQL Router is already installed, all we need to do is to bootstrap it with the metadata server,
calling mysqlrouter with the following command line option from the system’s shell:

Steps:

wget https://dev.mysql.com/get/Downloads/MySQL-Router/mysql-router-8.0.11-linux-glibc2.12-x86-64bit.tar.gz
< https://dev.mysql.com/get/Downloads/MySQL-Router/mysql-router-8.0.11-linux-glibc2.12-x86-64bit.tar.gz>

tar -xvf mysql-router-8.0.11-linux-glibc2.12-x86-64bit.tar.gz

mv mysql-router-8.0.11-linux-glibc2.12-x86-64bit/ mysqlrouter

cd mysqlrouter/

chown -R mysql:mysql mysqlrouter/

MySQL Router connects to the InnoDB cluster, fetches its metadata and configures itself for use. The generated
configuration creates 2 TCP ports: one for read-write sessions (which redirect connections to the PRIMARY) and
one for read-only sessions (which redirect connections to one of the SECONDARY instances, in a round-robin
fashion).

If your application opens a very high number of database sessions overloading the PRIMARY instance and
additionally does not need to perform updates, you can limit them to use read-only sessions and offload the
PRIMARY.

Once bootstrapped and configured, start MySQL Router (or setup a service for it to start automatically when the
system boots):

https://deepakmysqldba.wordpress.com/2020/08/19/mysql-innodb-cluster-setup-step-by-step/ 6/10
04/07/2024, 14:46 MySQL InnoDB Cluster Setup Step by Step – MySQL DataBase Administration

1. Bootstrap on one of the node

./mysqlrouter –user=mysql –bootstrap mysqldba@10.172.52.60:3306 –directory /database/mysql/mysqlrouter –

conf-use-sockets

2. Now go to the –directory mentioned and start the start.sh

./start.sh

You can now connect a MySQL client to one of the incoming MySQL Router ports and see how the client gets
transparently connected to one of the InnoDB cluster instances. To see which instance we are actually
connected to, we simply query the ‘port’ status variable.

Here we connected to the read-write port (6446), so the Router connected us to the PRIMARY server, which is
listening on port 3310. Now, try connecting to the read-only port (6447) and see what you get.

The \sql shell command switches the MySQL Shell from the default JavaScript mode to SQL mode, where you can
directly execute SQL statements.

Checking Status of the InnoDB Cluster

To check the status of the InnoDB cluster at a later time, you can get a reference to the InnoDB cluster object by
connecting to the cluster through the Router or directly to one of the instances. If you want to make changes to
the cluster, you would need to connect to the read-write port of the Router.

Putting the InnoDB Cluster to the Test

Next we want to challenge the InnoDB cluster and test if the failover really works. We are going to kill the
PRIMARY instance and see if one of the other instances takes over automatically. For this purpose we use the
dba.killLocalInstance() function (in \js mode).

mysql-js> dba.killLocalInstance(“mysqldba: 10.172.52.69:3306′)

After that we again check (from our test session to the MySQL Router) which instance we are now connected to.
The first SELECT fails since the connection to the original PRIMARY was lost. MySQL Shell automatically
reconnects for us and we simply execute the command again.

https://deepakmysqldba.wordpress.com/2020/08/19/mysql-innodb-cluster-setup-step-by-step/ 7/10
04/07/2024, 14:46 MySQL InnoDB Cluster Setup Step by Step – MySQL DataBase Administration

MySQL InnoDB Cluster internal concept

An important thing to know when managing an InnoDB cluster are the states that the cluster can be in,
especially to know how to interpret the reported status of the cluster and specifically what to do to recover
from failure various scenarios.

Group Replication Member States

The state that a group member can be in depends on whether you’re querying the member directly or indirectly
through other members.

States as visible to the member itself:

OFFLINE
RECOVERING
ERROR
ONLINE

States as visible to other members:

RECOVERING
UNREACHABLE
ONLINE

When an instance is running but has not yet joined any group, it’s in the OFFLINE state. Once you join a group, it
will switch to RECOVERING as the instance syncs up with the rest of the group.
Once the distributed recovery is done, it finally switches to ONLINE. If the distributed recovery fails for some
reason, it switches to ERROR.

If an ONLINE member stops responding to the other members (because it has crashed, networking problems,
extreme high load, timeout etc.), its state will be switched to UNREACHABLE. If the UNREACHABLE member
doesn’t return before a timeout, it will be removed from the group by the remaining members, and thus will
appear as (MISSING) in the cluster.status() command output. That process of expelling a member is done
through a vote, so it will only happen if there are enough members available for a quorum.

Group or Replica Set States

The following diagram gives an overview of the possible states that a group can assume and how state
transitions occur.

Solid transition lines show MySQL Shell commands that you can execute in each state, while dashed lines are
events triggered outside of our control.

Some possibilities are omitted from the diagram for the sake of reducing clutter. For example, you can execute
most commands, like addInstance() or rejoinInstance(), in any state as long as there is a quorum.

The group states shown by the cluster.status() command are:

OK– is shown when all members belonging are ONLINE and there is enough redundancy to tolerate at
least one failure.
OK_PARTIAL– when one or more members are unavailable, but there’s still enough redundancy to
tolerate at least one failure.
OK_NO_TOLERANCE– when there are enough ONLINE members for a quorum to be available, but there’s
no redundancy. A two member group has no tolerance, because if one of them becomes UNREACHABLE,

https://deepakmysqldba.wordpress.com/2020/08/19/mysql-innodb-cluster-setup-step-by-step/ 8/10
04/07/2024, 14:46 MySQL InnoDB Cluster Setup Step by Step – MySQL DataBase Administration

the other member can’t form a majority by itself; which means you will have a database outage. But
unlike in a single member group, at least your data will still be safe on at least one of the nodes.
NO_QUORUM– one or more members may still be ONLINE, but cannot form a quorum. In this state, your
cluster is unavailable for writes, since transactions cannot be executed. However, read-only queries can
still be executed and your data is intact and safe.
UNKNOWN– this state is shown if you’re executing the status() command from an instance that
is not ONLINE or RECOVERING. In that case, try connecting to a different member.
UNAVAILABLE– this state is shown in the diagram but will not be displayed by the status() command. In
this state, ALL members of the group are OFFLINE. They may still be running, but they’re not part of the
group anymore. This can happen if all members restart without rejoining, for example.

State transition diagram for InnoDB cluster. (PDF version < http://mysqlserverteam.com/wp-
content/uploads/2016/12/AvailabilityStateMachine.pdf> )

Group Partitions
One exceptional scenario that although rare can be confusing and sometimes dangerous is a group partition,
when members of a group are virtually broken down into 2 or more sub-groups. That can happen if, for
example, you have members in different networks and the communication between them is disrupted. In that
case, all members can be ONLINE, but the members of one group will be seen as UNREACHABLE by the other
and vice-versa. Because of the requirement for a majority, only one partition (at most) will be able to receive
updates, preserving consistency of the database.

But when using the cluster.forceQuorumUsingPartitionOf(), which is explained below, you must be careful that
you’re not in such a situation. If that is the case, you could be creating 2 separate groups that receive updates
separately, which would cause your database to process transactions independently and leave it in an
inconsistent state.

5 member group, all ONLINE5 member group, partitioned into two

Recovering from Failures

The following scenarios are some of the most common situations you can encounter in the event of a failure.
We explain how to identify them and how to recover from them using the InnoDB cluster API in the MySQL Shell.

Server Restarted
If mysqld restarts for any reason (crash, expected restart, reconfiguration etc), it will NOT be in the group
anymore when it comes back up. It needs to rejoin it, which may have to be done manually in some cases. For
that, you can use the cluster.rejoinInstance()command to bring it an instance back to the group. The parameter
it takes is the URI of the instance.

Example: cluster.rejoinInstance(“root@192.168.1.50”)

Loss of Quorum
If so many members of your replica set become UNREACHABLE that it doesn’t have a majority anymore, it will
no longer have a quorum and can’t take decisions on any changes. That includes user transactions, but also
changes to the group’s topology. That means that even if a member that became UNREACHABLE returns, it will
be unable to rejoin the group for as long as the group is blocked.

To recover from that scenario, we must first unblock the group, by reconfiguring it to only consider the
currently ONLINE members and ignore all others. To do that, use
the cluster.forceQuorumUsingPartitionOf() command, by passing the URI of one of the ONLINE members of your
replica set. All members that are visible as ONLINE from that member will be added to the redefined group.

https://deepakmysqldba.wordpress.com/2020/08/19/mysql-innodb-cluster-setup-step-by-step/ 9/10
04/07/2024, 14:46 MySQL InnoDB Cluster Setup Step by Step – MySQL DataBase Administration

Note that this is a dangerous command. As explained above, if you happen to have a partition in your group,
you could be accidentally be creating a split-brain, which will lead to an inconsistent database. Make sure that
any members that are UNREACHABLE are really OFFLINE before using this command.

Example: cluster.forceQuorumUsingPartitionOf(“root@192.168.1.30”)

All Members OFFLINE

The cluster.forceQuorumUsingPartitionOf() command requires at least one instance to be still ONLINE and part
of the group. If somehow all your members are now OFFLINE, you can only recover the group if you “bootstrap”
the group again, out of a single seed member. To perform that, you need to use
the dba.restoreFromCompleteOutage()command on a designed seed instance, and then rejoinInstance() on the
remaining members until you have your cluster fully restored.
Note: this command will be available starting from MySQL Shell 1.0.7

Conclusion

MySQL InnoDB cluster aims to make High Availability accessible to MySQL users of all levels of knowledge and
experience. While the cluster.status() command lets you monitor the state of your cluster at a glance, it’s very
important to know how to interpret it, so that you know when action is required from you and what you should
do to ensure that your MySQL databases remain functioning optimally.

Published by

deepakpadhi
View all posts by deepakpadhi < https://deepakmysqldba.wordpress.com/author/deepakmysqldba/>

1 Comment

Pingback: MySQL 8 Fail Senerio – MySQL DataBase Administration <

https://deepakmysqldba.wordpress.com/2021/05/30/mysql-8-failover-checks/>

Leave a Comment

MySQL DataBase Administration < https://deepakmysqldba.wordpress.com/> , Create a free website or blog at

WordPress.com. < https://wordpress.com/?ref=footer_website>

https://deepakmysqldba.wordpress.com/2020/08/19/mysql-innodb-cluster-setup-step-by-step/ 10/10