sys_WordMark_AT_Pag

e1
HDFS HA RUNBOOK FOR SLB
PROD/QA/DEV ENVIRONMENTS

© Copyright 2019, ATOS PGS sp. z o.o. All rights reserved. Reproduction in whole or in part is prohibited without the prior
written consent of the copyright owner. For any questions or remarks on this document, please contact Atos Poland Global
Services, +48 22 4446500.

AUTHOR(S) : Marcin Niewiatowski

DOCUMENT NUMBER :
VERSION : 1.0
STATUS : Final
SOURCE : Atos Poland Global Services
DOCUMENT DATE : 06 December 2019RELEASED FOR TRAININGRELEASED FOR
OPERATIONSREVIEW BEFORE
NUMBER OF PAGES : 11

OWNER
HDFS HA Runbook for SLB PROD/QA/DEV environments

sys_WordMark_AT_Continued
version: 0.1

Public

document number:

Contents
1 Audience and document purpose...................................................................
2 Components in scope...................................................................................
3 Ambari and Name Node................................................................................
3.1 Ambari database (PostgreSQL) backup and restoration....................................
3.2 Name Node HA configuration and backup.......................................................
4 Journal nodes..............................................................................................
4.1 Jounral node transactions backup..................................................................
5 Data Nodes...............................................................................................
6 Zookeeper................................................................................................

Atos Poland Global Services

06 December 2019
2 of 11
HDFS HA Runbook for SLB PROD/QA/DEV environments

sys_WordMark_AT_Continued
version: 0.1

Public

document number:

List of changes
version Date Description Author(s)
0.1 20.11.2019 Initial document structure created Marcin
Niewiatowski
0.2 29.11.2019 Content update Marcin
Niewiatowski
1.0 06.12.2019 Document finalized with the pdf documents included Marcin
(in case of no HDP docs available online) Niewiatowski

Atos Poland Global Services

06 December 2019
3 of 11
 HDFS HA Runbook for SLB PROD/QA/DEV environments

sys_WordMark_AT_Continued
version: 0.1

Public

document number:

1 Audience and document purpose

The document has been prepared for the SLB HDP platform administrators and ATOS team
responsible for maintaining the PROD/QA and DEV environments. End-user/business team was
not meant as a participant in the process nor the document recipient.
Scope of the document describes the current (for the date of document creation) configuration,
processes and detailed steps leading to archive, update and restore the services functionality in
case of HA/DR drill or real-life issue.
Processes described in this document were based on the vendor (HortonWorks) best practices
and/or documentation. Links to them are the integral part of the knowledge required to operate
the runbook.
During this runbook creation – authors followed the suggestions brought together in the
following articles:
Part one:
https://community.cloudera.com/t5/Community-Articles/Disaster-recovery-and-Backup-best-
practices-in-a-typical/ta-p/246641
and part two:
https://community.cloudera.com/t5/Community-Articles/Disaster-recovery-and-Backup-best-
practices-in-a-typical/ta-p/246651
For the knowledge systematization – the following vendor support case was created by Arshad
Amir Jamadar:
https://my.cloudera.com/cases/639712/comments

Atos Poland Global Services

06 December 2019
4 of 11
 HDFS HA Runbook for SLB PROD/QA/DEV environments

sys_WordMark_AT_Continued
version: 0.1

Public

document number:

2 Components in scope
To cover the HA/DR processes for HDFS service = the following components were described:
1. Ambari - configuration files for HDFS
2. NameNodes configuration backup (including HDFS data map image)
3. DataNode availability, commissioning and decommissioning
4. JournalNode (including journal data recreation)
5. Zookeeper – as the HDFS cluster managing and coordinating process

Atos Poland Global Services

06 December 2019
5 of 11
 HDFS HA Runbook for SLB PROD/QA/DEV environments

sys_WordMark_AT_Continued
version: 0.1

Public

document number:

3 Ambari and Name Node

The HDFS service is being managed by use of Ambari. That determines the configuration steps for
creating the high availability configuration for Name Node service – which is described in the
documentation:
https://docs.cloudera.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-user-guide/content/
namenode_high_availability.html

3.1 Ambari database (PostgreSQL) backup and restoration

The dedicated PostgreSQL runbook covers all the technical steps and proceedings related to
HA/DR in general. This part of the HDFS runbook describes the basic Ambari DB backup.

Vendor documentation proposes the following:

https://community.cloudera.com/t5/Community-Articles/Backing-up-the-Ambari-database-with-
Postgres/ta-p/246352

https://docs.cloudera.com/HDPDocuments/Ambari-2.1.1.0/bk_ambari_reference_guide/content/
_back_up_current_data.html

The Ambari server configuration files are located on Ambari server at

/etc/ambari-server/conf and /var/lib/ambari-server/.

These folders were backed up initially on NLXS5133 server at /home/backup location, for each
environment respectively. There is no need for restoration process of Ambari configuration,
because it is assumed that either:
- we will install the new Ambari instance and the old config files can just be
compared
- or we will work on the current instance of the Ambari server and saved files can
be used as a reference to the initial configuration only.

The substantial information about Ambari configuration is being stored in the two Ambari
databases, named ‘ambari’ and ‘ambarica’.

The tasks to perform the Ambari DB backup are:

1. Stop the existing Ambari server

2. Create the Ambari databases, named ‘ambari’ and ‘ambarica’ dumps (using pg_dump
command) to the two separated sql files
3. Start Ambari server

For the SLB HDP platform - the PostgreSQL DB dumps are being archived at NLXS5133
infrastructure server.

The root path to the PROD/QA/DEV backup is:

/home/backup/PRODbackups/
/home/backup/QAbackups
/home/backup/DEVbackups

Atos Poland Global Services

06 December 2019
6 of 11
 HDFS HA Runbook for SLB PROD/QA/DEV environments

sys_WordMark_AT_Continued
version: 0.1

Public

document number:
The backup files are located in the corresponding subfolders with PostgreSQL server name, for
example:
/home/backup/QAbackups/nlxs5146_postgres

The tasks to perform the restoration are:

1. Install the new instance of Ambari server on dedicated cluster node and verify the started
process and Ambari instance
2. Stop the Ambari server
3. Stop the Ambari agent
4. Connect to the PostgreSQL instance, containing the new Ambari server databases
5. Drop the newly created ambari and ambarica databases and verify if they were dropped
successful
6. Create manually two new databases named ambari and ambarica
7. Copy the sql backups of the old databases from backup server to PostgreSQL server
8. As the postgres user restore the contents of ambari and ambarica databases, by
executing the backed up sql files (you can use the psql -f <filename.sql> option)
9. Start the Ambari server
10. Start the Ambari agent
11. Connect to the recreated Ambari instance and verify the contents, configuration and
functionalities
12. Update all the cluster nodes Ambari agents with the new Ambari Server IP address/fqdn

3.2 Name Node HA configuration and backup

The Name Node service in DEV/QA and PROD is configured with HA and contains two
instances: Primary Name Node and Secondary Name Node. Only one instance is being marked as
the Active Name Node at a time, the other node is being marked as a Standby Name Node. The
Active Name Node role is assigned dynamically by the Zookeeper service.

3.2.1 fsimage backup/restore

The Name Node fsimage procedure is described in the following document:

As the alternative to the procedure above – to back up the checkpoint directories on Active Name
Node – make a copy the contents of the following directory:
/hadoop/hdfs/namenode/current

The fsimage copies are being captured from the Active Name Node location and stored at
infrastructure server nlxs5133, in the following directories:

/home/backup/PRODbackups/nn_metadata/archive
/home/backup/QAbackups/nn_metadata/archive
/home/backup/DEVbackups/nn_metadata/archive

for each environment respectively.

Atos Poland Global Services

06 December 2019
7 of 11
HDFS HA Runbook for SLB PROD/QA/DEV environments

sys_WordMark_AT_Continued
version: 0.1

Public

document number:
The restoration process is the opposite operation – having both instances of NN stopped (and
HDFS service stopped on the cluster, with all the dependant services, too) – restore the contents
of the /hadoop/hdfs/namenode/current directory from saved backup (on both Name Nodes) and
start the HDFS service.

REMARK

There is a potential risk – if the /Hadoop directory is placed on the root (/) partition of the
server. If this partition will be filled-out by the HDFS transactions – it could disable the whole
server functionality and prevent the operating system from correct boot-up process. To avoid this
potential risk – the Name Node dfs.namenode.name.dir can be placed on the different
partition/filesystem.

Atos Poland Global Services

06 December 2019
8 of 11
 HDFS HA Runbook for SLB PROD/QA/DEV environments

sys_WordMark_AT_Continued
version: 0.1

Public

document number:

4 Journal nodes
The Journal Nodes are distributed system to store edits of HDFS. Active Namenode as a client
writes edits to journal nodes and commit only when its replicated to all the journal nodes in a
distributed system. Standby NN need to read data from edits to be in sync with Active one. It can
read from any of the replica stored on journal nodes.
The Journal node configuration is determined in Ambari HDFS service configuration and
together with the Name Nodes they are managed by Zookeeper to ensure the HA for HDFS.

4.1 Jounral node transactions backup

The individual Journal node backup should contain the contents of dfs.journalnode.edits.dir,
which value is configured in Ambari hdfs config > advanced > advanced-hdfs site section.
In case of our clusters it has been set up as:
/hadoop/hdfs/journal

Since the contents of all three Journal nodes edits dir should be exactly the same – in case of
malfunction of one of them – it will be enough to
- Stop the journal node service on the impacted host
- copy data from another healthy Journal node
- start the Journal node service

The HDFS requires that two out of the three journal nodes should be always available.
Backing up the Journal nodes is not necessary – because Ambari can always create the proper
Journal node instance – using the appropriate wizard.
The Jorunal node management is described at the chapter 4 of the following document (describing
the HA configuration for HDP platform):

REMARK

There is a potential risk – if the /Hadoop directory is placed on the root (/) partition of the
server. If this partition will be filled-out by the HDFS transactions – it could disable the whole
server functionality and prevent the operating system from correct boot-up process. To avoid this
potential risk – the Name Node dfs.namenode.name.dir can be placed on the different
partition/filesystem.

Atos Poland Global Services

06 December 2019
9 of 11
 HDFS HA Runbook for SLB PROD/QA/DEV environments

sys_WordMark_AT_Continued
version: 0.1

Public

document number:

5 Data Nodes
Data Nodes are multiplied across the platform, we have a full redundancy of the singular
machine functions. Statistically – since we have the HDFS replication factor set to the default
value of 3 – the three Data Nodes can be unavailable at the same time – without any data loose
or hdfs degradation.

Currently – the clusters for DEV/QA and PROD are configured to accept on Data Nade malfunction
without the CRITICAL alert triggering. For some cases – as the service tasks related with the
hardware fxes/upgrades clusters can be safely configured to accept two Data Nodes unavailability.

In case of any Data Node malfunction – the node can by easily decommissioned from the cluster.
Then we can proceed with the service maintenance/investigating. After the investigation – the
operating system, node data and configuration settings should be cleared and the Data Node can
be re-added to the cluster infrastructure, with the common “Add new host” wizard.

There are no HA-related non-standard tasks or steps to reproduce were planned for Data Nodes.

Atos Poland Global Services

06 December 2019
10 of 11
 HDFS HA Runbook for SLB PROD/QA/DEV environments

sys_WordMark_AT_Continued
version: 0.1

Public

document number:

6 Zookeeper
Zookeeper in HDFS service context is responsible for managing the Name Nodes role
assignment and coordinates the distributed processes between the Name Nodes while the HA is
being used for hdfs.
Zookeeper servers are multiplied across the cluster (usually the three instances installed). The
2/3 of them should be running to guarantee the Zookeper standard operational status. Both Name
Nodes have Zookeeper Failover Controller service installed (during the HA enablement process)
and these services are responsible for reporting and negotiating the Name Node active/standby
status.

The Name Node HA enablement has been described in the following documentation (Chapter
5.5):
https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_hadoop-high-availability/
content/ch_HA-NameNode.html

Atos Poland Global Services

06 December 2019
11 of 11