Professional Documents
Culture Documents
Hadoop Migration and Upgradation
Hadoop Migration and Upgradation
Preparation Phase
Access Grants Assessment of Current Infrastructure Inventory of Current Setup Meta-Data Backup
New Cluster Setup with Hadoop 2.x . OR Inline Migration to Hadoop 2.x from Hadoop 1.x Methodology
Staging Environment
Replica of Current Setup in Staging Environment Upgrade in Staging Environment Validation of Migration in Staging Environment
Production Rollout
(c) Infinity
Upgrade a cluster by configuring a new cluster with same capacity and newer Hadoop version and then migrate the files/data from old cluster to new one.
(c) Infinity
CopyToLocal /CopyFromLocal
The process flows, the files are copied to local drive using Hadoop command CopyToLocal and the files are then pushed to the new cluster using CopyFromLocal, and the older cluster can be decommissioned.
This is a kind of cluster to cluster copy, using Hadoop cp command the files are transferred from one HDFS to other HDFS. As the version are different we need a mechanism called copy from HFTP where the command is executed from the target cluster by defining source as old cluster with HFTP protocol and target as HDFS protocol.
This is a kind of cluster to cluster copy, using Hadoop distcp command the files are transferred from one HDFS to other HDFS. As the version are different we need a mechanism called copy from HFTP where the command is executed from the target cluster by defining source as old cluster with HFTP protocol and target as HDFS protocol.
(c) Infinity
Upgrading
existing cluster from V1 to V2 inline by Installing/Configuring new cluster and updating metadata.
Hadoop V2
Hadoop V1
(c) Infinity
Shutdown of Cluster/Node s
Production Rollout
Upgrade Phase
(c) Infinity
http://wiki.apache.org/hadoop/Hadoop_Upgrade
(c) Infinity
Drop
(c) Infinity