Hadoop Distributed File

System Basics
Contents
• Hadoop Distributed File System Design Features
• HDFS Components
• HDFS Block Replication
• HDFS Safe Mode
• Rack Awareness
• NameNode High Availability
• HDFS Checkpoints and Backups
• HDFS Snapshots
• HDFS NFS Gateway
Hadoop Distributed File System
Design Features
• Big data processing
• write-once/read-many model
• No caching of data
• Design based on Google File System(GFS)
• Designed for Data Streaming
• Data Locality
• Moving Computation
HDFS Components
• Node Types
• Name Node - Manage Metadata
• Data Node – Store/Retrieve Data

•Design-Master/Slave Architecture
• Master(NN)-File System Namespace
• Slave(DN)-Read/Write Request
Various system roles in an HDFS deployment
• Disk Files-
• fsimage_*
• image of the file system state used only at startup by the NameNode.
• Stores metadata

• edit_*
• A series of modifications done to the file system after starting the NameNode.

• Location -
dfs.namenode.name.dir property in the hdfs-site.xml file.
HDFS Block Replication Cluster Replication
Factor
>8 3
• Replicates across the cluster >1 and <8 2

• amount of replication - dfs.replication Single

Machine
1

in hdfs-site.xml file
• Replicating each block across number of
machine(default=3)
• HDFS based on block size default is
64MB
• Splits are based on logical partitioning
of data.
HDFS Safe Mode
• When the NameNode starts
• enters a read-only safe mode
• blocks cannot be replicated or deleted
• Safe Mode enables the NameNode to perform two important
processes:
• Loading fsimage file into memory and replaying the edit log
• Mapping between blocks and data nodes is created,at least one copy of the
data is available before safe mode exit.
• Safe mode for maintenance - hdfs dfsadmin-safemode command
-Administrator maintenance
Rack Awareness
• Data Locality
• Hadoop MapReduce is to move the computation to the data
1. Data resides on the local machine (best).
2. Data resides in the same rack (better).
3. Data resides in a different rack (good).
Example- YARN Scheduler
Pros- improved fault tolerance
Cons-Entire rack failure then performance degraded
NameNode High Availability
• Earlier,NameNode was a single point of failure
• NameNode High Availability (HA) -to provide
true failover service
HA

Active Standby
Name node Name node

all client HDFS maintains enough state to

operations in the provide a fast failover
cluster

Apache ZooKeeper is used to monitor the NameNode

health.
 HDFS NameNode Federation
• Earlier,single
namespace/nameNode
• Federation addresses this
limitation
• Multiple nameNodes/namespaces
• Key benefits:
• Namespace scalability
• Better performance
• System isolation
HDFS Checkpoints and Backups
• Checkpoints:
• fsimage - stores metedata
• edit log file - file system
modifications
• Backups:
• Maintains an up-to-date copy of the file system namespace both in
memory and on disk
• NameNode supports one BackupNode at a time
• No CheckpointNodes may be registered if a Backup node is in use.
HDFS Snapshots
• Created by administrators using the hdfs dfs snapshot command.
• HDFS snapshots are read-only point-in-time copies of the file system
• Features:
• sub-tree of the file system or the entire file system
• data backup, protection against user errors, and disaster recovery
• Instant creation
• Data not copied
• record the block list and the file size
• Doesnot affect regular HDFS operations.
Hadoop MapReduce
Framework
Contents
• MapReduce Model
• MapReduce Parallel Data Flow
• Data flow for a word count program
• Process placement during MapReduce
• Fault Tolerance
• Speculative Execution
MapReduce Model
• Simple, powerful
• Stages:
• Mapping stage
• Reducing stage
• Example word count
• grep " <<word>>" <<filename>> | wc -l/c
• The mapper and reducer functions are both defined with respect to
data structured in (key, value) pairs.
• The mapper takes one pair of data with a type in one data domain,
and returns a list of pairs in a different domain:
• Map(key1,value1) → list(key2,value2)
• The reducer function is then applied to each key–value pair, which in
turn produces a collection of values in the same domain:
• Reduce(key2, list (value2)) → list(value3)
• Each reducer call typically produces either one value (value3) or an
empty response. Thus, the MapReduce framework transforms a list of
(key, value) pairs into a list of values.
• Properties:
• Data flow is in one direction (map to reduce)
• Original data is preserverd
• No dependency
• Hadoop accomplishes parallelism by using a distributed file system
(HDFS) to slice and spread data over multiple servers.
• Data slice are then combined in the reducer step
MapReduce Parallel Data Flow
• Input Splits-HDFS distributes and replicates data over multiple servers
• Map Step-Parallel nature of Hadoop
• Combiner Step-Key–value pairs are combined prior
• Shuffle Step-similar keys are combined and counted by reducer
• Reduce Step-Actual Reduction happens and outputs to HDFS
Simple Hadoop MapReduce data flow for a word
count program
The count for run can be
combined into (run,2) before
the shuffle. This optimization
can help minimize the
amount of data transfer
needed for the shuffle phase.
• Placement of mappers and reducers are done by Hadoop YARN
resource manager and MapReduce framework.
• Nodes can run both mapper and reducer tasks.
• The dynamic nature of YARN enables the work containers used by
completed map tasks to be returned to the pool of available
resources.
• The shuffle stage makes sure the necessary data are sent to each
mapper.
• Mappers can complete at any time.
Process placement during MapReduce
Fault Tolerance
• Strict control of data flow throughout the execution of the program.
• Design of MapReduce makes it possible to easily recover from the
failure of one or many map processes as it has no dependency.
• Failed reducers can be restarted.
• MapReduce ApplicationMaster will need to restart the reducer tasks
• Process is totally transparent to the user and provides a fault-tolerant
system to run applications.
Speculative Execution
• Challenges with large clusters-inability to predict or manage
unexpected system failures.
• Congested network, slow disk controller, failing disk, high processor
load, or similar problem lead to slow performance.
• One part of a MapReduce process runs slowly, then the application
cannot complete.
• As input data are immutable in the MapReduce process, start a copy
of a running map process without disturbing any other running
mapper processes.
• Speculative execution can reduce cluster efficiency,it can be turned
off and on in the mapred-site.xml configuration file.
 Node A Task
Node 1
Progress
Task(Slow)

Scheduler
Node 2
Node B Launch
Speculative
Task(Duplicate)

Node 3