UBP

– LIMOS

BIG DATA
EXERCISES – HDFS
L AURENT D ’O RAZIO
2015/2016

Installation
This section presents how to install Hadoop. The first step consists in downloading Hadoop’s binary.
Such a binary is available at the following address:

http://hadoop.apache.org

Once downloaded, extract its content using the following commands:

tar xvf hadoop<>.tar

Then update your JAVA_HOME and HADOOP_HOME using the following command:

export JAVA_HOME=<path_to_java_home>
export HADOOP_HOME=<path_to_hadoop>

Pseudo-distributed mode
This part aims at using HDFS in a pseudo-distributed mode (simulating communications on a single
node).

ssh keys generation

If ssh on localhost is not possible, then generate ssh keys with the appropriate rights in a terminal
with the following commands:

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

Configuration
Edit the following files:

etc/hadoop-env.sh

...
export JAVA_HOME=<path_to_java_home>
...
etc/hadoop/core-site.xml

<configuration>
<property>

Big Data – Laurent d’Orazio – 2015/2016 1 /3

 UBP – LIMOS

<name>hadoop.tmp.dir</name>
<value>/home/oth/Documents/work/data/hadoop/tmp</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
etc/hadoop/hdfs-site.xml

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>

Execution
Format the file system

bin/hdfs namenode -format

Start the nodes (name and data)

sbin/start-all.sh

Note
Shut down the nodes

sbin/stop-all.sh

File management
Make some HDFS directories

bin/hdfs dfs -mkdir /user

bin/hdfs dfs -mkdir /user/oth
Put one file

bin/hdfs dfs put <path> /user/oth

Bibliography
http://hadoop.apache.org

Big Data – Laurent d’Orazio – 2015/2016 2 /3

 UBP – LIMOS

Big Data – Laurent d’Orazio – 2015/2016 3 /3