HDFS Commands

1. jps: To check the Hadoop services are up and running

[cloudera@quickstart ~]$ jps

2. hadoop version : To check the the current version of hadoop installed

[cloudera@quickstart ~]$ hadoop version

3. help: To get the help about commands

[cloudera@quickstart ~]$ hadoop fs -help

-touchz <path> ... :

Creates a file of zero length at <path> with current time as the
timestamp of
that <path>. An error is returned if the file exists with non-zero
length

-usage [cmd ...] :

Displays the usage for given command or all commands if none is
specified.

Generic options supported are

-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|resourcemanager:port> specify a ResourceManager
-files <comma separated list of files> specify comma separated files to
be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar
files to include in the classpath.
-archives <comma separated list of archives> specify comma separated
archives to be unarchived on the compute machines.

The general command line syntax is

bin/hadoop command [genericOptions] [commandOptions]

4. help command: To get the help about specific command

[cloudera@quickstart ~]$ hadoop fs -help ls
-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [<path> ...] :
List the contents that match the specified file pattern. If path is not
specified, the contents of /user/<currentUser> will be listed. For a
directory a
list of its direct children is returned (unless -d option is specified).

Directory entries are of the form:

permissions - userId groupId sizeOfDirectory(in bytes)
modificationDate(yyyy-MM-dd HH:mm) directoryName

and file entries are of the form:

permissions numberOfReplicas userId groupId sizeOfFile(in bytes)
modificationDate(yyyy-MM-dd HH:mm) fileName

-C Display the paths of files and directories only.

-d Directories are listed as plain files.
-h Formats the sizes of files in a human-readable fashion
rather than a number of bytes.
-q Print ? instead of non-printable characters.
-R Recursively list the contents of directories.
-t Sort files by modification time (most recent first).
-S Sort files by size.
-r Reverse the order of the sort.
-u Use time of last access instead of modification for
display and sorting.

5. usage: To get the options for specific command

[cloudera@quickstart ~]$ hadoop fs -usage ls
Usage: hadoop fs [generic options] -ls [-C] [-d] [-h] [-q] [-R] [-t] [-S]
[-r] [-u] [<path> ...]

6. ls: This command is used to list all the files and directories
[cloudera@quickstart ~]$ hadoop fs -ls /
Found 7 items
drwxrwxrwx - hdfs supergroup 0 2017-10-23 10:29 /benchmarks
drwxr-xr-x - hbase supergroup 0 2021-08-01 23:58 /hbase
drwxr-xr-x - solr solr 0 2017-10-23 10:32 /solr
drwxrwxrwt - hdfs supergroup 0 2020-09-12 10:53 /tmp
drwxr-xr-x - hdfs supergroup 0 2017-10-23 10:31 /user
drwxr-xr-x - hdfs supergroup 0 2017-10-23 10:31 /var
drwxr-xr-x - cloudera supergroup 0 2020-09-23 10:11 /wordoutput

7. pwd: To check the present working directory

[cloudera@quickstart ~]$ pwd
/home/cloudera

8. mkdir: To create a directory in Hadoop dfs

[cloudera@quickstart ~]$ hadoop fs -mkdir demo

[cloudera@quickstart ~]$ hadoop fs -ls

Found 6 items
-rw-r--r-- 1 cloudera cloudera 72 2021-07-31 10:40 WCFile.txt
drwxr-xr-x - cloudera cloudera 0 2021-07-30 05:28 WCOutput
drwxr-xr-x - cloudera cloudera 0 2021-07-30 10:52 WCOutput1
drwxr-xr-x - cloudera cloudera 0 2021-07-31 10:43 WCOutput2
drwxr-xr-x - cloudera cloudera 0 2021-07-31 10:53 WCOutput3
drwxr-xr-x - cloudera cloudera 0 2021-08-02 00:18 demo

9. cat: To creates a file and display it.

[cloudera@quickstart ~]$ cat >sample.txt
This file is created for demo purpose.

[cloudera@quickstart ~]$ cat sample.txt

This file is created for demo purpose.

10. copyFromLocal : To copy files/folders from local file system to hdfs

store.
This is the most important command. Local filesystem means the files
present on the OS.
[cloudera@quickstart ~]$ hadoop fs -copyFromLocal
/home/cloudera/sample.txt /user/cloudera/demo1/

[cloudera@quickstart ~]$ hadoop fs -ls /user/cloudera/demo1/Found 1 items

-rw-r--r-- 1 cloudera cloudera 39 2021-08-02 00:24
/user/cloudera/demo1/sample.txt

[cloudera@quickstart ~]$ hadoop fs -cat /user/cloudera/demo1/sample.txt

This file is created for demo purpose.

[cloudera@quickstart ~]$ cat >sample.txt

This file is created to demo
the use of PUT command.

[cloudera@quickstart ~]$ cat sample.txt

This file is created to demo
the use of PUT command.

11. put: To copy files/folders from local file system to hdfs store.
[cloudera@quickstart ~]$ haddop fs -put sample.txt
/user/cloudera/demo1/sample1.txtbash: haddop: command not found
[cloudera@quickstart ~]$ hadoop fs -put sample.txt
/user/cloudera/demo1/sample1.txt
[cloudera@quickstart ~]$ hadoop fs -ls /user/cloudera/demo1/
Found 2 items
-rw-r--r-- 1 cloudera cloudera 39 2021-08-02 00:24
/user/cloudera/demo1/sample.txt
-rw-r--r-- 1 cloudera cloudera 53 2021-08-02 00:29
/user/cloudera/demo1/sample1.txt

[cloudera@quickstart ~]$ hadoop fs -cat /user/cloudera/demo1/sample.txt

This file is created for demo purpose.

[cloudera@quickstart ~]$ hadoop fs -cat /user/cloudera/demo1/sample1.txt

This file is created to demo
the use of PUT command.

[cloudera@quickstart ~]$ cat >sample3.txt

This file is created to demo
use ofmove command.

12. moveFromLocal: This command will move file from local to hdfs.
[cloudera@quickstart ~]$ hadoop fs -moveFromLocal sample.txt
[cloudera@quickstart ~]$ hadoop fs -moveFromLocal sample3.txt

[cloudera@quickstart ~]$ ls
a.out Downloads orders.java --warehouse-
dir
categories.java eclipse parcels WCFile.txt
cloudera-manager enterprise-deployment.json --password WCFile.txt~
cm_api.py express-deployment.json Pictures WordCount.jar
customers.java insertion.c~ products.java
wordinput.txt~
departments.java kerberos Public workspace
dept.java lib Templates
Desktop Music --username
Documents order_items.java Videos
[cloudera@quickstart ~]$ hadoop fs -ls
Found 9 items
-rw-r--r-- 1 cloudera cloudera 72 2021-07-31 10:40 WCFile.txt
drwxr-xr-x - cloudera cloudera 0 2021-07-30 05:28 WCOutput
drwxr-xr-x - cloudera cloudera 0 2021-07-30 10:52 WCOutput1
drwxr-xr-x - cloudera cloudera 0 2021-07-31 10:43 WCOutput2
drwxr-xr-x - cloudera cloudera 0 2021-07-31 10:53 WCOutput3
drwxr-xr-x - cloudera cloudera 0 2021-08-02 00:18 demo
drwxr-xr-x - cloudera cloudera 0 2021-08-02 00:29 demo1
-rw-r--r-- 1 cloudera cloudera 53 2021-08-02 00:33 sample.txt
-rw-r--r-- 1 cloudera cloudera 49 2021-08-02 00:34 sample3.txt

13. du: It will give the size of each file in directory.

[cloudera@quickstart ~]$ hadoop fs -du /user/cloudera/demo1
39 39 /user/cloudera/demo1/sample.txt
53 53 /user/cloudera/demo1/sample1.txt

14. df:: This command will give the total size of directory/file.
[cloudera@quickstart ~]$ hadoop fs -df
Filesystem Size Used Available Use%
hdfs://quickstart.cloudera:8020 58531520512 919380092 45821091840 2%
[cloudera@quickstart ~]$ hadoop fs -cat /user/cloudera/demo1/sample3.txt
cat: `/user/cloudera/demo1/sample3.txt': No such file or directory

15. cp: This command is used to copy files within hdfs

[cloudera@quickstart ~]$ hadoop fs -cp /user/cloudera/sample3.txt
/user/cloudera/demo1
[cloudera@quickstart ~]$ hadoop fs -cp /user/cloudera/sample3.txt
/use/cloudera/demo1cp: `/use/cloudera/demo1': No such file or directory
[cloudera@quickstart ~]$ hadoop fs -ls /user/cloudera/demo1
Found 3 items
-rw-r--r-- 1 cloudera cloudera 39 2021-08-02 00:24
/user/cloudera/demo1/sample.txt
-rw-r--r-- 1 cloudera cloudera 53 2021-08-02 00:29
/user/cloudera/demo1/sample1.txt
-rw-r--r-- 1 cloudera cloudera 49 2021-08-02 00:41
/user/cloudera/demo1/sample3.txt

16. mv: This command is used to move files within hdfs.

[cloudera@quickstart ~]$ hadoop fs -mv /user/cloudera/sample3.txt
/user/cloudera/demo
[cloudera@quickstart ~]$ hadoop fs -ls /user/cloudera/demo
Found 1 items
-rw-r--r-- 1 cloudera cloudera 49 2021-08-02 00:34
/user/cloudera/demo/sample3.txt

17. rm: This command deletes a file from HDFS.

It is very useful command when you want to delete a file.
[cloudera@quickstart ~]$ hadoop fs -rm /user/cloudera/demo1/sample3.txt
Deleted /user/cloudera/demo1/sample3.txt

18. rmr: This command deletes a file from HDFS recursively.

It is very useful command when you want to delete a non-empty directory.
[cloudera@quickstart ~]$ hadoop fs -rm -R /user/cloudera/demo1
Deleted /user/cloudera/demo1

19. copyToLocal (or) get: To copy files/folders from hdfs store to local
file system.
[cloudera@quickstart ~]$ hadoop fs -copyToLocal
/user/cloudera/demo/sample3.txt /home/cloudera/hdfs_bkp

20. moveToLocal (or) get: To move files/folders from hdfs store to local
file system.
[cloudera@quickstart ~]$ hadoop fs -moveToLocal
/user/cloudera/demo/sample3.txt /home/cloudera/hdfs_bkp

21. touchz: It creates an empty file.

[cloudera@quickstart ~]$ hadoop fs -touchz /user/cloudera/demo/sample4.txt
22. get: To copy files/folders from hdfs store to local file system.
[cloudera@quickstart ~]$ hadoop fs -get /user/cloudera/demo/sample4.txt
/home/cloudera/hdfs_bkp
get: `/home/cloudera/hdfs_bkp': File exists

23. setrep: This command is used to change the replication factor of a

file/directory in HDFS.
By default it is 3 for anything which is stored in HDFS (as set in hdfs
core-site.xml).
[cloudera@quickstart ~]$ hadoop fs -setrep -R 4
/user/cloudera/demo/sample4.txt