Professional Documents
Culture Documents
Commands Guide: Apache Hadoop Core Docs Stable1
Commands Guide: Apache Hadoop Core Docs Stable1
CommandsGuide
Apache>Hadoop>Core>docs>stable1
Search
LastPublished:08/04/201309:13:21
Searchthesitewithgoogle
CommandsGuide
Overview
GenericOptions
UserCommands
archive
distcp
fs
fsck
fetchdt
jar
job
pipes
queue
version
CLASSNAME
classpath
AdministrationCommands
balancer
daemonlog
datanode
dfsadmin
mradmin
jobtracker
namenode
secondarynamenode
tasktracker
Overview
Allhadoopcommandsareinvokedbythebin/hadoopscript.Runningthehadoopscriptwithout
anyargumentsprintsthedescriptionforallcommands.
Usage:hadoop[configconfdir][COMMAND][GENERIC_OPTIONS]
[COMMAND_OPTIONS]
http://hadoop.apache.org/docs/stable1/commands_manual.html
1/8
6/1/2016
CommandsGuide
Hadoophasanoptionparsingframeworkthatemploysparsinggenericoptionsaswellas
runningclasses.
COMMAND_OPTION
Description
configconfdir OverwritesthedefaultConfigurationdirectory.Defaultis
${HADOOP_HOME}/conf.
GENERIC_OPTIONS Thecommonsetofoptionssupportedbymultiplecommands.
COMMAND
Variouscommandswiththeiroptionsaredescribedinthefollowing
COMMAND_OPTIONS sections.ThecommandshavebeengroupedintoUserCommandsand
AdministrationCommands.
GenericOptions
Thefollowingoptionsaresupportedbydfsadmin,fs,fsck,jobandfetchdt.Applicationsshould
implementTooltosupportGenericOptions.
GENERIC_OPTION
Description
conf<configurationfile> Specifyanapplicationconfigurationfile.
D<property=value>
Usevalueforgivenproperty.
fs<local|namenode:port> Specifyanamenode.
jt
Specifyajobtracker.Appliesonlytojob.
<local|jobtracker:port>
files<commaseparated
Specifycommaseparatedfilestobecopiedtothemap
reducecluster.Appliesonlytojob.
listoffiles>
libjars<commaseperated Specifycommaseparatedjarfilestoincludeinthe
classpath.Appliesonlytojob.
listofjars>
archives<commaseparated Specifycommaseparatedarchivestobeunarchivedon
thecomputemachines.Appliesonlytojob.
listofarchives>
UserCommands
Commandsusefulforusersofahadoopcluster.
archive
Createsahadooparchive.MoreinformationcanbefoundatHadoopArchives.
Usage:hadooparchivearchiveNameNAME<src>*<dest>
COMMAND_OPTION
Description
archiveNameNAME Nameofthearchivetobecreated.
src
Filesystempathnameswhichworkasusualwithregularexpressions.
dest
Destinationdirectorywhichwouldcontainthearchive.
distcp
Copyfileordirectoriesrecursively.MoreinformationcanbefoundatHadoopDistCpGuide.
Usage:hadoopdistcp<srcurl><desturl>
COMMAND_OPTION
Description
SourceUrl
http://hadoop.apache.org/docs/stable1/commands_manual.html
2/8
6/1/2016
CommandsGuide
srcurl
desturl
DestinationUrl
fs
Usage:hadoopfs[GENERIC_OPTIONS][COMMAND_OPTIONS]
Runsagenericfilesystemuserclient.
ThevariousCOMMAND_OPTIONScanbefoundatFileSystemShellGuide.
fsck
RunsaHDFSfilesystemcheckingutility.SeeFsckformoreinfo.
Usage:hadoopfsck[GENERIC_OPTIONS]<path>[move|delete|
openforwrite][files[blocks[locations|racks]]]
COMMAND_OPTION
<path>
move
delete
openforwrite
files
blocks
locations
racks
Description
Startcheckingfromthispath.
Movecorruptedfilesto/lost+found
Deletecorruptedfiles.
Printoutfilesopenedforwrite.
Printoutfilesbeingchecked.
Printoutblockreport.
Printoutlocationsforeveryblock.
Printoutnetworktopologyfordatanodelocations.
fetchdt
GetsDelegationTokenfromaNameNode.Seefetchdtformoreinfo.
Usage:hadoopfetchdt[GENERIC_OPTIONS][webservice
<namenode_http_addr>]<path>
COMMAND_OPTION
<fileName>
webservice<https_address>
Description
Filenametostorethetokeninto.
usehttpprotocolinsteadofRPC
jar
Runsajarfile.UserscanbundletheirMapReducecodeinajarfileandexecuteitusingthis
command.
Usage:hadoopjar<jar>[mainClass]args...
Thestreamingjobsarerunviathiscommand.ExamplescanbereferredfromStreaming
examples
Wordcountexampleisalsorunusingjarcommand.ItcanbereferredfromWordcountexample
job
http://hadoop.apache.org/docs/stable1/commands_manual.html
3/8
6/1/2016
CommandsGuide
CommandtointeractwithMapReduceJobs.
Usage:hadoopjob[GENERIC_OPTIONS][submit<jobfile>]|[
status<jobid>]|[counter<jobid><groupname><counter
name>]|[kill<jobid>]|[events<jobid><fromevent#><#
ofevents>]|[history[all]<jobOutputDir>]|[list[all]]|
[killtask<taskid>]|[failtask<taskid>]|[setpriority
<jobid><priority>]
COMMAND_OPTION
submit<job
file>
status<jobid>
counter<job
id><groupname>
<countername>
kill<jobid>
events<jobid>
<fromevent#>
<#ofevents>
history[all]
<jobOutputDir>
list[all]
killtask
<taskid>
failtask
<taskid>
setpriority
<jobid>
<priority>
Description
Submitsthejob.
Printsthemapandreducecompletionpercentageandalljobcounters.
Printsthecountervalue.
Killsthejob.
Printstheevents'detailsreceivedbyjobtrackerforthegivenrange.
history<jobOutputDir>printsjobdetails,failedandkilledtipdetails.
Moredetailsaboutthejobsuchassuccessfultasksandtaskattempts
madeforeachtaskcanbeviewedbyspecifyingthe[all]option.
listalldisplaysalljobs.listdisplaysonlyjobswhichareyetto
complete.
Killsthetask.KilledtasksareNOTcountedagainstfailedattempts.
Failsthetask.Failedtasksarecountedagainstfailedattempts.
Changesthepriorityofthejob.Allowedpriorityvaluesare
VERY_HIGH,HIGH,NORMAL,LOW,VERY_LOW
pipes
Runsapipesjob.
Usage:hadooppipes[conf<path>][jobconf<key=value>,
<key=value>,...][input<path>][output<path>][jar<jar
file>][inputformat<class>][map<class>][partitioner
<class>][reduce<class>][writer<class>][program
<executable>][reduces<num>]
COMMAND_OPTION
conf<path>
jobconf<key=value>,<key=value>,...
input<path>
output<path>
jar<jarfile>
inputformat<class>
map<class>
partitioner<class>
http://hadoop.apache.org/docs/stable1/commands_manual.html
Description
Configurationforjob
Add/overrideconfigurationforjob
Inputdirectory
Outputdirectory
Jarfilename
InputFormatclass
JavaMapclass
JavaPartitioner
4/8
6/1/2016
CommandsGuide
reduce<class>
writer<class>
program<executable>
reduces<num>
JavaReduceclass
JavaRecordWriter
ExecutableURI
Numberofreduces
queue
commandtointeractandviewJobQueueinformation
Usage:hadoopqueue[list]|[info<jobqueuename>[
showJobs]]|[showacls]
COMMAND_OPTION
list
info<job
queuename>[
showJobs]
showacls
Description
GetslistofJobQueuesconfiguredinthesystem.Alongwithscheduling
informationassociatedwiththejobqueues.
Displaysthejobqueueinformationandassociatedscheduling
informationofparticularjobqueue.IfshowJobsoptionsispresenta
listofjobssubmittedtotheparticularjobqueueisdisplayed.
Displaysthequeuenameandassociatedqueueoperationsallowedfor
thecurrentuser.Thelistconsistsofonlythosequeuestowhichtheuser
hasaccess.
version
Printstheversion.
Usage:hadoopversion
CLASSNAME
hadoopscriptcanbeusedtoinvokeanyclass.
Usage:hadoopCLASSNAME
RunstheclassnamedCLASSNAME.
classpath
PrintstheclasspathneededtogettheHadoopjarandtherequiredlibraries.
Usage:hadoopclasspath
AdministrationCommands
Commandsusefulforadministratorsofahadoopcluster.
balancer
Runsaclusterbalancingutility.AnadministratorcansimplypressCtrlCtostoptherebalancing
process.SeeRebalancerformoredetails.
Usage:hadoopbalancer[threshold<threshold>]
http://hadoop.apache.org/docs/stable1/commands_manual.html
5/8
6/1/2016
CommandsGuide
COMMAND_OPTION
threshold
<threshold>
Description
Percentageofdiskcapacity.Thisoverwritesthedefault
threshold.
daemonlog
Get/Settheloglevelforeachdaemon.
Usage:hadoopdaemonloggetlevel<host:port><name>
Usage:hadoopdaemonlogsetlevel<host:port><name><level>
COMMAND_OPTION
Description
getlevel
Printstheloglevelofthedaemonrunningat<host:port>.This
<host:port><name> commandinternallyconnectstohttp://<host:port>/logLevel?log=
<name>
setlevel
Setstheloglevelofthedaemonrunningat<host:port>.This
<host:port><name> commandinternallyconnectstohttp://<host:port>/logLevel?log=
<name>
<level>
datanode
RunsaHDFSdatanode.
Usage:hadoopdatanode[rollback]
COMMAND_OPTION
rollback
Description
Rollsbackthedatanodetothepreviousversion.Thisshouldbeused
afterstoppingthedatanodeanddistributingtheoldhadoopversion.
dfsadmin
RunsaHDFSdfsadminclient.
Usage:hadoopdfsadmin[GENERIC_OPTIONS][report][safemode
enter|leave|get|wait][refreshNodes][finalizeUpgrade][
upgradeProgressstatus|details|force][metasavefilename][
setQuota<quota><dirname>...<dirname>][clrQuota<dirname>...
<dirname>][help[cmd]]
COMMAND_OPTION
Description
Reportsbasicfilesysteminformationandstatistics.
Safemodemaintenancecommand.SafemodeisaNamenodestatein
whichit
1.doesnotacceptchangestothenamespace(readonly)
2.doesnotreplicateordeleteblocks.
SafemodeisenteredautomaticallyatNamenodestartup,andleaves
safemodeautomaticallywhentheconfiguredminimumpercentageof
blockssatisfiestheminimumreplicationcondition.Safemodecanalso
beenteredmanually,butthenitcanonlybeturnedoffmanuallyaswell.
refreshNodes
RereadthehostsandexcludefilestoupdatethesetofDatanodesthat
areallowedtoconnecttotheNamenodeandthosethatshouldbe
decommissionedorrecommissioned.
finalizeUpgrade FinalizeupgradeofHDFS.Datanodesdeletetheirpreviousversion
workingdirectories,followedbyNamenodedoingthesame.This
report
safemodeenter
|leave|get|
wait
http://hadoop.apache.org/docs/stable1/commands_manual.html
6/8
6/1/2016
upgradeProgress
status|details
|force
metasave
filename
setQuota
<quota>
<dirname>...
<dirname>
clrQuota
<dirname>...
<dirname>
help[cmd]
CommandsGuide
completestheupgradeprocess.
Requestcurrentdistributedupgradestatus,adetailedstatusorforce
theupgradetoproceed.
SaveNamenode'sprimarydatastructuresto<filename>inthedirectory
specifiedbyhadoop.log.dirproperty.<filename>willcontainoneline
foreachofthefollowing
1.DatanodesheartbeatingwithNamenode
2.Blockswaitingtobereplicated
3.Blockscurrrentlybeingreplicated
4.Blockswaitingtobedeleted
Setthequota<quota>foreachdirectory<dirname>.Thedirectory
quotaisalongintegerthatputsahardlimitonthenumberofnamesin
thedirectorytree.
Besteffortforthedirectory,withfaultsreportedif
1.Nisnotapositiveinteger,or
2.userisnotanadministrator,or
3.thedirectorydoesnotexistorisafile,or
4.thedirectorywouldimmediatelyexceedthenewquota.
Clearthequotaforeachdirectory<dirname>.
Besteffortforthedirectory.withfaultreportedif
1.thedirectorydoesnotexistorisafile,or
2.userisnotanadministrator.
Itdoesnotfaultifthedirectoryhasnoquota.
Displayshelpforthegivencommandorallcommandsifnoneis
specified.
mradmin
RunsMRadminclient
Usage:hadoopmradmin[GENERIC_OPTIONS][refreshQueueAcls]
COMMAND_OPTION
Description
Refreshthequeueaclsusedbyhadoop,tocheckaccessduring
refreshQueueAcls submissionsandadministrationofthejobbytheuser.Theproperties
presentinmapredqueueacls.xmlisreloadedbythequeue
manager.
jobtracker
RunstheMapReducejobTrackernode.
Usage:hadoopjobtracker[dumpConfiguration]
COMMAND_OPTION
Description
DumpstheconfigurationusedbytheJobTrackeralongwithqueue
dumpConfiguration configurationinJSONformatintoStandardoutputusedbythe
jobtrackerandexits.
namenode
Runsthenamenode.Moreinfoabouttheupgrade,rollbackandfinalizeisatUpgradeRollback
Usage:hadoopnamenode[format[force][nonInteractive]]|[
upgrade]|[rollback]|[finalize]|[importCheckpoint]
http://hadoop.apache.org/docs/stable1/commands_manual.html
7/8
6/1/2016
CommandsGuide
upgrade]|[rollback]|[finalize]|[importCheckpoint]
COMMAND_OPTION
Description
format[force] Formatsthenamenode.Itstartsthenamenode,formatsitandthen
shutsitdown.Userwillbepromptedforinputifthenamedirectories
[
nonInteractive] existonthelocalfilesystem.
nonInteractive:Userwillnotbepromptedforinputifthename
directoriesexistinthelocalfilesystemandtheformatwillfail.
force:formatsthenamenodeandtheuserwillNOTbepromptedto
confirmformattingofnamedirectoriesinthelocalfilesystem.If
nonInteractiveoptionisspecifieditwillbeignored.
upgrade
Namenodeshouldbestartedwithupgradeoptionafterthedistribution
ofnewhadoopversion.
rollback
Rollsbackthenamenodetothepreviousversion.Thisshouldbeused
afterstoppingtheclusteranddistributingtheoldhadoopversion.
finalize
Finalizewillremovethepreviousstateofthefilessystem.Recent
upgradewillbecomepermanent.Rollbackoptionwillnotbeavailable
anymore.Afterfinalizationitshutsthenamenodedown.
Loadsimagefromacheckpointdirectoryandsaveitintothecurrent
importCheckpoint one.Checkpointdirisreadfrompropertyfs.checkpoint.dir
secondarynamenode
RunstheHDFSsecondarynamenode.SeeSecondaryNamenodeformoreinfo.
Usage:hadoopsecondarynamenode[checkpoint[force]]|[
geteditsize]
COMMAND_OPTION
checkpoint
[force]
geteditsize
Description
CheckpointstheSecondarynamenodeifEditLogsize>=
fs.checkpoint.size.Ifforceisused,checkpointirrespectiveofEditLog
size.
PrintstheEditLogsize.
tasktracker
RunsaMapReducetaskTrackernode.
Usage:hadooptasktracker
LastPublished:08/04/201309:13:21
Copyright2008TheApacheSoftwareFoundation.
http://hadoop.apache.org/docs/stable1/commands_manual.html
8/8