JeeraTechno
HDFS COMMANDS
1. copyFromLocal(Copy a file or directory from Local to HDFS)
hadoop fs –copyFromLocal /home/training/dvs/f1.txt dvs_hdfs
Note: Can be used for copying multiple files, similar pattern
files, all the files, a directory
2. moveFromLocal(move a file or directory from Local to HDFS)
Hadoop fs –moveFromLocal /home/training/dvs/f1.txt
dvs_hdfs
3. copyToLocal(Copy a file or directory from HDFS to Local)
hadoop fs –copyToLocal dvs_hdfs/f1.txt /home/training/dvs
4. moveToLocal(Not yet implemented)
5. cp (copy a file from one location to another location inside
HDFS)
hadoop fs –cp dvs_hdfs/file1 dvs_hdfs1
6. mv (move a file from one location to another location inside
HDFS)
hadoop fs –mv dvs_hdfs/file1dvs_hdfs1
7. put (Similar to copyFromLocal)
hadoopfs –put /home/training/dvs/file1dvs_hdfs
8. get (Similar to copyToLocal)
hadoopfs –get dvs_hdfs/file1 /home/training/dvs
2nd Floor, Alpha Block, Sigma SoftTech Park, Varthur Hobli, Bangalore, Karnataka, India 560066.
Facebook : /Jeeratechno | Instagram : /Jeeratechno | Mob: +91 90141 28319,
JeeraTechno
9. getmerge (writes multiple file contents in to a single file in
Local File system)
hadoop fs –getmerge dvs_hdfs/file1 dvs_hdfs/file2
/home/training/dvs/f3
Note : This command is not supported in this version of
Hadoop.
10. mkdir (Create a directory)
hadoop fs –mkdir dvs_hdfs
11. touchz ( can create n no: of empty files in HDFS)
hadoop fs –touchz dvs_hdfs/file1
12. rm (Remove a file)
hadoop fs –rm dvs_hdfs/file1
13. rmr (Can be used for removing a file or Directory
recursively)
hadoop fs –rmr dvs_hdfs/file
hadoop fs –rmr dvs_hdfs/Dir1
Note: Can be used to remove similar pattern files(*.sh, *.txt
etc), all the files(*)
14. ls (Lists all the files & directories)
hadoop fs –ls dvs_hdfs
ls|tail –n (Tail option with List)
hadoop fs –ls dvs_hdfs|tail -10
15. ls|head –n (head option with List)
2nd Floor, Alpha Block, Sigma SoftTech Park, Varthur Hobli, Bangalore, Karnataka, India 560066.
Facebook : /Jeeratechno | Instagram : /Jeeratechno | Mob: +91 90141 28319,
JeeraTechno
hadoop fs –ls dvs_hdfs|head -10
16. cat (Displays the content of a file)
hadoop fs -cat dvs_hdfs/file
17. text(Displays the content of zipped files)
hadoop fs -text dvs_hdfs/file.gz
18. cat|tail –n (Display bottom n lines of a file)
hadoop fs -cat dvs_hdfs/file|tail 10
19. cat|head–n (Display top n lines of a file)
hadoop fs -cat dvs_hdfs/file|head 10
20. cat|wc –l (Counts the no:of lines in a file)
hadoop fs -cat dvs_hdfs/file1|wc –l
21. cat|wc –w (Counts the no:of words in a file)
hadoop fs –catdvs_hdfs/file1|wc –w
22. cat|wc –c (Counts the no:of Characters in a file)
hadoop fs -cat dvs_hdfs/file1|wc –c
23. du (Disk Usage of a file or directory)
hadoop fs –du dvs_hdfs/file1.txt
24. du –h (formats & shows file or directory size in human
readable format)
hadoop fs –dur -hdvs_hdfs
2nd Floor, Alpha Block, Sigma SoftTech Park, Varthur Hobli, Bangalore, Karnataka, India 560066.
Facebook : /Jeeratechno | Instagram : /Jeeratechno | Mob: +91 90141 28319,
JeeraTechno
25. du –s(shows summary of the directories instead of each
file)
hadoop fs –du –s dvs_hdfs
26. df (Disk usage of the entire file system)
hadoop fs –df
O/P:
Filesystem Size Used Available Use%
hdfs://nameservice1 328040332591104 102783556870823
210750795833344 31%
27. df –h (Formats & shows in the human readable format)
hadoop fs -df –h
O/P:
Filesystem Size Used Available Use%
hdfs://nameservice1 298.4 T 93.5 T 191.7 T 31%
28. count(Counts all the Directories & Files in the given path)
hadoop fs –count dvs_hdfs
29. fsck(To check file system health)
Hadoop fsck dvs_hdfs
30. fsck –files –blocks (Displays corresponding Files& their
block level info)
Hadoop fsck dvs_hdfs/file1.txt –files -blocks
2nd Floor, Alpha Block, Sigma SoftTech Park, Varthur Hobli, Bangalore, Karnataka, India 560066.
Facebook : /Jeeratechno | Instagram : /Jeeratechno | Mob: +91 90141 28319,
JeeraTechno
31. fsck –files –blocks –locations (Displays files& block level
info including the block location)
32. hadoopfsckdvs_hdfs/f1.txt –files –blocks –locations -racks
33. setrep(used to change the replication factor a file or a
directory)
hadoop fs –setrep 5 dvs_hdfs/file1
Hadoop fs –setrep 5 –w dvs_hdfs/ABC
-w It requests that the command waits for the replication to
complete. This
can potentially take a very long time.
34. Controlling block size at file level without changing the
block size in hdfs-site.xml
Hadoop fs –D dfs.block.size=134217728 –put
source_pathdestination_path
35. Controlling replication at file level irrespective of the
default replication set to 3
Hadoop fs –D dfs.replication=2 –put
source_pathdestination_path
36. Setting replication factor for a directory in HDFS
Hadoop fs –setrep 5 –R dvs_hdfs/ABC
Note: All the files copied under this directory will be having a
replication factor of 5 irrespective of the default replication set.
37. Safe Mode
Hadoop dfsadmin –safemode leave
Hadoop dfsadmin –safemode enter
2nd Floor, Alpha Block, Sigma SoftTech Park, Varthur Hobli, Bangalore, Karnataka, India 560066.
Facebook : /Jeeratechno | Instagram : /Jeeratechno | Mob: +91 90141 28319,
JeeraTechno
Hadoop dfsadmin –safemode get
38. Delete all the files in trash
hadoop fs -expunge
39. Copying a file from one cluster to another cluster
hadoop fs -distcp hdfs://namenodeA/emp.csv
hdfs://namenodeB/
40.
2nd Floor, Alpha Block, Sigma SoftTech Park, Varthur Hobli, Bangalore, Karnataka, India 560066.
Facebook : /Jeeratechno | Instagram : /Jeeratechno | Mob: +91 90141 28319,
