Blog

Hey Guys!!! In this post, I’m going to walk you through some important HDFS shell commands which can be used to manage files present in Hadoop distributed file system. These command are also important if you are planning to take CCA-175 certification exam. Happy Learning 🙂

Check hadoop version.

Usage  : hdfs version 
Example: hdfs version

Check the health of the Hadoop file system.

Usage  : hdfs fsck <path> 
Example: hdfs fsck /user/user1001/    (Your user directory)
Example: hdfs fsck /                  (If you have access to root) 

List the contents of a directory.

Usage  : hdfs dfs -ls <path> 
Example: hdfs dfs -ls /user/user1001/retail_db

Create a new directory.

Usage  : hdfs dfs -mkdir <path/directory_name> 
Example: hdfs dfs -mkdir /user/user1001/test_dir
Here we are creating a directory named test_dir.

Delete a directory.

Below commands can be used to delete a directory in HDFS.

Usage  : hdfs dfs -rm -r <directory_path> 
Example: hdfs dfs -rm -r /user/user1001/test_dir
Here we are deleting directory named test_dir. This command will also delete all the sub-directories and data present under test_dir.

Usage  : hdfs dfs -rm  <directory_path> 
Example: hdfs dfs -rm  /user/user1001/test_dir
This command deletes a directory only when it is empty.

Usage  : hdfs dfs -rmdir <directory_path> 
Example: hdfs dfs -rmdir /user/user1001/test_dir
This command deletes a directory only when it is empty. 

Uploading a file to HDFS.

-put command can be used to copy single file, or multiple files from local file system to the HDFS.

Usage  : hdfs dfs -put <src_path> <destination_path> 
Example: hdfs dfs -put /home/data/log.txt /user/user1001/test_dir/
Here we are copying the file log.txt from /home/data/log.txt location to  /user/user1001/test_dir/ location.

-copyFromLocal command is similar to put command, except that the source is restricted to a local file reference.

Usage  : hdfs dfs -copyFromLocal <src_path> <destination_path> 
Example: hdfs dfs -copyFromLocal /home/data/log.txt /user/user1001/test_dir/
Here we are copying the file log.txt from /home/data/log.txt location to  /user/user1001/test_dir/ location.

-cp command allows multiple sources.

Usage  : hdfs dfs -cp <src_path> <destination_path> 
Example: hdfs dfs -cp /home/data/log1.txt /home/data/log2.txt /user/user1001/test_dir/
Here we are copying multiple files log1.txt and log2.txt from /home/data/ location to /user/user1001/test_dir/ location.

Download a file from HDFS.

-get command copies files to the local file system.

Usage  : hdfs dfs -get <hdfs_path> <local_path>
Example: hdfs dfs -get /user/user1001/test_dir/log.txt /home/
Here we are fetching a file named log.txt from HDFS location to local file system.

-copyToLocal command copies files to the local file system. Similar to -get command, except that the destination is restricted to a local file reference.

Usage  : hdfs dfs -copyToLocal <hdfs_path> <local_path> 
Example: hdfs dfs -copyToLocal  /user/user1001/test_dir/log.txt /home/ 

Move file from source to destination.

-mv command moves a file from one HDFS location to another. Moving files across file system is not permitted.

Usage  : hdfs dfs -mv <hdfs_path1> < hdfs_path1>
Example: hdfs dfs -mv /user/user1001/test_dir/log.txt  /user/user1001/test_dir1/

See contents of a file.

-cat command can be used to see contents of a file. It is same as unix cat command.

Usage  : hdfs dfs -cat <hdfs_path>
Example: hdfs dfs -cat  /user/user1001/test_dir/log.txt

Display last few lines of a file.

-tail command can be used to see last few lines of a file. Its is same as unix tail command.

Usage  : hdfs dfs -tail <hdfs_path>
Example: hdfs dfs -tail /user/user1001/test_dir/log.txt

Display the aggregate length of a file.

-du command can be used to see last few lines of a file. Its is same as unix tail command.

Usage  : hdfs dfs -du <hdfs_path>
Example: hdfs dfs -du /user/user1001/test_dir/log.txt