Hadoop 1.x and 2.x HDFS Basic Developer Commands, “hadoop” command

Filed Under: Big Data

In this post, we are going to discuss about Hadoop 1.x and Hadoop 2.x HDFS Basic commands with simple, suitable and real-time examples. These are useful for BigData Hadoop Developers. We will discuss Hadoop HDFS Admin Commands in my coming posts.

NOTE:- Both Hadoop 1.x and Hadoop 2.x uses almost same HDFS commands. We will mention if there are any changes from Hadoop 1.x to Hadoop 2.x.

Introduction

In Hadoop, we do the following operations on HDFS (Hadoop Distributed FileSystem) component.

  • We can create or remove folders
  • We copy files form external system to HDFS Filesystem
  • We can delete existing HDFS files
  • We can view existing HDFS Filesystem’s file content

We are going to explore all these operations in this post. Before starting discussions about HDFS commands, first we need to understand about “hadoop” command usage.

Hadoop “hadoop” Command

In Hadoop HDFS component, we can use “hadoop” command to execute all HDFS commands. “hadoop” command is implemented as a Script available at ${HADOOP_home}\bin path.

“hadoop” command Syntax:-


    hadoop [commands] [commands-options]

We are going to discuss this command in detail with examples in next section.

Hadoop HDFS Basic Commands

We use “hadoop fs” command to perform all Hadoop HDFS component operations. Here “fs” stands for FileSytem.

“hadoop fs” command syntax:-


    hadoop fs [command-options]

NOTE:-
Almost most of the “hadoop fs” command options are similar to Unix/Linux commands. If you are familiar with Unix/Linux commands, then it is very easy to use “hadoop fs” command. Now, We are going discuss this command to perform all operations mentioned in “Introduction” section. We need to use Terminal to execute all these commands in our CloudEra Environment VMWare setup.

To create a new directory in Hadoop HDFS FileSystem.


We use “hadoop fs -mkdir” option to create a new directory in the HDFS FileSystem.

“hadoop fs -mkdir” Syntax:-


    hadoop fs -mkdir [command-options]

Example-1:- We are going to create this directory structure: “/ram/mrv1/output/”
First, We need to create create “ram” directory, then “mrv1” then “output” as shown below.


    hadoop fs -mkdir /ram
    hadoop fs -mkdir /ram/mrv1
    hadoop fs -mkdir /ram/mrv1/output

hdfs-mkdir

To list the content of a directory in Hadoop HDFS FileSystem.

We use “hadoop fs -ls” option list the content of a directory in the HDFS FileSystem.
“hadoop fs -fs” Syntax:-


    hadoop fs -ls [dir-name]

Example-1:- We are going to list the content of a directory “/ram”


    hadoop fs -ls /ram

Please observe above image for this example output.

To copy a File from Local FileSystem to Hadoop HDFS FileSystem.

We use “hadoop fs -put” option to copy a File from External or Local FileSystem to Hadoop HDFS FileSystem. Here External or Local means outside Hadoop HDFS FileSystem.

“hadoop fs -put” command Syntax:-


    hadoop fs -put [fully-qualified-source-filename] [Hadoop-HDFS-Location-filename]

Example-1:- We are going to copy “NASDAQ_daily_prices_C.csv” from “/home/cloudera/workspace/” to Hadoop HDFS “/ram/mrv1/output” folder.

Please follow the following steps to perform this operation:

  • First copy “NASDAQ_daily_prices_C.csv” from our Windows or your OS to ClodEra VMWare Environment.
  • Here, we need to use same steps like Windows OS. Select “NASDAQ_daily_prices_C.csv” file with Mouse and select “Copy” option from Mouse-Menu, then move mouse pointer “/home/cloudera/workspace/” in CloudEra Environment, use same Mouse-menu and select “Paste” option.

    hdfs-local-filesystem-file

  • Now our “NASDAQ_daily_prices_C.csv” is available at “/home/cloudera/workspace/” in CloudEra Environment.
  • Next step is we need to move this file from Local FileSystem to HDFS FiseSystem by using “hadoop fs -put” command.
  • 
        hadoop fs -ls /ram/mrv1/output
        hadoop fs -put /home/cloudera/workspace/NASDAQ_daily_prices_C.csv /ram/mrv1/output
    

    First execute “hadoop fs -ls” command to know whether that directory contains this file or not. Then execute “hadoop fs -put” command to move file into HDFS FileSystem as shown below:

    hdfs-put-copy-file

To copy a File from Local FileSystem to Hadoop HDFS FileSystem.

We can also use “hadoop fs -copyFromLocal” option to copy a File from External or Local FileSystem to Hadoop HDFS FileSystem. Here External or Local means outside Hadoop HDFS FileSystem.

“hadoop fs -copyFromLocal” command Syntax:-


    hadoop fs -copyFromLocal[fully-qualified-source-filename] [Hadoop-HDFS-Location-filename]

NOTE:-
In Hadoop HDFS, Both “hadoop fs -put” and “hadoop fs -copyFromLocal” commands perform same operation. It’s upto our choice to choose one in our application.

To copy a File from Hadoop HDFS FileSystem to Local FileSystem.

We use “hadoop fs -copyToLocal” option to copy a File from Hadoop HDFS FileSystem to External or Local FileSystem.

“hadoop fs -copyToLocal” command Syntax:-


    hadoop fs -copyToLocal  [Hadoop-HDFS-Location-filename] [fully-qualified-source-filename] 

Example-1:- We are going to copy “NASDAQ_daily_prices_C.csv” from Hadoop HDFS “/ram/mrv1/output” to “/home/cloudera/workspace/” available in Local FileSystem.

Please follow the following command to perform this operation:


ls -lrt /home/cloudera/workspace/

Here we don’t find “NASDAQ_daily_prices_C.csv” file.


    hadoop fs -copyToLocal /ram/mrv1/output/NASDAQ_daily_prices_C.csv /home/cloudera/workspace/ 

This command copies “NASDAQ_daily_prices_C.csv” file from Hadoop HDFS “/ram/mrv1/output” to “/home/cloudera/workspace/” available in Local FileSystem.

NOTE:-

  • In Hadoop HDFS, “hadoop fs -put” Or “hadoop fs -copyFromLocal” commands are quite opposite to “hadoop fs -copyToLocal” command.
  • “hadoop fs -put” Or “hadoop fs -copyFromLocal” commands are used to copy data from Local FileSystem to Hadoop HDFS FileSytem.
  • “hadoop fs -copyToLocal” command id used to copy data from Hadoop HDFS FileSytem to Local FileSystem .

To view the content of File available in Hadoop HDFS FileSystem.

We use “hadoop fs -cat” command to view the content of File available in Hadoop HDFS FileSystem.

“hadoop fs -cat” command Syntax:-


    hadoop fs -cat [fully-qualified-file-path]

Example:-


    hadoop fs -cat /ram/mrv1/output/server.log

To remove a File available in Hadoop HDFS FileSystem.

We use “hadoop fs -rm” command to remove a File available in Hadoop HDFS FileSystem.

“hadoop fs -rm” command Syntax:-


    hadoop fs -rm [file-name]

Example:-


    hadoop fs -rm /ram/mrv1/output/NASDAQ_daily_prices_C.csv

To remove a Directory available in Hadoop HDFS FileSystem.

We use “hadoop fs -rmdir” command to remove a Directory available in Hadoop HDFS FileSystem.

“hadoop fs -rmdir” command Syntax:-


    hadoop fs -rmdir [dir-name]

If that directory is not empty, we will get error like “Directory is not empty”. In that case use “hadoop fs -rm [file-name]” name to remove files then remove that directory.

Example:-


    hadoop fs -rmdir /ram/mrv1/output

Hadoop HDFS Developer Commands List

Here we are going to list out all Hadoop Developer related HDFS commands and their usage in a tabular format for quick reference purpose.

Hadoop HDFS Command Command Usage
hadoop fs -ls [hdfs-path] To list the content of a directory in HDFS.
hadoop fs -mkdir [dir-name] To create a directory in HDFS.
hadoop fs -rmdir [dir-name] To remove a directory in HDFS.
hadoop fs -put [source-path] [dest-hdfs-path] To copy a file from Local FileSystem to HDFS FileSystem.
hadoop fs -copyFromLocal [source-path] [dest-hdfs-path] To copy a file from Local FileSystem to HDFS FileSystem.
hadoop fs -copyToLocal [source-path] [dest-hdfs-path] To copy a file from HDFS FileSystem to Local FileSystem.
hadoop fs -cat [filename] To view the content of a file available in HDFS FileSystem.

That’s it all about Hadoop 1.x or 2.x HDFS Basic Developer Commands with suitable Examples. We will use these commands in my coming posts.

We will discuss Hadoop HDFS Admin Commands in my coming posts.

Please drop me a comment if you like my post or have any issues/suggestions.

Comments

  1. Adi says:

    Thanks Very useful .please share some hadoop admin commands

  2. rajendra babu says:

    all the posts on big data are very good. Give some explanation on slots, containers and partitioning. i am not found any info about these concepts in your sections. if anything available on those topics please let me know. thankk you.

  3. bala says:

    very good tutorial sir thank u

Leave a Reply

Your email address will not be published. Required fields are marked *

close
Generic selectors
Exact matches only
Search in title
Search in content
Search in posts
Search in pages