Using Counters in Hadoop MapReduce API with Example

Counters in Hadoop MapReduce

 MAP REDUCE COUNTER provides a way to measure the progress or the number of operations that occur within MAP REDUCE programs.

  Advantages of counters are :

While Executing a task, if any one of them fails because of bad records with binary data then the whole job also fails.

At that point, we need to avoid the bad records, i.e, skip out the bad records using counters.

For counters, we have to configure some parameters within the mapper class to check for statistics.

At that time of running, parameters are used to find out which are the bad records.

Here we have built-in counters parameters.

• Max no of records in mapper I/P
• Max no of records in mapper O/P

We can have user-defined counters and we will use them as getting counters() which have to be mentioned in the driver code.

Bad records will be stored in temp/log files.

Simple Example for map reduce:

Part-m-00000 Map file

Part-r-00000 Reduce file

                       

Mapper Output:

Internally, it will sort and shuffle and give the desired output.

Mapper output                                  Sort and shuffle output

 

                                    (1- No of occurrences)

                                                 

< hadoop,1>                                     < hadoop,1,1,1,1>

                                               

< hadoop,1>                                         < good,1,1,1,1>

                                                           

Ι                                                                        Ι

 

Ι                                                                         Ι

 

Ι                                                                         Ι                

 

                                                   

  Reducer Output is                                  

Sum=0
Obj. hasmore elements()
{	< hadoop,4>
Value. get values()
Sum=sum+ value;
}	….
Context. Write(word, sum)	….

Map Reduce Programming

  The Driver code generally falls under the main() method of the map-reduce program.

As we are dealing with the below information in our driver code.

• The final output(key, value)details

                                                The data type will be changed.

                                                  Data type

• Input and output related information of HDFS (the input path from where the map-reduce program will expect the input file of HDFS and the output path of HDFS where exactly map-reduce will produce the results.

The program itself will create the HDFS output path after the successful execution of the program.

Any Mapper class should extend the base class of the mapper in order to provide the mapper business logic.

It will take 4 arguments and the first two correspond to mapper input (key, value)

 The last two-argument corresponds to mapper output(key, value)

 In any mapper class, we have to override a method called a Map.

How to add the External (dependent jars)

Right-click on Project name Select Build path configure build path Screenshot_1806 go to libraries tab click on add external Jab select the appropriate jar files click on ok.

How to Export the jar

Right click on project name click on Export go to Java select JAR file Click on Next browse the path where you wish to place the jar file save Screenshot_1806 finish

Execution commands:-

                               To create Directory

                               Copying the input file to HDFS

                                  To check the output

List of Big Data Courses: