Hadoop Apache Pig Execution Types

Execution Types

Pig has two execution types or run modes, local and hadoop (currently called mapreduce)

  1. Local mode
  2. Map Reduce mode.

1) Local mode

Capture 15 In Local mode, pig runs in a single JVM and accesses the local file system.

Capture 15 This mode is suitable only for small data sets and typing out pig.

Capture 15 To run in local mode, set the option to local i.e

%pig-x local

2) Map Reduce Mode

Capture 15 In Map Reduce mode, pig translates queries into map reduce jobs and runs them on a hadoop cluster and cluster may be pseudo or fully distributed cluster.

Capture 15 To use Map Reduce mode, we have to check the version of pig that is compatible with the versions of hadoop

Capture 15 Pig honors the HADOOP-HOME environmental variable for finding which Hadoop client to run and pig will also use a bundled copy of the hadoop libraries.

Capture 15 For pig, we have to set two properties in pig properties file which is pig’s cong directory.

i.e for psedudo distributed setup, example is

mapped. job. tracker=local host:8021

Capture 15 Once you have configured pig to connect to Hadoop cluster, you can launch pig, setting the  -x option to map reduce or a map reduce mode is default i.e. % pig grunt>

Running Pig Programs

Capture 15 This section shows you how to run Pig in local mode, using the Grunt shell, a Pig script, and an embedded program.

Capture 15 There are three ways of executing pig programs which work in both local and map reduce mode.

  1. Script-pig can run a script file that contains pig commands
  2. Grunt – Is an interactive shell for running pig commands
  3. Embedded – you can run pig programs from Java using the pig server class, much like you can use JDBC to run SQL Programs from JAVA.

Modes of Execution in Pig

You can execute Pig Latin statements.

  1. Grunt shell or command line
  2. Local mode or Map Reduce mode
  3. Either Interactively or in a batch

Local mode and MR or HDFS Mode

     In the local mode execution of pig, we expect all the inputs from the local environment (input from local mode and produce the output from the local mode but should not be from HDFS)

Syntax:- Pig-x local.

      In MR mode execution of pig, we expect all input files from HDFS path and also produce the output on top of hdfs only

Syntax:- Pig(or)

Grunt shell:

Capture 15 The grunt mode can also be called as interactive mode. Grunt is pig’s interactive shell. It is started when no file is specified for pig to run.

Capture 15 Grunt shell is an interactive base shell where you will expect the o/p then and there only, irrespective of the input.

Script mode Execution:

Capture 15 In script mode, pig runs the commands specified in a script file.

 Capture 15 Here, we will describe all transformations in a single file which ends with pig. All the commands will be executed one after another which are there in .pig file.

Local Map Reduce
Pig- x local abc.pig Pig abc.pig

Embedded mode:

Capture 15 In this mode, we will do the pig customization code if at all sane analyzer functionality is not archived through the user defined transformations.

Capture 15 Then we can write Java code to archive the same and reference the same Java code (.jar file) in our pig script by making use of the code.

“Register xyz.jar”

Note:- Register key word should be the first statement within our pig script code.

0 Responses on Hadoop Apache Pig Execution Types"

Leave a Message

Your email address will not be published. Required fields are marked *

Copy Rights Reserved © Mindmajix.com All rights reserved. Disclaimer.