Blog

Controlling Hadoop Jobs using Oozie

  • (4.0)
  •   |   561 Ratings

OOZIE:

 OOZIE is a server based work flow engine specialized in running work flow jobs with actions that run HADOOP MAPREDUCE and PIG jobs.

OOZIE is a Java web–application that runs in a Java servlet container.

For the purposes Of oozie, a work flow is a collection of action i.e hadoop mapreduce jobs, pig jobs are arranged in a control dependency DAG (Direct Acyclic Graph)

Control dependency from one action to another means that the second action can’t run until the first action has completed.

Oozie work flow actions start jobs in remote systems

Upon action completion, the remote systems call back oozie to notify the action completion, at this point, oozie proceeds to the next action in the work flow.

Oozie work flows contain control flow nodes and action nodes

Enthusiastic about exploring the skill set of Hadoop? Then, have a look at the HADOOP TRAINING COURSE together additional knowledge.

Control flow nodes define the beginning and the end of a work flow (start, end and fail nodes) and provide a mechanism to control the work flow execution path (decision, fork and join nodes)

Action nodes are the mechanism by which a work flow triggers the execution of computation or processing task.

Oozie provides support for different types of actions such as Hadoop map-reduce, Hadoop file system, pig, SSH, HTIP, email and oozie sub-work flow.

Oozie work flows can be parameterized using variables like (input dir) within the work flow definition.

When submitting a work flow job values, the parameters must be provided

Work flow Diagram(Word count work flow example):-

 

Installation and configuring oozie:-

To install and run oozie using an embedded to meat server and an embedded Derby database

System Requirements:-

Unix(tested in linux and Max osk)

Java 1.6 +

Hadoop 0.20 and 1.0.0

Ext Js Library (optional, to enable oozie web console)

                 ExtJS 2.2

Note:-Java 1.6+bin directory should be in the command path

Server Installation:-

oozie ignores any set value for oozie- HOME, and … oozie computes its home automatically

Download or build an oozie binary distribution

Download a hadoop binary distribution

Download ExtJS library[version 2.2]

Note:–

ExtJS library is not bundled with oozie, because it uses a different license and recommended to use a oozie Unix user for one oozie server.

Frequently asked Hadoop Interview Questions

Expand the oozie distribution tar.gz.

Expand the hadoop distribution far.gz(as the oozie Unix user)

Oozie is bundled without hadoop JAR file and ExtJS library

Hadoop JARs are requited to run oozie.

The ExtJS library is only  required for the oozie web –console to work

All oozie server scripts i.e.

oozie serup.sh

oozie start.sh                        all run only under the

oozie run.sh                           Unix user that owns the

oozie stop.sh                          oozie installation directory

Use the oozie- setup.sh script to add the Hadoop JARS and ExtJS Library to oozie.

     $bin/oozie – setup.sh-hadoop 0.20.200

                ${ hadoop-Home}-ExtJS hmp/ext-2.2.zip

To start oozie as a daemon process run the command as

$bin/oozie - start.sh

To start oozie as a foreground process, run the command as

$bin/oozie - run.sh

Check the oozie.log file logs/ oozie.log to ensure. Oozie. started properly.

To Check the status of oozie using the Oozie command like rool is

$bin/oozie admin- oozie HTTP://LOCALhost:11000/ oozie-status

oozie status should be normal when using the oozie web console

Related Article: Hadoop Installation

Client Installation:-

Copy and expand the oozie- client TAR.GZ bundled with the distribution.

5 Add the bin/direction to the PATH

Note: The oozie server installation includes the oozie client which should be installed in remote machines only.

Oozie share lib installation:-

Expand the oozie- sharelib TAR.COZ .file bundled with the distribution

Shared directory must be copied to the oozie HOME directory in HDFS:

$hadoop fs-put share

Note:

This must be done using the oozie hadoop (HDFS) and if shared directory already exists in HDRS, it must be deleted before copying  again.

Explore Hadoop Sample Resumes! Download & Edit, Get Noticed by Top Employers!Download Now!

List of Other Big Data Courses:

 Hadoop Adminstartion  MapReduce
 Big Data On AWS  Informatica Big Data Integration
 Bigdata Greenplum DBA  Informatica Big Data Edition
 Hadoop Hive  Impala
 Hadoop Testing  Apache Mahout

 


Popular Courses in 2018

Get Updates on Tech posts, Interview & Certification questions and training schedules