How to Write Hive UDF (User-Defined Functions)

Hive UDF

Capture 15 Sometimes the query you want to write can’t be expressed easily using the built–in functions that Hive provides.

Capture 15By writing UDF (User Defined function) hive makes it easy to plug in your own processing code and invoke it from a Hive query.

Capture 15 UDF’s have to be writhen in Java, the Language that Hive itself is written in.

There are three types of  UDF’s in Hive

  1. UDF’s (regular)
  2. UDF’s (user defined Aggregate Functions)
  3. UDF’s (user defined table – generating Functions)

Capture 15 They differ in the number of rows in which they accept input and produces output.

1) UDF Operates on a single row and produces a single row as its output has most of the functions, such as mathematical functions.

2) UDAF’S:-

Capture 15 UDAF works on multiple input rows and creates a single output row and aggregate functions which include functions such as count and MAX.

  • A UDTF:-Operates on a single row and produces multiple rows- a table- as output.

Capture 15 Table–generating function are less well known than the other two types.

Ex:- Consider a table with a single column x which contains arrays of strings.

ITEMS By’\002’;

Capture 15 After running a LOAD DATA Command, the following query confirms that the data was loaded correctly:

hive>SELECT * FROM arrays;

[“a”, ”b”]

[“c”, ”d” ,“e”]

Capture 15 Next, we can use the explode UDTF to transform this table

Capture 15 This function emits a row for each entry in the array.

Capture 15 So, in this case the type of the output column y is STRING.

Capture 15 The result is that the table is flattened into five rows:

Hive>SELECT explode(x)As y from arrays;

Capture 15 SELECT Statements using UDTFs have some restrictions such as not being able to retrieve additional column expressions.

Writing a UDF:-

Capture 15We can write a simple UDF by using characters from the ends of strings.

Capture 15 Hive already has a built- in function called, so we can call the strip

Capture 15 The code for the strip Java class is shown as below for stripping characters from the ends of strings

Package com. hadoop book .hive;
Import . org . apache. Common. Long. String URLS;
Import . org . apache. hadoop. Hive. ql. exec UDF;
Import . org . apache. hadoop. Io .text;
Public class strip extends UDF
Private Text result = new text();
Public. Text. evaluate(Text str)
Return null;
Result. set(string utils. Strip(str. To string()));
Return result:
Public. Text. evaluate(Text str, string strip chers)
Return null;
result. set(string utils. Strip(str. To string(),strip chars));
Return result;

Capture 15 A UDF must satisfy the following two properties:

  1. A UDF must be a sub class of org. apache. Hadoop. Hive ql. exec. UDF
  2. A UDF must implement at least one evaluate() method.

Capture 15The strip class has two evaluate() methods. Which are not defined by an interface

Capture 15 The first strips leading and trailing white space from the input while the second strip has set of supplied characters from the ends of the string.

Capture 15 To use MB UDF in Hive, Run as JAVA Application and register the file with Hive:

hive>ADD JAR/path/to/Hive-examples.jar;

Capture 15 We also need to create an alias for the java class name:

Hive)CREATE TEMPORARY FUNCTION strip As ‘com-hadoop book. Hive. strip.;

Capture 15 To call ADD JAR, you can specify at launch time a path where Hive looks for auxiliary JAR files to put on its class path.

Capture 15 This technique is used for automatically adding your own library of UDFs for every time you hive.

Capture 15 There are two ways of specifying the path either by passing the –aux path option to the hive command as below:

%hive—aux path/path/to/Hive-examples jar

or by setting the HIVE-AUX-JARS-PATH environment variable before involving Hive.

Capture 15 The UDF is now ready to be used, just like a built-in function:



hive>SELECT strip(‘banana’, ’ab’)FROM dummy;

Output is: non


0 Responses on How to Write Hive UDF (User-Defined Functions)"

Leave a Message

Your email address will not be published. Required fields are marked *

Copy Rights Reserved © Mindmajix.com All rights reserved. Disclaimer.