CSC 3211

BIG DATA ANALYTICS

A STUDY WAS CONDUCTED ON

USER DEFINED FUNCTION

Submit by :
• MOHAMED HILMI.M
200071601067
• MOHAMED JAFEER.M
200071601069
• B.TECH CSE B
 USER DEFINED FUNCTION

A user-defined function (UDF) refers to a function created by the user or

developer to perform custom operations on large datasets. UDFs are commonly
used in distributed computing frameworks like Apache Hadoop or Apache Spark
to process and transform data at scale. These functions allow users to extend the
functionality of the existing frameworks by defining their own logic and applying
it to the data processing pipelines.

Here's a general overview of how user-defined functions can be used in big data:

1. Data Transformation: UDFs can be used to transform or manipulate data

according to specific requirements. For example, you can define a UDF to extract
specific information from a text field, convert data types, or apply complex
calculations to derive new columns.

2. Data Cleansing: UDFs can be utilized for data cleaning tasks, such as
removing duplicates, handling missing values, or normalizing inconsistent data
formats. These functions can implement customized rules or algorithms to clean
the data based on the user's needs.

3. Feature Engineering: UDFs are often employed in big data pipelines for
feature engineering, where new features are derived from existing data to improve
the performance of machine learning models. UDFs can implement domain-
specific logic to extract relevant features or perform aggregations on the data.

4. Complex Analysis: Sometimes, standard built-in functions may not be

sufficient to perform advanced analysis on the data. UDFs enable users to define
complex analytical operations and apply them to large datasets. This could
include statistical analysis, pattern recognition, or any other custom calculations.
To use UDFs in big data frameworks like Apache Hadoop or Apache Spark, you
typically need to follow the framework's specific APIs and guidelines for creating
and registering your functions. These frameworks provide APIs in various
programming languages like Java, Scala, or Python to define UDFs and integrate
them into your data processing workflows.

It's worth noting that UDFs in big data frameworks are designed to operate in a
distributed and parallel manner, taking advantage of the distributed computing
capabilities of these frameworks. This allows for efficient processing of large
volumes of data across multiple nodes in a cluster.

In big data processing, user-defined functions (UDFs) play a crucial role in

performing custom operations on large datasets. UDFs allow users to extend the
functionality of big data processing frameworks by defining their own
computations, transformations, or analyses that are not available out of the box.

Here's an example of how a user-defined function can be used in big data

processing:

1. MapReduce: In the MapReduce paradigm, a UDF can be defined to perform

custom operations within the map and reduce phases. For example, let's say you
have a large dataset of text documents and you want to count the occurrences of
a specific word. You can define a UDF that takes a document as input and emits
key-value pairs, where the key is the word and the value is the count. The
MapReduce framework will then apply this UDF to each document in parallel
and consolidate the results.

2. Apache Spark: Apache Spark, a popular big data processing framework,

supports user-defined functions through its Spark SQL module. UDFs can be
defined using programming languages like Python, Java, or Scala. For instance,
if you have a DataFrame containing a column of numbers and you want to
calculate the square of each number, you can define a UDF that takes a number
as input and returns its square. You can then apply this UDF to the DataFrame,
and Spark will distribute the computation across the cluster.
3. Hive: Hive is another widely used big data processing tool that provides a SQL-
like interface for querying and analyzing data stored in distributed file systems.
Hive allows users to define UDFs in various programming languages, such as
Java or Python, to perform custom operations on the data. For instance, you can
define a UDF to extract specific information from a string or perform complex
mathematical calculations. These UDFs can then be invoked within Hive queries
to process the data.

Overall, user-defined functions empower big data practitioners to implement

custom logic and operations tailored to their specific requirements, allowing them
to extract insights and derive value from large datasets efficiently.

CASE STUDY:

Problem Statement:

A financial institution wants to detect fraudulent transactions in real-time to

prevent financial losses and protect their customers. They have a continuous
stream of transaction data coming in from various sources and need to analyze
each transaction as it occurs to identify potential fraud.

Solution:

To address this problem, the financial institution decides to utilize a big data
processing framework like Apache Flink and define a user-defined function
(UDF) to perform real-time fraud detection on the incoming transaction stream.

1. Data Stream Ingestion:

The first step is to ingest the transaction data stream into Apache Flink. This can
be achieved by connecting to real-time data sources like Apache Kafka or by
leveraging Flink's connectors for various streaming platforms.
2. Defining the UDF:

The financial institution defines a UDF called `fraud_detection` that takes a

transaction as input and returns a boolean value indicating whether the transaction
is fraudulent or not. The UDF implements a fraud detection algorithm, which may
involve analyzing transaction patterns, checking for anomalies, or applying
machine learning models.

3. Applying the UDF:

The defined UDF is applied to the incoming transaction stream using Flink's
DataStream API. The UDF is connected to a transformation operation, such as
`map` or `filter`, which applies the function to each transaction in real-time across
the Flink cluster. The result is a filtered stream containing only the potentially
fraudulent transactions.

4. Alerting and Actions:

Once a potentially fraudulent transaction is detected, the financial institution can

take immediate actions, such as generating an alert for further investigation,
blocking the transaction, or notifying the customer. These actions can be triggered
based on the output of the UDF in the streaming pipeline.

Benefits:

Utilizing a user-defined function for real-time fraud detection provides several

advantages:

Immediate Response: The UDF enables real-time analysis of transactions as

they occur, allowing the financial institution to respond swiftly to potential fraud
and minimize financial losses.

Customization: The UDF can be tailored to the financial institution's specific

fraud detection requirements, incorporating domain knowledge, rules, or
advanced machine learning models.
Scalability: Big data processing frameworks like Apache Flink offer scalability,
enabling the processing of high-volume transaction streams and the ability to
handle increasing data loads as the business grows.

Low Latency: The real-time processing capabilities of the big data framework
ensure minimal latency in detecting and responding to fraudulent transactions.

By leveraging a user-defined function for real-time fraud detection in this case

study, the financial institution can proactively identify and mitigate fraudulent
activities, ensuring the security and trustworthiness of their financial systems and
protecting both the institution and its customers from financial risks.

Using the UDF

After writing the UDF and generating the Jar file, follow the steps given below −

Step 1: Registering the Jar file

After writing UDF (in Java) we have to register the Jar file that contain the UDF
using the Register operator. By registering the Jar file, users can intimate the
location of the UDF to Apache Pig.

Syntax
Given below is the syntax of the Register operator.
REGISTER path;

Example
As an example let us register the sample_udf.jar created earlier in this chapter.
Start Apache Pig in local mode and register the jar file sample_udf.jar as shown
below.
$cd PIG_HOME/bin
$./pig –x local

REGISTER '/$PIG_HOME/sample_udf.jar'
Note − assume the Jar file in the path − /$PIG_HOME/sample_udf.jar

Step 2: Defining Alias

After registering the UDF we can define an alias to it using the Define operator.

Syntax
Given below is the syntax of the Define operator.
DEFINE alias {function | [`command` [input] [output] [ship] [cache] [stderr] ]
};

Example
Define the alias for sample_eval as shown below.
DEFINE sample_eval sample_eval();

Step 3: Using the UDF

After defining the alias you can use the UDF same as the built-in functions.
Suppose there is a file named emp_data in the HDFS /Pig_Data/ directory with
the following content.
001,Robin,22,newyork
002,BOB,23,Kolkata
003,Maya,23,Tokyo
004,Sara,25,London
005,David,23,Bhuwaneshwar
006,Maggy,22,Chennai
007,Robert,22,newyork
008,Syam,23,Kolkata
009,Mary,25,Tokyo
010,Saran,25,London
011,Stacy,25,Bhuwaneshwar
012,Kelly,22,Chennai
Verify the contents of the relation Upper_case as shown below.
grunt> Dump Upper_case;

(ROBIN)
(BOB)
(MAYA)
(SARA)
(DAVID)
(MAGGY)
(ROBERT)
(SYAM)
(MARY)
(SARAN)
(STACY)
(KELLY)

CONCLUSION:

User-defined functions (UDFs) in Apache Spark provide a powerful mechanism

to extend the functionality of Spark by allowing developers to define their custom
logic for data processing. UDFs enable complex transformations, aggregations,
or calculations to be applied to distributed datasets in a distributed and
parallelized manner. It's important to note that clustering alone does not provide
specific object-level labels or precise object recognition. It aids in grouping
visually similar objects, which can then be used as a preliminary step for
subsequent tasks like classification or detection to achieve more precise object
recognition results. Overall, user-defined functions in Spark empower users to
extend Spark's functionality and perform custom computations on distributed
data. By leveraging UDFs, users can implement complex data transformations,
incorporate their domain-specific logic, and achieve more flexible and tailored
data processing within Spark's distributed computing framework.