See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/343615047

Anomaly Detection (Data Science) Anomaly Detection, A Key Task for

AI and Machine Learning

Presentation · August 2020

CITATIONS READS

0 207

1 author:

Nilu Singh
K L University
89 PUBLICATIONS   245 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

speaker recognition View project

Useful research tools View project

All content following this page was uploaded by Nilu Singh on 22 August 2020.

The user has requested enhancement of the downloaded file.

Anomaly Detection (Data Science)
Anomaly Detection, A Key Task for AI
and Machine Learning

Dr. Nilu Singh

School of Computer Applications
Babu Banarasi Das University
Lucknow-UP
 Content
• Introduction
• Anomaly and Its Importance in Machine
Learning
• Examples of potential anomalies
• How to detect anomaly
• Types of Anomalies in Anomaly Detection
• Importance of anomalies
• Applications
• References
 Introduction
• One way to process data faster and more
efficiently is to detect abnormal events, changes
or shifts in datasets.
• Anomaly detection refers to identification of
items or events that do not conform to an
expected pattern or to other items in a dataset that
are usually undetectable by a human expert.
Cont…
• Anomaly detection is the process of identifying
unexpected items or events in data sets, which
differ from the norm.
• It is often applied on unlabeled data which is
known as unsupervised anomaly detection.
Cont…
It has two basic assumptions:

1. Anomalies only occur very rarely in the data.

2. Their features differ from the normal instances
significantly.
 Anomaly and Its Importance in
Machine Learning

• Data is becoming increasingly important in

almost every conceivable field and area.
• From business and healthcare to law enforcement
and sports, data is central to their operations.
• Instead, you need to make good use of it, and this
is where data science comes into play.
Cont…
• Anomaly detection is one of the most interesting
applications of data science.
 Examples of potential anomalies
• A leaking connection pipe that leads to the
shutting down of the entire production line.

• Multiple failed login attempts indicating the

possibility of fishy cyber activity.

• Fraud detection in financial transactions.

 How to detect anomaly
• How do we identify whether data points are
normal or anomalous?
• Let us take the following example-
Cont…
Cont…
• In the given example of two-dimensional data
(X and Y), it becomes quite easy to visually
identify anomalies through data points located
outside the typical distribution.
Cont…
• However, looking at the figures to the right, it is
not possible to identify the outlier directly from
investigating one variable at the time: It is
the combination of the X and Y variable that
allows us to easily identify the anomaly.
 Types of Anomalies in Anomaly
Detection
Not all anomalies are equal. they can be split
into three broad categories:
• Point anomalies
• Collective anomalies
• Contextual anomalies
Point Anomalies
Collective Anomalies
Contextual anomalies
 Importance of anomalies

• Modern businesses are beginning to understand

the importance of interconnected operations to get
the full picture of their business.
• Besides, they need to respond to fast-moving
changes in data promptly, especially in case of
cyber-security threats.
• Anomaly detection can be a key for solving such
intrusions
Applications of Anomaly Detection
There are many applications for anomaly detection:

Cyber Security– Network intrusion is a prominent

example. One way an anomaly detection algorithm
would do this would be by monitoring traffic to
establish normal levels and then identifying anything
that falls outside this norm.
Cont…
• Fraud detection – This was mentioned above
with the credit card example.
• Social media monitoring – Here anomaly
detection might identify that searches for a
particular topic spike at certain times of the year,
enabling advertisers and marketers to allocate
their budgets accordingly.
Cont…
• Machine performance – Digital
twin technologies are a good example in this
instance.
• Medical monitoring – This is everything from
identifying abnormal patterns or occurrences in
an individual (such as an irregular heartbeat).
 Machine learning and statistical
analysis for anomaly detection

• Approach 1: Multivariate statistical analysis

• Approach 2: Artificial Neural Network
 Anomaly Detection Methods
• Based on different machine learning algorithms,
anomaly detection methods are primarily
classified under the following two type-
1. Supervised methods
2. Unsupervised methods
• https://www.dictionary.com/browse/anomaly
• https://towardsdatascience.com/how-to-use-machine-
learning-for-anomaly-detection-and-condition-
monitoring-6742f82900d7
• https://thedatascientist.com/anomaly-detection-why-you-
need-it/
• https://www.kdnuggets.com/2019/10/anomaly-detection-
explained.html
View publication stats