OFFICIAL (CLOSED) \ NON-SENSITIVE

What is
Anomaly Detection
Mr Hew Ka Kian
hew_ka_kian@rp.edu.sg
 OFFICIAL (CLOSED) \ NON-SENSITIVE

What is Anomaly Detection

• Anomaly detection (also outlier detection) is the identification of rare items,
events or observations which raise suspicions by differing significantly from the
majority of the data.
• The anomalous items may
translate to some kind of problem such
as bank fraud, network breach, a structural
defect, medical problems or errors in a text.
• Anomalies are also referred to as outliers,
novelties, noise, deviations and exceptions.
• Anomaly detection is applied on unlabeled
data is known as unsupervised anomaly
detection although supervised anomaly
detection is possible with labeled data
Source: https://en.wikipedia.org/wiki/Anomaly_detection
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Anomaly Types
• Anomaly is a broad concept, which may refer to many different types
of events in time series.
• A spike of value, a shift of volatility etc. could all be anomalous or
normal, depending on the specific context.

Source:
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Time series data anomaly detection

• Successful anomaly detection hinges on an ability to accurately analyze time series data in real
time.
• Time series data is composed of a sequence of values over time. That means each point is
typically a pair of two items — a timestamp for when the metric was measured, and the value of
that metric.
• Time series data anomaly detection can be used
for valuable metrics such as: Seismic
Virus
infection reading
cases

Power
Transaction
generator
volume
output

Login Mobile app

attempts installs

Source:
 OFFICIAL (CLOSED) \ NON-SENSITIVE
Univariate vs. Multivariate
Univariate
• Looking at one variable
• If we want to look at
anomalous weather patterns,
univariate anomaly detection
will measure a single indicator,
such as temperature. We can
then ask questions like “is this
temperature strange for this
region?”
Multivariate
• Need to consider multiple
factors and the relationship
between them
• If we want to look at
anomalous weather patterns,
multivariate analysis will
consider a host of factors, like
precipitation, humidity and air
pressure.
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Anomaly Detection Toolkit

Anomaly Detection Toolkit (ADTK) is a Python package for unsupervised time

series anomaly detection.

This package offers a set of functions that makes the training of the dataset and
the detection of anomaly easier to code

It also provides some functions to process and visualize time series and
anomaly events.

ADTK is open sourced and its code and many examples of how to use the
package is at https://github.com/arundo/adtk
 OFFICIAL (CLOSED) \ NON-SENSITIVE

ADTK Anomaly Types

• ADTK can detect a point anomaly where there is a data point whose value is
significantly different from others.
• An outlier point in a time series time is one that exceed the normal range of this
series.
• To detect outliers, the normal range of time series values (baseline) is what a
detector needs to learn.
 OFFICIAL (CLOSED) \ NON-SENSITIVE

ADTK Anomaly Types

• Spike and Level Shift: In some situations, whether a time point is
normal depends on if its value is aligned with its near past.
• An abrupt increase or decrease of value is called a spike if the change
is temporary.

Source:
 OFFICIAL (CLOSED) \ NON-SENSITIVE

ADTK Anomaly Types

• An abrupt increase or decrease of value is called a spike if the change
is temporary.
• However we should use ADTK to detect a level shift if the change is
permanent.

Source: https://cloud.google.com/ai-platform/docs/ml-solutions-overview
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Workflow Source data

Data can come from the places You can also get your hands on
you have access to like the credit privileged data through Prepare data
card transaction if you work in the commercial arrangement like
bank paying for the data
Select the
algorithm

There are also plenty of open (public) dataset that is shared with the public free: Train the
https://kaggle.com is an online community for machine learning enthusiasts and it model
has many open dataset
https://data.gov.sg was first launched in 2011 as the government's one-stop portal
to its publicly-available datasets from 70 public agencies. To date, more than 100 Test the model
apps have been created using the government’s open data.

Use the model

for Inference
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Workflow Source data

Filter the data Transform

Prepare data

• The rows of interest like • Combine multiple source Select the

those above 65 years old • Extract feature by applying algorithm
• The columns of interest, mathematical functions
basically just the datetime like moving average of 20
and the feature columns data points Train the
model

Test the model

Use the model

for Inference
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Workflow Source data

What type of anomaly?

Prepare data

Global (point) Contextual anomalies Collective anomalies

Select the
algorithm

ADTK volatility shift

ADTK threshold detector
to detect values outside
ADTK quantile detector to
detect values outside
detector detects shift of
volatility by comparing 2
ADTK Seasonal detector
detects departure from a
Train the
certain threshold values certain percentile windows of values repeating pattern model

Test the model

Use the model

for Inference
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Workflow Source data

Train and Test

Prepare data

To train the model using ADTK, call the fit(df)

function Select the
algorithm

To test or use the model to detect anomaly, call Train the

the detect(df) function model

Convenient function that train followed by Test the model

detect, call the fit_detect(df) function

Use the model

for Inference
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Exercise D
Pandas is a fast and powerful Python library for data
manipulation.

Import the library

• import pandas as pd

Read the content of the comma separated values (CSV) into a

Pandas Series object
• s = pd.read_csv('dataset.csv', index_col="pr_date",
parse_dates=True, infer_datetime_format=True)
• index_col is the column that holds the datetime
• infer_datetime_format=True tries to guess the date format
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Exercise D

Use the ADTK validate_series(df) to check for error in the Series

Import the library

• from adtk.data import validate_series

Validate the Series

• s = validate_series(s)
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Exercise D Detecting Simple Threshold

Use ThresholdAD to detect outlier (point anomaly) that exceeds the
baseline threshold

Import the library

• from adtk.detector import ThresholdAD

Create the ThresholdAD object with the high and low threshold values. Values above the
high or below the low threshold are flagged as anomaly
• threshold_ad = ThresholdAD(high=30, low=15)

Detect anomalies

• anomalies = threshold_ad.detect(s)
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Exercise D

Use the plot() function to visualize the graph

Import the library

• from adtk.visualization import plot

Plot the graph with the DataFrame and anomalies. Can specify the anomaly
marker colour and the tag as marker (dot on the graph)

• plot(s, anomaly=anomalies, anomaly_color='red’,

anomaly_tag="marker");
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Exercise D Percentile
• Percentile: the value below which a percentage of data falls.

Source: https://www.mathsisfun.com/data/percentiles.html
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Exercise D Percentile
Use QuantileAD to detect outlier (point anomaly) that exceeds the
certain percentile of the series

Import the library

• from adtk.detector import QuantileAD

Create the object with the high and low percentile with values outside of
this boundary considered anomalous
• quantile_ad = QuantileAD(high=0.99, low=0.01)

Need to be trained on the range before detection. fit_detect() does this

in one step
• anomalies = quantile_ad.fit_detect(df)
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Exercise D
Use VolatilityShiftAD() to detect a change in the volatility in the
time series

Import the library

• from adtk.detector import VolatilityShiftAD

VolatilityShiftAD() compares the volatity between 2 windows next to each other. We

have to create the object specifying how many time points do the windows contain

• volatility_shift_ad = VolatilityShiftAD(window=30)
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Exercise E
• Read tsla.us.txt csv file and print the content.
s = pd.read_csv('data/tsla.us.txt’,
index_col="Date", parse_dates=True,
infer_datetime_format=True)
print(s)
• What are the columns
Open, High, Low, Close, volume, OpenInt
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Exercise E
• Write the code to drop the other columns except the Date and Volume columns.
s = s.drop(['High','Low','Open','Close','OpenInt'],axis=1)
• Write the code to detect the volatility shift with window of 60 values
s = validate_series(s)
volatility_shift_ad = VolatilityShiftAD(window=60)
anomalies = volatility_shift_ad.fit_detect(s)
• Plot the graph and do you have something like the graph shown in the worksheet that
detects the starts of increase volatility?
plot(s, anomaly=anomalies, anomaly_color='red’);
• What did you find that was the likely reason for the anomaly? Bare in mind that anomaly
does not have to be bad, simply something that deviates from the standard or norm.
• Tesla announced it is getting profitable in 2013.
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Exercise F
• Open the file weekly-infectious-disease-bulletin-cases.csv using Excel.
• What are the columns

epi_week: week of the year

disease: disease
no._of_cases: no of cases
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Exercise F
• How do we filter for the rows with ‘Dengue Fever’?
infectious = infectious[infectious['disease'] == 'Dengue Fever’]
• Set the epi_week as the index column.
infectious = infectious.set_index('epi_week')
• Drop the disease column.
infectious = infectious.drop('disease',axis=1)
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Student Activity Source data

In exercise G, we are going to do the whole ML workflow Prepare data

1. Source for the data at Data.gov.sg

Select the
2. Prepare the data algorithm
• Change the datetime format
• Choose a disease to examine and filter out the irrelevant columns Train the
model
3. Select the algorithm – Shift in volatility
4. Train and use the model for inference Test the
model

Use the model

for Inference

Source:
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Exercise G
• Acute Upper Respiratory Tract infections
infectious = pd.read_csv(
'data/average-daily-polyclinic-attendances-for-selected-diseases.csv’)
infectious['epi_week'] = pd.to_datetime(
infectious['epi_week'] + '-1 00:00:00', format='%Y-W%W-%w %H:%M:%S’)
infectious = infectious[
infectious['disease'] == 'Acute Upper Respiratory Tract infections’]
infectious = infectious.set_index('epi_week’)
infectious = infectious.drop('disease',axis=1)
volatility_shift_ad = VolatilityShiftAD(window=52)
anomalies = volatility_shift_ad.fit_detect(infectious)
plot(infectious, anomaly=anomalies, anomaly_color='red');
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Problem Solution
Is there is any abrupt change in trend for visitors to Singapore?

Source:
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Problem Solution
• Source data
• Get the visitor-international-arrivals-to-singapore-by-region-monthly.csv from Data.gov.sg
• The datetime column is well defined so no need to modify
s = pd.read_csv('data/visitor-international-arrivals-to-singapore-by-region-
monthly.csv', index_col="month",
parse_dates=True)
• Filter for a region like Africa
s = s[s['region']=='Africa']
• Drop the regions column
s = s.drop(['region'],axis=1)
• Train and inference
s = validate_series(s)
volatility_shift_ad = VolatilityShiftAD(window=12)
anomalies = volatility_shift_ad.fit_detect(s)
• Plot
plot(s, anomaly=anomalies, anomaly_color='red')
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Exercise H
• Any anomaly?
Yes, around 2002-2003 (depending on country)