random inverse undersampling

problem
Muhammad Usama Muhammad Ahmed Taha
18K-0200 18K-0231 18K-XXXX

June 16, 2022

Teacher: Waqas Sheikh

1
1 Research Goal
Our Research goal is to solve class imbalance
problem on a large data set that holds true to
the the concept of class imbalance and for this we
have used a novel approach known as inverse ran-
dom under sampling technique. This technique
has been previously used to solve class imbalance
problem for multi-label classification. Our data
set will have this technique applied upon.
Inverse Random Under sampling technique
works by inverting the cardinalities of major-
ity and minority class. Thus the majority class
is divided into subsets such that each subset is
smaller in size then the minority class.
Basically the concept behind Inverse Random
Under Sampling is to maintain a high rate of
True positive rate by imbalance inversion but
this also induces a high false positive rate. For
this by an ensemble technique known as bagging,
by combining the designed detectors using fusion
we can control false positive rate
2 Retrieving Data
The data set we intended to retrieve is called
Glass identification data set. This data set is
a multi-class data set which provides an analy-
sis of 7 different types of glasses. The motivation
behind the creation of this data set was for crim-
inological investigation, thus the glass could be
used as an evidence if correctly identified. This
data set consists of 10 attributes which then are
used to identify the type of glass based on these
10 attributes.
This data set was retrieved from github. But
the data set did not provide any attribute
name,which were then fetched separately from
the official UCI website. This data set does in-
1
deed contain a relatively large class imbalance
problem
3 Data Exploration
After the data was extracted it was thoroughly
observed for any unexpected or unusual findings.
The 10 attributes in our data set were index, Re-
fractive index, Sodium, Magnesium, Aluminum,
Potassium, Silicon, Calcium, Barium and Iron.
And the distinct 7 types of glass were building
windows processed, building windows non pro-
cessed, vehicle windows processed, vehicle win-
dows non processed, containers, tableware and
head lamps. All the attributes consisted of con-
tinuous values. All in All there were a total of
214 samples available in the data set.
After delving a little deep we started to ob-
serve the class imbalance in our data set. As
there were 7 distinct classes an ideal situation
would have been if every class consisted of equal
samples but that is certainly not the case with
our data set. So before evaluating further let’s
go over the per class sample size.
ˆ 1st class which is building window processed
contains a sample size of 70 (32.71 %)
ˆ 2nd class which is building window non pro-
cessed contains a sample size of 76 (35.51 %)
ˆ 3rd class which is vehicle window processed
contains a sample size of 17 (7.94 %)
ˆ 4th class which is vehicle window non pro-
cessed contains a sample size of 0 (0 %)
ˆ 5th class which is containers contains a sam-
ple size of 13 (6.07 %)
ˆ 6th class which is tableware contains a sam-
ple size of 9 (4.21 %)
 ˆ 7th class which is headlamps contains a that good relationship between the variables and
sample size of 29 (13.55 %) the target variable was found.Missing values are
another hindrance in a data set that provides a
The above points tell us exactly how much of negative impact when training a model. There
an imbalance there exists between each class. were no missing values in any column in our data
4th class, which is the vehicle windows non pro- set thus this eliminated the need for filling in the
cessed, does not even contain a single sample data set.
so even it should will not be considered as a
valid class anymore and will be dropped, leav-
ing us with only 6 classes. Now out of these 6 After finding correlation we started to under-
classes it can easily be observed that the differ- stand the distribution of each variable in the
ence between the class with maximum sample data set. Distribution tells us how the data set
size, which is building windows non processed values are distributed across a curve. A normal
with 76 samples, and the class with minimum distribution also known as Gaussian distribution
sample size, which is tableware with only 9 sam- is a normal distribution that is symmetric about
ples, is exceedingly large. the mean, basically telling us that the majority
But that is not all as there are 6 classes and of data distribution is near the mean, it usually
they can easily be be distributed in 2 groups occurs as a bell curve if plotted on a graph. Here
we find on our data set that most variables have
ˆ 1st group where the sample size is greater
a sort of skewed distribution which if unhandled
or equal to 70 samples. It 2 classes.
will produce incorrect results and will distort the
ˆ 2nd group where the sample size is smaller purpose of our project. We also used box plot
then even 30 samples. It contains 4 classes. and histograms to better understand the the dis-
tribution, obtain an idea about the outliers and
This produces a great disparity and the re- observe individual distribution of each each at-
sults obtained by using any other conventional tribute with all 6 classes. As distribution seemed
method would be incredibly biased and incor- to be the extent of the problems in our data set.
rect. The combined samples from all classes in We also decided to use violin plot integrated with
the 2nd group do not even sum up to a the sam- box plot for every attribute. This granted us
ple size of a single class in group 1. This is the with a deep perspective of every attribute and
primary problem that will be addressed in this how every attribute need to be catered individ-
our project. ually.
After this we did further exploration on our
data set. We applied Pearson Correlation be-
tween the variables and a heat map is used to A complete dashboard with all the minor de-
clearly isolate every correlation. This provided tails regarding the data set is created to examine
us with a general idea of variables and their re- every last detail that might have been missed or
lationship between other variables.Only the at- overlooked. Basically it informs us about the to-
tribute ’Calcium’ was found to be having no cor- tal number of values, distinct values, maximum
relation with the target variable. Apart from value, missing values etc regarding each column.

2
4 Data preparation
In this stage the first thing that will be needed to
be done is dropping the class of vehicle windows
non processed from our evaluation as it contains
0 samples. Another thing that we observed dur-
ing the correlation tests that attribute ’calcium’
and the target variable had no relation and thus
providing us a reason to drop it off and reduce
the size of our data set.
Now as we discovered in the data exploration
stage that data in most columns had a skewed
distribution and we had also further observed
that data was not on the same scale. So the
first priority was to normalize our data so that
a distribution can be achieved which is nearest
to normal distribution. For this purpose nor-
malize function is used. We again plot the dis-
tribution of the data after normalizing and ob-
serve how the skewness had been considerably
reduced. After normalizing another task was to
bring our data on the same scale. For this we
standardize our data, this brings the entire data
set to a similar scale. We used a standard scaler
function that transforms the entire data set into
standardized form. After this when again the
data is observed, a bell shaped curve is achieved
for every attribute.
Another problem we observed were during
the exploration stage was the outliers that were
found in considerable quantity. We removed the
outliers by finding out the z score and choos-
ing only the entries with the z score of less then
3. Distribution is again plotted and this time
the data distribution seems to be the closest to
Normal distribution. As of now data in it’s en-
tirety has been taken care off and the data is in
its cleanest form, of course apart from the class
imbalance problem.
3
5 Data modeling
Now here we will deal with the problem of class
imbalance. And for this problem we will use
the technique of Inverse Random Under Sam-
pling. Inverse Random Under Sampling tech-
nique has previously shown great promise while
dealing with class imbalance problem in multi
class dataset.
Before moving on how we modeled our data
set using Inverse Random Under Sampling, let
us understand how The mechanism behind this
technique works. This technique first increases
the true positive rate and decreases the false pos-
itive rate, which increases as a consequence of
increasing true positive rate, by using the tech-
nique called bagging.
Bagging is an ensemble classifier that works on
a relatively simple idea. bagging generates many
training subsets from a larger complete data set.
Each subset of samples is generated randomly
where each sample in a subset is selected with
replacement and equal probability. Each subset
has a prediction method applied to it. Then the
final prediction is obtained by majority voting.
It is usually the positive class with a rather
small amount of samples, thus in Inverse Ran-
dom Under Sampling data manipulation hap-
pens in such a way that we reduce the the sam-
ple size of majority class which is the negative
class such that the sample size of positive class
exceeds the negative class. This happens by con-
tinuously under sampling until each subset is
smaller then that of the positive class. Now the
samples of positive class exceeds that of nega-
tive class thus the focus shifts towards the posi-
tive class. Each training set yields one classifier
design with importance towards positive class.
Then by combining the different designs through
fusion a composite boundary can be constructed.
 Coming back to our data set we have used the
same approach of making One v/s all. We iterate
over all the minority classes and consider them
on a binary level.
We specify 3rd class, 5th class and 6th class
as a set of minority classes. We used the pseudo
code of Inverse Random Under Sampling in
the paper ”Inverse Random Under Sampling for
class imbalance problem and it’s application to
multi-label classification” which was provided to
us. We used decision tree as our base classifier
and for bagging we have used mean method for
combining.The pseudo code can be observed be-
low.
def IRUS.fit (attributes, prediction):
maj,min,d : majority class in prediction, mi-
nority class in prediction, total classes with value
counts in prediction
compute subsets(attributes, predictions)
for each subset created:
Xtrain : subset[remove class column]
ytrain : subset[only class column]
model : Decision Tree Classifer
model.fit(Xtrain, ytrain)
bagging.append(model)
6 Automation and presenta-
tion
We have used Kfold validation to further un-
derstand and present the most valid results we
could find. We used 5 fold validation for this pur-
pose. To evaluate our model with better meth-
ods of evaluation we used F1 score and ROC-
AUC curve.
ROC-AUC is a performance metric for classi-
fications problems at various thresholds. AUC
here stands for Area under the ROC curve,
which means that it measures the entire two-
dimensional area underneath the entire ROC
curve. F1 score is also known as the harmonic
mean of precision and recall. It is a measure of
model’s accuracy on a data set.
To better understand how the technique of In-
verse Random Under Sampling technique helps
us, we also used several other machine learning
models without using this technique to gather
how much exactly does the this technique per-
form in comparison to these models that do not
incorporate Inverse Random Under Sampling.
For this purpose the model we used are SVM,
KNN, logistic regression, decision tree-C4.5, de-
cision tree-Cart.
Finally we ran tests on the the model of In-
verse Random Under Sampling with other mod-
els not using this technique. The IRUS model
outperformed every single other model we used.
The IRUS model got and F1 score of 0.76 while
the 2nd highest score was only 0.67 which was
achieved by 2 different models KNN and SVM.
This result is evidence enough that the tech-
nique Inverse Random Under Sampling provides
a huge increase in performance. It helps to solve
the problem of class imbalance and provides and
evident jump in results then other conventional
method.
The full table of all the models with their F1
score and ROC score are shown below
Models F1 Score ROC AUC Score
IRUS 0.769 0.858
SVM 0.676 0.872
KNN 0.676 858
Logistic Regression 0.666 0.862
Decision Tree-C4.5 0.646 0.788
Decision Tree-Cart 0.610 0.794