Proceedings of the 5th International Conference on Smart Systems and Inventive Technology (ICSSIT 2023)

IEEE Xplore Part Number: CFP23P17-ART; ISBN: 978-1-6654-7467-2

Theoretical Evaluation of Ensemble Machine

Learning Techniques
M ilind Shah * Kinjal Gandhi Kinjal A Patel
Assistant Professor Assistant Professor Assistant Professor
2023 5th International Conference on Smart Systems and Inventive Technology (ICSSIT) | 978-1-6654-7467-2/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICSSIT55814.2023.10061139

Department of Computer Science & Department of Computer Science & Faculty of Computer Applications &
Engineering Engineering Information Technology
Krishna School of Emerging Krishna School of Emerging Gujarat Law Society University
Technology & Applied Research – Technology & Applied Research – Ahmedabad, Gujarat, India
Drs. Kiran & Pallavi Patel Global Drs. Kiran & Pallavi Patel Global kinjal5721@gmail.com
University (KPGU) University (KPGU)
Vadodara, Gujarat, India Vadodara, Gujarat, India
milindshahcomputer@gmail.com kinjal445@gmail.com

Harsh Kantawala
Assistant Professor
Department of Computer Engineering
G H Patel College of Engineering &
Technology –
Charutar Vidhya Mandal University
(CVM)
Vallabh - Vidhyanagar, Gujarat, India
harshkantawala@gcet.ac.in
Ankita Kothari
Rohini Patel
Assistant Professor
Assistant Professor
Department of Computer Science &
Department of Computer Science &
Engineering
Engineering
Krishna School of Emerging
Krishna School of Emerging
Technology & Applied Research –
Technology & Applied Research -
Drs.Kiran & Pallavi Patel Global
Drs. Kiran & Pallavi Patel Global
University (KPGU)
University (KPGU)
Vadodara, Gujarat, India
Vadodara, Gujarat, India
ankita.cor@gmail.com
rohini.d.patel@gmail.com

Abstract— The use of ensemble techniques is widely
recognized as the most advanced approach to solving a variety
of problems in machine learning. These strategies train many
models and combine the results from all of those models in
order to enhance the predictive performance of a single model.
During the period of the last several years, the disciplines of
artificial intelligence, pattern recognition, machine learning,
neural networks, and data mining have all given a considerable
consideration to the concept of ensemble learning. Ensemble
Learning has shown both effectiveness and usefulness across a
broad range of problem domains and in significant real -world
applications. Ensemble learning is a technique that involves the
construction of many classifiers or a group of base learners
and the merging of their respective outputs in order to
decrease the total variance. When compared to using only one
classifier or one base learner at a time, the accuracy of the
results achieved by combining numerous classifiers or the set
of base learners is greatly improved. It has been shown that the
use of ensemble methods may increase the predicted accuracy
of machine learning models for a range of tasks, including
classification, regression, and the identification of outliers. This
study will discuss about ensemble machine learning techniques
and its various methods such as bagging, boosting, and
stacking. Finally, all the factors involved in bagging, boosting,
and stacking are compared.
Keywords— En semble Learning, Bagging, Boosting,
Stacking, Machine Learning, Multiple Classifier, Blend of
Experts.
I. INT RODUCT ION
In the field of co mputers, machine learning is where art
and science come together. Understanding data and selecting
the appropriate algorithm are two crucial steps in developing
an effective machine learning tool. Why settle with just one
algorithm when you can choose several different ones and
have them all contribute to achieving the same goal?
Enhanced outcomes.
Ensemble methods are a set of procedures that include
the development of numerous models followed by their
combination in order to get superior outcomes. The results
that ensemble techniques give are often more accurate than
the solutions that a single model would produce. Ensemble
learning is able to effectively solve a broad range of issues
pertaining to machine learning, including estimate,
confidence, error correction, continuous learning, missing
features, and many more. When using ensemble learning,
you generate a number of base-level classifiers or learners
and combine them for training on your dataset. Base learners,
sometimes called weak learners, are generated by applying
the base learning algorithm to the training data. In most
cases, ensemble learning uses some type of learning
algorithm to produce what is known as a homogeneous
ensemble. However, in certain cases, it constructs what are
known as heterogeneous ensembles by applying many types
of learning algorith ms. The benefit of using ensemble
learning is that it is able to improve the performance of less
capable learners, which in turn helps to improve the overall
accuracy of the learning algorithm when applied to the
training data. There are a lot of applications out there that use
the ensemble learning approaches, and in the end, those
applications have come to the conclusion that the ensemble
method works better than the traditional way (single learner)
[5].
Finding a single model that can provide the most accurate
prediction of the intended outcome is the objective of every
machine learning problem, rather than utilizing any of the
traditional machine learning algorithms and expecting that
this model will be the best accurate predictor, ensemble
methods take into consideration a numerous models and then
generate one final model by averaging all of those models
together.
When trying to increase the reliability of one's predictions,
it is common practice to use ensemble methods of machine
learning. They are often used in situations in which a single
machine learning algorithm would be unable to effectively

978-1-6654-7467-2/23/$31.00 ©2023 IEEE 829

Authorized licensed use limited to: COCHIN UNIVERSITY OF SCIENCE AND TECHNOLOGY. Downloaded on March 17,2024 at 05:11:14 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the 5th International Conference on Smart Systems and Inventive Technology (ICSSIT 2023)
IEEE Xplore Part Number: CFP23P17-ART; ISBN: 978-1-6654-7467-2
understand the underlying connections due to the dataset
being either too large or too complex. The main reason to
choose ensemble machine learning techniques is that they
can help to avoid overfitting, it can also be less susceptible to
bias, and are more robust to changes in data.
Let's compare ensemble learning with a real-world case
study. Imagine if the co mpany's management decided to hold
a meeting to discuss a measure that had just been approved,
after wh ich they called the meet ing. Instead of the Managing
Director or HR taking the decision alone by themselves may
not be the best idea because this accounts for dictatorship,
and the results may not be in favor of the employees as much
as they could be. Therefore, this comparison may be used to
a single model in machine learn ing, such as a Decision Tree
or Logistic Regression, and they will always perform slightly
less than the ensemble approaches. On the other hand, if a
meeting is held between a number of different employees,
say four of them, each of them will offer an opinion along
with the benefits and drawbacks of the situation. And finally,
the most compelling viewpoint will be selected based on the
votes of the majority. The ensemble method is precisely the
same thing as described here.Similarly, we can say that
several models are utilized, and depending on the results of
each, a majority vote will be used to make the final decision.
Training
Model A
Data
Training
Model B Generalizer
Data
Training
Model C
Data
Fig 1. Summary of Ensemble Learning
This paper consists of six sections. Section I is Introduction,
Section II is Why we need ensemble techniques , Section III
is Related Work, Section IV Methodology, Section V is
Co mparative Analysis and finally, Conclusion which
concludes the paper.
II. WHY WE NEED ENSEMBLE TECHNIQUES?
It is well known that bias, variance, and noise all have a
severe impact on the errors and predictions produced by
mach ine learning models. To counteract these drawbacks,
ensemble techniques are used. And there are two primary
reasons, both of which are connected, to adopt an ensemble
rather than a single model. These are some of the
arguments: Performance: An ensemble of models may
produce more accurate predictions and achieve higher levels
of performance than any one of the individual models that
contribute to it. The use of an ensemble helps to bring the
spread or dispersion of the predictions and the performance
of the model closer together, which is what is meant by the
term "robustness." When trying to solve a problem with
predictive modeling, ensembles are used to do a better job
of predicting than a single predictive model could.
III. RELATED WORK
In [1] Emine Yaman et al, the objective of the research is
to discover if bagging and boosting ensemble classifiers are
capable of properly detecting neuromuscular disorders by
978-1-6654-7467-2/23/$31.00 ©2023 IEEE
Authorized licensed use limited to: COCHIN UNIVERSITY OF SCIENCE AND TECHNOLOGY. Downloaded on March 17,2024 at 05:11:14 UTC from IEEE Xplore. Restrictions apply.
making use of EM G s ignals as the data source. The purpose
of this research is to evaluate whether or not these classifiers
are capable of doing so. It is important to understand the fact
that the method can be divided into three distinct steps . The
first step is to calculate the wavelet packing coefficient
(WPC) for each type of EM G signal. Next, we need to
calculate the WPC statistics to represent the distribution of
wavelet coefficients. This can be done using a computer
program. To achieve this goal, wavelet transform can be
used. Ensemble classifiers were used during the final step of
the method to achieve d iagnosis of neuromuscular
conditions. The resulting features were used as input
variables in the classification process. In this research, a
comparison has been made between the bag ensemb le
learning strategy and the reinfo rcement ensemble learning
approach for automat ic recognition of electro myography
(EM G) data. The performance of ensemble classifiers for
real-world p roblems has been presented in many research
studies, but many focus on the feasibility of bagging and
boosting ensemble classifiers for the diagnosis of
neuromuscular disorders. Very few studies have been done.
Meanwhile, several research studies have shown the
effectiveness of ensemble classifiers in relation to real-
world p roblems. This is still true despite ensemble
classifiers being shown to be efficient. The results of this
research showed that ensemble classification performs better
in the diagnosis of neuromuscular disorders compared to
individual classifiers. The results are encouraging as the
AdaBoost algorithm co mbined with the random forest
ensemble method is shown to produce an accuracy of
99.08%, an F-measure of 0.99, an AUC of 1, and a kappa
statistic of 0.99.
In [2] Xianwei Gao et al, as its research object ive, the
NSL-KDD data collection is used in this research. After
that, it goes into the most recent advancements in the area of
intrusion detection technology as well as the obstacles that it
faces, and it wraps up with a suggestion for an adaptable
ensemble learning model. A ltering the quantity of data that
is used during the training phase of the MultiTree method
and creating several decision trees are both required steps in
the construction of this technique. We came to the
conclusion that the most effective way to improve the
overall detection impact would be to make use of a variety
of different fundamental classifiers, such as decision trees,
random forests, kNN, and DNN, in addition to developing
an adaptive voting mechanism for ensembles. When we put
our method through its paces using NSL-KDD TestC, we
discover that the MultiTree algorith m has an accuracy of
84.2%, and that the final accuracy of the adaptive voting
process is 85.2%. Previous research papers have been
proved to be inferior to our ensemble model in terms o f its
capability to effectively boost detection accuracy. This has
been shown via a series of tests. Furthermore, the analysis of
the data indicates that the quality of the data characteristics
is an important component in determining the efficacy of the
detection. This was discovered as a result of the discovery
that the quality of the data characteristics is an essential
component. If we want to obtain better results in the future,
we need to make sure that the feature selection and
preprocessing of the data that is used for intrusion detection
are optimized. Only then will we be ab le to achieve our goal
of getting better results.
830
 Proceedings of the 5th International Conference on Smart Systems and Inventive Technology (ICSSIT 2023)
IEEE Xplore Part Number: CFP23P17-ART; ISBN: 978-1-6654-7467-2
In [3] Thomas G Dietterich, in this research, an overview
of various approaches is provided, along with an
explanation of why ensembles may often do better
than anyone classifier. We discuss some recent research that
compared other ensemble approaches, and then we show
some recent experiments in an effort to determine the
factors that contribute to Adaboost's lack of quick
overfitting. This paper is offering a quick analysis of the
techniques for generating ensemb les and examined the three
key reasons why ensemble approaches are able to
outperform any single classifier that is included within the
ensemble. The research has also presented some
experimental data, which reveal on one of the secrets behind
AdaBoost's impressive level of performance.
In [4] Omer Sagi et al, the object ive of the research is
analyzes conventional, new, and state-of-the-art ensemble
approaches and explores current difficu lties and
developments in the area of ensemble learning. The purpose
of this article is to exp lain the notion of ensemble learning.
The main goal of these kinds of approaches is to increase
the accuracy of pred ictions by comb ining the results of a
number of separate models with their weighted averages. In
this research, we take a look at a nu mber of current trends as
well as potential future research areas in the field o f
ensemble learn ing. One current direction is to improve
widely used algorithms in order to make them mo re
effective and suited for "Big Data." These initiatives often
involve enhancing the usage of computing resources and
facilitating the spread of algorith ms among several
workstations. The translation of ensemble models into
models that are simp ler and more co mprehensive while
maintaining the pred icted accuracy of the ensemble they
were generated fro m is another promising research path. In
conclusion, we discussed current research that aim at
merging the ensemble model with deep neural networks, in
line with the accelerated pace of research into deep neural
networks. These researches were carried out very recently.
In [5] Thomas Rincy N et al, the objective of the research
is to conduct a comprehensive analysis of the many different
ensemble learning techniques that are widely used in
mach ine learning. The main contribution of the research is
to discuss a one of the most co mmon types of learning
methods in mach ine learn ing is termed ensemble learning,
an essential idea. The benefit of ensemb le learning and its
many methods, such as boosting, which construct a robust
classifier based on the number of base learners. Bagging is a
method that combines the bootstrap and aggregation into
one, and it is shown as a parallel ensemb le technique.
Essential contributions made by our research include
stacking, a method in wh ich independent learners are
merged by a learner and a variety of experts to train an
ensemble of classifiers using a method called sampling.
Stacking is one of the most important aspects of our wo rk.
In future, one of our goals is to imp lement a model based on
mach ine lea rning that makes use of ensemble learning
strategies.
In [6] Mohammad Zounemat-Kermani et al, the objective
of the research is to analy ze development of ensemb le
techniques in several applied areas of hydrology. These
techniques include ensemble resampling methods (bagging,
boosting, dogging, etc.), model averaging, and clustering.
Generalized accu mulation. Accu mulation is also an
978-1-6654-7467-2/23/$31.00 ©2023 IEEE
Authorized licensed use limited to: COCHIN UNIVERSITY OF SCIENCE AND TECHNOLOGY. Downloaded on March 17,2024 at 05:11:14 UTC from IEEE Xplore. Restrictions apply.
approach. This research review main ly focuses on
hydrological topics such as surface hydrology, river water
quality, rain fall runoff, debris flo w, river freezing, sediment
transport, groundwater, flood modeling and forecasting, and
river freezing escalation. This finding shows that the use of
ensemble methods in hydrology is definitely superior to the
traditional methods of model learning (which is done on an
individual basis). The use of ensemble techniques shows its
unquestionable superiority and proves the point. In addition,
boosting techniques such as boosting, AdaBoost, and steep
gradient boosting are often successfully applied to
hydrological problems rather than binning, stacking, and
dagging methods. This is because boost technology allows
you to squeeze more water into your system. This is because
the enhancement techniques are likely to increase the
gradient. One of the main conclusions that can be drawn
fro m this analysis is that during the past three years fro m
2018 to 2020, most researchers have used AdaBoost, XGB,
rotating forests, random subspaces, and mining. A lthough
AdaBoost and XGB boosting methods have been exploited
in many studies, more work is needed to evaluate the
performance of rotational and stochastic spatial bagging
methods as well as drilling in different hydrological
modeling. Th is can be achieved using Dagging. Integrating
mu ltip le ensemb le techniques into learning algorith ms to
improve performance is an interesting and potentially
fruitfu l research avenue. For examp le, co mbining stacking
methods can further improve the packing and reinforcement
approach.
In [7] Debachudamani Prusti et al, the objective o f the
research is researchers and industry professionals have come
up with a nu mber of different methods for detecting fraud,
all o f wh ich include the use of different algorith ms in order
to identify fraudulent patterns. This research presents the
use of several classification models using machine learning
techniques to find accuracy and other performance
characteristics to detect fraudulent transactions. These
models are developed to investigate performance. Using
classification algorith ms such as K-Nearest Neighbor (K-
NN), Extreme Learning Machine (ELM ), Rando m Forest
(RF), Mult ilayer Perceptron (M LP), and closed classifiers,
we critically examine their performance enhancement.
Created and evaluated. For its performance as a result of
improving the predict ion performance p rovided by this
model, we presented a prediction classification model which
is a set of five d ifferent algorith ms. The performance
evaluation of the presented model was established using
20% of the collected test data. Using a s et of machine
learning algorith ms is one of the unique methods developed
for cred it card fraud detection technology. Although there
are only minor d ifferences in accuracy between different
models, we found the percentage of p rediction accuracy o f
the proposed classification model to be 83.83%. Th is is a
significant imp rovement over previous percentages. The
suggested approach achieves a reduction in the fraud
detection error while simu ltaneously achieving an
improvement in the fraud prediction rate.
In [8] Arshad Jamal et al, the object ive of the research
investigated the possibility of using extreme gradient
boosting (XGBoost) models to assess crash injury severity
compared to several trad itional machine learning methods
(logistic regression, random forest, decision t ree). The data
831
 Proceedings of the 5th International Conference on Smart Systems and Inventive Technology (ICSSIT 2023)
IEEE Xplore Part Number: CFP23P17-ART; ISBN: 978-1-6654-7467-2
collected for this study were obtained fro m the Min istry of
Transportation (MOT) in Riyadh, Saudi Arabia.
Specifically, the data comes fro m the Traffic Safety
Admin istration. The data in this dataset is from 13,546
traffic accidents reported on 15 local highways between
January 2017 and December 2019. Experimental results
obtained using k-folds (k = 10) fo r d ifferent performance
measures show that the XGBoost method outperforms other
models. Collective pred iction performance and accuracy o f
individual damage severity classes. This was demonstrated
by the fact that the XGBoost method outperformed other
models in terms of group prediction performance and
individual class accuracy. These results were obtained using
the k-fo ld algorith m. Based on the findings of XGBoost
feature importance analysis, the impo rtant variables that can
accurately predict the consequences of crash damage
severity are crash type, weather conditions, road surface
conditions, damage type at location, lighting conditions, etc.
and vehicle type As a result, a co mparative XGBoost study
using different performance data showed that our model
outperforms most previous studies.
In [9] D.P. Gaikwad et al, the objective of the research is
to offer a unique approach for the identificat ion of intrusions
that is based on the ensemble method of machine learning.
The imp lementation of the intrusion detection system makes
advantage of the Bagging technique of ensemble, with
REPTree serving as the basis class. In order to raise the
accuracy of the classification and lower the number of false
positives, we have chosen the relevant characteristics from
the NSL KDD dataset. The suggested ensemble method's
performance is analyzed in terms of the accuracy of its
classifications, the amount of time it takes to create models,
and the number of false positives it produces. According to
the findings of the experiments, the Bagging ensemble
combined with the REPTree base class has the greatest
classification accuracy. Bu ild ing a model v ia the Bagging
approach results in a reduction in the amount of time needed
to complete the task. When co mpared to other approaches to
mach ine learning, the proposed ensemble method produces
false positive rates that are significantly lower than those
achieved by other methods. The proposed approach is tested
on a test dataset, and its validity is checked using a 10-fold
cross validation. Classifiers' abilit ies are evaluated in terms
of the accuracy of their classifications, the amount of time
they take to develop models, and the number of false
positives they produce. The effectiveness of the strategy is
evaluated in relation to that of other conventional machine
learning strategies. Based on the findings of the research,
the ensemble bagging machine learning approach achieves
the greatest classification accuracy of 99.67% when
subjected to 10-fold cross validation and 81.29% when
applied to the test dataset. The amount of time spent
creating the model and the number of false positives is two
of the most important aspects of the procedure. Co mpared to
the AdaBoost algorith m, which uses a decision -based
classifier and other machine learn ing strategies, the model
building time o f this method and the number of false
positives are much less. Overall, the developed ensemble
method is more efficient and provides the highest
classification accuracy with the lowest number of false
positives.
978-1-6654-7467-2/23/$31.00 ©2023 IEEE
Authorized licensed use limited to: COCHIN UNIVERSITY OF SCIENCE AND TECHNOLOGY. Downloaded on March 17,2024 at 05:11:14 UTC from IEEE Xplore. Restrictions apply.
In [10] K.M. Zubair Hasan et al, the objective of the
research is to provide a classifier that is based on an
ensemble technique in order to effectively enhance the
decision-making of existing classifiers and diagnose kidney
disease. Ensemble techniques are a way to improve
predicted performance beyond that which could be gained
fro m any of the component learning algorith ms by
combin ing a number of different learning algorith ms into
one solution. In addition, data are examined using 10-fold
cross-validation and system performance is evaluated using
receiver operating characteristic curves. Extensive testing
using the CKD dataset taken from the Un iversity of
California, Irvine Machine Learning Library shows that the
ensemble-based approach provides the best possible
performance. Unlike many conventional prediction
algorith ms, the classification accuracy of our proposed
ensemble learning method reaches 99%.
In [11] Sad i Evren Seker et al, in this research, for this
purpose, a total of 333 data sets including uniaxial
compressive strength and canvas shear force, 103 data sets
including RQD and 125 data sets including mach ine weight
were collected fro m relevant research literature. The
purpose of this study is to predict loader performance using
six d ifferent machine learn ing algorith ms and a co mbination
of different machine learning algorith ms obtained fro m the
ensemble techniques. The following algorith ms are
considered computationally useful: Zero R, random forest
(RF), Gaussian process, linear regression, logistic
regression, and mult ilayer perceptron (M LP). As a result,
MLP and RF p rovide better results compared to other
methods. Many road performance predict ion models have
been published in relevant academic literature, including
expert energy models, empirical models, and artificial
neural network models. On the other hand, these models are
rarely used as data because they mostly have little or
moderate correlation with the actual field perfo rmance of
loaders.
In [12] Nouf Rahimi et al, the objective of the research is
to further imp rove the accuracy and expand the availab ility
of FR representations, we introduce a novel ML
classification approach. This strategy uses higher precision
as the weight of the weighted group voting method. It is a
technique that combines different machine learning models.
Simp le Bayes models, support vector machines (SVM),
decision tree models, logistic regression models, and
support vector classification models create the five
combined models (SVC).The co llected data sets served as
the basis for method implementation, train ing and testing.
FR classification took 0.7 seconds and the classification
accuracy was 99.45%. SVM , SVC, simple Bayes, decision
tree and logistic regression (ML) machine learning
classifiers were used to develop this model. The proposed
ensemble using the most accurate mach ine learning based
classifiers (SVM, SVC and logistic regression) showed the
same accuracy (99.45%) as using five classifiers. The only
difference is the time, the decrease represents the
improvement when using fewer classifiers.
In [13] Xibin DONG et al, the objective of the research is
to evaluate the current state of research concerning the
conventional methods of ensemble learn ing and categorize
these methods based on the many qualit ies they exhib it. In
addition, we present challenges and potential research
832
 Proceedings of the 5th International Conference on Smart Systems and Inventive Technology (ICSSIT 2023)
IEEE Xplore Part Number: CFP23P17-ART; ISBN: 978-1-6654-7467-2

directions for each mainstream approach of ensemble good agreement with the experimental facts regarding the
learning, and we also provide an additional introduction for fluorescence wavelengths. This was shown during the
the combination of ensemble learning with various other processes of regression that were carried out on the
hotspots in machine learning, such as deep learning, fluorescence dataset. It was shown that blending had a
reinforcement learning, and so on. However, there is still stronger predictive performance for the classification o f
work to be done to better enhance the performances of liquid crystal behavior, and it provided insight into the
ensemble models, particularly in situations when the data behaviors of liquid crystals. The research came to this
includes pretty patterns. Following the completion of this conclusion as one of its results. Evidence that the linked
work, we believe that readers will have a fundamental processes in the data have been accurately defined may be
comprehension of the above current ensemble learning seen in the form of an increase in the accuracy of the
methodologies and will be able to use ensemble learning blending strategy. Although DT-based ensemble learning
fro m a variety of perspectives. We hope that by presenting models were powerful enough to consistently predict
some proposals for future ensemble learn ing paths, we may features, different DT -based ensemble learning models
be able to provide some insight on this emerging topic. produced contradictory predictions for the same co mpounds.
In [14] Xibin Dong et al, the objective of the research is This was the case despite the fact that DT-based ensemble
to analyze the current state of research concerning the learning models were used. Therefore, the inconsistency of
conventional methods of ensemble learning and to level-0 model predictions may be overcome by blending
categorize such methods according to their many defining techniques so long as more than two levels of models made
features. In addition, we present challenges and potential correct forecasts.
research directions for each mainstream approach of After literature survey it has been observed that both
ensemble learning, and we also provide an additional bagging and boosting are techniques that have the benefit of
introduction for the co mbination of ensemble learning with enhancing model stability while simultaneously reducing
various other critical areas in machine learning, such as deep variation. In addition, these strategies may be utilized to
learning, reinfo rcement learn ing, and so on. They presented deal with imbalanced datasets and provide more accurate
a range of concerns as well as potential research predictions. By training nu merous models on different
opportunities, regarding ensemb le learn ing. However, there subsets of the data, it is feasible to significantly reduce the
is still work to be done to better enhance the performances amount of time spent "overfitting" the data by using bagging
of ensemble models, particularly in situations when the data and boosting methods. Both bagging and boosting are
includes intricate patterns. Following the completion of this examples of algorith ms that may assist make models more
work, we believe that readers will have a fundamental interpretable by showing which attributes are most essential
comprehension of the aforementioned current ensemb le for the predictions.
learning methodologies and will be able to use ensemble So me drawbacks are also observed, that real-time
learning fro m a variety of perspectives. We hope that by applications are not a good fit for bagging and boosting
presenting some proposals for future ensemble learning algorith ms since these techniques are co mputationally costly
paths, we may be able to provide some insight on this and require a large number of training data points. As a
emerging topic. result, these algorithms are not well suited for real-time
In [15] Chia Hsiu Chen et al, the objective o f the research applications. Bagging and boosting are two different types
is when applied to binary classification and regression of algorithms, but both involve training mu ltiple models,
modeling tasks, they investigated the levels of predictability which may be costly co mputationally. In the event that the
and interpretability provided by four standard and well - models are not tuned appropriately, there is a possibility that
established ensemble learning methods (Random forest, they may overfit the data. Given that bagging and boosting
extreme randomized forests, adaptive boosting, and gradient algorith ms use many models, the results they provide might
boosting). After then, the approaches for blending were be difficult to understand.
established by assembling the main co mponents of four T ABLE 1. EXIST ING APPROACH LIMIT AT IONS
various ways of learning in an ensemble. The blending Author Publication Method / Limitations
Name with Year Algorithm
strategy was able to bring about an increase in performance
Used
as well as a unity of interpretation since it comp iled the
Emine Hindawi, Ensemble -
unique predictions that were given by the several learning Yaman et al 2019 classifiers
models. Th is research presents an in-depth assessment of the [1]
major parts of t wo case studies that supplied us with so me Omer Sagi Wiley, 2018 Ensemble Improve
valuable informat ion on co mpound properties. The case et al [4] methods standard
studies were carried out by the same researchers that algorithms to
conducted the previous research. These research findings make them
were used in order to provide us with the aforementioned more efficient
informat ion. It's possible that QSPR modeling with and
appropriate for
interpretable machine learn ing algorithms might help
"Big Data"
enhance chemical design so that it functions more Thomas IEEE, 2020 - One of our
efficiently, generate knowledge, and test ideas for better Rincy N et goals is to
results. According to the findings that were obtained, the al [5] implement a
combination of the QC descriptors and the fluorescence model based
dataset resulted in the production of a model that not only on machine
had good predictability and interpretability, but also had learning that

978-1-6654-7467-2/23/$31.00 ©2023 IEEE 833

Authorized licensed use limited to: COCHIN UNIVERSITY OF SCIENCE AND TECHNOLOGY. Downloaded on March 17,2024 at 05:11:14 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the 5th International Conference on Smart Systems and Inventive Technology (ICSSIT 2023)
IEEE Xplore Part Number: CFP23P17-ART; ISBN: 978-1-6654-7467-2

makes use of IV. METHODOLOGY

ensemble
learning There are many ensemble machine learning techniques
strategies. available but will discuss about the three widely used
Debachuda IEEE, 2019 Predictive The techniques: Bagging, Boosting & Stacking.
mani Prusti Classification Identification 1). Bootstrap Aggregation (Bagging)
et al [7] model by of fraud is Bagging, also known as bootstrap aggregation, is the
Ensemble dependent on process of combining numerous versions of a projected
methods the availability model. Each model is trained separately and then averaged
of real-time before being integrated. The process of bagging is useful to
datasets. If
many various types of machine learning models, such as
financial
organizations, decision trees, regression models, and neural networks,
make the amongst others. The basic objective of bagging is to achieve
sensitive less variation than any individual model.
datasets of When it comes to bagging, there are a few different
different approaches to produce less variance than any individual
fraudulent model:
operations  Use more models - When we utilize additional
public at all. models, the amount of variance in our results will
D.P. IEEE, 2015 Bagging When a model decrease.
Gaikwad et Ensemble identifies  Use a larger dataset. When we have a larger dataset,
al [9] method suspicious we will find that there is less variance.
packets, it  Use a more broad range of models. When we apply
sounds an several different models, we will see that there is a
alarm to warn trend toward less variance in the results.
the network  Emp loy a more advanced approach to the bagging
administrator. process. Some techniques for bagging, such as
All standard random subspace bagging, are able to minimize
packets are variance to an even greater extent.
permitted
 The variance may be reduced by bagging without
inside the
introducing any bias into the predictions.
system.
Unknown When learners are unstable and liable to overfit, bagging is
packets are particularly effective. Parallel bagging may be performed to
stored for prevent the waste of unnecessary computing resources. This
future research is one of its benefits, and it frequently serves as a catalyst for
in a log file. expanding the algorithm's use in a range of domains. The
bagging method is used extensively in a number of
In [4] Conventional algorith ms may be made more applications, including the financial sector, health-care data
effective and suitable for use with b ig data will depend on predictions, high-dimensional data, terrestrial cover
the algorith ms individually as well as the datasets to which mapping, scam detection, network intrusion detection
they are applied. One technique to make standard algorithms systems, and several medical fields, such as neuroscience
more suitable for use with big data is to enhance their and prosthetics, etc.
Bagging is represented using the following formula:
efficiency. Standard algorith ms for big data can be
improved in a nu mber of ways, including the fo llowing: f bagging = f 1 (Z) + f2 (Z) +........... + f n (Z) [eq.1]
making use of techniques for parallel co mputing in order to
speed up computation; designing algorith ms that are created In above formula, the keyword on the left represents the
specifically to the structure of the dataset; making use o f bagged prediction, while the terms on the right represent the
sampling techniques in order to reduce the amount of data individual learners.
that needs to be processed; and making use of _________________________________________________
Bagging Algorithm
approximation methods in order to get results more quickly.
In [5] the solution is to develop a mach ine learning model
that, in order to improve the accuracy of its predictions, will Input: dataset X = {(a 1 , b1 ), (a2 , b2 ), ……, (an, bn)}
make use of ensemble learning techniques. Base Learning Algorithm £;
In [7] the solution is it will be easier to identify potential Number of Base Learners L.
fraudsters. If financial organization makes the sensitive
datasets of different fraudulent operations, the data should Process:
be availab le with principle of anonymity and it wou ld be for c = 1, ……, L;
easier to identify fraud. hc = € (X, Xbs ) % Xbs is the bootstrap distribution
In [9] the solution for future research work is ensemble Exit
algorith ms, wh ich can be used is random forest for unknown
Output = I (a) = arg b€Y , max ∑ ( ( )
packets which is stored in log file.

2). Boosting

978-1-6654-7467-2/23/$31.00 ©2023 IEEE 834

Authorized licensed use limited to: COCHIN UNIVERSITY OF SCIENCE AND TECHNOLOGY. Downloaded on March 17,2024 at 05:11:14 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the 5th International Conference on Smart Systems and Inventive Technology (ICSSIT 2023)
IEEE Xplore Part Number: CFP23P17-ART; ISBN: 978-1-6654-7467-2

Boosting is a method used in ensemble learn ing to Zx+1 = Modify_Distribution (Zx , € x )

improve the accuracy of a model by co mbining the Exit
predictions of numerous models. The purpose of boosting is _________________________________________________
to transform weak learners into strong learners via multip le
iterations. The ensemble model in boosting is the weighted 3). Stacking
sum of all co mponent base learners. Adaptive Boosting Stacking is one of the most often used ensemble machine
(AdaBoost) and Gradient Boosting are two conceptual in learning approaches for predicting many nodes in order to
boosting that vary in how the base models are aggregated. construct a new model and enhance model performance.
Stacking allo ws us to train many models to address
Boosting Algorithm comparable issues, and based on their co mbined output, it
constructs a model with enhanced performance. In stacking,
Input = Sample Distribution Z an algorithm takes the outputs of sub-models as input and
Base Learning Algorithm £ tries to learn the optimal way to combine the input
predictions to get a more accurate output prediction.
Number of base learners X Stacking, often referred to as a stacked generalization, is an
expanded variant of the Model Averaging Ensemble
Process: approach in which all sub-models participate equally
Z1 = Z % Init Distribution according to their performance weights and construct a new
for x = 1, ….., X model with imp roved predictions . Stacking is the name of
cx = € (Zx ); % this new model since it is layered on top of the others.
€ x = Q y- Zx cx ((x) ≠ f(x))
V. COMPARAT IVE ANALYSIS
T ABLE II. COMPARAT IVE ANALYSIS
Factor Bagging Boosting Stacking
Complete Form Bootstrap Aggregating As its name indicates As its name indicates
Approach Consider mostly on Consider mostly on Considers heterogeneous
homogenous weak learners. homogenous weak learners. weak learners (combines
different learning
algorithms)
Bias & Variance The objective of bagging is Boosting is used when the Stacking is also used when
not to eliminate bias; rather, goal is to eliminate bias. the objective is to eliminate
it is to maintain bias stability bias.
and reduce variation.
Adaptability Bagging is most effectively Boosting works particularly A number of different weak
used to base models that effectively with less capable learners are fitted in
have a low bias but a large learners who have low separation from one another,
variance. variance but large bias. and then a meta-model is
trained on top of that to
predict outputs depending on
the outputs that are produced
by the base models.
Implementation Implementing the Boosting is a strategy Stacking is a practical
following algorithms will that may be put into practice concept that may be realized
allow for the practical use of by using one of the by the use of the following
bagging: following algorithms: algorithms:
1) Random forest 1) AdaBoost (Canonical 1) Stacked Models
Boosting) (Canonical Stacking)
2) Canonical Bagging
2) Gradient Boosting 2) Blending
3) Extra Trees
3) Stochastic gradient 3) Super Ensemble
boosting (XGBoost)
Type "Bagging" is an example of Boosting is an example of a Stacking is a method that
a parallel ensemble method Sequential ensemble trains many models
in which each model is approach. In this method, simultaneously and then
developed individually. each successive model integrates the results by
attempts to fix the errors that training a meta-model to
were made by the model that provide a prediction that is
came before it. based on the predictions of
multiple weak models.

978-1-6654-7467-2/23/$31.00 ©2023 IEEE 835

Authorized licensed use limited to: COCHIN UNIVERSITY OF SCIENCE AND TECHNOLOGY. Downloaded on March 17,2024 at 05:11:14 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the 5th International Conference on Smart Systems and Inventive Technology (ICSSIT 2023)
IEEE Xplore Part Number: CFP23P17-ART; ISBN: 978-1-6654-7467-2

Subset Selection Method During the bagging process, In the boosting procedure, It trains each individual
various training data subsets additional sub-datasets are dataset by applying the
are selected at random, and selected at random from the knowledge gained from the
then the original dataset is weighted dataset, and then overall dataset.
used to make replacements. replacement takes place.
Dependencies In Bagging, each model is When boosting, the It illustrates how a
developed independently performance of subsequent generalized model is created
from the others. models is dependent on the by integrating a number of
performance of the model different models into one.
that came before it.
Performance If the issue is overfitting, If you have an issue with Addressing overfitting in the
bagging would be the best underfitting, boosting might models used for stacking
option. provide better results. seems to be the superior
strategy.
Takeaways Bagging is the process of Boosting consists of Stacking includes fitting
fitting multiple decision sequentially adding many kinds of models to the
trees to separate samples of ensemble techniques that same data and use a second
the same dataset and correct the predicted output model to determine the
averaging the resulting provided by preceding optimal way to combine the
predictions. models and producing a predictions.
weighted average of the
predictions.
[2] X. Gao, C. Shan, C. Hu, Z. Niu, and Z. Liu, “An Adaptive Ensemble
CONCLUSION & FUTURE WORK Machine Learning Model for Intrusion Detection,” IEEE Access, vol.
The ability to mix the results of many different models is 7, pp. 82512–82521, 2019, doi: 10.1109/ACCESS.2019.2923640.
one of the primary reasons why ensemble techniques are so [3] T. G. Dietterich, “Ensemble methods in machine learning,” Lect. Notes
significant in the field of machine learning. Since of this, the Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes
Bioinformatics), vol. 1857 LNCS, pp. 1–15, 2000, doi: 10.1007/3-540-
performance of an ensemble of models may often be 45014-9_1.
enhanced over that of a single model because the ensemble [4] O. Sagi and L. Rokach, “Ensemble learning: A survey,” Wiley
can learn fro m the benefits and drawbacks of each individual Interdiscip. Rev. Data Min. Knowl. Discov., vol. 8, no. 4, pp. 1–18,
model. Because the mixture of models may produce a more 2018, doi: 10.1002/widm.1249.
accurate model than any single model, ensemble approaches [5] N. Thomas Rincy and R. Gupta, “Ensemble learning techniques and its
can also assist in lowering the likelihood of overfitting efficiency in machine learning: A survey,” 2nd Int. Conf. Data, Eng.
occurring..The most common types of mistake that may Appl. IDEA 2020, 2020, doi: 10.1109/IDEA49133.2020.9170675.
occur in learning models are noise, variation, and bias. The [6] M. Zounemat-Kermani, O. Batelaan, M. Fadaee, and R. Hinkelmann,
ensemble approaches that are used in machine learning assist “Ensemble machine learning paradigms in hydrology: A review,” J.
Hydrol., vol. 598, no. December 2020, 2021, doi:
to reduce the impact of these error-causing factors, which in 10.1016/j.jhydrol.2021.126266.
turn helps to ensure that the machine learning (ML)
[7] D. Prusti and S. K. Rath, “Fraudulent Transaction Detection in Credit
algorithms are accurate and reliable. Card by Applying Ensemble Machine Learning techniques,” 2019 10th
The main objective of this research is to determine why Int. Conf. Comput. Commun. Netw. Technol. ICCCNT 2019, pp. 6–11,
to use ensemble machine learning techniques instead of 2019, doi: 10.1109/ICCCNT45670.2019.8944867.
single machine learning models. [8] A. Jamal et al., “Injury severity prediction of traffic crashes with
And we just covered ensemble machine learning ensemble machine learning techniques: a comparative study,” Int. J.
techniques and also compared all the factors of bagging, Inj. Contr. Saf. Promot., vol. 28, no. 4, pp. 408–427, 2021, doi:
boosting, and stacking because our paper is only discussing 10.1080/17457300.2021.1928233.
three main techniques of ensemble machine learning [9] D. P. Gaikwad and R. C. T hool, “Intrusion detection system using
bagging ensemble method of machine learning,” Proc. - 1st Int. Conf.
techniques; other methods are out of the scope of th is Comput. Commun. Control Autom. ICCUBEA 2015, pp. 291–295,
research. 2015, doi: 10.1109/ICCUBEA.2015.61.
Future research opportunities include improving the [10] K. M. Z. Hasan, “Perfomance Evaluation of Ensemble-Based Machine
accuracy of the predictions produced by ensemble models , Learning Kidney Disease”, [Online]. Available:
using ensemble models while simu ltaneously reducing the http://dx.doi.org/10.1007/978-981-13-5953-8_34.
computational costs of training, creating new ensemble [11] S. E. Seker and I. Ocak, “Performance prediction of roadheaders using
methods that are more efficient than the approaches that are ensemble machine learning techniques,” Neural Comput. Appl., vol.
already in use and finally applying the use of ensemble 31, no. 4, pp. 1103–1116, 2019, doi: 10.1007/s00521-017-3141-2.
[12] N. Rahimi, F. Eassa, and L. Elrefaei, “An ensemble machine learning
methods to non-traditional methods of machine learning, technique for functional requirement classification,” Symmetry
such as unsupervised learning and reinforcement learning . (Basel)., vol. 12, no. 10, pp. 1–26, 2020, doi: 10.3390/sym12101601.
[13] X. Dong, Z. Yu, W. Cao, Y. Shi, and Q. Ma, “A survey on ensemble
REFERENCES learning,” Front. Comput. Sci., vol. 14, no. 2, pp. 241–258, 2020, doi:
[1] M. A. Yaman, F. Rattay, and A. Subasi, “Comparison of Bagging and 10.1007/s11704-019-8208-z.
Boosting Ensemble Machine Learning Methods for Face [14] X. Dong, Z. Yu, W. Cao, Y. Shi, and Q. Ma, “A survey on ensemble
Recognition,” Procedia Comput. Sci., vol. 194, pp. 202–209, 2021, learning,” Front. Comput. Sci., vol. 14, no. 2, pp. 241–258, 2020, doi:
doi: 10.1016/j.procs.2021.10.074. 10.1007/s11704-019-8208-z.
[15] C. H. Chen, K. Tanaka, M. Kotera, and K. Funatsu, “Comparison and
improvement of the predictability and interpretability with ensemble

978-1-6654-7467-2/23/$31.00 ©2023 IEEE 836

Authorized licensed use limited to: COCHIN UNIVERSITY OF SCIENCE AND TECHNOLOGY. Downloaded on March 17,2024 at 05:11:14 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the 5th International Conference on Smart Systems and Inventive Technology (ICSSIT 2023)
IEEE Xplore Part Number: CFP23P17-ART; ISBN: 978-1-6654-7467-2

learning models in QSPR applications,” J. Cheminform., vol. 12, no. 1, Teja, “Inhibiting Webshell Attacks by Random Forest Ensembles with
pp. 1–16, 2020, doi: 10.1186/s13321-020-0417-9. XGBoost,” J. Inf. Technol. Digit. World, vol. 4, no. 3, pp. 153–166,
[16] D. Sasikala, D. Chandrakanth, C. Sai Pranathi Reddy, and J. Jitendra 2022, doi: 10.36548/jitdw.2022.3.003.

978-1-6654-7467-2/23/$31.00 ©2023 IEEE 837

Authorized licensed use limited to: COCHIN UNIVERSITY OF SCIENCE AND TECHNOLOGY. Downloaded on March 17,2024 at 05:11:14 UTC from IEEE Xplore. Restrictions apply.