Exercise #5: Implementing Binary Logistic Regression model
using Neural Networks with Cross-Validation
John Vincent M. Cabugnason
Computer Science and Engineering
Jose Rizal University
Mandaluyong, Philippines
johnvincent.cabugnason@my.jru.edu
Abstract
This study focuses on improving the predictive performance of
models for complex binary classification problems in insurance. It
explores the use of neural network architectures, the selection of
appropriate activation functions, configuration of loss functions,
and regularization techniques to prevent overfitting. These
techniques aim to enhance model interpretability and accuracy for
different classification tasks.
Keywords — Binary Logistic Regression, Neural Networks,
Machine Learning, Classification models
I. INTRODUCTION
A. Conduct some research and introduce the problem of
Binary Logistic regression (BLogReg).
According to Pramoditha, R. (2022), [1] Logistic
regression is a neural network model with no hidden layers
[1]. Binary Logistic Regression, or BLogReg for short, is a
statistical technique that helps us understand how one or more
factors can impact a simple "yes" or "no" outcome. In other
words, it's a way to figure out what factors are most likely to
influence whether something happens or not. For example,
BLogReg can help us predict whether a customer will buy a
product or not based on certain characteristics, such as age,
gender, and income. In simpler terms, the purpose of
BLogReg is to figure out the likelihood of something
happening or not happening based on certain factors. For
example, we might use BLogReg to predict the likelihood of a
customer buying a product based on their age, gender, and
income. By analyzing these factors, we can estimate the
probability of the customer making a purchase or not.
Over the last ten years, there has been a significant rise in
the use of new technologies and big data analytics. These
advancements have started to change the way businesses
approach predictive modeling. In other words, companies are
now using these new tools and techniques to better understand
trends and patterns in their data, which can help them make
more informed decisions. This shift has impacted many
different industries, and we are likely to see even more
changes in the years to come. In [4], We consider the problem
of predicting the probability that an insurance claim will be
filed in a French motor third party liability (MTPL) insurance
portfolio consisting of features describing the vehicle, the
driver and the geographic area of the car registration and the
number of insurance claims observed during one accounting
year [4].
B. Explain the motivation for implementing BLogReg using
neural networks.
The concept of using machine learning algorithms to save
lives has a powerful impact on people. This study specifically
focuses on addressing the biggest threats to human health and
the leading causes of death. By using advanced technology
like machine learning, researchers hope to find new ways to
prevent or treat these health issues, ultimately improving
people's quality of life and saving lives. The potential for this
kind of research to make a difference is truly remarkable and
has the potential to change the world for the better. In [2],
Data set is generated by combining different type of sources,
in order to identify factors that can classify the status of the
smoker as smoker, former smoker and non-smoker. 33
Attributes were used with 10000 instances, class problem was
divided in three classes Smoker, Non-Smoker and Former
Smoker. The significance of an attribute obtained using the
odds ratio, and factors were identified using classification with
10-fold cross validation combined with 1-1 and 1-against all
method [2].
II. LITERATURE REVIEW
“Logistic regression and artificial neural network
classification models: a methodology review”
In the medical field, predictive models are being
increasingly used to help with diagnosing and predicting the
progression of various health conditions. These models are
designed to analyze data and identify patterns that can help
doctors make more accurate diagnoses and predictions about a
patient's health. By using these advanced tools, medical
professionals can provide better care and treatment options
that are tailored to each patient's unique needs. According to
Dreiseitl S. and Ohno-Machado L. (2003), [4] These models
are built from ‘‘experience’’, which constitutes data acquired
from actual cases [4]. The data can be prepared and organized
in different ways. In some cases, it can be transformed into a
set of rules, which is commonly seen in knowledge-based
expert systems. Alternatively, the data can also be used as
training material for statistical and machine learning models.
These models learn from the data to make predictions and gain
insights that can be valuable in various fields, including
healthcare.
In [3], There are now several implementations of
predictive modeling algorithms readily available, both as free
and commercial software [3]. The accuracy of the results
produced by these models relies on three main things: the
quality of the data used to create the model, how carefully the
model's adjustable parameters were selected, and the criteria
used to assess the model's performance. It's important to use
high-quality data when building a model, and to choose the
right settings for the model's parameters. Additionally, the
criteria used to evaluate the model's performance should be
carefully chosen to ensure that the results are meaningful and
accurate. By paying attention to these factors, we can create
models that are more reliable and effective in various fields,
including healthcare.
In [3], The task of classifying data is to decide class
membership y of an unknown data item x0 based on a dataset
D = (x1; y1),....,(xn, yn) of data items xi with known class
memberships yi [3]. There are two main ways to classify data.
The first way is to simply divide the data into two groups and
label them as either 0 or 1. This is known as dichotomous
classification. The second way is more complex, as it involves
modeling the probability of a data item belonging to a
particular class. This approach not only assigns a class label to
the data item, but also provides a probability of its
membership in that class. This probability can be useful in
many applications, including healthcare, where it can help
doctors make more accurate diagnoses and treatment
decisions.
Logistic regression, artificial neural networks, k-nearest
neighbors, and decision trees are all methods used to model
the probability of a data item belonging to a particular class.
These methods are part of the second approach to data
classification mentioned earlier. However, they differ in how
they approximate the probability based on the data. Each
method has its own strengths and weaknesses, and is better
suited for certain types of data and applications. By
understanding the differences between these methods, we can
choose the best one for a particular problem and obtain more
accurate results. In [4], Currently, logistic regression and
artificial neural networks are the most widely used models in
biomedicine, as measured by the number of publications
indexed in MEDLINE: 28,500 for logistic regression, 8500 for
neural networks, 1300 for k- nearest neighbors, 1100 for
decision trees, and 100 for support vector machines.
“Enhancing Logistic Regression Using Neural
Networks for Classification in Actuarial Learning”
The past decade has seen a surge in new technologies and
big data analytics, which have begun to transform the
landscape of predictive modeling in many enterprises. These
new technologies have made it possible to collect and analyze
vast amounts of data, which has led to the development of
more accurate and sophisticated predictive models. One of the
most significant developments in predictive modeling in
recent years has been the rise of machine learning. Machine
learning algorithms can learn from data without being
explicitly programmed, which makes them ideal for tasks such
as predicting customer behavior or identifying fraud.
According to Tzougas G. and Kutzkov K (2023), [4] The
ML methods which have been used so far for efficiently
addressing alternative regression and classification problems
in insurance include, for example, XGBoost, random forest
(RF), decision trees (DTs), naïve Bayes, K-nearest neighbor
(K-NN) model, AdaBoost (AB) model, stochastic gradient
boosting (SGB) model, support vector machine (SVM) model
and NNs [4]. In addition, a very interesting and novel research
direction is the combined actuarial neural network (CANN).
CANN is a type of neural network that is specifically designed
for actuarial applications. It combines the strengths of both
traditional actuarial methods and machine learning techniques.
All traditional actuarial models can be implemented in a
neural network. In particular, the CANN approach can be seen
as a neural network that boosts the traditional actuarial
regression model.
Another major development has been the growth of big
data. Big data refers to the large and complex datasets that are
now being generated by businesses and organizations. These
datasets can be used to train machine learning algorithms and
develop more accurate predictive models. A logistic regression
model is a type of statistical model that can be used to predict
the probability of an event occurring. It is a linear model,
which means that the predicted probability is a linear
combination of the explanatory variables. The model is
constrained to produce outputs between 0 and 1, which
represents the probability of the event occurring and not
occurring, respectively. According to Tzougas G and Kutzkov
K (2023), This approach enables us to consider several
extensions of the logistic regression model and utilize methods
for optimization and regularization that come as part of
platforms for neural network training [2].
The use of predictive modeling is becoming increasingly
widespread across a range of industries. For example, banks
use predictive modeling to identify customers who are at risk
of defaulting on their loans, and retailers use it to predict
which products are likely to be purchased by a given
customer. Overall, the past decade has seen a significant
transformation in the field of predictive modeling. New
technologies and big data analytics have made it possible to
develop more accurate and sophisticated predictive models.
“Neural Networks for Macroeconomic Forecasting:
A Complementary Approach to
Linear Regression Models”
Over the past few years, there has been a growing interest
among economists in neural networks. These powerful tools
have caught the attention of macroeconomic forecasters due to
their ability to identify and replicate both simple and complex
connections between different variables. After examining
many studies, it appears that neural networks are often better
at predicting economic output and financial variables like
stock prices than traditional linear models when it comes to
forecasting for the future. It has been discovered that the
neural network is better at predicting outcomes than a well-
known linear regression model that was created by the
department. In fact, the neural network has been found to
reduce errors by anywhere from 13 to 40 percent. Although
several tests have been conducted, they suggest that there isn't
enough proof to show that the increase in accuracy when using
neural networks for forecasting is statistically significant.
According to Gonzales S. (2000-07), [5] the human brain
is the most complex computer known to us, and in an effort to
gain a deeper understanding of it, researchers have sought to
replicate its various abilities through the development of
artificial intelligence [5]. According to recent research, neural
networks have the potential to be effective in predicting
volatile financial variables that are typically challenging to
forecast using conventional statistical methods. The author
suggests that the most basic forms of neural networks are
closely related to standard econometric techniques. In this
paper, the author will draw comparisons between neural
networks and econometric methods to help readers who are
familiar with econometrics better comprehend the topic. By
gaining a better understanding of neural networks, economists
can determine whether these models are useful for
macroeconomic and financial forecasting.
In [5], As cognitive scientists studied the brain and its
ability to learn, they identified some key characteristics that
seemed particularly important to the brain's success. Based on
these attributes, neural networks were developed as a
framework. To gain a deeper comprehension of these
networks, it is beneficial to briefly explore these essential
characteristics of the brain. According to Gonzales S. (2000-
07), the brain consists of billions of basic units known as
neurons that are arranged in an extensive network. These
neurons are believed to carry out the relatively uncomplicated
function of selectively transmitting electrical impulses to one
another [5].
“Comparison of artificial neural network and binary
logistic regression for determination of impaired
glucose tolerance/diabetes”
Since the inception of computing, artificial intelligence has
been suggested as a means of utilizing reasoning to assist in
clinical decision-making. According to Kazemnedjad et al.,
2008, Artificial neural networks are a computer modeling
technique that takes inspiration from the observed behaviors of
biological neurons [6]. Discriminant analysis and logistic
regression are statistical methods that have traditionally been
used to create models for clinical diagnosis and treatment.
However, recent studies have shown that artificial neural
networks can enhance prediction accuracy in various
situations, such as predicting the prognosis of breast cancer in
women post-surgery, developing models for surgical decision-
making for traumatic brain injury patients, and predicting
survival rates of alcoholic patients with severe liver disease.
Nevertheless, some studies have reported that artificial neural
networks and statistical models produce comparable results.
Binary logistic regression showed that all factors were
significantly associated with glucose tolerance status. Age,
sex, BMI, and WHR were significant risk factors for diabetes
mellitus (DM). Additionally, those with hyperlipidemia or
hypertension had a higher risk of DM and impaired glucose
tolerance (IGT). The mean response is the probability of a
binary outcome, such as whether or not a person has a disease.
The logit function is a transformation of the probability that
maps it to a linear scale. This allows us to use linear regression
to model the relationship between the logit of the probability
and the predictors. The predictors can be continuous variables,
such as age or weight, or categorical variables, such as gender
or race. The model can be used to predict the probability of the
outcome for a given set of predictors. It can also be used to
identify the factors that are most strongly associated with the
outcome. However, it is important to note that logistic
regression is a statistical model, and it is not a perfect
predictor. The results of a logistic regression model should be
interpreted with caution. In [5], If the only objective is to
make accurate predictions, then neural networks may be a
better choice than binary logistic regression. However, if it is
important to understand the relationship between the
predictors and the outcome, then binary logistic regression
may be a better choice.
Neural networks and binary logistic regression are both
machine learning models that can be used for classification or
prediction. Neural networks are typically more complex than
binary logistic regression, but they can be used to model more
complex relationships between the predictors and the outcome.
Binary logistic regression is often more interpretable than
neural networks, and it can be used to identify the effect of
individual factors on the outcome. The choice of which model
to use depends on the specific objectives of the analysis.
A. Summarize and Discuss the Highlights of the related
papers.
The four papers discuss the application of machine
learning models for classification and prediction. Logistic
regression and artificial neural networks are popular machine
learning models known for their effectiveness in various
domains. Logistic regression is a statistical model that predicts
the probability of an event occurring by combining
explanatory variables linearly. It produces outputs between 0
and 1, representing the probabilities of the event occurring or
not.
On the other hand, artificial neural networks are inspired
by the human brain and consist of interconnected nodes
capable of learning patterns in data. They are commonly used
for tasks like image recognition and natural language
processing. Both logistic regression and artificial neural
networks have strengths and weaknesses. Logistic regression
is simple and interpretable but may struggle with complex
problems and numerous predictors. Neural networks can
handle complex relationships but are harder to interpret and
prone to overfitting.
The choice between these models depends on the analysis
objectives. Neural networks are suitable for accurate
predictions, while logistic regression is better for
understanding predictor-outcome relationships. Other machine
learning models like decision trees, support vector machines,
and k-nearest neighbors can also be used for classification and
prediction, depending on the data characteristics and analysis
goals.
In conclusion, machine learning models are powerful tools
for accurate predictions and understanding variable
relationships. However, selecting the appropriate model is
crucial for the specific task at hand.
III. METHODOLOGY
In the dataset that I have, it has the sex, smoker, region,
age, bmi, children, and charges as my attributes where helps
me for predicting the insurance charges for smoking. The
features in this dataset may be used to predict or analyze the
target variable, which is likely to be the medical charges. This
dataset can be used for various purposes, such as identifying
factors that affect medical charges or developing models to
predict medical charges based on the given features.
A. Explain the preprocessing steps needed to prepare the
dataset for BlogReg modeling.
Before using the dataset for BlogReg modeling, several
steps need to be taken to prepare it. Firstly, the data needs to
be cleaned by removing any missing or irrelevant information
and addressing any outliers. Next, the most relevant features
that are likely to affect the target variable should be selected.
The data also needs to be normalized and scaled to ensure that
all features are on the same scale. Finally, the dataset should
be split into training and testing sets to evaluate the model's
performance on new data. These steps help ensure that the
data is appropriately prepared for modeling and that the model
performs well when applied to new data.
B. Detail the initial neural network architecture that can be
used for BLogReg.
To create an initial neural network architecture for
BlogReg, we can start with an input layer that has nodes
corresponding to the number of features in the dataset. We
can then add one or more hidden layers with different
numbers of nodes, using activation functions like relu or
sigmoid to introduce non-linearity. Finally, we'll have an
output layer with a single node that represents the
probability of the target variable. The network will be
trained using techniques like backpropagation and
stochastic gradient descent to optimize the model's
performance.
C. Discuss your strategy for adjusting the different
hyperparameters of a Neural Network to get the best
model. Limit only to the following Hyperparameters:
a. Learning Rate – 0.001
b. Epochs
c. Number of Neurons (10,20,30,40) for each Hidden
Layer
d. Training/Testing Size (Set Training and Testing Size to
70/30 only)
D. Discuss the different hyperparameters of a Neural
Network
Neural Networks are complex models that require careful
tuning to perform well. One of the key factors that impact their
performance is the learning rate, which determines how much
the model's weights are updated as it learns from the data. If
the learning rate is set too high, the model can overshoot the
optimal weights and perform poorly. On the other hand, if the
learning rate is too low, the model may take too long to
converge and may not learn effectively. So finding the right
balance is crucial for achieving good performance.
The hidden layers of a Neural Network are like the brain of
the model, where it learns to recognize patterns in the data.
The size of these layers is important, as a larger layer can learn
more complex features, but may also require more data to B. Display and analyze the performance charts of the top-
avoid overfitting. Additionally, the activation function used in performing model across all epochs.
each layer can have a big impact on how the model learns.
C. For additional points: Display and analyze the training
Different functions have different strengths and weaknesses,
chart for a specific fold.
so selecting the right one is crucial for achieving good
performance. By carefully tuning these hyperparameters, we
can ensure that our Neural Network is learning the most
important features from the data and making accurate
predictions.

IV. RESULTS
A. Run your chosen dataset in the NN and present the results
of the implementation.

V. ANALYSIS

A. Interpret the results based on your observation and

discuss whether or not the top model is showing signs of
overfitting.

The results show that the model was able to predict the
training examples in each fold perfectly, achieving an
accuracy score of 1.0000. This suggests that the model was
able to learn the training data well. However, it is important to
keep in mind that this doesn't necessarily mean that the model
will perform just as well on new data that it hasn't seen before.
To be confident that the model can make accurate
predictions on new data, we need to evaluate its performance
on unseen data. This is because sometimes a model can be too
good at learning the training data and become too specialized,
making it difficult to generalize well to new data. Therefore,
we need to evaluate the model's performance on validation or
test data to see if it can make accurate predictions on data that
it hasn't seen before.
Although the model achieved a perfect accuracy score on
the training data, it doesn't necessarily mean that it will
perform just as well on new data. To be sure that the model
can make accurate predictions on new data, we need to
evaluate its performance on validation or test data that it hasn't
seen before.
By assessing the model's performance on a separate
validation or test set, we can determine whether the model is
overfitting or not. If the model shows high accuracy on both
the training and validation/test sets, it suggests that the model
is not overfitting and can generalize well to new data. This
means that the model can make accurate predictions on new
data that it hasn't seen before, which is crucial for its practical
use.
B. Discuss any challenges or limitations you encountered
during the experiments.
During the experiments, one challenge I encountered
was the issue of consistently obtaining a train accuracy of
1.00. This means that the model was able to perfectly
predict the training examples in each fold, which may
seem ideal at first. However, it raised concerns about
potential overfitting and the generalizability of the model
to unseen data. Another challenge I faced during the
experiments was the lack of information about the model's
performance on unseen data. This means that I didn't have
enough data to fully evaluate the model's effectiveness in
making accurate predictions on new data that it hasn't seen
before. This made it difficult to determine if the model was
truly effective and reliable in real-world scenarios.
Another challenge I encountered was the lack of
detailed information about the dataset used for training and
validation. Having a clear understanding of the
characteristics and distribution of the data is essential in
evaluating the model's performance and identifying any
potential biases or limitations. Without this information, it
was challenging to assess the reliability and
generalizability of the model's predictions. It would have
been beneficial to have insights into the dataset to ensure
that the model was trained on diverse and representative
data, which would enhance its ability to perform well in
real-world scenarios.
VI. CONCLUSION
A. In your own words, summarize the things that you learned
in this exercise.
In this paper, I delve into the world of Binary Logistic
Regression (BLogReg), a powerful statistical tool that allows
us to understand how different factors can influence a simple
"yes" or "no" outcome. Imagine being able to predict whether
a customer will buy a product or not based on their age,
gender, and income. That's exactly what BLogReg enables us
to do. It's like peering into the future and estimating the
probability of an event happening or not happening based on
specific factors.
This paper highlights the incredible potential of this
combination, especially when it comes to healthcare. By
leveraging advanced technology like machine learning,
researchers aim to tackle the biggest threats to human health
and find innovative ways to prevent or treat these issues.
Imagine the impact it could have on saving lives and
improving people's quality of life. Furthermore, the paper
explores how BLogReg and neural networks have been
applied in various fields, such as medical diagnosis, accurate
predictive modeling, and even economic forecasting. It's
fascinating to see how these techniques are revolutionizing
industries and shaping the future of data analysis.
REFERENCES
[1] Pramoditha, R. (2022, May 19). Replicate a logistic regression
model as an artificial neural network in keras. Towards Data
Science. https://towardsdatascience.com/replicate-a-logistic-
regression-model-as-an-artificial-neural-network-in-keras-
cd6f49cf4b2c
[2] Dreiseitl, S., & Ohno-Machado, L. (2002). Logistic regression
and artificial neural network classification models: a
methodology review. Journal of Biomedical Informatics, 35(5–
6), 352–359. https://doi.org/10.1016/s1532-0464(03)00034-0
[3] No title. (n.d.). Aip.org. Retrieved October 14, 2023, from
https://pubs.aip.org/aip/acp/article/2203/1/020036/961282/Classi
fication-of-the-factors-for-smoking
[4] Tzougas, G., & Kutzkov, K. (2023). Enhancing logistic regression using
neural networks for classification in actuarial learning. Algorithms,
16(2), 99. https://doi.org/10.3390/a16020099
[5] Gonzalez, S. (n.d.). Neural networks for macroeconomic forecasting: A
complementary approach to linear regression models. Cloudfront.net.
Retrieved October 15, 2023, from
https://d1wqtxts1xzle7.cloudfront.net/34194665/Gonz00-libre.pdf?
1405316597=&response-content-disposition=inline%3B+filename
%3DNeural_Networks_for_Macroeconomic_Foreca.pdf&Expires=1697
361343&Signature=DyeJ9UA0rZUKIAgSulR2pmVfxSMKC~e5ufB8G
yFiwIU0sm5cUx7WMJXc9-
iGHj2Xq022tSC~2talZFMSaxq7yTHHdmOIjoW3kMJm7npROmTwW
L9HSjm8cl13I8WVbeG0gx9HK96qXx7c8XOGMbOtRDCfiWi1NYGS
1WvR5ZvqkW1RmsirJtP-
zsOe2tW5fW97YcxLfJAhO1zgmGmHk5oW2YvLTuarES3F0~uvCy4K
3Gvkptf~LexN10HMbOOCAu-
z65gGLNalxJ1k0TLCMZieHBzqwWVWwKKCPvUvwPweAElzt7FjGz
QYwcnei35BCMw7iDCC4pnG~b3kKft1UAfDMw__&Key-Pair-
Id=APKAJLOHF5GGSLRBV4ZA
[6] Kazemnejad, A., Batvandi, Z., & Faradmal, J. (n.d.). Comparison of
artificial neural network and binary logistic regression for determination
of impaired glucose tolerance/diabetes. Who.int. Retrieved October 15,
2023, from
https://iris.who.int/bitstream/handle/10665/117927/16_6_2010_0615_06
20.pdf?sequence=1&isAllowed=y