A Model for Predicting Pre-Delinquency using Extreme Gradient Boosting

By

Anthony M. Kisengese

A Research Thesis Submitted to the School of Computing and Engineering Sciences

in partial fulfillment of the requirement of the Degree of Master of Science in
Information Technology

Strathmore University

April 2020
 Declaration

I, Kisengese, Antony Mwawuganga declare that this thesis is my original work and has
not been submitted for an award of degree in any other university.

30th April, 2021

Signature ……………………………………… Date …………………………………………

122801

This thesis has been submitted to the School of Computing and Engineering Sciences for
examination with my approval as the university supervisor.

Signature ……………………………………… Date ………………………………………….

Name: Dr. Omwenga, Vincent

 List of Abbreviations

ATM Automated-Teller Machine

CSV Comma-Separated Values
CVV Card Verification Value
ISO International Organization for Standardization
MCC Merchant Category Code
OTP One-Time Password
PAN Primary Account Number
SVM Support Vector Machines
TPB Theory of Planned Behavior

Chapter 1:
 Abstract

Credit risk is one of the significant risks that financial institutions that advance credit in
credit cards are exposed to. Credit scoring has been one of the techniques that issuers
have been using to determine their customers' creditworthiness, making credit
management one of the critical functions to minimize collections and register positive
turnover. Most approaches target bad debts in the late collections cycle, which
negatively impacts the issuer's books when the credit card accounts become default.
The purpose of this study is to model delinquency scoring to minimize defaults in the
late collection stages by identifying card accounts that exhibit early signs of
delinquency way before the cardholder misses payments. Depending on each score, the
issuer will make informed decisions of how well to proactively engage the cardholder
to identify the best way of intervening in their financial situation and mitigate the risk
of missing payments. This study used Extreme Gradient Boosting to develop a model
that can learn from the observed data and perform predictions on newly submitted card
accounts based on learned information and achieved a prediction accuracy of 81.62%.

Keywords: Credit risk, Credit Scoring, Delinquency, Extreme Gradient boosting

 Table of Contents

Declaration....................................................................................................................................ii
List of Abbreviations...................................................................................................................iii
Abstract.........................................................................................................................................iv
Table of Contents..........................................................................................................................v
List of Figures............................................................................................................................viii
List of Tables.................................................................................................................................ix
List of Equations...........................................................................................................................x
Chapter 1: Introduction.............................................................................................................1
1.1 Background to the study...............................................................................................1
1.2 Problem Statement.........................................................................................................3
1.3 Objectives........................................................................................................................4
1.3.1 General Objective.......................................................................................................4
1.3.2 Specific Objectives......................................................................................................4
1.4 Research Questions........................................................................................................4
1.5 Justification......................................................................................................................5
1.6 Scope and Limitation.....................................................................................................5
Chapter 2: Literature Review...................................................................................................6
2.1 Introduction....................................................................................................................6
2.2 Theory of Planned Behavior (TPB)..............................................................................6
2.3 Cardholder characteristics that influence delinquency............................................7
2.4 Credit Card Consumption Patterns...........................................................................10
2.4.1 Card activity after a period of inactivity...............................................................12
2.4.2 Bounced Payments...................................................................................................12
2.4.3 Over limit for the first time in a given time horizon...........................................12
2.4.4 Cash Advances..........................................................................................................12
2.4.5 Change in the types of products or services purchased.....................................13
2.4.6 Credit utilization ratio.............................................................................................13
2.4.7 Frequency of direct debit cancellation...................................................................14
2.5 Empirical Review.........................................................................................................14
v
 2.5.1 Single Classifiers.......................................................................................................14
2.6 Conceptual Framework...............................................................................................20
2.7 Operationalization of the Study Variables...............................................................21
Chapter 3: Research Methodology.........................................................................................22
3.1 Introduction..................................................................................................................22
3.2 Research Design...........................................................................................................22
3.3 System Development Methodology..........................................................................22
3.4 System Analysis............................................................................................................22
3.5 System Design..............................................................................................................23
3.6 System Implementation..............................................................................................23
3.7 Target Population and Sampling...............................................................................23
3.8 Data Collection.............................................................................................................24
3.8.1 Data Dictionary.........................................................................................................24
3.9 Data Analysis Methods...............................................................................................26
3.10 Research Quality..........................................................................................................26
3.10.1 Reliability...............................................................................................................26
3.10.2 Validity...................................................................................................................27
3.11 Dissemination of the Study Results...........................................................................28
3.12 Utilization of the Study Results.................................................................................28
3.13 Ethical considerations..................................................................................................28
Chapter 4: System Analysis, Design and Architecture.......................................................29
4.1 Introduction..................................................................................................................29
4.2 System Analysis............................................................................................................29
4.1.1 Requirements Gathering.........................................................................................29
4.1.2 Functional Requirements........................................................................................36
4.1.3 Non-Functional Requirements...............................................................................36
4.3 System Architecture.....................................................................................................37
4.4 System Design..............................................................................................................38
4.4.1 Use Case Diagrams...................................................................................................38
4.4.2 Sequence Diagram....................................................................................................41
4.4.3 ERD.............................................................................................................................41
vi
 4.4.4 Database Schema......................................................................................................42
4.4.5 Class Diagram...........................................................................................................43
4.4.6 Wireframes of the system........................................................................................44
Chapter 5: System Implementation and Testing.................................................................46
5.1 Introduction..................................................................................................................46
5.2 Model Development....................................................................................................46
5.2.1 Model Tuning............................................................................................................47
5.3 System Implementation..............................................................................................49
5.4 System Testing..............................................................................................................52
5.5 System Validation........................................................................................................52
Chapter 6: Discussion..............................................................................................................54
6.1 Introduction..................................................................................................................54
6.2 Determinants of delinquency scoring.......................................................................54
6.3 Classifiers used in predicting delinquency of credit card accounts.....................55
6.4 Developing the model.................................................................................................55
6.5 Testing the performance of the model......................................................................55
Chapter 7: Conclusion and Recommendation.....................................................................56
7.1 Conclusions...................................................................................................................56
7.2 Recommendations........................................................................................................56
7.3 Future Work..................................................................................................................57
References....................................................................................................................................58

vii
 List of Figures

Figure 2:1 Theory of planned behavior.....................................................................................7

Figure 2:2 Support Vector Machines........................................................................................17
Figure 2:3 Conceptual Framework...........................................................................................20
Figure 4:1 Check for null values...............................................................................................29
Figure 4:2 Variable description.................................................................................................29
Figure 4:3 Probability of missing payment next month........................................................31
Figure 4:4 Distribution of age and limiting balance..............................................................32
Figure 4:5 Distribution of customers per age group..............................................................32
Figure 4:6 Repayment status for last 6 months vs probability of default...........................33
Figure 4:7Distribution of age with marital status vs tendency to default..........................33
Figure 4:8 Distribution of sex and tendency to default.........................................................34
Figure 4:9 Distribution of bill amounts vs payment amounts.............................................35
Figure 4:10 System Architecture...............................................................................................37
Figure 4:11 Use case diagram....................................................................................................38
Figure 4:12 Sequence Diagram..................................................................................................40
Figure 4:13 Entity relationship diagram (ERD)......................................................................41
Figure 4:14 Database Schema....................................................................................................42
Figure 4:15 Class diagram.........................................................................................................43
Figure 4:16 Login wireframe.....................................................................................................44
Figure 4:17 Prediction form wireframe....................................................................................44
Figure 5:1 Script for loading dataset........................................................................................45
Figure 5:2 Script for credit utilization ratio function.............................................................46
Figure 5:3 Script for overdraft function...................................................................................46
Figure 5:4 Script for parameter model tuning........................................................................47
Figure 5:5 Script for Bayesian optimization............................................................................48
Figure 5:6 Login Screen..............................................................................................................49
Figure 5:7 Prediction Form Screen...........................................................................................50
Figure 5:8 Prediction results......................................................................................................50

viii
 List of Tables

Table 3:1 UCI Credit card dataset dictionary.........................................................................24

Table 4:1 Use case UC1- Login..................................................................................................39
Table 4:2 Use case UC2 - Get Customer Classification..........................................................40
Table 4:3 Use case UC3 - Manage users...................................................................................40
Table 5:1 Testing Approaches...................................................................................................52
Table 5:2 Model Classification Results....................................................................................53
Table 5:3 Comparison of Prediction Accuracy.......................................................................53

ix
 List of Equations

Equation 2:1 Discriminant analysis function..........................................................................15

Equation 2:2 Logistic Regression Equation.............................................................................15
Equation 3:1 Model Accuracy...................................................................................................26
Equation 3:2 Precision Ratio......................................................................................................27
Equation 3:3 Recall Ratio...........................................................................................................27
Equation 4:1 Credit utilization ratio.........................................................................................31
Equation 4:2 Computation for credit utilization ratio...........................................................31

x
 Chapter 1: Introduction

1.1 Background to the study

Credit cards are payment instruments issued by financial institutions to their customers
for accessing credit limits for making payments, after which customers repay the issuer
sometime in the future based on a credit agreement. This credit limit is like a loan. The
card is a physical representation of a financial account held with the issuer. Therefore,
all transactions performed using the card are debited against this account. Every card
transaction reduces the available credit with the same transaction amount for which the
cardholder can access future payments. At the end of every month, the cardholder’s
total debt is the amount spent in that month, plus the interests accrued or any other
agreed fees. Cash withdrawals usually attract high transaction fees and interests to
discourage customers from doing so since this is not the primary use of these facilities.
Besides, a credit card offers a cardholder great payment convenience with various
benefits including but not limited to the ease of performing cashless transactions, the
ability to track all expenses at once, and offering an accurate documentation history of
one's creditworthiness (Lin et al, 2019; Wong and Lynn, 2019).

A cardholder is usually required to make the minimum payment agreed on their total
card debt at the end of every month, but also with the liberty of paying the whole debt
owed. When one fails to fulfill this credit agreement, the card is said to be delinquent
(Finlay, 2010). A preliminary assessment is usually taken by the issuer to determine the
account's status, if the account should fall into collections to recover the debt at the
earliest opportunity (Finlay, 2010) or other treatment strategies are to be used. This
assessment is undertaken to ascertain whether a failure to submit payments is due to
fraud or a cardholder is under financial stress. When the delinquency period is
exhausted, and the cardholder fails to make payments, the account falls into collections
in which an issuer follows recovery strategies that are part of their operations, from
internal debt collection and recovery teams to acquiring services of debt recovery
1
agencies, all in a bid to motivate the cardholder to repay their debt. When repayment is
impossible, the debt is written off, and the account becomes default, which is a terminal
state in the credit card life cycle.

Pre-delinquency is the state before the credit cardholder misses payments. Before the
expected minimum payment is made, this time frame offers the issuer an opportunity to
spot any form of financial duress that may result in delinquency. Finlay (2010, p.140)
argues that cardholders usually miss payments when their financial situations have
changed, such as loss of income, which consequently changes their credit usage
patterns, and these patterns can be detected with the use of a pre-delinquency scoring
model. A high pre-delinquency score defines a high propensity of the cardholder
missing a payment, while a low score the vice versa. Depending on the score, the issuer
can make informed decisions about whether to contact customers and take pre-emptive
interventions that may prevent the customer from becoming delinquent. For example,
since accounts with low pre-delinquency show more significant self-cure signs, the
issuer can send informative emails or messages to the cardholder, which reminds them
of upcoming bills and has the likelihood of triggering those customers who don't miss
payments to repay on time. On the other hand, accounts with high pre-delinquency
scores may require the issuer to contact the cardholder to determine any mitigating
actions that can be undertaken, such as suspending repayments for a while for the
customer to recover their financial footing (Finlay, 2010). Selecting the proper strategy,
message, and channel to approach a cardholder provides the issuer a far better
likelihood of securing repayments whereas maintaining a positive relationship with
them (Esgalhado et al, 2019).

Pre-delinquency scoring is a relatively new scoring model applied to existing customers

to identify those up-to-date customers who are likely to miss their next payment
(Finlay, 2010). This scoring model differs from credit scoring models in that the goal of
pre-delinquency scoring is not to decide about further lending, but rather to inform the
decision of whether to contact customers and take pre-emptive actions that may avoid

2
the customer from becoming delinquent. However, credit scoring techniques can also
be applied in pre-delinquency scoring as they are both concerned with predicting the
probability of undesirable customer behavior in the future (Lessmann et al, 2015).

Multiple studies have been undertaken to develop scoring models, with logistic
regression so far being the de-facto technique that has been used extensively (Onay, &
Öztürk, 2018). Lessmann et al (2015) performed a benchmark of 41 classification
algorithms and observed that ensemble classifiers had a better predictive performance
compared to single classifiers such as support vector machines, artificial neural
networks. This study will use an extreme gradient boosting(EGBoost) ensemble to fit
and build the delinquency scoring model.

1.2 Problem Statement

Credit risk management is one of the management practices that financial institutions
involved in consumer lending must undertake. Of all the different risk exposures,
including market risk, operational risk, and other risks, 50% of the total risk elements
relate to credit risk (Sharifi, Haldar & Rao, 2019). Credit scoring is one of the methods
that financial institutions have employed to efficiently manage credit risk and maximize
profitability by modeling a customer's creditworthiness. This technique aims to classify
a customer as either "good" or "bad" depending on the application's context: new
customers and existing customers. For new customers, the goal is to access the
customer's potential risk once they have been granted a credit facility, while for existing
customers, the goal is to forecast their future behavior over a given time horizon (Onay
& Öztürk, 2018). The former is called application scoring, and the latter, behavioral
scoring. All customers categorized as good are likely to repay their credit, while the bad
counterparts are likely to default in repayment. Weights are assigned to the
characteristics that define the customer's propensity to default as an indicator of the
customer's risk level to generate a credit score. This, therefore, informs the lending
decision and the account-treatment strategies in the collection cycle.

3
Delinquency management enables the issuer to evaluate the value at risk way earlier in
the collections cycle. Therefore, more efforts can be channeled to customers who show
non-self-cure signs using a proactive approach to intervene in their financial situations
using personalized arrangements. For example, in a case when the credit card account
has shown a likelihood of default, and there exists some headroom between the
customer's balance and their credit limit, the issuer can lower the credit limit to mitigate
the risk of exposure if the account was to fall into delinquency (Finlay, 2010). Other
interventions include and are not limited to account closures, payment reminder
notifications, and change in the minimum repayment amount.
This research proposes a prediction model that considers the personal characteristics of
the cardholder, their credit usage, and payment behavior to predict the likelihood of
missing payments in a given billing cycle, and possibly in upcoming billing cycles.

1.3 Objectives

1.3.1 General Objective

This research aims to minimize the number of credit card accounts that fall into
delinquency by developing a prediction model based on cardholder characteristics and
transaction activities.

1.3.2 Specific Objectives

i. To analyze the cardholder characteristics and transaction activities that influence

the delinquency credit card accounts
ii. To examine the classifiers used in predicting delinquency of credit card accounts
iii. To develop a model for predicting delinquency of credit card accounts
iv. To test the performance of the model in predicting the delinquency of credit card
accounts

4
1.4 Research Questions

1. How do cardholder behavioral characteristics and transaction activities influence

the delinquency of credit card accounts?
2. What are the classifiers used in predicting the delinquency of credit card
accounts?
3. How can the model be developed, and what algorithms and methodology will be
used in its implementation?

4. How can the performance of the developed model be tested?

1.5 Justification

Credit card debt has been rising as more and more credit accounts fall into delinquency
with revolving balances. Prediction models come handy to classify customers into
segments for more targeted interventions depending on the value at risk and customer
profile. This can lead to increased issuer-cardholder connection and raise the chances of
cardholders paying, and effective settlement approaches can also be designed especially
for cardholders under financial duress. Therefore, the issuer can position themselves to
expand on relationships demonstrating that they care about cardholders' financial
health and well-being. Also, this reduces strain on collections operations by keeping
low-risk and self-cure accounts out of delinquency.

1.6 Scope and Limitation

This research's scope is limited to learning from publicly available data related to credit
card accounts with has limited demographic, socio-economic, credit loan, and payment
history information. This is due to the sensitivity of the credit card data which banks
treat as confidential and private. The study will also be limited to predicting the

5
possibility of a credit cardholder missing a payment based on the patterns established
in the data.

6
 Chapter 2: Literature Review

2.1 Introduction

This chapter will cover relevant literature related to the study. It will look into the
theory of planned behavior to construct the independent variables that influence
behavioral scoring in the study. The empirical review will provide both statistical and
machine learning techniques that have been used in credit scoring.

2.2 Theory of Planned Behavior (TPB)

Icek Ajzen postulated the theory of planned behavior in 1985 as a model to predict and
understand human behavior in that behaviors are immediately determined by
behavioral intentions and, under certain circumstances, perceived behavioral control
(Kan & Fabrigar 2017). According to the theory, the determinants of behavioral
intentions are a combination of three factors: attitudes toward the behavior, subjective
norms encompassing the behavior execution, and personal perceived behavioral
control. A behavior is an individual’s overt action or set of actions that they perform
and is confined to the researcher's theoretical and/or applied objectives and can be
conceptualized in terms of an individual's action, target, and context, and the time it is
performed (Kan & Fabrigar, 2017). The behavior's attitude is an individual's evaluation
of performing the behavior as defined by these four components and assessed by
measuring one's behavioral beliefs. Subjective norms are determined by normative
beliefs and the motivation to conform with specific referents (Kan & Fabrigar, 2017).
The normative beliefs are an individual's perception of with regards to expectations of
the people important to them. Perceived behavioral control refers to people's perception
of the ease or difficulty of directly performing the behavior of interest and influencing
intentions or behavior (Kan & Fabrigar, 2017).

7
 Figure 2:1 Theory of planned behavior

The theory of planned behavior suggests that sometimes people may intend to perform
a given behavior but may lack complete control of their behaviors given the
circumstantial current internal or external controls constraining the behavior under
observation (Bhattacherjee, 2021). Internal controls constitute the person's ability to
perform the intended behavior, while external controls are the availability of external
resources needed to perform that behavior (Bhattacherjee, 2012).
Behaviors influence outcomes by partly contributing to the factors that lead to an
outcome under observation. An outcome is a combination of a person's behavior and
other external factors that positively and negatively influence the outcome.
Through this model, this study will investigate possible behaviors that influence the
pre-delinquency of credit cardholders using transaction-oriented activities such as
credit-utilization ratio, persistence debt, and cash advances.

2.3 Cardholder characteristics that influence delinquency

The personal cardholder characteristics that influence delinquency can be categorized

into demographic, socio-economic, and behavioral factors. According to Ming‐Yen et al

8
(2013, p.484), age, income, occupation, and marital status were posited to be the socio-
demographic factors that greatly influence credit cardholders’ propensity to default.
Older credit cardholders tend to be sound and conservative in their spending decisions
while their young counterparts tend to accumulate debt as they perceive their debt to be
a temporary state which they will repay in the future as they’re still young (Ming-Yen et
al, 2013, p.484). According to a study by Zainudin, Mahdzan, and Yeap (2019), most
young adults are technology savvy and digitally sophisticated than their older
counterparts due to great exposure to digital payments and online shopping. According
to this study young adults were found to be highly indebted due to excessive credit
card usage caused by the intensified tendency for instant gratification, compulsive
shopping behavior, high spending power and high debt commitment. The study also
noted that the young credit cardholders were likely to pay the minimum monthly
payment required while 50% admitted to falling into delinquency. In their research,
Kiarie, Nzuki, & Gichuhi (2013) found that there was a lower rate of default among
married cardholders compared to cardholders who were single. On gender, a study by
Ciunova-Shuleska (2012) noted that most credit cardholders were males with females
having a high tendency to repay their debt on time. Income was also significant
contributing factor of how much credit limit a cardholder can access and repay.
According to Stavins (2020), high-income credit cardholders were observed to have
significantly higher credit card balances but showed tendency to repay those balances
each month. Lower-income credit cardholders who were less educated exhibited a
pattern of carrying unpaid balances, falling into the convenient cardholders’ bracket.
The behavioral characteristics are a factor of the repayment patterns that are influenced
by the cardholder's attitude towards repayment of debt, issuer characteristics, policy,
and regulatory issues, and the prevailing economic conditions. The repayment action is
normative in the essence that the cardholder is expected to fulfill their contractual
agreement by paying up their dues. Singh, Rylander, & Mims (2018) conducted a study
on college students' behavior and defined credit card repayments as partial, full, or
minimum payments and payments made on time. Those credit cardholders who pay
full amounts on time are known as convenient users or transactors. On the other hand,

9
revolvers are those that make partial or minimal payments with the balance being
carried forward to the next payment period and therefore incur interest fees and
sometimes late payment fees. Revolvers can further be classified into non-defaulting,
and defaulting revolvers, where the latter have defaulted severally in paying their
credit. Convenient users usually show self-cure signs, while revolvers are the accounts
of concern since the risk of default is high, and accounts are likely to default since the
credit cardholder's willingness to repay their loan is arguably low. In their study,
Barboza, Smith, and Boubacar (2017) found that missing payments is a leading and
significant source of anxiety in all model specifications due to prior financial
mismanagement that creates adverse spillover effects in the current billing cycle while
partial payments indicated financial distress.
According to the study by Shapiro, & Burchell (2012), financial anxiety was found to be
a contributing factor to the financial administration of their finances, which
consequently influenced their attituded towards repayment of debt. Barboza, Smith,
and Boubacar (2017) argued that anxiety is inevitable, and one can experience anxiety in
multiple dimensions of their lives, which can be a motivating or negative stressor. The
depressive nature of financial anxiety influences the wrong assessment of outcomes
leading to poor choices, which are primarily contributed by easy access to credit cards,
low levels of financial literacy, which create pervasive incentives to live beyond one's
means, and financial difficulties (Barboza, Smith, and Boubacar, 2017). Some of these
choices have a negative financial impact, including over-borrowing, accumulation of
revolving debt, or failure to make full payments according to schedules.
Since credit cards offer cardholders the flexibility of how quickly they repay their credit
balance, individuals can choose whether to repay in full at the end of each month or
spread payments over a long period (Financial Conduct Authority, 2017). The
minimum payments made each month are based on the credit agreement with the
issuer. This flexible nature of making minimum payments each month means that credit
cardholders can carry a large balance for an extended period without significantly
reducing the debt. Holding a credit card balance for an extended period can also signify
that a customer may be trapped in a cycle of borrowing that they cannot afford to pay

10
down (Financial Conduct Authority, pp. 14-17, 2017). In some circumstances, a credit
cardholder making the monthly minimum repayment may have underlying financial
difficulties obscured by the repayment pattern (Financial Conduct Authority, pp. 2017).

2.4 Credit Card Consumption Patterns

A credit card is used to make payments, either in a card-present and card-not-present

environments, which defines the cardholder's interactivity and the payment platform. A
card-present transaction involves the cardholder interacting with their physical card to
complete payment by swiping their card through a reader or when a payment device
processes an EMV chip in the card. On the other hand, a card-not-present transaction
involves the cardholder making payments in a payment processing platform such as an
e-commerce site without physically presenting their credit card to complete the
payment. In a card-not-present environment, a combination of the credit card physical
information such as the primary account number (PAN), expiry date, and the card
verification value (CVV) are used to verify and authenticate the transaction. Some
issuers enforce a cardholder security protocol layer commonly referred to as 3D Secure
to prevent fraudulent activities on one's account by sending a one-time PIN (OTP) to
the cardholder's mobile device to complete the payment. In either of these two
scenarios, credit card transactions fall into either the following types: purchase, pre-
authorization, capture, void, reversal, and verify.
A purchase, commonly known as a sale, is the most common type of credit card
transaction where a cardholder uses their card to pay for goods or services. In this
transaction, the cardholder initiates the payment, either from a merchant's payment
device or in a payment gateway where the card data and the transaction amount is
submitted to the acquiring bank network, which is the financial institution that is
responsible for processing the transaction on behalf of the merchant. The acquiring
bank then sends this transaction to the card network, which reroutes the transaction to
the issuer's bank for approval. If approved, the merchant receives an approval code
signifying the completion of a successful financial transaction. This transaction type

11
reduces the cardholder credit limit, and the cardholder is obligated to settle the debt
according to the credit agreement with the issuer.
A pre-authorization transaction is similar to purchase with the exemption that, the
transaction amount is not debited from the credit card account, but rather, it is
"reserved" for a given number of days, usually between 7 to 10 days, as a guarantee of
the availability of funds to be claimed later using a capture transaction. If a pre-
authorization is not captured during the allotted period, the funds are released back to
the credit card account. This method is often used by gas stations, car rentals, and hotels
where the merchant desires to ensure that the transaction amount is available before
offering a service. A pre-authorization request doesn't affect the credit card limit as the
charge is not incurred in this leg. A capture transaction is the second leg of the pre-
authorization transaction and utilizes the approval code returned in the pre-
authorization phase to complete this transaction. The capture amount can be less or up
to the pre-authorized amount but cannot exceed the same. The net effect of both the
pre-authorization and capture requests is similar to the purchase transaction, with the
only difference being the time when the actual debit is done from the account.
Therefore, when the capture transaction is actualized, the credit limit is affected in
similar ways to purchase.
A refund performs a reversal of the purchase transaction, and the transaction amount is
credited back to the credit card account less the transaction fees. It is similar to an
internal funds transfer transaction as funds are moved from the merchant's pool
account to the credit cardholder's account to zero out the purchase transaction. A
refund increases the credit limit of the credit card account and reduces the outstanding
payable debt. A verification transaction is a zero-amount transaction processed into a
credit card account to ascertain for the first time the validity of the card in card-not-
present scenarios where a charge is to be incurred in the future. It's usually used by
merchants offering subscription services that are payable using card-on-file processing
architecture where the card information resides with the merchant for future payment
of services. The number of verification transactions in a credit card account depicts the

12
card's active use in making purchases in different e-commerce sites or the possibility of
payments to be debited into the account sometime in the future.
Other transactions that can be analyzed that determine a change in the cardholder's
behavior include cash advances from the credit card, first time over limits over a period,
a sudden credit card activity after a period of inactivity, and recent direct debit
cancellations. These transactions signify a shift of the average cardholder's behavior,
which is likely to be caused by changes in their financial status or deliberate intentions
resulting from lack of attention to their credit limits.

2.4.1 Card activity after a period of inactivity

Credit card inactivity occurs when the card is not used to perform any transactions over
a given time horizon. Inactivity in this aspect falls within the defined period of
inactivity by the issuer before the credit card account is closed. Inactivity is influenced
by several factors including and not limited to the cardholder having several credit
cards to their intention of using the credit card only for emergencies.

2.4.2 Bounced Payments

A payment transaction bounces when the transaction fails to complete successfully at

the point of service due to the card's limits, and the transaction is said to be rejected.
When a purchase or capture transaction is rejected, it usually reflects that the
cardholder has exhausted their credit card limit and that an increase of credit limit at
the time of performing a transaction has been denied for an account with no protected
overdraft facility. Rejected transactions are also an attribute of overspending, which is a
measure of whether a cardholder knowingly transacts and does not have sufficient
balance to fund the transaction.

2.4.3 Over limit for the first time in each time horizon

Over limits refers to when a credit card account surpasses its credit limit with a
transaction. When the transaction amount exceeds the credit limit, the issuer may

13
choose to decline the transaction, leading to a bounced payment or charge over-limit
fees.

2.4.4 Cash Advances

A cash advance occurs when a cardholder withdraws money from an ATM or performs
unprotected overdraft payments for issuers with credit overdraft facilities. Cash
advances usually attract higher interests than normal purchases, have no grace period,
and attract a direct cash advance fee. Cash advances usually follow the need to have
cash that could have otherwise been withdrawn from one's current or savings account.

2.4.5 Change in the types of products or services purchased

Each financial message in the network is transmitted with the merchant category code
(MCC), which is an ISO 18245 4-digit code that distinguishes the industry in which a
merchant operates regarding the goods or services they provide, and this makes it easy
to categorize the products or services a cardholder spends on. The type of merchant, the
amount spent at each, and how often a cardholder shops with the card are key
attributes that can reflect a significant change in the financial status of a cardholder
when observed over time. When a cardholder who uses their credit card for purchasing
non-essential services suddenly begins to buy essential goods or services such as
groceries, this sudden change can be observed and analyzed as a factor that the
cardholder's financial status has changed.

2.4.6 Credit utilization ratio

The credit utilization ratio is the credit card balance compared to the credit limit and
determines the card's balance level. As the cardholder makes more purchases, the credit
utilization ratio goes up, and so does the default risk. A cardholder with a low credit
utilization ratio has a lower chance of missing their payments than one with a higher
ratio when observed on the same time horizon. Ambrose et al. (2006, pp. 2-7) observed
that credit utilization increases during periods of economic distress and noticed a trend

14
in that initial credit utilization was lower for borrowers who expected future financial
credit deterioration, while their counterparts utilized a more significant percentage of
the total credit available.

2.4.7 Frequency of direct debit cancellation

Direct debits are payment arrangements that cardholders enter with a bank to facilitate
the transfer of money from their other accounts to their credit card account to settle
their outstanding debts on agreed debts. Credit cardholders are at liberty to cancel
direct debits per the arrangements they have with their banks. Cancellation of direct
debit before settling the credit card debt at the end of the billing cycle informs the
intention of preserving access to cash on the checking account due to factors such as
hoarding cash due to an uncertain economic difficulty. Simultaneously, it shows a low
probability of settling card debt, mostly if this was their main checking arrangement.

2.5 Empirical Review

2.5.1 Single Classifiers

Single classifiers can be utilized to solve a credit scoring problem, which is a binary
classification of two classes – “good” and “bad” customers (Tsai & Hung, 2014). Single
classifiers can be categorized into two: statistical techniques and machine learning
methods. The most used statistical techniques in credit scoring include linear
discriminant analysis, multiple discriminant analysis and logistic regression among
others. Machine learning techniques include artificial neural networks (ANN), K-
nearest neighbor, and support vector machines.

Discriminant Analysis
Fisher developed discriminant analysis in 1946 as a classical model for separating two
groups using a linear combination of variables by deriving a linear combination of
explanatory variables(ratios) that provides the maximum distance between the means
of the two-subsets (Khemais, Nesrine, & Mohamed, 2016). In scoring, discriminant

15
analysis can is used to separate credit cardholders into two subsets, those who are likely
to miss payments, and those who will self-cure over the same period. This method
allows the characteristics of those credit cardholders who are likely to miss payments to
be determined and distinguished from those who will self-cure, and from these, a new
cardholder pre-delinquency score can be determined. A discriminant analysis function
for this model can therefore be presented as follows:

Si=a1 X 1+ a2 X 2+ …+a n X n

Equation 2:1 Discriminant analysis function

where a1, a2, …, an are the discriminant coefficients, X1, X2, …, Xn are the discriminant
variables and Si is the variable defining each of the subsets. When the S score is high,
the cardholder is likely to miss payments, and when the S score is low, the cardholder is
likely to self-cure.
While discriminant analysis is easy to implement and straightforward in generating
results, one of its weaknesses is the assumption that the variables used normally
distributed and independent, which is restrictive, is real-life practice since the credit
data is categorical in nature. It also suffers from the fact that the subsets' covariances
under observation might not be equal (Mittal, Gupta, & Jain, 2011).

Logistic Regression
Logistic regression is a linear model that estimates the probability of discrete outcomes
based on several predictor variables. The most common logistic regression models
binary outcomes bounded by the range 0 or 1; yes/no, success/failure, and will occur/won’t
occur. Therefore, logistic regression can predict a customer's probability to default and
identify the variables related to this behavior. The mathematic representation of logistic
regression is as follows:
k
pi
log ( )
1−p i
=β 0 + ∑ ❑ β k X ji
j=1

Equation 2:2 Logistic Regression Equation

16
 (Adapted from de Paula et al., 2019)

where pi is the probability of the customer i being good, k is the number of independent
variables in the model, Xji is the value of the independent variable j for customer i and βj
are parameters, i = 1, ..., n, j = 1, ..., k.
According to Regis and Artes (2015), logistic regression is used to estimate the
probability of a customer transitioning from one state A, non-defaulting, to state B,
defaulting over a certain period, and other kinds of transitions are not considered.
However, a customer can possess other states other than these two, including non-
defaulting without revolving credit, non-defaulting with revolving credit use, in delay,
voluntary cancellation, and default (Regis & Artes, 2015).

K-Nearest Neighbor
The k-nearest neighbor classifier is a lazy, non-parametric, and instance-based learning
algorithm used for classification and regression. Its non-parametric characteristic
enables this technique not to make any assumptions in the underlying data, which is a
useful attribute in real-life practice. According to Khemais, Nesrine, & Mohamed (2016),
this technique assesses the similarities between the pattern identified in the training set
and the input pattern. The non-parametric nature of this method gives it an advantage
of modeling irregularities in the risk function over the feature space (Khemais, Nesrine,
& Mohamed, 2016).

Artificial Neural Networks (ANN)

ANN is a computational model originally inspired by how the human brain processes
information from cognitive and computer science research developments. ANN mimics
the human brain process by allowing for concurrent complex processing of inputs to
achieve an output (Ala'raj, Abbod, & Radi, 2018). An ANN model consists of a network
of mutually connected artificial neurons that transmit information to one another. The
neurons are represented by some state (0 or 1), and each node may also have some
weight assigned to them that defines its strength or importance in the model (Kaur &
Kumari, 2020). The neurons are organized in layers, with an input layer representing a

17
given input data vector, the hidden middle layers, and finally, the output layer which
provides the desired output of classification. The main advantage of ANN is that the
model does not take any prior assumptions about data distribution before learning,
which significantly promotes the usability of ANNs in various applications.

Support Vector Machines (SVMs)

Support vector machine (SVM) is a machine learning technique used in both
classification and regression. In an SVM linear classifier model, a data point is viewed
as a p-dimensional vector separated by a maximum of p-1 hyperplanes (Kaur &
Kumari, 2020). As many hyperplanes may be used to classify the data, the best
hyperplane with the maximum margin between the two classes it separates is selected
(Kaur & Kumari, 2020). This hyperplane is called the maximum margin hyperplane. In
credit scoring, it is used to find an optimal hyperplane that categorizes the training
input data into two classes – good or bad (Ala'raj, Abbod, & Radi, 2018).

Figure 2:2 Support Vector Machines

18
 (Ala’raj, Abbod, & Radi, p. 615, 2018)

The dashed lines are called margins from the above figure, and the training data that lie
on the margins are called support vectors. SVM works best when there is a clear margin
of separation between classes, and therefore, is more effective in high dimensional
spaces. However, it is not suitable where the dataset is considerably large or when the
dataset has more noise.

Ensemble Classifiers
Ensemble learning is one of the techniques used to optimize and improve the
performance of individual machine learning classifiers. An ensemble classifier consists
of multiple base models that are trained, and their predictions combined to achieve
higher predictive performance and reduce generalization error. The number of base
models is however kept small due to the computational training expense and
diminishing returns of the final model performance as more models are used. Different
ensemble techniques are chosen depending on the complexity of the problem at hand:
varying the training data, varying the base models, or varying the combinations in
generating the final ensemble output.
Machine learning model errors are defined by two properties: bias and variance. Bias is
a measure of how close the model can capture the mapping function between inputs
and outputs whereas variance refers to the amount by which the model changes when
different training data is used in fitting (Zhanga & Ma, 2012). This introduces a trade-off
with respect to the performance of a model – increasing the variance reduces the bias
and increasing the bias reduces the variance of the model. Ensembles, therefore, achieve
a better predictive performance in a prediction model than a single classifier by
reducing the variance of the prediction error (Zhanga & Ma, 2012).
According to Tsai & Hang (2014), ensemble approaches can be classified into 3
categories: bagging, boosting, and stacking.

Bagging

19
Bagging classifier is an ensemble technique that Leo Breiman proposed in 1994 that
handles classification and regression problems (Tsai & Hung, 2014). It is designed to
improve the stability and accuracy of machine learning algorithms by combining
classifications of randomly generated training sets to form a final prediction. Such
techniques can be typically be used as a variance technique by randomization into its
construction procedure and then creating an ensemble of out. Random forest is one of
the popular implementations of the bagging technique which is used for both
regression and classification based on a multitude of decision trees where each decision
tree gives a classification (Hartmann, 2021). The algorithm uses a bootstrapping
technique to train each tree in parallel on various subsets of the training dataset using
different subsets of available features. To classify a new instance, each generated
decision tree undergoes a voting process, and the final decision consists of the most
popular class. Since each decision tree in the random forest is unique, this reduces the
algorithm's overall variance. This consequently enhances the predictive accuracy of the
model and addresses the instability problem experienced in decision trees. Thus, they
remedy the problem of individual decision trees which suffer from high variance and
retain the benefit of a low-bias method making random forests to be appealing for both
practitioners and researchers (Hartmann, 2021).

Boosting
Bosting involves learning from previous base model mistakes to make better predictions. It
involves combing several weak base learners to form one strong learner, thereby improving the
predictability of the models (Hartmann, 2021). It works by arranging weak learners in a
sequence such that weak learners learn from the next learner in the sequence. The three widely
applied boosting algorithms are Adaptive Boosting (AdaBoost), gradient boosting and Extreme
Gradient Boosting (XGBoost). Adaboost combines multiple weak learners, which are decision
trees with a single split known as decision stumps, to create a single strong learner that can be
used for both regression and classification ( Hartmann, 2021). Initially, all observations are
weighted equally when creating the first decision stump. Observations that are incorrectly
classified are then assigned more weights to correct the prediction error in successive iterations.
In gradient boosting, predictors are added sequentially to an ensemble, with each set of

20
preceding predictors correcting their successor which increases the accuracy of the model. New
predictors are then fit to counter the residual errors in the previous predictor.
XGBoost is an implementation of gradient boosted trees that focus on the computational speed
and performance of the target model. In XGBoost, the weak learners are added in parallel using
a multithreaded pattern, which results in proper hardware utilization, greater speed, and
efficiency. This algorithm is commonly used due to its ability to handle missing values and
prevent overfitting, parallelization in constructing trees, out-of-core optimization computing for
large data sets, and cache optimization. XGBoost has been used successfully and across various
Kaggle competitions (Hartmann, 2021).

Stacking
Stacking, or commonly known as stacking generalization involves integrating multiple
classifiers with different classification algorithms (Tsai & Hung, 2014). In this method, a new
model learns how to best combine the prediction outputs from other multiple existing models.

2.6 Conceptual Framework

The proposed model will utilize a credit card dataset that will be split into two: training
and validation datasets. The training dataset will be used to fit the model and perform
exploratory data analysis to identify the patterns in the data. The validation dataset will
be used to evaluate the model while tuning hyperparameters. The proposed model
shall assign a score to a credit cardholder based on the applied prediction rules. The
conceptual framework is illustrated with Figure 2.4 below:

21
Figure 2:3 Conceptual Framework

22
 Chapter 3: Research Methodology

3.1 Introduction

This study aims to develop a model for predicting credit cardholders' pre-delinquency
using behavioral characteristics for a given time horizon. This chapter outlines the
research methodology that will be used to conduct the study.

3.2 Research Design

This study will take an exploratory research approach to establish the relationship
between the variables under observation and how they relate to the cardholder's
propensity to repay their card debt. It will also enable assigning weights to the variables
depending on their influence on their outcome in determining a credit cardholder's pre-
delinquency level.

3.3 System Development Methodology

The research will use an agile methodology, which is both iterative and incremental.
This methodology takes a fail-fast approach to fast-prototyping that allows the model to
be tested and validated as the system features are continually defined. This is suitable
for a machine learning project as they involve a high level of uncertainty and allows the
understanding of the business problem to be refined with each iteration through
feedforward, feature re-engineering, and modeling.

3.4 System Analysis

The system is intended to be used by banks, and other financial institutions that issue
credit cards and the system's main users will be the credit and collection officers. The
system's functional requirements include allowing users to import data using CSV
format, validating, and analyzing imported data, predicting the score of credit
cardholders, and presenting the results in a user-friendly format. The non-functional

23
requirements include secure access to the system through proper authorization and
authentication of users, ease of scale, and acceptable performance speeds. The system
shall have user interface from which all user intended actions will be submitted and
results obtained. 

3.5 System Design

Various tools will be used to capture system interactions between the different
components of the system. Use case diagrams will be used to describe the different
actors in the system and the main use cases that will be involved to achieve their system
needs. For each use case defined, sequence diagrams will be used to explore and show
the order of interactivity among the system's various components. An entity-
relationship diagram will define the relationships of the different entity types that are to
be modeled in the system's design.

3.6 System Implementation

This study will use publicly available datasets containing relevant credit card data
information to build the model since obtaining financial data from banks, and financial
institutions are difficult due to its sensitive nature and the legal privacy obligation tied
to them. The model will handle missing values in the data and apply feature selection.
The Python language will be used for data manipulation, data analysis, and running the
machine learning algorithms. Apache Cassandra, an open-source database, will be used
for persistent storage and provides fault-tolerant, scalable, and high availability
capabilities required for processing financial data. The ReactJs Framework will be used
to develop the front-end web interfaces that users will use to access the system due to
its fast, scalable, and simple characteristics.

3.7 Target Population and Sampling

Population

24
The target population of this study will be credit card cardholders. They will constitute
cardholders who have defaulted, with revolving balances, and those who have paid
their debt in full. The study will also focus on credit cardholders with a repayment
history over a period of 6 months. The source of this data will be the UCI Machine
Learning Repository which contains 30, 000 credit cardholders from a Taiwanese bank
with transaction activities spanning from April 2005 to September 2005.

Sampling
The study will focus on probability sampling to prevent bias and give each cardholder
in the population an equal chance of being selected. This will result in a sample with a
true representation of the population. The study will also adopt simple random
sampling and from this, 80% of the collected records will be used for training the
model, 10% for testing, and 10% for validation purposes.

3.8 Data Collection

This study will utilize a secondary dataset obtained from UCI Machine Learning
Repository which contains information on default payments, demographic factors,
credit data, history of payment, and bill statements of credit card customers from a
Taiwanese bank from April 2005 to September 2005. The dataset includes 30, 000
customer records and 24 attributes that will be significant in developing, testing, and
validating the model. Given the scarcity of credit card datasets, this is the most recent
publicly available data that suits the study.

3.8.1 Data Dictionary

The dataset contains 24 variables as described in the table below

Table 3:1 UCI Credit card dataset dictionary

SN. Variable Comments

1 ID ID of credit cardholder
2 LIMIT_BAL  Amount of given credit in NT dollars
(includes individual and
family/supplementary credit

25
3 SEX Gender (1=male, 2=female)

4 EDUCATION 1- Graduate School

2- University
3- High School
4- Others
5 & 6 Unknown
5 MARRIAGE Marital status (1=married, 2=single,
3=others)
6 AGE Age in Years

7 PAY_0  Repayment status in September 2005:

Scale (the same applies to PAY_2 TO
PAY_5)
-1 = pay duly
Other Positive Value – Payment delay for
the value in months
1 = payment delay for one month
2 = payment delay for two months
…..
8 = payment delay for eight months
9 = payment delay for nine months and
above

8 PAY_4 Repayment status in August 2005

9 PAY_3  Repayment status in July 2005

10 PAY_4 Repayment status in June 2005

11 PAY_5  Repayment status in May 2005
12 PAY_6  Repayment status in April 2005
13 BILL_AMT1  Amount of bill statement in September
2005 (NT dollar)
14 BILL_AMT2  Amount of bill statement in August 2005
(NT dollar)
15 BILL_AMT3  Amount of bill statement in July 2005 (NT
dollar)
16 BILL_AMT4 Amount of bill statement in June 2005
(NT dollar)
17 BILL_AMT5  Amount of bill statement in May 2005
(NT dollar)
18 BILL_AMT6 Amount of bill statement in April 2005
(NT dollar)

26
 19 PAY_AMT1 Amount of previous payment in
September 2005 (NT dollar)
20 PAY_AMT2 Amount in previous payment in August
2005(NT dollar)
21 PAY_AMT3 Amount in previous payment in July
2005(NT dollar)
22 PAY_AMT4 Amount in previous payment in June
2005(NT dollar)
23 PAY_AMT5 Amount in previous payment in May
2005(NT dollar)
24 default.payment.next.mont Default Status (0-No, 1-Yes)
h

3.9 Data Analysis Methods

The data will analyze the dataset information on default payments, demographic
factors, credit data, history of payment, and bill statements of the credit cardholders
using density plots, box plots, and heat maps. The density plots will be used to describe
the distribution of each feature in the dataset. Heat maps will be used to identify
patterns and perspectives of each feature in the dataset. Box plots will be used to
visualize the spread of values in the dataset. On data cleansing, the imputation method
will be used to handle missing values.

3.10 Research Quality

3.10.1 Reliability

Reliability is the degree of consistency to which the model will yield the same results for
the given set of inputs. This study will measure performance by evaluating the accuracy
of the model, the precision, and recall ratios. Accuracy gives the ratio of the correctly
classified observations to the total observations, which is given by the following
formula:

27
 TP+TN
Accuracy=
TP+TN + FP+ FN
Equation 3:3 Model Accuracy

Where TP (True Positives), which are the correctly predicted positive observations; TN
(True Negatives) is correctly predicted negative values; FP (False positives) which are
wrongly predicted positive values; and FN (False Negatives) which are wrongly
predicted negative values.
Precision is the ratio of all the correctly classified observations from the positive class
among all the observations classified by the model as the positive class. It is expressed
in the following formula:

TP
Precision=
TP+ FP

Equation 3:4 Precision Ratio

A recall is the ratio of the correctly classified observations from the positive class among
all samples from the positive class in the actual data. It is expressed as follows:

TP
Recall=
TP+ FN

Equation 3:5 Recall Ratio

3.10.2 Validity

The research shall endeavor to maintain the data's integrity by using and presenting the
training data at is, allowing for data preprocessing only to enable the model to work as

28
intended. The data shall not be manipulated to produce baked results, and the results
shall be presented as-is.

3.11 Ethical considerations

The study will use publicly available secondary datasets and shall therefore comply
with the license agreements, privacy, and copyright dispute requirements under which
the data is provided. Besides, proper accreditations shall be provided to the owners of
the data. The software to be used shall be free or acquired under educational licenses or
procured on need-be-basis in proprietary software.

29
 Chapter 4: System Analysis, Design, and Architecture

4.1 Introduction

This chapter describes the general architecture and design of the system with an in-
depth analysis of the proposed software solution. The study developed a web-based
system that credit officers will interact with to generate real-time delinquency scores for
their customers.

4.2 System Analysis

System analysis offered the researcher the capability to fulfill the software capabilities
vis-a-vis the outlined objectives. The system’s capabilities in this case is the software
system that will enable credit officers to adequately score a credit cardholder in a given
billing cycle and inform their decision of early interventions to avoid them falling into
delinquency.

4.1.1 Requirements Gathering

4.1.1.1 Dataset Source

This study will utilize a secondary dataset obtained from UCI Machine Learning
Repository due to the privacy and legal concerns associated with credit datasets from
financial institutions that make it difficult to acquire such data. This dataset presents the
most up-to-date credit card information that contains relevant demographic and
transaction history attributes that are significant to the study.

4.1.1.2 Data Pre-Processing

The dataset had no missing values and hence there was no need for imputation. All
categorical variables were already encoded accordingly as described to the data
dictionary, and the values were either an integer or a decimal(float) value.

30
Figure 4:4 Check for null values

Figure 4:5 Variable description

According to the dataset description, the marriage column should have 3 categories: 1-
married, 2-single, and 3-others. The dataset shows that the marriage column has an
extra value of 0. This was changed to 3, the others category as follows:

31
The same approach applied to marriage was also used for the education variable: the
extra value 0 was changed to 4 -the others category. Since 5 & 6 represented unknown
categories, this was also condensed to the others category as follows:

The sex column also had 2 extra unknown categories with values 3 and 4. A new
category, others, was introduced to group together these unknown categories with the
encoding 3, implying that rows with value 4 only were updated:

The credit utilization ratio is a measure of the total revolving credit already consumed
to the total revolving credit available. It can also be expressed as the total debt divided
by the credit limit in the case of a single credit card. This can be expressed using the
following equation:
Total Debt
Credit Utilization Ratio=
Available Credit (Credit Limit)

Equation 4:6 Credit utilization ratio

For this given dataset, the average credit utilization ratio was derived as follows:

BILL AMT 1 + BILL AMT 2+ BILL AMT 3 + BILLAMT 4 + BILLAMT 5+ BILL AMT 6
Credit Utilization Ratio=
Credit Limit∗6 months

Equation 4:7 Computation for credit utilization ratio

4.1.1.3 Exploratory data analysis

According to the dataset, 77.88% of the credit cardholders are not likely to miss
payments in the next billing cycle, while 22.12% are likely to default. This indicates how
imbalanced the dataset is and this must be accounted for when developing the model.

32
Figure 4:6 Probability of missing payment next month

The dataset consists of skewed data of limiting balance and age of credit cardholders as
shown below. The plot in figure 4.4 shows there are more cardholders having a limiting
balance between 0 and 200000, and with age between 20 and 40 years. The plot in figure
4.5 shows the highest number of customers fall between 21-30 and 31-40 age groups.
Therefore, we can conclude that with the increasing age group, the number of
customers that will default in the next payment month is decreasing. Thus, age is an
important feature in predicting the probability of default.

33
Figure 4:7 Distribution of age and limiting balance

Figure 4:8 Distribution of customers per age group

The following plot shows the proportion of customers likely to default based on
repayment history. We observe that, customers who paid on time had less probability
of missing payments, while the probability of missing payments increased as the
number of months in payment delay increased.

34
Figure 4:9 Repayment status for last 6 months vs probability of default

A plot of the distribution of age with marital status and the tendency to default showed
that, customers within 30 and 50 age brackets and single customers of age 20-30 tend to
default payment with the single customers having a higher probability to default.

Figure 4:10Distribution of age with marital status vs tendency to default

35
Females in the age group 20-30 had a very high tendency to default payment compared
to males in all age brackets. In addition, the proportion of females who were not likely
to default in the next billing cycle was also higher than the males.

Figure 4:11 Distribution of sex and tendency to default

A scatter plot of the bill amounts against the payment amounts over the observation
period revealed that there is a higher proportion of customers whose bill amount is
high, but the payment done for the same billing month is very low. This is inferred by
the concentration of datapoints along the y-axis and near to 0 on the x-axis:

36
Figure 4:12 Distribution of bill amounts vs payment amounts

4.1.2 Functional Requirements

The system shall provide the following primary functions:

a. The system shall allow credit officers to submit new customer details for
classification
b. The system shall determine the delinquency classification of credit cardholders
based on the submitted information
c. The system shall present the results of classification to the credit officers

4.1.3 Non-Functional Requirements

Credit card data represents one of the most sensitive information that financial
institutions must secure in day-to-day business operations. The system shall endeavor
to fulfill the following:

37
 a. System Security
The system shall follow secure coding standards, and high data security by complying
with payments security requirements as defined in the Payment Card Industry Data
Security Standard (PCI DSS) specification and other applicable security specifications.

b. Data Integrity
The system shall ensure accuracy, consistency, and completeness of data. Where
applicable, the system shall track all events through audit logs, and enforce approval
checks for the completion of system events.

c. System Reliability and Performance

The system shall provide high reliability and scalability and produce correct and
consistent results in its life cycle.

d. System Performance
The system shall have high performance, with the in-built capability serving high traffic
with low response time.

4.3 System Architecture

The system will consist of the frontend and backend components. The frontend
responsibility will be to render the presentation logic to the users, who will sign in into
the system and access the prediction form. The prediction form will allow the user to
key in new credit cardholder customer data for prediction. These data will be sent to the
backend using a REST protocol to the API component which will then sanitize and
validate input and finally forward to the prediction engine. The prediction engine
consists of an XGBoost classifier algorithm which will then generate prediction score,
either 0 or 1, which shall be returned as a response by the API to the frontend
component and presented to the user as feedback. This flow is captured in the
architectural diagram below:

38
[ CITATION Placeholder1 \l 1033 ]

Figure 4:13 System Architecture

39
4.4 System Design

4.4.1 Use Case Diagrams

Figure 4:14 Use case diagram

Description of Use cases

This section provides detailed descriptions of the above use cases using the two-column
fully dressed format:

Table 4:2 Use case UC1- Login

Use Case Login

Primary Actors Credit officer, System Admin

40
Post-conditions User verification and access granted
Main Success Scenario
Actor Intention System Responsibility
1. User submits email and password
combination
2. System authenticates users
Extensions
At any time when the user forgets their password:
a). A user will initiate a password recovery process by submitting their email
address
b). The system shall send an email notification with a link to set-up a new
password
c). The user shall submit their new password and attempt to login

Table 4:3 Use case UC2 - Get Customer Classification

Use case: Get Customer Classification

Primary Actors Credit Officer
Pre-Conditions User is successfully authenticated
Post-Conditions Customer classification result
Main Success Scenario
Actor actions System Responsibility
1.User submits customer details
2. System performs feature selection
3. System classifies customer with the
given features
4. Return customer classification

Table 4:4 Use case UC3 - Manage users

Use Case Manage Users

Primary Actors System Admin
Pre-Conditions User is successfully authenticated
Post-conditions User profiles are updated accordingly
Main Success Scenario
Actor Actions System Responsibility
1. Submits user profile information
2. System updates user profile

41
 information

4.4.2 Sequence Diagram

A sequence diagram highlights the order of operations from start to finish, and is
depicted in the figure below:

Figure 4:15 Sequence Diagram

42
4.4.3 ERD

Figure 4:16 Entity relationship diagram (ERD)

4.4.4 Database Schema

43
Figure 4:17 Database Schema

4.4.5 Class Diagram

44
Figure 4:18 Class diagram

4.4.6 Wireframes of the system

A wireframe is a mock-up or a skeletal visual representation of the system for

conceptualizing the system layout components. The following figure captures the login
feature of the system:

45
Figure 4:19 Login wireframe

Once a user has logged into the system, the system will redirect them to a screen that
allows them to enter customer details and predict the likelihood of this customer
missing payment in the current billing cycle. This is captured in the wireframe below:

Figure 4:20 Prediction form wireframe

46
 Chapter 5: System Implementation and Testing

5.1 Introduction

This chapter describes how the prediction system was developed, tested, and validated.
The system consisted of a web solution with frontend, and backend modules. The
frontend module provided users with graphical interfaces for interacting with the
system. The backend module was an integration of a REST API interface serving the
frontend requests and internal linkage to the prediction model. This chapter covers how
the model was developed and how the system was integrated to form a web-based
software product.

5.2 Model Development

The model was developed using the Python programming language and used the
scikit-learn library for implementation. The scikit-learn is an open-source machine
learning library that is extensive, accessible and provides high-level API interaction for
most machine learning methods and statistical modelling for classification, regression,
clustering, and dimensionality reduction.
Data pre-processing was first performed as described in section 4.1.1.2 and included
derivation of the credit utilization ratio and the overdraft amounts. The CSV dataset
was first loaded into a panda’s data frame as shown below:

Figure 5:21 Script for loading dataset

47
The credit utilization ratio was generated as follows:

Figure 5:22 Script for credit utilization ratio function

The overdraft amounts for each of the 6 observation months were computed as follows:

Figure 5:23 Script for overdraft function

5.2.1 Model Tuning

The XGBoost algorithm has several parameters that can influence the accuracy and the
training speed of a model, and this study utilized the following parameters: the
n_estimators, early_stopping_rounds, learning_rate and n_jobs. The value of n_estimators
influence underfitting and overfitting of the model - a low n_estimators value causes
underfitting which results to inaccurate predictions on both training and validation
data, while a large value causes overfitting which results to accurate predictions on the
training data but inaccurate predictions for any other set of data. The
early_stopping_rounds define the number of model iterations that can be performed
with significant improvements on the validation score. A good training practice is to set
a high value for n_estimators and then use the early_stopping_rounds to find the
optimal time to stop iterating. The learning_rate is a multiplier that is used as a factor

48
of prediction from each component model before summing up the predictions. A small
learning_rate value is usually used since it yields more accurate models, though at the
expense of longer training time. The n_jobs define the number of CPU machines that
can be allocated to the model for parallelism to build models faster. A good practice is
to set the value of n_jobs to be equal the number of processing cores in the host
machine.
The Bayesian optimization parameter-tuning algorithm was used to obtain a best-
optimized set of parameters due to its ability to constantly learn from previous
optimizations and requires fewer samples to learn or derive the best values. A 10-fold
validation was then performed for each specified set of parameters in each iteration.
This is as shown in the figure below:

Figure 5:24 Script for parameter model tuning

The best parameters were then extracted and used to fit the model as shown in the
following figure:

49
Figure 5:25 Script for Bayesian optimization

5.3 System Implementation

The frontend module was implemented using React, which is an open source JavaScript
library for building user interfaces. It offers fast, scalable and simplicity capabilities
when creating single page applications and the ability to re-use user interface
components in a clean way. The frontend module allowed users to be authenticated
using a login form, where they are required to submit valid authentication details, to
which they’re directed to the authorized resources as per their profile. System
administrators have the capability to define system configurations and manage users,
while credit officers have the capability to perform predictions based on customer
details and receive feedback from the system.
For all these interactions, the frontend module makes asynchronous requests to the
backend module through a REST API. The backend module was developed using the
Python programming language and used the Flask framework which provides a
minimalist approach in development, making it easy and fast to develop and deploy
backend modules. A MySQL database server was used to store all the data relating to

50
users, transaction logs – which are the predicted transaction events - and audit log
trails. The following figures capture the main screens that a credit officer must interact
with to generate a prediction of a given customer:

Figure 5:26 Login Screen

51
Prediction form

Figure 5:27 Prediction Form Screen

Prediction results

Figure 5:28 Prediction results

52
5.4 System Testing

The study performed the following testing techniques to ascertain the fulfillment of the
functional and non-functional requirements outlined in section 4.1.2 and 4.1.3:
Table 5:5 Testing Approaches

Testing Technique Focus

Unit testing Check internal data structures,
validation and sanitation of both input
and output data.
Integrated testing Ascertaining that the various integrated
system components are working
efficiently
Acceptance testing Checking the deliverables of the system
against the expected business
requirements
Security testing Eliminating security vulnerabilities and
compliance to security policies and
requirements for credit card data
Performance testing Ensuring the system responds within
considerable time when supplied with
input data.

5.5 System Validation

When training the model, each iteration of tuning was subjected to a 10-fold cross
validation to avoid overfitting and achieved a prediction accuracy 81.62% based on the
evaluation criteria of a confusion matrix. The following table shows the classification
results of the model:
Table 5:6 Model Classification Results

Classification Precision Score Recall F1-Score

0 0.84 0.94 0.89
1 0.64 0.37 0.47

53
The performance of the model was further compared with other classifiers by training
with the same data used in the model, and obtained the following results which showed
that the XGboost model outperformed the other classifiers and therefore, is a suitable
prediction model for the study:

Table 5:7 Comparison of Prediction Accuracy

Method Prediction Accuracy

KNN 77.42
Logistic Regression 77.87
Linear Discriminant Analysis 80.97
Random Forests 0.81

Chapter 6: Discussion

6.1 Introduction

This chapter discusses the results of the study according to the objectives set out at in
chapter one. The objectives of the study were to analyze the factors that influence

54
delinquency, examine the classifiers used in predicting delinquency, develop a model
for predicting delinquency and testing the performance of this model.

6.2 Determinants of delinquency scoring

The study examined socio-economic, socio-demographic characteristics and behavioral

characteristics of a cardholder and transaction activities relating to consumption and
repayment of the credit card debt. The socio-economic characteristics of age, marital
status and age were found to be key factors that influence the propensity to repay.
Behavioral characteristics such as the repayment pattern as exhibited by the pay
amounts show cased that cardholders with no revolving balance and with few months
in arrears were likely to repay their debt in the next billing cycle. Credit cardholders
with a consecutive high number of repayment months in arrears had a high likelihood
of missing payments and may point to the issue that the customer is in financial distress
or is caught up in a cycle of persistent debt.
The study derived two variables from the dataset: overdrafts and credit utilization ratio.
Overdrafts which were derived from the difference between the bill amounts and the
credit limits were highly significant towards the probability of missing payments while
the credit utilization ratio values were less significant.
The study limited to developing the model with respect to the available data and thus,
some variables were not captured in the model such as direct-debit cancellations, card
inactivity, cash advances, bounced payments and the type of products or services
purchased by a credit cardholder. The significance of these attributes, therefore, could
not be justified in this study.

6.3 Classifiers used in predicting delinquency of credit card accounts

The study investigated both single and ensemble classifier techniques that fall into
either statistical or machine learning algorithms that are used in predicting the
likelihood of a credit cardholder missing payment. Ensemble classifiers were found to

55
be outstanding classifiers due to their ability to improve on the weaknesses of single
classifiers by trading-off variance and bias factors of the base models used.

6.4 Developing the model

The model was developed using the XGBoost algorithm and implemented using the
python programming language. The first step was to perform data-preprocessing and
exploratory data analysis for identifying the key variables for the study. The model
hyperparameters were tuned using the Bayesian optimization approach and then fitted
with all the dataset attributes plus the derived attributes. The model was then
integrated into a REST API built with the Flask framework, and the functionalities
exposed using graphical user interfaces built using the React framework.

6.5 Testing the performance of the model

The model achieved 81.62% prediction accuracy and outperformed a number of

classifiers that were benchmarked against it. This result is suitable for prediction and an
issue can adapt it and extend it to assign a probability score which can inform the
business the intervention to be taken to reduce the potential risk exposure. The model
can be further be extended to define a recommendation strategy which will be used for
the credit management team for proactive engagement with the cardholders.

Chapter 7: Conclusion and Recommendation

7.1 Conclusions

The study had the following 4 objectives with the goal of developing a pre-delinquency
model for credit cardholders:
a) To analyze the cardholder characteristics and transaction activities that influence
the delinquency credit card accounts. The study analyzed socio-economic, socio-
demographic and transaction-oriented behavior and concluded that, age, gender,

56
 repayment patterns and overdrafts activities had significant influence on the
propensity to miss payment.
b) To examine the classifiers used in predicting delinquency of credit card accounts.
The study examined both single and ensemble classifiers that are used for
predicting delinquency of credit cardholders.
c) To develop a model for predicting delinquency of credit card accounts. The goal
of the study was achieved by developing an XGBoost model for predicting
delinquency.
d) To test the performance of the model in predicting the delinquency of credit card
accounts. The model achieved 81.62% prediction accuracy and outperformed a
number of single classifiers. This made this model suitable for classification
purposes in this problem domain.

7.2 Recommendations

From the study, the following recommendations were deduced:

a) Financial anxiety is a key contributor for credit cardholders accumulating debt
and financial institutions should include key indicators of financial anxiety from
transactions that span beyond the credit card account related activities such as
direct debit cancellations, overall credit ratings, revolving loan debts from other
accounts, etc.
b) To minimize risk exposure, financial institutions should revise the credit limits
and the minimum payment amounts depending on the repayment trends that a
given credit cardholder exhibits over a given observation period.
c) Pre-delinquency scoring modeling can perform well with aggregation of a
cardholders activities across different domains, with adjustments done with
respect to the current prevailing economic conditions, governing policies and
frameworks, and the risk appetite of the financial institutions.

57
7.3 Future Work

The developed model can be extended to calculate the actual probability of missing
payment and segment customers accordingly based on these values to determine the
nature of contact. This can further be broken down into stages within a given billing
cycle for trends to be easily observed and the right strategy be adopted based on the
trend.

58
 References

Agarwal, S., Ambrose, B. W. (Brent W., & Liu, C. (2006). Credit Lines and Credit
Utilization. Journal of Money, Credit, and Banking, 38(1), 1–22.
https://doi.org/10.1353/mcb.2006.0010

Ala’raj, M., Abbod, M., & Radi, M. (2018). The applicability of credit scoring models in
emerging economies: an evidence from Jordan. International Journal of Islamic and Middle
Eastern Finance and Management, 11(4), 608–630. https://doi.org/10.1108/IMEFM-02-
2017-0048

Barboza, G., Smith, C. and Boubacar, I., 2017. A Contribution to the Empirics of
Consumers' Anxiety Behavior on and in Credit Card Repayment. Credit Card
Management and Financial Literacy Among College Student. Journal of Financial
Management, Markets and Institutions, (1), pp.35-66. https://doi.org/10.12831/87059

Bhattacherjee, A. (2012). Social science research principles, methods, and practices.

University of South Florida. http://scholarcommons.usf.edu/cgi/viewcontent.cgi?
article=1002&context=oa_textbooks

Boughaci, D., & Alkhawaldeh, A. A. K. (2020). Appropriate machine learning

techniques for credit scoring and bankruptcy prediction in banking and finance: A
comparative study. Risk and Decision Analysis, 8(1–2), 15–24.
https://doi.org/10.3233/RDA-180051

Ciunova-Shuleska, A. (2012). The Impact of Demographic, Socio-economic and

Behavioral Characteristics on Attitudes Toward Credit Cards in Macedonia.
Mediterranean Journal of Social Sciences, 3(9), 199-206.

59
de Paula, D. A. V., Artes, R., Ayres, F., & Minardi, A. M. A. F. (2019). Estimating credit
and profit scoring of a Brazilian credit union with logistic regression and machine-
learning techniques. RAUSP Management Journal, 54(3), 321–336.
https://doi.org/10.1108/RAUSP-03-2018-0003

Esgalhado, B., Higginson, M., Jacques, F., Matecsa, M. and Selandari, F. (2019). Getting a
grip on bad debt: Practical steps to help utilities boost their resilience: McKinsey Digital‌.

Financial Conduct Authority (2017). Credit card market study: consultation on persistent
debt and earlier intervention remedies.

Finlay S. (2010) Collections (Early Stage Delinquency). The Management of Consumer

Credit. Palgrave Macmillan, London.

Firafis, H. (2015). Determinants of loan repayment performance: Case study of Harari

microfinance institutions. Journal of Agricultural Extension and Rural Development, 7(2),
56–64. https://doi.org/10.5897/JAERD2014.0622

Hartmann, J. (2021). Classification Using Decision Tree Ensembles, Einhorn, M., Löffler,
M., de Bellis, E., Herrmann, A. and Burghartz, P. (Ed.) The Machine Age of Customer
Insight, Emerald Publishing Limited, Bingley, pp. 103-117.
https://doi.org.ezproxy.library.strathmore.edu/10.1108/978-1-83909-694-520211011
Kan M.P.H., Fabrigar L.R. (2017) Theory of Planned Behavior. In: Zeigler-Hill V.,
Shackelford T. (eds) Encyclopedia of Personality and Individual Differences. Springer,
Cham. https://doi.org/10.1007/978-3-319-28099-8_1191-1

Kaur, H. and Kumari, V. (2020), "Predictive modelling and analytics for diabetes using a
machine learning approach", Applied Computing and Informatics, Vol. ahead-of-print
No. ahead-of-print.
https://doi.org.ezproxy.library.strathmore.edu/10.1016/j.aci.2018.12.004

Khemais, Z., Nesrine, D., & Mohamed, M. (2016). Credit Scoring and Default Risk
Prediction: A Comparative Study between Discriminant Analysis & Logistic Regression.

60
International Journal of Economics and Finance, 8(4), 39.
https://doi.org/10.5539/ijef.v8n4p39

Kiarie, F. K., Nzuki, D. M., & Gichuhi, A. W. (2013). Influence of Socio-Demographic

Determinants on Credit Cards Default Risk in Commercial Banks in Kenya. International
Journal of Science and Research 4(5), 1611-1615.

Kumar, V. (2013). Collections scorecards and risk segmentation [Online]. Available at

https://www.experian.co.uk/blogs/latest-thinking/decisions-and-credit-
risk/collections-scorecards-and-risk-segmentation/ (Accessed: 31 May 2020)

Lada M & Wejer-Kudelko M (2018). Success factors and barriers to the effective debt
collection process. Scientific Works of the Wrocław University of Economics, (51`5), pp.
147-155.

Lessmann, S., Baesens, B., Seow, H.-V., & Thomas, L. C. (2015). Benchmarking state-of-
the-art classification algorithms for credit scoring: An update of research. European
Journal of Operational Research, 247(1), 124–136.
https://doi.org/10.1016/j.ejor.2015.05.030

Lin, L., Revindo, M.D., Gan, C. and Cohen, D.A. (2019), "Determinants of credit card
spending and debt of Chinese consumers", International Journal of Bank Marketing, Vol.
37 No. 2, pp. 545-564.

Ming‐Yen Teoh, W., Chong, S., & Mid Yong, S. (2013). Exploring the factors influencing
credit card spending behavior among Malaysians. International Journal of Bank
Marketing, 31(6), 481–500. https://doi.org/10.1108/IJBM-04-2013-0037

Mittal, S., Gupta, P., & Jain, K. (2011). Neural network credit scoring model for micro
enterprise financing in India. Qualitative Research in Financial Markets, 3(3), 224–242.
https://doi.org/10.1108/17554171111176921

61
Onay, C., & Öztürk, E. (2018). A review of credit scoring research in the age of Big Data.
Journal of Financial Regulation and Compliance, 26(3), 382–405.
https://doi.org/10.1108/JFRC-06-2017-0054

Régis, D. E., & Artes, R. (2015). Using multi-state markov models to identify credit card
risk. Production, 26(2), 330–344. https://doi.org/10.1590/0103-6513.160814

Shapiro, G. K., & Burchell, B. J. (2012). Measuring financial anxiety. Journal of

Neuroscience, Psychology, and Economics, 5(2), 92–103. https://doi.org/10.1037/a0027647

Singh, S., Rylander, D. H., & Mims, T. C. (2018). Understanding credit card payment
behavior among college students. Journal of Financial Services Marketing, 23(1), 38–49.
https://doi.org/10.1057/s41264-018-0042-0

Stavins, J. (2020). Credit card debt and consumer payment choice: What can we learn
from credit bureau data? Journal of Financial Services Research, 58(1), 59-90.
https://doi.org/10.1007/s10693-019-00330-8

Taneta- Skwiercz, D. (2018). Sustainable development indicators - Poland against the

background of European Union countries. Scientific Works of the Wrocław University
of Economics, (516), pp.121–132.

Team, S., 2016. Debt Management and Collection Analytics (Post Delinquency Vintage) |
Scoredata. [online] Scoredata.com. Available at: https://scoredata.com/debt-
management-and-collection-analytics-post-delinquency-vintage/ [Accessed 5 June
2020].

Thomas, L., Edelman, D. and Crook, J., 2017. Credit Scoring and Its Applications. 2nd
ed. Philadelphia: Siam, ISBN 978-1-611-97455-3, pp. 157-177.

Tsai, C.-F. and Hung, C. (2014). Modeling credit scoring using neural network ensembles,
Kybernetes, Vol. 43 No. 7, pp. 1114-1123.
https://doi.org.ezproxy.library.strathmore.edu/10.1108/K-01-2014-0016

62
Varma, R., 2015. Bolstering Credit with Pre-Delinquency Management. [online] BAI.
Available at: https://www.bai.org/banking-strategies/article-detail/bolstering-credit-
with-pre-delinquency-management/ [Accessed 5 June 2020].

Wong, K.Y. and Lynn, M. (2019), "Credit card cue effect: How mere exposure to credit
card cues promotes consumers’ perceived financial well-being and spending",
International Journal of Bank Marketing, Vol. 38 No. 2, pp. 368-383.

Zainudin, R., Mahdzan, N. S., & Yeap, M.-Y. (2019). Determinants of credit card misuse
among Gen Y consumers in urban Malaysia. International Journal of Bank Marketing,
37(5), 1350–1370. https://doi.org/10.1108/IJBM-08-2018-0215

Zhanga, C., & Ma, Y. (2012). Ensemble machine learning: Methods and applications.
Springer.

63