Comparative Study of Random Forest and Neural Network for Prediction in Direct

Marketing
Sandeep Deswal
Prince Shri Venkateshwara Padmavathy Engineering College, Chennai

Abstract: This paper deals with prediction in direct marketing for banks. Nowadays direct
marketing methods are used to reach potential customers as they involve reduced promotional
cost. For getting maximum business from the huge customer data sets accurate prediction of
response becomes crucial. The data considered here is of a Portuguese banking institution. Input
is a twenty attribute data set about bank clients while response is to predict if the client will
subscribe to term deposit or not. Comparative analysis is done between random forest and neural
networks in R platform for the stated prediction problem.

Key Words: Direct Marketing, Prediction, Random Forest, Neural Networks, Term Deposit.

1. INTRODUCTION
Direct marketing is a method to reach customers without use of an advertising middleman.
Direct marketing can be done in various ways like telephone calls, emails, brochures,
newsletters, text messages, emails and many others. Direct marketing aims in reducing
campaigning cost and maximizing return rate. For direct marketing, it is important to identify
target audience. In a particular area, the said audience is identified as per our needs and their
ability.

Once potential customers are identified it is always easy to identify the customers need and its
expectation from the product. The client can be explained on the benefits of the product by direct
interacting with him either meeting him directly or giving him utmost attention over phone. By
this the mood of the customer can be easily judged and all his misconception or preconceived
notions can be corrected immediately. With direct marketing the company representative
develops a personal touch with the customer and can expect the results faster. In today’s scenario
relationship, between the customer and company representative, plays a very important role in
success or failure of the product. One can expect results faster and more. By direct marketing the
company can make the feel of the product to the customer, which in turn clears any doubt for the
same. Thus it can be stated that profitability of direct marketing largely depends upon accurate
prediction of the potential customer.

Data considered here for classification and prediction of potential customers is of a Portuguese
banking institution. The marketing campaigns were based on phone calls. Inputs are twenty
different attributes about the client(mentioned in detail in section 4) and output is if client has
opted for term deposit (yes or no).Often, more than one contact to the same client was required,
in order to access if the product (bank term deposit) would be ('yes') or not ('no') subscribed.
We have done prediction for the mentioned data set using random forest (RF) and neural
network (NN). Leo Breiman [1] developed random forest. Literature shows that since then RF
has been used for classification and prediction for various problems like financial forecasting,
remote sensing, and genetic and biomedical analysis [2-9]. Neural network has also successfully
used in prediction problems [10, 12-14]. A general block diagram for data training and prediction
is shown in figure 1. Detail of techniques used are explained in the succeeding sections.

Input Output
Data Pre- Model Building Prediction
Attributes processing RF / NN RF / NN

Fig.1 Block diagram for prediction

Section 2 and 3 brief about theory of random forest and neural networks respectively. Section 4
explains the data of Portuguese bank. Section 5 gives the results and finally section 6 concludes
the paper.

2. RANDOM FOREST
Random forest (RF) is an effective algorithm for prediction problems and is widely used for data
regression and classification. Advantage of using random forest is that accuracy is maintained
even with inconsistent data; is easy to use, fast and robust. RF saves data pre processing and data
preparation time for numerical data. Drawback is that if there are many wrong tress then
prediction may go wrong and it is difficult to interpret. RF is widely implemented in R or
Python. In our work we are working on R platform. R is a open source software.

In RF algorithm various decision trees are made from random subsets of data. Or put in other
words RF is based on bagging approach to get a combination of decision trees with same
distribution for all the trees in the forest. This is pictorially depicted in figure 2.

Fig. 2. Random Forest – Supervised Classification Machine Learning Algorithm

RF algorithm can be divided in two stages:

Stage 1: Boosting: In this stage random data is bagged together to create ‘n’ random tress.
Collection of these trees makes a random forest.

Stage 2: Bagging: In this stage outcome of each tree for same variable is combined to get the
final predicted output. Final prediction is either based on result of each decision tree or that
appears maximum times in decision trees.

Figure 3 below shows steps used in our study for data pre-processing, training and prediction
using random forest.
Installing packages Shuffle Data Row wise
dataset <-
(randomForest and dataset[sample(nrow(d
readtext) ataset)),]

Defining Target Defining Input

Variable variables

Training the Dataset

Including Library trainDataset <-
dataset[1:(totalDataset *
(randomForest and
training/100),c(inputs, Data Pre-
readtext) target)]
Processing

Selection of testing Dataset

Reading testDataset <-
dataset[(totalDataset *
Data from
training/100):totalDataset,c(i
CSV file nputs, target)]
Model
Building

Model building

formula <-
as.formula(paste(target, "~",
paste(c(inputs), collapse =
"+")))

model <-
randomForest(formula,
trainDataset, ntree=500,
mtry=2)

Prediction of the Model Extracting Actual

Predicted <- Actual <-
predict(model, as.double(unlist(testD
testDataset) ataset[target]))

Model Evaluation Results

Fig. 3. Flow chart for prediction using random forest.

 3. NEURAL NETWORK
A neural network can be thought of as a pattern classifier because it maps an unfamiliar input
pattern to the closest (best represented) output pattern. In other words, a neural network can
identify the output that best represents an unknown input pattern in cases where an exact input
output functional relationship is not easily defined. The input and output patterns may be binary
or analog signals [12-14]. Backbone of Central Nervous System (CNS) is the neuron. An
individual neuron is interconnected to many other neurons forming a densely connected network
[13]. Mathematical model of a neuron is as follows:

X(1,2….n)=Input samples fpn (k)=Output sample of the neural network

n
 (k )   wk uk  B
k 0 (1)
fpn(k)=  [ (k)]
()is non linear activation function;
w and u are the weights and the input vectors,
B is the threshold value
fpn is output of neural network.

Although a single neuron can perform certain simple pattern detection function, the power of
neural computation comes from the number of neurons connected in network structure. The most
commonly used neural network architecture in system identification and control is multilayer
feedforward neural network (MFFNN) with an error backpropogation (BP) algorithm as shown
in figure 4. The neural network is thus a pattern classifier or associator, receiving patterns
directly from data at the input and delivering output patterns that will give some quantitative
information about the system. The problem is to “train” the network by adjusting the weights and
thresholds so that the proper prescribed prediction will be made [10].
Fig. 4 . Multilayer feed forward Neural Network.

4. THE DATASET UNDER STUDY[11]

Data under study is of a Portuguese banking institution for direct marketing of term deposits.
Phone calls are the basis of the data. Twenty different attributes which include information about
potential customer like age, occupation etc were inputs of the model and output is if the client
would take term deposit or not (yes or no). The data set is a .csv file with 41188 examples
(number of clients), 20 inputs ordered date wise from may 2008 to november 2010. Details of
various attributes are given in Table 1.First twenty attributes are input and last attribute the target
output.

Table 1. Attribute Information

S.No. Feature Description Type

1 Age Age of client in years Numeric

2 Balance Client’s average annual balance in Euros (€) Numeric

3 Day Client’s last contact day Numeric

4 Duration How long it takes to contact the client in Numeric

seconds
S.No. Feature Description Type

5 Campaign Number of contacts performed for client Numeric

during the current campaign

6 Pdays Number of days elapsed since last contact Numeric

from previous campaign

7 Previous Number of contacts performed for the client Numeric

before current campaign

8 Employment Variation rate - quarterly indicator Numeric

9 Consumer price Monthly indicator Numeric

index

10 Consumer confidence Monthly indicator Numeric

index

11 Number of Quarterly indicator Numeric

employees

12 Job Type of job held by client Categorical-'admin.','blue-

collar','entrepreneur','housemaid','m
anagement','retired','self-
employed','services','student','techn
ician','unemployed','unknown'

13 Marital Client’s marital status Categorical-

'divorced','married','single','unknow
n'; note: 'divorced' means divorced
or widowed

14 Education Client’s highest educational qualification Categorical-

‘basic.4y','basic.6y','basic.9y','high.
school','illiterate','professional.cour
se','university.degree','unknown'

15 Default Is the client in default of credit facility? Categorical- 'no','yes','unknown'

16 Housing Does the client have housing loan? Categorical- 'no','yes','unknown'

17 Loan Does the client have personal loan? Categorical-'no','yes','unknown

18 Contact Client’s contact communication type Categorical-'cellular','telephone'

S.No. Feature Description Type

19 Month Client’s last contact month Categorical-'jan', 'feb', 'mar', ...,

'nov', 'dec'
S.No. Feature Description Type

20 Poutcome Outcome of the previous marketing Categorical-:

campaign 'mon','tue','wed','thu','fri'

21 Output variable y or Has the client subscribed a term deposit? binary: 'yes','no'
n (desired target)

5. RESULTS
Training and prediction was successfully done for the above mentioned Portuguese banking
institution data using random forest and neural network.

5.1 Prediction using Random Forest

Prediction results were analysed in terms of out-of-bag (OOB) error, classification error (CE)
and root mean square error (RMSE) in random forest with different number of trees. Out-of-bag
is the error rate of the out-of-bag classifier on the training set. Classification error (CE) is
computed as in equation 2.

(2)

Where g = number of incorrectly classified samples;

a = total number of samples.

Table 2. Performance of Random Forest

Model (RF) OOB CE RMSE Execution

time
Number of trees (n) (seconds)

n = 50 9.25 % 4.56% 9.45% 5.12

n = 100 8.8% 4.44% 9.27% 6.28

n = 200 8.87% 4.40% 8.95% 10.58

n = 300 8.92% 4.52% 8.93% 16.8

n = 400 8.82% 4.43% 8.45% 19.69

n = 500 8.78% 4.42% 8.04% 23.09

Table 2 shows performance of random forest for prediction of Portuguese Banking Institution
with different number of trees (n= 50,100,200,300,400,500). This variation is directly related to
number of randomly selected variables at each node for cluster building (tree). Literature shows
that performance of RF is enhanced with increasing number of trees[10]. Our results for
prediction also verify the same. But there are always exceptions [10]; as in our case when
number of trees were increased from 200 to 300, OOB decreases by 5%.

5.2. Prediction using Neural Network.

Prediction and results of the given data using NN were analysed in terms of sum of squared
errors (SSE), root mean square error (RMSE). In both cases error is between actual output and
predicted output. Neural network’s performance is sensitive towards the number of layers in the
network. As number of layers is increased the performance of neural network is improved at the
cost of its complexity. In our study we have varied number of layers of NN in the range of 3 to
10. For NN program execution categorical attributes were changed to numeric values.

Table 3. Performance of Neural Network

Model (NN) SSE RMSE Execution

time
(seconds)

Neural Network 10.26% 4.50

Layers (l = 3)

Neural Network 9.56% 5.68

Layers (l = 5)

Neural Network 9.15% 9.96

Layers (l =7)

Neural Network 8.25% 16.18

Layers (l = 10)

Table 3 summarizes the various errors under study (SSE, RMSE) and execution time for
different number of layers in NN (3, 5, 7, and 10).

5.3 Comparative analysis of RF and NN

RMSE and execution time were computed in the prediction problem of Portuguese banking
institution using RF and NN. In table 4 comparative analysis of RF and NN is done on this basis.
From table 2 the least value of RMSE using RF is compared with the least Value of RMSE using
NN from table 3.
Table 4. Comparative analysis of RF and NN

Model RMSE Execution Time

RF (n=500) 8.04% 23.09

NN (l = 10) 8.25% 16.18

From table 4 it is seen that value of RMSE is slightly lesser in RF as compared with NN.

6. CONCLUSION
This paper explored random forest and neural network for prediction in direct marketing for a
Portuguese banking Institution. We cannot choose best machine learning among the various
techniques available in literature. Different techniques need to be validated for a particular
problem. This study compares well known neural network and random forest for the stated
problem. In RF prediction analysis was done by taking different number of trees and in neural
network different number of layers were considered. For both the techniques prediction results
were good; but in case of RF value of RMSE is slightly lesser than with NN. So we can conclude
that for our particular Portuguese banking institution problem RF is giving better results than NN
for prediction.

REFERENCES

[1] Breiman, L. 2001. Random Forests. Machine Learning, Vol. 45 Issue 1, pp. 5-32.

[2] Goldstein, B., Polley, E., and Briggs, F. 2011. Random forests for genetic association studies.
Statistical Applications in Genetics and Molecular Biology, Vol.10, Issue 1, pp. 1–34.

[3] Siroky, D. 2009. Navigating random forests and related advances in algorithmic modeling.
Statistics Surveys, 3:147–163.

[4] Jiang, P., Wu, H., Wang, W., Ma, W., Sun, X., and Lu, Z. 2007. Mipred: classification of real
and pseudo microrna precursors using random forest prediction model with combined features.
Nucleic Acids Research, Vol.35, Issue 2. pp. 339–344.

[5] Palmer, D., O’Boyle, N., Glen, R., and Mitchell, J. 2007. Random forest models to predict
aqueous solubility. J Chem Inf Model, Vol.47, Issue 1, pp. 150–158.

[6] Shi, T., Seligson, D., Belldegrun, A., Palotie, A., and Horvath, S. 2005. Tumor classification
by tissue microarray profiling: Random forest clustering applied to renal cell carcinoma. Modern
Pathology, Vol.18, Issue 4, pp. 547–557.

[7] Baoxun Xu, Junjie Li, Qiang Wang, Xiaojun Chen, 2012. A Tree Selection Model for
Improved Random Forest, Bulletin of advanced technology research, Vol. 6 No.2 Apr. 2012.
[8] Olatunji Apampa, Tilburg University. Evaluation of Classification and Ensemble Algorithms
for Bank Customer Marketing Response Prediction, Journal of International Technology and
Information Management, Vol. 25 No.4, 2016.

[9] Esra Akdeniz Duran, Ayça Pamukcu, Hazal Bozkurt. Comparison of Data Mining
Techniques for Direct Marketing Campaings, Journal of Engineering and Natural Sciences, 32,
2014,pp 142-152.

[10] Muhammad Waseem Ahmad, Monjur Mourshed,ANN for high resolution Yacine Rezgui.
Trees Vs Neurons: comparison between random forest and prediction of building energy
consumption, Journal of Energy and Building, Elsiever, Vol. 147, 2017, pp. 77-89.

[11] S. Moro, P. Cortez and P. Rita. A Data-Driven Approach to Predict the Success of Bank
Telemarketing. Decision Support Systems, Elsevier, Vol.62, June 2014,pp 22-31

[12] S.A. Kalogirou, M. Bojic, Artificial neural networks for the prediction of the energy
consumption of a passive solar building, Energy Vol.25 No.5, 2000, pp 479–491.

[13] A. Azadeh, S. Ghaderi, S. Sohrabkhani, Annual electricity consumption forecasting by

neural network in high energy consuming industrial sectors,Energy Convers. Manag. Vol.49
No.8 2008,pp 2272–2278.

[14] Y. Cheng-wen, Y. Jian, Application of ANN for the prediction of building energy
consumption at different climate zones with HDD and CDD, in: Future Computer and
Communication (ICFCC), 2010 2nd International Conference on, vol. 3, 2010, pp. 286–289,