Machine Learning Techniques Final Report

(Fall, 2020)
Team:
ML_Explorer

Member:
Min-Hung Lo (B05502062)
Pai-Hung Cheng (R09522717)
Ting-Yi Chen (R09543028)

Instructor:
Hsuan-Tien Lin

by the beautiful mathematics of the traditional machine

Abstract—This report provides three different machine
learning models for predicting the daily revenue of
specific hotel booking platform. The selected models are
Deep Learning (DL), Gradient Boosted Machine (GBM)
and Support Vector Machine (SVM). Apart from the
methods above, linear regression (LR) had been tried
out at first to make sure whether the task could easily be
solved linearly. A brief theoretical review to these
models and comparison from different perspectives are
presented. We also introduce how to pre-process the
dataset, split the original data into train and validation
subset and choose the best tuning parameters with
minima validation error calculated by mean absolute
error (MAE). The best result is achieved by GBM with
public score showing 0.407895 and private score showing
0.402597.
I. INTRODUCTION
achine learning is now a widely used technique
M in several fields. In this competition, we are
asked to help the hotel booking platform to predict the
future daily revenue, which is the average daily rate
(ADR) multiplied by the number of days the customer
is going to stay in the room. The goal of the prediction
is to accurately infer the future daily revenue of the
platform, where the daily revenue is quantized to 10
scales. For example, revenue lies between 0 to 10000
will be labeled 1, and 10001 to 20000 will be 2, and so
on. To solve this problem, DL is gaining popularity
due to its supremacy in terms of accuracy when dealing
with huge amount of data, so we considered DL to be
one of the three candidates to conquer these 90000-
training data. On the other hand, we are also impressed
learning models. Hence, we come up with an idea to
make an inner competition between DL and other 2
traditional models. Here, SVM and GBM are chosen
because the former has no aggregation tricks, while the
latter has.
II. MODEL SELECTION
A. Deep Learning
DL is a class of machine learning algorithms that uses
multiple layers to progressively extract higher-level
features from raw data, which is proved to be able to
model complex non-linear relationship.
There are diverse types of neural networks, but they
always consist of the same components: neurons,
weights, biases, activation functions, learning rate and
optimizer. A variety of outcomes are generated,
resulting from different combination of parameters.
Since it takes times to train a neural network, the vital
task for us is to select the best combination of hyper-
parameter.
B. Gradient Boosted Machine
GBM is a supervised learning model which
combines gradient descent with boosting and can deal
with both regression and classification problems [1].
Take classification problem as an example, GBM starts
with a weak learner such as decision stump, and then
increases the weights of misclassifications while
lowering the weights for those correctly classified [2].
We repeat this process for a specified number of
iterations. Those misclassifications arise from the
 TABLE I TABLE II
COMPARISON BETWEEN DL, GBM AND SVM ADVANTAGES AND DISADVANTAGES OF GBM AND SVM

Performance DL GBM SVM Model Advantages Disadvantages

aLow bHigh cMedium
Efficiency Makes weak learner Improves to minimize
dMedium dLow powerful and provides all errors which causes
Scalability High
Interpretability Low dMedium dHigh predictive accuracy overfitting
Popularity High Medium Low Can optimize on Trains sequentially
a
The time complexity of DL is rather complicated and is affected by several different error with respect to errors
factors [3]. GBM functions which consumes time
b
The time complexity is 𝑂(𝑁𝑙𝑜𝑔𝑁 ∗ 𝑑 ∗ 𝑚) for GBM [4]. Has high flexibility
c
The training time complexity is 𝑂(𝑁 2 ∗ 𝑑) and the testing time Works great with which results in
complexity is 𝑂(𝑁𝑆𝑉 ∗ 𝑑) for SVM [5]. categorical and complicated
dThe structure for GBM is tree-based and for SVM is generalized linear
numerical values interactions between
model. parameters
Suitable for medium size
Weak resistance to noise
previous iteration is identified as gradients which tell dataset
us how to improve the model in the next iteration. The SVM The classifying
More effective in high
error function is given as hyperplane gives no
dimensional spaces
specific explanation [9]
𝑚𝑖𝑛 𝑚𝑖𝑛 1 𝑁
𝜂
∑
ℎ 𝑁 𝑛=1
𝑒𝑟𝑟(∑𝑡−1
𝜏=1 𝛼𝜏 𝒈𝜏 (𝒙𝑛 ) + 𝜂ℎ(𝒙𝑛 ), 𝑦𝑛 ), (1)
E. Discussion
where 𝜂 is the step of gradient descent, ℎ is called Due to the variety of machine learning models, three
functional gradient, 𝑁 is the number of rows in the specific perspectives should be pointed out to select
dataset, 𝑡 is the number of optimizing counts, 𝛼𝜏 the most suited model for this problem.
represents the weight of misclassification mentioned 1) Efficiency
1−𝜖𝜏 Computational time consumption of a model on an
above, and is defined as 𝑙𝑛 (√ ) , 𝜖𝜏 is the input of size N to arrive at the output.
𝜖𝜏
prediction error rate, 𝒈𝜏 is hypothesis, 𝒙𝑛 is the training 2) Scalability
data and 𝑦𝑛 is the training label. The error here is The ability to deal with any amount of data.
calculated with 0/1 error [6]. 3) Interpretability
The ability of a model for human to go through each
C. Support Vector Machine step of the algorithm and check if each step is
SVM has been used to define a space in which reasonable to them [10].
different classes are maximally separable, i.e., SVM In addition, the popularity is considered as a
classifier (SVC) learns from the training dataset and reference. Table I compares DL, GBM and SVM with
project into a higher dimensional space where two these four aspects, where GBM reveals the highest
classes can be separated by a hyperplane which efficiency with proper scalability and interpretability.
maximizes the margin separating the classes. Besides from those superiorities, the tree-based
Combined with the kernel trick, SVC can solve non- structure also makes GBM a well-known model to
linear problem as well. handle categorical data. Thus, we recommend GBM
As a branch of SVM, SVM regression (SVR) the best model. The cons and pros of GBM and SVM
supports the regression problem. Instead of finding a are summarized in Table II.
hyperplane with largest margin, SVR aim to decide a
decision boundary at a distance from the original III. DATA PREPROCESSING
hyperplane such that data points closest to the
hyperplane or the support vectors are within that A. Preliminary work
boundary line [7]. 1) Extract stays_in_week_nights and
stays_in_weekend_nights from the
D. Linear Regression original train.csv before feature transformation.
LR is a linear approach to model the relationship 2) Create necessary .csv files by popping out
between a scalar response and one or more explanatory ADR and cancellation labels.
variables [8]. This method is mostly used for
forecasting and finding out cause and effect
relationship between variables.
 TABLE III
PARAMETERS AND VALIDATION ERRORS OF DL
Parameter I Parameter II Parameter III
Validation Error
batch_size epochs init_mode
10 100 uniform 0.390
10 200 uniform 0.403
10 100 lecun_uniform 0.426

B. Feature Transformation x) company

1) Drop invalid rows xi) arrival_date_year
a) No revenue contribution b) Method
b) adr < 0 For each feature, combining all the
c) stays_in_week_nights + stays_in_weekend_ categories in the train and test data as the
nights = 0 scaler.
d) adults + children + babies = 0 i) fillna('N/A')
2) Drop useless fields ii) LabelEncoder+MinMaxScaler(-0.5,
a) Data without meaning 0.5)
i) ID iii) OneHotEncoder
b) Data not given in test data iv) TargetEncoder
i) reservation_status 6) Given range categorical features
ii) reservation_status_date a) Repeated time
3) Pop important fields i) month
a) Revenue components b) Method
i) is_canceled i) Map with dictionary {'Jan': 1, ..., 'Dec':
ii) adr 12} + MinMaxScaler(0, 1)
4) Numerical features
a) Time period, people or status change C. Encoding Effectiveness and Discussion
i) lead_time 1) OneHotEncoding
ii) arrival_date_week_number Several features include plenty of categories, such as
iii) stays_in_weekend_nights/stays_in_week Country and Agent. Compare to the 28 features
_nights originally, OneHotEncoding brings up the number of
iv) days_in_waiting_list features up to 900 which creates redundancy and
v) adults, children, babies causes memory issues.
vi) previous_cancellations 2) TargetEncoding
vii) prev_booking_not_canceled In target recording, features are replaced with a
viii) booking_changes blend of posterior probability of the target given
ix) total_of_special_requests categorical value and the prior probability of the target
x) required_car_parking_spaces over all the training data [10]. This method takes in
b) Method target-explainable value while encoding but requires
For each feature, combining all the different careful validation in case of overfitting.
numerical value in the train and test data as Apparently, TargetEncoding is preferred over
the scaler. OneHotEncoding to reduce training time and increase
i) fillna(0) physical meaning. Numerical features are processed
ii) MinMaxScaler(0, 1) with MinMaxScaler which eliminates the effect of
5) Unknown range categorical features different numerical intervals.
a) Categories, Time
i) hotel IV. IMPLEMENTATION
ii) country
iii) market_segment, A. Validation
iv) distribution channel Validation is a touchstone of model performance.
v) reserved_room_type We find the best model and corresponding parameters
vi) assigned_room_type by calculating the validation error before uploading
vii) deposit_type
viii) customer_type
ix) agent
 TABLE IV
PARAMETERS AND VALIDATION ERRORS OF GBM
Parameter I Parameter II
max_depth Validation Error Public Score Private Score
n_estimators
3 0.317 - -
10 0.232 0.329 0.442
50 50 - -
0.298
100 0.304 - -
3 0.291 0.408 0.403
10 0.245 0.355 0.455
100
50 0.291 - -
100 0.291 - -
3 0.298 - -
10 0.218 0.355 0.455
200
50 0.298 - -
100 0.298 - -

our result, so we can improve the correctness without

entry limitations.
sklearn.model_selection.train_test_split provides an
efficient way to help us calculate the validation error.
We choose to split the data with respect to the date into
two folds, where the first fold contains 80% of the
original one and the second fold contains the
remaining 20%.
Setting train_test_split(shuffle=False) brings us a
benefit since older data is used for training while
newer data is used for validation, which makes sense
to this time-dependent problem.
In this problem, we take advantage of the validation
tricks twice. One for predicting cancellation and the
other for predicting ADR.
B. Data Grouping
Initially, each row is grouped date by date, so all the
features are summed up for one single day. For
example, the total number of adults in one day is
condensed into one cell. The categorical features are
also added up after label encoding. And the prediction
target would be the daily revenue label. With this
manner, it is very efficient for training. Since the
classification label directed corresponds to the train
label.
However, it is found that this method somehow
misses or mixes the features. Thus, we turn to let each
row data remained as one unit, and the predicted term
become cancellation and ADR rather than integers.
C. Cancellation Prediction
Since test data has no cancellation records, the prior
work is to predict whether the order is canceled. We
aim to find the smallest validation error which is
calculated using MAE.
D. ADR and Label Prediction
Next, with cancellation features added, we can
predict ADR for those valid orders and extract output
labels. We aim to find the smallest validation error
which is calculated using MAE.
E. Parameters Tuning
1) GridSearchCV for DL
DL have hyper-parameters that we can adjust to
customize our model to fit the data. Hyper-parameter
optimization is an effective approach to objectively
search different combination of hyper-parameter and
choose a subset that results in a model that achieves
the best performance, which is available in the scikit-
learn Python library.
First, we set up our model with KerasClassifier,
where the neural network inside is built with 5 layers,
each with Dropout layer with dropout rate of 0.05. By
searching different batch size, epoch, and initial mode
of weights, GridSearchCV will select the best
combination of them with minimum loss.
2) For loops for GBM and SVM
GBM and SVM are rather simple models, so we
execute for loops to find parameter combinations
which lead to the best score. For example,
the n_estimators of GBM and the kernel function of
SVM.
V. EXPERIMENTAL RESULTS
A. Linear Regression
Starting with the easiest approach, some of the data
information are revealed and the baseline is set-up.
One advantage of LR is that the weight could be
obtained, which tells the importance of each feature.
And it is also with the weight observation, we find that
 TABLE V
PARAMETERS AND VALIDATION ERRORS OF SVM
Parameter I
Validation Error Public Score Private Score
C
1 0.464 0.687 0.753
10 0.464 0.592 0.688
100 0.497 - -
500 0.630 - -

TABLE VI
PARAMETERS AND VALIDATION ERRORS OF SVM ( FOR IS_CANCELED LABEL)
Parameter I Parameter 2
Validation Error
C Kernel
linear 0.181
1 poly 0.139
rbf 0.144
linear 0.174
10 poly 0.132
rbf 0.133
linear 0.174
100 poly 0.126
rbf 0.124
linear 0.174
200 poly 0.126
rbf 0.122

OneHotEncoder is unfavorable. And settle us on C. Gradient Boosted Machine

TargetEncoder. The reason is given below.
Consider using the MinMaxScaler and
OneHotEncoder for numerical and categorical features,
respectively. Due to the sparsity of data (some data are
left blank), the linear model tends to compensate the
blank by increasing the weight of other columns. In
one single row, the transformed one-hot-encoded value
is normally way higher than that of min-max-scaled
between [0,1]. Hence, from LR’s perspective,
modifying the weight of one-hot-encoded feature is
more efficient, which causes some parts of the one-hot-
encoded weights went to extremely high or negative.
B. Deep Learning
With the help of GridSearchCV, we adjust various
parameters — batch_size, epoch and init_mode, trying
every combination to find the minimum validation
error. Table III shows the best combination of
parameters. After checking validation error of each
combination of parameters, we find that there is a
trade-off between epoch and init_mode for our pre-
processed dataset. By adjusting different pre-
processing method, we can find the best combination
to better-fit our data [12].
We adjust two parameters in this experiment —
n_estimators and max_depth — while holding others.
Table IV shows the tuned parameters and the
validation error of each model, where public and
private scores are shown with the smallest validation
error and the default. We can observe that the
validation error becomes smaller when two terms
increase until reaching some specific values.
Therefore, we can draw a short conclusion that the
appropriate model complexity helps to strengthen the
model, but care must be taken to avoid overfitting.
D. Support Vector Machine
SVM has several types of kernel function such as
linear, sigmoid, polynomial, and radial basis function.
Particularly, radial basis function is the most popular
one, hence, we only choose the regularizer C as the
variable while other minor parameters remaining
unchanged. Table V gives a summary of the result,
where public and private scores are shown under the
same conditions as Table IV.
Besides from the result, the cancellation prediction
is also shown in Table VI with different combinations
of C and kernel.
E. Discussion
From the discussion above, we can find that GBM is
superior to other methods. From the whole practical
experience, we know that data pre-processing is a key
point to the model performance. If data pre-processed
is set up properly, SVM or DL may also reach the same
level to GBM.
VI. CONCLUSION
This work demonstrates three different methods to
solve the predicting problem. The process of feature
transformation and implementation are also described
in detail. Although DL and SVM are admittedly
powerful tools, GBM in this problem achieves the
lowest error as we expect because of the
straightforward data preprocessing, which is
unfavorable to other two models. In addition, the
efficiency of GBM gives us more chances to trial and
error with respect to the validation error.
VII. WORK LOADS
A. Min-Hung Lo
Data preprocessing, LR, some parts of SVM and
report writing.
B. Pai-Hung Cheng
Data preprocessing, DL and report writing.
C. Ting-Yi Chen
Data preprocessing, GBM, some parts of SVM and
report writing.
Available from:
https://www.sciencedirect.com/topics/engineering/sup
port-vector-machine
[8] Linear Regression Introduction.
Available from:
https://towardsdatascience.com/introduction-to-
machine-learning-algorithms-linear-regression-
14c4e325882a
[9] Advantage and Disadvantage of SVM
Available from:
https://dhirajkumarblog.medium.com/top-4-
advantages-and-disadvantages-of-support-vector-
machine-or-svm-a3c06a2b107
[10] Zachary C. Lipton, The Mythos of Model
Interpretability. 2016, Carnegie Mellon University
[11] Target Encoding
Available from:
https://medium.com/analytics-vidhya/target-encoding-
vs-one-hot-encoding-with-simple-examples-
276a7e7b3e64)
[12] Grid Search.
Available from:
https://docs.h2o.ai/h2o/latest-stable/h2o-docs/grid-
search.html

REFERENCES
[1] A Gentle Introduction to Gradient Boosting.
Available from:
http://www.ccs.neu.edu/home/vip/teach/MLcourse/4_
boosting/slides/gradient_boosting.pdf
[2] Understanding Gradient Boosted Machines.
Available from:
https://towardsdatascience.com/understanding-
gradient-boosting-machines-9be756fe76ab
[3] Rich, Lee and Ing-yi, Chen, Time Complexity of DL.
2020 International Conference on Mathematics and
Computers in Science and Engineering (MACISE)
[4] Time Complexity of GBM.
Available from:
https://medium.com/towards-artificial-
intelligence/time-and-space-complexity-of-machine-
learning-models-df9b704e3e9c
[5] Marc Claesen, Frank De Smet, Time Complexity of
SVM with RBF Kernel. 2014, Katholieke Universiteit
Leuven
[6] Machine Learning Techniques Lecture 11: Gradient
Boosted Decision Tree.
Available from:
https://www.csie.ntu.edu.tw/~htlin/mooc/doc/211_han
dout.pdf
[7] SVM Introduction.