sensors

Article
Comparison of Random Forest, k-Nearest Neighbor,
and Support Vector Machine Classifiers for Land
Cover Classification Using Sentinel-2 Imagery
Phan Thanh Noi 1,2, * and Martin Kappas 1
1 Cartography, GIS and Remote Sensing Department, Institute of Geography, University of Göttingen,
Goldschmidt Street 5, 37077 Göttingen, Germany; mkappas@gwdg.de
2 Cartography and Geodesy Department, Land Management Faculty, Vietnam National University of
Agriculture, Hanoi 100000, Vietnam
* Correspondence: phan.thanh.noi@gmail.com; Tel.: +49-551-399-805

Received: 20 November 2017; Accepted: 20 December 2017; Published: 22 December 2017

Abstract: In previous classification studies, three non-parametric classifiers, Random Forest (RF),
k-Nearest Neighbor (kNN), and Support Vector Machine (SVM), were reported as the foremost
classifiers at producing high accuracies. However, only a few studies have compared the
performances of these classifiers with different training sample sizes for the same remote sensing
images, particularly the Sentinel-2 Multispectral Imager (MSI). In this study, we examined and
compared the performances of the RF, kNN, and SVM classifiers for land use/cover classification
using Sentinel-2 image data. An area of 30 × 30 km2 within the Red River Delta of Vietnam with
six land use/cover types was classified using 14 different training sample sizes, including balanced
and imbalanced, from 50 to over 1250 pixels/class. All classification results showed a high overall
accuracy (OA) ranging from 90% to 95%. Among the three classifiers and 14 sub-datasets, SVM
produced the highest OA with the least sensitivity to the training sample sizes, followed consecutively
by RF and kNN. In relation to the sample size, all three classifiers showed a similar and high OA
(over 93.85%) when the training sample size was large enough, i.e., greater than 750 pixels/class or
representing an area of approximately 0.25% of the total study area. The high accuracy was achieved
with both imbalanced and balanced datasets.

Keywords: Sentinel-2; Random Forest (RF); Support Vector Machine (SVM); k-Nearest Neighbor
(kNN); classification algorithms; training sample size

1. Introduction
There is undoubtedly a high demand for land use/cover maps for the monitoring and
management of natural resources, development strategies, and global change studies [1–4]. Land
use/cover maps are one of the most important documents that provide information for various
applications, such as land use policy development, ecosystem services, urban planning, conservation,
agricultural monitoring, and land use/cover dynamic assessment [5–9].
Remote sensing satellite images are considered as one of the most important data sources for
land use/cover mapping [10] due to their extensive geographical coverage at an efficient cost while
providing irreplaceable information on the earth’s surface [11]. Land use/cover maps are usually
produced based on remote sensing image classification approaches [12–14]. However, the accuracy
and processing time of land use/cover maps using remote sensing images is still a challenge to the
remote sensing community [15].
Sentinel-2 is the latest generation Earth observation mission of the ESA (European Space Agency)
designed for land and coastal applications, and it includes the identical Sentinel-2 A and Sentinel-2 B

Sensors 2018, 18, 18; doi:10.3390/s18010018 www.mdpi.com/journal/sensors

Sensors 2018, 18, 18 2 of 20

satellites which launched in June 2015 and March 2017, respectively [16]. Sentinel-2 remains active
and enhances the mission of Landsat and SPOT (Systeme Probatoire d’Observation de la Terre) [17].
It is a system with a wide-swath, high spatial resolution (10–60 m), temporal resolution (ten days/five
days for Sentinel-2 A, B/Sentinel-2 A + B), and multi-spectral (13 spectral bands) capabilities. It has
also gained great attention in research due to its free access and global coverage. A wide range of
applications have been studied with Sentinel-2 A (and Sentinel-2 simulation data), such as soil moisture
mapping [18], mapping urban surface water bodies [19], forest stress monitoring [20], and quantifying
above ground biomass [21]. Clevers et al. [22] used Sentinel-2 data for retrieving LAI and leaf and
canopy chlorophyll content of potato crops. Particularly, in land use and land cover mapping, the
practicality of Sentinel-2 has been tested and showed the high potential of application [23,24]. However,
because it is a new type of satellite imagery, there are only a handful of studies using Sentinel-2 for
land use/cover mapping, and thus more research is necessary to conduct and evaluate the usefulness
of this imagery.
According to Lu and Weng [25], it is not only the imagery appropriateness but also the right choice
of classification method that affects the results of land use/cover mapping. In literature, a variety of
classification methods have been developed and tested for land use/cover mapping using remote
sensing data [26–28]. These methods range from unsupervised algorithms (i.e., ISODATA or K-means)
to parametric supervised algorithms (i.e., maximum likelihood) and machine learning algorithms such
as artificial neural networks (ANN), k-Nearest Neighbors (kNN), decision trees (DT), support vector
machines (SVM), and random forest (RF).
In the last decade, the nonparametric methods (machine learning based algorithms) have
gained great attention of remote sensing based applications. To understand the rise of machine
learning methods in land use/cover mapping, we searched the number of articles in the ISI
Web of Knowledge (Indexes in SCI-EXPANDED and SSCI) with different keywords (“land
cover”* AND “classification”* AND “Maximum Likelihood (/Artificial Neural Networks/k-Nearest
Neighbors/Decision Trees/Support Vector Machines/Random Forest)” during the timespan from
2007 to 2017.
Figure 1 shows that in this timespan, the use of SVM and RF classification algorithms increased
significantly. The number of articles using MLC and ANN have fluctuated throughout the years, but
has generally remained steady. In recent years (2014, 2015, and 2017), there were a handful of studies
using kNN, however, we have not found any publications using kNN in 2016 with our searched
keywords. It should be mentioned that in Figure 1, the high number of papers with the keyword MLC
does not mean that there is much research using MLC for classification. In fact, most studies from
our searched list used the MLC method as one of the criteria to compare to other machine learning
algorithms [11,29,30].
The results shown in Figure 1 reflect the studies in recent literature. Prasad et al. [31] stated
that DT is too sensitive to small changes in the training dataset. They also reported that DT is
occasionally unstable and tends to overfit in the model. The ideal value of k for the kNN classifier is
difficult to set [32]. The ANN method contains a high level of complexity in computational processing,
causing it to become less popular in remote sensing based classification applications. SVM and RF are
insensitive to noise or overtraining, which shows their ability in dealing with unbalanced data [33].
Therefore, among the nonparametric methods, SVM and RF are becoming increasingly popular in
image classification studies [34].
Several studies have been implemented in order to find the best classification algorithm for land
use/cover studies by comparing the performance of these classifiers either among themselves or
with other classification algorithms. However, their conclusions are quite different. For example,
in the studies by Adam et al. [34] and Ghosh and Joshi. [35], SVM and RF showed similar results of
classification. Khatami et al. [11] found that SVM, kNN, and RF generally outperform other traditional
supervised classifiers. Pouteau et al. [36] compared 6 machine learning algorithms (SVM, Naïve Bayes,
C4.5, RF, Boosted Regression Tree, and kNN) with 6 satellite data sets from different sensors (Landsat-7
choice of classification method that affects the results of land use/cover mapping. In literature, a
variety of classification methods have been developed and tested for land use/cover mapping using
remote sensing data [26–28]. These methods range from unsupervised algorithms (i.e., ISODATA or
K-means) to parametric supervised algorithms (i.e., maximum likelihood) and machine learning
Sensors 2018, 18, 18 3 of 20
algorithms such as artificial neural networks (ANN), k-Nearest Neighbors (kNN), decision trees (DT),
support vector machines (SVM), and random forest (RF).
ETM+, In the
SPOT, lastAirSAR,
decade,TerraSAR-X,
the nonparametric methods
Quickbird, (machine learning
and WorldView-2) basedEcosystems
for Topical algorithms)Classification
have gained
great
and stated that kNN better performs for the Landsat-7 ETM+ classification. Most recently, learning
attention of remote sensing based applications. To understand the rise of machine Heydari
methods in land use/cover mapping, we searched the number of articles in the ISI Web of Knowledge
and Mountrakis [37] studied the effects of the classifier selection, reference sample size, reference class
(Indexes
distribution, in and
SCI-EXPANDED andin per-pixel
scene heterogeneity SSCI) with different
classification keywords
accuracy using 26(“land
Landsatcover”* AND
sites with five
“classification”* AND (Naïve
classification algorithms “Maximum Likelihood
Bayesian, kNN, SVM, Tree(/Artificial Neural
ensemble, and Networks/k-Nearest
Artificial Neural Network).
Neighbors/Decision
They concluded that Trees/Support
SVM and kNN Vector
were theMachines/Random Forest)” during
best classification methods the classification.
for Landsat timespan from
2007 to 2017.

Figure 1.
Figure Number of
1. Number of articles
articles in
in the
the ISI
ISI Web
Webof
ofKnowledge
Knowledgefor
fora ageneral
generalsearch
searchonon
different keywords
different in
keywords
the last decade (2007–2017*). *Accessed on 13 August 2017.
in the last decade (2007–2017*). *Accessed on 13 August 2017.

In addition, to the best of our knowledge, only a limited amount of research was published
that compared and evaluated the performance of RF, SVM, and kNN with different training
sample strategies using Sentinel-2 imagery, especially in Vietnam. Therefore, it is practical for
a study to compare and evaluate the performance of RF, SVM and kNN for land use/cover
mapping over North Vietnam using the new satellite data, Sentinel-2 A. The objectives of this study
are: (i) to evaluate the performance of the three most increasing classifiers, RF, kNN, and SVM,
when applied to a Sentinel-2 image and (ii) to assess the effects of the training samples size,
strategies, and type (balanced/imbalanced) on the accuracy of the classification results of the three
aforementioned classifiers.

2. Materials and Methods

The overall methodology of the study is described in Figure 2. To fulfill the study objectives,
a study area was selected based on the land cover characteristics and the availability of remote sensing
imagery data. The remote sensing image was preprocessed, atmospherically corrected, and clipped to
the study area (Figure 3).
Sensors 2018, 18, 18 4 of 20
Sensors 2018, 18, 18 4 of 20

Figure 2. Flowchart of the study methods.

Figure 2. Flowchart of the study methods.

2.1. Study Area

2.1. Study Area
In this study, in order to compare the performance of different classification algorithms on
In thisdata
different study, in order
training to compare
sample the performance
strategies, an area of2of30different
× 30 km classification algorithms
2 of a peri-urban andon different
rural with
data training sample strategies, an area of 30 × 30 km of a peri-urban and rural with
heterogeneous land cover area in the north of the Red River Delta (RRD), Vietnam was chosen (Figure heterogeneous
land cover
3). This is aarea in theland
typical north of the Red
use/cover River
of the Delta
RRD (RRD),
area, Vietnam
slightly wasfrom
sloping chosen
the(Figure 3). This
southwest is
to the
anortheast.
typical land use/cover of the RRD area, slightly sloping from the southwest to the northeast.
The study area mainly includes six typical classes: resident (fragment and distribution over The
study area mainlyimpervious
the study area), includes sixsurface
typical classes:
(includingresident (fragment
factory, and distribution
block building over the study
and transportation, area),
roads),
impervious surface
agriculture, bare (including
land, factory,
forest, and water.block building and transportation, roads), agriculture, bare
land, forest, and water.
 Sensors 2018, 18, 18 5 of 20
Sensors 2018, 18, 18 5 of 20

Figure
Figure3.3.Location
Locationofofthe
thestudy
studyarea
areaininthe
thenorth
northof
ofthe
theRed
RedRiver
RiverDelta
Delta(RRD),
(RRD),Vietnam.
Vietnam.

2.2. Data Used

2.2. Data Used
Sentinel-2 A, level 1 C (ID = L1C_T48QXJ_A010168_20170603T034152) acquired on 3 June 2017,
Sentinel-2 A, level 1 C (ID = L1C_T48QXJ_A010168_20170603T034152) acquired on 3 June 2017,
was downloaded from the United States Geological Survey (USGS) website [38]. Spectral bands of
was downloaded from the United States Geological Survey (USGS) website [38]. Spectral bands of
the Sentinel-2 A satellite imagery are shown in Table 1. The imagery was atmospherically corrected
the Sentinel-2 A satellite imagery are shown in Table 1. The imagery was atmospherically corrected
using the Sen2cor tool, which is available in the Sentinel Application Platform (SNAP) toolbox [22,39].
using the Sen2cor tool, which is available in the Sentinel Application Platform (SNAP) toolbox [22,39].
After atmospheric correction, 10 bands (2–8, 8A, 11 and 12) were composited with 20 m resolution
After atmospheric correction, 10 bands (2–8, 8A, 11 and 12) were composited with 20 m resolution
and clipped to the study area (30 × 30 km22). The composited-10 band-20m imagery was used for
and clipped to the study area (30 × 30 km ). The composited-10 band-20m imagery was used for
classification in this study.
classification in this study.
Table 1. Spectral bands of the Sentinel-2 A satellite imagery.
Table 1. Spectral bands of the Sentinel-2 A satellite imagery.
Spectral Band Center Wavelength (nm) Band Width (nm) Spatial Resolution (m)
Spectral
Band 1Band Center Wavelength
443 (nm) Band Width
20 (nm) Spatial Resolution
60 (m)
Band
Band 21 443490 2065 60 10
Band
Band 32 490560 6535 10 10
Band 3 560 35 10
Band
Band44 665665 3030 10 10
Band
Band55 705705 1515 20 20
Band
Band66 740740 1515 20 20
Band 7 7
Band 783783 2020 20 20
Band 8 842 115 10
Band 8 842 115 10
Band 8a 865 20 20
Band
Band8a9 945865 2020 60 20
Band
Band910 945
1380 3020 60 60
Band
Band 1011 1610
1380 9030 20 60
Band 12 2190 180 20
Band 11 1610 90 20
Band 12 2190 180 20
Sensors 2018, 18, 18 6 of 20

2.3. Training and Testing Sample Datasets

The training data (training and testing samples) was collected based on the manual interpretation
of the original Sentinel-2 data and high-resolution imagery available from Google Earth.
To collect training sample data, the create polygon tool in the ArcGIS 10.5 toolbox was used to
create 135 polygons for each land cover class. Due to the different polygon sizes, the number of pixels
for each land cover class also differed (Table 2).

Table 2. Training and testing sample sizes used in this study.

Land Cover Training (polygon/pixels) Testing (pixels)

Residential 135/1410 625
Impervious surface 135/1645 427
Agriculture 135/2619 614
Bare land 135/1274 605
Forest 135/1267 629
Water 135/1704 628

For an accurate assessment of the classification results, 650 points for every land cover class was
collected. However, to ensure that the training and testing datasets were independent, we buffered
15 m for all point samples (testing dataset) and removed points which had buffered-points intersecting
with (or belonging to) polygon samples. As results, we obtained the number of testing points (pixels)
as shown in Table 2.
To evaluate the effect of the training sample sizes, as well as the performance of classification
algorithms on the classification accuracies, we randomly divided training sample data into 14 different
datasets (Table 3), in which seven-imbalanced datasets (iset_1, iset_2, iset_3, iset_4, iset_5, iset_6,
and iset_7) had the corresponding sizes of 5%, 10%, 20%, 40%, 60%, 80%, and 100% of the total
training data. The create Data Partition function in the caret package [40] was used to guarantee
that all datasets had the same proportion-training sample of each land cover class. The remaining
seven-balanced datasets were created as follows: bset_1 (50 pixels/class), bset_2 (125 pixels/class),
bset_3 (250 pixels/class), bset_4 (500 pixels/class), bset_5 (750 pixels/class), bset_6 (1000 pixels/class),
and bset_7 (1250 pixels/class). The number of pixels in each class for every sub-dataset was chosen to
keep the most consistent size between the imbalanced and balanced training sample size; for example,
the lowest number of pixels among the 6 land cover classes was 1267 pixels, leading bset_7 to have
1250 pixels for the balanced data for each class.

Table 3. Samples data for training classification.

Imbalanced_data iset_1 iset_2 iset_3 iset_4 iset_5 iset_6 iset_7

No. pixels 5% 10% 20% 40% 60% 80% 100%
Balanced data bset_1 bset_2 bset_3 bset_4 bset_5 bset_6 bset_7
No. pixels 50 100 250 500 750 1000 1250

2.4. Classification Algorithms and Tuning Parameters

Tuned parameters play an important role in producing high accuracy results when using SVM, RF,
and kNN. Each classifier has different tuning steps and tuned parameters. For each classifier, we tested
a series of values for the tuning process with the optimal parameters determined based on the highest
overall classification accuracy. In this study, the classified results under the optimal parameters of each
classifier were used to compare the performance of classifiers [41].
Sensors 2018, 18, 18 7 of 20

2.4.1. Support Vector Machine (SVM)

In land cover classification studies, according to Knorn et al. [42] and Shi and Yang. [43], the radial
basis function (RBF) kernel of the SVM classifier is commonly used and shows a good performance.
Therefore, we used the RBF kernel to implement the SVM algorithm. There are two parameters that
need to be set when applying the SVM classifier with RBF kernel: the optimum parameters of cost
(C) and the kernel width parameter (γ) [41,44]. The C parameter decides the size of misclassification
allowed for non-separable training data, which makes the adjustment of the rigidity of training data
possible [45]. The kernel width parameter (γ) affects the smoothing of the shape of the class-dividing
hyperplane [46]. Larger values of C may lead to an over-fitting model [35], whereas increasing the
γ value will affect the shape of the class-dividing hyperplane, which may affect the classification
accuracy results [35,47]. Following the study of Li et al. [28] and pretested to our dataset, in this study,
to find the optimal parameters for SVM, ten values of C (2−2 , 2−1 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 ), and ten
values of γ (2−5 , 2−4 , 2−3 , 2−2 , 2−1 , 20 , 21 , 22 , 23 , 24 ) were tested. This procedure was applied to all
14 sub-datasets.

2.4.2. Random Forest (RF)

In order to implement the RF [33], two parameters need to be set up: the number of trees (ntree)
and the number of features in each split (mtry). Several studies have stated that the satisfactory results
could be achieved with the default parameters [12,48–50]. However, according to Liaw & Wiener [48],
the large number of trees will provide a stable result of variable importance. In addition, Breiman [33]
stated that using more than the required number of trees may be unnecessary, but this does not harm
the model. In addition, Feng et al. [51] stated that with ntree = 200, RF could achieve accurate results.
√
Regarding the mtry parameter, there are many studies that use the default value mtry = p, where
p is the number of predictor variables [12]. However, in this study, to find the optimal RF model for
classification, a range of values for both parameters were tested and evaluated: ntree = 100, 200, 500,
and 1000; mtry = 1:10 with a step size of 1.

2.4.3. k-Nearest Neighbor (kNN)

The kNN approach is a non-parametric [52] that has been used in the early 1970’s in statistical
applications [53]. The basic theory behind kNN is that in the calibration dataset, it finds a group of k
samples that are nearest to unknown samples (e.g., based on distance functions). From these k samples,
the label (class) of unknown samples are determined by calculating the average of the response
variables (i.e., the class attributes of the k nearest neighbor) [54,55]. As a result, for this classifier, the k
plays an important role in the performance of the kNN, i.e., it is the key tuning parameter of kNN [41].
The parameter k was determined using a bootstrap procedure. In this study, we examined k values
from 1 to 20 to identify the optimal k value for all training sample sets.

2.5. Accuracy Assessment and Comparisons

In order to assess the accuracy of classification performance, there are many metrics available in
the literature. The two most popular metrics are overall accuracy (OA) and Kappa statistic. Recently
however, the Kappa statistic is becoming less common in remote sensing classification accuracy
assessment [37]. One of the drawbacks of solely using the OA metric is that it does not show the
specific performance of classes. He and Garcia [56] stated that if input datasets (training samples) are
highly imbalanced, the OA value might be deceiving, because the rare classes may be classified very
poorly. He and Garcia [56] also suggested that when choosing OA as the criterion metric, the best class
distribution is followed by the naturally occurring.
In this study, we used a stratified sampling approach; moreover, we divided training data into
several sub-datasets, including imbalanced and balanced datasets. This approach matches the OA
metric. In addition, to compare the performance of each classifier, we used the same training (input)
Sensors 2018, 18, 18 8 of 20

and testing (validation) datasets; thus, the effect of individual class’s distribution on OA does not
bias the results. We also calculated the 95% confidence interval (error tolerance) δ of the probability
estimate [57] for every OAs. Because we used the same testing datasets for all classification accuracy
assessment, thus the δ were not significant different. (The detail of OAs and δ are shown in Appendix B).
Therefore, to 2018,
Sensors assess and compare the performance of the classifiers and the different datasets,8 we
18, 18 of 20used
overall accuracy (OA) as the criterion. In total, we have seven imbalanced datasets, seven balanced
assessment, thus the δ were not significant different. (The detail of OAs and δ are shown in Appendix
datasets, and three classifiers. Consequently, each classifier had 14 classification results, totaling 42
B). Therefore, to assess and compare the performance of the classifiers and the different datasets, we
overall classification results.
used overall accuracy (OA) as the criterion. In total, we have seven imbalanced datasets, seven
balanced datasets, and three classifiers. Consequently, each classifier had 14 classification results,
3. Results
totaling 42 overall classification results.
3.1. The Effects of Tuning Parameters on Classification Accuracies
3. Results
Due to the limitation of space and the consistency of the results, only the results of eight
3.1. The Effects of Tuning Parameters on Classification Accuracies
sub-datasets—4 balanced (bset_1, bset_3, bset_5, bset_7) and 4 imbalanced (iset_1, iset_3, iset_5,
iset_7)—areDue to the
shown. Thelimitation
results of
of space and the consistency
the remaining of the
sub-datasets results,
were still only theand
tested results of eight
showed sub-
a consistent
datasets—4
trend in results. balanced (bset_1, bset_3, bset_5, bset_7) and 4 imbalanced (iset_1, iset_3, iset_5, iset_7)—
are shown. The results of the remaining sub-datasets were still tested and showed a consistent trend
in results.
3.1.1. The kNN Classifier
With the
3.1.1. ThekNN
kNNclassifier,
Classifier to classify one object, the algorithm bases the class attributes of its k
nearest neighbors
With the kNNTherefore,
[41]. classifier, tothe k value
classify one plays
object, an
theimportant role in
algorithm bases thethe performance
class of kkNN,
attributes of its
and is the key tuning
nearest parameter
neighbors of kNN
[41]. Therefore, thealgorithm. In this
k value plays study, werole
an important tested a range
in the of k values
performance (1 to 20)
of kNN,
for choosing
and is the optimal
the key tuningparameter
parameter of of the
kNNkNN classifier
algorithm. using
In this different
study, sub-datasets.
we tested a range of k values (1 to
20) for4 choosing
Figure shows the the results
optimal ofparameter
the kNN of the kNN classifier
classifier using different
error when appliedsub-datasets.
to different sub-datasets.
The lowest error was achieved with k = 1 for all datasets. As shown in Figure different
Figure 4 shows the results of the kNN classifier error when applied to 4, when sub-datasets.
k increases from
The lowest error was achieved with k = 1 for all datasets. As shown in Figure 4, when k increases
1 to 20, the error of the kNN classifier subsequently increases. This finding is consistent with the study
from 1 to 20, the error of the kNN classifier subsequently increases. This finding is consistent with
by Qian et al. [46]. Therefore, the optimal k for the kNN classifier was chosen as k = 1.
the study by Qian et al. [46]. Therefore, the optimal k for the kNN classifier was chosen as k = 1.

Figure 4. Cont.
Sensors 2018,18,
Sensors2018, 18,18
18 99 of
of 20
20
Sensors 2018, 18, 18 9 of 20

Figure 4. The relationship between classification error (y-axis) and k value (x-axis) parameters (1 to
Figure 4. The
4.
Figure20) The relationship
relationship
of the
between
between
kNN classifier
classification error
classification
obtained
(y-axis)
error
from the bootstrap
and
(y-axis) k value
and
resampling k (x-axis)
value
approach
parameters
(x-axis) parameters
using different
(1 to 20)
sub- (1 to
of the kNN classifier
20) of datasets
the kNN obtained
of training sample
classifier from the bootstrap resampling approach using different sub-datasets
data. from the bootstrap resampling approach using different sub-
obtained of
training
datasetssample data.sample data.
of training
3.1.2. The RF Classifier
3.1.2.
3.1.2. The
The RF
AsClassifier
RF Classifier
stated in section 2.4.2., there are two parameters that significantly affect the performance of
the RF classifier: ntree and mtry. In this study, we used Sentinel-2 with ten bands for classification,
As stated
stated in
Asmeaning in Section
section 2.4.2,
2.4.2., there
there are
aretwotwo parameters that significantly affect the
the performance of
the input data has 10 variables. To parameters
find the optimal thatparameters
significantly affect
for the performance
RF classifier, several of
the
the RF classifier:
RFvalues
classifier: ntree
ntree and
and mtry.
mtry. In
In this
this study,
study, we
we used
used Sentinel-2
Sentinel-2 with
with ten
ten bands
bands for
for classification,
classification,
(mtry = 1:10; ntree = 100, 200, 500, and 1000) were tested for all 14 sub-datasets. The highest
meaning
meaning the
the input
results input
for data
data has
has 10
all sub-datasets variables.
10were
variables.
obtained To find
Towith the
the optimal
findmtry optimal
equal to 2, parameters
parameters
3, or 4 (Figure for the
the RF
for5). RF classifier,
classifier, several
several
values (mtry
values (mtry = 1:10;
As =shown
1:10; inntree
ntree =
Figure 100,
= 100, 200, 500,
200,the
5, when 500, and
and
mtry 1000)
was1000) were
were
2, 3, or tested
4, thetested
results for
for all
of all 14
ntree sub-datasets.
14200,
sub-datasets.
500, and 1000The
The highest
highest
were
results for
for all
resultssimilar.all sub-datasets were
In addition, Figure
sub-datasets wereobtained
6 shows that
obtained with
with mtry
mtryequal
out-of-bag to
(OOB)
equal to2,2,error
3,3,or 44(Figure
ordecreased 5).
(Figuresharply
5). when ntree
As increased from
in 1 to
Figure 100.
5, When
when ntree
the increased
mtry was from
2, 3,101
or to
4, 400,
As shown in Figure 5, when the mtry was 2, 3, or 4, the results of ntree 200, 500, 500,
shown the different
results sub-datasets
of ntree 200,had
and slightly
and 1000
1000 were
were different trends, however, generally, thethat
OOBs were slightly reduced at all sub-datasets. All OOBs of
similar. In addition, Figure 6 shows that out-of-bag (OOB) error decreased sharply when ntree
similar. In addition, Figure 6 shows out-of-bag (OOB) error decreased sharply when ntree
all sub-datasets were almost remain stable when ntree increase from 400 to 500. Therefore, ntree =
increased
increased from
from 11 to to 100.
100. When
When ntree
ntree increased
increased from
from 101 101 toto 400,
400, different
different sub-datasets
sub-datasets had had slightly
slightly
500 was used as the optimal value for RF classifiers. The mtry value was chosen based on the highest
different
different trends,
trends, however,
however, generally, the
the OOBs were slightly reduced at all sub-datasets. All OOBs of
OOBs were slightly reduced at all sub-datasets. All OOBs
results of mtry = 2, 3, orgenerally,
4. of
all
all sub-datasets
sub-datasetswere werealmost
almostremain
remain stable
stable when
when ntree increase
ntree increase from 400400
from to 500. Therefore,
to 500. ntreentree
Therefore, = 500=
was usedused
500 was as the optimal
as the optimalvalue for for
value RF RF classifiers. The
classifiers. The mtry
mtry value
valuewas waschosen
chosenbased
basedon onthe
thehighest
highest
results of mtry = 2, 3,
results of mtry = 2, 3, or 4. or 4.

Figure 5. Cont.
Sensors 2018, 18, 18 10 of 20
Sensors 2018, 18, 18 10 of 20
Sensors 2018, 18, 18 10 of 20

Figure 5.
Figure 5. Effect
Effect of
of the
the number
number of
of trees
trees and
and the
the number
number of of random
random split
split variables
variables at
at each
each node
node (mtry)
(mtry)
Figure
on the 5. Effectaccuracy
overall of the number
for RFof trees and the
classification number
using all of random
training splitdata.
sample variables at each node (mtry)
on the overall accuracy for RF classification using all training sample data.
on the overall accuracy for RF classification using all training sample data.

Figure 6. Cont.
Sensors 2018, 18, 18 11 of 20
Sensors 2018, 18, 18 11 of 20

Figure 6. The relationship between OOB error (y-axis) and ntree parameter (x-axis) of the RF classifier
Figure
Figure 6.
6. The
Therelationship
relationship between
between OOB
OOB error
error (y-axis)
(y-axis) and
and ntree
ntree parameter
parameter (x-axis)
(x-axis) of
of the
the RF
RF classifier
classifier
using different sub-datasets of training sample data.
using
using different
different sub-datasets of training sample data.

3.1.3. The SVM Classifier

3.1.3.
3.1.3. The
The SVM SVM Classifier
Classifier
In order to find the optimal values for the SVM model, several values were examined for C and
In
In order to find the the optimal
optimal values valuesfor forthe theSVM SVMmodel, model,several severalvalues valueswere wereexamined
examined forfor
CCandand
γ:
γ: C−(22−2−2, −2−1
−1, 20, 21, 22, 23, 24, 25, 26, 27), γ (2−5, 2−4, 2−3, 2−2, 2−1, 20, 21, 22, 23, 24).
γ: C (2 , 2, 2 1 , 200, 211,, 2222, ,2233, ,224,42, 52,52, 62, 62,72),7γ
C (2 ), (2 (2, −25−4,, 22−−34,,22−2−,32, −12,−220,, 22−1,12, 22, 02,32, 12,4).
γ −5 22 , 23 , 24 ).
Figure 7 shows the relationship between the OOB error and the SVM parameters. Based on these
Figure
Figure 7 shows shows the the relationship
relationship between the OOB error and the SVM parameters. Based Based onon these
these
results, the optimal parameters of the 14 sub-datasets were chosen according to the lowest OOB error
results,
results, the optimal
optimal parameters
parameters of the the 1414 sub-datasets
sub-datasets were were chosenchosen according according to to the
the lowest
lowest OOB
OOB error
error
for each result. In general, a high value of C and a low value of γ produced the lowest error. The
for
for each result. In general, a high value of C and a low
result. In general, a high value of C and a low value of γ produced the lowest error. The higher value of γ produced the lowest error. The
higher error was observed with a low C value with both high (greater than eight) and low (less than
higher
error was error was observed
observed with a with low Ca value low C with valueboth withhigh both(greaterhigh (greater than eight) than eight)
and low and lowthan
(less (less0.125)
than
0.125) value of γ (Figure 7). The optimal parameters (C and γ) of different sub-datasets varied (Table
0.125)
value ofvalue of γ (Figure
γ (Figure 7). The7). The optimal
optimal parametersparameters (C and(C γ)and γ) of different
of different sub-datasetssub-datasets
variedvaried
(Table (Table
4).
4).
4). The optimal models (using optimal parameters) were applied to the whole image for
The optimal models (using optimal parameters) were applied to the whole image for
The optimal
classification results. models (using optimal parameters) were applied to the whole image for
classification results.
classification results.

Figure 7. Cont.
Sensors 2018, 18, 18 12 of 20
Sensors 2018, 18, 18 12 of 20

Figure
Figure 7.
7. The
Therelationship
relationshipbetween
between classification
classification error
error and
and parameters
parameters (C
(C and
and γ)
γ) of
ofthe
theSVM
SVMclassifier
classifier
obtained
obtained from
from different
different sub-datasets of training sample data.

Table
Table 4.
4. The
The optimal
optimal parameters
parameters (C
(C and γ)
γ) of
of different sub-datasets varied.

Imbalanced Dataset
Imbalanced Dataset γ
γ C
C Balanced Dataset
Balanced Dataset
γγ C C
iset_1 0.03125 64 bset_1 0.03125 64
iset_1 0.03125 64 bset_1 0.03125 64
iset_2 iset_2 0.5 0.5 32 32 bset_2
bset_2 0.125
0.125 32 32
iset_3 iset_3 0.03125 0.03125128128 bset_3
bset_3 0.125
0.125 64 64
iset_4 0.25 32 bset_4 0.125 64
iset_4 0.25 32 bset_4 0.125 64
iset_5 0.125 64 bset_5 0.125 128
iset_6 iset_5 0.5 0.125 32 64 bset_5
bset_6 0.125
0.25 128 64
iset_7 iset_6 0.25 0.5 64 32 bset_7
bset_6 0.25
0.25 64 128
iset_7 0.25 64 bset_7 0.25 128
3.2. The Performance of Different Classifiers on Imbalanced Datasets
3.2. The
As Performance of Different
shown in Figure 8, withClassifiers
all seven on Imbalanced(iset_1
sub-datasets Datasets
to iset_7), SVM always showed the most
accurate results,in
As shown followed
Figure by RF and
8, with all kNN.
sevenHowever,
sub-datasets the three
(iset_1 highest accuracies
to iset_7), SVM of all classifiers
always showedwere the
only slightly different. The accuracy results of SVM were not
most accurate results, followed by RF and kNN. However, the three highest accuracies significantly different among different
of all
training sample
classifiers sizes;slightly
were only from iset_1 to iset_7,
different. the lowestresults
The accuracy was iset_1of SVM datawere at 93.76% and the highest
not significantly was
different
iset_5 data at 95.32%. In contrast, the classification accuracy of kNN and
among different training sample sizes; from iset_1 to iset_7, the lowest was iset_1 data at 93.76% and RF were significantly different
between
the highestsmall
was sample sizesat(iset_1)
iset_5 data 95.32%.and large sample
In contrast, sizes (iset_7).
the classification With small
accuracy of kNN andandimbalanced
RF were
training samples
significantly (iset_1,
different iset_2, small
between and iset_3),
sample there is (iset_1)
sizes a difference between
and large sample classification accuracies.
sizes (iset_7). SVM
With small
produced a significantly higher accuracy than that of RF and kNN.
and imbalanced training samples (iset_1, iset_2, and iset_3), there is a difference between This is consistent with the results
reported fromaccuracies.
classification Shi and Yang SVM[43]. From aasmall
produced sample higher
significantly size (iset_1)
accuracy to a than
largerthatsample
of RFsize
and(iset_7),
kNN.
the accuracy significantly increased with RF and kNN, whereas the results
This is consistent with the results reported from Shi and Yang [43]. From a small sample size (iset_1) of SVM were only slightly
increased.
to a larger It is indicated
sample that thethe
size (iset_7), sample size and
accuracy imbalanced
significantly data of training
increased with RFsamples
and kNN, has whereas
more impact the
on the classification
results of SVM were accuracy
only slightlyfor kNN and RF
increased. It than for SVM.
is indicated that the sample size and imbalanced data
The highest
of training samples accuracy
has more forimpact
the three on classifiers occurred
the classification when the
accuracy for training
kNN andsample RF than size
forwasSVM. large
enough, i.e., iset_5;
The highest kNN, for
accuracy RF,the and SVM
three were 94.59%,
classifiers occurred 94.70%,whenand the95.32%,
trainingrespectively
sample size (Figure
was large 8).
However,
enough, when
i.e., thekNN,
iset_5; trainingRF,sample
and SVM sizeswere
increased
94.59%, further
94.70%, (iset_6
and and
95.32%,iset_7), the overall
respectively accuracy
(Figure 8).
of the classifiers
However, when the slightly
trainingdecreased;
sample sizes kNN,increased
RF, and further
SVM for iset_6and
(iset_6 (iset_7)
iset_7), were
the 93.85% (94.13%),
overall accuracy
94.32%
of (94.44%), slightly
the classifiers and 95.12% (95.07%),
decreased; respectively.
kNN, RF, and SVM It is suggested
for iset_6 that if the
(iset_7) training
were 93.85% sample data
(94.13%),
is imbalanced between classes, the training sample sizes should be large
94.32% (94.44%), and 95.12% (95.07%), respectively. It is suggested that if the training sample data enough to achieve the best
is
performance of classifiers. If the training sample size is too large,
imbalanced between classes, the training sample sizes should be large enough to achieve the best it could change the proportion
between classes,
performance which lead
of classifiers. If to
thethe decrease
training in overall
sample size isaccuracy.
too large,For all three
it could classes
change theinproportion
this study,
between classes, which lead to the decrease in overall accuracy. For all three classes in this study, the
Sensors 2018, 18, 18 13 of 20

Sensors 2018, 18, 18 13 of 20

the highest accuracies were achieved when the training sample size represented approximately 0.26%
highest accuracies
of the total wereThis
study area. achieved when the
is consistent training
with sample
research size represented
of Colditz approximately
[58]; in which they stated0.26% of
that for
the total study area. This is consistent with research of Colditz [58]; in which they stated that
the RF classifier, the training sample size should account for 0.25% of the total study area. Our results for the
RF
showclassifier,
that thisthe training
case sample
is not only size
valid forshould
the RF account
classifierfor
but0.25% of the
also for total study
the SVM and kNNarea. Our results
classifiers.
show that this case is not only valid for the RF classifier but also for the SVM and kNN classifiers.

Figure 8. The
The performance
performance of
of the
the kNN, SVM, and RF classifiers on different imbalanced training
sample sizes.

3.3.
3.3. The
The Performance
Performance of
of Different Classifiers on
Different Classifiers on Balanced
Balanced Datasets
Datasets
For balanced datasets
For balanced datasets (bset_1
(bset_1totobset_7),
bset_7),thetheSVMSVMclassifier
classifierstill
stillproduced
produced thethe highest
highest accuracy
accuracy at
at 95.29%, followed by RF at 94.59% and kNN at 94.10%. However, the
95.29%, followed by RF at 94.59% and kNN at 94.10%. However, the performance of each classifier on performance of each classifier
on different
different training
training sample
sample sizessizes
was was
onlyonly slightly
slightly different
different (Figure (Figure
9). For9).the
ForkNN
the kNN classifier
classifier with awith
small a
small training sample size (bset_1 to bset_4), the training sample size
training sample size (bset_1 to bset_4), the training sample size had a strong impact on the accuracy had a strong impact on the
accuracy of classification.
of classification. The overallThe trend
overallshowed
trend showed
that the that the larger
larger the training
the training sample sample
size, size, the higher
the higher the
the accuracy.
accuracy. Increasing
Increasing the sample
the sample size from size50from 50 pixels/class
pixels/class (bset_1)(bset_1) to 500 pixels/class
to 500 pixels/class (bset_4)
(bset_4) resulted
resulted in an accuracy
in an accuracy increaseincrease from 89.85%
from 89.85% to 93.45%.to 93.45%.
However, However,
when the when the training
training sample sample sizehigh
size was was
high
enough enough
(more(more
than 750 than 750 pixels/class),
pixels/class), as it is as
foritbset_5,
is for bset_6,
bset_5, andbset_6,
bset_7,andthebset_7, the classification
classification accuracy
accuracy
was stablewas stable at94.10%,
at 93.96%, 93.96%,and 94.10%,
94.02%, andrespectively.
94.02%, respectively.
With
With thethe RFRF classifier,
classifier, the
the larger
larger training
training sample
sample alsoalso produced
produced higherhigher accuracy
accuracy with with the
the first
first 44
sub-datasets (bset_1 to
sub-datasets (bset_1 to bset_4).
bset_4). However,
However, the the difference
difference at at bset_1
bset_1 andand bset_4
bset_4 waswas not
not asas large
large as
as with
with
the kNN classifier. With the smallest training sample (bset_1), RF produced
the kNN classifier. With the smallest training sample (bset_1), RF produced a higher accuracy (91.47%) a higher accuracy
(91.47%)
than thatthanof kNN that(89.95%);
of kNN (89.95%);
however,however,
with bset_4 with bset_4
(750 (750 pixels/class),
pixels/class), the accuracy
the accuracy results of results
RF and of
RF
kNN and kNN
were were at
similar similar
93.61% at and
93.61% and 93.45%,
93.45%, respectively.
respectively. The stable Theaccuracy
stable accuracy
results of results of bset_5,
bset_5, bset_6,
bset_6, and bset_7 (93.47%, 94.42%, 94.59%) were also observed
and bset_7 (93.47%, 94.42%, 94.59%) were also observed with the RF classifier. with the RF classifier.
The
The SVMSVM classifier
classifiershowed
showeddifferent
differentresults.
results. When
When thethe training
training sample
sample sizesize
waswassmall small (bset_1
(bset_1 and
and bset_2),
bset_2), the classification
the classification accuracyaccuracy
was high was
andhigh anddifferent
slightly slightly atdifferent
92.63% and at 92.63%
92.35%,and 92.35%,
respectively.
respectively.
However, when However, whensample
the training the training sample size
size increased fromincreased from 125(bset_2)
125 pixels/class pixels/class
to 250(bset_2) to 250
pixels/class
pixels/class (bset_3) and 500 pixels/class (bset_4), the training sample size had a strong impact on
classification accuracy. This is an interesting finding; because the results showed a contrast with
Sensors 2018, 18, 18 14 of 20

(bset_3) and
Sensors 2018, 18,500
18 pixels/class (bset_4), the training sample size had a strong impact on classification 14 of 20
accuracy. This is an interesting finding; because the results showed a contrast with previous studies
previous
which studies
stated which
that the stated
training that the
sample training
size was lesssample size
sensitive wasSVM.
with less sensitive
This mightwith SVM.
be true This
if the might
training
be true if
sample theistraining
size sample(less
small enough sizethan
is small enough (less or
125 pixels/class) than 125enough
large pixels/class) or than
(greater large750enough (greater
pixels/class).
than 750
Figure pixels/class).
9 shows Figure
that when the 9balanced
shows that whensample
training the balanced training from
size increasing sample
0.20%size(bset_5)
increasing from
to 0.33%
0.20% (bset_5)
(bset_7) of theto 0.33%
total (bset_7)
study area,ofthe
theperformance
total study area, theclassifiers
of the performance of the
were classifiers
similar were
between similar
different
between sample
training different training
sizes. When sample sizes. When
comparing comparing
the three theSVM
classifiers, threeproduced
classifiers,the
SVM produced
highest the
accuracy,
highest accuracy,
followed by RF and kNN.followed by RF and kNN.

9. The
Figure 9. Theperformance
performanceofofkNN,
kNN, SVM,
SVM, and
and RF RF classifiers
classifiers on different
on different balanced
balanced training
training sample
sample sizes.
sizes.
4. Discussion
4. Discussion
Figure 10 shows the difference between OA of imbalanced and balanced sub-datasets from the 42
Figure
results of the1014shows thetraining
different difference between
sample sizesOA for of
eachimbalanced
classifier. and
Two balanced sub-datasets
different trends are clear:from the
when
42 resultssample
training of the sizes
14 different
were largetraining sample
(greater thansizes for each
or equal classifier.
to 500 Two different
pixels/class) trends are
the performance clear:
of kNN,
when training sample sizes were large (greater than or equal to 500 pixels/class)
RF, and SVM on balanced and imbalanced datasets was not significantly different, except for kNN the performance of
kNN, RF, and SVM on balanced and
with dataset_5 (approximately 750 pixels/class).imbalanced datasets was not significantly different, except for
kNNDue withtodataset_5
the actual(approximately
proportion of750 landpixels/class).
cover type on the landscape, the rare classes have a low
number of pixels in the training sample cover
Due to the actual proportion of land (Tabletype on the landscape,
2); therefore, the rare classes
in all sub-datasets (set_1 have a low
to set_7),
number of pixels in the training sample (Table 2); therefore, in all sub-datasets
the number of pixels in balanced datasets is always smaller than those of imbalanced sub-datasets. (set_1 to set_7), the
number ofaspixels
However, in balanced
mentioned earlier,datasets
when the is training
always sample
smaller size
thanisthose
large of imbalanced
enough, sub-datasets.
the performance of
However, as mentioned earlier, when the training sample size is large enough,
classifiers on balanced and imbalanced sub-datasets was similar. In other words, the classifiers are lessthe performance of
classifierstoon
sensitive balanced
the and training
imbalanced imbalanced datasub-datasets
if the training was similar.
sample sizeInis other
large words,
enoughthe(i.e.,classifiers
greater thanare
less sensitive to
750 pixels/class). the imbalanced training data if the training sample size is large enough (i.e., greater
than 750 pixels/class).
Sensors 2018, 18, 18 15 of 20
Sensors 2018, 18, 18 15 of 20

Figure
Figure 10. 10.
TheThe differenceininOA
difference OA using
using balanced
balancedand
andimbalanced
imbalanceddatasets.
datasets.

Figure 10 shows that the performance of kNN at all sub-datasets for dataset_1, dataset_6, and
Figure
dataset_7 10wasshows that
similar forthe performance
balanced of kNNdatasets,
and imbalanced at all sub-datasets
whereas the resultsfor dataset_1,
of kNN on dataset_6,
the
and dataset_7 was similar
remaining datasets for balanced
were significantly and imbalanced
different. It is indicated datasets,
that the kNNwhereas the results
classifier of kNN on
is less sensitive
the remaining datasets
with imbalanced were
training significantly
sample different.
data, although it variesIt with
is indicated that the
training sample sizes.kNNWithclassifier
the RF and is less
sensitive
SVMwith imbalanced
classifiers, trainingsample
when training samplesizedata, although
is large enough it varies withdataset_6,
(dataset_5, training sample sizes. With
and dataset_7) the the
RF andresults
SVMofclassifiers,
imbalancedwhen and balanced
training were
samplesimilar
size and veryenough
is large high (greater than 94.32%).
(dataset_5, dataset_6, and dataset_7)
Many studies investigated the performance of the RF
the results of imbalanced and balanced were similar and very high (greater than classifier on different training sample sizes
94.32%).
Many studies investigated the performance of the RF classifier on different training [58]
and strategies of different satellite images, but the conclusions are contradictory. Colditz sampleand sizes
Mellor et al. [59] found a trend for the area proportional allocation of training samples, in which the
and strategies of different satellite images, but the conclusions are contradictory. Colditz [58] and
greater the land cover class area is the more training samples are needed to produce the best
Mellor et al. [59] found a trend for the area proportional allocation of training samples, in which
classification accuracy. In contrast, Dalponte et al. [60] and Jin et al. [61] stated that the RF classifier
the greater the landbetter
would perform cover class
with area is
balanced the more
training sample training
data. Our samples are needed
results (Figure 9) show to that
produce the best
if training
classification accuracy. In contrast, Dalponte et al. [60] and Jin et al. [61]
sample size is small (less than 500 pixels/classes) the difference in accuracy of imbalanced and stated that the RF classifier
would perform
balanced better
data would with balanced
be large training
(dataset-1, sampleordata.
dataset-3), smallOur results However,
(dataset-2). (Figure 9)whenshowthe that if training
training
sample size is
sample small
size (lessenough
is large than 500 pixels/classes)
(dataset-5, dataset-6, theanddifference
dataset-7) inthe
accuracy of imbalanced
performance and balanced
of RF on balanced
and imbalanced
data would training samples
be large (dataset-1, were or
dataset-3), similar.
smallIt(dataset-2).
must be mentioned
However,that when forthe
those different
training sample
conclusions,
size is large enough the RF classifier was
(dataset-5, used with
dataset-6, anddifferent
dataset-7) satellite imagery. Dalponte
the performance of RFeton al. balanced
[60] used and
HySpex-VNIR
imbalanced training 1600 and HySpex-SWIR
samples were similar.320i data, Jin
It must et al. [61] used
be mentioned thatLandsat
for thoseimagery, Colditz
different [58]
conclusions,
the RF classifier was used with different satellite imagery. Dalponte et al. [60] used HySpex-VNIR 1600
and HySpex-SWIR 320i data, Jin et al. [61] used Landsat imagery, Colditz [58] used MODIS imagery,
and Mellor et al. [59] used Landsat TM composited with topographic and climate variables. Therefore,
Sensors 2018, 18, 18 16 of 20

used MODIS imagery, and Mellor et al. [59] used Landsat TM composited with topographic and
climate variables. Therefore, it is suggested that the performance of the RF classifier on different
Sensors 2018, 18, 18 16 of 20
satellite imagery data with different training sample strategies (balanced versus imbalanced) is
different.
With the that
it is suggested SVMtheclassifier, as shown
performance of theinRF
Figure 10 among
classifier the 42satellite
on different classification
imageryresults, the 8different
data with highest
accuracies
training belonged
sample to the
strategies SVM versus
(balanced classifier (SVM_set4,
imbalanced) SVM_set5, SVM_set6, and SVM_set7).
is different.
Particularly,
With thethe SVMclassifier,
SVM classifier as
had the superior
shown performance
in Figure 10 amongcapability with small training
the 42 classification results,sample
the 8
sizes (set_1
highest and set_2
accuracies with to
belonged approximately 50 pixels
the SVM classifier per class).
(SVM_set4, SVM produced
SVM_set5, SVM_set6, overall accuracies
and SVM_set7).
ranging fromthe
Particularly, 93.76%
SVMto 93.96% (92.35%
classifier had the to 92.63%)performance
superior for imbalanced (balanced)
capability withsub-datasets,
small trainingcompared
sample
to 89.80%
sizes (set_1toand
92.06% (89.85%
set_2 to 90.50%) and 90.71%
with approximately 50 pixelsto per
92.34% (91.47%
class). SVMtoproduced
92.58%) for the kNN
overall and RF
accuracies
classifiers,
ranging fromrespectively. This is (92.35%
93.76% to 93.96% consistentto with thefor
92.63%) study of Shao and
imbalanced Lunettasub-datasets,
(balanced) [62]. compared
to 89.80% to 92.06% (89.85% to 90.50%) and 90.71% to 92.34% (91.47% to 92.58%) for the kNN and RF
5. Conclusions
classifiers, respectively. This is consistent with the study of Shao and Lunetta [62].
Classification of Sentinel-2 imagery (ten bands, 20 m) using three different machine learning
5. Conclusions
algorithms were implemented, evaluated, and compared. Fourteen different sub-datasets, including
Classification
balanced of Sentinel-2
and imbalanced, imagerytraining
with different (ten bands,
sample20 m) using
sizes fromthree
50 to different
more than machine learning
1250 pixels/class
algorithms were implemented,
were performed. evaluated,
All classification and compared.
results (OA) were high,Fourteen different
approximately sub-datasets,
ranging from 90% including
to 95%,
balanced and imbalanced, with different training sample sizes from 50 to more than
with the SVM classifier on average producing the highest OA with the least sensitivity to training 1250 pixels/class
were
sampleperformed. All classification
size, followed by the RF and results
kNN (OA) were The
classifiers. high, approximately
difference ranging from
in OA between kNN90%and to RF95%,
was
with
largethe SVMthe
when classifier on sample
training average size
producing the highest
increased OA with the least
from sub-dataset-1 sensitivity to for
to sub-dataset-4 training
both
sample size, and
imbalanced followed by the
balanced RF and
cases; kNN classifiers.
however, the differenceThebetween
difference in OAtraining
various betweensample
kNN and sizesRFofwas
the
large
SVM when the was
classifier training sample size
insignificant. Forincreased
all threefrom sub-dataset-1
classifiers, when the to training
sub-dataset-4
sampleforwas
bothlarge
imbalanced
enough
and balanced
(greater than cases; however, thewith
750 pixels/class), difference
both between
imbalanced various
and training
balancedsample sizes(iset_5/bset_5,
datasets of the SVM
classifier was insignificant.
iset_6/bset_6, For all three
and iset_7/bset_7), the classifiers,
OA was when the trainingsimilar
approximately sample and was large
high enough (greater
(over 93.85%).
than 750 pixels/class),
Furthermore, with both that
it is recommended imbalanced and balanced
in land cover datasets
classification using (iset_5/bset_5,
remote sensing iset_6/bset_6,
images and
and iset_7/bset_7),
machine the OA was
learning algorithms, theapproximately
training sample similar and high
size should (over 93.85%).
represent Furthermore,
approximately 0.25% ofittheis
recommended
total study area.that in land cover classification using remote sensing images and machine learning
algorithms, the training sample size should represent approximately 0.25% of the total study area.
Acknowledgments: We would like to acknowledge the Vietnamese government for its financial support. We
Acknowledgments: We would
are most grateful to Stefan like(Institute
Erasmi to acknowledge the Vietnamese
of Geography, government
University for its
of Göttingen) forfinancial support.
his helpful We are
suggestions.
most
This grateful to Stefan
publication was Erasmi (Institute
supported of Geography,
financially University
by the Open of Göttingen)
Access for hisof
Grant Program helpful suggestions.
the German This
Research
publication was supported financially by the Open Access Grant Program of the German Research Foundation
Foundation
(DFG) (DFG)
and the Open and the Open
Access Access Fund
Publication Publication Fund of theofUniversity
of the University Göttingen.ofWeGöttingen.
also thankWethe
also thank the
anonymous
anonymous reviewers for their valuable comments, which greatly
reviewers for their valuable comments, which greatly improved our paper.improved our paper.

Author Contributions:
Author Contributions: Phan
Phan Thanh
Thanh Noi
Noi analyzed
analyzed the
the data,
data, performed
performed the
the experiments
experiments and
and computed
computed thethe
statistical
statistical analysis. Martin Kappas
analysis. Martin Kappas helped
helped to
to conceive, design the
conceive, design the statistical
statistical analysis
analysis and
and corrected
corrected the
the final
final
manuscript.
manuscript. All
All authors
authors developed
developed and
and discussed
discussed the
the manuscript
manuscript together
together and
and finally
finallywrote
wrotethe
thepaper.
paper.
Conflicts of Interest: The authors declare no conflict of interest.
Conflicts of Interest: The authors declare no conflict of interest.
Appendix A
Appendix A

Figure A1. Cont.

Sensors 2018, 18, 18 17 of 20
Sensors 2018, 18, 18 17 of 20

Figure A1. The classification results with the highest overall accuracy for each classifier.
Figure A1. The classification results with the highest overall accuracy for each classifier.

Appendix
AppendixBB

TableA1.
Table A1.Overall
OverallAccuracy
Accuracyand
and95%
95% confidence
confidence interval
interval (error
(error tolerance)
tolerance) of the
of the probability
probability estimate
estimate of
of 42
all allclassification
42 classification results.
results.
Classifier set1 set2 set3 set4 set5 set6 set7
Classifier set1 set2 set3 set4 set5 set6 set7
RF_iset 90.71 ± 1.0 92.34 ± 0.9 93.64 ± 0.8 93.97 ± 0.8 94.70 ± 0.7 94.32 ± 0.8 94.44 ± 0.8
RF_iset
RF_bset 90.71 ± 1.0
91.47 ± 0.9 92.34 ± 0.9
92.58 ± 0.9 93.64 ± 0.8
92.60 ± 0.9 93.97 ± 0.8
93.61 ± 0.8 94.70 ± ±0.7
94.59 0.7 94.32 ± ±0.8
94.42 0.8 94.44 ± ±0.8
94.37 0.8
RF_bset
SVM_iset 91.47 ± 0.9
93.96 ± 0.8 92.58 ± 0.9
93.76 ± 0.8 92.60 ± 0.9
94.33 ± 0.8 93.61 ± 0.8
94.78 ± 0.7 94.59 ± ±0.7
95.32 0.7 94.42 ± ±0.8
95.12 0.7 94.37 ± ±0.8
95.07 0.7
SVM_iset
SVM_bset 93.96 ± 0.8
92.63 ± 0.9 93.76 ± 0.8
92.35 ± 0.9 94.33 ± 0.8
93.71 ± 0.8 94.78 ± 0.7
94.95 ± 0.7 95.32 ± ±0.7
95.10 0.7 95.12 ± ±0.7
95.07 0.7 95.07 ± ±0.7
95.29 0.7
SVM_bset 92.63 ± 0.9 92.35 ± 0.9 93.71 ± 0.8 94.95 ± 0.7 95.10 ± 0.7 95.07 ± 0.7 95.29 ± 0.7
kNN_iset 89.80 ± 1.0 92.06 ± 0.9 92.60 ± 0.9 93.37 ± 0.8 94.59 ± 0.7 93.85 ± 0.8 94.13 ± 0.8
kNN_iset 89.80 ± 1.0 92.06 ± 0.9 92.60 ± 0.9 93.37 ± 0.8 94.59 ± 0.7 93.85 ± 0.8 94.13 ± 0.8
kNN_bset 89.85 ± 1.0 90.50 ± 1.0 91.81 ± 0.9 93.45 ± 0.8 93.96 ± 0.8 94.10 ± 0.8 94.02 ± 0.8
kNN_bset 89.85 ± 1.0 90.50 ± 1.0 91.81 ± 0.9 93.45 ± 0.8 93.96 ± 0.8 94.10 ± 0.8 94.02 ± 0.8

References
References
1. DeFries, R.S.; Foley, J.A.; Asner, G.P. Land-use choices: Balancing human needs and ecosystem function.
1. DeFries,
Front. Ecol.R.S.; Foley, J.A.;
Environ. 2004,Asner, G.P. Land-use choices: Balancing human needs and ecosystem function.
2, 249–257.
2. Front. Ecol. Environ. 2004, 2, 249–257.
Foley, J.A.; DeFries, R.; Asner, G.P.; Barford, [CrossRef]C.; Bonan, G.; Carpenter, S.R.; Chapin, F.S.; Coe, M.T.; Daily,
2. Foley, J.A.; DeFries, R.; Asner, G.P.;
G.C.; Gibbs, H.K.; et al. Global consequences Barford, C.; Bonan,
of landG.; Carpenter,
use. S.R.;309,
Science 2005, Chapin, F.S.; Coe, M.T.; Daily, G.C.;
570–574.
3. Gibbs,
Verburg, H.K.; et al.
P.H.; Global consequences
Neumann, of land use.inScience
K.; Nol, L. Challenges using 2005, 309,and
land use 570–574. [CrossRef]
land cover [PubMed]
data for global change
3. Verburg, P.H.; Neumann, K.; Nol,
studies. Glob. Chang. Biol. 2011, 17, 974–989.L. Challenges in using land use and land cover data for global change
4. studies. Glob. Chang. Biol. 2011, 17, 974–989. [CrossRef]
Hansen, T. A review of large area monitoring of land cover change using Landsat data. Remote Sens.
4. Hansen,
Environ.T.2012,
A review of large area monitoring of land cover change using Landsat data. Remote Sens. Environ.
122, 66–74.
5. 2012, 122, K.J.;
Wessels, 66–74. [CrossRef]
Reyers, B.; Jaarsveld, A.S.; Rutherford, M.C. Identification of potential conflict areas between
5. Wessels, K.J.; Reyers,
land transformation and B.; Jaarsveld, A.S.;conservation
biodiversity Rutherford, M.C. Identification
in north-eastern of potential
South Africa. conflict areas between
Agric. Ecosyst. Environ.
land transformation
2003, 95, 157–178. and biodiversity conservation in north-eastern South Africa. Agric. Ecosyst. Environ.
6. Fry, 95,
2003, 157–178.
J.; Xian, G.Z.;[CrossRef]
Jin, S.; Dewitz, J.; Homer, C.G.; Yang, L.; Barnes, C.A.; Herold, N.D.; Wickham, J.D.
6. Fry, J.; Xian, of
Completion G.Z.;
the Jin,
2006S.; Dewitz,land
National J.; Homer, C.G.; Yang,
cover database for theL.;conterminous
Barnes, C.A.;United
Herold, N.D.;Photogramm.
States. Wickham, Eng.
J.D.
Completion of the 2006
Remote Sens. 2011, 77, 858–864. national land cover database for the conterminous United States. Photogramm. Eng.
7. Remote
Burkhard, Sens.B.;
2011, 77, F.;
Kroll, 858–864.
Nedkov, S.; Müller, F. Mapping ecosystem service supply, demand and budgets.
7. Burkhard, B.; Kroll, F.;
Ecol. Indic. 2012, 21, 17–29. Nedkov, S.; Müller, F. Mapping ecosystem service supply, demand and budgets.
8. Ecol. Indic. 2012,
Gebhardt, 21, 17–29. [CrossRef]
S.; Wehrmann, T.; Ruiz, M.A.M.; Maeda, P.; Bishop, J.; Schramm, M.; Kopeinig, R.; Cartus, O.;
8. Gebhardt, S.; Wehrmann,
Kellndorfer, J.; Ressl, R.; et al.,T.; Ruiz, M.A.M.; Automatic
MAD-MEX: Maeda, P.; wall-to-wall
Bishop, J.; Schramm,
land cover M.; Kopeinig,for
monitoring R.;the
Cartus, O.;
Mexican
Kellndorfer, J.; Ressl, R.; et al. MAD-MEX: Automatic wall-to-wall
REDD-MRV program using all Landsat data. Remote Sens. 2014, 6, 3923–3943. land cover monitoring for the Mexican
9. REDD-MRV
Guidici, D.; program
Clark, M.L. using all Landsat data.
One-Dimensional Remote Sens.neural
convolutional 2014, 6, 3923–3943.
network [CrossRef]
land-cover classification of multi-
seasonal hyperspectral imagery in the San Francisco Bay Area, California. Remote Sens. 2017, 9, 629.
Sensors 2018, 18, 18 18 of 20

9. Guidici, D.; Clark, M.L. One-Dimensional convolutional neural network land-cover classification of
multi-seasonal hyperspectral imagery in the San Francisco Bay Area, California. Remote Sens. 2017, 9,
629. [CrossRef]
10. Topaloğlu, H.R.; Sertel, E.; Musaoğlu, N. Assessment of classification accuracies of SENTINEL-2 and
LANDSAT-8 data for land cover/use mapping. In International Archives of the Photogrammetry, Remote Sensing
and Spatial Information Sciences; ISPRS: Prague, Czech Republic, 2016; Volume XLI-B8, pp. 1055–1059.
11. Khatami, R.; Mountrakis, G.; Stehman, S.V. A meta-analysis of remote sensing research on supervised
pixel-based land cover image classification processes: General guidelines for practitioners and future
research. Remote Sens. Environ. 2016, 177, 89–100. [CrossRef]
12. Duro, D.C.; Franklin, S.E.; Dubé, M.G. A comparison of pixel-based and object-based image analysis with
selected machine learning algorithms for the classification of agricultural landscapes using SPOT-5 HRG
imagery. Remote Sens. Environ. 2012, 118, 259–272. [CrossRef]
13. Xia, J.S.; Mura, M.D.; Chanussot, J.; Du, P.; He, X. Random subspace ensembles for hyperspectral image
classification with extended morphological attribute profiles. IEEE Trans. Geosci. Remote Sens. 2015, 53,
4768–4786. [CrossRef]
14. Chen, Y.; Dou, P.; Yang, X. Improving land use/cover classification with a multiple classifier system using
AdaBoost integration technique. Remote Sens. 2017, 9, 1055. [CrossRef]
15. Gomez, C.; White, J.C.; Wulder, M.A. Optical remotely sensed time series data for land cover classification:
A review. Int. Soc. Photogramm. 2016, 116, 55–72. [CrossRef]
16. Martins, V.S.; Barbosa, C.C.F.; de Carvalho, L.A.S.; Jorge, D.S.F.; Lobo, F.L.; Novo, E.M.L.M. Assessment of
atmospheric correction methods for Sentinel-2 MSI images applied to Amazon floodplain lakes. Remote Sens.
2017, 9, 322. [CrossRef]
17. Wang, Q.; Blackburn, G.A.; Onojeghuo, A.O.; Dash, J.; Zhou, L.; Zhang, Y.; Atkinson, P.M. Fusion of Landsat
8 OLI and Sentinel-2 MSI data. IEEE Trans. Geosci. Remote Sens. 2017, 55, 3885–3899. [CrossRef]
18. Gao, Q.; Zribi, M.; Escorihuela, M.J.; Baghdadi, N. Synergetic use of Sentinel-1 and Sentinel-2 data for soil
moisture mapping at 100 m resolution. Sensors 2017, 17, 1966. [CrossRef] [PubMed]
19. Yang, X.; Zhao, S.; Qin, X.; Zhao, N.; Liang, L. Mapping of urban surface water bodies from Sentinel-2 MSI
imagery at 10 m resolution via NDWI-based image sharpening. Remote Sens. 2017, 9, 596. [CrossRef]
20. Eitel, J.U.; Vierling, L.A.; Litvak, M.E.; Long, D.S.; Schulthess, U.; Ager, A.A.; Krofcheck, D.J.; Stoscheck, L.
Broadband red-edge information from satellites improves early stress detection in a New Mexico conifer
woodland. Remote Sens. Environ. 2011, 115, 3640–3646. [CrossRef]
21. Sibanda, M.; Mutanga, O.; Rouget, M. Examining the potential of Sentinel-2 MSI spectral resolution in
quantifying above ground biomass across different fertilizer treatments. ISPRS J. Photogramm. Remote Sens.
2015, 110, 55–65. [CrossRef]
22. Clevers, J.G.P.W.; Kooistra, L.; van den Brande, M.M.M. Using Sentinel-2 data for retrieving LAI and leaf
and canopy chlorophyll content of a potato crop. Remote Sens. 2017, 9, 405. [CrossRef]
23. Pesaresi, M.; Corbane, C.; Julea, A.; Florczyk, A.J.; Syrris, V.; Soille, P. Assessment of the added-value of
Sentinel-2 for detecting built-up areas. Remote Sens. 2016, 8, 299. [CrossRef]
24. Lefebvre, A.; Sannier, C.; Corpetti, T. Monitoring urban areas with Sentinel-2A data: Application to the
update of the copernicus high resolution layer imperviousness degree. Remote Sens. 2016, 8, 606. [CrossRef]
25. Lu, D.; Weng, Q.A. Survey of image classification methods and techniques for improving classification
performance. Int. J. Remote Sens. 2007, 28, 823–870. [CrossRef]
26. Friedl, M.A.; Brodley, C.E. Decision tree classification of land cover from remotely sensed data.
Remote. Sens. Environ. 1997, 61, 399–409. [CrossRef]
27. Waske, B.; Braun, M. Classifier ensembles for land cover mapping using multitemporal SAR imagery. ISPRS J.
Photogramm. Remote Sens. 2009, 64, 450–457. [CrossRef]
28. Li, C.; Wang, J.; Wang, L.; Hu, L.; Gong, P. Comparison of classification algorithms and training sample
sizes in urban land classification with Landsat Thematic Mapper imagery. Remote Sens. 2014, 6, 964–983.
[CrossRef]
29. Jhonnerie, R.; Siregar, V.P.; Nababan, B.; Prasetyo, L.B.; Wouthuyzen, S. Random forest classification for
mangrove land cover mapping using Landsat 5 TM and Alos Palsar imageries. Procedia Environ. Sci. 2015,
24, 215–221. [CrossRef]
Sensors 2018, 18, 18 19 of 20

30. Basukala, A.K.; Oldenburg, C.; Schellberg, J.; Sultanov, M.; Dubovyk, O. Towards improved land use
mapping of irrigated croplands: Performance assessment of different image classification algorithms and
approaches. Eur. J. Remote. Sens. 2017, 50, 187–201. [CrossRef]
31. Prasad, A.; Iverson, L.; Liaw, A. Newer classification and regression tree techniques: Bagging and random
forests for ecological prediction. Ecosystems 2006, 9, 181–199. [CrossRef]
32. Naidoo, L.; Cho, M.A.; Mathieu, R.; Asner, G. Classification of savanna tree species, in the Greater Kruger
National Park region, by integrating hyperspectral and LiDAR data in a random forest data mining
environment. ISPRS J. Photogramm. Remote Sens. 2012, 69, 167–179. [CrossRef]
33. Breiman, L. Random forests. Mach. Learn. 2001, 45, 5–32. [CrossRef]
34. Adam, E.; Mutanga, O.; Odindi, J.; Abdel-Rahman, E.M. Land-use/cover classification in a heterogeneous
coastal landscape using RapidEye imagery: Evaluating the performance of random forest and support vector
machines classifiers. Int. J. Remote Sens. 2014, 35, 3440–3458. [CrossRef]
35. Ghosh, A.; Joshi, P.K. A comparison of selected classification algorithms for mapping bamboo patches in
lower Gangetic plains using very high resolution WorldView 2 imagery. Int. J. Appl. Earth Obs. Geoinf. 2014,
26, 298–311. [CrossRef]
36. Pouteaua, R.; Collinb, A.; Stolla, B. A Comparison of Machine Learning Algorithms for Classification of Tropical
Ecosystems Observed by Multiple Sensors at Multiple Scales; International Geoscience and Remote Sensing
Symposium: Vancouver, BC, Canada, 2011.
37. Heydari, S.S.; Mountrakis, G. Effect of classifier selection, reference sample size, reference class distribution
and scene heterogeneity in per-pixel classification accuracy using 26 Landsat sites. Remote Sens. Environ.
2018, 204, 648–658. [CrossRef]
38. U.S. Geological Survey. Available online: https://earthexplorer.usgs.gov/ (accessed on 22 July 2017).
39. Louis, J.; Debaecker, V.; Pflug, B.; Main-Knorn, M.; Bieniarz, J.; Mueller-Wilm, U.; Cadau, E.; Gascon, F.
Sentinel-2 Sen2Cor: L2A Processor for Users. In Proceedings of the Living Planet Symposium (Spacebooks
Online), Prague, Czech Republic, 9–13 May 2016; pp. 1–8.
40. Kuhn, M. Building predictive models in R using the caret package. J. Stat. Softw. 2008, 28, 1–26. [CrossRef]
41. Qian, Y.; Zhou, W.; Yan, J.; Li, W.; Han, L. Comparing machine learning classifiers for object-based land cover
classification using very high resolution imagery. Remote Sens. 2015, 7, 153–168. [CrossRef]
42. Knorn, J.; Rabe, A.; Radeloff, V.C.; Kuemmerle, T.; Kozak, J.; Hostert, P. Land cover mapping of large areas
using chain classification of neighboring Landsat satellite images. Remote. Sens. Environ. 2009, 113, 957–964.
[CrossRef]
43. Shi, D.; Yang, X. Support vector machines for land cover mapping from remote sensor imagery. In Monitoring
and Modeling of Global Changes: A Geomatics Perspective; Springer: Dordrecht, The Netherlands, 2015;
pp. 265–279.
44. Ballanti, L.; Blesius, L.; Hines, E.; Kruse, B. Tree species classification using hyperspectral imagery:
A comparison of two classifiers. Remote Sens. 2016, 8, 445. [CrossRef]
45. Exelis Visual Information Solutions. ENVI Help; Exelis Visual Information Solutions: Boulder, CO, USA, 2013.
46. Melgani, F.; Bruzzone, L. Classification of hyperspectral remote sensing images with support vector machines.
IEEE Trans. Geosci. Remote Sens. 2004, 42, 1778–1790. [CrossRef]
47. Huang, C.; Davis, L.S.; Townshend, J.R.G. An assessment of support vector machines for land cover
classification. Int. J. Remote Sens. 2002, 23, 725–749. [CrossRef]
48. Liaw, A.; Wiener, M. Classification and regression by randomForest. R News 2002, 2, 18–22.
49. Immitzer, M.; Atzberger, C.; Koukal, T. Tree species classification with random forest using very high spatial
resolution 8-Band WorldView-2 satellite data. Remote Sens. 2012, 4, 2661–2693. [CrossRef]
50. Zhang, H.K.; Roy, D.P. Using the 500 m MODIS land cover product to derive a consistent continental scale
30 m Landsat land cover classification. Remote Sens. Environ. 2017, 197, 15–34. [CrossRef]
51. Feng, Q.; Liu, J.; Gong, J. UAV remote sensing for urban vegetation mapping using random forest and texture
analysis. Remote Sens. 2015, 7, 1074–1094. [CrossRef]
52. Duda, R.; Hart, P. Pattern Classification and Scene Analysis; John Wiley & Sons: New York, NY, USA, 1973.
53. Franco-Lopez, H.; Ek, A.R.; Bauer, M.E. Estimation and mapping of forest stand density, volume and cover
type using the k-Nearest Neighbors method. Remote Sens. Environ. 2001, 77, 251–274. [CrossRef]
54. Akbulut, Y.; Sengur, A.; Guo, Y.; Smarandache, F. NS-k-NN: Neutrosophic Set-Based k-Nearest Neighbors
classifier. Symmetry 2017, 9, 179. [CrossRef]
Sensors 2018, 18, 18 20 of 20

55. Wei, C.; Huang, J.; Mansaray, L.R.; Li, Z.; Liu, W.; Han, J. Estimation and mapping of winter oilseed rape LAI
from high spatial resolution satellite data based on a hybrid method. Remote Sens. 2017, 9, 488. [CrossRef]
56. He, H.; Garcia, E.A. Learning from imbalanced data. IEEE Trans. Knowl. Data Eng. 2009, 21, 1263–1284.
57. Baraldi, A.; Puzzolo, V.; Blonda, P.; Bruzzone, L.; Tarantino, C. Automatic spectral rule-based preliminary
mapping of calibrated Landsat TM and ETM+ images. IEEE Trans. Geosci. Remote Sens. 2006, 44, 2563–2586.
[CrossRef]
58. Colditz, R.R. An evaluation of different training sample allocation schemes for discrete and continuous land
cover classification using decision tree-based algorithms. Remote Sens. 2015, 7, 9655–9681. [CrossRef]
59. Mellor, A.; Boukir, S.; Haywood, A.; Jones, S. Exploring issues of training data imbalance and mislabelling
on random forest performance for large area land cover classification using the ensemble margin. ISPRS J.
Photogramm. Remote Sens. 2015, 105, 155–168. [CrossRef]
60. Dalponte, M.; Orka, H.O.; Gobakken, T.; Gianelle, D.; Naesset, E. Tree species classification in boreal forests
with hyperspectral data. IEEE Trans. Geosci. Remote Sens. 2013, 51, 2632–2645. [CrossRef]
61. Jin, H.; Stehman, S.V.; Mountrakis, G. Assessing the impact of training sample extraction on accuracy of an
urban classification: A case study in Denver, Colorado. Int. J. Remote Sens. 2014, 35, 2067–2081.
62. Shao, Y.; Lunetta, R.S. Comparison of support vector machine, neural network, and CART algorithms for
the land-cover classification using limited training data points. ISPRS J. Photogramm. Remote Sens. 2012, 70,
78–87. [CrossRef]

© 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).