biocybernetics and biomedical engineering 39 (2019) 613–623

Available online at www.sciencedirect.com

ScienceDirect

journal homepage: www.elsevier.com/locate/bbe

Original Research Article

A deep learning model integrating SK-TPCNN and

random forests for brain tumor segmentation in MRI

Tiejun Yang, Jikun Song *, Lei Li

College of Information Science and Technology, Henan University of Technology, Zhengzhou, Henan, China

article info abstract

Article history: The segmentation of brain tumors in magnetic resonance imaging (MRI) images plays an
Received 8 July 2018 important role in early diagnosis, treatment planning and outcome evaluation. However,
Received in revised form due to gliomas' significant diversity in structure, the segmentation accuracy is low. In this
4 January 2019 paper, an automatic segmentation method integrating the small kernels two-path convolu-
Accepted 4 June 2019 tional neural network (SK-TPCNN) and random forests (RF) is proposed, the feature extrac-
Available online 12 June 2019 tion ability of SK-TPCNN and the joint optimization capability of model are presented
respectively. The SK-TPCNN structure combining the small convolutional kernels and large
convolutional kernels can enhance the nonlinear mapping ability and avoid over-fitting, the
multiformity of features is also increased. The learned features from SK-TPCNN are then
applied to the RF classifier to implement the joint optimization. RF classifier effectively
integrates redundancy features and classify each MRI image voxel into normal brain tissues
and different parts of tumor. The proposed algorithm is validated and evaluated in the Brain
Tumor Segmentation Challenge (Brats) 2015 challenge Training dataset and the better
performance is achieved.
© 2019 Nalecz Institute of Biocybernetics and Biomedical Engineering of the Polish
Academy of Sciences. Published by Elsevier B.V. All rights reserved.

images of brain tumors is important to the early diagnosis,

1. Introduction
Gliomas are the most common malignant tumor in brain
tumors with high mortality and rapid dispersion. For those
advanced tumors, the clinical population has a general
survival duration of two years or less, so the immediate
treatment is required [1,2]. The accurate segmentation in MRI
treatment planning and evaluation.
Because of its strong resolution to soft tissues, magnetic
resonance imaging (MRI) is one of the most commonly used
imaging method for diagnosing brain tumors in clinical
treatments, which provides detailed descriptions of brain
images. The MRI image has multiple modalities, different
modality shows the different image information from

* Corresponding author at: College of Information Science and Technology, Henan University of Technology, No.100 LianHua Street,
High-Tech Zone, Zhengzhou, Henan, China.
E-mail address: 201692304@stu.haut.edu.cn (J. Song).
https://doi.org/10.1016/j.bbe.2019.06.003
0208-5216/© 2019 Nalecz Institute of Biocybernetics and Biomedical Engineering of the Polish Academy of Sciences. Published by Elsevier
B.V. All rights reserved.
614 biocybernetics and biomedical engineering 39 (2019) 613–623
different sides and emphasizes the different contents. The T2-
weighted fluid attenuated inversion recovery (Flair) modality
can obtain the T2-weighted (T2) image which has cerebrospi-
nal fluid for low signal, is more sensitive to lesions. T1-
weighted (T1) modality image has a good display on tissue
anatomy. T1-weighted contrast-enhanced (T1c) modality has
an obvious boundary texture feature difference. The lesion
signal in T2 is prominent, which is suitable for observation on
lesion and its localization, quantitative and qualitative
analysis [3]. The single modality is difficult to provide
sufficient identifiable information of lesion tissues, so Flair,
T1, T1c and T2 are usually used as the diagnostic basis for brain
tumors.
Due to the interference of noise, bias field distortion and
tissue movement in the process of magnetic resonance
imaging, the image is blurred and the grayscale is not uniform.
Moreover, the number of images generated by MRI is huge,
brain tumor itself has an infiltrative growth with no obvious
boundary and other characteristics, many automatic segmen-
tation algorithms are not fully competent. Although manual
segmentation is time-consuming with poor repeatability, the
current clinical usually employ experienced doctors to manual
delineate tumor areas layer by layer with computer-aided
software on the MRI multiple modality images. Therefore, the
precise segmentation of brain tumor images by making use of
the machine is an inevitable trend in the development of
medical images.
At present, brain tumor segmentation methods can be
divided into generative model based methods and discrimi-
native model based methods [4].
Brain tumor segmentation based on generative model
generally requires to obtain prior information of a specific field
from probabilistic atlases, which can be modeled as an outlier
detection problem [5–7]. Menze et al. [8] pointed out that the
generative model has been well explored in the unseen data,
but it may be difficult to transform the prior knowledge
explicitly into an appropriate probability model.
The methods based on the discriminative model establish a
classification model by using relationship between image
features and voxels. The performance of segmentation
depends on complex features and classification algorithms.
Brain tumor segmentation algorithms use a variety of image
features, including local histograms, first-order textures,
gradients and structural tensors, etc. The most common
discriminative algorithms include support vector machine
(SVM) [9], decision trees [10], RF [11] etc. However, most
traditional classifiers can't provide usable segmentation
results in practice. SVM is good at processing small dataset
and simple binary-class classification task, while brain tumor
segmentation task is a multi-class problem and features are
complex. Decision trees are prone to over-fitting. RF is an
effective algorithm for handling multi-class classification
problems and high-dimensional feature vectors, shows the
best segmentation results in brain tumor segmentation [4,12].
Tustison et al. [13] developed a two-phase segmentation
framework based on RFs to improve the segmentation results
of the second classifier using the output of the first classifier.
Amiri et al. [14] established a multi-layer architecture to learn
tumor characteristics better, in which a powerful structured
random forests and Bayesian network were embed. Most brain
tumor segmentation techniques use hand-designed features
and input them into classifiers. The limitation is that a large
number of features must be used in order to describe the
images better, which results in the curse of dimensionality,
time-consuming and other issues. To solve this problem, the
CNN model is adopted, which has the ability of self-learning, is
suitable to deal with complex problems with large data
volume.
CNN can learn characteristics from huge amounts of data
directly by means of extracting features layer by layer,
achieves remarkable results in the field of image recognition,
and has been widely used in brain tumor segmentation.
Additionally, the fully convolutional network (FCN) model and
its variant have also been extensively studied and achieved
excellent segmentation results. It is notable that researchers
mainly explore several aspects as follow: (1) Design the
network structure to extract features better. Zikic et al. [15]
proposed a simple CNN with two convolutional layers and one
fully connection layer. Because of the relatively simple
structure, no more accurate segmentation results are
obtained. Pereira et al. [16] proposed a deeper CNN model
using 3x3 small convolutional kernels to segment brain tumor
images. Havaei et al. [17] established a cascade structure, the
results of the pixel-by-pixel probability obtained by the basic
CNN are used as additional input to its subsequent CNN. At the
same time, [17] also attempted to establish a two-pathway
CNN structure and can extract more contextual information
from the image. Hussain et al. [18] constructed the inception
linear (ILinear) nexus structure, in which the inception
structure was added to achieve a satisfactory segmentation
effect. In [19], an improved FCN algorithm was proposed to
solve the fuzziness of tumor boundary, the up-sampling phase
was divided into boundary tasks and regional tasks. Iqbal et al.
[20] proposed three improved network structures based on
SegNet and achieved satisfactory segmentation results.
Pereira et al. [21] added segmentation Squeeze-and-Excitation
(SegSE) blocks to FCN to achieve competitive segmentation
results. (2) Add post-processing procedures to improve
segmentation results. Kamnitsas et al. [22] used conditional
random fields (CRF) as a post-processing procedure to process
3D brain tumor images, the classification effect was greatly
improved. However, 3D-CNN can make full use of the 3D
information of the image, but it also increases the computa-
tional cost. Therefore, 2D-CNN has been more widely used.
Zhao et al. [23] integrated FCN and CRF in a unified framework
and specified CRF as a recurrent neural network (CRF-RNN).
Rao et al. [24] connected the final layer of the CNN with the
softmax layer to produce output for training RF classifiers. (3)
Explore a special loss function for brain tumor image, such as
weighted cross-entropy function and sensitivity function, etc.
Dong et al. [25] evaluated an automatic segmentation method
based on U-Net using Soft Dice loss function to solve the
unbalanced samples. Sudre et al. [26] developed a Generalized
Dice overlap loss function to mitigate the data unbalanced
issue. Xue et al. [27] proposed a new adversarial-critical
network with multi-scale L1 loss function.
It can be found that the research on brain tumor image
segmentation is generally carried out from three aspects:
model structure improvement, post-processing procedure and
loss function. Feature optimization is generally achieved
 biocybernetics and biomedical engineering 39 (2019) 613–623 615

by improving the model structure, then feature integration can

be optimized by adding post-processing program or integrat-
ing the label spatial dependency. These are two problems that
the CNN model needs to solve for brain tumor image
segmentation task.
In this paper, a joint optimization model that integrates an
improved CNN and RF for the automatic brain tumor
segmentation is presented. The main contributions can be
summarized as follows:

1) An improved CNN model combining small convolutional

kernels and large convolutional kernels, named SK-TPCNN,
is proposed. It integrates the nonlinear mapping capability
of small convolutional kernels and the advantage of large
convolutional kernels to extract contextual background
information, can extract more abundant image texture
features and spatial information.
2) A joint optimization model combining SK-TPCNN and RF
was employed to implement the automatic segmentation
of brain tumors. It can effectively integrate redundancy
features, improve the segmentation results obviously.
3) The proposed model is validated and evaluated on the Brats
Fig. 1 – Flow diagram of the proposed algorithm.
2015 challenge training dataset, promising segmentation
result is achieved.

The remainder of this paper is organized as follows.

In Section 2, the proposed method is presented. The databases
used for evaluation and the experimental setup are detailed in
Section 3. Results are presented and discussed in Section 4.
Finally, the main conclusions are presented in Section 5.
2. Method
The flow diagram of the proposed method is shown in Fig. 1.
There are three main steps: Pre-processing, Feature Extraction
and Post-processing.
the same tissue between images has different gray values.
In this paper, N4ITK method is used to conduct bias field
correction for images [28]. However, this is not sufficient to
ensure that the final intensity distribution is within a similar
range for different subjects in the same MRI sequence.
Therefore, intensity normalization algorithm is adopted to
bring the intensity values range to [0, 1]. In this way, the gray
histogram of each sequence will be more similar. After the
normalized operation on each MRI image, the gray mean and
standard deviation of the patches are calculated and normal-
ized to zero mean and unit variance.

Step 1. The 2D MRI image patches are pre-processed to
produce an image patches training dataset with four
modality.
Step 2. The training dataset is input into the constructed
SK-TPCNN model to complete the feature extraction, and
the loss function is minimized by the optimization
algorithm to obtain the SK-TPCNN optimal parameter
model.
Step 3. The output features of the SK-TPCNN optimal
parameter model are fed into the RF which used as the post-
processing procedure to complete the integrated optimi-
zation.
Step 4. Morphological operation is performed on the
segmentation result to smooth the map edges.
2.2. Convolutional neural network (CNN)
CNN is used to implements the whole segmentation procedure
by processing a 2D M
M image patch centered on a certain
pixel to predict the label of this pixel. Therefore, the training
dataset with four modalities is forwardly propagated through
the stacked convolutional layers, fully connected (FC) layers
and other layers. Each layer produces a certain number of
feature maps as outputs, uses a nonlinear activation function
at the end of the network to convert the features into the
probability distribution of five prediction labels, and then
minimizes the loss function by a back-propagation optimiza-
tion algorithm. The followings are the important information
of the CNN involved in this paper.

1. Convolutional layer (convolutional kernel): is used to

2.1. Pre-processing
Firstly, the 0.5% highest and 0.5% lowest intensity values of
each image sequence are removed in order to make the
intensity values of different image sequence within a coherent
range. Because MRI images are altered by bias field distortion,
extract features from different input planes. The feature
figure Os is calculated as follows:
Os ¼ bs þ Sr Wsr  Xr (1)
where, Xr is the rth input channel of the input term, Wsr is the
weight matrix of the channel rs convolutional kernel, * is the
616 biocybernetics and biomedical engineering 39 (2019) 613–623
convolutional operation, and bs is the bias term of a
convolutional kernel. The CNN kernels can be designed in
any sizes, such as 3
3, 7
7, 14
14, etc. Different kernel
sizes take into account different underlying background
information. In order to combine the local texture informa-
tion with contextual background information of the image,
one of the effective ways is to combine the small
convolutional kernels with the large convolutional kernels.
Inspired by [29], it is found that 3
3 small convolutional
kernel has a great potential to obtain the deepest convolu-
tional layer with the least number of parameters, which can
enhance the nonlinear mapping ability and prevent over-
fitting.
2. Activation function: the activation function can obtain the
nonlinear transformation of the input image Os. Common
rectified linear units (ReLU) are defined as follows:
Zs ¼ maxð0; Os Þ
Experiments show that ReLU has a better effect than the
classic sigmoid method and accelerates the training
convergence [30]. However, the constant 0 applied in ReLU
will affect the gradient flow and the corresponding weight
adjustment. Therefore, in this paper, a variant of ReLU—
LeakyReLU (LReLU) [31], was used to deal with these
limitations. The experiments also demonstrate that LReLU
has a lower validation loss than ReLU. It introduces a small
slope value in the negative part of the function, which was
defined as:


Os if Os  0
Zs ¼ (3)
alpha  Os if Os < 0
where, f(Os)s is the rectifying linear unit function, alpha is the
slope of the third quadrant oblique line in the activation
function image, and Os is the input image.
3. Pooling Layer: pooling layer makes the feature representa-
tion more concise and stable, reduces the computational
burden of the next stage. The max-pooling is calculated
Fig. 2 – Detail diagram of SK-TPCNN structure.
as follows:
Hs;i;j ¼ max Zs;iþp;jþp (4)
p
where i,j represents the spatial position on the output
feature map, p represents the size of the pooling window, Zs,
i,j is the output feature map after the activation function, Hs,i,
j is the output feature map after pooling operator. We add
overlapping max-pooling layers to reduce feature redun-
dancy and prevent the elimination of valid information.
4. Parameter Initialization: parameter initialization is crucial
for network convergence. In this paper, Xavier uniform
initialization method [32] is adopted. This method keeps the
activation value and gradient within a controllable range,
otherwise the gradient may diffuse even explode in the back
propagation.
(2) 5. SK-TPCNN Structure: This paper designs a SK-TPCNN
model, which is shown in Fig. 2, model structure parameters
are shown in Table 1. Through the experiments and
literature, we observe that as the number of path increases,
the learning ability of the network will be enhanced.
Meanwhile, the two-path model has become a popular
structure recently due to its advanced performance in
general image recognition and medical image segmentation
[33,34,35]. However, due to the differences in the image
properties of natural images and brain tumor images, the
two-path structure must be constructed based on the
characteristics of brain tumor images. Our intention for the
choice of two-pathway structure is that we hope the
prediction of a pixel's label will be affected by two sides:
the visual details of the area surrounding the pixel which
are represented as the tumor structure details [8] and its
larger contextual information, including the approximate
location of the patches in the brain. Therefore, a two-path
model is designed to achieve the optimal result. The two
paths designed in this paper are called the local path and
the global path, respectively. The local path stacks the 3
3
 biocybernetics and biomedical engineering 39 (2019) 613–623 617

Table 1 – Structure parameters of SK-TPCNN.

Layer Layer Filter Stride Num. Input size Output size
type size of filters (channel
W
H) (channel
W
H)
Local path Layer1 Conv. 3
3 1 64 4
33
33 64
33
33
Layer2 Conv. 3
3 1 64 64
33
33 64
33
33
Layer3 Conv. 3
3 1 64 64
33
33 64
33
33
Layer4 Max-pool. 3
3 2 64
33
33 64
16
16
Layer5 Dropout 64
16
16 64
16
16
Layer6 Conv. 3
3 1 128 64
16
16 128
16
16
Layer7 Conv. 3
3 1 128 128
16
16 128
16
16
Layer8 Conv. 3
3 1 128 128
16
16 128
16
16
Layer9 Max-pool. 2
2 2 128
16
16 128
8
8
Layer10 Dropout 128
8
8 128
8
8

Global path Layer1 Conv. 14
14 1 64 4
33
33 64
20
20

Layer2 Conv. 13
13 1 64 64
20
20 64
8
8
Layer3 Dropout 64
8
8 64
8
8

Concatenation Layer1 Merge 128
8
8+ 64
8
8 192
8
8

Layer2 FC 1
1 128 12,288 128
Layer3 Dropout 12,288 128
Layer4 FC 1
1 256 128 256
Layer5 Dropout 128 256
Layer6 FC 1
1 5 256 5

convolutional layers and forms a deeper network structure

with fewer parameters. The nonlinear mapping capability is
enhanced and it is not easy to appear over-fitting. If we
stack several 3
3 convolutional layers, the extracted
features become higher-level and more abstract with the
increasing depth. However, the feature extraction ability
reaches the maximum threshold when the number of
layers is stacked to a certain number, and does not increase
infinitely as the number of layers increases. Experiments
show that the six layers will get good results. The Global
Path can take into account the context background
information of pixels. It has large double convolutional
kernels of 14
14 and 13
13, which is the large which can
be used in global path to capture global information. The
double convolutional kernels can fully nonlinearly trans-
form the feature and increase the diversity of features. By Fig. 3 – DSC comparison of the image patches size.
convolving and pooling the input feature graph, the output
feature graphs at the top of the two paths are the same size.
Then, the output feature maps of the top hidden layers of
the two paths are linearly superimposed, the superimposed the closer the two probability distributions are. Because the
feature map is connected to a fully connected layer for loss function is highly nonlinear, adaptive moment estima-
output, and the softmax layer is only used during the tion algorithm [38] which is good at dealing with the sparse
training feature phase. Inspired by [16,17], we choose image gradients and non-stationary is used as the optimization
block with 33
33 as the training dataset, and we select the algorithm, and adaptive learning rates for different param-
size of 25
25 and 41
41 for comparison experiments to eters are calculated.
prove the superiority of 33x33 image patches, the compari-
son is shown in Fig. 3. Due to the large number of parameters and the deep layer
6. Training: The SK-TPCNN model uses the Brats2015 Training network, every block is followed by a dropout layer whose
dataset for training, the categorical cross entropy [36,37] is value is set to 0.1 to prevent over-fitting. The dropout layer
used as the loss function, cross entropy is used to measure drops a certain percentage of activation randomly at each
the difference between the actual distribution and the update during training by setting dropped units to zero. This
expected distribution which is defined as follow: reduces over-fitting as units have to learning independently
X X instead of relying on each other for producing an output.
Hðh; h0 Þ ¼  hi;j logðh0i;j Þ (5)
i 2 voxels j 2 classes
Although the two-path model of this paper has a large number
of parameters, our experimental results have shown that SK-
where h0 represents the predicted probability distribution TPCNN has been effectively trained and has good effects
and h is the true distribution. The lower the cross entropy is, without strong over-fitting. At first, a regularization method
618 biocybernetics and biomedical engineering 39 (2019) 613–623

such as the dropout layer is used. Secondly, the characteristics

of different patients' brains MRI images are similar, and the
changes among these brain tumor images are far less than the
general image data, so the amount of data we need is not
particularly large.
Using the grid search method and cross-validation to fine-
tune the other hyper-parameters, the selected hyper-param-
eters make the network perform best on the verification set,
optimize the network performance. Specific model hyper-
parameters are shown in Table 2.

2.3. Integrated optimization of SK-TPCNN and RF

Rao et al. [24] take the output of the last hidden layer of each
CNN as the representation of a pixel in each modality, and use
the concatenation of the representations of all the modalities
as features to train a random forest classifier. In this paper, SK-
TPCNN is used as the feature extraction unit, RF is used as the
classifier to complete the post-processing procedure, classify
and determine each pixel type of brain tumor image.
Integrated optimization framework of SK-TPCNN and RF is
shown in Fig. 4.
The multi-output features dataset U of SK-TPCNN were
used as the training dataset of RF, which is the concatenation
of Local Path output, Global Path output and the FC layer
output. By combining the shallow features and deep features, Fig. 4 – Integrated optimization framework of SK-TPCNN
the higher level features can be obtained. If the features are and RF.
more abundant, the classification accuracy of RF will be
improved as the feature dimensions increase. The feature
vector for voxels in concatenation representation U are fed 2.4. Morphological operator
into each decision tree Ti(i = 1, 2, . . ., N1, N1 = 50) to obtain five
probability distributions c 2 {1, 2, 3, 4, 5}, each degree of We use the morphological operator to improve the segmenta-
confidence p(i, s)( f(u) = c) represents the probability that tion result weakly. Due to the high intensity values around the
sample s belongs to class c. Based on the classes assigned skull part of the brain image, some false positives may occur
for each voxel in the test dataset, the final segmentation is during the segmentation process. The morphological opening
created by mapping the voxel estimated class to the operation method is used to erosion and expand the
segmentation volume. The classification results are based segmentation result, and some small false positives at the
on the average of all decision tree results and are voted as edge of the segmented image are removed, so that the
follows: segmentation edge tends to be smooth. The comparison
results of the morphological operation are shown in Fig. 5:
From Fig. 5, it can be seen that the segmentation edge after
1 X N1
morphological operation is smoother, and the number of false
FðsÞ ¼ argmaxpc ðpÞ ¼ argmaxc pði; uÞð f ðuÞcÞ (6)
N1 i¼1 positives points are reduced to some extent. Specifically, the Dice
metrics of Fig. 5(b) in three regions are: 0.877, 0.807, 0.968; and the
RF parameters which mainly include the number and depth Dice metrics of Fig. 5(c) in three regions are: 0.883, 0.815, 0.968.
of decision trees are continuously adjusted by testing the brain
tumor training dataset, then the results are verified using
multiple cross-validation. In the end, 50 decision trees are 3. Experiment step
built, each with depth of 30.
3.1. Database

The method presented in this paper is validated by the

Table 2 – Hyper-parameters of SK-TPCNN.
Brats2015 Training dataset [8]. Each patient in the Brats
Aspect Parameters Values database has four modality sequences, as shown in Fig. 6: Flair,
Initialization Bias 0.1 T1, T1c, T2, and all have different positions, size and different
Weights (0.005, 0.005) shapes of the tumor regions, all the images sequence of the
LRelu alpha 0.3 same object were already aligned with the T1c modality and
Dropout 0.1 skull stripped. Brats2015 Training dataset includes 220 high
L2 regularization Regularizer_params 0.01
grade gliomas (HGG) and 54 low grade gliomas (LGG). Each
Adam lr 0.0001
group of images has an artificial segmentation label. Manually
 biocybernetics and biomedical engineering 39 (2019) 613–623 619

Fig. 5 – Comparison of segmented results. From left to right: (a) ground truth; (b) segmented results without morphological
operator; (c) segmented results with morphological operator.

Fig. 6 – MRI image modal sequence. From left to right: flair modality; T1 modality; T1C modality; T2 modality.

segmentation labels divide tumor images into five categories:
normal tissue, edema, necrosis, enhancing tumor, and non-
enhancing tumor. However, the validation criteria were
divided into enhancing tumor, core tumor (necrosis + non-
enhancing tumor + enhancing tumor) and complete tumor
(except normal tissue).
includes dice similarity coefficient (DSC), positive predictive
value (PPV) and sensitivity (Sen) [8,27].
The DSC measures the overlap between the artificially
segmented labels and the automatically segmented results,
is defined as:

2TP
3.2. Experiment details DSC (7)
FP þ 2TP þ FN

The brain tumor segmentation problem proposed in this paper PPV measures the number of FP and TP, is defined as:
is a multi-class segmentation problem that includes five types
of regions: normal tissue, necrosis, edema, non-enhancing
TP
tumor and enhancing tumor. However, there is a high degree PPV ¼ (8)
TP þ FP
of data imbalance in the brain tumor segmentation problem,
healthy tissues account for about 98% of the entire brain. The sensitivity helps to evaluate the quantity of TP and FN,
Selecting patches from actual data distribution will cause the is defined as:
model to be overwhelmed by healthy tissue patches data and
make it to be biased toward the healthy pixels. Therefore, this
TP
paper uniformly extracts the same amount of data from all Sensitivity ¼ (9)
TP þ FN
categories so that all categories are fully trained, healthy pixels
tend to be normal distribution. Among them, TP, FP and FN indicate the number of true
The methods experiment on Keras framework based on the positive, false positive, false negative points, respectively.
Tensorflow. The experimental machine uses an Intel Core i7
3.5GHZ processor and is equipped with an NVIDIA GeForce
4. Results and discussion
GTX1080 GPU.

3.3. Evaluation and validation To evaluate the performance of the proposed method, two
comparative experiments are established. In the first scenario,
The 5-fold cross-validation method is used to validate the the images that are directly segmented by the SK-TPCNN
performance of the model. The evaluation criteria mainly model are used as the final segmentation result. In the second
620 biocybernetics and biomedical engineering 39 (2019) 613–623
scenario, the image features are automatically extracted by the
SK-TPCNN model and then classified by RF. It demonstrates
the excellent classification capability of RF and the superior
segmentation capability of the joint optimization model
consisted of SK-TPCNN and RF. The segmentation results
are compared as shown in Fig. 7, where (a) is the patient's T1
modality imaging, (b) is the ground truth, used as the gold label
in this paper, (c) is the segmentation result of the SK-TPCNN
model, and (d) is the segmentation result of the SK-TPCNN
+ RF model. Among them, red is the necrosis region, green is
the edema region, blue is the non-enhancing tumor region,
and yellow is the enhancing tumor area. It can be seen from
Fig. 7, the SK-TPCNN model has already achieved a relatively
accurate segmentation result, which has a strong feature
extraction ability and can effectively integrate features from
two paths. The segmentation results of the enhancing tumor
region is the best, but the other regions are relatively rough,
especially the edema region is over-segmented. From Fig. 7(d),
we can see that after the pixel classification by the RF classifier,
the performance is improved, and the segmentation edge of
each tumor region is more refined. Further, the box plots and
comparison plots of the two evaluation results for the main
evaluation factors are shown in Figs. 8 and 9. The above
comparative experiment results show that SK-TPCNN has
excellent feature extraction ability, and integrate of RF and SK-
TPCNN can greatly improve the classification accuracy as well
as achieve an accurate segmentation of brain tumors.
In order to further quantify the performance of our
algorithms, we compared it with several classic brain tumor
segmentation algorithms, all of which ranked among the top
in recent Brats competitions.
Table 3 lists the evaluation results of models proposed in
this paper and other models. It can be seen that no algorithm
can rank first in all metrics in all regions, but the SK-TPCNN
Fig. 7 – Comparison of segmented results. From left to right: (a) TI sequences; (b) ground truth; (c) SK-TPCNN results; (d) SK-
TPCNN + RF results.
+ RF model has excellent performances on most metrics, and
rank first among these algorithms for several metrics.
Specifically, in the first scenario, the segmentation results
directly obtained by the SK-TPCNN model works well, the DSC
overlap is medium in many algorithms, and the enhancing
region performs better. Among them, [20,25,27] do not add any
post-processing procedures, only use the designed convolu-
tional model to extract the feature and then use softmax for
classification. So they can compare the feature extraction
ability with SK-TPCNN to some extent without considering the
influence of classifiers and other factors. It can be seen from
the table that SK-TPCNN is highly competitive in these feature
extraction algorithms. The PPV is low, shows that the feature
extraction ability still has a lot of room for improvement, but
the enhancing region performs better than most algorithms.
Sensitivity shows good results, the complete and core regions
perform better in these algorithms. This shows that the SK-
TPCNN model can detect regions including tumors, but it
cannot accurately detect boundaries. It can be seen from the
table that SK-TPCNN excessively segment the tumor area,
especially the core area.
In the second scenario, using the RF classifier for pixel
classification, it can be seen that the accuracy of joint model is
much better than SK-TPCNN model. The DSC in all three
regions has been improved, and the complete and core regions
have produced comparable results among these algorithms,
the improvement of the enhancing region is particularly
noticeable. According to Menze et al. [8], the segmentation of
enhancing tumor region is one of the most difficult tasks in
brain tumor segmentation tasks, the proposed model has
outperformed other algorithms in terms of DSC in the
enhancing region. Furthermore, the sensitivity of the complete
and core areas remains unchanged roughly, the PPV has a
visible increase, and the enhancing area also performs well
 biocybernetics and biomedical engineering 39 (2019) 613–623
Fig. 8 – Indicator box diagram of SK-TPCNN and SK-TPCNN + RF segmentation results.
in all algorithms. It shows that the joint optimization of SK-
TPCNN and RF can effectively integrate the features and
reduce the feature redundancy.
From the above analysis, we can see that the SK-TPCNN
model designed in this paper has a strong feature extraction
capability and obtains relatively accurate results. Although the
Fig. 9 – DSC comparison of SK-TPCNN and SK-TPCNN + RF
segmentation results.
621
SK-TPCNN model is over-segmented, this situation can be
improved by increasing the amount of training data. The
combination of stacked 3
3 convolutional kernels with large
convolutional kernels is a viable strategy. The integration of RF
and SK-TPCNN model has greatly improved the segmentation
results which makes the segmentation result stand out in
many classic algorithms.
5. Summary
According to the precise segmentation of brain tumors in MRI
image, this paper proposes a novel SK-TPCNN model and an
integrated model combined SK-TPCNN with RF. Experiments
show that the SK-TPCNN model combines the 3
3 convolu-
tional kernels and large convolutional kernels, can improve
the ability of image feature extraction. Therefore, the SK-
TPCNN model proposed in this paper is still a very potential
convolutional network structure. The joint optimization of SK-
TPCNN and RF greatly improves the classification accuracy
and reduces the feature redundancy. Since SK-TPCNN is
designed and trained according to the characteristics of brain
tumor images, the general segmentation verification experi-
622 biocybernetics and biomedical engineering 39 (2019) 613–623

Table 3 – Quantitative results of proposed methods compared to the results from the classic brain tumor segmentation
algorithms published recently. The bold numbers highlight the scores best among these algorithms on the Brats 2015
Training dataset.
Algorithm Data DSC PPV Sensitivity

Complete Core Enh. Complete Core Enh. Complete Core Enh.

SK-TPCNN BRATS2015 Training Datasets 0.86 0.74 0.81 0.78 0.61 0.87 0.95 0.91 0.76
SK-TPCNN + RF BRATS2015 Training Datasets 0.89 0.80 0.87 0.83 0.70 0.92 0.96 0.92 0.83
Dong [25] BRATS2015 Training Datasets 0.85 0.76 0.74 * * * * * *
Lun [39] BRATS2015 Training Datasets 0.87 0.67 0.85 * * * * * *
Kamnitsal [22] BRATS2015 Training Datasets 0.90 0.75 0.73 0.92 0.86 0.76 0.89 0.72 0.74
Xue [27] BRATS2015 Training Datasets 0.85 0.70 0.66 0.80 0.65 0.62 0.92 0.69 0.80
Iqbal [20] BRATS2015 Training Datasets 0.88 0.81 0.81 0.83 0.84 0.84 0.86 0.80 0.80

ment is not performed on other lesion datasets. However, due
to the complexity of brain tumor images relative to other
lesion images and the advancement of SK-TPCNN itself, SK-
TPCNN has a certain generalization segmentation ability. For
the segmentation of lesions in other parts, the structure and
parameters should be fine-tuned according to the datasets.
The algorithm proposed in this paper is verified by the
Brats2015 challenge Training dataset and has achieved
satisfactory performance for whole regions, produces a better
performance for enhancing region in particular.
Author statement
Tiejun Yang contributed to concept and opinion of work,
critical revision of the paper, and final approval.
Jikun Song contributed to image analysis, software design,
development techniques, segmentation accuracy, and draft-
ing the paper.
Lei Li contributed to concept of work and critical revision
of the paper.
Acknowledgments
This study was supported by the Natural Science Foundation
of Henan Province Science and Technology Committee [Grant
No. 162102210189].
references
[1] Bauer S, Wiest R, Nolte LP, Reyes M. A survey of MRI-based
medical image analysis for brain tumor studies. Phys Med
Biol 2013;58(13):97–129.
[2] Meir EGV, Hadjipanayis CG, Norden AD, Shu HK, Wen PY,
Olson JJ. Exciting new advances in neuro-oncology: the
avenue to a cure for malignant glioma. CA Cancer J Clin
2010;60(3):166–93.
[3] Goetz M, Weber C, Binczyk F, Polanska J, Tarnawski R,
Bobek-Billewicz B, et al. DALSA: domain adaptation for
supervised learning from sparsely annotated MR images.
IEEE Trans Med Imaging 2016;35(1):184–96.
[4] Pinto A, Pereira S, Correia H, Oliveira J, Rasteiro MD, Carlos
AS. Brain tumor segmentation based on extremely
randomized forest with high-level features. Proc. 2015 37th
Annual International Conference of the IEEE. 2015. pp.
3037–40.
[5] Gooya A, Pohl KM, Bilello M, Cirillo L, Biros G, Melhem ER,
et al. GLISTR: glioma image segmentation and registration.
IEEE Trans Med Imaging 2012;31(10):1941–54.
[6] Menze BH, Leemput KV, Lashkari D, Weber MA, Ayache N,
Golland P. A generative model for brain tumor
segmentation in multi-modal images. Proc. International
Conference on Medical Image Computing and Computer-
Assisted Intervention. 2010. pp. 151–9.
[7] Prastawa M, Bullitt E, Ho S, Gerig G. A brain tumor
segmentation framework based on outlier detection. Med
Image Anal 2004;8(3):275–83.
[8] Menze BH, Jakab A, Bauer S, Kalpathy CJ, Farahani K, Kirby J,
et al. The multimodal brain tumor image segmentation
benchmark (BRATS). IEEE Trans Med Imaging 2015;34
(10):1993–2024.
[9] Bauer S, Nolte LP, Reyes M. Fully automatic segmentation of
brain tumor images using support vector machine
classification in combination with hierarchical conditional
random field regularization. Proc. International Conference
on Medical Image Computing and Computer-Assisted
Intervention. 2011. pp. 354–61.
[10] Zikic D, Glocker B, Konukoglu E, Criminisi A, Demiralp C,
Shotton J, et al. Decision forests for tissue-specific
segmentation of high-grade gliomas in multi-channel MR.
Medical Image Computing and Computer-Assisted
Intervention (MICCAI). 2012. pp. 369–76.
[11] Tustison NJ, Shrinidhi KL, Wintermark M, Durst CR, Kandel
BM, Gee JC, et al. Optimal symmetric multimodal templates
and concatenated random forests for supervised brain
tumor segmentation (simplified) with ANTsR.
Neuroinformatics 2015;13(2):209–25.
[12] Gotz M, Weber C, Blocher J, Stieltjes B, Meinzer HP,
Maier-Hein K. Extremely randomized trees based
brain tumor segmentation. Proc. MICCAI Multimodal
Brain Tumor Segmentation Challenge (BraTS); 2014. p.
006-011.
[13] Tustison NJ, Shrinidhi KL, Wintermark M, Durst CR, Kandel
BM, Gee JC, et al. Optimal symmetric multimodal templates
and concatenated random forests for supervised brain
tumor segmentation (Simplified) with ANTsR.
Neuroinformatics 2015;13(2):209–25.
[14] Amiri S, Mahjoub MA, Rekik I. Bayesian network and
structured random forest cooperative deep learning for
automatic multi-label brain tumor segmentation. Proc.
International Conference on Agents and Artificial
Intelligence. 2018. pp. 183–90.
[15] Zikic D, Ioannou Y, Brown M, Criminisi A. Segmentation of
brain tumor tissues with convolutional neural networks.
Proc. MICCAI Multimodal Brain Tumor Segmentation
Challenge (BraTS). 2014. pp. 36–9.
 biocybernetics and biomedical engineering 39 (2019) 613–623
[16] Pereira S, Pinto A, Alves V, Silva CA. Brain tumor
segmentation using convolutional neural networks in MRI
images. IEEE Trans Med Imaging 2016;35(5):1240–51.
[17] Havaei M, Davy A, Wardefarley D, Biard A, Courville A,
Courville A, et al. Brain tumor segmentation with deep
neural networks. Med Image Anal 2015;35:18–31.
[18] Hussain S, Anwar SM, Majid M. Segmentation of glioma
tumors in brain using deep convolutional neural network.
Neurocomputing 2018;282:248–61.
[19] Shen H, Wang R, Zhang J, Mckenna SJ. Boundary-aware fully
convolutional network for brain tumor segmentation. Proc.
International Conference on Medical Image Computing and
Computer-Assisted Intervention. 2017. pp. 433–41.
[20] Iqbal S, Ghani MU, Saba T, Rehman A. Brain tumor
segmentation in multi-spectral MRI using convolutional
neural networks (CNN). Microsc Res Tech 2018;81(10):419–27.
[21] Pereira S, Alves V, Silva CA. Adaptive feature recombination
and recalibration for semantic segmentation: application to
brain tumor segmentation in MRI; 2018, arXiv:1806.02318.
[22] Kamnitsas K, Ledig C, Newcombe VF, Simpson JP, Kane AD,
Menon DK, et al. Efficient multi-scale 3D CNN with fully
connected CRF for accurate brain lesion segmentation. Med
Image Anal 2016;36:61–78.
[23] Zhao XM, Wu YH, Song GD, Li ZY, Zhang YZ, Fan Y. A deep
learning model integrating FCNNs and CRFs for brain tumor
segmentation. Med Image Anal 2017;43:98–111.
[24] Rao V, Sharifi M, Jaiswal A. Brain tumor segmentation with
deep learning. Proc. MICCAI Multimodal Brain Tumor
Segmentation Challenge (BraTS). 2015. pp. 56–9.
[25] Dong H, Yang G, Liu F, Mo Y, Guo Y. Automatic brain tumor
detection and segmentation using U-Net based fully
convolutional networks. Proc. Conference on Medical
Image Understanding and Analysis. 2017. pp. 506–17.
[26] Sudre CH, Li W, Vercauteren T, Ourselin S, Cardoso MJ.
Generalised dice overlap as a deep learning loss function
for highly unbalanced segmentations; 2017, arXiv:1707.
03237v3.14.
623
[27] Xue Y, Xu T, Zhang H, Long R, Huang X. SegAN: adversarial
network with multi-scale L1 loss for medical image
segmentation. Neuroinformatics 2018;1–10.
[28] Tustison NJ, Avants BB, Cook PA, Zheng Y, Egan A,
Yushkeich PA, et al. N4ITK: improved n3 bias correction.
IEEE Trans Med Imaging 2010;29(6):1310–20.
[29] Simonyan K, Zisserman A. Very deep convolutional
networks for large-scale image recognition. Comput Sci
2014.
[30] Krizhevsky A, Sutskever I, Hinton GE. ImageNet
classification with deep convolutional neural networks.
Proc. International Conference on Neural Information
Processing Systems, vol. 60, no. 2. 2012. pp. 1097–105.
[31] Maas AL, Hannun AY, Ng AY. Rectifier nonlinearities
improve neural network acoustic models. Proc. 30th
International Conference on Machine Learning, vol. 30;
2013.
[32] Glorot X, Bengio Y. Understanding the difficulty of training
deep feedforward neural networks. J Mach Learn Res
2010;9:249–56.
[33] Bai S, Li ZH, Hou JJ. Learning two-pathway convolutional
neural networks for categorizing scene images. Multimed
Tools Appl 2017;76(15):16145–62.
[34] Sun TC, Wang YL, Yang J. Convolution neural networks
with two pathways for image style recognition. IEEE Trans
Image Process 2017;26(9):4102–13.
[35] Wang MM. Multi-path convolutional neural networks for
complex image classification; 2015.
[36] Pieter TB, Dirk PK, Shie M, Reuven YR. A tutorial on the
cross-entropy method. Ann Oper Res 2005;134(1):19–67.
[37] Rubinstein RY. Optimization of computer simulation
models with rare events. Eur J Oper Res 1997;99(1):89–112.
[38] Kingma DP, Ba J. Adam: a method for stochastic
optimization. Comput Sci 2014.
[39] Lun TK, Hsu W. Brain tumor segmentation using deep
convolutional neural network. Proc. MICCAI Multimodal
Brain Tumor Segmentation Challenge (BraTS); 2016.