IET Image Processing
Research Article
Semantic image segmentation using an
improved hierarchical graphical model
Neda Noormohamadi1, Peyman Adibi1 , Sayyed Mohammad Saeed Ehsani1
1Artificial
Intelligence Department, Faculty of Computer Engineering, University of Isfahan, Isfahan, Iran
E-mail: adibi@eng.ui.ac.ir
Abstract: Hierarchical graphical models can incorporate jointly several tasks in a unified framework. By applying this approach,
information exchange among tasks would improve the results. A hierarchical conditional random field (CRF) is proposed here to
improve the semantic image segmentation. Although this newly proposed model applies the information of several tasks, its run
time is comparable with the contemporary approaches. This method is evaluated on MSRC dataset and has shown similar or
better segmentation accuracy in comparison with models where CRFs or hierarchical models are adopted.
1 Introduction
The objective of computer vision is to provide the capability of
understanding the world by intelligent processing. Such a vision is
obtained through extraction of useful information from digital
images and performing complex inferences on input data [1]. This
task is performed by constructing a statistical model relating image
data to high-level information. The probabilistic graphical models,
as a consolidated group of statistical modelling tools widely
applied in computer vision problems such as stereo matching [2]
and class segmentation [3], constitute the appropriate candidates
for this purpose.
There exist many studies on isolated solving of individual
machine vision problems, including tasks like object detection,
scene understanding, and pose estimation. Obviously, there exists a
meaningful relation among these tasks. For example, if the type of
image scene is known, the objects expected to be seen will be
limited. Thus, if these tasks are solved in a unified framework, in a
sense that information is exchanged among them, then expecting to
improved results is justified. By applying this approach, the system
can accomplish a holistic perception from the image. Holistic scene
understanding is an excellent objective in machine vision [4].
Different methods have been proposed to solve this problem. Some
of them apply the output of each of the tasks (i.e. scene
recognition, depth estimation, and segmentation) in the model in a
sequential manner, by applying the output of one task (e.g. object
detection) as the features for the other tasks (e.g. image
segmentation) [5–7]. Another method, depicted here applies jointly
several tasks in a unified framework in a hierarchical structure set
up [1, 8, 9]. Many researchers formulate the joint reasoning for
multiple tasks, as an inference problem in an undirected graphical
model. The most popular graphical model adopted in this area is
the conditional random field (CRF) [10, 11], where often the lower
layer corresponds to the image segmentation task. The ith node of
this layer represents the semantic label of the ith pixel/super-pixel
(SP) in the image.
In the probabilistic graphical models adopted in such multi-task
vision problems, the image information is embodied into the model
by unary potentials. Moreover, pairwise potentials are applied in
the model to smooth the labels of adjacent nodes. If several tasks
are applied in the model, each task is added to the model as a
separate layer. According to each CRF, a problem-specific energy
function is defined, which should be minimised, in order to solve
the problem.
In the most of the related literature, information is extracted
from small regions of an image like the SPs. Although this
information, named local, is effective, it is not efficient and many
IET Image Process., 2018, Vol. 12 Iss. 11, pp. 1943-1950
© The Institution of Engineering and Technology 2018
ISSN 1751-9659
Received on 12th July 2017
Revised 3rd February 2018
Accepted on 12th June 2018
E-First on 16th July 2018
doi: 10.1049/iet-ipr.2017.0738
www.ietdl.org
ambiguities still remain in the model. To overcome this drawback,
the idea of non-local interest regions [12, 13] is suggested to define
the pairwise potentials although unary and pairwise potentials are
not adequate to eliminate all ambiguities in the image. New
approaches are proposed where global information is integrated
into the problem [14–16]. Adding global information to the model
as a layer in the graph [17] is one of the suggestions in eliminating
ambiguities.
Besides these approaches, the deep learning techniques [18, 19]
are being proposed to capture more contextual cues, where a multi-
layer convolutional neural network (CNN) is applied to generate
hierarchical features in order to segment the scenes. Most recently,
an attempt is made to combine the strengths of CRFs and CNNs by
the authors in [20, 21]. To this end, the problem is formulated as a
CRF, while feature extraction, inference, or training is dealt with
through CNN. They produce promising segmentation results.
The objective of this study is to assign a semantic label to each
pixel of an input image. To this end, our suggested method applies
multiple tasks including object detection, scene recognition, and
semantic segmentation in a simultaneous manner. Here, it is
assumed that a basic set of pixel labels, a probabilistic list of scene
types for each image, and a set of candidate objects detected in the
image are available; accordingly a method with the following
characteristics is being proposed:
(i) The proposed model not only can improve the segmentation
results by utilising several potential functions but also simplifies
the model as explained in the experimental section.
(ii) Taking inspiration from [22], an improved unary potential is
applied to this proposed model for each SP of the image. Unlike
the findings in [22], both the local and global information are
combined as a term in energy function.
(iii) A class of potential functions is the shape prior for the objects.
The quality of the shape prior in this paper is enhanced.
The proposed method has similar ideas with the work of Yao et
al. [17], in presenting a holistic approach to scene understanding
which jointly inferences about segmentation, the scene type,
presence of a class in the image, and presence of objects in the
image which are the instances of the classes (e.g. an image
including three horses has three objects of the class ‘horse’).
However, there exist differences between the method of [17] and
our newly proposed approach. One important difference is that they
apply scene type in the model by adding a separate layer to the
graph, while, here the scene type in the model is applied by
combining it with the local unary potential of SPs. This approach
1943
promotes the model simplicity. In spite of their model, here the
contextual relations among candidate objects are of concern.
The rest of the paper is organised as follows: the proposed
approach is introduced in Section 2; the experimental results are
discussed in Section 3 and the conclusion is presented in Section 4.
2 Proposed approach
The image semantic segmentation problem is formulated as an
inference problem in a CRF model. Since this suggested model
extends a basic CRF, we first provide a brief background material
on the basic CRF and hierarchical CRF (HCRF) and then explain
our proposed method.
2.1 Basic CRF
On a given image, the basic CRF [10] is defined on a set of random
variables X = xi , i ∈ V where V corresponds to the image
elements like pixels, SPs, and patches. For segmentation task, each
xi can take any label in a discrete label space L representing the set
of object categories like a bottle, a car, a person, and so on. In
correspondence with each CRF, an energy function E(X) is
defined. Most of the inference methods seek to minimise the
following energy function:
E(X) = ∑ λcψ c(Xc) (1)
c∈C
where λc is the weight vector, C is the set of cliques, and ψ c is the
cost function (named potential function), defined over a clique c of
a set of variables Xc involved in this clique (e.g. the set of pixels in
a SP). The potential functions assign scores to a clique for different
labelling.
The following refers to the commonly applied potential
functions. A unary potential (named data term) is defined over just
one variable. This function calculates the scores through the
appearance-based or location-based features. A pairwise potential
(named smooth term) is defined over each pair of the neighbouring
variables vi and v j. This function encourages the neighbouring
variables to get the similar labels from the label space. The
objective of the pairwise potential functions is to cause a spatial
smoothness in final labelling.
In addition to these common potential functions, higher order
potential functions may be defined on bigger cliques. Bigger
connectivity can incorporate complex priors into the model, and
reduce discretisation artefacts. Although the higher order potentials
are useful in the models, they lead to too complex inferences. Thus,
in most studies, the cliques with a size of greater than two are not
considered and if are considered each of the higher order cliques is
decomposed into a set of the pairwise ones [17]. A clique of pixels
that belongs to a SP is an example of a higher order clique.
2.2 Hierarchical CRF
Researchers like Sun et al. [8] have proposed methods where
information about several tasks are applied jointly in computer
vision problem. HCRF [8, 23] is such a multi-task model
developed on top of a basic CRF, where each layer corresponds to
one task. In HCRF, information is exchanged between tasks
through pairwise potentials defined among layers.
2.3 Proposed extended HCRF
In order to jointly take advantage of scene classification, object
detection, and object segmentation, the problem is modelled
through a HCRF. The architecture of this newly proposed model is
shown in Fig. 1. Its main components are described in the
following.
There are three modules in Fig. 1. Module 1 runs two
segmentation procedures in a coarse to fine manner over an input
image, which results in two sets of image segments, called SPs and
super-segments (SSs). In this model, for both of the segmentation
levels, the pixels that lie inside the same segment are assumed to
1944
take the same label. It is clear that if the segment generator
algorithm is not accurate sufficiently, a segment may include
several objects in the image, which results in a big fault in the
model. However, if it is accurate enough, its application decreases
the model size.
Moreover, the label of a pixel may be subject to the long-range
information, i.e. to determine the label of a green pixel (‘grass’ or
‘tree’), information about the surrounding pixels should be known.
In module 2, the candidate objects of the image are detected by a
part-based object detector model, which assumes an object has
several parts [24]. For each one of the detected objects, the method
of [24] specifies an object class, the location of the detected object,
a score, and the index of a root mixture component. Module 3
computes a probability vector for the image, containing the
classifier scores for each scene class.
By applying the extracted segments and the detected objects, a
multi-level graph is drawn for this proposed model, as illustrated in
Fig. 1, where the low-level nodes correspond to the smaller
segments of the image named super-pixel (SP). The second level
nodes are associated with extracted segments in a larger size. The
bigger segments are named super-segment (SS). Each SS is
connected to the SPs which lie in that (SS). In the higher level, the
model consists of several binary variables bl ∈ {0, 1}, assigned to
each detected object, which allow the model to accept or reject
these detections. For each detected object its constituent SPs are
known. Knowing the class of an object influences the prediction
about the classes of its corresponding SPs. Therefore, at the higher
level, each object is connected to its constituent SPs in the graph.
By using the outputs of the modules, the energy function over
the generated graph which consists of six parts that related to the
model scene classification, object detection, and object
segmentation in a simultaneous manner is presented as
E(x, y, b) = w1E1 + w2E2 + w3E3 + ∑ w4clsE4 + ∑ wrE 5 5
cls ∈ C r∈R
(2)
+ ∑ w6clsE6
cls ∈ C
where x and y are the random variables associated with the image
SPs and SSs, respectively, b is the set of candidate objects in the
image, C consists of image semantic labels, R consists of semantic
relations among candidate objects, and each E1 to E6 is one term of
energy function which will be explained separately in the following
subsections. A different weight of w1, w2, w3, w4cls, w5r, and w6cls, is
defined for each term of the energy function that will be learned in
the training step through the structure prediction framework [25].
2.3.1 Unary potential for each SP and SS: In order to take
advantage of the image segmentation task, information is
introduced into the model by defining the terms E1 and E2 as
follows:
E1 = ∑ ∅i(xi) , (3)
i ∈ SP
E2 = ∑ ∅ j(y j) (4)
j ∈ SS
where ∅i and ∅ j encode the unary potential functions over each SP
and SS, respectively. By employing SSs, we take advantage of the
long range dependencies in this proposed model. To calculate ∅i,
local unary potential and global feature extracted from the entire
image must cooperate. The local unary potential corresponding to
each SP is computed by averaging the TextonBoost [26] pixel
potentials inside each SP. Markov random fields are mostly able to
model the local interactions between nodes, consequently, they
only apply local information. Although they contain useful
information, it is clear that they do not contain all the hidden
information in the image, making global information necessary for
better inference. In order to apply global information in the model,
the image label classification is calculated through
IET Image Process., 2018, Vol. 12 Iss. 11, pp. 1943-1950
© The Institution of Engineering and Technology 2018
 ∅scene(scene = cls) = σ(tscene) (5) −β, if xi ≠ y j
∅i, j xi, y j = (8)
0, otherwise .
where tscene is the classifier score assigned to the scene type cls and
σ(x) = 1/(1 + exp( − 1.5x)) is a logistic function. where β is learned as the weight of this potential.
In [22], the local and global information are considered as the
two separate terms: data and global terms, respectively. 2.3.3 Unary potential for object detection: As previously stated,
Nevertheless, in this paper, the global feature in (5) is combined we assign a binary variable bl to each detected object. The model is
with the local unary potential for each SP as follows: allowed to accept or reject these detections by defining the fourth
term of energy function as follows:
∅i(xi = cls) = ∅li(xi = cls) + ∅scene(scene type = cls) (6)
E4 = ∑ ∅l bl (9)
where ∅li is the unary local potential for the ith SP xi. In (4), the l ∈ BB
unary potential ∅ j(y j) is computed for each SS by averaging the
TextonBoost pixel potentials inside each SS. Note that the scene where ∅l bl is the unary potential defined over candidate object bl.
types can be generally different from the pixel labels, but they are A logistic function f (x) = 1/(1 + exp( − 1.5x)) is applied in
very similar. Thus, the scene types could be mapped to the pixel calculating ∅l(bl). If this score is high for the lth candidate object,
labels that are semantically most similar to them (e.g. the scene bl is encouraged to be visible. It is expected that the estimation of
type ‘person’ can be mapped to the pixel label ‘body’). the candidate objects to be a strong high-level guide for
segmentation task (e.g. if a ‘bicycle’ object is detected with a high
2.3.2 Pair-wise potential between SP and SS: In order to make detection probability, it is more likely that some pixels take the
the assigned labels between SP and SS consistent, each SS is class ‘bicycle’).
connected to the SPs which lie in that SS using an edge in the
graph and define their covering energy term E3 as follows: 2.3.4 Spatial pairwise potential among the candidate
objects: The fifth term of energy function models the spatial
E3 = ∑ ∅i j(xi, y j) (7) relationship between two objects as follows:
(i, j)
E5 = ∑ ∅rlk(bl, bk) (10)
where ∅i j is the pair-wise potential between ith SP xi and jth SS y j. (l, k) ∈ BB
The Pn potential of [27] is applied in calculating ∅i j expressed as

Fig. 1  Overview of our proposed method for image semantic segmentation. First, the raw input image is fed to three modules. Module 1 partitions the image
into a set of segments in two levels (SPs and SSs), module 2 extracts the candidate objects of the image. The third module computes a probability vector for the
image containing the classifier scores for each scene class. Next, the graph is established based on the outputs of these modules. Thereafter, the outputs of
these modules and the constructed graph are used to compute the terms of the energy function. Finally, the training and testing procedures are performed
using the produced models. In this figure SupPix, SupSeg, and obj stand for super-pixel, super-segment, and object, respectively. Also, E_1 to E_6 are the
terms of the energy function in (2)

IET Image Process., 2018, Vol. 12 Iss. 11, pp. 1943-1950 1945
© The Institution of Engineering and Technology 2018
where BB is the set of all detected objects in the image, and ∅rlk is
defined between two candidate objects which have a semantic
relationship of type r. This will be an effective measure since, i.e. if
it is known that the first object is a table located ‘below’ the second
object, the second one is more likely to be a dish rather than a boot.
The following spatial relationship commonly applied in related
studies [8, 28] is of concern here:
R = {next to, above, below, overlap} (11)
To determine the geometric relationship between two candidate
objects, the bounding box of one candidate object is set first as the
reference bounding box, next, the spatial relationship of the second
object related to the reference object is identified as follows: first,
draw additional bounding boxes similar to the reference object for
each of the mentioned spatial relationships (above: draw on top of
a box; next-to: draw on left and right side of a box, and below:
draw below of a box). Next, check which one of the drawn
bounding boxes has an overlap more than 50% with the reference
bounding box. If a bounding box meets this condition, the model
assigns the corresponding relationship to the given pair of the
boxes and if none of the above relationships are met and the two
original boxes have an overlap more than 50% then, there exist an
‘overlap’ relationship between them and at the end a score is
computed for the assigned relationship. For example, if the
relationship ‘above’ is of concern, its score is calculated as follows:
countAbove
scoreAbove = (12)
∑r countr
where countr is the frequency of relation r occurrence among
objects in training data. The score of the other relationships
mentioned in (11) is computed similarly to (12).
2.3.5 Pair-wise potential between segmentation and
detection: When several visual processes are run in a combined
manner to solve the problem, the objective is to improve the
performance and resolve ambiguities in each visual task by
applying the information from the other tasks. In this paper, the
relation between detection and segmentation tasks as the sixth term
of (2), in a more effective manner, is defined as follows:
E6 = ∑ ∅il xi, blcls (13)
(i, l)
where ∅il is the pair-wise potential between the corresponding
segmentation and detection. Inspired by Yao et al. [17], a new
potential ∅il is defined as follows:
∅il xi, blcls =
1 (14)
Ai p∑
μ(p, compl) ∗ ∅l bl , if xi = cls ∩ bl = 1
∈A i
1 ∅l bl
Ai p∑
μ(p, compl) ∗ , if xi = cls ∩ bl ≠ 1
∈A
γ ‘Nature’ can be associated with the pixel labels ‘Body’, ‘Road’,
i
0, otherwise .
where γ ≥ 0 encourages bl to be detected, and Ai is the set of pixels
in the ith SP. In this potential, the pose and shape of mixture
component compl that has detected object bl is applied. In this
Table 1 Standard train/test split of MSRC-21 dataset [29]
No. of classes No. of training images No. of testing images
21 335 256
1946
manner, the mean mask μ(p, compl) of component compl is
calculated for all pixels p which lie inside the detected bounding
box.
This new potential encourages the labelling of support region to
correspond to the candidate object and to be consistent with the
object shape average. Moreover, this term encourages the candidate
object to be visible (take label 1), if a sufficient number of SPs take
the class of the detector.
3 Experiments
3.1 Experimental setting
The MSRC-21 dataset [29], the popular benchmark for scene
labelling is applied here for evaluation (Table 1). MSRC-21
consists of classes like water and sky which do not have a specific
shape, as well as more shape-defined classes such as car and bird.
These classes are commonly referred to as ‘stuff’ and ‘things’,
respectively. The standard train/test split [29] is applied to the
experiments.
In this newly proposed algorithm, three basic modules are
applied in calculating the energy terms of (2): in the first module,
the algorithm provided by Arbelaez et al. [26] is applied to
construct a hierarchical region tree. In this algorithm, first, the
initial regions are constructed from an oriented contour signal;
next, an agglomerative clustering procedure forms these regions
into a hierarchical region tree. That the generated regions respect
the given boundaries is the advantage of this algorithm. The
threshold parameter is set at 0.08 and 0.16 for the two layers,
which determines the regions’ number and size. The second
module represents an object detector, where the well-known
deformable part-based model of [24] is employed to detect the
candidate image objects. For each detected object, the method of
[24] specifies object class, location, score, and index of root
mixture component. For this module, we need to train a few
detectors with different components and compute the shape masks
comp for each class on the MSRC dataset. We used the trained
models and computed the shape masks provided by Yao et al. [17].
In the third module, to generate the global image classification
information, we apply the method of a standard bag-of-words
spatial pyramid and train a linear one-vs-all SVM classier. The
used features in this module consist of: SIFT features, colour SIFT,
RGB histograms, and colour moment invariants. The output for
each image is a probability vector containing the scores for each
scene type. In training phase, the learning method provided by
Hazan and Urtasun [25] is applied. They present an intuitive
approximation for structured prediction problems using Fenchel
duality based on a local entropy approximation and define a
message-passing algorithm for it. In the experiments, a CRF with l2
regularisation is applied. The inference is run by a message-passing
algorithm [30] which splits the graph-based optimisation program
into several local optimisation problems by introducing additional
Lagrange multipliers. This algorithm guarantees to converge [30].
The basic parameter ε is 1 in this method. In the experiments here,
the scene types similar to that of [17] are applied (Table 2).
Although the scene type can be different as to the pixel label, there
can exist a similarity between them. As observed in Table 2, there
exists a one-to-one mapping between 18 scene types and 18 pixel
labels (white columns). The scene types with no equivalent pixel
labels (grey columns) are mapped to the most similar unaligned
pixel labels. Therefore, the scene types ‘Person’, ‘City’, and
and ‘Sky’, respectively. The γ is chosen empirically, beginning
with a relatively small value, ending with a constant step size
increase at each iteration, that is, γ = 10. The training and testing
processes are run on an Intel i7-4702MQ 2.20 GHz processor.
The labelling models are assessed according to the standard
measures of average-based class accuracy (ACA) and global-based
pixel accuracy (GPA) [14], presented as follows, respectively:
1 N m, m
c∑
ACA = (15)
m
∑ n N m, n
IET Image Process., 2018, Vol. 12 Iss. 11, pp. 1943-1950
© The Institution of Engineering and Technology 2018
 N unary potential, SS unary potential, global feature, object detection
GPA = ∑ ∑ m mn Nmm n
,
,

,
(16)
m
where Nm, n is the number of pixels of category m labelled as class
n, and C is the class total.
The advantages of this approach are revealed when compared to
some of the state-of-the-art models as baselines. We ran some of
the baselines using the published codes and default parameter
settings given by the authors, while the experimental results of
others directly were borrowed for comparison. The baselines are of
three categories: (i) modelling scene labelling through CRFs [17,
31], (ii) modelling scene labelling through CNN-based approaches,
like fully convolutional networks [19], hierarchical deep learning
[32], and (iii) modelling based on fusion model, including a fusion
model where flexible segmentation graph is applied [33].
Our framework has been implemented in MATLAB, and the
source code for this work is available online at https://github.com/
neda-noor-m/Seg_hgm.
3.2 Results and analysis
3.2.1 Quantitative results: The performance of this model on
MSRC-21 dataset and its comparison with the considered baselines
are presented in Table 3. The results clearly reveal that this method
is comparable with state-of-the-art methods. This approach
outperforms other counterparts in terms of ACA measure and
training time. The training time is an important factor, due to the
application of algorithm for larger datasets. In this context, a
reduction in training and testing times leads to a decrease in the
complexity of this newly proposed model (e.g. here 33 components
are compared to 52 components applied in [17]). This proposed
algorithm is comparable with others of its kind in terms of GPA
and testing time criteria.
More detailed quantitative results on the MSRC dataset are
tabulated in Table 4, where, x, y, g, o, c, xy, and xo represent the SP
unary potential, spatial pairwise potential between the candidate
objects, pair-wise potential between SP and SS, and pair-wise
potential between segmentation and detection, respectively. For
example, in the third row ‘x + g’ indicates the combination of SP
information with global information. The contribution of each
potential on the performance over the entire MSRC dataset is
shown in Table 4. Also, we have shown these contributions for a
few images in Fig. 2.
3.2.2 Analysis of the energy terms: In the following, we discuss
the impact of energy terms, which directly affect the performance.
Unary potential for each SP: Calculating unary potentials for
the SPs using local information is usual. Also, many researches
apply the global information to the model by adding a separate
layer to the graph. Here, we combine global information and local
potential of each SP (including local structure information) to
calculate their unary potentials. Global information involves long-
range dependencies in the image and local potential of each SP
includes local structure information. As observed in Table 4,
integration of these information improves the results, since this is
an efficient and easy solution for capturing long-range
dependencies. By adopting this new potential in the model, the
hidden information of the image can easily be considered.
Comparison between the second and third rows indicates that
applying global features in SP potential improves the performance
of the ‘thing’ classes. The comparison between the fourth and fifth
rows confirms this observation.
Pair-wise potential between segmentation and detection: This
new potential encourages support region labelling corresponding to
the candidate object to be consistent with the object shape average.
Moreover, this term encourages the candidate object to be visible if
a sufficient number of SPs are of the class detector. In fact, by
incorporating this potential, we aim to exchange the information
between segmentation and detection tasks. The proposed shape

Table 2 Mapping between the used scene types and pixel labels
Pixel building grass tree cow sheep sky aeroplane water face car bicycle flowers sign bird book chair road cat dog body boot
label
Scene type building grass tree cow sheep nature aeroplane water face car bicycle flowers sign bird book chair city cat dog person boot

Table 3 Performance on MSRC-21 in terms of recognition accuracy and efficiency

Methods ACA, % GPA, % Training, h Testing, s
this method 80.01 86 0.5 4.68
[33] 79.4 87.3 6.8 0.68
[19] 77.9 91.4 N/A 0.2
[34] N/A 81.7 N/A N/A
[31] 78.2 86.8 5.4 16.8
[32] 74.6 80.4 5.8 0.7
[17] 79.3 86.2 2.5 14.06
The bold values indicate which method performs the best in the given category.
The average training times are for the whole training set (per hour), and the average testing times are per image (per second).

Table 4 Quantitative results on the MSRC dataset

Model Building Grass Tree Cow Sheep Sky Aeroplane Water Face
x 72 97 90 78 85 96 84 84 83
x + y + xy 70 96 90 76 81 93 86 87 83
x + g + y + xy 68 64 90 77 83 90 88 85 86
x + y + o + xy + xo 69 97 90 83 87 96 88 84 90
x + g + y + o + xy + xo 66 96 89 84 87 96 90 82 90
full model 66 96 89 85 87 96 90 82 90
[17] 71 98 90 79 86 93 88 86 90
[33] 88 96 93 87 84 92 73 69 93
[31] 81 96 89 74 84 99 84 92 90
[19] — — — — — — — — —
[32] — — — — — — — — —

IET Image Process., 2018, Vol. 12 Iss. 11, pp. 1943-1950 1947
© The Institution of Engineering and Technology 2018
 
Model Car Bicycle Flower Sign Bird Book Chair Road Cat Dog Body Boat ACA
x 81 91 97 69 49 95 59 90 81 53 65 00 76.1
x + y + xy 83 92 97 71 50 95 59 85 83 53 65 00 76.14
x + g + y + xy 83 93 97 72 52 95 63 85 81 53 63 00 76.29
x + y + o + xy + xo 85 94 97 76 51 96 77 89 84 52 70 08 79.25
x + g + y + o + xy + xo 85 95 97 83 42 96 81 84 83 55 74 12 79.57
full model 85 95 97 87 43 97 81 88 84 55 74 12 80.01
[17] 84 94 98 78 54 97 71 89 83 54 68 17 79.18
[33] 80 78 88 92 85 65 84 81 65 67 65 43 79.4
[31] 86 92 98 91 35 95 53 90 62 77 70 12 78.2
[19] — — — — — — — — — — — — 77.9
[32] — — — — — — — — — — — — 77.6
The table shows pixel recall in per cent for different object classes. Full model in the seventh row points to a model with the contribution of all potentials (x + g + y + xy + o + xo + c).
The bold values indicate which method performs the best in the given category.

Fig. 2  Segmentation result on MSRC-21 dataset. The last right panel indicates the mapping between the colours and labels
(a) Image, (b) Ground-truth, (c) x + y, (d) Our model

prior presented in (6) is better than the one proposed in [17],
because here the object detection probability is applied in the
equation. In the experiments, it is observed that applying this
proposed shape prior increases ACA measure from 78.69 to 79.25.
Spatial pairwise potential between the candidate objects: The
geometric relationships (next to, above, below and overlap)
between candidate objects are considered in this energy term. Thus,
the candidate objects whose relationship is unusual are penalised,
e.g. the relationship ‘top’ between ‘cow’ and ‘water’ is strange. We
apply the mentioned method in Section 2.3.4 to extract the
geometric relationships between objects. Next, we calculate the
score of each relation using (12). The details of how these relations
are learned and how the scores are calculated were explained
comprehensively in Section 2.3.4. In MSRC dataset, the most
common geometric relationship is ‘next to’ which is 40% of all the
extracted relations. Also, the ‘above’, ‘below’ and ‘overlap’ consist
of 20, 10, and 30% of all relations, respectively. Adding this
potential to the model (sixth row of Table 4) improves the result as
compared to the model associated with fifth row, in Table 4.
3.2.3 Qualitative results: Some visual examples of our labelling
outputs on MSRC-21 dataset are shown in Fig. 2; some examples
of failure cases are shown in Fig. 3. As observed in Table 4, most
of the failure cases occur in classes ‘bird’ and ‘boat’. One of the
main reasons for causing an error is related to the equal
contribution of local and global information to calculate unary
potential for SP. Assigning unequal weights, which are learned in
the training step, to local and global information contributions can
be effective to decrease this error (Fig. 4).
4 Conclusion
The objective of this study is to improve the segmentation
precision of the images. To accomplish this, a hierarchical CRF
model where segmentation, detection information and scene type
are applied is being proposed. The unary potentials for each SP are
promoted by combining local and global information extracted
from the image. The spatial relationships between objects are
applied to improve shape prior in this model. This proposed
method is evaluated on MSRC dataset, and reveal similar or better

1948 IET Image Process., 2018, Vol. 12 Iss. 11, pp. 1943-1950
© The Institution of Engineering and Technology 2018
Fig. 3  Examples of failure cases. The figure depicts that the bird and boat are classified as a dog and a building, respectively.
(a) Image, (b) Ground-truth, (c) x + y, (d) Our model
Fig. 4  Quantitative results on several sample images in MSRC
pixel-wised labelling accuracy against the competing models where
CRFs or hierarchical models are applied. As a future work on this
method, learning more suitable weights in training step for
contributing local and global information can be considered, which
may improve the results.
5 References
[1] Wang, C., Komodakis, N., Paragios, N.: ‘Markov random field modeling,
inference & learning in computer vision & image understanding: a survey’,
Comput. Vis. Image Underst., 2013, 117, (11), pp. 1610–1627
[2] Sun, J., Zheng, N.-N., Shum, H.-Y.: ‘Stereo matching using belief
propagation’, IEEE Trans. Pattern Anal. Mach. Intell., 2003, 25, (7), pp. 787–
800
[3] Fulkerson, B., Vedaldi, A., Soatto, S., et al.: ‘Class segmentation and object
localization with superpixel neighborhoods’. Proc. 2009 IEEE 12th Int. Conf.
on Computer Vision, Kyoto, Japan, September 2009, pp. 670–677
[4] Mottaghi, R., Fidler, S., Yuille, A., et al.: ‘Human-machine CRFs for
identifying bottlenecks in holistic scene understanding’, arXivPrepr, 2014
[5] Brox, T., Bourdev, L., Maji, S., et al.: ‘Object segmentation by alignment of
poselet activations to image contours’. Proc. CVPR 2011, Colorado Springs,
USA, June 2011, pp. 2225–2232
[6] Gu, C., Lim, J.J., Arbeláez, P., et al.: ‘Recognition using regions’. Proc. 2009
IEEE Conf. on Computer Vision and Pattern Recognition, Miami, USA, June
2009, pp. 1030–1037
[7] Dai, J., He, K., Sun, J.: ‘Instance-aware semantic segmentation via multi-task
network cascades’. Proc. 2016 IEEE Conf. on Computer Vision and Pattern
Recognition, Las Vegas, USA, June 2016, pp. 3150–3158
[8] Sun, M., Kim, B., Kohli, P., et al.: ‘Relating things and stuff via
objectproperty interactions’, IEEE Trans. Pattern Anal. Mach. Intell., 2014,
36, (7), pp. 1370–1383
[9] Eigen, D., Fergus, R.: ‘Predicting depth, surface normals and semantic labels
with a common multi-scale convolutional architecture’. Proc. 2015 IEEE Int.
Conf. on Computer Vision, Santiago, Chile, December 2015, pp. 2650–2658
IET Image Process., 2018, Vol. 12 Iss. 11, pp. 1943-1950
© The Institution of Engineering and Technology 2018
[10] Lafferty, J., McCallum, A., Pereira, F.: ‘Conditional random fields:
probabilistic models for segmenting and labeling sequence data’. Proc. 18th
Int. Conf. on Machine Learning, Williamstown, USA, June 2001, pp. 282–
289
[11] Koltun, V.: ‘Efficient inference in fully connected CRFs with Gaussian edge
potentials’, Adv. Neural Inf. Process. Syst., 2011, 24, pp. 109–117
[12] Torralba, A., Murphy, K.P., Freeman, W.T.: ‘Contextual models for object
detection using boosted random fields’. Adv. Neural Inf. Process. Syst., 2004,
17, (1), pp. 1401–1408
[13] Winn, J., Shotton, J.: ‘The layout consistent random field for recognizing and
segmenting partially occluded objects’. Proc. 2006 IEEE Computer Society
Conf. on Computer Vision and Pattern Recognition, New York, USA, June
2006, pp. 37–44
[14] Ladicky, L., Russell, C., Kohli, P., et al.: ‘Graph cut based inference with co-
occurrence statistics’. Proc. 11th European Conf. on Computer Vision,
Heraklion, Crete, September 2010, pp. 239–253
[15] Russell, C., Kohli, P., Torr, P.H.S., et al.: ‘Associative hierarchical CRFs for
object class image segmentation’. Proc. 2009 IEEE 12th Int. Conf. on
Computer Vision, Kyoto, Japan, September 2009, pp. 739–746
[16] Plath, N., Toussaint, M., Nakajima, S.: ‘Multi-class image segmentation using
conditional random fields and global classification’. Proc. 26th Annual Int.
Conf. on Machine Learning, Montreal, Canada, June 2009, pp. 817–824
[17] Yao, J., Fidler, S., Urtasun, R.: ‘Describing the scene as a whole: joint object
detection, scene classification and semantic segmentation’. Proc. 2012 IEEE
Conf. on Computer Vision and Pattern Recognition, Providence, USA, June
2012, pp. 702–709
[18] Shuai, B., Wang, G., Zuo, Z., et al.: ‘Integrating parametric and non-
parametric models for scene labeling’. Proc. 2015 IEEE Conf. on Computer
Vision and Pattern Recognition, Boston, USA, June 2015, pp. 4249–4258
[19] Sharma, A., Tuzel, O., Jacobs, D.W.: ‘Deep hierarchical parsing for semantic
segmentation’. Proc. 2015 IEEE Conf. on Computer Vision and Pattern
Recognition, Boston, USA, June 2015, pp. 530–538
[20] Chen, L.-C., Papandreou, G., Kokkinos, I., et al.: ‘Semantic image
segmentation with deep convolutional nets and fully connected CRFs’, arXiv
Prepr, 2014
1949
[21] Zheng, S., Jayasumana, S., Romera-Paredes, B., et al.: ‘Conditional random
fields as recurrent neural networks’. Proc. 2015 IEEE Int. Conf. on Computer
Vision, Santiago, Chile, December 2015, pp. 1529–1537
[22] Lucchi, A., Li, Y., Boix, X., et al.: ‘Are spatial and global constraints really
necessary for segmentation?’. Proc. 2011 Int. Conf. on Computer Vision,
Barcelona, Spain, November 2011, pp. 9–16
[23] Wang, H., Koller, D.: ‘Multi-level inference by relaxed dual decomposition
for human pose segmentation’. Proc. CVPR 2011, Providence, USA, June
2011, pp. 2433–2440
[24] Felzenszwalb, P.F., Girshick, R.B., McAllester, D., et al.: ‘Object detection
with discriminatively trained part-based models’, IEEE Trans. Pattern Anal.
Mach. Intell., 2010, 32, (9), pp. 1627–1645
[25] Hazan, T., Urtasun, R.: ‘A primal-dual message-passing algorithm for
approximated large scale structured prediction’. Adv. Neural Inf. Process.
Syst., 2010, 158, (2), pp. 838–846
[26] Arbelaez, P., Maire, M., Fowlkes, C., et al.: ‘Contour detection and
hierarchical image segmentation’, IEEE Trans. Pattern Anal. Mach. Intell.,
2011, 33, (5), pp. 898–916
[27] Kohli, P., Kumar, M.P., Torr, P.H.S.: ‘Solving energies with higher order
cliques’, 2007
1950
[28] Anand, A., Koppula, H.S., Joachims, T., et al.: ‘Contextually guided semantic
labeling and search for three-dimensional point clouds’, Int. J. Robot. Res.,
2012, 32, (1), pp. 19–34
[29] Shotton, J., Johnson, M., Cipolla, R.: ‘Semantic Texton forests for image
categorization and segmentation’. Proc. 2008 IEEE Conf. on Computer Vision
and Pattern Recognition, Anchorage, USA, June 2008, pp. 1–8
[30] Schwing, A., Hazan, T., Pollefeys, M., et al.: ‘Distributed message passing for
large scale graphical models’. Proc. CVPR 2011, Colorado Springs, USA,
June 2011, pp. 1833–1840
[31] Ladickỳ, L., Russell, C., Kohli, P., et al.: ‘Associative hierarchical random
fields’, IEEE Trans. Pattern Anal. Mach. Intell., 2014, 36, (6), pp. 1056–1077
[32] Farabet, C., Couprie, C., Najman, L., et al.: ‘Learning hierarchical features for
scene labeling’, IEEE Trans. Pattern Anal. Mach. Intell., 2013, 35, (8), pp.
1915–1929
[33] Zhou, Q., Zheng, B., Zhu, W., et al.: ‘Multi-scale context for scene labeling
via flexible segmentation graph’, Pattern Recognit.., 2016, 59, pp. 312–324
[34] Saito, M., Okatani, T.: ‘Transformation of Markov random fields for marginal
distribution estimation’. Proc. 2015 IEEE Conf. on Computer Vision and
Pattern Recognition, Boston, USA, June 2015, pp. 797–805
IET Image Process., 2018, Vol. 12 Iss. 11, pp. 1943-1950
© The Institution of Engineering and Technology 2018