Embedded System Design for Visual Scene
Classification
Sumair Aziz*, Zeshan Kareem, Muhammad Umar Khan, Muhammad Atif Imtiaz
Department of Electronic Engineering,
University of Engineering and Technology Taxila, Pakistan
Email: *sumair.aziz@uettaxila.edu.pk, zeshanali6454@gmail.com, umar.khan@uettaxila.edu.pk, atif.imtiaz@uettaxila.edu.pk
Abstract— Computer vision and robotics community is
experiencing growing interest in visual scene classification
due to availability of low cost and compact visual sensing
devices. This paper presents framework aimed at embedded
system design for visual scene classification. In the proposed
framework we used data fusion of local and global
descriptors as feature vectors for scene classification. We
construct feature vector by integrating Local Quinary
Patterns (LQP), Bag of Visual Words (BoW) and Histogram
of Oriented Gradients (HOG). For classification multiclass
Support Vector Machines (SVM) is used. Experiments are
performed on publicly available MIT indoor scene
classification database. Comparison of our approach with
other methods show that our approach is efficient in terms of
overall accuracy.
Keywords—Local quinary patterns, Bag of visual words,
histogram of gradients, SVM, scene classification
I. INTRODUCTION
Scene understanding is one of the most important problem
in computer vision. Although remarkable progress has
been achieved in the past decades, general-purpose scene
understanding is still considered to be very challenging
[1]. One of the incredible skills of the human visual system
is how accurately it can identify and understand the
complex visual world. The quick extraction of semantic
understanding of a scene requires a few hundred
milliseconds to be consolidated into the memory [2]. The
purpose of the scene recognition is to discover the overall
scene category by putting stress on understanding its
global characteristics [3, 4]. Visual scene classification and
understanding is to examine a scene by considering the
geometric and semantic context of its contents and to find
the core relationships between them. In scene recognition
there is a need for a model that classifies the scene,
identifies and parts each component of the object, as well
as explains the image with a list of labels [5]. Scene
recognition can be very useful in the progressive driver
support system, and generally for robotics. Scene
understanding can be applied to a multi-sensor system
containing a stereo camera and a LIDAR, which are to be
978-1-5386-7266-2/18/$31.00 ©2018 IEEE
calibrated. It also involves the road detection, pedestrian
detection, traffic signs recognition and so on [6].
There are many challenges that affect the
computer vision-based algorithms designed for scene
classification. Variation in lighting conditions in which
image is captured, affect the image quality and
thresholding parameters used for image segmentation.
Foggy and hazy conditions also affect the image quality.
Viewpoint from which image is captured by the camera is
the most important aspect for recognizing the scene
accurately and it continuously remains to be the problem
for researchers.
II. LITERATURE REVIEW
Solution for urban scene classification based on fusion
framework that works on parts of over-segmented images
based on Dempster-Shafer theory is presented in [6].
Three types of sensors namely camera, LIDAR used in this
approach. KITTI [7] dataset is used for results validation.
Eextended multi-structure local binary pattern based
feature extraction approach is proposed for high-resolution
image classification in [8]. Experimental results show that
the used method work efficiently in capturing spatial
pattern and local contrast, consistently outperforming more
than a few state-of-the-art classification algorithms. In [9]
authors provided an overview of the several technologies
that have been established in the recent years to help the
visually impaired in recognizing the general objects in an
indoor environment on the methods based on computer
vision. Since the only piece of the visual data that system
extract, store and compare for each object is much less
than that of non-tag based systems, which deal with much
more detailed information, such as color, size, shape, etc.
for each object.
The combination of audio and video features, as well as
combination of different type of classifiers are allocated to
attain frame by frame classification of personal video
recordings into semantically meaningful categories such as
data about the environment (indoor, outdoor, etc.), the
presence or the absence of the people and their activities
like sports or partying [10]. A system designed that learns
which classifiers and parameters are suitable for this task.
739
 Figure 1: Proposed System Block Diagram

Two classifiers are used, GMM classifier for audio
features and SVM classifier for image features. Classifier
combination schema is implemented using the meta-
classification. Meta-classification gets the better results
over the different approaches performed, resulting a
promising f-measure higher than 57% in average for all
categories and higher than 73% over the diverse
categories.
In [11] scene classification problem is addressed
that by using histogram features extracted from the
saliency maps to predict the existence of interesting objects
in images and to quickly prune uninteresting images. For
the validation of this approach, a database is constructed
that consists of 1000 background and object images
captured in the environment in which robot is working.
Results are evaluated in terms of overall performance of
the proposed approach for the different saliency maps
using Precision, Recall and F1-Measure. Recently,
Convolutional neural networks (CNN) [12] and transfer
learning is also employed for scene recognition [10].
Authors performed experiments on MIT indoor scene
recognition dataset and compared differences in properties
of indoor and outdoor scene recognition. In [13] scene
classification is performed using convolutional neural
networks [12] that works on the technical approaches
which are Baseline, ResNet [14], Dropout and L2
Regularization, Batch Normalization, Loss and Optimizer
and Saliency Maps. Experiments were performed on
organized as follows. Section III describes the proposed
visual classification system. Experimental setup and results
are discussed in Section IV. Finally, conclusions of our
work are presented in Section V.
III. PROPOSED VISUAL CLASSIFICATION SYSTEM
General architecture of the visual classification system is
shown in Fig. 1. In the first step, color image having Red,
Green and Blue channels is captured from camera which is
passed to the embedded processor (Raspberry Pi) [21] for
further processing. Local Quinary Patterns (LQP), Bag of
visual words (BoVW) and Histogram of Gradients (HOG)
features are extracted from image. Finally, scene
classification is performed using multiclass Support vector
machines (SVM) classifier.
A. Feature Extraction using Local Quniary Patterns
Let the image captured from camera be I (i, j ) having
red, blue and green channels. LQP features are computed
independently for each channel. For a given image I ,
LQP features are extracted from each window of 3× 3
size. Finally, histograms bins are created for all computed
LQP features in order to reduce the data dimensionality
and increase computation speed.
The local quinary values were obtained using Eq. (1),
+2 I ≥L c 2

+1 L ≤ (I ) > L
Places2 dataset [15]. In [16] novel method is presented for 
1 c 2

assisted living environment and human are tracked by x( I , L , L ) =  0 − L ≥ ( I ) < L

c 1 2 1 c 1
(1)
video cameras and arrays of microphones for audio −1 L < ( I ) ≤ − L

2 c 1
recording. Results are reported for audio monitoring and
video tracking individually. −2 I ≤ −L c 2

In our work we use feature fusion of multiple

descriptors that best describes the image for scene
55 21 12 1 10 0 0 1
classification. Local Quinary Patterns (LQP) [17] based
34 7 21 89 45 1 0
feature descriptor was originally proposed for content-
17 56 22 25 98 1 0 0
based image retrieval problems. LQP effectively captures
81 31 19 14 34
the local neighborhood information present in texture 0 0 0
61 88 49 0 43
images. The bag-of-visual words (BoVW) technique is 0 1
50

mainly inspired from problem of text document analysis 0 0 0 8 4 2 1

[18]. BoVW computes the feature vector by capturing the 16 1
type of important interest points in an image. Histogram of -2 0 +2
32 64 128
64
0 0 0
Oriented Gradients (HOG) [19, 20] operator computes the +2 +1
0 0
gradients present in image patches. We used feature fusion +2 -1 -2 136
0 1 0
approach for scene image classification by combining
strong features from these descriptors. Multiple categories 1 0 0

of indoor scenes are classified using multiclass Support 0 0

Vector Machines (SVM) classifier. Rest of this paper is 0 0 1

Figure 2: Local Quinary Pattern Feature Descriptor

740
Where L1 and L 2 are user defined upper and lower
threshold parameters. I c is the center pixel of 3 × 3
window in shown in Fig. 2. Finally, whole image is
represented by histogram of all features computed from
each window.
B. Bag of Visual Words (BoVW)
The bag-of-words (BoW) methodology was basically
proposed for problem of text document analysis, later on it
was adopted for image processing and computer vision
applications [18]. A visual analogue of a word is used in
BoW model for image analysis, that is based on clustering
low-level visual features of local regions such as color or
texture and finally feature vector of desired length (words)
is constructed as a result.
BoVW feature extraction scheme is shown graphically in
Fig. 3 and it involve following processing steps:
(i) Detect regions or points of interest
(ii) Compute local features descriptors over those
regions
(iii) Quantize the feature descriptors into words to
form the visual vocabulary, and
(iv) Compute the appearance of specific word in
image in the vocabulary for creating the BoW
feature
Given a training dataset F consists of k images represented
by T = f 1, f 2, f 3,..., fk ,where f is extracted SURF
features, k-means clustering algorithm group F based on
fixed number of visual categories or words V represented
by V = v1, v 2, v 3,..., vc , where c is the cluster number. Data
is summarized as vector of N × C occurrence table of
counts Cij = k ( vi, fj ) ,where n ( vi, fj ) denotes how frequent
the word vi appeared in an image fj .
C. Histogram of Oriented Gradient (HOG)
HOG feature descriptor is very popular in computer vision
domain and its mainly used for human detection [19, 20].
Images K-means
Feature Vectors
V3 V1
V2
Bag-of-Words
Figure 3: Feature Extraction through Bag of Visual Words
It also proved effective for object detection [22]. To extract
HOG features from input scene image, first image is
divided into overlapping blocks. Afterwards, orientations
of gradients are computed from each block, followed by
quantization of features into histogram bins. Each bin has
its own orientation range. Finally, features from all image
blocks are concatenated to form a feature vector that
represents whole scene image.
D. Classification - Support Vector Machines (SVM)
Scene image classification is performed using multiclass
SVM. SVM finds a hyperplane that separates D-
dimensional data into its two categories [23]. It is a
discriminative model for classification that principally
depends on two basic assumptions. First, complex
classification problems can be classified through simple
linear discriminative functions by transforming data in to a
high-dimension space. Second, training patterns used in
SVMs are only those that are close to the decision surface
supposing they provide the most relevant information for
classification. Support vector machines were proposed as
binary classifiers. However, in real scenarios, data is to be
classified into multiple classes. This is done by using
multiclass SVM. Either One Against One(OAO) or One
Against All (OAA) approach can be used [24]. For image
scene classification, features extracted in previous stage
are used to train multiclass SVM (OAO) classifier.
E. Proposed Algorithm
Input: scene image
Output: classify scene
1. Input image from attached camera
2. Compute LQP, BoW and HOG features and
construct histogram
3. Concatenate all three feature histograms to form a
single feature vector.
4. Perform SVM multiclass classification to classify
the image.
IV. EXPERIMENTS AND RESULTS
We evaluate our proposed method for scene classification
on MIT indoor 67 dataset [25]. It consists of total 67
indoor scene classes, out of which 15 are selected for
evaluation purpose. The images in dataset are best
characterized by the objects they contain and their global
layout. MIT indoor dataset is a challenging dataset that
illustrates the significance of using semantically
meaningful representation for complex scene cluttered
with objects. Figure 4 shows selected images from 15
categories used for evaluation.
We tested our algorithm using MATLAB 2016 on i3
workstation having 8 GB RAM. We also performed
experimentation on Raspberry Pi 3 embedded computer
[21] using OpenCV [26]. We used 5-fold cross validation
for experimental evaluation and results are discussed in
terms of accuracy, precision and recall.
 TP + TN 
Accuracy =   ×100 (2)
 TP + TN + FP + FN 
741
 TABLE I. RESULTS OF INDIVIDUAL CLASSES IN TERMS OF RECALL
AND PRECISION

True Positive Rates Positive Predicted

Classes
(Recall) Value (Precision)
Bedroom 0.93 0.72
Bathroom 0.77 0.98
Child room 0.67 0.90
Closet 0.81 0.92
Clothing store 0.79 0.87
Computer room 0.74 0.86
Dining room 0.80 0.99
Game room 0.71 0.97
Garage 0.97 1
Kitchen 0.76 0.92
Living room 0.67 0.99
Figure 4: Scene Images from MIT Indoor Database Lobby 0.78 0.64

 TP  computational speed and discriminative performance that

Precision=   × 100 (3)
 TP + FP 
 TP 
Recall =   × 100 (4)
 TP + FN 
The results in terms of accuracy of applying our proposed
integrated framework that consists of LQP, HOG and BoW
feature sets is presented in Fig. 5. Our proposed approach
achieved overall 81.6% accuracy. The difference between
texture of a living room image and bedroom is less, so the
it become very difficult to distinguish. This results in
predication inaccuracy i.e. 18% living room scene images
predicted as bedroom. We also compared our results with
widely used local feature descriptors, Local Binary
Patterns [27], Local Ternary Patterns [28] and Local Tri-
directional patterns [29]. Comparison results are shown in
Fig. 6. LQP features are very effective in terms of
makes it suitable choice for embedded systems. While
HOG is successful in terms of capturing gradient
information about the edges present in image. BoVW
technique is used to capture the global information
available in image. It computes the interest points in whole
image and cluster them in groups. Feature fusion of global
and local feature descriptors gives us better result as
compared to earlier work. Results of individual classes in
terms of recall and precision are presented in table 1.
V. CONCLUSIONS
In this paper, we presented an integrated feature fusion
approach for indoor scene classification. Our approach
fuses useful features from local and global descriptors, i.e.
LQP, BoVW and HOG features are fused to form a strong
representation of scene image. LQP and HOG are robust to
noise and extract local neighborhood information from
image patches while BoVW represents the global interest
points present in an image. Multiclass SVM is applied for
classification on MIT indoor scene dataset. In future, we
will implement Field Programmable Gate Arrays (FPGA)

Figure 5: Confusion matrix showing accuracy of individual classes

Figure 6: Comparison of proposed framework with other methods

742
based hardware accelerator for LQP and HOG feature
descriptors to enhance the overall system performance in
terms of throughput which is critical requirement of high
performance and real time embedded systems.
REFERENCES
[1] S. Song, S. P. Lichtenberg, and J. Xiao, "Sun rgb-d: A rgb-d
scene understanding benchmark suite," in Proceedings of the
IEEE conference on computer vision and pattern recognition,
2015, pp. 567-576.
[2] L. Fei-Fei, A. Iyer, C. Koch, and P. Perona, "What do we
perceive in a glance of a real-world scene?," Journal of
vision, vol. 7, pp. 10-10, 2007.
[3] A. Oliva and A. Torralba, "Modeling the shape of the scene:
A holistic representation of the spatial envelope,"
International journal of computer vision, vol. 42, pp. 145-
175, 2001.
[4] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva,
"Learning deep features for scene recognition using places
database," in Advances in neural information processing
systems, 2014, pp. 487-495.
[5] L.-J. Li, R. Socher, and L. Fei-Fei, "Towards total scene
understanding: Classification, annotation and segmentation in
an automatic framework," in Computer Vision and Pattern
Recognition, 2009. CVPR 2009. IEEE Conference on, 2009,
pp. 2036-2043.
[6] P. Xu, F. Davoine, J.-B. Bordes, H. Zhao, and T. Denoeux,
"Information fusion on oversegmented images: An
application for urban scene understanding," in Thirteenth
IAPR International Conference on Machine Vision
Applications, 2013, pp. 189-193.
[7] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, "Vision meets
robotics: The KITTI dataset," The International Journal of
Robotics Research, vol. 32, pp. 1231-1237, 2013.
[8] X. Bian, C. Chen, Q. Du, and Y. Sheng, "Extended multi-
structure local binary pattern for high-resolution image scene
classification," in Geoscience and Remote Sensing
Symposium (IGARSS), 2016 IEEE International, 2016, pp.
5134-5137.
[9] R. Jafri, S. A. Ali, H. R. Arabnia, and S. Fatima, "Computer
vision-based object recognition for the visually impaired in
an indoors environment: a survey," The Visual Computer,
vol. 30, pp. 1197-1222, 2014.
[10] A. M. Barbancho, L. J. Tardón, J. López-Carrasco, J. Eggink,
and I. Barbancho, "Automatic classification of personal video
recordings based on audiovisual features," Knowledge-Based
Systems, vol. 89, pp. 218-227, 2015.
[11] C. Scharfenberger, S. L. Waslander, J. S. Zelek, and D. A.
Clausi, "Existence detection of objects in images for robot
vision using saliency histogram features," in Computer and
Robot Vision (CRV), 2013 International Conference on,
2013, pp. 75-82.
[12] W. Liu, Z. Wang, X. Liu, N. Zeng, Y. Liu, and F. E. Alsaadi,
"A survey of deep neural network architectures and their
applications," Neurocomputing, vol. 234, pp. 11-26, 2017.
[13] J. King, V. Kishore, and F. Ranalli, "Scene classification with
Convolutional Neural Networks."
[14] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning
for image recognition," in Proceedings of the IEEE
conference on computer vision and pattern recognition, 2016,
pp. 770-778.
[15] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba,
"Places: A 10 million image database for scene recognition,"
IEEE transactions on pattern analysis and machine
intelligence, 2017.
[16] A. Karpov, L. Akarun, H. Yalçın, A. Ronzhin, B. E. Demiröz,
A. Çoban, et al., "Audio-visual signal processing in a
multimodal assisted living environment," in Fifteenth Annual
Conference of the International Speech Communication
Association, 2014.
[17] S. K. Vipparthi and S. K. Nagar, "Color directional local
quinary patterns for content based indexing and retrieval,"
Human-centric Computing and Information Sciences, vol. 4,
p. 6, 2014.
[18] C.-F. Tsai, "Bag-of-words representation in image annotation:
A review," ISRN Artificial Intelligence, vol. 2012, 2012.
[19] N. Dalal and B. Triggs, "Histograms of oriented gradients for
human detection," in Computer Vision and Pattern
Recognition, 2005. CVPR 2005. IEEE Computer Society
Conference on, 2005, pp. 886-893.
[20] Q. Zhu, M.-C. Yeh, K.-T. Cheng, and S. Avidan, "Fast
human detection using a cascade of histograms of oriented
gradients," in Computer Vision and Pattern Recognition,
2006 IEEE Computer Society Conference on, 2006, pp. 1491-
1498.
[21] V. Menezes, V. Patchava, and M. S. D. Gupta, "Surveillance
and monitoring system using Raspberry Pi and SimpleCV,"
in Green Computing and Internet of Things (ICGCIoT), 2015
International Conference on, 2015, pp. 1276-1278.
[22] W. Zhang, G. Zelinsky, and D. Samaras, "Real-time accurate
object detection using multiple resolutions," in Computer
Vision, 2007. ICCV 2007. IEEE 11th International
Conference on, 2007, pp. 1-8.
[23] S. Amarappa and S. Sathyanarayana, "Data classification
using Support vector Machine (SVM), a simplified
approach," International Journal of Electronics and
Computer Science Engineering. ISSN-2277-1956, 2014.
[24] J. Milgram, M. Cheriet, and R. Sabourin, "“One against one”
or “one against all”: Which one is better for handwriting
recognition with SVMs?," in Tenth international workshop
on frontiers in handwriting recognition, 2006.
[25] A. Quattoni and A. Torralba, "Recognizing indoor scenes," in
Computer Vision and Pattern Recognition, 2009. CVPR
2009. IEEE Conference on, 2009, pp. 413-420.
[26] K. Pulli, A. Baksheev, K. Kornyakov, and V. Eruhimov,
"Real-time computer vision with OpenCV," Communications
of the ACM, vol. 55, pp. 61-69, 2012.
[27] M. Pietikäinen, A. Hadid, G. Zhao, and T. Ahonen, Computer
vision using local binary patterns vol. 40: Springer Science &
Business Media, 2011.
[28] X. Tan and B. Triggs, "Enhanced local texture feature sets for
face recognition under difficult lighting conditions," in
International Workshop on Analysis and Modeling of Faces
and Gestures, 2007, pp. 168-182.
[29] M. Verma and B. Raman, "Local tri-directional patterns: A
new texture feature descriptor for image retrieval," Digital
Signal Processing, vol. 51, pp. 62-72, 2016.
743