APPROACH TO THE EXTRACTION OF DESIGN FEATURES OF

INTERIOR DESIGN ELEMENTS USING IMAGE RECOGNITION

TECHNIQUE

JIN SUNG KIM1 , JAE YEOL SONG2 and JIN KOOK LEE3
1,2,3
Department of Interior Architecture & Built Environment, Yonsei
University, Republic of Korea
1,2
{wlstjd1320|songjy92}@gmail.com 3 leejinkook@yonsei.ac.kr

Abstract. This paper aims to propose deep learning-based approach to

the auto-recognition of their design features of interior design elements
using given digital images. The recently image recognition technique
using convolutional neural networks has shown great success in the
various field of research and industry. The open-source frameworks and
pre-trained image recognition models supporting image recognition task
enable us to easily retrain the models to apply them on any domain. This
paper describes how to apply such techniques on interior design process
and depicts some demonstration results in that approaches. Furniture
that is one of the most common interior design elements has sub-feature
including implicit design features, such as style, shape, function as well
as explicit properties, such as component, materials, and size. This paper
shows to retrain the model to extract some of the features for efficiently
managing and utilizing such design information. The target element is
chair and the target design features are limited to functional features,
materials, seating capacity and design style. Total 3933 chair images
dataset and 6 retrained image recognition models were utilized for
retraining. Through the combination of those multiple models, inference
demonstration also has been described.

Keywords. Deep learning; Image recognition; Interior design

elements; Design feature; Chair.

1. Introduction
1.1. RESEARCH OBJECTIVE
The artificial intelligence (AI) and the machine learning have been applied in many
fields of research and industry. Especially, deep artificial neural networks have
achieved great success on many contests about pattern recognition (Schmidhuber
2015). The advent of deep convolutional neural networks(CNNs) and the powerful
graphics processing unit(GPU) are an important role of successful large-scale
image recognition. General-purpose image recognition model like Alexnet
(Krizhevsky et al. 2012) and GoogLeNet(a.k.a Inception) (Szegedy et al. 2015)
were proposed and opened to people to utilize them for the various domains.

T. Fukuda, W. Huang, P. Janssen, K. Crolla, S. Alhadidi (eds.), Learning, Adapting and Prototyping,
Proceedings of the 23rd International Conference of the Association for Computer-Aided Architectural
Design Research in Asia (CAADRIA) 2018, Volume 2, 287-296. © 2018 and published by the Association
for Computer-Aided Architectural Design Research in Asia (CAADRIA) in Hong Kong.
288 J.S. KIM, J.Y. SONG AND J.K. LEE

The framework for deep learning such as Tensorflow (Abadi et al. 2016) and
its related library also was provided as open source and have helped people in
various domains to retrain image recognition model. Furthermore, deep learning
model could catch some object on image or video. Through the object detection
approach like Faster R-CNN (Ren et al. 2015) and YOLO (Redmon et al. 2016), it
is possible to analyze a scene and extract object and its information like the label.
This could be very useful for the various domain where image processing is an
important task.
To go beyond the general purpose of image recognition and object detection,
domain knowledge-based approach is needed. In terms of interior design, furniture
is one of the most common interior design elements in the interior design process
or for designer and client. It is usually carried out to analyze each design element
and its feature from the indoor scene or product image for various activities related
to interior design. In this regard, it is more meaningful to recognize design features
of each element such as function, materials, style and etc. than to classify whether
it is chair or table. Sub-feature of furniture includes implicit design features,
such as design type, style, shape and usage, and also explicit properties, such
as component, color, materials, and size. Interior design experts who trained
with sufficient and good resource could successfully recognize such things. For
successful automated extraction of such things, good training resources are needed
to learn what features are and train to recognize and classify. The ultimate purpose
of this research is to develop a system or applications to efficiently manage and
utilize design information about the interior design and its elements on the digital
image. The applications may include database system managing interior images
with interior design information, tools for analysis on interior design and design
recommendation for interior designer and client. The objective of this paper is to
look for possibilities of deep learning-based approach to automated recognition
design features of the interior design element. We developed image dataset for
training some design features of the chair, retrained some models using that
dataset and pre-trained model and showed the results of testing automate extraction
through application demonstration.

1.2. RESEARCH SCOPE AND PROCESS

This research ultimately aims to develop automated extraction of whole interior
design element and its design feature from the image and utilize it for interior
design. Among the needed researches for that, detecting interior element is already
possible and has shown high accuracy. As the basic research for next step, this
paper describes to simply survey necessary technique and approach and utilize
them to test to retrain deep learning model to recognize interior design element
and its design features (e.g. figure 1). We surveyed on available techniques such
as CNNs and image recognition library and used them for our purpose. For the
demonstration, target interior design element was limited to seating things like
chair, sofa, and stool. Design features were limited to some features that could be
visually identified on image. Image data for training was collected from Google
image search and relevant professional website like the Houzz, one of the most
popular interior e-commerce website. The process of this paper is summarized as
 APPROACH TO THE EXTRACTION OF DESIGN FEATURES OF 289
INTERIOR DESIGN ELEMENTS USING IMAGE RECOGNITION
TECHNIQUE

follows:
1. Training Automated extraction of design features of chair: Furniture and its
feature are identified and interesting criteria features which can be checked on
visual. Next, image dataset has been built according to features from various
web-based source. Multiple image recognition models are retrained for each
objective.
2. Demonstration of automate chair and its some design features using re-training
models: to collaborate multiple automized extraction models, the various design
features are inferred from given image.

Figure 1. Scope of work for automated extraction of interior design elements and their design
features.

2. Background
Fully reviewing deep learning and convolutional neural networks is out of the
scope of this paper. In this paper, the survey on important background and the
way to use such approach and tools for interior design domain are covered.

2.1. DEEP LEARNING-BASED IMAGE RECOGNITION

Deep convolutional neural networks (CNNs) is the method to process image
with pixel data. It has lots of layer such as convolution filter, pooling filter
and fully connected layer (e.g. figure 2). Passing through these layers made
from given image, the weighs are trained for optimizing performance of image
feature recognition task (LeCun et al. 2015) After success of “Supervision” of
Kreizhevsky et al. (2012) in ILSVRC2012 image classification challenge, interest
in CNNs were dramatically increased and new model and application approach
has been proposed. Recently, the outstanding result of general purpose image
classification and object detection has been shown. CNNs have been applied to
290 J.S. KIM, J.Y. SONG AND J.K. LEE

various problems with high performance such as classifying image photography

style (Karayev et al. 2013) and recognizing faces (Taigman et al. 2014). The
Inception-v3 (Szegedy et al. 2016) that was utilized in this paper is recent
successful image recognition model. This model and retrain package are opened
to people by Google, so it greatly helps for us to use it for the purpose. The
Inception-v3 has more convolution and pooling layer than other models, but its
accuracy was proved and it is easy to retrain with its library and transfer learning.

Figure 2. The structure of the typical convolutional neural networks for image recognition
(LeCun and Bengio 1995).

2.2. APPLICATIONS OF IMAGE RECOGNITION FOR INTERIOR DESIGN

2.2.1. Learning visual similarity of interior design element and recommendation
system.
Bell and Bala (2015) proposed an approach to learning visual similarity of interior
design product and image retrieval. They developed a visual search mechanism to
match product image in an indoor scene with the clear well-light image on white
background. They showed CNNs can successfully extract visual features, training
visual similarity, such as shape, color, etc. and retrieve some image that visually
very similar to inputting image. Instead of classifying and extracting the features of
interior products, it was carried out to recommend similar products. This is useful
to search some furniture of interest, but it is manually processed to recognize of
interesting information of interior design product.

2.2.2. Automate classification of furniture design style

Hu et al. (2017) proposed an approach to automate classification of furniture image
according to some design style. They utilize a series of machine learning methods
for exploring the meaningful features of furniture styles, in terms of the aesthetic
role of furniture in indoor design. Going beyond the general perspective, various
details of furniture such as the color, line shape, materials, size and so on are
considered. They developed Training image database limited to 16 Categories of
furniture style which are most popular. For this, handcrafted classification and
learning-based classification was used. To increase accuracy, combination of top
two model showed final accuracy of 70.2%.
These two researches showed potentials of utilization of deep learning-based
approach for interior design elements, such as learning similarity of product
 APPROACH TO THE EXTRACTION OF DESIGN FEATURES OF 291
INTERIOR DESIGN ELEMENTS USING IMAGE RECOGNITION
TECHNIQUE

images or some furniture design styles. As a little different application of such

image recognition technique, this paper proposes to approach to extraction of more
fine-grained design feature.

3. Automized extraction of design features of interior design element

This chapter describes definition features of furniture that can be visually identified
and approach to training automated extraction model using deep learning-based
CNN. As a basic research, target indoor element is limited to the chair and its
feature. It is introduced to prepare training image data according to the feature
categories and train a series of automated extraction model. Figure 3 shows the
framework of automated extraction of furniture and its feature.

Figure 3. The framework of combinations of multiple models for the design features extraction.

3.1. FURNITURE AND ITS DESIGN FEATURES

Furniture is the main one of the interior design element. As the product, it has
some features that can be identified by designers, customers, and users. Furniture
often has similar structural, technological, functional, operational and aesthetic
features, so dividing distinctively and clearly, them is difficult (Smardzewski
2015). Despite its difficulty, Classification of furniture could help to understand
it and even design, purchase and use it. Accordingly, the various researches
were carried out to identify and classify features of them. McDonagh et al.
(2002) categorized the features of product design into practical applications of the
product, human factors engineering, and aesthetic style. Lee (2010) categorized
the of the furniture product designs into a functional design feature, human factor
design feature and external design feature. In summary, features of furniture are
categorized into three features: 1) related to function such as usage, supporting
some action 2) related to feeling like safety, comfort and etc. 3) related to
appearances such as style, color, and materials.
The first feature is related to supporting some human action or position, so it is
the core feature. This feature is determined by whether the furniture has specific
functional components. These kinds of components can be easily identified
visually. As an example of a chair, legs, sheet, backrest, armrest, headrest, and
292 J.S. KIM, J.Y. SONG AND J.K. LEE

frame could be included in the components. This feature is very explicitly shown.
On the other hand, the other features were defined in various perspective. This is
the because those non-functional features are subjective and implicit. The feature
related to human‘s feeling, for example, is affected by one‘s characteristic and
preference. It is also difficult to distinguish clearly style that is the main one of
the most important feature of appearance. However, these features obviously help
people to understand the furniture.
Hence, this research aims to check whether a computer could learn these
features from furniture images and then recognize them. Automated extraction
of them helps many people like a designer, customer, and user to make a decision
on interior design development. As the basic study for that, the target is limited
to chair and its features. The categories of target features of the chair that can be
identified on the image are as follows (e.g. figure 4):
1. Functional features that can be check by additional components of chair such as
armrest, backrest, headrest
2. Materials that are limited to five materials commonly used, such as leather, fabric,
wood, plastic, and velvet
3. Seating capacity that can be checked by the number of seats
4. Design style that are limited most popular, such as classic, eclectic, modern,
Scandinavian style

Figure 4. The categories and their detailed classes of target design feature for training
automized extraction.

3.2. PREPARATION TRAINING DATA

Training the automated extraction model requires specific dataset. Through
category of features proposed in the previous section, images were collected
from Google image search and www.Houzz.com that one of the biggest interior
and furniture e-commerce website. To build each dataset, we utilized automate
image scraper API and easily download on the website. As the unsupervised
learning method, after collecting data with the keywords of features suggested
in the previous session, they were manually classified again by interior design
experts. It is easy to classify them according to functional features, because arm,
back, head component is very visually explicit. we selected 5 detailed classes of
materials that common and well-identified. Classification of seating capacity was
 APPROACH TO THE EXTRACTION OF DESIGN FEATURES OF 293
INTERIOR DESIGN ELEMENTS USING IMAGE RECOGNITION
TECHNIQUE

processed according to count of sheet division and length up to four in this paper.
In case of design style, we selected only four different classes. Classification of
design style is even controversial because its definition is implicit and may vary
from person to person. Hence, we collected and classified not some chairs of
modern style but chairs that can seem modern style according to criteria of Miller
(2010). Total 3933 images were prepared for training and the number of training
images per detailed classes of each category is shown in figure 5.

Figure 5. The number of training images per detailed classes of each category .

3.3. RE-TRAINING MULTIPLE IMAGE RECOGNITION MODELS

This section suggests an approach to automated extraction of chair features using a
combination of deep learning-based image recognition models. Using the subsets
of dataset developed in the previous chapter, multiple recognition models were
developed by training a series of other CNN models. General purpose image
recognition model that has been already developed by other researchers is a single
model and have multiple classes more than 10 classes. That approach is effective
to classify a category of an object but it is impossible to recognize various category
from features. For recognition of various features of an object, it is important to
divide features set that each CNN model train. For example, training single model
for the presence of armrest is better rather than one for the presence of armrest
and backrest. Combination of Multiple models is effective to the recognition of
various types category. We retrained total 6 models, three for functional features,
one for each of materials, seating capacity and style. Table 1 shows accuracy and
train steps of training and validation of each model. The Accuracies of the models
for functional features are averagely higher than the others. This is because those
functional features are shown as more explicit by additional components like an
armrest.
294 J.S. KIM, J.Y. SONG AND J.K. LEE

Table 1. The results of validation and training of each model.

4. INTERFACE OF AUTOMIZED EXTRACTION OF DESIGN

FEATURES OF INTERIOR DESIGN ELEMENTS
We developed GUI tool for automated extraction of design features of interior
design elements for visually well-showing extraction results and making it easy
for anyone to use this. This was This was developed in Python and PyQt5 for
cross-platform. This tool yet supports only automated extraction phase. This
application looks like Figure 6. The user can open lists of image files and click the
run button for automated extraction of design features of each file. The results of
each image are shown in the table of inference result. Now, the results of extraction
are only six that are derived from retrained models in the previous section.

Figure 6. Interface of GUI tool for automized extraction of design features.

 APPROACH TO THE EXTRACTION OF DESIGN FEATURES OF 295
INTERIOR DESIGN ELEMENTS USING IMAGE RECOGNITION
TECHNIQUE

5. Conclusion
This paper proposed deep learning-based approach for automized extraction of
design feature of from image. This research aims ultimately to develop automized
extraction of such features and make application for in interior design, such
as supporting case study of interior design or interior element database and
recommend system. As the basic research for that, we utilized deep learning-based
CNNs model for extracting design features of interior design elements on digital
image. Using well-trained model, we made multiple model learn four different
categories of design features. We categorized design feature of interest, such as
functional features, materials, seating capacity and design style, and collected
image dataset and subset according to the category. Each category includes
detailed classes. functional features include armrest, backrest and headrest.
materials do fabric, leather, plastic, velvet and wood. Seating capacity do seat
of division and length. Finally, design style does classic, modern eclectics,
Scandinavian. Total 3933 images were used and 6 models were re-trained. Mean
training accuracies of functional features is 94.3% and higher than other models.
This is because that such features are more explicit by existence and nonexistence
of components. However, detailed texture of material, seating capacity and design
style were not well recognized. We developed GUI tool for extracting design
features of given images. Through demonstration of the tool, it is checked for
anyone to easily use them for extracting some series of images. For future works,
the effective training methods will be needed as each feature with expansion
of target design element and features. With these, approach to applications for
interior design will be able to be proposed, such as digital image-based interior
design element and information database or recommendation system of interior
design element according to requirements.

Acknowledgements
This work was supported by the National Research Foundation of Korea
Grantfunded by the Korean Government (NRF-2015R1C1A1A01053497).

References
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A.,
Dean, J., Devin, M. and Ghemawat, S.: 2016, Tensorflow: Large-scale machine learning
on heterogeneous distributed systems, OSDI, 16, 265-283.
Bell, S. and Bala, K.: 2015, Learning visual similarity for product design with convolutional
neural networks, ACM Transactions on Graphics (TOG), 34(4), 98.
Dazkir, S.S. and Read, M.A.: 2012, Furniture forms and their influence on our emotional
responses toward interior environments, Environment and Behavior, 44(5), 722-732.
Hu, Z., Wen, Y., Liu, L., Jiang, J., Hong, R., Wang, M. and Yan, S.: 2017, Visual Classification
of Furniture Styles, ACM Transactions on Intelligent Systems and Technology (TIST), 8(5),
67.
Karayev, S., Trentacoste, M., Han, H., Agarwala, A., Darrell, T., Hertzmann, A. and
Winnemoeller, H.: 2013, Recognizing image style, arXiv preprint arXiv:1311.3715.
Krizhevsky, A., Sutskever, I. and Hinton, G.E.: 2012, Imagenet classification with deep
convolutional neural networks, Advances in neural information processing systems,
1097-1105.
296 J.S. KIM, J.Y. SONG AND J.K. LEE

LeCun, Y. and Bengio, Y. 1995, Convolutional networks for images, speech, and time series,
in M.A. Arbib (ed.), The handbook of brain theory and neural networks, MIT press.
LeCun, Y., Bengio, Y. and Hinton, G.: 2015, Deep learning, Nature, 521(7553), 436-444.
Lee, A.S.: 2010, A study on predicting consumers‘satisfaction based on the features of furniture
product designs, International Journal of Organizational Innovation (Online), 2(3), 138.
McDonagh, D., Bruseberg, A. and Haslam, C.: 2002, Visual product evaluation: exploring
users’ emotional relationships with products, Applied Ergonomics, 33, 231-240.
Miller, J.: 2010, Furniture: World Styles from Classical to Contemporary, Dorling Kindersley.
Redmon, J., Divvala, S., Girshick, R. and Farhadi, A.: 2016, You only look once: Unified,
real-time object detection, Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, 779-788.
Ren, S., He, K., Girshick, R. and Sun, J.: 2015, Faster R-CNN: Towards real-time object
detection with region proposal networks, Advances in neural information processing
systems, 91-99.
Schmidhuber, J.: 2015, Deep learning in neural networks: An overview, Neural networks, 61,
85-117.
Smardzewski, J.: 2015, Furniture design, Springer.
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke,
V. and Rabinovich, A.: 2015, Going deeper with convolutions, Proceedings of the IEEE
conference on computer vision and pattern recognition, 1-9.
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. and Wojna, Z.: 2016, Rethinking the inception
architecture for computer vision, Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, 2818-2826.
Taigman, Y., Yang, M., Ranzato, M. and Wolf, L.: 2014, Deepface: Closing the gap to
human-level performance in face verification, Proceedings of the IEEE conference on
computer vision and pattern recognition, 1701-1708.
Zehtaban, L., Elazhary, O. and Roller, D.: 2016, A framework for similarity recognition of CAD
models, Journal of Computational Design and Engineering, 3(3), 274-285.