Research Proposal for PhD 2021-2024

Talking Buildings: A smart IoT based Interactive Semantic Web

Proposed by- Suhas Devmane

Supervised by- Dr Charith Perera, Prof O F Rana

Cardiff School of Computer Science & Informatics, Cardiff university

Overview

My research will be the junction of three areas: Knowledge Engineering, IoT and
Machine Learning. I will Investigate how can we implement knowledge engineering and IoT
devices together in smart Buildings. Various Smart buildings are re-equipped with IoT devices
for management of building. Changes in the building, state and health of the physical
infrastructure can be identified to measure condition of building which increases functionality,
comfort, safety and reduces cost etc. Smart buildings should provide the efficient and
intelligent operation and management of the building physical infrastructure. Moreover, using
Machine Learning (ML) we can detect anomalies. My work aims to contribute to the smart
homes using knowledge representation techniques which allow the building to understand as
its own to become a ‘smart’ one. My idea is to implement varies IoT devices connected
through semantic web where building will give us answers regarding its own status,
anomalies etc. My research could assist building managers to implement this concept as well.

Introduction

In the era of IoT, modern buildings are being integrated with sensors and equipments.
Most of the cases we want to implement some techniques which will help us to achieve goals
like demand response, occupant interaction, predictive control, fault detection. So, IoT
devices can improve security, convenience, accessibility, energy efficiency operations etc.
More than 80 applications described by Buildsys. (5) IoT devices generate structured,
unstructured data and stores in various systems. Every building is unique and consist of
custom-designed controls, architectures, use cases. Multiple equipments are made by
different vendors having different capabilities and features. Hence, point should be noted that
‘smart’ solution should be applicable to heterogeneous buildings.
Another challenge is to understand the data in terms of metadata-the data about the
data. For example, what are the ‘things’ building contains, what are the locations of the
‘things’, how ‘things’ are interrelated to each other and configuration parameters of control
systems etc. ML will give us the ability to recognize the patterns which helps to detect
anomalies. It will improve the health and security, captures suspicious activity and uneven
patterns.
So, how will we observe and take action on some unexpected condition? How will we
find if any rare thing happening in building? isn’t that great if buildings tell us about its health?
I mean, If I ask the building, can you tell me temperature of room_no5? Is somewhere more
noise observed today? how is the energy consumption? Is any room overused? human health
is highly dependent on the building climate. (15)(16) so, tell me how is the health of building.
Will it be great if building self describes itself to become a ‘smart’ one? Can building tell us
whenever there is anything unexpected thing happens? It is only possible when building
learns knowledge about his capabilities and their relationships. For that, building should
understand the data captured by devices and understand their location and relations with
each other.
Literature review

Supervisory Control & Data Acquisition (SCADA) was introduced in early 2000s,
Technological boom occurred and SCADA technologies became antiquated over time.
Building energy management systems (BEMS) sometimes called building management
systems (BMS) is implemented to save energy bills (7) followed by many others. (8)(9) It
reduces the demand and consumption of energy succesfully. (12)
BIM Infrastructure (32) Is developed to design and operations which capture structural
information, but unable to describe of how the constituent equipment and points function
together. (44) While BMS, SCADA systems are custom to the deployment sites, Project
Haystack [2] aims to address heterogeneity in buildings using tags.
Project Haystack is a popular tagging system for describing building assets using
semi-structured sets of tags. Think of it as a “MARKUP LANGUAGE” for data Because there
are no formal rules for how tags can be used. It focuses on capturing topological concepts in
buildings. (45) Smart Appliances REFerence Ontology (SAREF) is an ontology capturing high
level aspects of smart and connected appliances,
Metadata schemas like Project Haystack (1), IFC (2) and semantic sensor web (3) are
compared by Dr. Bhattacharya (4) and conclude that none of these metadata schemas
investigated is complete or expressive enough to capture all the tags and semantic
information in buildings, they aren’t flexible to capture novel sensors and semantic sensor.
(4) This concludes that, existing schema fall short in capturing important relationships and
concepts necessary for applications.

 BMS requires site-specific conventions and implicit domain knowledge.

 IFC is having inextensible data model with limited query mechanisms for software.
 SAREF unable to capture the full spectrum of equipment and sensors that exist in
buildings.
 Project Haystack is popular tagging system but there is lack of formal rules for tag
composition, lack of improvement over unstructured labels and it unable to cover
HVAC, electrical subsystems. It is difficult to standardize the use of tags across many
buildings.

To overcome this metadata problems and considering properties of standard researchers

developed BRICK Schema.

Brick is an open-source metadata schema that semantically describes various

components, systems infrastructures within a building to enable innovative applications
across the building. It standardizes semantic descriptions of the physical, logical and virtual
assets in buildings and their relationships.
Brick uses Semantic Web technology to describe features, assets and subsystems.
Brick models describe data sources and their context - what they are, where they are, what
they mean, and how they relate to the building processes and structures that contain them.
(43) A Brick model of a building is a labelled, directed graph in which the nodes are entities
and the edges are relationships.
Brick has several advantages over Project Haystack and IFC. (35)
 BOT and Brick are built upon Semantic Web technology and it’s easy to translate
concepts between the two data models.
 SAREF models can be easily integrated into Brick.
 Brick developed a taxonomy and an ontology which improve problem solving within
that domain. Ontologies limit complexity and organize data into information and
knowledge.
 The Brick ontology is distributed as a set of Turtle files which is a compact textual
format that is understood by most Semantic Web tools. (33)
 Brick based metadata model that can be maintained throughout the building lifecycle
(34) and we can create our own Brick models with ML.
 Recently, a common platform for Brick metadata inference Is developed from some
existing metadata sources and modelled to a central integration server, which merges
the metadata into a valid Brick model. (33)
 It uses Resource description framework (RDF) which is structured statements about
resources consists of declaration of properties of resources and their relationships to
other resources.
 Brick uses SPARQL query language.

Several existing and prior efforts to make database (36-41) are limited in terms of span
and context. Hence, researchers made open testbed for storing, describing, updating,
discovering and retrieving building data using the Brick metadata schema and Introduced as
mortar. The Mortar platform used by algorithm developers, researchers, building managers.
the platform will soon support user-provided datasets and metadata models. (35)

By considering all of the above researches and technologies, I come to know that the
combination of these Knowledge representation techniques, machine learning technologies
and IoT devices can be implemented in the building to form a network which will talk to us
using different tools.
Research Questions:

In the light of the above discussion my work aims to contribute to the smart buildings which
will address some of the following research questions.
 How we can give knowledge to the building so that building can talk to us? What will
be a unique solution for all types of buildings?
 How will we implement different types of knowledge representation techniques like
semantic web in smart buildings?
 What are the different ways to find anomalies in real time and what kind of devices will
need to deploy?
 How to implement and analysis different types of ML algorithms and use deep learning
to the detect anomalies?
 How to store and process the data in efficient ways for quick and accurate predictions?
 How to program different sensors, ML algorithms, semantic web and their query
languages to heterogeneous buildings? How to use different tools?
 How will we connect semantic web to IoT devices? How to implement RDF data model,
SPARQL queries, RDF schema and OWL standards, URI’s for talking buildings?
Methodology
ML based anomaly detection developed earlier (10) where autoencoder and long
short-term memory encoder decoder (LSTM-ED) models are tested. This study showcases
that the advancement in ML and IoT can be fully utilized to solution development. I will use
following three pillars in my research.

Why IoT
IoT infrastructure reflects the strong connectivity of assets between physical and
digital. (17) The Devices data can be centrally stored in the cloud database which enables
further data analytics on historical and real time data. Earlier experiment with IoT (17) found
reliable, efficient, flexible, economical. Number of battery operated or plugged heterogeneous
IoT devices can be implemented in every type of building with minimum energy consumption.
Cisco predicts that more than 50 billion devices will be connected to internet by 2020. IoT
aims to intelligently interlink these devices to develop new custom and understandable
communication between human and among things themselves. These devices records,
stores, understands inhabitant’s behaviour to improve their health and safety.

Why Machine learning

Earlier described Brick Schema can be used with the ML to detect the anomalies.
Benefiting from the evolving cloud computing technology, mature ML techniques such as
neural networks that rely heavily on powerful computing resource are able to be exploited by
a variety of applications in the building environment to optimize operations [19], energy
consumption [20] and human comfort [21], etc. Earlier authors developed (ARIMA) (22)
model for anomaly detection is effective [23] but requires heavy computation for thousands
of data points. k-Nearest Neighbour (KNN) method to predict and detect anomalies is
implemented earlier. (24)
Different attempts are made to implement artificial Neural networks ANN, regression
models, hybrid ARIMA-ANN in the buildings earlier. (25)(26)(27). Further studies
(28)(29)(30)(31) have shown the promising potentials of machine learning, especially deep
neural network-based anomaly detection algorithms used in isolated areas. Therefore, a
comprehensive study on applying ML based anomaly detection approaches to smart ‘talking’
buildings is significant.

Knowledge Representations
Information is data that has been given meaning by way of relational connection.
Information answers to questions that begin with such words as who, what, when, where and
how many. Knowledge is the appropriate collection of information, such that its intent is to be
useful. Natural language is so difficult to represent the knowledge due to the paraphrasing
and ambiguity. Formal knowledge presentation is a field of AI which captures the
semantics(meaning) of concepts, properties, relationship and entities of specific knowledge
domains as structured data. Machines (computers) must able to understand formal
knowledge representation.
Understanding depends on syntax, semantics, context, pragmatics and experience.
The semantics of information is expressed with the help of knowledge representations
(Ontologies). The semantic web is an extension of the current web. The meaning of
information (semantics) is made explicit by formal (structured) and standardized knowledge
representations(ontologies). The semantic web is kind of a global database that contains a
universal network of semantic propositions.
Objectives:
 My Research objective is to make a talking building by combining above three
technologies and make a prototype of infrastructure which can be implemented for any
types of buildings with minimum human intervention.
 To make a robust solution with maximum accuracy and minimum latency.
 To play with different types of ML algorithms and to perform several operations on
data for minimum computation and quick response.
 To experiment and implement different upcoming strategies and techniques so that
my research will help/assist to under-development technologies in smart buildings.
My research task includes:

To inspect the number of IoT devices which can be implemented for the building
To review and find alternatives for data acquisition and storage and processing
To study and review knowledge representation techniques
To study types of anomalies, their sources and detection strategies
To study machine learning algorithms and inspect their performance
To deploy devices and experimentation, analysis
To setup and describe prototype for the building
To Document research findings and future scope
However, I have to change some strategies or techniques depending upon the
current and upcoming technologies to make better models.

Time Table

Year Oct-Dec Jan-Mar April-Jul Aug-Oct

1 Review Research
proposal
Discussion and setting
goals, planning
Literature review
Skill learning
Monthly supervisories
2 Re-visit literature review
Prototype implementation
On-side visits, Fieldwork
and investigations
Experimentation, analysis
Testing, Modifications
and review
Monthly Supervisories
Skill implementations
3 Re-visit literature review
Transcription, analysis
Writeup
Viva preparations
Monthly supervisories
Bibliography
1) Project haystack. http://project-haystack.org/.
2) ISO 16739:2005 - Industry Foundation Classes, Release 2x, Platform Specification (IFC2x Platform),
2005.
3) M. Compton, P. Barnaghi, L. Bermudezc, and et al. The SSN ontology of the W3C semantic sensor
network incubator group. Web Semantics: Science, Services and Agents on the World Wide Web,
2012.
4) Arka Bhattacharya, Joern Ploennigs, and David Culler. 2015. Short Paper: Analyzing Metadata
Schemas for Buildings: The Good, the Bad, and the Ugly. In Proceedings of the 2nd ACM
International Conference on Embedded Systems for Energy-Efficient Built Environments. ACM.
5) https://ibm.biz/UCB-IBM-Apps
6) T. Cerovsek. A review and outlook for a ’building information model’(BIM): A multi-standpoint
framework for
technological development. Advanced engineering informatics, 25(2):224–244, 2011
7) Khairy sayed, Hossam A. Gabbar. etc BUILDING ENERGY MANAGEMENT SYSTEMS (BEMS),
University of Ontario Institute of Technology, 2000 Simcoe Street North, Oshawa, L1H7K4, ON,
Canada
8) Nguyen, N.-H., Tran, Q.-T., Leger, J.-M., and Vuong, T.-P. (2010) A real-time control using wireless
sensor network for intelligent energy management system in buildings.IEEE Workshop on
Environmental Energy and Structural Monitoring Systems
9) Javed, A., Larijani, H., Ahmadinia, A., Emmanuel, R., Mannion, M., and Gibson, D. (2017) Design and
implementation of a cloud enabled random neural network-based decentralized smart controller with
intelligent sensor nodes for HVAC.
10) Yu Liu, Linköping University,∗, Zhibo Pang Corporate Research, Sweden Magnus Karlsson a,
Shaofang Gong , Anomaly detection based on achine learning in IoT-based vertical plantwall for
indoor climate control
11) Iveta Puķītea*, Mg. sc., Ineta Geipeleb, Prof., Dr. oec, Different Approaches to Building Management
and Maintenance Meaning Explanation, Riga Technical University, Institute of Civil Engineering and
Real Estate Economics, 6 Kalnciema Str., Riga, LV-1048, Latvia
12) S. Tepic, P. Pejic, J. Domšic, H. Mihaldinec, H. Džapo, IBMS – Intelligent Building Management
System Framework, Information and Communication Technology, Electronics and Microelectronics
(MIPRO) 2015 143 – 148.
13) B. Yu, Z. Hu, M. Liu, H. Yang, Q. Kong, Y. Liu, Review of research on airconditioning systems and
indoor air quality control for human health, Int. J. Refrig. 32 (1) (2009) 3–20, [Online]. Available:
http://www.sciencedirect.com/ science/article/pii/S0140700708000984
14) J. Kim, M. Kong, T. Hong, K. Jeong, M. Lee, Physiological response of building occupants based on
their activity and the indoor environmental quality condition changes, Build. Environ. 145 (2018) 96–
103, [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0360132318305663
15) J. Kim, T. Hong, J. Jeong, C. Koo, M. Kong, An integrated psychological response score of the
occupants based on their activities and the indoor environmental quality condition changes, Build.
Environ. 123 (2017) 66–77, [Online]. Available:
http://www.sciencedirect.com/science/article/pii/S0360132317302810
16) J. Kim, T. Hong, M. Kong, K. Jeong, Building occupants’ psycho-physiological response to indoor
climate and CO2 concentration changes in office buildings, Build. Environ. 169 (2020) 106596,
[Online]. Available: http://www. sciencedirect.com/science/article/pii/S036013231930808X
17) P. Ray, A survey on Internet of Things architectures, J. King Saud Univ.-Comput. Inf. Sci. 30 (3)
(2018) 291–319, [Online]. Available: http://www.sciencedirect.
com/science/article/pii/S1319157816300799
18) Ghayvat, H., Mukhopadhyay, S., Gui, X., & Suryadevara, N. (2015). WSN- and IOT-Based Smart
Homes and Their Extension to Smart Buildings. Sensors, 15(5), 10350–
10379. doi:10.3390/s150510350
19) X. Dai, J. Liu, X. Zhang, A review of studies applying machine learning models to predict occupancy
and window-opening behaviours in smart buildings, Energy Build. 223 (2020) 110159, [Online].
Available: http://www.sciencedirect.com/science/article/pii/S0378778820303017.
20) K. Mason, S. Grijalva, A review of reinforcement learning for autonomous building energy
management, Comput. Electr. Eng. 78 (2019) 300–312, [Online]. Available:
http://www.sciencedirect.com/science/article/pii/S0045790618333421
21) M. Ayoub, A review on machine learning algorithms to predict daylighting inside buildings, Sol. Energy
202 (2020) 249–275, [Online]. Available:
http://www.sciencedirect.com/science/article/pii/S0038092X20303509.
22) A.H. Yaacob, I.K.T. Tan, S.F. Chien, H.K. Tan, ARIMA based network anomaly detection, in: 2010
Second International Conference on Communication Software and Networks, 2010, pp. 205–209.
23) S. Ahmad, A. Lavin, S. Purdy, Z. Agha, Unsupervised real-time anomalydetection for streaming data,
Neurocomputing 262 (2017) 134–147, Online Real-Time Learning Strategies for Data Streams,
[Online]. Available: http://www.sciencedirect.com/science/article/pii/S0925231217309864
24) Z. Wang, T. Parkinson, P. Li, B. Lin, T. Hong, The Squeaky wheel: Machine learning for anomaly
detection in subjective thermal comfort votes, Build. Environ. 151 (2019) 219–227, [Online]. Available:
http://www.sciencedirect.com/science/article/pii/S0360132319300861
25) X. Shi, W. Lu, Y. Zhao, P. Qin, Prediction of indoor temperature and relative humidity based on cloud
database by using an improved BP neural network in chongqing, IEEE Access 6 (2018) 30559–
30566.
26) A. Hamrani, A. Akbarzadeh, C.A. Madramootoo, Machine learning for predicting greenhouse gas
emissions from agricultural soils, Sci. Total Environ. (2020)140338, [Online]. Available:
http://www.sciencedirect.com/science/article/pii/S0048969720338602.
27) Umit. Çavuş, Büyükşahin, Şeyda. Ertekin, Improving forecasting accuracy of time series data using a
new ARIMA-ANN hybrid method and empirical mode decomposition, Neurocomputing 361 (2019)
151–163, [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0925231219309178.
28) P. Malhotra, V. Tv, A. Ramakrishnan, G. Anand, L. Vig, P. Agarwal, G. Shroff, Multi-sensor
prognostics using an unsupervised health index based on LSTM encoder-decoder, in: 1st SIGKDD
Workshop on Machine Learning for Prognostics and Health Management, 2016
29) D. Park, Y. Hoshi, C.C. Kemp, A multimodal anomaly detector for robot-assisted feeding using an
LSTM-based variational autoencoder, IEEE Robot. Autom. Lett.3 (3) (2018) 1544–1551.
30) N. Ding, H. Ma, H. Gao, Y. Ma, G. Tan, Real-time anomaly detection based on long short-Term
memory and Gaussian Mixture Model, Comput. Electr. Eng.79 (2019) 106458, [Online]. Available:
http://www.sciencedirect.com/science/article/pii/S0045790618334372.
31) C. Xu, H. Chen, Abnormal energy consumption detection for GSHP system based on ensemble deep
learning and statistical modeling method, Int. J. Refrig. 114 (2020) 106–117, [Online]. Available:
http://www.sciencedirect.com/science/article/pii/S0140700720300992
32) Oti, A. H., Kurul, E., Cheung, F., & Tah, J. H. M. (2016). A framework for the utilization of Building
Management System data in building information models for building design and operation.
Automation in Construction, 72, 195–210. doi:10.1016/j.autcon.2016.08.043
33) Gabe Fierro, Anand K. Prakash, Cory Mosiman, Marco Pritoni, Paul Raftery, Michael Wetter, and
David E. Culler. 2020. Shepherding Metadata Through the Building Lifecycle. In Proceedings of the
7th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and
Transportation.
DOI: https://doi.org/10.1145/3408308.3427627
34) Demo Abstract: Interactive Metadata Integration with Brick Gabe Fierro, Anand Krishnan
Prakash, Cory Mosiman, Marco Pritoni, Paul Raftery, Michael Wetter, David E Culler. In Proceedings
of the 7th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and
Transportation (BuildSys 2020). November 18, 2020, Virtual Event.
DOI: https://doi.org/10.1145/3408308.3431125
35) Mortar: An Open Testbed for Portable Building Analytics, GABE FIERRO, UC Berkeley MARCO
PRITONI, Lawrence Berkeley National Laboratory, MOUSTAFA ABDELBAKY, DANIEL LENGYEL,
and JOHN LEYDEN, UC Berkeley, ACM Transactions on Sensor Networks, Article No.: 7,
https://doi.org/10.1145/3366375
36) S. Barker, A. Mishra, D. Irwin, E. Cecchet, P. Shenoy, and J. Albrecht. 2012. Smart*: An open data
set and tools for enabling research in sustainable homes. SustKDDAugust (2012), 6.
DOI:https://doi.org/adf
37) Nipun Batra, Jack Kelly, Oliver Parson, Haimonti Dutta, William Knottenbelt, Alex Rogers, Amarjeet
Singh, and Mani Srivastava. 2014. NILMTK: An open source toolkit for non-intrusive load monitoring.
In Proceedings of the 5th International Conference on Future Energy Systems (e-Energy’14), 265–
276. DOI:https://doi.org/10.1145/2602044.2602051arxiv:1404.3878
38) EIA. 2016. Commercial buildings energy consumption survey (CBECS) user’s guide to the 2012
CBECS public use microdata file. August
(2016).https://www.eia.gov/consumption/commercial/data/2012/index.cfm?view=microdata.
39) J. Zico Kolter and Matthew J. Johnson. 2011. REDD: A public data set for energy disaggregation
research. SustKDD Workshop 25, 1 (2011), 1–6.
http://users.cis.fiu.edu/lzhen001/activities/KDD2011Program/workshops/WKS10/doc/SustKDD3.pdf.
40) Clayton Miller and Forrest Meggers. 2017. The building data genome project: An open, public data set
from nonresidential building electrical meters. Energy Procedia 122 (2017), 439–444.
DOI:https://doi.org/10.1016/j.egypro.2017.07.400. CISBAT 2017 International Conference on Future
Buildings & Districts—Energy Efficiency from Nano to Urban Scale.
41) S. E. Womble, J. R. Girman, E. L. Ronca, R. Axelrad, H. S. Brightman, and J. F. Mccarthy. 1995.
Developing baseline information on buildings and indoor air quality (BASE’94). Part II (1995), 1–8.
42) https://hackernoon.com/anomaly-detection-strategies-for-iot-sensors-l31x321t
43) Bharathan Balaji, Arka Bhattacharya, Gabriel Fierro, Jingkun Gao, Joshua Gluck, Dezhi Hong, Aslak
Johansen, Jason Koh, Joern Ploennigs, Yuvraj Agarwal, et al. 2016. Brick: Towards a unified
metadata schema for buildings. In Proceedings of the ACM International Conference on Embedded
Systems for Energy-Efficient Built Environments (BuildSys). ACM
44) https://brickschema.org
45) https://w3c-lbd-cg.github.io/bot/