The 9th IEEE International Conference on Dependable Systems, Services and Technologies, DESSERT’2018
24-27 May, 2018, Kyiv, Ukraine
Integration of Neural Networks
Into Smart Sensor Networks
Ivan Lobachev1,2, Roman Maleryk1, Svitlana Antoschuk1, Denys Filiahin1 and Mykhaylo Lobachev1
1
ONPU, Odesa, Ukraine, 65044
2
Intel Corp., Vancouver BC, Canada
Abstract—Within the scope of sensor networks, there
exists a number of obstacles in the face of lower processing
power, if focusing on power consumption, as well as
maintenance and deployment issues when operating on a
larger scale. Such issues as bandwidth availability as well as
communication latency are also important factors to
consider. This article discusses a distributes sensor network
model that utilizes a low power approach along with neural
networks to facilitate more wholesome local processing fro
faster responses as well as improve overall system efficiency.
The physical deployment model used the Intel Movidius
VPU modules and Raspberry Pi microcontrollers to
investigate and model the neural network integration. As
part of the architecture, Caffe and TensorFlow frameworks
were used the most to process graphical and numerical data.
Certain abstraction layers had to be developed to facilitate
smoother horizontal communication between the units. The
system was tested to operate with both graphical inputs such
as images or video feeds, as well as numerical information,
such as analog sensor data.
Keywords—Neural Networks, Sensor Networks, Machine
Learning, Deep Learning, Smart Sensors, Distributed
Systems, Low-Power Solutions, Scalable Architecture
I. INTRODUCTION
In today’s environment neural networks are finding
new applications in many branches of the industry.
However, while major solutions apply the cloud
computing approach, where the data is funnelled to central
servers for processing, this introduces issues of bandwidth
limitations, network access and latency when considered
in the context of larger scale data acquisition and
transducer networks. For example, a smart building may
wish to uphold the privacy of the collected data, while
other tasks such as image and video processing would
require significant amounts of computational power.
These sorts of tasks can greatly benefit from the use of a
compact and smart data processing method, for large data
sets or big data structures, as well as dynamic resource
allocation or system expansion. Case-wise examples can
include a situation where currently allocated system
resource are insufficient for the presented task, may
require new classes of supplied information, graphical or
otherwise, or simply a task that this particular system has
not dealt with before. In order to solve this problem the
software layer, based on a neural network, would be
implemented in such a way that if a new device, or
978-1-5386-5903-8/18/$31.00 ©2018 IEEE
computing resource is connected to the system it would
“inherit” the necessary knowledge and information from
the other modules present. This approach would allow
making the system more easily scalable, and as a result it
would more easily deal with unexpected situations and
more complex problems.
II. OVERVIEW OF RELATED WORK
There are various ways can be accelerated using a
number of approaches, either by means of specialized
hardware or via software optimization. Nowadays there
are a number of processors and devices developed
specifically to work with neural networks. Below is a
quick overview of currently present solutions. Mobileye
EyeQ is a specialized processor to accelerate machine
vision processing with the intended use lying in
autonomous vehicles. Google Tensor Processing Unit
(Google TPU) is a tensor processor of the neural
processing class, which is a specialized integral system
developed by Google, and intended to be used in tandem
with the machine learning library TensorFlow, as part of
new approach development in big data processing. The
Intel Nervana Neural Network Processor (NNP) is the first
commercially available tensor processor, which is
intended for training neural networks using deep learning.
Another complimentary example to such an approach can
be IBM TrueNorth, which is a neuromorphic processor,
implemented based on neuron interaction rather than
arithmetic modeling. Lastly in terms of hardware there is
the Intel Movidius Myriad 2 which is a multicore AI-
accelerator based on the VLIW (Very Long Instruction
Word) architecture. With additional tie-ins for video
processing it serves as a basis for the Movidius Neural
Compute Stick, a device that supports USB interface and
employs the technology of deep neural networks [1].
Similarly, in terms of software there are a few prominent
frameworks at the moment that were also employed in this
work. They are the TensorFlow framework, which is an
open source library for machine learning developed by
Google and aimed at establishing and training neural
networks, with the goal of automatic location and
classification of figures nearing perception levels close to
those of a human. The other framework that was used is
Caffe (Convolution Architecture For Feature Extraction)
is a deep learning environment, developed by Yangqing
544
Jia from Berkley [2]. Caffe is an open source platform, While in order to save time the basic dataset, for the
distributed under the BSD license. Being developedin sake of this particular example road sign recognition was
C++ and supporting python, it provides sufficient degree used as seen in Figure 1, can be found online, it can be
of versatility for a range of applications. Based on the further developed and increased by means of manipulating
analysis of the available resources and analogous existing data. The dataset also becomes more robust when
solutions, it was decided to use Rasperry Pi 3 as the the introduced variety includes higher presence of noise,
Microcontroller Node within the sensor network distortion and unusual angles.
prototype, and the Movidius NCS as the supporting
module for neural network related tasks. This combination
allows to demonstrate the aspects of both self-sufficient
processing power of small-scale nodes, which in turns also
greatly benefits mobility where necessary, as well as low
power consumption, with the Movidius module requiring
less than 1W of power. In addition, the overall cost per
unit would be noticeably less than similar alternatives,
given the requirement of local processing and cloud
porting avoidance is applied. Figure 1. Examples of graphics from the training dataset

III. IMAGE RECOGNITION AND GRAPHIC PROCESSING To perform image recognition, during the initial stages
CAPABILITY INTEGRAITON the classical Convolutional Neural Network (CNN) model
Computer vision allows to discover, analyze track and can be used, for example AlexNet Figure 2 [6] [7].
classify objects. However in many applications, the main
task is offloaded to a remote server, or may induce high
performance and monetary costs if done locally. A few
examples of applications include medicine, industrial
applications as well as in the military [3] [4] [5]. For
example, some of the most advanced systems are capable
of sending the missile to a given area, instead of a specific
target. Upon arrival the missile will locate the targets
locally, after navigating to the correct location based on
received video data. Another new area of application is
the autonomous vehicle industry, including submersible Figure 2. Network architecture illustration
vehicles, aircrafts as well as land-based vehicles such as
As a starting point for testing, the dataset from The
cars, trains and robots. In these applications the level of
German Traffic Sign Recognition Benchmark [8] was
autonomy can vary from fully autonomous, for example
used. However in the future other types of datasets, as
UAVs, to assistive where the machine vision data is used
well as developments to the current one are in progress.
to support the driver or pilot and is limited by the
application and available computing and power resources. To minimize the time needed to train a neural network, the
To test graphic processing capabilities being integrated method of adaptation and improvement of already
established network models using the Caffe framework is
into the structure, the above applications were considered
employed. The framework describes the network in a
and used as a basis for the testing example. Given the fact
specialized “prototxt” file format, where the layers are
that the compact size of RPi and Movidius NCS positively
described in a style similar to JSON, as seen in Figure 3.
impact their mobility capabilities, the nodes with the
added functionality could also be used in self-driving cars The trained network is stored under a “caffemodel” file
or traffic-based applications. As mentioned previously the extension.
low power consumption of the overall unit also positively
reflects on the viability of such application. In order to
validate the core functionality being integrated, the
following stages are used.
 Search and creation of the dataset to facilitate the
learning
 The training of the deep neural network
 Converting the input graphical flow (either static
images or video) into required format
 Graphical processing of the data using the neural
network based on Movidius NCS Figure 3. An example of a layer within the neural network

545
 To implement the aforementioned system, with
The obtained prototxt and caffemodel files are required dynamic architecture, the following set-up structure was
to create the “graph” file, which is in turn an input file for established. A microntroller board consisting of a
the neural chip interface. The file can be generated using a Raspberry Pi, the Movidius NCS modules and the Emotiv
specialized utility “MyNCCompile”. In order to utilize the Epoc+, the graphical representation of the setup can be
established infrastructure for video feed processing, a few seen in Figure 5 below.
slight modifications need to take place. After the video
feed is routed to the network, the “categories” and graph
files need to be modified to reflect the names of classes
within the trained network, in the same order as during its
learning procedure. This process can be automated the
same way as the dataset generation. By running the
modified software in tandem with the Movidius NCS we
are able to recognize the road sign presented to the unit
via a video feed as can be seen in Figure 4.

Figure 5. The representative structure of the setup

As a sanity check reference point, to begin the

Figure 4. An example of the module at work with a video feed
The above test allows us to verify the viability of use
of such a system within a wider scope. For example in the
future a similar approach can be used to classify room
states, occurring events or item location using an ongoing
video feed from existing surveillance systems or other
image collection sources.
IV. PROCESSING COMPLEX NUMERICAL SENSOR DATA
As an example to test the numerical data processing
using neural networks, a complex signal source in the
form of EEG array was chosen. Nowadays there are e
number of devices, even outside of the medically available
equipment to obtain EEG data, such as for example neural
helmets that are used by the gaming industry. Usually
such a device would contain a combination of sensors
within, including not only an array of EEG electrodes but
also motion sensors and gyroscopes to track the head
motions in addition to brain activity and signals, this as a
result, allows the device to be used in a wide range of
applications. To provide a brief overview of possible
applications, a neural helmet could be used by people with
movement disabilities [9], medical practitioners to
monitor brain activity of the patient [10], or to monitor the
emotional state of the patient when, for example, verbal
communication may not be possible. Alternatively it could
be the next generation of joystick controllers for games,
according to recent trends [11].
investigation of this application, the initial dataset was
obtained from an open source resource. The chosen
dataset contains raw EEG data, that was gathered using
the Emotiv Epoc+ which contains 14 electrodes
monitoring the points F7, F8, F3, F4, FC5, FC6, T7, T8,
P3, P4, P7, P8, O1, O2. The action of interest was the
state of the eyes, either open or closed, was externally
measured using a camera concurrent to the continuous
EEG measurement and then physically supplemented to
the dataset, upon investigation of the obtained contour
signs. A binary output result was obtained where “1”
defines the eyes being closed and “0” them being open.
The set contains 14976 measurements, with 14 signals per
measurement each signal supplied by one channel of the
Emotiv helmet EEG sensor. The recording time was 117
seconds while the discrete frequency was 128 Hz. The
sample of the data across all sensors, and the resulting
signal can be found on the graph in Figure 7.
Figure 6. A sample spectrogram containing over 6592 records of
data

546
 Figure 7. The graph of the recorded signals

In this experiment a neural network was used to

construct the classifier of the eye state, while the
implementation method employed the TensorFlow
framework due to its affinity for numerical applications,
the necessary software used Python as the programming
language of choice for the implementation. While using
the data from the instrumental peripherals a perceptron
was constructed with the network structure of 14x40x1,
which demonstrated a 94% accuracy on the tested
datasets. However, in order for the dynamic system to
utilize the benefits of the Movidius NCS, which is
optimized for use with CNN based networks, the above
numerical dataset needed to pass through a software
abstraction layer before entering the NCS hardware. This
pre-processing involved converting the obtained data into
a spectrogram form in the frequency domain, by means of
a Fourier transform, and post process the obtained data in
graphical form from that point on. The resulting data lies
in the spectrum of 0.5 Hz to 30 Hz, due to the fact that this
scope encompasses all of the data of interest, delta
rhythms (0.5-4 Hz), theta rhythms (4-8 Hz)as well as
alpha (8-13 Hz) and beta (13-30 Hz) rhythms. Such an
approach improves efficiency, as the processing would
occur in batches, for data collected over a period of 1-10
seconds. It is worth noting that the measurement of each
of the sensors is assumed to be independent, and this
allows to also increase the dataset size for the training
stages. The conceptual architectural diagram of the
software abstraction layer for the spectrogram conversion
of numerical data, in this case using EEG as the data
source, can be found in Figure 8.
As can be interpreted from earlier introduction of the
added software abstraction layer used for preprocessing, it
can condense a large amount of information into a single
entity that can more optimally be used within the scope of
CNN based networks. An example of such a resulting
graphic can be found in Figure 6. The spectrogram was
constructed using 6592 records of data from 16 separate
sources.
Figure 8. Representative structural diagram of the software
abstraction layer
As will also be discussed later, including in Section V,
this approach demonstrates the viability of use of
acceleration of processing of large numerical data sets
using socialized hardware based on CNN architecture for
higher degree of adaptability to the acquired data, and
malleability of the application itself.
V. OVERALL ARCHITECTURE
The overall architecture of the system would build on
top of
the established transducer network described in [12].
Each of the nodes is still able to function on a power
budget of less than 5W, even given the prototype nature of
component selection. Furthermore this power efficiency
has the potential to be further improve by applying the
architecture to a more industry specialized application,
where singular integration as well as additional
developments in firmware with regards to power saving
can further improve the overall power efficiency of the
units. Nevertheless, even within the current configuration
each node now possesses the ability to individually
process graphical data and large sets of complex
numerical data, which can allow to decrease the amount of
nodes needed by simply raising the load on a smaller
number of existing ones. As mentioned previously there is
also the possibility for further optimization of node

547
utilization and placement based on specific applications
and use model, however they will not be discussed in
detail within the scope of this work. The above described
structure will also further benefit from leveraging the
network aspect of the architecture overall, as will be
described in Section VI.
VI. FUTURE WORK
Future development of the functionality introduced in this
work is currently encompassing the following:
 Incorporating load distribution across multiple
NCS units
 Integrating the video processing component into a
more complex system
 Improving the dynamic resource allocation
 Improving the “shared networks” architecture
implementation
The last point of the aforementioned list refers to the
ability of our system to “share” the networks it has already
trained previously with other nodes or units, if the unit
determines that it is undertaking a similar task. The
improvement to the obtained network may then be made
based on new data that is received, and an updated version
of the network will be shared back via the “common
library” function which is implemented by relying on the
network structure of the system, where the nodes are able
to communicate internally without the need for external
network connections.
VII. CONCLUSION
In conclusion, the nodes of the transducer network were
able to assimilate the ability to use neural network
approaches to process bigger volumes of more complex
data. Furthermore, by incorporating the Visual Processing
Units (VPUs) into the new architecture the nodes have
gained the ability to perform more sophisticated analysis
and decision making based on graphical data such as
static imaging or dynamic video feeds. The improved
processing capability allows for the system to become
more adaptable to different environment and more
efficient in terms of necessary units to cover a specific
area or obtain a desired volume of data. Furthermore, the
increased processing power aided by the neural network
integration adds less than 1W of power budget increase
per NCS [13], which means that the overall power
efficiency of the system as a whole, taking into
548
consideration the reduction of nodes necessary has also
improved. The system has performed block level testing
on the ability to perform graphical data analysis using the
architecture outlined in Section III as well as complex
numerical data processing discussed in Section IV, which
also included an incorporation of a software layer to
allow more versatile data inputs.
REFERENCES
[1] M. H. Ionica and D. Gregg, “The movidius myriad architecture’s
potential for scientific computing,” IEEE Micro, vol. 35, no. 1, pp.
6–14, 2015.
[2] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R.
Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional
architecture for fast feature embedding,” in Proceedings of the
22nd ACM international conference on Multimedia. ACM, 2014,
pp. 675–678.
[3] A. Makaenko and V. Kalaida, “Technique of localization of face
image for the systems of videocontrol on the bases of neuronet,”
Thompson Polytechnic University Journal, vol. 309, no. 8, 2006.
[4] G. J. Beach, G. Moody, J. Burkowski, and C. J. Jacobus, “Portable
composable machine vision system for identifying projectiles,”
Jan. 11 2018, uS Patent App. 15/675,763.
[5] M. Nielsen, “Using neural nets to recognize handwritten digits,”
Neural Networks and Deep Learning, 2015.
[6] H.-C. Shin, H. R. Roth, M. Gao, L. Lu, Z. Xu, I. Nogues, J. Yao,
D. Mollura, and R. M. Summers, “Deep convolutional neural
networks for computer-aided detection: Cnn architectures, dataset
characteristics and transfer learning,” IEEE transactions on
medical imaging, vol. 35, no. 5, pp. 1285–1298, 2016.
[7] R. Liao, A. Schwing, R. Zemel, and R. Urtasun, “Learning deep
parsimonious representations,” in Advances in Neural Information
Processing Systems, 2016, pp. 5076–5084.
[8] J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel, “Man vs.
computer: Benchmarking machine learning algorithms for traffic
sign recognition,” Neural Networks, no. 0, pp. –, 2012. [Online].
Available:
http://www.sciencedirect.com/science/article/pii/S0893608012000
457
[9] J. Meng, S. Zhang, A. Bekyo, J. Olsoe, B. Baxter, and B. He,
“Noninvasive electroencephalogram based control of a robotic
arm for reach and grasp tasks,” Scientific Reports, vol. 6, p.
38565, 2016.
[10] L. Yuan and J. Cao, “Patients’ eeg data analysis via spectrogram
image with a convolution neural network,” in International
Conference on Intelligent Decision Technologies. Springer, 2017,
pp. 13–21.
[11] F. Shir, “Mind-reading system-a cutting-edge technology,” Mind,
vol. 6, no. 7, 2015.
[12] I. Lobachev and E. Cretu, “Smart sensor network for smart
buildings,” in Information Technology, Electronics and Mobile
Communication Conference (IEMCON), 2016 IEEE 7th Annual.
IEEE, 2016, pp. 1–7.
[13] F. Conti and L. Benini, “A ultra-low-energy convolution engine
for fast brain-inspired vision in multicore clusters,” in Design,
Automation & Test in Europe Conference & Exhibition (DATE),
2015. IEEE, 2015, pp. 683–688.