Edge Processing and Enterprise Integration:
Closing the Gap on Deployable Industrial
Sensor Networks
Robert P. Adler, Jonathan Huang, Raymond Kong, Philip Muse, Lama Nachman, Rahul C. Shah,
Chieh-Yih Wan, and Mark Yarvis
Intel Corporation, USA
Abstract-The next frontier of sensor networking is
integration: building operational systems that meet business
requirements and integrate with existing business processes. We
describe an industrial deployment and the system architecture
that overcomes challenges in energy efficiency, system reliability,
data fidelity, and flexible integration. We evaluate the
performance of these deployments and consider various system
enhancements that impact performance.
Index Terms- sensor networks, deployments, architecture
I. INTRODUCTION
While sensor networks may be valuable in a wide
variety of applications in the long term, the industrial
market represents an excellent short term opportunity for
this technology. Predictive maintenance of industrial
machinery is one area in which the return on investment
of wireless sensing has been demonstrated [1]. The cost
of deploying a wireless sensor network is both lower
than the cost of a wired sensing capability and easily
offset by the decrease in factory downtime (enabled by
increased sensing granularity). By decreasing the barrier
to entry, wireless sensor networks enable a wider
applicability of predictive maintenance systems.
Prior work has demonstrated the viability of industrial
sensor networking using a proof of concept deployment
[1]. The goal of that deployment was to validate the
overall network architecture, including reliability,
deployability, and cost. In addition, it demonstrated that
by including additional memory and reliable
communications in the sensing platform the overall
network lifetime could be increased.
In this paper, those results are extended,
demonstrating the practicality of an operational
deployment capable of replacing existing predictive
maintenance capabilities on a day-to-day basis. Our
architecture is based on the Intel Mote 2 sensor node [2].
We demonstrate the ability of this platform to provide
Other names and brands may be claimed as the property of others.
1-4244-1268-4/07/$25.00 t2007 IEEE
Thiis full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the IEEE SECON 2007proceedings.
620
additional energy savings through true edge processing,
providing real-time decimation and filtering, FFT, and
spike energy computation [3] on the sensing platform.
We describe the implementation of a generic query
capability, which enables an application agnostic
architecture. Finally, we show that by utilizing an
online, XML-based query mechanism, the sensor
network can be customized to specific application
requirements and also easily integrated as part of
existing business applications and processes.
We describe and evaluate this architecture in the
context of an industrial deployment that provides
predictive maintenance for thrusters and payload pumps
onboard an oil tanker. This deployment was evaluated
side by side with an existing predictive maintenance
capability, which was based on manual data collection.
We evaluate the performance of this system in terms of
its maintainability and overall lifetime, and its ability to
reliably deliver high-quality data. Finally, we evaluate
the impact of various system tradeoffs and optimizations
on system performance.
The contributions of the paper are threefold. First, we
demonstrate techniques for increasing system lifetime
through processing at the edge of the network for certain
applications, enabled by a platform like the Intel Mote 2.
Second, we show the advantage of using XML as an
input/output mechanism for sensor networks to easily
integrate with existing business processes while allowing
for customizable applications and queries. Finally we
identify key issues needed to make industrial sensor
networks a reality - system reliability, integration with
industry systems, and application-specific edge
processing.
The rest of the paper is organized as follows. Section
II describes the application and deployment. Section III
reviews related work, while Section IV describes the
overall system architecture and data flow. A more
detailed look at some specific system components is
presented in Section V. Section VI presents an analytical
model that calculates system lifetime, evaluates the
performance of specific software components and
provides some results on system reliability. Section VII
details lessons learned from the deployment experience
while Section VIII concludes the paper.
II. APPLICATION DESCRIPTION
We have designed, implemented, and evaluated a
sensor network for predictive maintenance of propulsion
and payload equipment onboard an oil tanker. The
deployment consisted of 22 sensor motes and 91 sensors
on 21 machines in a multi-level engine room of a large
oil tanker to facilitate condition-based maintenance [4].
The system was intended to replace the existing manual
collection mechanism, in which people with handheld
devices walk to each sensor point to collect data. This
process required roughly half an hour per machine.
Historically, collected data was imported into the
Enshare Emonitor software package [5], which
performed trend analysis of the vibration signature and
identified any potential problems. This time-consuming
process was performed about once per month.
We replaced the manual data collection process with a
wireless sensor network, which allows more frequent
data collection (hourly in this deployment) while
maintaining integration with the backend Enshare
Emonitor analysis software. Frequent data collections
can greatly increase the probability of detecting faults in
equipment before catastrophic failures occur. While
data collections have previously been automated using
wired networks [6], the cost of retrofitting the
environment with wires can be significant.
In this deployment, each sensor node (mote) collects
data from multiple sensing points per machine. Data
from each sensing point includes time and frequency
domain vibration signals, temperature, RPM, and Spike
Energy (gSE). The battery status of each mote is also
collected.
Multiple engine compartments on the ship were
isolated from each other by water-tight bulkheads. Since
periodic closure of these bulkheads made RF
communication virtually impossible, the software must
function autonomously and adapt to unpredictable
disruptions to network connectivity. In addition, the
metallic surfaces created a multipath-rich environment.
These two factors resulted in an RF-unfriendly
environment.
III. RELATED WORK
Many researchers have reported experiences with
specific sensor networks application deployments. In
621
previous work, we reported on an oil tanker deployment
that demonstrated concept viability [1] and the
reliability, cost, and performance advantages of using a
resource-rich sensing platform. However, the previous
deployment was merely a proof of concept. That system
did not support business class integration or run-time
user configurable data acquisition. In addition, the
system had relatively low data rate (3000 data points per
measurement), which provided insufficient resolution for
diagnostic analysis (1/24th the density of current
offerings in the handheld market). The deployment
collected about 80% of the expected sensor data, with
individual nodes delivering anywhere between 30% and
90% of their data. In contrast, the current system is fully
business integrated, supports complex queries, in-
network data analysis (edge processing, triggering), and
consistently collects 96% of requested data.
There have also been a number of outdoor sensor
network deployments that are targeted to environmental
monitoring. One of the first such deployments was on
Great Duck Island [7], where 43 nodes collected 36 byte
readings every 70 seconds from Leach Storm Petrel
nests. In Sonoma, California another network gathered
data from 33 motes in redwood trees every 5 minutes for
44 days [8]. In Zebranet [9], zebras in a nature preserve
were monitored via collar-mounted motes, forming a
sparse gossip-based communication topology. Other
successful deployments include environmental
monitoring in the James Reserve in California [10], the
ExScal deployment [11], as well as fire monitoring in
the Bitterroot National Forest in Idaho [12].
While environmental monitoring systems are viable
sensor network applications, their requirements are
rather different from our application. For example,
typical environmental sensor networks are designed for
continuous low (scalar) data rate monitoring (tens of
bytes per node per minute), whereas our application
requires bulk data at relatively infrequent intervals
(hundreds of kilobytes per node every hour). Reliable
data collection is not typically critical in environmental
monitoring since it is possible to compensate by over-
sampling (either in time or space). In our application,
losing even a single packet of data (if not properly
recovered) could render a given data collection for a
given machine/unit useless. A clear exception to typical
environmental monitoring applications is the volcano
monitoring application reported in [13]. This application
required continuous data rates of approximately 300
bytes per second and achieved a data reliability rate
above 95%, albeit in the context of a small network (4
nodes).
Other researchers have proposed prototype sensor
network integration frameworks centered on broad/
public access in a green-field technical environment
[14], [15]. In contrast, our work was a real-world
implementation of a business-critical system that
integrated with business applications in a production
environment. We did not have the flexibility to make
changes to the existing data consumer to support
application-side integration. Further, broad access to this
sensor data is not desirable to the business customer;
rather, the goal is seamless integration to business
systems.
There have also been a number of proposals on
different approaches to sensor network query
mechanisms. Yao and Gehrke propose mechanisms for
tasking sensor networks through declarative queries
[16]. TinyDB [17] is an integrated query-based solution
for generic environmental monitoring applications.
TinyDB incorporates network routing, database
querying, node scheduling and data collection and
aggregation features into its highly database-centric
query processing framework. It uses a SQL-like query
language for data acquisition and offers a rich set of
database-centric primitives to support in-network data
processing. In contrast, we propose a query engine that
uses a lightweight, but highly flexible query interface
that is optimized for industrial sensing, as discussed in
Section V.C. The TinyDB approach is also more suitable
for scalar data than bulk data, since there is no easy way
to sequentially schedule queries and data retrieval in the
network. For bulk data, this approach would result in
congestion in the network.
IV. SYSTEM OVERVEW
Rather than building a highly customized sensor
network, we have created a generic distributed
architecture that supports a broad class of industrial
monitoring and sensing applications. Key requirements
of this system include low cost, low upkeep and
maintenance, easy integration with existing enterprise
applications, and highly flexible configuration.
A. System Architecture
Our network architecture is hierarchical, with three
logical layers (Figure 1). The lowest layer, Tier 1,
consists of Intel Mote 2 sensor nodes that interface with
the physical world. Sensor nodes are responsible for
autonomous data acquisition and the forwarding of
acquired data to nodes in Tier 2. To maintain low cost
and power efficiency, nodes at this layer have no long
term storage and limited radio bandwidth and battery
capacity. TinyOS was chosen as the software
framework for this layer, enabling extremely low duty
622
1. Obtain query
parameter 8. Convert to
updates ULF and
import
(XML)
2. Transport
7. Transport
query output data
parameters
(via SFB)
(via SFB)
3. Deliver 6. Generation
I
of output
query
parameters data (XML)
4. Inject 5. Data
queries collection
Figure 1. A three-tiered network architecture and system data flow.
cycle operation. For simplicity, all sensor nodes follow
the same user-configurable sleep/wake schedule, waking
only to sense and transmit sensed data. Communication
at this layer is via an 802.15.4 network.
The Intel Mote 2 features an Intel XScaleg processor
(PXA271), 256 kB of SRAM, 32 MB of SDRAM, 32
MB of Flash, and an 802.15.4 radio (CC2420). The
processor supports frequency and voltage scaling,
enabling the application to dynamically adjust its power
consumption and performance based on its changing
needs. We have taken advantage of this feature by
scaling up the frequency during high computation phases
and scaling it back down during radio communication.
Sensor nodes are subdivided into groups called
clusters. Each cluster forms an ad hoc DSDV routing
tree, rooted by a cluster head that provides the interface
to Tier 2. Routing decisions are made using an end-to-
end success rate metric, as described in [18]. During
each data acquisition cycle, nodes select a cluster either
dynamically, based on the quality of available routes to
cluster heads [19], or via static pre-assignment. This
routing structure allows data from sensors (which make
up the bulk of traffic) to efficiently and reliably reach the
cluster head, routing around any obstructions or low
quality links. Flooding is used for communication from
cluster heads into the network. Each cluster operates on
an orthogonal communication channel (a different radio
frequency), allowing communication in each cluster to
proceed in parallel, without contention from other
clusters. Communication at this tier utilizes a data
representation, data transport, and query and control
capability that is optimized for low bandwidth and high
packet loss rates.
The middle layer, Tier 2, consists of the cluster heads
from Tier 1. Each cluster head leads autonomous data
acquisition for a cluster and provides a modest amount
of persistent storage (a flash memory card) for incoming
data. Each cluster head is composed of a line-powered
Stargate embedded PC platform [20], running Linux,
coupled via a UART interface to an Intel Mote 2,
running TinyOS. The Intel Mote 2 provides the
802.15.4 radio for communication with sensor nodes.
The network stack is split between the Stargate and the
Intel Mote 2 to maximize reuse of TinyOS software
components, primarily MAC, network, and query
components.
Each cluster head controls the formation of sensor
node clusters, directs the sleep/wake cycle of the cluster,
initiates sensing on each sensor node in turn by injecting
queries, and provides persistent storage for resulting
sensor data. To provide fault tolerance and allow for
periodic disconnection (as might occur when the
watertight doors of a ship are closed), a disconnection
tolerant transport over an IP network (over Ethernet or
802.11) is used for communication between cluster
heads and the top tier of the network. When the
communication channel is available, cached sensor data
is transmitted to, and configuration updates are received
from, the top tier. Devices at this tier receive instructions
in an XML format that describes the sensing task
required and export an XML representation of the
requested data to applications. XML was chosen for
ease of integration with existing enterprise applications.
The top layer, Tier 3, provides a standards-based and
IP addressable interface to the enterprise. This tier also
provides the management and diagnostic interface to the
sensor network. Finally, this tier provides long term
archival storage (typically hard disk based) for data
received from the sensor network.
The three-tiered architecture described above allows
the sensor network to balance several competing goals.
The top tier provides a persistent interface to the
enterprise network, while still allowing the second tier to
be disconnection tolerant and the lower tier to conserve
power much of the time. In addition, the top tier
provides standards compliance (XML data structure, and
IP communication), while hiding the power-optimized
protocols and data formats used in the lower tiers.
Finally, the storage capacity of individual nodes grows
from the bottom tier to the top, allowing a balance
between reliability, cost and energy constraints.
In the above architecture, instructions to the sensor
network are simply sensing parameters, abstracting away
the specifics of the application. This approach
minimizes the amount of custom software and
preprocessing needed within the sensor network itself.
B. Data Flow
A high level view of data flow through the system is
623
shown in Figure 1. Queries are configured using a GUI
which runs in the Enterprise network. The GUI creates
an XML representation of queries and other network
configuration parameters (described in Section V.A),
which is then pushed to all the cluster heads using a file-
oriented disconnection tolerant transport called the
Secure File Bridge (SFB).
Once the input XML reaches a given cluster head, it is
serialized into a binary format and stored in a database
for transmission to cluster nodes. This compact
representation is used in the lowest layer of the network
architecture for energy, communication, and storage
efficiency.
Each cluster head is responsible for querying and
gathering data from the sensor nodes in a given cluster
during each data collection cycle. At the beginning of
each data collection cycle, nodes form clusters around
the cluster heads based on an end-to-end success rate
metric [18]. During the cluster formation process, all
nodes in the network operate on a default radio channel
which is statically defined when the network is
provisioned. Once the clusters are formed, however,
cluster members switch to a cluster-specific radio
channel. The cluster formation process is described in
detail in Section V.B.
After the channel change, each cluster head
individually queries each node in its cluster, sending it
the relevant query parameters and gathering the sensor
data from the node. Details of the query protocol are
described in Section V.C. The resulting data is converted
into an XML format at the cluster head and cached for
transmission to the enterprise network. When
connectivity allows, the XML data files are pushed to
the Enshare server using the SFB disconnection tolerant
transport.
After all nodes in a cluster are queried, the cluster
head notifies the cluster of the time designated for the
next data collection, after which all the sensor nodes go
to sleep. When the next collection cycle occurs, sensor
nodes wake up and, using the default communication
channel, begin the process of cluster formation again.
V. ARCHITECTURAL DETAILS
A. Integration with External Systems
Sensor networks, particularly those intended for
industrial applications, should have an interface that fits
into existing business processes. This is particularly true
for sensor networks that are replacing an existing
technology or process. In our particular deployment, it
was important to easily integrate with the customer's
existing commercial off-the-shelf (COTS) industrial

Gateway
Cluster Head wD
monitoring,tool z [5]. C
Enshare Emonitor1gRggg1CwI
.>$,$,$,$,
NodeBB X" Sf_
Figure 4. The phases of cluster formation and data acquisition.
monitoring tool Enshare Emonitor [5]. Changes to
Enshare were not possible. Our integration design
focused on the following requirements to allow broad
adoption with minimal overhead: require no changes to
the existing production environment, maintain an open
architecture for easy COTS and custom-application
integration, and leverage existing standards and data
formats. Our solution implements an XML interface to
the sensor network for both input and output. The
interface is application independent, and could be
applied to a wide variety of different sensing tasks.
Figure 2 shows a simplified example of the XML
input format. At a high-level the input XML provides
the settings for each active cluster head and a list of
queries to be performed by specific sensor nodes. The
cluster specification identifies and names each cluster
head in the network and controls the collection
frequency, operating channel, and other cluster-centric
parameters. For each node, the input XML specifies
both a logical and physical address of the node (allowing
physical replacements when failures occur), and the
sensing task that the node should perform, in the form of
queries. In addition, the node specification can provide
default values to be used in queries and enable or disable
node health monitoring.
Each query is made up of one or more sensing tasks.
Each sensing task specifies the channel to be sensed, the
Figure 2. Example Input XML
type of sensing to perform (e.g., temperature, vibration),
----
the post processing functions required, and calibration
and scaling parameters. A given sensing task can also
specify that multiple measurements should be
S to
performed, and an average result returned. Multiple
X1
sensing tasks form a query group. One measurement
within a query group can be configured to be a trigger.
Subsequent measurements are skipped if the trigger
value is not reached. For example, vibration
measurements can be skipped if a motor's shaft is not
spinning (and thus the motor is not operating).
Individual sensing tasks within a query group can be
performed simultaneously, allowing values to be
correlated in time. Finally, since many sensing tasks use
similar parameters, the input XML also allows reusable
sets of query parameters to be defined. In summary, a
query is a set of generic instructions for controlling a
sensor driver and setting input parameters.
The output XML format provides a similar generic
format for representing the resulting output data from a
sensor node. Figure 3 shows a simplified example of an
XML output record. The XML output specifies one or
more output values, as well as metadata that describes
the sensing context. The metadata includes the identity
of the sensor that produced the data, the query that
triggered the measurement, the unit of measurement, and
other attributes of the output data. As required, this
generic output XML format can then be converted into a
specific input format for a target application. In our
particular application, we converted the XML output
into a proprietary input format for Enshare Emonitor.
B. Cluster Formation
As described in Section IV, during each data
collection cycle, sensor nodes form clusters that operate
independently to deliver requested data to a cluster head.
The goal of clustering is to allow each sensor node to
find a high-quality route to a cluster head and for each
node to participate in sensing exactly once per cycle.
Cluster formation follows the phases shown in Figure 4.
>1C
Figure 3. Example output XML
624
 All cluster heads and sensor nodes begin cluster
formation at the same point in time across the entire
network. The cluster head reads the configuration stored
in persistent storage. If a radio channel has been
assigned to a given cluster head, that cluster head begins
sending DSDV route updates to the sensor nodes. Per
the DSDV protocol, sensor nodes rebroadcast route
updates whenever they choose a new route. Route
selection is based on an end-to-end routing metric that
optimizes the packet delivery rate. Because route
updates are being broadcast from all cluster heads at the
same time, all sensor nodes are able to select the lowest
cost route to reach any cluster head. At this point, each
sensor node thus knows the cluster to which it belongs.
Each cluster consists of a tree of sensor nodes, rooted at
a cluster head.
Once all nodes have had an opportunity to select a
cluster, the channel change phase begins, in which a
beacon sent by the cluster head instructs all cluster
members to switch to the channel assigned to the cluster
head in the configuration. This message could be
integrated with routing messages, but for simplicity a
separate message is used.
At this point, all sensor nodes have a route to the
cluster head and are operating in the designated channel.
However, the cluster heads do not know which sensor
nodes have chosen this cluster. During cluster
identification, each node sends a traceroute message
along the routing tree to the cluster head. Each node
along the route appends its own identity to this message.
When received by the cluster head, these messages
identify the membership and topology of the cluster.
The cluster head uses this information to send specific
queries sequentially to each node in the cluster. After
the querylresponse phase, the cluster is instructed to
return to the default operating channel, go to sleep, and
wake at a future time for the next cluster formation.
C. Query Mechanism
The query mechanism is driven by the cluster head
during each data collection cycle. It serializes complex
query parameters specified in the input XML' and
reliably polls each sensor in the cluster. The query
parameters are encoded and packed into a compact
binary string consist of multiple segments. Each
segment is a triplet [type, length, value] that encodes a
query parameter as shown in Figure 5. Using this
encoding method, complex query specification (in XML
Note that not all parameters specified in the input XML go into the
network. For example, in Figure 2, the parameters targeted for cluster head
configuration, such as "CollectionPeriod, CollectionPeriodUnit,
OperatingChanner' stay at the cluster head.
625
CIompact query structure
Ipe enci laue
Params string
Figure 5. Packing and unpacking a query.
format) of a few hundred bytes in length can be packed
into a binary string of less than 50 bytes. This approach
greatly reduces the size of the query messages, reducing
communication overhead and thus increasing the
operational lifetime of battery-powered sensor nodes.
During each collection cycle, a query engine running
on the cluster head accesses the query parameter
database, and for each cluster member identifies the
types of queries required, schedules query transmission,
serializes the queries into a compact binary format, and
reliably delivers queries with appropriate parameters.
The query mechanism uses a reliable handshake to
transmit queries and receive data from each node. The
cluster head queries one node at a time. Following each
successful data capture according to the specified query
parameters, the data is immediately transmitted to the
cluster head via a reliable bulk transport. This process
ensures a congestion-free data transmission environment
for bulk data retrieval.
Unlike TinyDB, our query mechanism is acquisition-
centric rather than database-centric. Instead of
specifying queries in a generic SQL-like language, as in
TinyDB, our queries include a set of well defined
parameters (Figure 5) that configure the sensor hardware
for data acquisition on the sensor node. Our query
engine is lightweight but yet generic enough to support a
broad class of industrial sensing applications.
The generic nature of the query language is enabled by
a generic sensor driver interface. This interface abstracts
the sampling parameters (e.g., sampling rate, number of
samples, trigger source, target of triggers, trigger logic,
etc) from the low level implementation. The query
processor on the sensor node unpacks the query
parameters into a well-defined data structure (Figure 5)
and passes the structure to the sensor driver through this
interface. Post-processing functions can be specified by
the calling application (e.g., perform a 1024-point FFT
on the acquired data). By using different parameters with
the same physical channel, multiple sensors of different
types can be overloaded on top of a specific physical
sensor channel. By specifying multiple channels in the
same query, multiple data channels can be captured
simultaneously. Finally, a flexible notion of triggering is
provided, similar to that of an oscilloscope.
D. Low Power Network Association
In our architecture, sensor nodes sleep and wake
together. However, a new node joining the network
(called an orphan node) would not know the sleep/wake
cycles of the network until it participates in one cycle of
cluster formation and data collection. A node is
typically an orphan when it is newly installed or when its
batteries are changed. A low power network association
algorithm allows orphan nodes to align with the network
sleep/wake cycles while conserving their energy,
compared to the simple approach of staying awake till
the next data collection cycle.
As shown in Figure 6, an orphan node duty cycles as it
searches for a network. The duty cycle (on + off) period
is chosen so that it is impossible for an orphan node to
sleep through a cluster formation process. At the same
time, the "on" time is the minimum time required for the
orphan node to recognize that a cluster formation is in
process, which would stop its duty cycling.
A second feature is that cluster heads which are line
powered transmit periodic messages while the network
is sleeping. These messages indicate the time remaining
until the next network discovery period. Thus any
orphan node that is within radio range of a cluster head
can immediately align with the network sleep/wake
cycle. Note that this only benefits nodes that are within
one hop of a cluster head; however, those nodes were a
substantial portion of the network in our deployment.
VI. SYSTEM PERFORMANCE
A. Analytical Model
When designing the system, we wanted to estimate the
effect of various system parameters on energy
consumption and battery life. To achieve this goal, we
built an analytical model that estimates battery life and
system performance. This model was needed for two
reasons. First, we wanted to enable the customer to make
informed tradeoffs when configuring the system
parameters (e.g. battery replacement frequency versus
collection frequency). Second, we wanted to get a better
insight into the inefficiencies of the system and make
system level optimizations in the final phases of the
development.
We divided the application into multiple operating
modes and measured the average power consumed by
the application in each mode. In sleep mode the
626
Orphan Node:
Discovery Time Node is associated with the net ork
Awake
Sleeping *
Wake
Time
Network:
Awake
Sleeping
Discovery Time Querying and data
collection
Figure 6. A timing diagram for low-power network association.
processor is in deep sleep, and the radio and sensor
board turned off
are = 0.548 mA). During sensor
warm up, the analog portions of the sensor board, the
processor (in 13 MHz mode) and radio are all on ('warmup
= 150.2 mA). In the data collection mode, the analog
and digital portions of the sensor board are on, the radio
is also on, and the processor is bumped up to 104 MHz
to satisfy the high computing demands of our data
processing algorithms (Icollection 220.9 mA).
Collectively, the sensor warm-up and data collection
modes are called the active mode, since the node is
actively collecting data, and the average current
consumption is calculated by taking into account the
current consumption and time spent in these two modes
(Iactive = 157.14 mA). In the data transfer or passive
mode, the sensor board is off while the processor
(running at 13 MHz) and radio are on (Ipu=ve 57.9
mA). Note that for simplicity, the data transfer mode is
assumed when the mote is either routing data for other
nodes or simply waiting for other nodes to finish
transferring data. Hence the average current
consumption of a node is given by,
I Tsleep Isleep +Tactive iactive +Tpasive passive
avg T T +T (1)
1sleep +Tactive passive
Next, we estimated the time spent in each mode
during the collection cycle (T=1 hour). The sensor
warm-up mode was a system parameter dictated by the
sensor board and the type of sensor (Twarm=p 6.5 sec.).
To estimate the time spent in the data collection mode,
we picked typical sampling parameters and data
processing requirements provided by the customer. We
estimated the time spent in the data transfer mode using
the average sustained network data rate. Since the
average rate is clearly dependent on network topology,
we averaged the data rate from several configurations
(measured to be 4 kbps). In the case of the 10 node
cluster (Nm=otes10), with 5 sensors per node (Nsensors=5),
the total active time (Tactive) by this method was
calculated to be 168 seconds while the passive time
(Tpassive) was 1581 seconds. Finally, we measured the
network formation time (Tnetform = 65 sec.).
Tactive = Nsensors (Twarmup + Tcollection) (2)
Tpassiv = (Nm'otes-1) Tctive+ntoter lVensors lansfe,Tiettorn
e
_forlt~~~~e~r~trn~e
Tileep= T-(-noteN Tactive+Nmotes Nsensors Ttransfi+etToret
Using the above values and the battery capacity
(16000 mAh), the model was validated by performing
multiple runs in different environments. In fact, the on
board ship data demonstrated that after 15 days of
operation, the battery voltage had dropped from 4.4V to
3.8V, which is very close to the 3.75V value predicted
by the model. The duration of a collection cycle is
clearly constrained by the total active time as a lower
bound. If the application requires a collection cycle
smaller than Tactive, some of the parameters mentioned
above will need to be tweaked to obtain a shorter Tactive
period.
B. Effect ofEdge Processing
One of the differentiating features of this system is its
edge processing capability. The Intel Mote2 platform
with its Intel XScaleg processor has high energy
efficiency for computationally intensive applications.
When considering energy consumption, it is important to
evaluate not only the power consumption of the
processor, but also the amount of time it takes to
perform a certain task. The actual energy consumed is
the product of the two. The tradeoff between edge data
processing (at the point of sensing) and transmission of
raw data to the backend for processing depends on
energy consumption, latency constraints, and data
sharing needs. In the case of this application, the data
from different machines do not need to be correlated,
and there is no tight latency constraint. Hence, the main
issue in this implementation was one of energy
optimization. We chose the edge processing approach
because it allows the amount of data flowing through the
system to be reduced considerably, while adding
minimal compute cycles to support data reduction,
resulting in a significant total energy reduction. The
resulting energy savings is a function of the energy
efficiency of both the radio and the processor.
In this application, we needed to monitor different
machinery with varying operating speeds, thus forcing
the data acquisition system to support a wide variety of
sampling rates. The maximum sampling rate of 100 kHz
was dictated by the spike energy (gSE) calculation,
which demodulates resonant frequencies in the 10's of
kHz range. While it is possible to provide a variable
sampling rate in either software or hardware, supporting
a wide range of sampling rates (along with all the
corresponding filters) in hardware significantly increases
(3) system complexity. Instead, our implementation uses
hardware with a fixed sampling rate (at the maximum
required: 100 KHz), with down-sampling and
(4)
decimation filtering performed in software when needed.
It is important that the decimation (i.e. reducing
sampling rate) must be preceded by an anti-aliasing filter
in order to satisfy the Nyquist criteria [21]. The anti-
aliasing filters are designed to have 0.01 dB passband
ripples, 80 dB stopband attenuation, and passband width
of 0.4 times the target sampling frequency.
A typical example is shown in Table I for a machine
with anfiiax (maximum frequency span) of 1 kHz. In our
example, a 1600 line FFT measurement has been
requested, with magnitude spectrum averaged over four
separate time frames. The first column assumes that the
raw data is being captured at the node and transferred to
the backend, where it is processed. In the second
column, the down-sampling and filtering are performed
in the sensor node. In this case, transmitted data was
reduced by 39x, at the cost of less than 2 additional
seconds of processing time. It results in a 27x reduction
in energy consumption. A further reduction in energy
consumption is obtained by performing the FFT analysis
and averaging on the sensor node. This reduced the
amount of data transmitted (particularly through
averaging across multiple samples), while increasing
processing time by less than one second. The third
column shows 250x reduction in data transmission and
48x reduction in energy consumption compared to not
doing processing at the edge.
Note that we made an optimistic assumption of data
TABLE I: ENERGY EFFICIENCY OF DATA PROCESSING AT TUE EDGE
Backend Edge Down- Edge Down-
Down- sampling sampling + FFT
sampling
Final Sampling
Rate (KH-z)
100 2.56 2.56
Bytes Transmitted 1280 32.768 5.12
(kB)
Data Transfer 204.8 5.2 0.8
Time (sec)
Data Processing 0 1.76 2.48
Time (sec)
Total Energy (mJ) L 38011 1424 J 787
627
transmission throughput (50 kbps). Much greater energy measurements per channel (velocity and Spike Energy).
savings from on-node processing is expected in more The customer was interested in collection cycles
realistic scenarios. between once per day and once per hour. In the case of
Energy efficient signal processing was a key criterion a single collection per day, the 3 D-batteries will last for
in processor selection. The Wireless MMXTM (WMMX) 354 days, due to a low duty cycle of 2%, resulting in an
instructions of the PXA27 1 processor proved to be of average current of 1.9 mA. In this case, the average
great importance in energy efficiency of the power consumption is highly influenced by the sleep
implementation. In order to keep memory consumption current (0.5 mA). This effect can be seen by examining
within the limits of internal SRAM (external RAM tends the difference between the actual battery life plot and the
to be much more power hungry), the decimation filtering battery life plot assuming the sleep current goes to 0.
must be performed in real time, without loss of samples. However if we consider the case of 24 collections per
Furthermore, it is desirable to meet this real time day, the duty cycle rises to 49%, resulting in an average
requirement with as low a CPU clock frequency as current of 33 mA. In this case, the power consumption
possible to minimize power. Figure 7 evaluates two is dominated by the active current.
different implementations of the same multi-stage D. Effect of Low Power Network Association
decimation algorithm: 1) a C implementation with
object code generated by GCC and 2) a hand-optimized To understand the impact of low power network
WMMX implementation. The difference between these association on overall performance, first consider the
two implementations is about an order of magnitude in duty cycling of orphan nodes. If the network wakes
favor of WMMX. While part of the difference is due to every T seconds for data collection, and new nodes can
hand-coded assembly optimizations, a large part is due join at any time, the average wait by an orphan is T/2. If
to the WMMX vector multiply-and-accumulate the time an orphan node remains awake to check for
operation which performs four 1 6x16 multiplications network formation is Tdc,awake and the time it sleeps is
and sums the results in a single cycle, greatly Tdc,sleep, the energy saved by duty cycling is given by
accelerating the filtering operation. The performance of
the WMMX implementation enabled real time E I- Ppassive 0 T Tdc,awake
computation at the lowest clock speeds of 13 MHz, savings-2 + (5)
while the C implementation would require 104 MHz.
dc,awake dc,sleep)
C. Effect of Collection Frequency For a cluster of 10 nodes and a data collection period
The typical collection frequency for predictive of 24 hours, duty cycling increases network lifetime by
maintenance varies from a few collections per day to one 2.2%, while the gain in lifetime if collections are weekly
collection per week. We designed our system to allow is 13.388%. The second power saving mechanism is the
dynamic collection frequency adjustment. periodic broadcast of sleep messages by the cluster head
Figure 8 shows the effect of the collection frequency nodes. Since any node that is within radio range of a
on the battery life for a typical cluster configuration of cluster head can immediately synchronize with the
10 nodes, 5 channels per node, and two vibration network, the energy savings in this case would be
120

0 100
a)
0
U)
en

80 600
Q EC
en 60 500
L- * WM MX
Battery Life
0 400
40
0~
._
realtime > 300
20 deadline Battery Life as
13 MHz .) 200 sleep current -> 0
0
z e o0 '0
0,' zoo/Us m
loo
EC 89.4 83.5 51.6 50.2 32.9 41.8 28.9 26.6
0
IMWMMX 10.9 5.9 5.2 4.1 3 3.8 2.9 2.4 1 3 6 9 12 24
Decimation factor Number of Daily Collections
Figure 7. C versus WMMX implementations Figure 8. Collection frequency vs. battery life for a 10-node cluster

628

1 In those rare cases, the failed sensor nodes would miss
savings 2 passive
The result is an increase of 4.6% in the lifetime for a
cluster of 10 motes collecting data every 24 hours and
30.4% for a weekly collection period.
E. Evaluation of System Reliability
The reliability of the system was measured in detail in
a testbed environment that approximated the oil tanker
environment over 8 weeks of continuous operation. The
testbed consisted of 10 sensor nodes deployed in three
clusters in an industrial space that provided chilled water
and other environmental support for a large office
building. The steel-and concrete construction and large
machinery provided a close approximation to the
deployment environment.
Depending on the machine it was attached to, each of
the 10 sensor nodes performed either 17 or 21 data
acquisitions every 90 minutes. The amount of data to be
collected from each mote was either 78.3 kB or 97.9 kB.
Figure 9 shows a histogram of successful data
collections for each node over the 8 week test. A
successful collection was defined as a cycle in which all
acquisitions for a sensor node were completed and
successfully transferred. In Figure 9 the average data
collection rate was 95.6%. System reliability can also be
evaluated by the collection reliability factor which is the
reliability with which we can expect a single acquisition
to complete. The collection reliability factor for the
testbed over the 8 week test was measured to be 98.40%.
We identified two major issues that contributed to the
failure of an acquisition or data collection. First, the poor
radio communication channel, which was time-varying,
sometimes caused specific nodes to fail to associate with
the cluster head, thus missing a query cycle.
Furthermore, a very poor communication channel could
cause the reliable bulk transport protocol to fail when the
maximum number of retransmissions was exceeded.
Second, on rare occasions, sensor nodes became
100.00%
98.00%
96.00% Mean =956%
94.00%
92.00%
90.00%
88.00%
86.00%
84.00%
nl n3 n5
n7 n9
Configured Mote
(nl-nlO)
Figure 9. Histogram of the percentage of successful data collection cycles
for each mote
unresponsive due to unknown reasons at random time.
(8) queries. However, a watchdog timer would later reset
the node and automatically restore it to normal
operation.
VII. LESSONS LEARNED
Our system performance evaluation uncovered a few
shortcomings of the design and identified some potential
solutions. We discuss briefly some of these results
below.
The query mechanism described above tightly couples
sensing and data delivery to a query. While it is
beneficial to perform bulk data transfers sequentially to
reduce the contention for the RF channel, sensing
(sensor warm-up and data acquisition) can clearly occur
in parallel across all nodes within the system. This
approach could reduce the length of a collection cycle,
hence reducing the total energy consumption
significantly, particularly in large clusters. A simple
approach would be to perform two passes across the
nodes in each cluster. In the first pass, queries would be
delivered, and in the second pass, results would be
gathered. We estimate that for the oil tanker
deployment, this would increase the lifetime from 20.5
days to 61 days, an increase of approximately 197%.
A second shortcoming is the simple sleep mechanism
that we implemented in this application, which requires
all nodes in the cluster to be awake for the entire
collection cycle. As the cluster size grows, nodes have
to wait longer for other nodes to finish, hence increasing
the passive time and reducing the opportunity for nodes
to return to the lowest energy mode (sleep mode). Figure
10 shows the energy spent in the three phases as a
function of the number of nodes within the cluster
(single collection per day case). Clearly the passive
energy grows considerably with the cluster size.
Implementing a more sophisticated sleep mechanism
could recover some of this passive energy. However,
because some nodes need to be awake to serve as
200000
) 150000 Sleep Mode
E
Passive Mode
> 100000
Active Mode
50000
0
10 8 6 4 2
Number of nodes in a cluster
Figure 10. Effect of cluster size on energy consumption
629
routers, the potential gain may be limited.
VIII. CONCLUSION
This paper described the development and deployment
of an operational sensor network for predictive
maintenance of equipments onboard an oil tanker. The
system used XML for flexible input and output, enabling
easy integration into existing business processes and
providing an application agnostic way to configure the
sensor devices. A model was presented to compute the
lifetime of the system, which was also used to evaluate
the performance tradeoffs of different system features
and to aid the customer in making informed system
configuration choices. The paper also demonstrated the
energy advantage of features such as edge processing
and low power network association, leading to
substantially increased network lifetime.
REFERENCES
[1] L. Krishnamurthy, et al., "Design and Deployment of Industrial Sensor
Networks: Experiences from a Semiconductor Plant and the North Sea,"
Proc. of the 3rd International Conference on Embedded Networked
Sensor Systems (SenSys), Nov. 2005.
[2] R. Adler, et al., "Demo Abstract: Intel Mote 2: An Advanced Platform
for Demanding Sensor Network Applications," Proceedings of the 3rd
International Conference on Embedded Networked Sensor Systems,
SenSys'05, San Diego, CA, Nov. 2005.
[3] M. Xu, "Spike EnergyT Measurement and Case Histories," ENTEK
IRD International Corporation Technical Report, Online:
http://domino.automation.rockwell.com/applications/gs/region/EntekWe
bST.nsffiles/Xu99.pdf/$file/Xu99.pdf
[4] A. B. Holmes, "Making Maintenance Invisible," A-B Journal, Putman
Media, Sept. 2004.
[5] Rockwell Automation, "EmonitorR An Integrated Suite of
Maintenance Data Functions," http://www.rockwellautomation.com/
services/conditionmonitoring/emonitor.html
[6] Bentley Nevada, "Asset Condition Monitoring Systems,"
http://www.bently.com/prod/products/monitoringsystems.htm
[7] R. Szewczyk, A. Mainwaring, J. Polastre, J. Anderson, and D. Culler,
"An Analysis of a Large Scale Habitat Monitoring Application,"
630
Proceedings ofthe SecondACM Conference on Embedded Networked
Sensor Systems (SenSys '04), Baltimore, MD, November 2004.
[8] G. Tolle, et al., "A Macroscope in the Redwoods," Proc. of the 3rd
International Conference on Embedded Networked Sensor Systems,
(SenSys'05), San Diego, CA, Nov. 2005.
[9] T. Liu, C. Sadler, P. Zhang, and M. Martonosi, "Implementing Software
on Resource-Constrained Mobile Sensors: Experiences with Impala and
ZebraNet," Proc. ofthe Second International Conference on Mobile
Systems, Applications and Services, Boston, MA, June 2004.
[10] R. Szewczyk, E. Osterweil, J. Polastre, M. Hamilton, A. Mainwaring,
and D. Estrin, "Habitat Monitoring with Sensor Networks,"
Communications of the ACM, Vol. 47, No. 6, June 2004.
[11] A. Arora, et al., "Exscal: Elements of an Extreme Scale Wireless Sensor
Network," Proceedings of the 11th IEEE International Conference on
Embedded and Real-Time Computing Systems and Applications (RTCSA
2005), Hong Kong, August 2005.
[12] C. Hartung, C. Seielstad, S. Holbrook, and R. Han, "FireWxNet: A
Multi-Tiered Portable Wireless System for Monitoring Weather
Conditions in Wildland Fire Environments," Proceedings of the Fourth
International Conference on Mobile Systems, Applications, and Services
(MobiSys 2006), Uppsala Sweden, June 2006.
[13] G. Werner-Allen, J. Johnson, M. Ruiz, O. Marcillo, J. Lees, and M.
Welsh, "Monitoring volcanic eruptions with a wireless sensor network",
In Proc. EWSN, Jan. 2005.
[14] J. Shneidman, P. Pietzuch, J. Ledlie, M. Roussopoulos, M. Seltzer, M.
Welsh, "Hourglass: An Infrastructure for Connecting Sensor Networks
and Applications," Harvard Technical Report TR-21-04.
[15] P. Gibbons, B. Karp, Y. Ke, S. Nath and S. Seshan. IrisNet: An
Architecture for a Worldwide Sensor Web. Pervasive Computing, Oct-
Dec 2003.
[16] Y. Yong and J. E. Gehrke, "The Cougar Approach to In-Network Query
Processing in Sensor Networks," Sigmod Record, Volume 31, Number
3, September 2002.
[17] S. Madden, M. Franklin, J. Hellerstein, and W. Hong, "TinyDB: An
Acquisitional Query Processing System for Sensor Networks," ACM
Transactions on Database Systems, Vol. 30, No. 1, March 2005, pp 122-
173.
[18] M.D. Yarvis, W.S. Conner, L. Krishnamurthy, J. Chhabra, B. Elliott,
and A. Mainwaring, "Real-World Experiences with an Interactive Ad
Hoc Sensor Network," Proceedings International Workshop on Ad Hoc
Networking (IWAHN 2002), Vancouver, B. C., Canada, August, 2002.
[19] M. Yarvis, N. Kushalnagar, H. Singh, A. Rangarajan, Y. Liu, and S.
Singh, "Exploiting Heterogeneity in Sensor Networks," Proceedings of
the IEEE International Conference on Computer Communication
(INFOCOM2005), Miami, FL, March 2005.
[20] Crossbow Technology, Inc., "Stargate X-Scale Processor Platform
Datasheet," Online: http://www.xbow.com/Products/Product_pdf files/
Wireless_pdf/Stargate_Datasheet.pdf
[21] A.V. Oppenhein and R.V. Schafer, Discrete-Time Signal Processing,
2nd Edition. Englewood Cliffs, NY: Prentice Hall, 1999.