You are on page 1of 14

Fog Computing

A New Era for Cities with


Fog Computing
The Quadruple Silo (QS) problem refers to four silos that result from deploying
commercial solutions for smart cities: physical, data, and service management
silos, and their implications in administrative silos. As this analysis of a Barcelona
fog computing initiative shows, a converged cloud/fog paradigm not only helps
solve the QS problem, it also meets the requirements of a growing number of
decentralized services where traditional cloud models fall short. In addition to
exposing cases where fog computing is a must, this article shows that reasons
for deploying fog are centered much more on operational requirements than
on cloud-related performance issues.

T
Marcelo Yannuzzi, Frank he design of platforms capable of plausible models. Among these is an
van Lingen, Anuj Jain, and managing urban services holis- increasingly popular model that we
Oriol Lluch Parellada tically and uniformly across dif- recently demonstrated in Barcelona.
Cisco Systems ferent city departments is at the heart Our model offers a common and distrib-
of the innovations that smart cities are uted data fabric across the city for mul-
Manel Mendoza Flores expected to bring. However, the rapid tiple departments (that is, tenants), and
Barcelona City Council evolution of technologies developed is based on a platform that seamlessly
for capturing the opportunities around combines cloud computing and fog com-
David Carrera and Juan smart cities has led to the proliferation puting (for more details on fog, see www.
Luis Prez of proprietary and application-specific openfogconsortium.org).
Barcelona Supercomputing Center systems, or siloed solutions, for services Here, we analyze the technical chal-
such as parking, lighting, traffic, energy lenges that cities are facing and outline
Diego Montero management, and video analysis. Each the design principles, main results, and
Technical University of Catalonia of these systems is typically deployed as lessons learned after embarking on an
BarcelonaTech a vertical solution from the things up ambitious co-innovation project with
to the cloud, giving rise to independent the Barcelona City Council, several
Pablo Chacin management ecosystems that show poor industrial partners, and academia. In
Sensefields integration with each other. particular, we examine the requirements
The collateral effects of deploy- of a growing number of urban services
Angelo Corsaro ing dedicated solutions are starting to where cloud-centric approaches are
PrismTech become apparent especially among insufficient and fog computing is man-
early adopters. For instance, cities that datory. We describe use cases (UCs) that
Albert Olive invested in vertical solutions for mak- were implemented and demonstrated in
Schneider Electric ing certain urban services smarter are Barcelona, providing supporting evi-
now shifting their focus to other more dence for fog computing. To the best of

54 Published by the IEEE Computer Society 1089-7801/17/$33.00 2017 IEEE IEEE INTERNET COMPUTING
A New Era for Cities with Fog Computing

Related Work in Smart City Platforms

I n 2013, Nice, France launched the Connected Boulevard,1 a


smart city platform to optimize all aspects of city manage-
ment, including parking, traffic, street lighting, waste disposal,
(US), Tianjin (China), and Singapore (for details, see www.tech-
nologyreview.com/business-report/cities-get-smarter). The
approaches differ in each city, but its clear that all will be built
and environmental quality. Although the platform included around secure services distributed between the edge and data
advanced data-sharing capabilities based on the Data Distribu- centers, and that all will be managed in a coherent way. 3 For
tion Service (DDS) standard, it didnt address enhanced service more details, see www.openfogconsortium.org.
lifecycle management or standardized models for edge nodes.
A different approach, SmartSantander, 2 focused on a Euro- References
pean facility for research and experimentation of architectures, 1. A. Corsaro, Connected Boulevard Its What Makes Nice, France, a Smart
technologies, and applications for smart cities, while FIWARE City, blog, 9 Sept. 2014; http://blog.iiconsortium.org/2014/09/connected-
(www.fiware.org) aimed to build an open ecosystem to ease boulevard-its-what-makes-nice-france-a-smart-city.html.
the development of new applications for cities and other 2. L. Snchez et al., SmartSantander: Experimentation and Service Provision
domains. However, neither SmartSantander nor FIWARE in the Smart City, Proc. 16th Intl Symp. Wireless Personal Multimedia Com-
focused on fog computing. munications, 2013, pp. 16.
Other smart city-related projects include those announced 3. Y.C. Hu et al., Mobile Edge Computing: A Key Technology towards 5G, white
for Songdo (South Korea), Masdar City (Abu Dhabi, United paper, ETSI, 2015; www.etsi.org/images/files/ETSIWhitePapers/etsi_wp11_
Arab Emirates), Paredes (Portugal), Manchester (UK), Boston mec_a_key_technology_towards_5g.pdf.

our knowledge, our project is the first to deliver dashboards exploiting a rich set of APIs
a complete cloud/fog architecture that addresses through which the solution provider exposes
the challenges currently facing many cities. real-time analytics, billing information, ser-
vice management functions, and so on (for
The Quadruple Silo (QS) Problem more on this, see, for example, www.world-
The predominant model for installing solutions sensing.com).
in urban spaces is based on the deployment of
vendor-specific devices at the edge. The model In such scenarios, the in-ground sensors often
typically includes sensors, control and actua- rely on Low-Power Wide-Area Network tech-
tion elements, and the corresponding gateways. nologies such as LoRa (www.lora-alliance.org)
In many cases, the data produced at the edge and therefore dont require the allocation of
can be backhauled directly to the cloud, where compute resources close to them. An alternative
different types of analytics and data manage- is to utilize more sophisticated sensors, such as
ment processes can be performed in a central- cameras (as in Figure 1), with the advantage
ized way. However, other cases require more that they can be used for parking and video
elaborated installations. A common practice is surveillance simultaneously. To detect vehicle
to place compute nodes closer to the data pro- presence, these solutions rely on video analyt-
ducers, in addition to the resources required in ics, which typically run in resources embedded
the cloud. Existing solutions in the marketplace in the cameras themselves.
typically come in the form of dedicated (propri- As Figure 1 shows, commercially available
etary) nodes, which are installed at the network solutions for other services follow similar pat-
edge for a specific application. terns, both in the type of deployment and their
Figure 1 illustrates different vertical solu- management structure. As we describe in our
tions combining these elements. For example, a UCs later, this is true of vertical solutions for
parking system might be composed of energy management (that is, systems for moni-
toring and controlling the power of different
self-powered sensors installed in the pave- devices in public spaces; see www.schneider-
ment for each parking spot, along with gate- electric.com/ww/en) and systems for traffic
ways for collecting the sensors status within monitoring and regulation, such as those devel-
their coverage range; oped by Sensefields (www.sensefields.com).
a back-end platform hosted either in a public Overall, production-ready solutions for mak-
or private cloud for managing the parking ing urban services smarter have traditionally
service and its data; and focused on dedicated systems that, depending

march/april 2017 55
Fog Computing

City administrators and


decision makers

S4 Administrative Budget and Budget and Budget and


$ $ $
silos roadmap roadmap roadmap

Service admin Smart Service admin Smart energy Service admin Smart traffic
parking management management
Service
S3 management
silos

1010101
010101010
S2 Data silos 10011100
Cloud 1011 Cloud Cloud

Gateways/Proprietary computing Gateways/Proprietary computing Gateways/Proprietary computing

S1 Physical
silos

Sensors, control and actuation Sensors, control and actuation Sensors, control and actuation

Vertical solution Vertical solution Vertical solution

Figure 1. The quadruple silo (QS) problem. The problem refers to four silos that result from
deploying commercial solutions for smart cities. Physical silos (S1) are composed of vendor-specific
sensors, control, and actuation systems; proprietary gateways; and compute resources at the edge.
Data silos (S2) are usually stored in the cloud and in proprietary resources at the edge. Service
management silos (S3) are independent management ecosystems for managing parking, energy,
traffic, and so on. Administrative silos (S4) are independent departments with different budgets,
objectives, and roadmaps.

on the segments covered and their correspond- istrators rarely have access to information
ing requirements, may or may not be accompa- produced by data workflows that cross differ-
nied by proprietary compute resources at the ent administrative domains because vertical
edge. This poses complex challenges for cities, solutions werent conceived for that purpose.
because existing technologies ultimately lead to Although some departments might manage
service fragmentation and the segmentation of their services using a common data center, the
management competencies. This is clearly vis- logical separation between vertical solutions
ible in Figure 1, and fuels what we define as the remains unsolved. Thus, even if the corre-
QS problem: four categories of silos that cities sponding back ends are hosted by a converged
face after deploying multiple vertical solutions: infrastructure, departments will suffer from
physical (hardware) silos (S1), data silos (S2), data segmentation (S2).
service management silos (S3), and administra- Siloes 13 make it harder to align the pri-
tive silos (S4). orities, budgets, and roadmaps across different
In this scenario, basic aspects such as allo- departments (S4). Our solution addresses the
cating compute resources for processing and challenges of the first three silos holistically,
managing data or for managing the life- reducing barriers for this fourth silo by provid-
cycle of the services hardware, software, and ing more transparency both on how resources
firmware are vertical-specific. Moreover, are allocated and on the impact of uniformly
siloed systems demand significant efforts to deploying and managing services in various
analyze relevant information across differ- departments. We believe this will pave the way
ent departments, especially in real time. In for more cohesive decisions and better align-
fact, as the top of Figure 1 shows, city admin- ment among city departments.

56 www.computer.org/internet/ IEEE INTERNET COMPUTING


A New Era for Cities with Fog Computing

Connectivity limitations &


Criticality Policy/regulation Local processing
bounded delay
A B C D
Autonomous operation Data privacy policies and/or Anomaly
Physical constraints
at the edge operational reasons detection

A A B C D A B B
UC1 UC2 UC3 UC4 UC5
Power monitoring Access control and
and control of Event-based Traffic analysis and Connectivity
telemetry of sensors
elements in video regulation on demand
through street cabinets
public spaces

Figure 2. The five Barcelona use cases (UCs) and their fog computing requirements category. The pictures
at the bottom show one of the street cabinets used during the field trials, as well as the in-ground sensors
installed for traffic monitoring. As noted in the figure, (A) represents an autonomous operation at the edge;
(B) represents data privacy policies and operational reasons; (C) represents physical constraints; and (D)
represents anomaly detection.

Fog Computing in Barcelona point, the goal is to streamline urban services


As S1 in Figure 1 illustrates, with the advent of and reduce hardware and service maintenance
IoT-driven solutions, the number of devices to overheads. This approach is especially impor-
be installed, powered, and maintained in urban tant in scenarios that require compute resources
spaces is increasing dramatically. This is leading at the edge. In many cases, it will make more
to space and operational problems, as cities cant sense to aggregate multiple computing tasks in
continue adding hardware for each new service generic processing nodes hosted in the cabinets,
deployed at the edge. Mindful of these challenges, which can also provide other functions, such as
Barcelona realized that the more than 3,000 street security, distributed analytics, monitoring, data
cabinets deployed in the city form a natural infra- normalization, and brokering.
structure to build out its smart city vision. Figure As we describe later, we addressed five dif-
2 shows one of these cabinets; the pictures were ferent UCs in Barcelona; to better explain their
taken during our field trials in Barcelona. requirements, we classify them into categories (see
A new generation of street cabinets can offer Figure 2). A given UC requiring compute at the
strategic control points for the city and can help edge such as fog nodes hosted in self-driving
accomplish goals that are essential for scaling cars might contain some or even all of the
smart city plans. From a technical perspective, requirements from the following four categories.
the objective is to have a single, extensible, and
distributed infrastructure that provides the nec- Autonomous Operation at the Edge
essary flexibility to address opportunities for This category encompasses urban services that
current and future urban services technologies must remain operative even if backhaul con-
in an integrated way. This infrastructure would nectivity to the cloud is lost. This UC is typi-
have nodes in three locations: the cabinets, cal for services where the criticality of tasks
metropolitan network, and data centers. More and operations demand self-sufficient means for
importantly, it should enable instantiation and monitoring, analyzing, and controlling different
management of urban services and their data in processessuch as for smart and autonomous
a consolidated way. From an operational stand- control of energy distribution boards inside

march/april 2017 57
Fog Computing

cabinets. Here, we label this UC1; two additional This category applies to the case of event-based
UCs also required autonomous operation at the video in Barcelona (UC3).
edge in Barcelona: access control to the street
cabinets and telemetry of sensors within and Use Cases
outside them (UC2); and traffic analysis and reg- In Barcelona, we focused on five UCs, which we
ulation (UC4). demonstrated using the converged architecture
described later.
Data Privacy Policies/Operations
This category includes cases where: the data UC1: Power Monitoring/Element Control
owner doesnt allow specific data to be moved The new generation of cabinets must provide a
to the cloud; the data owner wants to enforce means to monitor and autonomously manage the
local control and brokering for specific data energy distribution boards they house. These
exchanges; or operational or regulatory con- boards will plug and protect elements located
straints preclude particular actions, such both within and outside the cabinets. The sys-
as issu i ng specif ic ac t uat ion com ma nd s tems currently offering this functionality are
from the cloud. Many IoT systems are mis- supplied in the form of vertical solutions (see
sion critical and pose grave risks if hacked, Figure 1). These systems typically include pro-
so cyber-physical security and data privacy prietary controllers that should also be housed
are among the main reasons for deploying in the cabinets, a back end running in the cloud
fog computing. In the case of Barcelona, this for gathering data from distribution boards
category applies to event-based video (UC3), scattered around the city, and proprietary dash-
traffic regulation (UC4), and connectivity on boards for displaying measurements, alarms,
demand (UC5). and so on (the controllers and distribution dash-
boards were provided by Schneider Electric; see
Physical Constraints www.schneider-electric.com/ww/en).
This category includes cases in which data must Virtualizing controllers and other func-
be processed locally due to physical limitations tions to manage energy systems offers several
that prevent sending and processing all of the advantages, such as the possibility of offering
data in the cloud. We further divide this cat- catalogs of products that can be dynamically
egory as follows: instantiated as microser vices in fog nodes.
These products might include functions such as
Connectivity limitations. In some cases, the
things at the edge have either limited making local decisions to keep only critical
bandwidth capability to the cloud or com- services operative during an outage (even in
munications are simply unreliable; they case of loss of cloud connectivity);
thus require localized compute resources to monitoring power consumption and analyzing
ensure the desired functionality. In Barce- specific key performance indicators in real time
lona, for example, this applies to UC3. (such as the quality of power supplied); and
Bounded delay. In some cases, very low managing new resources (such as when an
latency and/or close to real-time response uninterruptible power supply is added).
is sometimes needed; this requires localized
analytics, closed-loop control, and depend- As (A) in Figure 2 shows, this is considered
able actuation on physical systems (such as a critical service in Barcelona and will lever-
for self-driving cars). age the fog nodes horizontal functions, includ-
ing service assurance for critical functions that
Anomaly Detection require high availability, security, analytics,
This category covers cases in which only a data management, and routing and switching.
small fraction of the data collected is important. As we now show, the other UCs also exploit
This is particularly relevant when UCs involve these horizontal functions.
the ingestion of large data volumes at the edge,
which usually requires real-time analysis and UC2: Access Control and Cabinet Telemetry
data filtering because the data scanned becomes Cabinets are exposed to the natural elements, as
useful only when an irregularity is detected. well as to physical abuse or even unauthorized

58 www.computer.org/internet/ IEEE INTERNET COMPUTING


A New Era for Cities with Fog Computing

entry. It is therefore important that the cabi- UC4: Traffic Management


nets state be monitored, covering aspects that Sensors in the street can register traffic and
go beyond power control, including humidity, monitor aspects such as the number of vehicles,
temperature, and the status of the door (open their speed, length, and other variables includ-
or closed). Entry to the cabinets can be elec- ing estimations of queue sizes and waiting
tronically controlled, and it should be possible to time at signalized intersections. Current solu-
access their compartments even if connectivity tions in this space rely on proprietary gateways
to the cloud is lost. to gather data from the sensors, and theyre
Automated entry control and environmen- typically installed as vertical solutions. These
tal monitoring within the cabinets are con- gateways host sophisticated algorithms and
sidered critical services (A in Figure 2). Thus, analytics that are required for performing the
the collection of data from different families measurements with high accuracy.
of sensors (such as http://plat.one), data nor- By pushing the computation to a fog node,
malization, analysis, and access control were vendors could focus on their applications and
performed by processes running in virtual- software components, and spend less effort on
ized environments in the fog nodes. In this UC, ancillary elements, such as maintaining their
the status of certain sensors can be associated gateways, security, and so on. As Figure 2 shows,
with an alarm that will go off when unauthor- this UC is considered critical (A). In addition, due
ized access is detected. As we show in UC3, this to legal regulation, its not possible to issue com-
alarm was one of the triggers used for video mands to actuate on traffic lights directly from
streaming. the cloud (B). The outsourcing of traffic regula-
tion tasks to the fog nodes is still in its infancy
UC3: Event-Based Video and requires a comprehensive study address-
The city has a large base of installed cameras. ing both technical and nontechnical challenges
Although some are connected to a high-capacity (including safety, security, resilience, and busi-
backbone network, others are located in places ness and operational benefits). The UC dem-
where the only available option is cellular con- onstrated in Barcelona covered only the traffic
nectivity. This UC focuses on the latter (C in monitoring part.
Figure 2). These cameras record continuously
and send their video signals to the fog nodes, UC5: Connectivity on Demand
where theyre stored in circular buffers. Only Given the number of activities that take place
when an event occurs (D), such as when a cer- in Barcelona (concerts, sports events, confer-
tain noise threshold is reached or an unauthor- ences, and so on), the city council gets frequent
ized entry to a cabinet is detected, the fog-based connectivity requests from local TV stations,
system triggers two video streams that show police, and so on. The connectivity is for trans-
mitting video, covering the citys activities in
what happened before the event (stored in real time. These requests are typically for net-
the circular buffer in the fog node), and work capacity that cant be provided through
whats happening right after the event in mobile networks or local WiFi, so theyre cur-
real time. rently provisioned through networking equip-
ment hosted in the cabinets. This means that
The videos are streamed to a back end (in the video crews, police officers, and so on must
cloud or other destinations). open the cabinets and get a wired connection
The picture shown under UC3 in Figure 2 is through an internal switch/router (B).
a snapshot captured from one of these cameras; This is a recurring need for a city that
the video was initiated after the emulation of hosts many events, but provisioning the con-
an unauthorized cabinet entry (in the pictures nectivity, bandwidth, and quality of service
upper left corner). The reasons for not stream- (QoS) required typically takes days. The city
ing continuously are typically cost, privacy, wanted a self-service portal that let request-
and data storage overheads (B). In this UC, a ors (with proper authentication) reserve band-
fog node can aggregate video feeds from mul- width on demand and leverage the fog nodes in
tiple cameras nearby, and perform analysis and the cabinets to access a high-speed connection
stream only when events are detected. with QoS support in minutes. In Barcelona, the

march/april 2017 59
Fog Computing

3. The design should focus on common needs


Openness S1 S4 across verticals and industries. Our target was
to create a platform that can serve multiple
purposes and be ported and reused in other cit-
S3
ies and other IoT domains. The street cabinets
S2 S3 in Barcelona are a means to an end; theyre
S1
Uniform simply the point of presence (PoP). Thus, the
Virtualization Data-app management cabinets should not constrain the platforms
and box centric model and service
reach and applicability. Other cities might rely
consolidation consolidation
on different PoPs for hosting fog nodes, includ-
ing poles, underground installations, and so
Figure 3. The fog platforms foundational principles. These basic on, and should be able to leverage our designs.
principles address the four silos (S1S4) in the quadruple silo problem,
Based on these objectives, the design was
which results when deploying commercial solutions for smart cities.
guided by four principles (see Figure 3). Its worth
highlighting that Figure 3 shows only the basic
fog nodes will replace the legacy routers and pillars for fulfilling the three objectives men-
switches inside the cabinets and will host virtu- tioned earlier. Other fundamental principles for
alized versions of them. designing distributed systems of this nature
This is also part of the consolidation the including end-to-end security, policy manage-
city wants to enforce at the edge. Thus, when a ment, and scalability are an integral part
request is made, the connectivity service must of the platform, and hence traverse the pillars
be automatically orchestrated and granted, shown in Figure 3.
which requires the configuration of fog nodes The platforms openness is essential. In
and network tunnels to reach their correspond- our model, openness has a two meanings. The
ing endpoints. And, more importantly, the platform offers not only multivendor support
cloud/fog system must correlate information at the fog, network, and cloud levels, but also
from these sometimes dynamic connectivity enables the deployment of homologated third-
requests with data produced by sensors that lie party applications on the city platform. More-
beneath UC2 and UC3 (such as to avoid setting over, our platform should provide for uniform
off certain alarms when the door of the cabi- management of different tenants (mainly city
net is opened). The capacity is reserved only departments, in the case of Barcelona) and their
for the requested period; after that, resources services. The goal of multitenancy is to let ten-
are released automatically. In Figure 2, the pic- ants deploy and manage their services autono-
ture under UC5 shows how authorized person- mously, while benefiting from the economies
nel could request such connectivity through a of scale of integrated equipment for simultane-
mobile phone or a tablet, using an application ously providing compute resources and services
that was developed during the field trials. to multiple departments. This includes virtual-
ized environments that can run in the devices at
Design Principles the edge, network, and data center. This design
The work conducted in Barcelona focused on the principle is captured as Virtualization and box
design, implementation, and demonstration of a consolidation in Figure 3, and addresses silo S1
fog computing platform guided by three objectives: in the QS problem.
Another important aspect is that our design
1. The platform should provide a solution for is based on the fusion of data-centric and appli-
problems S1, S2, and S3 as Figure 1 shows, cation-centric models. This fusion is essential
while paving the way for more cohesive in IoT scenarios because, in most cases, a cloud/
decisions and better alignment among city fog platform must efficiently manage the life-
departments affected by problem S4. cycle of both the data and IoT applications. To
2. The platform should manage urban services sub- this end, we have coined the term data-app
ject to the requirements and UCs described centric, which reflects the fact that our model
earlier and, more importantly, demonstrate borrows some of the design principles and best
this in the streets of Barcelona. practices that the two models offer.

60 www.computer.org/internet/ IEEE INTERNET COMPUTING


A New Era for Cities with Fog Computing

Our data-app centric approach provides These consolidations can dramatically reduce
advanced policy management for both data the complexity, cost (through capital expendi-
and applications. This approach facilitates tures and operating expenses), maintenance,
data sharing across user-provided applica- and time required to launch and deploy solutions
tions in a controlled way. In turn, it enables throughout the city, while effectively address-
new and powerful analytics, leading to action- ing the challenges posed by the QS problem. In
able business intelligence across agencies and this context, cost refers to the comparison with
administrations. Weve particularly focused the alternative siloed approach. Whether a UC is
on controlling and mediating data exchanges addressed with a siloed solution or through our
and creating sophisticated data workflows in platform, the monetary benefit of the individual
a secure way. For instance, data produced by UC is the same. However, once multiple UCs are
one department on a specific data topic can be deployed, our platform has two main advantages.
consumed by other departments, or even third First, it offers economies of scale, because the
parties, based on roles and policies that control same edge and cloud infrastructure can be lever-
the access to the data (we delve into details of aged for multiple UCs. Second, our infrastructure
this later). enables sharing of data between services of dif-
In addition to the policy framework we ferent UCs, which the city can leverage to add
designed for analyzing and extracting value additional value on top of the individual UCs.
from the data, the platform supports the instal-
lation of applications wherever required (that Converged Architecture
is, at the network edge, the backhaul network, Figure 4 shows a simplified version of our model.
or the cloud). The data-app centric approach As the model shows, Barcelona initially requires
addresses silos S2 and S3 of the QS problem. a flat model composed of only two layers: fog
Furthermore, we focused on designing a nodes inside the cabinets and a common back
horizontal platform that offers uniform orches- end in the cloud. More complex arrangements,
tration, security, distributed analytics, and including a hierarchy of fog layers and hybrid
management APIs across departments; the plat- clouds, might be required once the deployment
form particularly addresses decentralized ser- of urban services begins to scale.
vices that require compute resources beyond the The architecture represents a fully distrib-
data centers. One of the designs central aspects uted system, where even the individual blocks
was to make city services easy to operate and in the figure might be internally distributed. Its
maintain through scalable orchestration and composed of three main groups of components:
proper automation. the fog nodes, the back end, and a set of cross-
The design principles we followed allow domain functions, including security, service
fog computing to elevate IoT from point solu- assurance, and network management.
tions to managed ser v ices at the edge. We
contend that this approach will make it easier Fog Nodes
to align roadmaps and optimize budgets and The main components inside the fog nodes
purchases across departments, which will help include instances of virtual domains (VDs).
city administrators tackle silo S4 of the QS
problem. Instances of virtual domains (VDs). The VDs
Compared to our converged cloud/fog plat- represent the execution environments for the
form, service silos from different solution pro- different services running concurrently in a
viders beyond the cloud considerably increase fog node. Each VD is application-specific and
the operational expenses for cities. Our design belongs to a single tenant. A tenant might have
offers multiple levels of consolidation, including multiple instances of VDs running in the same
fog node.
box consolidation (that is, the integration of In our implementation, we support both
routers, switches, and compute nodes in the Docker containers and virtual machines (VMs) on
continuum between edge and cloud); kernel-based virtual machines (KVMs). Depend-
the consolidation of data and its management; ing on the service requirements, a VD could
and be instantiated as a single container or a single
uniform service management. VM, or it could be composed of several instances

march/april 2017 61
Fog Computing

Dashboards

Dashboard back end


Distributed orchestration and management platform
REST API (back end)

ETSI MANO Service models 1010101

(YANG) Data Policy 010101010


10011100
Cross-domain management management 1011
Mapping Cloud
orchestrator
Device models
NETCONF (YANG) Application
Analytics
management
Service assurance

VNFM
Network
Security

REST Topology Monitoring


Fog node northbound interfaces
VIM
Fog nodes Data 1010101
010101010
10011100 Analytics
management 1011

VD1 VD2 VD3 VD4 VSR


Fog nodes inside
the cabinets

Energy Sensor Event-based Traffic Connectivity


control telemetry video control on demand

Communications infrastructure
Fog node southbound interfaces

Things Things

Figure 4. The architecture and its mapping to the case of Barcelona. The fog nodes on the right consolidate the five use
cases covered in the project. (ETSI = European Telecommunications Standards Institute; MANO = Management and
Orchestration; NETCONF = Network Configuration Protocol; VD = Virtual Domain; VIM = Virtualized Infrastructure
Manager; VNFM = Virtual Network Functions Manager; and YANG = Yet Another Next Generation, which is a
standardized data modeling language.)

configured and chained as a tenant-specific exe- across VDs. The analytics also assist other key
cution environment within a fog node. building blocks for managing urban services
end to end, such as security, service assur-
Data management block. The data manage- ance, and networking. In our implementation,
ment block represents the internal buses, we used Cisco ParStream (www.cisco.com/c/
brokers, and data policy enforcement rules en/us/products/analytics-automation-software/
required to locally share and correlate data parstream/index.html), an historian database
between specific microservices and applica- that supports big data analytics in distributed
tions. For data distribution and brokering, we deployments at very large scale. Different algo-
deployed and tested both the Data Distribu- rithms for application-specific analytics can be
tion Service (DDS) and RabbitMQ. We also embedded in ParStreams core, thus enabling
used their multiprotocol capabilities to sup- highly efficient ways to support analytics in
port Message Queuing Telemetry Transport multitenant environments, including service
(MQTT)-based applications. Regarding policy assurance, power control, and security.
enforcement, we used Lightweight Directory Data sharing between different tenants
Access Protocol (LDAP) in the back end and was ma naged t h roug h Pa rSt rea ms data-
LDAP replicas at the fog node level. The rep- base, while data access control was handled
licas allow the fog nodes to continue autho- through LDAP. In Barcelona, we demonstrated
rizing the defined data sharing policies, even UCs that required data-sharing policies and
when backhaul connectivity is lost. access control among tenants at the fog node
level. For instance, the cameras in Barce-
Analytics. Analytics can run locally, enabling lona are managed by a specific administra-
autonomous operation and decision-making tion (tenant x), while the sensors connected

62 www.computer.org/internet/ IEEE INTERNET COMPUTING


A New Era for Cities with Fog Computing

through the cabinets are managed by another supports NETCONF but also provides other key
administration (tenant y). interfaces, such as command-line interface (CLI),
As Figure 2 shows, we handled the data REST, Simple Network Management Protocol
sharing workflows required to implement UC3 (SNMP), and a Web user interface. All of these
as follows. When processes running in VD2 (see interfaces are automatically rendered thanks to
Figure 4) detect an anomaly, an event is trig- the YANG models supplied with the fog nodes.
gered (and stored in ParStream), which is con-
sumed by VD3. In this example, an instance Southbound interfaces. The southbound inter-
of tenant y (VD2) is the data writer, while an faces depend on the sensor families and data
instance of tenant x (VD3) is one of the data collection elements to be connected. In Bar-
readers. This is a part of a UC that requires celona, the strategy we followed was to pre-
anomaly detection, data privacy policies, local integrate with various vendors to cover the
processing, and video streaming that will be requirements of the different UCs implemented.
sent through 3G/4G connectivity because the
fog nodes are located in areas without wired Back-End Cloud Support
connectivity. The second group of components supports the
back end in the cloud. Given the distributed
Virtual switch/router (VSR). As previously nature of our model, some of the blocks in the
explained, the fog nodes will consolidate and back end might be hosted in the fog nodes and
replace legacy networking gear within the cabi- vice versa. The back end thus supports several
nets. Therefore, the VSRs shown on the right in components, including the following.
Figure 4 support the multiple networks required
and act as a gateway for the applications run- Orchestration system. As Figure 4 shows, the
ning in the different VDs. Multiple indepen- model here is the NFV Management and Orches-
dent instances of VSRs can run concurrently tration (MANO) stack, which was recently stan-
in a single fog node if required. In Barcelona, dardized by the European Telecommunications
we used software routers in a virtual form fac- Standards Institute (ETSI). ETSI MANO is based
tor, including Ciscos CSR1kv. For isolating the on three layers:
multiple networks involved, we used both vir-
tual local area networks (VLANs) and a virtual an orchestrator that enables service defini-
extensible LAN (VXLAN). tion and automation across three domains
(fog, network, and data center),
Northbound interfaces. In the architecture that a Virtual Network Functions Manager (VNFM),
we conceived, all the fog nodes are provisioned and
with a standard Network Configuration Pro- a Virtualized Infrastructure Manager (VIM).
tocol (NETCONF) interface; the data modeling
language we use in tandem with NETCONF is In our architecture, we used Cisco tail-f Network
the IETF standard Yet Another Next Genera- Services Orchestrator (NSO) as the cross-domain
tion (YANG). NETCONF and YANG are widely orchestrator, Cisco Elastic Services Controller
adopted in industry, especially in the service (ESC) as the VNFM, and OpenStack as the VIM.
provider and enterprise space. Theyre also an As Figure 4 shows, one of the cross-domain
integral part of most efforts around network orchestrators main roles is to map service defi-
functions virtualization (NFV) and software- nitions modeled in YANG to device-specific
defined networking (SDN). However, the advan- configurations based on device models that are
tages of choosing NETCONF and YANG for also specified in YANG. The combination of
fog computing go far beyond interoperability ETSI MANO, NETCONF, and YANG is essential
enablement, as they let us strategically create a not only in terms of openness but also to enable
common architecture for NFV, 5G, fog, and IoT. a seamless convergence of cloud and fog.
We followed this approach in Barcelona.
We created the first YANG models for a fog Data management. This module works in con-
node, and for IoT services involving fog com- cert with the ones hosted in the fog nodes and
puting. All fog nodes used in Barcelona were enables data workflows across city departments.
endowed with Cisco ConfD, which not only Aspects such as data normalization, persistency,

march/april 2017 63
Fog Computing

Table 1. Validation of fog through use case requirements.


Use case (UC) Aspects that require fog The cloud alternative
UC1: Power monitoring Autonomous operation and control of Impossible to ensure autonomous operation
power supplies for critical services; real-time
intervention
UC2: Cabinet telemetry Autonomous operation (access control must be Impossible to ensure autonomous operation
guaranteed, even without backhaul connectivity)
UC3: Event-based video Poor connectivity, local data analysis, and Unfeasible due to cost and practical reasons
anomaly detection
UC4: Traffic management Autonomous operation (intervene on Impossible to ensure autonomous operation;
traffic lights, even when backhaul network regulation prevents actuation on traffic regulators
is unavailable); run complex algorithms to from the cloud
accurately monitor vehicle flows
UC5: Connectivity Automated configuration of devices at the edge Can centrally control the configuration of
on demand to accommodate connectivity requests standalone switches and routers, but at the
cost of losing the practical advantages of box
consolidation

and brokering are managed by the distributed Figure 4 (such as operation and business sup-
data management blocks shown in Figure 4. port systems) are also an integral part of the
As we mentioned, in our implementation, we back end. In addition, the back end offers a rich
mainly used DDS and RabbitMQ. set of APIs that abstract entirely the internals
and the complexity of managing the lifecycle
Application enablement framework. This frame- of city services and the underlying infrastruc-
work manages the lifecycle of the applications ture. The platform also lets us build dashboards
instantiated in the VDs hosted by the fog nodes. for the different tenants. Through these dash-
It also manages other applications that are boards, the appropriate authenticated user can
required for functions such as networking, secu- define, deploy, update, and remove virtualized
rity, and service assurance. Among these are the services, define access policies, create new ten-
VSRs instantiated in the fog nodes, as well as ants, get an overview of the deployments in the
other virtualized applications for malware detec- city, and so on. Our project demonstrated the
tion, anti-virus, passive and active monitoring, different views of the platform from a tenants
and so on, each of which can be instantiated in perspective (for example, the department in
the fog or in the cloud, depending on the service charge of energy management cant access data
requirements. from traffic sensors, and vice versa).

Policy management. Together with adequate security, Cross-Domain Functions


it glues data and application management functions. In addition to the former two groups, three addi-
This component is key for endowing the architecture tional blocks are crucial to ensuring that the
with the desired data-app centric functionality. As architecture is secure, remains connected, and
mentioned earlier, in our implementation, the policies offers services that are resilient to potential fail-
for accessing resources and defining identities and ures that might occur during their lifecycle (see
roles are managed through LDAP. the left side of Figure 4). For example, we con-
figured the fog nodes using secure bootstrap-
Analytics. This module works in concert with ping and zero touch provisioning processes,
the ones hosted in the fog nodes and enables including remote attestations supported through
historian analytics in a fully distributed way. In trusted platform modules.
Barcelona, we used Cisco ParStream.
Validation and Lessons Learned
Additional functions. Other functions, includ- One of the goals of the Barcelona initiative was
ing topology discovery and management, moni- to validate the relevance and necessity of fog
toring, and additional subsystems not depicted for cities. Although cloud is seen as the go-to

64 www.computer.org/internet/ IEEE INTERNET COMPUTING


A New Era for Cities with Fog Computing

solution for many IoT-related challenges, we ing the cloud to the edge means that fog provid-
have come to understand by talking to vari- ers can offer various as-a-service models (such
ous cities and industries that fog is crucial. as for analytics, security, and data sharing).
Here, we discuss some of these findings, as well These as-a-service models reduce the software
as lessons learned. and maintenance complexity for solution provid-
Table 1 shows the five UCs, highlighting the ers. Indeed, weve already confirmed that sev-
various requirements and the need for fog com- eral third-party solution providers are willing to
puting. The last column of the table describes, revise their strategy and possibly change their
where possible, whether a viable cloud alternative business models, as their (siloed) focus on hard-
exists for the requirements defined. As the table ware, security, and analytics has now migrated
shows, fog is mandatory in four of the five UCs to the platform.
and is clearly desirable in the fifth for practical
reasons, such as box consolidation. Platform Flexibility Leads to Innovation
Overall, the table highlights the necessity for Because of fog, cities can consolidate services
fog to enable several smart city UCs. The project on a single platform, thereby reducing the
in Barcelona led to new insights into the impact infrastructure and operational costs. One of the
of fog computing. Following are the most impor- findings of our project was that, by lowering
tant lessons learned from this project. the complexity and incremental cost to deploy
new services at the edge, cities are becoming
Fog is Needed for Practical Reasons more agile in deploying new services and are
Fog computing was introduced a few years ago more inclined to innovate.
in response to challenges posed by many IoT
applications, including requirements such as Pre-integration Is Essential
very low latency, real-time operation, large geo- Service providers and system integrators play
distribution, and mobility.1 As Table 1 shows, an important role in the digitization of cit-
the reality in cities is quite different at the ies, and they will be important partners when
moment. The drivers for fog computing today launching platforms like the one we describe
are mainly rooted in aspects such as autono- here. However, no two cities are the same.
mous operation, regulatory limitations, and box While the premise of uniform service man-
and service consolidation. Although the driv- agement, automation, and de-silofication is
ers for cities are currently different from those valid for all cities, the types of sensors and
originally forecasted, the need for distributed third-party applications used will differ from
computation at the edge is unquestionable. city to city. Pre-integration that is, provid-
ing connectors to sensors and their protocols,
Box Consolidation and De-silofication and deployment of third-party services is
Are Key therefore key, and would enable service pro-
Cities see the potential of connecting devices and viders and system integrators to rapidly roll
sensors for their citizens, but they also recognize out such a platform in multiple cities and
that they cant deploy new boxes (gateways) in countries.
the city each time a new solution is introduced.
In the case of Barcelona, the cabinets have lim- Fog Computing Isnt Limited to
ited space, but even if cities mount boxes in light Constrained Devices
poles or walls, they face the aesthetics factor and Fog computing, by virtue of the networks
a maintenance burden of managing multiple ser- edge, is sometimes associated with constrained
vice silos, as described by the QS problem in Fig- devices (limited CPU, memory, disk space, and
ure 1. In summary, box consolidation is a must so on). Our project showed that the cabinets in
to avoid the proliferation of siloed boxes from Barcelona can house multicore industrial PCs
different solution providers. with multiple gigabytes of memory and up to
terabytes of disk space, not only to enable the
Fog Offers New Business Models deployment of various virtualized services but
for Solution Providers also to plan for new ones. A spectrum of edge
Fog reduces the need for these providers to build computing devices will emerge, ranging from
and maintain their own hardware, while extend- constrained devices to full-blown data center-

march/april 2017 65
Fog Computing

Budget and
S4
roadmap City administrators and References
decision makers
1. F. Bonomi et al., Fog Computing: A Platform for Inter-
$
Budget
udget and net of Things and Analytics, Big Data and Internet of
roadmap
Things: A Roadmap for Smart Environments, vol. 546,
$ Budget and 2014, pp. 169186.
roadmap
2. W. Shi et al., Edge Computing: Vision and Chal-
$ lenges, IEEE Internet of Things J., vol. 3, no. 5, 2016, pp.
Cloud/network/fog 637646.
Analysis 3. M. Hassan et al., Help Your Mobile Applications with
S1 S2 S3
Fog Computing, Proc. Fog Networking for 5G and IoT
Control
C t l
Workshop, 2015; doi:10.1109/seconw.2015.7328146.
Sensing

Marcelo Yannuzzi is a principal engineer at Ciscos Chief


Strategy Office, where he works on strategic innovation
in the areas of fog computing, IoT, and security, and
provides strategic advice on new business opportunities
and technologies for Cisco. Yannuzi has a PhD in com-
puter science from the Technical University of Catalonia
BarcelonaTech. Contact him at mayannuz@cisco.com.
Internet of Things (IoT)

Frank van Lingen is a technology strategist at Cisco Sys-


tems. His research interests include distributed sys-
Figure 5. The proposed platform. By fusing cloud, network, and fog, tems, analytics, and fog computing within the context
this common platform breaks down silos S1, S2, and S3 in the QS of the Internet of things. Van Lingen has a PhD in
problem. computer science from the University of Technology
Eindhoven. Contact him at fvanling@cisco.com.

like servers. One of the challenges going for- Anuj Jain is a director at Ciscos Chief Strategy Office. He
ward is to ensure that the platform we developed leads an innovation team working in the areas of fog
is capable of supporting this wide variety of computing, IoT, next-generation computing, artificial
edge devices. intelligence, and security, with a focus on disrup-
tive technologies and strategic business opportunities
for Cisco. Jain has an MS in micro-engineering from

T he project carried out in Barcelona demon-


strates that a platform like the one shown in
Figure 5 can efficiently address the QS problem.
EPFL. Contact him at januj@cisco.com.

Oriol Lluch Parellada is a technology strategist at Cisco


We believe this approach can be applied to various Systems. His research interests include cloud infra-
other domains, including oil and gas, manufac- structure, fog computing, ser vice orchestration,
turing, and utilities, and that it can also help and artificial intelligence. Lluch Parellada has an
address some of the main challenges that edge MSc in telecommunications engineering from the
computing will face.2 One of these challenges is Technical Universit y of CataloniaBarcelonaTech
to determine how data might be shared between and an EM BA in management of technolog y from
many different processes running at the edge t he E PF L a nd H EC Lau sa n ne. Contac t h i m at
and the cloud, particularly to support functions orilluch@cisco.com.
such as smart offloading.3
Manel Mendoza Flores is a net work and securit y archi-
Acknowledgments te c t at t he Ba r c e lon a C it y Cou nc i l , whe r e hes
We thank the city of Barcelona for its support during the involved in multiple projects aimed at improving
project, as well as many other partners involved, including the qualit y of life of Barcelonas citizens using
H. Antunes, R. Milito, I. Errando, J. Balagu, J. L. Ferrer, S. ne w te c h nolog ie s . Me ndoz a F lor e s h a s a BE i n
Figuerola, O. Trullols, K. Macpherson, J. Bienert, D. Eckstein, te le com mu n icat ion s f rom t he Te c h n ica l Un ive r-
H. Werner, C. Byers, R. Zheng, R. Irons-Mclean, F. Osman, sit y of Cata lon iaBa rce lonaTec h . Contac t h i m at
J. Greengrass, H. vant Hag, and K. Steele. mmendozaf@bcn.cat.

66 www.computer.org/internet/ IEEE INTERNET COMPUTING


A New Era for Cities with Fog Computing

David Carrera leads the Datacentric Computer Research has a PhD in computer science from Technical Univer-
Group at the Barcelona Supercomputing Center (BSC). sity of CataloniaBarcelonaTech. Contact him at pcha-
His research interests include performance management cin@sensefields.com.
of data-centric platforms. Carrera has a PhD in computer
science from the Technical University of Catalonia Angelo Corsaro is the chief technology officer at ADLINK
BarcelonaTech. Contact him at david.carrera@bsc.es. Technology, where he looks after technology strategy
and innovation for the companys Industrial Inter-
Juan Luis Prez is a senior research engineer in the Data- net of Things (IIoT) platform. His research interests
centric Computer Research Group at Barcelona Super- include large-scale mission/business critical distrib-
computing Center. His research interests include the uted systems, real-time systems, functional program-
model-driven IoT. Luis Prez has an MS in computer ming, IoT, and fog and cloud computing. Corsaro has
architecture, networks, and systems from the Technical a PhD in computer science from Washington Uni-
University of CataloniaBarcelonaTech. Contact him at versity in St. Louis. Contact him at angelo.corsaro@
juan.luis.perez@bsc.es. prismtech.com.

Diego Montero is a PhD candidate at the Networking and Albert Olive is a systems architecture expert at Schneider
Information Technology Lab (NetITLab), where his Electric. He has focused on several national and inter-
research interests include network security, SDN, national projects to design control and power distri-
network virtualization, fog computing, and mobil- bution architectures within Schneider. Contact him at
ity. Montero has an MSc in computer architecture, albert.olive@schneider-electric.com.
networks, and systems from Technical University of
CataloniaBarcelonaTech. Contact him at dmontero@
ac.upc.edu.

Pablo Chacin is a researcher and practitioner with more


Read your subscriptions through
than 20 years of experience in industry and academia.
the myCS publications portal at
His areas of interest include distributed systems, soft-
http://mycs.computer.org.
ware architecture, and middleware services. Chacin

Recognizing Excellence in High-Performance Computing


Nominations are Solicited for the
SEYMOUR CRAY, SIDNEY FERNBACH & KEN KENNEDY AWARDS

SEYMOUR CRAY COMPUTER ENGINEERING AWARD


Established in late 1997 in memory of Seymour Cray, the Seymour Cray Award is awarded to recog-
nize innovative contributions to high-performance computing systems that best exemplify the creative
spirit demonstrated by Seymour Cray. The award consists of a crystal memento and honorarium of Deadline: 1 July 2017
US$10,000. This award requires 3 endorsements.
All nomination details available at
awards.computer.org

SIDNEY FERNBACH MEMORIAL AWARD


Established in 1992 by the Board of Governors of the IEEE Computer Society, this award honors the
memory of the late Dr. Sidney Fernbach, one of the pioneers on the development and application of
high-performance computers for the solution of large computational problems. The award, which con-
sists of a certificate and a US$2,000 honorarium, is presented annually to an individual for an out-
standing contribution in the application of high-performance computers using innovative approach-
es. This award requires 3 endorsements.

ACM/IEEE-CS KEN KENNEDY AWARD


This award was established in memory of Ken Kennedy, the founder of Rice Universitys nationally
ranked computer science program and one of the worlds foremost experts on high-performance
computing. A certificate and US$5,000 honorarium are awarded jointly by the ACM and the
IEEE Computer Society for outstanding contributions to programmability or productivity in high-
performance computing together with significant community service or mentoring contributions. This
award requires 2 endorsements.

MARCh/APRIl 2017 67