Bell Labs

Technical
Journal
Volume 24 | July 2019

Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
 The network OS:
Carrier-grade
SDN control of
multi-domain,
multi-layer
networks
Marina Thottan, Catello Di Martino, Young-Jin Kim, Gary Atkinson, Nakjung Choi,
Nishok Mohanasamy, Lalita Jagadeesan, Veena Mendiratta, Jesse E. Simsarian, and
Bartek Kozicki
Nokia Bell Labs

Bell Labs Technical Journal | V o l u m e 2 4 | 2

Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
 Marina Thottan is the Group Leader for the Control Research Group in Nokia Bell Labs. She joined Bell Labs
Research in 1999 and has contributed to a wide variety of projects, including Online Gaming, Content Distribution,
Routing protocols, Data over Optical networks, High Speed Router Design, Network Management and Anomaly
Detection. Most recently she has been leading work on Reliable Software-Defined Networks (SDN). She has also
made significant contributions to the field of Smart Grid Communication Networks and Analytics. Her contributions
were recognized by two Bell Labs Teamwork Awards. Marina received a Ph.D. in Electrical and Computer Engineering
from Rensselaer in 2000. She has published over 60 papers in scientific journals, book chapters and conferences
and holds several patents in the areas of network management, interactive network applications, routing
algorithms, data analytics and network architectures. She is co-author of a recent book Communication Networks for Smart Grids: Making
Smart Grids Real and has also co-edited a book Algorithms for Next Generation Networks. Marina is an IEEE Fellow and a Bell Labs Fellow.

Catello Di Martino is Member of Technical Staff at Nokia Bell Labs, Murray Hill, NJ, USA. He obtained an MSc ‘06 and
PhD ‘09, both in computer and control engineering from “Federico II” University of Naples. Currently, he is working
on the creation of resilient communication platforms and networks. Across different institutions, he has contributed
to several key research projects related to resiliency in the areas of high-performance computing, cloud computing,
SDNs, air traffic control management system and sensor networks.

Young-Jin Kim was until recently a Member of Technical Staff of the Control Research Department at Nokia Bell
Labs, Murray Hill, NJ, USA. Dr. Kim received his B.S. and M.S. degree in Computer Science from Yonsei University,
Korea and a Doctorate in Computer Science from University of Southern California. Since joining Bell Labs in 2010,
he has been contributing to data center networks, IP-optical networks and device-to-device communications such
as plug-in electrical vehicles and smart grids. His work typically involves the design of software platforms, security
measures and routing algorithms on aspects of information-centrism, software-defined network paradigm and
virtualization. Prior to joining Bell Labs, he had worked as senior software engineer at the Telecommunication R&D
center of Samsung Electronics, Korea. His research has been published in IEEE/ACM conference proceedings and
journals and has been distributed as publicly available software.

Gary Atkinson is a member of technical staff in the Network Control Research Department of Nokia Bell Labs.
Since joining Bell Labs, he has worked in the areas of wireless, optical and data network design and analysis, typically
cross-disciplinary and involving mathematical modeling and algorithm development for network planning, design
and tradeoff analysis, including research into and application of algorithmic methods for challenging network
combinatorial optimization and control problems. He holds a B.S. in Math and a B.S. in Physics from the University of
Texas, an M.S. in Physics from Northeastern University and a D.Sc. in Operations Research from George Washington
University, where he produced an award-winning doctoral dissertation in applied mathematical chemistry. He has
authored over 40 conference papers and journal articles in mathematics, physics, telecommunications and power
engineering. He is an inventor with over a dozen patents or filings.

Nakjung Choi is currently a member of technical staff in Nokia Bell Labs, at Murray Hill, NJ, USA, where he has
worked for Bell Labs since April 2010. He received his B.S. (magna cum laude) and Ph.D. at the School of Computer
Science and Engineering, Seoul National University in 2002 and 2009, respectively. Also, he has received several
awards such as Best Paper Awards and Awards of Excellence (formerly Bell Labs, Alcatel-Lucent). His research
focused on Future Internet, SDN/NFV/Cloud, 4G/5G/IoT and future converged services.

Bell Labs Technical Journal | V o l u m e 2 4 | 3

Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
 Nishok Mohanasamy is a Research Engineer specialized in Software Systems Research in the Control Research
Department at Nokia Bell Labs, Murray Hill, NJ, USA. Nishok received his M.S degree in Computer Science from
Montclair State University, New Jersey. Currently, he is responsible for designing, developing and prototyping
new disruptive concepts, tools or systems and in implementing and demonstrating future SDN-based carrier-grade
networking platforms.

Lalita Jagadeesan is a Distinguished Member of Technical Staff in the Network Control Research Department
at Nokia Bell Labs. Her current research interests include software verification, security and programmable (SDN)
networks. Lalita has held a variety of technical and managerial positions at Bell Labs in computer science research,
software and security and is a co-inventor of a dozen patents (issued and pending). She is a senior member of the
IEEE and holds a Ph.D., S.M and S.B. in computer science, all from the Massachusetts Institute of Technology.

Veena Mendiratta is a Consulting Member of Technical Staff in the Network Control Research Department at
Nokia Bell Labs, Naperville, IL, USA. Her work has focused on the reliability and performance analysis for
telecommunications systems products, networks and services to guide system architecture solutions. Her
research interests include architecture, system and network dependability analysis, software reliability engineering,
programmable networks (SDN) resiliency and telecom data analytics. Current research is focused on network
reliability and analytics — architecting and modeling the reliability of next-generation programmable networks
and development of analytics-based anomaly detection algorithms for improving network performance and
reliability. She is a member of IEEE, INFORMS and SIAM. She holds a B.Tech in engineering from the Indian Institute
of Technology, New Delhi, India and a Ph.D. in operations research from Northwestern University, USA.

Jesse Simsarian is a Member of Technical Staff at Nokia Bell Labs, Murray Hill, NJ, USA. Since joining Bell Labs in
2000, he has been researching optical networks and switching, including software-defined transport networks, fast-
switching coherent optical receivers and scalable optical packet routers. From 1998 to 2000 he had a National
Research Council Fellowship at the National Institute of Standards and Technology in Gaithersburg, MD, USA. He
received the Masters and Ph.D. degrees in physics from SUNY Stony Brook in 1995 and 1998, respectively, and is
a Senior Member of the OSA and IEEE. He has authored or co-authored 80 publications and holds several patents.

Bartek Kozicki is a Member of Technical Staff at Nokia Bell Labs, Antwerp, Belgium. He received his MSc from the
Technical University of Lodz, Poland, in 2004 and PhD degree from Osaka University, Japan, in 2008. From May
2008 to December 2010, he worked as an associate researcher at NTT Network Innovation Laboratories. Since
2011 he has been with Bell Labs focusing on design and modeling of high-speed electrical and optical networks.
His specialty is advanced modulation formats. Current research interests include access network architecture and
software-defined networking. Dr. Kozicki has published over 50 papers in international journals and conferences.
He has filed over 20 patents in the areas of optics and telecommunications.

Bell Labs Technical Journal | V o l u m e 2 4 | 4

Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
 Driven by the growth of cloud and virtualization technologies,
telecom services are under pressure to become increasingly
dynamic. Cloud services can be instantiated in a matter of
minutes on computing platforms, which can rapidly schedule
and allocate virtualized physical resources. A similar flexibility
to adapt to dynamic service requests remains a challenge with
network resources. Recently, centralized controllers have been
deployed in data center networks to adapt network
connectivity to computational needs. However, due to the
complexity and variety of carrier networking technologies,
carrier networks have been slow to move towards centralized
control. More recently, carriers are evolving to address
network complexity by pushing for the disaggregation of
network elements and the introduction of equipment with
more control flexibility. Carriers require a new control
paradigm to benefit from this new flexibility, which extends far
beyond today’s OpenFlow network controllers.
The lack of a global network controller with a consistent
network view slows down the development and deployment
of the more dynamic network services required by extensive
cloud technologies. To address the scaling requirements of the
carrier environment, the envisioned network controller would
be implemented as a distributed system. In this article, we
describe a global network controller designed to function as a
network operating system with multi-domain, multi-layer
capabilities that will transform carrier-grade telecom services.
We define the requirements for a carrier-grade network
operating system (OS) and describe a prototype system
Bell Labs Technical Journal | V o l u m e 2 4 | 5

Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
 (NetUNIX) that can control heterogeneous networks at large
scale and ensure high service reliability. We highlight the
advantages of the NetUNIX platform in the context of other
open source platforms and illustrate the use of the NetUNIX
platform with two use cases: a metro-distributed data center
and 5G next-generation fronthaul.
Introduction
All aspects of our modern lives have been
impacted by innovations in data
communications. The new connected world
offers a myriad of interfaces such as audio,
video, text, and virtual and augmented reality
to create a rich, context-aware communication
environment. Consumers of these digital
communication platforms continue to demand
more data, higher quality and controllable
priority. The increase in digital services has
largely been met by a vast data network
comprised of equipment and software from
many thousands of different equipment
providers, all expected to interwork at some
level of compatibility. However, future network
services will be increasingly dynamic, making
the rapidly growing combination of software
and hardware components difficult to control
and, consequently, less able to satisfy the
needs of the consumers on a reasonable time
scale with reasonable costs.
Today, telecom operators are experiencing an
economic squeeze created by the inexhaustible
demand for low-cost network bandwidth, and the
demand for ubiquitous access to the cloud is
forcing them to think beyond merely adding
capacity. Cloud services require the restructuring
of networks following similar principles to those in
the computing industry [1]. Network functions,
systems and individual network elements must be
disaggregated, realigned and reintegrated at
different levels via new software platforms and
Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
interfaces [2]. The essential elements of this
transformation are:

A new automation-optimized network fabric,
with a rich and verifiable set of configuration
parameters that fuses data center (DC) fabric,
scalable routing and flexible optical
interconnection

A network OS to control network functions
and the network fabric, making them agile
and adaptable to varying services and user
demands.
Today’s network architectures are distant
from this vision. Customers requesting a
network service must still go through a lengthy
provisioning process. The delay stems from
insufficient automation of the tools needed
for efficient and reliable configuration of
network resources. Typically, the end-to-end
communication service is established over
multiple network segments managed by different
teams or operators through individual element
configuration. Tools are optimized for specific
services, which are tightly coupled to the
custom-built, vendor-specific network, making
rapid, end-to-end, reliable and secure service
creation nearly impossible and providing almost
no support for dynamic reconfiguration.
In response, network hardware equipment
providers have proposed more programmable
network elements with open control interfaces.
One of the early efforts to simplify
network elements was the Bell Labs Soft
Router [3]. More recent examples include
Bell Labs Technical Journal | V o l u m e 2 4 | 6
 OpenFlow-enabled switches, flexible optical
transponders and cross-connects, and fully
virtualized network functions. These
programmable elements are the building blocks
of the new network fabric. However, to leverage
the flexibility of this new hardware and
to provide visibility of multiple network
segments at scale, a new control platform
will be needed — one that ensures the same
carrier-grade service reliability as before.
Future control plane – the
network operating system
The goal of the network operator is to have a
future control plane that makes networks truly
programmable with rapid service provisioning that
gives rise to a new network-automation paradigm.
The network operator would be able to run new
services on the network as easily as installing a
new app on an iPhone, thus making it analogous to
a computer OS that partitions resources (memory,
compute, etc.) to various applications/services.
In contrast with the computing paradigm, network
operators today provision the network manually
with specific scripts and processes accessed from
a command line interface and laboriously
configure individual ports and network layers. The
provisioning process requires hours or even days
to accomplish complicated changes. To realize the
vision of agile, seamless and efficient operations
the network needs an OS: a coordinated, scalable
management platform with an intuitive interface
for clients and operators that gives the perception
of the network as an adaptive resource supporting
service automation.
This programmable network control plane would
be the equivalent of a computer OS for multi-
layer, multi-domain networks. Such a network
OS would allow customers to precisely express
their requested network services and empower
network operators to automatically and reliably
provision those network services. The role of
the network OS is to satisfy all service level
agreements (SLAs) for new and already-
deployed network services, and ensure efficient
resource allocation across multiple network
Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
layers and domains such as IP, MPLS, Layer 2,
optical, and mobile and fixed access. To fully
realize the benefits of automation, control
should be technology-agnostic and encompass
all network elements in the end-to-end service
delivery chain. This multi-layer, multi-domain
aspect of a network OS is illustrated in figure 1.
Requirements for a multi-layer,
multi-domain network OS
To fulfill the vision of a multi-layer, multi-domain
control platform, the functionality of the network
OS must go beyond that of an OpenFlow network
controller [4]. OpenFlow controllers translate low-
level packet flow decisions into flow table rules
and install them in OpenFlow-enabled switches.
With an OpenFlow controller, the level of
interaction between the customer and the
network is too granular and does not provide
mechanisms to ensure reliable service delivery or
efficient network utilization.
The differentiating characteristics of a network
OS compared with an OpenFlow software-
defined networking (SDN) controller are
summarized here.

Network abstraction: a multi-layer network
modeling language enabling seamless
operation across heterogeneous network
technologies and virtual network functions
with consistent topology representation
across network layers, e.g., unified
representation of heterogeneous
resources in a computing OS, such as a “file”
in UNIX.

Intent framework: a secure northbound
interface that is bidirectional to express
network service requests in the form of
intents and to monitor fulfillment of SLAs.
It is a programmable interface for controlling
the abstracted network slices allocated to
the authorized customer, e.g., high-level,
easy-to-use APIs in a computing OS such as
“POSIX C libraries” in UNIX. The intent
framework hides the multi-layer aspects of
the network to allow expression of resource
needs that may span different network
layers.
Bell Labs Technical Journal | V o l u m e 2 4 | 7
 FIGURE 1. Network fabric controlled by a multi-layer, multi-domain network OS.

Network resource manager (NRM): a
real-time mapper of network service requests
into hardware configurations. The resource
manager is a multi-layer path computation
element that can optimize goals without
human intervention, and includes topology
and hardware configuration compilers with
correctness verification. The resource
manager also keeps track of active and
pending network services, their
characteristics, performance and resources. It
includes algorithms for predicting future
network demands, e.g., resources (CPU,
memory, etc.) and their scheduling; similar in
a computing OS to paging and to process
scheduling in UNIX. These required network
OS functionalities will be further described in
what follows.
Related work
Two major SDN controller platforms attract the
most attention from academia and industry:
Open Daylight (ODL) [5] and Open Network
Operating System (ONOS) [6]. Both are modular,
open-source SDN controller platforms to
customize and automate SDN. At a high level, they
share many architectural concepts, such as high
availability (HA) and model-driven, cluster-based
intent frameworks, but differ on some design
principles. For example, the core of the ODL
architecture is the Model-Driven Service
Abstraction Layer (MD-SAL), which enables the
modeling of network devices/applications using
Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
the industry-standard modeling language YANG
[7] to interact in an abstract way. The ONOS core
has a simple graph abstraction using Java objects
so that YANG-based data models are translated
into Java objects outside of ONOS using the YANG
Management System (YMS). In addition, with
regard to southbound protocols, ODL has strong
support for legacy network devices using industry
standard protocols, such as NetConf/YANG and
TL1 [8], but ONOS supports innovative SDN
protocols such as ONF Transport API [9] and new
programmable hardware forwarding elements P4
[10] [11].
From a use-case perspective, ODL has automated
service delivery, such as transport SDN, cloud and
an SDN controller for Open Network Function
Virtualization (OPNFV), and network resource
optimization, such as DC interconnect, which are
major challenges for many equipment providers.
ONOS is more focused on the CORD (Central
Office Rearchitected as Datacenter) use cases
with R (Residential)-CORD, E (Enterprise)-CORD,
M (Mobile)-CORD and A (Analytics)-CORD [12].
CORD use cases are more aligned with the
operators’ evolution towards using central offices
as distributed data centers, which they can
achieve by combining a data center infrastructure
controller, such as OpenStack, with an
orchestrator, such as XOS. In this sense, ODL is
more equipment provider-centric while ONOS is
more operator-centric. Some research
papers have investigated a performance
Bell Labs Technical Journal | V o l u m e 2 4 | 8
 comparison between the two SDN controller
architectures [13].
From an operating system perspective, the
ONOS platform aims to provide more general
features, such as resource management and
intent framework, and is therefore closer to
NetUNIX. But ONOS still has missing pieces
from a network OS perspective, such as multi-
layer, multi-domain awareness of network
resources represented as layered network
abstractions and real-time SLA satisfaction.
Figure 2 captures the key differentiators of
NetUNIX from other network OSes. In the following
sections, we present the architecture and further
details on key components of NetUNIX.
Architecture
The software architecture of NetUNIX
presented in figure 2 is aimed at providing an
extensible, distributed network OS. The key
advancements that the proposed architecture
brings to current network controllers and
operating systems are:

Native support for network description
through a multi-layer network abstraction
layer

An intent-based multi-layer service
description Application Programming
Interface (API)

Dynamic, multi-layer network resource
management

Transparent support for advanced service,
system and network resiliency features
through novel continuous testing and smart
failover techniques

Native security.
The proposed architecture is based on the
Long-Term Support Linux kernel (LTS) and will
run on commodity server hardware. On top of
the Linux kernel there is the southbound (SB)
layer that is used to communicate with the
network devices. The SB implements an equipment
provider-agnostic interface through a specific
network element abstraction layer (network drivers
as discussed below).
Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
The core of the architecture controls the network
elements and discovers, manages and allocates
network resources (e.g., capacity, links, nodes and
network topology). Above the core, the system
bus delivers data, commands and signals to other
elements of the architecture.
Network abstractions
Network abstraction generalizes the operation of
disparate network technologies and domains by
providing a comprehensive network view
that enables unified, end-to-end network service
control and optimization. It also makes the
network OS equipment provider-agnostic. One of
the key requirements of network abstractions is
to provide a consistent view of the network
resources and connectivity to the resource
manager of the OS. In the market, the de-facto
standard for network models is NETCONF/YANG
[14]. The primary focus of these models is to
configure and monitor the network equipment.
These data models lack flexibility for describing
explicit network connectivity, layering and layer
encapsulation information and lack a defined
mathematical structure for ensuring consistent
network representations.
For the multi-layer, multi-domain network OS,
we have developed a formal network modeling
language to describe network equipment,
topology, connectivity and performance that
consistently represents a comprehensive
network view.1 Our modeling language captures
a snapshot across all layers and domains in an
equipment provider-agnostic manner. It is
flexible enough to generate different levels of
abstraction while mathematically guaranteeing
consistency of the abstracted network views.
More details on formal verification have been
provided by Fortune [15].
In addition, the language semantic check and
visualization programs aid in path validation
over multiple links for multi-layer networks
1. It is based on extensible markup language (XML) for
automatic syntactic and semantic validation. Also, it can
be easily converted between JSON and XML formats by
using many open source tools and library, e.g., JSON.org.
Bell Labs Technical Journal | V o l u m e 2 4 | 9
 FIGURE 2. Network OS architecture.
and automate the presentation of network
resources. Simsarian et al describe an example
of a multi-layer IP and optical network
representation using NetUNIX modeling
language (NetGraph) [16]. Figure 3a shows
a NetGraph visualization of two carrier routers
interconnected by an optical network with
reconfigurable optical add-drop multiplexers
(ROADMs). Figure 3b shows an abstracted view of
the network with only the optical network
termination points. In addition, NetGraph can
optionally contain performance data, such as link
utilization, to enable other modules to better
understand network states, for example, NRM for
dynamic network optimization.
Intent framework
The envisioned control platform is intended for
heterogeneous networks, covering technologies
from optical through IP, MPLS and L2 to wireless.
An aggregation of explicit packet and flow handling
primitives specific to each of these technologies
would lead to a very complex service-provisioning
interface. Moreover, it would require the network
customer to predetermine the exact technology
Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
over which the service is to be transported. This
defeats the purpose of network abstraction and
would limit the ability to dynamically optimize
network resources. Therefore, for the multi-layer
network OS, it is necessary to establish a set
of service description primitives to reflect the
requirements of the service, rather than the
underlying network elements. This should take
the form of a high-level intent language forming
the interface between the application and the
control platform.
The intent language enables network customers
to describe the characteristics of the requested
network service in absolute, technology-agnostic
terms. The role of interpreting those
specifications, finding paths that satisfy
service requirements and compiling them into
network element configurations is delegated
to the control platform. When the requested
service is committed, it is the role of the controller
to track the performance of the service and report
it to the customer. To express the requirements of
most typical services (service SLAs) we have
defined the following set of parameters:
Bell Labs Technical Journal | V o l u m e 2 4 | 1 0
 FIGURE 3. NetGraph visualization of carrier routers interconnected with an optical network: a) L1–L7 refer to different network layers and the
triangles are adaptations between the layers; b) shows an abstracted view of the network.

Connectivity (endpoints, point-to-point, in its use of high-level, service parameter types.

point-to-multipoint, broadcast, path We propose a more complete set of parameters
diversity, load balance) to express a broad range of network services.

Bandwidth symmetry Ferguson et al proposed delegating service

Explicit routes characteristic parsing to the controller, as well

Reliability (availability, protection, latency as the scheduling of future services [18]. In
in recovering from failures) addition to the concepts described in Ferguson,

Security attributes the multi-layer network OS platform performs

Service scheduling and holding time the important role of multi-layer traffic

Guaranteed bit rate placement optimization as well as quality of

Maximum bit rate (maximum bit-rate duty service (QoS) awareness.
cycle)
For example, consider a simple intent request to

Maximum data-plane latency
create a L3 virtual network topology among

Fidelity (upper bound bit-error rate (BER)).
nodes A, D, and C in the IP/optical network
This intent-based approach enables network shown in figure 4 where the network
customers to define the requirements of connections have bandwidth, latency, and
interest while delegating the configuration of mutually diverse requirements. This service
network parameters entirely to the automated intent could also have a further requirement of
control platform. This approach is similar to the path diversity. The intent framework translates
concept of software-friendly networks [17] this high-level intent specification into three

Bell Labs Technical Journal | V o l u m e 2 4 | 1 1

Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
 FIGURE 4. Mapping an intent request for a virtual subnetwork among the nodes A, D, and C into mutually disjoint connections between those
nodes that are then configured onto the network.

point-to-point intents that satisfy the service
criteria. The service criteria are each captured
in a specific grammar as shown in figure 5. The
intent framework compiles the set of grammars
to define the specific resource requests to the
network OS.
Network resource manager (NRM)
The managed network is composed of standalone
networking communications equipment
distributed over a (potentially large) geographic
area. Efficient use of such resources has
traditionally required careful pre-deployment
planning, first to understand the potential traffic
and services, and then to design network
interconnections and capacities, including
protection and restoration capacity, using
sophisticated combinatorial algorithms to find
reasonable, cost-efficient designs. Once
deployed, the resulting network then operates
with real-time traffic having a combination of
distributed protocols in the higher layers (e.g.,
MPLS and IP) for adapting to dynamic flows that
ultimately are transported over a quasi-static and/
or pre-planned OTN and optical infrastructure.
Realizing the promise of programmable networks,
however, requires the network resources
themselves to be managed much more
dynamically and adaptively at all layers on a time
scale much shorter than for offline planning.
Algorithms and associated decision logic is needed
to enable efficient management. And control of
programmable network resources and services
takes a more central role than in traditional
computer OSes.
Essential for network resource management are
algorithms that produce and manage a mapping
of services to network resources. Generally, a
user’s service request has components that
require fulfillment by network and non-network
resources such as application servers, web
servers, data centers or computation farms.
The focus here is only on the network
component of a service request (see figure 6).
Thus, a network service (hereafter referred to
as a service) is a delivery of digital content or
bandwidth by a service provider network to
and/or from user-designated interface locations
according to user-required performance
characteristics and attributes. Services are

Bell Labs Technical Journal | V o l u m e 2 4 | 1 2

Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
 FIGURE 5. Intent grammar processed by the Intent Framework for the example in figure 4.
assumed to have associated SLAs specifying
their performance requirements.
The foregoing definition allows for a broad range
of services, from high-level content or cloud
interconnection to lower-level intersystem
federation or point-to-point transport wavelength
services. Resources are installed network
capacities that can be assigned tasks confusing to
support fulfillment of service requests. Resources
can be:

Elemental, such as wavelengths on specified
point-to-point optical links

IP or optical ports on specified devices at a
node

A bandwidth fraction of an IP port or
composite, such as a light-path, depending
on which objects an algorithm is designed to
operate

Radio channels or codes

Timeslots in fixed access systems.
Importantly, resources of like kind are fungible
as far as the services are concerned. As long as a
service’s SLA is being met, it does not matter
exactly which network resources are being used
Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
or whether they remain fixed throughout the
service fulfillment.
All requests for services by network users or
third parties are assumed to enter the network
through a logically centralized point via the
northbound interface (NBI) or the Intent API
(see figure 4). A request entering the NBI will
have already been separated out by a higher
layer application interface into the network-only
intent, which will be converted into (if not
already so expressed in) a standardized
service description or intent language. The
requests are then mapped to the network in three
stages.
In the first stage, as shown in figures 4 and 5, the
Intent Framework translates the network service
intent requests into a set of valid resource-
consuming demands. Demands associated with a
service are a set of network interface points that
must be interconnected in accordance with the
service’s SLA to provide for its fulfillment. They are
data objects that comprise inputs suitable for
algorithms. Demands are expressed in connection
units consistent with the network’s transport
Bell Labs Technical Journal | V o l u m e 2 4 | 1 3
 FIGURE 6. Overall service request is decomposed into network and non-network components. In stage 1, the network component is pre-
sented to the NetOS, where the Intent Framework (IF) translates the network intents into valid network resource-consuming requests, as
shown in figures 4 and 5.
resources and capabilities, such as units of
bandwidth for Label-Switched Paths (LSP) or units
of wavelengths (at maximum line rate), and they
can include other attributes and parameters
needed for algorithm selection and constraint
instantiation. The intent framework, with inputs
from the policy engine, performs the conversion of
service requests from the standardized service
description language form into algorithm-
understandable demand data objects. The SLA
characteristics and operator policies are also
compiled into the demands as attributes, such as
performance and constraint parameters. The result
of the first stage is a validated resource allocation
problem that needs to be solved for the service
requests to be realized.
The second stage of the mapping, pictured in figure
7, consists of assigning demands to network
resources consistent with the demands’ attributes.
Using the demands as inputs, this stage is
performed by a family of algorithms available in
NetUNIX. They operate on abstractions of actual
network resources and their associated capabilities
to produce, as output, assignments
of demands to suitable network resource
abstractions. The resource allocation problems
to be solved are typically challenging network
optimization problems of varying types.
Some of these may be most efficiently solved by
leveraging new or existing specialized or
purpose-built algorithms best suited to address the
Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
problem type. To exploit such potential efficiency
gains, demands present in a provisioning cycle
would be classified according to pre-defined
provisioning problem types for which specialized
algorithms are available in NetUNIX. The order of
execution of each sub-problem would be based on
logic that accounts for considerations such as
demand type prevalence, priorities and sub-rate
containment. Demands with additional
performance characteristics (such as delay
tolerance) could request end-to-end connections
over full wavelength or sub-wavelength LSPs. Based
on traffic classification by the policy engine, the
demands are mapped to corresponding network
resources using special-purpose algorithms for
the traffic type.
The third stage of the process first verifies that the
abstracted allocation plan of the intents can be
realized on the network. It then translates the
abstract plan into a realizable resource allocation.
This latter step is addressed by the southbound
interface (SBI) described below.
Data analytics in all its forms, both classical
methods as well as cognitive-based methods, are
a key part of the functioning of the system and will
serve to provide key decision support throughout
the NetOS. For example, to reduce the impact of
some of the more time-consuming control and
decision algorithms, data analytics will allow for
the anticipation of service requests or network
issues. Although analytics can be used to trigger
Bell Labs Technical Journal | V o l u m e 2 4 | 1 4
 FIGURE 7. In Stage 2, the resource allocation problem defined by the Intent Framework (IF) is solved by the Network Resource Manager (NRM).
The result is a mapping of the demands required for the network services to network resources.
service reconfiguration, the changes are still
provisioned as “demands”. Analytics may also be
used to support demand classification by the
policy engine, for instance, the use of historical
traffic monitoring data to help classify the most
appropriate algorithm family to use for realizing a
service request. Cognitive-based methods such as
various forms of machine learning will be exploited
for their pattern recognition abilities, such as to
identify recurrent traffic request patterns or to
recognize emerging resource usage patterns that
should be avoided. To obtain the data to employ
such analytics capabilities, it is assumed that there
will be a scalable performance and traffic-
monitoring infrastructure architected and
designed. It will harvest, filter and feed essential
IP, optical and service performance and traffic
information into the data analytics system. Then
fast, scalable analytics and machine-learning
algorithms will be developed to mine the data
for actionable decision inputs.
There are many challenges to creating
algorithms for programmable networks. The
algorithms include the coordination of service
provisioning across network layers, for instance
between the IP/MPLS layers and the OTN/optical
Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
layers or between IP layer flows and the TDM of
a fixed access domain, to ensure optimum
resource utilization. This may entail grooming
lower rate services in and out of higher speed
facilities multiple times in transit, confusing. It is
well known that solving this kind of problem,
even offline, for multiple demands in a
provider-scale network is computationally
complex. However, to be successful for network
control, the targeted algorithms need to be
sufficiently responsive and agile to maintain
synchronization with the network state dynamics
while providing a net benefit to network
resource and service management.
Thus, what will be needed are fast, scalable,
possibly cross-layer-aware combinatorial
algorithms and processes that provide
sufficiently high-quality solutions to NP-hard
(non-deterministic polynomial-time hard)
problems that are time bound. Further, the overall
algorithmic approach needs to be suitable for an
online, on-demand network-programming
paradigm and maximize network value to its
operator. In other words, fast algorithms need to
be able to generate good solutions whose quality
depends on the amount of time available to
Bell Labs Technical Journal | V o l u m e 2 4 | 1 5
 compute them. One way to realize this is to have
algorithm design and development guided by the
following principles.
Maximize utilization of installed network
capacity
The network should run as “hot” as the network
control will allow. Installed network capacities are
fungible resources. They should be adapted as
needed to accommodate present and
anticipated network services and events, while
allowing sufficient headroom for the unexpected.
Expected services can be designed and
provisioned under a given failure scenario, and
installed protection capacity can be used for
lower priority, pre-emptible or reconfigurable
traffic. This requires development of agile
algorithms for near-real-time resource-usage
management and frequently retuning service
provisioning to maintain service reliability and
efficient network utilization.
Analytics-driven capacity utilization via service
anticipation and fault prediction
Data analytics are used to buy time for
optimization and possibly other complex
NetUNIX processes by anticipating event
occurrences and initiating or preparing for such
processes as early as possible. This would entail
leveraging near-real-time streaming analytics of
traffic/usage data or performance data to
inform event prediction. To extract the
maximum information from the available data,
analytics should leverage all methodologies such
as cognitive-based techniques including, but not
limited to, machine learning.
Speed versus quality optimization solution
tradeoff
Network optimization algorithms should be
designed to run under time constraints, trading
off optimization solution quality for speed.
To offset the loss of design-quality optimizations,
frequent retuning of service provisioning is
performed on the network.
Creating algorithmic solutions according to these
principles is a key ingredient to enabling an online,
Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
on-demand programmable networking paradigm.
Figures 8 illustrates an integrated view of the
resource management system interworking with
the intent framework, the policy engine, the SBI,
as well as data analytics both external and internal
to the NRM component.
Network drivers (SBI)
The computational output of NetUNIX consists
of data objects that are abstractions of network
resources. For connectivity service requests from
NetUNIX users or apps, the stage of mapping
services to the network is achieved by
converting those data object outputs into
actionable configuration directives for the
network elements and then executing them, as
shown in figure 9.
The abstractions must first be mapped to actual
network resources, managed by the resource
manager and then a schedule of (re)configuration
tasks, which are managed by the intent manager,
must be generated to efficiently sequence the
provisioning operations and to avoid conflicts
or wasteful configuration changes. Once all
configuration operations are scheduled, they
must be converted into network-element-specific
instructions understandable by the affected
elements, which are managed by the flow
manager and device drivers and then executed via
southbound protocols such as OpenFlow,
Representational State Transfer (REST), or open
source Remote Procedure Call (gRPC). In
addition to installing actionable
configuration directives into network elements,
network drivers are responsible for automatically
populating network topology information (e.g.,
device, port, link and host) into NetUNIX and
continuously collecting flow statistics and port
statistics through communication with network
elements, including OpenFlow devices and non-
OpenFlow devices.
Resilience of the network OS
In carrier-grade legacy systems, such as telephone
switches, the switch hardware is more reliable than
the control software because the hardware
includes self-checking circuitry and hardware-level
Bell Labs Technical Journal | V o l u m e 2 4 | 1 6
 FIGURE 8. In Stage 3, the resource allocation plan is verified as feasible and then handed over to the Southbound Interface (SBI) where it is
converted into device-understandable instructions and then provisioned into the network. Also shown is a more detailed unified view of the
interworking of the NBI, IF, NRM, SBI as well as the Policy Engine and various analytics supporting overall NetOS operation.

FIGURE 9. A prototypical southbound interface (SBI) converting data object outputs into actionable configuration directives for the network
elements.

Bell Labs Technical Journal | V o l u m e 2 4 | 1 7

Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
 fault masking and recovery capabilities. Only a few
errors are handled by the software stack, which in
turn is typically equipped with simple recovery
mechanisms that include test-retry techniques or
warm/cold replica activation.
As in legacy systems, the dependability of the
network OS is also determined by the
effectiveness of its resiliency mechanisms.
However, in contrast to legacy systems, the
distributed nature of the network OS core
requires more advanced resiliency mechanisms
that are able to guarantee consistency across
replicas while keeping low failure detection and
recovery times. In addition, due to the different
configurations (e.g., multi-vendor) and different
requirements of delivered services, resiliency
mechanisms have to be flexible and
continuously adapt to different working
conditions.
As described in the architecture section, the
resiliency of the network OS is achieved by means
of the Resiliency Framework that includes the
following components: error detection manager,
high availability manager, failover manager and
continuous testing framework, as shown in
figure 10. These components work in a pipeline as
follows: the error detection manager deploys
service-level and system-level error detection
on-demand, e.g., whenever a new service is
provisioned, customized to the specific SLA; the
high-availability manager computes specific KPIs
to detect system-level dependability threats and
trigger recovery actions; the failover manager,
triggered by the high-availability manager, selects
the best system- or service-level recovery action
to perform recovery without impacting the
provisioned services; and the continuous testing
manager, continuously injects defects (e.g.,
failures) at the system and service level to identify
potential resiliency bottlenecks and solutions in a
controlled fashion.
The overall goal of the Resiliency Framework is to
detect controller errors with low latency and to
failover to an active standby component without
impact on the provisioned services and without
Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
losing state information. It ensures resiliency
against failure of control functionality with respect
to dependability metrics such as availability,
reliability and resilience.

The availability metric is the probability that a
system is operating as expected at any given
time. This metric represents the percentage
of time (e.g., 99.99 percent or 4 “nines”) in
which services are provisioned, as specified in
the SLA with the customer, for example with
specified latency and capacity.

The reliability metric is the probability of
continuous successful service delivery for a
given time t. This metric represents the
probability that a service can be provisioned
without interruption for t. This metric
captures the probability that a service
completes successfully with required
termination, including release of used
resources. This is related to the inverse metric
of the dropped-call rate per million (DCR) in
legacy systems, i.e., the fraction of telephone
calls that, due to technical reasons, were
terminated before the service ended.

The resilience metric is the probability of
continuous successful service delivery in spite
of failures that occurred in the system. This
metric depends on the specific delivered
service and captures the capability of the
system to recover from a given number of
consecutive failures without impact on the
delivered service.
The resiliency framework computes the above
metrics with data provided by the data-driven
architecture of the network OS, which logs events
at fine granularity and performs analytics to
detect anomalous, service-impacting conditions.
The level of delivered service can be monitored
and compared against SLAs to track compliance
through the data provided by the data-driven
architecture and the metrics computed by the
resiliency framework. SLAs and service
requirements are specified with intents that are
rich in reliability and availability attributes, which
allow for translating SLAs to a set of
corresponding low-level rules that are enforceable
Bell Labs Technical Journal | V o l u m e 2 4 | 1 8
 FIGURE 10. Continuous testing framework consisting of an error detection manager, high availability manager and failover manager.
in various elements of the OS and the network.
Modeling enables service quality and performance
under failure conditions to be evaluated and
dynamically validated by a continuous testing
framework that leverages context-aware fault
injections.
The resiliency of the system is continuously
measured by analyzing the response of the
system to unpredicted events. The continuous
testing framework simplifies creation of failure
(failure injection) within the network OS ecosystem
and provides a greater degree of precision to
identify resiliency bottlenecks and proper recovery
actions for the created failure. The main objective
of the continuous testing framework is to reduce
the offline failure testing time of the network OS
(services and applications) by doing it online, i.e.,
during production hours. This is done by injecting
realistic failure conditions in a controlled fashion
to establish confidence that the system degrades
gracefully with tolerable (i.e., within the SLA
parameters) or no impact on the provisioned
services. The workflow of the continuous testing
framework is shown in figure 10.
Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
The fault-injection compiler selects failures according
to the operational context and history. It then
customizes the failure injection and activation
policies. The system watcher component performs
continuous monitoring of the failure, and evaluates
the impact of the injected failure with respect to
detection, recovery and downtime (see figure 11).
Data from different network OS components are
collected after failure injection and are delivered
to the data-driven architecture that stores them
in the form of failure-fingerprints, i.e., a set of
features extracted by the data associated with
the injected failure. The resiliency framework then
uses the failure fingerprints to continuously train
error-detectors specific to the injected failure
in order to increase the failure coverage of the
resiliency framework.
Based on the assessment of the controlled failure
injection, the appropriate failure recovery
procedures are implemented to address any
impending failure scenarios. The types of failure
injections supported on the continuous testing
framework are shown in figure 11.
Bell Labs Technical Journal | V o l u m e 2 4 | 1 9
 FIGURE 11. Types of failure injections supported by the continuous testing framework.
Security of the network OS
Security of the network OS is essential as
programmable networks can be dynamically
reconfigured through software. Security issues
can arise in the control, application and data
planes and in the interactions between these
planes. The network OS should address
complementary aspects such as:

Access control through rules governing the
degree of dynamic network programmability
given to network applications

Identification of security vulnerabilities in all
the planes through analytics

Infrastructure that enables real-time
mitigation of these identified vulnerabilities
through dynamic network reconfiguration

Analytics-enhanced, automated verification
to identify security issues early in the
software lifecycle.
As network OS software enables real-time
network programmability, the power of this
programmability also intensifies the potential
impacts that software faults can have on
the security, reliability and performance of
programmable networks. We have demonstrated
that subtle faults in network applications can have
disastrous impacts on programmable networks
and that faulty behaviors can be detected
and identified during testing and deployment
through machine learning [19].
Given the significant impact on programmable
network behavior, it is imperative to have a high level
Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
of confidence that network OS controller software
and applications do not contain subtle software
faults. At the same time, it is not possible to
exhaustively analyze network OS software. Given the
high degree of multi-threading and distribution, the
number of potential states and execution paths is
astronomically high. For the same reasons, subtle
software bugs are also unlikely to be discovered
during system testing. We have created an approach
and framework for combining automated code
verification with machine-learning-based analytics
to detect and identify SDN software faults that can
compromise the network. Our approach is based on
probabilistic sampling of timed execution of SDN
software combined with anomaly detection.
Identified anomalous execution paths can then
be replayed to detect potential software faults.
Our high-level approach is depicted in figure 12.
Multi-domain federation
As described in the introduction to the network
OS, we envision end-to-end (E2E) communication
service delivery. Typically, the E2E service is
deployed across multiple domains including DC,
access and metro/core networks.
A comprehensive and abstracted view of
reachability, network topology, security policy and
computing resources must be provided for the
service deployment across multiple domains.
There is, to the best of our knowledge, no study on
the multi-domain aspect of the network OS. ONOS,
as a network OS, provides a clustering of multiple
controller instances in a single domain with their own
Bell Labs Technical Journal | V o l u m e 2 4 | 2 0
 FIGURE 12. Detecting reliability, performance and security issues by learning on timed-execution paths.
distributed state-sharing tools. The CORD model can
provide service chaining in a single data-center [20],
together with ONOS. However, these approaches do
not address the E2E multi-domain requirement
described above. Rather, they focus on independent
scenarios, such as CORD for residential, enterprise
and mobile.
Motivated by the need for multi-domain control,
we developed a federation framework that can
provide E2E service such as service chaining, load
balancing or replication across multiple domains.
As shown in figure 13, our framework is
implemented as an E2E network hypervisor
over the top of multiple network operating
systems and is a key application of the network
OS. It enables interaction between multiple
domain controllers, including multiple DC
controllers, metro network controllers and
Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
core network controllers, if necessary, thus
exposing a comprehensive network view to
other applications such as a service-chaining
orchestrator.
Use cases
The following section illustrates the use of the
multi-layer, multi-domain network OS for
potential carrier application scenarios. We
explain how the described concepts are
applicable to metro-data center interconnects
and to the convergence of wireless and wireline
access networks.
Metro-distributed data center
The expected move towards edge-cloud
services will result in increased metro-network
traffic and a need for dynamic multi-layer
control. This is reflected in a study that
Bell Labs Technical Journal | V o l u m e 2 4 | 2 1
 FIGURE 13. Multi-domain federation enabled by Network OS.
has predicted 560 percent growth in
metro traffic, which includes forecasted
data center (user-to-DC and
DC-interconnect) traffic increase of more
than 440 percent [21].
Flexible and reconfigurable DC optical
interconnection
Flexible physical-layer optical transport
technologies have emerged to keep pace with the
growing metro-network bandwidth needs.
Flexible-rate transponders have been introduced
that support 400G using 64-quadrature
amplitude modulation (64-QAM) as well as 100G
with quadrature phase-shift keying (QPSK).
The 64-QAM modulation has shorter reach
than the 100G QPSK format, which is a natural
consequence of optical transmission approaching
the Shannon limit on single-mode fiber. However,
the shorter reach and high spectral efficiency of
the higher-order modulation formats are well
suited to the shorter metropolitan network
reaches — creating a further incentive towards
edge-cloud DCs. In addition, flexible and
transparent optical networking equipment, such as
reconfigurable optical add-drop multiplexers
(ROADM), have been deployed in metro networks
to allow for dynamic wavelength routing between
DC sites.
Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
Until now, the flexible capabilities of physical-layer
hardware, such as adjustable transponder bit-rate
ROADM wavelength routing and transponder
wavelength tuning, have been largely under-utilized.
Optical networks are typically operated in a “set
and forget” manner where the flexibility is useful
for network planning and deployment,
but the connections remain static thereafter.
The network OS has the potential to improve
network operation efficiency by allowing dynamic
reconfiguration of the IP/optical network to adapt
to bandwidth demands. Furthermore, network OS
applications can optimize the transponder
operating parameters such as the bit rate,
modulation format, or even the policy parameters
for probabilistic constellation shaping.
Intelligent dynamic traffic engineering
Our network OS also enables new, low-cost
approaches to achieve high network resiliency using
multi-path transport [22] [23]. It extends traffic
engineering and routing to accommodate inter-DC
traffic as well as intra-DC traffic. Key components
are dynamic and have elastic wavelength
management and federation among multiple-
domains. Presently, the utilization of inter-DC links
is known to be on average less than 20 percent [24].
Here the link utilization is measured from the DC
operator’s perspective, which does not account for
possible additional over-provisioning such as 1þ1
Bell Labs Technical Journal | V o l u m e 2 4 | 2 2
 FIGURE 14. Next-generation fronthaul traffic management with flow-delay prioritization.
wavelength protection by the network operator.
With traffic engineering and routing powered by a
network OS, link utilization can be significantly
improved compared to existing DC operator
solutions, such as B4 [25] and SWAN [26], which do
not interact with the wide-area network.
Using SDN-enabled multi-path transport, data
between the DC locations can be evenly
distributed across the paths using the topology
information supplied by the network OS
combined with traffic engineering at the edge
switches. The method gives high utilization of
available network resources with built-in
resiliency for low-cost DC interconnection.
Competing methods for distributing flows across
paths, such as equal-cost multipath (ECMP)
routing, require a statistically large number of
flows to achieve equal traffic distribution, which
is not always the case for DC interconnection.
Most alternatives, including ECMP, operate
statically and do not adjust to changes in the
network, such as flow arrival variability, outages
or impairments, with resulting inefficiencies (e.g.,
bandwidth over-provisioning or protection
wavelengths). By contrast, this dynamic traffic
distribution is based on real-time monitoring of
traffic flow statistics and network status allowing
it to adjust to the network state and preserve
high-availability without the need for backup
wavelengths.
Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
5G next-generation fronthaul
The evolution of access networks points to a
converged transport infrastructure with the fiber
access network as the cornerstone. Operators are
seeking to simplify the networks, for instance by
reducing the overhead associated with operating
multiple access networks for different services
(residential, enterprise or mobile transport) and in
terms of connectivity (e.g., improving utilization of
networks that have been over-provisioned for
peak capacity) but still some flexibility to satisfy
latency requirements from different architectural
choices.
In line with these ambitions, the industry is
working on several network architectures in the
medium term, such as consolidating fixed access
end offices to reduce real estate and
operational costs, and distributing the
architecture to minimize the access-specific
portion of the network. The key element driving
this evolution is the need to support new types
of services derived from the advent of 5G. It is
commonly understood that to implement 5G
features, such as with an extended set of
collaborative multi-point (COMP) schemes,
architectural changes will be introduced in the
form of RAN centralization. This will not only
enable maximizing spectral efficiency (capacity)
and network resource utilization, but will also
enable operators to profit from the advantages
Bell Labs Technical Journal | V o l u m e 2 4 | 2 3
 brought by the cloud paradigm such as
processing pooling gains and cost savings. But
this savings will come at a cost, massively
increasing demand for transport capacity
accompanied by an equally challenging decrease
in required latencies.
Delay-aware, next-generation fronthaul
traffic management
A centralized RAN configuration splits the
cellular base-station functionality, placing each
portion in different locations (cell site, central
office). The distribution of functions can take
several flavors as introduced in [27] [28]. Yaun
et al argue that segmenting the signal
processing within the physical layer delivers all
the benefits of key 5G features, while
considerably relaxing the bandwidth
requirements (compared with existing fronthaul
(FH) approaches such as CPRI) to the levels of
commercial ethernet links [29]. The so-called
next-generation FH interface, otherwise known
as mid-haul (MH), also has the characteristic
of varying with traffic load and, thus,
allows exploiting multiplexing gains at
the transport level.
To this end, multi-wavelength TDM-PON (e.g.,
NG-PON2) is well suited for aggregating traffic
of different services and to building converged
access within a converged infrastructure. The
advantages of leveraging a mature technology,
such as PON, for MH transport are manifold:
inbuilt key functions, such as QoS,
synchronization and security; and alleviation of
operations and management challenges such as
scalability, upgrades and utilization.
Injecting a new service will, however, call
for improvements in the traffic-handling
capabilities. A low-latency protocol operating
over the physical layer must be ensured, as
demonstrated by Anthapadmanabhan et al
[30]. In addition, the traffic manager, which
occupies a central role in providing QoS at
the aggregation level, becomes one of the
elements where SDN can be applied in the
access domain [31].
Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
Whereas the current approach to QoS is to
provision each fixed access network element for
extended periods, a dynamic and automatic
traffic manager will (semi-) persistently configure
the QoS function following traffic variations that
occur in short time scales. Such functionality can
be enabled by a control entity, which has visibility
on the E2E service performance and allocation of
timing/bandwidth constraints on the
intermediate heterogeneous network domains,
as illustrated on the left-hand side of figure 14.
Besides maximizing network utilization, this
operational mode permits prioritization of MH
packets with little time left, given the stringent
delay budget (on the order of hundreds of
microseconds [32]), as illustrated on the
right-hand side of figure 14.
Summary
A network OS with multi-domain, multi-layer
capabilities is an essential part of the future network
fabric. It will simplify and optimize network
performance while managing faults, dynamic
demand and ongoing system evolution. We have
discussed the requirements and described the
implementation of NetUNIX, an example of a
network OS that can control heterogeneous
networks at large scale and with high reliability. We
demonstrated how NetUNIX can be applied in just
two of several use cases — data center
interconnection and 5G fronthaul — to span
different network technology domains, such as one
might find in a typical telecom carrier network.
NetUNIX can provide true dynamic network
reconfiguration and enhanced service agility, both
urgent requirements being placed on operators by
the ever-growing embrace of cloud-based services,
which are prerequisite for coming 5G services, e.g.,
network slicing.2
2. Network slicing stands for a concept in which the
physical network infrastructure is logically partitioned to
serve multiple, possibly disparate, sets of business
requirements so service-centric definition is an end-to-
end virtual private service as a fundamental enabler for
new value generation, offering a platform for agile net-
work service creation in response to specific require-
ments of vertical segments, e.g., enterprise, industry.
Bell Labs Technical Journal | V o l u m e 2 4 | 2 4
 References
[1] M. Weldon (ed.), The Future X Network: A Bell
Labs Perspective, Boca Raton, FL, USA: CRC
Press, 2015.
[2] AT&T, “Domain 2.0 Vision White Paper,” AT&T,
13 November 2013. Available: https://www.
att.com/Common/about_us/pdf/AT&T
Domain 2.0 Vision White Paper.pdf
[3] R. Ramjee, F. Ansari, M. Havemann,
T.V. Lakshman, T. Nandagopal, K. Sabnani and
T. Woo, “Separating Control Software from
Routers,” Int. Conf. on Communication
System Software and Middleware, Delhi,
2006.
[4] D. Kreutz, F. M. V. Ramos, P. Esteves Verissimo,
C. Esteve Rothenberg, S. Azodolmolky and
S. Uhlig, “Software-Defined Networking: A
Comprehensive Survey,” Proc. of the IEEE, vol.
103, no. 1, pp. 14–76, 2015.
[5] OpenDaylight. Available: https://www.
opendaylight.org/
[6] Open Network Operating System. Available:
http://onosproject.org/
[7] M. Bjorklund (ed.), YANG - A Data Modeling
Language for the Network Configuration
Protocol (NETCONF), IETF Proposed Standard,
RFC 6020, October, 2010.
[8] Transaction Language 1. Available: https://en.
wikipedia.org/wiki/Transaction_ Language_1
[9] C. Qiaogang, E. Segev, E. Varma, G. Zhang,
H. Ding, I. Busi, J. He, K. Sethuraman, L. Ong,

pez,
N. Davis, R. Vilalta, S. Bellotti and V. Lo
Functional Requirements for Transport API,
Open Networking Foundation Standard,
TR-527, Jun. 2016.
[10] The P4 Language Consortium, Available:
http://p4.org/
[11] P. Bosshart, D. Daly, G. Gibb, M. Izzard,
N. McKeown, J. Rexford, C. Schlesinger,
D. Talayco, A. Vahdat, G. Varghese and
D. Walker, “P4: Programming Protocol-
Independent Packet Processors,” ACM
Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
Sigcomm Computer Communications Review
(CCR), vol. 44, no. 3, Jul. 2014.
[12] CORD: Reinventing Central Offices for
Efficiency & Agility. Available: http://
opencord.org/
[13] D. Suh, S. Jang, S. Han, S. Pack, M.-S. Kim,
T. Kim and C.-G. Lim, “Toward Highly Available
and Scalable Software-Defined Networks
for Service Providers,” IEEE Communications
Magazine, vol. 55, no. 4, pp. 100–107,
Apr. 2017.
€ nwa
[14] J. Scho €lder, “Network Configuration
Management with NETCONF and YANG”
presented at 84th IETF Meeting, Vancouver,
2012.
[15] S. Fortune, “Equivalence and generalization
in a layered network model,” Journal of
Computer and System Sciences, vol. 81,
no. 8, pp. 1698–1714, 2015.
[16] J. E. Simsarian, N. Choi, Y.-J. Kim,
S. Fortune and M. Thottan, “NetGraph Data
Model Applied to Multilayer Carrier
Networks,” OFC 2016, Anaheim, CA, USA,
20–24 Mar. 2016.
[17] K.-K. Yap, T.-Y. Huang, B. Dodson, M. S. Lam
and N. McKeown, “Towards software-friendly
networks,” Proceedings of the First ACM
Asia-Pacific Workshop on Systems, ser.
APSys ’10, New York, NY, USA,
pp. 49–54, 2010.
[18] A. D. Ferguson, A. Guha, C. Liang, R. Fonseca
and S. Krishnamurthi, “Participatory
networking: an API for application control
of SDNs,” Proceedings of the ACM SIGCOMM
2013 Conference on SIGCOMM, ser.
SIGCOMM ’13, New York, NY, USA, pp. 327–
338, 2013.
[19] L. Jagadeesan and V. Mendiratta,
“Programming the Network: Application
Software Faults in Software-Defined
Networks.” Proceedings of the IEEE
International Symposium on Software
Bell Labs Technical Journal | V o l u m e 2 4 | 2 5
 Reliability Engineering Workshops (ISSREW),
pp. 125–131, 2016.
[20] “Central Office Re-architected as a
Datacenter (CORD),” AT&T and
Open Networking Lab White Paper,
Jun. 2015. Available: https://wiki.
onosproject.org/download/attachments/
3444016/Technical Whitepaper-CORD.pdf?
version=1&modificationDate=
1434143584069&api=v2
[21] “Metro Network Traffic Growth: An
Architecture Impact Study,” Bell Labs
Strategic White Paper, Murray Hill,
NJ, USA, Dec. 2013. Available: http://www.
tmcnet.com/tmc/whitepapers/documents/
whitepapers/2013/9378-bell-labs-
metro-network-traffic-growth-an-
architecture.pdf
[22] Y.-J. Kim, J. Simsarian and M. Thottan,
“Cross-Layer Orchestration for Elastic and
Resilient Packet Service in a Reconfigurable
Optical Transport Network,” Proc. OFC 2015,
paper Tu2B.2, 2015.
[23] Y.-J. Kim, J. Simsarian and M. Thottan,
“Software-Defined Traffic Load Balancing for
Cost-Effective Data Center Interconnection
Service,” Proc. IEEE Inter. Sym. on Integrated
Network Management, May 2017.
[24] S. Jain, A. Kumar, S. Mandal, J. Ong,
L. Poutievski, A. Singh, S. Venkata,
J. Wanderer, J. Zhou, M. Zhu and
J. Zolla, “Experience with a globally-deployed
software-defined WAN,” ACM SIGCOMM
Computer Communication Review, vol. 43,
no. 4, pp. 3–14, Aug. 2013.
[25] S. Jain, A. Kumar, Su. Mandal, J. Ong,
L. Poutievski, A. Singh, S. Venkata,
J. Wanderer, J. Zhou, M. Zhu, J. Zolla,
Authorized licensed use limited to: UNIVERSIDAD CARLOS III MADRID. Downloaded on February 20,2020 at 07:32:52 UTC from IEEE Xplore. Restrictions apply.
€ lzle, S. Stuart and A. Vahdat, “B4:
U. Ho
Experience with a globally-deployed
software defined WAN,” Proc. ACM
SIGCOMM 2013 Conference on SIGCOMM,
Aug. 2013.
[26] C.-Y. Hong, S. Kandula, R. Mahajan, M. Zhang,
V. Gill, M. Nanduri and R. Wattenhofer,
“Achieving high utilization with software-
driven WAN,” Proc. ACM SIGCOMM 2013
Conference on SIGCOMM, Aug. 2013.
[27] Next Generation RAN Architecture. Available:
http://www.xran.org/
[28] R. Knopp, N. Nikaein, C. Bonnet,
F. Kaltenberger, A. Ksentini and R. Gupta,
“Prototyping of Next-Generation Fronthaul
Interfaces (NGFI) Using Openairinterface,”
OpenAireInterface, 2018. Available: http://
www.openairinterface.org/?page_id = 1695
[29] Y. Yuan, I. Chih-lin, J. Huang, S. Ma, C. Cui
and R. Duan, “Rethink fronthaul for soft RAN,”
IEEE Communications Magazine, vol. 53,
no. 9, pp. 82–88, Sep. 2015.
[30] N. Prasanth Anthapadmanabhan, A. Walid and
T. Pfeiffer, “Mobile fronthaul over
latency-optimized time division
multiplexed passive optical networks,”
IEEE Inter. Conf. on Communication
Workshop (ICCW), 2015.
[31] B. Kozicki, N. Olaziregi, K. Oberle, R. B. Sharpe
and M. Clougherty, “Software-defined
networks and network functions virtualization
in wireline access networks,” GLOBECOM
Workshops (GC Wkshps), vol. 2014,
pp. 595–600, 8–12 Dec. 2014.
[32] ITU-T Recommendation G.989.1 “40-Gigabit-
capable passive optical networks (NG-PON2):
General requirements, Amendment 1”,
Aug. 2015.
Bell Labs Technical Journal | V o l u m e 2 4 | 2 6