NEW EBOOK:

DATA CENTER NETWORK

AUTOMATION. SIMPLIFIED.
Mike Capuano
Table of Contents
Introduction...........................................................................................................................................3

Why Every Size IT Team Should Strive to Implement a Software-Defined Data Center...........................4
Public vs. Private Cloud......................................................................................................................................................... 4
Software-Defined Data Center............................................................................................................................................. 5
Pluribus Networks – Putting SDDC and Private Cloud Within Reach......................................................................... 6

A Foundation of Open Networking for Network Automation..................................................................7

Open Networking Has Matured........................................................................................................................................... 7
Open Networking Switches Are Becoming More Powerful........................................................................................... 9
Support has Matured...........................................................................................................................................................10
Brownfield or Greenfield.....................................................................................................................................................10
Bottom Line............................................................................................................................................................................10

The “Easy Button” for SDN Control of Physical and Virtual Data Center Networks...............................11
The Underlay..........................................................................................................................................................................11
Controller-based vs. Controllerless SDN Underlay Automation................................................................................12
The Overlay.............................................................................................................................................................................12
A New Approach to Overlay Fabric Networking.............................................................................................................14
Underlay and Overlay Unified – It Just Works................................................................................................................16

Network Analytics Without Probes, TAPs and Packet Brokers..............................................................17

Traditional Approaches to Analytics.................................................................................................................................17
Another Approach to Telemetry and Analytics..............................................................................................................19
Insight Analytics....................................................................................................................................................................19

The Importance of Network Segmentation for Security and Multi-Tenancy.........................................21

Automating Network Segmentation................................................................................................................................21
More Efficient Utilization of Security Devices.................................................................................................................22
Integrated Telemetry and Analytics..................................................................................................................................23

Summary.............................................................................................................................................24

Additional Resources...........................................................................................................................25

2  |  Data Center Network Automation. Simplified. pluribusnetworks.com

Introduction
We are living in a multi-cloud world where certain workloads work best in specific cloud
environments – large hyperscale public clouds, private hosted clouds or on-premises
(on-prem) or colocation (colo)-based private clouds. A recent survey by IDC found that
approximately 75% of workloads will reside in hosted private cloud or in on-prem or
colo-based data centers supporting end-user-owned private clouds. In light of this, data
center operators need to build IT foundations for private cloud that are agile and that enable
IT teams to focus on outcomes that support the digital transformation of their businesses.
Thus, on-prem or colo-based single-site or multi-site data centers must be fully automated
and orchestrated as software-defined data centers (SDDC) to support private cloud.
Unfortunately, while compute and storage automation and virtualization have made leaps
and bounds over the last decade, software-defining and virtualizing the data center network
has continued to be extremely complicated and expensive. This has put SDDC and private
cloud out of reach for lean IT teams operating small and medium single-site or multi-site data
centers. This eBook takes a look at these challenges and outlines some solutions that can
help small IT teams achieve SDDC cost effectively.

3  |  Data Center Network Automation. Simplified. pluribusnetworks.com

CHAPTER 1
Why Every Size IT Team Should Strive to
Implement a Software-Defined Data Center

In a fast-moving and increasingly digital world, businesses of all sizes are working toward digital
transformation (DX) to stay competitive and grow profitably (Figure 1). This results in thinking through
existing application workloads and new applications that might be required for DX.

Digital Investments Are Paying Off

Financial Impact Of Digital Transformation

(2013-2017 CAGR)
3.0% 2.3%
2.0% 1.2%
1.0%
0.0%
Digital Manufacturers Non Digital Manufacturers
-1.0%
-2.0%
-3.0% -2.1%

-4.0% -3.1%

Revenue Growth Profit Growth

Figure 1: IDC’s “DX Reinvention — The Race to the Future Enterprise,” Doc #DR2019_GS4_MW, March 2019

Public vs. Private Cloud
On this DX journey, the allure of the hyperscale
public cloud is undeniable: no need for
infrastructure, spin up workloads quickly, easily
spin them down (if you remember) and scale
nearly infinitely. This has driven more and
more businesses to use public hyperscalers for
development and testing (dev and test), as well
as production workloads, either sponsored by IT
or as shadow IT projects directly out of business
units. That said, many IT teams have now built
experience and an understanding of which type
of workloads make sense in the public cloud
and which do not, based on criteria such as cost,
performance, security, data privacy and more.
As a result, while many businesses continue to
move workloads to hyperscalers, they are also
making decisions to keep specific workloads
on-prem or in colo facilities running on
end-user-owned data

4  |  Data Center Network Automation Simplified pluribusnetworks.com

center infrastructure, or alternatively, moving to
private hosted clouds run by managed service
providers (MSPs) and cloud service providers
(CSPs). In fact, IDC’s “Cloud Repatriation Accelerates
in a Multicloud World,” Doc #US44185818, August
2018, showed that 81% of enterprises had initiated
some sort of repatriation activity (pulling back
workloads from the public cloud). So, while the
growth of hyperscale public cloud continues,
and even as we see enterprises consolidating or
closing down their data center real estate, it is
clear that on-prem and especially colo data center
infrastructure owned by end-users as well as
hosted private cloud are both growing and here to
stay as part of a multi-cloud world. As can be seen
in figure 2, from the same IDC survey, the majority
of workloads will be in private cloud environments
in the “ideal” state.

Substantial Workload Shifts to Cloud Environments

Private Cloud a strong focus for on- and off-premises solutions
Q. What percent of your organization’s applications are Q. Ideally, if your organization could start over tomorrow without legacy
currently deployed in the following venues: sum to 100 IT decisions, what percent of your organization’s applications would be
% of Applications deployed in the following venues sum to 100

Saas 11% 13% PUBLIC Focus shifts from efficiencies &

Iaas/Paas 10% CLOUD cost savings to speed &
12%
Hosted perfomance
14%
Private Cloud 17%

PRIVATE Application churn, particularly

On-Premises 30% CLOUD for non-cloud on premises apps
Private Cloud favors private clouds
31%
Off-Premises
Non-Cloud 10%
10% Cloud management the glue for
the increasingly complex
On-Premises 26% NON-CLOUD
Non-Cloud application portfolio
17%

TODAY IDEAL STATE

Figure 2: The majority of workloads will reside in on-prem or reside in private hosted cloud environments.
From IDC’s Cloud Repatriation Accelerates in a Multicloud World, Doc #US44185818, August 2018

Software-Defined Data Center high-performance and highly automated SDDC

What is important, then, whether you are an
end-user business owner of data center
infrastructure or a non-hyperscale MSP or CSP, is
to ensure that your on-prem or colocation data
center infrastructure can provide a cost-effective,
foundation to support private cloud. SDDCs
generally consist of software-defined networking
(SDN) of the physical network (underlay) along with
virtualized network (overlay), compute and storage,
all coordinated by a higher-level orchestrator such
as vCenter, OpenStack or Kubernetes.

5  |  Data Center Network Automation. Simplified. pluribusnetworks.com

Compute and storage virtualization, and now
containerization, have made leaps and bounds
over the last decade. Yet software-defining and
virtualizing the data center network has continued
to be extremely complicated and expensive –
putting SDDC and private cloud out of reach for lean
IT teams operating small and medium single-site or
multi-site data centers. This is unfortunate because
digital transformation is critical, and it is clear that
businesses that have automated their data center
networks with SDN have seen clear economic
benefits, as highlighted in the Enterprise Strategy
Group’s survey results displayed in figure 3.

How would you characterize your company’s timeliness developing and launching
new products and services, relative to its competition? (percent of respondents)

Organizations implementing SDN (N=556) Organizations not implementing SDN (N=3,340)

55%
49%

39%
32%

13% 11%

We are usually significantly ahead We are often ahead of our We are usually in line with or
of our competition competition behind our competition
Figure 3: Organizations that deploy SDN in the data center are significantly more competitive.
From Enterprise Strategy Group, 2018

SDN provides new levels of network automation to Pluribus Networks – Putting SDDC
accelerate IT transformation: and Private Cloud Within Reach
• 
97% of transformed companies have At Pluribus Networks, we believe there is an
committed to SDN approach based on open networking and open
• 
SDN users are 3.5 times more likely to be source principles for businesses of all sizes to
significantly ahead of their competitors in time achieve SDN, network virtualization and network
to market (49% versus 13%) analytics cost-effectively – putting SDDC and
• 
2.5 times more SDN users made excellent private cloud within reach of every IT team, even
progress enabling a rapidly elastic data center for small data centers and lightly staffed IT teams.
environment (46% versus 18%) It is clear that breaking through the barrier of
automating the network is the critical hurdle to
supporting SDDC and private cloud.

6  |  Data Center Network Automation. Simplified. pluribusnetworks.com

CHAPTER 2
A Foundation of Open Networking
for Network Automation
In Chapter 1, I talked about the fact that, while many
workloads are moving to hyperscale public clouds,
many will continue to run either in data centers with
end-user owned infrastructure (on-prem or in colos) or
in hosted private clouds. I also reviewed the business
benefits of transforming to an SDDC and, in particular,
focused on the challenges of providing a software-
defined underlay and virtual overlay networking
infrastructure, which has been the Achilles heel for IT
teams in terms of achieving SDDCs.
Given the clear benefits of SDDC transformation,
is there an affordable and simple approach to
get there? Is there an “Easy Button” that makes it
feasible for even small and medium data center
operators? We believe the answer is clearly yes.
To understand how, I will look at three sets
of questions:
1: Should I deploy open networking or go with a
vertically integrated vendor when it comes time
to upgrade, expand, migrate or consolidate my
data center and to support SDDC? How much
risk is there in open networking, and what is the
support model?
2: Do I want my leaf-and-spine physical network
to be deployed as an SDN fabric, or am I
comfortable with box-by-box configuration,
operations and troubleshooting? Is the cost and
complexity of deploying SDN worth the effort?
What are the various approaches?
7  |  Data Center Network Automation Simplified
3: Do I want to create a virtualized network overlay
fabric that abstracts the physical network into
a set of software-based virtual tunnels between
all switches supported by software-based
routers, switches and load balancers that offer
the ability to establish new network topologies
and services in seconds? Is the cost and
complexity of deploying a virtual network worth
the effort? What are the various approaches?
These questions can be asked in any order, and
often can and should all be asked and investigated in
parallel. Here I will focus on the first set of questions,
about open networking, and then address SDN and
network virtualization in Chapter 3.
Open Networking Has Matured
Over the last decade, some customers have
been reluctant to take a risk on open networking
because of its perceived immaturity and concerns
with service and support in a model where
software comes from one company and hardware
from another. This is juxtaposed against the
tremendous benefits that have been achieved from
disaggregating software from hardware, including
driving capital costs down by up to 50% and, more
importantly, speeding hardware and software
innovation by enabling separate innovation
paths and leveraging an open source community
approach. For example, Pluribus utilizes Free Range
pluribusnetworks.com
Routing (FRR), which is an open source codebase down feature velocity. Pluribus and other open
that sits under the auspices of the Linux Foundation software-only solutions can innovate quickly in a
and which gives us the base routing code for our DevOps model and issue frequent software releases
Netvisor® ONE Network Operating System (NOS). with new capabilities, while the hardware is also
Because we leverage this core set of code from innovated in parallel from the likes of chip vendors
FRR, we are able to apply our software engineering such as Broadcom and system-level hardware
resources to quickly innovate around the edges, solutions from Edgecore, Dell EMC and Celestica.
focusing on key use cases for our customers and our
Open networking has been widely deployed by
unique approach to SDN and network virtualization,
the hyperscalers and is now moving into the
and contribute code back upstream for others to
mainstream as IT teams become more comfortable
leverage. There is no doubt that disaggregation
with the technology’s performance and quality, as
itself speeds innovation and, unlike with vertically
well as support from open networking vendors.
integrated vendors where hardware and software
As a result, open networking – also referred to as
are highly intertwined, there is no hardware
“white box” or “bare metal” switching – is growing
dependency that increases complexity and slows
faster than other switching categories (figure 4).

Open networking adoption with bare metal switching grows

Data Center Ethernet Switch Port Forecast by Type
Key takeaways
70%
Bare metal switches to reach 31% of ports
shipped in CY23, up from 20% in CY18 60%
Expected YoY revenue growth 46% for
CY19 50%

Open networking switch ecosystems

Ports (%)

40%
selling to enterprise are expanding
Bare Metal 46% YoY
30%
Open switch designs certified by the Open
Compute Project continue to increase 20%

The maturity and availability of options for 10%

switch OSs continues to improve
0%
CY17 CY18 CY19 CY20 CY21 CY22 CY23

General Purpose Purpose-built Blade Bare metal

Source: IHS Market Data Center Networks Intelligence Service - March 2019

Figure 4: White box switching is growing faster than any other switching category

For example, AT&T has completely committed Pluribus is deployed in over 240 customers today,
to the white box and open source path across including deployments in over 60 virtualized
multiple places in the network publicly. Many (NFVi) 4G/5G mobile cores of Tier 1 service
other institutions, from cloud service providers providers, where our software is carrying the
to K-12 school districts and local governments to traffic of hundreds of millions of mobile data
enterprises, have deployed open networking with subscribers. These sorts of mission-critical, large-
great success. scale deployments have allowed the software and
hardware technology to mature and be hardened.

8  |  Data Center Network Automation. Simplified. pluribusnetworks.com

Open Networking Switches Are
Becoming More Powerful
This large number of deployments has led to
important feedback going to the open networking
hardware vendors and resulted in rapid innovation
as well as increased performance, not only in terms
of data plane but also control plane processing
power, memory and architectural innovations.
For example, 32x 100 GbE white box switches can
now be sourced with Intel Xeon Broadwell 12 core
processors, 8/16/32G of RAM and 256G or larger
SSDs, providing a powerful server-like platform
that complements high-performance Broadcom
network processing units (NPUs) like the
Trident 3. The system-level architecture of these
platforms, shown in figure 5, has also matured,
with some manufacturers implementing two
parallel 10G network interfaces between the Intel
CPU and the Broadcom ASIC providing high-
speed links to support significant control and
management plane traffic. This has resulted in
not only wire-rate performance in the data plane
but the ability to run containerized workloads
with significant traffic throughput in the control
plane, such as tens of virtual routers with high
performance, making these switches suitable
in the most demanding single- or multi-tenant
network environments.

Next-Gen Open Networking/Whitebox Switching

Enabling Consolidation Of Multiple Infrastructure Layers
Ethernet Switch
CPU Complex
Switch designs leverage server design elements
Flash BMC Powerful multicore x86 CPU architectures
DDR SSD TPM
Internal 10G NICs
Multi-Core X86 Onboard solid state storage

Dedicated 10G Links Baseboard management controller (BMC)

Trusted platform module (TPM)
Switch ASIC
Enables NOS to offer containerized/virtualized
applications with high-speed data workloads e.g.:
Front Panel Ports
Network Services (like virtual FW leveraging internal 10G NICs)
Application Flow Telemetry & PCAP capture
Open networking switch hardware
Multi-tenant routing engines
Figure 5: White box switches become more server-like and include dual 10G internal NICs

In spite of all of our talk of network virtualization,
we will always need the physical underlay for
connectivity. Our view at Pluribus is that if this
latent server-like processing power is being
deployed anyway, it should not go to waste.
With clever software, one can leverage these
platforms to run containerized applications like
SDN, network virtualization, virtual network
functions (VNFs), virtual routers, network analytics
and more. This novel approach is covered in more
detail in Chapter 3.

9  |  Data Center Network Automation. Simplified. pluribusnetworks.com

Support has Matured
Customers have become more comfortable with
support from open networking software and
hardware vendors. The support model does
depend on the vendor partnership structure, but
these have been set up, exercised and polished
over the last decade. For example, in the case
of Pluribus’ partnership with Dell EMC, which
has an extremely large global sales and support
infrastructure, Dell EMC will take first- and
second-level support, with Pluribus providing
third-level technical software support. In the case
of Edgecore or Celestica, Pluribus takes first- and
second-line support and brings in the hardware
vendors if needed. Pluribus has a follow-the-sun
model, with 24×7 support out of our offices in
Santa Clara, California and Bangalore, India.
Brownfield or Greenfield
Any solution can be used to build a greenfield
leaf-and-spine data center network once basic
proof-of-concept lab testing is complete. However,
in many instances, IT teams will want to insert a
few leaf switches at the top of one or two racks
into a brownfield data center to get a feel for open
networking performance, stability and usability.
In such a case, the data center network might
have a pair of existing spine switches from a
traditional vertically integrated vendor like Cisco,
Arista or Juniper. Many open network operating
systems, including Pluribus’ Netvisor ONE OS,
are designed to use standards-based Layer 2 and
Layer 3 protocols and can easily insert into such a
scenario. An exception is when open networking
solutions use a centralized SDN controller
running on multiple servers to hold network state
10  |  Data Center Network Automation. Simplified.
and program the switches with the OpenFlow
protocol. In this case, the spine switches must
be replaced with white box spines running the
same OpenFlow- based OS that is running on the
leaves, effectively limiting this type of solution to
greenfield-only deployments. Another approach
that effectively requires greenfield environments
is hardware-bound SDN implementations like
Cisco ACI. ACI requires specific switches with
specific hardware, typically requiring a rip-and-
replace of existing infrastructure to deploy. The
hardware dependency adds a layer of complexity
and fragility that can hamstring and overwhelm IT
teams trying to deploy SDN.
Bottom Line
Open networking provides tremendous innovation
and has been operating in mission-critical
networks around the globe for a number of
years – the code has been stressed and hardened
in real world deployments at scale. For example,
the traffic from hundreds of millions of mobile
subscribers is running through the Pluribus
Netvisor ONE OS and Adaptive Cloud Fabric™
today as we are deployed in over 60 virtualized
4G and 5G mobile cores and over 240 customer
environments. In addition to being hardened, the
cost-to-performance ratio and feature velocity of
the hardware and software available from open
networking solutions is compelling. There has
never been a better time to take a hard look at
this technology in pursuit of lowering your CapEx,
benefiting from modern automation, breaking free
of vendor lock-in and enjoying an increasing rate
of innovation.
pluribusnetworks.com
 CHAPTER 3
The “Easy Button” for SDN Control of
Physical and Virtual Data Center Networks
(Especially for Space- and Cost-Constrained Environments)
In Chapter 2, I posed three questions regarding
automating the data center network in pursuit
of building an SDDC to support private cloud.
There I focused on question 1 – should I deploy
open networking? – and highlighted how open
networking switches have developed powerful,
server-like control planes featuring multi-core
CPUs complemented with plenty of RAM and
flash storage.
In this chapter I will focus on two more sets
of questions:
•  Do I want my leaf-and-spine physical network
to be deployed as an SDN fabric, or am I
comfortable with box-by-box configuration,
operations and troubleshooting? Are the cost
and complexity of deploying SDN worth the
effort? What are the various approaches?
•  Do I want to create a virtualized network overlay
fabric that abstracts the physical network into a
set of software-based virtual tunnels between all
switches, supported by software-based routers,
switches and load balancers that offer the
ability to establish new network topologies and
services in seconds? Is the cost and complexity
of deploying a virtual network worth the effort?
What are the various approaches?
11  |  Data Center Network Automation. Simplified.
The Underlay
When first rolling out a data center leaf-and-spine
network, one needs to deploy physical switches
for connectivity. Typical deployments have two
10 GbE/25 GbE connections from each server to a
top-of-rack (TOR) switch, which in turn connects
via 100 GbE uplink to a spine switch. In an open
networking world, the network operator loads
their choice of open source NOS via the Open
Networking Install Environment (ONIE) onto the
switch, with the NOS running on the switch CPU
(e.g., from Intel) and programming the forwarding
ASIC (e.g., from Broadcom).
In a leaf-and-spine network without SDN
automation, each of these switches must be
configured, managed and monitored individually,
typically through command line interface (CLI),
which consumes a lot of time from deeply
technical networking experts. There are DevOps
tools like Red Hat Ansible that can help automate
some steps, but fundamentally provisioning,
operations and troubleshooting is still happening
box by box. On the other hand, SDN can
completely automate the underlay and make
10, 20, 30 or more switches look like one logical
programmable entity. In other words, instead of
pluribusnetworks.com
managing 32 switches, for example, the operator
manages a single logical switch with a single IP
management address, dramatically increasing
agility, simplifying management, reducing human
errors and enabling less skilled techs to operate
the network.
Controller-based vs. Controllerless
SDN Underlay Automation
How automation is achieved is where things get
interesting. The more traditional SDN approach
is based on the Open Networking Foundation’s
(ONF) OpenFlow protocol. In this architecture, all
the network fabric state is kept in a centralized SDN
controller, physically separate from the network
switches, and the forwarding tables in the switches
are programmed via the OpenFlow protocol.
Examples of solutions that use this approach
come from Big Switch Networks or leverage open
source controllers such as Open Daylight (ODL)
or ONOS. To achieve high availability, typical best
practice dictates three redundant controllers per
data center location. This is certainly a workable
approach for a single large data center, but it
can run into economic feasibility challenges
with smaller or geographically distributed sites,
because the network team needs to pay for and
deploy three servers along with licensing for
three controllers at every site. Two sites equals
six controllers, three sites equals nine controllers
and so on.
The alternate approach is to deploy a distributed
SDN function that runs as an application
in the user space of every switch in the
network – leveraging the power of distributed
computing – with the state of the fabric distributed
12  |  Data Center Network Automation. Simplified.
into an extremely space-efficient micro database
resident on each switch. This is the approach
Pluribus takes with our Netvisor ONE and
Adaptive Cloud Fabric. With this “controllerless”
approach, there is no set of external controllers
and thus no associated hardware or software
controller costs or unnecessary consumption of
space and power, which can be problematic in
constrained environments. There are a number of
other benefits that come with the controllerless
approach, including the ability to easily insert into
brownfield networks, improved resiliency with
in-band control and the ability to seamlessly
stretch across geographically distributed sites
regardless of distance, with no need for multiple
controllers at every site.
Once SDN is deployed, the underlay can be
managed as a single fabric, so it’s easy to make
configuration changes or troubleshoot across all
switches in the fabric with a single command or
query. SDN can be used to set up any topology for
the underlay, including Layer 2 or Layer 3
leaf and spine, and automation increases agility
and reduces operational costs and human
errors significantly.
The Overlay
Once you have the underlay established, the
next step is to virtualize the network by creating
an overlay network. The overlay network is
constructed by creating software tunnels between
virtualized endpoints with an encapsulation
technology such as VXLAN, software-based
switches and software-based routers. This
method of software abstraction is agnostic to the
physical server connections and the underlying
network topology.
pluribusnetworks.com
Overlay network virtualization has a number
of important benefits, including scalability,
multi-tenant security and operational agility.
Underlay networks, including OpenFlow-based
networks that only implement VLAN-based flow
redirection without an overlay, are typically limited
to 4094 VLANs, while VXLAN-based overlays enable
scaling to over 16 million VLANs. This is especially
valuable for multi-tenant service provider networks
because it enables each tenant to control its own
VLAN numbering and scale independently of other
tenants up to the full range of 4094 VLANs.
Overlay networks also improve security in those
multi-tenant networks by providing an additional
layer of abstraction and full isolation between
customers, as shown in figure 6. That level of
isolation is also highly valuable in other use cases,
Figure 6: Overlay networking creates software-based networks that can be defined per tenant
13  |  Data Center Network Automation. Simplified.
including separating Internet of Things (IoT) traffic
from mission-critical corporate traffic to reduce
attack surfaces and efficiently utilizing firewall
ports and other security devices. In chapter 5, I
discuss the importance and benefits of network
segmentation in greater detail.
Overlays also enable far better scalability and
service agility when stretching services across
multiple sites and arbitrary topologies, because
they decouple logical service provisioning from
the underlying network topology and take
advantage of the optimized resilience and efficient
link utilization of standard Layer 3 underlay
technologies, such as equal-cost multipath (ECMP)
load balancing. With a virtualized overlay network
running completely in software, new network
services can be quickly established without having
to touch the underlay.
pluribusnetworks.com
Like SDN for the underlay, there are multiple
approaches to network virtualization. One
approach has the software switches and routers
running on the same servers that run the
application workloads. In this case, the VXLAN
tunnels terminate into VXLAN tunnel endpoints
(VTEPs) that run on those same servers as well.
The pricing model for this approach is typically
on a per-processor basis for each of the servers,
often adding thousands of extra dollars per CPU.
Then, in addition, a number of separate external
servers are needed for management, network
controllers, edge services gateways and more. This
means extra cost, space and power for the servers,
as well as multiple software licenses that must
be purchased. Again, like SDN for the underlay,
these overlay controllers need to be deployed
in clusters of three for redundancy. Finally,
because the packet processing in these solutions
consumes host compute power that would
otherwise support application workloads, many
in the industry are advocating that SmartNICs
be deployed. These “smart” network interface
cards (NICs) include CPUs and memory to offload
network processing from the main server CPUs,
increasing processing power but adding significant
expense per server along with complexity,
requiring multiple integration and configuration
steps. Solutions that take this approach are
14  |  Data Center Network Automation. Simplified.
Juniper Contrail, Nokia Nuage and VMware NSX.
Some data center operators will see benefits to
this approach that outweigh the added cost and
complexity, but generally that will only be true
for larger data center environments where having
multiple servers for management and control is
acceptable and where there are usually large IT
teams that can manage integration complexity. In
those cases, the software overlay solutions can be
deployed over a Pluribus SDN controller underlay
or even over a standard manually configured
underlay.
A New Approach to Overlay
Fabric Networking
The alternative approach for overlay fabric
networking is to leverage the power of the CPU
and the packet processing power of the forwarding
ASIC in the TOR switches that are being deployed
anyway. Just like the SDN underlay, this is the
approach that Pluribus takes with our Adaptive
Cloud Fabric for the overlay fabric. In this case, the
VXLAN tunnels terminate on the white box TOR
switch and leverage the specialized ASIC from
Broadcom to hardware-accelerate the packet
processing and termination of VXLAN tunnels
into VTEPs
pluribusnetworks.com
 Traditional Controller-Based Controllerless

Network
Monitoring
Infrastructure

Network
Virtualization and
Segmentation

SDN Underlay
Control

API
White Box
Packet L2/L3 Packet L2/L3 SDN Network DCGW
Analytics
Processing NOS Processing NOS Control Virtualization Router

Broadcom CPU / Memory Broadcom Intel CPU / Memory

Figure 7: Leveraging the distributed compute power of white box switches to implement SDN and network virtualization

Layer 2 and Layer 3 unicast and multicast services expense of other network virtualization solutions,
are distributed throughout the overlay fabric using it also is controllerless – again, where no external
an Anycast Gateway approach, again leveraging controllers are needed – reducing space, power
the distributed processing power of the switches. and cost, which is especially critical in smaller data
Not only does this eliminate the per-CPU license center environments.
expense and the optional per-server SmartNIC

Analytics Analytics Analytics

Packet broker Switch Packet broker Switch Packet broker Switch

Large amounts of external VIM Controller VIM Controller VIM Controller

Cluster Overlay Cluster Overlay Cluster Overlay
hardware and software Controller cluster Controller cluster Controller cluster
Underlay Underlay
licenses to automate the Controller cluster Controller cluster
Underlay
Controller cluster

underlay and overlay

DC1 DC2 DC3

Bookended leaf-
spine with closed 1c1c1b
Protocols
greenfield fit

Optional: SmartNics per server for network virtualization data plane acceleration

VIM Controller
Cluster

Leverage the processing Single UMUM Network

Management & Insight
power of the switch to save Analytics
DC1 DC2 DC3

cost, space and power

UNUM

Any WAN

Figure 8: A dramatic reduction in cost, space and power for constrained environments and a pre-integrated solution
that is simple for resource-constrained IT teams to deploy.

15  |  Data Center Network Automation. Simplified. pluribusnetworks.com

Underlay and Overlay Unified –
It Just Works
In the case of Pluribus’ Netvisor ONE OS and
Adaptive Cloud Fabric, the typical deployment
is with unified SDN control of the underlay and
overlay fabric, with no external controllers needed,
as shown in figures 7 and 8. Not only is this
approach very cost effective, with extremely low
space and power consumption, the unification of
these two automation layers results in a simple
deployment with minimal integration required.
The solution works out of the box, from zero-touch
provisioning (ZTP) of the switches to building
an SDN-automated underlay to deploying the
virtualized network fabric. It is simple, fast and
efficient, as you can see in this short four-minute
video that shows the deployment of a simple
four-switch fabric using our UNUM management
system – UNUM Day-0 Automation. Once
deployed, the solution is easily integrated through
a northbound RESTful API with orchestration
systems such as vCenter, Red Hat OpenStack
or Kubernetes.
Data centers are becoming more distributed into
colocation facilities and even out to the edge
of the network - what we call distributed cloud.
Fundamentally, the industry is seeing more and
16  |  Data Center Network Automation. Simplified.
smaller data centers moving workloads closer to
users and things to improve customer experience
and enable new latency-sensitive applications.
Most approaches to SDN control of the underlay
and overlay are complex, costly and consume
a lot of space and power, and are not really
well designed for smaller and geographically
distributed mini data center environments. By
using the processing power of the switches
that are being deployed anyway for physical
connectivity, Pluribus enables a very cost-,
space- and power-efficient SDN underlay and
virtualized overlay, which makes it feasible to
deploy an SDDC in small/medium data centers
and constrained edge data center deployments.
Now this is not the final layer of automation we
need, however, so in Chapter 4 I will talk about the
importance of network analytics and how, once
again, the power of open networking switches can
be leveraged with clever software to deliver
cost-effective yet very granular network telemetry.
pluribusnetworks.com
CHAPTER 4
Network Analytics Without Probes,
TAPs and Packet Brokers
In Chapter 3, I wrote about a controllerless
implementation for SDN automation of the
underlay and a virtualized network overlay fabric
that leverages the distributed processing power
of open networking switches. The result of this
approach is a very efficient and highly integrated
network automation solution for smaller data
center environments where traditional SDN
approaches are simply too expensive, consume
too much space and power and struggle to span
geographically distributed multi-site or edge data
center locations.
This novel approach is very powerful and
necessary but not sufficient – there is another
layer of functionality required to support
comprehensive data center automation. In order
to monitor the network and quickly identify
and troubleshoot performance issues, granular
telemetry on every flow that traverses the fabric
is essential. In fact, major vendors like Cisco, with
their Tetration offering, have heavily validated the
need for application analytics for today’s modern
applications. But these traditional approaches
17  |  Data Center Network Automation. Simplified.
are not optimal for smaller environments as they
require a set of external test access points (TAPs),
probes and packet brokers that effectively overlay
the network fabric, not to mention a number of
servers to execute the analytics. This results, again,
in high cost and space and power consumption, as
well as additional complexity.
Traditional Approaches
to Analytics
Traditional switches and routers switch billions of
packets per second between servers and clients at
sub-microsecond latencies using custom ASICs but
have limited capability to record enough telemetry
detail to provide a truly useful picture of network
performance over time. It is a very similar story for
OpenFlow-based switches, which use merchant
silicon but have insufficient telemetry. As such,
external TAPs and monitoring networks have to be
built to get a sense of what is actually going on in
the infrastructure. The figure below shows what
monitoring today looks like.
pluribusnetworks.com
 Monitoring
Traditional Network Monitoring Fabric

Selectively place Mirroring

Taps and mirroring
ports

Tap

Tools
VM VM VM VM VM VM VM VM VM

EXPENSIVE Fabric to
VM VM VM VM VM VM VM VM VM aggregate and filter
traffic
Server Server Server

Server Server Server

Server Server Server

Server Server Server

Figure 9: A traditional approach to network telemetry and analytics requires external TAPS, probes and packet brokers.

This is where challenges arise. A typical data center
network that connects servers runs a combination
of 10, 25, 40 and 100 GbE today. These switches
typically have many servers connected to them
that are pumping traffic at high speed.
Some possible approaches to instrumenting the
network today are as follows:
1. Provision a copper or fiber optic TAP at every
link and divert a copy of every packet to a packet
broker fabric, which in turn routes traffic to
the monitoring tools. With the fiber optics TAP
and passives, every packet is mirrored, and the
3.  Using the networking switches themselves
monitoring tools need to deal with a few Tb/s
or 1B+ packets per second from each switch.
However, the reality is that this approach is
impossibly expensive, and thus no one deploys it.
2.  Selectively place copper or fiber optic TAPs
at uplinks or edge ports. Mirror these edge
packets to a packet broker fabric, which in turn
routes traffic to the monitoring tools. While
this is less costly, it means the inner network
becomes a black hole with no visibility. Many
of us have learned the hard way over time that
without 100% visibility, you can’t fix a problem
very efficiently. In addition, even this selective
deployment of hardware makes the cost go up
dramatically, as more switches are deployed
and require monitoring – the monitoring fabric
needs more capacity and the monitoring
software gets more complex and needs more
hardware resources.
to selectively sample traffic (e.g., sFlow with
standard hardware or NetFlow with proprietary
hardware) and send this traffic and flow
information to monitoring tools. This approach
is built upon the premise of sampling, where
the sampling rates are typically 1 in 5,000 to
10,000 packets – any more than this runs into

18  |  Data Center Network Automation. Simplified. pluribusnetworks.com

 scale challenges. This approach is better than
nothing, but does not really have enough raw
detail to attain a full picture of the network.
Another Approach to Telemetry
and Analytics
As described in the previous chapter, it makes
sense in constrained environments to leverage
the distributed processing power of white box
switches when possible. Similar to what can
be done with SDN and network virtualization,
one can write clever software that leverages the
CPU and memory of the switch as well as the
packet processing ASIC to monitor every TCP
connection, including traffic within VXLAN tunnels,
at wire speed across the entire fabric. This allows
the tracking of all east-west and north-south
traffic flows to expose important network and
application performance characteristics and
quickly isolate and fix problems.
Specifically, rich telemetry from the SDN and
virtual network fabric can be gathered, where each
switch in the fabric collects the metadata for every
flow, stores some amount of metadata on the
switch and then sends the bulk of the metadata to
an analytics application via REST API. In particular,
more recent open networking switches feature
dual 10G NICs that run between the CPU and
the network processing ASIC, providing plenty of
throughput to transport the data to the CPU. The
bulk of the processing would happen on the local
OS instance, and only metadata would be peeled
off and sent to the analytics application, which
allows this solution to scale to billions of flows.
19  |  Data Center Network Automation. Simplified.
With this approach one can effectively capture
every TCP flow across the fabric at wire speed,
including TCP connection states (SYN, SYNACK,
EST, FIN, etc.) by service, client, domains and many
other options over time, and store the metadata in a
repository for deep analysis. Also, multiple options
to tag IP addresses, VLANs, MAC addresses and
switch ports with metadata/contextual tags can be
offered, and then one can aggregate or filter flows
based on the custom tags. In addition to flows, of
course, it is important to have port telemetry and
device diagnostics via a selection of searchable
options such as fabric node, switch port, vport
(virtual port) and state, including a dashboard of all
ports in the fabric. This is extremely valuable as it
provides real-time and/or historical data analytics
to identify performance concerns and root-cause
network outages or to quickly understand security
threats like DDOS attacks.
We had a large DDoS attack against the main
district website. With Pluribus, we were able to
determine in minutes that it was a couple of
IP addresses from a nearby college and shut it
down quickly before it had any impact.
Sr. System Engineer,
Large Midwest K-12 School District
Insight Analytics
Pluribus Netvisor ONE has implemented this novel
software approach described above, leveraging
the CPU, memory and packet processing ASIC to
provide comprehensive flow and switch telemetry.
pluribusnetworks.com
Performance metrics are stored within the fabric
and delivered as lightweight metadata that can
be viewed using CLI from the fabric or delivered
via API or IPFIX to other monitoring systems,
security information and event management
(SIEM) platforms or the Pluribus Insight Analytics
solution, figure 10, which is an optional software
module in our UNUM product offering. This
solution can store from 100 million up to 2.5 billion
Figure 10: One of many prepackaged reporting displays in UNUM Insight Analytics
As applications become more distributed, with
both east-west and north-south traffic, and
services are deployed within private clouds, the
ability to monitor each and every connection is of
paramount importance for both performance and
security reasons (find more on security in chapter
5). Given the amount of data, traditional sampled
data network analytics sources do not scale. More
traditional packet monitoring solutions designed
to overcome this limitation unfortunately require
20  |  Data Center Network Automation. Simplified.
flows over a time window and has an analytics
engine and a rich set of reports that allow network
operations teams to drill down to a single flow
and identify performance issues or bad actors. Its
powerful search engine UI and simple query syntax
can help isolate and filter specific flows among
millions in a fraction of a second. This can help
quickly identify and rectify performance issues for
regular reporting to senior management.
significant hardware overlay infrastructures that are
expensive, complex and consume space and power
– not ideal for smaller data center environments.
The best approach is to leverage the distributed
processing power of the network switches
themselves with some clever software to provide
the data sources and analytics tools the ability to
observe every packet and flow at a fraction of the
cost of traditional hardware-based solutions.
pluribusnetworks.com
CHAPTER 5
The Importance of Network Segmentation
for Security and Multi-Tenancy
We can’t talk about data center networking
automation without addressing the topic of
security. Firewalls sit at the perimeter of the data
center network and can protect against
north-south traffic entering the data center fabric,
but do not address east-west traffic and threats
moving laterally once inside the network. For
example, sophisticated malware can hide within
encrypted data and be missed by conventional
firewalls, and once inside can create significant
damage. IoT is one example of an application
with many new endpoints generating traffic and a
potentially immature security model that results
in a new, large attack surface. There are already
examples of successful attacks through
IoT devices, such as an attack on a casino via a
WiFi-connected fish tank temperature sensor, as
well as a massive retail attack via a WiFi-connected
HVAC system.
Consequently, the industry has moved toward
leveraging virtual routing and forwarding instances
(VRFs) to segment the network and isolate traffic.
VRFs can be deployed in a traditional underlay or
on top of a VXLAN-based overlay. In either case,
one of the challenges traditional networking
solutions face is the complexity of provisioning and
deploying the VRFs. If deployed in the underlay,
21  |  Data Center Network Automation. Simplified.
it is necessary to configure multiple VRFs per
switch on multiple switches across the data center
or campus – a nightmare of complexity that is
very prone to human error. In addition, because
of the heavy protocol exchange in a typical VRF
implementation, traditional solutions run into
VRF scale challenges. Similarly, setting up a VXLAN
fabric using, for example, BGP-EVPN requires N x
tens of steps per switch, and then adding VRFs
on top of that adds another N x tens of steps
per switch.
Automating Network
Segmentation
On the other hand, Pluribus’ open SDN approach
with the Adaptive Cloud Fabric sets up a mesh of
VXLAN tunnels automatically. Once deployed,
VRFs can be programmed to run across the fabric
on every switch within a VXLAN segment with
a single atomic command. Literally only one
command is needed – a dramatic simplification.
In addition, Pluribus’ VRF scale is limited only by
hardware because the Adaptive Cloud Fabric’s SDN
approach does not need the protocol exchange
typically required by VRFs.
pluribusnetworks.com
Ultimately, this simple-to-deploy and highly
scalable network segmentation approach
significantly reduces system attack surfaces
so that endpoints only see the resources and
services necessary to perform their tasks,
limiting accessibility and mitigating risk. With the
programmability and ease of use of the Adaptive
Cloud Fabric, network or security team members
can quickly add segments and VRFs to control
traffic flowing across the fabric without having
to reconfigure the underlying physical
network infrastructure.
More Efficient Utilization of
Security Devices
This segmentation also allows the more efficient
use of firewalls, intrusion detection systems (IDS),
intrusion prevention systems (IPS) and other physical
or virtual security devices. Instead of deploying many
of these devices throughout the network, they can
be centralized for easy management, while also
allowing the pooling and sharing of these typically
expensive resources. Furthermore, traffic can be
steered to specific resources such as a separate IoT
analytics system or to external cloud services for
processing, with confidence that it is separated from
higher-value traffic.

Ext Network

CORE ROUTERS

MULTIPLE vFLOW
VRF SERVICE
VRF VRF VRF VRF
INSTANCES INSERTION
(COLORS) POLICY

PERIMETER
VM-10 VM-20 VM-30 VM-20 VM-20 FIREWELL (OR
VM-10 VM-10 IPS/IDS)
VM-10 VM-20 VM-20
VM-20 VM-30
VM-20 VM-10 VM-30 CLUSTER
VM-30

Figure 11: VRFs distributed across the fabric with Anycast Gateways to better leverage pooled firewall resources

22  |  Data Center Network Automation. Simplified. pluribusnetworks.com

Segmentation can also be used for multi-tenancy.
There are different levels of segmentation, and
the Adaptive Cloud Fabric is unique in its ability to
offer not only VXLAN overlay supported by VRFs
for segmentation across the data and control
planes but also deep slicing. Deep slicing leverages
a construct called vNETs (virtual networks) that
slice the fabric across the data, control and
management planes. Deep slicing allows each
tenant, if desired, to use its own automation tools
to control its slice fabric wide. For example, if you
are a regional cloud service provider that deployed
ACF across five data centers, you could define a
slice for one customer with a set of physical or
virtual ports at three of those five data centers, and
that tenant could configure its slice as it sees fit.
23  |  Data Center Network Automation. Simplified.
Integrated Telemetry
and Analytics
Finally, as discussed in Chapter 4, the integrated
telemetry monitors every TCP connection and
flow at wire speed, including traffic within overlay
tunnels orchestrated by the Adaptive Cloud Fabric,
to expose important network and application
behavior characteristics. Visibility is provided on
a per-segment basis with complete separation
of data for compliance requirements. Enabling a
comprehensive view of the fabric, the integrated
visibility greatly improves situational awareness
while eliminating the costs and complexity
associated with hardware-based monitoring tools.
pluribusnetworks.com
Summary
The world is moving to a hybrid multi-cloud model, with IDC estimating that 75% of workloads
will be deployed in on-prem, colo-based or hosted private clouds for cost, security, performance
and data sovereignty reasons. These on-prem, colo and hosted private cloud environments
require a completely automated data center foundation – the software-defined data center or
SDDC. While storage and compute virtualization and automation have made great strides over
the last decade, networking has lagged. Data center networking automation solutions today
have been designed for large data centers operated by large IT teams that have the budget to
buy, and the resources to integrate and deploy, layers of external hardware and software to
achieve network automation and virtualization. Unfortunately, these traditional approaches do
not suit the larger universe of data center and private cloud operators that have smaller IT teams.

The Pluribus Netvisor ONE operating system and Adaptive Cloud Fabric have been designed
to deliver a superior level of network automation to small IT teams while simultaneously fitting
into cost-, space- and power-constrained environments. Pluribus Networks takes advantage of
the underutilized distributed computational power, memory and packet processing inherent
in leaf-and-spine network switches distributed across one or multiple data center sites. By
leveraging these resources, Pluribus delivers a unique “controllerless” approach to SDN
automation of the physical network, provides a service-rich and secure VXLAN virtual fabric
and enables comprehensive and granular telemetry and analytics. Not only is the solution
cost-, space- and power-efficient, it is unified and pre-integrated, so it just works out of the box.
Supporting well-known orchestration systems, including VMware vCenter, Red Hat OpenStack
and Kubernetes, Pluribus Networks puts fully automated SDDC within reach for small IT teams
with constrained physical environments - the Easy Button for SDDC.

24  |  Data Center Network Automation. Simplified. pluribusnetworks.com

Additional Resources
Webinar Replay: Realizing the SDDC: Simple, Affordable SDN and Network Virtualization for
Any Size Data Center
Web Page: Netvisor ONE Operating System
Web Page: Adaptive Cloud Fabric
Web Page: Insight Analytics

Copyright © 2019 Pluribus Networks, Inc. All Rights Reserved. Netvisor is a registered trademark, and The Pluribus
Networks logo, Pluribus Networks, Freedom, Adaptive Cloud Fabric, and VirtualWire are trademarks of Pluribus
Networks, Inc. All other brands and product names are registered and unregistered trademarks of their respective
owners. Pluribus Networks reserve the right to make changes to its technical information and specifications at any
time, without notice. This informational document may describe features that may not be currently available or Pluribus Networks, Inc.
may be part of a future release, or may require separate licensed software to function as described. www.pluribusnetworks.com