Hyperautomated AI-based

Security vs VMware NSX

Manuel Nedbal

Those tasked with building network security appliances know that network traffic processing is
accomplished in several different stages. Each NextGen Firewall, IPS, and DLP appliance sends traffic
through a pipeline that becomes progressively deeper in their packet analysis. As such, each stage requires
a different amount of compute, storage and memory, and appliance vendors over the years have mastered
how to anticipate required budgets for typical traffic flows. Although the following example oversimplifies
such a processing sequence, and each stage conveys just a glimpse of the complexities involved, we will use
these as a baseline and apply them to our evaluation of the VMware NSX architecture.

1. Packet reception/transmission: picking up and distributing incoming packets to a set of subsequent

processing stages. Receiving outgoing packets and putting them back onto the network

2. Flow processing: reassembling, reordering, re-fragmenting packets to present a stream to

subsequent layers

3. TLS decryption: decrypting traffic and re-encrypting it outbound

4. Deep packet inspection: parsing protocols and delivering the intended appliance features

5. Object handling: extracting files, domains, URLs, etc. and analyzing each

6. Machine Learning: behavioral analysis and deriving anomalies

7. Event processing and correlation: deduplicating, suppressing events, and creating more meaningful
outcomes for security teams

Such security richness is required for detecting
and countering modern threats. Industry-leading
solutions can provide highly effective security if they
are placed and configured properly. However, as I
noted in the last blog, “Hyperautomation of AI-based
Security in the Distributed Cloud,” security teams
are not always able to achieve the required security
effectiveness, and this eventuality leads to
a different set of challenges. For this blog, we
will focus on how VMware’s traffic processing
approaches fare in its delivery of rich security
functionality and requirements.
VMware has been advocating against monolithic
appliances as the security solution for dynamic data
centers, and they have made significant inroads in
this direction. Their solution suggests coupling an
ever-richer set of security controls to the hypervisor.
Let’s review these solution benefits and cautions,
and then contrast it with a system that supports
Hyperautomated AI-based Security.
VMware’s Networking and Security offering, named
NSX, comes in two flavors, NSX-V and NSX-T. NSX-V
is the VMware product that allows traditional
appliance vendors to integrate security via an API

called NetX. Via NetX, virtual appliances that run
on VMware’s hypervisor, can apply traffic inspection
and control (see Figure 1). In parallel, Access Control
Lists (ACLs) can be enforced inside the VMware
hypervisor, which makes sense when considering
the deterministic lookup times required per packet
and the lean processing overhead.
NSX-V WITH TRADITIONAL SECURITY
Host 1
3rd party
VM1 VM2 VM3 Security VM
NetX
pg 10 pg 11
VLAN 10 VLAN 11
VMware Hypervisor ACLs
Physical Network
Figure 1: NSX-V with Traditional Security Appliance
While NSX-V provides a major leap in the right
direction, there are few issues still left to address:
•Third-party security VMs are monolithic in nature,
thus putting a burden of compute, storage and
memory on each host (all “stages” stick together)
•Customers must manage a growing set of virtual
appliances from a third-party vendor and make
sure they are supporting the VMware versions they
want to deploy (there is tight coupling between an
NSX-V agent needed on each security appliance
and the hypervisor version)
•Chaining multiple monolithic services increases the
latency and overhead problems as the same traffic
processing stages are repeated
HYPERAUTOMATED AI-BASED SECURITY VS VMWARE NSX | 2
This solution delivers an architecture that scales
out with the number of physical hosts in a data
center. Rather than aggregating traffic at an
artificially created choke point in the network,
traffic is inspected close to where it is generated,
and security capacities keep growing at the same
rate as the data center.
Host 2
3rd party
VM4 VM5 Security VM
NetX
pg 10 pg 11
VLAN 10 VLAN 11 Virtual Switch
VMware Hypervisor ACLs
•Scaling to the right number of security appliance
instances depending on traffic flows and policies
does not exist. This starts to become an issue when
a single security appliance cannot handle the
traffic of all the VMs it needs to protect on the same
host. The more “steps” need to be processed, and
the more cores Intel adds to a CPU, and the more
bandwidth can be run over networks, the more
likely a bottleneck occurs
•Upgrades and appliance failures will incur outages
as there is no High Availability within the same host

Several years after the VMware acquisition of Nicira
in 2012, the release of NSX-T hit the market with the
promise to deliver uniform networking and security
controls across multi-cloud environments. This was
extended in November 2019 as threat prevention
was introduced as part of NSX, a feature that packs
more of the previously mentioned “stages” into the
hardware abstraction layer, as depicted in Figure 2.
NSX-T WITH SECURITY
Host 1 Host 2
VM1 VM2 VM3
pg 10 pg 11
VLAN 10 VLAN 11
VMware Hypervisor Security
Physical Network
Figure 2: NSX-T with Security
Now let’s look at the architectural challenges
imposed when moving a richer set of security
controls into the hypervisor:
•Most of the same problems that monolithic
traditional security VMs experience apply here as
well (scaling, HA, upgrades). Hypervisors are bound
to certain compute, storage and memory limits
available on the host and compete with workload
VMs for those provisions. Especially hard to predict
are the capacity demands of threat prevention as
they are a function of policy and the actual traffic
observed. This can lead to starving workloads or
security controls
HYPERAUTOMATED AI-BASED SECURITY VS VMWARE NSX | 3
Given that such an approach requires support from
the hypervisor, VMware’s technology stack had to
be made available in public clouds as well. This, of
course, allows customers to experience a consistent
look and feel even when they expand into public
clouds, all using the VMware technology stack.
VM4 VM5
pg 10 pg 11
VLAN 10 VLAN 11 Virtual Switch
VMware Hypervisor Security
•Security controls run only on VMware hypervisors,
which requires the use of Amazon and Azure
proprietary hardware carveouts that are running
VMware technology stacks. Those stacks are
different from the cloud provider offerings.
Customers must pay for cloud providers and
VMware as well as incur a different feature set than
offered by clouds
•Traditionally, hypervisors were just focused on
hardware abstractions to keep the code minimal
and robust. Extending the footprint of a hypervisor
also extends the attack surface and carries the
potential for stability issues as well as breaches
 HYPERAUTOMATED AI-BASED SECURITY VS VMWARE NSX | 4

To sum it up one could argue that the NSX-T Figure 3 depicts how only a tiny presence per
solution resembles a physical network security host is required for the initial processing stage.
appliance running a hypervisor with security that We call this an FV for FlowVisor. A FlowVisor
leaves some space for other workloads to run. does to network flows what a hypervisor does to
hardware: it abstracts and unifies packet reception
Contrast the VMware solution with a ShieldX
and transmission independent of the underlying
ESP Hyperautomated AI-based security approach.
networking architecture. By making that FlowVisor
The art and science of looking deep into network
smart and intent-driven, it knows what traffic to
traffic flows is CPU and memory intense, whereas
process and how deep in a flow to go. It can send
just packet reception and transmission requires
that traffic on to subsequent stages that sit in the
lots of network IO. To devise the ideal solution, it
Elastic Backend. The backed can contain pooled
is important to realize that each individual traffic
sets of instances representing dynamically scaled
processing stage requires different scaling, and its
sets of stages. That pool enjoys the same luxury of
efficacy depends on varying hardware capabilities
modern virtualized or containerized infrastructure
like CPU, network IO or disk IO. Further, the
as the workloads they protect. This approach truly
amount of data exchanged decreases significantly
distributes all processing stages throughout the
the deeper the analysis steps go, thus making it
VMware data center, Amazon VPC, Azure VNET
possible to allow for pooling later stage security
infrastructures, etc., and does not impose the need
services across hosts. With that in mind we can
for a uniform hypervisor.
already guess what an ideal architecture looks like
Hyperautomated FlowVisor
that will profoundly overcome the previously listed
shortcomings.

HYPERAUTOMATED FLOWVISOR Host m-n

Elastic Backened
(N+1 HA)

Host 1 Host 2

VM1 VM2 VM3 FV VM4 VM5 FV

pg 10 pg 11 pg 10 pg 11
VLAN 10 VLAN 11
pg 12
VLAN 10 VLAN 11
pg 12 Virtual Switch

VMware Hypervisor VMware Hypervisor

Physical Network

Figure 1: NSX-V with Traditional Security Appliance

 HYPERAUTOMATED AI-BASED SECURITY VS VMWARE NSX | 5

This architecture combines the FlowVisor with a
scalable Elastic Backend that is tightly coupled
with changes happening in the environment for
continuous adaptation. As such it outperforms
any other monolithic design as it delivers the
following advantages:
•Ability to transparently insert, agent-less, into
the network traffic without disruption to the
applications and hypervisors but with automation
and no user intervention
•Auto-scaling out and in on an as-needed basis for
only the stage that needs help or is underutilized
•N+1 High Availability vs. 2N in traditional models
(provision only one extra for each stage)
•Most efficient resource consumption (no need for
having all stages on each host)
•Automated and orchestrated policy computation
and generation by learning groups and policies
using ML
•Non-disruptive upgrades by supporting
rolling upgrades
•Richest set of security controls easily extensible
without compromising hypervisor security, capacity
or safety
•Native integration with Amazon EC2 and Microsoft
Azure not requiring proprietary hypervisors that are
incompatible with native cloud APIs
•Providing superior security to its own infrastructure
as it has two shells around each component: the
VM, and containers within VMs
•More effective security controls as they can share
state across instances. Other approaches would
require state synchronization across hypervisors
and 3rd party appliances, which is very hard to
accomplish if at all attempted
To summarize, the ideal approach is to decouple
network security from the underlying hypervisor.
This delivers most reliability, effectiveness, flexibility
and agility to manage the solution over time.
Especially as public cloud providers expose APIs
for network traffic monitoring, they natively built
the underpinnings of the FlowVisor and there is
no driver to unify a technology stack across clouds
for security. The proposed solution herein helps
customers additionally by capturing their intent
and configuring components automatically to
achieve the intended security posture continuously
and optimally.

ShieldX Networks, Inc. +1 408.758.9400

2025 Gateway Place, Suite 400 info@shieldx.com
San Jose, CA 95110 USA www.shieldx.com

© 2018 ShieldX Networks, Inc. All rights reserved. For trademark, copyright, patent, November 2018
and other intellectual property and legal information, visit www.shieldx.com/legal