1 Openstack Neutron Distributed

Virtual Router (DVR) - Part 1 of 3

In this 3-post series I cover how DVR works in (just about enough)
detail.

I assume you know how virtual routing is managed in Neutron.

Up until Juno, all L3 traffic was sent through the network node, including
even traffic between VMs residing on the same physical host.
This, of course, created a bottleneck, as well as a single point of failure
(SPOF).

In Juno, DVR was introduced to overcome these problems.

As I described in my previous post, L3 networking in Neutron is divided

into 3 main services:

1. East-West communication: IP traffic between VMs in the data

center
2. Floating IP (aka DNAT): The ability to provide a public IP to a
VM, making it directly accessible from public networks (i.e. internet)
3. Shared IP (aka SNAT): The ability to provide public network
access to VMs in the data center using a shared (public) IP address

The main idea of DVR is to distribute the L3 Agent functionality to all

the compute nodes, distributing the load and eliminating the SPOF.

DVR successfully distributes the first two services.

However, SNAT distribution is not covered by DVR, and remains to be

handled by hitting the Network Node in a centralized manner.

The main challenges for SNAT distribution were maintaining the FWaaS
statefulness and conserving IP addresses from the public network address
pool (since supporting these will require each compute node to have a
public IP address).
 Neutron components with DVR entities are shown in red

DVR implements a virtual router element on every compute node using

the Linux network namespace (similarly to L3 Agent virtual router).

This is achieved by cloning the current centralized L3 implementation onto

every compute node.
For each service, this is handled in a different manner.

Inter subnet routing East-West

All the cross subnet traffic (between VMs of the same tenant) in DVR is
now handled locally on the compute node using the router namespace.

A Linux namespace is created for every virtual router, on each compute

node that hosts VMs that are connected to that router.

In order to configure these local DVR namespaces, the DVR L3 Agent

(which used to be deployed only to the network node) is deployed onto all
compute nodes as you can see in the diagram below.
Now, extra OVS flows were required to enable the DVR functionality. To
do that,enhancements were applied to L2 OVS Agent. These
enhancements are described below.

In order to simplify management, the same IP and MAC addresses are

reused on all the compute node DVRs, i.e. a single {IP, MAC} per virtual
router port).
Owing to this, ARP traffic for the virtual router ports is kept local to the
compute node.

All the ARP entries for networks attached to each DVR are proactively
populated by the L3 DVR Agent on all the cloned virtual router
namespaces, i.e. on all the compute nodes that participate.

In order to avoid having traffic from different physical hosts using the
same MAC (which, as we said, we are reusing), each of the cloned DVRs
is assigned a system-wide unique MAC address.

I will reuse the example from my previous

post: the popular cloud deployment
3-tier web application design pattern (www-app-db).

In Horizon, we set up 3 subnets (in addition to the public) with one virtual
router to bind them all.
On each compute node, the DVR namespace is pre-configured with the
following:
• Port MAC and IP addresses
• Routing tables (default route per subnet and static routes)
• ARP table entries for all the VMs in the connected networks
In addition to the namespace, OVS flows are
installed into the integration bridge and into the tunnel bridge to:
• Isolate DVR broadcast domain to be local to the integration bridge
• Translate the globally assigned MAC of the DVR into the local MAC
address and vice versa
• Redirect traffic correctly per VM (new flows are installed for every
new VM started on the compute node)
 Realization of our topology on 2 Compute nodes

As we can see, VM1 on Web subnet communicates with VM3 on App

subnet:
1. The packet reaches the subnet GW port on the local compute
node
• It is routed using the DVR namespace routing table into the
destination subnet
• Linux namespace then performs the followings actions on the
packet:
• Use the pre-populated ARP table to set the destination
MAC address
• Set the source MAC to the local DVR GW port of the
destination subnet
2. On the tunnel bridge the source MAC is replaced with
the global DVR MAC address
3. The packet reaches the compute node that hosts
the destination VM:
• The segmentation ID is replaced with the local vlan
• The packet is matched by the global source MAC address and
forwarded to the integration bridge
4. The packet source MAC is replaced with the appropriate local
DVR GW MAC and forwarded directly to the destination VM port
In the reverse flow from VM3 to VM1, packets are routed through the
local DVR instance of the compute node that hosts VM3.
Note that the reverse flow goes in a different route (see the blue line).
Due to this behavior, we cannot create stateful East-West rules,
hence FWaaS for East-West traffic is not supported by DVR Juno.

In my next posts, I will go into the North-South scenario, covering

the DNAT and SNATservices.

2 Openstack Neutron Distributed

Virtual Router (DVR) - Part 2 of 3
In this post, the 2nd of a 3-post series about DVR, I go into the North-
South DNAT scenario in details.
Up until Juno, all L3 traffic was sent through the network node. In
Juno, DVR was introduced to distribute the load from the network node
onto the compute nodes.

The L3 networking in Neutron is divided into 3 main services:

1. East-West communication: IP traffic between VMs in the data

center
2. Floating IP (aka DNAT): The ability to provide a public IP to a
VM, making it directly accessible from public networks (i.e. internet)
3. Shared IP (aka SNAT): The ability to provide public network
access to VMs in the data center using a shared (public) IP address
In my previous post, I covered how DVR distributes the the East-West L3
traffic.
In this post I am going to begin covering the North-South
traffic starting with Floating IP (DNAT).

DNAT Floating IP North-South

In order to support Juno's DVR local handling of floating IP DNAT traffic in
the compute nodes, we now require an additional physical port that
connects to the external network, on each compute node.

The Floating IP functionality enable direct access from the public network
(e.g. Internet) to a VM.

Let's follow the example below, where we will assign a floating IP to the
web servers.

When we associate a VM with a floating IP, the following actions take

place:
1. The fip-<netid> namespace is created on the local compute
node (if it does not yet exist)
2. A new port rfp-<portid> is created on the qrouter-
<routerid> namespace (if it does not yet exist
3. The rfp port on the qrouter namespace is assigned the
associated floating IP address
4. The fbr port on the fip namespace is created and linked via
point-to-point network to the rfp port of the qrouter namespace
5. The fip namespace gateway port fg-<portid> is assigned
an additional address from the public network range (the floating IP
range)
6. The fg-<portid> is configured as a Proxy ARP
Now, lets take a closer look at VM4 (one of the web servers).

In the diagram below, the red dashed line shows the outbound network
traffic flow from VM4 to the public network.

The flow goes through 5 steps:

1. The originating VM sends a packet via default gateway and
the integration bridge forwards the traffic to the local DVR gateway
port (qr-<portid>).
2. DVR routes the packet using the routing table to the rfp-
<portid> port
3. The packet is applied NAT rule using IPTables, changing the
source-IP of VM4 to the assigned floating IP, and then it is sent
through the rfp-<portid> port, which connects
to the fip namespace via point-to-point network (e.g.
169.254.31.28/31)
4. The packet is received on the fbr-<portid> port in
the fip namespace and then routed outside through the fg-
<portid> port
At this point you may be confused with the descriptions, so let's try to
simplify this a bit with a concrete example:
[Public Network range]=10.100.100.160/28
[Web Network]=10.0.0.0/24
[VM4 floating IP]=10.100.100.163
[VM4 private IP]=10.0.0.6
As you can see in the diagram, routing consumes an additional IP from
the public range per compute node (e.g. 10.100.100.164).

The reverse flow will go in the same route, the fg-<portid> act as
a proxy ARP for the DVR namespace.

In the next post, I will go into the North-South scenario using Shared IP
(SNAT).

Please feel free to leave comments, questions and corrections.

3 Openstack Neutron Distributed

Virtual Router (DVR) - Part 3 of 3
In this post, the last of a 3-post series about Openstack Juno DVR, I go
into the North-South SNAT scenario in details.
Up until Juno, all L3 traffic was sent through the network node. In
Juno, DVR was introduced to distribute the load from the network node
onto the compute nodes.

The L3 networking in Neutron is divided into 3 main services:

1. East-West communication: IP traffic between VMs in the data
center
2. Floating IP (aka DNAT): The ability to provide a public IP to a
VM, making it directly accessible from public networks (i.e. internet)
3. Shared IP (aka SNAT): The ability to provide public network
access to VMs in the data center using a shared (public) IP address
In my previous posts, I covered how DVR distributes the the East-West L3
traffic and the DNAT North-South.
In this post I finish covering the North-South traffic with the Shared IP
(SNAT).
SNAT Shared Gateway North-South
The SNAT North-South shared public gateway functionality was
not distributed by DVR. It remains centralized on the Network Node, as
you can see in the diagram below.

In order to configure the SNAT namespaces on the Network node, the

SNAT-DVR L3 agent is deployed.

An additional private address is assigned in the SNAT namespace for each

of the DVR connected subnets, thus providing the centralized SNAT
functionality.

Lets follow a communication flow from VM3 in the App network to the
public network.
1. The packet is forwarded to the DVR namespace via the local
DVR default gateway port
2. If the packet destination is not private, then it is routed to the
appropriate default public gateway port (based on the source
subnet) on the SNAT namespace
3. An OVS flow converts the local VLAN into the App network
segmentation ID
4. The packet is forwarded into the SNAT namespace and NAT-
ed using the SNAT public address
Let's look at a concrete example:
[Public Network range]=10.100.100.160/28
[App Network]=192.168.200.0/24
[Shared Public IP (SNAT)]=10.100.100.162
[VM3 private IP]=192.168.200.6

In my next post I will summarize the DVR solution in Openstack Juno.

Questions and comments are welcome.