Download Checkmk free

Quickly gain a complete view of your IT infrastructure, no matter

complex.

Checkmk Do

VMware » Hardware Storage » Software/OS VMware Info » Hosted@KensyCloud

Multi-tenant/VLANs behind a virtualized pfS

firewall in ESXi
Home Networking Multi-tenant/VLANs behind a virtualized pfSense firewall in ESXi

About Me

Posted By Jon on Nov 26, 2015 | 15 comments

 Facebook 0  Twitter 6  Google+ 1  reddit 65

 LinkedIn 1
most intere
storage, an
hardware.
leveraging
cloud tech
Amazon AW
and vReali
Free AI Video Creator Automatio
When not
technology
performan
Discover our AI video creator with real actors. Create videos with photo
in less than 5 minutes checking o

Colossyan Creator See More

I struggled to title this blog post because the purpose of this implementation for me was to modify my existing
vSphere environment so that I could host VMs in their own segregated network. Really, you don’t need that
requirement for this to be useful – this post also documents how to segment your network with VLANs while
using a virtual pfSense firewall. I found it confusing enough that I thought it’d make for a good blog post
regarding a VM of pfSense and VLANs. Certified P
Issuer: VM
Why bother? PROVIDE

I wanted to be able to spin up a small Linux VM and let a friend remote into it to test an application but I didn’t
want him being able to run free on my network. There’s many ways to accomplish this, so don’t take my
instructions below as the sole method. Because I run a virtual pfSense firewall, the networking portion of this is a
little more confusing than if the firewall were physical.
Originally, I had this concept working by simply creating an additional vSwitch on my ESXi host. I then added a
NIC to my pfSense VM and connected it to a port group on the newly created vSwitch. After creating VMs and
putting them in this new port group, I was able to control access to and from this new interface in pfSense.
However, if you read the Configuration Maximums documentation for vSphere 6.0 you’ll find that there is a 10 NIC
limit per VM so this method will not scale well at all. So, while this would work, I could potentially run into issues
with total NIC counts on the pfSense VM and, even worse, each time I added a NIC I had to reboot the firewall
because FreeBSD does not support hot-add of NICs yet. I don’t suspect needing that many NICs/networks on my
firewall, but it’s not best-practice if we’re working toward real-world concepts.
Tags
Working at a cloud hosting company, I am very familiar with creating a multi-tenant setup in vSphere. However,
the configurations I am accustomed to always involve a physical firewall, router, or Layer 3 switch at the top of the
bash Cloud Co
infrastructure making it more straightforward for me to understand. Consider the following basic diagram:
Dell Dell R
ESXi 6.0
Intel LogInsig
Networking NS
PowerConnect
Replication S
Supermicro S
Preview ubuntu
6.5 vmotion v
vRealize vs
vSphere 6.5
Windows S

The above diagram is pretty conventional – all trunk lines will carry the necessary VLANs into the hosts; just
substitute your favorite hardware, maybe add a secondary firewall, maybe put them on their own switch stack,
maybe not, you get the idea. From there, a port group will be created on the vSwitch with the associated uplinks
and that port group will be tagged for the appropriate VLAN. The VLAN will traverse from the firewall all the way
down to the VM within the host. Access to and from the VLAN will be controlled by the firewall as a sub-interface
and every VM on that VLAN/port group will use the sub-interface as its gateway. But, what happens when you
move pfSense inside the a host as a VM? Well, functionally, not much – but, conceptually it is a little confusing at
first (or was to me). I’ll post a diagram I came up with to best describe what’s going on in my working setup:
As you can see, the diagram above is a bit busier. That’s mainly because I decided to include more detail in order
to better explain the solution. Starting at the top, you’ll find my internet connection (Verizon Fios) lands on my Dell
PowerConnect 5448. I have the Fios ONT box land on the switch in a small 3-port VLAN because I want the ability
to fail the firewall over to another host if I have to – having the switch involved means I don’t have to have the
connection from the ONT plugged directly into a single host. We need to create a VLAN and place a few ports on
it because we don’t want the DHCP from Verizon Fios being available to any old device on the switch. Note that
the WAN VLAN does not leave the Dell PowerConnect switch. You’ll see above that vmnic6 on my ESXi host
connected to the switch as part of the small VLAN. The PowerConnect 5448 also has all of the VLANs created on
it that I am planning to use. To start, I’ve created VLAN2, VLAN51, and VLAN200.

Let’s make it work!

The ESXi host connects to the switch over two physical NICs (vmnic0 and vmnic4). The ports that these two
interfaces connect to are configured as trunk ports and have VLANs 2, 51, and 200 added to the trunk. Because
the pfSense VM has a NIC on vSwitch0, which links to the PowerConnect switch via vmnic0/4, all you need to do
is create port groups tagged with the VLANs created on the switch and then create a sub-interface in pfSense
tagged for the same VLANs and you’ll be ready to go. Below are screenshots showing that I’ve labeled interfaces
g3 and g4 on my PowerConnect switch for the associated vmnics along with the VLAN configuration :
It’s always a good idea to label your ports with descriptions so that you can configure them later. With Cisco
switches you can enable CDP and then identify which vmnic goes to which port of your switch. With Dell you can
enable LLDP but I don’t currently have it enabled. In order to identify the ports you can use the show bridge
address-table command on a PowerConnect switch to show the MAC address table and correlate the MACs of
your ESXi host to the ports on your switch. If you have two NICs in an active/active configuration you may need
to set one NIC to “standby” or “unused” for the remaining MAC to show up. The MAC table will only show
information once traffic is initiated and the switch learns the MAC.
There’s on “gotcha” that threw me for a second. Your pfSense is going to be straddling two port groups up front,
right? If you’ve already deployed pfSense it should have a NIC on your default VM port group and your WAN port
group, as shown below:

pfSense NIC settings

Please ignore Network Adapter 3 above as that was for testing and I haven’t rebooted the VM to remove it yet.
That said, here’s where the visualization aspect gets tricky. If the pfSense inside interface is on the “LAN” port
group, and that port group isn’t tagged for any VLANs (or if it is tagged for a single VLAN) then how will pfSense
know about the other VLANs we want to create? Remember, we are trying to not add more NICs to pfSense, so
how do we get the single “LAN” NIC to see more than just the default (VLAN1) VLAN? Glad you asked! See the
screenshot below:
 vSwitch port group set up as a “trunk”

Remember earlier when we focused on how a physical setup would trunk the VLANs all the way down to the
VMs? Well, we still need to do that – except we can’t make a “trunk” port going into a vSwitch in the conventional
sense. Instead, we can tag the port group that a VM is a member of for “All (4095)” VLANs – in doing this, the
pfSense will receive any packets upstream that are tagged for any VLAN. This step is crucial as it basically
“trunks” all VLANs through to the pfSense VM “LAN” NIC. Something worth noting is that “All (4095)” VLANs
include the default VLAN1/untagged traffic. I have a bunch of VMs already on the “LAN” portgroup when it did
not have a VLAN ID set. Setting it the “All (4095)” did not disturb the VMs already there.
Once you’ve got your trunk interfaces configured all the way to your ESXi host(s) and have your pfSense inside
interface port group setup for all VLANs, you’re ready to create your new port groups. It’s important to know
where to create the port groups or this won’t work. If you have more than one vSwitch with uplinks to your
physical switch, make sure you’re using the right vmnics/vSwitch for this. To create the port groups on vSwitch0,
just open your vSphere Web Client, select your host, go to Manage, then the Networking tab, Virtual Switches,
and click the symbol for “Add Host Networking” – when the window opens, select the bottom option for creating
“Virtual Machine Port Group for a Standard Switch“:

Adding Port Group to Standard Switch

On the next window you’ll simply select the existing standard vSwitch (vSwitch0 in this case):
 Select the existing vSwitch

Then, when prompted to create the new Port Group, just give it a name that makes sense (you’ll be selecting this
“network” when you configure VMs you wish to associate) and give it the proper VLAN ID that you’ve already
created:

Creating new Port Group with VLAN

Once you’ve named and set the VLAN ID for the new port group, click next and Finish and you’ll have your new
port group waiting for you. Sticking with my VLANs of 51 and 200, see the screenshot below:
 Host Port Groups with VLANs

Above you’ll see that I created a “Conway DR” and “HPS Network” port group each tagged for VLAN 51 and 200,
respectively.
The next step is to tell pfSense what the heck to do with traffic that is tagged for these VLANs. To do that, we’ll
log into our pfSense GUI. Once logged in, we’ll hover over Interfaces and select (assign):

pfSense interfaces

If you refer back to the second diagram I posted in this entry you’ll note that the diagram shows pfSense with
vmx0 and vmx1 interfaces. Specifically, it shows that pfSense has vmx1 on the “LAN” port group. I made it easy
in pfSense by labeling vmx1 as “LAN” but you may not have done that – either way it’s important to know which
interface corresponds to the port group that was set for the VLAN ID of “All (4095)“. Once you’ve identified that,
click the VLANs tab:

pfSense Interface VLANs

You can see above that I already have my two VLANs created and they are associated with the vmx1 interface. In
order to create that, you simply click the “+” symbol at the bottom of the window which will open a window where
you specify the interface to associate the VLAN with (important!), the VLAN ID, and a description (note, VLAN 999
does not exist on my switch this is just a demonstration):
 Create VLAN within pfSense

Once you click save, you’re done. You’ll see the VLAN is created but we’re not done yet. We have to go back to
the Interface Assignments tab so that we can create a sub-interface from the newly created/associated VLAN:

Create New Sub-Interface

As above, click the drop-down, select your new VLAN you created, and hit the “+” symbol on the right. The result
will be a new interface with the VLAN associated with it, called Optx:

Sub-Interface Added

Once you see the new interface, simply click on it (it’s a URL), and enable the interface, set the name of the
interface, and create a static IPv4 address/subnet for this interface to use:
 New interface enabled

That’s it! Now, you have a port group associated with VLAN 999, a sub-interface in pfSense associated with VLAN
999, and when you hover over Firewall and then select Rules you will see the interface as a tab and you can
create rules! You can also setup a DHCP server for this new interface if you choose.
From here, you’d simply create a new VM and when you go to configure the NIC you will see your newly created
port group in the drop down – simply select the port group you want, IP the VM (or let DHCP do it), and you’re set.
Remember that the gateway for the VM will be the sub-interface IP we set in the screenshot above (10.0.99.1 in
the example).
Going forward, you could create a new port group/sub-interface for each “client” or network you want to isolate
(remembering to trunk the VLAN from the switch all the way into the pfSense VM) and create rules as needed.
Or, you could create a “client network” and simply drop whatever VMs are not part of your
personal/management network into that port group and IP them accordingly. There’s potential for the VMs to see
each other this way, but the access rules can be configured so that they won’t be able to see anything outside of
their own VLAN. If you were a large cloud host you could create these many “client networks” with 24-bit masks
so as to keep broadcast domains small, for example.
The whole point of doing it this way is not only to make it easier to manage from a vSphere perspective, but so to
also not run up against the NIC limitation at the VM level. And, as mentioned before, adding/removing NICs from
a FreeBSD VM will require the VM be rebooted for the changes to take effect. The “sub-interface” method is
much more production-friendly and most similarly matches the first diagram in this blog entry dealing with
conventional physical network equipment.
I hope you found this entry useful – let me know if anything needs clarification or is incorrect! One last thing I
wanted to add – I did this configuration 100% remote. Everything from identifying the ports on the switch to
making them trunks, to adding sub-interfaces is all non-invasive (if you do it right). If you think ahead, you can
move everything around to a properly secured/zoned VLAN setup without disturbing any network connectivity
already in place!
 Author: Jon

Share This Post On

15 Comments
CNC kantbank machine
MAY 6, 2019

you have got an incredible weblog here! would you like to make some invite posts on my weblog?

Post a Reply

Mark
DECEMBER 31, 2017

Happy New Year Jon!

Thank you so much for sharing your wealth of knowledge on how to configure a pFSENSE VM on
vSphere. Using the information in your blog, I am trying to configure a pFS to act as an router/firewall
for my single server home vsphere lab. I want to simulate a multi-datacenter network representing 2
physical sites. Is something like this even possible on a single host or I would need to have a second
host, whereby each host has its own pFS appliance and I simply connect the two WAN ports of the
routers together with the same IP address range.
Could do with your input.

Post a Reply

Mark
AUGUST 11, 2017

Great article are you still using PF

Post a Reply

Jon
AUGUST 14, 2017

Thanks! Yes, still using pfSense. It’s been through many upgrades but this site is still
served behind one!

Post a Reply

Mark
AUGUST 14, 2017

I am a huge PF fan however I am in the process of moving my lab network

over to a Edge Router Lite. Still having seconds thoughts since PF just works.
 Post a Reply

Jon
AUGUST 15, 2017

I’ve researched the ERL when making recommendations to

friends and colleagues. The problem I have with it is that it’s
really just a router. It can do some but not all of what pfSense
does. Because I run my pfSense as a VM on VMware hosts with
10 Gbps interfaces, I get very fast routing/firewalling that I can’t
afford to recreate on hardware.

Post a Reply

yes-sir
JUNE 14, 2017

Hi,
Great article.
We have a similar setup and we tried following along with your example. However, we’ve ran into
problems whereby the protected machine and pfSense cannot see each other. We’ve posted a
detailed description on Stack Overflow (Networking) where if you can you could give feedback on
what we could have missed. Thanks.
https://networkengineering.stackexchange.com/questions/41910/pfsense-vmware-esxi-vlan-
integration

Post a Reply

manolis manolidis
MAY 30, 2017

great article

Post a Reply

Don
FEBRUARY 8, 2017

Excellent write-up. I’m considering a similar setup but perhaps eliminating vSwitch1 and moving
vmnic06 to vSwitch0, essentially creating a 3-port trunk. I would then define a new VLAN for WAN
traffic, say WLAN VLAN ID:300, and add a corresponding port group to vSwitch0 in ESXi. The pfSense
VM would next be added to this port group to receive WAN traffic on vmx0 (i.e., with ESXi untagging
WAN traffic as it enters the VM through this interface and tagging traffic as it leaves). The LAN (VLAN
ID:All) port group would also be connected to the pfSense VM as vmx1 and then sub-interfaced in
pfSense just as you have done.
I suppose this is really just a trade-off between physically separating WAN traffic on a dedicated NIC
(i.e., more security but with potentially a single physical point of failure) versus combining WAN traffic
on the trunk (i.e., theoretically less security but with better physical redundancy by pooling the NICs).
Thoughts?

Post a Reply
 Jon
FEBRUARY 8, 2017

Hey Don – the only problem with a 3-port uplinked vSwitch is that it is expected that any
portgroup (and thus VLAN) defined on the vSwitch is available to all uplinks unless you
override the teaming/failover policy for the portgroup. That said, there’s nothing keeping
you from putting VLAN 300 on the uplinks as part of a trunk, and landing your WAN on
an access port on the physical switch in that same VLAN. Not sure if you get that, but
basically, you don’t have to have a dedicated NIC for uplink to WAN, you can just trunk it
as you mentioned. Personally, it’s all 802.1q and secure – there haven’t been instance of
VLAN hopping outside of very specific situations so I think you’d be fine either way.
Personally, I prefer to not mess with the teaming/failover overrides if possible! Good
thinking!

Post a Reply

Bugatti
DECEMBER 3, 2016

Nice write up. I came across your article while trying to figure out how to set my stuff up. Would
appreciate any feedback as I am stuck ATM. I have 2 dell r610 running some internal VMs for my
house and also running some external web services. I am trying to put together a more extensive lab
inside this environment. I currently have 6 esxi 6 nested hosts that will serve 3 separate vCenters so I
can test NSX. my physical network is a flat network with a 3560g cisco switch. I know I can set vlans in
it but would like to avoid it if at all possible. what I want to do is setup PFsense as a VM internal router
to my nested esxi hosts. I have created it with 5 nics, 1 is a san nic, 1 is a mgmt nic, seperate network
from my internal home network, 1 will be a vmotion network and the last 2 will be used for the NSX
network across all hosts that will manage the VXLANs. The issue I am having is when I go in and set
the mgmt lan in pfsense to vlan 10 and allow traffic, then set my mgmt nic on my esxi host to vlan 10
and set the port group on the physical esxi host to vlan 10 and then set my windows server nic to vlan
10 none of them can communicate. DO I really need to set the port on the physical cisco 3560g to
vlan 10 for all of that to work and can the pfsense control all the vlan tagging only for nested
environment. Thanks for any help anyone can add

Post a Reply

Salman
JANUARY 22, 2017

Bugatti, any luck … i’m also on the same track to achieve it!

Post a Reply

Jon
JANUARY 24, 2017

So the problem you’re having is that your switch needs to support this! The way it works
is that your switch is still used even though pfSense is a VM with virtual interfaces. If you
set your switch ports to trunk, all all vlans, native vlan 1, this will work. Also remember,
you can’t double tag. So when you’re creating a port group setup for VLAN 10 then
everything in that port groups is being tagged for VLAN 10. So don’t tag within the
Windows guest. If everything is VLAN 10 then the switch port can be tagged for VLAN 10
and everything else untagged. A diagram would be helpful – feel free to post a link. Sorry
it took me a while to reply. Your best bet to keep things flexible is to trunk.

Post a Reply
 Fracnk
SEPTEMBER 27, 2016

I’m thinking about a similar setup but with the particularity that each VLAN will host the exact same
VMs and IP settings.
The goal is to have an identical development environment for our developers that we could clone
when a new person starts. We will then implement PAT on the pfsense to give access to each
developer to the VMs. Developer 1 will connect using port 3391, dev2 port 3392 etc…
I haven’t tried yet but do you foresee some issues in having that sort of setup where the IP settings
are always the same but we isolate the traffic in using VLANs?

Post a Reply

Ben
SEPTEMBER 24, 2016

Great read! I’m doing a similar setup!

Post a Reply

Trackbacks/Pingbacks
In Other BSDs for 2015/11/28 – DragonFly BSD Digest - […] Multi-tenant/VLANs behind a virtualized pfSense firewall in ESXi. (via)
[…]

Submit a Comment
Your email address will not be published. Required fields are marked *

Comment *

Name *

Email *

Website

Save my name, email, and website in this browser for the next time I comment.
Notify me of follow-up comments by email.

Notify me of new posts by email.

Submit Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.
0 Follows

YouTube
Jon
Followers

Home Hardware Networking Software/OS Storage VMware Hosted by KensyCloud

POGA Lux
World‘s #1 leading gaming case. The luxury home for your c
go

INDIGAMING

Designed by Elegant Theme