vCloud Director Networking for Dummies

BY MASSIMO, ON SEPTEMBER 14TH, 2010

During the Beta phase of vCloud Director (aka Redwood) I put together a small deck called “Redwood Networking for Dummies”. I have received a number of positive feedbacks so I decided to turn that document into a blog post. Networking in vCloud Director is certainly a controversial matter. I believe it is fair to describe it both complex and rich at the same time. There have been many attempts lately to describe it from the like of Duncan Epping and Hany Michael on their own Blogs. They have done a great job in getting into the details. However I‟d like to try to give a different perspective on the subject. While I won‟t be able to avoid all of the technicalities, I‟d like to give you a sense of the philosophy behind what we have built into the product. Last but not least note that there are a couple of approaches to describe networking in vCloud Director. The first approach starts with the cloud end-user in mind and describes how networking works in support of certain application deployments use cases. From there you can walk all the way down to describe what happens at the vSphere platform level. The second approach starts with the vSphere administrator in mind and describes how networking works building up from the vSphere constructs, all the way to what gets exposed to the cloud end-user. In this post I am going to use the second approach. This is not because I believe it is the right one but simply because it is the one I am more comfortable with and the one that may serve better the readers of this blog. So let‟s get started. Introduction to vCloud Director Networking Before we get into the matter, you need to step back and think about the vCloud Director philosophy for a moment. Cloud is all about giving the end-users an unprecedented level of flexibility that allows them to do things that were only available to vSphere administrators before. In a way you can think of vCloud Director as an interface (or a proxy) into the virtual infrastructure. This allows vSphere administrators to give end-users a lot more flexibility, but at the very same time it allows them to keep full control of what end-users can do. Achieving this level of cooperation and flexibility in the networking subsystem is no trivial task. Think about how it is difficult to implement something that allows an end-user to create, in self-service mode, separate layer 2 network segments, define custom layer 3 IP policies, configure services such as DHCP, NAT and Firewall… all without having to ask the vSphere / cloud administrator to do that for you, all without messing up with the cloud-wide setup, all without causing conflicts with the other tenants on the cloud. This is a titanic effort, believe me. Explaining how networking in vCloud Director (vCD from now on) works is really like pealing an onion. If I was to explain it with the cloud end-user in mind I‟d start from the outer part getting into the middle of it. Since I am going to explain it from the vSphere administrator point of view, I will have to start from the inner part of the onion building up the abstraction levels that the end-user will see in the end. This document will try to explain the three major networking levels within vCD. They are External Networks, Organization Networks and vApp Networks. These are in fact the type of networks you can instantiate. Before we start discussing these three network layers we need to introduce another concept that is of paramount importance for vCloud Director operations: Network Pools. Think about it for a minute. How can we give an end-user a controlled way to deploy layer 2 networks? Layer 2 networks are usually vSphere PortGroups with an associated VLAN. How can you keep control of that? How can you let different tenants deploy these PortGroups keeping track of what‟s going on in the cloud (and in turn on the vSphere layer) and, in doing so, avoiding conflicts with similar

External Networks The vCD inner networking component is called External Networks. That‟s in fact what you do when you create a vCD External Network: you point to an existing vSphere PortGroup. As the word implies. This is a picture that shows what it is. he/she would chose an available VLAN and would create a PortGroup based on that. a DNS address and a pool of IP addresses. that CMDB. I usually suggest to name the vCD External Network after the vSphere PortGroup for ease of tracking. a default gateway. It‟s easy: One of the most confusing points about the creation of the External Network is that the wizard asks for some layer 3 configuration parameters. If the enduser can deploy layer 2 networks there must be a CMDB that can be programmatically be accessed under the covers (by the end-user). More on this later. For naming convention you will be calling this External Network something like Internet or ExtNet-Internet. A Network Pool is in fact a set of layer 2 networks that the cloud administrator has declared as “available”. Essentially you are telling vCloud Director that there is a PortGroup that is able to provide external connectivity to your cloud environment. these are networks that are managed by someone that is typically external to the vCD environment and are identified by a vSphere PortGroup. we are just “registering” these networks into vCD. In a cloud self-service model it doesn‟t work like this. Yes Network Pools are.deployments in other tenants (aka Organizations)? vCloud Director solves this problem using what we call Network Pools. He/she would then connect the vNIC of this end-user‟s VM to the newly created PortGroup and advise the end-user that the change was made to the VM. The typical example is a PortGroup with VLAN 233 (for instance) which can support native Internet traffic. If you want your Organization (and in turns your vApps) to have connectivity to the external world you need to have External Networks. In particular the wizard asks for a subnet mask. in a way. Think about it: in the old days when the end-user needed something like this he/she had to go to the vSphere admin which would in turns look into his/her VLAN CMDB (typically an EXCEL spreadsheet ). What are these parameters for? Well. What we are doing while filling this . remember that we said External Networks are networks that are built and maintained by an external entity.

Each tenant can have three type of networks configured as you can depict from the picture below (you may not immediately get some of the acronyms and colored labels – no worries – it will be more clear later): . For this you need to be patient.e. In particular the IP pool that you need to configure is a pool of IP addresses that vCD will use to distribute IP addresses (and related layer 3 info) to VMs connecting to this network. So how do you fill that pool? You have to turn to the folks that administer that specific network segment and you need to ask them something like “can you reserve me a set of IP addresses that no one else will be using on your network and that I can dedicate to the vApps I will be instantiating on vCD?”. In the previous section we have created cloud-wide external connectivity (i. External Networks).wizard is essentially telling vCD what layer 3 information to use when VMs will be connected to this network directly. In other words. Cloud end-users are defined inside each Organization. With Organization Networks things start to become more “interesting”. We will get there. Now we are zooming inside an Organization. how would you be able to instantiate vApps directly onto that network if you don‟t know which IP address to use? That‟s what that Static IP pool is. Organization Networks External Networks are easy. An Organization (or Org) is a logical construct within vCD that describes a tenant or a customer. This doesn‟t tell the whole story on how VMs get their IPs when deployed.

Note that in this case you are still asked for those layer 3 IP information. this is the layer 2 dedicated segment that will support my Org Network. In this case nothing happens at the vSphere layer. So how is this implemented at the vSphere layer? When you create such a network a good learning exercise is to switch to the vSphere client interface. This network really represents a dedicated layer 2 segment that has its own private IP schema that the Organization can chose arbitrarily (for example 192.x. There you will see a number of things happening: first a new PortGroup is deployed on-the-fly.x). The second network is called External Organization Network (NAT / Routed). however this time you can create them based on your specific needs because this segment is private and its layer 3 info are not going to overlap nor to be shared with anything else anyway.168. this type of Org Network is a logical construct created inside vCD but doesn‟t really have any counterpart in the vSphere world: if you connect a vApp to this Org Network. Not a big deal. This private network is then routed to the External Network I have chosen to route to. Consequently a new vShield Edge appliance is automatically deployed by vShield . the vNIC gets configured to connect to the Internet PortGroup (VLAN 233) in the example above.The first network is called External Organization Network (Direct Connect) and it‟s the simplest way to connect to the external world through an External Network.

The Edge device is then configured with the appropriate layer 3 info you have filled in the wizard when creating this Organization Network. vApps that are deployed to connect to this network cannot go outside through the External Network. Remember that Edge also provides DHCP service to that segment so that‟s why Edge is optionally used if the Organization Administrator decides to enable DHCP on that segment (note DHCP is disabled by default). They come from the Network Pools we have briefly mentioned above. The third network that it is possible to created within an Organization is called Internal Organization Network. This appliance is effectively the routing device connecting your dedicated layer 2 network to the External Network (see picture). At this point you may wonder how vCD and vSphere can “deploy a new PortGroup on the fly” to back this dedicated layer 2 network we need to create. So. As the name implies this network is only available internally to the Organization. in summary.Manager. At this point you may wonder why there is an Edge deployed onto that PortGroup since there is no need to do routing. Firewall and DHCP functionalities to protect and serve this dedicated layer 2 segment. When creating this network in fact the wizard will ask you for the layer 3 private schema as well as the Network Pool where to grab an available layer 2 network (think of it as an available VLAN for the moment). this is how you can connect your VM vNIC: . The Edge license provided with vCD supports NAT. In fact this type of network is similar to the External Organization Network (NAT / Routed) with the only exception that it doesn‟t connect to the external world.

This is something that you may want to do to either create and support secure n-tier applications deployments or to fence a vApp to an Organization Network. Organizationwise networks (aka Organization Networks) and now we are going to investigate what we call vApp Networks which are. Also consider that vCD treats everything as a vApp. Which brings us to the third type of network. guess what. Sometimes you will be using more VMs in a single vApp. In a way. However it is possible for a VM to have two or more vNICs. fencing is a shortcut given to end-users in the vCD user .Note that. Fencing a vApp allows you to instantiate many times the same vApp onto an Organization Network preserving layer 2 and layer 3 information. for simplicity. A single VM is in fact a vApp with one VM in it. vApp Networks So far we have seen cloud-wide networks (aka External Networks). the picture shows a VM that can connect to different Organization Networks. Most of the time VMs will have only one vNIC connected to either one of the Org Networks. networks that are only available within a single vApp.

This sounds familiar if you think at the different Organization Network options. the vApp Network can connect using NAT technologies to the Org Network (in which case a new layer 2 network is being deployed from a specified Network Pool and a new Edge is instantiated to connect to the Organization Network) or the vApp Network can be left isolated from the rest of the world (in which case a new layer 2 network is being deployed from a specified Network Pool and a new Edge is instantiated only if DHCP gets enabled). That‟s because. right now.interface to achieve transparently this cloning operation. From a vSphere perspective creating a vApp Network explicitly or taking the “fence shortcut” in the UI translates into the deployment of Edge devices as well as separate layer 2 networks from a vCD pre-defined Network Pool. While the configuration wizards may seem to be slightly different. As a matter of fact we are effectively creating a similar stack at the vApp level and we could then plug this stack on top of the other stacks we created at the Org level. note that the relationship between vApp Networks and Organization Networks is somewhat similar compared to the relationship between Organization Networks and External Networks. I am focusing more on what happens at the vSphere layer rather than focusing on the different end-user options vApp Networks and Fenced vApps have to offer. You remember the onion? The picture below shows a VM that connects to a vApp Network where DHCP was enabled (note the presence of the Edge device): . Note I am oversimplifying a matter that is more complex than what I am trying to picture. By this I mean that vApp Networks can connect directly to an Org Network (in which case the VM connects to the Organization Network PortGroup).

shows a VM connected to a vApp Network with external connectivity. Again: only one of these three connections can be active at any point in time. In fact the Edge system vm always have a maximum of two vNICs: one that connects to the network to be protected and the other one connected to the network it needs to route/NAT too.This picture below. External Org Net (NAT/Routed) and Internal Org Net. Don‟t be confused by this picture: an Edge can Route/NAT to one and only one network at any point in time. on the other hand. . The picture below shows all the possible configurations for the second Edge vNIC: External Org Net (Direct Connect). Note how a VM can be potentially NATted twice if connected to a NATted vApp Networks which in turns connect to a NATted Organization Network.

This is helpful to set limits on what an Organization can do (and possibly charge accordingly). This also comes handy to keep control and keep track of layer 2 network usage. Org Networks are created by the cloud administrator (on behalf of the Organization administrator) and when the cloud admin starts the creation process the wizard asks interactively for “which Network Pool to use to grab an available layer 2 segment”. you don‟t want someone creating 100 vApp Networks in half a day and consume the entire Network Pool immediately. When you associated a Network Pool to an Organization vDC you can set a limit on the number of segments any user in that organization can grab.It may be interesting to call out that there are a few philosophical differences between how you create. In this case when a user creates a vApp Network a layer 2 network is grabbed from the Network Pool associated to the Organization vDC the user is deploying the vApp to. . After all this end-user may not even have a clue what a Network Pool is and perhaps it may not even know what a layer 2 network is. In fact if you associate a Network Pool with 100 networks in it. configure and deploy vApp Networks compared to Organization Networks. To overcome this we associate a network pool to the Organization vDC. We do not want to expose that question to the end-user when he/she creates a vApp Network. I am not going to cover use cases of where and how to use combinations of vApp and Org Networks to create secure deployments because in this post I wanted to give you more the sense of what happens from a vSphere and cloud administrator perspective rather than from a cloud end-user perspective.

We use a technique called Mac-in-Mac to create layer 2 separated networks without using VLANs. VLAN-backed Network Pools: this is the easiest to get. This is extremely useful for big environments where VLAN management is problematic. You can. And so on. for example. PortGroup-backed Network Pools: it is similar to the VLAN-backed. Now the question is: what happens when you connect a vNIC to either an Organization Network or a vApp Network? How do you control the layer 3 behavior? As we said. vCD will tell vCenter “please create on the fly a PortGroup. In the example below I have connected it to a vApp Network as you can depict from the name (vAppInternal). Yeah that‟s right. Now the acronym VCD-NI and the labels Preprovisioned and Created-on-the-fly in the pictures above should make more sense to you. and give it VLAN ID 100″. If you chose to connect it to a network you have three choices on how to get an IP. This is correct but it doesn‟t tell the whole story. Virtual Machines IP management First of all note you cannot connect a vNIC to an External Network directly. Try to go back and have a look at them again. you have a choice of connecting each vNIC of the VM to an Organization Network. a vApp Network or leave the vNIC not connected. create a Network Pool and give it a range of VLAN ID 100 to 199. You can however connect the vNIC to either an Organization Network or a vApp Network. and give it VLAN ID 101″. So far we kept referring to a “layer 2 segment” as a PortGroup with an associated VLAN id. either because there is a limited number of VLANs available or because keeping track of VLANs is a big management overhead (especially if you use an excel spreadsheet to do that ). The difference is that the PortGroups need to be pre-provisioned on the vSphere infrastructure and they need to be imported into vCloud Director. When you create such a Network Pool you only specify how many of these layer 2 networks you want this Network Pool to have and you are done. they are already there pre-provisioned.Network Pools At this point you may have an overall understanding of what a Network Pool is and why it is used. the corresponding VLAN ID gets put back into the pool of available networks to be deployed. This is the case when you use vSphere Standard Switches (as opposed to Distributed Switches) or when you use the Nexus 1000v (at the moment vCD cannot manipulate programmatically Port Profiles). In summary it is a small CMDB that contains layer 2 segments available to vCD administrators and end-users. Of course if one of these networks is destroyed during the lifecycle of the cloud. Why using this? Well there are some circumstances where vCenter cannot easily (programmatically) create PortGroups on the fly. Whenever you grab one of these IDs because you need to deploy a new layer 2 segment. When vCD starts to deploy PortGroups from this Network Pool you won‟t see any VLAN associated to them but they are indeed different layer 2 segments. The next time there is a need for another layer 2 segment vCD will tell vCenter “please create on the fly a PortGroup. Note Network Pools need to be created before we start deploying the actual networks we have described above (with the exception of the External Networks because they don‟t use Networks Pools). There are really three different type of Network Pools one can create. vCloud Director Network Isolation Network Pools: This is when things start to get interesting (again). See the “IP Mode” drop-down in the picture: . So vCD won‟t tell vCenter to create these on the fly.

an External Organization Network Routed/NAT or an Internal Organization Network. vCloud and the Meaning of being “Open” » 23 comments to vCloud Director Networking for Dummies  Diagram: VMware vCloud Director Networking Architecture | HYPERVIZOR September 14. It is important to understand that from a VM perspective this is considered a Static IP Address. UNCATEGORIZED « vCloud Director: Catalog Experiments vSphere. Massimo. 2010 at 6:13 PM · Reply . This is the private IP Pool range you had to configure when creating a vApp Network. It goes without saying that this manual IP address cannot fall into the same range of the DHCP scope nor the Static IP Pool if you want to avoid potential IP conflicts. If it‟s an External Organization Network Direct Connect it will have to be a DHCP that is available on that PortGroup associated to the External Network (in which case this would be out of the scope of vCloud Director). is directly proportional to the richness of configurations and options available to the cloud end-user to consume “self-service”.Static IP Pool: this is the pool of IP addresses that you have configured when you created the network you are connecting to. In that case the vNIC will search for a DHCP lease on the network it connects to. In case of an External Organization Network Direct Connect the IP Pool range configured when creating the External Network it connects to will be used. an External Organization Network Routed/NAT or an Internal Organization Network. If it‟s a vApp Network. in this case. DHCP: I guess this is self-explanatory. it only happens to come from a pool that vCD controls. I covered the three main network layers and I have then focused a bit on the concept of the Network Pools and how Virtual Machines can be configured to connect to the available networks inside the Organization (including vApp Networks). Static Manual: This is used in those situations where you do not want or cannot use either one of the two above. this will have to come from the Edge DHCP service. Conclusions In conclusion I hope I managed to give you a different perspective on how vCD networking works and especially the logic behind it. You have to manually enter the IP address into the vCD interface and make sure it is the same you have entered into the Guest OS of the VM you are working on. Remember that complexity. The first IP available in the Static IP Pool gets “plugged” into the VM (as a static address) at Guest Customization time.

vCloud Director.]  Naresh September 16. o Massimo September 17. Any help is appreciated. From a VM IP stack perspective using an IP from the static IP Pool turns into a static IP address at the VM level (vCD will chose it from the Pool).  sanjai September 17.[. If the vNIC connect to an External Org Network (NAT/Routed) or to an Internal Org Network then you need to turn on the DHCP service at the Edge device (or have a DHCP server on the same network as a vApp for that matter). Another MUST-READ. The only thing I can think of is that you created Network Pools off a vDS that is not available in the cluster where your Oragnization vDC (hence the Provider vDC) originates from.. 2010 at 9:37 AM · Reply Hi Naresh. even though there are networks available in the Network Pools at the cloud level. This may be the case if you have two vCenter servers driving different clusters and you created a Network Pools leveraging a switch on the first vCenter but the RP backing the Org vDC comes from a different vCenter.. HTH. I still can not differentiate between Static IP Pool and DHCP as both looks same in Virtual Machines IP management. [.. Posted in Diagrams. Please shed some light … o Massimo September 17. .] The networking part of the presentation I‟ve mentioned above has been rewritten by the master at this link. On the screen in the Org vDC propertiesm where you can select network pool. 2010 at 12:52 PM · Reply your presentation is great and detailed. If you select DHCP you are basically setting the VM to get a dynamic address on the Network. Massimo. 2010 at 9:27 AM · Reply Thanks for your effort in explaining the vCD Network concepts. I was trying to setup the cloud and i am stuck at a point where i am not able to allocate network pool to a Organization vDC. it is none and doesn‟t list the networks that were created at the cloud level. If the vNIC connects to an External Org Network (Direct Connect) the DHCP service needs to be available on the External Network. vCloud Director | Tags: diagrams.. 2010 at 9:44 AM · Reply Hi Sanjai.

.] – Networking part 1 – Intro Creating a VMware vCloud Director Cluster vCloud Director Architecture vCloud Director Networking for Dummies Automating vCloud Director and Oracle DB [.HTH.]  Eric September 30.The weblog of an IT pro specializing in virtualization.. I came back several times. Any way thanks for the help i have to overcome some more issues in setting this monster up  Brad Hedlund September 24. I have a pretty good understanding of vCD networking now. 2010 at 10:48 PM · Reply Great post Massimo! Your style is perfect for spoon feeding. I didn‟t had two vcenter server but i had to reconfigure my Vsphere environemnt and create a new cluster. 2010 at 2:16 PM · Reply Thanks Massimo .blog. 2011 at 11:12 AM · Reply Yes it did helped to understand the underlying concept. Sanjai  Naresh September 17. Unfortunately.  sanjai March 18. You helped me think differently and could resolve the issue and proceed further. Massimo. is it possible to configure 2 Vshield zones based on the same IP Subnet via VShield Manager ? . storage. thanks to you.scottlowe. 2010 at 7:48 PM · Reply Hi all I‟m a vShield beginner and I need some help. Cheers. For 2 given customers using an overlapping IP subnet to assign their VMs.org . 2010 at 3:57 PM · Reply [. Brad  A Collection of vCloud Director Links . i took a very long time to reply hope you don‟t mind.. and servers September 29.. I think the issue was that the resources were not available to all the ESX hosts.

. I am not sure what you are talking about. there is no single „edge‟ device like in your pictures. Massimo.  wrong. Who ever said you can only have a “single” edge per Org? Also “Isolated” may have different meanings to a lot of different people. this is not supported today. vShield Zones/App wouldn‟t work with overlapping IP schemas across different tenants. they are exactly the same as „external direct‟ . look for lab-manager encapsulation to see why it can not be isolated. meaning multiple „edge‟ devices for the same ORG (per port-group) . 2010 at 8:27 PM · Reply [.sorry November 20.How is the isolation performed between the 2 customers.]  wrong. I am glad it‟s so “clear” in your mind… however stating that an Internal network and External Direct networks are exactly the same… makes me wonder… Massimo.] Networking part 1 – IntroCreating a VMware vCloud Director ClustervCloud Director ArchitecturevCloud Director Networking for DummiesAutomating vCloud Director and Oracle DB InstallationDiagram: The VMware vCloud Director Cell [. Best Regards. you are making it difficult when it is actually very clear o Massimo November 22. „internal networks‟ are not internal at all . vshield-edge device (VM) is created for every port-group that use it ..  Want to know more about vCloud Director? « Trent Steele's Blog October 6.sorry November 20... just a vlan on a portgroup sent to external switch when moving between ESX servers. This is something we are looking at for future versions. 2010 at 9:31 AM · Reply Eric. 2010 at 9:54 PM · Reply this is all very wrong sorry . also „isolated‟ to define VCDNI is wrong . Eric o Massimo October 1. 2010 at 9:57 PM · Reply . 2010 at 12:41 PM · Reply I have approved the above comment but to be honest I am not sure whether it‟s spam or what. For someone not even a dedicated port on a shared switch is “isolated” (go figure VLANs or MAC-in-Mac).

It‟s rather in the integration of all of them (CPU/MEMORY/DISK/NETWORK) into something that an end-user can easily consume from a portal or from an API. so all NAT caveat apply here.  covertp April 24.  Alex May 27. Perhaps your 50$ supermarket firewall will have more features than the Edge today … but it doesn‟t fit into the picture above. Massimo.also – when you use vshiled edge in vcloud . Note that the External Network could also be an internal company-wide backbone where you want to attach Orgs and keep them protected (from other internal workloads) via the same Edge device. compare it to a 50$ cheep FW and tell who is better in terms of functionality and features o Massimo November 22. I lose connectivity… . 2011 at 4:53 AM · Reply Hi. and Cloud Isolated pools. 2011 at 4:22 PM · Reply It depends on the context. VSE is NOT a router. I still have a question about the network pools. the physical network switches on the network need to be configured for the relevant VLANs? I have tried both VLAN backed. and as soon as I put 2 VMs in the same org network on different ESX hosts. 2011 at 5:43 AM · Reply As an example of the information that you will need to fill out for an “External Network”–is this public IP info given to you by your ISP? o Massimo April 27. you are forced to NAT your servers . If you are an Enterprise deploying a Private Cloud and you want to give your Orgs access to the Internet (via the Edge security gateway) then yes those would be the public IPs your internet provider gave you (unless you have some sort of additional NATting at the perimiter of your physical network in which case it would be the NATted addresses that your perimeter physical device is exposing internally). very helpful article. Am I to assume that for any of the 3 types. also – VSE as a FW ? it is not even certified as a fire-wall so why call it so ? look for the capabilities of VSE. 2010 at 12:50 PM · Reply The features will improve for sure down the road but the value is not in the features themselves.

Thanks a lot! o Massimo June 23. Do I need a VLAN configured so that the port group on the dvswitch can be shared amongst the ESX hosts? Thanks!  Boris June 23. The port group gets created on the dvswitch.  Alex May 30. For the VCD-NI pool it depends if you specify a VLAN ID for the pool. thanks a lot! Helped me understand the inner workings a bit more. then yes that VLAN need to be defined on the ports. without a VLAN ID configured. 2011 at 3:44 PM · Reply Wow.com/2010/10/06/vcd-%E2%80%93-networking-part-3%E2%80%93-use-case-2/ . but I can‟t tell whether it‟s just me misunderstanding the product. I suppose. 2011 at 8:01 AM · Reply Hi Massimo. Thanks you! What about some post about how to set up properly organizations from scratch (including all the PortGroup and Network Pool configured)? It would be really helpful. I have a VCD-NI pool configured for 50 networks.o Massimo May 27. 2011 at 8:08 AM · Reply Alex.yellowbricks. If you do. for the VLAN backed and the PG-backed yes the VLANs need to be defined/preprovisioned on the switch ports (basically you need to create a trunk). Thanks for your response. Hope this helps. that‟s very helpful. but VMs on that internal org network cannot see each other unless they are residing on the same ESX host. 2011 at 4:57 PM · Reply See this (and the previous “parts” linked in it): http://www. Massimo. I‟m still having connectivity problems.

Sign up to vote on this title
UsefulNot useful